Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this submission, the author proposed a data augmentation method for ViT. Specifically, two samples will be mixed in a global level, where patches of an image will be replaced with the patches from another images at corresponding locations. Then, the labels are generated using both the attention map generated by the ViT and the binary mask to avoid the negative effect using only attention map in the beginning of training process. The experimental results have illustrated the effectiveness of the proposed method. The approach is further demonstrated in the downstream applications. 
strength_and_weaknesses: Pros:
1. The paper is well written and easy to follow.
2. The experiments are thorough and the results are pretty good.
3. The proposed method can be generalized to different visual tasks and boost performance.
Cons:
1. In the downstream tasks, seems like the proposed method is only employed for pretraining. Can it be used for directly finetuning for downstream applications?
2. Minor: in Generalization analysis: "Fig2(c) depicts their loss and top-1....." should be Fig2(c) and (d) depicts their loss and top-1...."
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. The authors have conducted plenty of experiments to demonstrate the effectiveness of the proposed method and the proposed approach can be reproduced based on the information given in the paper.
summary_of_the_review: To sum up, the paper is well written and easy to follow. It proposed a data augmentation method for ViT which can further boost the performance of ViT on various tasks. Experiments on different tasks and applications were conducted and the results were well analyzed. I have some questions which can be referred to in Strength And Weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new variant of CutMix, namely MixPro to further improve the previous state-of-the-art (TransMix) for Vision Transformers. The proposed method, including MaskMix and Progressive Attention Labeling, aims to tackle three primary drawbacks they found in TransMix. Extensive experiments show that MixPro can lead to consistent and remarkable improvement on various tasks and datasets.
strength_and_weaknesses: ### Strength

**1. The proposed method is simple yet effective, which makes it simple to follow.** MixPro basically follows the paradigm designed in TransMix, which only requires several lines of code to implement. Personally I like ideas that are simple and sweet. The major changes in MixPro, including MaskMix and Progressive Attention Labeling, are easy to be implemented in principle. However, the authors did not include the pseudo-code in their manuscript. It would be more clear if they can include the pseudo-code in the appendix.

**2. The proposed method exhibit strong and consistent improvement** for different tasks and models even when compared with a strong baseline TransMix.

**3. The proposed method is fast and applicable to many different Transformer-based networks.** As shown in Table 2, the proposed method can lead to a significant gain with minimal computational introduced.

**4. The ablation studies and experiments are comprehensive and extensive.** The comparison between MixPro and TransMix makes it clear to understand the efficacy of the proposed method.

### Weakness

**1. I am not fully persuaded by the whole story and the explanation given by the author in the introduction.** The authors argue that TransMix has three major drawbacks including region-based mixing, boundary artifact and unreliable attention matrix.
- 1.1 For region-based mixing, I am a bit confused about why *region-based mixed images may provide insufficient image contents* and why the proposed MaskMix can help to alleviate this problem as MaskMix will also mix just a portion of the entire image, which may be insufficient as well.
- 1.2 For boundary artifact, the authors argue that it will introduce *sharp rectangular borders that
are clearly distinguishable from the background.* However, the solution MaskMix will also lead to these sharp artifacts and the patch-wise mixture will spread this kind of noise everywhere in the mixed image, which may even aggravate such problem. Instead of echoing the argument by the authors, I would suggest that MaskMix may introduce more noisy patterns into the image mixture so that the network can be better regularized by these noises. 
- 1.3 For Progressive Attention Labeling (PAL), the authors argue that it is better than TransMix since *attention maps may not always be reliable during the training process.* However, the $\lambda$ of TransMix is actually the mean of attention weights and the original $\lambda$ introduced by CutMix (a.k.a. $\lambda_{area}$ in the manuscript). This indicates that TransMix also considers this problem and uses the area of the cropped image as an extra condition. Meanwhile, TransMix also evaluates the effect of using different attention maps in calculating $\lambda_{attn}$. However, using a highly-discriminative attention map pre-trained by DINO to calculate $\lambda_{attn}$ does not result in any improvement in terms of the top-1 accuracy on ImageNet (see in Table 6 in TransMix paper). So I would doubt whether the label noise introduced in the label space would also be helpful.

**2. The illustration for details of the proposed method is a bit unclear to me.** The authors did a good job in describing their overall pipeline and the basic idea. However, I am a bit confused about whether the introduced image will be first shuffled or not before mixing. The cat in Figure 2 does not follow the original geometric structure, but there seems to be no shuffling operation in Eq. (7).

**3. May the authors do more analysis on why and how MaskMix works better than TransMix?** My assumption is that **MixPro creates more complex scenarios in the image mixture such that the network can be better regularized**.  Thereby I am curious about whether introducing more random patterns will further improve the performance. These random patterns may include but not be limited to:
- random shuffling (or shifting) on the cropped patches
- random shuffling (or shifting) on the patches of the canvas image

**4. The proposed method is only applicable to Vision Transformers with class token.**
clarity,_quality,_novelty_and_reproducibility: I recognize the novelty and the efficacy of the proposed method. The paper is basically clear but lacks some implementation details. The reproducibility cannot be guaranteed since the authors do not promise to make their code publicly available.
summary_of_the_review: I hold a double feeling of this paper. On the one hand, MixPro demonstrates its strong performance on various tasks and datasets., meanwhile, I like ideas that are simple yet sweet. On the other hand, I cannot agree with the explanation stated in the introduction and related work. I would first leave a borderline here and see how the authors respond.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes MixPro, an improvement of TransMix, a mixup method designed for vision transformers where the label interpolation weights are determined by the attention maps. It contributes two ideas: 1) (MaskMix) in the input space, the mask is random over patches rather than rectangular (as in CutMix). 2) (Progressing Attention Labeling, PAL) The label interpolation weights evolve from area-based (as in CutMix) to attention-based (as in TransMix) as the confidence of the classifier increases during training, thus attention maps becoming more reliable.
strength_and_weaknesses: The paper is well written and a pleasure to read. The ideas are very simple and the two improvements make sense. There are extensive experiments and comparisons, mostly following TransMix.

I find the two improvements incremental. They are both of the kind one would expect to find between a conference paper and a journal submission, not in a new paper.

The most critical missing experiment is an independent evaluation of the two components. As it stands, it is impossible to assess the benefit of either idea (MaskMix and PAL). There is an ablation in Table 6 (Appendix) but only on ImageNet-1k accuracy, where the gains of the two ideas are only 0.3% and 0.4%. Independent evaluation should follow on all experiments.

Another limitation of the empirical study is the narrow focus on vision transformers. TransMix may have initiated this but there are many ways one may obtain an attention map from a convolutional network, so I see no reason for this narrow focus. Extending to convolutional networks will also allow a comparison with previously published results. Currently, the only comparison with methods prior to TransMix is on ImageNet-1k classification (Table 2). All other comparisons are limited to TransMix and the baseline, which is not sufficient.

MaskMix is motivated by the global receptive field of vision transformers, but I find this motivation insufficient. For one thing, region-level crops appear to be more difficult for the local receptive field of convolutional networks; for another, an independent evaluation of MaskMix and a study on convolutional networks could at least validate the claim.

Concerning PAL:
- The use of cosine similarity in (8) is not elegant or justified, since the two probability vectors are l1-normalized. More appropriate is the histogram intersection (which is to l1 norm what is dot product to l2 norm). Or any other similarity measure based on a proper distribution metric.
- The probability vector p of the interpolated input is not a good indicator of classifier confidence. More reliable are the probability vectors of the clean images, which could be compared with the clean labels (resulting in the predicted probability of the ground truth classes). I am not sure how this could be implemented, given that TransMix only processes mixed images. Maybe confidence could be estimated globally by considering some clean images per epoch. This could be at least a baseline for Table 8, even if it does not improve.
- Since patches are chosen at random from one or the other image, it is not clear if the attention map of a mixed image makes as much sense as that of a clean image or at least that of TransMix (CutMix) style mixing (by rectangular area). At least some visualization is in order. Another baseline is using always the area-based label interpolation as in CutMix.
- Of course, PAL should be evaluated independently of MaskMix (with rectangular area image mixing).

Another idea or baseline that relates to both improvements is to use attention maps not only for label but also for input image interpolation, as in SaliencyMix.

Other benchmarks that were considered in prior works and not in TransMix or this work include robustness to adversarial attacks, calibration and out-of-distribution detection on datasets other than ImageNet, where the network is trained. Such experiments are important not just for completeness but because they have margin for considerable gains for a new method. For example, the gain of MixPro over TransMix is in the order of 0.5% in most experiments. Given that MixPro consists of two ideas, the gain of each is even less.
clarity,_quality,_novelty_and_reproducibility: As discussed above, the paper is of high quality and the method very clear. However originality is limited to incremental ideas and small improvements. The paper relies too heavily on TransMix.
summary_of_the_review: Overall, this work proposes two simple and interesting ideas and is well implemented and well written, but given its limited novelty and limited benefit, one would expect as compensation at least a wider range of empirical studies that would allow a deeper understanding of the related methods. The current study is not mature enough and I am not sure if it can make it for this ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a data augmentation method for ViT, where two samples are mixed on a global level, by mixing patches between the images This is strongly related to previous data augmentation methods like cut-mix for  convolutional neural networks. The experimental results show that the proposed method is effective.

Overall, the proposed ideas are simple and interesting, the paper is well written. Yet, the novelty is limited and the results could have been validated on more datasets. Overall, the evaluation is however, still sufficient. 
justification_for_why_not_higher_score: The novelty of the proposed approach is somewhat limited. 
justification_for_why_not_lower_score: The proposed methods extends/specifies existing approaches to vision transformers and the achieved results are promising. The paper is well written.
Decision: Accept: poster