Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel data augmentation technique for visual transformers.
strengths_and_weaknesses: Strengths:

- Interesting and novel idea
- Faster performance
- SOTA results
- Comprehensive ablation studies

Weaknesses:

- It is not entirely clear what does it take and how easy it is to add this method to an arbitrary transformer. It would help to have an appendix with python code showing how a typical transformer is modified with the proposed algorithm. Is it as simple as adding a layer for which good hyperparameter values are known, or additional hyperparameter search is really necessary on case by case basis?
- The work explores 2 baseline transformer methods applied to 3 datasets. There are other models that are cited in the paper such as ViT-Lite, NesT-T, NesT-B. It would be of interest to have your work applied to those as well.
- I feel this method could have wider applicability, beyond Transformer based models (e.g. ResNets). It would be of interest to either see some empirical results exploring this area, or if this is impossible due to the limitations of the method, to have a discussion of this limitation and perhaps an outline of how this limitation could be overcome in future work.

questions: - Can you encapsulate your staff in a layer and show how to add it? The source code is based on a function. If you want your code to be used by others, having it as a standard module and a simple way to incorporate it in existing models would be a great plus to your work
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors proposed a novel saliency-aware token-level data augmentation method for assisting the training of vision transformers. Specifically, they proposed TokenMixup, which takes the self-attention as the inherent saliency detector, to mix intermediate token sets so that the saliency level of a batch is maximized. With TokenMixup, they achieve significant improvement on image classification tasks and meanwhile obtain higher efficiency than other gradient-based mixup approaches. 
strengths_and_weaknesses: Pros:

1. This paper proposes TokenMixup, a novel saliency-aware token-level mixup strategy for improving vision transformers. It ingeniously adopts self-attention, the basic module in transformers, to serve as a substitute for conventional computationally-heavy saliency detector. 

2. The authors conduct thorough experiments for evaluating the effectiveness of the proposed method by comparing different network backbones and different augmentation approaches. The ablation study is sufficient for validating the importance of each component of TokenMixup as well. 

3. Based on the proposed TokenMixup, the authors propose a variant: vertical TokenMixup. By incorporating the features from previous layers, it can easily captures multi-scale feature for further improvement, which is simple and effective. 

4. The authors provide some explanations and discussions from a perspective of curriculum learning, which leaves some inspirations for further research. 

Cons: 

1. Since ScoreNet is trained simultaneously with the main branch, there may exist some biases during the calculation of overall saliency. 
questions: 1. Is there any other methods for replacing the ScoreNet for filtering data samples with higher confidence? 
limitations: There is no potential negative societal impact for this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a token-level data augmentation method that is tailored to networks that have attention layers. The authors argue that recent gradient-based mixup methods are costly and propose a horizontal (mixing tokens between different samples) and vertical (mixing tokens between different layers) mixup schemes using attention. Both proposed methods improve the performance of baseline models on CIFAR and ImageNet.
strengths_and_weaknesses: Strengths
- As attention-based architectures (e.g. transformer) are becoming more and more important in many deep learning domains, finding a mixup method tailored for those architectures is timely and valuable.
- The proposed method is much more cost-efficient compared to the previous saliency-based methods, which is especially important as transformers models are usually very large.

Weaknesses
- Since the proposed method consists of many components (Sample difficulty assessment, optimal assignment, token-level mixup, saliency-based label reassignment), I would like to see more ablations to verify if each component is necessary. For example, how does (sample difficulty assessment + Cutmix) work compared to HTM or VTM?
- The most direct comparison to the saliency-based mixup baselines are given in Table 3 and L228-233. The authors note that the proposed attention-based approach, while being much more cost-efficient, shows higher performance than the gradient-based methods such as Co-Mixup. However, the authors do not provide any intuition on why the attention-based approach is better. Therefore, it is hard to interpret the performance gap especially since the authors motivate their approach as an efficient alternative to the gradient-based methods.
- According to the results in Appendix C Table 1, the proposed method seems somewhat sensitive to the choice of hyperparameters. If one or two of the hyperparameters are not set properly, the performance drops similar to the baseline (CCT-7/3x1 + Input Mixup & Cutmix in main paper Table 5.).
questions: - The authors provide results on HTM and VTM, but does not provide results when both HTM and VTM are applied simultaneously. How is the performance of HTM + VTM?
- Can the authors provide results on other transformer architectures to see if the proposed methods are generalizable?
limitations: The main limitation of the method is that it is only applicable to attention-based models, which the authors address clearly on the appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an efficient attention-guided data augmentation method, named TokenMixup. As the name implies, the goal is to provide an efficient mixup method based on attentions to improve the performance of classification models. Vertical TokenMixup is also introduced, which considers using tokens from different feature levels for mixup. Experiments on CIFAR and ImageNet show that the proposed approach performs better than the original mixup method.
strengths_and_weaknesses: Overall speaking, the writing quality of this paper is good and it is easy to follow. Some strengths and weaknesses of this paper are listed as follows.

Pros:

- The idea of this paper is interesting. Unlike previous follow-ups of the mixup method, this paper follows the curriculum learning framework and introduces a tiny scorenet to measure the saliency of this token. Besides, the Vertical TokenMixup method is also interesting. This is the first work I see that uses tokens from different token levels to conduct mixup operation and results in good performance in image classification.

- The analysis of this paper is great. Both the main paper and the supplementary material provide sufficient experiments to demonstrate how the proposed approach works. I really like the analysis and thank the authors for providing these experiments.

Despite the strengths in novelty and method analysis, I still have the following concerns.

Cons:

- Despite the efficiency, the proposed method needs an extra ScoreNet to compute saliency. This would inevitably introduce extra training time, no matter how much it is.

- In the original mixup method, there is only one hyper-parameter that needs to be tuned. However, in this paper, there are three new hyper-parameters as analyzed in the supplementary material. For CIFAR and ImageNet, the authors have provided suggestions on how to select them. However, when applied to other datasets, users still need to determine how to select them.

- At present, only CCT and the original ViT are used as baselines. Since the first ViT work, there are a lot of new ViT models. I really think the authors should conduct experiments on more powerful ViT models, like Swin Transformer, Pyramid Vision Transformer, and more recent VOLO, etc. Results on powerful baseline models would definitely make this paper stronger.

- In addition, from Table 5, we can see that the improvement is mostly from Input Mixup & Cutmix. Could the authors explain why the proposed approach succeeds while Manifold Mixup fails in lifting model performance?
questions: More analysis on the ImageNet dataset would make the analysis more convincing.

For more questions, please refer to the Strengths And Weaknesses part.
limitations: Limitations have been mentioned in the supplementary material. No potential negative societal impact is found.
review_title: null

Meta-review: metareview: The paper is about speeding up the saliency computation used in gradient based mixup algorithms(Puzzlemix, Co-mixup, etc). The authors propose employing the attention layer output of the transformer to replace the expensive saliency computation. 

Since focus of the the paper is on improving the speed and accuracy aspects of gradient based mixup algorithms, the tables 1 and 2 should remove

I strongly suggest the authors to remove tables 1 and 2 as it only shows irrelevant accuracy results across different network architectures. Since the main focus of the paper is on improving over the previous mixup algorithms, I suggest the authors to run the following experiment and insert it as table 1 in the main paper. Current table 5 does not show fair comparison across different mixup methods.

1) fix the network architecture. Denote it as A (i.e. CCT-7/3x1)
2) report the following accuracy results.
A;
A + input mixup;
A + manifold mixup;
A + cutmix;
A + puzzlemix;
A + co-mixup;
A + horizontal TM;
A + vertical TM;
A + horizontal TM + vertical TM

Here, you would need to make sure the backpropagate all the way through the individual pixels as opposed to tokens to properly run the gradient based mixup algorithms.

Also, I suggest the authors to add the end-to-end forward-backward computation time per image in addition to the avg. latency in table 3.

Overall, I like the proposed method of speeding up the saliency computation via attention maps. However, the experiment protocol needs a lot of improvement.
confidence: Certain
Decision: Accept