TokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformers
Authors: Hyeong Kyu Choi
Joonmyung Choi
Hyunwoo J. Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: data augmentation
mixup
vision transformer

Abstract: Mixup is a commonly adopted data augmentation technique for image classification. Recent advances in mixup methods primarily focus on mixing based on saliency. However, many saliency detectors require intense computation and are especially burdensome for parameter-heavy transformer models. To this end, we propose TokenMixup, an efficient attention-guided token-level data augmentation method that aims to maximize the saliency of a mixed set of tokens. TokenMixup provides ×15 faster saliency-aware data augmentation compared to gradient-based methods. Moreover, we introduce a variant of TokenMixup which mixes tokens within a single instance, thereby enabling multi-scale feature augmentation. Experiments show that our methods significantly improve the baseline models’ performance on CIFAR and ImageNet-1K, while being more efficient than previous methods. We also reach state-of-the-art performance on CIFAR-100 among from-scratch transformer models. Code is available at https://github.com/mlvlab/TokenMixup.

Token Efficient Attention-guided Token-level Data Augmentation for Transformers
Hyeong Kyu Choi", Joonmyung Choi', Hyunwoo Kim Department of Computer Science and Engineering, Korea University (imhgchoi, pizard,
ac. kr
Abstract
Mixup is commonly adopted data augmentation technique for image classification. Recent advances in mixup methods primarily focus on mixing based on saliency. However, many saliency detectors require intense computation and are especially burdensome for parameter-heays transformer models. To this end, we propose TokenMixup, efficient attention- attention-guiddd token-level data augmentation method that aims to maximize the saliency of a mixed set of tokens. TokenMixup provides x15 faster saliency-aware data augmentation compared gradient-based methods. Moreover, we introduce a variant of TokenMixup which mixes tokens within a sin- gle instance, thereby enabling multi-scale feature augmentation. Experiments show that our methods significantly improve the bascline models' performance on CI- FAR and ImageNet- IK, while being more efficient than previous methods. We also reach state-of-the-ar performance on CIFAR- 100 among from-scratch transformer models. Code is available at https: //gi thub. wot.///vaaab//Toeeaauu
Introduction
Various data augmentation methods have been proposed for computer vision tasks. One of the most successful augmentation methods is mixup [ which is commonly applied to image classification. It attempts augment the input by taking a convex combination of two random instances, and by reassigning the ground truth label correspondingly. Building on mixup, many subsequent works [ have been introduced. They focus on mixing instances in a more meaningful way (e.g., salieney- aware mixup, manifold level mixup, submodular diversity maximization etc.), while a great majority attempts perform mixup based saliency.
These saliency-awan mixup methods often entail gradient computation. To detect salient regions, the input is proppaat-d once and back propapappe to extract the gradient map. However, such mechanism tends to be computationally heavy. This is especially pertinent to transformer based models [10), as they generally retain high parameter volume. Keeping this in mind, we aim to present saliency- mixup method favorable for transformers. Specifically, we regard self self-attention as the inherent saliency detector, which serves as an efficient means for augmentation.
We accordingly propose ToxenMixup. an efficient attention-guided token-level mixup method. Its objective is mix intermediate token sets so that the saliency level of batch is maximized. To detect salient tokens, we take advantage of the attention map to achieve a x15 speed-up compared to the gradient-basee saliency is maximized by optimally matching instance pairs, accomplished by an algorithm that exactly solves the optimization problem. We also introduce ScoreNet, a simple module that measures the difficulty of an instance. Based on ScoreNet, TokenMixup is applied selectively with respect to sample difficulty, which can be viewed as a type of curriculum learning method. To the best of
saliency estimator, without compromising model performance. The batch
"The first two authors contributed equally i corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

our knowledge, this is the first attempt in the literature to discuss mixup from curriculum learning perspective. Furthermore, we present novel mixup approach that combines intermediate tokens across transformer layers within single instance. We name this Vertical ToxenMixup (VTM) since vertically mixes tokens to provide rich multi-scale information to the attention layer.
Experiments on image classification datasets imply that our methods are both effective and efficient. Also, an advantage of TokenMixup that it can be used in parallel with other mixup methods. In combination with those methods, we improve our baselines' performance by significant margins on CIFAR- 10, CIFAR-I00, and ImageNet- K. Especially for CIFAR- 100, we achieve new state-of- the-art performance among transformer-basee models that do not use pre-trained models.
Then, contributions are fourfold:
We propose Tixun an efficient attention-guided token-level mixup for transformers. By incorporating the self-attention map, we achieve x15 faster saliency-awan data augmentation compared to gradient-basee methods.
We incorporate curriculum learning into mixup, to adaptively perform TokenMixup in an interme- diate layer based on the confidence score of ScoreNet.
We also studied a variant, Vertical TokenMixup, which performs mixup with single sample and enables multi-scale feature augmentation.
We achieve state-of-the-art performance on CIFAR-100 among transformer models trained from scratch. We also gain significant improvements over baseline models for image classification.
2 Related Works
Mixup. Input Mixup [ is data augmentation method widely adopted for image classification. The classification model is trained with convex combination of input images and labels. A special case of Input Mixup is CutMix (2). which can be seen as pixel-wise mixup method with binary masks. Recent advances in these mixup methods mainly focus on appropriately utilizing saliency information, and on mixing in image feature levels.
The core motivation of saliency-based mixup is that salient regions should be preserved when mixed, to retain a sufficient amount of information and learn more natural feature representations. SaliencyMix [5] adopts various saliency detectors to directly extract salient regions. Puzzle Mix 3 attempts to mix images using saliency information while retaining local statistics. Co-Mixup maximizes gradient-basee based saliency while encouraging supermodular diversity of the mixed images.
SuperMix [9] takes advantage of supervised signals to mix input images based on saliency.
On the other hand, Manifold Mixup [6] has provided theoretical grounds the advantages of mixing images in the higher level of features. Manifold mixing helps the model learn flatter representation and smoother decision boundaries. Moreover, it is noted that it enhances robustness to adversarial attacks. Recently, AlignMix 7 was proposed to further align geometric properties of objects by mixing instances in the feature space. Furthermore, StyleMix 8 proposed to manipulate content and style information disparately to generate more abundant and robust samples.
Vision Transformers. Originating from the natural language processing field, the Transformer [T] model enjoyed tremendous success in numerous computer vision applications, e.g. image classifica- tion, object detection [2], and human-object interaction detection [13 14] et cetera. Specifically, image classification is one of the most fundamental computer vision tasks. Starting from the ViT [55. many works have attempted assimilate the convolutional operation with transformer blocks 21) Other variants have effectively incorporated the convolutional neural network into the transformer to achieve superior performance 22.26
3 Methods
Here, we introduce TokenMixup, simple but effective mixup method for transformer models. The goal of our method is to augment intermediate tokens while maximizing the saliency level. This can be accomplished in three major steps: 1) Sample difficulty assessment, 2) Attention-guided saliency detection & optimal assignment, 3) Token-level mixup. The following subsections provide detailed descriptions of the steps.

---

Optimal Assignment
Token-level Mixup
X
ScoreNet
Token
Pooling
X
Classifier
X
Alg. line
Alg. line 6
Sample level
Token level
Alg. line
Figure 1: Toxen Batch samples X are first filtered with respect to difficulty scores evaluated by
ScoreNet in F. resulting in easy samples X. Then, images in X are optimally paired with samples in X via Hungarian matching, that the overall saliency the mini-batch maximized after mixup. For a matched pair (X x(), tokens of {() with sufficiently lower saliency levels than the tokens of x are replaced, resulting in new token set (1) containing greater salieney level.
mix
3.1 Sample Difficulty Assessment
We first assess the difficulty of the input to adaptively decide whether to apply augmentation. To evaluate sample difficulty, a measuring function F needs to be defined. We utilize parameterized module, ScoreNet, which is simple MLP that predicts the target value y based on the intermediate tokens ((),, where is the index of the sample in a batch (See the supplement for figure). Given the ScoreNet output, the difficulty score is evaluated with the prediction loss computed
F(X(1, (()) CrossEnntopy(SSorreettt ")) (()).
If the score is greater than a threshold т, the sample is deemed sufficiently hard and no mixup is performed. On the other hand, the set of easy samples with F (x y()) < t denoted as will be augmented by Token-level mixup after pairing with other samples in mini-batch, as illustrated in Figure I Note that ScoreNet is trained simultaneously with the main model and training ScoreNet to assess sample difficulty at the intermediate layer can be viewed as an auxiliary task.
3.2 Attention-guided Saliency Detection & Optimal Assignment
have discussed the imbalanced information of tokens. Due to this imbalance, random token mixing would potentially cause significant information loss and meaningless token replacements (See the supplement for relevant analysis). Thus, we aim to mix tokens based on saliency.
Several works 19 27 29
Attention-guided saliency detection. Instead of the computationally heavy gradient- saliency detectors 3 4. take advantage of the attention map, an inherent saliency approximator within transformer modules. Specifically, we can infer the saliency the tokens in the th layer via Attention Rollout 30,, which computes the attention imposed on the tokens from layer i to i (E 0)
(i)
g (i+1) (efl).....(..................... (i+())
such that (1) = H 1 where H/ is the number of heads in the multi-head attention layer, and () is the 1tth attention head of layer i. In order to retrieve A, the samples need to be propagated for the subsequent é layers with stop-gradient. Since each additional forward propagation is an overhead, we approximate Attention Rollout by setting f 0. That is. we only use the attention map of the following layer as the saliency estimator. We find this a sufficient approximation of the full Attention Rollout. See section [5. for analysis on the soundness of the approximated saliency map. Then, the saliency score S, can be computed as
n
Σ
Air
(3)
n
1m1
where = 1,2,... ... ... denotes the token index.
3

---

Algorithm Mixup
Input: R hhxnxd Y E R hxc
T, P
Output: X Rbxuxd Rbbx
u - evaluate difficulty of x
s.t. 1, 1,2......
Eq.
2: (( - evaluate saliency of each token with b-step attention rollout
> Eq.
3: "0,0 - select easy samples w.r.t U/ and r s.t. = 1.2.....
4: Cij < Σ, max(( Stth p. 0)
s. ........b and j 1,2...... s.t. m = 1.2.....b b' and (m) E {1, ..... b}
Eq. (
5: (m)) - HungarianMatthingg(C (Cy))
6: X x(i)
Mix-token(XX), x(r())p))
s.t. = ..
Eq. 10
mix
7: pu)
mix = Relabel(P()) y(r(i))
Eq.
9: return X mix to X, Y
8: restore x 7(0)
s.t. 1,2..,,
Optimal assignment. Based on the estimated saliency of tokens, we aim to maximize the overall saliency level by optimally assigning a different mixup target for each instance. So, we first define Pi, R2 as the saliency difference between a random instance pair j), which is computed as
Pi.j S St.
where i,j 1.2.....b and R" the token saliency map from the j instance. Then, represent our objective as an optimization problem written as
max max OEEM B
Σ (Pr.ori - p) prrri. ri.
(5)
where M is the set of all possible batch permutations of the instances, and or E M refers to an arbitrary permutation. is the threshold hyperparameter that controls the minimum saliency gain required for a token be mixed, and r; is a binary decision vector for the jh instance. i.e., token of i is replaced with token f of (1) when ri, = 1, and preserved when Fi, = 0 (r = 1,2,............................ n).
The above optimization problem be exactly solved by utilizing the Hungarian Matching algo- rithm 31
The matching algorithm requires the score matrix, E Rbx which computed as
C1. Σ max 0),
(6)
where each item in C,,, refers the maximum saliency gain resulting from mixing (0) and X The actual maximum gain will be reached when p= O, but we set a positive threshold to control the minimum saliency gain required for each token to be replaced. Then, by applying the Hungarian algorithm, we can find the optimal batch permutation o² such that
o' arg max Σ C
Then, by mixing 80 and (1), our objective is optimized.
By incorporating sample difficulty assessment (section) 3.1 into 5 our final objective is written as
b Σ
max max oEEM FrB
(Pi,or(0 PPP,o(t)-00 - p)Tr Tri
s.t. 1 s nx (F(X")
where F is the difficulty measurer, and T the difficulty threshold. 1 denotes the indicator function, while 1 is vector. If the sample is hard, i.e., (( y()) enforcing all elements of r, to be O. In that case, none of the tokens will be mixed. On the other hand, if the sample easy, Le.. ((x(), y(i) < T, the constraint is 1 T & a which satisfied by any random vector. Then, the objective becomes identical to 5
т. the constraint becomes 1 O.

---

K,V: :
Encoder
K
Multi-scale feature
Encoder
* V
Cross-Attention
Feed Forward
Z Softmax (olinax((KK)
Encoder
Figure 2: Vertical TokenMixup. By applying VTM, the x most salient tokens from each previous layer are brought up to form the key and value tokens. By utilizing cross-attention mechanism, the dimension is preserved while being able aggregate multi-scale information for token augmentation. The figure demonstrates the case when VTM applied layer 3.
3.3 Token-level Mixup
Now, the remaining question how the tokens of (0) and x(o' (1) are going to be mixed. As defined in objective (5). we use hard replacement of tokens with respect decision vector r;. Then, mixing is performed by replacing token index / from the original instance i, with token from the paired instance = or a that Pi. is satisfied. In other words, only the token indices that increase saliency level greater than p are mixed. This is accomplished by defining appropriate binary mask M, E IB" (t= 1.2..... n.) containing
s( 5(0 P
M,
otherwise
where 510) is the saliency score of x Then, the mask is employed as
00 M X (1 M) O x()
(10)
mix
where operator O denotes element-wist multiplication and (1, x() refer to the paired instances. We also reassign labels based on the replaced tokens" saliency score. The new mixed label is computed as
Σ M, S,,0
Σ (1 M.) 5()
p( mix
(11)
ΣR M, 5(0
(1 M,) s(
Σ
M, (()) + +(1 M,) s(
Consequently, relatively unimportant tokens will be replaced with salient tokens from another instance, and its label will be adjusted with respect to the change in overall saliency level. See Figure ffor visualization, and Algorithm [ for overall computation process. More detailed pseudocode is provided
the supplement.
3.4 Vertical TokenMixup
Mixing multiple instances is not the only way to mixup. We also present simple variant of TokenMixup, which provides token-level augmentation within single sample by utilizing the tokens from previous layers. Similar to (5) we can define a new objective for Vertical TokenMixup as
max max 16
Σ (PLN - T) fi.
(12)
EIB
5

---

where L is the set of previous layer indices, and (i) returns an arbitrary layer index for the 1th instance. This objective can optimized by a similar scheme as Algorithm[[] However, considering that the tokens from each layer with an identical index tend to contain similar information, saliency difference matrix (r, S10) - S; (as in 44) will most likely constant across indices. This may lead to meaningless mixup without much increase in saliency level. Therefore, we take different approach to mix tokens vertically.
The simplest method would be to concatenate all tokens from previous layers and apply self-attention. But due to the quadratic complexity of self-attention, naive concatenation may induce excessive computation. To reduce overhead, we selectively pool the most salient tokens from each previous layer and concatenate them the original token set. Also, preserve input dimension, we adopt the cross tross-attention mechanism. If X1 is the original token set and X₂ is the concatenated token set, vertical cross-attention ex expressed as
Z Sofumax (X, Wg (T) X2Wy,
(13)
where Wg. Wk. W, denote the projection layers for query, key, and value, respectively. See Figure for visualization, and the supplement for pseudocode.
3.5 Discussions
Moxun as Curriculum Learning. A general framework for curriculum learning consists of two main components: Difficulty Measurer and Training Scheduler [32]. TokenMixup satisfies these conditions, as ScoreNet measures the difficulty of each input, and augmenting the tokens based input difficulty naturally schedules training. In the early training phase where the model parameter not optimized, most instances will be evaluated difficult and TokenMixup is performed. As the model converges, on the other hand, many samples will become easy, triggering augmentation to make the task more challenging overall. Also, TokenMixup is applied discriminatively based on individual sample difficulty which enables more intricate curriculum scheduling. See section 5.3 for empirical demonstration. These properties will be especially useful when other mixup methods are used in parallel. For instance, if input mixup is already applied. further augmentation may be unnecessary. In such case, ScoreNet will regard the sample sufficiently difficult, and no further mixup will be performed.
4 Experiments
4.1 Preliminaries
Baseline. In experiments on CIFAR we used Compact Convolution Transformer (CCT) as our baseline. For ImageNet- 1k 34 experiments, we used the vanilla VIT-B/16 [15] as baseline. Then, our augmentation methods are evaluated on three representative image classification datasets: CIFAR-10. CIFAR-100, and ImageNet-IK.
Experimental setup. We evaluate TokenMixup and its variant, Vertical TokenMixup (VTM), on CIFAR and ImageNet with different settings. To avoid confusion, denote the original Toxen as Horizontal TokenMixup (HTM) in the following sections. For CIFAR experiments, we adopt the 1500-epoch version of CCT. We modified the learning rate scheduler and positional embedding type to achieve better performance than original papers (denoted in Table TT. Other experiment settings follow (22). and all experiments on CIFAR datasets were conducted on a single RTX A6000 GPU. For ImageNet-IK IK experiments, we used the ViT-B/16 model pre-trained and fine-tuned on ImageNet-21k and ImageNet-1k, where we took advantage of the officially released pre-trained weights. Experiments for Horizontal TokenMixup were conducted on single NVIDIA A100 GPU, and RTX 3090 GPUs were used parallel for Vertical TokenMixup. Other experiment settings are reported in the supplement.
4.2 Experimental Results
First, we demonstrate the image classification performance on CIFAR-10 and CIFAR-100 when TokenMixup applied to CCT. From Table we can see that applying Horizontal TokenMixup and Vertical Token simultaneously improves its strict CIFAR- 100 baseline, by a significant margin of 0.70%. HTM and VTM each surpasses the baseline performance by margin of 0.69% and 0.67% % as well. Also, we outperform the previous state-of-ther the model (among from- models) by

---

Table 1: Experimental results on CIFAR. Experiment results with TokenMixup on CCT compared with state-of-the- models that do for each model, and baseline models denoted with reports retrained performance. Also, the number parenthesis indicates the numher of epochs used training, and the parameter numbers are retrieved from CIFAR-100 models. Best performances are highlighted in yellow.
pretrained models for initialization. Top-I validation accuracy is reported
Models
Params MACs
CIFAR-10 CIFAR-100
Convolutional Network based
ResNet18
11.18 M
0.04
90.27
63.41
ResNet50
23.53 M
0.08 G
90.60
61.68
ResNec1001-v2
10.33 M
1.55 G
95.08
77.29
MobileNetV222..0
36
8.72 M
0.02 G
91.02
67.44
WRN-28-10 37
36.5 M
96.00
80.75
WRN-40-4 37
8.9 M
95.47
78.82
ResNeXI-29-8xxfe
34.4
96.35
82.23
ResNeXt-229- 19x164 38
68.1 M
96.42
82.69
Transformer based
ViT-Lite-6/4
3.19 M
0.22 G
93.98
73.33
ViT-Lite-7/4 22
3.72 M
0.26
93.57
73.94
NesT-T
17.0
96.04
78.69
NesT-B
68.0
97.20
82.56
CVT-6/4 [22
3.19 M
0.22G G
93.60
74.23
CCT-7/3x1(15000 22
3.76 M
0.95
97.48
82.72
CCT-7/3x1(1100)
3.78 M
0.95
97.48
82.87
CCT-7/3x1(1500) Horizontal TM (ours)
3.81 M 0.95
97.57
83.56
CCT-7/3xI(1500) Vertical TM (ours)
0.95
97.78
83.54
CCT-7/3x1(1500) HTM VTM (ours)
3.81 M
0.95
97.75
83.57
Table Experimental results on ImageNet. Experiment results with TokenMixup on VIT are compared. We report Top-1 and Top-5 validation accuracy on ImageNet-Ik, For the baseline model, we used ViT-B/16 (224 224). denotes fine-tuned performance officially reported by the authors of VIT.
Models
Params MACs
Top-1 Ace. Top-5 Acc.
Convolutional Network based
ResNet50 40
25.5 M 60.19 M
4.3 G
76.20
ResNet152 40
11.58 G
78.57
DenseNct-264
79.20
94.71
WRN-55---bbttleneck
37
68.9 M 21 M
78.10
93.97
ReGNerY-44 42
2.0
80.0
ResNeeXt-50-xx40d
38
77.00
ResNect--01-64x4d 38
79.60
94.70
Transformer based
ViT-B/16-2241 [15]
86.6 M
16.9
81.2
ViT-B/16-224 Horizontal TM (ours)
87.3
16.9
82.37
96.29
ViT-B/16-224 Vertical TM (ours)
86.6
82.30
96.33
ViT-B/16-224 HTM + VTM (ours)
87.3 M
16.9 G
82.32
96.27
huge margin of 0.85%. the case of CIFAR-10, we achieved higher accuracy of 97.57% and 97.78% with HTM and VTM, respectively. Considering the highly saturated performance on this dataset, 0.30% improvement over the baseline is not trivial.
Performance on ImageNet-1K 1K is evaluated on the vanilla ViT [5] model. The officially reported Top-1 accuracy is 81 .2, which achieved by using weights pre-trained on ImageNet-21K and fine-tuning on ImageNet- IK. By fine-tuning the model further by employing our augmentation methods, we achieve Top-1 accuracy of 82.37% with Horizontal TokenMixup. 82.30% with Vertical TokenMixup. and 82.32% by using both. Also note that HTM marginally increases parameter size by only 0.7M, while VTM does not require any additional parameters. See the supplement for further experiments other baselines.
7

---

Input Image
Attention Map
Attention Rollout
Gradiennt
Image
Attention
Attention Rollout
Gradient
Figure 3: Qualitative comparison of saliency detectors. The saliency map derived for each method is provided. We can observe that the methods render similar outputs. Note, the values are minmax scaled to [0.1].
Table Saliency detector comparison. Our method (HTM) achieves high accuracy while heing x15 as fast as the gradient method. Also, we observe only x3 latency compared to the random baseline.
Saliency type
Gradient-based
Attention-based
Random baseline
Avg. Latency (ms)
236 (x14.8)
16 (x1.0)
5 (x0.3)
CIFAR- 100 Acc.
83.28
83.56
83.05
Analysis
Here, we analyze TokenMixup to answer the following research questions: Q1. Is the Attention an appropriate saliency approximation? Q2. What are the effects of each component in TokenMixup? Q3. What properties does TokenMixup have?
5.1 Soundness of Saliency Estimation
In TokenMixup, we estimate the saliency using the attention map from the subsequent layer. The soundness of this approximation would be one of the main questions regarding saliency detection. So, we provide relevant analyses on the appropriateness of our saliency approximation method to answer the research question Q1.
Qualitative examples. Here, we provide a qualitative analysis on the performance each saliency detector. In Figure 3 we observe that there is significant difference between the full attention rollout and the 1-step approximation, i.e., the subsequent attention map. Slight discrepancy in the sharpness of the distribution saliency threshold Moreover, attention baased methods seem to be more accurate than the gradient- based detector in certain cases. See the supplement for additional qualitative examples, and also the quantified comparison on the sharpness measures of the two methods.
be observed, but it does not greatly affect our method that utilizes
Efficiency comparison. We compare the efficiency of three saliency detectors: gradient- based, attention-basee based, and random baseline. For the gradient-han hased detector, we follow the method used in Co-Mixup 44, and random baseline refers random selection of tokens. In Table we report the average latency of saliency detection per iteration in our CIFAR- 100 experiment setting with a batch size 128. Compared to the gradient-based method, attention- saliency detection with 1-step rollout achieves approximately x x15 speed-up while demonstrating even better performance.
5.2 Component Analysis
In this section, we analyze the effect each component in Horizontal TokenMixup and Vertical TokenMixup so as to answer Q2. In Table 4 we provide ablation studies on components of HTM: Sample difficulty assessment (ScoreNet). Optimal assignment, Token-level mixup, and Saliency- based label reassignment. By ablating ScoreNet, all instances are mixed regardless of sample difficulty; note, ScoreNet still receives supervision signals. By getting rid of optimal assignment, mixup pairs are not assigned via Hungarian matching; pairs are randomly selected. Token-level mixup and label reassignment relevant to the method by which tokens are selected. Then, by ablating mixup, we randomly select tokens to replace, while label reassignment ablation derives new labels based on the number of tokens that have been replaced. We also conducted sensitivity tests on key hyperparameters p, and K, but tables and discussions were moved to the supplement due to spatial constraints.

---

Table 4: Ablation study on CIFAR-100. Performance on CIFAR- 100 is evaluated by ablating each component
one-by-one. We observe that optimal assignment and label reassignment plays significant role.
Ablated Component
CIFAR-101 Accuracy
CCT-7/3x1 + HTM
83.56 (- (-000)
(-) Sample difficulty assessment (ScoreNet)
82.85 (- 0.71)
Optimal assignment
82.61 0.95)
Token-level mixup
83.05 0.51)
Salieney-bass label reassignment
82.62 0.94)
Table 5: Mixup combinations. CCT performance on CIFAR-100 is compared for mixup method combinations.
denotes values reported in the original paper. Best performanees among comparable settings are highlighted.
Mixup Combination
CIFAR-I00 Accuracy
CCT-7/3x1
76.80
CCT-7/3xI Manifold Mixup
76.72
CCT-7/3x1 Horizontal TM
77.34
CCT-7/33x Vertical TM
78.18
CCT-7/3x1 Input Mixup & Cutmix
82.87
CCT-7/3x1 Input Mixup & Cutmix Manifold Mixup
80.08
CCT-7/3x1 Input Mixup & Cutmix Horizental TM
83.56
CCT-7/3x + Input Mixup & Cutmix Vertical TM
83.54
CCT-7/3x1 Input Mixup Cutmix HTM VTM
83.57
5.3 What TokenMixup Learns
Here, we answer our research question Q3 by analyzing what our mixup methods are learning. We demonstrate a curriculum learning perspective terms of HTM, and compare our method with direct mixup alternative, manifold mixup [6]
A Curriculum Learning perspective. In Fig- ure 4 we plot the trend in the number of sam- ples per batch TokenMixup is activated based on ScoreNet. In the first few epochs (regime (A)), the number of mixed samples is relatively low, as there less need for augmentation in the early phase of learning. But the learn- able parameters warm up. the number increases abruptly. In regime (B), the average number of mixed tokens slowly increases as model perfor- mance steadily improves. In the last learning phase (regime (C)), mixup frequency converges.
Avg. mixed samples
(A)
(8)
(C)
200
400
600
800
1000
1200
1400
Epochs
Figure 4: Trend in the number of mixed instances.
Comparison with Manifold Mixup. TokenMixup is applied to the intermediate tokens, similarly to manifold mixup ( which is an input mixup 1 method applied to intermediate features. By using manifold mixup, transformer tokens are mixed via linear interpolation, instead of the hard replacement of tokens as TokenMixup. Table our methods are compared with manifold mixup, along with various combinations of original mixup methods (i.e., input mixup and cutmix). The best setting in the CCT paper adopted input mixup and cutmix. By applying HTM and VTM on top of that setting, we achieved sate-of-theeart performance. On the other hand, by applying manifold mixup, we observed severe deterioration accuracy. We conjecture that soft mixing of transformer tokens leads ever-smoothing of features, which may be inadequate for self-attention layers. See the supplement for visual aid.
6 Conclusion
We proposed TokenMixup for transformers, which adopts the attention map in place of computa- tionally heavy gradient-based saliency detectors. By adopting the attention as the saliency map, we achieve x x15 speed-up compared the gradient-based method with even better performance. We also introduce a novel perspective of curriculum learning to mixup, which enables adaptive augmentation based on sample difficulty and training schedule. To add diversity, Vertical TokenMixup introduced, which mixes tokens from different layers for multi-scale feature augmentation within single sample.

---

With TokenMixup, we achieve state-of-the- aht performance on CIFAR-100, and improve baselines by significant margins on CIFAR-I0 and ImageNet- IK..
Acknowledgments and Disclosure of Funding
This work was partly supported by ICT Creative Consilience program (ITP--202--2000-001181)) supervised by the IITP; the National Supercomputing Center with supercomputing resources including technical support SSC--022-CRR--1000 and Kakao Brain Corporation.
References
[1] Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Par Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018
[2] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[3] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statisties for optimal mixup. In ICMI., 2020.
[4] Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. arXiv preprint arXiv:2102.0300 2021.
[5] AFM Uddin, Mst Monira. Wheemyung Shin, TaeChoong Chung, Sung- Ho Bae, et al. Saliencymix: A saliency guided data augmentation strategy for better regularization. arXiv preprint arXiv:2006.01791, 2020.
[6] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICMI., 2019.
[7] Shashanka Venkataramanan, Yannis Avrithis, Ewa Kijak, and Laurent Amsaleg. Alignmix: Improving representation by interpolating aligned features. ICLR, 2022.
[8] Minui Hong, Jinwoo Choi, and Gunhee Kim. Stylemix: Separating content and style for enhanced data augmentation. In CVPR, 2021.
[9] Ali Dabouei, Sobhan Soleymani. Fariborz Taherkhani, and Nasser M Nasrabadi. Supermix: Supervising the mixing data augmentation. In CVPR, 2021.
[10] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017.
[1I] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[12] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiangang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020.
[13] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end human-object interaction detection with transformers. In CVPR, 2021.
[14] Jihwan Park, SeungJun Lee, Hwan Heo, Hyeong Kyu Choi, and Hyunwoo J. Kim. Consistency learning via decoding path augmentation for transformers in human object interaction detection. In CVPR, 2022.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2021.
[16] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[17] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[18] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu, Incorporating convolution designs into visual transfomers. In 2021.
10

---

[19] Byeongho Heo. Sangdoo Yun, Dongyoon Han. Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In ICCV, 2021.
[20] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. NeurIPS, 2021.
[21] Stéphane d' Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In ICML, 2021.
[22] Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:204405704, 2021.
[23] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In ICCV, 2021.
[24] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In ICCV, 2021.
[25] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint 0xXiv22122.10885 2021.
[26] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. NewrIPS, 2021.
[27] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Ffffciient vision transformers with dynamic token sparsification. NeurlPS, 2021.
[28] Byungseok Roh, Jae Shin, Wuhyun Shin, and Sachoon Kim. Sparse detr: Efficient end-to-end object detection with learnable sparsity. In ICLR, 2021.
[29] Tao Wang, Li Yuan, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Pup-detr: towards efficient visual analysis with transformers. ICCV, 2021.
[30] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00 00928, 2020.
[31] James Munkres.
Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics, 5(1): 1957.
[32] Xin Wang, Yudong Chen, and Wenwu Zhu, A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[33] Alex Krizhevsky, Geoffrey Hinton, et Learning multiple layers features from tiny images, 2009.
[34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
[36] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chied Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018
[37] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 165.0766.. 2016.
[38] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In CVPR, 2017.
[39] Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, Serean Arik, and Tomas Plister. Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding. In AAAI, 2022.
[40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[41] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[42] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.
11

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section
(b) Did you describe the limitations of your work? [Yes] See the supplement
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the supplement
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and the supplement for technical details
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A] Our experiment settings do not conduct each experiment multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4 and the supplement for technical details.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See the supplement or Section 4 baseline.
(b) Did you mention the license of the assets? [Yes] See the supplement
(c) Did you include any new assets either the supplemental material or as URL? [Yes] See the supplement
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [Yes] See the supplement
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See the supplement
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "sHLKpL7c7V9", "reviewer": "Reviewer_Muyi", "summary": "The paper proposes a novel data augmentation technique for visual transformers.", "questions": "- Can you encapsulate your staff in a layer and show how to add it? The source code is based on a function. If you want your code to be used by others, having it as a standard module and a simple way to incorporate it in existing models would be a great plus to your work", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Interesting and novel idea\n- Faster performance\n- SOTA results\n- Comprehensive ablation studies\n\nWeaknesses:\n\n- It is not entirely clear what does it take and how easy it is to add this method to an arbitrary transformer. It would help to have an appendix with python code showing how a typical transformer is modified with the proposed algorithm. Is it as simple as adding a layer for which good hyperparameter values are known, or additional hyperparameter search is really necessary on case by case basis?\n- The work explores 2 baseline transformer methods applied to 3 datasets. There are other models that are cited in the paper such as ViT-Lite, NesT-T, NesT-B. It would be of interest to have your work applied to those as well.\n- I feel this method could have wider applicability, beyond Transformer based models (e.g. ResNets). It would be of interest to either see some empirical results exploring this area, or if this is impossible due to the limitations of the method, to have a discussion of this limitation and perhaps an outline of how this limitation could be overcome in future work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "kk9v7etWaum", "reviewer": "Reviewer_NLuh", "summary": "This paper presents a token-level data augmentation method that is tailored to networks that have attention layers. The authors argue that recent gradient-based mixup methods are costly and propose a horizontal (mixing tokens between different samples) and vertical (mixing tokens between different layers) mixup schemes using attention. Both proposed methods improve the performance of baseline models on CIFAR and ImageNet.", "questions": "- The authors provide results on HTM and VTM, but does not provide results when both HTM and VTM are applied simultaneously. How is the performance of HTM + VTM?\n- Can the authors provide results on other transformer architectures to see if the proposed methods are generalizable?", "limitations": "The main limitation of the method is that it is only applicable to attention-based models, which the authors address clearly on the appendix.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- As attention-based architectures (e.g. transformer) are becoming more and more important in many deep learning domains, finding a mixup method tailored for those architectures is timely and valuable.\n- The proposed method is much more cost-efficient compared to the previous saliency-based methods, which is especially important as transformers models are usually very large.\n\nWeaknesses\n- Since the proposed method consists of many components (Sample difficulty assessment, optimal assignment, token-level mixup, saliency-based label reassignment), I would like to see more ablations to verify if each component is necessary. For example, how does (sample difficulty assessment + Cutmix) work compared to HTM or VTM?\n- The most direct comparison to the saliency-based mixup baselines are given in Table 3 and L228-233. The authors note that the proposed attention-based approach, while being much more cost-efficient, shows higher performance than the gradient-based methods such as Co-Mixup. However, the authors do not provide any intuition on why the attention-based approach is better. Therefore, it is hard to interpret the performance gap especially since the authors motivate their approach as an efficient alternative to the gradient-based methods.\n- According to the results in Appendix C Table 1, the proposed method seems somewhat sensitive to the choice of hyperparameters. If one or two of the hyperparameters are not set properly, the performance drops similar to the baseline (CCT-7/3x1 + Input Mixup & Cutmix in main paper Table 5.).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dkX4KprRGN7", "reviewer": "Reviewer_nzAU", "summary": "In this paper, the authors proposed a novel saliency-aware token-level data augmentation method for assisting the training of vision transformers. Specifically, they proposed TokenMixup, which takes the self-attention as the inherent saliency detector, to mix intermediate token sets so that the saliency level of a batch is maximized. With TokenMixup, they achieve significant improvement on image classification tasks and meanwhile obtain higher efficiency than other gradient-based mixup approaches. ", "questions": "1. Is there any other methods for replacing the ScoreNet for filtering data samples with higher confidence? ", "limitations": "There is no potential negative societal impact for this work. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. This paper proposes TokenMixup, a novel saliency-aware token-level mixup strategy for improving vision transformers. It ingeniously adopts self-attention, the basic module in transformers, to serve as a substitute for conventional computationally-heavy saliency detector. \n\n2. The authors conduct thorough experiments for evaluating the effectiveness of the proposed method by comparing different network backbones and different augmentation approaches. The ablation study is sufficient for validating the importance of each component of TokenMixup as well. \n\n3. Based on the proposed TokenMixup, the authors propose a variant: vertical TokenMixup. By incorporating the features from previous layers, it can easily captures multi-scale feature for further improvement, which is simple and effective. \n\n4. The authors provide some explanations and discussions from a perspective of curriculum learning, which leaves some inspirations for further research. \n\nCons: \n\n1. Since ScoreNet is trained simultaneously with the main branch, there may exist some biases during the calculation of overall saliency. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aeaxyCT2hk", "reviewer": "Reviewer_pM5D", "summary": "This paper presents an efficient attention-guided data augmentation method, named TokenMixup. As the name implies, the goal is to provide an efficient mixup method based on attentions to improve the performance of classification models. Vertical TokenMixup is also introduced, which considers using tokens from different feature levels for mixup. Experiments on CIFAR and ImageNet show that the proposed approach performs better than the original mixup method.", "questions": "More analysis on the ImageNet dataset would make the analysis more convincing.\n\nFor more questions, please refer to the Strengths And Weaknesses part.", "limitations": "Limitations have been mentioned in the supplementary material. No potential negative societal impact is found.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall speaking, the writing quality of this paper is good and it is easy to follow. Some strengths and weaknesses of this paper are listed as follows.\n\nPros:\n\n- The idea of this paper is interesting. Unlike previous follow-ups of the mixup method, this paper follows the curriculum learning framework and introduces a tiny scorenet to measure the saliency of this token. Besides, the Vertical TokenMixup method is also interesting. This is the first work I see that uses tokens from different token levels to conduct mixup operation and results in good performance in image classification.\n\n- The analysis of this paper is great. Both the main paper and the supplementary material provide sufficient experiments to demonstrate how the proposed approach works. I really like the analysis and thank the authors for providing these experiments.\n\nDespite the strengths in novelty and method analysis, I still have the following concerns.\n\nCons:\n\n- Despite the efficiency, the proposed method needs an extra ScoreNet to compute saliency. This would inevitably introduce extra training time, no matter how much it is.\n\n- In the original mixup method, there is only one hyper-parameter that needs to be tuned. However, in this paper, there are three new hyper-parameters as analyzed in the supplementary material. For CIFAR and ImageNet, the authors have provided suggestions on how to select them. However, when applied to other datasets, users still need to determine how to select them.\n\n- At present, only CCT and the original ViT are used as baselines. Since the first ViT work, there are a lot of new ViT models. I really think the authors should conduct experiments on more powerful ViT models, like Swin Transformer, Pyramid Vision Transformer, and more recent VOLO, etc. Results on powerful baseline models would definitely make this paper stronger.\n\n- In addition, from Table 5, we can see that the improvement is mostly from Input Mixup & Cutmix. Could the authors explain why the proposed approach succeeds while Manifold Mixup fails in lifting model performance?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept