Adversarial AutoMixup
Authors: Huafeng Qin
Xin Jin
Yun Jiang
Mounîm El-Yacoubi
Xinbo Gao
Venue: ICLR 2024 spotlight (2024)
Keywords: Data Augmentation
Mixup
Image Classification
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Data mixing augmentation has been widely applied to improve the generalization ability of deep neural networks. Recently, offline data mixing augmentation, e.g. handcrafted and saliency information-based mixup, has been gradually replaced by automatic mixing approaches. Through minimizing two sub-tasks, namely, mixed sample generation and mixup classification in an end-to-end way, AutoMix significantly improves accuracy on image classification tasks. However, as the optimization objective is consistent for the two sub-tasks, this approach is prone to generating consistent instead of diverse mixed samples, which results in overfitting for target task training. In this paper, we propose AdAutomixup, an adversarial automatic mixup augmentation approach that generates challenging samples to train a robust classifier for image classification, by alternatively optimizing the classifier and the mixup sample generator. AdAutomixup comprises two modules, a mixed example generator, and a target classifier. The mixed sample generator aims to produce hard mixed examples to challenge the target classifier, while the target classifier's aim is to learn robust features from hard mixed examples to improve generalization. To prevent the collapse of the inherent meanings of images, we further introduce an exponential moving average (EMA) teacher and cosine similarity to train AdAutomixup in an end-to-end way. Extensive experiments on seven image benchmarks consistently prove that our approach outperforms the state of the art in various classification scenarios. The source code is available at 
https://github.com/JinXins/Adversarial-AutoMixup.

Published as a conference paper at ICLR 2024
ADVERSARIAL AUTOMIXUP
Huafeng Qin Xin Jin 1.x Yun Jiang' Mounim A. El-Yacoubi
Xinho Gao
1 CChongqing Technology and Business University
TTelecom SudParis, Institut Polytechnique de Paris
SChongqing University of Posts and Telecommunications
* Equal contribution
Corresponding author
ABSTRACT
Data mixing augmentation has been widely applied improve the generalization ability deep neural networks. Recently, offline data mixing augmentation, e.g. handcrafted and saliency information-base mixup, has been gradually replaced by automatic mixing approaches. Through minimizing two sub-tasks, namely, mixed sample generation and mixup classification in end-to-end way, AutoMix significantly improves accuracy on image classification tasks. However, as the op- timization objective is consistent for the sub-tasks, this approach is prone generating consistent instead of diverse mixed samples, which results in overfit- ting for target task training. In this paper, we propose AdAutomixup, an adversar- ial automatic mixup augmentation approach that generates challenging samples to train a robust classifier for image classification, by alternatively optimizing the classifier and the mixup sample generator. AdAutomixup comprises two modules, a mixed example generator, and a target classifier. The mixed sample generator aims to produce hard mixed examples to challenge the target classifier, while the target classifier's 's aim is to learn robust features from hard mixed examples to im- prove generalization. To prevent the collapse of the inherent meanings of images, we further introduce an exponential moving average (EMA) teacher and cosine similarity to train AdAutomixup in end-to-end way. Extensive experiments on seven image benchmarks consistently prove that our approach outperforms the state of the art in various classification scenarios. The source code is available wttpa//u..ooo/nnnnnn//eeeeeeeeee
INTRODUCTION
Due to their robust feature representation capacity, Deep neural network models, such as convo- lutional neural networks (CNN) and transformers, have been successfully applied in various tasks, e,g., image classification (Li et al., 2022c; Krizhevsky et al., 2012: Li et al., 2022a; 2024), object detection (Bochkovskiy et al., 2020), and natural language processing (Vaswani et al., 2017). One of the important reasons is that they generally exploit large training datasets to train massive network parameters. When the data is insufficient, however, they become prone to over-fittiing and make overconfident predictions, which may degrade the generalization performance on test examples.
To alleviate these drawbacks, data augmentation (DA) is proposed to generate samples to improve generalization downstream target tasks. Mirup (Zhang et al., 2017), recent DA scheme, has received increasing attention as it can produce virtual mixup examples via a simple convex com- bination of pairs of examples and their labels effectively train a deep learning (DL) model. DA approaches (Li et al., 2021; Shorten & Khoshgoftaar, 2019; Cubuk et al., 2018; 2020; Fang et al., 2020: Ren et al., 2015; et al., 2020), proposed for image classification, can be broadly split into three categories: 1) Handerafted-
mixup augmentation approaches, where patches from one image are randomly cut and pasted onto another. The ground truth label of the latter is mixed with the label of the former proportionally to the area of the replaced patches. Representative approaches include CutMix ( (Yun et al., 2019), Cutout (DeVries & Taylor, 2017), ManifoldMixup (Verma al., 2019), and ResizeMix (Qin et al., 2020). CutMix and ResizeMix, shown in Fig. 1, generate mixup samples by randomly replacing a patch in an image with patches from another; 2) Saliency- guided mixup augmentation approaches that generate, based on image saliency maps, high-quality sam- ples by preserving regions of maximum saliency. Representative approaches (Uddin et al., 2020; Walawalkar et al., 2020; Kim et al., 2020; Park et al., 2021; Liu al., 2022c) learn the optimal

---

Published as a conference paper at ICLR 2024
SuegleA
HiACp
PHix
Rokieii
Andeliis
Top-1 Accuracy(%)
xx 50
Saepled
Staodbiiic
Iter
Figure 1: Mixed images of various approaches. (a) Accuracy of ResNet18 trained by different mixup approaches with 200 epochs on CIFARIO0. (b) Mixed images of various mixup-based approaches.
mixing policy by maximizing the saliency regions; 3) Automatic Mixup-based augmentation ap- proaches, that learn a model, e.g. DL model, instead of a policy, to automatically generate mixed images. (Liu al., 2022d) for example, proposed an AutoMix model for DA, consisting of tar- get classifier and a generative network, to automatically generate mixed samples to train a robust classifier by alternatively optimizing the target classifier and the generative network.
The handcrafted mixup augmentation approaches, however, randomly mix images without consid- ering their contexts and labels. The target objects, therefore, may be missed in the mixed images, resulting in a label mismatch problem. maliency-guided-basee mixup augmentation methods can al- leviate the problem as the images are combined with supervising information, namely the maximum saliency region. These mixup models, related to the first two categories above, share the same learn- ing paradigm: an augmented training dataset generated by random or learnable mixing policy and a DL model for image classification. As image generation is not directly related to the target task, iee. classification, the generated images guided by human prior knowledge, i.e. saliency- sllsency-based,, may not be effective for target network training. Moreover, it is impossible to generate all possible mixed instances for target training. The randomly selected synthesized samples thus may not be represen- tative of the classification task, ultimately degrading classifier generalization. Besides, such gener- ated samples will be input to the target network repeatedly, resulting in inevitable overfitting over long epoch training. To overcome these problems, automatic mixup-based augmentation approaches generate augmented images by a sub-network with a good complexity-acuuren proach comprises two sub-tasks: mixed sample generation module and classification module, conjointly optimized by minimizing the classification loss in an end-to-end way. As the optimiz- ing goal is consistent for the two sub-tasks, however, the generation module may not be effectively guided and may produce, consequently, simple mixed samples to achieve such goal, which limits sample diversification. The classifier trained on such simple examples is prone, therefore, to suffer from overfitting, leading to poor generalization performance on the testing set. Another limitation is that current automatic mixup approaches mix two images only for image generation, where the rich and discriminating information is not efficiently exploited.
trade-off. This ap-
To solve these problems, we propose in this paper AdAutomixup, an adversarial automatic mixup augmentation approach to automatically generate mixed samples with adversarial training in an end-to-end way, as shown in Fig. 2. First, an attention-based generator is investigated to dynami- cally learn discriminating pixels from a sample pair associated with the corresponding mixed labels. Second, we combine the attention-basee network, where the generator and the classifier are alternatively updated by adversarial training. Unlike AutoMix (Liu et al., 2022d). generator is learned to increase the training loss of the tar- get network through generating adversarial samples, while the classifier learns more robust features from hard examples to improve generalization. Furthermore, any set of images, instead of two im- ages only, can be taken as an input to our generator for mixing image generation, which results in more diversification of the mixed samples. Our main contributions are summarized as follows.
generator with the target classifier to build an adversarial
(a) We present an online data mixing approach based on an adversarial learning policy, trained end-to-end to automatically produce mixed samples.
(b) We propose an adversarial framework to jointly optimize the target network training and the mixup sample generator. The generator aims to produce hard samples to increase the target net- work loss while the target network, trained on such hard samples, learns a robust representation to improve classification. To avoid the collapse of the inherent meanings of images, we apply exponential moving average (EMA) and cosine similarity to reduce the search space.
(c) We explore an attention-based mix sample generator that can combine multiple samples instead of only two samples generate mixed samples. The generator is flexible as its architecture is not changed with the increase of input images.
2

---

Published as a conference paper at ICLR 2024
2
RELATED WORK
Hand-crafted based mixup augumentaion Mixup (Zhang et al., 2017), the first hybrid data aug- mentation method, generates mixed samples by subtracting any two samples and their one-hot la- bels. ManifoldMixup (Verma et al., 2019) extended this mixup from input space to feature space. To exploit spatial locality, CutMix (Yun et al., 2019) crops out region and replace it with a patch of another image. To improve MixUp and CutMix, FMix (Harris et al., 2020) uses random binary masks obtained by applying threshold to low-frequency images sampled from the frequency space. RecursiveMix (Yang et al., 2022) iteratively resizes the input image patch from the previous itera- tion and pastes it into the current patch. To solve the strong "edge" problem caused by CutMix, SmoothMix (Jeong et al., 2021) blends mixed images based on soft edges, with the training labels computed accordingly.
Saliency guided based mixup augmentation SalieneyMix (Uddin et al., 2020), SnapMix (Huang et al., 2020) and Attentive-CutMix (Walawalkar et al., 2020) generate mixed images based on the salient region detected by the Class Activation Mapping(CAM) (Selvaraju et al., 2019) or saliency detector. Similarly, PuzzleMix (Kim et al., 2020) and Co-Mixup (Kim et al., 2021) propose op- timization strategy to obtain the optimal mask by maximizing the sample salieney region. These approaches, however, suffer from lack of sample diversification as they always deterministically select regions with maximum saliency. To solve the problem, Saliency Grafting (Park et al., 2021) scales and thresholds the saliency map grant all salient regions are considered increase sam- ple diversity. Inspired by the success of Vit (Dosovitskiy et al., 2021; Liu et al., 2021) in com- puter vision, adaptive mixing policies based on attentive maps, e.g. TransMix (Chen et al., 2021), TokenMix (Liu et al., 2022a), TokenMixup (Choi et al., 2022), MixPro (Zhao et al., 2023), and SMMix (Chen et al., 2022), were proposed to generate mixed images.
Automatic Mixup based augmentation Mixup approaches the first two categories allow a trade-off between precise mixing policies and optimization complexity, the image mixing task is not directly related to the target classification task during the training process. To solve this prob- lem, AutoMix (Liu et al., 2022d) divides the mixup classification into two sub-tasks, mixed sample generation and mixup classification, and proposes an automatic mixup framework where the two sub-tasks optimized jointly, instead of independently. During training, the generator continu- ously produces the mixed samples while the target classifier is preserved for classification. In recent years, adversarial data augmentation (Zhao et al., 2020) and generative adversarial networks (Anto- niou et al., 2017) were proposed to automatically generate images for data augmentation. To solve the domain shift problem, Adversarial MixUp (Zhang al., 2023; Xu et al., 2019) have been inves- tigated to synthesize mix samples or features for domain adaptation. Although there are very few works for automatic mixup, it will become a research trend in the future.
3 ADAUTOMIX
In this section, present the implementation of AdAutoMix, which is composed of target clas- sifier and a generator, as shown in Fig. 2. First, we introduce the mixup classification problem and define the loss functions. Then, we detail our attention-based generator that learns dynamically the augmentation mask policy for image generation. Finally, we show how the target classifier and the generator are jointly optimized in an end-to-end way.
3.1 DEEP BHARNING-BASER CLASSIFIERS
Assume that S = [a.|s- = 1,2, 2.., is a training set, where S is the number of the images. We
select any N samples from S to obtain sample set X { #N, with Y = (y1, -31,32, y.. its corresponding label set. Let vw be any feature extraction model, e.g., ResNet (He et al., 2016), where W is a trainable weight vector. The classifier maps example x E X into label y Y. A DL classifier zh is implemented to predict the posterior class probability, and W are learned by minimizing the classification loss, i.e. the cross entropy (CE) loss in El):
Loe (tOw, vv) ylogg w (x)).
3

---

Published as a conference paper at ICLR 2024
Update a target model
Fixed Generator module Gg(-]
Dalasels
Feature map
Updated
Tarpet module shy (
Mix block
mintleres abeees albers
Concat
Allernatively
Update a generator model
Updated
Fixed
Gemerator module Gp(-)
Feeture mep
map
module
Myy [+]
p....
Mix block
module
Tarpet
mislshonse (1 Blhcounn
Concet
Shp
Figure 2: Illustration of AdAutoMix framework. AdAutoMix consists of a generator module and target module, which are alternatively trained end-to-end. The generator module aims to produce hard samples to challenge the target network while the target network, trained on such hard samples, learns robust feature representation for classification.
For N samples in sample set X. the average cross-entropy (ACE) loss is computed by Eq,
Luce (1bw, Y)
(1o((//(11))) Un) * An). 1==1
where is the scalar multiplication. In the mixup classification task, we input any N images associated with mixed ratios À to generator Go(-) that outputs mixed sample Tmirs de- fined in (8) from section 3.2. Similarly, the label for such mixed image Imiz is obtained by Umin En Un ④ An. vww is optimized by average mixup cross-entropy (AMCE) loss in (3):
Lamce (tb,, Y) = Lee ( (tw (Fmix) ) Umiz).
(3)
Similarly, we also compute the mixup cross-entropy (MCE) by Eq.
T.meo (V/w, (mnen) = Tooe (1ow Σ (In An)) Mmmix)
(4)
3.2 GENERATOR
As described in Section most existing approaches mix two samples by manually designed policies automatic learning policies, which results in insufficient exploitation of the supervised informa- tion that might be provided by the training samples for data augmentation. In our work. we present a universal generation framework to extend the two-image mixing multiple-image mixing. To learn a robust mixing policy matrix, we leverage self-attention mechanism to propose an attention-based 1,2,... N) is a sample set with N original training samples and Y = (Y./n 1, ..., N} are the correspond- ing labels. We define À {Ai, .... AN as the mixed ratio for the images with their sum constrained to be equal to 1. As shown in Fig. 3, each image in an image set is first mapped to feature map with encoder Eg, which is updated by an exponential moving average of the target classifier, i.. E + (1 - W", where W' is the partial weight of the target classifier. In our experiments, existing classifiers. ResNet18, ResNet34, and ResNeXt50, are used as target classi- fiers, and W' is the weight vector of the first three layers in the target classifier. Then, the mixed ratios are embedded into the resulting feature map to enable the generator to learn mask policies for image mixing. For example, given nith image In E RW x where W and H represent image size, we input to an encoder and take outputs from its Ith layer as feature map 2/ E RCxuxh where C' is the number of channels, and w and h represent map size. Then, we build matrix with size w x with all values equal to I, multiplied by the corresponding ratio An to obtain embedding matrix My.- We embed An with the Ith feature map in simple and efficient way by concatenating 2 concat(M, 2,) R x)x The embedding feature map 2{_ is mapped to three em- bedding vectors by three CNNs with x 1 kernel (as shown in Fig. 3), respectively. Therefore, we obtain three vectors gas kns and e. for the nthh image Fn. Note that the channel number is reduced
mixed sample generator, as shown in Fig. 3. As described in Section 3.1, X = {Ir,,
4

---

Published as a conference paper at ICLR 2024
CAB module
Cony 1=1
Faature maap
CA3
8
X Metrell
kc
SettMax
Cany 1x1
Add
[0. 10.1,.m.
Upainggi
Cany 1x]
%
Figure 3: Mixed module: the cross attention block (CAB), used to learn the policy matrix for each image, is combined with v, ..,,...,N N) values to compute the policy matrix for image mixing.
to its half for On and hn to save computation time and is set to 1 for Un- In this way, the embedding vectors of all images computed and denoted by q1, 92."..0, K1, k2....,N. and 21 Ug The cross attention block (CAB) (as shown in Fig. 3) for the 1ith image is computed by Eq. (5):
Lait on hi
Pr = Softmar
Un
(5)
Vd
where d is the normalization term. We concatenate N attention matrices by Eq. (6):
(ComecalCCooal P= Sofooorr(((CC. (Concal (P1, Pa,..,, Pw)).
(6)
The matrix P RNxwhxuk is resized to P E RN*W*H by upsampling. We split N matrices, Pl, P1 2..,.PPP Ply from P', treated as mask policy matrices to mix images in the sample set X by (7.
Zmir Σ Ia O
where @ denotes the Hadamard product. To facilitate representation, the mixed image generation procedure is denoted generator Gg by Eq.(8):
Emiz
Go (X,
(8)
where 0 represents all the learnable parameters of the generator.
3.3 ADVERSARIAL AUGMENTATION
This section provides the adversarial framework propose to jointly optimize the target network w/w and the generator Go through adversarial learning. Concretely, the generator Ga attempts to produce augmented mixed image set increase the loss of target network v'w while target network v'w aims to minimize the classification loss. An equilibrium can be reached where the learned representation reaches maximized performance.
3.3.1 ADVERSARIAL LOSS
As shown in (8), the generator takes X and the of mixed ratio À input and outputs synthe- sized image Zinix to challenge the target classifier. The latter receives either a real or a synthesized image from the generator as its input and then predicts its probability of belonging to each class. The adversarial loss is defined by the following minimax problem to train both players by Eq.
W* 0r =argminmaxt
Lamce (vw, Y)]]. XES
0
where S and X are the training set and image set, respectively. A robust classifier should correctly classify not only the mixed images, but also the original ones, we incorporate two regularization terms Lmnee (+/ (Eminss Mmiz)) )) and (Vow. Y) to enhance performance. Accordingly, the objec-
tive function rewritten as shown by Eq.(10):
W* 0r =argminmax| E [Lamce(tve Y) + aLmmce Ymix) + (1 - La)c (Uw, Y)J]. (10)
KES
To optimize parameter 9, G.(.) produces images with given image sets to challenge the classifier. is possible, therefore, that the inherent meanings of images (i.e. their semantic meaning) collapse.
5

---

Published as a conference paper at ICLR 2024
To tackle this issue, introduce cosine similarity and teacher model as two regularization terms to control the quality of mixed images. The loss is changed accordingly. as shown by Eq.(11):
W 0 argminmax
Lamce (1W, Y) + aLmmce Fmix) = (1 - a)Lacce (tw, Y) X
E
(11)
gameo (W +1 Lecsiiill))
where Leosine En cosine( 1hpp (Xmin)) uh ( An. cosine(.) is cosine similarity function, and "' teacher model whose weights are updated as an exponential moving average of the
target (EMA) models weights, i.e. W < ς (1-5) Notice that Lee(ww y) is the standard cross-entropy loss. (1',, Y) loss facilitates the backbone to provide a stable feature map at early stage that it speeds up convergence. Target loss Lamce Y) aims to learn task-relevan information the generated mixed samples. Lmce (s'w, Fmii) facilitates the capture of task-relevant information in the original mixed samples. Teasine and Lamcc W) are used to control the quality of generation mix images.
3.4 ADVERSARIAL OPTIMIZATION
Similarly many existing adversarial training algorithms, is hard directly find a saddle point (W*, 0*) solution to the minimax problem in (q)(1 Alternatively, a pair of gradient descent and ascent are employed to update the target network and the generator.
Consider target classifier ziw (-) with loss function L-e(-), where the trained generator Go(+) maps multiple original samples to mixed sample. The learning process of the target network can be defined as the minimization problem in (12):
W =argmin [Lance (vw, Y) + aLmcce (t'w, (mnie) + (1 a LLLcace (vw, Y)
XES
(12)
Lammm (w Y) + (1 (1--))1ae Loosiei
The problem in Eq. (12) is usually solved by vanilla SGD with learning rate of 5 and a batch size of B, and the training procedure for each batch can be computed by (13))
W(t + (t) JVW
1 K
Σ
Lamoe (vw, Y) + Aummeeeee Mmice) + (1 a a)) (1bw, Y) (13)
BT amce (1/w, V) +1+(1- 3)
where K is the number of mixed images or image sets produced from patch set B. As the cosine similarity and the teacher model are independent of W, (1.(13) can be rewritten as (q.()
K
(1++1)=W(0 = 5V
Σ
Lamoe Y) + a.Lmme (thw, Umin) + (1 - L (Vw. Y)] (14)
K
Note that the training procedure be regarded as average over K instances of gradient com- putation, which can reduce gradient variance and accelerate the convergence of the target network. However, training may suffer easily from over-fitting due to the limited training data over long training epoch. To overcome this problem. different from AutoMix (Liu et al., 2022d), our mixup augmentation generator generates set of harder mixed samples increase the loss of the target classifier, which results in minimax problem to self-train the network. Such a self-supervised ob- jective may be sufficiently challenging to prevent the target classifier from overfitting the objective. Therefore, the objective is defined as the following maximization problem in (15):
0" =argmax E
[Lamco(//:: Y) LLamc (yi7 Y) + (1 Lecsiiee]]]
(15)
XEB
To solve the above problem, employ gradient ascent to update the parameter with learning rate of 7, which is defined in (16))
K
0 1) 0(1) + +
1 K
Σ
(1/m Y) am (vip, Y) (1 Leosinen
(16)
Intuitively, the optimization of (16) is the combination of two sub-tasks, the maximization of I (1w (Emiz Umiz)) and the minimization of amae (1 Y) - 001-130eccsinmn This tends to push

---

Published as a conference paper at ICLR 2024
the synthesized mixed samples far away from the real samples to increase diversity, while ensuring the synthesized mixed samples are recognizable for a teacher model and kept, within a constraint to avoid collapsing the inherent
similarity to the feature representation of original images, meanings of images. This scheme enables generating challenging samples by closely tracking the updates of the classifier. We provide some mixed samples in Appendix B.2 and B.3.
EXPERIMENTS
To estimate our approach performance, we conducted extensive experiments seven classifica- tion benchmarks, namely CIFARI00 (Krizhevsky al., 2009), Tiny-ImageNet (Chrabaszez al., 2017), ImageNet-1K (Krizhevsky et al., 2012), CUB-200 (Wah et al., 201 1), FGVC-Airerati (Maji al., 2013) and Standford-Cars (Krause et al., 2013) (Appendix A. For fair assessment, we compare our AdAutoMixup with some current Mixup methods, i.e. Mixup (Zhang et al., 2017), CutMix (Yun et al., 2019), ManifoldMix (Verma et al., 2019), FMix (Harris et al., 2020), Re- sizeMix (Qin et al., 2020), SaliencyMix (Uddin et al., 2020), PuzzleMix (Kim et al., 2020) and AutoMix (Liu et al., 2022d). To verify our approach generalizability, five baseline networks, namely ResNet18, ResNet34, ResNet50 (He et al., 2016). ResNeXt50 (Xie al., 2017), Swin- Transformer (Liu et al., 2021) and ConvNeXt(Liu et al., 2022b), are used to compute classification accuracy. We have implemented our algorithm 2022b). Some common parameters follow the experimental settings of AutoMix and we provide our own hyperparameters in Appendix A.2. For all classification results, we report the mean perfor- mance of 3 trials where the median of top-1 I test accuracy in the last 10 training epochs is recorded for each trial. To facilitate comparison, we mark the best and second best results in bold and cyan.
the open-source library OpenMixup (Li al.,
4.1 CLASSIFICATION RESULTS
4.1.1 DATASET CLASSIFICATION
We first train R8 and ResNeX150 on CIFARI00 for 800 epochs, using the following perimental setting: The basic learning rate is 0.1, dynamically adjusted by cosine scheduler, SGD (Loshchilov & Hutter, 2016) optimizer with momentum of 0.9, weight decay of 0.0001, batch size of 100. To train ViT-based models, e.g. Swin-Tiny Transformer and ConvNeXt-Ting we train them with AdamW (Loshchilov & Hutter, 2019) optimizer with weight decay of 0.05, batch size of 100, 200 epochs. On Tiny-ImageNet, except for a learning rate of 0.2 and training over 400 train
epochs, training settings similar to the ResNet18, ResNet34 and ResNet50 for 100 epochs using PyTorch-style setting. The experiments implementation details are provided in Appendix A.3
used in CIFAR100 On ImageNet-1K,
Table and Fig. show that our method outperforms the existing approaches on CIFARI00. After training by our approach, ResNet18 and ResNeXt50 achieve an accuracy improvement of 0.28% and 0.58% w.r.t the second best results, respectively. Similarly. ViT-based approaches achieve the highest classification accuracy of 84.33 % and 83.54% and outperform the previous best approaches with an improvement of 1.66% and 0.24%. On the Iiny-ImageNet datasets, our AdAutoMix consistently outperforms existing approaches terms of improving the classification performance ResNet18 and ResNeXt50, i.e. 1.86 & and 2.17% significant improvement w.r.t the second best approaches. Also, Table 1 shows that AdAutoMix achieves an accuracy improvement (0.36% for ResNet18, 0.3% for ResNet34, and 0.13% ResNet50) the ImageNet- large scale dataset.
Table 1: Top-1 accuracy (%) of mixup methods on CIFAR-100, Imageee and ImageNet- 1..
10
CIFAR100
Imag-Im
ImageNet-1k -
Method
ResNet18
ResNeX150
Swin-T
Conv NeXi-T
ResNet18
ResNeXi50
ResNet18
ResNet34
ResNet50
Vanilaa
78.04
81.09
78.41
78.70
61.68
65.04
70.04
73.85
76.83
MixUp
79.12
82.10
76.78
81.13
63.86
66.36
69.98
73.97
77.12
CutMix
78.17
80.64
65.53
66,47
68.95
73.58
77.17
SaliencyMix
79.12
81.53
80.40
82.82
64.60
66.55
69.16
73.56
77.14
FMix
79.69
81.90
80.72
81.79
63.47
65.08
69.96
74.08
77.19
PuzzleMix
81.13
82.85
80.33
82.29
65.81
67.83
70.12
74.26
77.54
ResizeMix
80.01
81.82
80.16
82,53
63.74
65.87
69,50
73.88
77.42
AutoMix
82.04
83.64
82.67
83.30
67.33
70.72
70.50
74.52
77.91
AdAutoMix
82.32
84,22
84,33
83.54
69.19
72.89
70,86
74.82
78.04
Gain
+0.28
+0.58
+1.66
+0.24
+1.86
+2.17
+0.36
+0.30
+0.13

---

Published as a conference paper at ICLR 2024
Table 2: Accuracy (%)f of mixup approaches on CUB-200, FGVC-Aireratt and Standford-
CUB-200
FGVC-Anerafts
Standford-Cars
Method
ResNet18
ResNe150
ResNet18
ResNeXt50
ResNet18
ResNeXt50
Vanilla
77.68
82.38
80.23
85,10
86.32
90.13
MixUp
78.39
82.98
79.52
85.18
86.27
90.81
CutMix
78.40
83.17
78.84
84.55
87 48
91.22
ManifoldMix
79.76
83.76
80.68
86.60
85.85
90.20
MaliencyMix
77.95
81.71
80.02
84.31
86.48
90.60
FMix
77.28
83.34
79.36
86.23
87.55
90,90
PuzzleMix
78.63
83,83
80.76
86.23
87.78
91.29
ResizeMix
78.50
83.41
78.10
84.08
88.17
91.36
AutoMix
79.87
83.88
81.37
86.72
88.89
91.38
AdAutoMiN
80.88
84.57
81.73
87.16
89.19
91.59
Gain
+1.01
+0.69
+0.36
+0,44
+0.30
+0.21
4.1.2 FINE-GRAINED CLASSIFICATION
On CUB-200, FGVC-Aircrafts, and Standford-Cars, we fine-tune pretrained ResNet18, ResNet50, and ResNeXi50 using SGD optimizer with momentum of 0.9, weight decay of 0.0005, batch size of 16, 200 epochs, learning rate of 0.001, dynamically adjusted by cosine scheduler. The results in Table 2 show that AdAutoMii achieves the best performance and significantly improves the perfor- mance of vanilla (3.20% on CUB-200, 1.5%/2.06% on Aircraft and 2.87% on Cras), which implies that AdAutoMix also robust to more challenging scenarios.
4.2 CALIBRATION
DNNs are prone suffer from getting overconfident in classification tasks. Mixup methods can effectively alleviate this problem. To this end, we compute the expected calibration error (ECE) of various mixup approaches on the CIFARI00 dataset. It can be seen from the experimental results in Fig. 4 that our method achieves the lowest ECE, i.e. 3.2%, w.r.t existing approaches. We provide more experimental results in Table Appendix A.5
MiXUp
CutMix
FMix
GridMMix
PuzzeMix
AutoMix
AdAutoMix
Acouaay
DCE-077
Confidence
Figure 4: Calibration plots of Mixup variant CIFARI00 using ResNer18.
4.3 ROBUSTNESS
We carried out experiments on CIFAR100-C (Hendrycks & Dietterich, 2019) to verify robustness against corruption. A corrupted dataset is manually generated include 19 different corruption types (noise, blur, fog, brightness, etc.). We compare our AdAutoMix with some popular mixup algorithms: CutMix, FMix, PuzzleMix, and AutoMix. Table shows that our approach achieves the highest recognition accuracy for both clean and corrupted data, i.e. 1.53% and 0.40% classification accuracy improvement w.r.t AutoMix. We further investigate robustness against the FGSM (Good- fellow et al., 2015) white box attack of 8/255 loo epsilon ball following (Zhang et al., 2017). Our AdAutoMix significantly outperforms existing methods, as shown Table 4.
4.4 OCCLUSION ROBUSTNESS
Sverlay Tranfora Janke Nacktoo
Sardine Paich
To analyze the AdAutoMix robustness against random occlusion (Naseer et al., 2021), we build image sets by randomly masking images from datasets CIFAR100 and CUB200 with 16x patches, using different mask ratios (0- 100%). We input the resulting occluded images into two classifiers, Swin-Tiny Transformer and ResNet-50, trained by various Mixup models to compute test accuracy. From the results in Fig. 5 and in Table in Appendix A.6, we observe that MiXutoMix achieves the highest accuracy with different occlusion ratios.
Cockace (83)
Figure 5: Robustness against image occlusion
with different occlusion ratios.
8

---

Published as a conference paper at ICLR 2024
Table 3: Top-1 accuracy (%)t
with ResNet50 on CUB200 and Standford-Can
Dataset
Vanilla
MixUp
CutMix
PuzzleMix
AutoMix
AdAutoMix
CUB
81.76
82.79
81.67
82.59
82.93
83.36(+0.43)
Cars
88.88
89.45
88.99
89.37
88.71
89.65(+0.20)
4.5 TRANSFER LEARNING
We further study the transferable abilities of the features learned by AdAutoMix for downstream classification tasks. The experimental settings in subsection 4.1.2 are used for transfer learning on CUB-200 and Standford-Cars except for training now over 100 epochs. ResNet50 trained on ImageNet- IK is finetuned on CUB200 and Standford-Cars for classification. Table shows that AixutoMix achieves the best performance, which proves the efficacy of our approach for down- stream tasks.
4.6 ABLATION EXPERIMENT
Table 4: Top-1 aceuracy and FGSM er-
ror of ResNet18 with other methods.
In AdAutoMix, four hyperparameters, namely the num- ber of input images N, the weights a, 3, and mixed ratios X, which are important achieve high performance, are fixed in all experiments. To save time, we train the clas- sifier 8 for 200 epochs by our AdAutoMixup. The accuracy of R with different o, (3, N, and are shown in Fig. (a), (b), (c), and (d). Also, the classification accuracy of AdAutoMixup with different and N are depicted in Table. 9 and Table. 10 in Ap- pendix A.8. AdAutoMix, with N=3, a =0.5, f=0.3, and =I default, achieves the best performances on the various datasets. In addition, two regularization terms, Lywe (y/W,mui)) and LVre Y), attempt to improve classifier robustness, and another two regularization terms, namely cosine similarity Leosine and EMA model Lamce (w Y). aim to avoid the collapsing of the inherent meaning of images in AdAutoMix. We thus carry experiments to evaluate the performance of each module concerning classifier performance improvement. To facilitate the description, we remove the four modules from AiXutoMix and denote the resulting approach as basic AdAutoMix. Then,
Cleun
Coruuption
FGSM
Method
Acc($))
Acc(S)]
Frro((S)) X
CulMix
79.45
46.66
88.24
FMix
78.91
50.58
88.35
PuzzleMix
79.96
51.04
80.52
AutoMix
80.02
50.75
82.67
AdAutoMix
81.55
51.44
75.66
Table 5:
Ablation experiments on based on ResNet18 and
CIFARIO0 100 ResNext50.
CIFART00
Method
ResNet18
ResNeX150
Base 3)
79.38
82.84
+0. + 0.5Lace
80.04
84.12
-.33 + O.
81.55
84.40
gradually incor- porate the two modules Umee(ww: Y) and Lace (1/w, Y). and the two modules Lamce (177 Y) and Leosine. and compute the classification accuracy. The experimental results in Table. show that the Ly/,, Fmiz) and Lace (2/w. Y) improve classifier accuracy by about 0.66%. However, after in- corporating Lemce (v'p. Y) and Leosine to constraint the synthesized mixed images, we observe that the classification accuracy is significantly increased, namely 1.51% accuracy improvement, which implies that these two modules are capable of controlling the quality of generated images in the adversarial training. Also, we show the accuracy of our approach with gradually increasing indi- vidual regularization terms in Table. in the Appendix. A.8. There is a similar trend that each
regularization term improves accuracy.
(a). weight
(b). B weight
(c). N samples
(d). ratio
1op-1Acm Aocuray (%)
Narten
Pack
Mester
Nutto Auc Taa
Hyperparameter sensitivity analysis
Figure 6: Ablation of hyperparameter a, B, input samples and À of AdAutoMix on CIFAR100.
CONCLUSION
In this paper, we have proposed AdrutoMirup, a framework that jointly optimizes the target clas- sifier and the mixing image generator in an adversarial way. Specifically, the generator produces hard mixed samples to increase the classification loss while the classifier is trained on the hard sam- plcs to improve generalization. In addition, the generator can handle multiple sample mixing cases. The experimental results the six datasets demonstrate the efficacy of our approach.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported in part by the Scientific Innovation 2030 Major Project for New Generation of Al under Grant 2020AAA0107300, in part by the National Natural Science Foundation of China (Grant No. 61976030), the Science Fund for Creative Research Groups of the Chongqing University (Grant No. CXQT21034),
part by the National Natural Science Foundation of China (Grant No. 62221005), in part by the National Natural Science Foundation of China (Grant No. U22A2096), part by the Research on JY human-machine hybrid enhanced intelligence theory and method for command and decision-making (Grant No. 8091B0121 (2), and in part by the Fund of Henan Provincial Science and Technology Department (Grant No. 222102210301). We thank all members who contribute to the Open community.
REFERENCES
Antreas Antoniou, Amos J. Storkey, and Harrison Edwards. Data augmentation generative adver- sarial networks. ArXiv, abs/1711.043400 2017.
Alexey Bochkovskiy, Chien-Yae Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. ArXiv, abs/2004. 10934, 2020.
Jie-Neng Chen, Shuyang Sun, Ju He, Philip Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers, 2021. 3
Mengzhao Chen, Mingbao Lin, Zhihang Lin, Yu xin Zhang, Fei Chao, and Rongrong Ji. Smmix: Self-motivated image mixing for vision transformers. ArXiv, abs/2212.12977: 2022. 3
Hyeong Kyu Choi, Joonmyung Choi, and Hyunwoo Kim. Tokenmixup: Efficient attention-guida
token-level data augmentation for transformers. ArXiv, abs/2210.07562. 2022. 3
Patryk Chrabaszez, Iya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an
alternative to the cifar datasets. arXiv preprint xrXiv:1707.08819 2017. 7, 14
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 19X5v11805,09501, 2018.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Terrance Vrics and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1oo.ii:::08..... 04552, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Jiemin Fang, Yuzhu Sun, Kangjian Peng. Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. Fast neural network adaptation via parameter remapping and architecture search. arXiv preprint arXiv:2001.02525, 2020.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference Learning Representations (ICLR), 2015.
Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, and Adam Prügel-
Bennett Jonathon Hare. Fmix: Enhancing mixed sample data augmentation. arXiv preprint arXiv:2002. /2047, 2(3):4, 2020. 3,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. 3,
10

---

Published as a conference paper at ICLR 2024
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. arXiv preprint arXiv: 2019. 8
Shaoli Huang, Xinchao Wang, and Dacheng Tao. Snapmix: Semantically proportional mixing for augmenting fine-grained data. AAAI Conference on Artificial Intelligence, 2020.
Jongheon Jeong, Sejun Park. Minkyu Kim, Heung-Chang Lee, Do-Guk Kim, and Jinwoo Shin. Smoothmix: Training confidence-calibratee smoothed classifiers for certified robustness. In Neu- ral Information Processing Systems, 2021. 3
lang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pp. 5275-5285. PMLR, 2020. 7.
Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity, arXiv preprini arXiv:2102.088 13200365, 2021. 3
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for line-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR- 13), 2013. 14
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7, 14
Alex Krizhevsky, Ila Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012. 1, 7, 14
Siyuan Li, Zicheng Liu, Zedong Wang. Di Wu, Zihan Liu, and Stan Z. Li. Boosting discriminative visual representation learning with scenario agnosii mixup. ArXiv, abs/2111.1555544 2021. 1
Siyuan Li, Zedong Wang, Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiang- bin Zheng, and Stan Z. Li. Moganet: Multi-order gated aggregation network. arXiv preprint arXiv:2211.03295, 2022a.
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, and Stan Z. Li. Openmixup: Open mixup toolbox and benchmark for visual representation learning. arviv preprint arXiv:2209.04851, 2022b. 7
Siyuan Li, Di Wu, Fang Wu, Zelin Zang, Kai Wang, Lei Shang, Baigui Sun, Haoyang Li, and Z.Li. Architecture-agnostic masked image modeling from vit back to cnn. arXiv preprint arXiv:2205.13943 2022c.
Siyuan Li. Weiyang Jin, Zedong Wang, Fang Wu, Zicheng Liu, Cheng Tan, and Stan Z. Li. Semire- ward: A general reward model for semi-supervised learning. In International Conference on Learning Representations, 2024.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Differentiable automatic data augmentation. In European Conference Computer Vision, pp. 580-595. Springer, 2020. 1, 16
Jihao Liu, B. Liu, Hang Zhou, Hongsheng Li, and Yu Liu. Tokenmix: Rethinking image mixing for data augmentation in vision transformers. European Conference on Computer Vision, 2022a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Con- ference on Computer Vision (ICCV), 2021. 3,7
Zhuang Liu, Hanzi Mao, Chao-Yure Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s, 2022b. 7
Zicheng Liu, Siyuan Li, Ge Wang, Cheng Tan, Lirong Wu, and Stan Li. Decoupled mixup for data-efficient learning. ArXiv, abs/2208.10761. 2022c.

---

Published as a conference paper at ICLR 2024
Zicheng Liu, Siyuan Li, Di Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, and Stan Z Li. Automix: Unveiling the power of mixup for stronger classifiers. In European Conference on Computer Vision, pp. 441-458. Springer, 2022d. 2, 2,
Iyya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.033: 2016. 7
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5157, 2013. 7, 14
Muzammal Naseer, Kanchana Ranasinghe, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. In Neural Information Processing Systems, 2021. 8
Joonhyung Park, June Yong Yang, Jinwoo Shin, Sung Ju Hwang, and Eunho Yang. Saliency graft- ing: Innocuous attribution attribution-guided guided mixup with calibrated label mixing. In AAAI Conference on Artificial Intelligence, 2021. 1,3
Jie Qin, Jiemin Fang, Qian Zhang, Wenyu Liu, Xingang Wang, and Xinggang Wang. Resizemix: Mixing data with preserved object information and true labels. arXiv preprint 20.i::0022.11001, 2020. 1,7 7
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint 1rX:v:1506.01497, 2015. 1
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. arXiv preprint 161.:2161,.02391, 2019. 3
Connor Shorten and Taghi M Khoshgoftaar. survey on image data augmentation for deep learning- Journal of Big Data, 6(1):1-48, 2019. 1
AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, et al. Salien- cymix: A saliency guided data augmentation strategy for better regularization. arXiv preprint SEKV:2000601799. 2020. 3.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp.
2019. 3,7
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. California Institute of Technology, 2011. 7, 14
Devesh Walawalkar, Zhiqiang Shen, Zechun Liu, and Marios Savvides. Attentive cutmix: An en- hanced data augmentation approach for deep learning based image classification. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 3642-3646, 2020. 1,3
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang- Adversarial domain adaptation with domain mixup. AAAI Conference on Artificial Intelligence, 2019. 3
Lingfeng Yang, Xiang Li, Borui Zhao, Renjie Song, and Jian Yang. Recursivemix: Mixed learning with history. ArXiv, 00/2203306844 2022. 3
12

---

Published as a conference paper at ICLR 2024
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceed- ings of the International Conference Computer Vision (ICCV), pp. 6023-6032, 2019. 1, 3,
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. 2017. 7, 8, 16
Jiajin Zhang, Hanqing Chao, Amit Dhurandhar, Pin- Yu Chen, Ali Tajer, Yangyang Xu, and Pingkun Yan. Spectral adversarial mixup for few-shot unsupervised domain adaptation. ArXiv, abs/2309.01207. 2023. 3, 16
Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entropy adversarial data aug- mentation for improved generalization and robustmess. ArXiv, abs/2010.08001, 2020. 3, 16
QiHao Zhao, Yangyu Huang, Wei Hu, Fan Zhang, and J. Liu. Mixpro: Data augmentation with maskmix and progressive attention labeling for vision transformer. ArXiv, abs//304. 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.I DATASET INFORMATION
We briefly introduce image datasets used in this paper. (1) CIFAR-100 (Krizhevsky al., 2009) contains 50,000 training images and 10,000 test images in 32 x 32 resolutions, with 100 class settings. (2) Tiny-ImageNet (Chrabaszez et al., 2017) contains 10,000 training images and 10,000
validation images of 200 classes in 64 x 64 resolutions. (3) ImageNet- IK (Krizhevsky et al.. 2012)
contains 1,281,167 training images and 50,000 validation images of 1000 classes. (4) CUB-200- 2011 I (Wah et al., 201 I) contains 1 11,788 images from 200 wild bird species. FGVC-Airerafts (Maji al., 2013) contains 10,000 images of 100 classes of aricrafts and Standford- -ars (Krause et al., 2013) contains 8, 1.4 training images and 8,041 test images of 196 classes.
A.2 EXPERIMENTS HYPERPARAMETEN DETAILS
In our work, the feature layer / is set to 3, and the momentum coefficient starts from & 0.999 and increased to in a cosine curve. Also, AdAutoMix uses the same set of hyperparameters in all experiments follows: a=0.5, 3=0.3, >=1.0, N=3 or N=2.
A.3 EXPERIMENTS IMPLEMENTATION DETAILS
On CIFARI00, RandomFlip and RandomCrop with 4-pixel padding are used as basic data aug- mentations for images with size experimental setting: SGD optimizer with momentum of 0.9, weight decay of 0.0001, batch size of 100, and training with 800 epochs. The basic learning rate is 0.1, dynamically adjusted by the co- sine scheduler; CIFAR version of ResNet variants are used, i.e., replacing the 7 x convolution and MaxPooling by a 3 3 convolution. To train Vit-based approaches, e.g. Swin-Tiny Transformer, we resize images to 224 224 and train them with Adam optimizer with weight decay of 0.05, batch size of 100, and total training 200 epochs. The basic learning rate is 0.0005, dynamically adjusted by the cosine scheduler. For ConyNeXt-Ting training, the images keep the 32 x 32 resolution, and we train it based on the setting of Vit-based approaches except for the basic learning rate of 0.002. the a and are set 0.5 and 0.3 for CIFAR on ResNet18 and ResNeXt50.
32. For ResNet18 and ResNeXt50, we use the following
On Tiny-Imagel NeageNet, RandomFlip and RandomResizedCrop for 64 x 64 are used as basic data aug- menting. Except for learning rate of 0.2 and training over 400 epochs, training settings are similar to the ones used in CIFAR100.
On ImageNet- IK, use a Pytorch-style training setup. which optimizes the model for 100 epochs by SGD optimizer with a batch size of 256, basic learning rate of 0.1, the SGD weight decay of 0.0001, and the SGD momentum of 0.9.
On CUB-200, FVGC-Airerafts and StandfordCCass we use the official PyTorch pre-trained models on ImageNet- 1k are adopted initialization, using SGD optimizer with momentum of 0.9, weight decay of 0. 0.0005, batch size of 16, 200 epochs, learning rate of 0.001, dynamically adjusted by cosine scheduler. the a and are set to 0.5 and O.I.
A.4 DETAILS OF THE EXPERIMENTS FOR THE OTHER MIXUP
You can access detailed experimental settings and results at https://github com/ Westlake- Al/openmixup, which also provides the open-source code for most of the compared Mixup methods.
A.5 RESULTS OF CALIBRATION
Table 6: The expected calibration error (ECE) of ResNet18 and Swin- Swin-Tiny Transformer (Swin- (Swin-Tiny) with various Mixup methods trained CIFAR100 dataset for 200 epochs.
Classifiers
Mixup
CutMix
FMix
GndMix
PuzzleMix
AutoMix
AdAutoMix
ResNed18
15.3
4.4
8.9
6.5
3.7
3.4
(-0.2)
Swin-Tiny
13.4
10.1
9.2
9.3
16.7
10.5
9.2 (-0.0)
14

---

Published as a conference paper at ICLR 2024
MpxUp
FMix
GddMix
PuzzleMiix
AutoMix
AdAutaMixx
Accuracy
EC5-001%
8C5412%
1677
DV2: 10.5%
Confidence
Figure 7: Calibration plots of Mixup variants and AdAutoMix on CIFAR-100 using ResNet- The red line indicates the expected prediction tendency.
A.6 THE ACCURACY OF VARIOUS MIXUP APPROACHES ON OCCLUSION IMAGE SET
Table 7:
The accuracies of ResNet50 and Swin-Tiny Transformer trained by various Mixup ap-
proaches on CIFAR100 and CUB200 datasets with different occlusion ratios.
Swin-Tiny Transformer on CIFARI00
Method
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
MiXUP
76.82
74.54
71.88
67.98
63.18
55.26
44.20
30.07
15.69
6.14
PuzzleMix
80.45
78.98
77.52
75.47
71.16
64.42
53.40
38.53
21.39
7.91
AutoMix
82.68
81.40
79.05
75.44
70.61
64.30
55.25
40.92
23.09
9.73
AdAutoMix
84.33
82.41
80.16
76.84
72.09
66.74
58.09
46.48
28.02
9.91
ResNet-50
CUB200
Method
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Vanilla
82.15
74.75
61.89
46.24
30.81
16.67
8,94
4.63
2.23
1.07
CutMix
83.05
76.45
64.44
50.86
39.47
28.99
20.78
14.46
8.64
2.21
PuzzleMix
84.01
80,99
76.01
68.45
58,15
43.44
28.41
15.38
5.76
2.39
AutoMix
84.10
81,90
78.05
73.18
64.96
51.21
36.85
22.35
8.63
3.88
AdAutoMix
84.57
82.46
80.16
75.84
66.19
55.74
40.19
25.44
10.04
4.39
0% Occlusion
30% Occlusion
50% Occlusion
70% Occlusion
90% Occlusion
Figure 8: The images with different occlusion ratios.
A.7 THE CURVES OF EFFICIENCY AGAINST ACCURACY
The training time of various mixup data augmentation approaches against accuracy is shown in Fig. 9. AdAutoMix take more computation time, but it consistently outperforms previous state-of- the-art methods with different ResNet architectures on different datasets.
ResNet-50 ImageNet: 1K
ResNet- 18 CIFAR100
ResNet- 18 Standfond-Car
Top- Accuracy (%)
Askuaoiic
Aukiiii
Aubolii
Centic
Ca-Ming
Maltp
21-
Cabla
353
Venta
Mestlp
Nixio
72.
06.5
Training Time (hours)
Figure 9: The plot of efficiency vs. accuracy
A.8 ADAUTOMIX MODULES EXPERIMENT
Table 8 lists the accuracy of our AdAutoMix by gradually increasing regularization terms. The experimental results imply that each regularization term capable of improving the robustness of AdxutoMix.
15

---

Published as a conference paper at ICLR 2024
Table 9 shows the accuracy of our MixutoMix with different X. The experimental results show that AdAutoMix with A= 1 as default achieves the best performances on CIFARI00 dataset.
Table 10 shows the accuracy of our AdAutoMix with different input image number N. From Table 10, we can see that AdAutoMix achieve the highest accuracy with N=3 on CIFARI00.
Table 8: Loss function experiments on CIFAR100 based on ResNet18.
Method
Base
Base 0.5
Base | 0.5 0.5
Base 0. 0.5 0.5Lomee
Buse 10.5L 0.5lmma
0 0. memor
0..3 umerr + wpwine
ResNer18
79.38
79,98
80:04
81.32
81.35
Table 9: Classification accuracy of ResNet 18 with different À ratio.
CIFARI00
Method
0.2
1.0
2.0
5.0
10.0
ResNet18
82.27
82.32
81.73
80.02
81.05
ResNeXt50
84.22
84.40
83.99
84.31
83.63
Table 10: Classification accuracy of ResNet18 trained by AdAutoMix with different input image number N, where N = means that it is vanilla method.
CIFAR 100
inputs
Top1-Acc(*)
Top5-Acc(%)
Times s/iter
== 1
78.04
94.60
0.1584
N 2
82.16
95.88
0.1796
N = 3
82.32
95,92
0.2418
4
81.78
95.68
0.2608
N = 5
80.79
95.80
0.2786
A.9 ACCURACY OF RESNET-18 TRAINED BY ADAUTOMIX WITH AND WITHOUT ADVERSARIAL METHODS.
Figure 10 shows the accuracy of ResNet-18 18 trained by our AdAutoMix with and without adversarial training on CIFAR100. The experimental results demonstrate AdAutoMix with adversarial training achieves higher classification accuracy on CIFAR 100 dataset, which implies that the proposed ad- versarial framework is capable of generating harder samples to improve the robustness of classifier.
ResNet- 18 on CIFAR100
Top-1 Accuracy(%)
30
20
AdkutaMix Mdkurshiscwtthott
19
2500
5000
7500
10000
12500
15000
17500
20000
Iter
Figure 10: The T accuracy plot of AdAutoMix training with and without adversarial methods.
A.10 COMPARISON WITH OTHER ADVERSARIAL DATA AUGMENTATION
We further compare Mixup (Zhang et al., 2017) and our AdAutoMix with existing Adversarial Data Augmentation methods, e.g. DADA (Li et al., 2020), ME-ADA (Zhao et al., 2020), and SAMix (Zhang et al., 2023). Table 11 depicts the classification accuracy of various approaches. The experimental results in Table 11 demonstrate that our AdAutoMix outperforms existing Adver- sarial Data Augmentation methods and achieves the highest accuracy the CIFAR 100 dataset.
16

---

Published as a conference paper at ICLR 2024
Table 11: Experiments with AdAutoMix and other Adversarial Data Augmentation methods.
Baseline
Mp
DADA
ME-ADA
SAMiX
AdAutoMix
ResNet-18
76.42
78.52
78.86
77.45
54.01
81.55
A.11 ALGORITHM OF ADAUTOMIX
Algorithm AdAutoMix training process
Input: Encoder Eg, Ezg Classifier tww tw. Samples S, lambda X, Generator Gg(-), coefficient
and feature map 2n
1: Ez- E3.-armas Eg.params
2: for X, in loder do
3:
z 5x(%)
4: Imiz = Go(z), A)
5: Lamce thw (Fmizz X, Y)
6:
Lamos Leooine Vw (Iminy X, Y)
7:
for 1 14th
do Update W(t+ + 1) according to Eq. 14
8:
9:
end for
10:
for 1< to < T2 do
Update ( + 1) according to Eq. 12: end for
13:
Update (agiteeEE-uuu Bgg-arams, Es.prrass
14:
Ez-parama = * eg.pprams + (1 {) Earpamsss
15: end for
B VISUALIZATION OF MIXED SAMPLES
B. CLASS ACTIVATION MAPPING (CAM) DIFFERENT MIXUP SAMPLES.
The Class activation mapping (CAM) of various Mixup models are shown in Fig. 11.
MixUp
CutMix
PuzzleMix
AutoMix
AdAutaMix
Methods 2. =05
CAM Purple Finch Predication
CAM for Eastem Towhse Predication
Figure 11: The class activation map of various Mixup models (A = 0.5).
17

---

Published as a conference paper at ICLR 2024
B.2 MIXED SAMPLES ON CUB-200
The mixed samples generated by our approach trained on CUB200 dataset are depicted in Fig. 12.
Figure 12: Visualization of mixed samples on CUB-200.
B.3 MIXED SAMPLES ON CIFARION
The mixed samples generated by our approach trained on CIFAR100 dataset are shown Fig. 13.
Figure 13: Visualization of mixed samples on CIFAR100.
18

---

Published as a conference paper at ICLR 2024
B.4 DIVERSITY OF SAMPLES GENERATED BY VARIOUS APPROACHES
To demonstrate that AdAutoMix is capable of generating diversity samples, we show the synthesis- ing images of AdAutoMix and AutoMix ImageNet-I K. From Fig. 14, we can that AdAu- toMix produces mixed samples with more differences. By contrast, AutoMix generates similar images different iteration epochs, which implies that the proposed AdAutoMix has the capacity to produce diverse images by adversarial training.
20 epoch
50 epoch
100 epoch
AdAutoMix
AutoMix
Figure 14: Mixed samples with different epochs.
19


Reviews:
[{"review_id": "YJ5yQf8OwG", "reviewer": "Reviewer_vYYM", "summary": "Mixup data augmentations are widely used and usually require well-designed sample mixing strategies, e.g., AutoMix optimized in an end-to-end manner. However, using the same mixup classification loss as the learning objective for both the mixed sample generation and classification tasks might cause consistent and unitary samples, which lack diversity. Based on AutoMix, this paper proposes AdAutomixup, an adversarial automatic mixup augmentation approach that generates challenging samples to train a robust vein classifier for palm-vein identification by alternatively optimizing the classifier and the mixup sample generator. Meanwhile, the authors introduce an EMA teacher with cosine similarity to train AdAutomixup preventing the collapse of the inherent meanings of images. Extensive experiments on five mixup classification benchmarks demonstrate the effectiveness of the proposed methods.", "questions": "* (**Q1**) Do the authors provide the hyper-parameter settings of AdAutoMix (e.g., the mixing ratio $\\lambda$, the mixed sample number $N$, and $\\beta$ in Eq. (12)? The authors might provide a sensitivity analysis of the hyper-parameters in the Appendix.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* (**S1**) This paper provides an interesting view of improving mixed sample qualities through adversarial training in the close-loop optimized mixup augmentation framework. The overall presentation of the manuscript is easy to follow, and the proposed methods are well-motivated.\n\n* (**S2**) Extensive experiments on mixup benchmarks verify the performance gains of the proposed AdAutoMix compared to existing mixup methods. Popular Transformer architectures are included in experiments.", "weaknesses": "* (**W1**) More empirical analysis of the proposed methods can be added. Despite the authors visualizing the mixed samples and CAM maps of various mixup methods, it can only reflect the overall performances and characteristics of different methods. I suggest the authors provide a fine-grained analysis of each proposed module to demonstrate its effectiveness, e.g., plotting the classification accuracy of using adversarial training or not.\n\n* (**W2**) Small-scale experiments. The authors only provide comparison results on CIFAR-10/100, Tiny-ImageNet, and fine-grained classification datasets. More experiments on ImageNet-1K or other large-scale datasets are required. Meanwhile, the evaluation tasks or metrics can be more diverse, such as more robustness evaluations with adversarial attacks and transfer experiments to downstream tasks.\n\n* (**W3**) Some minor drawbacks in writing formats, and I suggest the authors take more time to polish the writing. As for Section 3, the arrangement of Sec. 3.1 and Sec. 3.2 can be reversed. Or the authors can provide a Preliminary section to introduce the background knowledge (e.g., mixup classification problem). As for equations, the text subscripts (e.g., $argmin_{\\theta}$, $L_{amce}$) should be in bold format, i.e., using `\\mathrm{}` as $\\mathrm{argmin}_{\\theta}$. As for tables and figures, there are some untidy arrangements, like Table 3, 4, and 5, and Figure 5 and 6. The author might zoom in on Figure 1 to show the detailed improvement of AdAutoMix.\n\n================== Post-rebuttal Feedback ==================\n\nSince the rebuttal feedback and revised version have almost addressed the weaknesses and concerns I mentioned, I raise my score to 8 and encourage the authors to further polish the writing issues and add more discussion of the limitations & future works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rmaDAurcs5", "reviewer": "Reviewer_xmzW", "summary": "The paper proposes an adversarial data augmentation strategy that builds on top of AutoMix. The framework alternates between training a mixed example generator and a target classifier. The mixed sample generator aims to produce hard examples and the target tries to improve generalization by learning robust features. With automatic mixup approaches (based on saliency or otherwise) the combination is deterministically selected, and there is no sample diversification. To mitigate this, the method proposes an adversarial generator instead. \nTo prevent a collapse from the generator, an EMA teacher and a weighted cosine similarity term between the mixed sample and individual samples is used for end-to-end learning.", "questions": "The method compares the median of the top-1 test accuracy in the last 10 training epochs. Since adversarial methods are generally brittle and have unstable training dynamics, does the mean test accuracy fluctuate a lot? Also, it seems that there is no validation set used to choose the best checkpoint. This evaluation criteria is not justified in the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Results are consistently better than AutoMixup and the evaluation (Table 1) is thorough.\n\n\n\n--------\nPost-rebuttal:\n\nThe authors have adequately addressed the concerns in the review. Useful experiments and ablations have been added as well. I'm still a little skeptical about the actual impact of the paper, from the methods and corresponding evaluation numbers in the paper I believe that we're at the point of diminishing returns. \n\nI've therefore increased my score to a 6.", "weaknesses": "There is no evaluation compared to Adversarial data augmentation approaches [1, 2, 3, 4]. At least an introduction or related works section should be added as relevant approaches to the problem.\n\nThe term “cross attention module” (CAM) should not be used as it can be confused with “class activation mapping” (CAM) which is generally used in saliency-based data augmentation methods. \n\nSome notation is confusing - the encoder weight is updated with an EMA of the weights of the classifier - $\\hat{\\phi} = \\xi \\hat{\\phi} + (1-\\xi) W$. Is it unclear if the encoder refers to the generator or the classifier. Later near Equation 12, $\\psi$ is referred to as a target classifier with weights $W$. \n\nEquation 7 and 8 refer to the same value of $y$ used in cross entropy. It is better to keep the form of the loss consistent, since $y_{mix} = \\sum_i y_i \\lambda_i$, implies \n\n$\\sum_i L_{ce}(\\psi(x_{mix}), y_i) \\lambda_i = \\sum_i -\\lambda_i y_i \\log(\\psi(x_{mix})) = \\log(\\psi(x_{mix})) \\sum_i -\\lambda_i y_i = -\\log(\\psi(x_{mix})) y_{mix} = L_{ce}((\\psi(x_{mix}), y_{mix}))$\n\n\nEquations 10 through 15 are badly formatted and hard to read. It is also unclear what the individual contribution of the four cross-entropy terms are, and a suitable way to choose $\\alpha$ and $\\beta$. \n\nSection 4.2 mentions the proposed method has the lowest calibration error, but there is no table showing the ECE of other baselines. Fig.4. shows the ECE of only the proposed method.\n\n\nTypos and minor mistakes:\n“to facility representation”\nMxiup → mixup\nbad formatting  in eq 5\nnotation in eq 6, 7 is unclear. is the $*$ scalar multiplication?\nwhat is meant by “inherent meaning of images” - this sounds slightly unscientific, this should be explained in a bit more detail \n\nCurrently, I think the paper needs a lot of work - both in terms of coherence and motivation for the method. There are too many elements all over the place and it is unclear what the improvement actually comes from. The evaluation criteria is not standard (see Questions) and needs more justification. Therefore, I recommend an initial reject.\n\n_______\n\n\n[1] Zhang, Jiajin, et al. \"Spectral Adversarial MixUp for Few-Shot Unsupervised Domain Adaptation.\" International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023.\n\n[2] Xu, Minghao, et al. \"Adversarial domain adaptation with domain mixup.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.\n\n[3] Zhao, Long, et al. \"Maximum-entropy adversarial data augmentation for improved generalization and robustness.\" Advances in Neural Information Processing Systems 33 (2020): 14435-14447.\n\n[4] Antoniou, Antreas, Amos Storkey, and Harrison Edwards. \"Data augmentation generative adversarial networks.\" arXiv preprint arXiv:1711.04340 (2017).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6NcpfhI1xc", "reviewer": "Reviewer_sHE4", "summary": "The paper proposes a new augmentation technique. First it proposes hard samples to train and secondly a robustification of the classifier. The method is evaluated on 4 datasets", "questions": "None. I see this paper as a clear contribution.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The idea to augment with hard examples is interesting. Furthermore, to iterate between augmentation and classifier is also interesting.\nShowed results are strong.", "weaknesses": "I do not see any significant weakness. The method is harder to implement and it requires more resources that other augmentation techniques, but given the timeline of augmentation, it is expected", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)