SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This subsection evaluates performance gains of DM on six image classification benchmarks, including CIFAR-100 [25], Tiny-ImageNet (Tiny) [10], ImageNet-1k [48], CUB-200-2011 (CUB) [59], FGVC-Aircraft (Aircraft) [42]. There are mainly two types of mixup methods based on their mixing policies: static methods including Mixup [77], CutMix [74], ManifoldMix [57], SaliencyMix [56], FMix [17], and ResizeMix [47], and dynamic mixup methods including PuzzleMix [23], AutoMix [38], and SAMix [31]. For a fair comparison, we use the optimal  $\alpha$  in  $\{0.1, 0.2, 0.5, 0.8, 1.0, 2.0\}$  for all mixup
Table 1: Top-1 Acc  $(\%)\uparrow$  of small-scale image classification on CIFAR-100 and Tiny-ImageNet datasets based on ResNet variants.
Datasets	CIFAR-100	Tiny-ImageNet
Methods	R-18	RX-50	WRN-28-8	R-18	RX-50
MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)
Mixup	79.12	80.44	82.10	82.96	82.82	83.51	63.86	65.07	66.36	67.70
CutMix	78.17	79.39	81.67	82.39	84.45	84.88	65.53	66.45	66.47	67.46
ManifoldMix	80.35	81.05	82.88	83.15	83.24	83.72	64.15	65.45	67.30	68.48
FMix	79.69	80.12	81.90	82.74	84.21	84.47	63.47	65.34	65.08	66.96
ResizeMix	80.01	80.26	81.82	82.96	84.87	84.72	63.74	64.33	65.87	68.56
Avg. Gain	+0.78	+0.77	+0.34	+1.18	+1.62
Table 2: Top-1 Acc (\%)↑ of image classification on Table 3: Top-1 Acc (\%)↑ of image classification ImageNet-1k with ResNet variants using PyTorch- on ImageNet-1k based on ResNet-50 using RSB style 100-epoch training recipe. A3 100-epoch training recipe.
Methods	R-18	R-34	R-50
MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)
Vanilla	70.04	-	73.85	-	76.83	-
Mixup	69.98	70.20	73.97	74.26	77.12	77.41
CutMix	68.95	69.26	73.58	73.88	77.07	77.32
ManifoldMix	69.98	70.33	73.98	74.25	77.01	77.30
FMix	69.96	70.26	74.08	74.34	77.19	77.38
ResizeMix	69.50	69.90	73.88	74.00	77.42	77.65
Avg. Gain	+0.32	+0.24	+0.25
Methods	MCE	DM(CE)	MBCE (one)	MBCE (two)	DM(BCE) (one)
RSB	76.49	77.72	78.08	76.95	78.43
Mixup	76.01	76.69	77.66	77.42	78.28
CutMix	76.47	77.22	77.62	67.54	78.21
ManifoldMix	76.14	76.93	77.78	67.78	78.20
FMix	76.09	76.87	77.76	73.44	78.11
ResizeMix	76.90	77.21	77.85	77.30	78.32
Avg. Gain	+0.76	-4.38	+0.47
Table 4: Top-1 Acc  $(\%)\uparrow$  of classification on ImageNet-1k with ViTs.
Methods	DeiT-S	Swin-T
MCE	DM(CE)	MCE	DM(CE)
DeiT	79.80	80.37	81.28	81.49
Mixup	79.65	80.04	80.71	80.97
CutMix	79.78	80.20	80.83	81.05
FMix	79.41	79.89	80.37	80.54
ResizeMix	79.93	80.03	80.94	81.01
Avg. Gain	+0.39	+0.19
Table 5: Top-1 Acc  $(\%)\uparrow$  of fine-grained image classification on CUB-200 and FGVC-Aircrafts with ResNet variants.
Datasets	CUB-200	FGVC-Aircrafts
Methods	R-18	RX-50	R-18	RX-50
MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)
Mixup	78.39	79.90	84.58	85.04	79.52	82.66	85.18	86.68
CutMix	78.40	78.76	85.68	85.97	78.84	81.64	84.55	85.75
ManifoldMix	79.76	79.92	86.38	86.42	80.68	82.57	86.60	86.92
FMix	77.28	80.10	84.06	84.85	79.36	80.44	84.85	85.04
ResizeMix	78.50	79.58	84.77	84.92	78.10	79.54	84.08	84.51
Avg. Gain	+1.19	+0.35	+2.07	+0.73
algorithms and follow original hyper-parameters in papers. We adopt the open-source codebase OpenMixup [32] for most mixup methods. The detailed training recipes and hyper-parameters are provided in Appendix B. We also evaluate the adversarial robustness of CIFAR-100 in Appendix C.2.
Small-scale Classification Benchmarks. For small-scale classification benchmarks on CIFAR-100 and Tiny, we adopt the CIFAR version of ResNet variants and train with SGD optimizer following the common training settings [23, 38]. Table 1 and A2 show small-scale classification results. The proposed DM(CE) significantly improves MCE based on various mixup algorithms. Based on CIFAR-100 and three different CNNs (R-18, RX-50, and WRN-28-8), the decoupled mixup brings an average performance gain of  $0.78\%$ ,  $0.77\%$ , and  $0.34\%$ , respectively. More notably, the gains on the Tiny dataset are significant, with average performance gains of:  $1.18\%$  and  $1.62\%$  on R-18 and RX-50.
ImageNet and Fine-grained Classification Benchmarks. For experiments on ImageNet-1k, we follow three popular training procedures: PyTorch-style setting [19], DeiT [55] setting, and RSB A3 [63] setting to demonstrate the generalizability of decoupled mixup. As shown in Table 2, 3, and 4, DM(CE) improves consistently over MCE in all mixup algorithms on three training settings we considered. The relative improvements have been calculated in the last row of tables. For example, DM(CE) yields around  $+0.4\%$  for mixup methods based on ResNet variants using PyTorch-style and RSB A3 settings; around  $+0.5\%$  and  $+0.2\%$  for all methods based on DeiT-S and Swin-T using DeiT setting. Notice that MBCE(two) denotes using two-hot encoding for corresponding mixing classes, which yield worse performance than MBCE, and DM(BCE) adjusts the labels for the mixing classes by Equation 6. It verifies the necessity of DM(BCE) in the case of using MBCE. As for fine-grained benchmarks, we follow the training settings in AutoMix and initialize models with the official PyTorch pre-trained models (as supervised transfer learning). Table 5 and A6 show that DM(CE) noticeably boosts the original MCE for eight popular mixup variants, especially bringing  $0.53\% \sim 3.14\%$  gains on Aircraft based on ResNet-18.
5.2 Semi-supervised Transfer Learning Benchmarks
Following the transfer learning (TL) benchmarks [71], we perform TL experiments on CUB, Aircraft, and Stanford-Cars [24] (Cars). Besides the vanilla Fine-Tuning baseline, we compare current
Table 6: Top-1 Acc  $(\%)\uparrow$  of semi-supervised transfer learning on various TL benchmarks (CUB-200, FGVC-Aircraft, and Standfold-Cars) using only  $15\%$ ,  $30\%$  and  $50\%$  labels based on ResNet-50.
Methods	CUB-200	FGVC-Aircraft	Stanford-Cars
15%	30%	50%	15%	30%	50%	15%	30%	50%
Fine-Tuning	45.25±0.12	59.68±0.21	70.12±0.29	39.57±0.20	57.46±0.12	67.93±0.28	36.77±0.12	60.63±0.18	75.10±0.21
+DM	50.04±0.17	61.39±0.24	71.87±0.23	43.15±0.22	61.02±0.15	70.38±0.18	41.30±0.16	62.65±0.21	77.19±0.19
BSS	47.74±0.23	63.38±0.29	72.56±0.17	40.41±0.12	59.23±0.31	69.19±0.13	40.57±0.12	64.13±0.18	76.78±0.21
Co-Tuning	52.58±0.53	66.47±0.17	74.64±0.36	44.09±0.67	61.65±0.32	72.73±0.08	46.02±0.18	69.09±0.10	80.66±0.25
+DM	54.96±0.65	68.25±0.21	75.72±0.37	49.27±0.83	65.60±0.41	74.89±0.17	51.78±0.34	74.15±0.29	83.02±0.26
Self-Tuning	64.17±0.47	75.13±0.35	80.22±0.36	64.11±0.32	76.03±0.25	81.22±0.29	72.50±0.45	83.58±0.28	88.11±0.29
+Mixup	62.38±0.32	74.65±0.24	81.46±0.27	59.38±0.31	74.65±0.26	81.46±0.27	70.31±0.27	83.63±0.23	88.66±0.21
+DM	73.06±0.38	79.50±0.35	82.64±0.24	67.57±0.27	80.71±0.25	84.82±0.26	81.69±0.23	89.22±0.21	91.26±0.19
Avg. Gain	+5.95	+2.77	+1.34	+5.65	+4.52	+2.65	+7.22	+4.22	+2.35
Table 7: Top-1 Acc  $(\%)\uparrow$  of semi-supervised learning on CIFAR-100 (using 400, 2500, and 10000 labels) based on WRN-28-8. Notice that DM denotes using DM(CE) and AS, Con denotes various unsupervised consistency losses, Rot denotes the rotation loss in ReMixMatch, and CPL denotes the curriculum labeling in FlexMatch.
Methods	Losses	CIFAR-10	CIFAR-100
250	4000	400	2500	10000
Pseudo-Labeling	CE	53.51±2.20	84.92±0.19	12.55±0.85	42.26±0.28	63.45±0.24
MixMatch	CE+Con	86.37±0.59	93.34±0.26	32.41±0.66	60.24±0.48	72.22±0.29
ReMixMatch	CE+Con+Rot	93.70±0.05	95.16±0.01	57.15±1.05	73.87±0.35	79.08±0.27
MixMatch+DM	CE+Con+DM	89.16±0.71	95.15±0.68	35.72±0.53	62.51±0.37	74.70±0.28
UDA	CE+Con	94.84±0.06	95.71±0.07	53.61±1.59	72.27±0.21	77.51±0.23
FixMatch	CE+Con	95.14±0.05	95.79±0.08	53.58±0.82	71.97±0.16	77.80±0.12
FlexMatch	CE+Con+CPL	95.02±0.09	95.81±0.01	60.06±1.62	73.51±0.20	78.10±0.15
FixMatch+Mixup	CE+Con+MCE	95.05±0.23	95.83±0.19	50.61±0.73	72.16±0.18	78.75±0.14
FixMatch+DM	CE+Con+DM	95.23±0.09	95.87±0.11	59.75±0.95	74.12±0.23	79.58±0.17
Average Gain	+1.44	+0.95	+4.74	+2.30	+2.13
state-of-the-art TL methods, including BSS [68], Co-Tuning [71], and Self-Tuning [67]. For a fair comparison, we use the same hyper-parameters and augmentations as Self-Tuning, detailed in Appendix B.2. In Table 6, we adopt DM(CE) and AS for Fine-Tuning, Co-Tuning, and Self-Tuning using Mixup. DM(CE) and AS steadily improve Mixup and the baselines by large margins, e.g.,  $+4.62\% \sim 9.19\%$  for  $15\%$  labels,  $+2.02\% \sim 5.67\%$  for  $30\%$  labels, and  $+2.09\% \sim 3.15\%$  for  $50\%$  labels on Cars. This outstanding improvement implies that generating mixed samples efficiently is essential for data-limited scenarios. A similar performance will be presented as well in the next SSL setting.
5.3 Semi-supervised Learning Benchmarks
Following [52, 76], we adopt the most commonly used CIFAR-10/100 datasets among the famous SSL benchmarks based on WRN-28-2 and WRN-28-8. We mainly evaluate the proposed DM on popular SSL methods MixMatch [2] and FixMatch [52], and compare with Pesudo-Labeling [27], ReMixMatch [1], UDA [65], and FlexMatch [76]. For a fair comparison, we use the same hyperparameters and training settings as the original papers and conduct experiments with the open-source codebase TorchSSL [76], detailed in Appendix B.2. Table 7 shows that adding DM(CE) and AS significantly improves MixMatch and FixMatch: DM(CE) brings  $1.81\sim 2.89\%$  gains on CIFAR-10 and  $1.27\sim 3.31\%$  gains on CIFAR-100 over MixMatch while bringing  $1.78\sim 4.17\%$  gains on CIFAR-100 over FixMatch. Meanwhile, we find that directly applying mixup augmentations to FixMatch brings limited improvements, while FixMatch+DM achieves the best performance in most cases on CIFAR-10/100 datasets. Appendix C.3 provides further studies with limited labeled data. Therefore, mixup augmentations with DM can achieve data-efficient training in SSL.
5.4 Ablation Study and Analysis
Hyperparameters and Proposed Components. Since we have demonstrated the effectiveness of DM in the above experiments, Figure 1 and 5 verified that DM could well explore hard mixed samples. We then verify whether DM is robust to hyper-parameters (full hyper-parameters in Appendix C.5) and study the effectiveness of AS in SSL:
Figure 5: Top-1 Acc  $(\%)$  of mixed samples on ImageNet-1k validation set.
Table 8: Ablation of the proposed asymmetric strategy (AS) and DM(CE) upon Self-Tuning for semi-supervised transfer learning on CUB-200 based on R-18.
Methods	15%	30%	50%	100%
Self-Tuning	57.82	69.12	73.59	75.08
+MCE	63.36	72.81	75.73	76.67
+MCE+AS(λ≥0.5)	59.04	69.67	74.89	75.96
+MCE+AS(λ≤0.5)	62.97	72.46	75.40	76.34
+DM(CE)+AS(λ≤0.5)	66.17	74.25	77.68	78.52
(1) The only hyper-parameter  $\eta$  in DM(CE) and DM(BCE) can be set according to the types of mixup methods. We grid search  $\eta$  in  $\{0.01, 0.1, 0.5, 1, 2\}$  on ImageNet-1k. As shown in Figure A2 left, the static (Mixup and CutMix) and the dynamic methods (PuzzleMix and AutoMix) prefer  $\eta = 0.1$  and  $\eta = 1$ , respectively, which might be because the dynamic variants generate more discriminative and reliable mixed samples than the static methods.
(2) Hyper-parameters  $\xi$  and  $t$  in DM(BCE) can also be determined by the characters of mixup policies. We grid search  $\xi \in \{1,0.9,0.8,0.7\}$  and  $t \in \{2,1,0.5,0.3\}$ . Figure A2 middle and right show that cutting-based methods (CutMix and AutoMix) prefer  $\xi = 0.8$  and  $t = 1$ , while the interpolation-based policies (Mixup and ManifoldMix) use  $\xi = 1.0$  and  $t = 0.5$ .
(3) Table 8 shows the superiority of  $\mathrm{AS}(\lambda \leq 0.5)$  in comparison to MCE and  $\mathrm{AS}(\lambda \geq 0.5)$ , while using DM(CE) and  $\mathrm{AS}(\lambda \leq 0.5)$  further improves MCE.
(4) The experiments of different sizes of training data are performed to verify the data efficiency of DM. We can observe that decoupled mixup improves by around  $2\%$  accuracy without any computational overhead. The detailed results are shown in Appendix C.3.
Occlusion Robustness We also analyzed robustness against random occlusion [43] for models trained on ImageNet-1k using the official implementation. Concretely, the classifier is thought to be robust if it predicts the correct label given an occluded version of the image. In other words, the network learns essential features (e.g., semantic regions) that discriminate each class. For occlusion, we consider patch-based random masking. In particular, we split the image of  $224 \times 224$  resolutions into patch size  $16 \times 16$  and randomly mask  $M$  patches out of the total number of  $N$  patches, where the occlusion ratio is defined as  $\frac{M}{N}$ . As shown in Figure 6, the
proposed DM helps various mixup methods achieve better occlusion robustness, indicating DM can force the model to learn discriminative features, e.g., image patches with semantic information that is deterministic when the occlusion ratio is high.
Figure 6: Robustness against different occlusion ratios of images for mixup augmentations using the MCE and our DM(CE) loss based on ResNet-50 (left) and DeiT-S (right) on ImageNet-1k. RSB and DeiT denote using CutMix+Mixup (static mixup policies) in RSB A3 [63] and DeiT [55] training settings. DM(CE) improves mixups by exploring hard mixed samples.
6 Conclusion, Limitations, and Border Impacts
Decoupled Mixup and Dynamic Mixups. We investigate and show two limitations of the decoupled mixup. Different from static mixup methods, dynamic mixup spends extra time to optimize mixing masks in input space to align the mixed samples and labels. Although the optimized mixing policies can enhance the model to find discriminative features [38], their predic
Table 9: Top-1 Acc  $(\%)\uparrow$  on CIFAR-100 and Tiny.
Datasets Backbone	CIFAR-100 WRN-28-8	Tiny-ImageNet RX-50
Methods	MCE	DM(CE)	MCE	DM(CE)
PuzzleMix	85.02	85.25	67.83	68.04
+RSB	85.24	85.61	68.17	68.86
AutoMix	85.18	85.38	70.72	71.56
+RSB	85.35	85.54	70.98	72.37
tions are also under-confident. In Table 9, we tried some advanced dynamic mixup policies, e.g.,
$^{2}$ https://github.com/Muzammal-Naseer/Intriguing-Properties-of-Vision-Transformers
PuzzleMix [23], and AutoMix [38], with decoupled mixup, and found the improvement is limited. The main reason is there will not be many hard mixed samples in dynamic mixups. Therefore, we additionally incorporate two static mixups in RSB training settings, i.e., a half probability that Mixup or CutMix will be selected during the training. As expected, the improvements from the decoupled mixup are getting obvious upon static mixup variants. This is a very preliminary attempt that deserves more exploration in future works, and we provide more results of dynamic mixups in Appendix C.
Meanwhile, we further conduct comprehensive comparison experiments with modern Transformer-based architectures on CIFAR-100, considering the concurrent work TransMix [5] and TokenMix [36]. As shown in Table 10, where results with  $\dagger$  denote the official implementation and the other are based on OpenMixup [32], DM(CE) enables DeiT (CutMix and Mixup) to achieve competitive performances as dynamic mixup variants like AutoMix and SAMix [31] based on ConvNeXt-S without introducing extra computational costs, while still performing worse than them based on DeiT-S. Compared with specially designed label mixing methods using attention maps, DM(CE) also achieves compet
Table 10: Top-1 Acc (\%)↑ of on CIFAR-100 training 200 and 600 epochs based on DeiT-S and ConvNeXt-T. Underlines denote the top-3 best results. Total training hours and GPU memory are collected on a single A100 GPU.
Methods	DeiT-Small	ConvNeXt-Tiny
200 ep	600 ep	Mem.	Time	200 ep	600 ep	Mem.	Time
Vanilla	65.81	68.50	8.1	27	78.70	80.65	4.2	10
Mixup	69.98	76.35	8.2	27	81.13	83.08	4.2	10
CutMix	74.12	79.54	8.2	27	82.46	83.20	4.2	10
DeiT	75.92	79.38	8.2	27	83.09	84.12	4.2	10
SmoothMix	67.54	80.25	8.2	27	78.87	81.31	4.2	10
SaliencyMix	69.78	76.60	8.2	27	82.82	83.03	4.2	10
AttentiveMix+	75.98	80.33	8.3	35	82.59	83.04	4.3	14
FMix	70.41	74.31	8.2	27	81.79	82.29	4.2	10
GridMix	68.86	74.96	8.2	27	79.53	79.66	4.2	10
ResizeMix	68.45	71.95	8.2	27	82.53	82.91	4.2	10
PuzzleMix	73.60	81.01	8.3	35	82.29	84.17	4.3	53
AutoMix	76.24	80.91	18.2	59	83.30	84.79	10.2	56
SAMix	77.94	82.49	21.3	58	83.56	84.98	10.3	57
DeiT+TransMix	76.17	79.33	8.4	28	-	-	-	-
DeiT+TokenMix†	76.25	79.57	8.4	34	-	-	-	-
DeiT+DM(CE)	76.20	79.92	8.2	27	83.44	84.49	4.2	10
itive performances to TransMix and TokenMix. How to further improve the decoupled mixup with the salient regions or dynamic attention information to research similar performances of dynamic mixing variants can also be studied in future works.
The Next Mixup. In a word, we introduce Decoupled Mixup (DM), a new objective function for considering both smoothness and mining discriminative features in mixup augmentations. The proposed DM helps static mixup methods (e.g., MixUp and CutMix) achieve a comparable or better performance than the computationally expensive dynamic mixup policies. Most importantly, DM raises a question worthy of researching: is it necessary to design very complex mixup policies? We also find that decoupled mixup could be the bridge to combining static and dynamic mixup. However, the introduction of additional hyperparameters may take users some extra time to check on other than images or other mixup methods. This also leads to the core question of the next step in the development of this work: how to design a more elegant and adaptive mixup training objective that connects different types of mixups to achieve high data efficiency? We believe these explorations and questions can inspire future research in the community of mixup augmentations.
This work was supported by National Key R&D Program of China (No. 2022ZD0115100), National Natural Science Foundation of China Project (No. U21A20427), and Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University. We thank the AI Station of Westlake University for the support of GPUs and thank all reviewers for polishing the manuscript.
[1] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785, 2019. 3, 8
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv:1905.02249, 2019. 3, 8
[68] Chen Xinyang, Wang Sinan, Fu Bo, Long Mingsheng, and Wang Jianmin. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 3, 8
[69] Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. Advances in Neural Information Processing Systems, 35:3361-3376, 2022. 3
[70] Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y. Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3
[71] Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 7, 8
[72] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations (ICLR), 2020. 19
[73] Hao Yu, Huanyu Wang, and Jianxin Wu. Mixup without hesitation. In International Conference on Image and Graphics (ICIG), pages 143-154. Springer, 2021. 4
[74] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6023-6032, 2019. 2, 3, 6, 19, 20, 22
[75] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016. 6
[76] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3, 8, 19
[77] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017. 2, 3, 6, 20, 21, 22
In the Appendix sections, we provide proofs of proposition 1 ( $\S$ A.1) and proposition 2 ( $\S$ A.2), implementation details ( $\S$ B), and more results of comparison experiment and empirical analysis ( $\S$ C).
A Proof of Proposition
Proposition 1. Assuming  $x_{(a,b)}$  is generated from two different classes, minimizing  $\mathcal{L}_{MCE}$  is equivalent to regress corresponding  $\lambda$  in the gradient of  $\mathcal{L}_{MCE}$ :
$$
\left(\nabla_ {z _ {(a, b)}} \mathcal {L} _ {M C E}\right) ^ {l} = \left\{ \begin{array}{l l} - \lambda + \frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {c} \exp \left(z _ {(a , b)} ^ {c}\right)}, & l = i \\ - (1 - \lambda) + \frac {\exp \left(z _ {(a , b)} ^ {j}\right)}{\sum_ {c} \exp \left(z _ {(a , b)} ^ {c}\right)}, & l = j \\ \frac {\exp \left(z _ {(a , b)} ^ {l}\right)}{\sum_ {c} \exp \left(z _ {(a , b)} ^ {c}\right)}, & l \neq i, j \end{array} \right. \tag {7}
$$
Proof. For the mixed sample  $(x_{(a,b)},y_{(a,b)})$ ,  $z_{(a,b)}$  is derived from a feature extractor  $f_{\theta}$  (i.e  $z_{(a,b)} = f_{\theta}(x_{(a,b)})$ ). According to the definition of the mixup cross-entropy loss  $\mathcal{L}_{MCE}$ , we have:
$$
\begin{array}{l} \left(\nabla_ {z (a, b)} \mathcal {L} _ {M C E}\right) ^ {l} = \frac {\partial \mathcal {L} _ {M C E}}{\partial z _ {(a , b)} ^ {l}} = - \frac {\partial}{\partial z _ {(a , b)} ^ {l}} \left(y _ {(a, b)} ^ {T} \log \left(\sigma \left(z _ {(a, b)}\right)\right)\right) \\ = - \sum_ {i = 1} ^ {C} \left(y _ {(a, b)} ^ {i} \frac {\partial}{\partial z _ {(a , b)} ^ {l}} \left(\log \left(\frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {j = 1} ^ {C} \exp \left(z _ {(a , b)} ^ {j}\right)}\right)\right)\right) \\ = - \sum_ {i = 1} ^ {C} \left(y _ {(a, b)} ^ {i} \frac {\sum_ {j = 1} ^ {C} \exp (z _ {(a , b)} ^ {j})}{\exp (z _ {(a , b)} ^ {i})} \frac {\partial}{\partial z _ {(a , b)} ^ {l}} \left(\frac {\exp (z _ {(a , b)} ^ {i})}{\sum_ {j = 1} ^ {C} \exp (z _ {(a , b)} ^ {j})}\right)\right) \\ = - \sum_ {i = 1} ^ {C} \left(y _ {(a, b)} ^ {i} \left(\delta_ {i} ^ {l} - \frac {\exp \left(z _ {(a , b)} ^ {l}\right)}{\sum_ {j = 1} ^ {C} \exp \left(z _ {(a , b)} ^ {j}\right)}\right)\right) \\ = \frac {\exp (z _ {(a , b)} ^ {l})}{\sum_ {j = 1} ^ {C} \exp (z _ {(a , b)} ^ {j})} - y _ {(a, b)} ^ {l}. \\ \end{array}
$$