SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left(\nabla_ {z (a, b)} \mathcal {L} _ {D M}\right) ^ {l} = \frac {\partial \mathcal {L} _ {D M}}{\partial z _ {(a , b)} ^ {l}} = - \frac {\partial}{\partial z _ {(a , b)} ^ {l}} \left(y _ {[ a, b ]} ^ {T}) \log \left(H \left(z _ {(a, b)}\right)\right) y _ {[ a, b ]}\right) \\ = - \frac {\partial}{\partial z _ {(a , b)} ^ {l}} \Big (\sum_ {i, j = 1} ^ {C} y _ {a} ^ {i} \log \left(\frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {k \neq j} ^ {C} \exp \left(z _ {(a , b)} ^ {j}\right)}\right) y _ {b} ^ {j} + \sum_ {i, j = 1} ^ {C} y _ {a} ^ {j} \log \left(\frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {k \neq i} ^ {C} \exp \left(z _ {(a , b)} ^ {j}\right)}\right) y _ {b} ^ {i} \Big) \\ = - \sum_ {i, j = 1} ^ {C} \left(y _ {a} ^ {i} y _ {b} ^ {j} \frac {\partial}{\partial z _ {(a , b)} ^ {l}} \left(\log \left(\frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {k \neq j} ^ {C} \exp \left(z _ {(a , b)} ^ {k}\right)}\right) + \log \left(\frac {\exp \left(z _ {(a , b)} ^ {j}\right)}{\sum_ {k \neq i} ^ {C} \exp \left(z _ {(a , b)} ^ {k}\right)}\right)\right)\right) \\ = - \sum_ {i, j = 1} ^ {C} \left(y _ {a} ^ {i} y _ {b} ^ {j} \left(\delta_ {i} ^ {l} - \frac {\sum_ {k \neq j} \exp \left(z _ {(a , b)} ^ {k}\right) \delta_ {k} ^ {l}}{\sum_ {k \neq j} \exp \left(z _ {(a , b)} ^ {k}\right)} + \delta_ {j} ^ {l} - \frac {\sum_ {k \neq i} \exp \left(z _ {(a , b)} ^ {k}\right) \delta_ {k} ^ {l}}{\sum_ {k \neq i} \exp \left(z _ {(a , b)} ^ {k}\right)}\right)\right) \\ = \frac {\sum_ {k \neq i} \exp (z _ {(a , b)} ^ {k}) \delta_ {k} ^ {l}}{\sum_ {k \neq i} \exp (z _ {(a , b)} ^ {k})} + \frac {\sum_ {k \neq j} \exp (z _ {(a , b)} ^ {k}) \delta_ {k} ^ {l}}{\sum_ {k \neq j} \exp (z _ {(a , b)} ^ {k})} - \delta_ {i} ^ {l} - \delta_ {j} ^ {l}. \\ \end{array}
$$
Thus, for  $\mathcal{L}_{DM}$  loss:
$$
\left(\nabla_ {z _ {(a, b)}} \mathcal {L} _ {M C E}\right) ^ {l} = \left\{ \begin{array}{l l} - 1 + \frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {c \neq j} \exp \left(z _ {(a , b)} ^ {c}\right)}, & l = i \\ - 1 + \frac {\exp \left(z _ {(a , b)} ^ {j}\right)}{\sum_ {c \neq i} \exp \left(z _ {(a , b)} ^ {c}\right)}, & l = j \\ \frac {\exp \left(z _ {(a , b)} ^ {l}\right)}{\sum_ {c \neq i} \exp \left(z _ {(a , b)} ^ {c}\right)} + \frac {\exp \left(z _ {(a , b)} ^ {l}\right)}{\sum_ {c \neq j} \exp \left(z _ {(a , b)} ^ {c}\right)}, & l \neq i, j \end{array} \right. \tag {8}
$$
Proposition 2. With the decoupled Softmax defined above, decoupled mixup cross-entropy  $\mathcal{L}_{DM(CE)}$  can boost the prediction confidence of the interested classes mutually and escape from the  $\lambda$ -constraint:
$$
\mathcal {L} _ {D M (C E)} = \sum_ {i = 1} ^ {c} \sum_ {j = 1} ^ {c} y _ {a} ^ {i} y _ {b} ^ {j} \Big (\log \big (\frac {p _ {(a , b)} ^ {i}}{1 - p _ {(a , b)} ^ {j}} \big) + \log \big (\frac {p _ {(a , b)} ^ {j}}{1 - p _ {(a , b)} ^ {i}} \big) \Big).
$$
Proof. For the mixed sample  $(x_{(a,b)},y_{(a,b)})$ ,  $z_{(a,b)}$  is derived from a feature extractor  $f_{\theta}$  (i.e.  $z_{(a,b)} = f_{\theta}(x_{(a,b)})$ ). According to the definition of the mixup cross-entropy loss  $\mathcal{L}_{DM(CE)}$ , we have:
$$
\begin{array}{l} \mathcal {L} _ {D M (C E)} = y _ {[ a, b ]} ^ {T} \log \left(H \left(Z _ {(a, b)}\right)\right) y _ {[ a, b ]} \\ \triangleq y _ {a} ^ {T} \log \left(H \left(Z _ {(a, b)}\right)\right) y _ {b} + y _ {b} ^ {T} \log \left(H \left(Z _ {(a, b)}\right)\right) y _ {a} \\ = \sum_ {i, j = 1} ^ {C} y _ {a} ^ {i} \log (\frac {\exp (z _ {(a , b)} ^ {i})}{\sum_ {k \neq j} ^ {C} \exp (z _ {(a , b)} ^ {j})}) y _ {b} ^ {j} + \sum_ {i, j = 1} ^ {C} y _ {a} ^ {j} \log (\frac {\exp (z _ {(a , b)} ^ {i})}{\sum_ {k \neq i} ^ {C} \exp (z _ {(a , b)} ^ {j})}) y _ {b} ^ {i} \\ = \sum_ {i, j = 1} ^ {C} y _ {a} ^ {i} y _ {b} ^ {j} \left(\log \left(\frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {k \neq j} ^ {C} \exp \left(z _ {(a , b)} ^ {j}\right)}\right) + \log \left(\frac {\exp \left(z _ {(a , b)} ^ {j}\right)}{\sum_ {k \neq i} ^ {C} \exp \left(z _ {(a , b)} ^ {i}\right)}\right)\right) \\ = \sum_ {i, j = 1} ^ {C} y _ {a} ^ {i} y _ {b} ^ {j} \big (\log (\frac {\exp (z _ {(a , b)} ^ {i})}{\overline {{\sum_ {k = 1} ^ {C} \exp (z _ {(a , b)} ^ {k})}}})}{\frac {\sum_ {k \neq j} ^ {C} \exp (z _ {(a , b)} ^ {j})}{\sum_ {k = 1} ^ {C} \exp (z _ {(a , b)} ^ {k})}}) + \log (\frac {\exp (z _ {(a , b)} ^ {j})}{\overline {{\sum_ {k = 1} ^ {C} \exp (z _ {(a , b)} ^ {k})}}}) \big) \\ = \sum_ {i, j = 1} ^ {C} y _ {a} ^ {i} y _ {b} ^ {j} \left(\log \left(\frac {p _ {(a , b)} ^ {i}}{1 - p _ {(a , b)} ^ {j}}\right) + \log \left(\frac {p _ {(a , b)} ^ {j}}{1 - p _ {(a , b)} ^ {i}}\right)\right), \\ \end{array}
$$
where  $p_{(a,b)} = \sigma (z_{(a,b)})$
We briefly introduce used image datasets. (1) Small scale classification benchmarks: CIFAR-10/100 [25] contains 50,000 training images and 10,000 test images in  $32 \times 32$  resolutions, with 10 and 100 classes settings. Tiny-ImageNet [10] is a rescaled version of ImageNet-1k, which has 10,000 training images and 10,000 validation images of 200 classes in  $64 \times 64$  resolutions. (2) Large scale classification benchmarks: ImageNet-1k [26] constrains 1,281,167 training images and 50,000 validation images of 1000 classes in  $224 \times 224$  resolutions. (3) Small-scale fine-grained classification scenarios: CUB-200-2011 [59] contains 11,788 images from 200 wild bird species for fine-grained classification. FGVC-Aircraft [42] contains 10,000 images of 100 classes of aircraft. Standford-Cars [24].
B.2 Training Settings
Small-scale image classification. As for small-scale classification benchmarks on CIFAR-100 and Tiny-ImageNet datasets, we adopt the CIFAR version of ResNet variants, i.e., using a  $3 \times 3$  convolution instead of the  $7 \times 7$  convolution and MaxPooling in the stem, and follow the common training settings [23, 38]: the basic data augmentation includes RandomFlip and RandomCrop with 4 pixels padding; SGD optimizer and Cosine learning rate Scheduler [39] are used with the SGD weight decay of 0.0001, the momentum of 0.9, and the Batch size of 100; all methods train 800 epochs with the basic learning rate  $lr = 0.1$  on CIFAR-100 and 400 epochs with  $lr = 0.2$  on Tiny-ImageNet.
Fine-grained image classification. As for fine-grained classification experiments on CUB-200 and Aircraft datasets, all mixup methods are trained 200 epochs by SGD optimizer with the initial learning rate  $lr = 0.001$ , the weight decay of 0.0005, and the batch size of 16. We use the standard augmentations RandomFlip and RandomResizedCrop, and load the official PyTorch pre-trained models on ImageNet-1k as initialization.
ImageNet image classification. For large-scale classification tasks on ImageNet-1k, we evaluate mixup methods on three popular training procedures, and Tab. A1 shows the full training settings of the three settings. Notice that DeiT [55] and RSB A3 [63] settings employ Mixup and CutMix with a switching probability of 0.5 during training. (a) PyTorch-style setting. Without any advanced training strategies, a PyTorch-style setting is used to study the performance gains of mixup methods: SGD optimizer is used to train 100 epochs with the SGD weight decay of 0.0001, a momentum of 0.9, a batch size of 256, and the basic learning rate of 0.1 adjusted by Cosine Scheduler. Notice that we replace the step learning rate decay with Cosine Scheduler [39] for better performances following [74]. (b) DeiT [55] setting. We use the DeiT setting to verify the DM(CE) effectiveness in training Transformer-based networks: AdamW optimizer [41] is used to train 300 epochs with a batch size of 1024, the basic learning rate of 0.001, and the weight decay of 0.05. (c) RSB A3 [63] setting. This setting adopts similar training techniques as DeiT to ConvNets, especially using MBCE instead of MCE: LAMB optimizer [72] is used to train 100 epochs with the batch size of 2048, the basic learning rate of 0.008, and the weight decay of 0.02. Notice that DeiT and RSB A3 settings use the combination of Mixup and CutMix (50% random switching probabilities) as the baseline.
Semi-supervised transfer learning. For semi-supervised transfer learning benchmarks, we use the same hyper-parameters and augmentations as Self-Tuning<sup>3</sup>: all methods are initialized by PyTorch pre-trained models on ImageNet-1k and trained 27k steps in total by SGD optimizer with the basic learning rate of 0.001, the momentum of 0.9, and the weight decay of 0.0005. We reproduced Self-Tuning and conducted all experiments in OpenMixup [32].
Semi-supervised learning. For semi-supervised learning benchmarks (training from scratch), we adopt the most commonly used CIFAR-10/100 datasets among the famous SSL benchmarks based on WRN-28-2 and WRN-28-8 following [52, 76]. For a fair comparison, we use the same hyperparameters and training settings as the original papers and adopt the open-source codebase TorchSSL [76] for all methods. Concretely, we use an SGD optimizer with a basic learning rate of
<sup>3</sup>https://github.com/thuml/Self-Tuning
Table A1: Ingredients and hyper-parameters used for ImageNet-1k training settings.
Procedure	PyTorch	DeiT	RSB A3
Train Res	2242	2242	2242
Test Res	2242	2242	2242
Test crop ratio	0.875	0.875	0.95
Epochs	100/300	300	100
Batch size	256	1024	2048
Optimizer	SGD	AdamW	LAMB
LR	0.1	1 × 10-3	8 × 10-3
LR decay	cosine	cosine	cosine
Weight decay	10-4	0.05	0.02
optimizer momentum	0.9	β1, β2 = 0.9, 0.999	X
Warmup epochs	X	5	5
Label smoothing ε	X	0.1	X
Dropout	X	X	X
Stoch. Depth	X	0.1	0.05
Repeated Aug	X	✓	✓
Gradient Clip.	X	1.0	X
H. flip	✓	✓	✓
RRC	✓	✓	✓
Rand Augment	X	9/0.5	6/0.5
Auto Augment	X	X	X
Mixup alpha	X	0.8	0.1
Cutmix alpha	X	1.0	1.0
Erasing prob.	X	0.25	X
ColorJitter	X	X	X
EMA	X	0.99996	X
CE loss	✓	✓	X
BCE loss	X	X	✓
$lr = 0.03$  adjusted by Cosine Scheduler, the total  $2^{20}$  steps, the batch size of 64 for labeled data, and the confidence threshold  $\tau = 0.95$ .
B.3 Hyper-parameter Settings
We follow the basic hyper-parameter settings (e.g.,  $\alpha$ ) for mixup variants in OpenMixup [32], where we reproduce most comparison methods. Notice that static methods denote Mixup [77], CutMix [74], ManifoldMix [57], SaliencyMix [56], FMix [17], ResizeMix [47], and dynamic methods denote PuzzleMix [23], AutoMix [38], and SAMix [31]). Similarly, interpolation-based methods denote Mixup and ManifoldMix while cutting-based methods denote the rest mixup variants mentioned above. We set the hyper-parameters of DM(CE) as follows: For CIFAR-100 and ImageNet-1k, static methods use  $\eta = 0.1$ , and dynamic methods use  $\eta = 1$ . For Tiny-ImageNet and fine-grained datasets, static methods use  $\eta = 1$  based on ResNet-18 while  $\eta = 0.1$  based on ResNeXt-50; dynamic methods use  $\eta = 1$ . As for the hyper-parameters of DM(BCE) on ImageNet-1k, cutting-based methods use  $t = 1$  and  $\xi = 0.8$ , while interpolation-based methods use  $t = 0.5$  and  $\xi = 1$ . Note that we use  $\alpha = 0.2$  and  $\alpha = 2$  for the static and dynamic methods when using the proposed DM.
Table A2: Top-1 Acc  $(\%)\uparrow$  of small-scale image classification on CIFAR-100 and Tiny-ImageNet datasets based on ResNet variants.
Datasets	CIFAR-100	Tiny-ImageNet
Methods	R-18	RX-50	WRN-28-8	R-18	RX-50
MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)
SaliencyMix	79.12	79.28	81.53	82.61	84.35	84.41	64.60	66.56	66.55	67.52
PuzzleMix	81.13	81.34	82.85	82.97	85.02	85.25	65.81	66.52	67.83	68.04
AutoMix	82.04	82.32	83.64	83.94	85.18	85.38	67.33	68.18	70.72	71.56
SAMix	82.30	82.40	84.42	84.53	85.50	85.59	68.89	69.16	72.18	72.39
Avg. Gain	+0.19	+0.40	+0.15	+0.95	+0.56
Table A3: Top-1 Acc  $(\%)\uparrow$  of image classification on ImageNet-1k with ResNet variants using PyTorch-style 100-epoch training recipe.
Methods	R-18	R-34	R-50
MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)
SaliencyMix	69.16	69.57	73.56	73.92	77.14	77.42
PuzzleMix	70.12	70.32	74.26	74.51	77.54	77.71
AutoMix	70.51	70.64	74.52	74.77	77.91	78.15
SAMix	70.85	70.90	74.96	75.10	78.11	78.36
Avg. Gain	+0.20	+0.25	+0.23
Table A4: Top-1 Acc  $(\%)\uparrow$  of image classification on ImageNet-1k based on ResNet-50 using RSB A3 100-epoch training recipe.
Methods	MCE	DM(CE)	MBCE (one)	MBCE (two)	DM(BCE) (one)
SaliencyMix	76.85	77.25	77.93	72.74	78.24
PuzzleMix	77.27	77.60	78.02	77.19	78.15
AutoMix	77.45	77.82	78.33	77.46	78.62
SAMix	78.33	78.45	78.64	77.58	78.75
Avg. Gain	+0.30	-1.99	+0.04
Table A5: Top-1 Acc (\%)↑ of classify- Table A6: Top-1 Acc (\%)↑ of fine-grained image classification cation on ImageNet-1k with ViTs. on CUB-200 and FGVC-Aircrafts with ResNet variants.
Methods	DeiT-S	Swin-T
MCE	DM(CE)	MCE	DM(CE)
DeiT	79.80	80.37	81.28	81.49
SaliencyMix	79.32	79.86	80.68	80.83
PuzzleMix	79.84	80.25	81.03	81.16
AutoMix	80.78	80.91	81.80	81.92
SAMix	80.94	81.12	81.87	81.97
Avg. Gain	+0.32	+0.13
Datasets	CUB-200	FGVC-Aircrafts
Methods	R-18	RX-50	R-18	RX-50
MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)
SaliencyMix	77.95	78.28	83.29	84.51	80.02	81.31	84.31	85.07
PuzzleMix	78.63	78.74	84.51	84.67	80.76	80.89	86.23	86.36
AutoMix	79.87	81.08	86.56	86.74	81.37	82.18	86.69	86.82
SAMix	81.11	81.27	86.83	86.95	82.15	83.68	86.80	87.22
Avg. Gain	+0.45	+0.42	+0.94	+0.36
C More Experiment Results
C.1 Image Classification Benchmarks
Small-scale classification benchmarks. For small-scale classification benchmarks on CIFAR-100 and Tiny-/ImageNet, we also conduct experiments of applying the proposed DM(CE) to dynamic mixup methods even though these algorithms have achieved high performance in Table A2: DM(CE) brings  $0.23\% \sim 0.36\%$  on CIFAR-100 for the previous state-of-the-art PuzzleMix and brings  $0.21\% \sim 0.27\%$  on Tiny-/ImageNet for the current state-of-the-art method SAMix. Overall, the proposed DM(CE) produces  $+0.15 \sim 0.4\%$  and  $0.56 \sim 0.95\%$  average gains on CIFAR-100 and Tiny-/ImageNet, demonstrating its generalizability to advanced mixup augmentations.
ImageNet and fine-grained classification benchmarks. For experiments on ImageNet-1k, we also employ the proposed DM(CE) to dynamic mixup approaches on ImageNet-1k with PyTorch-style [19], DeiT [55], and RSB A3 [63] training settings to further evaluate the generalizability of decoupled mixup. As shown in Table A3 and Table A4, DM(CE) gains  $+0.2 \sim 0.3\%$  top-1 accuracy over MCE in average for four dynamic mixup methods based on ResNet variants on ImageNet-1k; Table A5 show DM(CE) also improves dynamic methods based on popular DeiT-S and Swin-T backbones with modern training recipes. These results indicate that the proposed decoupled mixup can also boost these dynamic mixup augmentations with high performances on ImageNet-1k. Moreover, the proposed DM(CE) can improve dynamic mixup variants on fine-grained classification benchmarks, as shown in Table A6, with around  $+0.4 \sim 0.9\%$  average gains over MCE based on ResNet variants.
Table A7: Top-1 Acc  $(\%)\uparrow$  and FGSM error  $(\%)\downarrow$  on CIFAR-100 and Tiny-IIdealNet based on ResNet-18 training 400 epochs.
Datasets	CIFAR-100	Tiny-ImageNet
Methods	Acc(%)↑	Error(%)↓	Acc(%)↑	Error(%)↓
MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)
Mixup	79.34	79.70	70.28	70.05	63.86	65.07	89.06	88.91
CutMix	79.58	79.77	87.43	86.84	65.53	66.45	89.14	88.79
ManifoldMix	80.18	81.06	72.50	72.19	64.15	65.45	88.78	88.52
PuzzleMix	80.22	80.58	79.76	79.53	65.81	66.13	91.83	92.05
AutoMix*	81.78	81.96	69.94	69.80	67.33	68.18	88.37	88.34
C.2 Adversarial Robustness
Since mixup variants are proven to enhance the robustness of DNNs against adversarial samples [77], we compare the robustness of the original MCE and the proposed DM(CE) by performing the
Figure A1: Experimental overviews of hard mixed sample mining. Left: Top-1 and top-2 accuracy of mixed data based on ResNet-50 trained 100 epochs on ImageNet-1k. Prediction is counted as correct if the top-1 prediction belongs to  $\{y_a, y_b\}$ ; prediction is counted as correct if the top-2 predictions are equal to  $\{y_a, y_b\}$ . Compared with static policies like Mixup [77] and CutMix [74], the dynamic method AutoMix [38] significantly reduces the difficulty of mixup classification and alleviates the label mismatch issue [23] by providing more reliable mixed samples but also requires a large computational overhead. Right: Taking Mixup as an example, our proposed decoupled mixup cross-entropy, DM(CE), significantly improves training efficiency by exploring hard mixed samples and alleviates the label mismatch issue.
FGSM [15] white-box attack of  $8 / 255\ell_{\infty}$  epsilon ball following [23]. Table A7 shows that DM(CE) improves top-1 Acc of MCE while maintaining the competitive FGSM error rates for five popular mixup algorithms, which indicates that DM(CE) can boost discrimination without disturbing the smoothness properties of mixup variants.
C.3 Data-efficient Mixup with Limited Training Labels
To further DM whether data-efficient mixup training can be truly achieved, we conducted supervised experiments on CIFAR-100 with different sizes of training data.  $15\%$ ,  $30\%$ , and  $50\%$  of the CIFAR-100 data are randomly selected as training data, and the test data are unchanged. The proposed decoupled mixup uses DM(CE) as the loss function by default. From Table A8, we can see that DM improves performance consistently without any computational overhead. Especially when using only  $15\%$  of the data, DM can improve accuracy by  $2\%$ . Therefore, combined with the experimental results of semi-supervised learning in Sec. 5.3 and Sec. 5.2, we can say that mixup training with DM is more data-efficient with limited data.
Table A8: Top-1 Acc  $(\%)\uparrow$  of image classification on CIFAR-100 with ResNet-18 using  $15\%$ ,  $30\%$ , and  $50\%$  labeled training sets.
Methods	15%	30%	50%
MCE	DM(CE)	MCE	DM(CE)	MCE	DM(CE)
Vanilla	42.48	-	56.41	-	64.32	-
Mixup	42.23	44.39	55.61	56.78	64.55	65.92
CutMix	43.81	44.85	55.99	57.14	64.38	65.87
SaliencyMix	42.95	44.01	55.42	56.51	64.56	66.10
PuzzleMix	42.67	43.87	56.19	57.36	64.74	66.26
Avg. Gain	+1.36	+1.14	+1.48
C.4 Empirical Analysis
In addition to occlusion robustness in Figure 6, we analyze the top-1 and top-2 mixup classification accuracy and visualize validation accuracy curves during training to empirically demonstrate the effectiveness of DM in Figure A1.
Figure A2: Ablation of hyper-parameters on ImageNet-1k based on ResNet-34. Left: analyzing the balancing weight  $\eta$  in DM(CE); Middle: analyzing  $\xi$  in DM(BCE) when  $t$  is fixed to 1 and 0.5; Right: analyzing  $t$  in DM(BCE) when  $\xi$  is fixed to 1 and 0.8.
Figure A3: Sensitivity analysis of hyper-parameters on different datasets based on ResNet-18.
C.5 Ablation Study and Analysis
Ablation of hyper-parameters We first provide ablation experiments of the shared hyper-parameter  $\eta$  in DM(CE) and DM(BCE). In Figure A2 left, the static (Mixup and CutMix) and the dynamic methods (PuzzleMix and AutoMix) prefer  $\eta = 0.1$  and  $\eta = 1$ , respectively, which might be because the dynamic variants generate more discriminative and reliable mixed samples than the static methods. Then, Figure A2 middle and right show that ablation studies of hyper-parameters  $\xi$  and  $t$  in DM(BCE), where cutting-based methods (CutMix and AutoMix) prefer  $\xi = 0.8$  and  $t = 1$ , while the interpolation-based policies (Mixup and ManifoldMix) use  $\xi = 1.0$  and  $t = 0.5$ .
Sensitivity Analysis To verify the robustness of hyper-parameter  $\eta$ , extra experiments are conducted on CIFAR-100, Tiny-ImageNet, and CUB-200 datasets. Figure A3 shows the results consistent with our ablation study in Sec. 5.4. Dynamic mixup methods prefer the large value of  $\eta$  (e.g., 1.0), while static ones are more like a small value (e.g., 0.1). The main reason for this is the dynamic methods generate mixed samples where label mismatch is relatively rare, relying on larger weights to achieve better results, while the opposite is true in static methods.