Harnessing Hard Mixed Samples with Decoupled Regularizer
Authors: Zicheng Liu
Siyuan Li
Ge Wang
Lirong Wu
Cheng Tan
Stan Z. Li
Venue: NeurIPS 2023 poster (2023)
Keywords: mixup
data augmentation
classification
data efficiency
TL;DR: We introduce a decoupled mechanism to enhance the training efficiency of hand-crafted mixups by mining hard mixed samples without any extra cost.

Abstract: Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. Recently, dynamic mixup methods have improved previous \textit{static} policies effectively (e.g., linear interpolation) by maximizing target-related salient regions in mixed samples, but excessive additional time costs are not acceptable. These additional computational overheads mainly come from optimizing the mixed samples according to the mixed labels. However, we found that the extra optimizing step may be redundant because label-mismatched mixed samples are informative hard mixed samples for deep models to localize discriminative features. In this paper, we thus are not trying to propose a more complicated dynamic mixup policy but rather an efficient mixup objective function with decoupled regularizer, named decoupled mixup (DM). The primary effect is that DM can adaptively utilize those hard mixed samples to mine discriminative features without losing the original smoothness of mixup. As a result, DM enables static mixup methods to achieve comparable or even exceed the performance of dynamic methods without any extra computation. This also leads to an interesting objective design problem for mixup training that we need to focus on both smoothing the decision boundaries and identifying discriminative features. Extensive experiments on supervised and semi-supervised learning benchmarks across seven datasets validate the effectiveness of DM.

Harnessing Hard Mixed Samples with Decoupled Regularizer
Zicheng Liu
Siyuan Li Ge Wang
Chen Tan
Lirong Wu Stan Z. Li
AI Lab, Research Center for Industries of the Future, Hangzhou, China; 'Zhejiang University; "Westlake University; (liuzicheng; lisiyuan; wangge; tanchen; lirongwu; stan. aa.n.zq.1i. @westlake wwwaakeeeed..... cn
Abstract
Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. Re- cently, dynamic mixup methods have improved previous static policies effectively (e.g., linear interpolation) by maximizing target-related salient regions mixed samples, but excessive additional time costs are not acceptable. These additional computational overheads mainly come from optimizing the mixed samples ac- cording to the mixed labels. However, found that the extra optimizing step may be redundant because aabel-mismatched mixed samples are informative hard mixed samples for deep models localize discriminative features. In this paper, we thus are not trying to propose a more complicated dynamic mixup policy but rather efficient mixup objective function with decoupled regularizer named Decoupled Mixup (DM). The primary effect that DM can adaptively utilize those hard mixed samples to mine discriminative features without losing the original smoothness mixup. As a result, DM enables static mixup methods to achieve comparable
exceed the performance of dynamic methods without any extra computation. This also leads to an interesting objective design problem for mixup training that we need focus on both smoothing the decision boundaries and identifying discriminative features. Extensive experiments on supervised and semi-supervised learning benchmarks across seven datasets validate the effective- ness of DM as plug- and-play module. Source code and models are available at https://gi //gttub.. cwm/ Com/Westlakee A.e--AI/ AI/ /I/openmixup.
Squirrel
Squiinel
Introduction
MxUp
Deep Learning has become the bedrock of mod- em AI for many tasks in machine learning [3] such as computer vision [19, 18], natural lan- guage processing [12]. Using a large number of learnable parameters, deep neural networks (DNNs) can recognize subtle dependencies in large training datasets to later leveraged to perform accurate predictions on unseen data. However, models might overfit the training set without constraints enough data [53]. To this
OuAMix
Mixed Cross-entropy
Decoupled Mxxpp Loss
visualization of hard mixed sample mining by class activation mapping (CAM) [49] of ResNet-50 on ImageNet. From left to right, CAM of top-2 predicted classes using mixup cross- entropy (MCE) and decoupled mixup (DM) loss.
Figure 1:
Eqqual contribution.
Stan Z. Li (Stan Zo,L. westlake.edu.cn) is the corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(1.0 Panda)
(1.0 Squirrel)
(0.3 Squirel Panda)
(0.7 Squirrel Panda)
Za
36b
Hard Mixed Sample for Squimal
Hard Mixed Sample for Panda
Illustration of the two types of hard mixed samples in CutMix with ''Suiirel' and 'Panda" as an example. Hard mixed samples indicate that the mixed sample contains salient features of a class, but the value of the corresponding label is small. MCE loss fails to leverage these samples.
Figure 2:
end, regularization techniques have been deployed to improve generalization [61], which can be categorized into data-independent or data-dependent ones [16]. Some data-independent strategies, for example, constrain the model by punishing the parameters' norms, such as weight decay [40]. Among data-dependent strategies, data augmentations [51] are widely used. The augmentation policies often rely on particular domain knowledge [58] in different fields.
Mixup [77]. a data-dependent augmentation technique, is proposed to generate virtual samples by linear combination of data pairs and the corresponding labels with the mixing ratio À [0, 1]. Recently, a line of optimizable mixup methods are proposed to improve mixing policies to generate object-aware virtual samples by optimizing discriminative regions in the data space to match the corresponding labels [56, 23, 22] (referred to as dynamic methods). However, although the dynamic approach brings some performance gain, the extra computational overhead degrades the efficiency of mixup augmentation significantly. Specifically, the most computation of dynamic methods is spent on optimizing mabel-nismatched samples, but the question of why these masel-mismatched samples should be avoided during the mixup training has rarely been analyzed. In this paper, we find these mismatched samples are completely underutilized by static mixup methods, and the problem lies in the loss function, mixed cross- cross-entropy loss (MCE). Therefore, samples are not only not static mixup disadvantages but also hard mixed samples full of discriminative information. Taking CutMix [74] an example, two types of hard mixed samples are shown on the right of Figure 2. Since MCE loss forces the model's 's predictions to be consistent with the soft label distribution, i.e., the model cannot give tigh-confidence predictions for the relevant classes even if the feature is salient in hard mixed samples, we can say that these hard samples are not fully leveraged.
argue that these mismatched
From this perspective, we expect the model to be able to mine these hard samples, i.e., to give confident predictions according to salient features for localizing discriminative characteristics, even if the proportion of features is small. Motivated by this finding, introduce simple yet effective Decoupled Mixup (DM) loss, a mixup objective function for explicitly leveraging the hard samples during the mixup training. Based on the standard mixed cross-entropy (MCE) loss, extra decoupled regularizer term is introduced to enhance the ability to mine underlying discriminative statistics the mixed sample by independently computing the predicted probabilities of each mixed class. Figure 1 shows the proposed DM loss can empower the static mixup methods to explore more discriminative features. Extensive experiments demonstrate that DM achieves data-efficiency training supervised and semi-supervised learning benchmarks. Our contributions are summarized below:
Unlike those dynamic mixup policies that design complicated mixing policies, we propose DM, mixup objective function of mining discriminative features adaptively.
Our work contributes more broadly to understanding mixup training: it essential focus not only on the smoothness by regression of the mixed labels but also on discrimination by encouraging the model to give reliable and confident predictions.
Not only in supervised learning but the proposed DM can also be easily generalized to semi-supervised learning with a minor modification. By leveraging the unlabeled data, it can reduce the conformation bias and significantly improve performance.
Comprehensive experiments on various tasks verify the effectiveness of DM, egg., DM- based static mixup policies achieve a comparable or even better performance than dynamic methods without the extra computation.

---

2 Related Work
Mixup Augmentation. As data-dependent augmentation techniques, mixup methods generate new samples by mixing samples and corresponding labels with well-designed mixing policies [77, 57, 69, 64]. The pioneering mixing method is Mixup [77], whose mixed samples are generated by linear interpolation between pairs of samples. ManifoldMix variants [57, 14] extend Mixup to the latent space of DNNs. After that, cut-based methods [74] are proposed to improve the mixup for localizing important features, especially in the vision field. Many researchers explore using nonlinear or optimizable sample mixup policies to generate more reliable mixed samples according to mixed labels, such PuzzleMix variants [23, 22, 45], SaliencyMix variants [56, 60), AutoMix variants [38, 311, and SuperMix [11]. Concurrently, recent works try to generate more accurate mixed labels with saliency information [20] attention maps [5, 9, 7] for Transformer architectures, which require prior pre-trained knowledge or attention information. On the contrary, the proposed decoupled mixup is a pluggable learning objective for mixup augmentations. Moreover, mixup methods extend to more than two elements [22, 11 and regression tasks [70]. Some researchers also utilize mixup augmentations to enhance contrastive learning [8, 21, 28, 50, 31] or masked image modeling [33, 6] to learn general representation in a self-supervised manner.
Semi-supervised Learning and Transfer Learning. Pseudo-Labeling [27] is a popular semi- supervised learning (SSL) method that utilizes artificial labels converted from teacher model predic- tions. MixMatch [2] and ReMixMatch [1] apply mixup on labeled and unlabeled data enhance the diversity of the dataset. More accurate pseudo-labeling relies on data augmentation techniques to introduce consistency regularization, e.g.. UDA [65] and FixMatch [52] employ weak and strong augmentations to improve the consistency. Furthermore, CoMatch [29] unifies consistency regular- ization, entropy minimization, and graph-based contrastive learning to mitigate confirmation bias. Recently proposed works [62, 4] that improve FixMatch by designing more accurate confidence- pscudo-label selection strategies, e.g., FlexMatch [76] applying curriculum learning for updating confidence threshold dynamically and class-wisely. More recently, SemiReward [30] proposes reward model to filter out accurate pseudo labels with reward scores. Fine-tuning a pre-trained model on labeled datasets is widely adopted form of transfer learning (TL) in various applications. Previ- ously, [13, 44] show that transferring pre- pre-trained AlexNet features to downstream tasks outperforms hand- hand-crafted features. Recent works mainly focus better exploiting the discriminative knowledge of pre-trained models from different perspectives. L2-SP [35] promotes the similarity of the final so- lution with pre-trained weights by a simple 1.2 penalty. DELTA [34] constrains the model by a subset of pre-trained feature maps selected by channel-wise attention. BSS [68] avoids negative transfer by penalizing smaller singular values. More recently, Self-Tuning variants [67, 54] combined contrastive learning with TL to tackle confirmation bias and model shift issues in one-stage framework.
3 Decoupled Mixup
3.1 Preliminary
Mixed Cross-Entropy Underutilizes Mixup Let us detine y RC as the ground-truth label with C categories. For labeled data point x E WXx whose embedded representation Z is obtained from the model M and the predicted probability p can be calculated through Softmax function p = a22). Given the mixing ratio A E [o. 1] and A-related mixup mask H E RW*H the mixed sample (a(a,)): M)) can be generated (a.b)) = H @ Fa + (1 H) @ xbb and Ma..) Ayo + (1 - A)yb- where @ denotes element-wise product, (aa)ya) and (Is, yb) are sampled from a labeled dataset = {xr., ya Va)) Note that superscripts denote the index; subscripts indicate the type of data, e.g., Z(a,b) represents mixed sample generated from va and rs; y' indicates the label value on i-th position. Since the mixup labels are obtained by somehow X-based interpolation, the standard CE loss weighted by X, LCe = vir.)) log (z(,,))) is typically used the objective in the mixup training:
LMCE = Σ (Al(sé 1) 1)logp((.)) + +(1 -))l(( =1) log P(a.))
(I)
where T(-) E (0, 1) is an indicator function that values one and only if the input condition holds. Noticeably, these two items of Equation I are classifying No and yb while keeping the linear
3

---

consistency with mixing coefficient A. As a result, DNNs with this mixup consistency prefer relatively less confident results in high-entropy behaviour [46] and longer training time in practice. The main reason is that in addition to À constraint, the competing relationships defined by Softmax in LMCE are the main cause of the confidence drop, which is more obvious when dealing with hard mixed samples. Precisely, the competition between the mixed class and b in Equation can severely affect the prediction of single class; that is, interference from other classes prevents the model from focusing its attention. This typically causes the model to be insensitive to the salient features of the target and thus undermines model performance, as shown in Figure 1. Although the dynamic mixup alleviates this problem, the extra time overhead is unavoidable if only focusing mixing policies on the data level. Therefore, the key challenge is to design an ideal objective function for mixup training that maintains the smoothness of the mixup and can simultancously explore the discriminative features without any computation costs.
3.2 Decoupled Regularizer
To achieve the above goal, we first dive into the LMCE and propose the efficient decoupled mixup. Proposition 1. Assuming I(a.b) is generated from two different classes, minimizing LMCE equiva- lent to regress corresponding X in the gradient:
exp(=ta,)) D. (xpp)%a.a))
= a
(V s(o, fMCE)*
A) + Eg (xpp(F(o.))
exp(=
i= b
exp(=/u..)) Σ. exp(=(
+ a,
Softmax Degrades Confidence. As we can see from Proposition 1, the predicted probability of Ta(..) will be consistent with X, and the probability is computed from the Softmax directly. The Softmax forces the sum of predictions to one (winner takes all), which is undesirable in mixup classification, especially when there are multiple and non-salient targets in mixed samples, e.g. hard mixed samples, as shown in Figure 2. The standard Softmax in LME deliberately suppresses confidence and produces high-entropy predictions by coupling all classes. As a consequence, LMCE makes many static mixup methods require longer epochs than vanilla training to achieve the desired results [57, 73]. Based on previous analysis, novel mixup objective, decoupled mixup (DM), is raised to remove the Coupler and thus utilize the hard mixed samples adaptively, finally improving the performance of mixup methods. Specifically, for mixed data points =(a.b) generated from a random pair in labelled dataset L, an encoded mixed representation b(a (a((((),))) is generated by feature extractor fo. A mixed categorical probability of i-th class is attained:
exp(F(a,))
( = =E. exp(2ff.))
(3)
Decoupled Softmax. where a(.) is standard Softmax. Equation 3 shows how the mixed probabil- ities are computed for mixed sample. The competition between and b is the main reason that results in low confidence of the model, i.e., the sum of semantic information of hard mixed samples are larger than "I" defined by Softmax. Therefore, we propose simply remove the competitor class in Equation 3 to achieve decoupled Softmax. The score on i-th class is not affected by the j-th class:
exp(2fa.))
olz(abi) (xp( Enp exp( ((a.)))
where o6-) is the proposed decoupled Softmax. In Equation 4, by removing the competitor, compared with Equation 1, the decoupled Softmax makes all items associated with become -1 in gradient, the derivation is given in the A. Our Proposition 2 verifies that the expected results are achieved with decoupled Softmax.
Proposition 2. With the decoupled Softmax defined above, decoupled mixup cross-entropp LDM can boost the prediction confidence of the interested classes mutually and escape from the X-constraint:
Pla.)) ) + log 1
)
P(a.))
LDM ΣΣ vlaud
log (
(5)
1- 1-P(a.)) P(a.)
P(a.b)
1=1

---

714
Top-I Accuracy (56)
71.
*MiRUs + DDACI
CoMisup
14
united
*CuMix DMICE)
77,4
ProdeeMii
712
212
Top
OMICE
Mixl?
Top-4
12
60
Epochs
Training Time Mlours/100ep)
Mixup
ManifolMMx
Figure 3: Results illustration of applying decouple mixup. Left: taking MixUp as an example, our proposed decoupled mixup cross-entropy, DM(CE), significantly improves training efficiency by exploring hard mixed sample; Middle: Acc vs. cost on ImageNet-lk: Right: Top-2 acc is calculated when the top-2 predictions equal {b).
The Decoupled Mixup. the original smoothness of LACE should not be lost, and thus the proposed DM is regularizer for discriminability. The final form of decoupled mixup be formulated as follows:
The proofs of Proposition 1 and are given in the Appendix. In practice,
CDMI(CE) = (V(a,, (z((6 + (
log(@(=()))) Ma
LMCE
CDM
where D(a,b) indicates the mixed label while V[a.,] is two-hot label encoding, is a trade-off factor. Notice that v is robust and can be set according the character of mixup methods (see Sec. 5.4).
Practical consequences of such simple modification on mixup and the performance:
Make What Should be Certain More Certain. As we expected, mixup training with a decoupling mechanism will be more accurate and confident handling hard mixed samples with our artificially constructed hard mixed samples by using PuzzleMix. Figure 3 right demonstrates the model trained with decoupled mixup mostly doubled the top-2 accuracy on these mixed samples, which also verifies the information contained in mixed samples is beyond the "I" defined by standard Softmax. More interestingly, this advantage of decoupled mixup. i.e., higher confidence and accuracy, can be further amplified in superviid learning due the uncertainty of pseudo-labeling.
Enhance the Training Efficiency. It straightforward to notice that there no extra computation cost when using DM in vanilla mixup training, and the performance we can achieve is the same or even better than optimizable mixup policies, i.e., PuzzleMix, CoMixup, etc. Figure left and middle show decoupled mixup unveils the power of static mixup for more accurate and faster.
4 Extensions of Decoupled Mixup
With the high-accurate nature of decoupled mixup for mining hard mixed samples, semi-supervised learning is suitable scenario to propagate the accurate label from labeled space to unlabeled space by using asymmetrical mixup. In addition, we can also generalize the decoupled mechanism into the binary cross-entropy for boosting the multi-classificaiinn task.
4.1 Asymmetrical Strategy for Semi-supervised Learning
Based labeled data 7. {(ra, Va ) if we further consider unlabeled data U ((ua: va) no decoupled mixup can the strong connection between I. and U. Recall the confirmation bias [67] problem of SSL: the performance of the student model is restricted by the teacher model when learning from inaccurate pseudo-labels. To fully use the L and strengthen the teacher model to provide more robust and accurate predictions, the unlabeled data with large À can be used to mix with the labeled data to form hard mixed samples. With these hard mixed samples, we can employ decoupled mixup into semi -ume-iueeeee learning effectively. Since only the label of is accurate, we need to make little asymmetric modification to the decoupled mixup, called Asymmetrical Strategy(AS) Formally, given the labeled and unlabeled datasets I. and U, AS builds reliable connection by generating hard mixed samples between T. and U in asymmetric manner (A < 0.5):
((a.b) Aσa + (1 A)us; )(a.)) xya + (1 X)ob.
5

---

Due to the uncertainty of the pseudo-label, only the labeled part is retained in LDM:
ÊDM = log (o(z(a,b))) yb.
where Ya and yb are one-hot labels from L. AS could be regarded as a special case of DM that only decouples on labeled data. Simply replacing LDM with LDM can leverage the hard samples and alleviate the confirmation bias in semi semi-supervised learning.
4.2 Decoupled Binary Cross-entropy Loss
Binary Cross-entropy Form of DM. Differ- ent from Softmax-based classification, we can also build decoupled mixup in multi-label clas- sification tasks (1-vs-all) by using mixup binary
1.0
0-0
3.0
label scale
0.8
8-0.8
cross-entropy (MBCE) loss [63] (a(+) denotes Sigmoid rather Softmax in this case). Proposi- tion 2 demonstrates the decoupled CE mu- tually enhance the confidence of predictions for the interested classes and be free from limi- tations. Similarly, for MBCE, since it is not in- herently bound to mutual interference between classes by Softmax, we have to preserve partial consistency and encourage more confident pre- dictions, and thus propose a decoupled mixup binary cross-entropy loss, DM(BCE).
2.0
0.6
0=
1-0
0.4
p= 0.5
t-0,8
p-0
0.0
E=1.
1-0
value
Figure 4: Rescaled label of different À value.
To this end, a rescaling function r ,,,,5 + X' is designed to achieve this goal. The mixed label rescaled by r(-): Amié = daya + A6y0. where Aa and \ are rescaled. The rescaling function is
defined as follows:
r(,, = (x (2) 05405E<1,
where & the threshold, f an index to control the convexity. shown in Figure 4, Equation has three situations: (a) when ç = 0, O, the rescaled label is always equal to 1, as two-hot encoding: (b) when 5 1, r(.) a linear function (vanilla mixup): (c) the rest curves demonstrate the parameter that changes the concavity and & is responsible for truncating.
Empirical Results. In the case of interpolation-
mixup methods (egg. Mixup, ManifoldMix, etc.) that keep linearity between the mixed label and sample, the decoupled mechanism can be introduced by only adjusting threshold t. In the case of cutting- mixing policies (e.g., CutMix, etc.) where the mixed samples and labels have square relationship (generally convex function), we can approximate the convexity by adjusting ç. which are detailed in Sec. 5.4 and Appendix C.5.
Experiments
We adopt two types of top-1 classification accuracy (Acc) metrics (the mean of three trials): (i) the median top-1 Ace of the last 10 epochs [52, 38] for supervised image classification tasks with Mixup variants, and (ii) the best top-1 Acc in all checkpoints for SSL tasks. Popular ConvNets architectures are used as backbone networks: ResNet variants including
and Transformer-basee ResNet [19] (R), Wide-ResNet (WRN) [75], and ResNeXt-32x4d (RX) [66], Vision Transformers including DeiT [55] and Swin Transformer (Swin) [37].
5.1 Image Classification Benchmarks
This subsection evaluates performance gains of DM on six image classification benchmarks, including
CIFAR- 100 [25], ImageNet (Tiny) [10], ImageNet- 1k [48], CUB-200-201 (CUB) [59], FGVC- Aircraft (Aircraft) [42]. There are mainly two types of mixup methods based on their mixing policies: static methods including Mixup [77], CutMix [74), ManifoldMix [57]. SaliencyMix [56], FMix [17],
and ResizeMix [47], and dynamic mixup methods including PuzzleMix [23], AutoMix [38], and
SAMix [3 For fair comparison, we the optimal in {0.1, 0.2, 0.5, 0.8, 1.0, 2.0} for all mixup

---

Table 1: Top-1 Acc (%)1 of small-scale image classification on CIFAR-100 and Finy-ImageNet datasets based on ResNet variants.
Dutasels
CIFAR-100
Iny-lemgeNet
R-18
RX-50
WRN-28-8
R-18
RX-50
Methods
MCE
DM(CE)
MCE
DM(CE)
МСE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
Mixup
79.12
80.44
82.10
82.96
82.82
83.51
63.86
65.07
66.36
67.70
CutMix
78.17
79.39
81.67
82.39
84.45
84.88
65.53
66.45
66.47
67.46
ManifoldMix
80,35
81,05
82.88
83,15
83,24
83.72
64.15
65.45
67.30
68.48
FMix
79.69
80.12
81.90
82.74
84.21
84.47
63.47
65.34
65.08
66.96
ResizeMix
80.01
80.26
81.82
82.96
84.87
84.72
63.74
64.33
65.87
68.56
Avg. Gain
+0.78
+0.77
+0.34
+1.18
+1.62
Table 2: Top-1 Acc (%)f of image classification on Table 3: Top-1 Acc (%))* of image classification ImageNet-Ik with ResNet variants using PyTorch- ImageNet-Il Ik based on ResNet-50 using RSB A3 100-epoch training recipe.
style 100-epoch training recipe.
R-18
R-34
R-50
Methods
MCE DM(CE)
MBCE
MRCE
DM(BCE)
Methods
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
(une)
(Lwu)
(one)
Vanila
70.04
73.85
76.83
RSB
76.49
77.72
78.08
76.95
78,43
Mixup
69.98
70.20
73.97
74.26
77.12
77.41
Mixup
76.01
76.69
77.66
77.42
78.28
CutMix
68.95
69.26
73.58
73.88
77.07
77.32
CutMix
76.47
77.22
77.
67.54
78.21
ManifoldMix
69.98
70.33
73.98
74.25
77.01
77.30
ManifoldMix
76.14
76.93
77.78
67.78
78.20
70.26
74.08
74.34
77.19
77.38
FMix
76.09
76.87
77.76
73.44
78.11
ResizeMix
69.50
69.90
73.88
74.00
77.42
77.65
ResizeMix
76.90
77.21
77.85
77.30
78.32
Avg. Gam
+0.32
+0.24
+0.25
Avg. Gaun
+0.76
-4.3
+0.47
Table 4: Top-1 I Acc (%)f of classifi-
Table 5: Top-1 Acc (%)) of fine-grained image classification on CUB-200 and FGVC-Aircrafts with ResNet variants.
cation on ImageNet-Ik with ViTs.
DeiT-S
Swin-T
Datasers
CUB-200
RGVC-Airerass
Methods
MCE
DM(CK)
MCE
DM(CK)
R-18
RX-50
R-18
RX-50
Deil
79.80
80.37
81.28
81.49
Methods
MCF
DM(CE)
MCE
DMICE)
MCF
DM(CE)
MCF
DM(CE)
Mixup
79.65
80.04
80.71
80.97
Mixup
CulMix
78.40
78.39
79.90
78.76
85.68
$4.58
85.97
85.04
78.84
79.52
81.64
82.66
85.18
84.55
85,75
86.68
CutMix
79.78
80.20
80.83
81.05
ManifoldMix
79.76
79.92
80.68
82.57
86.60
FMix
79.41
80.37
80.54
FMix
77.28
80.10
84.06
84.85
79. 36
80.44
84.85
85.04
ResizeMix
79.93
80.03
80.94
81.01
ResizeMix
78.50
79.58
84.77
84.92
78.
79.54
84.08
84.51
Avg. Gain
+0.39
+0.19
Avg. Gain
+1.19
+0.35
+2.07
+0.73
algorithms and follow original sapper-prameterss in papers. We adopt the open-source codebase
OpenMixup [32] for most mixup methods. The detailed training recipes and provided in Appendix B. We also evaluate the adversarial robustness of CIFAR-100 in Appendix C.2.
are
Small-scale Classification Benchmarks. For small-scale classification benchmarks on CIFAR-100 and Tiny, we adopt the CIFAR version of ResNet variants and train with SGD optimizer following the common training settings [23, 38]. Table 1 and A2 show amall-scale classification results. The proposed DM(CE) significantly improves MCE based on various mixup algorithms. Based on CIFAR- 100 and three different CNNs (R-18. RX-50, and WRN-28-8) 8), the decoupled mixup brings an average performance gain of 0.78%. 0.77%, and 0.34%, respectively. More notably, the gains on the Tiny dataset are significant, with average performance gains of: 1.18% and 1.62% on R-18 and RX-50.
ImageNet and Fine-grained Classification Benchmarks. For experiments on ImageNet- 1k, we follow three popular training procedures: PyTorch-style setting [19], DeiT [55] setting, and RSB A3 [63] setting to demonstrate the generalizability of decoupled mixup. As shown in Table 2, 3, and 4, DM(CE) improves consistently over MCE in all mixup algorithms on three training settings we considered. The relative improvements have been calculated in the last of tables. For example, DM(CE) yields around +0.4% for mixup methods based on ResNet variants using Py'oocch-style and RSB A3 settings; around +0.5% and +0.2% for all methods based on DeiT-S and Swin-T using DeiT setting. Notice that MBCE(two) denotes using two-hot encoding for corresponding mixing classes, which yield worse performance than MBCE, and DM(BCE) adjusts the labels for the mixing classes by Equation 6. It verifies the necessity of DM(BCE) in the case of using MBCE. As for fine-grained benchmarks, we follow the training settings in AutoMix and initialize models with the official Py Torch pre-trained models (as supervised transfer learning). Table 5 and A6 show that DM(CE) noticeably boosts the original MCE for eight popular mixup variants, especially bringing 1.53%~3.14%% gains on Aircraft based on ResNet-18.
5.2 Semi-supervised Transfer Learning Benchmarks
Following the transfer learning (TL.) benchmarks [71], we perform TL. experiments on CUB, Aircraft, and Stanford-Cars [24] (Cars). Besides the vanilla Fine-Tuming baseline, compare current
7

---

Table 6: Top-1 Acc (%)f of semi supervipe transfer learning various TL benchmarks (CUB-200,
FGrCaAir and Standfold- using only 15%, 30% and 50% labels based on ResNet-50.
CUB-200
FGVC-Airrrat
Stanford-Can
Methods
15%
30%
50%
15%
30%
50%
15%
30%
50%
Fine- TineiT
45.2511.12
19.68
20.12_00.9
39.57.10.2v 20
57.46.0.12
67.93.1022
36.77.10.1
60.63-10.18
75.1010.21
+DM
50.04:0.11
61.39:0.24
7187-0023
43.15
61.02:1..15
70.38
41.30
62.65
77.19.0.19
BSS
47.74
63.38
72.56=0.17
40.41
59.23
69.19
40.57+0.12
64.13+0.18
76.78=0.21
Co- Tuning
52.58+0.53 66.47+0.17
74.64-0.56
44.09+0.67
61.65+00522
72.73+0006
46.02+0.18
69.09
80.66+0.25
+DM
54,9622.65
68.25
75.72-0.37
49.27=0.83
65.60.20.11
=0.89
51.78:t0.34
74.15
83.02-20.29
Self-Tuming
64.17:1047
75.13
80.22.0036
64.11
76.03:10:2
81.22.10.2
72.5010:45
83.5810:25
88.1110.29
+Mixup
62.3810.32
74.65
81, 0.227
59.38.10.1
14.65
81.4610.27
70.31
83.6310.25
88.6610.21
+DM
73.0610.38
29.50
82.64
67.57...27
80.71
84.82
81.69
89.22
91.26:10.19
Avg. Gain
+5.95
+2.77
+1.34
+5.65
+4.52
+2.65
+7.22
+4.22
+2.35
Table 7: Top-1 Acc (%)f of semi-supervised learning on CIFAR-100 (using 400, 2500, and 10000
labels) based on WRN-28-8. Notice that DM denotes using DM(CE) and AS, Con denotes various unsupervised consistency losses, Rot denotes the rotation loss in Methh and CPL denotes the curriculum labeling in FlexMatch.
CIFAR-10
CIFAR-100
Methods
Losses
250
4000
400
2500
10000
Pseudo-Labeling
CE
53.51=2.20
84.92=0.19
12.55=00.5
42.26=0.28
63,45=024
MixMatch
CE+Con
86.37=0.59
93.34=026
$2.41::0.6
60.24=0.48
72.22=029
ReMixMatch
CE+Con+Rot
93.70=oas 16=0mm
1..15
73.87=035
79.08=027
MiXMatch+DM
CB+Con+DM
89. 89.16=071 20
35.72:
62.51
74.70=028
UDA
CE+Con
94.84=0.6 95.71
53.61
72.27=021
7025
FixMatch
CE+Con
95.14=0.05 95.79
53.58
71.97=0.16
77.80=0.12
FlexMatch
CE+Con+CPL.
95.02=009 1
60.06:1.62
73.51
78.
10..10
FixMutch+Mixup
CE+Con+MCE
95.05=023
95
50.63
72.16-0188
18
FixMatch+DM
CE+Con+DM
95.23 0.09
95.87=0.11
39.75
74.12-0.23
79.58 o0.17
Average Gain
+1.44
+0.95
+4.74
+2.30
+2.13
state-of-the-art TL methods, including BSS [68], Co-Tuning [71]. and Self-Tuning [67]. For a fair comparison,
the same hyper-parameters and augmentations as Self- detailed in Appendix B.2. In Table 6, we adopt DMH and AS for Fine-Tuning, Co-Tuning, and Self- using Mixup. DM(CE) and AS steadily improve Mixup and the baselines by large margins, e-g., +4.62% 9.1%% for 15% labels, +2.02% for 30% labels, and 12.09%~...15%% 3.15% for 50% labels on Cars. This outstanding improvement implies that generating mixed samples efficiently is essential for data-limited scenarios. A similar performance will be presented as well in the next SSL setting.
5.3 Semi-supervised Learning Benchmarks
Following [52, 76), we adopt the most commonly used CIFAR- datasets among the famous SSL benchmarks based on WRN-28-2 and WRN-28-8. We mainly evaluate the proposed DM on popular SSL methods MixMatch [2] and FixMatch [52], and compare with Pesudo-Labeling [27], ReMixMatch [1], UDA [65], and FlexMatch [76]. For a fair comparison,
use the same hyperpa- rameters and training settings as the original papers and conduct experiments with the open-source codebase TorchSSL [76]. detailed in Appendix B. Table shows that adding DM(CE) and AS significantly improves MixMatch and FixMatch: DM(CE) brings 1.8 1.81~2.89% gains on CIFAR-10 and 1. 1.27~33311% % gains on CIFAR- over MixMatch while bringing 1.78~4.17%% gains on CIFAR- 100 over FixMatch. Meanwhile, we find that directly applying mixup augmentations to FixMatch brings limited improvements, while Fi achieves the best performance in most cases on UUFAR-10/100 datasets. Appendix C.3 provides further studies with limited labeled data. Therefore, mixup augmentations with DM can achieve data-efficient training in SSL.
5.4 Ablation Study and Analysis
Hyperparameters and Proposed Components. Since we have demonstrated the effectiveness of DM in the above experiments, Figure and 5 verified that DM could well explore hard mixed samples. We then verify whether DM robust to hyper-parameters (full hyper-parameters in Appendix C.5) and study the effectiveness of AS in SSL:
8

---

Figure 5: Top-1 Acc
Table 8: Ablation of the proposed asymmetric strategy (AS) and DM(CE) upon Self-Tuning for semi-supervised transfer learning on CUB-200 based on R-18.
(%) of mixed samples on
ImageNet-Ie Ik validation set.
Thp- acc. of mikedd (S)
70
Methods
15%
30%
50%
100%
Self-Tuning
57.82
69.12
73.59
75.08
+MCE
63.36
72.81
75.73
76.67
10
+MCE+AS(A 2 0.5)
59.04
69.67
74.89
75.96
+MCE+AS(A < 0.5)
62.97
72.46
75,40
76.34
Mattoo-M
cuttic
+DM(CE)AA(A S 0.5)
66.17
74.25
77.68
78.52
0.6
0.2
3.4
0.6
0.8
10
Mixing ntlo 2.
(1) The only typer-paramedd 1) in DM(CE) and DM(BCE) can be set according to the types of mixup methods. We grid search 7) in (0.01, 0.1, 0. 1. on ImageNet-1k, As shown in Figure A2 left. the static (Mixup and CutMix) and the dynamic methods (PuzzleMix and AutoMix) prefer 7/ = 0.1 and rj = 1, respectively, which might be because the dynamic variants generate more discriminative and reliable mixed samples than the static methods. (2) Hyper-parameters { and in DM(BCE) can also be determined by the characters of mixup policies. We grid search { E {1. 0.9, 0.8, 0.7] and / E (2, 1.0.5, 0.3). Figure A2 middle and right show that cutting-hased -hannd methods (CutMix and AutoMix prefer { = 0.8 and while the interpolation- based policies (Mixup and ManifoldMix) use { = 1.0 and = 0.5. (3) Table 8 shows the superiority of AS(A S 0.5) in comparison to MCE and AS(A 2 0.5), while using DM(CE) and AS(A I 0.5) further improves MCE.
1,
(4) The experiments of different sizes of training data are performed to verify the data efficiency of DM. We can observe that decoupled mixup improves by around 2% accuracy without any computational overhead. The detailed results are shown in Appendix C.3.
Occlusion Robustness We also analyzed robustness against random occlusion [43] for models trained on ImageNet-IK using the of- ficial implementation". Concretely, the clas- sifier is thought be robust if it predicts the correct label given an occluded version of the image. In other words, the network learns es- sential features (e.g., semantic regions) that discriminate each class. For occlusion, we consider patch-based random masking. In particular, we split the image of 224 224 resolutions into patch size 16 x 16 and ran- domly mask M patches out of the total num- ber of N patches, where the occlusion ratio defined as N. As shown in Figure 6, the
ResNedo Rusion PochDDep
DeiT Ruaiom Putchbop
Aucene (8)
Purkidu (955)
pkm
Tig
EuT 3000
Oodanten
Figure 6: Robustness against different occlusion ratios of images for mixup augmentations using the MCE and our DM(CE) loss based on ResNet-50 (left) and DeïT-S (right) on ImageNet- RSB and DeiT denote using CutMix+Mixup (static mixup policies) in RSB A3 [63] and DeiT [55] training settings. DM(CE) improves mixups by exploring hard mixed samples. proposed DM helps various mixup methods achieve better occlusion robustness, indicating DM can force the model to learn discriminative features, e.g., image patches with semantic information that is deterministic when the occlusion ratio is high.
Conclusion, Limitations, and Border Impacts
Decoupled Mixup and Dynamic Mixups. We investigate and show two limitations of the
Table 9: Top-1 Acc (%)1 on CIFAR- 100 and Tiny.
Datasets
CIFAR-100
Imageett
decoupled mixup. Different from static mixup methods, dynamic mixup spends extra time to optimize mixing masks in input space align the mixed samples and labels. Although the op- timized mixing policies can enhance the model to find discriminative features [38], their predic- tions are also under-confident. In Table 9, we tried some advanced dynamic mixup policies, e.g.,
Backbone
WRN-28-8
RX-50
Methods
MCE
DM(CE)
MCE
DM(CE)
PuzzleMix
85.02
85.25
67.83
68.04
+RSB
85.24
85.61
68.17
68.86
AutoMix
85.18
85.38
70.72
71.56
+RSB
85.35
85.54
70.98
72.37
Phttps:///gte com/Muzammal Naseee phopreettttttttggggrrrrrtrteeeee www.-iiinoo-Tannfrrrreee

---

PuzzleMix [23], and AutoMix [38], with decoupled mixup, and found the improvement is limited. The main reason is there will not be many hard mixed samples in dynamic mixups. Therefore, we additionally incorporate two stafic mixups in RSB training settings, i.e., half probability that Mixup or CutMix will be selected during the training. As expected, the improvements from the decoupled mixup are getting obvious upon static mixup variants. This is a very preliminary attempt that deserves more exploration future works, and we provide more results of dynamic mixups in Appendix C.
Meanwhile, we further conduct com- prehensive comparison experiments archi-
Table 10: Top-1 Acc (%)f of on CIFAR- 100 training 200 and 600 epochs based on DeiT-S and ConvNeXt-T. Underlines denote the top-3 best results. Total training hours and GPU memory are collected on a single A100 GPU.
with modern Transformer-basee tectures on CIFAR- 100, considering the concurrent work TransMix [5] and TokenMix [36]. As shown in Table 10, where results with f denote the offi- cial implementation and the other are based OpenMixup [32], DM(CE) enables DeiT (CutMix and Mixup) to achieve competitive performances as dynamic mixup variants like AutoMix and SAMix [31] based on ConvNeXt- without introducing extra compu- tational costs, while still performing worse than them based on DeiTT-S. Compared with specially designed la- bel mixing methods using attention maps, DM(CE) also achieves compet- itive performances to TransMix and TokenMix. How to further improve the decoupled mixup with the salient regions dynamic attention information to research similar performances of dynamic mixing variants can also be studied in future works.
Methods
DeiT-Small
ConvNeXi- Tiny
2001 ep 600 ep Mem.
Time
200 ep 600 ep Mem.
Time
Vanilla Mixup CutMix
68.50
27
78.70
80.65
4.2
10
69.98
76.35
8.2
27
81.13
83.08
4.2
10
74.12
79.54
8.2
27
82.46
83.20
4.2
10
DeiT
75.92
79.38
27
83.09
84,12
4.2
SmoothMix
67.54
80.25
8.2
27
78.87
81.31
4.2
10
SaliencyMix
69.78
76.60
8.2
27
82.82
83.03
4.2
10
AttentieMixx+
75.98
80.33
35
82.59
83.04
4.3
14
PMix
70.41
74.31
8.2
27
81.79
82.29
4.2
10
GridMix
68.86
74.96
8.2
27
79.53
79.66
4.2
10
ResizeMix
68.45
71.95
8.2
27
82.53
82.91
4.2
10
PuzzleMix
73.60
81.01
8.3
35
82.29
84.17
4.3
53
AutoMix
76.24
80.91
18.2
59
83.30
84.79
10.2
56
SAMix
77.94
82.49
21.3
58
83.56
84.98
10.3
DeiT+ TransMix
76.17
79.33
8.4
28
DeiT TokenMix
76.25
79.57
8.4
34
DeT++MI(CE)
76.20
79.92
8.2
27
83.44
84.49
4.2
10
The Next Mixup. In word, we introduce Decoupled Mixup (DM), a new objective function for considering both smoothness and mining discriminative features in mixup augmentations. The proposed DM helps static mixup methods (e.g., MixUp and CutMix) achieve comparable or better performance than the computationally expensive dynamic mixup policies. Most importantly, DM raises a question worthy of researching: is il necessary to dexign very complex mixup policies? We also find that decoupled mixup could be the bridge to combining static and dynamic mixup. However, the introduction of additional hyperparameters may take users some extra time to check on other than images other mixup methods. This also leads to the core question of the next step in the development of this work: how to design a more elegant and adaptive mixup training objective that connects different types of mixups to achieve high data efficiency? We believe these explorations and questions can inspire future research in the community of mixup augmentations.
Acknowledgement
This work was supported by National Key R&D Program of China (No. 20222D0115100, National Natural Science Foundation of China Project (No. U21A20427), and Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University. We thank the Al Station of Westlake University for the support of GPUs and thank all reviewers for polishing the manuscript.
References
[1] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi- cemi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv: /911:9//1100785 2019. 3,8
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint
19X5v:190..02249, 2019. 3,8
10

---

[3] Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
[4] Hao Chen, Ran Tao, Yue Fan, Yidong Wang. Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-uuality tradeoff in semi- supervised learning. In The Eleventh International Conference on Learning Representations, 2022.
[5] Jie-Neng Chen, Shuyang Sun, Ju He, Philip Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. In Proceedings of the Conference Computer Vision and Pattern Recognition (CVPR), 2022. 3, 10
[6] Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit- Yeung. Mixed autoencoder for supervie visual representation learning. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3
[7] Mengzhao Chen, Mingbao Lin, Zhihang Lin, Yu xin Zhang, Fei Chao, and Rongrong Ji. Smmix: Self-motivated image mixing for vision transformers. Proceedings of the International Conference on Computer Vision (ICCV), 2023. 3
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In Proceedings of the International Conference on Machine Learning (ICMI.) 2020. 3
[9] Hyeong Kyu Choi, Joonmyung Choi, and Hyunwoo J. Kim. Tokenmixup: Efficient attention- guided token-level data augmentation for transformers. Advances in Neural Information Processing Systems (NeurIPS), 2022. 3
[10] Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint 1 2017. 6, 19
[11] Ali Dabouei, Sobhan Soleymani, Fariborz: Taherkhani, and Nasser M Nasrabadi. Supermix: Supervising the mixing data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2021. 3
[12] Jacob Devlin, Ming- Beig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
[13] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman. Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the International Conference on Machine Learning (ICML), 2014.
[14] Mojtaba Faramarzi, Mohammad Amini, Akilesh Badrinaaraayanan, Vikas Verma, and Sarath Chandar. Patchup: A regularization technique for convolutional neural networks. arXiv preprint arXiv:2006.07794 2020. 3
[15] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing ad- versarial examples. In International Conference on Learning Representations (ICLR), 2015.
22
[16] Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup locally linear out-of-manifold regularization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 3714-3722, 2019. 2
[17] Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, and Adam Prügel- Bennett Jonathon Hare. Fmix: Enhancing mixed sample data augmentation. arXiv preprint arXiv:2002. 12047, 2(3):4, 2020. 20
[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. Proceedings of the International Conference on Computer Vision (ICCV), 2017. 1
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. 7,21

---

[20] Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3
[21] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[22] Jang- Hyyn Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. In International Conference on Learning Representa- lions (ICLR), 2021. 2,
[23] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning (ICML), pages 5275-5285. PMLR, 2020. 3, 6, 7, 10, 19, 20, 22
[24] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13), 2013. 7, 19
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6, 19
[26] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (NeurIPS), pages 1105, 2012. 19
[27] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, page 896, 2013. 3, 8
[28] Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. I-mix: domain-agnostic strategy for contrastive representation learning. In International Conference on Learning Representations (ICLR), 2021.
[29] Junnan Li, Caiming Xiong, and Steven Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. Proceedings of the International Conference on Computer Vision (ICCV), 2021. 3
[30] Siyuan Li, Weiyang Jin, Zedong Wang, Fang Wu, Zicheng Liu, Cheng Tan, and Stan Z. Li. Semireward: A general reward model for semi-supervised learning. ArXiv, abs/2111.15454, 2023.
[31] Siyuan Li, Zicheng Liu, Zedong Wang, Di Wu, Zihan Liu, and Stan Z. Li. Boosting discrimina- tive visual representation learning with scenario-agnostic mixup. ArXiv, abs/2111 0tos/2111.1545 15454, 2021.
3. 6, 10, 20
[32] Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, and Stan Z. Li. Openmixup: Open mixup toolbox and benchmark for visual representation learning. e- AI AAAIopeemxxuu 2022. 7, 10, 19,20 20
[33] Siyuan Li, Di Wu, Fang Wu, Zelin Zang. Kai Wang, Lei Shang. Baigui Sun, Haoyang Li, and Stan.Z.Li. Architecture-agnostic masked image modeling from vit back to cnn. In International Conference on Machine Learning (ICMI), 2023.
[34] Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. In International Conference on Learning Representations (ICLR), 2019.
[35] Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. Proceedings of the International Conference on Machine Learning (ICML), 2018.
12

---

[36] Jihao Liu, B. Liu, Hang Zhou, Hongsheng Li, and Yu Liu. Tokenmix: Rethinking image mixing for data augmentation in vision transformers. In European Conference on Computer Vision (ECCV), 2022. 10
[37] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision (ICCV), 2021. 6
[38] Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Liu Zihan, and Stan Li. Automix: Unveiling the power of mixup for stronger classifier. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2022. 3, 6, 7, 9, 10, 19, 20, 22
[39] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016. 19
[40] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXtv:1711.05101, 2017. 2
[41] Uly Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations (ICLR), 2019. 19
[42] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint 1rXiv:1306.5751, 2013. 6, 19
[43] Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296-23308, 2021. 9
[44] Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivie. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2014. 3
[45] Joonhyung Park, June Yong Yang, Jinwoo Shin, Sung Ju Hwang, and Eunho Yang. Saliency grafting: Innocuous attribution-guided mixup with calibrated label mixing. In AAAI Conference on Artificial Intelligence, 2022. 3
[46] Francesco Pinto, Harry Yang, Ser-Nam Lim, Philip HS Torr, and Puneet K Dokania. Regmixup: Mixup as a regularizer can surprisingly improve accuracy and out distribution robustness. arXiv preprint arXiv: 1450. 2022. 4
[47] Jie Qin, Jiemin Fang. Qian Zhang, Wenyu Liu, Xingang Wang, and Xinggang Wang. Resizemix: Mixing data with preserved object information and true labels. arXiv preprint 1rXiv:2012.11101, 2020. 6, 20
[48) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large seale visual recognition challenge. International journal of computer vision (IJCV), pages 21 211-252, 2015.
[49] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- localization. arXiv preprint 1r100:161..02391, 2019. 1
[50] Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un- mix: Rethinking image mixtures for unsupervised visual representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021. 3
[51] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data 6(1):1-48 2019. 2
[52] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 3, 6. 8. 19
13

---

[53] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research (JMLR), 15(1):192--1958 2014.
[54] Cheng Tan, Zhangyang Gao, Lirong Wu, Siyuan Li, and Stan Li. Hyperspherical consistency regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7244-7255, 2022. 3
[55] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML), pages 10347-10357, 2021. 6, 9, 19, 21
[56] AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bac, et al. Salien- eymix: A saliency guided data augmentation strategy for better regularization. arXiv preprint 1xXiv:200600199/ 2020. 2, 6, 20
[57] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning (ICML), pages
PMLR, 2019.
4. 6, 20
[58] Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoe Le. Towards domain- agnostic contrastive learning. In International Conference Machine Learning (ICML), pages 10530-10541. PMLR, 2021. 2
[59] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011. 6, 19
[60] Devesh Walawalkar, Zhiqiang Shen, Zechun Liu, and Marios Savvides. Attentive cutmix: An enhanced data augmentation approach for deep learning based image classification. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3642-3646, 2020. 3
[61] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning (ICML), pages 1058-1066. PMLR, 2013. 2
[62] Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang. Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, et al. Freematch: Self-adaptive thresh- olding for supersupe learning. In The Eleventh International Conference on Learning Representations, 2022. 3
[63] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm, 2021. 6, 9, 19, 21
[64] Lirong Wu, Jun Xia, Zhangyang Gao, Haitao Lin, Cheng Tan, and Stan Z Li. Graphmixup: Improving class-imbalanced node classification by reinforcement mixup and self-supervised context prediction. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pages 519-535 Springer, 2022.
[65] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv: 12848, 2019. 3, 8
[66] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 1492-1500, 2017.
[67] Wang Ximei, Gao Jinghan, Long Mingsheng, and Wang Jianmin. Self-tuning for data-efficient deep learning. Proceedings of the International Conference on Machine Learning (ICML., 2021. 3, 5, 8
14

---

[68] Chen Xinyang, Wang Sinan, Fu Bo, Long Mingsheng, and Wang Jianmin. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 3, 8
[69] Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. Advances Neural Information Processing Systems, 35:3361-
3376, 2022. 3
[70] Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y. Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[71] Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 7,8
[72] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference Learning
Representations (ICLR), 2020. 19
[73] Hao Yu, Huanyu Wang, and Jianxin Wu. Mixup without hesitation. International Conference on Image and Graphics (ICIG), pages 143-154. Springer, 2021. 4
[74] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6023-6032, 2019. 2, 3, 6, 19, 20, 22
[75] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.
[76] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting supervupe learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3, 8, 19
[77] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17Xiv:1710.09441 2017. 2, 3, 20, 21, 22
15

---

Appendix
In the Appendix sections, we provide proofs of proposition (SA.1) and proposition 2 ($A.2), implementation details ($B), and more results of comparison experiment and empirical analysis ($C).
A Proof of Proposition
A.1 Proof of Proposition
Proposition 1. Assuming *(a,)) is generated from two different classes, minimizing LMCE is equivalent to regress corresponding À in the gradient of LMCE:
exp(+ia.s))
A Ee exp(=(n.0)
1=
expls
(V (Vic..o fACE)*
-(1 A) + Eo expl (xp(=(o,))) exple(n.0)) Ee explefa.b))
1 1=1 j
+ i.j
Proof. For the mixed sample (I(a.))) M(),6)), (a) is derived from a feature extractor fo (i.e 2(a,b)
fo(F(a)))). According to the definition of the mixup cross-entropy loss LMCE, we have:
OCMCE 0z(a,b)
(Vaaa)))MCC))
(Ma.s log (o(z(a)))))
Dz(a,s)
2 (log( explz(as))
(iad dzt
ES= exp(e(ass)) 1)
(a.b)
Es-eexpleee exple(a.))
2 exp(s(as))
(Mlai))
Oz((mb ( exp(= (a.))
exp(z(an))
Σj= exp(e(a)))
exp( (xp(((a,)))
ΣØ= == exp(=(a.b)
Similarly, have:
(V (Vs(a,))LMM)) dLDM
θ
MaoD log (H (H(aa.))))aa
T
Dz(a,s)
dz(a.)
vn log( exp(s(a.)))
vdl log( expls(es))
(Σ
)ad +
Σ
DeSt exp(= 1bh)
0z(a.b)
De Daysy expl (ai))
i,==1
i1=1
(log( expl<(a.6)
exp(<(at))
Whith 0= 0=((a))
+ log(
Ligs exp(=ta.)))
De
kapi exp(sta.b)
Ener exp(=
E Exei explet (x.()(
Σ
Whutd (6)
+ 5
EKety exp(=t
Eeprexp exp(=ta.)))
Exys (xp(=((,o))0
Enys exp(=ta,))
)8%
+
8
-8/1 81.
Eeys exp(st..))
Eeeg exp(2 exp(2ta.)) (a)))
16

---

Thus, for DDM loss:
exp(=(a,b)
-1 + Eeeey expl=(u.))
1=
expl=(a.s))
(V Vi.a, fMCK)*
-1 Zes exp(
1 [=j j
(8)
exp(=(a.0)) Lezi exp(=(a.)))
exp(s(a.0)
1 i.j
Lezs expleian))
17

---

A.2 Proof of Proposition 2
Proposition 2. With the decoupled Softmax defined above, decoupled mixup cross-entropy LDM(CR) can boost the prediction confidence of the interested classes mutually and escape from the X- constraint:
sauu ( log ( P(a.0) log ( P(a,b)
P(a.,) P(a.0)
LDM(CE)
1-P(a.))
i=1==1
Proof. For the mixed sample (d M(a.)),, F(a,b) is derived from a feature extractor So (i.e [(a.))=/ol((.o)))> According to the definition of the mixup cross-entropy loss LDMICE), we have:
LDM(CE) V/o.b) log (H(Z [ia.b]
- Va log (11 (Z(,)))))) ub + ve log (H((),))))
Va log exp(z(av)
exp(=(a,)) Eees expl= )ub
vbb +
vá log(
E (xp(=((,)))
w=1
exp(z(a.))
expl (a.1)
Σ
vaul (log(
+ log
Eleg exp(= (a,b))
Exzfi exp(=
1==1
exp(sto,)))
exp(=(o.
En exp(+
Eiiny exp(s
Σ
yavl logg(
+ log
Ens expl= 2 exp(=
2 expls
i
E== exp(=
Pla.))
P(aa.)
vaul logg(
) + log(
1),
--p(a,)) P((a.b)
1 P((ab))
where P(a.t) (z(a,))).
18

---

B Implementation Details
B.1
Dataset
We briefly introduce used image datasets. (1) Small scale classification benchmarks: CIFAR- 10/100 [25] contains 50,000 training images and 10,000 test images in x2x32 resolutions, with 10 and 100 classes settings. Tiny-ImageNet [10] is a rescaled version of ImageNet-Ik, which has 10,000 training images and 10,000 validation images of 200 classes in 64x64 resolutions. (2) Large scale classification benchmarks: ImageNet-1K [26] contrains 1,281,167 training images and 50,000 validation images of 1000 classes in 224x224 resolutions. (3) Small-scale fine-grained
classification scenarios: CUB-200-2011 [59] contains 11,788 images from 200 wild bird species for
fine-grained classification. FGVC-Aircraft [42] contains 10,000 images of 100 classes of aircraft.
Standford- [24].
B.2 Training Settings
Small-scale image classification. As for small-scale classification benchmarks on CIFAR-100 and Tiny-ImageNet datasets, adopt the CIFAR version of ResNet variants, i.e., using 3 x 3 convolution instead of the 7 x 7 convolution and MaxPooling in the stem, and follow the common training settings [23, 38]: the basic data augmentation includes RandomFlip ip and RandomCrop with 4 pixels padding: SGD optimizer and Cosine learning rate Scheduler [39] are used with the SGD weight decay of 0.0001, the momentum of 0.9, and the Batch size of 100; all methods train 800 epochs with the basic learning rate Ir- = 0.1 on CIFAR-100 100 and 400 epochs with Ir = 0.2 on Tiny-ImageNet.
Fine-grained image classification. As for fine- aine-griiedd classification experiments on CUB-200 and Aircraft datasets, all mixup methods are trained 200 epochs by SGD optimizer with the initial learning rate ir 0.001, the weight decay of 0.0005, and the batch size of 16. We use the standard augmentations RandomFlit ip and wenddoRResizedCree and load the official Py pre-trained models on ImageNet- 1k as initialization.
ImageNet image classification. For large-scale classification tasks on ImageNet-Ik, we evaluate mixup methods on three popular training procedures, and Tab. Al shows the full training settings of the three settings. Notice that DeiT [55] and RSB A3 [63] settings employ Mixup and CutMix with a switching probability of 0.5 during training. (a) Py Pyrcch-style setting. Without any advanced training strategies, PyTorch-style setting is used to study the performance gains of mixup methods: SGD optimizer is used to train 100 epochs with the SGD weight decay of 0.0001, momentum of 0.9, batch size of 256, and the basic learning rate of 0.1 adjusted by Cosine Scheduler. Notice that we replace the step learning rate decay with Cosine Scheduler [39] for better performances following [74]. (b) DeiT [55] setting. We the DeiT setting verify the DM(CE) effectiveness in training Transformer-base batch size of 1024, the basic learning rate of 0.001, and the weight decay of 0.05. (c) RSB A3 [63] setting. This setting adopts similar training techniques as DeiT to ConvNets, especially using MBCE instead of MCE: LAMB optimizer [72] is used train 100 epochs with the batch size of 2048, the basic learning rate of 0.008, and the weight decay of 0.02. Notice that DeiT and RSB A3 settings use the combination of Mixup and CutMix (50% random switching probabilities) as the baseline.
networks: Adam W optimizer [41] is used to train 300 epochs with a
Semi-supervised transfer learning. For semi-supervised transfer learning benchmarks, we use the and augmentations as Self-Tuning": all methods are initialized by PyTorch pre-trained models on ImageNet-I k and trained 27k steps in total by SGD optimizer with the basic learning rate of 0.001, the momentum of 0.9, and the weight decay of 0.0005. We reproduced Self- Self-Tuning and conducted all experiments in OpenMixup [32].
same
Semi-supervised learning. For semi-superriied learning benchmarks (training from scratch), adopt the most commonly used CIFAR- 10/100 datasets among the famous SSL benchmarks based on WRN-28-2 and WRN-28-8 following [52, 76]. For fair comparison, we use the same hyperparameters and training settings the original papers and adopt the open-source codebase TorchSSL [76] for all methods. Concretely, we use an SGD optimizer with a basic learning rate of
>https//ggihhbb wttpcc/ww...oeeumm///ettt/ttiite
19

---

Table Al: Ingredients and typer-parameters used for ImageNet-Ile Ik training settings,
Procedure
PyTorch
DeiT
RSB A3
Train Res
2242
224
2242
Test Res
2242
224
2242
Test crop ratio
0.875
0.875
0.95
Epochs
100/300
300
100
Batch size
256
1024
2048
Optimizer
SGD
AdamW
LAMB
LR
0.1
1 x 10-3
8 x 10-3
LR decay
cosine
cosine
cosine
Weight decay
10-4 4
0.05
0.02
optimizer momentum
0.9
31, 82
0.9, 0.999
X
Warmup epochs
X
5
Label smoothing
X
0.1
X
Dropout
X
Stoch. Depth
0.1
0.05
Repeated Aug
X
/
/
Gradient Clip.
1.0
X
H. flip
RRC
/
Rand Augment
X
9/0.5
6/0.5
Auto Augment
X
Mixup alpha
0.8
0.1
Cutmix alpha
x
1.0
1.0
Erasing prob.
x
0.25
X
ColorJitter
x
EMA
0.99996
X
CE loss
X
BCE loss
X
V
Ir 0.03 adjusted by Cosine Scheduler, the total 2 steps, the batch size 64 for labeled data, and the confidence threshold 0.95.
B.3 Hyper-paranee parameter Settings
We follow the basic
settings (e.g., a) for mixup variants in OpenMixup [32]. where reproduce most comparison methods. Notice that static methods denote Mixup [77]. CutMix [74], ManifoldMix [57], Saliency Mix [56], FMix [17], ResizeMix [47], and dynamic methods denote PuzzleMix [23]. AutoMix [38], and SAMix [31]). Similarly, interpolation based methods denote Mixup and ManifoldMix while suuting-based methods denote the rest mixup variants mentioned above. We the hyper-parameters of DM(CE) follows: For CIFAR-100 and ImageNet-Ile 1k, static methods use = 1, and dynamic methods use 1. For Imay-ImageNet and fine-grained datasets, static methods use 1. As for the interpolation- based methods use 0.5 and Note that we use t and 0.8. while
7j based on ResNet- 18 while 77 0.1 based ResNeXt-50: dynamic methods of DM(BCE) on ImageNet- Ik, cutting-based methods use
0.2 and a for the static and dynamic methods when using the proposed DM.
Table A2: Top-1 Acc (%)1 of small-scale image classification on CIFAR-100 and Tiny-ImageNet datasets based on ResNet variants.
Datasets
CIFAR- 100
Tiny ImageNet
R-18
RX-50
WRN-28-8
R-18
RX-50
Methods
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
SaliencyMix
79.12
79.28
81.53
82.61
84.35
84.41
64.60
66.56
66.55
67.52
PuzzleMix
81.13
81.34
82.85
82.97
85.02
85.25
65.81
66.52
67.83
68.04
AutoMix
82.04
82.32
83.64
83.94
85.18
85.38
67.33
68.18
70.72
71.56
SAMix
82.30
82.40
84.42
84.53
85.50
85.59
68.89
69.16
72.18
72.39
Avg. Gain
+0.19
+0.40
+0.15
+0.95
+0.56
20

---

Table A3: Top-1 Acc (%)f of image classification on Table A4: Top-1 Acc (%)t of image classification ImageNet-Ik 1k with ResNet variants using PyTorch- on ImageNet- Ik based on ResNet-50 using RSB style 100-epoch training recipe.
A3 100-epoch training recipe.
R-18
R-34
R-50
Methods
MCE
DM(CK)
MBCE
MBCR
DM((CE)
Methods
MCE
DM(CE)
MCE
DM
MCE
DM(CE)
(one)
(two)
(one)
SaliencyMix
69.16
69.57
73.56
73.92
77.14
77.42
MaliencyMix
76.35
77.25
77.93
72.74
78.24
PuzzleMix
70.12
70.32
74.26
74.51
77.54
77.71
PuzzleMix
77.27
77.60
78.02
77.19
78.15
AutoMix
70.51
70.64
74.52
74.77
77.91
78.15
AutoMix
77.45
77.82
78.33
77.46
78.62
SAMix
70.85
70.90
74.96
75.10
78.11
78.36
SAMi
78.33
78.64
77.58
78.75
Avg. Gain
+0.20
+0.25
+0.23
Avg. Gain
+0.30
-1.99
+0.04
Table A5: Top-1 Ace (%)) of classifi- Table A6: Top-1 1 Acc (%)J of fine-grained image classification
cation on ImageNet- k with ViTs.
on CUB-200 and C-rrrrfsss with ResNet variants.
DeïT-S
Swin-T
Datasets
CUB-200
FGVC-Airenfts
Methods
MCE
DM(CE)
MCE
DM(CE)
R-18
RX-50
R-18
RX-50
Deil
79,80
80.37
81.28
81.49
Methods
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
MalieneyMix
79.32
79.86
80.68
80.83
SaliencyMix
77.95
78,28
83.29
84.51
50.02
81.31
84.31
85.07
PuzzleMix
79.84
80.25
PuzzleMix
78.63
84.51
84.67
80.76
86.23
86.36
AutoMix
80.78
80.91
81.80
81.92
AutoMix
79.87
81.08
86.56
86.74
81.37
82.18
86.69
86.82
SAMix
80.94
81.12
81.87
81.97
SAMix
81.11
81.27
86.83
86.95
82.15
83.68
86.80
87.22
Avg. Gain
+0.32
+0.13
Avg. Gain
+0.45
+0.42
+0.94
+0.36
C More Experiment Results
C.1 Image Classification Benchmarks
Small-scale classification benchmarks. For small-scale classification benchmarks on CIFAR- 100 and Tiny-Imagee ImageNet, we also conduct experiments of applying the proposed DM(CE) to dy- namic mixup methods even though these algorithms have achieved high performance in Table A2: DM(CE) brings 0.23%~0.36%% on CIFAR-100 for the previous state-of-the-ee PuzzleMix and brings 0.21%~O..2 0.27% on Tiny-mmageNet for the current sfate-of-the-an art method SAMix. Overall, the proposed DM(CE) produces +0. 15~0..% and 0.56~00.5% average gains on CIFAR-100 and Tiny-ImageNet, demonstrating its generalizability to advanced mixup augmentations.
ImageNet and fine-grained classification benchmarks. For experiments ImageNet-Ik, we also employ the proposed DM(CE) to dynamic mixup approaches on ImageNet-Ik with PyTorch-style [19], DeiT [55], and RSB A3 [63] training settings to further evaluate the generalizability of decoupled mixup. As shown in Table A3 and Table A4, DM(CE) gains +0.2~~0.9% top-1 accuracy over MCE in average for four dynamic mixup methods based on ResNet variants ImageNet- 1k; Table A5 show DM(CE) also improves dynamic methods based on popular DeiT-S and Swin-T backbones with modern training recipes. These results indicate that the proposed decoupled mixup can also boost these dynamic mixup augmentations with high performances on ImageNet- 1 Moreover, the proposed DM(CE) can improve dynamic mixup variants fine-grained classification benchmarks, as shown in Table A6, with around 00.4%0.9% average gains over MCE based on ResNet variants.
Table A7: Top-1 1 Ace (%)* and FGSM error (%)4 CIFAR-100 and Tiny-ImageNet based ResNet-18 training 400 epochs.
Datasets
CIFAR-100
Tiny- ImageNet
Acc(%)t
Error( %)..
Acc(%))
Error(
Methods
MCE
DM(CE)
MCE DM(CE)
MCE
DM(CE)
MCE DM(CE)
Mixup
79.34
79.70
70.28
70.05
63.86
65.07
89.06
88.91
CulMix
79.58
79.77
87.43
86.84
65.53
66.45
89.14
88.79
ManifoldMix
80.18
81.06
72.50
72.19
64.15
65.45
88.78
88,52
PuzzleMix
80.22
80.58
79.76
79.53
65.81
66.13
91.83
92.05
AutoMix *
81.78
81.96
69.94
69.80
67.33
68.18
88.37
88.34
C.2 Adversarial Robustness
Since mixup variants are proven to enhance the robustness DNNs against adversarial samples [77], compare the robustness of the original MCE and the proposed DM(CE) by performing the
21

---

Top-1 acc. mixed (%)
70
60
50
50
40
Acc. mixed
40
MixUp top-1
MixUp top-2
CutMix top-1
30
CutMix
MCE
20
AutaMix top-1
DMICE)
X top-2
10
10
0.0
0.4 0.6 0.8 1.0
20 40
60 80
100
Mixing ratio X Figure A1: Experimental overviews of hard mixed sample mining. Left: Top-I and top-2 accuracy of mixed data based on ResNet-50 trained 100 epochs on ImageNet- 1k. Prediction is counted as correct if the top-1 prediction belongs {yo: 3: prediction is counted as correct if the top-2 predictions are equal to (3a. }. Compared with static policies like Mixup [77] and CutMix [74], the dynamic method AutoMix [38] significantly reduces the difficulty of mixup classification and alleviates the label mismatch issue [23] by providing more reliable mixed samples but also requires large computational overhead. Right: Taking Mixup as an example, our proposed decoupled mixup cross-entropy, DM(CE), significantly improves training efficiency by exploring hard mixed samples and alleviates the label mismatch issue.
Epochs
FGSM [15] white-box attack of 8/255 &o epsilon ball following [23]. Table A7 shows that DM(CE) improves top-1 Acc of MCE while maintaining the competitive FGSM error rates for five popular mixup algorithms, which indicates that DM(CE) can boost discrimination without disturbing the smoothness properties of mixup variants.
C.3 Data-efficient Mixup with Limited Training Labels
To further DM whether data-cfficient mixup training can be truly achieved, conducted supervised experiments on CIFAR-I00 with different sizes training data. 15%, 30%, and 50% of the CIFAR- 100 data are randomly selected training data, and the test data unchanged. The proposed decoupled mixup uses DM(CE) as the loss function by default. From Table A8, we can see that DM improves performance consistently without any computational overhead. Especially when using only 15% of the data, DM can improve accuracy by 2%. Therefore, combined with the experimental results of semi-supervin supervised learning in Sec. 5.3 and Sec. 5.2, say that mixup training with DM more data-efficient with limited data.
Table A8: Top-1 Ace (%)t of image classification CIFAR-100 with ResNet- 18 using 15%. 30%, and 50% labeled training sets.
15%
30%
50%
Methods
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
Vanilla
42.48
56.41
64.32
Mixup
42.23
44.39
55.61
56.78
64.55
65.92
CutMix
43.81
44.85
55,99
57.14
64.38
65.87
SaliencyMix
42.95
44.01
55.42
56.51
64.56
66.10
PuzzleMix
42.67
43.87
56.19
57.36
64.74
66.26
Avg. Gain
+1.36
+1.14
+1.48
C.4 Empirical Analysis
In addition occlusion robustness in Figure 6, we analyze the top-1 and top-2 mixup classification accuracy and visualize validation accuracy curves during training to empirically demonstrate the effectiveness of DM in Figure Al.
22

---

DM/CE)
DM|BCE)
145
782
78.0
78.0
Top-1
73.
77.8
1
Misuppe-11
73.0
CatMortaa. 5)
77.6
VonfokaMiw(o013
725
74.5
77.4
0.3
1.0
value
value
vvaluc
Figure A2: Ablation of hyper-parameters on ImageNet-Ik based on ResNet-34, Left: analyzing the balancing weight 7 in DM(CE); Middle: analyzing & in DM(BCE) when is fixed and 0.5; Right: analyzing t in DM(BCE) when is fixed to and 0.8.
CIFAR-100
Tiny-lmageNet
CUB-200
68
80.5
Top-1 nec. (96)
67
79.5
7M
78.5
78.0
AutoMix
64
77.
0.5
0.1
0.01
2
0.5
0.1
0.01
0.5
0.1
0.01
rl value
value
Figure A3: Sensitivity analysis of paper--arrmeters on different datasets based on ResNet-18 18.
C.5 Ablation Study and Analysis
Ablation of hyper-parameters We first provide ablation experiments of the shared hyper- prper-prrameter 7 in DM(CE) and DM(BCE). In Figure A2 left, the static (Mixup and CutMix) and the dynamic methods (PuzzleMix and AutoMix) prefer 7 0.1 and 7) 1, respectively, which might be because the dynamic variants generate more discriminative and reliable mixed samples than the static methods. Then, Figure A2 middle and right show that ablation studies of syper-parameters & and in DM(BCE), where cutting-based methods (CutMix and AutoMix) prefer & 0.8 and 1, while the interpolation-basee based policies (Mixup and ManifoldMix) use ç 1.0 and t 0.5.
Sensitivity Analysis To verify the robustness of hyper-parameter 7/, extra experiments are conducted CIFAR-100, Imag-ema and CUB-200 datasets. Figure A3 shows the results consistent with our ablation study in Sec. 5.4. Dynamic mixup methods prefer the large value of (e.g. 1.0), while static ones more like small value (e.g., 0.1). The main reason for this is the dynamic methods generate mixed samples where label mismatch is relatively rare, relying on larger weights to achieve better results, while the opposite is in static methods.
23


Reviews:
[{"review_id": "04ByVpgBAC", "reviewer": "Reviewer_JzuZ", "summary": "The authors point out that while $\\textit{dynamic}$ mixup methods are shown to be effective, they induce too much computational cost. To address this issue, they propose a $\\textit{static}$ method called Decoupled Mixup, which utilizes the hard mixed samples. The authors suggest that the Softmax function will suppress the model's confidence on hard mixed samples. Based on this idea, the authors propose a decoupled mixup cross-entropy loss which uses a decoupled version of Softmax that ease the \"sum-to-one\" constraint of Softmax. This loss is then added in the standard mixup loss as a regularization term.\n\nEmpirically, the authors show that decoupled mixup improves the top-1 accuracy performance beyond some standard mixup methods on a variety of benchmark datasets. They also show that this method can be generalized with good performance on semi-supervised learning.", "questions": "1. Line 72. Should the word \"conformation\" be \"confirmation\"?\n\n2. Proposition 1, line 133. What does it mean by saying \"$\\textit{to regress corresponding}\\ \\lambda$\" in the gradient Equation (2)?\n\n3. Line 144, \"... for mixed data point $z_{(a,b)}$\". Should it be \"$x_{(a,b)}$\"?\n\n4. Line 147, \"$\\textbf{Decoupled Softmax}$\". The subtitle seems not in the right place. Probably layout error.\n\n5. Line 144-147. From my understanding the text here is to provide the definition of Softmax function and to introduce $\\sigma(\\cdot)$ as its denotation. I would simply put it in Section 3.1 before Equation (1), along with the descriptions of all other notations. \n\n6. Line 148-150. The text here is basically telling the same story as line 137-140, that Softmax suppresses the confidence of the model on hard mixed samples, the sum of whose semantic information should be more than $1$. I think they can be combined together, rather than having the idea repeated twice in one paragraph.\n\n7. Equation (4), \"$\\phi(z_{(a,b)})^{i,j}$\". Since it is suggested in the previous text that superscripts denote the index, I think here $j$ can be put as a subscript of $\\phi$ as an indication of the function, making the expression clearer.\n\n8. Line 153, \"the decoupled Softmax makes all items associated with $\\lambda$ becomes $-1$ in gradient\". Though the proof of this statement should be straightforward, can you provide it in the Appendix as well, since it's mentioned that \"the derivatiopn is given in the A.1\" while there is only a proof of Equation 1 in A.1?\n\n9. Line 180, \"multi-classification task\". Should it be \"multi-lable classification task\" to be more specific?\n\n10. Line 186, \"... the unlabeled data with large $\\lambda$ ...\" . Does it mean unlabeled data with large combination weight? i.e. the $(1-\\lambda)$ as in \"$\\hat{x}_{(a,b)}=\\lambda{x_a}+(1-\\lambda)u_b$\" actually?\n\n11. Equation under line 192. The expression in RHS (particularly $z_{(a,b)}$) is not clear enough to indicate that only the labeled part is retained in $\\mathcal{L}_{DM}$.\n\n12. Line 220 \"threshold $t$\" and line 222 \"$\\xi$\". The notations are used exchanged.\n\n13. Appendix A.1, first line of the equations under line 526. Should the minus sign \"$-$\" here be the equal sign \"$=$\"?\n\n14. The idea of raising confidence for hard mixed samples is interesting, but conventionally from the perspective of calibration, one may wish the confidence to not be too large. Will there be a contradict (or a trade-off) between leveraging hard mixed samples and improving the models' calibration performance?\n\n15. In some tasks (datasets), mixup may not necessarily create hard mixed samples, especially when the type of data doesn't have apparent semantic information, for example points on a 2D plane. Also, when manifold intrusion occurs, the true label of a mixed sample may differs from both the labels of the pair of real samples. In these cases, DM may provide no significant improvement, or even degrade the performance. Do you have any insight or principle to determine the \"level of necessity\" or \"effectiveness\" of applying DM in a given task and dataset?", "limitations": "Not many obvious limitations. The derivation of some theory statement is not complete.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The idea of utilizing hard mixed samples from the perspective of Softmax function is novel and interesting.\n\n2. The theoretical explanation of the effectiveness of the proposed algorithm is solid.\n\n3. The experiments are conducted thoroughly on plenty of tasks, datasets and Mixup methods.\n\n4. Experiments settings are explained in details, especially the configurations of the hyperparameters of the proposed new method DM, leaving great convenience for future practitioners trying to reproduce the work.\n\n5. The improvement of DM beyond standard Mixup methods is empirically shown to be significant. Also, while there isn't much improvement of DM beyond dynamic Mixup, the saved computational costs are also valuable.\n\n", "weaknesses": "1. The typesettings of some figures and tables are a bit too dense. \n\n2. A few typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your great effort and very constructive comments to help us improve the manuscript. We will address your questions one by one and make the corresponding changes in the reversion. Please note that due to compilation issues, $L$ denotes $\\mathcal{L}$\n\n---\n### Answers to questions\n> 1, 3, 4, 5, 6, 7, 9, 12, and 13, typos and writing suggestions.\n\n- Thanks so much for your careful feedback! We'll correct all typos and adopt your suggestions to polish the writing of this paper in the revised version.\n\n> 2. Proposition 1, line 133. What does it mean by saying \"to regress corresponding $\\lambda$\" in the gradient Equation (2)?\n\n- Here “regress to corresponding $\\lambda$” means equation (2) can be regarded as a regression task with the $\\lambda$ as a target when $i=a$ or $b$. Because there are gradients if the predicted probability is not exactly equal to $\\lambda$. Maybe proposition 1 could be more clear if it becomes “Assuming $x_{(a,b)}$ is generated from two different classes, minimizing $L_{MCE}$ is equivalent to a regression task taking $\\lambda$ as labels”. Besides, this equation is an explanation that implies $L_{MCE}$ suppresses prediction confidence.\n\n> 8. Line 153, \"the decoupled Softmax makes all items associated with $\\lambda$ becomes -1 in gradient\". Can you provide it in the Appendix as well?\n- Sure, similar with A.1 we have:\n\n$$\n\\\\begin{align*}\n        \\big( \\nabla_{z(a, b)} L_{DM} \\big)^{l}\n        &=\\frac{\\partial L_{DM}}{\\partial z_{\\tiny(a, b)}^l} = -\\frac{\\partial}{\\partial z_{\\tiny(a, b)}^l} \\Big(y_{[a,b])}^{T}\\log\\big(H(z_{(a, b)})\\big) y_{[a,b]} \\Big) \\\\\\\\\n        &=-\\frac{\\partial}{\\partial z_{\\tiny(a, b)}^l} \\Big( \\sum_{i,j=1}^{C} y_a^i \\log(\\frac{\\exp(z_{\\tiny(a,b)}^i)}{\\sum_{k \\neq j}^{C}\\exp(z_{\\tiny(a,b)}^j)}) y_b^j+\\sum_{i,j=1}^{C} y_a^j \\log(\\frac{\\exp(z_{\\tiny(a,b)}^i)}{\\sum_{k \\neq i}^{C}\\exp(z_{\\tiny(a,b)}^j)}) y_b^i \\Big) \\\\\\\\\n        &=-\\sum_{i,j=1}^{C}\\Big( y_{a}^{i}y_{b}^j \\frac{\\partial}{\\partial z_{\\tiny(a, b)}^l}\\big(\\log(\\frac{\\exp(z_{\\tiny(a,b)}^i)}{\\sum_{k \\neq j}^{C}\\exp(z_{\\tiny(a,b)}^k)}) + \\log(\\frac{\\exp(z_{\\tiny(a,b)}^j)}{\\sum_{k \\neq i}^{C}\\exp(z_{\\tiny(a,b)}^k)}) \\big) \\Big) \\\\\\\\\n        &=-\\sum_{i,j=1}^{C}\\Big( y_{a}^{i}y_{b}^j \\big(\\delta_i^l - \\frac{\\sum_{k \\neq j}\\exp(z_{\\tiny(a,b)}^k)\\delta_k^l}{\\sum_{k \\neq j}\\exp(z_{\\tiny(a,b)}^k)} + \\delta_j^l - \\frac{\\sum_{k \\neq i}\\exp(z_{\\tiny(a,b)}^k) \\delta_k^l}{\\sum_{k \\neq i}\\exp(z_{\\tiny(a,b)}^k)} \\big) \\Big) \\\\\\\\\n        &=\\frac{\\sum_{k \\neq i}\\exp(z_{\\tiny(a,b)}^k)\\delta_k^l}{\\sum_{k \\neq i} \\exp(z^k_{(a,b)})}+\\frac{\\sum_{k \\neq j}\\exp(z_{\\tiny(a,b)}^k)\\delta_k^l}{\\sum_{k \\neq j} \\exp(z^k_{(a,b)})} - \\delta_i^l - \\delta_j^l.\n\\\\end{align*}\n$$\nThus, for $L_{DM}$ loss:\n\n$$\n\\\\begin{align}(\\nabla_{z_{(a,b)}} L_{MCE})^l=\n    \\\\begin{cases}\n        -1+\\frac{\\exp(z^i_{(a,b)})}{\\sum_{c \\neq j} \\exp(z^c_{(a,b)})}, & l=i \n        \\\\\\\\\n        -1+\\frac{\\exp(z^j_{(a,b)})}{\\sum_{c \\neq i} \\exp(z^c_{(a,b)})}, & l=j \n        \\\\\\\\\n        \\frac{\\exp(z^l_{(a,b)})}{\\sum_{c \\neq i} \\exp(z^c_{(a,b)})}+\\frac{\\exp(z^l_{(a,b)})}{\\sum_{c \\neq j} \\exp(z^c_{(a,b)})}, & l \\neq i, j\n    \\\\end{cases}\n\\\\end{align}\n$$\n\n> 10. Line 186, does it mean unlabeled data with large combination weight?\n- Yes, it is. As we stated in Line 191, we set $\\lambda<0.5$ to achieve this result. I'm sorry for confusing your understanding, the description here is not rigorous, we would change “large $\\lambda$” to “large combination weight” in our revised version.\n\n> 11.  Equation under line 192. The expression is not clear to indicate that only the labeled part is retained in $L_{DM}$.\n- Given $ z_{(a,b)}$, the (pseudo) label index of $x_a,u_b$ is $i,j$, we have:\n\n$$\n\\hat{L_{DM}}=log(\\frac{exp(z^i_{(a,b)})}{\\sum_{k\\neq j}^C exp(z^j_{(a,b)})})\n$$\nThis means calculating the prediction of class $i$ by decoupling class $j$. Written in matrix form this becomes $y_a^T log(\\phi(z_{(a,b)}))y_b$, which means accessing the results of ground-truth label $y_a$ after coupling off the pseudo-label $y_b$. Thus, we say this formula retains the labeled part. To make it more clear, we will add an intuitive example in the revised version.\n\n> 14. Will there be a contradict (or a trade-off) between leveraging hard mixed samples and improving the models' calibration performance?\n- This is the fundamental reason why we introduced the hyperparameter $\\eta$ in $\\mathcal{L}_{DM(CE)}$. This parameter is the very trade-off between smoothness(calibration) and discrimination(leveraging hard mixed samples). the value of $\\eta$ can be referred to as 5.4(1), which generally takes 0.1 by default for static methods and 1.0 for dynamic methods.\n\n> 15. Do you have any insight or principle to determine the \"level of necessity\" or \"effectiveness\" of applying DM in a given task and dataset?\n- Although the semantic information is not as centralized in these tasks as in concrete object classification, aggregating some dispersed local features can implicitly construct semantic features with sufficient discriminative information, thus also forming implicit hard mixed samples. Therefore, we have done experiments on Place205 dataset for Decoupled Mixup and show that DM is also gainful for all kinds of mixup algorithms. We evaluate the performance gain of DM(CE) upon various mixup methods based on ResNet-18 on Place205. We follow the settings of the Place205 mixup benchmark in OpenMixup. The results are shown in the following table. We can thus conclude that our proposed DM(CE) loss can also mine hard mixed samples in the scenic tasks.\n\n| Methods     | $\\alpha$ | (MCE) | +DM(CE) |\n|-------------|----------|-------|---------|\n| MixUp       | 0.2      | 59.33 | +0.38   |\n| CutMix      | 0.2      | 59.21 | +0.49   |\n| ManifoldMix | 0.2      | 59.46 | +0.36   |\n| SaliencyMix | 0.2      | 59.50 | +0.27   |\n| FMix        | 0.2      | 59.51 | +0.22   |\n| ResizeMix   | 1        | 59.66 | +0.18   |\n| PuzzleMix   | 1        | 59.62 | +0.19   |\n| AutoMix     | 2        | 59.74 | +0.11   |"}, {"review_id": "BXMA9Qit3Y", "reviewer": "Reviewer_y8Fo", "summary": "This paper introduced a simple strategy decoupled mixup (DM) to improve the effectiveness of Mixup and its variants. Regarding the softmax result of a mixed image with a pair of classes, one class is removed from the denominator when computing the loss of the other class.  Authors provided both theoretical and empirical analyses to confirm DM has the effect of increasing the confidence of the predicted classes. DM can also be applied in semi-supervised learning and multi-label classification. Extensive experiments have been conducted covering standard image classification, semi-supervised learning and semi-supervised fine-tuning. ", "questions": "See Weaknesses", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths\n1. The work is well motivated with the idea of making confident predictions for Mixup training. \n2. The proposed idea is novel and can be combined with existing Mixup variants.\n3. DM is proved to be effective on various tasks. \n", "weaknesses": "1. Authors claimed the smoothness of Mixup can be preserved by DM, but I didn't see detailed discussions about this. The idea of DM seems contradict with smoothness of Mixup or label smoothing. To this end, the mechanism of DM is not totally clear. \n2. Why PuzzleMix and AutoMix are not evaluated in image classification tasks?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your great efforts, these valuable questions and constructive suggestions, which are exactly what the paper needs. We will take your suggestions and solve your problems one by one, and all corresponding changes will be reflected in the revision.\n\n---\n\n### Answers to questions\n\n>1.  Authors claimed the smoothness of Mixup can be preserved by DM, but I didn't see detailed discussions about this. The idea of DM seems contradict with smoothness of Mixup or label smoothing. To this end, the mechanism of DM is not totally clear.\n\n- The final form of $L_{DM(CE)}$ is composed of two parts: $L_{MCE}$ and $L_{DM}$. Our claim in Line 160 is that $L_{DM}$ is the role of the regularizer in mining hard mixed samples to improve the discriminability of the model. According to Equation 2, we can clearly see that the optimization of $L_{MCE}$ can be regarded as a regression task with $\\lambda$ as the target, which brings smoothness to decision boundaries. Therefore, we say that decoupled mixup (DM) has both the properties of smoothness and enhanced discrimination at the same time. In a word, The mechanism of DM is that when dealing with mixed samples with information greater than \"1\" (Line 135-139), it can break the limit of $L_{MCE}$ to fully utilize the extra information from hard mixed samples.\n\n> 2. Why PuzzleMix and AutoMix are not evaluated in image classification tasks?\n\n- Due to the efficiency of data augmentation, static methods are still mainly used in the main text. However, according to L331 **\" View results of dynamic mixups in the Appendix.\"**, in our supplemental submission, we have experimented with not only AutoMix and PuzzleMix in full accordance with the setup of the main text, but also included other dynamic methods, SaliencyMix and SAMix, as detailed in Tables A2, A3, A4, A5, A6, A7, and A8."}, {"review_id": "aUDj6LLsQY", "reviewer": "Reviewer_sDcS", "summary": "The authors propose a new objective function with decoupled regularizer named decoupled mixup (DM) to harness hard mixed samples and mine discriminative features adaptively. This method is available on supervised learning and semi-supervised learning. Unlike the previous approaches which propose a more complicated dynamic mixup policy with extra computation, the proposed DM can adaptively utilize those hard mixed samples to mine discriminative features without losing the original smoothness of mixup.", "questions": "Please refer to the weaknesses.", "limitations": "The authors described potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\tHarnessing hard mixed samples without losing the original smoothness of mixup is an interesting idea.\n2.     The proposed DM enables static mixup methods to achieve comparable or even exceed the performance of dynamic methods without any extra computation.\n3.\tThe authors provide lots of experiments to demonstrate the effectiveness of the proposed method.", "weaknesses": "1.\tIt is confusing about the important notations, such as i，j, a, b. The author should follow some notational conventions. \n2.\tSection 4.2 is abrupt with a poor description, confusing notation definitions, and low contextual relevance. \n3.\tEquation (2) is not closely related to the context. Please explain it in detail.\n4.\tSince Table 6 reports the experiments on transfer learning, the authors should describe that their proposed DM can be adapted to transfer learning in the related work. In addition, the title of Section 5.2 should be changed to \"Transfer Learning Benchmarks\". It might be more reasonable if the authors exchange Sections 5.2 and 5.3 since the authors pay more attention on semi-supervised learning.\n5.\tIt would be better if the authors move lines 164 - 175 and Figure 3 to Section 5. Figures 5 and 6 are too small in the font and the colors of the lines are hard to distinguish.\n6.\tSome important equations are not numbered, and there should be a ',' link between the equation and the \"where\" in the same sentence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious time and great efforts. Your insightful suggestions and professional questions are the key to improving the quality of the paper. We will address your questions one by one and make the corresponding changes in the reversion.\n\n---\n### Answers to questions\n\n> 1. It is confusing about the important notations, such as i，j, a, b. The author should follow some notational conventions.\n\nThanks for your constructive suggestion for improving the readability, we will make the notation more clear in our revised version. Actually, we have followed the famous mixup method CutMix to define the notations of mixed samples from different two samples $x_a, x_b$ and use $i,j$ as indices to access vectors/matrices, which are conventional notations in most machine learning papers. To be more clear, taking the label $y$ as example, one-hot label $y_a\\in\\mathbb{R}^C$, mixed label $y_{(a,b)}\\in\\mathbb{R}^C$. Introduce $i,j$ to access specific values , $y^i_a\\in\\mathbb{R}$. Specifically,\n$$\ny_a^k=\\\\left\\\\{\n\\\\begin{aligned}\n1& &&k=i\\\\\\\\\n0& &&k\\neq i\n\\\\end{aligned}\\right.\n$$\n$$\ny_{(a,b)}^k=\\\\left\\\\{\n\\\\begin{aligned}\n\\lambda &&k=i\\\\\\\\\n1-\\lambda &&k=j\\\\\\\\\n0 && k\\neq i,j\n\\\\end{aligned}\\right.\n$$\n\n> 2. Section 4.2 is abrupt with a poor description, confusing notation definitions, and low contextual relevance.\n- Section 4.2 is an extension of DM to multi-label classification. Since the manuscript has been too compact, some of the detailed statements have been overlooked resulting in poor presentation in this section. To improve the presentation, in an updated version we will simplify section 3.2 (e.g., merge Equation 3 into 3.1 and reduce the length). Then in section 4.2, we will add back-to-back statements to improve the coherence, e.g., \"softmax-based models cannot deal with multi-label problems, thus how to introduce DM mechanism in multi-label classification is a question worth considering.” To make the notation more clear, the rescaled $\\lambda$ is harmonized as $\\lambda'$ then in Line 212, the equation becomes $y_{(a,b)}=\\lambda'_a y_a+\\lambda'_by_b$.\n\n> 3. Equation (2) is not closely related to the context. Please explain it in detail.\n- Equation (2) shows minimizing $L_{MCE}$ is equivalent to a regression task taking $\\lambda$ as labels during the optimization. Therefore, this equation is direct evidence that $L_{MCE}$ suppresses prediction confidence. This is the motivation why we need to propose DM. We will explain this more clear at the beginning of section 3.2 in our updated version.\n\n> 4. Since Table 6 reports the experiments on transfer learning, the authors should describe that their proposed DM can be adapted to transfer learning in the related work. In addition, the title of Section 5.2 should be changed to \"Transfer Learning Benchmarks\". It might be more reasonable if the authors exchange Sections 5.2 and 5.3 since the authors pay more attention on semi-supervised learning.\n- Thanks for your constructive again! We will adopt your suggestions in the revision: a) enriching the related works about transfer learning. b) change the title of section 5.2.\n\n> 5. It would be better if the authors move lines 164 - 175 and Figure 3 to Section 5. Figures 5 and 6 are too small in the font and the colors of the lines are hard to distinguish.\n- Sure, we will re-organize the paper according to your valuable suggestions.\n\n> Some important equations are not numbered, and there should be a ',' link between the equation and the \"where\" in the same sentence.\n- Thanks for this careful review, we will fix them in the revision.\n"}, {"review_id": "rs1mh5H8Mr", "reviewer": "Reviewer_s9eU", "summary": "This paper proposes Decoupled Softmax(Eq. 4), which is an interesting improvement to the previous mixup method, which mitigates the impact of noise in mixed samples by modifying the loss.", "questions": "I have no other issues and the paper is written clearly and easily understood.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed idea is simple and effective. The manuscript has a high degree of completion and is rich in experiments.", "weaknesses": "I see no obvious disadvantages. \n\nThere are some related literatures that I think are close to the author's claim:\nThe Benefits of Mixup for Feature Learning, It found that modifying the lambda of y in the mixup loss does not significantly affect the performance of the model.\nUMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup, This paper also found that appropriate modifications can be made in the mixup loss to improve the generalization of the model.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work! These two papers you have mentioned are very interesting works, but many of the differences are worth discussing. \n\n- Although The Benefits of Mixup for Feature Learning argues the different linear interpolation parameters for features and labels can still achieve similar performance, their analysis is still limited to the standard Softmax, ignoring the fact that the semantic information of the mixed samples should be greater than \"1\". If the setting for the sum of the weights is greater than 1 and directional (not a completely random linear interpolation), then I suspect the conclusion is likely to change. \n- UMIX proposes a new mixup loss function from an uncertainty perspective, but his need for additional training time and hyperparameters to obtain the importance weight makes the method not so convenient. \n\nWe will discuss them in the related works in the revised version. Please feel free to ask any other questions!"}]

Author rebuttal:

Meta-review: This work highlights a drawback of the existing $\lambda$ mixup, which suppresses the confidence of the lower weighted class by proposing a decoupled mixup loss. The proposed solution is a simple and elegant loss (decoupled-mixup loss). It involves the removal of the opponent class in the standard softmax. The need for the proposed $\mathcal{L}_{DM}$ is well motivated (Proposition 1) 

The authors extensively show results on the supervised and semi-supervised setup and demonstrate its ability to use various existing mixup-based methods. They also show their ability to scale across different architectures ranging from ResNets, WideResNets, and Transformers. The proposed method provides significant savings in computational costs which have been a deterrent in dynamic mixup-based methods that show comparable results. 

The authors have taken into account the detailed critique by JzuZ regarding the writing which we expect to be fixed in the camera-ready version of the manuscript.
Decision: Accept (poster)