Decoupled Mixup for Data-efficient Learning
Authors: Zicheng Liu
Siyuan Li
Ge Wang
Cheng Tan
Lirong Wu
Stan Z. Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Data Augmentation
Mixup
Data-efficient Learning
Semi-supervised Learning

Abstract: Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. Recently, dynamic mixup methods have improved previous static policies effectively (e.g., linear interpolation) by maximizing salient regions or maintaining the target in mixed samples. The discrepancy is that the generated mixed samples from dynamic policies are more instance discriminative than the static ones, e.g., the foreground objects are decoupled from the background. However, optimizing mixup policies with dynamic methods in input space is an expensive computation compared to static ones. Hence, we are trying to transfer the decoupling mechanism of dynamic methods from the data level to the objective function level and propose the general decoupled mixup (DM) loss. The primary effect is that DM can adaptively focus on discriminative features without losing the original smoothness of the mixup while avoiding heavy computational overhead. As a result, DM enables static mixup methods to achieve comparable or even exceed the performance of dynamic methods. This also leads to an interesting objective design problem for mixup training that we need to focus on both smoothing the decision boundaries and identifying discriminative features. Extensive experiments on supervised and semi-supervised learning benchmarks across seven classification datasets validate the effectiveness of DM by equipping it with various mixup methods.

Under review as a conference paper at ICLR 2023
DECOUPLED MIXUP FOR DATA- LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Mixup is efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. Recently, dynamic mixup methods have improved previous static policies effectively (e.g., linear interpolation) by maximizing salient regions or maintaining the target in mixed samples. The discrepancy is that the generated mixed samples from dynamic policies are more instance discriminative than the static ones, e.g., the foreground objects are decoupled from the background. However, optimizing mixup policies with dynamic methods in input space is an expensive computation compared to static ones. Hence, we are trying to transfer the decoupling mechanism of dynamic methods from the data level to the objective function level and propose the general decoupled mixup (DM) loss. The primary effect that DM can adaptively focus discriminative features without losing the original smoothness of the mixup while avoiding heavy computational overhead. As a result, DM enables static mixup methods achieve comparable or even exceed the performance of dynamic methods. This also leads to an interesting objective design problem for mixup training that we need to focus on both smoothing the decision boundaries and identifying discriminative features. Extensive experiments on supervised and semi- supervised learning benchmarks across seven classification datasets validate the effectiveness of DM by equipping it with various mixup methods.
INTRODUCTION
Deep Learning has become the bedrock of modern Al for many tasks in machine learning (Bishop, 2006) such as computer vision (He et al., 2016; 2017), natural language processing (Devlin et al., 2018). Using a large number of learnable parameters, deep neural networks (DNNs) can recognize subtle dependencies in large training datasets be later leveraged to perform accurate predictions on unseen data. However, models might overfit the training set without constraints or enough data (Srivastava et al., 2014). To this end, regularization techniques have been deployed to improve generalization (Wan et al., 2013), which can be categorized into iata-indeeendent or data-dependent ones (Guo al., 2019). Some data-independent strategies, for example, constrain the model by punishing the parameters' norms, such as weight decay (Loshchilov & Hutter, 2017). Among data-dependent strategies, data augmentations (Shorten Khoshgoftaar, 2019) are widely used.
Mixup (Zhang et al., 2017: Yun et al., 2019), data-dependent augmentation technique, is proposed generate virtual samples by a linear combination of data pairs and the corresponding labels with the mixing ratio À [0, 1]. DNNs trained with this technique are typically more generalizable and calibrated (Thulasidasan al., 2019), whose prediction accuracy tends be consistent with confidence. The main reason is that mixup heuristically smooths the decision boundary to improve
(1.0 Panda)
(1.0 Squirrel)
CuiMix (0.6 Squirel 0.4 Panda)
MiXUp (0.2 Squirrel 0.8 Panda)
Semantic inconsistency between mixed labels and sample
Figure I: Illustration the problem of label mismatch. The red mixed labels are the ground truth.

---

Under review as a conference paper at ICLR 2023
Top-1 acc. of mixed data (%)
70
60
Acc. of mfixxdd data (%)
60
50
50
MixUp top-1
40
MixUp top-2
30
CutMix top-1
30
CutMix top-2
MCE
20
AutoMix top-1
DM((E))
AutoMix top-2
10
0.0
0.2
0.4
20
100
Mixing ratio h
Epochs
Experimental overviews of the label mismatch issue. Compared with static policies like Mixup and CutMix, the dynamic method AutoMix significantly reduces the difficulty of mixup classification and alleviates the label mismatch problem by providing more reliable mixed samples, but also brings a large computational overhead. Left: Top-1 1 and Top-2 accuracy of mixed data on ImageNet- k with 100 epochs. Prediction is counted as correct if the Top-1 prediction belongs to {ya: 3b): prediction is counted correct if the Top-2 predictions are equal to (ya: Right: Taking Mixup as an example, decouple mixup cross-entropy (DMCH) significantly improves training efficiency by alleviating the label mismatch issue from the perspective of designing a loss function.
Figure 2:
the overall robustness by regressing the mixing ratio À in mixed labels. However, a completely handcrafted mixing policies ( (Verma et al., 2019; Uddin et al., 2020; Harris et al., 2020) (referred to as static methods) may result in a mismatch issue between the mixed labels and the mixed samples, which leads to problems such as instability or slow convergence, namely label mismatch (Liu et al., 2022), as shown in Figure 1. To mitigate this problem, line of time-consuming mixup methods are proposed to improve mixing policies to generate object-aware virtual samples by optimizing discriminative regions in the data space (Uddin et al., 2020; Kim et al., 2020; 2021) (referred as dynamic methods). For example, in Figure 2 left, the dynamic methods like AutoMix (Liu et al., 2022) improve mixup accuracy significantly. The basic idea is to decouple the foreground from the background and mix their corresponding features to avoid label mismatch. However, these data-level decoupling methods require a complex deployment and an additional 1.5 times the training time as static methods, which may violate the mixup augmentations ease of use and lightness. Thus, leaving aside the design of a new mixup policy, a new question raises that can we design a loss function for mixup that takes into account both the smoothness of mixup and the discriminatory of instances without heavy computation? From a different perspective, we first regard the label- mismatched samples as hard mixed samples. In other words, even though there are not sufficient features in the mixed sample, we still expect the model to have the ability to mine these hard features to make confident predictions. Therefore, make full use of mixed samples without introducing additional computational effort to achieve data-efficient mixup training.
Motivated by this intuition, introduce Decoupled Mixup (DM), a mixup objective function for explicitly leveraging the target-relevant information of mixed samples without losing original smoothness. Based on the standard cross-entropy loss, an extra decoupled term is introduced to enhance the ability to mine underlying discriminative statistics in the mixed sample by independently computing the predicted probabilities of each mixed class. As a result, DM can further emphasize the contribution of each involved class in mixup to explore the efficient usage of existing data, Extensive experiments demonstrate that DM achieves data-efficiency training on supervised and semi-supervised learning benchmarks. Our contributions are summarized below:
Unlike those static mixup policies that suffer from the label mismatch problem, we propose DM, mixup objective of mining discriminative features while maintaining smoothness.
Our work contributes more broadly to understanding mixup training: it is essential to focus not only on the smoothness by regression of the weight of mixing but also on discrimination by encouraging the network to give a highly confident prediction when the evidence is clear.
The proposed DM can be easily generalized to semi-supervised learning by a minor modifi- cation. By leveraging the unlabeled data efficiently, it can reduce the conformation bias and significantly improve overall performance.
Comprehensive experiments various tasks verify the effectiveness of DM, e,g., DM- based static mixup policies achieve a comparable or even better performance than dynamic methods without the extra computational overhead.
2

---

Under review as a conference paper at ICLR 2023
2 PRELIMINARIES
2.1 À WEIGHTED CROSS-ENTROPY UNDERUTILIZES MIXUP
Firstly, we dive into analyzing the generation of mixed samples and the original loss function. Let us define y E RC as the ground-truth label with C' categories. For data point I R whose embedded representation z is obtained from the model .M and the predicted probability p can be calculated through a Softmax function p o(2). Given the mixing ratio X E [0.1] and À related mixup mask H E RW**, the mixed sample (a(a, M(a.)) can be generated as F(a,b) = H O Ta + (1 H) @ Tb, and V(a.b) = Aya + (1 A)ye. where @ denotes element-wise product, (Na, ya) and (rt, yb) are sampled from a labeled dataset I, = ((Ta, ya) ya)) Note that superseripts denote the class index; subscripts indicate the type of data, e.g., F((.b)) represents mixed sample related to Ta and zzi y the value on i-th position. Since the mixup labels are obtained by
linear interpolation, the standard CE loss weighted by X, LMCE = V(a,s) log or (2(a.))), ), is typically used as the optimization objective in the mixup training process, which can be rewritten as:
LMCE =
- E (AI(yá = log + +11 - 1) log P(o,))
(1)
where T(.) E (0. an indicator function that values one if and only the input condition holds; P(a,b) Noticeably, these two items of LMCE are classifying Yo and yb according to mixup coefficient A. The effect of weighting in standard cross-entropy loss is to smooth the decision space between two classes related in mixed samples. However, the condition for LMCE to be optimal is Ra T-x Ps which means LMCF only focuses on regressing À value, but not encouraging the model mine underlying statistics of mixed samples, e.g., small À value suppresses the confidence of predictions even if the mixed features are evident. Therefore, the first challenge is improve and design general and efficient objective function for mixup training that maintains the smoothness of mixup and can simultancously explore the key features related to the downstream task.
2.2 UNRELIABLE MIXUP AMPLIFIES CONFIRMATION BIAS
If we further consider semi-supervised learning (SSL), combining mixup and pscudo-labeling tech- niques can effectively improve performances with limited labeled data (Berthelot et al., 2019a). For each Ua in an unlabeled dataset U {ua, va) """ where va its corresponding pseudo label, taking MixMatch (Berthelot et al., 2019b) as an example, pseudo-label ÿ is generated by computing the average of the models predicted class distributions across all the K basic augmentations (e.g., image cropping, etc.) of un:
K
i. = arg max
M (yalkin
K
After obtaining the pseudo labels, the same mixup process as before to generate virtual samples to augment the dataset. However, if the reliability of pseudo labels is not consid- ered, the generated samples with false pseudo labels (FPLs) can even mislead the model and aggravate confirmation bias (Arazo et al., 2020), as shown in Figure 3 left. This becomes worse when mixup is employed on FPL.s (supported by Sec. 4.3); either side with larger or smaller weights can further reinforce the wrong deci- sion hyperplane and make it irreparable. Yet, if mixup is used only on labeled data, it will not be able to mine deeper into the potential information in large amount of unlabeled data. Thus, not only expect DM to improve the training efficiency in the supervised framework as Sec. 2.1 but also want to solve the confirmation bias under semi-supervision, shown in Figure 3 right.
use
classificr
False classifier
Falsc pscudo label
Figure Left: unconstrained mixup misleads the model by generating bad samples. Right: the decision bound can be correctly guided by the DM.
3

---

Under review as a conference paper at ICLR 2023
3 DATA-EFFICIENT MIXUP LEARNING
In data-cfficient mixup learning, various scenarios derived from supervised learning (SL) will be considered, including transfer learning (TL) and SSL. The mentioned label mismatch and unreliable mixup are the two main challenges in these scenarios where mixup plays an important role. Both of them will be addressed by decoupled mixup simply and effectively.
3.1 DECOUPLED MIXUP
As stated in the preliminaries 2, the major problem of À weighted CE (CMCe) that, while preserving the consistency between (((,b) and 3(a.6) to smooth the predictions but neglect the potential information of mixed samples when the mixed label and sample are mismatching.
Proposition 1. Assuming ((a,b) generated from two different classes, minimizing LMCE is equiv- alent to regress corresponding À in the gradient of LMCE:
exp(=(a,1))
+ Σ. (xp(>
1=i
exp(s(u.b)
(V f), =
11 + Ee expl=ians)
1=
exp(=ia,0))
1 + i.j
Ee exp(>((.6)
where / is the index of our interested class. As coupled with À in Equation 2. Such design deliberately suppresses the prediction confidence for the class with smaller X, although it builds linear correspondence according X. In other words, LMCE does not encourage the model to focus on the class when its information is insufficient (label mismatch), but it could have been good hard mixed sample. As consequence, LaCcE makes many static mixup methods require a long training time to achieve the desired results (Verma et al., 2019; Yu et al., 2021). Based on previous analysis, novel mixup objective, decoupled mixup, raised to learn the comprchensive features of the mixed data adaptively and improve the mixup training efficiency. Before that, we first start with an original Softmax function. For mixed data points 2(a.8) generated from random pair in labeled dataset an encoded representation =(a,b) (a())),)) is generated by feature extractor fa. A mixup categorical probability of i-th class is attained:
see, the predicted probability of ((a,b) is
(xp(((a.b))
((((,))) = Σ. exp(2(
where the result of a(-) is vector of categorical probabilities. Empirically, well-designed dynamic mixup policy that decouples the target object from the background effectively reduces label mismatch and thus improves performance. This motivated us introduce decoupling the objective function of mixup. For a mixed sample, we can achieve decoupling by omitting one of the mixed classes and thereby calculating the prediction probability distribution independently. Even if the mixed sample does not contain distinct class characteristics, the model will make relatively high confidence predictions without the competitor class, e.g., another class in the mixup. That is, the model will be encouraged to treat that sample versa, if the characteristics are obvious enough, the model's prediction is not limited by the value. Therefore, propose remove the competitor class in the standard Softmax achieve decoupled prediction. The score on i-th class is not affected by the j-th class:
hard sample, expecting to mine the underlying feature. And vice
exp(=(a.))
((=(a M1) exp(=las))) + Eej, exp(2(a.b)
where the result of ((-) matrix of categorical probabilities. By removing the opponent class, the two desired effects are achieved simultaneously: (1) breaking the À restriction on prediction; (2) encouraging more confident prediction on the current class. Compared with Equation 2, the decoupled Softmax makes all items associated with À become -1, the derivation is given in the A.
With the decoupled Softmax defined above, decoupled mixup cross-entropy LDMICE) can boost the prediction confidence of the interested classes mutually and escape from the X- constraint:
Proposition 2.
Plam Σ Noue log
log ( P(a.))
--P(a.)) )
LDM(CE) ΣΣ
1 1-p(a.)) (
P(a.t)
4

---

Under review as a conference paper at ICLR 2023
The proofs of Proposition and are given in Appendix. In this way, the scope and confidence of our target class probability are successfully decoupled from the "background", Further, according to the property of decoupled cross-entropy loss which only pays attention to the current classes, the original linear correspondence between the mixed sample and label could be corrupted. Therefore, preserve the consistency, LMCE loss is employed in the final Decoupled Mixup (DM) loss. The overall loss function of decoupled mixup can be formulated as follows:
DAM -VV(,.)) log(o(Z(a.b)) +7 Vya (og(((((,
LMOR
(DMICE)
where b(a.b) indicates the mixed label while b]a,b] is two-hot label encoding, is a trade-off factor. Notice that 7) is robust and can be set according to the character of mixup methods (see Sec. 4.4).
3.2 ASYMMETRICAL STRATEGY
Although LDM can improve the learning efficiency in labeled data, how effectively exploit the unlabeled data and reduce the confirmation bias is still problem. Recall the confirmation bias problem of SSL: the performance of the student model is restricted by the teacher model when learning from inaccurate pseudo-labels. To strengthen the teacher to provide more accurate predictions, the unlabeled data with larger À can be used to mixup with the labeled data form hard samples. Specifically, a larger À can make the unlabeled data obscure most of the information of the labeled data, but since the label is assured, the whole process of mixup can be treated hard sample generation. With the addition of these hard samples, the teacher model will have better performance and thus give more reliable pseudo-labels, Formally, given the labeled and unlabeled datasets L and U, AS builds reliable mixed samples between L and U with an asymmetric manner (A < 0.5);
(a,b) = Am + (1 A)us; 9(a.)) = Aya + (1 A)0b-
Due to the uncertainty of the pseudo-label, only the labeled part is retained in LDM(CE):
ÉDM(CE) = yı log (o(z(a.))) yo.
where Ya and yb are one-hot labels. AS could be regarded as special case of DM that only decouples the hard samples with ground-truth labels. Replacing (DMICE) with (DM(CE) can leverage the hard samples in mixed samples between and U while alleviating the problem of confirmation bias.
3.3 GENERALIZATION TO BINARY PROSS-ENTRONE
Binary cross-entropy form of DM. If we treat mixup training as a multi-label classifica- tion task (1-vs-all) using mixup binary cross- entropy (MBCE) loss (Wightman et al., 2021) (a(.) denotes Sigmoid), then can generate the decoupled loss for each class. Proposition 2 demonstrates the decoupled bofted CE can mutually enhance the confidence of predic- tions for the interested classes and be free from A limitations. For MBCE, since it is not in- herently bound to mutual interference between classes by Softmax, we need to preserve partial consistency and encourage more confident pre- dictions to propose decoupled mixup binary cross-entropy loss, DMBBEE).
0-0.8.
m.30
label scale
0.8
==
2-0.
0.6
t-0.8.
r=1.0
0.4
t-0.8 8.
0.2
c=
0.0
1=1.0
0.0
0.2
0.4
0.6
0.8
1.0
A value
Figure 4: Rescaled label of different À value.
To this end, rescaling function X, t, + X' is designed to achieve this goal. The mixed label enhanced by r(-): Umiz = Auya + A636. where da and As are rescaled. The rescaling function is defined as follows:
-
where & is the threshold, t an index to control the convexity. As shown in Figure 4, there are three situations: (a) when { 0, 0, the rescaled label is always equal to I, two-hot encoding; (b) when & = 1, = 1, r(-) is linear function (vanilla mixup); (c) the rest curves demonstrate / is the parameter that changes the concavity and & is responsible for truncating.
5

---

Under review as a conference paper at ICLR 2023
Empirical results. In the case of interpolation-bast mixup methods (e.g., Mixup, ManifoldMix, etc.) that keep a linear relationship between the mixed label and sample, the decoupled mechanism can be introduced by only adjusting the threshold t. In the case of cutting- based mixing policies (e.g. CutMix, PuzzleMix, etc.) where the mixed samples and labels have a square relationship (generally a convex function),
approximate the convex function by adjusting &.
EXPERIMENTS
We adopt two types of top-1 classification accuracy (Acc) metrics (the mean of three trials): (i) the median top-1 Ace of the last 10 epochs (Sohn et al., 2020; Liu et al., 2022), and (ii) the best top-1 Acc in all checkpoints. We report the median top- 1 Acc for image classification tasks with Mixup variants and the max top-1 Ace for SSL tasks. Popular ConvNets and Transformer- based architectures are used backbone networks: ResNet variants including ResNet (He et al., 2016) (R), Wide-ResNet (WRN) (Zagoruyko & Komodakis, 2016), and ResNeXt-32x4d (RX) (Xie et al., 2017), Transformer- based architectures including DeiT (Touvron et al., 2021) and Swin Transformer (Swin) (Liu et al., 2021). Double horizontal line in the tables splits the static and dynamic methods.
4.1
IMAGE CLASSIFICATION BENCHMARKS
This subsection evaluates performance gains of DM on six image classification benchmarks, including CIFAR-10/100 (Krizhevsky et al., 2009), Imageee (Chrabaszcz al., 2017), ImageNet-IK (Rus- sakovsky et al., 2015), CUB-200-201 1 (CUB) (Wah et al., 201 1), FGVC-Aircraft (Aircraft) (Maji et 2013). There mainly two types of mixup methods based on their mixing policies: static methods including Mixup (Zhang et al., 2017), CutMix (Yun et al., 2019), ManifoldMix (Verma et al., 2019), SaliencyMix (Uddin et al., 2020), FMíx (Harris et al., 2020), and ResizeMix (Qin et al., 2020), and dynamic mixup methods including PuzzleMix (Kim et al., 2020), AutoMix (Liu et al., 2022), and SAMix (Li al., 2021b). For fair comparison, we use the optimal a in {0.1, 0.2, 0.5, 0.8, 1.0,2.03 for all mixup algorithms and follow original spper-parameters in papers. We adopt the open-souree codebase OpenMixup (Li et al., 2022) for most mixup methods. The detailed training recipes are provided in Appendix A.3.
Table 1: Top-1 Ace (%)t of scall-scale image classification on CIFAR-100 and ImageemteNNet datasets based on ResNet variants.
Dutasets
CIFAR-100
Tiay-Nm
R-18
RX-50
WRN-28-8
R-18
RX-50
Methods
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
MCE
DM(CE)
Mixup
79.12
80.44
82.10
82.96
82.82
83.51
63.86
65.07
66.36
67.70
CutMix
78.17
79.39
81.67
82,39
84.45
84.63
65.53
66.45
66.47
67.46
ManifoldMix
80.35
80.95
82.88
83.15
83.24
83.42
64.15
65.45
67.30
68.48
FMix
79.69
80,00
81.90
82.74
84.21
84.28
63.47
65.34
65.08
66.96
ResizeMix
80.01
80,06
81.82
82.96
84.87
84.72
63.74
64.33
65.87
68.56
MalhencyMix
79.12
79,28
81.53
82.61
84.35
84.41
64,60
66.56
66.55
67.52
PuzzleMix
81.13
81.34
82.85
82.97
85.02
85.25
65.81
66.52
67.83
68.04
AutoMix
82.04
82.32
83.64
83,94
85.18
85.38
67.33
68.18
70.72
71.56
SAMix
82.30
82.40
84.42
84.53
85.50
85.59
68.89
69.16
72.18
72.39
Small-scale classification benchmarks. Table I shows small-scale classification results, the pro- posed DM(CE) significantly improves MCE based on various mixup algorithms: it meets our expectation that DM(CE) brings more performance gains for most static mixup variants, e.g., 11.32% on CIFAR-100 and +1.21% .34% on Tiny for Mixup, except for ResizeMix. It might be because ResizeMix tries preserve the full target information of the two mixing classes. Meanwhile, DM(CE) still enhances dynamic mixup methods even though these algorithms have achieved high performance: DM(CE) brings 1.23%~0036%% on CIFAR-100 for the previous state-of- the-art PuzzleMix and brings 0.21%~0277%% on Tiny for the current best method SAMix.
ImageNet and fine-grained classification benchmarks. As shown in Table 2, 3, and 4, DM(CE) improves over MCE in all mixup algorithms on three training settings: around +0.4% for mixup methods based on ResNet variants using PyTorch-style and RSB A3 settings; around +0.5% and +0.2% for all methods based DeiT-S and Swin-T using DeiT setting. Mcanwhile, Table shows that DM(BCE) noticeably boosts the original MBCE, e.g., +1.21%%/+0.68% 7..%% for RSB//CuMiss/Mnifooda and +0.37%/+0. 12% for AutoMix/SAMix. Notice that MBCH((wo)

---

Under review as a conference paper at ICLR 2023
Table 2: Top-1 1 Ace (%)f of image classification on Table 3: Top-1 Ace (%)t of image classification ImageNet-Ik lk with ResNet variants using PyTorch-on ImageNet-Ik Ik based on ResNet-50 using RSB style training recipe.
A3 training recipe.
R-18
R-34
R-50
Methods
MCE MD(CE)
MBCE
MBCE
MD(BCE)
Methods
MCE MD(CE)
MCE
MD(CE)
MCE
MD(CE)
(one)
(two)
(one)
Vanilla
70.04
73.85
76.83
RSB
76.49
77.72
78.08
78,43
Mixup
69.98
70.20
73.97
74.26
77.12
77.41
Mixup
76.01
76.69
77.66
77.42
78.28
CutMix
68.95
69.26
73.58
73.88
77.07
77.32
CutMix
76.47
77.22
77.62
67.54
78.21
ManifoldMix
69.98
70.33
73.98
74.25
77.01
77.30
ManifoldMix
76.14
76.93
77.78
67.78
78.20
FMix
69.96
70.26
74,08
74.34
77.19
77.38
FMix
76.09
76.87
77.76
73.44
78.11
ResizeMix
69.50
69.90
73.88
74.00
77.42
77.65
ResizeMix
76.90
77.21
77.85
77.30
78.32
SaliencyMia Mix
69.16
69.57
73.56
73.92
77.42
SaliencyMix
76.85
77.25
77.93
72.74
78.24
PuzzleMix
70.12
70.32
74.26
74.51
77.54
77.71
PuzzleMix
77.27
77.60
78.02
77.19
78.15
AutoMix
70.51
70.64
74.52
74.77
77.91
78.15
AutoMix
77.45
77.82
78.33
77.46
78.62
SAMix
70.85
70.90
74.96
75.10
78.11
78.36
SAMiX
78.33
78.45
78.64
77.58
78.75
Table Top-1 Acc (%)1 of cation on ImageNet-Ik Ik with ViTs.
5: Top-1 Ace (%) of fine-grained image classification on
CUB-200 and FGVC-Aircrafts datasets with ResNet variants.
DeiT.S
Swin T
Datasets
CUB-2000 200
PGVC-Airenats
Methods
MD(CE)
MCE
MD(CE)
R-18
RX-50
R-18
RX-50
Delf
79.80
80.37
81.28
81,49
Methods
MCE
DM(CE)
MCE
DMICE)
MCE
DM(CE)
MCE
DM(CE)
Mixup
79.65
80.04
80.71
80.97
Mixup
78.39
79.90
84.58
85,04
79.52
82.66
85.18
86.68
CutMix
79.78
80.20
80.83
81.05
CurMix
78.40
78.76
85.68
85.97
78.84
81.64
84.55
85.75
FMix
79.41
79.89
80.37
80.54
ManifoldMix
79.76
79.92
86.38
86,42
80.68
82.57
86.60
86.92
ResizeMix
79.93
80.03
80,94
81.01
FMix
77.28
80.10
84.06
84.85
79.36
80.44
84.85
85.04
Saliency Mix
79.32
79.86
80.68
80.83
ResizeMix
84.77
84.92
78.10
79.54
84.51
PuzzleMix
79.84
80,25
81,03
81.16
SaxiencyMix
77.95
78.28
83.29
84.51
80.02
81.31
84.31
85.07
AntoMix
80.78
80.91
81.80
81.92
AutoMix
79.87
86.56
86.74
81.37
82.18
86.82
SAMix
80.94
81.12
81.87
81,97
SAMiX
81.11
81.27
86.83
86.95
82.15
83.68
86.80
87.22
denotes using two-hot encoding for corresponding mixing classes, which yield worse performance than MBCE, and DM(BCE) adjusts the labels for the mixing classes by Equation 3. It verifies the necessity of MD(BCE) in the case of using MBCE. Table 5 shows that DM(CE) noticeably boosts the original MCE for eight popular mixup variants, especially bringing 0.53%^~ 144% gains on Aircraft based on ResNet-18.
4.2 TRANSFER LEARNING BENCHMARKS
Following the transfer learning (TL) benchmarks (You et al., 2020), we perform TL experiments on CUB, Aircraft, and Stanford-Cars (Krause et al., 2013) (Cars). Besides the vanilla Fine-Tuning baseline, we compare current state-of-the-art TL methods, including BSS (Xinyang et al., 2019), Co-Tuning (You et al., 2020), and Self Tuning (Ximei et al., 2021). For a fair comparison, we use the same hyper-parameters and augmentations as Self-Tuning: all methods are initialized by PyTorch pre-trained models on ImageNet-I k and trained 27k steps in total by SGD optimizer with the basic learning rate of 0.001, the momentum of 0.9, and the weight decay of 0.0005. In Table 6, we adopt DM(CE) and AS for Fine- Fune-nuning, Co-Tuning, and Self- using Mixup. DM(CE) for 15%
and AS steadily improve Mixup and the baselines by large margins, e.g., +4.62%~9.11 labels, 2-.02%~556677 for 30% labels, and 23.1%% for 50% labels Cars. This outstanding improvement implies that generating mixed samples efficiently is essential for data-efficient training in data-limited scenarios. A similar performance will be presented as well in the next SSL setting.
4.3 SE LEARNING BENCHMARKS
Following (Sohn et al., 2020: Zhang al., 2021), we adopt the most commonly-used CIFAR-10/100 datasets among the famous SSL benchmarks based on WRN-28-2 and WRN-28-S We mainly evaluate
Table Top-1 Acc (%)f of transfer learning on various TL benchmarks using only 15%, 30% and 50% labels based on ResNet-50.
CUB-200
FGVC-Aireraf
Stanford-
Methods
15%
30%
50%
15%
30%
50%
1.5%
30%
50%
Fine- FineiTuning
45.25
$9.68
22
39.5720.2
57.4610.12
67.93
36.774a.12
60.63
70.00.2.21
+DM
50.04:1o.17
61.3920.21
71.87=a23
43.15
61.02.1.13
70.38
41.30
62.65
77.19
BSS
47.7410.23
63.3810.29
72.36_017
40.4110.12
59.2310.31
69.1910.15
40.57.10.12
64.
14.1310.18
76.78.1.21
Co- Tuning
52.5810.53 66.47aa.17
74.64_0.36
44.09
61.65
72.73
46.02:0.18
69.09
80.66.0.25
+DM
54.9620.65
68.25
75.72-0.37
49.27
65.60:0.41
74.89+0.17
51.78
24.15
83.02=0.25
Sell-Tuning
64.1710.47 15.13
80.22_0.6
64.1110.32
76.0310.25
81.22.1029
72.5010.45
13.5810.28
18.11
+Mixup
62.3810.32 74.65
01.46_0.27
59.3810.31
74.65.10.5
81.46
70.3110.22
83,63
88.6610.21
+DM
73.06:0..35
79.50=a.a5
82.64-024
67.57
80.71
84.82
81.69
89.22=0.21
91.26=:0.19
7

---

Under review as a conference paper at ICLR 2023
Table 7: Top-1 Acc (%) of semi-supervised learning on CIFAR-10 (using 250 and 4000 labels) and
CIFAR-I00 (using 400, 2500, and 10000 labels) based on WRN-28-2 and WRN-28-8, respectively.
Notice that DM denotes using DM(CE) and AS, Con denotes various unsupervised consistency losses, Rot denotes the rotation loss in ReMixMatch, and CPL denotes the curriculum labeling Fatx.
CIFAR-10
CIFAR-100
Methods
Losses
250
4000
400
2500
10000
Pseudo-Labeling
CE
$3.51
84.92
12.55
42.26::0.28
63.45 0.22
Mix.Match
CE+Con
86.37
23.34
32.41 #:0.66
60.24:10.48
72.22
Matc
CE+Con+Rot
93.70=0005
95.16
57.15
73.87
79.08
MixMatch+DM
CE+Con+DM
89.16-271
95.15
35.72
62.51+4037
74.70=0.28
UDA
CE+Con
94.84=20.6
90.71
53.61
72.27:40:1
77.51 =0.23
FixMatch
CE+Con
95. 14.00.
95. 05.79=000
53.58:20.82
71.97:10.16
77.80=0.12
FlexMatch
CE+Con+CPL
95.02
05.81
60.06::162
73.51
78. 10.110
FixMutch-Mixup
CE+Con+MCE
95.05
95.83
50.61 1:0.3
72.161:0.18
78.75
FixMatch+DM
CE+Con+DM
95.23
95.17=10.11
59.75
74.12+.023
79.58=0.17
the proposed DM on popular SSL methods MixMatch (Berthelot et al., 2019b) and FixMatch (Sohn al., 2020), and compare with Pesudo-Labeling (Lee et al., 2013). ReMixMatch (Berthelot et al., 2019a), UDA (Xie et al., 2019), and FlexMatch (Zhang et al., 2021). For a fair comparison, we use the same hyperparameters and training settings as the original papers and adopt the open-source codebase TorchSSL (Zhang et al., 2021) for all methods. Concretely, we use an SGD optimizer with a basic learning rate of Ir 0.03 adjusted by Cosine Scheduler, the total 220 steps, the batch size of 64 for labelled data, and the confidence threshold = 0.95. Table 7 shows that adding DM(CE) and AS significantly improves MixMatch and FixMatch: DM(CE) brings 1.81~2.8998 29% gains on CIFAR-10 and 1.27~3331 31% % gains on CIFAR-IO0 100 over MixMatch while bringing 1.8~4.117%% gains on CIFAR- 100 over FixMatch. Meanwhile, we find that directly applying mixup augmentations to FixMatch brings limited improvements, while Di achieves the best performance in most cases on CIFAR- 10/100 datasets. Appendix A.5 provides further studies with limited labelled data. Therefore, mixup with DM achieve data-efficient training in semi-supervised scenarios.
4.4 ABLATION STUDY
Since we have demonstrated the effectiveness of DM in the above four subsections and Figure 6, we verify whether DM is robust to study the effectiveness of AS in SSL tasks:
(see full hyper-parameterd in Appendix A.4) and
(1) The only hyper-parameter 7j in DM(CE) and DM(BCE) can be set according to the types of mixup methods. We grid search 1) in {0.01, 0.1, 0.5, 1,2] ImageNet- 1k. As shown in Figure left, the static (Mixup and CutMix) and the dynamic methods (PuzzleMix and AutoMix) prefer 1 = 0.1 and 1 1, respectively, which might be because the dynamic variants generate more discriminative and reliable mixed samples than the static methods.
(2) Hyper-parameters & and in DM(BCE) can also be determined by the characters of mixup policies. We grid search E {1 [1, 0.9, 0.8, 0.7] and E (2, 1, 0.5, 0.33. Figure middle and right show that cutting-based methods (CutMix and AutoMix) prefer & 0.8 and f = 1, while the interpolation- bassdd policies (Mixup and ManifoldMix) use { = 1.0 and t = 0.5.
DM(CE)
DM(BCE)
78.5
76.0
780
Top-1 acc.
73.5
77.5
77.8
MidUp
OutMbit-11
MisUp((11)
71.0
PurziaMix
Outbix
77.0
AutoMist-111 -11
77.6
ManifoldMist-111
MixUptL-0.89
AutoNint
72.5
76.5
77.4
0.5
0.1
0.01
3.5
0.8
0.7
0.3
0.5
1.0
2.0
11 value
value
value
Figure 5: Ablation of hyper-parameters, on ImageNet- based on ResNet-34. Left: analyzing the balancing weight y in DM(CE); Middle: analyzing ş DM(BCE) when is fixed to 1 and 0.5; Right: analyzing f in DM(BCE) when is fixed to and 0.8.
8

---

Under review as a conference paper at ICLR 2023
Figure 6: Top-1 Acc of mixed samples on ImageNet-IK Ik vali- dation.
Table 8: Ablation of the proposed asymmetric strategy (AS)
and DM(CE) of semi-supervised transfer learning on CUB-
200 based on ResNet-18.
Top-1 asc. mixtdd (1)
Methods
15%
30%
50%
Self-Tuning
57.82
69.12
73.59
+MCE
63.36
72.81
75.73
+MCE+AS(A 2 0.5)
59.04
69.67
74.89
Helio
+MCE+AS() S 0.5)
62.97
72.46
75.40
Maxts DD
Cutlicc
+DM(CE)AAS(A S 0.5)
66.17
74.25
77.68
6.0
3.6
10
Mixing ratio
(3) Table shows the superior of AS(A & 0.5) in comparison to MCE and AS(A 2 0.5), while using DM(CE) and AS(A - 0.5) further improves MCE.
(4) The experiments of different sizes of training data are performed to verify the data- efficientness of DM. We can observe that decoupled mixup improves by around 2% accuracy without any computational overhead. The detailed results are shown in Appendix ??
5 RELATED WORK
Mixup. As data-dependent augmentation techniques, mixup methods generate new samples by mixing two or more samples and corresponding labels with well-designed mixing policies (Zhang et al., 2017; Yun et al., 2019; Verma et al.. 2019; Kim et al., 2020; 2021; Liu et al., 2022; Li et al., 2021b). The pioncering mixing method is Mixup (Zhang et al., 2017), whose mixed samples are generated by linear interpolation between pairs of samples. After that, cut-based methods are proposed to improve the mixup for localizing important features, especially the vision field. In other works, authors explore using nonlinear optimizable interpolation mixup policies, such as PuzzleMix (Kim et al., 2020), Co-Mixup (Kim et al., 2021), AutoMix (Liu et al., 2022), and SAMix (Li et al., 2021b). Moreover, mixup methods extend to more than two elements (Kim et al., 2021; Dabouei et al., 2021), 1), and are utilized in contrastive learning to learn discriminative visual representation (Kalantidis et al., 2020; Lee al., 2021; Shen al., 2021; Li al., 2021b).
Semi-supervised Learning and Transfer Learning. MixMatch (Berthelot et al., 2019b) and ReMixMatch (Berthelot et al., 2019a) apply mixup on labelled and unlabeled data enhance the diversity of the dataset. More accurate pseudo-labelling relies on data augmentation techniques to introduce consistency regularization, e.g., UDA (Xie al., 2019) and FixMatch (Sohn al., 2020) employ weak and strong augmentations to improve the consistency. Furthermore, CoMatch (Li al., 2021a) unifies consistency regularization, entropy minimization, and graph-based contrastive learning to mitigate confirmation bias. Fatx (Zhang et al., 2021) improves FixMatch by applying curriculum learning for dynamically updating confidence threshold class-wisely wasss-iseely... Fine- tuning a pre-trained model on labelled datasets is widely adopted form of transfer learning (TL) in various applications. Previously, (Donahue et al., 2014; Oquab et al., 2014) show that transferring pre-trained AlexNet features to downstream tasks outperforms hand-crafted features. More recently, Self- Tuning (Ximei et al., 2021) introduced contrastive learning into TL to tackle confirmation bias and model shift issues in one-stage framework.
CONCLUSION AND LIMITATIONS
In this paper, we introduce Decoupled Mixup (DM), a new objective function for considering both smoothing the decision boundaries and mining discriminative features. The proposed DM helps early static mixup methods (e.g., MixUp and CutMix) achieve a comparable or better performance than the computationally expensive dynamic mixup policies. Most impotently, DM raises question worthy of researching: is it necessary to design very complex mixup policies to achieve expected results? ? Based on our analysis and experiments, we argue that the loss function of mixup training is a new question worthy of studying. However, the introduction of additional hyperparameters may take users some extra time to check whether the default hyperparameters This also leads to the core question of the next step in the development of this work: how to design a more elegant and adaptive mixup training objective by leveraging label-mismatched samples?
reliable datasets other than images.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E OConnor, and Kevin McGuinness. Pseudo- and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv: 19117978, 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint urXiv:I905.m 02249, 2019b.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint 11XXv::707.088/1 2017.
Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, and Nasser Nasrabadi. Supermix: Super- vising the mixing data augmentation. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 13794-13803, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the International Conference Machine Learning (ICMI.), 2014.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regular- ization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 3714-3722. 2019.
Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, and Adam Prügel- Bennett Jonathon Hare. Fmix: Enhancing mixed sample data augmentation. arXiv preprint vrXiv:2002.12047, 2(3)44 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the International Conference Computer Vision (ICCV), 2017.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning, In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning (ICML), pp. 5275-5285. PMLR, 2020.
Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song, Co-mixup: Saliency guided joint mixup with supermodular diversity. International Conference on Learning Representations (ICLR), 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13),
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems (NeurIPS), pp. 1097-1105, 2012.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supereised learning method for deep neural networks. In Workshop challenges in representation learning, ICML, pp. 896, 2013.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. I-mix: A domain-agnostic strategy for contrastive representation learning. In International Conference on Learning Representations (ICIR), 2021.
Junnan Li, Caiming Xiong, and Steven Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the International Conference on Computer Vision (ICCV), 2021a.
Siyuan Li, Zicheng Liu, Di Wu, Zihan Liu, and Stan Z. Li. Boosting discriminative visual representa- tion learning with scenario-agnostic mixup. arXiv preprint arXiv:2111. 2021b.
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, and Stan Z. Li. Openmixup: Open mixup toolbox and benchmark for visual representation learning. https: httpp:///ithhn www//est AI/ sup, 2022.
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision (ICCV), 2021.
Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Liu Zihan, and Stan Z Li. Automix: Unveiling the power of mixup for stronger classifier. Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2022.
lya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint a60iv 1003::6608.03383, 2016.
lyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 130.v51306.515/ 2013.
Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivie. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
Jie Qin, Jiemin Fang, Qian Zhang, Wenyu Liu, Xingang Wang, and Xinggang Wang. Resizemix: Mixing data with preserved object information and true labels. arXiv preprint arXiv:2012.11101, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision (IJCV), pp. 211-252, 2015.
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing, Un-mix: Rethinking image mixtures for unsupervised visual representation learning. Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2021.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi- superviie learning with consistency and confidence. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
11

---

Under review as a conference paper at ICLR 2023
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research (JMLR), ((11):1929-1958 2014.
Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. arXiv preprint vrXiv:1955.1101., 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML), pp. 10347-10357, 2021.
AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, et al. Salien- cymix: A saliency guided data augmentation strategy for better regularization. arXiv preprint arXiv:2006.01791, 2020.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning (ICML), pp. 6438 6447. PMLR, 2019.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongic. The caltech-ucsd birds-200-2011 dataset. 2011.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning (ICML.) pp. 1058- 1066. PMLR, 2013.
Ross Wightman, Hugo Touvron, and Herv Jgou. Resnet strikes back: An improved training procedure in timm, 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. arXiv preprint 19X4::1904.12848. 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 1492-1500, 2017.
Wang Ximei, Gao Jinghan, Long Mingsheng, and Wang Jianmin. Self-tuning for data-efficient deep learning. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Chen Xinyang, Wang Sinan, Fu Bo, Long Mingsheng, and Wang Jianmin. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Hao Yu. Huanyu Wang, and Jianxin Wu. Mixup without hesitation. In International Conference on Image and Gruphics (ICIG), pp. 143-154. Springer, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5023-6032 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting sunissuper learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 170..01710 2017.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
In this Appendix, we provide proofs of the proposition (BA.1) and proposition 2 (5A.2) and implementation details (BA.3).
A.1 PROOF PROPOSITION
Proposition 1. Assuming ((),b) is generated from two different classes, minimizing LMCE is equivalent to regress corresponding À in the gradient of LMCE:
expl+in,))
+ En exp(+(a.a)
1=
exp(=
- X) Ee exp(
1=
(4)
exp(=ia.8)
+ i,j
expl
Proof. For the mixed sample (I(a.)), M(a,))), ((a is derived from a feature extractor fo (i.e 2(a,b) fo(M(a)))) According to the definition of the mixup cross-entropy loss LMCEE we have:
=
OLNCE
(MCE))
dzt (a,b))
(vTa,s) log (o(z(a))))
dz((am)
δ (log( exp(2(a,))
(H(ai)) 0z(a,i) Ej exp(<(as)) 1)
((ab)) Ej exp(((a))
2 explo(a.))
exp(z(av))
defa,b) Egm exp(=(a.n)
expl<tam)
(oa Eg exp(=(a)))
exp (p((aa)))
Blass
ク exp(=(a.)))
13

---

Under review as a conference paper at ICLR 2023
A.2 PROOF OF PROPOSITION
Proposition 2. With the decoupled Softmax defined above, decoupled mixup cross-entropy LDMICE) boost the prediction confidence of the interested classes mutually and escape from the X- constraint:
Pla.)) + log (1 PlaN)) P(a.))
LDM(CE)
yo
log
1-1 P(aa0)
==1==1
Proof. For the mixed sample (ITa,)) Ma.,)); F(a.8) is derived from a feature extractor So (i.e ((o.))=/o(e..,))> According to the definition of the mixup cross-entropy loss LDM(CE), we have:
LDMICE) V(a log (H(Zaa)))))aa - va log (H(Z)),))) yb vo log (H (H(Z)),))) yo
C
exp(=(a.)))
C
exp(=(a.1) IRet exp exp(((a0))
Va log(
1o6 +
Σ
Vd log(
)m(
kØt exp(2 (a.b)
4==1
C
exp(=
exp(=(a.0))
Σ
v.yl log(
+ log
Enety (a.b)
Exets exp(z(a,b) (a,
4-=1
exp(s(n.0))
exp(=(a.es)
=
Σ
C
Vayl log(
Σ exp(=?
Ei explx(a.br)
+ log
Σ Rxz exp(
Nu expls(a.b))
w-=1
2 exp(=(a.6))
k- exp(=/u.1)
C
P(a.))) ) + log(
P(a,b)
Σ
Viri log(
P(a,)) ))
=
where P(a.b) (2(a,)))
14

---

Under review as a conference paper at ICLR 2023
A.3 IMPLEMENTATION DETAILS
Dataset. We briefly introduce used image datasets. (1) Small scale classification benchmarks:
CIFAR-10/100 (Krizhevsky et al., 2009) contains 50,000 training images and 10,000 test images in 32x32 resolutions, with 10 and 100 classes settings. Tiny- ImageNet (Chrabaszcz et al., 2017) rescaled version ImageNet-Ik, which has 10,000 training images and 10,000 validation images of 200 classes in 64x66 resolutions. (2) Large scale classification benchmarks: ImageNet- Ik (Krizhevsky et al., 2012) contrains 1, 1,281,167 167 training images and 50,000 validation images of 1000 classes. (3) Small-scale fine-grained classification scenarios: CUB-20 CUB-200-200 201--200 (Wah al.,
2011) contains 11,788 images from 200 wild bird species for fine-grained classification. FGVC- Aircraft (Maji et al., 2013) contains 10,000 images of 100 classes of aircraft. Standford-Cars (Krause et al., 2013).
Training settings on ImageNet-Ik. Tab. Al shows the full training settings of PyTorch, DeiT, and RSB A3 on ImageNet- Ik. Following (Yun et al., 2019), we replace the step learning rate decay with Cosine Scheduler (Loshchilov & Hutter, 2016) for better performances.
Hyper-parameter settings.
follow basic hyper-parameter settings (e.g., ()) for mixup variants in OpenMixup (Li al., 2022), where reproduce most comparison methods. Notice that static methods denote Mixup (Zhang et al., 2017), CutMix (Yun et al., 2019), ManifoldMix (Verma al., 2019), SaliencyMix (Uddin et al., 2020), FMix (Harris et al., 2020), ResizeMix (Qin et al., 2020), and dynamic methods denote PuzzleMix (Kim et al., 2020), AutoMix (Liu et al., 2022), and SAMix (Li al., 2021b)). Similarly, interpolation-bassel methods denote Mixup and ManifoldMix while cutting-based methods denote the rest mixup variants mentioned above. We set the hyper- parameters of DM(CE) as follows: For CIFAR-100 100 and ImageNet-Ik static methods use 1j 0.1, and dynamic methods use 77 1. For Lmay-ImaeeNet and tne-grained datasets, static methods use r based on ResNet- while 7) = 0.1 based on ResNeXt-50;; dynamic methods use 17 = As for the apeerpprraeetrss of DM(BCE) on ImageNet- 1k, cutting-based based methods use / = 1 and { = 0.8, while interpolation- based methods use 0.5 and { Note that we use a 0.2 and a 2 for the static and dynamic methods when using the proposed DM.
Table Al: Ingredients and hyper-parametens used for ImageNet- Ik training settings.
Procedure
PyTorch
DeiT
RSB A3
(Yun et al., 2019) (Touvron et al., 2021)
(Wightman et al., 2021)
Train Res
224"
224"
224"
Test Res
2242
2242
2242
Test crop ratio
0.875
0.875
0.95
Epochs
100/300
300
100
Batch size
256
1024
2048
Optimizer
SGD
Adam
LAMB
LR
0.1
1 10-3
8 10-3
LR decay
cosine
cosine
cosine
Weight decay
10
0.05
0.02
optimizer momentum
0.9
B1. B2 = 0.9, 0.999
X
Warmup epochs
X
5
5
Tahel smoothing
X
0.1
Dropout
X
X
Stoch. Depth
X
0.1
0.05
Repcated Aug
X
/
Gradient Clip.
X
1.0
H. flip
RRC
/
Rand Augment
X
9/0.5
6/0.5
Auto Augment
X
X
X
Mixup alpha
X
0.8
0.1
Cutmix alpha
1.0
1.0
Frasing prob.
X
0.25
ColorJitter
X
X
X
HMA
X
0.99996
CE loss
BCE loss
X
X
15

---

Under review as a conference paper at ICLR 2023
A.4
SENSITIVITY ANALYSIS
CIFAR-100
Tiny-ImageNet
CUB-200
01.9
60
80.5
67
80.0
Top-1 scc.
80
66
79.0
Mixup
OutMix
65
CuiMi
MidUp
78.5
GutMix
Mixip
Puzzlattii
Puzzlekii
Puzzlattix
78
64
77.5
0.5
0.2
0.01
0.5
0.01
0.5
0.1
0.01
value
1 value
value
Figure A1: Sensitivity analysis of hyper-parametee on different datasets based on ResNet- 18.
In order to verify the robustness of hyper-parameter 7), extra experiments are conducted various datasets: CIFAR-100, Tiny-ImageNet, and CUB-200. Figure Al shows the results that are consistent with our ablation study in Section 4.4. The dynamic mixup methods prefer the large value of 1) (1.0), while the static ones are more like a small value (0.1). The main reason for this is that dynamic methods generate mixed samples where label mismatch is relatively rare, relying larger weights to achieve better results, while the opposite true static methods.
A.5 ANALYSIS OF DATA-EFFICIENT MIXUP
To further DM whether data-efficient mixup training can be truly achieved, we conducted supervised experiments CIFAR-100 with different sizes of training data. 15%, 30% and 50% of the CIFAR- 100 data are randomly selected as training data, and the test data are unchanged. decoupled mixup uses DM(CE) as the loss function by default. From Table A2, performance consistently without any computational overhead. Especially when using only 15% of the data, DM can improve accuracy by 2%. Therefore, combined with the experimental results of semi-supervised learning in Sec. 4.3 and Sec. 4.2, we can say that mixup training with DM is more data-efficient with limited data.
that DM improves the
Table A2: Top-1 Acc (%)f of image classification on CIFAR- 100 dataset with ResNet- 18 using only 15%, 30%, and 50% labeled training sets.
15%
30%
50%
Methods
MCE
DM(CE)
MCE DM(CE)
MCE
DM(CE)
Vanilla
42.48
56.41
64.32
Mixup
42.23
44.39
55.61
56.78
64.55
65.92
CutMix
43,81
44.85
55,99
57.14
64.38
65.87
SaliencyMix
42.95
44.01
55.42
56.51
64.56
66.10
PuzzleMix
42.67
43.87
56.19
57.36
64.74
66.26
16


Reviews:
[{"review_id": "bnaJeznOYO", "reviewer": "Reviewer_4Ujo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper revisits the commonly used mixup technique and proposes to modify the loss function to achieve similar effect as the time-consuming search based methods. The proposed method is simple and proved effective across various tasks, including classification, semi-supervised learning on both small and large scale datasets. ", "summary_of_the_review": "Overall this paper is good: it extends an existing widely-used technique with simple modification that produces good results across various tasks. There are some issues in the experiments, which I do not think undermine the value of this work. \n\n\nPost rebuttal comments: After reading the comments from other reviewers, I tend to reject this submission. While this submission indeed proposes a novel technique that generalizes well across various tasks, the technical contribution is slightly below the bar this prestigious conference. So, I would like to join other reviewers and vote for a rejection. ", "strength_and_weaknesses": "### Strength\n1. The proposed method is well motivated, simple, technically sound,  of generalizability. \n2. Extensively experiments are conducted to verify the proposed method across different tasks, datasets, and other alternative choices. \n3. The paper is well presented, with professional figures, table and language use. \n\n### Weaknesses\n1. Setting the parameter looks very important to the performance. Sensitivity analysis has been conducted in the one dataset, which shows the parameter is important to the performance. It is unclear how the parameters are set in other tasks. Considering that in many cases the. proposed method is slightly better than the naive mixup method, the value of this method would significantly reduce if it is non-trivial to tune the parameter to get results.\n\n2. The weakness of this work has not been discussed. As a plug-and-play tool like this, it would be highly valuable if the authors can reveal the weakness so that people can decide whether to adopt it for their own projects.   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KcLO-wmfOFP", "reviewer": "Reviewer_5tSp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a variant of the Mixup method that considers the label mismatch problem by decoupling (i.e., removing) the competitor's class in the softmax which can potentially cause mismatching between the mixed label and sample. The authors formulate the decoupled mixup cross-entropy loss ($L_{DM(CE)}$) that becomes an element of the proposed decoupled mixup (DM) loss  ($L_{DM}$) with the standard mixed cross-entropy loss ($L_{MCE}$). The authors claim that using the DM loss can address the label mismatch problem and argue a similar effect as dynamic mixup methods. The authors apply the DM loss on Image classification tasks on the datasets of CIFAR-100, Tiny-ImageNet, and ImageNet-1k with diverse network architectures and baseline Mixup methods. The authors apply the proposed loss not only to the static Mixup methods but to the so-called dynamic Mixup methods such as SaliencyMix, PuzzleMix, AutoMix, and SAMix. Moving further from the conventional image classification experiments, the authors further evaluate fine-tuning classification on FGVC datasets and semi-supervised learning methods. All the experimental results consistently back the effectiveness of the proposed loss.", "summary_of_the_review": "This work provides a novel Mixup-based loss for consistent performance improvements on several datasets with many models. I like the constant improvements over diverse setups and scenarios. However, what I'm most concerned about is that there is actually novelty in the sense of a new loss of performance gains, but there doesn't seem to be enough evidence to support the author's claim. I recommend the authors revise the paper by providing more materials that show the link between the decoupling by the proposed DM loss and the label mismatch problem; consider more on the gap between embellishing the proposed method as a dynamic aspect and the ones categorized into dynamic methods.", "strength_and_weaknesses": "Pros)\n+ The idea looks simple yet practical for the practitioners using Mixup in training.\n+ This paper provides extensive empirical studies to support the effectiveness of the proposed method.\n+ All the experimental results consistently improve when adopting the proposed loss upon the baseline Mixup methods.\n\nCons)\n- There is no clear evidence that decoupling the class can reduce the label mismatch. It hardly agrees that only performance improvements cannot support the claim. Instead, the authors should provide any empirical or theoretical backups of whether this is actually addressed through the proposed DM loss.\n- Some notations are not clearly presented:\n  - $p(a,b)$ seems to be the probability of the mixed samples; $p_a$ and $p_b$ must be scalars. I could understand it only after reading the manuscript carefully.\n  - l, i, and j in eq.(2)\n  - M in Section 2.2\n  - $y^i_a$, $y^j_b$ in proposition 2\n\nComments)\n- The authors claim that the proposed method can adaptively focus on discriminative features as the dynamic Mixup methods, but they seem to be conceptually different. If not, there should exist any backups for this. Presumably, the performance improvements upon the dynamic Mixup methods in Table 1 seem to support my concern.\n- How does \"the discriminatory of instances\" in p.2 be achieved through the proposed method?\n- Please explain in more detail why the regressing lambda is the problem.\n- Vanishing $\\lambda$ in eq.(2) release the gradient magnitude not to become having a dependency on $\\lambda$, but still unclear that 1) it affects the prediction; 2) it is actually connected with the label mismatching. Please clarify this.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DrOdoTVh1x9", "reviewer": "Reviewer_baqd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In  this paper,  to solve the expensive computing resources problem of current dynamic mixup method, it proposed a method trying to transfer the decoupling mechanism of dynamic methods from the data level to the objective function level and propose the general decoupled mixup loss. The experimental results on supervised and semi-supervised tasks proved the effectiveness of DM.\n", "summary_of_the_review": "In summary, the paper proposed a novel loss function and verified the effectiveness on different dataset and different tasks.  There are some minor clarity issues on the formulations. However, the paper doesn’t provide proof on how “data-efficient” of the proposed method is and there are no Quantitative results to show the computation efficiency advantage.", "strength_and_weaknesses": "The paper proposed a new loss function, and extends to the semi-supervised case and binary cross entropy form. the experimental results show that it’s better than previous benchmarks. \nSome Questions:\n1. Some annotations are misleading.  like in formula (2),  where is l defined. For the asymmetrical strategy DM loss, the log(\\phi) is a matrix or a vector?\n2. What’s the meaning of “data-efficient”? Can you show the model performance under different sizes of dataset to prove the data-efficiency advantage over the baselines?\n3. As the paper argued, one of the advantages is that it’s efficient computing, can you show experiments on how compute-efficient compared with the baselines?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7NnCfs7osv1", "reviewer": "Reviewer_A9VU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new objective function for mixed sample data augmentation (MSDA). Technically, the proposed Decoupled Mixup (DM) loss is equivalent to the summation of the original mixup loss and the additional losses that only consider one mixed target label while the other mixed target labels are ignored. For example, assuming a mixed label [0.7, 0.3, 0, 0] and a prediction probability [0.5, 0.45, 0.05, 0], then the DM loss is equivalent to `MixedCrossEntropy([0.7, 0.3, 0, 0], [0.5, 0.45, 0.05, 0])` (the original mixup loss) + `CrossEntropy([1, 0, 0], [0.91, 0.09, 0])` + `CrossEntropy([1, 0, 0], [0.9, 0.1, 0])` (called as \"decoupled\" mixup loss). The motivation of the proposed method is that the objective function of MSDA is highly correlated to the mixing ratio $\\lambda$: by ignoring the other mixed targets, the DM loss is \"decoupled\" with the mixing ratio $\\lambda$. The extensive study on image classification tasks (supervised and semi-supervised) shows that the proposed method is effective.", "summary_of_the_review": "This paper shows great empirical improvements on various benchmarks. The extensive studies are impressive and show consistent improvements. As my comments in \"Strength\", this paper will be helpful for many practitioners working on ImageNet-1K.\n\nHowever, I think this paper has a large room for improvement in terms of academic publication. Especially, I think this paper needs a connection between motivation and the proposed method. I cannot find any reason why the proposed method will work. In terms of the label mismatch problem, I think this method will still suffer from the label mismatch problem because the label mismatch problem can be solved only by making the mixed sample and the mixed label dependent. Furthermore, I think this paper should compare using $L_{DM(CE)}$ alone as well. If using $L_{DM(CE)}$ alone does not work well, then it means that we need a better explanation (and motivation) of the effect of the combination of $L_{MCE}$ and $L_{DM(CE)}$, not only for $L_{DM(CE)}$.\n\nOverall, I think this paper does not reach the acceptance threshold of the ICLR conference.\n\n----\n\nDuring the reviewer-author discussion period, the authors provide additional experiments to resolve my concerns. However, as my last comment, I am still skeptical about revising my score because my concerns still remain. I will maintain my initial recommendation.", "strength_and_weaknesses": "## Strength\n\n- Because the proposed method is invariant to the choice of the mixing strategy, the proposed method is easy to be adopted to any MSDA method.\n- The strongest strength of this paper is the expensive experimental results and consistent improvements.\n    - The extensive study shows that the proposed method is effective to the existing MSDA methods. Especially, this paper studies the effect of the proposed objective function on various benchmarks with state-of-the-art optimization techniques. For example, this paper explores not only ImageNet training with 300 epochs and cosine lr scheduling (a standard one) but also ResNet Strike Back (RSB)-ish advanced optimization settings or DeiT optimization settings.\n    - In my opinion, the ImageNet-1k experimental results (Table 2, 3, 4) will be helpful to researchers who work with ImageNet-1k benchmarks; as far as the reviewer knows, there are not many works that compare the existing state-of-the-art MSDA methods (Mixup, CutMix, ManifoldMixup, FMix, ResizeMix, SaliencyMix, PuzzleMix, AutoMix, SAMix) with the most advanced optimization settings, such as RSB or DeiT settings. E.g., considering the heavy computational resources of \"dynamic MSDA\" methods (SaliencyMix, PuzzleMix, AutoMix, SAMix), practitioners can choose ResizeMix + DM loss.\n    - This paper also explores transfer learning and semi-supervised learning scenario in state-of-the-art settings.\n\n\n## Weakness\n\n### Weak connection between label mismatching problem and the proposed method\n\nI cannot find any connection between the \"label mismatching problem\" and the \"decoupled Mixup loss\". Assuming we use a static Mixup method (e.g., Mixup), even if we use the DM loss, we still suffer from the label mismatching problem because the loss function still contains the original loss function. Recall that the label mismatching problem means that there exists a mismatch between a mixed sample and a mixed label; it means that if we want to solve this problem, we have to make a mixed label depending on a mixed sample. However, the mixed samples and the mixed labels by the proposed method are invariant; it will still suffer from the label mismatching problem if the base method suffers from the problem.\n\nAlthough this paper has a meaningful empirical contribution, in terms of academic publication, I think this paper should need more verifications for the statements and the motivation. I cannot find any connection between the proposed method and the motivation.\n\nAlso, as a minor comment, I cannot find any connection between \"data-efficient\" (the title) and the proposed method. I think there need to be more statements for why this method is data-efficient compared to the original mixup loss.\n\n### Weak motivation why we should \"decouple\" lambda and the objective function\n\nI presume that the motivation of this paper is removing (or reducing) the dependency of $\\lambda$ and the objective function. If the main motivation of this paper is \"addressing label mismatch\", then there should be more explanations of how removing the dependency of $\\lambda$ and the objective function (\"decoupling\") and label mismatch are related. More specifically, it is not clear to me how \"decouple\" fixes the label mismatch problem.\n\nMoreover, the proposed method still uses the original loss function. What happens if we only use $L_{DM(CE)}$ alone without $L_{MCE}$? If using $L_{DM(CE)}$ alone is worse than using $L_{MCE}$ alone, then I think it is hard to say that the proposed $L_{DM(CE)}$ itself is effective.\n\nFurthermore, to argue that the \"decoupling\" (ignoring the other mixing targets) is effective, then I think this paper should include the following baseline:\n\n- `CrossEntropy(target_1, prediction)` + `CrossEntropy(target_2, prediction)` => removing $\\lambda$ in the objective function\n\nIt is the same as \"Complete-label CutMix\" in Table 8 of CutMix paper, and it is known to be worse than the $\\lambda$ mixing strategy (following Table 8 in the CutMix paper). I think this result can make the submission stronger.\n\n### Minor\n\n- It seems that the title \"data-efficient\" and the motivation are not well-matched. I suggest using a different term rather than data-efficient in the title and Section 3.\n- $l$ of Equation (2) is not defined. Please fix the equation.\n- Suggestions for a better formatting\n    - Please specify the full terminology of AS (page 5). I presume AS stands for \"Asymmetrical strategy\".\n    - I would like to recommend inserting \"improvements\" in the tables (e.g., as Table 1 of CutMix paper).\n    - Please do not use `\\vspace{-XX}` between sections (e.g., page 5 \"3.2 Asymmetrical strategy\")\n    - Please do use `\\vspace{XX}` between a caption and a table (I usually recommend adding a space `~` between a caption and a table)\n    - Please make more spaces (e.g., `\\hfill`) between Table 2 and 3 (as well as Table 4 and 5).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject