Bypassing the Random Input Mixing in Mixup
Authors: Hongyu Guo
Venue:  (2021)
Keywords: Deep Learning
Data Augmentation
Mixup

Abstract: Mixup and its variants have promoted a surge of interest due to their capability of boosting the accuracy of deep  models. For a random sample pair, such approaches generate a set of synthetic samples through interpolating both the  inputs and their corresponding one-hot labels. Current methods either interpolate random features from an input pair or learn to mix salient features  from the  pair. Nevertheless, the former methods can create misleading synthetic    samples or remove important features from the given inputs, and the latter strategies incur significant computation cost for selecting descriptive input regions. In this paper, we show that the effort needed for the input mixing can be bypassed. For a given sample pair, averaging the features from the two inputs and then assigning it with a set of  soft labels can  effectively regularize the training. We empirically show that the proposed  approach performs on par with state-of-the-art strategies in terms of predictive accuracy. 

Under review as a conference paper at ICLR 2021
BYPASSING THE RANDOM INPUT MIXING IN MIXUP
Anonymous authors
Paper under double-blind review
ABSTRACT
Mixup and its variants have promoted a surge of interest due to their capability boosting the accuracy of deep models. For random sample pair, such approaches generate set of synthetic samples through interpolating both the inputs and their corresponding one-hot labels. Current methods either interpolate random features from an input pair or learn to mix salient features from the pair. Nevertheless, the former methods features from the given inputs, and the latter strategies incur significant computation cost for selecting descriptive input regions. In this paper, we show that the effort needed for the input mixing can be bypassed. For a given sample pair, averaging the features from the two imputs and then assigning it with set of soft labels can effectively regularize the training. We empirically show that the proposed approach performs on par with state-of-the-- strategies in terms predictive accuracy.
create misleading synthetic samples remove important
INTRODUCTION
Deep neural networks have demonstrated their profound successes in many challenging real-world applications, including image classification (Krizhevsky et al. 2012), speech recognition (Graves al. 2013), and machine translation (Bahdanau al. [2015] Sutskever al. (014). One key factor attributing to such successes is the deployment of effective model regularization techniques, which empower the learning avoid overfitting the training data and generalize well to unseen samples. This is because current deep models typically embrace high modeling freedom with very large number of parameters. To this end, many regularizers for deep models have been introduced. including weight decay (Hanson & Pratt 1988), dropout (Srivastava et al. (2014), stochastic depth (Huang et al.
2016 batch normalization (loffe Szegedy 2015), and data augmentation schemes (Cubuk et
2019
Hendrycks et al. 2020 Inoue 2018 Lecun et al. 1998 Simard et al. 1998).
Among those effective reg- ularizers, Mixup (Zhang) jet al.] (2018)
sim-
ple and yet effective, data- augmentation based regu- larizer for enhancing the deep classification models. Through linearly interpo- lating random input pairs and their training targets in one-hot representation, Mixup generates a set of synthetic examples with soft labels regularize the training. Such pairwise, label-variant data augmen- tation techniques (Guo)
neural network
dissification
imput2
mixed Input
global soft tabel
triining saft tanget for the clasiffur
local soft label
Illustration of the proposed method. The mixed input is the average of the two inputs; the training target for the averaged input is the combination of the local soft label (average of the two one-hot targets) and the global soft label (dynamically generated during training).
Figure 1:
2020; Guo et al.] 2019
Kim al. 2020; Li et al.
2020al Tokozume et al. 2018a b. Verma et al. 2019 Yun et al. 2019: Zhang et al., 2018) have
attracted surge of interest and shown their effectiveness boosting the accuracy of deep networks.

---

Under review as a conference paper at ICLR 2021
Nevertheless, unlike label- preee-ipesg data augmentation such rotation, flip, and crop, there is still limited knowledge about how to design better mixing policies for sample pairs for effective label-variant regularization. Current Mixup-based approaches either mix pair of inputs using random mixing coefficients (Guo) 2020; Guo et al.
2019 Summerrs & Dinneen 2019 Tokozume
et al. [208bb Verma et al. 2019 Yun et al. [209 Zhang al. (018)) or learn mix salient features from the given pair (Dabouci et al. 2020 Kim et al. 2020 Li et al., 2020b; Walawalkar et al., 2020)
create set of synthetic samples. Nonetheless, the former methods may create misleading synthetic
samples (Guo et al., (019) or remove important features from the given inputs (Kim et al., 2020) and the latter strategies can incur significant computation cost for identifying and selecting the most descriptive input regions (Kim et al. 2020 Walawalkar et al. 2020).
In this paper, we show that the effort needed for the input mixing with a range of mixing ratios can be bypassed. For a given input pair, one can average the features from the two inputs, and then assign it with set of soft labels. These soft labels are adaptively learned during training to incorporate class information beyond the provided label pair as well as the evolving states of the training. The method is illustrated in Figure | where the mixed input is the pixel-wise average of features from the two inputs and its training target is the combination of the local soft label, which is the average of the two one-hot targets, and the global soft label, which is generated by the networks during training.
We empirically show that the proposed approach performs par with state-of-thee the-art methods with random input mixing policy or learning to mixing strategy, in terms of predictive accuracy. We also demonstrate that the synthetic samples created by method keep tuning the networks long after the training error on the original training set is minimal, encouraging the learning to generate, for each class of the training samples, tight representation clusters. Also, the Class Activation Mapping (CAM) (Zhou et al. (016) visualization suggests that our method tends to focus on narrower regions of an image for classification.
BYPASSING THE INPUT MIXING IN MIXUP
2.1 MIXUP- BASED DATA AUGMENTATION
For a standard classification setting with training data set (X; the objective the task is to develop a classifier which assigns every imput x E X label y E Y. Instead of using the provided training set (X; Y), Mixup Zhang et al. 2018) generates synthetic samples with soft labels for training. For pair of random training samples (z'; y²) and (z';8)), where is the input and y the one-hot encoding of the corresponding class, Mixup creates a synthetic sample as follows.
xx = Az + \)æ
ز Vx A + (1-)))²
(2)
where À is scalar mixing policy for mixing both the inputs and the modeling targets of the sample pair. X is sampled from a Beta(a, a) distribution with a hyper-paramettr a. The generated samples (xx. oi are then fed into the model for training minimize the cross-entropy loss function. VX
Current variants of Mixup focus on the introduction of a representation function & for input mixing:
F
[ A) <0> x² X).
The state-of-the the random input mixing variant CutMix (Yun et al. 2019) defines the & function to form a binary rectangular mask applying to a randomly chosen rectangle covering À proportion of the input image. PuzzleMix (Kim et al., 2020) is the state-of-the-are learning to mixing variant. This method defines the zi function to compute the saliency map of the input pair, find the optimal mask, and optimize the transport plans for generating the mixed example. This is ensure that the mixed image contain sufficient target class information corresponding to the mixing ratio À while preserving the local statisties of each input.
2.2 FORMING THE TRAINING INPUT
Similar to Mixup and its variants, our method also creates mixed samples from random sample pair, but for a given sample pair our method does not generate a set of inputs with different features. Instead, for range of mixing ratios As, our approach uses the same mixed input, invariant to the
2

---

Under review as a conference paper at ICLR 2021
mixing ratios provided. We term our method Label driven Mixup (denoted LaMix). To this end, the mixed input H in LaMix has two forms. For the random imput mixing methods such as Mixup and CutMix, it is formed by pixel-wise average of the raw features of the input pair (æ*, xx)):
/ 0.5x* + 0.5z²
(4)
For the learning to mixing methods such as PuzzleMix, the mixed input is formed by pixel-wise average of the saliency features computed with À value as 0.5:
3 (> |->
0.5) + "
(5)
The newly resulting input ~V is then used for training, by passing through neural network model fo (parameterized with (p) to generate the mn-dimensional input embedding S RT:
S (T")))
(6)
Next, the resulting 5 then fed into a linear fullyconnected layer Wi E Rexm mm to produce the predicted classification distribution over the classification target classes:
Softmax(W,5*)
(7)
The error of the training is computed by comparing the prediction 7 and the training target of the input 25, which is discussed next.
2.3 FORMING THE TRAINING TARGET
For the same mixed input 3M created from a random input pair (x', w.f,,"), LaMix associates it with various soft target label ÿ which is learned by the networks with two steps: 1) obtaining global soft label and 2) integrating with local soft label. detail, the dynamically assigned soft target label j is a function of (x', x² y. y², A), as follows.
xX = T(x' x2 X, 10, ,3)).
(8)
As illustrated in Figure LaMix implements the r(-) function through adding an additional fullycon- nected layer W, R xm to the original networks. That is, for the given input 3 with A, LaMix first computes its probabilities over the classification targets, denoted px:
pX Softmax(((W,55*))
where a denotes the Sigmoid function, and Süj is the same input embedding as that Equation 7 In other words, the two predictions (i.e., Equations 7 and ( share the same networks except for the last layer. Because the predictions p are about all the target classes of the given data, such predictions thus reflect the class information beyond the provided local label pair, namely (v' x²) ) For description purpose, we term PK global soft label. The Sigmoid function here will provide the probability of associating the input to a particular label.
After having the global soft label px we then integrate it with the local label of the given sample pair, i.e., o described in Equation as follows:
BDX + (1-
(10)
where B is a scalar coefficient between [0,1]. Doing so, we enable the synthetic sample (F ) to have information from both the local labels of the provided sample pair (i.e., 7 and the class information beyond the labels of the given pair (i.e., )) This process also illustrated in Figure
2.4 PARAMETER OPTIMIZATION
For training, LaMix minimize, with gradient descent, the cross entropy loss E between the assigned training soft target eX and the network's 's predicted classification distribution y
E
リ logg'
(11)
During training, we also need to prevent LaMix from assigning target soft labels too far away from the gold labels in the original training set. To attain this goal, we alternatively feed samples the networks with either mini-batch with original inputs, i.e., " or mini-batch from the mixed inputs, i.e., T Note that, when training with the former, the networks still need learn to assign the training target ū to the sample T44
3

---

Under review as a conference paper at ICLR 2021
MNIST
Fashion
SVHN
Cifar10
Cifarl00
Vanilla
0.62110.05
4.7810.19
3.64.00.4
5.19-0.30
24.1911.27
Mixup
0.56++0.01
4.18+0.02
3.37+0049
3.88=0332
21.10+0.21
PreAct ResNet-18
CutMix
0.59+00006
4.06+0.06
2.82+0022
4.09+
19.77+0.26
LaMix
0.50
4.03++0.09
2.69+0.08
3.82-0.18
09.1
Vanilla
0.61
4.55+0.11
3.22+-0.05
4.83-0.30
23.10-00.62
Mixup
0.57+0.03
4.31+0.05
2.85++0.07
0.29
19.48+0048
ResNet-50
CutMix
0.49+00.33
4.11+0.11
2.61+0008
4.57-0.31
18.98+0.37
LaMix
0.47:10.04
4.10
1.39
3.20-0.24
18.6010.69
Table 1: Error rate (%) of methods with PreAct ResNet- 18 and ResNet-50 as baselines. We report
mean scores over runs with standard deviations (denoted =). Best results highlighted in Bold.
3 EXPERIMENTS
We compare method with both random input mixing and learning to mixing Mixup variants.
3.1 COMPARE WITH RANDOM INPUT MIXING METHODS
We use five image classification tasks. MNIST digit (1-10) recognition dataset with 60,000 training and 10,000 test gray-level, dataset having the same scale as MNIST, containing 10 classes fashion product pictures. SVHN is the Google street view house numbers recognition data set with 73,257 digits, 32x32 color images for training. 26,032 for testing, and 531, 131 additional, easier samples. We did not use the additional images. Ci is an image classification task with 10 classes with 50,000 training and 10,000 test samples. Cifar100 is similar to Cifar 10 but with 100 classes and 600 images each.
images. Fashion is an image recognition
We experimented using networks PreAct ResNet-18 (He al. 2016) and ResNet-50 (He al. 2016) (denoted as Vanilla model). To fit the MNIST and Fashion data into the same convolutional network architecture, we just simply resize the 28x28 image to 32x32 as that of C0far10, Cifar100, and SVHN. Each reported value (accuracy or eгror rate) is the mean of five runs, on NVIDIA GTX TitanX GPU with 12GB memory. The discussed in Equation 10 was set as 0.5 unless otherwise specified.
We compare with Mixup ZZhagg et al. 2018) and the -tate-of-the-art random input mixing method CutMix (Yun al., 2019). For Mixup, we use the code from Facebook at For CutMix, we use the authors' implementation in For LaMix, the added fully connected layer is the same as the fullyconnected layer of the original network with a Softmax function on the top. For all the comparison baselines, the mixing ratio À is sampled uniformly between [0.1]. All models are trained using
(128 examples) backprop, as specified in (Zhang et al. 2018 and with the exact settings in the Facebook codes, for 400 epochs.
3.1.1 PREDICTIVE ACCURACY
The predictive eтror rates obtained by the Vanilla, Mixup, CutMix, and LaMix with PreAct ResNet- 18 and ResNet-50 as baselines are in Table
0.45
0.40
Cifa
Cifar10
0.35
Table shows that LaMix performed on par with outperformed the comparison models on all the five datasets. Results also indicate that the performance of CutMix and Mixup was sensitive to the datasets used: CutMix outperformed Mixup on some cases, but loss on some other datasets. This may due to the noisy mixed samples these two methods gen- erated during training. For example, on Cifar10 and MNIST with PreAct ResNet- 18 or ResNet-50, CutMix obtained lower accuracy than Mixup. But
0.30
0.25
0.20
0.15
0.10
0.00
Gassl
Class2
Claaa3
Clazs
Class5
Figure 2: Average values (y-axis) of the top five soft targets (x-axis) used by LaMix for training on Cifar100 (green bars) and Cifar10 (red bars).
on Cifar 00 and SVHN with PreAct ResNet- 18, CutMix outperformed Mixup with a large margin. In contrast, LaMix performed consistently better than CutMix and Mixup all the cases.
These results suggest that, although using the same mixed image for a wide range of mixing ratios, LaMix is able to assign set of appropriate soft labels to the same image, so that they can effectively
'https//githuu.com/// http:///hhhccmmmatccccccc/e////e https:/gggthnaaccommmoooooiiiaaa
4

---

Under review as a conference paper at ICLR 2021
regularize the training to achieve regularization effect effective as that of random input mixing methods, which rely on mixing different features from a given image pair to form a set of images.
3.1.2 EVOLVING SOFT TRAINING TARGETS
In Figure 2 we visual- ize the top five largest training targets used by LaMix. We provide the average values of the top targets during training for both Ci- for100 (left) and Ci- far10 (right). Results in Figure Zindicate that LaMix re-distributes large portion of the one- hot distribution to the other classes, For both C0 and Cifarl0, half of the one-hot dis- tribution mass is re-assigned the other targets. For example, as shown in Figure 2 the second largest targets for the C0 and Cifarl0 are 0. 14 and 0.25, respectively.
Figure 3: Soft targets used for training (left subfigure) and the global soft labels generated (right subfigure) for a batch of training samples over 400 epochs. The x-axis depicts the soft target values and the y-axis is the distribution mass over the 400 training epochs (from top to down).
In Figure 3 we also visualize the assigned soft labels of a random mini-batch of training samples in LaMix with PreAct ResNet-18 18 on Cifarl0 across
81.0
the 400 epochs of training. The left depicts the soft targets used for training by LaMix, and the right shows the learned global soft labels. In these figures, the x-axis depicts the soft target values and the y-axis is the distribution mass over different training epochs (from top to bottom).
80.5
80.0
79.5
79.0
78.5
Results in the left subfigure of FigureBjnnicon that the soft target labels assigned to the same batch of training samples keep changing during the training over the 400 epochs (from top to bottom), with the largest soft target training signals laying between 0.55 and 0.65. The right subfigure suggests that the learned global targets for the batch training samples tend to have small values, containing large group with values between 0.14 to 0.18, during the early phases of training (top) and then having more values laying between 0.18 and 0.20 as the training progresses (bottom of the figure).
0.1
0.2
0.3
0.4
0.5
0.6 0.7 0.8 0.9
weighting ratio
Figure 4: Accuracy (%) obtained by LaMix while varying the weight ratio 3 on Cifar100.
These results imply that for the same mixed image, LaMix able to assign it with different soft targets, thus increasing the size of the training set but bypassing the efforts needed for imput mixing.
15.0
80.0
75.0
LaMix valitation accursy
10.0
3.1.3 RE-WEIGHT
Misup validation rcurray
CutMix valicition accuray
THE LOCAL AND GLOBAL SOFT LABELS
70.0
CuMMis waining
Mauu tmining loss
5.0
We here evaluate the impact of different weight factors for 3 discussed in Equation I0 for the two soft labels, by varying it from 0.1 to 0.9. The results obtained by LaMix using PreAct ResNet-18 on Cifar 100 are Figure
LaMix texining loss
65.0
60.0
0.0
201
401
601
801
1001
1201 14011660 1901 2001 epoch
The results in Figure suggest that averaging the two sets of soft labels provides better accuracy than other weighting ratios for this dataset. Also, the
Figure 5: Training loss (right y-axis) and vali- dation accuracy (left y-axis) of LaMix, Mixup, and CutMix across 2K training epochs.
5

---

Under review as a conference paper at ICLR 2021
LaMix obtained similar accuracy when the / fell into the region between [0.3, 0.8].
Network Architecture LeNet NideResNett28-100 DenseNet-121
Vanilla
Mixup
CutMix
LaMix
58.9442.88
56.4+1.42
62.5
1.83+11822
19.9+00411
17.8++0.31
17,9+0030
17.2++0.26
19.5=0.34
17.340.13
17.3
16.9:0009
Table 2: Error rate (%) LaMix, Mixup, and CutMix with LeNet, WidcRessNe12-10 and DenseNet- 121 Cifar100. We report mean over runs with standard deviations (denoted +:)
3.1.4 TRAINING CHARACTERISTICS
In F we plot the training loss and valida- tion accuracy across extended training period,
79.0
namely 2k training epochs, of LaMix, Mixup,
78.5
and CutMix with PreAct ResNet-18 on Cifar100.
78.0
Figure shows that the training loss LaMix (thin red curve) maintains relatively higher level than the other two methods, allowing the model keep tuning the networks. Promisingly, as shown by the accuracy curve (wide red), even training for long time, LaMix is not overfitting the training set.
77.5
76.5
76.0
75.5
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
smoothing value
Figure Accuracy (%) of label smoothing with various smoothing factors on PreAct ResNet-18.
3.1.5 PERFORMANCE ON SHALLOWER, DEEPER, AND WIDER NETWORKS
We also test LaMix using different network ar- chitectures on Cifar 100, including a shallow network LeNet (Lecun et al.) 1998), a wider network WedeResNet-22 28- 10 (Zagoruyko & Komodakis 2016), and the 121 layers deep network DenseNet- 121 (Huang et al. 2017). D. For these networks, we use the implementation from Facebook
Similar to that of using PreAct ResNet-18 18 and ResNet-50, LaMix performed consistently par with or superior to the comparison models on these three additional network architectures.
3.1.6 COMPARISON WITH LABEL SMOOTHING TECHNIQUE
We also compare our method LaMix with label smoothing techniques presented in (Müller
al. 2019 Pereyra et al. 2017] Szegedy et al. 2016). These techniques train models with soft targets that are a weighted average of the hard targets and the uniform distribution over labels with a smoothing factor, aiming at preventing the network from becoming over-confident. We varies the smoothing factor from 0.1 to 0.9 on PreAct ResNet-18 18 for C00. Accuracy re- sults are presented in Figure 6
LAVA
Figure 7: Left: two classes of Cifar10 testing images; The right three depict the visualized CAMs of LaMix, Mixup, and CutMix, respectively.
Results in Figure 6 show that label smoothing
can improve the predictive accuracy of PreAct ResNet- 18 if the smoothing factor being tuned properly. but even the best accuracy obtained is still far below that achieved by LaMix. For example, the best accuracy 78.47% obtained by PreAct ResNet-18 is with label smoothing factor of 0.2, which is still below the 80.62% achieved by LaMix.
3.1.7 CLASS ACTIVATION MAPPING (CAM) VISUALIZATION
After the 400 training epochs, in Figure 7 depict the CAM maps (Zouu et al. 2016) of two classes (bird and frog) using testing samples from C0 The test images are on the left, and the

---

Under review as a conference paper at ICLR 2021
visualization CAMs created by LaMix, Mixup, and CutMix are depicted the second left, second right, and the right, respectively. One can see from Figure 7 that, LaMix tends to focus on narrower discriminative regions of the images than Mixup and CutMix.
3.1.8 REGULARIZATION EFFECT OF THE SYNTHETIC SAMPLES
We also visualize the impact of the synthetic samples created by LaMix. We present the results of three classes (i.e., Cat, Dog, and Deer) in Cifar10 with a 2D bottleneck hidden representation in the middle of the network. In specific, following the 512 filters generated by the PreAct ResNet- 18 (namely the layer before the Softmax layer), we use a fully-connected Tanh layer of 2 units as bottleneck layer, followed by two fully-connected Tanh layers with 100 and 512 units respectively. We visualize the 2D bottleneck representation for the original input samples at the 100th, 400th, 700t, and 1 training epoch. For the last three, the training errors were 100%. Figure 8 depicts the results, with different colors for the three classes.
ribe
life
40m
Figure 8: 2D data embeddings of three classes in Cifarl0 at the 100th 400th, 700th and Ikth epoch.
Results in FigureBishor showw that, the learning progresses, the embeddings for the training samples tuned gradually towards separating the three classes and the model achieves 100% accuracy at the 400th epoch (second left). On the right two subfigures (with training accuracy of 100%). the network keeps tuning using the newly created samples. At the 1000 epoch (right), the input embeddings are separated into three tight clusters with distance far away than that of the 400th epoch (second left). These results suggest that the synthetic samples help LaMix create tight representation cluster for training samples in each class and widen the gap between these clusters. These turnings happened long after the training accuracy on the original training set was not able to further increase.
3.2 COMPARE WITH LEARNING TO MIXING METHOD
In this section, we compared LaMix with the state-of-the-art learning to mixing variant of Mixup. namely the PuzzleMix (Kim et al. 2020). We use the authors' code published at [ We trained the networks for 600 epochs as implemented in the provided source codes, and kept all the settings in the codes. Each reported error is the mean of five runs, on a NVIDIA GTX TitanX GPU with 12GB memory. In this set of experiments, the in LaMix was set as 1.0 unless otherwise specified.
MNIST
Fashion
SVHN
Cifar10
Cifar100
Vanilla
0.6210.05
4.78-0.19
3.64=0.42
5.19::0.0
24.19:1.27
PreAct ResNet-18 18
Puzzle
0.57+0.04
4.18=0.06
2.80=0.04
3.1610.09
19.5640.30
LaMix
0.56+10.14
4.12-0.04
2.924-0.99
3.1740.10
19.91+0.28
Vanilla
0.61+0.05
4.55=0.14
3.22-00.5
4.83+0.30
23.10+0.62
ResNet-50
Puzzle
0.49+0.02
3.99=0.07
2.44=0.04
1.
16.75+0.22
LaMix
0.5110.02
3.09
2.33.10.04
2.80.10.02
16.6910.17
Table 3: Error rate (%) of the testing methods with PreAct ResNet- 88 and ResNet-50 as baselines. We report mean scores over runs with standard deviations (denoted +:).
Results on the five testing datasets as used to evaluate the random input mixing methods Section 3.1 are presented in Tables 3 Tables shows that LaMix performed on par with the PuzzleMix in terms of accuracy obtained. As shown in the table, LaMix obtained very similar accuracy on all the testing cases the PuzzleMix method, except for the ResNet-50 on the Cifar 10 dataset, which PuzzleMix slightly outperformed LaMix with 0.25%. These results suggest that mixing the input features with a range of mixing ratios As in PuzzleMix can be bypassed without degrading the predictive accuracy.
7

---

Under review as a conference paper at ICLR 2021
3.2.1 EXPERIMENTS ON
The authors' codes for PuzzleMix also in-
clude settings for running PuzzleMix on Imng--mageNet& (Chrabaszez et al. (2077). so here we also evaluated LaMix us- ing this dataset. This dataset has 200 classes, each with 500 training and 50 test 64x64x3 images. We presented the results in Table
Kiny-ImageNer
PuzzleMix
36.38:10.23
PreAct ResNet-18
LaMix
36.32-t0.36
PuzzleMix
30.73++022
ResNet-50
LaMix
30.68=00.24
Table 4: Error rate (%) on Imny-ImageNet.
Results in Table show that LaMix ob- tained similar error rates when compared with PuzzleMix on this dataset. For both testing network architectures, namely PreAct ResNet- 18 and ResNet-50, the accuracy difference between LaMix and PuzzleMix is marginal. These results further indicate that the input mixing effort with a range mixing ratios As in the learning to mixing variant PuzzleMix can be bypassed to conduct effective model regularization.
RELATED WORK
Various variants of the Mixup Zhang et al. 2018) method have been introduced, devoting to creating
a set mixed imputs by either randomly mixing the inputs (Guo) 2020 Guo et al. [019 Tokozume
cet al. 201 Sa b, Verma et al. 2019; Yun et al. 2019 Zhang et al. 2018) ) or learning to mixing the
informative features from an input pair (Dabouei et al. 2020; Kim et al 2020 Li et al. 2020a b
Walawalkar et al. 2020). In the former category, Mixup (Zhang et al. 2018) mixes all the input
features with a random mixing ratio. CutMix (Yun et af, 2019) creates mixed input by copying box of pixels from one image and pasted it to the other image. The second family of approaches
aim to learn to mixing important input features (Dabouei et al. 2020; Kim et al., 2020 Li et al.
2020a Walawalkar al. 2020). For example, the recent method PuzzleMix (Kim et 2020
leverages the optimal transport to combine the identified salient features from the imputs. There is also a method called Manifold Mixup (Verma et al.) (219)). which conducts feature interpolation in addition to mixing the input pair. This approach performs feature mixing randomly selected layer of the networks. Unlike these methods which generate a set of mixed inputs from a sample pair, our strategy creates only one mixed input by averaging the features from given sample pair.
Our method also relates to the label smoothing techniques, which regularize the output distribution of neural models to penalize high-confidence softmax distributions (Müller et al.) 2019: Pereyra et al. 2017. Szegedy et al. 2016). These techniques train models with soft targets that are a weighted average of the hard targets and the uniform distribution over labels with smoothing factor. These smoothing methods, however, do not relate the smooth labels with the associated features but utilize uniform label smoothing. Our approach learns to form soft targets based the features of the input pair and the states of the evolving networks.
The generation of the global soft labels in LaMix is also related to how the training targets generated by self-distillation methods (Ahn et al. 2019 Furlanello et al. [2018] Hinton et al.] 2015 Mobahi treats the final predictions as target labels for
al. 2020 Yang et 2019). However, self-distillate a new round of training, and the teacher and student architectures are identical (Mobahi et al., 2200)). In LaMix, the classifier network and the network for the global soft label have different architectures, and the training targets for the classification model are augmented by the global soft labels.
CONCLUSION AND FUTURE WORK
We contributed a finding that one can bypass the effort needed for input mixing with a range of mixing ratios in Mixup- based approaches conduct effective pairwise, variint model regularization. Consequently, one can mitigate creating noisy synthetic samples from the given inputs or reduce the computation cost required for selecting salient regions from the inputs.
Our observation may shed light on how one may conduct label-variant data augmentation in domains that are expensive perform input mixing with range of mixing ratios.
8

---

Under review as a conference paper at ICLR 2021
ADDITIONAL EXPERIMENTS
6.1 MIXUP PLUS LABEL SMOOTHING
We conducted experiments of stacking uniform label smoothing (with coefficients of 0.1, 0.2, 0.3 and 0.4) on top of Mixup, using PreAct ResNet-18 18 and ResNet-50 on Tiny ImageNet, Cifar 1 100 and Cifar10. 10. We presented the results in Table
Dataset
network
LaMix
Mixup
Mixup+ULSS
ULSO1
Mixup=ULS022
Mixup+ULS03.
Mixup+USSOX
TinylmageNet
ResNet18
34.40+0.266 26
41.68-0.33
42.31 -0.4
45.19+4..68
60.58+0.39
78.48-00553
ResNer50
85.0.115
39.09-0.28
42.58 =0.39
59.25=#091
81,06=0.24
91.82+0.37
ResNer18
19.38+0.
21.10+0.22
21.51+0.51
21.4++0.5
20.94-0.49
20.95-+0.49
Cifar100
ResNet50
18.60=00699
19448=0448
21.58 -0.86
20,8740.51
21.64=0.41
21.18 +0.58
Cifarl0
R8sNet18
3.824-0.18
3.88+-0.2
4.00+-0.17
3.95+0.13
4.06-1-004
4.06--0.03
ResNet50
3.60=20.24
4.29:40.28
4.02:40.27
4.09=20.10
4.19:40.18
4.68=0079
Table 5: Error rate (%) of the testing methods with PreAct ResNet-18 and ResNet-50 as baselines. We report mean scores over runs with standard deviations (denoted =).
These results show that label smoothing degraded the accuracy of vanilla Mixup in the cases of 100a Cifar10, and Tiny ImageNet. Such accuracy degradation is due the two layers of label smoothing in Mixup. Research has shown that the mixing label in the original Mixup has the effect of label smoothing (Carratino al. 2020). this sense, adding another label smoothing regularizer on top of the vanilla Mixup means that the Mixup will have two layers of label smoothing regularizer, which may mess up the regularization effect, thus degrading the accuracy as shown in the results above.
6.2 RANDOM INPUT MIXING IN TINY IMAGENET
We conducted additional experiments of random input mixing on Tiny ImageNet, and present the results in Table Similar that in Table D results here again suggest that, although using the same mixed image for a wide range of mixing ratios, LaMix is able to assign a set of appropriate soft labels to the same image, that they effectively regularize the training achieve regularization effect as effective as that of random input mixing methods, which rely on mixing different features from a given image pair to form set of images.
Tiny ImageNet
Vanilla 42.6110.41
Mixup
CutMix
LaMix
PreAct ResNet18 ResNet50
41 11.8510.55
43.01 #0.36
34.40
40.39
39.0910.28
41.36:1..4
28.85
Table Error rate (%) of the testing methods with PreAct ResNet-18 and ResNet-50 baselines on Tiny ImageNet. We report mean scores over 5 runs with standard deviations (denoted +.).
6.3 WITHOUT ORIGINAL. TRAINING SAMPLES
Without the original samples, the LaMix method significantly degraded its accuracy. We conducted the experiments and provided the results in Table 7
Dataset
Network
LaMix
LaMix w/o original samples
PreAct ResNet18
19.38 1-0.66
2..74
Cifar 100
ResNet50
18.60:10.69
25.57=0.36
PreAct ResNet18
3.
5.90-10.31
10
ResNet50
3.60-1-024
6.39-10.59
Table 7: Error rate (%) of the testing methods with PreAct ResNet-15 18 and ResNet-50 as baselines. We report mean scores over runs with standard deviations (denoted +).
The reason here that, without the original training samples, the training samples could be very different from the testing samples. Also, it is interesting to see that ResNet50 obtained inferior results than PreAct ResNet18 in some cases when removing the original training samples. This indeed suggests that Resnet50 fits the mixed training samples better due to its expressive power, but those

---

Under review as a conference paper at ICLR 2021
mixed training samples may be far from the testing samples due to the lack of the original training samples.
6.4 WITHOUT GLOBAL SOFT LABEL
Without the global soft label, namely pt,, the LaMix model significantly degraded the accuracy. We conducted the experiments and provided the results in Table g
Dataset
Network
LaMix
LaMix w/o global soft label
PreAct ResNerIO
19.3810.16
23,03:10.35
Cifar100
ResNet50
18.60+0.69
21.34++0.70
PreAct ResNetIO
3.82::0118
5,07 0.38 4.78:10.14
Ci
ResNet50
3.60=:0224
Table 8: Error rate (%) of the testing methods with PreAct ResNet-18 18 and ResNet-50 as baselines. We report mean scores over runs with standard deviations (denoted +).
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. arXiv, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.
Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regulariza- tion. 2020.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the CIFAR datasets. arXiv, 2017.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In CVPR, pp. 113-123, 2019.
Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, and Nasser M Nasrabadi. Supermix: Super- vising the mixing data augmentation. arXiv preprint arXivv2003.05034, 2020.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandku- Born again neural networks. In /CML, 2018.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey E. Hinton. Speech recognition with deep recurrent neural networks. CoRR, 2013.
Hongyu Guo. Nonlinear mixup: Out-of-manifold data augmentation for text classification. In AAAI, pp. 4044-4051. AAAI Press, 2020.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regulariza- tion. In AAAI, pp. 3714-3722. AAAI Press, 2019.
Stephen Jose Hanson and Lorien Y. Pratt. Comparing biases for minimal network construction with back-propagation. In NIPS, 1988.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. CoRR, abs/1603.05007, 2016.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In ICLR, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In arXiv, 2015.
10

---

Under review as a conference paper at ICLR 2021
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pP- 2261-2269, 2017.
2017.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. CoRR. bbs/1603.09382, 2016.
Hiroshi Inoue. Data augmentation by pairing samples for images classification. CoRR,
bbss/1001.0292) 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In ICML, 2020.
Alex Krizhevsky, Nya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, pp. 1097-1105, USA, 2012. Curran Associates Inc.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)):278-2222 1998.
Boyi Li, Felix Wu, Ser-Nam Lim, Serge Belongie, and Kilian Q. Weinberger. On feature normalization and data augmentation. In arXiv, 2020a.
Hao Li, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Attribute mix: Semantic data augmentation for fine grained recognition. CoRR, 2020b.
Hossein Mobahi, Mehrdad Farajtahar, and Peter L.. Bartlett. Self-distillation amplifies regularization in hilbert space. In arXiv, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In NIPS, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regularizing neural networks by penalizing confident output distributions. In ICLR workshop, 2017.
Patrice Simard, Yann LeCun, John S. Denker, and Bernard Victorri. Transformation invariance in pattern recognition tecognttiontanngent distance and tangent propagation. In Newral Networks: Tricks of the Trade, This Book an Outgrowth of a 1996 NIPS Workshop, pp. 239-27, 1998.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1): 1929-1958, January 2014. ISSN 1532-4435.
Cecilia Summers and Michael J. Dinneen. Improved mixed-exampa data augmentation. In IEEE Winter Conference on Applications of Computer Vision, WACV 2019, Waikoloa Village, HI, USA, January 7-11, 2019, pp. 1262-1270. IEEE, 2019.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence sequence learning with neural networks. CoRR, abs31140,, 2014.
C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Learning from between-class examples for deep sound recognition. In ICLR, 2018a.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-class learning for image classifica- tion. In CVPR, pp. 5486-5494, 201 8b.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi. Ioannis Mitliagkas, David Lopez-Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ICMI., 2019.
11

---

Under review as a conference paper at ICLR 2021
Devesh Walawalkar, Zhiqiang Shen, Zechun Liu, and Marios Savvides. Attentive cutmix: An enhanced data augmentation approach for deep learning based image classification. CoRR, abs/2003.13048, 2020.
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L. Yuille. Training deep neural networks in generations: A more tolerant teacher educates better students. In AAAI, pp. 5628-5635. AAAI Press. 2019.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp. 6022-6031. IEEE, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016, 2016.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
12


Reviews:
[{"review_id": "yOfuXt4gI0a", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to change the labels for the mixed examples in mixup. My major concerns are as follows.\n1.\tThe motivation of adopting soft label for mixup is not clear. Label smoothing is helpful for generic training but why it can benefit mixup?\n2.\tThe proposed method is more like a combination of mixup and label smoothing. The improvement may come from label smoothing as a generic trick rather than mixup itself.\n3.\tThe performance of proposed method is very close to mixup, where the improvement is not significant. Additional experiments on ImageNet can make the results more convincing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "rJYZcVupnBq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Main Claim:**\n\nThe authors propose to use soft labels on naive mixup as an alternative to sophisticated mixup strategy. In experiments on 5 small datasets, the proposed method can achieve better accuracy than baselines.\n\n**Strong points:**\n\nThe authors propose to use soft labels to overcome the mislearning features in mixup images. The idea is simple and straightforward. \n\nExperiment results show the method works well on small datasets.\n\n\n**Weak points:**\n\nThe contribution of this work is incremental.\n\nExperiment results on ImageNet are missing. On Tiny-ImageNet, it’s also helpful to show the performance of Mixup and CutMix.\n\nSome decisions are made without justification. Some details are not clearly explained. See questions.\n\n\n\n**Recommendation:**\n\nReject.\n\nThe proposed method is incremental. The method should be evaluated on ImageNet. \n\n**Questions:**\n\nIn Eq(9), a sigmoid function is applied before the softmax function. So the unnormalized logits of this distribution is in [-1, 1]. Why?\n\nWhy is “target soft labels too far away” an issue for the model? why does “a mini-batch with original inputs“ prevent “LaMix from assigning target soft labels too far away”? If the trick is not applied, what will happen to the model? Will the model take more time to converge or it won't converge?\n\n“For LaMix, the added fully connected layer is just a copy of the fully-connected layer of the original network with a Softmax function on the top.” Is the network pre-trained? Are the two matrices sharing weights?\n\nAs shown on Figure 2, the top 2 classes occupy a large portion of the probability. So I’m curious which part actually contributes to the improvements. Is it (A) the reweighting of y_i and y_j, or (2) the introduction of other labels? I.e. After computing Eq (10), keep the value for y_i and y_j, set all other dimensions to zero, (then renormalize the distribution), and use it as the training label, what will happen?  My impression is it may solve the problem of  “target soft labels too far away”.\n\n**Comments:**\n\nEq (3) is confusing to me. I think authors can follow the convention in (Yun et al.), rewrite the equation as $x_\\lambda^{i,j}=\\Phi(x_i, x_j, lambda) * x_i + (1 - \\Phi(x_i, x_j, lambda) * x_j$\n\nhas tow forms -> has two forms\n\n**After rebuttal:**\n\nThanks to the author for providing additional experimental data. But without the results of imagenet, it is difficult to judge the effectiveness of this method on complicated data. So I decided to keep the original score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kF7r-Ky5Ty", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe previous advanced Mixup methods, such as CutMix and PuzzleMix, involve input mixing. This paper suggests a new Mixup approach, called LaMix, that does not require input mixing. The solution is combining the original target label (interpolation of two one-hot targets) and generated target labels from an additional network to use it for training. The authors argue that LaMix achieves superior performance without input mixing.\n\nReasons for score:\nThe authors should explain the reason why using global soft labels is theoretically plausible before describing the method. Although empirical results look nice, I am suspicious about the experimental settings for the comparison with other methods.\n\nPros:\n- The paper includes diverse results on probing experiments.\n- The paper is clearly written.\n\nConcerns/Questions:\n- As far as I understood, neural network parameters for the global soft label (W_t) are not trained because they are only used for training labels. Then, these parameters are just randomly initialized values. Is it right? If so, isn’t the final effect sensitive to the initialization?\n- Sigmoid activation is used in equation (9), different from equation (7). However, there is no explanation about the reason for using it. My guess is to make artificial labels similar to each other.\n- I don’t understand why beta in LaMix is 1.0 for Section 3.2 experiments. First, I think beta = 1.0 means not using the global soft label, and it is equivalent to standard Mixup. Second, the authors mention that setting beta to 0.5 is a good choice in Section 3.1.3 (Figure 4). Moreover, they use beta as 0.5 for the experiments of Section 3.1.1 (Table 1). The settings of Table 1 and Table 3 (model architectures and datasets) are the same except for the beta value of LaMix.\n- Could you provide hyperparameters used for other Mixup methods as a baseline? Are they well-tuned?\n- I am curious whether the combination with the global soft label can be done after input mixing. If then, I think providing these results would be helpful to check whether the regularization effect is orthogonal to input mixing.\n- To compare LaMix with label smoothing, I think the author should apply label smoothing to Mixup rather than the vanilla setting.\n\nMinor comment:\n- “Puzzle Mix” -> “PuzzleMix”", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ph6zhjnK-eA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper simply combines mixup and self-distillation to achieve more adaptive soft label, which effectively regularize the training. In the manuscript, authors argue that the existed mixup-based approaches has two mainly efforts, may create misleading training samples or meet computation cost issue on creating samples. Motivated by this, they propose \"LaMix\", which can leverage the information of self-distillation, to solve those two efforts and achieve competitive performance with SOTA \"Puzzle-mixup\".   \n\n\nComment\n\n1. This paper introduces a combination method between mixup and self-distillation, and simply use an additional FC layer to have adaptive soft label, which is intuitive and clear but lack of novelty. Can you give more insightful comment about how adaptive label can help mixup soft label?\n\n2. In Figure 3, I think the provided evidence for the effect of considering adaptive soft label with mixup approach is promising.\n\n3. For experimental results, in section 3.2, the proposed \"Lamix\" seems not achieve significant improvement compared to the SOTA \"Puzzle-mix\" on CIFAR-10 and 100.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to improve Mixup by using soft labels, removing the need for input mixup. The reviewers found the paper was clear and found the experiments promising. The reviewers raised concerns about the lack of experiments comparing this approach to Mixup+Label smoothing, which were addressed during the rebuttal by the authors. However, the reviewers did not find the empirical evidence strong enough given that this is mostly an empirical contribution. The authors do not necessarily need to train on the full Imagenet, but it would be beneficial to evaluate on more standard settings on the dataset considered to facilitate comparison to previous work.
Decision: Reject