Teach me how to Interpolate a Myriad of Embeddings
Authors: Shashanka Venkataramanan
Ewa Kijak
laurent amsaleg
Yannis Avrithis
Venue: Submitted to ICLR 2023 (2023)

Abstract: Mixup refers to interpolation-based data augmentation, originally motivated as a way to go beyond empirical risk minimization (ERM). Yet, its extensions focus on the definition of interpolation and the space where it takes place, while the augmentation itself is less studied: For a mini-batch of size $m$, most methods interpolate between $m$ pairs with a single scalar interpolation factor $\lambda$.

In this work, we make progress in this direction by introducing MultiMix, which interpolates an arbitrary number $n$ of tuples, each of length $m$, with one vector $\lambda$ per tuple. On sequence data, we further extend to dense interpolation and loss computation over all spatial positions. Overall, we increase the number of tuples per mini-batch by orders of magnitude at little additional cost. This is possible by interpolating at the very last layer before the classifier. Finally, to address inconsistencies due to linear target interpolation, we introduce a self-distillation approach to generate and interpolate synthetic targets.

We empirically show that our contributions result in significant improvement over state-of-the-art mixup methods on four benchmarks. By analyzing the embedding space, we observe that the classes are more tightly clustered and uniformly spread over the embedding space, thereby explaining the improved behavior.

Under review as a conference paper at ICLR 2023
TEACH ME HOW TO INTERPOLATE A MYRIAD OF EM- BEDDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
Mixup refers to interpolation-base data augmentation, originally motivated as a way to go beyond empirical risk minimization (ERM). Yet, its extensions focus on the definition of interpolation and the space where it takes place, while the augmentation itself is less studied: For mini-batch of size m, most methods interpolate between m pairs with a single scalar interpolation factor X.
In this work, we make progress in this direction by introducing MultiMix, which interpolates an arbitrary number n of tuples, each of length m, with one vector X per tuple. On sequence data, we further extend to dense interpolation and loss computation over all spatial positions. Overall, we increase the number of tuples per mini-batch by orders of magnitude at little additional cost. This is possible by interpolating at the very last layer before the classifier. Finally, to address inconsistencies due to linear target interpolation, approach to generate and interpolate synthetic targets.
introduce delf-distillation
We empirically show that our contributions result in significant improvement over sfate-of-the-art mixup methods on four benchmarks. By analyzing the embedding space, we observe that the classes are more tightly clustered and uniformly spread over the embedding space, thereby explaining the improved behavior.
INTRODUCTION
Mixup (Zhang et al., 2018) is a data augmentation method that interpolates between pairs of training examples, thus regularizing a neural network to favor linear behavior in-between examples. Besides improving generalization, it has important properties such as reducing overconfident predictions and increasing the robustness to adversarial examples. Several follow-up works have studied interpola- tion in the latent embedding space, which is equivalent to interpolating along manifold in the input space (Verma et al., 2019), and a number of nonlinear and attention-batt
interpolation mech- anisms (Yun et al., 2019; Kim et al., 2020; 2021; Uddin et al., 2021; Hong et al., 2021). However, little progress has been made in the augmentation process itself, i.e., the number of examples being interpolated and the number of interpolated examples being generated.
Mixup was originally motivated as a way to go beyond empirical risk minimization (ERM) ( (Vapnik, 1999) through a vicinal distribution expressed as an expectation over an interpolation factor which is equivalent to the set of linear segments between all pairs of training inputs and targets. In practice however, in every training iteration, a single scalar À drawn and the number of interpolated pairs is limited to the size of the mini-batch, as illustrated in Figure (a)) This is because, if interpolation takes place in the input space, would be expensive to increase the number of examples per iteration. To our knowledge, these limitations exist in all mixup methods.
In this work, we argue that a data augmentation process should augment the data seen by the model, or at least by its last few layers, as much as possible. In this sense, we follow manifold mixup (Verma al, 2019) and generalize it in number of ways to introduce MultiMix, as illustrated in Figure 1(b). First, rather than pairs, we interpolate tuples that are as large as the mini-batch. Effectively, instead of linear segments between pairs of examples in the mini-batch, we sample on their entire corver hull. Second, we draw different vector À for each tuple. Third, and most important, increase the number of interpolated tuples per iteration by orders of magnitude by only slightly decreasing the actual training throughput in examples per second. This is possible by interpolating at the deepest.

---

Under review as a conference paper at ICLR 2023
mixed clean
(a) mixup
(b) MultiMix (ours)
Figure 1: Data augmentation from mini- batch B consisting of m = 10 points in two dimensions. (a) mixup: sampling of m points on linear segments between pairs of points in B, using the same interpolation factor X. (b) MultiMix: sampling of n 300 points in the convex hull of B.
layer possible, i.e., just before the classifier, which also happens to the most effective choice. The interpolated embeddings are thus only processed by single layer.
Apart from increasing the number of examples seen by the model, another idea to increase the number of loss terms per example. In many modalities of interest, the input is sequence in one or more dimensions: pixels or patches images, voxels in video, points or triangles in high- dimensional surfaces, name few. The structure of input data expressed in matrices or tensors, which often preserve a certain spatial resolution until the deepest network layer before they collapse e.g. by global average pooling (Szegedy et al., 2015; He al., 2016) or by taking the output of classification token (Vaswani et al., 2017; Dosovitskiy et al., 2020).
In this sense, we choose to operate at the level of sequence elements rather than representing ex- amples by single vector. We introduce dense MultiMix, which is the first approach of this kind in mixup-based data augmentation. sequence elements and we also apply the loss densely, illustrated in Figure 2. This is extreme form of augmentation where the number of interpolated tuples and loss terms increases further by one two orders of magnitude, but little cost.
particular, we interpolate densely the embeddings and targets of
Finally, linear interpolation of targets, which is the norm in most mixup variants, has a limitation: Given two examples with different class labels, the interpolated example may actually lie in region associated with a third class in the feature space, which is identified as manifold intrusion (Guo al., 2019). In the absence of any data other than the mini-batch, straightforward way address this limitation is to devise targets originating in the network itself. This naturally leads to self- distillation, whereby moving average of the network acts as teacher and provides synthetic soft targets (Tarvainen & Valpola, 2017), to be interpolated exactly like the original hard targets.
In summary, we make the following contributions:
1. We introduce MultiMix, which, given a mini-batch of size m, interpolates an arbitrary number n m of tuples, each of length m, with one interpolation vector À per tuple- compared with m pairs, all with the same scalar À for most mixup methods (subsection 3.2).
2. We extend to dense interpolation and loss computation over all spatial positions (subsec- tion 3.4).
3. We use online self-distillation to generate and interpolate soft targets for mixup-compare with linear target interpolation for most mixup methods (subsection 3.3).
4. We improve over sfte-of-thh-ar mixup methods on image classification, robustness to adversarial attacks, object detection and out-of-distribution detection (section 4).
2 RELATED WORK
Mixup: interpolation methods In general, mixup interpolates between pairs of input exam- ples (Zhang et al., 2018) or embeddings (Verma et al., 2019) and their corresponding target labels. Several follow-up methods mix input images according to spatial position, either at random rectan- gles (Yun et al., 2019) or based on attention (Uddin et al., 2021; Kim et al., 2020; 2021). in an attempt to focus different object in each image. We also use attention in our dense MultiMix variant, but
2

---

Under review as a conference paper at ICLR 2023
in the embedding space. Other definitions of interpolation include the combination of content and style from two images (Hong et al., 2021) and the spatial alignment of dense features ( (Venkatara- manan et al., 2022). Our dense MultiMix variant also uses dense features but without aligning them, hence it can mix a very large number of images and generate a large number of interpolated sam- ples. Our work is orthogonal to these methods rather than on the definition of interpolation.
focus the sampling process of augmentation
Mixup: sampling To the best of our knowledge, the only methods that interpolate more than two examples for image classification are OptTransMix (Zhu al., 2020), SuperMix (Dabouei et al., 2021) and (-Mixup (Abhishek et al., 2022). All three methods operate in the input space and limit the number of interpolated examples to the mini- batihbacch size, m: whereas our MultiMix generates an arbitrary number of interpolated examples (n = 1000 in practice) in the embedding space. To determine the interpolation weights, OptTransMix involves a complex optimization process and only applies to images with clean background; ç-Mixup uses random permutations of a fixed vector; and SuperMix uses Dirichlet distribution over not more than 3 samples practice. We also sample weights from the Dirichlet distribution but interpolate many examples as the mini-batch, m.
Beyond classification, m-Mix (Zhang et al., 2022) uses graph neural networks in a self-supervised setting with pair-based loss functions. The interpolation weights are deterministic and based on pairwise similarities. Essentially, this operation resembles a layer of graph neural network.
Self-distillation Distillation refers to a two-stage knowledge transfer process where a larger teacher model ensemble is trained before predicting soft targets to train a smaller student model on the same (Buciluă et al., 2006; Hinton et al., 2015; Romero et al., 2014; Shen et al., 2019; Zagoruyko & Komodakis, 2016a) or different (Radosavovic et al., 2018; Xie et al., 2020) training data. The architecture of the two models may be the same with training at multiple stages, for example in co-distillation,
continual learning (Rebuffi et al., 2017; Li & Hoiem, 2018). In self-distillation not only the models are the same, but the knowledge transfer process is also online, e.g. between layers of the same model (Zhang al., 2019) between two versions of the model (Anil et al., 2018), where the teacher parameters may be obtained from the student rather than learned (Tar- vainen & Valpola, 2017). The latter approach has been successful in self-supervised representation learning (Grill et al., 2020; Caron et al., 2021; Zhou et al., 2022).
As far as we know, distillation has only been used for mixup in SuperMix (Dabouei et al., 2021) as a two-stage process from a large pre- pre-trained teacher model to a small student model. We are the first to use single-stage, online self-distillation where the teacher model is moving average of the student.
this context, following (Tarvainen & Valpola, 2017),
Dense loss functions Although standard in dense tasks like semantic segmentation (Noh et al., 2015; He et al., 2017), where dense targets commonly exist, dense loss functions are less common otherwise. Few examples are in few-shot learning (Lifchitz et al., 2019; Li et al., 2019), where data augmentation is of utter importance, and in unsupervised representation learning, e.g. dense contrastive learning (O Pinheiro et al., 2020; Wang et al., 2021), learning from spatial correspon- dences (Xiong et al., 2021; Xie et al., 2021a) and masked language or image modeling (Devlin et al., 2019; Xie et al., 2021b; Li et al., 2021; Zhou et al., 2022). Some of these methods use dense dis- tillation (Xiong et al., 2021; Zhou et al., 2022), which is also studied in continual learning (Dhar al., 2019; Douillard et al., 2020). To our knowledge, we are the first use dense interpolation and a dense loss function for mixup. Our setting is supervised, similar to dense classification (Lifchitz al., 2019), but we also use dense distillation (Zhou al., 2022).
METHOD
3.1 PRELIMINARIES AND BACKGROUND
Problem formulation Let E X be an input example and y E y its one-hot encoded target, where X' R is the input space, } {0, 1] and is the total number of classes. Let fo X' - Bd be an encoder that maps the input x to an embedding z = fo(x), where d is the dimension of the embedding. A classifier gw Rd c-1 maps vector p gw gw(2) of predicted probabilities
3

---

Under review as a conference paper at ICLR 2023
over classes, where A" C R'+1 is the unit n-simplex, i.e. p 0 and 1. 1, and 1. R is an all-ones vector. The overall network mapping is f := gw o fo.
Parameters (0, W) are learned by optimizing plcs, lct X = (I1,.... (m)) E IR DR be the inputs, Y = (yi..... ...) E Rex" the targets and P == (p...... Pm)) Rexm the predicted probabilities of the mini-batch, where P f(X) ((Tm)) The objective is to minimize the cross-entropy
mini-batches. Given a mini-batch of m exam-
H(Y, P) == -1? (Y @ log(()))
(1)
of predicted probabilities P relative to targets Y averaged over the mini-batch, where C is the Hadamard (element-wise) product. In summary, the mini-batch loss is
L(X,Y; HHY,HHYY,W, lww.gw((f( (fo(X))).
(2)
Mixup Mixup methods commonly interpolate pairs of inputs or embeddings and the correspond- ing targets the mini-batch level while training. Given mini-batch of m examples with inputs X and targets Y. let Z = (Z1...., 2m) E Rdx= be the embeddings of the mini-batch, where Z = fx( Manifoll mixup (Verma et al., 2019) interpolates the embeddings and targets by form- ing a convex combination of the pairs with interpolation factor X E [0, 1]:
7 = 2(A + (1- A)IT)
(3)
- Y + (1 X)I),
(4)
where À ~ Beta(a, a), 1 is the identity matrix and I1 R=xx is a permutation matrix. Input mixup (Zhang et al., 2018) interpolates inputs rather than embeddings:
X = ( + (1 A)IT).
(5)
Whatever the interpolation method and the space where it is performed, the interpolated data, e.g. X (Zhang et al., 2018) 7 (Verma et al., 2019), replaces the original mini-batch data and gives rise to predicted probabilities P = (PI. Pm) E Rx over classes, e.g. P f(X) (Zhang et al., 2018) or P gw (2) (Verma al., 2019). Then, the average cross-entropy H( P) (I) between the predicted probabilities P and interpolated targets Y is minimized. The number of interpolated data is m, same as the original mini-batch data.
3.2 MULTIMIX
Interpolation Given mini-batch of m examples with embeddings Z and targets Y, we draw interpolation vectors A Dir(a) for k I.....m. n, where Dir(a) the symmetric Dirichlet distribution and A E A A that is, A 2 0 and 1 A 1. We then interpolate embeddings and targets by taking n convex combinations over all m examples:
= ZA
(6)
-=
where A (Ai...., An) E Rmxx, We thus generalize manifold mixup (Verma et al., 2019);
1. from pairs tuples of length m, long the mini-batch: m-term convex combina- tion (6).() vs. 2-term in (3), Dirichlet vs. Beta distribution;
2. from dx in (3), interpolated targets Y Rex= (7) vs. Rexm in (4);
arbitrary number n of tuples: interpolated embeddings Ž E dxn (6)
3. from fixed X across the mini-batch differrnnt X for each interpolated item.
Loss Again, we replace the original mini-batch embeddings Z by the interpolated embeddings
7 and minimize the average cross-entropy H(Y, , P) (1) between the predicted probabilities P gw (Z) and the interpolated targets Y (7). Compared with (2), the mini-batch loss becomes
T (X, Y: W) H(YA, gw (fo(X)A))
(8)
4

---

Under review as a conference paper at ICLR 2023
norm
201
encoder fe
norm
72
322
Figure 2: Dense MultiMix (subsection 3.4) for the special case m 2 (two examples), n (one interpolated embedding), r = 9 (spatial resolution 3 x 3). The embeddings Z1, 22 E 12dx9 of input images I1, 22 are extracted by encoder fo- Attention maps 01, 02 E R are extracted (10), multiplied element-wise with interpolation vectors X, (1 X) E R (11) and éj-normalized per spatial position (12). The resulting weights are used to form the interpolated embedding E RR4x9 as convex combination of 21, per spatial position (13). Targets are interpolated similarly (14).
3.3 MULTIMIX WITH SELF-DISTILLATION
Networks We use online self-distillation approach whereby the learned network / == gw becomes the student, whereas a teacher network f := gw o før the same architecture obtained by exponential moving average of the parameters (Tarvainen & Valpola, 2017; Grill et al., 2020).
The teacher parameters (0', W") are not learned: We stop the gradient the computation graph.
Views Given two transformations T and T', we generate two different augmented views v = ((x) and v' = f((x) for each input z, where
T and t' T". Then, given mini-batch of m examples with inputs X and targets Y, let V = f(X), V (()) E R Dxm the mini-batch views corresponding to the two augmentations and Z fo(V), Z' = (V') Rdxm the embeddings obtained by the student and teacher encoders respectively.
Interpolation We obtain the interpolated embeddings Ž, Z' from Z, Z' by (6) and targets Y from by (7), using the same A. The predicted class probabilities are given by gw (Z) and Pr gw (Z'), again obtained by the student and teacher classifiers, respectively.
Loss We learn parameters (0, W) by minimizing a classification and self-distillation loss:
y (1-
where 7 E [0, 1]. The former brings the probabilities P predicted by the student close to the targets Y. as in (8). The latter brings P close to the probabilities Fr predicted by the teacher.
3.4 DENSE MULTIMIX
We now extend to the case where the embeddings are structured, e.g. in tensors. This happens e.g- with token sentence embeddings in NLP and patch vs. image embeddings in vision. This works by removing spatial pooling and applying the loss function densely over all tokens/patches. The idea illustrated in Figure 2. For the sake of exposition, either by example or by spatial position. practice, all operations are on tensors.
formulation uses sets of matrices grouped
Preliminaries The encoder now fo X' + dxr mapping the input & to an embedding z = fo(#) E Rddr, where d is the number of channels and is its spatial resolution if there are more than one spatial dimensions, these are flattened.
Given mini-batch of examples, have again inputs X (Ii, (Ii,... Fm) Dxm and targets Y (y),..,, Dm) E Rexm Each embedding 2; = fo(x;) = (z.,..., +f)) E Rdxr for i 1,...,m consists of features =f E Rd for spatial position j 1, r.. We group features by position in matrices Zt ,..., Z", where Z (zf.-.., 2/n)) E Rdxm for j 1, 1,
5

---

Under review as a conference paper at ICLR 2023
Each feature vector will inherit the target of the corresponding input example. However, we also attach a level of confidence according to an attention map. Given an embedding z E Rdxr with target y y and vector E Rd, the attention map
Attention
h(z (izzTue ER"
(10)
measures the similarity of features of to where h is a non-linearity, egg. softmax or ReLU followed by & normalization. There are different ways define vector u. For example, u z1,/r
by global average pooling (GAP) of z. or u Wy assuming a linear classifier with W E Raxe similar class activation mapping (CAM) (Zhou et al., 2016). In the case of no attention, a 1,/r is uniform.
Given a mini-batch, let a; (a), (a.,...,af) .,)) E R be the attention map of embedding z, (10). We group attention by position in vectors a ,.... a", where (a)...., a E R for j = .....,
Interpolation For each spatial position j = 1, ,...., draw interpolation vectors X Dir(a)
for k = 1, ...... n and define A' = (Xi...., M/) Rmxx, Because input examples are assumed to
contribute according to the attention vector a² R" we scale the rows of A accordingly and then we normalize its columns back to A m-1 so that they can define convex combinations:
M = diag(a')
(11)
M M diag(IT.
(12)
We then interpolate embeddings and targets by taking convex combinations over m examples:
Ø = M
(13)
y Y
(14)
This is similar to (6),(7) but there a different interpolated embedding matrix Z E [g as well as target matrix y E Rexn per position, even though the original target matrix Y is one.
Classifier The classifier is now gw Rdxr + Rexr, maintaining the same spatial resolution the emhedding and generating one vector of predicted probabilities per spatial position. This is done by removing average pooling or any down-sampling operation. The interpolated embeddings ž' "!...,Z Z" (13) are grouped by example into ži.... Zn E Rdxr, mapped by Iw to predicted probabilities Pi...., Pn, E Rexr and grouped again by position into PL pi,.... . Pr E Rexx
In the simple case where the original classifier is linear, i.e. W E Rdxe it seen convolution and applied densely to each column (feature) of Z for l....,r.
Loss Finally, learn parameters 0, W by minimizing the weighted cross-entropy H(Y', Pl:s) of pi relative to the interpolated targets y again densely at each position j, where
H P:s) = - (Y 2
(15)
generalizes (1) and the weight vector defined as s = 1 M E This is exactly the vector used to normalize the columns of M in (12). The motivation is that the columns of M are the original interpolation vectors weighted by attention: A small & norm indicates that for the given position j. are sampling from examples of low attention, hence the loss is to be discounted.
4 EXPERIMENTS
4.1 SETUP
We use a mini-batch of size m = 128 examples in all experiments. For every mini-batch, we ap- ply MultiMix with probability 0.5 input mixup otherwise. For MultiMix, the default settings are given in subsection 4.4. We follow the experimental settings of AlignMixup (Venkataramanan et al., 2022) and use PreActResmet-11 18 (R-18) (He al., 2016) and WRN16-8 (Zagoruyko & Komodakis,

---

Under review as a conference paper at ICLR 2023
DATASET NETWORK
CIFAR-10
CIAR--10
TI
R-18
W16-8
R-18
W16-8
R-18
Buseline!
95.41=20.22
94.93-0006
76.69
78.80+00.55
56.49:6-221
Manifold mixup (Verma et al, 2019y!
97.00:10.05
96.44.10.02
80.00..0.44
80.77:10.26
59,31
PuzzleMix (Kim et al,, 2020)*
97.04:10.04
97.00
79.98 0.005
80.78 1:0.23
63.52 63.52.10.42 1:042
Co-Mixup (Kim al., 2021)²
97. 07.10=20.03
96.44:00.88
80.28
80.39 54
64. 13:10.4
AlignMixup (Venkatarammmmn et al., 2022)*
97.06:00.04
96.91 ::000
81.71 0:000
81.24
66,88 0:07
(-Mixup (Abhisbok 2022y*
96.26
96.35
80,46
79.73 00:13-1:0.15
63.18 63.18:10.14
MultiMix (ours)
0..07:00.03
97. 07.06:20.02
0..02.00044
81.44::0.03
14:004
distil
97. 17.0222
97.12:10.03
82.18 1:1.1
82.06
68.06:10.00
dense
97.09-40.02
97.09 --000
81.93
9.993--0004
81.77+40008
68.44-1-0.05
dense disiil
97.16
97.20=0.02
82.32
69.11
+0.64
+1.08
+2.26
Table 1: Image classification on CIFAR-10/100 and TI (Tinylmagenet), Mean and standard devia- tion of Top-1 accuracy (%) for 5 runs. R: PreActResnet, W: WRN. *: reproduced, t: reported by AlignMixup, Bold black: best; Blue: second best; underline: best baseline. Gain: improvement over best baseline. Comparison with additional baselines is given in subsection A.2
NETWORK
REsNET-50 SPEED
VYT-S/16
METHON
ERROR
SPEED
FRROR
Baseline
1.17
76.32
1.01
73.9
Manifold mixup (Verma et al., 2019)"
1.15
77.50
0.97
75.2
PuzzleMix (Kim at., 2020)*
0.84
78.76
0.73
Co-Mixup (Kim all, 2021>*
0.62
0.57
75.9
AlignMixup (Venkataramanan et all, 2022y!
1,03
79.32
MultiMix (ours)
1,16
78.81
0.98
75.2
+ distil
1.06
80.12
0.93
76.7
0,95
79.32
dense + distil
0.83
80.21
0.81
76.9
Gain
+0.89
+1.0
Table Image classification and training speed ImageNet. Top-1 accuracy (%): higher is better. Speed: images/sec (x x100): higher is better. 1: reported by AlignMixup. Bold black: best; Blue: second best; underline: best baseline. Gain: improvement over best baseline. Comparison with additional baselines is given in subsection A.2
2016b) as encoder on CIFAR-10 and CIFAR-100 datasets Krizhevsky et al. (2009); R-18 on Tiny- Imagenet (Yao Miller, 2015) (TI); and Resnet-50 (R-50) and ViT-S/16 (Dosovitskiy et al., 2020) on ImageNet (Russakovsky et al., 2015).
We report the mean and standard deviation of the top-1 accuracy (%) for five runs on image clas- sification. On robustness to adversarial attacks (subsection 4.2), we report the top-1 error. We also experiment on object detection (subsection 4.3) and out-of-distribution detection (subsection A.3).
4.2 RESULTS: IMAGE CLASSIFICATION AND ROBUSTNESS
Image classification In Table we observe that MultiMix and Dense MultiMix already outper- form SoTA on all datasets except CIFAR-I0 with R-18, where they are on par with Co-Mixup. The addition of distillation increases the gain and outperforms SoTA all datasets. Both distillation and dense improve over vanilla MultiMix and their effect is complementary on all datasets. On TI for example, distillation improves by 0.95%, dense by 1.33% and their combination by 2.0%. This combination brings an impressive gain of 2.26% over the previous SoTA AlignMixup. We provide additional analysis of the embedding space on 10 classes of CIFAR- 100 in subsection A.4.
In Table 2 we observe that on ImageNet with R-50, vanilla MultiMix outperforms all methods except Aligniixup. Adding dense, distillation or both outperforms all SoTA with both R-50 and ViT-S/16. Importantly, it brings an overall gain of 4% over the baseline with R-50 and 3% with ViT-S/16.
Training speed Table shows the training speed of MultiMix and its variants compared with SoTA mixup methods, measured on NVIDIA V-100 GPU, including forward and backward pass. In
7

---

Under review as a conference paper at ICLR 2023
ATTACK
FGSM
PGD
DATASET
CIFAR-1O
CIFAR-I00
TI
CIFAR-10
CIFAR-100
NETWORK
R-18
W16-8
R-18
W16-8
R-18
R-18
W16-8
R-18
W16-8
Baseline!
883:::0:11
3834-033
8722:0.10
72.6:0222
91.9-0006
999::00
99.9:0001
99.-0001
99.900001
Manifold mixup (Verma et all. 201997
76.91:0.14
76.01:0.044
9:.20:0.06
1.00
89.0
97.2:00011
98.41:0.03
99.61:0011
98.4::0.03
PuzzieMix
2020g
57.49:0.22
607-00002
788:00099
97.7-00001
97.04:001
9644:002
95.24:0033
Co Mixup (Kim et 2021)
60:01:0:05
58.8:0.10
07.51:0.022
56.5:00.04
97.000
96.11:0:00
95.31:0000
91.2:100011
AlignMisup (Venkataamazan et al., 2022y?
54.8--0.00
560-+-005
74.1+H0044
55.0+-0.33
0..0
95, 95.34-00.04
96.7
904+-000
+2.1
M-Mixup (Abblshck or el., 2022)*
72.82:023
67.3220344
75.3::021
68.020221
84.7:00.18
98.0220.06
98 38.62:0.03
97.420.00
96.1200.10
Mala (ours)
$4.12:0.09
55.34:0044
75:8::0044
$4.5:00011
77.5:00011
94.24:0.04
94.80:0001
90.00:0011
916:00001
+ distillation
52.5 4:0.5
5144:0001
52.71-0.02
76.2-:0.05
02.01
93.96-0.02
888-1-0011
90.000001
1:.11:00:1
75.5
92.9
92.61:001
88.61:0.00
dense distilition
$2.0 0000
50.1 4-004
7300
0.02
52.1 0.02
75.1 0:001
90.8
+0001
90.5
87.5 0:001
90.1::0003
Gain
+2.8
+5.9
+1.1
+4.5
45.6
+2.0
Table Robustness to FGSM PGD attacks. Mean and standard deviation of Top-1 error (%) for 5 runs: lower is better. ": reproduced, t: reported by AlignMixup. Bold black: best; Blue: second best; underline: best baseline. Gain: reduction error over best baseline. TI: Tiny Imngenttt R: PreActResnet, W: WRN. Comparison with additional baselines is given in subsection A.2
terms of training speed, the vanilla MultiMix is on par with the bascline, bringing a gain of 2.49%. The addition of distillation is on par with SoTA AlignMixup, bringing gain of 0.80%. Adding both dense and distillation brings gain of 0.89% over AlignMixup, while being 19.4% slower. The inference speed is the same for all methods.
Robustness to adversarial attacks We follow the experimental settings of Align- Mixup (Venkataramanan al., 2022) and use 8/255 Loo e-ball for FGSM (Goodfellow et al., 2015) and 4/255 Loc e-ball with step size 2/255 for PGD (Madry et al., 2018) attack. In Table 3 we observe that vanilla MultiMix is already more robust than SoTA on all datasets and settings except FGSM on CIFAR-100 with R-18, where it is on par with AlignMixup. The addition of dense, distillation or both again increases the robustness and shows that their effect is complementary. The overall gain is more impressive than in classification error. For example, against the strong PGD attack on CIFAR- 10 with W16-8, the SoTA Co-Mixup improves the baseline by 3.8% and our best result improves the baseline by 9.4%, which is more than double.
4.3 RESULTS: TRANSFER LEARNING OBJECT DETECTION
We evaluate the effect of mixup on the generaliza- tion ability of a pre-trained network to object detec- tion as downstream task. Following the settings of CutMix (Yun al., 2019), we pre-train R-50 on Ima- geNet with MultiMix and its variants and use as the backbone for SSD (Liu al., 2016) with fine-tuning on Pascal VOC07+12 12 (Everingham et al., 2010) and Faster-RCNN (Ren et al., 2015) with fine-tuning on MS-COCO (Lin et al., 2014).
DATASET
VOCU7+112
MS-COCO FASTER CNN
Baseline*
76.7
33.27
Input mixup"
76.6
34.18
CuiMix!
77.6
35.16
AlignMixup'
35.84
MultiMi (ours)
77.9
35.73
+ distil
78.7
35.97
+ dense
78.5
35.89
dense distil
79.1
36.41
In Table 4, we observe that, while vanilla MultiMix is slightly worse than AlignMixup, dense and dis- tillation bring improvements over the SoTA on both datasets and are still complementary. This is con- sistent with classification results. Compared with the baseline, our best setting brings gain of 2.40% mAP on Pascal VOCO7+ 12 and 3. 14% on MS-COCO.
Gain
+0.7
+0.57
Table 4: Transfer learning to object detec- tion. Mean average precision (mAP, %): higher is better. t; reported by AlignMixup. Bold black: best; Blue: second best; under- line: best baseline. Gain: increase in mAP.
4.4 ABLATIONS
All ablations are performed using R-18 CIFAR-100. For MultiMix, we study the effect of the layer where we interpolate, the number of tuples n and a fixed value of Dirichlet parameter a. More ablations are given in the supplementary material.
8

---

Under review as a conference paper at ICLR 2023
MuhiMix
MultiMix + distill
Dense MultiMix
Dease MuhiMix + distill
83
83
82
Accuracy
82
81
80
81
78
80
80
00.1
0,2
0,3
0.4
10'
10a
10a
10
10
0.5
1.5
(a) Mixing layers
(b) tuples v
(c) Dirichlet parameter
Figure 3: Ablation study of MultiMix and its variants on CIFAR-100 using R-18. (a) Interpolation
layers (R-18 block; O: imput mixup). (b) Number tuples n. (c) Dirichlet parameter a.
Interpolation layer For MultiMix, we use the entire network as the encoder fe by default, except for the last fully-connected layer, which we use as classifier gw- Thus, we interpolate embeddings in the deepest layer by default. Here, study the effect of different decompositions of the network f = Sw o fo. such that interpolation takes place at a different layer. When using distillation, we interpolate the same layer for both the teacher and the student. In Figure 3a), observe that mixing at the deeper layers of the network significantly improves performance. The same behavior observed when adding dense, distillation, or both. This validates our default choice.
is interesting that the authors of input mixup (Zhang et al., 2018) found that convex combinations
of three or more examples in the input space with weights from the Dirichlet distribution do bring further gain. This agrees with the finding of SuperMix (Dabouei et al., 2021) for four or more examples. Figure 3(a) suggests that further gain emerges when mixing in deeper layers.
Number of tuples n Since our aim is to increase the amount of data seen by the model, or least part of the model, it is important to study the number n of interpolated embeddings. We observe from Figure 3(b) that accuracy increases overall with n and saturates for n 2 1000 for all variants of MultiMix. Our best setting. Dense MultiMix with distillation, works best at 1000. We choose this as default, given also that the training cost increases with n. The training speed as a function of is given in the supplementary material and is nearly constant for 1000.
Dirichlet parameter Our default setting is to draw uniformly at random from (0.5, 2| for every interpolation vector (column of A). Here we study the effect of a fixed value of a. In Figure 3(C), we observe that the best accuracy comes with uniform distribution over the convex hull of the mini-batch embeddings. However, all mcasurements lower than the default ~ U0..5, 2]. For example, from Table (CIFAR-100 100, R-18), dense MultiMix + distillation has accuracy 82.52, compared with 82.23 in Figure 3(c) for a =
1 for most MultiMix variants, corresponding to the
5 CONCLUSION
In terms of input interpolation, the take-home message of this work is that, instead of devising smarter and more complex interpolation functions the input space or the first layers of the repre- sentation, it is more beneficial to just perform linear interpolation in the very last layer where the cost minimal, and then increase as much as possible the number of interpolated embeddings for mixup. This is more in line with the original motivation of mixup as way to go beyond ERM. In terms of target interpolation, the take-home message the opposite: instead of linear interpolation of original targets, find new synthetic targets for the interpolated embeddings with the help of the network itself, then interpolate them linearly. This idea fits nicely with self-distillation, which is popular in settings such self-supervised representation learning and continual learning. Interest- ingly, self-distillation can be seen as yet another form of augmentation, but in the model space.
A natural extension of this work is to settings other than supervised classification. A limitation is that it is not straightforward to combine the sampling scheme MultiMix with complex interpolation methods, unless they are fast compute in the embedding space.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kumar Abhishek, Colin Brown, and Ghassan Hamarnch. Multi-sample Ç-mixup: Richer, more realistic synthetic samples from a p-series interpolant. arXiv preprint 2tti::2200003323,,, 2022. 3, 7, 8, 1.2
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, and Geoffrey E Hinton. Large scale distributed neural network training through online distillation. arXiv preprint arXiv: 0a...1004..3333 2018. 3
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In ACM SIGKDD, 2006. 3
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 3
Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, and Nasser M. Nasrabadi. Supermix: Super- vising the mixing data augmentation. In CVPR, 2021. 3, 9, 1, 2, 6,
Jacob Devlin, Ming-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In CVPR, 2019. 3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929, 2020. 2,7
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. PODNet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, 2020. 3
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The paseal visual object classes (voc) challenge. IJCV, 2010.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015. 8
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. NeurlPS, 2020. 3,
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regular- ization. In AAAI, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016. 2,
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 3
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0255: 2015. 3
Minui Hong. Jinwoo Choi, and Gunhee Kim. Stylemix: Separating content and style for enhanced data augmentation. In CVPR, 2021. 1,3, 2,
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In ICML,, 2020. 1, 2, 7, 3,5
Jang-Hye Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. In ICI.R 2021. 1, 2, 7, 3,
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 7
Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to-class measure for few-shot learning. In CVPR, 2019.
Z. Li and D. Hoiem. Learning without forgetting, IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-29477 Dee 2018. 3
Zhaowen Li, Zhiyang Chen, Fan Yang. Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, al. MST: Masked self suff--uueeeised transformer for visual representation. In NeurlPS, 2021. 3
Yann Lifchitz, Yannis Avrithis, Sylvaine Picard, and Andrei Bursuc. Dense classification and im- planting for few-shot learning. In CVPR, 2019. 3
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV, 2016. 8
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, 2018.
Leland Mclnnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 2018. 4
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for seman- tic segmentation. In ICCV, 2015.
Pedro O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron Courville. Un- supervised learning of dense visual representations. NeurlPS, 2020.
Ilija Radosavovic, Piotr Dollar, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. In CVPR, 2018.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In CVPR, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. In ICLR, 2014. 3
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. I/CV, 2015. 7
Zhiqiang Shen, Zhankui He, and Xiangyang Xue. MEAL: Multi-model ensemble via adversarial learning. AAAI, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In NewrIPS, 2017. 2, 3,
A F M Uddin, Mst. Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Ho Bae. SaliencyMix: A saliency guided data augmentation strategy for better regularization. In /CML, 2021. 1, 2, 3, 4,
11

---

Under review as a conference paper at ICLR 2023
VN Vapnik. An overview of statistical learning theory. Neural Networks, IEEE Transactions on, 10 (5):988-95) 999, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. Alignmixup: Im- proving representation by interpolating aligned features. In CVPR, 2022. 3, 6,
4,
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019. 1, 2, 8,
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In ICML, 2020. 4
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-super visual pre-training. In CVPR, 2021.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 4
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020. 3
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Ex- ploring pixel-level consistency for unsupervised visual representation learning. In CVPR, 2021a.
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint 1tdi::2/11.098800 2021b.
Yuwen Xiong, Mengye Ren, Wenyuan Zeng, and Raquel Urtasun. Self-supervised representation learning from flow equivariance. In ICCV, 2021.
Leon Yao and John Miller. Tiny imagenet classification with convolutional neural networks. Tech- nical report, Standford University, 2015.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of scale image dataset using deep learning with humans in the loop. arXiv preprint 506.031506.03365, 2015. 2
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019. 1, 3,
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 201 3
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016b.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lazez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018. 1. 4, 3,5,5
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019. 3
Shaofeng Zhang, Meng Liu, Junchi Yan, Hengrui Zhang, Lingxiao Huang. Pinyan Lu, and Xiaokang Yang. m-mix: Generating hard negatives via multiple samples mixing for contrastive learning. In ACM SIGKDD, 2022. 3
12

---

Under review as a conference paper at ICLR 2023
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016. 6,5
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image bert pre-training with online tokenizer. In ICLR, 2022.
Jianchao Zhu, Liangliang Shi, Junchi Yan, and Hongyuan Zha. Automix: Mixup networks for sample interpolation via cooperative barycenter learning. In ECCV, 2020. 3
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A MORE EXPERIMENTS
A. MORE ON SETUP
Settings and hyperparameters We train MultiMix and its variants with mixed examples only- We use a mini-batch of size m = 128 examples in all experiments. For every mini-batch, we apply MultiMix with probability 0.5 or input mixup otherwise. For input mixup, we interpolate the standard v pairs (5), For MultiMix, we use the entire network as the encoder fo by default, except for the last fully-connected layer, which we use as classifier gw. We use n 1000 tuples and draw different ~ U|0.5, 2.0] for each example from the Dirichlet distribution by default. For multi-GPU experiments, all training hyperparameters including m and n are per GPU.
For dense MultiMix, the spatial resolution is 4 x 4 (r = 16) on CIFAR-1O/100 and 7 x 7 = 49) on Imagenet by default. We obtain the attention map by (10) using GAP for vector u and ReLU followed by & normalization as non-linearity h by default. To predict class probabilities and compute the loss densely, we use the classifier gw as x convolution by default; when interpolating at earlier layers, we follow the process described in subsection 3.4. For distillation, both the teacher and student networks have the same architecture. By default, we use y = in (9), that is, equal contribution of original labels and teacher predictions.
CIFAR-10/100 training Following the experimental settings of AlignMixup (Venkataramanan et al., 2022), we train MultiMix and its variants using SGD for 2000 epochs using the same random seed as AlignMixup. We set the initial learning rate to 0.1 and decay it by factor of 0.1 every 500 epochs. The momentum is set to 0.9 and the weight decay to 0.0001. We use a batch size m = 128 and train on single NVIDIA RTX 2080 TI GPU for 10 hours.
ImageetteNet training Following the experimental settings of PuzzleMix (Kim al., 2020), we train MultiMix and its variants using SGD for 1200 epochs, using the same random seed as AlignMixup. We set the initial learning rate to 0.1 and decay it by factor of 0.1 after 600 and 900 epochs. The momentum is set to 0.9 and the weight decay to 0.0001. We train on two NVIDIA RTX 2080 TI GPUs for 18 hours.
ImageNet training Following the experimental settings of PuzzleMix (Kim et al., 2020), we train MultiMix and its variants using the same random seed as AlignMixup. We train R-50 using SGD with momentum 0.9 and weight decay 0.0001 and V using AdamW with default parameters. The initial learning rate is set to 0.1 and 0.01, respectively. We decay the learning rate by 0.1 at 100 and 200 epochs. We train on 32 NVIDIA V100 GPUs for 20 hours.
Tasks and metrics We use top-1 error (%, lower is better) top-1 accuracy (%, higher is better) as evaluation metric on image classification and robustness to adversarial attacks (subsection 4.2 and subsection A.2). Additional datasets and metrics are reported separately for transfer learning to object detection (subsection 4.3) and out-of-distributian detection (subsection A.3).
A.2 MORE RESULTS: CLASSIFICATION AND ROBUSTNESS
Using the experimental settings of subsection A.1, we extend Table 1, Table and Table 3 of subsec- tion 4.2 by comparing MultiMix and its variants with additional mixup methods in Table 6 and Ta- ble 7. The additional methods are Input mixup (Zhang et al., 2018), Cutmix (Yun et al., 2019), SaliencyMix (Uddin et al., 2021), StyleMix (Hong et al., 2021), StyleCutMix (Hong et al., 2021), SuperMix (Dabouei et al., 2021) and (-Mixup (Abhishek et al., 2022). We reproduce (-Mixup and SuperMix using these settings. For Supermix, we use the official code which first trains the teacher network using clean examples and then the student using mixed. For fair comparison, we use the same network the teacher and student models.

---

Under review as a conference paper at ICLR 2023
In Table 6 and Table 7, observe that MultiMix and its variants outperform all the additional mixup methods on image classification. Furthermore, they are more robust to FGSM and PGD attacks as compared to these additional methods. The remaining observations in subsection 4.2 are still valid.
DATASET
CIFAR-10
CIFAR-100
NETWORK
R-18
W16-8
R-18
W16-8
R-
Buseline!
=5.41
94.93=00.6
76.69:00.26
78.80:00.55
$6.49::0.21
Input mixop (Zhang et al., 2018y?
95.98
96.18.10.00
0.040
80.16 80.16:10.1
56.60.10.16
CuiMix (Yun 2019y?
96.79-4000
96.48440.02
80.56
56.87-4-.299
Manifold mixup (Verma
2019y?
97.00.10.05
96.44.10.2
00.00.0..44
80.77::0:26
59.31-10.49
PuzzleMix (Kim ot al,, 2020y
97.04:10.04
97.00
79.98 0.0.5
80.78 1:0.23
63,52.10.48
Co-Mixup (Kim al,, 2021)T
97.10::0.03
90.04
80.2 80.28 00
80.39 19:0000.
64, (4:10::::44
SaliencyMix (Uddin et at., 2021)*
96.94:00.05
96.27 0.005
80.36 0055
80.29 +:000
66.14-40.51
StyleMix (Hong et ol., 2021)*
96.25:00:4
96.27 0.004
80.01 0:0.7
79.77
1:.27
StyleCutMix (Hong et al., 2021)
96.94:10.05
96.95 0.004
80.67 0.0.7
1:004
66.55 66.55:100.3 1:013
SuperMix (Dabouci
1)
96.03-10.05
96. 13.005
79.07 00.077:0066
79 0:000
64.43:10.39
AlignMixup (Venkataramanan ct 2022)*
97. 97.06:10.01
1:001
81.71 0.007
81 81.24 0:000
66.85 06005
Ç (-Mixup (Abhishek et ol., 2022)*
96.26:20.04
96.35:20044
80.46
79.73:20.15
63. 18:20.001
MultMMx (ours)
97.07
97.06:
0:00
81.44::0008
67.11 +7:014
distil
97.12:20.02
00:111
82.06
68.06-10..
dense
97.0 19.09 0:0002
97 97.09.10.02
81 :1.00:10.04
81.77:10.03
68. 68.441:0.05
dense distil
97.16 0.00
97.20 0.00
82.35 0.133
82.32 0.000
69.11 1.005
Gain
+0.06
+0.20
+0.64
+1.08
+2.26
Table 5: Image classification on CIFAR-10/100 and TI (Tiny Top-1 accuracy (%): higher better. R: PreActResnet, W: WRN. *: reproduced, i: reported by AlignMixup, *: reproduced with same teacher and student model. Bold black: best; Blue: second best; underline: best baseline. Gain: improvement over best baseline.
RESNET-50 SPEED ERROR
VIT-S/16
METHOD
SPEED
ERROR
Raseline 1
76.32
1.01
Inpot mixup (Zhong ol., 2018)
1.14
77.42
0.99
74.7
CalMix (Yun et al., 2019)*
1,16
78.60
0.99
74.4
Manifold mixup (Verma et al., 2019)"
1.15
77.50
0.97
75.2
PuzzleMix (Kim et al,, 2020)*
0.84
78.76
0.73
75.7
Co-Mixup (Kim ot al,, 2021)?
0.62
0.57
75.9
SalicacyMix (Uddin al., 2021y?
1.14
78.74
0.96
75.8
StyleMix (Hong et al., 2021yT
0.99
75.94
0.85
StyleCutMix (Hong al., 202
0.76
77.29
0.71
75.1
SuperMix (Dahouei et alll,,222222 2021)*
0,92
77 77.60
AlignMixup (oappnVeeenn (Venkataramanan et al., 2022y!
1,03
79.32
MultiMix (ours)
1.16
78.81
1.0
75.2
$0.12
0.93
dense
0.95
79.32
0.88
76.1
dense + distil
0.83
80.21
0.81
76.9
Gain
+0.89
+1.0
Table 6: Image classification and training speed on ImageNet. Top-1 accuracy (%); higher is better. Speed: images/sec (x10²): higher is better. reproduced with same teacher and student model, i: reported by AlignMixup. Bold black: best; Blue: second best; underline: best baseline. Gain: improvement over best baseline.
A.3 MORE RESULTS: OUT OF DISTRIBUTION DETECTION
This is standard benchmark for evaluating over-confidence. Here, in-distribution (ID) examples on which the network has been trained, and out-of-distribute (OOD) are examples drawn from any other distribution. Given mixture of ID and OOD examples, the network should predict ID example with high confidence and an OOD example with low confidence, i.e., the confidence of the predicted class should be below certain threshold.
Following AlignMixup (Venkataramanan et al., 2022), we compare MultiMix and its variants with SoTA methods trained using R-18 CIFAR-100 ID examples, while using LSUN (Yu et al.,
2

---

Under review as a conference paper at ICLR 2023
FGSM
DATASET
CIFAR-10 10
CIFAR-
CIFAR- 10
CIFAR- 100
W16-8
W16-8
W16-8
W16-8
Baseline
88.8::0.11
88.31:0.33
87.2.10.10
72.6::0222
91.9:10.06
99:0::0
99.9110.01
99.91:0:01
99.9:10.01
Input mixap (hhng 2019y?
79.14-0:07
79.1
81.4::0.23
67.3--0.06
08.7
99.74-0.02
99.4--0.01
999--0001
99.0--0.22
CaMix (Yin et all, 20199*
77.:1:0.06
78.3::0:05
96.9:10006
60.2:100:1
58.6:10.03
99.81:0.03
98. 1:0000
98.6 98.61:0.01
97.0:10.1
Manifeld mixup (Verma
201997
89,3
97.2--0.01
98.4--0.00
99.6--0001
PazzieMix (Kim al, 20200
57.4::0.22
60.7
0:.::00009
57.8+:0.33
03.8
97.7
97.0+-0.11
9644:0002
95. 0:000
Co Mixup (Kim et al. 2021)1
60.11:0.05
58.8 18.81:00.00
01.5::0.02
56.5 0:00
97.51:0.02
96.1110.00
95.31:0:00
91.2100011
SaliencyMix (1ddin
202177
5741:006
68.01:005
16
$1.1:20.06
974::0000
97 07:04:0044
95 0::000
91.7::0008
StyleMix (long et al,, 2021)?
80.01:0:23
7121:021
$0.61:0.15
68.2:10.17
85.1:10.16
98.11:0.09
97.51:0.07
98.3 3:0009
98.3:100099
StyleCurMix (Hong al,, 2022yy
80.5
96.74-0002
SaperMix (Dahoncci et al,, 2021))
60.0:::111
582:2012
56.3:0.19
81.120.12
97442:002
97:2:::000
91 01:40:000
92.7:00011
AlignMixpp (Yenkataramaan
2022y1
74.1+-004
55.0--0.03
78.0+00.3
95.34-00044
96.7--0000
904-00001
92.1+-0003
(-Mixup (Abbishok et al., 2022y*
728
67.3
75.322021
26
54.7.2018 18
98.020000
98 08.6:2:0.03
97.420110
96.1200 10
MulaMia (ouns)
$4.11:0:09
55.11:004
75:8:10:1
54.0100011
07.510001
94.2:10.04
91.0110001
90:0::0:01
91:61000
+ distiliaionn
02.5::00:5
$1410001
03:5:00:0
52.7:1002
76.2:1005
9261:001
91.9:10.00
88 88.8::0001
90::0000
dense
$4.1 1:001
53.31:0:00
74.5.10.03
52.9:00:1
15.5
929::0004
92:01:0001
88.61:000
90.8:00001
0.00
52.1:10.02
75.1::0.01
90.5::0.03
87.5100001
90.1::0.03
Gaia
+2.8
+5.9
+4.5
+2.0
Table 7: Robustness to FGSM & PGD attacks. Top-1 error (%): lower is better. reproduced, f reported by AlignMixup. ": reproduced, same teacher and student model. Bold black: best; Blue: second best; underline: best baseline. Gain: reduction of error over best baseline. TI: Tinylmagenet. R: PreActResnet, W: WRN.
DIISSTIDDITIIIIIIIONN DETECTION
DATASET
LSUN (CROP)
ISUN
(CROP)
DET AUROC
AUPR
AUPR DET AUROC AUPR
DET AUROC (OOD)
AUPR
AUPR
AUPR
METRIC
(ID) (OOD)
(ID)
(ID)
(OOD)
Baseline'
54.0
47.1
54.5
45.6
66.5
72.3
74.5
69.2
61.2
64.8
67.8
60.6
Inpat mixup (Zhing ol., 2018)
57.5
59.3
61.4
55.2
59.6
63.0
60.2
63.4
58.7
62.8
63.0
62.1
Catmix (Yun et all. 2019)*
63.8
63.1
61.9
63.4
67.0
76.3
81.0
77.7
70.4
84.3
87.1
80.6
Manifold mixup (Verna et al,, 20199
58.9
60.3
59.5
73.1
80.7
76.0
67.4
69.9
69.3
70,5
PuzzleMix (Kim
2020)*
64.3
73.7
71.1
78.2
81.9
Co-Mixup (Kim et al,, 2021yt
70.4
75.6
82.3
70.3
68.6
80.1
82.5
75.4
71.5
84.8
86.1
80.5
SaliencyMix (Uddin et al,, 2021>?
68.5
79.7
82.2
64.4
65.6
76.9
78.3
79.8
73.3
83.7
87.0
82.0
StyleMix (long et al., 2021)?
62.3
64.2
70.9
63.9
61.6
68.4
67.6
60.3
67.8
73.9
71.5
78.4
StyleCutMix (Hong et al., 2021)
70.8
78.6
83.7
74.9
70.6
82.4
83.7
76.5
75.3
82.6
82.9
78.4
SuperMix (Dabouei al., 2021)
78.6
AlignMixup (Venkataramanan
2022y1
74.2
79.9
84.1
75.1
72.8
83.2
84.1
80.3
77.2
85.0
87.8
85.0
(-Mixup (Abbishek et al., 2022)*
68.1
73.2
80.8
73.1
72.2
82.3
82.2
79.4
74.4
84.3
82.2
77.2
MultiMix (ours)
79.2
82.6
85.2
77.6
75.6
85.1
87.8
83.1
78.3
86.6
89.0
88.2
+ distillation
80.3
54.4
86.3
76.4
79.0
85.6
88.2
84.9
50.7
87.8
89.9
88.2
+ dense
80.8
84.3
85,9
78.0
76.8
85,4
88.0
84.6
81.4
89.0
90.8
88.0
+ dense + distillation
81.0
84.9
86.4
78.2
79.2
86.0
88.5
84.8
81.9
89.3
90.3
88.3
Gain
+6.8
+5.0
+2.3
+3.1
+5.3
+2.8
+4.4
+4.6
+4.7
+4.3
+3.0
+3.3
Table 8: Out-of-distribution detection using R-18. Det Acc (detection accuracy), AuROC, AuPR (ID) and AuPR (OOD): higher is better. ": reproduced, t; reported by Align.Mixup. ": reproduced, same teacher and student model. Bold black: best; Blue: second best; underline: best baseline. Gain: increase in performance. TI: Tiny Emnylmaeeet..
3

---

Under review as a conference paper at ICLR 2023
(a) Baseline
(b) Manifold mixup (Verma et al, 2019)
(c) SaliencyMix (Uddin al, 2021)
(d) AlignMixup (Venkataramanan al,, 2022)
(e) Dense MultiMix +
distillation (ours)
Figure 4:
Embedding space visualizution for 100 test examples per class of randomly chosen classes of CIFAR-100 with PreActResnet-18, using UMAP (Mclnnes et al., 2018).
2015), iSUN (Xiao et al., 2010) and TI to draw OOD examples. We use detection accuracy, Area under ROC curve (AuROC) and Area under precision-recall curve (AuPR) evaluation metrics. In Table 8, we observe that MultiMix and its variants outperform SoTA on all datasets and metrics by large margin. Although the gain of vanilla MultiMix and Dense MultiMix over SoTA mixup methods is small on image classification, these variants significantly reduce over-confident incorrect predictions and achieve superior performance out-of-distribution detection.
A.4 ANALYSIS OF THE EMBEDDING SPACE
Qualitative analysis We qualitatively analyze the embedding space on 10 CIFAR-100 classes in Figure 4. We observe that the quality of embeddings of the baseline is extremely poor with severely overlapping classes, which explains its poor performance on image classification. All mixup meth- ods result in clearly better clustered and more uniformly spread classes. Manifold mixup (Verma et al., 2019) produces five tightly clustered classes but the other five are still severely overlapping. SaliencyMix (Uddin et al., 2021) and AlignMixup (Venkataramanan al., 2022) yield four some- what clustered classes and 6 moderately overlapping ones. Our best setting, i.e., dense MultiMix with distillation, results in five tightly clustered classes and another five somewhat overlapping but less than all competitors. More plots including variants of MultiMix are given in the supplementary material.
Quantitative analysis We also quantitatively assess the embedding space on the CIFAR-100 test set using aligmment and uniformity (Wang & Isola, 2020). Alignment measures the expected pair- wise distance of examples in the same class. Lower alignment indicates that the classes are more tightly clustered. Uniformity measures the (log of the) expected pairwise similarity of all examples using a Gaussian kernel as a similarity function. Lower uniformity indicates that classes are more uniformly spread in the embedding space. On CIFAR-100, we obtain alignment 3.02 for baseline, 1.27 for Manifold Mixup (Verma et al., 2019), 2.44 for SaliencyMix (Uddin et al., 2021), 2.04 for AlignMixup and 0.92 for Dense MultiMix with distillation. We also obtain uniformity -1.94 for the baseline, -2.38 for Manifold Mixup (Verma et al., 2019), -2.82 for SaliencyMix (Uddin et al., 2021), -4.77 for AlignMixup (Venkataramanan al., 2022) and -5.68 for dense MultiMix with distillation. These results validate the qualitative analysis of Figure 4.
A.5 MORE ABLATIONS
As in subsection 4.4, all ablations here are performed using R-18 CIFAR-100.
Mixup methods with distillation In subsection 4.2 and Table 6, we observe that distillation sig- nificantly improves the performance when used with MultiMix. Here, we also study its effect when applied to SoTA mixup methods.
Given a mini-batch m examples with inputs X and targets Y, we obtain the augmented views V and V discussed in subsection 3.3. We then follow the mixup strategy of each mixup method and obtain the corresponding predicted class probabilities P. P' from the student and teacher classifier, respectively. E.g., for manifold mixup (Verma et al., 2019), we interpolate the embeddings Z = fo(V), 2' = (V') using (3) and obtain = gw (Z) and P' gw In each case, we obtain the interpolated targets Y using (4) and train the student network using (9).
4

---

Under review as a conference paper at ICLR 2023
METHOD
VANILLA
DISTIL
DENSE
DENSE + DISTIL
Baseline
76.76
78.28
78.16
79.07
Ingui mixup (Zhung et al., 2018)
79.79
80.19
80.21
80,54
CuMMxx (Yin al, 2019)
Manifold mixup (Verma ct at, 2019)
80.20
81.32
80.87
81.47
PuzzlcMix (Kim ot al., 2020)
79.99
81.26
80.62
81.44
Co-Mixup (Kim et at.,, 2021)
80.19
81.39
80.84
81.69
SaliencyMix (Uddin al,, 2021)
2021)
80.31
81.57
81.21
81.73
StyleMix (Hong al, 2021)
79.96
81.22
80.76
81.30
StyleCutMix (Hong 2021)
80.66
81.60
81.41
81.75
SuperMix (Dahonei al, 2021)"
79.01
AlignMixap (Venkataramanan al., 2022)
81.71
81.80
81.36
81.40
MultiMix (nurs)?
MultiMx (ours)
81.81
82.28
81.88
82.52
Table 9: Image classification CIFAR- 100 using R-18: The effect of distillation, dense loss and
both on SoTA mixup methods. Top-1 accuracy (%). higher is better. ": 'vanilla' refers to teacher pre-training and 'distil' to self-distillation where teacher and student trained concurrently from scratch. 4 Instead of dense MultiMix, we only apply the loss densely.
Uniform
81.33
81.59
CAM
softmax
81.21
81.45
81.63
Atlention (10)
suftmax 81.78
GAP
f relu
81.88
82.52
Table 10: Variants of spatial attention in dense MultiMix, with and without distillation, CIFAR- 100 using R-18. Top-I accuracy (%); higher is better. GAP: Global Average Pooling; CAM: Class Activation Maps (Zhou al., 2016); & relu: ReLU followed by & normalization.
In Table 9, we observe that with distillation, the performance of all SoTA mixup methods improve. For example, the baseline improves by 1.52% accuracy (76.76 78.23) and manifold mixup by 1.12% (80.20 ~> 81.32). average, we observe gain of % brought by distillation. An exception AlignMixup (Venkataramanan et al., 2022): distillation brings a marginal improvement of 0.09% (81.71 + 81.80), making it on-par with vanilla MultiMix.
Mixup methods with dense loss Table 6 we observe that dense interpolation and dense loss improve vanilla MultiMix. Here, we study the effect of the dense loss when applied to SoTA mixup methods.
Given mini-batch of examples, follow the mixup strategy of the SoTA mixup methods to obtain the mixed embedding Z ddxm for each spatial position discussed in subsection 3.4, we obtain the predicted class probabilities pi E gexm again for each j I,...,r. r. Finally, we compute the cross-entropy loss HY, Pi) (1) densely at each spatial position j. where the interpolated target label Y E Rexm is given by (4).
1,...,r. Then, as
In Table observe that using dense loss improves the performance of all SoTA mixup methods.
The baseline improves by 1.4% accuracy (76.76 + 78. 16) and manifold mixup by 0.67% (80.20 + 80.87). On average, we observe a gain of 0.7% brought by the dense loss. An exception is AlignMixup (Venkataramanan et al., 2022), which drops by 0.35% (81.71 + 81.36). This may be due to the alignment process, whereby the interpolated dense embeddings are very far from the original.
Finally, we study the effect of using a dense distillation loss on SoTA mixup methods. Here, simi- larly with (9), the loss has two terms for each spatial position j: the first is the dense cross-entropy above and the second is the dense distillation loss H ((P') where Pr is ob-
loss H HY,, ( tained by the teacher. Table 9, we observe that dense distillation further improves the performance of SoTA mixup methods compared to using the dense loss only.
5

---

Under review as a conference paper at ICLR 2023
Two-stage distillation Following SuperMix Dabouei et al. (2021), also study the effect of using a two-stage distillation process with MultiMix, rather than online self-distillation.
In the first stage, train the teacher using only clean examples for 300 epochs, and achieve top-1 accuracy of 75.62%. This is slightly lower than the 76.76% of the baseline from Table 9, which trained for 2000 epochs. In the second stage, fix the teacher parameters and train the student using the predictions from the teacher network as targets. In particular, we use the second term H(P', P) of (9), that is, y 0. At inference, the t accuracy drops by 16% (75.62 + 59.77). This shows that using the setting of SuperMix is not effective, while also being computationally expensive because of the two-stage training.
We also study the effect of training the student with both the interpolated labels Y(7) and the in- terpolated predictions P' of the pretrained teacher targets. In particular, use (9) with default 5. At inference, the top-1 accuracy improves by 4.7% compared with the teacher (75.62 80.35). However, the student accuracy of 80.35% is still inferior our 82.28% by online self-distillation (Table 9). This shows that joint training of teacher and student is beneficial.
1,800
1,600
images/ez
1,400
1,200
Baseline
MuliiMix
1,000
MuliMix distill
Donse MuluMix
800
Dense MuliiMxx + distill
10'
10a
10a
10a
10a
tuples
Figure 5: Training speed (images/sec) of MultiMix and its variants vs. number of tuples n on CIFAR- 100 using R-18. Measured on NVIDIA RTX 2080 TI GPU, including forward and backward pass.
Training speed In Figure 5, we analyze the training speed of MultiMix and its variants a func- tion of number tuples n. In terms of speed, vanilla MultiMix is on par with the baseline up to accuracy gain of 5%. The best performing variant- dense Multi-
= 1000, while bringing Mix with distillation -s only slower by 15.6% at n = 1000 as compared to the baseline, which arguably worth given the impressive 5.8% accuracy gain. Further increasing beyond n 1000 brings a drop in training speed, due to computing A and then using it to interpolate (6),7). Because 1000 default
1000 also brings little performance benefit according to Figure 3(b), we for all MultiMix variants.
Dense MultiMix: Spatial attention In subsection 3.4, we discuss different options for attention in dense MultiMix. In partieular, attention amounts to defining uniform 1,/r. Otherwise, a is defined by (10). The vector u can be defined as u = // by global average pooling (GAP) of z, which is the default, or Wy assuming linear classifier with W E Rx The latter similar to class activation mapping (CAM) Zhou et al. (2016), but here the current value of W is used online while training. The non-linearity h can be softmax or ReLU followed by l normalization (e, relu), which is the default. Here, we study the affect of these options on the performance of dense Multimix.
In Table 10, we observe that using GAP for and ε o relu as h yields the best performance overall. Changing GAP to CAM or & o relu to softmax is inferior, more so in the presence distillation. The combination of CAM with softmax is the weakest, even weaker than uniform attention. CAM may fail because using the non-optimal value of W while training; softmax may fail because of being too selective. Compared to our best result, uniform attention is clearly inferior, by nearly 1% in the presence distillation. This validates that the use of spatial attention in dense MultiMix is clearly beneficial. The intuition the same as in weakly supervised tasks: In the absence of dense

---

Under review as a conference paper at ICLR 2023
targets, assuming the same target of the entire example at every spatial position naively implies that the object of interest is present everywhere, whereas spatial attention provides a better hint as to where the object may really be.
Dense MultiMix: Spatial resolution We study the effect of spatial resolution dense MultiMix. By default, we use a resolution of x 4 at the last residual block of R-18 on CIFAR-100. Here, we additionally investigate 1 (downsampling by average pooling with kernel size 4, same GAP), 2 x 2 (downsampling by average pooling with kernel size 2) and x 8 (upsampling by using stride 1 in the last residual block). We measure accuracy 81.07% for spatial resolution 1 81.43% for for 2 x 2. 81.88% for x 4 and 80.83% for 8 x 8. We thus observe that performance improves with spatial resolution up to x 4, which the optimal, and then drops to assuming the same target at each spatial position. The resolution 8 x 8 is also more expensive computationally.
8. This drop may be due


Reviews:
[{"review_id": "otUYVs0gSB0", "reviewer": "Reviewer_Vnzf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes MultiMix which is a variant of the Mixup data augmentation technique. While Mixup creates augmented data by only considering interpolation between two data points, MultiMix considers the interpolation between $m$ data samples. In greater detail, MultiMix creates $n$ new augmented samples from $m$ data samples by linear mixing them with random weight vectors that are drawn from a Dirichlet distribution.  Moreover, the authors extend MultiMix to Dense-Multimix by removing spatial pooling and applying the loss function densely over all tokens/patches in some special deep-learning architectures. In the case that has label information, the author uses online self-distillation for creating soft labels for augmented data. On the application side, the authors conduct experiments in image classification, robustness to adversarial attacks, object detection, and out-of-distribution detection. The proposed techniques improve the classification accuracies on CIFAR10, CiFAR100, and TI better than baselines in both clean and adversarial attacked data. The proposed techniques also yield better accuracy in objective detection on VOC07+12 (SSD model) and MS-COCO (FASTER R-CNN model). ", "summary_of_the_review": "The paper misses some important baselines that also consider $m$ samples for creating new augmented data as mentioned in the weakness part. Moreover, the trade-off between performance and computation is not well-studied. A figure is needed with the x-axis as computational time and the y-axis as the performance. ", "strength_and_weaknesses": "## Strength\n\n* The paper is well-organized and easy to follow.\n* The experiments are conducted on various datasets and applications. The proposed techniques improve the performance slightly.\n* Multimix is more robust to the adversarial attack than the previous baselines. \n* The dense Multimix idea is interesting to perform mixup techniques in deep learning.\n\n## Weaknesses\n\n* The idea of using more samples to do mixing is not new. The authors should include those variants in the comparison. The authors have mentioned $m$-Mixup. The other variant that the paper should include is $k$-MIXUP [1] which also considers $m$ samples as MultiMix, however, $k$-mixup finds the optimal mapping between two mini-batches of size $m$ via optimal transport.\n\n[1] k-Mixup Regularization for Deep Learning via Optimal Transport, Kristjan Greenewald, et al\n\n* MultMix needs to sacrifice the computation time for the performance. However, I see that the improvement in performance is not very significant while the computational time is only reported in the image classification application. The authors should report the computational time for all other applications for completeness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ikv-0y6DiO", "reviewer": "Reviewer_rQBR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an extension of the mixup data augmentation method where all examples in a batch are mixed with coefficients sampled from Dirichlet distribution, and the number of mixup examples can be as many as one wishes with more Dirichlet samples. On top of this, the method further adds ensemble distillation and dense augmentation and losses, which is shown to further improve the accuracy. \n\nThe paper argues that mixup on the top layer before logit is the best and also most efficient.\n", "summary_of_the_review": "This paper proposed an extension of the mixup data augmentation method combined with some other techniques such as dense augmentation/loss and self-distillation with ensemble prediction. Though it demonstrates slightly better accuracy, it’s unclear how the different parts (mixup, dense augmentation, self-distillation) interact with each other. \n\nquestions\n* in experiments, half of the time it’s using put augmentation, the other half MultiMix, unclear how this hyperparameter affect performance. Do we must have input mixup? Why?\n* What prevents us from applying dense augmentation and self-distillation to other mixup baselines?\n* for manifold mixup, did you also apply mixup in the last layer?\n* previous papers show mixup more examples doesn't help, which is contrary to what this paper argues. But there is no ablation study to still mixup two examples, but mixup more batches. How do we know the gain is from mixing up more examples or simply because we have more tuples? \n* high-level question: penultimate layer is already linear representation (i.e., input of logistic regression), and mixup is intend to regularize by enforcing linearity of the representation, in this case, why would it still help to apply mixup? In another work, would it help to apply mixup for logistic regression? (or softmax for multi-lass)", "strength_and_weaknesses": "strengths\n* proposed an extension of mixup with better accuracy and higher efficient \n\nWeakness\n* the contribution seems a bit incremental \n* it’s unclear how the dense mixup and self-distillation are integral part of the proposed method, can we apply the same techniques to the baseline methods?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7Y76CpaTbS", "reviewer": "Reviewer_BWYW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a suite of Mix-based augmentation techniques that improve the performance of neural networks.", "summary_of_the_review": "Overall, I think the proposed augmentation techniques are empirically sound, which expands the toolbox of augmentations for training neural networks. However, none of these techniques has been theoretically justified; therefore, it is hard to judge whether these techniques are applicable in a given (new) scenario.", "strength_and_weaknesses": "Strength:\n1. The notations in the paper are rigorous.\n2. The writing of the paper is easy to follow.\n3. The proposed techniques are empirically solid.\n\nWeakness:\n1. None of the proposed techniques is theoretically justified. In other words, it is hard to judge in advance whether an approach helps before actually trying it out. Therefore, describing some scenarios where these techniques FAIL may be helpful.\n2. It looks to me that the paper is an ensemble of tricks for training neural networks --- It does not solve any problem.\n3. The scale of the experiments is limited --- the largest dataset is Tiny ImageNet. While this scale may be sufficient for a theoretical paper, it is less convincing for an empirical paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject