Embedding Space Interpolation Beyond Mini-Batch, Beyond Pairs and Beyond Examples
Authors: Shashanka Venkataramanan
Ewa Kijak
laurent amsaleg
Yannis Avrithis
Venue: NeurIPS 2023 poster (2023)
Keywords: Interpolation based data augmentation
mixup
dense interpolation
robustness
representation learning

Abstract: Mixup refers to interpolation-based data augmentation, originally motivated as a way to go beyond empirical risk minimization (ERM). Its extensions mostly focus on the definition of interpolation and the space (input or feature) where it takes place, while the augmentation process itself is less studied. In most methods, the number of generated examples is limited to the mini-batch size and the number of examples being interpolated is limited to two (pairs), in the input space.

We make progress in this direction by introducing MultiMix, which generates an arbitrarily large number of interpolated examples beyond the mini-batch size and interpolates the entire mini-batch in the embedding space. Effectively, we sample on the entire convex hull of the mini-batch rather than along linear segments between pairs of examples.

On sequence data, we further extend to Dense MultiMix. We densely interpolate features and target labels at each spatial location and also apply the loss densely. To mitigate the lack of dense labels, we inherit labels from examples and weight interpolation factors by attention as a measure of confidence.

Overall, we increase the number of loss terms per mini-batch by orders of magnitude at little additional cost. This is only possible because of interpolating in the embedding space. We empirically show that our solutions yield significant improvement over state-of-the-art mixup methods on four different benchmarks, despite interpolation being only linear. By analyzing the embedding space, we show that the classes are more tightly clustered and uniformly spread over the embedding space, thereby explaining the improved behavior.

Embedding Space Interpolation Beyond Mini-Batch, Beyond Pairs and Beyond Examples
Shashanka Venkataramanan'
Ewa Kijak!
Laurent Amsaleg²
Yannis Avrithis
Inria, Univ Rennes, CNRS, IRISA Institute of Advanced Research Artificial Intelligence (IARAI)
Abstract
Mixup refers to interpolation-based data augmentation, originally motivated as way to go beyond empirical risk mininization (ERM). Its extensions mostly focus the definition of interpolation and the space (input embedding) where it takes place, while the augmentation process itself is less studied. In most methods, the number of generated examples is limited the mini-batch size and the number of examples being interpolated is limited to two (pairs), in the input space.
We make progress in this direction by introducing MultiMix, which generates an arbitrarily large number of interpolated examples beyond the mini-batch size, and interpolates the entire mini-batch in the embedding space. Effectively, we sam- ple on the entire convex hull of the mini-batch rather than along linear segments between pairs of examples.
On sequence data we further extend to Dense MultiMix. We densely interpolate features and target labels at each spatial location and also apply the loss densely. To mitigate the lack of dense labels, inherit labels from examples and weight interpolation factors by attention as a measure confidence.
Overall, we increase the number of loss terms per mini-batch by orders of mag- nitude little additional cost. This is only possible because of interpolating the embedding space. We empirically show that our solutions yield significant improvement over state-of-the-art mixup methods on four different benchmarks, despite interpolation being only linear. By analyzing the embedding space, we show that the classes are more tightly clustered and uniformly spread over the embedding space, thereby explaining the improved behavior.
1 Introduction
Mixup [65] a data augmentation method that interpolates between pairs training examples, thus regularizing neural network to favor linear behavior in-between examples. Besides improving gen- cralization, it has important properties such as reducing overconfident predictions and increasing the robustness to adversarial examples. Several follow-up works have studied interpolation in the latent or embedding space, which is equivalent to interpolating along a manifold in the input space [51], and number of nonlinear and attention- ever, little progress has been made in the augmentation process itself, i.e., the number n of generated examples and the number of examples being interpolated.
interpolation mechanisms [62, 26, 25, 46, 7]. How-
Mixup was originally motivated as a way to go beyond empirical risk minimization (ERM) [47] through vicinal distribution expressed equivalent to the set of linear segments between all pairs of training inputs and targets. In practice however, in every training iteration, a single scalar À drawn and the number of interpolated pairs is limited to the size b of the mini-batch (n = b). as illustrated in Figure ((a). This is because
expectation
an interpolation factor X, which is
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

METHOD
SPACE
TERMS
MIXED
FACT
DISTR
Mixup [65]
input
b
2
Beta
Manifold mixup [51]
embeddiing
b
Betu
(-Mixup [I]
input
25
RandPerm
SuperMix [11]
input b
Dirichlet
MultiMix (owrs)
embeddiigg embedding
6
Dirichlet
Dense MultiMix (ours)
Dirichlet
Table 1: Interpolation method properties. SPACE: Space where interpolation takes place; TERMS:
number of loss terms per mini-batche MIXED: maximum number m of examples being interpolated; FACT: number of interpolation factors per mini-batcho DISTR: distribution used to sample in- terpolation factors; RandPerm: random permutations of fixed discrete probability distribution. b: mini-batch size; n: number of generated examples per mini-batch; r: spatial resolution.
interpolation takes place in the input space, it would be expensive to increase the number of pairs per iteration. To our knowledge, these limitations exist in all mixup methods.
In this work, we argue that a data aug- mentation process should increase the data seen by the model, or at least by its last few layers, as much as possi- ble. In this sense, we follow manifold mixup [51] and generalize it in num- ber of ways to introduce MultiMix.
dkes
First, we increase the number n of gen- erated examples beyond the mini-batch size b, by orders of magnitude (n >> b). This possible by interpolating deepest layer, i.e., just before the clas- sifier, which happens to be the most ef- fective choice. To our knowledge, we are the first to investigate n > b.
(a) mixup
(b) MultiMix (ours)
Figure 1: Data augmentation in a mini-batch B of b 10
the
points in two dimensions. (a) mixup: sampling n = b points on linear segments between b pairs of points using
the same interpolation factor A. (b) MultiMix: sampling
n = 300 points in the convex hull of B.
Second, we increase the number m of examples being interpolated from m 2 (pairs) to m = (a single tuple containing the entire mini-batch). Effectively, instead of linear segments between pairs of examples in the mini-batch, we sample on their entire convex hull as illustrated in Figure (bb). This idea has been investigated in the input space: the original mixup method [65] found non- effective, while [11] found it effective only up to m = 3 examples and [1] went up to m = 25 but with very sparse interpolation factors. our knowledge, we are the first to investigate m > the b.
embedding space and to show that it is effective up to
Third, instead of using a single scalar value of À per mini-batch, we draw a different vector À R for each interpolated example. A single À works for standard mixup because the main source of randomness the choice of pairs (or small tuples) out of b examples. In our case, because we use single tuple of size m 6, the only source of randomness being A.
We also argue that, what matters more than the number of (interpolated) examples is the total number of loss terms per mini-batch. common way to increase the number of loss terms per example is by dense operations when working sequence data, e.g. patches in images voxels in video. This is common in dense tasks like segmentation [41] and less common in classification [27]. We are the first investigate this idea in mixup, introducing Dense MultiMix.
In particular, this is an extension of MultiMix where we work with feature tensors of spatial resolu- tion r and densely interpolate features and targets at each spatial location, generating interpolated features per example and nr > n per mini-batch. We also apply the loss densely. This increases the number of loss terms further by factor r, typically one or two orders of magnitude, compared with MultiMix. Of course, for this to work, we also need a target label per feature, which we inherit from the corresponding example. This is weak form of supervision [70]. To carefully select the most representative features per object, we use an attention map representing our confidence in the target label per spatial location. The interpolation vectors À are then weighted by attention.
Table summarizes the properties of our solutions against existing interpolation methods. Overall, make the following contributions:
2

---

1. We generate an arbitrary large number of interpolated examples beyond the mini- bathh size, each by interpolating the entire mini-batch the embedding space, with one interpo- lation vector per example. (subsection 3.2).
2. We extend to attention- attention-weighted dense interpolation in the embedding space, further in- creasing the number of loss terms per example (subsection 3.3).
3. We improve over state-of-the-ar (SoTA) mixup methods on image classification, robust- ness to adversarial attacks, object detection and out-of-distribution detection. Our solu- tions have little or no additional cost while interpolation is only linear (section 4).
4. Analysis of the embedding space shows that our solutions yield classes that are lightly clustered and uniformly spread over the embedding space (section 4).
2 Related Work
Mixup: interpolation methods In general, mixup interpolates between pairs of input exam- ples [65] or embeddings [51] and their corresponding target labels. Several follow-up methods mix input images according to spatial position, either at random rectangles [62] or based on at- tention [46, 26, 25], in an attempt to focus on different object in each image. Other follow-up method [35] randomly cuts image at patch level and obtains its corresponding mixed label using content-based attention. We also attention in our dense MultiMix variant, but in the embedding space. Other definitions of interpolation include the combination of content and style from two im- ages [23], generating out-of-manifold samples using adaptive masks [37], generating binary masks by thresholding random low-frequency images [18] and spatial alignment of dense features [48]. Our dense MultiMix also uses dense features but without aligning them and can interpolate large number of samples, thereby increasing the number of loss-terms per example. Our work is or- thogonal to these methods we focus on the sampling process of augmentation rather than on the definition of interpolation. We refer the reader to [30] for a comprehensive study of mixup methods.
Mixup: number of examples to interpolate To the best of our knowledge, the only methods that interpolate more than two examples for image classification are OpfTransMix [71], SuperMix [11], (-Mixup [1] and DAML [45]. All these methods operate in the input space and limit the number of generated examples to the mini-batch size: whereas MultiMix generates an arbitrary number of interpolated examples (more than 1000 in practice) in the embedding space. To determine the inter- polation weights, OptTransMix uses a complex optimization process and only applies to images with clean background; ç-Mixup uses random permutations of a fixed vector; SuperMix uses Dirichlet distribution over not more than 3 examples in practice; and DAML uses a Dirichlet distribution to interpolate across different domains. We also use Dirichlet distribution but over many examples as the mini-batch size, Similar to MultiMix, [6] uses different À within a single batch. Beyond classification, m-Mix [66] uses graph neural networks in self-supervised setting with pair-based loss functions. The interpolation weights are deterministic and based on pairwise similarities.
Increasing the number of loss terms more than b per mini-batch is not very common in classification. In deep metric learning [40, 28, 53], it is common to have loss term for each pair of examples in a mini-batch in the embedding space, giving rise to b(b - 1)/2 loss terms per mini-batch, even without mixup [49]. Contrastive loss functions are also common in self- supervised learning [8, 5] but have been used for supervised image classification too [24]. Ensemble methods [55, 14, 19, 3] increase the number of loss terms, but with the proportional increase of the cost by operating in the input space. To our knowledge, MultiMix is the first method to increase the number of loss terms per mini-batch to n >> for mixup. Dense MultiMix further increases this number to nr. By operating in the embedding space, their computational overhead is minimal.
Number of loss terms
Dense loss functions Although standard in dense tasks like semantic segmentation [41,20], where dense targets commonly exist, dense loss functions are less common otherwise. Few examples are in weakfy-supervisee segmentation [70, 21. few-shot learning [33, 311, where data augmentation is of utter importance, attribution methods [27] and unsupervised representation learning, e.g. dense contrastive learning [42, 54), learning from spatial correspondences [59, 57] and masked language or image modeling [12, 58, 32, 69]. To our knowledge, we are the first to use dense interpolation and dense loss function for mixup.
3

---

3 Method
3.1 Preliminaries and background
Problem formulation Let E A' be an input example and E y its one-hot encoded target, where X RDD is the input space, Y' {0, 1] and is the total number of classes. Let Se ''' TR be an encoder that maps the input to an embedding 2 fe(z), where d is the dimension of the embedding. A classifier gw : R - A maps 2 to a vector = gw (z) of predicted probabilities over classes, where A " C R the unit n-simplex, i.e., p 2 and 1 p 1, and 1, E Ie an all-ones vector. The overall network mapping is == gw fe. Parameters (0, W) are learned by optimizing over mini-batches.
e-1
Given mini-batch of b examples, let X = (Z).... mb) E Dxb be the inputs, Y = . yb) E Rexb the targets and P (Pi,.... ......Ph.. PP)) E Rexb the predicted probabilities of the mini-batch, where P [(X) (J(z)) .... [ (xx)). The objective is to minimize the cross-entropy
H(Y. P) - 1! (Y à log( 1))
(1)
of predicted probabilities /' relative to targets averaged over the mini-batch, where is the Hadamard (element-wise) product. In summary, the mini-batch loss is
L(X,Y;6,N)
HY, 9w (fo(X))).
The total number of loss terms per mini-bater
Mixup Mixup methods commonly interpolate pairs of inputs or embeddings and the correspond- ing targets at the mini-batch level while training. Given mini-batch of examples with inputs X and targets Y. let Z (Z1...... . 26) E Rdd be the embeddings of the mini-batch, where = (X)...
Manifold mixup [51] interpolates the embeddings and targets by forming a convex combination of
pairs with interpolation factor E [0, 1]:
7 = ( + (1- (1-A)11)
(3)
Y = Y + (1- A)II,,
(4)
where ~ Beta a). I is the identity matrix and 11 E bbbb is permutation matrix. Input mixup [65] interpolates inputs rather than embeddings:
x X(I (1 X)II).
(S)
Whatever the interpolation method and the space where it is performed, the interpolated data, e.g. X [65] or Z [51], replaces the original mini-batch data and gives rise to predicted probabilities P P (pi....,, E Rexb over classes, egg. P S(X) [65] or P = =_gw (7) [51]. Then, the average cross-entropy H(Y, P) (1) between the predicted probabilities P and interpolated targets Y is minimized. The number of generated examples per mini-batch
= b, same as the original mini-batch size, and each is obtained by interpolating m = 2 examples. The total number of loss terms per mini-batch is again b.
3.2 MultiMix
Interpolation The number of generated examples per mini-batch is now n >> and the number of examples being interpolated is b. Given mini-batch of b examples with embeddings Z and targets Y, we draw interpolation vectors A ~ Dir(a) for k = I,... n, where Dir(a) is the symmetric Dirichlet distribution and d E A mm-1, that is, A 2 and A = 1. We then interpolate embeddings and targets by taking n convex combinations over all examples:
Ž = ZA
Y = YA,
(7)
where A (Ai...., An) E Rdxn We thus generalize manifold mixup [51]:
1. from b to an arbitrary number n > b of generated examples: interpolated embeddings
Z ædxn (6) vs. ddx in (3), targets Y E ex (7) vs. gexb in (4);

---

21
encoder fa
norm
2 (two examples), (one interpolated embedding), r = (spatial resolution x 3). The embeddings Z E Rdx9 of input images #1,22 extracted by encoder fo- Attention maps 41, 02 IR.h are extracted (9), multiplied element-wise with interpolation vectors X, (1 X) E R (10) and éj-normalized per spatial position (II). The resulting weights are used to form the interpolated embedding E IRdx9 as convex combination of 21, Z2 per spatial position (12). Targets are interpolated similarly (13),
Figure 2: Dense MultiMix (subsection 3.3) for the special
2. from pairs (m 2) to tuple of length b. containing the entire mini- m-term convex combination (6),() vs. 2-term in (3),(4) Dirichlet vs. Beta distribution;
3. from fixed À across the mini-batch different X for each generated example.
Loss Again, replace the original mini-bated embeddings Z by the interpolated embeddings Z and minimize the average cross-entropy H(Y, P)) (1) between the predicted probabilities P gw (Z) and the interpolated targets Y (7). Compared with (2), the mini-batch loss becomes
LM (X, Y; W) := H(Y A, gw (fo()))))
(8)
The total number of loss terms per mini-batch now >>
3.3 Dense MultiMix
We now extend to the case where the embeddings are structured, e.g. in tensors. This happens e.g. with token vs. sentence embeddings in NLP and patch vs. image embeddings in vision. works by removing spatial pooling and applying the loss function densely over all tokens/patches. The idea illustrated Figure 2. For the sake of exposition, our formulation uses sets of matrices grouped either by example by spatial position. In practice, all operations
tensors.
The encoder is now fo X + Rdxr, mapping the input fo(x) E Rdx,, where is the number of channels and r its spatial resolution there are more than one spatial dimensions, these are flattened.
Preliminaries
embedding
(aI,...,, xb) E Dxb and targets Y (y1,.... ........38... 3/b) Rexb Each embedding z; fo(x,) (2] ........ 2f) Rdxr for i 1...., consists of features 2 E Rd for spatial position j I....,r. We group features by position in 2",
Given a mini-batch of examples, we have again inputs
matrices 21, 22....,2"* where Z = (=)....,,f)) E Rdx for I....,r.
Attention In the absence of dense targets, each spatial location inherits the target of the corre- sponding input example. This is weak supervision, because the target object is visible every- where. To select the most reliable locations, we define a level of confidence according to an attention map. Given an embedding z E ddxx with target y E y and vector Rdd. the attention map
= h(z u) E IR"
measures the similarity of features of to u, where a non-linearity, e.g. softmax or ReLU followed by & normalization. There are different ways to define vector u. For example, u z1,/r by global average pooling (GAP) of z, similar to class activation mapping (CAM) [68]. In case no attention, 1,/r is uniform.
Wy assuming a linear classifier with W E ddxe.
Given a mini-batch, let a; (a:...,, ,....,,, a.) E R" be the attention map of embedding z; (9) for i I,...,b. b. We group attention by position in vectors a ..... a", where a' (af...., .)) B&
5

---

for = =o.j=,.,,,, Figure 6 in the supplementary shows the attention obtained by (9). We observe high confidence on the entire or part of the object. Where confidence is low, we assume the object is not visible and thus the corresponding interpolation factor should be low.
Interpolation
There are again > b generated examples per mini-batch, with m b examples being densely interpolated. For each spatial position j = 1....,77 V, we draw interpolation vectors X ~ Dir(a) for = I,.... n and define A' = (A....., X2) E Rmxx. Since input examples are assumed to contribute according to the attention vector a² E I we scale the rows of A accordingly and normalize its columns back to A A to define convex combinations:
M diag(a) (oog((a))> )A
(10)
M M diag(1,,,
(11)
We then interpolate embeddings and targets by taking n convex combinations over m examples:
Z Z 2'1
(12)
y Y
(13)
This is similar to (6).(7, but there is a different interpolated embedding matrix Z E Bgdx well as target matrix YJ Rxxx per position, even though the original target matrix Y one. The total number of interpolated features and targets per mini-batch is now nr.
The classifier is now gw Rdxr - Rexr, maintaining the same spatial resolution as the embedding and generating one vector of predicted probabilities per spatial position. This done by removing average pooling or any down-sampling operation. The interpolated embeddings Z' ..... Z" (12) are grouped by example into ž1 ..... Zn E Raxr, mapped by gw to predicted probabiliiiiss PI..... Pr. E _2ex and grouped again by position into pi ....... pr Bex.
Classifier
In the simple case where the original classifier is linear, i.e. W E gdxe, is seen as 1x 1 convolution and applied densely to each column (feature) of 2 for j = I....,T.
Loss Finally, we learn parameters 8, W by minimizing the weighted cross-entropy H(Ys, Pl;s) of pi relative to the interpolated targets Y again densely at each position j. where
H(,, P: s) == - (V @ log( (1)T)
(14)
generalizes (1) and the weight vector defined as s 1 M E TR.. This is exactly the vector used to normalize the columns of M in (11). The motivation is that the columns of M are the original interpolation vectors weighted by attention: A small & norm indicates that for the given position we are sampling from examples of low attention, hence the loss is to be discounted. The total number of loss terms per mini-batch is now nr.
4 Experiments
4.1 Setup
We use mini-hatch of size b 128 examples in all experiments. Following manifold mixup [51], for every mini-batch, we apply MultiMix with probability 0.5 or input mixup otherwise. For Mul- tiMix, the default settings are given in subsection 4.6. We use PreActResnet-188 (R-18) [21] and WRN16-8 [63] as encoder on CIFAR-10 and CIFAR-100 datasets [29]; R-18 on TinyImagenet [60] (TI); and Resnet-50 (R-50) and VIT-S/16 [13] on ImageNet [44]. To better understand the effect of mixup in ViT, we evaluate MultiMix and Dense MultiMix on ImageNet without using strong aug- mentations like Auto- Augment [9], RanmeAugment [10]. random erasing [67] and CutMix [62]. We reproduce TransMix [7] and TokenMix [35] using these settings.
4.2 Results: Image classification and robustness
Image classification In Table 2 we observe that MultiMix and Dense MultiMix already outper- form SoTA on all datasets except CIFAR-IO with R-18, where they are on par with Co-Mixup. all datasets.
Dense MultiMix improves over vanilla MultiMix and its effect is complementary On TI for example, Dense MultiMix improves over MultiMix by 1.33% and SoTA by 1.59%. We provide additional analysis of the embedding space 10 classes of CIFAR- 100 in subsection 4.4.

---

DATASET
CIFAR-10
CIFAR-100
T1
NETWORK
R-18
W16-8
W16-8
Baseline!
95 2:002
94.93=00066
06.69=0.26
78,80 0..55
56,49::0:21
Manifold mixup [51]'
97.00
9..44:20.02
80.00:10.34
80.77:10.66
:9:31::0:49
PuzzleMix [26]
97.04=00:44
97.00=0005
79.98:20.05
80.78:0023
63.52:00.2
Co-Mixup 125)
97.10
96.44 0.008
80.28 00.13
80.39 0.334
64.12.::00
AlignMixup [48]]
97.06620.04
96.91::0000
81.01
81.24:10.02
66.85:10.07
(-Mixup [1]'
96.26.1:0.44
96.35:20.04
80,46.20..6
79.73
63.18:10.14
MultiMix (ours)
97.07::0:03
97. 07.0620002
81.82:20.04
81.44:20.03
67.11:1004
Donse MultiMix (ours)
07.09::0.00
97.09.10002
81.93.20000
81.77:10.03
68.44::0.0
Gain
-0.01
+0.09
+0.22
+0.53
+1.59
Table 2: Image classification on CIFAR-10/100 and TI (Tinylmagenet), Mean and standard devia- tion of Top-1 accuracy (%) for runs. R: PreActResnet, W: WRN. *: reproduced, I; reported by AlignMixup, Bold black: best; Blue: second best; underline: best baseline. Gain: improvement over best baseline. Additional comparisons are in subsection A.2.
ATTACK
FGSM
PGD
DATASET
CIFAR-10
CIFAR-100
TI R-18
CIFAR- 10
CIFAR-100
NETWORK
R-18
W16-8
R-18
W16-8
R-18
W16-3
R-18
W16-8
Baseline!
88.8:20.11
88.3
87.2=0010
72.6::022
91.9:1006
99.9-20.0
99.9-1001
99. 99.9::01
99.9::001
Manifold mixup 1511*
76.94-0.14
76.0
80.2-0.06
56.34+0.10
89.34-0066
97.2
98.4440.33
99.6:10001
98.4+0000
PuzzleMix [26]*
57.4+-0.2
60.7:0002
78.8=0.99
57.8
83.8::005
97.7
97.0=0.01
96.44o002
95.2-:008
Co-Mixup (25)*
60.1::0.05
58.810.10
0.05=0:02
$6.5:10004
97.5
96.1 1:.00
95.3:10.03
94.2.1001
AlignMixup [48]*
54.8:10.03
56.0=0.05
7..1=0.04
55.0:10.03
08
95.3:20.04
96.7:2003
92.1 90.4::0001
(-Mixup [1]'
72.8
67,3
75.3 0.22-
68.0-+0.1
84.7=+0.18
98.0==0.06
98.6
97.4-+-.10
96.1
MultiMix (ours)
14.0
0..0
$4.0110.01
77.5
1:001
94.2
94.8
1:001
90.0:10.01
91.6.:000
Dense MuliiMix (ours)
54.1 0.001
53.3 0.003
73.5
0.03
52.9
75.5 0.004
92.9
0.004
92.6
0.00
88.6-
90.8
90.00
Gain
+0.7
+2.7
+0.6
+2.1
+3.3
+2,4
+3.5
+1.4
+1.3
Table Robustness to FGSM & PGD attacks. Mean and standard deviation of Top-1 error (%) for 5 runs: lower is better. ": reproduced, 1: reported by AlignMixup. Bold black: best; Blue: second best; underline: best baseline. Gain: reduction of error over best baseline. TI: Tiny Imagenet. R: PreActResnet, W: WRN. Comparison with additional baselines is given in subsection A.2.
NETWORK MISHODD
RESNET-SO
VIT-S/16
In Table we observe that on ImageNet with R- 50, vanilla MultiMix outperforms all methods ex- cept AlignMixup. Dense MultiMix outperforms all SoTA with both R-50 and ViT-S/16, bringing an overall gain of 3% over the baseline with R-50 and 2.2% with VIT-S/16. The gain over AlignMixup with R-50 is small, but it is impressive that it comes with only linear interpolation. To better isolate the effect of each method, we reproduce TransMix [7] and ToxenMix [35] with ViT-S/16 using their offi- cial code with our settings, i.e., without strong reg- ularizers like CutMix, Auto-Augment, Random- Augment etc. MultiMix is on par, while Dense MultiMix outperforms them by 1%.
SPEED
AcC
SPEED
Acc
Baseline 1
1.17
76.32
1.01
Manifold mixup [51]
1.15
77.50
0.97
PuzzleMix [26]
0.84
78.76
0.73
74.7
Co-Mixup [25]]
0.62
0.57
74.9
Transiix [7]"
1.01
75.1
TokenMix [35]*
0.87
75.3
AlignMixup [48]]
1.03
79.32
MultMMx (ours)
1.16
78.81
0.98
75.2
Dense MultiMix (ours)
0.95
79.42
0.88
76.1
Gain
+0.1
+1.2
Table 3: Image classification and training
speed on ImageNet. Top-1 accuracy (%); higher is better. Speed: images/sec (x 1x10)): higher is better. i reported by AlignMixup: *: reproduced. Bold black: best; Blue: second best; underline: best baseline. Gain: improve- ment over best baseline. Comparison with ad- ditional baselines is given in subsection A.2.
Training speed Table also shows the training speed as measured on NVIDIA V-100 GPU in- cluding forward and backward pass. The vanilla MultiMix has nearly the same speed with the base- line, bringing an accuracy gain of 2.49% with R- 50. Dense MultiMix is slightly slower, increasing the gain to 3.10%. The inference speed is the same for all methods.
Robustness to adversarial attacks We follow the experimental settings of AlignMixup [48] and use 8/255 Loo e-ball for FGSM [16] and 4/255 lo e-ball with step size 2/255 for PGD [38] attack.
7

---

Baseline
AlignMixup
MuliMix
MultiMix (ours)
Figure 3:
Embedding space visualization for 100 test examples per class of randomly chosen classes of CIFAR-100 with PreActResnet-Is using UMAP [39].
In Table we observe that MultiMix is already more robust than SoTA on all datasets and settings. Dense MultiMix also increases the robustness and is complementary.
The overall gain is impressive than in classification according Table For example, against the strong PGD attack on CIFAR-10 with W16-8, the SoTA Co-Mixup improves the baseline by 3.8% while Dense MultiMix improves it by 7.3%, which double. MultiMix and Dense MultiMix outperform Co-Mixup and PuzzleMix by 3-6% in robustness on CIFAR- 10, even though they are on- par on classification. There also a significant gain over SoTA AlignMixup by 1-3% in robustness to FGSM on TinylmageNet and to the stronger PGD.
4.3 Results: Transfer learning to object detection
We evaluate the effect of mixup on the gener- alization ability of pre-trained network ob- ject detection as a downstream task. Following the settings of CutMix [62], we pre-train R-50 on ImageNet with mixup methods and use it as the backbone for SSD [36] with fine-tuning on
DATASET
VOCO7+12
MS-COCO FR-CNN SPEED
DETECTOR
SSD SPEED
Baseline
76.7
9.7
33.27
23.6
Input mixup'
CutMix!
23,2
AlignMixup'
78.4
35.84
20.4
Pascal VOC07+12 12 [15] and Faster-RChn RCNN [43]
MuliiMix (ours)
23.2
with fine-tuning on MS-COCO [34].
Dense MultiMix (ours)
79.2
8.8
36.19
19.8
In Table 5, we observe that, while MultiMix slightly worse than AlignMixup on Pascal V0C07+12, Dense MultiMix brings improve- ments over the SoTA both datasets and is still complementary. This consistent with classification results. Dense MultiMix brings a gain of 0.8 mAP on Pascal VOC07+12 and 0.35 mAP on MS-COCO.
Gain
+0.8
+0.35
Table 5: Transfer learning object detection. Mean average precision (mAP, %): higher is bet- ter. I; reported by AlignMixup. Bold black: best; Blue: second best; underline: best baseline. Gain: increase in mAP. Speed: images/sec: higher is better.
4.4 Analysis of the embedding space
Qualitative analysis We qualitatively analyze the embedding space 10 CIFAR-100 classes in Figure 3. We observe that the quality of embeddings of the baseline is extremely poor with severely overlapping classes, which explains its poor performance on image classification. All mixup meth- ods result in clearly better clustered and more uniformly spread classes. AlignMixup [48] yields five somewhat clustered classes and five moderately overlapping ones. Our best setting, i.e., Dense MultiMix, results in five tightly clustered classes and another five somewhat overlapping but less than all competitors.
Quantitative analysis We also quantitatively assess the embedding space on the CIFAR-100 test set using alignment and uniformity [52]. Aligument measures the expected pairwise distance of examples in the same class. Lower alignment indicates that the classes are more tightly clustered. Uniformity measures the (log of the) expected pairwise similarity of all examples using a Gaussian kernel as a similarity function. Lower uniformity indicates that classes are more uniformly spread in the embedding space.

---

MultiMix
Dense MaltiMix
Default e Standard
82
Accuracy
81
81.5
81.5
80
81
80.5
79
80
80.5
80
0.1
0,2
0,3
0,4
10
10a
10a
10a
10
2
25
50
100
128
0.5
1.5
(a) Mixing layers
(b) # mixed examples
(c) # examples interpolated m
(d) Dirichlet parameter o
Figure 4: Ablation study on CIFAR-100 using R-18. (a) Interpolation layers (R-18 block; 0: input mixup). (b) Number n interpolated examples per mini-batch with m = b (Default) and m 2
(Standard). (c) Number of examples being interpolated, with 1000 (Default) and 100 (Standard). (d) Fixed value of Dirichlet parameter a.
On CIFAR-10M 100, we obtain alignment 3.02 for baseline, 2.04 for AlignMixup. 1.27 for MultiMix and 0.92 for Dense MultiMix. We also obtain uniformity -1.94 for the baseline, -2.38 for Align- Mixup [48], -4.77 for MultiMix and -5.68 for Dense MultiMix. These results validate the qualitative analysis of Figure 3.
4.5 Manifold intrusion analysis
Manifold intrusion [17] can occur when mixed examples are close classes other than the ones being interpolated in the embedding space. To evaluate for manifold intrusion, we define the intrusion distance (ID) as the minimum distance of mixed embedding to the clean embed- dings of all classes except the ones being interpolated, averaged over a mini-batch: ID (2, Z)
[ E3E min _ - =11 Here, is the set of mixed embeddings in a mini-batch and Z is the set of clean embeddings from all classes other than the ones being interpolated in the mini-batch. In- tuitively, a larger ID Z, Z) denotes that mixed embeddings in are farther away from the manifold of other classes in Z, thereby preventing manifold intrusion.
Averaged over the training set of CIFAR-100 100 using Resnet-18, the intrusion distance 0.46 for Input Mixup. 0.47 for Manifold Mixup. 0.45 for AlignMixup. 0.46 for MultiMix and 0.47 for Dense MultiMix. This is roughly the same for most SoTA mixup methods. This may be due to the fact that true data occupy only tiny fraction of the embedding space, thus generated mixed examples lie in empty space between class-specific manifolds with high probability. The visualization in Figure indeed shows that the embedding space is sparsely populated, even in two dimensions. This sparsity expected to grow exponentially the number dimensions, which in the order of 103.
4.6 Ablations
All ablations are performed using R-18 on CIFAR- 100. We study the effect of the layer where we interpolate, the number of generated examples per mini-batch, the number m of examples being interpolated and the Dirichlet parameter a. More ablations are given in the supplementary.
Interpolation layer For MultiMix, we use the entire network as the encoder fo by default, except for the last fully-connected layer, which we use as classifier gw- Thus, we interpolate embeddings in the deepest layer by default. Here, we study the effect of different decompositions of the network f= =g gw fe, such that interpolation takes place at a different layer. In Figure 4(a), we observe that mixing at the deeper layers of the network significantly improves performance. The same behavior observed with Dense MultiMix, which validates our default choice.
is interesting that the authors of input mixup [65] found that convex combinations of three or more examples in the input space with weights from the Dirichlet distribution do not bring further gain. This agrees with the finding of SuperMix [11] for four or more examples. Figure 4(a) suggests that further gain emerges when mixing in deeper layers.
Number n of generated examples per mini-batch This is important since our aim to increase the amount of data seen by the model, or at least part of the model. We observe from Figure 4(b) that accuracy increases overall with n and saturates for n 1000 for both variants of MultiMix. The improvement is more pronnounced when m = 2, which is standard for most mixup methods.

---

Our best solution, Dense MultiMix. works best at = 1000 and = 10, 000. We choose = 1000 as default, given also that the training cost increases with n. The training speed as a function of is given in the supplementary and is nearly constant for S 1000.
Number of examples being interpolated We vary m between 2 (pairs) and b = 128 (entire mini-batch) by using A' Rmxx drawn from Dirichlet along with combinations (subsets) over the mini-batch to obtain A E Raxa with m nonzero elements per column in (6),(7). We observe in Figure 4(c) that for both MultiMix and Dense MultiMix the performance increases with m. The improvement is more pronnounced when 100, which is similar to the standard setting (n b 128) of most mixup methods. Our choice of m = b = 128 brings an improvement of 1-1.8% over m 2. We use this as our default setting.
Dirichlet parameter interpolation vector (column of A). Here we study the effect of a fixed value of a. In Figure 4(d), we observe that the best accuracy comes with a 1 for most MultiMix variants, corresponding the uniform distribution over the convex hull of the mini-batch embeddings. However, all measurements lower than the default [00.5 2]. For example, from Table 2(a) (CIFAR-100, R-18), Dense MultiMix has accuracy 81.93, compared with 81 81.59 in Figure 4(d) for a =
Our default setting is to draw a uniformly random from [0.5, 2] for every
5
Discussion
The take-home message of this work is that, instead of devising smarter and more complex inter- polation functions in the input space intermediate features, it is more beneficial to use MultiMix, even though its interpolation is only linear. In this work, we combine three elements:
1. Increase the number n of generated mixed examples beyond the
size b.
2. Increase the number m of examples being interpolated from m. = 2 (pairs) to m = b.
3. Perform interpolation in the embedding space rather than the input space.
Figure 4 shows that all three elements are important in achieving SoTA performance and removing any one leads to sub-optimal results. We discuss their significance and interdependence here.
Increasing the number n of generated examples The expected risk is defined as an integral over the underlying continuous data distribution. Since that distribution is unknown, the integral is ap- proximated by finite sum, Le., the empirical risk. better approximation is the vicinal risk, where number of augmented examples is sampled from distribution in the vicinity of each training ex- ample, thus increasing the number of loss lerms per training example. Input mixup [65] is inspired by the vicinal risk. However, as a practical implementation, all mixup methods still generate b mixed examples and thus incur loss terms for a mini-batch of size b. As discussed in section 2, previous works have used more loss terms than b per mini-batch. but not for mixup.
Our hypothesis for the significance of element is that more mixed examples, thus more loss terms by interpolation, provide better approximation of the expected risk integral. Dense interpolation further increases the number of loss terms, thus further improving the quality of approximation.
Increasing the number m of examples being interpolated works, starting from input mixup [65] have attempted to interpolate > examples in the input space by sampling from Dirichlet or other distributions but have found the idea not effective for large m. Our finding is that element 2 becomes effective only by interpolating in the embedding space, that is, element 3.
discussed in section 2, previous
Interpolating in embedding space Element 3 is originally motivated by Manifold Mixup [51], where "interpolations in deeper hidden layers capture higher level information [64]." ACAI [4] explicitly studies interpolation in the latent space of an autoencoder to produce a smooth semantic warping effect in data space. This suggests that nearby points in the latent space are semantically similar, which in turn improves representation learning. Mixed examples generated by sampling in the embedding space lie on the learned manifold. We hypothesize that the learned manifold is good surrogate of the true, unknown data manifold.
A natural extension of this work is to settings other than supervised classification. A limitation is that is not straightforward to combine the sampling scheme of MultiMix with complex interpolation methods, unless they are fast to compute in the embedding space.
10

---

Transformers for image 6 Acknowledgements
This work was in part supported by the ANR-19-CE23-0 MEERQAT project and was performed using the HPC resources from GENCI-IDRIS Grant 2021 ADO1101225
References
[1] Kumar Abhishek, Colin Brown, and Ghassan Hamarneh. Multi-sample <-mixup: Richer, more realistic synthetic samples from a p-series interpolant. arXiv preprint arXiv:2204.03323, 2022. 2, 3, 7. 14, 15, 16, 17
[2] Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance seg- mentation with inter-pixel relations. In CVPR, 2019. 3
[3] James Urquhart Allingham, Florian Wenzel, Zelda E Mariet, Basil Mustafa, Joan Puigcerver, Neil Houlsby, Ghassen Jerfel, Vincent Fortuin, Balaji Lakshminarayanan, Jasper Snoek, et al. Sparse MoEs meet efficient ensembles. arXiv preprint 1rXiv:2/10.03366 2021.
[4] David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow, Understanding and improving interpolation in autoencoders via an adversarial regularizer. In ICLR, 2019. 10
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020. 3
[6] Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regularization. JMLRR, 2022. 3
[7] Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, Alan Yuille, and Song Bai. TransMix:
Attend to mix for vision transformers. In CVPR, 2022. 1, 6, 7, 15
[8] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI,, 2020.
[9] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. AutoAug-
ment: Learning augmentation policies from data, arXiv preprint 1805: 180500000 2018. [10] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Rand Practical
automated data augmentation with reduced search space. In CVPRW, 2020. 6
[11] Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani. and Nasser M. Nasrabadi. SuperMix:
Supervising the mixing data augmentation. In CVPR, 2021. 2, 3, 9. 14, 15, 16. 17, 18
[12] Jacob Devlin, Ming-We Weii Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words:
recognition at scale. In ICLR, 2021. 6
[14] Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with
rank- factors. In ICML, 2020. 3
[15] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisser- man. The pascal visual object classes (voc) challenge. IJCV, 2010. 8
[16] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver-
sarial examples. In ICLR, 2015. 7
[17] Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup locally linear out-of-manifold
regularization. In AAAI, 2019. 9
[18] Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, Adam Prügel-Bennett, and Jonathon Hare. FMix: Enhancing mixed sample data augmentation. arXiv preprint
arXiv:2002.12441 2020. 3
[19] Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for
robust prediction. arXiv preprint arXiv:2010.0661 2020. 3
[20] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV,
2017.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[22] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 1912.02781, 2019. 15, 16, 17

---

[23]
Minui Hong, Jinwoo Choi, and Gunhee Kim. StyleMix: Separating content and style for enhanced data augmentation. In CVPR, 2021. 3. 14, 15, 16, 18
[24] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola,
Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS,
2020. 3
[25] Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song, Co-Mixup: Saliency guided
joint mixup with supermodular diversity. In ICLR, 2021. 1, 3. 7. 15, 16, 17, 18
[26] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle Mix: Exploiting saliency and local
statisties for optimal mixup. In ICML, 2020. 1.3, 7. 14. 15, 16, 17, 18
[27] Jae Myung Kim, Junsuk Choe, Zeynep Akata, and Scong Joon Oh. Keep calm and improve visual feature attribution. In ICCV. 2021. 2,3
[28] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR. 2020. 3
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 6
[30] Siyuan Li, Zedong Wang. Zicheng Liu, Di Wu, and Stan Z. Li. OpenMixup: Open mixup toolbox and benchmark for visual representation learning. arXiv preprint (tti::2200004851,,,
2022.
[31] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to-class measure for few-shot learning. In CVPR, 2019. 3
[32] Zhaowen Li, Zhiyang Chen, Fan Yang. Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. MST: Masked self-supervised transformer for visual representation. In NeurIPS, 2021. 3
[33] Yann Lifchitz, Yannis Avrithis, Sylvaine Picard, and Andrei Bursuc. Dense classification and implanting for few-shot learning, In CVPR, 2019.
[34] Tsung-Y. Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[35] Jihao Liu, Boxiao Liu, Hang Zhou, Hongsheng Li, and Yu Liu. TokenMix: Rethinking image
mixing for data augmentation in vision transformers. In ECCV, 2022. 6, 7, 15
[36] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Theng--Yagg Fu, and Alexander C Berg. SSD: Single shot multibox detector. In ECCV, 2016. 8
[37] Zicheng Liu, Siyuan Li, Di Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, and Stan Li. AutoMix:
Unveiling the power of mixup for stronger classifiers. In ECCV, 2022. 3
[38] Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018. 7
[39] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. UMAP: Uniform mani-
fold approximation and projection. The Journal of Open Source Software, 2018. 8
[40] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In
ECCV, 2020. 3
[41] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation, In ICCV, 2015. 2,
[42] Pedro O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations. In NeurlPS, 2020. 3
[43] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time
object detection with region proposal networks. In NIPS, 2015. 8
[44] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
[45] Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain-augmenter
meta-learning. In CVPR, 2021. 3. 16, 17
[46] A F M Uddin, Mst. Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bae. Salien- cy Mix: A saliency guided data augmentation strategy for better regularization. In ICMI. 2021.
1. 3. 14, 15, 16, 17, 18
[47] VN Vapnik. An overview of statistical learning theory. IEEE Transactions on Neural Networks,
1999.
[48] Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. Align Improving representation by interpolating aligned features. In CVPR, 2022. 3, 7, 8, 9, 14, 15,
16. 17, 18
[49] Shashanka Venkataramanan, Bill Psomas, Ewa Kijak, Laurent Amsaleg, Konstantinos Karantzalos, and Yannis Avrithis. It takes two to tango: Mixup for deep metric learning.
12

---

In ICLR, 2022. 3
[50]
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.
Deep hashing network for unsupervised domain adaptation. In CVPR, 2017. 16. 17
[51] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold Mixup: Better representations by interpolating
hidden states. In ICML. 2019. 1, 2, 3, 4. 6. 7. 10, 14, 15, 16, 17, 18
[52] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through
alignment and uniformity on the hypersphere. In JCML, 2020.
[53] Xun Wang, Xintong Han, Weilin Huang. Dengke Dong, and Matthew R Scott. Multi-similarity
loss with general pair weighting for deep metrie learning. In CVPR, 2019. 3
[54] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learn-
ing for self-supervised visual pre-training. In CVPR, 2021. 3
[55] Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: an alternative approach to efficient
ensemble and lifelong learning, arXiv preprint Skkiv:2002.067155 2020. 3
[56) Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 16
[57] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate your- self: Exploring pixel-level consistency for unsupervised visual representation learning. In
CVPR, 2021. 3
(58) Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu, Simmim: A simple framework for masked image modeling. arXiv preprint
1XXv:2111,098881 2021. 3
[59] Yuwen Xiong. Mengye Ren, Wenyuan Zeng, and Raquel Urtasun. Self-supervised representa- tion learning from flow equivariance. In ICCV, 2021. 3
[60] Leon Yao and John Miller. Tiny imagenet classification with convolutional neural networks.
Technical report, Standford University. 2015.
[61] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 53365,336,, 2015. 16
[62]
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In
ICCV, 2019. 3, 6, 8. 14. 15, 16, 17, 18
[63]
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016. 6
[64]
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
ECCV, 2014. 10
[65] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond
empirical risk minimization. In ICLR, 2018. 1, 2. 3, 4, 9, 10, 14, 15, 16, 17, 18
[66] Shaofeng Zhang, Meng Liu, Junchi Yan, Hengrui Zhang, Lingxiao Huang, Xiaokang Yang,
and Pinyan Lu. M-Mix: Generating hard negatives via multi- sallie mixing for contrastive
learning. In SIGKDD, 2022. 3
[67] Zhun Zhong, Liang Zheng. Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data
augmentation. In AAAI, 2020. 6
[68] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR. 2016. 5, 19
[69] Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong- iBOT: Image bert pre-training with online tokenizer. In ICLR, 2022. 3
[70] Yanzhao Zhou, Yi Zhu, Qixiang Ye. Qiang Qiu, and Jianbin Jiao. Weakly supervised instance
segmentation using class peak response. In CVPR, 2018. 2,3
[71] Jianchao Zhu, Liangliang Shi, Junchi Yan, and Hongyuan Zha. AutoMix: Mixup networks for sample interpolation via cooperative barycenter learning. In ECCV, 2020.
13


Reviews:
[{"review_id": "4oDFDaaCqO", "reviewer": "Reviewer_Xd45", "summary": "The authors introduce MultiMix, an approach aimed at enhancing the Mixup class of techniques, which involve interpolating between input examples and their corresponding targets. This enhancement is achieved through the interpolation of a substantially more training examples, increased number of loss terms for each minibatch, and a more flexible interpolation factors. Further, the authors expand their proposed method with DenseMultiMix, a variation capable of interpolating examples in sequential data. Empirical evidence indicates that the proposed MultiMix and DenseMultiMix not only boost accuracy but also bolster the robustness metric across a diverse set of tasks, including but not limited to classification and object detection.", "questions": "Please see the weakness section.", "limitations": "One limitation as mentioned above is the application of DenseMultiMix in the realm of text data.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "There's a noticeable diminishing return when interpolating additional input examples in Mixup. This presents a limitation to the advancement of Mixup-related methodologies, which focus on interpolating more training examples. However, the approach introduced in this paper appears to overcome this limitation by implementing interpolation in the embedding space, as opposed to the input space, evidenced by Fig 4c. I regard this as a significant contribution of this paper.\n\nFurthermore, the adaptation of Mixup to sequential data like text has always been frustrated. There are three primary obstacles: 1. The blending of text tokens doesn't carry the same significance as mixed images. 2. How to interpolate along the temporal dimension or the spatial dimension. 3. Discrepancies in length among examples within a mini-batch. The introduced MultiMixup provides solutions to at least the first two challenges. Interpolating within the embedding space prevents the creation of nonsensical input tokens, while MultiMixup along with attention delivers an effective means of interpolating representations in the spatial dimension.", "weaknesses": "There are several weaknesses in this manualscript, as explaint in the following.\n\n1. The motivation for the suggested method is insufficiently compelling within the manuscript. The only statement regarding to the motivation is ```In this work, we argue that a data augmentation process should increase the data seen by the model, or at least by its last few layers, as much as possible. ```\n\n    This reasoning seems to be superficial as it fails to distinguish itself from the preceding data augmentation methods. Moreover, it doesn't elaborate on how the proposed approach enhances accuracy and robustness. More explicitly, while it's true that any data augmentation method would increase the volume of data processed by the model, not all such methods that do so effectively improve the model performance. Thus, the motivation fails to pinpoint the core of the research problem.\n\n2. Regarding the empirical evaluation, it would be appropriate for the authors to include comparison with methodologies such as AugMix [1]. Similar to the proposed method, AugMix enhances the volume of data points processed by the model through interpolating examples derived from varying combinations of image pre-processors using the Dirichlet distribution. Moreover, it offers a substantial gain in both accuracy and robustness metrics. Also, the divergence loss could potentially counter the scarcity of dense labels.\n\n     [1]: Hendrycks, Dan et al. “AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty.” ICLR 2020.\n\n3. The performance improvement over the established baseline is rather modest. In such a scenario, it would be beneficial if the manuscript offered more profound insights to the community to better understand Mixup-oriented techniques. Such insights could include how the proposed method is connected to existing methods like AugMix, as well as an exploration of why the proposed methodology shows greater promise. For instance, while Section 4.4 and Figure 6 in the appendix provide useful information, they lack a comparison between the proposed method and existing methods, as well as a comparison among different variants of MultiMix as done in Section 4.5. Empirical insights of this would offer more compelling evidence that the proposed methodology is a valuable contribution to the community.\n\n4. A less significant shortcoming pertains to the scope of the empirical evaluation. The batch size is held to be 128 in the domain of image classification. Understanding the robustness of the proposed method to batch size selection is crucial, especially as we move towards models with more # parameters where batch size tends to decrease. Specifically, when the batch size is small, MultiMix should offer more advantages over standard Mixup due to its ability to draw from a larger pool of interpolated examples. Another dimension worth exploring is the application of DenseMultiMix in the realm of text data. The manuscript would significantly benefit from showing evidence that the proposed method can address some of the challenges of extending Mixup-style techniques to the domain of text. I understand that interpolating dense text labels is a non-trivial task, preliminary studies on tasks such as text classification could suffice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback from R-Xd45. We address the concerns as follows:   \n\n### 1. Motivation\n Please refer to our comment on *Author Rebuttal by Authors*.\n\n### 2. Comparison with AugMix\n\nThanks for sharing this work.  \n **AugMix vs. MultiMix.** AugMix applies a convex combination of more than two examples, similar to MultiMix. The interpolation is between three distinct augmentations applied to an image, followed by a convex combination with the initial clean image. Similar to MultiMix, the interpolation weights are drawn from a Dirichlet distribution. However, AugMix interpolates examples in the *input space*, whereas MultiMix in the *embedding space*; the number of convex combinations is *limited to the batch size* in AugMix and *arbitrarily large* ($n=1000$ in practice) in MultiMix; and each convex combination is *limited to 3 examples* in AugMix and *as large as the mini-batch* in MultiMix.  Crucially, Fig. 4 shows that all 3 elements are needed to improve accuracy: embedding space, number $n$ of generated examples and number $m$ of examples being interpolated.  We shall add AugMix to the discussion of L55-61 and to Table 1. Then, AugMix will be closer to SuperMix [6].\n\n **Experimental evaluation.** We follow the experimental setting of AugMix and evaluate the robustness of MultiMix and Dense MultiMix on CIFAR-100-C using WideResnet and ImageNet-C using Resnet-50. We report the classification error on CIFAR-100-C and mean corruption error on ImageNet-C. We also compare MultiMix and Dense MultiMix with AugMix using our experimental settings (Sec. 4.1) for image classification on CIFAR-100 using Resnet-18 and ImageNet using Resnet-50.  The classification accuracy results are summarized as:\n|Dataset|CIFAR100-C|ImageNet-C|CIFAR100|ImageNet|\n|-|-|-|-|-|\n|Input Mixup|50.4|72.8|79.8|77.4|\n|AugMix|35.9|68.4|80.1|77.7|\n|MultiMix(ours)| 33.8| 67.9| 81.8| 78.8|\n|Dense MultiMix(ours)| 31.7| 67.1| 81.9| 79.4|\n\nOn robustness to corruptions, we observe that both MultiMix and Dense MultiMix outperform AugMix up to 4\\% on CIFAR-100-C, and are also robust to corruptions on ImageNet-C. While AugMix is more robust to corruptions than Input Mixup, it does not bring a significant improvement in accuracy over CIFAR-100 and ImageNet. This limitation of AugMix is also validated in Table 6 (Appendix) of the AugMix paper. By contrast, MultiMix and Dense MultiMix improve classification accuracy and are robust to corruptions. We shall add these results to the final paper.\n\n### 3. Additional empirical evidence  \nWe discuss the differences and compare MultiMix and SoTA mixup methods in Table 1, Sec. 2, Sec. 4.2 and 4.3, as well as Sec. A.2 and A.3. While the performance improvement over SoTA on classification is modest in Tables 2 and 3, it is significant on robustness in Table 4, reaching 3.5\\%. We further show the following achievements of MultiMix and Dense MultiMix:\n\n- **Improving out-of-distribution detection and model calibration.** A standard benchmark for evaluating over-confidence of a model is to measure its ability to detect out-of-distribution examples. In Sec. A.4 and Table 9 of the Appendix, we compare MultiMix and Dense MultiMix against SoTA mixup methods for out-of-distribution detection. We show that although the gain of MultiMix and Dense MultiMix over SoTA mixup methods is small on image classification, they *significantly reduce over-confident incorrect predictions* and achieve superior performance on out-of-distribution detection, with gain ranging from 1.8 to 6.6\\%.\n  Another way to evaluate over-confidence is *model calibration*. We evaluate calibration of the model using MultiMix and Dense MultiMix with Resnet-18 on CIFAR-100. We report *mean calibration error* (mCE) and *overconfidence error* (OE) below:\n|Metric|ECE|OE|\n|-|-|-|\n|Baseline|10.25|1.11|\n|Input Mixup|18.50 |1.42|\n|Manifold Mixup|18.41|0.79|\n|Co-Mixup|5.83|0.55|\n|AlignMixup|5.78|0.41|\n|MultiMix(ours)|5.63|0.39|\n|Dense MultiMix(ours)|5.28|0.27|\n\nBoth MultiMix and Dense MultiMix achieve lower ECE and OE than SoTA methods, indicating that they lead to a better calibrated model.\n\n- **Generalizing to unseen target domains.** We show that MutliMix and Dense MultiMix generalizes to unseen target domains in our response to R-Hzfq -- *Novelty on Dirichlet's distribution and Comparison with DAML*.\n\n- **Improving inter-class separability.** We show that MutliMix and Dense MultiMix improve inter-class separability in our response to R-6vc1 -- *Metrics to improve class separability*.\n\n- **Dense classification loss improving SoTA methods.** In Sec. A.4 (Appendix) under ``Mixup methods with dense loss'', we study the effect of the dense loss on SoTA mixup methods. In Table 10, we show that on average, the dense loss brings a gain of 0.7\\% over all mixup methods.\n\n### 4. Robustness with smaller batch size\nThanks for the suggestion. We compare Input Mixup, Manifold Mixup and MultiMix on CIFAR-100 using ResNet-18, using batch size $b<128$ for image classification. By default, we use $n=1000$ generated examples and $m=b$ examples being interpolated, following the experimental settings of Sec. 4.1. The results are as follows:\n|$m$ |2|25|50|100|\n|-|-|-|-|-|\n| Input Mixup|77.4| 78.3|79.0|79.5|\n| Manifold Mixup|78.6| 79.4|79.9|80.3|\n| MultiMix(ours)|80.9| 81.3|81.6|81.8|\n\nThe accuracy increase of MultiMix with batch size $b$ is similar to the accuracy increase with number $m$ of examples being interpolated (Fig. 4(c)). This in expected as, in the new experiment, $m$ and $b$ are increasing together. We validate the reviewer's hypothesis that the performance improvement of MultiMix over Input or Manifold Mixup is higher for smaller batch size.\n\n### 5. MultiMix for text classification\nThanks for the suggestion. Dense MultiMix is not straightforward to apply to text, since text is represented by sequences of discrete tokens of variable length, which do not have a fixed spatial resolution. Such extension is interesting as future work.\n\n"}, {"review_id": "qf4oF4SnRa", "reviewer": "Reviewer_Hzfq", "summary": "The paper aims to increase the performance and robustness of deep learning model by increasing sample diversity. The authors propose data augmentation techniques which interpolate samples from a convex hull. The experimental evaluations show that the algorithm outperforms existing techniques.\n\n**I have read the rebuttals. Please see the comments below for the discussion.**", "questions": "1. It would be nice to see the analysis on larger/multi-modal models. I am wondering if the gain will be more significant since the sampling diversity will be crucial there. For example, could you conduct experiments on larger Vision Transformer models or language-augmented vision models such as CLIP?\n2. In the ablation study, the performance seems to be sensitive to the hyper-parameters selected, especially since the performance difference from the second best algorithm is between 1-2%. Could you expand more on how the parameters are chosen for each experiment?", "limitations": "The authors discuss the limitation of the algorithm in the concluding paragraph.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Intuitive idea: the use of convex hull is natural and simple.\n2. Applicability: the algorithm can be generally applied in many settings.", "weaknesses": "1. Presentation: the presentation of the paper needs significant improvement. For example, the abstract should succinctly summarizes the key ideas of the paper.\n2. Novelty: sampling from a convex hull is not novel. The use of Dirichlet distribution has also been done in many other works. Could you please compare your work with the work by Shu et al. ?\n\n\n\n\n\n[1] Shu, Y., Cao, Z., Wang, C., Wang, J., & Long, M. (2021). Open domain generalization with domain-augmented meta-learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9624-9633).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback of Reviewer Hzfq. We address the concerns as follows:\n\n### 1. Presentation\nWe would like to point out that the other reviewers appreciate the writing and quality of the presentation. R-Xd45 and R-SS7F rate presentation as *excellent*. R-6vc1 acknowledges that The paper is *well-written* and *easy to understand*,  the formulations in the paper are *quite neat and well-organized*. R-SS7F says *High quality of presentation* of results: the presentation follows a *simple and logical flow*,  methods, notations, \\& results are *clearly stated and easy to understand* for the reader.''\n\nWe thank the reviewer for the suggestion on the abstract. For completeness, we discuss our motivation, the drawback of prior mixup methods, our two contributions--MultiMix and Dense MultiMix--as well as their performance on different benchmarks, all in 22 lines. We kindly ask the reviewer to help us on how can we further make our abstract more succinct without losing critical information. We believe that without the necessary background, the abstract might be difficult to follow. We also kindly ask the reviewer to point out any other examples where the presentation may be improved.\n\n### 2. Novelty of using Dirichlet distribution  \nWe never claim that sampling the interpolation weights from Dirichlet distribution is our contribution (L81-90). In fact, in Table 1 and L111-112, we mention that SuperMix also uses a Dirichlet distribution over not more than 3 examples in practice.  \n\n### 3. Comparison with DAML  \nThanks for sharing the work by Shu *et al.*. We use the official settings of DAML and compare MultiMix and Dense MultiMix on the Office-Home dataset under the open-domain setting using Resnet-18 as the backbone. We report the accuracy in the table below:  \n\n| Domain               | Clipart | Real-World | Product | Art    |\n|----------------------|---------|------------|---------|--------|\n| DAML                 | 45.13   | 65.99      | 61.54   | 53.13  |\n| MultiMix (ours)      | 46.01   | 66.59      | 60.99   | 54.58  |\n| Dense MultiMix (ours)| 46.32   | 66.87      | 62.28   | 56.01  |  \n\nWhile both MultiMix and DAML sample from the convex hull and use the Dirichlet distribution to sample interpolation weights, MultiMix and Dense MultiMix generalize to unseen domains better than DAML, due to sampling an arbitrarily large number of samples. In addition, Dense MultiMix brings significant gain, up to nearly 3\\%.\n\n### 4. Using Larger models  \n\nWe experiment with MultiMix and Dense MultiMix using ViT-B/16 on Imagenet for 100 epochs using our experimental settings (Section 4.1). The baseline, MultiMix and Dense MultiMix achieves a top-1 accuracy of 78.12\\%, 78.84\\% and 79.53\\% respectively. Thus, MultiMix and Dense MultiMix achieve a gain of 0.6\\% and 1.4\\% over the baseline. We shall add comparisons with SoTA mixup methods in the final paper.\n\n### 5. Sensitivity to hyper-parameters\n\nBy observing the plots of Fig. 4(a-d), we find that MultiMix and Dense MultiMix are not sensitive to hyperparameters. The plots are smooth and do not exhibit any local maxima on sharp peaks. Thus, only 4-5 measurements are enough to choose the best value for each hyperparameter. We have observed the same behavior in all our preliminary experiments while developing the method. This indicates the stability and robustness of our methods to the chosen configurations.\n"}, {"review_id": "AUE0lT8QLt", "reviewer": "Reviewer_6vc1", "summary": "This paper considers a generalized form of Manifold mixup by expanding the concept of a pair-wise interpolation method to the multi-sample-based interpolation on the entire convex hull. The proposed methods called MultiMix and Dense MultiMix show the best performance over prior mixup-based methods for the various classification benchmarks. Also, they are empirically shown to have better robustness than prior methods. \n", "questions": "$\\textbf{Question 1:}$ Would you provide a further explanation for the reason why the multi-sample mixup becomes when the interpolation is done in the deeper layer? Again, I felt that the main significance of this work is the finding of the effectiveness of multi-sample-based mixup on deeper layers. As mentioned in this paper, the authors of Mixup and SuperMix had tried to interpolate multiple samples but they are limited in the input space which leads to minimal performance gain. If the novelty becomes more clear and strong after the rebuttal, I will change the rating of the paper.\n\n$\\textbf{Question 2: }$ According to Weakness 3, the metrics used in Section 4.4 seem not to capture the separability between different classes. I think that when the 'Alignment' and 'Uniformity' values are divided into the similarities between different classes, they eventually consider the compactness and separability of the embedding. To be specific, when considering class $k$, we can measure the distance between the per-class averaged feature vector of class $k$ and the per-class averaged feature of the nearest interfering class. When we divide `Alignment' with the distance to the nearest different class, then the value will consider both the similarity between intra-class features and the dissimilarity between inter-class features. What happens when we compute the modified 'Alignment' and 'Uniformity'?\n\n$\\textbf{Question 3: }$ Would you provide a computational cost of the proposed methods and the comparison with other baselines? The results of the training speed guarantee that the proposed method does not suffer from further latency in practice, but I wonder if MultiMix and Dense MultiMix show a similar level of computational cost to the baselines.\n\n$\\textbf{Question 4: }$ Would you provide a discussion of the thought of manifold intrusion of the proposed methods? I imagine that the multi-sample-based mixup tries to cover the entire convex hull so it probably causes manifold intrusion more frequently.", "limitations": "Weakness 1 and Question 1 are for pointing out the limitation of the proposed work in view of novelty.\n\nAlso, I provide a suggestion for improving Related Work in Weakness 2.\n\nAs better metrics for quantitative analysis in section 4.4, I provide the modified measurements in Weakness 3 and Question 2.\n\nFor emphasizing the computational efficiency of MultiMix and Dense MultiMix, providing the computational costs would be beneficial (as aforementioned in Question 3).\n\nFor the manifold intrusion issue, I described the limitation and the corresponding question in Weakness 4 and Question 4.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "$\\textbf{Strength 1:}$ The main strength of this paper is the suggestion of the general form of the prior mixup-based methods to cover the entire convex hull which can be formed by a given set of training samples. Because the power of the mixup methods originated from the interpolation of the space, i.e., either input or embedding, in-between different samples, expanding the concept of mixup into the entire convex hull of the given samples are clear and convincing direction.\n\n$\\textbf{Strength 2:}$ The paper is well-written and easy to understand. To be specific, Fig. 1 is simple yet effective to understand the main strength of the proposed method. Also, the formulations in the paper are quite neat and well-organized.\n\n$\\textbf{Strength 3:}$ The experiments are done for various kinds of architectures ranging from the ResNet family to ViT, and benchmarks including CIFAR and ImageNet cases.\n", "weaknesses": "$\\textbf{Weakness 1:}$ The most crucial weakness of this paper is the novelty of the proposed method. Specifically, I admit that the expansion of the sample interpolation method to the generalized form, i.e., MultiMix, is convincing and the authors show a well-organized formulation of it, but I also feel that the expansion of the concept is quite easy to be anticipated from the prior Mixup methods. For example, in the original Mixup paper, the authors already tried to utilize multiple samples by weighting them with a simplex coefficient in the input space. Although it is reported not to show a further gain, it is hard to find a noticeable difference between the `trial' of the original Mixup paper and the proposed method except for the target space, i.e., input for Mixup vs. embedding space for MultiMix. When focusing on the difference, i.e., mixing on the embedding space, it is natural to ask for the reason why the multi-sample-based mixup becomes effective in the embedding space rather than the input space. However, I cannot find a deep insight into the phenomenon.\n\n$\\textbf{Weakness 2:}$ A more precise comparison in the Related Work part would be beneficial to highlight the significance of the proposed method. To be specific, in Related Work, a bunch of works are grouped into their attributes and briefly mentioned. The authors drop the detailed explanation of the prior works with a suggested survey paper but I am sure that the authors must provide a compact but clear explanation of the related works. As a simple suggestion, mentioning the names of prior mixup methods along with short descriptions would make it more clear. When a set of important baselines including AlignMix, Co-Mixup, PuzzleMix, and $\\zeta$-Mixup are called with names in the Related Work part, readers can easily match the name of the priors appeared in Experiments and the descriptions in Related Work.\n\n$\\textbf{Weakness 3:}$ The metrics in 4.4 Quantitative analysis, i.e., 'Alignment' and 'Uniformity' do not fully represent the quality of the embedding space. These measurements can evaluate the similarity of embedded features of intra-class samples but do not capture the separation between different classes, i.e., the separability of inter-class distributions. \n\n$textbf{Weakness 4:}$ Manifold intrusion is the main challenge of mixup-based methods. However, I cannot find any analysis for the effect of manifold intrusion of MultiMix. The experimental gains empirically show that the effect of manifold intrusion is minimal in practice. However, further discussion is needed on the effect of manifold intrusion of the multi-sample-based mixup approaches. I wonder if the chance of the intrusion might be higher than the pair-wise mixup of prior works. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback of R-6vc1. We address the concerns as follows:\n### 1. Why multi-sample-based mixup is effective\nUsing the description of our contributions presented in global thread **Author Rebuttal by Authors**, we elaborate on motivation here.\n\n``R-6vc1 - why the multi-sample-based mixup becomes effective in the embedding space rather than the input space``\n\nOur motivation for element 3 i.e. interpolating in the embedding space, is twofold: First, generating $n$ samples (element 1) is computationally expensive when performed in the input space or in shallow layers; generating arbitrarily many samples without significantly increasing the cost is only feasible in the embedding space (L49-52, Fig. 5). Second, *\"interpolation in the latent or embedding space is equivalent to interpolating along a manifold in the input space [36]''* (L27-28). We elaborate on this second motivation here.\n\nInterpolation in the embedding space is originally motivated by Manifold Mixup [36], which *\"leverages semantic interpolations''*, In particular, it *\"leverages interpolations in deeper hidden layers, which capture higher-level information\n(Zeiler \\& Fergus, 2013) to provide additional training signal.''* Indeed, all nonlinear interpolation mechanisms proposed for mixup [45, 16, 15, 33, 3] can be thought of as performing a linear interpolation on a hand-engineered nonlinear manifold. In this sense, Manifold Mixup performs linear interpolation on the manifold learned from the data. *Adversarially constrained autoencoder interpolation* (ACAI) [3*] explicitly studies interpolation in the latent space of an autoencoder to produce a smooth semantic warping effect in data space. This suggests that nearby points in the latent space are semantically similar, which in turn improves representation learning.\n\nBased on the above, mixed examples generated by sampling in the embedding space across different classes lie on the learned manifold. We hypothesize that the learned manifold is a good surrogate of the true, unknown data manifold, which should improve the accuracy and robustness of the model. We also hypothesize that using the learned manifold becomes more important when interpolating more examples. This hypothesis motivating element 3 is validated in Fig. 4(a), where accuracy increases continuously as we shift interpolation to the deepest layer.  \n\n[3*] Berthelot *et al.*, Understanding and improving interpolation in autoencoders via an adversarial regularizer. ICLR 2019.\n\n### 2.  Metrics to measure class separability\nThanks for the interesting suggestion. We formulate the modified alignment metric as\n\n$$\\frac{\\sum_{(x, x') \\sim p_{\\text{pos}}} ||f(x) - f(x') ||^2_2}{\\sum_{(x, x') \\sim p_{\\text{neg}}} ||f(x) - f(x')||^2_2}.$$\n\nHere, $p_\\text{pos}$ denotes the probability distribution of positive pairs (of the same class) and $p_\\text{neg}$ the distribution of negative pairs (of different classes). \n\nWe compare different methods by this modified metric on CIFAR-100 using Resnet-18. We obtain 0.63 for baseline, 0.44 for AlignMixup, 0.39 for MultiMix and 0.35 for Dense MultiMix. This shows that along with having small intra-class distance, MultiMix and Dense MultiMix have higher inter-class separability. We shall add this analysis to the final paper.\n\n### 3. Manifold intrusion\nManifold intrusion can occur when mixed examples are close to classes other than the ones being interpolated in the embedding space. To evaluate for manifold intrusion, we compute the *intrusion distance*, that is, the minimum distance between a mixed embedding and the clean embeddings of all classes except the ones being interpolated. This is formulated as \n\n$$\\text{ID}(\\tilde{Z},Z) = \\frac{1}{|\\tilde{Z} |}\\sum_{\\tilde{z} \\in \\tilde{Z}} \\min_{z \\in Z} ||\\tilde{z} - z||^2.$$\n\nHere, $\\tilde{Z}$ denotes a set of mixed embeddings in a mini-batch and $Z$ denotes the set of clean embeddings from all classes other than the ones being interpolated in the mini-batch. Intuitively, a larger $\\text{ID}(\\tilde{Z},Z)$ denotes that mixed embeddings in $\\tilde{Z}$ are farther away from the manifold of other classes in $Z$, thereby preventing manifold intrusion.\n\nWe compare different methods by this proposed metric on CIFAR-100 using Resnet-18. Averaged over the training set, the intrusion distance is 0.46 for Input Mixup, 0.47 for Manifold Mixup, 0.45 for AlignMixup, 0.46 for MultiMix and 0.47 for Dense MultiMix. This means that most SoTA mixup methods have roughly the same intrusion distance, thus MultiMix and Dense MultiMix do not suffer from a higher manifold intrusion than other methods. \n\nWe thus validate the reviewer's observation that manifold intrusion is minimal in practice. This may be due to the fact that true data occupy only a tiny fraction of the embedding space, thus generated mixed examples may lie in empty space between class-specific manifolds with high probability. The visualization of Fig. 3 indeed shows that the embedding space is sparsely populated, even in two dimensions. This sparsity is expected to grow exponentially in the number of dimensions, which is in the order of $10^3$.  \n\n### 4. Computational Cost\nWe compare the training speed of MultiMix and Dense MultiMix in Section 4.2 under \"Training speed\" (L243-249). We also ablate the training speed of MultiMix and its variants $vs.$ the number of tuples $n$ on CIFAR100 using R-18 in Section A.4 (Appendix) under \"Training speed\" (L543-550). In summary, MultiMix has nearly the same speed with the baseline, whereas Dense MultiMix is only slightly slower.\n\n### 5.  Elaborating on related works\nThanks for this suggestion. We organized related works based on the fact that most SoTA mixup methods focus on defining the interpolation operation, whereas we focus on the number and size of tuples used per mini-batch, as highlighted in Table 1. In the final paper, we shall provide names and discuss specific mixup methods that we have used for comparison in our experiments."}, {"review_id": "inh3ISqBrA", "reviewer": "Reviewer_SS7F", "summary": "The paper introduces a novel data augmentation method called MultiMix, which aims to go beyond the limitations of existing mixup methods. The authors propose interpolating an arbitrarily large number of examples beyond the mini-batch size in the embedding space, rather than along linear segments between pairs of examples. They also extend the method to sequence data with Dense MultiMix, which densely interpolates features and target labels at each spatial location. The paper demonstrates that their proposed solutions outperform state-of-the-art mixup methods on various benchmarks.\n\n*Contribution and novelty* The scientific contribution of the paper lies in the introduction of MultiMix and Dense MultiMix, which address the limitations of existing mixup methods. The authors propose novel approaches for generating an arbitrary number of interpolated examples beyond the mini-batch size, both in the embedding space and for sequence data. The empirical results, as described in the additional explanations, validate the effectiveness of these methods. Overall, this paper seems to improve the research frontier on data augmentation, but empirically and gives some new intuitive understanding of their results as their method leads to a more uniform distribution of embeddings within each class.  I do not see any clear weaknesses in the methods that may challenge the main claim of this work, but I have raised some issues in the questions section. Finally, since I have not read some of the papers that are cited in this work or other related papers in the literature, I defer the judgment about novelty of the ideas presented here to other reviewers.", "questions": "I have the following questions from the authors:\n- There is a lot of emphasis in the paper about the importance of number of loss terms and its importance for obtaining the results reported in this paper. However, one could achieve similarly high number of high loss terms by only incorporating several steps of pairwise mixup of the embeddings. This will have both random-ness in selection of pairs and value of $\\lambda$ and will be equivalent to imposing a sparsity on $\\lambda$'s that are sampled in Multimix. Is this point addressed in your evaluations or perhaps some cited works? In other words, what is exactly the benefit of having a dense $\\lambda$ if doing mixup in the embedding space? \n- It seems that the main benefits of the proposed approach is that it imposes new restrictions on the loss such that the solutions have uniform convex hulls per each input class. If authors agree with this statement, can they think of an alternative approach of achieving this? As a hypothetical experiment, can they think of an adding an explicit term in the loss that will encourage this uniform convex hulls, ie, some kind of a uniform convex hull regularisation term?", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Novelty: The paper introduces a new data augmentation method, MultiMix, that extends the concept of mixup beyond the limitations of existing methods. The idea of interpolating in the embedding space and densely interpolating features and labels in sequence data is original and contributes to the field of data augmentation in machine learning.\n\n- Empirical Results: The authors provide empirical evidence to support the effectiveness of their proposed solutions. They demonstrate significant improvements over state-of-the-art mixup methods on multiple benchmarks. The results validate the scientific contribution of the work and highlight its practical relevance.\n\n- High quality of presentation of results: the presentation of the results follows a simple and logical flow, with main contributions and their difference to the previsous works clearly stated. The Methods, notations,  & results are all clearly stated and easy to understand for the reader. \n\n- Intuitions about results: The paper provides some justifications & intuitions about  why their approach works. They do so by analyzing the embedding space. The authors show that the classes become more tightly clustered and uniformly spread over the embedding space, explaining the improved behavior of their solutions. This analysis adds depth to the scientific contribution of the work.", "weaknesses": "Since I have not read some of the papers that are cited in this work or other related papers in the literature, I defer the judgment about novelty of the ideas presented here to other reviewers. Besides that, I have raised a few issues in the questions section as they are not direct criticism of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback of R-SS7F. We address the concerns as follows:\n\n### 1. Importance of using dense $\\lambda$ in embedding space\n\nUsing the description of our contributions presented in global thread **Author Rebuttal by Authors**, we elaborate on motivation here.\n\n``R-SS7F - one could achieve similarly a high number of high loss terms by only incorporating several steps of pairwise mixup of the embeddings``\n\nThis refers to using $m = 2$ rather than $m = b$ (element 2) for the same number $n$ of generated examples (element 1). In Fig. 4(c), we experiment with the configuration $m = 2, n = 1000$, which has a high number of loss terms, each obtained by mixing pairs, as in most mixup methods. This configuration results in 0.7\\% lower accuracy compared to our default choice $m = b = 128, n = 1000$, thus empirically validating element 2.\n\n``R-SS7F - what is exactly the benefit of having a dense $\\lambda$ if doing mixup in the embedding space?``\n\nUsing a dense $\\lambda \\in \\Delta^{m-1}$ with $m = b$ sampled from the Dirichlet distribution *vs.* $\\lambda \\in [0,1]$ with $m = 2$ sampled from the Beta distribution refers to element 2.\n\nThe motivation for element 2 is similar to the motivation of element 1, that is, to better approximate the expected risk integral by using a more uniform distribution for $\\lambda$. The difference is best illustrated in Fig. 1: Even if we generate an arbitrarily large number $n$ of mixed examples with $m = 2$, these will always be constrained to linear segments between pairs of examples in the mini-batch, as shown in Fig. 1(a). By contrast, by using a dense $\\lambda$ as in MultiMix, generated examples span the entire convex hull of the mini-batch examples, as shown in Fig. 1(b). Thus, we hypothesize that the expected risk integral is better approximated. This hypothesis motivating element 2 is validated in Fig. 4(c), as explained above.\n\n### 2. Using a uniform convex hull regularizer\n\nThis statement about a uniform convex hull per class does not describe what MultiMix is doing. The closest statement about MultiMix is best depicted in Fig. 1(b): We draw mixed examples from a distribution that is uniform (more precisely, Dirichlet with parameter $\\alpha$ drawn uniformly at random) in the convex hull of the mini-batch examples, regardless of their class. Class labels are separately interpolated linearly as usual, see eq. (4)."}]

Author rebuttal:
As summarized in our first contribution (L81-83), MultiMix consists of the following three elements:

- Increase the number $n$ of generated mixed examples beyond the mini-batch size $b$ (L46-54, L71-78, Table 1/column *terms*, Fig. 4(b)).
 - Increase the number $m$ of examples being interpolated from $m = 2$ (pairs) to $m = b$ (a single tuple containing the entire mini-batch) (L55-61, Table 1/column *mixed*, Fig. 4(c)).
 - Perform interpolation in the *embedding* space rather than the *input* space (L27-28, L43-45, Table 1/column *space*', Fig. 4(a)).

Dense Multimix (L66-78) is our second contribution (L84-85). It is an additional application of element 1, further increasing the number of generated mixed examples to $nr$ per mini-batch (L71-78, Table 1/column *terms*, Fig. 4(a-c)), where $r$ is the spatial resolution.

All three elements are important in achieving the performance of MultiMix beyond the SoTA. Removing any element leads to sub-optimal results, as shown in the ablation of Fig. 4(a-c). Previous works have missed this combination. For example, Manifold mixup [36] is element 3 alone (L27-28). A number of works [1,6,47] have attempted element 2 alone (L58-60) and found it not effective for $m > 3$. To our knowledge, element 1 has not been investigated before. As requested, we elaborate below on our motivation of element 1. More on the motivation of elements 2 and 3 follow in the responses to R-SS7F and R-6vc1, respectively.

`` R-Xd45: The only statement regarding to the motivation... fails to distinguish itself from the preceding data augmentation methods.``

Our motivation for element 1 is stated in L32-38, which we elaborate on here. The motivation stems from the original mixup paper [47], where the *expected risk* is defined as an integral over the underlying continuous data distribution. Since that distribution is unknown, the integral is approximated by a finite sum over the training data, i.e, the *empirical risk*, where each training example gives rise to one loss term. A better approximation is the *vicinal risk*, where a number of augmented examples is sampled from a distribution in the vicinity of each training example, thus *increasing the number of loss terms per training example*. Mixup applies the same idea to pairs of training examples, where the vicinal distribution of each training example is defined over linear segments to all other training examples. However, as a practical implementation, the authors of [47] *"convex combine a minibatch with a random permutation of its sample index.''* Thus, the opportunity of increasing the number of loss terms per training example is lost: each mini-batch of size $b$ gives rise to precisely $b$ loss terms. Unfortunately, this choice is adopted by all follow-up works on mixup.

In our work, *"we argue that a data augmentation process should increase the data seen by the model, or at least by its last few layers, as much as possible''* (L39-43). By this, we mean that even a small part of the model should see more inputs and targets than $b$ per mini-batch, giving rise to more loss terms than $b$ per mini-batch (L46-53). Our particular solution is that the classifier head $g_W$ sees $n \gg b$ mixed inputs and targets, giving rise to $n$ loss terms per mini-batch. This is clarified in column *terms* of Table 1, where MultiMix differs from all previous mixup works. The motivation is precisely that $n \gg b$ loss terms provide a better approximation of the expected risk integral. Even if the encoder $f_\theta$ only sees $b$ clean examples per mini-batch, it is still affected by backpropagating gradients from all $n$ mixed examples during training.

``R-Xd45: it doesn't elaborate on how the proposed approach enhances accuracy and robustness``

Using more loss terms than $b$ per mini-batch is not very common, at least not in classification. In *deep metric learning*, it is common to have a loss term for each pair of examples in a mini-batch, giving rise to $\frac{1}{2} b (b-1)$ loss terms per mini-batch $b$, even without mixup [1*]. In image classification, *supervised contrastive learning* [2*] involves a loss term for every positive pair of each example in the mini-batch, thus also resulting in more than $b$ loss terms per mini-batch. *Dense loss functions* (L116-122) also increase the number of loss terms per training example by a factor $r$, where $r$ is the spatial resolution.

We hypothesize that increasing the number of loss terms beyond $b$ per mini-batch improves the accuracy and robustness of the model by providing a better approximation of the expected risk integral. MultiMix is the first method to do so for mixup, increasing the number of loss terms per mini-batch to $n \gg b$. Dense MultiMix is the first dense mixup operation, further increasing the number of loss terms per mini-batch to $nr$ (Table 1/column ``terms''). Our hypothesis motivating element 1 is validated in Fig. 4(b), where accuracy increases continuously as we increase $n$, both for MultiMix and Dense MultiMix. 

``R-6vc1 - it is hard to find a noticeable difference between the `trial' of the original Mixup paper and the proposed method except for the target space``

The `trial' of the original Mixup paper [47] refers to element 2, $m > 2$, where $m$ is the number of examples being interpolated. Interpolating in the embedding space refers to element 3. None of these two elements alone is the main contribution of our work. As stated above, a number of works [1,6,47] have attempted element 2 alone (L58-60), while Manifold mixup [36] is element 3 alone (L27-28). The main contribution is the observation that all three elements, where element 1 has not been attempted before. We have made our best effort to highlight element 1: L46-64, L71-78, Table 1/column *terms*.

[1*] Venkataramanan et al., It Takes Two To Tango: Mixup for Deep Metric Learning, ICLR 2022  

[2*] Khosla et al., Supervised Contrastive Learning, NeurIPS 2020

Meta-review: The reviewers and meta reviewer all carefully checked and discussed the rebuttal. They thank the authors for their response and their efforts during the rebuttal phase.

The reviewers and meta reviewer all acknowledge the original and sound contribution of combining different (subtle) aspects of mixup that had been considered so far in isolation and unsuccessfully. Similarly, the reviewers and meta reviewer all call out the overall good quality of the manuscript (writing, structure).

The rebuttal has considerably strengthened some aspects of the work (e.g., comparison with AugMix and DAML, analysis of smaller batch sizes, sensitivity with respect to the selection of the hyperparameters). However, there are still some remaining important concerns. As a result, the reviewers and the meta reviewer are weakly inclined to accept the paper.

In particular, the authors are urged to carefully update their final manuscript with the following points:

* _“Third, instead of using a single scalar value of lambda per mini-batch….”_: Using different lambda’s within a single batch seems to have been used by default in previous work with standard mixup, e.g., [A]
* _“...Using more loss terms than per mini-batch is not very common…”_: It should be mentioned that in the literature on efficient ensembles (e.g., [B, C, D, E]), the batch size with b examples is often turn to M*b loss terms, where M denotes the number of ensemble members.
* Systematically add the additional empirical evidence, and corresponding discussions, obtained during the rebuttal phase
* _“...MultiMix is thus a better approximation of the expected risk integral…”_: This is an argument that has been repeatedly used during the rebuttal. It is thus important to provide more insights and an empirical analysis (e.g., based on a synthetic setting where the true risk integral can be empirically estimated) and/or some theoretical arguments.
* The discussion with reviewer R-SS7F suggests that the hyperparameters have been selected in a *test-aware* fashion (on CIFAR-100, before then being identically applied to other settings). This departs from a clean 3-fold (train, validation, test) selection. As a result, the analysis started in https://openreview.net/forum?id=HKueO74ZTB&noteId=F5gh77Hr2x should be extended to all other hyperparameters.
* What happens with larger batch sizes? A study with b > or >> 128 (as used in many standard training procedures) would help understand if the benefits of the proposed approach plateau  

If the paper was submitted to a journal, it would be accepted conditioned on those key changes, the meta reviewer thus expects all those changes to be carefully implemented.

[A] Carratino et al., 2021, On Mixup Regularization. (https://github.com/google/uncertainty-baselines/blob/main/baselines/imagenet/deterministic.py#L64)

[B] Wen et al, 2019, BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

[C] Dusenberry et al, 2020, Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors

[D] Havasi et al, 2021, Training independent subnetworks for robust prediction

[E] Allingham et al, 2021, Sparse MoEs meet Efficient Ensembles
Decision: Accept (poster)