Title: Embedding Space Interpolation Beyond Mini-Batch, Beyond Pairs and Beyond Examples
Venue: NeurIPS 2023
Accepted: yes
Keywords: Interpolation based data augmentation

mixup

dense interpolation

robustness

representation learning
Abstract: Mixup refers to interpolation-based data augmentation, originally motivated as a way to go beyond empirical risk minimization (ERM). Its extensions mostly focus on the definition of interpolation and the space (input or feature) where it takes place, while the augmentation process itself is less studied. In most methods, the number of generated examples is limited to the mini-batch size and the number of examples being interpolated is limited to two (pairs), in the input space.

We make progress in this direction by introducing MultiMix, which generates an arbitrarily large number of interpolated examples beyond the mini-batch size and interpolates the entire mini-batch in the embedding space. Effectively, we sample on the entire convex hull of the mini-batch rather than along linear segments between pairs of examples.

On sequence data, we further extend to Dense MultiMix. We densely interpolate features and target labels at each spatial location and also apply the loss densely. To mitigate the lack of dense labels, we inherit labels from examples and weight interpolation factors by attention as a measure of confidence.

Overall, we increase the number of loss terms per mini-batch by orders of magnitude at little additional cost. This is only possible because of interpolating in the embedding space. We empirically show that our solutions yield significant improvement over state-of-the-art mixup methods on four different benchmarks, despite interpolation being only linear. By analyzing the embedding space, we show that the classes are more tightly clustered and uniformly spread over the embedding space, thereby explaining the improved behavior.

Embedding Space Interpolation Beyond Mini-Batch, Beyond Pairs and Beyond Examples

Abstract

Mixup refers to interpolation-based data augmentation, originally motivated as a way to go beyond empirical risk minimization (ERM). Its extensions mostly focus on the definition of interpolation and the space (input or embedding) where it takes place, while the augmentation process itself is less studied. In most methods, the number of generated examples is limited to the mini-batch size and the number of examples being interpolated is limited to two (pairs), in the input space.

We make progress in this direction by introducing MultiMix , which generates an arbitrarily large number of interpolated examples beyond the mini-batch size, and interpolates the entire mini-batch in the embedding space. Effectively, we sample on the entire convex hull of the mini-batch rather than along linear segments between pairs of examples.

On sequence data we further extend to Dense MultiMix . We densely interpolate features and target labels at each spatial location and also apply the loss densely. To mitigate the lack of dense labels, we inherit labels from examples and weight interpolation factors by attention as a measure of confidence.

Overall, we increase the number of loss terms per mini-batch by orders of magnitude at little additional cost. This is only possible because of interpolating in the embedding space . We empirically show that our solutions yield significant improvement over state-of-the-art mixup methods on four different benchmarks, despite interpolation being only linear. By analyzing the embedding space, we show that the classes are more tightly clustered and uniformly spread over the embedding space, thereby explaining the improved behavior.

1 Introduction

Mixup [65] is a data augmentation method that interpolates between pairs of training examples, thus regularizing a neural network to favor linear behavior in-between examples. Besides improving generalization, it has important properties such as reducing overconfident predictions and increasing the robustness to adversarial examples. Several follow-up works have studied interpolation in the latent or embedding space, which is equivalent to interpolating along a manifold in the input space [51], and a number of nonlinear and attention-based interpolation mechanisms [62, 26, 25, 46, 7]. However, little progress has been made in the augmentation process itself, i.e ., the number n of generated examples and the number m of examples being interpolated.

Mixup was originally motivated as a way to go beyond empirical risk minimization (ERM) [47] through a vicinal distribution expressed as an expectation over an interpolation factor λ , which is equivalent to the set of linear segments between all pairs of training inputs and targets. In practice however, in every training iteration, a single scalar λ is drawn and the number of interpolated pairs is limited to the size b of the mini-batch ( n = b ), as illustrated in Figure 1(a). This is because if

interpolation takes place in the input space, it would be expensive to increase the number of pairs per iteration. To our knowledge, these limitations exist in all mixup methods.

In this work, we argue that a data augmentation process should increase the data seen by the model, or at least by its last few layers, as much as possible. In this sense, we follow manifold mixup [51] and generalize it in a number of ways to introduce MultiMix .

First, we increase the number n of generated examples beyond the mini-batch size b , by orders of magnitude ( n /greatermuch b ). This is possible by interpolating at the deepest layer, i.e ., just before the classifier, which happens to be the most effective choice. To our knowledge, we are the first to investigate n > b .

(a) mixup

(b) MultiMix (ours)

Second, we increase the number m of examples being interpolated from m = 2 (pairs) to m = b (a single tuple containing the entire mini-batch). Effectively, instead of linear segments between pairs of examples in the mini-batch, we sample on their entire convex hull as illustrated in Figure 1(b). This idea has been investigated in the input space: the original mixup method [65] found it noneffective, while [11] found it effective only up to m = 3 examples and [1] went up to m = 25 but with very sparse interpolation factors. To our knowledge, we are the first to investigate m> 2 in the embedding space and to show that it is effective up to m = b .

Third, instead of using a single scalar value of λ per mini-batch, we draw a different vector λ ∈ R m for each interpolated example. A single λ works for standard mixup because the main source of randomness is the choice of pairs (or small tuples) out of b examples. In our case, because we use a single tuple of size m = b , the only source of randomness being λ .

Wealso argue that, what matters more than the number of (interpolated) examples is the total number of loss terms per mini-batch. A common way to increase the number of loss terms per example is by dense operations when working on sequence data, e.g . patches in images or voxels in video. This is common in dense tasks like segmentation [41] and less common in classification [27]. We are the first to investigate this idea in mixup, introducing Dense MultiMix .

In particular, this is an extension of MultiMix where we work with feature tensors of spatial resolution r and densely interpolate features and targets at each spatial location, generating r interpolated features per example and nr > n per mini-batch. We also apply the loss densely. This increases the number of loss terms further by a factor r , typically one or two orders of magnitude, compared with MultiMix. Of course, for this to work, we also need a target label per feature, which we inherit from the corresponding example. This is a weak form of supervision [70]. To carefully select the most representative features per object, we use an attention map representing our confidence in the target label per spatial location. The interpolation vectors λ are then weighted by attention.

Table 1 summarizes the properties of our solutions against existing interpolation methods. Overall, we make the following contributions:

We generate an arbitrary large number of interpolated examples beyond the mini-batch size, each by interpolating the entire mini-batch in the embedding space, with one interpolation vector per example. (subsection 3.2).

We extend to attention-weighted dense interpolation in the embedding space, further increasing the number of loss terms per example (subsection 3.3).

We improve over state-of-the-art (SoTA) mixup methods on image classification , robustness to adversarial attacks , object detection and out-of-distribution detection . Our solutions have little or no additional cost while interpolation is only linear (section 4).

Analysis of the embedding space shows that our solutions yield classes that are tightly clustered and uniformly spread over the embedding space (section 4).

2 Related Work

Mixup: interpolation methods In general, mixup interpolates between pairs of input examples [65] or embeddings [51] and their corresponding target labels. Several follow-up methods mix input images according to spatial position, either at random rectangles [62] or based on attention [46, 26, 25], in an attempt to focus on a different object in each image. Other follow-up method [35] randomly cuts image at patch level and obtains its corresponding mixed label using content-based attention. We also use attention in our dense MultiMix variant, but in the embedding space. Other definitions of interpolation include the combination of content and style from two images [23], generating out-of-manifold samples using adaptive masks [37], generating binary masks by thresholding random low-frequency images [18] and spatial alignment of dense features [48]. Our dense MultiMix also uses dense features but without aligning them and can interpolate a large number of samples, thereby increasing the number of loss-terms per example. Our work is orthogonal to these methods as we focus on the sampling process of augmentation rather than on the definition of interpolation. We refer the reader to [30] for a comprehensive study of mixup methods.

Mixup: number of examples to interpolate To the best of our knowledge, the only methods that interpolate more than two examples for image classification are OptTransMix [71], SuperMix [11], ζ -Mixup [1] and DAML [45]. All these methods operate in the input space and limit the number of generated examples to the mini-batch size; whereas MultiMix generates an arbitrary number of interpolated examples (more than 1000 in practice) in the embedding space . To determine the interpolation weights, OptTransMix uses a complex optimization process and only applies to images with clean background; ζ -Mixup uses random permutations of a fixed vector; SuperMix uses a Dirichlet distribution over not more than 3 examples in practice; and DAML uses a Dirichlet distribution to interpolate across different domains. We also use a Dirichlet distribution but over as many examples as the mini-batch size. Similar to MultiMix, [6] uses different λ within a single batch. Beyond classification, m -Mix [66] uses graph neural networks in a self-supervised setting with pair-based loss functions. The interpolation weights are deterministic and based on pairwise similarities.

Number of loss terms Increasing the number of loss terms more than b per mini-batch is not very common in classification. In deep metric learning [40, 28, 53], it is common to have a loss term for each pair of examples in a mini-batch in the embedding space, giving rise to b ( b -1) / 2 loss terms per mini-batch, even without mixup [49]. Contrastive loss functions are also common in selfsupervised learning [8, 5] but have been used for supervised image classification too [24]. Ensemble methods [55, 14, 19, 3] increase the number of loss terms, but with the proportional increase of the cost by operating in the input space. To our knowledge, MultiMix is the first method to increase the number of loss terms per mini-batch to n /greatermuch b for mixup. Dense MultiMix further increases this number to nr . By operating in the embedding space, their computational overhead is minimal.

Dense loss functions Although standard in dense tasks like semantic segmentation [41, 20], where dense targets commonly exist, dense loss functions are less common otherwise. Few examples are in weakly-supervised segmentation [70, 2], few-shot learning [33, 31], where data augmentation is of utter importance, attribution methods [27] and unsupervised representation learning , e.g . dense contrastive learning [42, 54], learning from spatial correspondences [59, 57] and masked language or image modeling [12, 58, 32, 69]. To our knowledge, we are the first to use dense interpolation and a dense loss function for mixup.

3 Method

3.1 Preliminaries and background

Problem formulation Let x ∈ X be an input example and y ∈ Y its one-hot encoded target, where X = R D is the input space, Y = { 0 , 1 } c and c is the total number of classes. Let f θ : X → R d be an encoder that maps the input x to an embedding z = f θ ( x ) , where d is the dimension of the embedding. A classifier g W : R d → ∆ c -1 maps z to a vector p = g W ( z ) of predicted probabilities over classes, where ∆ n ⊂ R n +1 is the unit n -simplex, i.e ., p ≥ 0 and 1 /latticetop c p = 1 , and 1 c ∈ R c is an all-ones vector. The overall network mapping is f := g W ◦ f θ . Parameters ( θ , W ) are learned by optimizing over mini-batches.

Given a mini-batch of b examples, let X = ( x 1 , . . . , x b ) ∈ R D × b be the inputs, Y = ( y 1 , . . . , y b ) ∈ R c × b the targets and P = ( p 1 , . . . , p b ) ∈ R c × b the predicted probabilities of the mini-batch, where P = f ( X ) := ( f ( x 1 ) , . . . , f ( x b )) . The objective is to minimize the cross-entropy

of predicted probabilities P relative to targets Y averaged over the mini-batch, where /circledot is the Hadamard (element-wise) product. In summary, the mini-batch loss is

The total number of loss terms per mini-batch is b .

Mixup Mixup methods commonly interpolate pairs of inputs or embeddings and the corresponding targets at the mini-batch level while training. Given a mini-batch of b examples with inputs X and targets Y , let Z = ( z 1 , . . . , z b ) ∈ R d × b be the embeddings of the mini-batch, where Z = f θ ( X ) . Manifold mixup [51] interpolates the embeddings and targets by forming a convex combination of the pairs with interpolation factor λ ∈ [0 , 1] :

˜ where λ ∼ Beta( α , α ) , I is the identity matrix and Π ∈ R b × b is a permutation matrix. Input mixup [65] interpolates inputs rather than embeddings:

˜ X = X ( λ I +(1 -λ ) Π ) . (5) Whatever the interpolation method and the space where it is performed, the interpolated data, e.g . ˜ X [65] or ˜ Z [51], replaces the original mini-batch data and gives rise to predicted probabilities ˜ P = ( p 1 , . . . , p b ) ∈ R c × b over classes, e.g . ˜ P = f ( ˜ X ) [65] or ˜ P = g W ( ˜ Z ) [51]. Then, the average cross-entropy H ( ˜ Y , ˜ P ) (1) between the predicted probabilities ˜ P and interpolated targets ˜ Y is minimized. The number of generated examples per mini-batch is n = b , same as the original mini-batch size, and each is obtained by interpolating m = 2 examples. The total number of loss terms per mini-batch is again b .

3.2 MultiMix

Interpolation The number of generated examples per mini-batch is now n /greatermuch b and the number of examples being interpolated is m = b . Given a mini-batch of b examples with embeddings Z and targets Y , we draw interpolation vectors λ k ∼ Dir( α ) for k = 1 , . . . , n , where Dir( α ) is the symmetric Dirichlet distribution and λ k ∈ ∆ m -1 , that is, λ k ≥ 0 and 1 /latticetop m λ k = 1 . We then interpolate embeddings and targets by taking n convex combinations over all m examples:

˜ where Λ = ( λ 1 , . . . , λ n ) ∈ R b × n . We thus generalize manifold mixup [51]:

from b to an arbitrary number n /greatermuch b of generated examples: interpolated embeddings ˜ Z ∈ R d × n (6) vs . R d × b in (3), targets ˜ Y ∈ R c × n (7) vs . R c × b in (4);

x

1

z

1

/circledot

norm





a

1

/circledot

λ



encoder

f

θ

⊕

˜

z

x

2

z

2

/circledot

norm





a

2

/circledot

1

λ



-

from pairs ( m = 2 ) to a tuple of length m = b , containing the entire mini-batch: m -term convex combination (6),(7) vs . 2-term in (3),(4), Dirichlet vs . Beta distribution;

from fixed λ across the mini-batch to a different λ k for each generated example.

Loss Again, we replace the original mini-batch embeddings Z by the interpolated embeddings ˜ Z and minimize the average cross-entropy H ( ˜ Y , ˜ P ) (1) between the predicted probabilities ˜ P = g W ( ˜ Z ) and the interpolated targets ˜ Y (7). Compared with (2), the mini-batch loss becomes L M ( X,Y ; θ , W ) := H ( Y Λ , g W ( f θ ( X ) Λ )) . (8)

The total number of loss terms per mini-batch is now n /greatermuch b .

3.3 Dense MultiMix

We now extend to the case where the embeddings are structured, e.g . in tensors. This happens e.g . with token vs . sentence embeddings in NLP and patch vs . image embeddings in vision. It works by removing spatial pooling and applying the loss function densely over all tokens/patches. The idea is illustrated in Figure 2. For the sake of exposition, our formulation uses sets of matrices grouped either by example or by spatial position. In practice, all operations are on tensors.

Preliminaries The encoder is now f θ : X → R d × r , mapping the input x to an embedding z = f θ ( x ) ∈ R d × r , where d is the number of channels and r is its spatial resolution-if there are more than one spatial dimensions, these are flattened.

Given a mini-batch of b examples, we have again inputs X = ( x 1 , . . . , x b ) ∈ R D × b and targets Y = ( y 1 , . . . , y b ) ∈ R c × b . Each embedding z i = f θ ( x i ) = ( z 1 i , . . . , z r i ) ∈ R d × r for i = 1 , . . . , b consists of features z j i ∈ R d for spatial position j = 1 , . . . , r . We group features by position in matrices Z 1 , . . . , Z r , where Z j = ( z j 1 , . . . , z j b ) ∈ R d × b for j = 1 , . . . , r .

Attention In the absence of dense targets, each spatial location inherits the target of the corresponding input example. This is weak supervision, because the target object is not visible everywhere. To select the most reliable locations, we define a level of confidence according to an attention map. Given an embedding z ∈ R d × r with target y ∈ Y and a vector u ∈ R d , the attention map

measures the similarity of features of z to u , where h is a non-linearity, e.g . softmax or ReLU followed by /lscript 1 normalization. There are different ways to define vector u . For example, u = z1 r /r by global average pooling (GAP) of z , or u = Wy assuming a linear classifier with W ∈ R d × c , similar to class activation mapping (CAM) [68]. In case of no attention, a = 1 r /r is uniform.

Given a mini-batch, let a i = ( a 1 i , . . . , a r i ) ∈ R r be the attention map of embedding z i (9) for i = 1 , . . . , b . We group attention by position in vectors a 1 , . . . , a r , where a j = ( a j 1 , . . . , a j b ) ∈ R b

for j = 1 , . . . , r . Figure 6 in the supplementary shows the attention obtained by (9). We observe high confidence on the entire or part of the object. Where confidence is low, we assume the object is not visible and thus the corresponding interpolation factor should be low.

Interpolation There are again n /greatermuch b generated examples per mini-batch, with m = b examples being densely interpolated. For each spatial position j = 1 , . . . , r , we draw interpolation vectors λ j k ∼ Dir( α ) for k = 1 , . . . , n and define Λ j = ( λ j 1 , . . . , λ j n ) ∈ R m × n . Since input examples are assumed to contribute according to the attention vector a j ∈ R m , we scale the rows of Λ j accordingly and normalize its columns back to ∆ m -1 to define convex combinations:

We then interpolate embeddings and targets by taking n convex combinations over m examples:

Classifier The classifier is now g W : R d × r → R c × r , maintaining the same spatial resolution as the embedding and generating one vector of predicted probabilities per spatial position. This is done by removing average pooling or any down-sampling operation. The interpolated embeddings ˜ Z 1 , . . . , ˜ Z r (12) are grouped by example into ˜ z 1 , . . . , ˜ z n ∈ R d × r , mapped by g W to predicted probabilities p 1 , . . . , p n ∈ R c × r and grouped again by position into P 1 , . . . , P r ∈ R c × n .

˜ This is similar to (6),(7), but there is a different interpolated embedding matrix ˜ Z j ∈ R d × n as well as target matrix ˜ Y j ∈ R c × n per position, even though the original target matrix Y is one. The total number of interpolated features and targets per mini-batch is now nr .

˜ ˜ ˜ ˜ In the simple case where the original classifier is linear, i.e . W ∈ R d × c , it is seen as 1 × 1 convolution and applied densely to each column (feature) of Z j for j = 1 , . . . , r .

generalizes (1) and the weight vector is defined as s = 1 /latticetop m M j ∈ R n . This is exactly the vector used to normalize the columns of M j in (11). The motivation is that the columns of M j are the original interpolation vectors weighted by attention: A small /lscript 1 norm indicates that for the given position j , we are sampling from examples of low attention, hence the loss is to be discounted. The total number of loss terms per mini-batch is now nr .

˜ Loss Finally, we learn parameters θ , W by minimizing the weighted cross-entropy H ( ˜ Y j , ˜ P j ; s ) of ˜ P j relative to the interpolated targets ˜ Y j again densely at each position j , where H ( Y, P ; s ) := -1 /latticetop c ( Y /circledot log( P )) s/ ( 1 /latticetop n s ) (14)

4 Experiments

4.1 Setup

We use a mini-batch of size b = 128 examples in all experiments. Following manifold mixup [51], for every mini-batch, we apply MultiMix with probability 0 . 5 or input mixup otherwise. For MultiMix, the default settings are given in subsection 4.6. We use PreActResnet-18 (R-18) [21] and WRN16-8 [63] as encoder on CIFAR-10 and CIFAR-100 datasets [29]; R-18 on TinyImagenet [60] (TI); and Resnet-50 (R-50) and ViT-S/16 [13] on ImageNet [44]. To better understand the effect of mixup in ViT, we evaluate MultiMix and Dense MultiMix on ImageNet without using strong augmentations like Auto-Augment [9], Rand-Augment [10], random erasing [67] and CutMix [62]. We reproduce TransMix [7] and TokenMix [35] using these settings.

4.2 Results: Image classification and robustness

Image classification In Table 2 we observe that MultiMix and Dense MultiMix already outperform SoTA on all datasets except CIFAR-10 with R-18, where they are on par with Co-Mixup. Dense MultiMix improves over vanilla MultiMix and its effect is complementary on all datasets. On TI for example, Dense MultiMix improves over MultiMix by 1.33% and SoTA by 1.59%. We provide additional analysis of the embedding space on 10 classes of CIFAR-100 in subsection 4.4.

DATASET

CIFAR-10

CIFAR-100

TI

NETWORK

R-18

W16-8

R-18

W16-8

R-18

Baseline

†

95.41

±

0.02

94.93

±

0.06

76.69

±

0.26

78.80

±

0.55

56.49

±

0.21

Manifold mixup [51]

†

97.00

±

0.05

96.44

±

0.02

80.00

±

0.34

80.77

±

0.26

59.31

±

0.49

PuzzleMix [26]

†

97.04

±

0.04

97.00

±

0.03

79.98

±

0.05

80.78

±

0.23

63.52

±

0.42

Co-Mixup [25]

†

97.10

±

0.03

96.44

±

0.08

80.28

±

0.13

80.39

±

0.34

64.12

±

0.43

AlignMixup [48]

†

97.06

±

0.04

96.91

±

0.01

81.71

±

0.07

81.24

±

0.02

66.85

±

0.07

ζ

-Mixup [1]

/star

96.26

±

0.04

96.35

±

0.04

80.46

±

0.26

79.73

±

0.15

63.18

±

0.14

MultiMix (ours)

97.07

±

0.03

97.06

±

0.02

81.82

±

0.04

81.44

±

0.03

67.11

±

0.04

Dense MultiMix (ours)

97.09

±

0.02

97.09

±

0.02

81.93

±

0.04

81.77

±

0.03

68.44

±

0.05

Gain

-0.01

+0.09

+0.22

+0.53

+1.59

ATTACK

FGSM

PGD

DATASET

CIFAR-10

CIFAR-100

TI

CIFAR-10

CIFAR-100

NETWORK

R-18

W16-8

R-18

W16-8

R-18

R-18

W16-8

R-18

W16-8

Baseline

†

88.8

±

0.11

88.3

±

0.33

87.2

±

0.10

72.6

±

0.22

91.9

±

0.06

99.9

±

0.0

99.9

±

0.01

99.9

±

0.01

99.9

±

0.01

Manifold mixup [51]

†

76.9

±

0.14

76.0

±

0.04

80.2

±

0.06

56.3

±

0.10

89.3

±

0.06

97.2

±

0.01

98.4

±

0.03

99.6

±

0.01

98.4

±

0.03

PuzzleMix [26]

†

57.4

±

0.22

60.7

±

0.02

78.8

±

0.09

57.8

±

0.03

83.8

±

0.05

97.7

±

0.01

97.0

±

0.01

96.4

±

0.02

95.2

±

0.03

Co-Mixup [25]

†

60.1

±

0.05

58.8

±

0.10

77.5

±

0.02

56.5

±

0.04

-

97.5

±

0.02

96.1

±

0.03

95.3

±

0.03

94.2

±

0.01

AlignMixup [48]

†

54.8

±

0.03

56.0

±

0.05

74.1

±

0.04

55.0

±

0.03

78.8

±

0.03

95.3

±

0.04

96.7

±

0.03

90.4

±

0.01

92.1

±

0.03

ζ

-Mixup [1]

/star

72.8

±

0.23

67.3

±

0.24

75.3

±

0.21

68.0

±

0.21

84.7

±

0.18

98.0

±

0.06

98.6

±

0.03

97.4

±

0.10

96.1

±

0.10

MultiMix (ours)

54.1

±

0.09

55.3

±

0.04

73.8

±

0.04

54.5

±

0.01

77.5

±

0.01

94.2

±

0.04

94.8

±

0.01

90.0

±

0.01

91.6

±

0.01

Dense MultiMix (ours)

54.1

±

0.01

53.3

±

0.03

73.5

±

0.03

52.9

±

0.04

75.5

±

0.04

92.9

±

0.04

92.6

±

0.01

88.6

±

0.03

90.8

±

0.01

Gain

+0.7

+2.7

+0.6

+2.1

+3.3

+2.4

+3.5

+1.4

+1.3

In Table 3 we observe that on ImageNet with R50, vanilla MultiMix outperforms all methods except AlignMixup. Dense MultiMix outperforms all SoTA with both R-50 and ViT-S/16, bringing an overall gain of 3% over the baseline with R-50 and 2.2% with ViT-S/16. The gain over AlignMixup with R-50 is small, but it is impressive that it comes with only linear interpolation. To better isolate the effect of each method, we reproduce TransMix [7] and TokenMix [35] with ViT-S/16 using their official code with our settings, i.e ., without strong regularizers like CutMix, Auto-Augment, RandomAugment etc . MultiMix is on par, while Dense MultiMix outperforms them by 1%.

Training speed Table 3 also shows the training speed as measured on NVIDIA V-100 GPU including forward and backward pass. The vanilla MultiMix has nearly the same speed with the baseline, bringing an accuracy gain of 2.49% with R-

50. Dense MultiMix is slightly slower, increasing the gain to 3.10%. The inference speed is the same for all methods.

Robustness to adversarial attacks We follow the experimental settings of AlignMixup [48] and use 8 / 255 l ∞ /epsilon1 -ball for FGSM [16] and 4 / 255 l ∞ /epsilon1 -ball with step size 2/255 for PGD [38] attack.

Baseline

AlignMixup

MultiMix (ours)

Dense MultiMix (ours)

In Table 4 we observe that MultiMix is already more robust than SoTA on all datasets and settings. Dense MultiMix also increases the robustness and is complementary.

The overall gain is more impressive than in classification according to Table 2. For example, against the strong PGD attack on CIFAR-10 with W16-8, the SoTA Co-Mixup improves the baseline by 3.8% while Dense MultiMix improves it by 7.3%, which is double. MultiMix and Dense MultiMix outperform Co-Mixup and PuzzleMix by 3-6% in robustness on CIFAR-10, even though they are onpar on classification. There is also a significant gain over SoTA AlignMixup by 1-3% in robustness to FGSM on TinyImageNet and to the stronger PGD.

4.3 Results: Transfer learning to object detection

We evaluate the effect of mixup on the generalization ability of a pre-trained network to object detection as a downstream task. Following the settings of CutMix [62], we pre-train R-50 on ImageNet with mixup methods and use it as the backbone for SSD [36] with fine-tuning on Pascal VOC07 + 12 [15] and Faster-RCNN [43] with fine-tuning on MS-COCO [34].

In Table 5, we observe that, while MultiMix is slightly worse than AlignMixup on Pascal VOC07 + 12, Dense MultiMix brings improvements over the SoTA on both datasets and is still complementary. This is consistent with classification results. Dense MultiMix brings a gain of 0.8 mAP on Pascal VOC07 + 12 and 0.35 mAP on MS-COCO.

4.4 Analysis of the embedding space

Qualitative analysis We qualitatively analyze the embedding space on 10 CIFAR-100 classes in Figure 3. We observe that the quality of embeddings of the baseline is extremely poor with severely overlapping classes, which explains its poor performance on image classification. All mixup methods result in clearly better clustered and more uniformly spread classes. AlignMixup [48] yields five somewhat clustered classes and five moderately overlapping ones. Our best setting, i.e ., Dense MultiMix, results in five tightly clustered classes and another five somewhat overlapping but less than all competitors.

Quantitative analysis We also quantitatively assess the embedding space on the CIFAR-100 test set using alignment and uniformity [52]. Alignment measures the expected pairwise distance of examples in the same class. Lower alignment indicates that the classes are more tightly clustered. Uniformity measures the (log of the) expected pairwise similarity of all examples using a Gaussian kernel as a similarity function. Lower uniformity indicates that classes are more uniformly spread in the embedding space.

MultiMix

Dense MultiMix

Default

Standard

0

0,1

0,2

0,3

0,4

79

80

81

82

(a) Mixing layers

10

1

10

2

10

3

10

4

10

5

80

80.5

81

81.5

82

(b) # mixed examples

n

2

25

50

100

128

80

80.5

81

81.5

82

(c) # examples interpolated

m

0.5

1

1.5

2

80.5

81

81.5

(d) Dirichlet parameter

α

On CIFAR-100, we obtain alignment 3.02 for baseline, 2.04 for AlignMixup, 1.27 for MultiMix and 0.92 for Dense MultiMix. We also obtain uniformity -1.94 for the baseline, -2.38 for AlignMixup [48], -4.77 for MultiMix and -5.68 for Dense MultiMix. These results validate the qualitative analysis of Figure 3.

4.5 Manifold intrusion analysis

Manifold intrusion [17] can occur when mixed examples are close to classes other than the ones being interpolated in the embedding space. To evaluate for manifold intrusion, we define the intrusion distance (ID) as the minimum distance of a mixed embedding to the clean embeddings of all classes except the ones being interpolated, averaged over a mini-batch: ID ( ˜ Z,Z ) = 1 | ˜ Z | ∑ ˜ z ∈ ˜ Z min z ∈ Z ‖ ˜ z -z ‖ 2 . Here, ˜ Z is the set of mixed embeddings in a mini-batch and Z is the set of clean embeddings from all classes other than the ones being interpolated in the mini-batch. Intuitively, a larger ID ( ˜ Z,Z ) denotes that mixed embeddings in ˜ Z are farther away from the manifold of other classes in Z , thereby preventing manifold intrusion.

Averaged over the training set of CIFAR-100 using Resnet-18, the intrusion distance is 0.46 for Input Mixup, 0.47 for Manifold Mixup, 0.45 for AlignMixup, 0.46 for MultiMix and 0.47 for Dense MultiMix. This is roughly the same for most SoTA mixup methods. This may be due to the fact that true data occupy only a tiny fraction of the embedding space, thus generated mixed examples lie in empty space between class-specific manifolds with high probability. The visualization in Figure 3 indeed shows that the embedding space is sparsely populated, even in two dimensions. This sparsity is expected to grow exponentially in the number of dimensions, which is in the order of 10 3 .

4.6 Ablations

All ablations are performed using R-18 on CIFAR-100. We study the effect of the layer where we interpolate, the number n of generated examples per mini-batch, the number m of examples being interpolated and the Dirichlet parameter α . More ablations are given in the supplementary.

Interpolation layer For MultiMix, we use the entire network as the encoder f θ by default, except for the last fully-connected layer, which we use as classifier g W . Thus, we interpolate embeddings in the deepest layer by default. Here, we study the effect of different decompositions of the network f = g W ◦ f θ , such that interpolation takes place at a different layer. In Figure 4(a), we observe that mixing at the deeper layers of the network significantly improves performance. The same behavior is observed with Dense MultiMix, which validates our default choice.

It is interesting that the authors of input mixup [65] found that convex combinations of three or more examples in the input space with weights from the Dirichlet distribution do not bring further gain. This agrees with the finding of SuperMix [11] for four or more examples. Figure 4(a) suggests that further gain emerges when mixing in deeper layers.

Number n of generated examples per mini-batch This is important since our aim is to increase the amount of data seen by the model, or at least part of the model. We observe from Figure 4(b) that accuracy increases overall with n and saturates for n ≥ 1000 for both variants of MultiMix. The improvement is more pronnounced when m = 2 , which is standard for most mixup methods.

Our best solution, Dense MultiMix, works best at n = 1000 and n = 10 , 000 . We choose n = 1000 as default, given also that the training cost increases with n . The training speed as a function of n is given in the supplementary and is nearly constant for n ≤ 1000 .

Number m of examples being interpolated We vary m between 2 (pairs) and b = 128 (entire mini-batch) by using Λ ′ ∈ R m × n drawn from Dirichlet along with combinations (subsets) over the mini-batch to obtain Λ ∈ R b × n with m nonzero elements per column in (6),(7). We observe in Figure 4(c) that for both MultiMix and Dense MultiMix the performance increases with m . The improvement is more pronnounced when n = 100 , which is similar to the standard setting ( n = b = 128 ) of most mixup methods. Our choice of m = b = 128 brings an improvement of 1-1.8% over m = 2 . We use this as our default setting.

Dirichlet parameter α Our default setting is to draw α uniformly at random from [0 . 5 , 2] for every interpolation vector (column of Λ ). Here we study the effect of a fixed value of α . In Figure 4(d), we observe that the best accuracy comes with α = 1 for most MultiMix variants, corresponding to the uniform distribution over the convex hull of the mini-batch embeddings. However, all measurements are lower than the default α ∼ U [0 . 5 , 2] . For example, from Table 2(a) (CIFAR-100, R-18), Dense MultiMix has accuracy 81.93, compared with 81.59 in Figure 4(d) for α = 1 .

5 Discussion

The take-home message of this work is that, instead of devising smarter and more complex interpolation functions in the input space or intermediate features, it is more beneficial to use MultiMix, even though its interpolation is only linear. In this work, we combine three elements:

Increase the number n of generated mixed examples beyond the mini-batch size b .

Increase the number m of examples being interpolated from m = 2 (pairs) to m = b .

Perform interpolation in the embedding space rather than the input space.

Figure 4 shows that all three elements are important in achieving SoTA performance and removing any one leads to sub-optimal results. We discuss their significance and interdependence here.

Increasing the number n of generated examples The expected risk is defined as an integral over the underlying continuous data distribution. Since that distribution is unknown, the integral is approximated by a finite sum, i.e ., the empirical risk . Abetter approximation is the vicinal risk , where a number of augmented examples is sampled from a distribution in the vicinity of each training example, thus increasing the number of loss terms per training example . Input mixup [65] is inspired by the vicinal risk. However, as a practical implementation, all mixup methods still generate b mixed examples and thus incur b loss terms for a mini-batch of size b . As discussed in section 2, previous works have used more loss terms than b per mini-batch, but not for mixup.

Our hypothesis for the significance of element 1 is that more mixed examples, thus more loss terms by interpolation, provide a better approximation of the expected risk integral. Dense interpolation further increases the number of loss terms, thus further improving the quality of approximation.

Increasing the number m of examples being interpolated As discussed in section 2, previous works, starting from input mixup [65] have attempted to interpolate m > 2 examples in the input space by sampling λ from Dirichlet or other distributions but have found the idea not effective for large m . Our finding is that element 2 becomes effective only by interpolating in the embedding space, that is, element 3.

Interpolating in embedding space Element 3 is originally motivated by Manifold Mixup [51], where 'interpolations in deeper hidden layers capture higher level information [64].' ACAI [4] explicitly studies interpolation in the latent space of an autoencoder to produce a smooth semantic warping effect in data space. This suggests that nearby points in the latent space are semantically similar, which in turn improves representation learning. Mixed examples generated by sampling in the embedding space lie on the learned manifold. We hypothesize that the learned manifold is a good surrogate of the true, unknown data manifold.

Anatural extension of this work is to settings other than supervised classification. A limitation is that it is not straightforward to combine the sampling scheme of MultiMix with complex interpolation methods, unless they are fast to compute in the embedding space.

References

Kumar Abhishek, Colin J Brown, and Ghassan Hamarneh. Multi-sample ζ -mixup: Richer, more realistic synthetic samples from a p -series interpolant. arXiv preprint arXiv:2204.03323 , 2022. 2, 3, 7, 14, 15, 16, 17

Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In CVPR , 2019. 3

James Urquhart Allingham, Florian Wenzel, Zelda E Mariet, Basil Mustafa, Joan Puigcerver, Neil Houlsby, Ghassen Jerfel, Vincent Fortuin, Balaji Lakshminarayanan, Jasper Snoek, et al. Sparse MoEs meet efficient ensembles. arXiv preprint arXiv:2110.03360 , 2021. 3

David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. In ICLR , 2019. 10

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS , 2020. 3

Luigi Carratino, Moustapha Ciss´ e, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regularization. JMLR , 2022. 3

Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, Alan Yuille, and Song Bai. TransMix: Attend to mix for vision transformers. In CVPR , 2022. 1, 6, 7, 15

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020. 3

Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. AutoAugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 , 2018. 6

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. RandAugment: Practical automated data augmentation with a reduced search space. In CVPRW , 2020. 6

Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, and Nasser M. Nasrabadi. SuperMix: Supervising the mixing data augmentation. In CVPR , 2021. 2, 3, 9, 14, 15, 16, 17, 18

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL , 2019. 3

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021. 6

Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In ICML , 2020. 3

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010. 8

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2015. 7

Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regularization. In AAAI , 2019. 9

Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, Adam Pr¨ ugel-Bennett, and Jonathon Hare. FMix: Enhancing mixed sample data augmentation. arXiv preprint arXiv:2002.12047 , 2020. 3

Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for robust prediction. arXiv preprint arXiv:2010.06610 , 2020. 3

Kaiming He, Georgia Gkioxari, Piotr Doll´ ar, and Ross Girshick. Mask R-CNN. In ICCV , 2017. 3

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016. 6

Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781 , 2019. 15, 16, 17

Minui Hong, Jinwoo Choi, and Gunhee Kim. StyleMix: Separating content and style for enhanced data augmentation. In CVPR , 2021. 3, 14, 15, 16, 17, 18

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS , 2020. 3

Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-Mixup: Saliency guided joint mixup with supermodular diversity. In ICLR , 2021. 1, 3, 7, 15, 16, 17, 18

Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle Mix: Exploiting saliency and local statistics for optimal mixup. In ICML , 2020. 1, 3, 7, 14, 15, 16, 17, 18

Jae Myung Kim, Junsuk Choe, Zeynep Akata, and Seong Joon Oh. Keep calm and improve visual feature attribution. In ICCV , 2021. 2, 3

Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR , 2020. 3

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 6

Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, and Stan Z. Li. OpenMixup: Open mixup toolbox and benchmark for visual representation learning. arXiv preprint arXiv:2209.04851 , 2022. 3

Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to-class measure for few-shot learning. In CVPR , 2019. 3

Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. MST: Masked self-supervised transformer for visual representation. In NeurIPS , 2021. 3

Yann Lifchitz, Yannis Avrithis, Sylvaine Picard, and Andrei Bursuc. Dense classification and implanting for few-shot learning. In CVPR , 2019. 3

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV , 2014. 8

Jihao Liu, Boxiao Liu, Hang Zhou, Hongsheng Li, and Yu Liu. TokenMix: Rethinking image mixing for data augmentation in vision transformers. In ECCV , 2022. 3, 6, 7, 15

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. SSD: Single shot multibox detector. In ECCV , 2016. 8

Zicheng Liu, Siyuan Li, Di Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, and Stan Z Li. AutoMix: Unveiling the power of mixup for stronger classifiers. In ECCV , 2022. 3

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018. 7

Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. UMAP: Uniform manifold approximation and projection. The Journal of Open Source Software , 2018. 8

Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In ECCV , 2020. 3

Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In ICCV , 2015. 2, 3

Pedro O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations. In NeurIPS , 2020. 3

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS , 2015. 8

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV , 2015. 6

Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain-augmented meta-learning. In CVPR , 2021. 3, 16, 17

A F M Uddin, Mst. Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bae. SaliencyMix: A saliency guided data augmentation strategy for better regularization. In ICML , 2021. 1, 3, 14, 15, 16, 17, 18

VNVapnik. An overview of statistical learning theory. IEEE Transactions on Neural Networks , 1999. 1

Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. AlignMixup: Improving representation by interpolating aligned features. In CVPR , 2022. 3, 7, 8, 9, 14, 15, 16, 17, 18

Shashanka Venkataramanan, Bill Psomas, Ewa Kijak, Laurent Amsaleg, Konstantinos Karantzalos, and Yannis Avrithis. It takes two to tango: Mixup for deep metric learning.

In ICLR , 2022. 3

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR , 2017. 16, 17

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold Mixup: Better representations by interpolating hidden states. In ICML , 2019. 1, 2, 3, 4, 6, 7, 10, 14, 15, 16, 17, 18

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML , 2020. 8

Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR , 2019. 3

Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR , 2021. 3

Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715 , 2020. 3

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR , 2010. 16

Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In CVPR , 2021. 3

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:2111.09886 , 2021. 3

Yuwen Xiong, Mengye Ren, Wenyuan Zeng, and Raquel Urtasun. Self-supervised representation learning from flow equivariance. In ICCV , 2021. 3

Leon Yao and John Miller. Tiny imagenet classification with convolutional neural networks. Technical report, Standford University, 2015. 6

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015. 16

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In ICCV , 2019. 1, 3, 6, 8, 14, 15, 16, 17, 18

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC , 2016. 6

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV , 2014. 10

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR , 2018. 1, 2, 3, 4, 9, 10, 14, 15, 16, 17, 18

Shaofeng Zhang, Meng Liu, Junchi Yan, Hengrui Zhang, Lingxiao Huang, Xiaokang Yang, and Pinyan Lu. M-Mix: Generating hard negatives via multi-sample mixing for contrastive learning. In SIGKDD , 2022. 3

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI , 2020. 6

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR , 2016. 5, 19

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image bert pre-training with online tokenizer. In ICLR , 2022. 3

Yanzhao Zhou, Yi Zhu, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Weakly supervised instance segmentation using class peak response. In CVPR , 2018. 2, 3

Jianchao Zhu, Liangliang Shi, Junchi Yan, and Hongyuan Zha. AutoMix: Mixup networks for sample interpolation via cooperative barycenter learning. In ECCV , 2020. 3

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel data augmentation method called MultiMix, which aims to go beyond the limitations of existing mixup methods. The authors propose interpolating an arbitrarily large number of examples beyond the mini-batch size in the embedding space, rather than along linear segments between pairs of examples. They also extend the method to sequence data with Dense MultiMix, which densely interpolates features and target labels at each spatial location. The paper demonstrates that their proposed solutions outperform state-of-the-art mixup methods on various benchmarks.\n\n*Contribution and novelty* The scientific contribution of the paper lies in the introduction of MultiMix and Dense MultiMix, which address the limitations of existing mixup methods. The authors propose novel approaches for generating an arbitrary number of interpolated examples beyond the mini-batch size, both in the embedding space and for sequence data. The empirical results, as described in the additional explanations, validate the effectiveness of these methods. Overall, this paper seems to improve the research frontier on data augmentation, but empirically and gives some new intuitive understanding of their results as their method leads to a more uniform distribution of embeddings within each class.  I do not see any clear weaknesses in the methods that may challenge the main claim of this work, but I have raised some issues in the questions section. Finally, since I have not read some of the papers that are cited in this work or other related papers in the literature, I defer the judgment about novelty of the ideas presented here to other reviewers.\n\nSTRENGTHS:\n- Novelty: The paper introduces a new data augmentation method, MultiMix, that extends the concept of mixup beyond the limitations of existing methods. The idea of interpolating in the embedding space and densely interpolating features and labels in sequence data is original and contributes to the field of data augmentation in machine learning.\n\n- Empirical Results: The authors provide empirical evidence to support the effectiveness of their proposed solutions. They demonstrate significant improvements over state-of-the-art mixup methods on multiple benchmarks. The results validate the scientific contribution of the work and highlight its practical relevance.\n\n- High quality of presentation of results: the presentation of the results follows a simple and logical flow, with main contributions and their difference to the previsous works clearly stated. The Methods, notations,  & results are all clearly stated and easy to understand for the reader. \n\n- Intuitions about results: The paper provides some justifications & intuitions about  why their approach works. They do so by analyzing the embedding space. The authors show that the classes become more tightly clustered and uniformly spread over the embedding space, explaining the improved behavior of their solutions. This analysis adds depth to the scientific contribution of the work.\n\nWEAKNESSES:\nSince I have not read some of the papers that are cited in this work or other related papers in the literature, I defer the judgment about novelty of the ideas presented here to other reviewers. Besides that, I have raised a few issues in the questions section as they are not direct criticism of this work.\n\nQUESTIONS:\nI have the following questions from the authors:\n- There is a lot of emphasis in the paper about the importance of number of loss terms and its importance for obtaining the results reported in this paper. However, one could achieve similarly high number of high loss terms by only incorporating several steps of pairwise mixup of the embeddings. This will have both random-ness in selection of pairs and value of $\\lambda$ and will be equivalent to imposing a sparsity on $\\lambda$'s that are sampled in Multimix. Is this point addressed in your evaluations or perhaps some cited works? In other words, what is exactly the benefit of having a dense $\\lambda$ if doing mixup in the embedding space? \n- It seems that the main benefits of the proposed approach is that it imposes new restrictions on the loss such that the solutions have uniform convex hulls per each input class. If authors agree with this statement, can they think of an alternative approach of achieving this? As a hypothetical experiment, can they think of an adding an explicit term in the loss that will encourage this uniform convex hulls, ie, some kind of a uniform convex hull regularisation term?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a generalized form of Manifold mixup by expanding the concept of a pair-wise interpolation method to the multi-sample-based interpolation on the entire convex hull. The proposed methods called MultiMix and Dense MultiMix show the best performance over prior mixup-based methods for the various classification benchmarks. Also, they are empirically shown to have better robustness than prior methods.\n\nSTRENGTHS:\n$\\textbf{Strength 1:}$ The main strength of this paper is the suggestion of the general form of the prior mixup-based methods to cover the entire convex hull which can be formed by a given set of training samples. Because the power of the mixup methods originated from the interpolation of the space, i.e., either input or embedding, in-between different samples, expanding the concept of mixup into the entire convex hull of the given samples are clear and convincing direction.\n\n$\\textbf{Strength 2:}$ The paper is well-written and easy to understand. To be specific, Fig. 1 is simple yet effective to understand the main strength of the proposed method. Also, the formulations in the paper are quite neat and well-organized.\n\n$\\textbf{Strength 3:}$ The experiments are done for various kinds of architectures ranging from the ResNet family to ViT, and benchmarks including CIFAR and ImageNet cases.\n\nWEAKNESSES:\n$\\textbf{Weakness 1:}$ The most crucial weakness of this paper is the novelty of the proposed method. Specifically, I admit that the expansion of the sample interpolation method to the generalized form, i.e., MultiMix, is convincing and the authors show a well-organized formulation of it, but I also feel that the expansion of the concept is quite easy to be anticipated from the prior Mixup methods. For example, in the original Mixup paper, the authors already tried to utilize multiple samples by weighting them with a simplex coefficient in the input space. Although it is reported not to show a further gain, it is hard to find a noticeable difference between the `trial' of the original Mixup paper and the proposed method except for the target space, i.e., input for Mixup vs. embedding space for MultiMix. When focusing on the difference, i.e., mixing on the embedding space, it is natural to ask for the reason why the multi-sample-based mixup becomes effective in the embedding space rather than the input space. However, I cannot find a deep insight into the phenomenon.\n\n$\\textbf{Weakness 2:}$ A more precise comparison in the Related Work part would be beneficial to highlight the significance of the proposed method. To be specific, in Related Work, a bunch of works are grouped into their attributes and briefly mentioned. The authors drop the detailed explanation of the prior works with a suggested survey paper but I am sure that the authors must provide a compact but clear explanation of the related works. As a simple suggestion, mentioning the names of prior mixup methods along with short descriptions would make it more clear. When a set of important baselines including AlignMix, Co-Mixup, PuzzleMix, and $\\zeta$-Mixup are called with names in the Related Work part, readers can easily match the name of the priors appeared in Experiments and the descriptions in Related Work.\n\n$\\textbf{Weakness 3:}$ The metrics in 4.4 Quantitative analysis, i.e., 'Alignment' and 'Uniformity' do not fully represent the quality of the embedding space. These measurements can evaluate the similarity of embedded features of intra-class samples but do not capture the separation between different classes, i.e., the separability of inter-class distributions. \n\n$textbf{Weakness 4:}$ Manifold intrusion is the main challenge of mixup-based methods. However, I cannot find any analysis for the effect of manifold intrusion of MultiMix. The experimental gains empirically show that the effect of manifold intrusion is minimal in practice. However, further discussion is needed on the effect of manifold intrusion of the multi-sample-based mixup approaches. I wonder if the chance of the intrusion might be higher than the pair-wise mixup of prior works.\n\nQUESTIONS:\n$\\textbf{Question 1:}$ Would you provide a further explanation for the reason why the multi-sample mixup becomes when the interpolation is done in the deeper layer? Again, I felt that the main significance of this work is the finding of the effectiveness of multi-sample-based mixup on deeper layers. As mentioned in this paper, the authors of Mixup and SuperMix had tried to interpolate multiple samples but they are limited in the input space which leads to minimal performance gain. If the novelty becomes more clear and strong after the rebuttal, I will change the rating of the paper.\n\n$\\textbf{Question 2: }$ According to Weakness 3, the metrics used in Section 4.4 seem not to capture the separability between different classes. I think that when the 'Alignment' and 'Uniformity' values are divided into the similarities between different classes, they eventually consider the compactness and separability of the embedding. To be specific, when considering class $k$, we can measure the distance between the per-class averaged feature vector of class $k$ and the per-class averaged feature of the nearest interfering class. When we divide `Alignment' with the distance to the nearest different class, then the value will consider both the similarity between intra-class features and the dissimilarity between inter-class features. What happens when we compute the modified 'Alignment' and 'Uniformity'?\n\n$\\textbf{Question 3: }$ Would you provide a computational cost of the proposed methods and the comparison with other baselines? The results of the training speed guarantee that the proposed method does not suffer from further latency in practice, but I wonder if MultiMix and Dense MultiMix show a similar level of computational cost to the baselines.\n\n$\\textbf{Question 4: }$ Would you provide a discussion of the thought of manifold intrusion of the proposed methods? I imagine that the multi-sample-based mixup tries to cover the entire convex hull so it probably causes manifold intrusion more frequently.\n\nLIMITATIONS:\nWeakness 1 and Question 1 are for pointing out the limitation of the proposed work in view of novelty.\n\nAlso, I provide a suggestion for improving Related Work in Weakness 2.\n\nAs better metrics for quantitative analysis in section 4.4, I provide the modified measurements in Weakness 3 and Question 2.\n\nFor emphasizing the computational efficiency of MultiMix and Dense MultiMix, providing the computational costs would be beneficial (as aforementioned in Question 3).\n\nFor the manifold intrusion issue, I described the limitation and the corresponding question in Weakness 4 and Question 4.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to increase the performance and robustness of deep learning model by increasing sample diversity. The authors propose data augmentation techniques which interpolate samples from a convex hull. The experimental evaluations show that the algorithm outperforms existing techniques.\n\n**I have read the rebuttals. Please see the comments below for the discussion.**\n\nSTRENGTHS:\n1. Intuitive idea: the use of convex hull is natural and simple.\n2. Applicability: the algorithm can be generally applied in many settings.\n\nWEAKNESSES:\n1. Presentation: the presentation of the paper needs significant improvement. For example, the abstract should succinctly summarizes the key ideas of the paper.\n2. Novelty: sampling from a convex hull is not novel. The use of Dirichlet distribution has also been done in many other works. Could you please compare your work with the work by Shu et al. ?\n\n\n\n\n\n[1] Shu, Y., Cao, Z., Wang, C., Wang, J., & Long, M. (2021). Open domain generalization with domain-augmented meta-learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9624-9633).\n\nQUESTIONS:\n1. It would be nice to see the analysis on larger/multi-modal models. I am wondering if the gain will be more significant since the sampling diversity will be crucial there. For example, could you conduct experiments on larger Vision Transformer models or language-augmented vision models such as CLIP?\n2. In the ablation study, the performance seems to be sensitive to the hyper-parameters selected, especially since the performance difference from the second best algorithm is between 1-2%. Could you expand more on how the parameters are chosen for each experiment?\n\nLIMITATIONS:\nThe authors discuss the limitation of the algorithm in the concluding paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce MultiMix, an approach aimed at enhancing the Mixup class of techniques, which involve interpolating between input examples and their corresponding targets. This enhancement is achieved through the interpolation of a substantially more training examples, increased number of loss terms for each minibatch, and a more flexible interpolation factors. Further, the authors expand their proposed method with DenseMultiMix, a variation capable of interpolating examples in sequential data. Empirical evidence indicates that the proposed MultiMix and DenseMultiMix not only boost accuracy but also bolster the robustness metric across a diverse set of tasks, including but not limited to classification and object detection.\n\nSTRENGTHS:\nThere's a noticeable diminishing return when interpolating additional input examples in Mixup. This presents a limitation to the advancement of Mixup-related methodologies, which focus on interpolating more training examples. However, the approach introduced in this paper appears to overcome this limitation by implementing interpolation in the embedding space, as opposed to the input space, evidenced by Fig 4c. I regard this as a significant contribution of this paper.\n\nFurthermore, the adaptation of Mixup to sequential data like text has always been frustrated. There are three primary obstacles: 1. The blending of text tokens doesn't carry the same significance as mixed images. 2. How to interpolate along the temporal dimension or the spatial dimension. 3. Discrepancies in length among examples within a mini-batch. The introduced MultiMixup provides solutions to at least the first two challenges. Interpolating within the embedding space prevents the creation of nonsensical input tokens, while MultiMixup along with attention delivers an effective means of interpolating representations in the spatial dimension.\n\nWEAKNESSES:\nThere are several weaknesses in this manualscript, as explaint in the following.\n\n1. The motivation for the suggested method is insufficiently compelling within the manuscript. The only statement regarding to the motivation is ```In this work, we argue that a data augmentation process should increase the data seen by the model, or at least by its last few layers, as much as possible. ```\n\n    This reasoning seems to be superficial as it fails to distinguish itself from the preceding data augmentation methods. Moreover, it doesn't elaborate on how the proposed approach enhances accuracy and robustness. More explicitly, while it's true that any data augmentation method would increase the volume of data processed by the model, not all such methods that do so effectively improve the model performance. Thus, the motivation fails to pinpoint the core of the research problem.\n\n2. Regarding the empirical evaluation, it would be appropriate for the authors to include comparison with methodologies such as AugMix [1]. Similar to the proposed method, AugMix enhances the volume of data points processed by the model through interpolating examples derived from varying combinations of image pre-processors using the Dirichlet distribution. Moreover, it offers a substantial gain in both accuracy and robustness metrics. Also, the divergence loss could potentially counter the scarcity of dense labels.\n\n     [1]: Hendrycks, Dan et al. “AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty.” ICLR 2020.\n\n3. The performance improvement over the established baseline is rather modest. In such a scenario, it would be beneficial if the manuscript offered more profound insights to the community to better understand Mixup-oriented techniques. Such insights could include how the proposed method is connected to existing methods like AugMix, as well as an exploration of why the proposed methodology shows greater promise. For instance, while Section 4.4 and Figure 6 in the appendix provide useful information, they lack a comparison between the proposed method and existing methods, as well as a comparison among different variants of MultiMix as done in Section 4.5. Empirical insights of this would offer more compelling evidence that the proposed methodology is a valuable contribution to the community.\n\n4. A less significant shortcoming pertains to the scope of the empirical evaluation. The batch size is held to be 128 in the domain of image classification. Understanding the robustness of the proposed method to batch size selection is crucial, especially as we move towards models with more # parameters where batch size tends to decrease. Specifically, when the batch size is small, MultiMix should offer more advantages over standard Mixup due to its ability to draw from a larger pool of interpolated examples. Another dimension worth exploring is the application of DenseMultiMix in the realm of text data. The manuscript would significantly benefit from showing evidence that the proposed method can address some of the challenges of extending Mixup-style techniques to the domain of text. I understand that interpolating dense text labels is a non-trivial task, preliminary studies on tasks such as text classification could suffice.\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nOne limitation as mentioned above is the application of DenseMultiMix in the realm of text data.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}