$k$-Mixup Regularization for Deep Learning via Optimal Transport
Authors: Kristjan Greenewald
Anming Gu
Mikhail Yurochkin
Justin Solomon
Edward Chien
Venue: Accepted by TMLR ()

Abstract: Mixup is a popular regularization technique for training deep neural networks that improves generalization and increases robustness to certain distribution shifts. It perturbs input training data in the direction of other randomly-chosen instances in the training set. To
better leverage the structure of the data, we extend mixup in a simple, broadly applicable way to $k$-mixup, which perturbs $k$-batches of training points in the direction of other $k$-batches. The perturbation is done with displacement interpolation, i.e. interpolation under
the Wasserstein metric. We demonstrate theoretically and in simulations that $k$-mixup preserves cluster and manifold structures, and we extend theory studying the efficacy of standard mixup to the $k$-mixup case. Our empirical results show that training with $k$-mixup
further improves generalization and robustness across several network architectures and benchmark datasets of differing modalities. For the wide variety of real datasets considered, the performance gains of $k$-mixup over standard mixup are similar to or larger than the
gains of mixup itself over standard ERM after hyperparameter optimization. In several instances, in fact, $k$-mixup achieves gains in settings where standard mixup has negligible to zero improvement over ERM.

Published in Transactions on Machine Learning Research (09/2023)
k- Mixux Regularization for Deep Learning via Optimal Transport
Kristjan Greenewald
MIT-IBM Watson AT Lab, IBM Research
Anming Gu
agu2002 edu
Boston Universiby
Mikhail Yurochkin
mikhail sunthall.arrcckii@@Bmm.coom
MIT-IBM Watson AT Lab, IBM Research
Justin Solomon
jsolomon&@iiz edu
Massachusetls Institute of Technology
Edward Chien
edchien @bu. edu
Boston University
Reviewed on OpenReview: https: // openreveew. net/ forum? id= W0gPKSS004
Abstract
Mixup popular regularization technique for training deep neural networks that improves generalization and increases robustness to certain distribution shifts. It perturbs input training data in the direction of other randomly-chosen instances in the training set. To better leverage the structure of the data, we extend mixup in simple, broadly applicable way to k-mirup, which perturbs k-batches of training points in the direction of other k- batches. The perturbation is done with displacement interpolation, i.c. interpolation under the Wasserstein metric. We demonstrate theoretically and in simulations that k-mixup preserves cluster and manifold structures, and we extend theory studying the efficacy of standard mixup to the k-mixup case. Our empirical results show that training with k-mixup further improves generalization and robustness across several network architectures and benchmark datasets of differing modalities. For the wide variety of real datasets considered, the performance gains of k-mixup over standard mixup are similar to or larger than the gains of mixup itself over standard ERM after hyperparameter optimization. In several instances, in fact, k-mixup achieves gains in settings where standard mixup has negligible to zero improvement over ERM.
1 Introduction
Standard mixup (Zhang et al. 2018 data augmentation approach that trains models on weighted averages of random pairs of training points. Averaging weights are typically drawn from beta distribution B(a, a),, with parameter a such that the generated training set is vicinal, i.e., does not stray too far from the original dataset. Perturbations generated by mixup may be in the direction of any other data point instead of being informed by local distributional structure. As shown in Figure 1 this property is key weakness of mixup that can lead to poor regularization when distributions are clustered or supported on a manifold. With larger a, the procedure can result in averaged training points with incorrect labels in other clusters or in locations that stray far from the data manifold.
To address these issues, we present k-mirup, which averages random pairs of sets of l samples from the training dataset. This averaging is done using optimal transport, with displacement interpolation. The sets

---

Published in Transactions on Machine Learning Research (09/2023)
No mixup
1-mixup
32 22-miixu
Figure 1: Outputs of a fully-connected network trained three synthetic datasets for binary classification, with no mixup (ERM), 1-mixup, and 32-mixup regularization (a = =). Note that ERM under-regularid (visible through the jagged boundaries), 1-mixup over-regularizes (visible through over-smoothing). and that 32-mixup better captures local structure (visible through less blur, increased contrast) while retaining reasonable smoothing between the classes.
of samples are viewed as discrete distributions and are averaged as distributions in geometric sense. If k = 1, recover standard mixup regularization. Figures [ and L illustrate how k-mixup produces perturbed training datasets that better match the global cluster manifold support structure of the original training dataset. The constraints of optimal transport are crucial, as for instance nearest-neighhors approach would avoid the cross-cluster matches necessary for smoothing Figure illustrates the distribution of possible matchings for a sample point and shows non-zero likelihood for these cross-cluster matches. In Section [ we provide empirical results that. justify the above intuition. The resulting method is easy to implement, computationally cheap, and versatile. Our contributions are as follows.
Empirical results:
We show improved generalization results on standard benchmark datasets showing k-mixup with k > consistently improves on standard mixup, where a is optimized for both methods. The improvements are consistently similar in magnitude or larger than those of 1-mixup over basic ERM.
On image datasets, heuristic of k = 16 outperforms 1-mixup in nearly all cases.
We show that k-mixup significantly improves robustness to certain distribution shifts (additive noise and adversarial samples) over 1-mixup and ERM.
Theoretical results:
We argue that as k increases, the interpolated samples are more and more likely to remain within the data manifold (Section 3.1
In the clustered setting, we provide an argument that shows inter-cluster regularization interpo- lates nearest points and better smooths interpolation of labels (Section 3.2
We extend the theoretical analysis of Zhang et al. (2020) and Carratino et al. (020) to our k-mixup setting, showing that leverages local data distribution structure (D; of Eq. to make more informed regularizations (Section
Related works. We tackle issues noted in the papers adaptive mixup (uuo et al. 2019) and manifold mixup (Verma et al. 2018). The first refers to the problem as "manifold intrusion" and socks to address it by training data point-specific weights a and considering convex combinations of more than 2 points. Manifold mixup deals with the problem by relying on the network to parameterize the data manifold, interpolating in the hidden layers of the network. We show Section 5 that k-mixup can performed in hidden layers boost performance of manifold mixup. related approach that of GAN-mixup (Sohn al.) 2020), which trains a conditional GAN and uses to generate data points between different data manifolds. The approaches above require training additional networks and are far more complex than our k-mixup method.
TTo this, note that because neireet- be many-to-one matching, nearly all matches would be intra-cluster between points of the class and thus provide few/no interpolated labels, particularly missing any interpolations the voids between classes where interpolation most important. As additional support, we Monte Carlo trials of the CIFAR-10 experiment below with discussion presented supplement Section J with analogue Figure for nengarest-neighbors strategy.
16 -6-nnerest--eel neighesss strategy. failed outperform even 1-mixup (0.09% worse). Further
2

---

Published in Transactions on Machine Learning Research (09/2023)
1: 1-mixup
k = 32: k-mixup
Figure 2:
Optimal transport couplings and vicinal datasets for A == (left) and k = 32 (right) in 3 simple datasets. In the bottom row, was used to generate vicinal datasets of size 512.
The recent local mixup method (Baena al.) 2022) uses a distance-based approach for modifying mixup. This method retains the random matching strategy of mixup, but scales the contribution of vicinal point to the objective function according to its distance from the original training point. As noted previously, such methods that employ random matching will fail to provide substantive data smoothing for the output function between clusters or high-density regions in the training data.
PuzzleMix (Kim et al. 2020) also combines optimal transport ideas with mixup, extending CutMix Yun et al, 2019) to combine pairs of images. PuzzleMix uses transport to shift saliency regions of images, producing meaningful combinations of input training data. Their use of optimal transport fundamentally different from ours and does not generalize to non-image data, There several other image-domain specific works that are in this vein, including CoMix (Kim et al. 2021 Automix (Liu et al. 2021 and Stackmix (Chen
ct al. 2021
Performing optimal transport between empirical samples of distributions has been considered in studies of the sample compleziity of Wasserstein distances (e.g. Weed & Bach (2119)) Unlike most settings, in our application the underlying source and target distributions are the same; the thcoretical investigation of a generalization of variance called k-variance by /Solomon et al. (2020) considers similar setting. In other works, transport between empirical samples has been dubbed minibatch optimal transport and has been used
in generative models (Genevay et al. 2018 Fatras et al. 2020 and domain adaptation DDmoddraa et al.
2018 Fatras et al. 2021
2 Generalizing Mixup
Standard mixup. Mixup uses a training datasel of feature-targel pairs ((x,, N where the target y is a one-hot vector for classification. Weighted averages of training points construct a vicinal dataset:
(2)) (Azz + A)æ5, Ay. + A)8y).
is sampled from a beta distribution, 3(a, a): with parameter a > 0 usually small so that the averages are near an endpoint. Using this vicinal dataset, empirical risk minimization (ERM) becomes:
Emir (f) = Baga [e (6)),
where i,j ~ ~111,111111.. NY, a), I proposed feature-target map, and E a loss function. Effectively, one trains on datasets formed by averaging random pairs of training points. As the training points are randomly selected, this construction makes it likely that the vicinal data points may not reflect the local structure of the dataset, in the clustered manifold-support setting.
k-mixup. To generalize mixup, we sample two random subsets of & training points {(x7 v/ Pa and ((ff of ) For compactness, let x {a7) and y" [b
1 denote the feature and target sets
3

---

Published in Transactions on Machine Learning Research (09/2023)
Figure 3: Locally-infonnn matching distributions D, for k 32, for randomly selected point (see Equation for explicit definition). These distributions reflect local manifold and cluster structure. Note that a neerestss approach would not provide cross-cluster matchings (see Figure 7 in Appendix J
(and likewise for 5). A weighted average of these subsets is formed with displacement interpolation and used as a vicinal training set. This concept is from optimal transport (see, e.g., Santambrogio (2015) and considers (x7, ,/²) and (5) as uniform discrete distributions fi7, fā over their supports. In this setting, the optimal transport problem becomes a linear assignment problem (Peyré & Cuturi 2019 The optimal
map is described by a permutation a E S₁ minimizing the cost:
k
W
1
k
Here, a be found efficiently using the Hungarian algorithm (Bertsimas & Tsitsiklis, 1997.. Figure gives intuition for this identification. When compared to the random matching used by standard mixup, pairing is more likely to match nearby points and to make matchings that better respect local daructur--especiall) especially by having nontrivial probability for cross-cluster matches between nearby points on the two clusters (see Figure 3
Given a permutation a and weight X, the displacement interpolation between (x y and (x5, y is:
DI*((> .y (a () = { (x7 v2) + +1 (1- U 1=1
k
As in standard mixup, draw X B(a, a). For the loss function, consider sampling k-subsets of the N samples al random, which can mathematically describe choosing 7,5 u(,..... (2) for which
(2) (2) are the possible subsets. 2 This yields an expected loss
{((=),0) DDMD)
Ey (f) = Ex,c, [LL((Isx (,),)))),,))))))))))(((())))(( DIXV", ,)))]
The localized nature of the matchings makes it more likely that the averaged labels will smoothly interpolate over the decision boundaries. A consequence is that k-mixup is robust to higher values of since it is no longer necessary to keep À close to 0 or 1 to avoid erroneous labels. This can be seen in our empirical results in Section and theoretical analysis in Section
k; a, and Root Mean Squared Perturbation Distance. When a is kept fixed and is increased, the perturbations become more local, and the distance of matchings (and thus perturbations) decreases. We found that more sensible comparisons across values of & can be obtained by increasing in concert with k 7 so that the average perturbation's squared distance remains constant-bearte squared distance values may not be achievable for high values of k. In effect, this is viewing the root mean squared perturbation distance as the parameter, instead of a. Computation details are found in Section 5.1 Throughout training, this tuned a is kept fixed. We typically pick an for A = 1, get the associated root. mean squared distance (E), and increase a as increases to maintain the fixed value of &.
in mind that large
2These possible subsets need not he enumerated since optimize the loss with stochastic gradient descent. Note that this adjustment is of not necessary for parameter tuning practice.
4

---

Published in Transactions on Machine Learning Research (09/2023)
S
(c(
(S)
<< Rs
Rs
6> Rs
Figure 4: Injectivity radius example. Various e-neighborhoods, (S),, have been illustrated for curve 8 C TR2. Intuitively, the topology of (S) no longer reflects that of S when Rs, the injectivity radius. To be precise, (S) is no longer homotopy equivalent to (see Hatcher (2000) for definition).
Pscudocode and Computational Complexity. We emphasize the speed and simplicity of our method. We have included brief PyTorch pseudocode in Section 5.1 below and note that with CIFAR-10 and k = 32, the of k-mixup added about one second per epoch GPU, Section 5.1 also contains more extended analysis of computational cost, showing that there is little computational overhead to incorporating k-mixup in training compared to ERM alone.
3 Manifold & Cluster Structure Preservation
Using an optimal coupling for producing vicinal data points makes it likely that vicinal data points reflect. local dataset structure. Below we argue that as increases, the vicinal couplings will preserve manifold support, preserve cluster structure, and interpolate labels between clusters.
3.1 Manifold support
Suppose our training data drawn from a distribution a on a d-dimensional embedded submanifold S X' y C MRM, where X' and y denote feature and target spaces. We define an injectivity radius: Definition 3.1 (Injectivity radius). Let B,( {p E X' d/ E} denote the e-neighborhood of S where d(p.S) S) is the Euclidean distance from p to S. Define the injectivity radius Rs of S to be the infiruum of the c's for which (S) is not homotopy equivalent to S.
As S is embedded, (S) is homotopy equivalent to S for small enough e, Rs 0. Essentially, Definition 3.1 grows e-neighborhood until the boundary intersects itself. See Figure for a schematic example that. illustrates this. We have the following:
Proposition 3.2. For a data distribution supported on an embedded submanifold S with injectivity radius Rs, with high probability any constant fraction 1 - 5 (for any fixed 8 E (0,1)) of the interpolated samples induced by k-mirup will remain within (,) for large enough.
The proof of this proposition in supplement Section [D] Hence, for large enough k, the interpolation induced by optimal transport will approximately preserve the manifold support structure. While the theory requires high & to achieve tight bound, our empirical evaluations show good performance in the small i regime. Some schematic examples are shown in Figures and
3.2 Clustered distributions
With clustered data distribution, preserve global structure by including many within-cluster and inter-cluster matches, where inter-cluster matches correspond (approximately) to the nearest points across clusters. This contrasts with 1-mixup, which, when the number of clusters is large, is biased towards primarily inter-cluster matches and does not seek to provide any structure to these random matches. These approximate characterizations of k-mixup are achieved exactly as + 20, as argued below.
To make precise the notion of clustered distribution, we adopt the (m, A-cchuterable definition used by Weed & Bach (2019): Solomon et al. (2020) In particular, distribution μ is (m, A)-cuusteralle if supp(μ)
5

---

Published in Transactions on Machine Learning Research (09/2023)
lies in the union of m balls of radius at most A. Now, if our training samples (2,, V1) are sampled from such distribution, where the clusters are sulficiently separated, then optimal transport will prioritize intra-cluster matchings over cross-cluster matchings.
Lemma 3.3. Drow two botches of samples {pr}i= and {qi from a (m, -lcltstrraalle distribution, where the distance between any pair of covering balls is at least 2A. If r, and s denote the number of samples in cluster in batch and 2, respeclively, then the optimal transport matching will have 2 E. -8:] cross-cluster matchings.
The proof of the above (supplement Section involves the pigeonhole principle and basic geometry. We also argue that the fraction of cross-cluster identifications approaches zero as k + oo and characterize the rate of decrease. The proof (supplement Section F follows via Jensen's inequality:
Theorem 3.4. Given the setting of Lemma 3.3, with probability masses Pi.....Pm. and two batches of size k matched with optimal transport, the expected fraction of cross-cluster identifications is O E (.)
Note that the absolute number of cross-cluster matches still increases as k increases, providing more information in the voids between clusters. We emphasize that the (m, A) assumption corresponds to setting where clusters are well-scparated such that cross-cluster identifications are as unlikely as possible. Hence, in real datascts, the fraction of cross-cluster identifications will be larger than indicated in Theorem 3.4 Finally, we show that these cross-cluster matches are of length close to the distance between the clusters with high probability (proved in supplement Section G). i.e., the endpoints of the match lie in the parts each cluster closest to the other cluster. This rests upon the fact that we are considoring the W2 cost, for which small improvements to long distance matchings yield large cost reductions in squared Euclidean distance.
Theorem 3.5. Suppose densily p has support on disjoinl compact sels (clusters) A, B whose boundaries are smooth, where p > O throughout A and B. Let D be the Fuclidean distance between A and B, and let RA. Ra be the radūi of A, B respectively. Define A, to be the subset of set A that is less than (1 + E)D distance from B, and define B, similarly. Consider two batches of size k drown from and matched with optimal transport. Then, for l large enough, with high probability all cross-cluster matches will have an endpoint each in A, and Be, where max(RA.Rn..
D
Theorem 3.5 implies that for large k; the vicinal distribution created by sampling along the cross-cluster matches will almost entirely lie in the voids between the clusters. If the clusters correspond to different classes, this will directly encourage the learned model to smoothly interpolate between the class labels as one transitions across the void between clusters. This is in contrast. to the random matches of 1-mixup, which create vicinal distributions that can span a line crossing any part of the space without regard for intervening clusters. The behaviors noted by Theorems 3.4 and 3.5 are visualized Figures] [ 2 and 3 showing that k-mixup provides smooth interpolation between clusters, and strengthens label preservation within clusters.
4 Regularization Expansions
Two recent works analyze the efficacy of 1-mixup perturbatively (Zhang et al. 2020 Carratino et al. 2020 Both consider quadratic Taylor series expansions about the training set a simple transformation of it, and they characterize the regularization terms that arise in terms of label and Lipschitz smoothing. We adapt these expansions to k-mixup and show that the resulting regularization is more locally informed via the optimal transport coupling.
In both works, perturbations are sampled from a globally informed distribution, based upon all other samples in the data distribution. In k-mixup, these distributions are defined by the optimal transport couplings. Given a training point 2i, we consider all k-samplings 7 that might contain it, and all possible k-samplings Ç that it may couple to. A locally-informor distribution is the following:
(E
D.
Σ
Sone(z))
(1)
6

---

Published in Transactions on Machine Learning Research (09/2023)
where dng denotes the optimal coupling between k-samplings ? and Ç. This distribution will be more heavily weighted on points that I; is often matched with. An illustration of this distribution for a randomly selected point in our synthetic examples is visible in Figure 3 We use "locally-inormmed" in the sense of upweighting of points closer to the point of interest that are likely to be matched to it by k-mixup.
Rn), and the perturba- tions in the regularization terms are sampled from D. We generalize their characterization to k-mixup, with D replaced by D.. Focusing the binary classification problem for simplicity, we assume loss of the form f(f(=), h(f(z)) y f(z) for some twice differentiable h and f. This broad class of losses includes the cross-cntropy for neural networks and all losses arising from generalized lincar models.
Zhang et al. (2020) expand about the features the training dataset Dx ==
Theorem 4.1. Assuming loss E as above, the k-mirup loss can be written as:
ET (ʃ) Estd
ΣR, R3 Exxpla+1a) -1) A]} j=1
where lim - (aa) 0, Estd denotes the standard ERM loss, and the three R2 regularization terms are:
R1 Ex-Bla+1a)11- x
(N(((())) - "/V xx
E Bx-B(o+1a))111 2)2]
R2
f((f(=)) [(r x()(r 2;)()=;;)))))) (V))))((((
2n
E Bx 111-)
N
R3 =
(M((f(x)) - y;) [(r V2) - =x,)1
2n
A proof is given in Section H of the supplement and follows from some algebraic rearrangement and a Taylor expansion in terms of 1 1 X. The higher-order terms are captured by Ex~S(o+1.o)) 1,0))(1 A) A)]- Eatd represents the constant term in this expansion, while the regularization terms R. represent the linear and quadratic terms. These effectively regularize fx(x)) and (x;) with respect to local perturbations r 2 sampled from D. ensuring that our regularizations are locally-informed. In other words, the regularization terms vary over the support of the dataset, at each point penalizing the characteristics of the locally-informed distribution rather than those of the global distribution. This allows the regularization to adapt better to local data (e.g. manifold) structure. For example, R₂ and R3 penalize having large gradients and Hessians respectively along the directions of significant variance of the distribution D; of points x; is likely to be matched to. When = 1, this Dj will not be locally informed, and will instead effectively be a global variance measure. As incrcases, the D; will instcad be dominated by matches to nearby clusters, better capturing the smoothing needs in the immediate vicinity of x,, Notably, the expansion is in the feature space alone, yielding theoretical results in the case of 1-mixup on generalization and adversarial robustness.
An alternative approach by Carratino et al. 2020) characterizes mixup as combination of reversion to mean followed by random perturbation. In supplement Section | we generalize their result to k-mixup via a locally-informed mean and covariance.
5
Implementation and Experiments
5.1 k-mixup Implementation & Computational Cost
Figure shows pseudocode for our implementation of one epoch of k-mixup training, in the style of Figure of Zhang et al. (2018) Also provided in Algorithm ] is the procedure used in the experiments to adjust a as k increases (denoted ).g) While cannot. be evaluated in closed form as function of a, since & monotonically increases with a it is sufficient to iteratively increase ag until the desired is reached. The while loop involves only computing the empirical expectation of a simple function of scalar beta-distribated random variable and is therefore fast to compute.
4Python code for applying k-mixup CIFAR10 can be found at httpss//gggtaaa...c////aaaaaaaaa
7

---

Published in Transactions on Machine Learning Research (09/2023)
y1, y2 should be one-hot vectors
for (x1, y1), (x2, y2) in zip(loader1, loader2):
idx numpy.zeros l_lkeekkiy (eepy--eros__kke((11 for i range (x1 (xt. // k):
cost scipy.spatial.
distance_ destance_matrixcae (xi k: (i+1) k], x2[i * k: (1+1) * x])
ix = scipy. opt.mize........... linearr____r sum_ assignment
sww.arsssssssgnnttttommmmmm (cost)
idx[i k: (i+1) k] = ix + i
x2 x2 [idx]
y2 y2 [idx]
1am numpy. random....dm....m beta (alpha, alpha)
Variable (1amm xi (1 lam) x2) y Variable (lam y1 (1 lam) * y2)
optimizer ())dd)
loss (net y). backward
aptimizer.step(
Figure k-mixup implementation
Algorithm Choosing og to maintain constant
Require: Chosen an, desired k trials N, constant c >> 1, Chreshold dalasel of interest.
1: Using N trials, compute & as the empirical average of the squared distance between two random points in the dataset.
Using N/k trials, compute & as the empirical average of the squared Wasscrstein-2 distance between random k-samples drawn from the dataset. Using N trials, compute empirical average Àι of (Ai, (1 A1)²) where d ((a),
4: & + E
a, Ås = Å 6: while d&Ea < and as < do
5: Initialize
7: O'k +- agc
Using N trials, compute empirical average d of min(A --x)))) where dn B(ax, ag).
end while
10: Output ag.
Computational cost. While the cost of the Hungarian algorithm is O(k3), it provides k data points for regularization, yielding an amortized O(k²) complexity per data point. For the smaller values of that we empirically consider, approximate Sinkhorn- Hungarian cost remains small relative to that of gradient computation (c.g. for CIFAR-10 and k: 32, the Hungarian algorithm costs 0.69 seconds per epoch in total). Computing the distance matrix input to the OT matching costs (kk2d) where d is dimension, yielding amortized O(kd) complexity per data point. With the high dimensionality of CIFAR, naive GPU implementation of this step of the computation adds about 0.5 seconds per epoch. Note that, on our hardware, the overall cost of an epoch is greater than 30 seconds. Moreover, training convergence speed is unaffected in terms of epoch count, unlike in manifold mixup Verma et al. 2018), which in our experience converges slower and has larger computational overhead. Overall, therefore, there is little computational downside to generalizing 1-mixup k-mixup, and, as we will see, the potential for performance gains.
methods (Cuturi 2013 are slower in practice, and the
5.2 Empirical Results
The simplicity of our method allows us to test the efficacy of k-mixup over a wide range datasets and domains: standard benchmark image datasots, UCI tabular datasots, and a speech datasot; employing a variety of fully connected and convolutional neural network architectures. Across these experiments we find that k-mixup for l consistently improves upon 1-mixup after hyperparameter optimization. The gains
8

---

Published in Transactions on Machine Learning Research (09/2023)
are generally on par with the improvements of 1-mixup over no-mixup (ERM). Interestingly, when the mean perturbation distances (squared) are constrained to be at a fixed level 5² k-mixup for k > still usually improves upon 1-mixup, with the improvement especially stark for larger perturbation sizes. In the tables below, 5 is used to denote the root mean square of the perturbation distances, and a denotes the a used in the k= = 1 case to achieve this g. We show here sweeps over and &, choosing h and & in practice is discussed in Supplement Section A
Unless otherwise stated, our training is done over 200 epochs via standard SGD optimizer, with learning rate 0.1 decreased at epochs 100 and 150, momentun 0.9, and weight decay 10-4, Note that throughout, we focus on comparing ERM, 1-mixup, and k-mixup on different architectures, rather than attempting to achieve a new state of the art on these benchmarks.
Image datasets. Our most extensive testing was done on image datasets, given their availability and the efficacy of neural networks in this application domain. In Table [ we show our summarizod error rates across various benchmark datasets and network architectures (all results are averaged over 20 random (rials) For each combination of dataset and architecture, report the improvement of k-mixxp over 1-mixup, allowing for optimization over relevant parameters. In this table, hyperparameter optimization refers to optimizing over a discrete set of the designated hyperparameter values listed in the detailed hyperparameter result tables shown below for each dataset and setting. As can be seen, the improvement is on par with that of 1-mixup over ERM. We also show the performance of k = 16 (a heuristic choice for k), which performed well across all experiments, outperforming 1-mixup in nearly all instances. This = 16-mixup performance is notable as it only requires optimizing a single hyperparameter a / E, the same hyperparameter used for 1-mixup.
Table 1: Summary of image dataset error rate results (in percent, all results averaged over 20 trials). Note that for each setting, k-mixup with hyperparameter optimization outperforms both ERM and optimized 1-mixup. Wc also show 16-mixup as a heuristic that only involves optimizing a / E, i.c. the hyperparameter space is the same as 1-mixup. This heuristic outperforms ERM and 1-mixup in all but one instance, where it matches the performance of 1-mixup. Note that, average, the improvement of k-mixup over 1-mixup tends to be similar in magnitude to the improvement of 1-mixup over ERM.
Dataset (Confidence)
Architecture
ERM 1-mixup (E opt.) K-Mixup ( opt.)
16-Mixup (heuristic, & opt.)
MNIST (1.02)
LeNet
0.95
0.76
0.66
0.74
CIFAR 10 (t.03)
Re
5.6
4.18
4.02
4.02
(+.03)
DenseNet- 190
3.70
3.29
2.85
2.85
+
WideResNNet-101
11.6
11.53
11.25
11.25
CIFAR- 100
(1.05)
MenseNet-BC-190
18.91
18.9
18.31
18.31
SVHN (1.02)
ResNet18 18
3.37
2.93
2.78
2.93
Tiny ImageNet (+.06)
ResNet18
38.50
37.58
35.67
35.67
Full parameter sweeps for all settings shown in Ta- ble D are in Appendix [ with two presented below that are indicative of the general pattern that we see as we vary a / & and k. In Table 2 we show results for MNIST (LeCun & Cortes 2010 with a slightly modified LeNet architecture to accommodate grayscale images, and for Tiny ImageNet with a Pre- Act ResNet-18 architecture in Zhang et al. (2018). Each table entry is averaged over 20 Monte Carlo trials. Here and throughout the remainder of the paper, error bars are reported for the performance results | In the MNIST results, for each & the best generalization performance is for some > 1, iee.
Table 2: Results for MNIST with LeNet architecture (no mixup (ERM) error: 0.95%), averaged over 20 trials (#.02 confidence on test error). Note the best k-mixup beats the best 1-mixup by 0.1 0.1%; on the same order as the 0.19% improvement of 1-mixup over ERM.
a
.05
a=.1
a=.2
2
a=.5
a=1 1
a 10
a 100
k
5
=
0.5
g = 0.6
g 0.8
g = 1.2
= 1.4
g = 2.1
2.2
T
0.79
0.79
0.76
0.86
0.83
1.05
1.26
2
0.85
0.79
0.90
0.76
0.83
0.94
1.14
0.91
0.80
0.83
0.81
0.85
0.89
0.95
0.80
0.81
0.78
0.79
0.78
0.81
0.83
16
0.77
0.82
0.80
0.75
0.80
0.74
0.75
32
0.79
0.75
0.77
0.79
0.80
0.76
0.71
64
0.80
0.82
0.77
0.78
0.78
0.66
0.71
5 As result, Table summarizes results from training 3680 neural uetwork models in total. 6 AAchieved with =0, 0, ic. equivalent to ERM. Thhese error bars are the one standard deviation of the Monte Carlo average, where for brevity we report only the worst such variation over the elements of the corresponding results table.
9

---

Published in Transactions on Machine Learning Research (09/2023)
70-
90-
80
92
casiicctoo
cassiicatoon
70-
% Caeeitaates
82-
% Cassitatte
80-
50-
50-
62-
40-
60
53
k-1
30-
100 Booch
150
200
100
150
200
100
150
200
100
150
200
Esoch
Ecech
Epoch
(a) Test acc., a= 1. (b) Train acc., a = 1.
(c) Test acc,, a = 10.
(d) Train acc., a 10.
Figure 6: Training convergence of k = and = 32-mixup on CIFAR-10, averaged over 20 random trials. Both train roughly the same rate (k = 32 slightly faster), the train accuracy discrepancy is due to the more class-accurate vicinal training distribution created by higher k-mixup.
k-mixup outperforms 1-mixup for all perturbation sizes. The lowest crror rate overall is achieved with k = and & = 2.1: 0.66%, an improvement of 0.1% over the best 1-mixup and an improvement of 0.29% over ERM.
For Tiny ImageNet, we again see that for each / &, the best generalization performance is for some k 1 (note that the & are larger for Tiny ImageNct than MNIST duc to differences in normalization and size of the images). The lowest error rate overall is achieved with k = 16 and & = 1000: 35.67%, and improvement of 1.91% over the best 1-mixup and an improvement of 2.83% over ERM. In both datasets, we see that for each perturbation size value E, the best generalization performance is for some k >> 1, but that the positive effect of increasing & is seen most clearly for larger &. This strongly supports the notion that k-mixup is significantly more effective than 1-mixup at designing an effective vicinal distribution for larger perturbation budgets. The lowest overall error rates are achieved for relatively high & and high k.
Additionally, we show training curves (performance as a. function of epoch) demonstrating that the use of k-mixup does not affect training convergence. These are shown for ResNet-18 on CIFAR-10 in Figure which is indicative of what is seen across all our experiments. The training speeds (test accuracy in terms of epoch) for 1-mixup and 32-mixup show no loss of convergence speed with k = 32-mixup, with (if anything) k = 32 showing a slight edge. The discontimuity at epoch 100 is due to our reduction of the learning rale at epochs 100 and 150 to aid con- vergence (used throughout our image experiments). The train accuracy shows a similar convergence pro- file between 1- and 32-mixup; the difference in absolute accuracy here (and the reason it is less than the test accuracy) is because the training distribution is the mixup-modified vicinal distribution. The curve for k = 32 is higher, especially for a = 10, because the induced vicinal distribution and labels are more consistent with the true distribution, due to the better matches from optimal transport. The large improvement in train accuracy remarkable given the high dimension of the CIFAR-10 data space, since it indicates that, k = 32-mixup is able to find significantly more consistent matches than k 1-mixup.
Table 3: Results for Tiny ImageNet with ResNet18 ar- chitecture (no mixup (FRM) error: 38.50%), averaged over 20 trials (#.06 confidence on test error). Note the best k-mixup beats the best 1-mixup by 1.91%; better than the 0.92% improvement of 1-mixup over ERM.
a=
0=.2
55
a=1
10
9.7
{
=
[
=
{
{
==0
38.47
38.26
37.79
37.61
37.58
2
38.48
38.06
37.89
37.43
37.00
4
38.51
38.13
37.67
37.44
36.21
8
38.42
38.05
37.67
37.19
35.86
16
38.47
38.10
37.60
37.28
35.67
32
38.45
38.04
37.60
37.29
35.88
UCI datasets. To demonstrate efficacy non-image data, we tried k-mixup UCI datasets Duua & Graff 2017 of varying size and dimension: Iris (150 instances, dim. 4), Breast Cancer Wisconsin-Diagnostia (569 instances, dim. 30), Abalone (4177 instances, dim. 8), Arrhythmia (452 instances, dim. 279), and Phishing (11055 instances, dim. 30). For Iris, we used a 3-ayyr net work with 120 and 84 hidden units; for Breast Cancer, Abalone, and Phishing, we used a 4-layer network with 120, 120, and 84 hidden units; and lastly, for Arrhythmia we used a 5-layer network with 120, 120, 36, and 84 hidden units. For these datasets we used a learning rate of 0.005 instead of 0.1. Each entry is averaged over 20 Monte Carlo trials. Test error rate is shown in Figure 4 k-mixup improves over 1-mixup in each case, although for Arrhythmia mixup
10

---

Published in Transactions on Machine Learning Research (09/2023)
outperforms both. In these small datasets, the best performance seems to be achieved with relatively small a (0.05, 0.1) and larger & (4 or greater).
Table 4: Test error on UCI datasets using fully connected networks, averaged over 20 random trials.
Datasct
a
=
1
k
k
=
4
8
16
Abalone
0.05
.07
71.37
71.32
71.23
71.54
71.47
(ERM: 72.32)
0.1
.09
71.32
71.41
71.82
71.05
71.07
Datasct
E
k:
k:
4
k
8
16
1.0
.20
71.85 71.56
71.26
71.29
70.90
Tris
0.05
.07
6.70
4.20
3.30
2.50
3.70
(FRM: 4.10)
0.1
_10
6.00
2.90
2.70
3.50
3.60
Arrhythmia
0.05
13.97
35.15
34.84
33.88
35.16
34.97
1.0 .22
7.00
3.90
2.90
3.80
2.90
(FRM: 32.06)
0.1
17.83
34.14
36. 13
34.29
35.64
33.39
1.0
42.28
34.25
35.51
33.78
33.55
33.64
Phishing (FRM: 3.43)
0.05 .28
3.30
3.14
3.05
3.17
3.06
0.1 138
3.30
3.36
3.35
3.34
3.26
Cancer
0.05
19
10.32
9.08
9.52
8.63
9.35
1.0 .85
4.69
4.55
4.27
4.18
4.20
(FRM: 11.25)
0.1
.26
10.42
9.75
9.92
9.14
10.25
1.0
.59
12.31
10.83
10.23
9.85
9.15
Speech dataset. Performance is also tested on a speech dataset: Google Speech Commands (Warden 2018
Results with simple LeNet architecture are in Table Each table entry is averaged over 20 Monte Carlo trials (+.014 confidence on test performance). We augmented the data in the same way as Zhang et al. 201
i.c. we sample the spectrograms from the data using a sampling rate of 16 kHz and equalize their sizes at 160 x 101. We also use similar training parameters: we train for 30 epochs using a learning rate of 3 x 10-3 that is divided by 10 every 10 epochs. The improvement of the best m-mixup over the best 1-mixup is 1.11%, with the best k-mixup performance being for 16 and large a / g. Note this improvement is greater than the 0.83% improvement of 1-mixup over ERM.
Toy datasets. Note that for completeness, in Appendix B we
Table 5: Google Speech Commands test error using LeNet architecture (no mixup (ERM) error: 12.26%), aver- aged over 20 Monte Carlo trials. Note the best k-mixup beats the best 1- mixup by 1.11%, greater than the 0.83% improvement of 1-mixup over ERM.
provide quantitative results for the toy datasets of Figures 2 and
3. confirming the qualitative analysis therein.
Distribution shift. As mentioned above, a key benefit of mixup is that (approximately) smoothly linearly interpolates between classes and thereby should provide degree of robustness to distribution shifts that involve small perturbation of the features Zhang et al. (2018). Here we test two such forms of distribution shift: additive Gaussian noise and FGSM white box adversarial allacks (a full exploration of the infinite set of possible distribution shifts is beyond the scope of this paper). Additive Gaussian noise can arisc from unexpected noise in the imaging sensor, and testing adversarial FGSM attacks on mixup was introduced in the original mixup paper Zhang let al. (2018) as a much more difficult perturbation test than i.i.d. additive noise. As in Zhang et al. (2018), we limit the experiment to basic FGSM attacks, since ilerative PGD attacks are too strong, making any performance improvements seem less relevant in practice and calling into question the realism of using PGD as a proxy for distribution shifts.
==
.2
10
k
5
14
5
19
5
26
31
45
11.89
11.55
11.43
11.55
12.16
2
11.76
11.48
11.32
11.38
11.62
4
11.68
11.36
11.18
11.19
11.20
8
11.83
11.36
11.14
11.14
10.84
16
11.72
11.36
11.11
11.04
10.32
Additive Gaussian noise results for CIFAR-10 and various levels e of noise and mixup parameter a are shown in Figure o Note that the h = 16 outperforms k for all noise levels by as much as 4. 4.29%, and ERM by as much as 16%.
Following the experimental setup of Zhang et al. (0118). Figure 7 shows results white-box adversarial attacks generated by the FGSM method (implementation of Kim (T for various values of maximum adversarial perturbation. As in Zhang et al. (2018). the goal of this test is not to achieve results comparable to the much more computationally intensive hods of adversarial training. This would be impossible for a aon-adversarial regularization approach. We show CIFAR-10 accuracy on white box FGSM adversarial data (10000 points), where the maximum adversarial perturbation is set to c/255; performance averaged over 30 Monte Carlo trials (+066 confidence). Note that the highest k outperforms k uniformly by as much
SSmaller values of a were tried, but this decreased performance for all values. SSftware has MIT License
11

---

Published in Transactions on Machine Learning Research (09/2023)
Table 6: Additive noise: Error for CIFAR-10 ResNet-18 with additive Gaussian noise of standard deviation c/255 (results averaged over 30 trials, =0.6 confidence). h 16-mixup outperforms 1-mixup and ERM in each instance (best performance for each e in bold).
10
Noise
ERM
k=1
k=4
1=
16
k=
k=4
k=16
k=1
k=4
k= 16
8
15.72
11.10
11.65
11.00
11.39
11.19
11.28
11.88
11.28
11.53
22.21
15:47
16.06
14.34
15.78
14.69
14.20
15.43
14.56
14.37
t= 12
29.55
21.60
21.75
18.75
21.19
19.06
17.71
20.56
18.45
17.69
14
37.93
29.69
28.34
23.86
27.43
23.72
21.62
25.90
22.62
21.61
6.12%, and all k >> 1-mixups outperform or statistically match k = 1-mixup for all attack sizes. Similar results for MNIST are in Table b), with the FGSM attacks being somewhat less effective. Here again, the highest l performs the best for all attack sizes.
The improved robustness shown by k-mixup speaks to a key goal of mixup, that of smoothing the predictions in the parts of the data space where no/few labels are available. This smoothness should make adversarial attacks require greater magnitude to successfully "break" the model.
Table 7: Adversarial shifts: error on white-box FGSM attacks. Parameter a chosen to maximize k performance. Large k-mixup outperforms 1-mixup in each instance.
k
<=4
c=8
16
k
==
5
1=1
2=
4=4
==8
=
16
1.23
1.66
2.37
3.88
7.70
18.10
20.63
26.25
31.07
36.72
48.55
74.78
2
1.12
1.50
2.11
3.58
7.44
19.81
20.40
25.70
29.89
34.34
43.78
72.22
4
1.07
1.37
1.94
3.25
7.30
22.02
20.73
26.29
30.59
34.94
43.94
71.20
8
1.08
1.40
1.88
3.12
6.92
21.00
20.58
26.06
30.31
34.50
43.51
70.18
16
0.88
1.11
1.50
2.42
5.28
16.44
16
20.21
25.50
29.57
33.80
43.32
68.66
32
0.85
1.07
1.39
2.27
4.91
14.95
(a) CIFAR-10
(+0.6 confidence). a = 0.5.
(b) MNIST (#0.1 confidence). a === 10.
Manifold mixup. We have also compared to manifold mixup (Verma et al. 2018), which aims to interpolate data points in deeper layers of the network to get more meaningful interpolations. This leads to the natural idea of doing k-mixup in these deeper layers. We use the settings in Verma et al. (2018). i.e, for ResNet18, the mixup layer is randomized (coin flip) between the input space and the output of the first residual block
Results for CIFAR-10 with a RosNet18 architecture are in Table Note that in this experiment, the a's shown are used for all le without adjustment since the matches across k-samples happen at random layers and cannot be standardized as simply. Numbers in this experiment are averaged over 20 Monte Carlo trials (+.03 confidence test performance). Manifold 1-mixup is matched by manifold k-mixup with k 4, but both are outperformed by standard k-mixup (Table Th. We therefore do not find a benefit to using demmanffold-mixpp our proposed k-mixup. We also tried randomizing over (a) the outputs of all residual blocks and (b) the outputs of (lower-limensional) deep residual blocks only, but found that performance of both manifold 1-mixup and manifold k-mixup degrades in these cases. This latter observation underscores that mixup in hidden layer manifolds is not guaranteed to be effective and can require tuning.
Table 8: Comparison to manifold mixup approach. "k-manifold mixup" test crror CIFAR-10 using ResNet18 architecture, averaged over 20 Monte Carlo trials (+.03 confidence). Compare Table ! and observe that this "manifold k-mixup" matches the standard manifold mixup but both under- perform our proposed k-mixup, which achieves a 4.02% error rate with k = 16.
over
a
22a
10
5.00
4.73
4.41
4.26
4.81
2
5.09
4.78
4,44
4.31
4.71
5.01
4.78
4.45
4.25
3.02
5.08
4.78
4,42
1.28
4.71
16
5.14
4.82
4.42
4.31
4.53
32
5.14
4.91
4.58
4.47
4.54
10 httppe///hhhho....o///aaaaarrtra
12

---

Published in Transactions on Machine Learning Research (09/2023)
6 Conclusions and Future Work
The experiments above demonstrate that k-mixup improves the generalization and robustness gains achieved by 1-mixup. This seen across a diverse range of datasets and network architectures. implement, adds little computational overhead conventional 1-mixup training, and may also be combined with related mixup variants. As seen the theory presented in Sections 3 and | as i increases, the induced regularization more accurately reflects the local structure of the training data, especially in the manifold support and clustered settings. Empirical results show that performance is relatively robust to variations in k, especially when normalizing for similar perturbation distances (squared), ensuring that extensive tuning is not necessary.
is simple to
With the notable exception of the larger improvement on Tiny ImageNet, our experiments show the improvement on high-dimensional datasets is sometimes smaller than on lower dimensional datasets (recall that classic mixup also has somewhat small gains over ERM in these settings). This difference may be influenced by the diminishing value of Euclidean distance for characterizing dataset. geometry in high dimensions (Aggarwal et al. 2001), but intriguingly this effect not remedied by doing OT in the lower- dimensional manifolds created by the higher layers in our manifold mixup experiments. In future work we will consider alternative metric learning strategies, with the goal of identifying alternative high-dimensional metries for displacement interpolation of data points.
Acknowledgements
The MIT Geometric Data Processing group acknowledges the generous support of Army Research Office grants W911NF2010168 and W911NF2110293, of Air Force Office of Scientific Research award FA9550-19-1-031, of National Science Foundation grants TIS- and CH5-1955697, from the CSAIL Systems that Learn program, from the MIT IBM Watson AI Laboratory, from the Toyota Toyota-CSAIL CSAIL Joint Research Center, from gift from Adobe Systems, and from Google Rescarch Scholar award.
References
Charu C Aggarwal, Alexander Hinneburg, and Daniel A Keim. On the surprising behavior of distance metrics in high dimensional space. In International conference on database theory, pp. 420-434. Springer, 2001.
Raphael Baena, Lucas Drumetz, and Vincent Gripon. Preventing manifold intrusion with locality: Local mixup, 2022.
Dimitris Bertsimas and John Tsitsiklis. Introduction to Linear Optimization. Athena Scientific, 1st edition, 1997. ISBN 1886529191.
Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On Mixup Regularization. arXiv:2006.060) fcs, stat], June 2020. URL
06049 arXiv: 2006.06049.
John Chen, Samarth Sinha, and Anastasios Kyrillidis. Stackmix: A complementary mix algorithm, 2021.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300 2013.
Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. DeepJDOT: Deep Joint Distribution Optimal Transport for Unsupervised Domain Adaptation. Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 467-483, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01225-0.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
Kilian Fatras, Younes Zine, Rémi Flamary, Remi Gribonval, and Nicolas Courty. Learning with minibatch Wasserstein asymptotic and gradient propertics. In Proceedings of the Twenty Third International
Conference on Artificial Intelligence and Statistics, pp. 2131-2141. PMLR, June 2020. URL https ISSN: 2640-3498.
7/proceedings.. mlr....m.r.... mlr hal..prrea///O00/aaaaaa000 ort html
13

---

Published in Transactions on Machine Learning Research (09/2023)
Kilian Fatras, Thibault Sejourne, Rémi Flamary, and Nicolas Courty. Unbalanced minibatch Optimal Transport; applications to Domain Adaptation. In Proceedings of the 38th International Conference on Machine Learning, pp. 3186 3197. PMLR, July 2021. URL htttps://proceeddggg: mlr. hil..press/v19// fatras21a.htm html ISSN: 2640-3498.
Aude Genevay, Gabriel Peyre, and Marco Cuturi. Learning Generative Modcls with Sinkhorn Divergences. In Proceedings of the Twenly-Pire International Conference on Arlificial Intelligence and Slatistics, pp. 1018-1617. PMLR, March 2018. URL https://proceddggs.ou /trpoooeddnng.....pprrrs/////.e ppress/844geeevayye html ISSN: 2640-3498.
Hongyu Guo, Yongyi Mao, and Richong Zhang. MixUp Locally Linear Out-of-Manifold Regularization. Proceedings of the AAAI Conference Artificial Intelligence, 33(01):3714 3722, July 2019. ISSN 2374-3468. doi: 10.1609/aase aaaai.333i01.3300 Number: 01.
URL httpps://jst aaai. index.dexxx.. http://///////atttticl//////////
Allen Hatcher. Algebraic topology. Cambridge Univ. Press, Cambridge, 2000. URL https://ccs.cerr. ch/ record/478079
S. K. Katti. Moments of the Absolute Difference and Absolute Deviation of Discrete Distributions. The Annals of Mathematical Statistics, 31(1):78-85, 1960. ISSN 0003-4851. URL htttp:://ww.jsstor stable/2237499 Publisher: Institute of Mathematical Statistics.
Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.0195. 2020.
Lang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup. In International Conference on Machine Learning, pp. 5275 5285. PMLR, November 2020. URL http::/prooceedings.oo mlr. http:/eeediiiss......rrsss//s///
html
ISSN: 2640-3498.
JangHyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. In International Conference Learning Represenlations, 2021. URL
https://openreviiet wtt.://www.......ooooooo///////a
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann.lece
com/ con/exdb/mnist/
Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Jianzhu Guo, and Stan Z. Li. Unveiling the power of mixup for stronger classifiers, 2021.
Gabriel Peyré and Marco Cuturi. Computational Optimal Transport: With Applications to Data Science. Foundations and Trends0 in Machine Learning, 11(5-6):355 607, February 2019. ISSN 1935-8237, 1935- 8245. doi: 10.1561 10.1561/22000000 2200000007.. URL htttps://ww nowpublishers Publisher: Now Publishers, Inc.
Filippo Santambrogio. Optimal Transport for Applied Mathemalicions: Calculus of Variations, PDEs, and Modeling. Progress in Nonlinear Differential Equations and Their Applications. Birkhäuser Basel, 2015. ISBN 978-3-319-20827-5 doi: 10.1007/978-3-20 2--2-------11-------2 URL 9783319208275
com/ hatn/gppboook.
Jy Yong Sohn, Jaekyun Moon, Kangwook Lee, and Dimitris Papailiopoulos. Gan-mixup: Augmenting across data manifolds for improved robustness. ICML Workshop on Uncertainity and Robustness in Deep Learning, 2020.
Justin Solomon, Kristjan Greenewald, and Haikady N Nagaraja. k-variance: A clustered notion of variance. arXiv preprint ar arXic:2012.0093 06958, 2020.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Aaron Courville, Ioannis Mitliagkas, and Yoshua Bengio. Manifold Mixup: Learning Better Representations by Interpolating Hidden States. September 2018. URT. http:/ppwrerieeeteeee////rrrrraa
14

---

Published in Transactions on Machine Learning Research (09/2023)
Table 9: Test error on toy datasets, averaged over 5 Monte Carlo trials.
a
.25
a=1
a=4 4
a
16
a
=4
64
a 25 a=1
a=
a=
16
64
a=
25
a=
a=4
16
64
155
g
g
.25
k
= 5
= .88
1.0g
1.1g
1.2
1.2g
1.9
2.4g
2,8
3.0
5.01
7.88
10.28
9.75
0.00
0.00
6.79
49.60
49.80
0.83
31.84
37.38
36.15 18
4.28
5.19
5.87
6.19
5.79
0.01
0.00
1.53
47.82
49.74
0.49
3.30
28.41
32.21
32.75
2.93
3.34
4.10
3.86
3.92
4
0.00
0.00
1.41
8.25
9.58
0.32
1.34
15.49
27.53
29.23
2.50
2.77
3.24
3.15
2.78
8
0.00
0.00
0.38
0.62
0.63
0.35
0.45
2.67
8.14
2.76
16
2.42
2.26
2.54
2.43
2.30
16
0.00
0.00
0.08
0.07
0.22
16
0.40
0.33
0.45
0.27
0.27
(a)
One Ring
(b) Four Bars
(c) Swiss Roll
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition. arXiv preprint
urXiv:1804.03900 2018.
Jonathan Weed and Francis Bach. Sharp asymptotic and finite-sample rates of convergence of empirical measures in wasserstein distance. Bernoulli, 25(4A):2620 2648, 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. pp. 6023 6032, 2019. URL https://openaccesss http://pppeaaccessstee.e.e.. thecvf com/ Strategy to to_ Train Strong Str Str.onggCOasssrrrrrrttttt.tt.oo Withh Features ICCV__CCCC _019__apeer_______p_ppp____a_ html
wtt. content ICCV 2019/html/Yun CutMix_ Seiiix_Reglaariztiiin
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Zaz. mixup: Beyond Empirical Risk Minimization. February 2018. http://pee.pppppeeeeeeeeeee///rr
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and Jamos Zou. How Does Mixup Help
With Robustness and Generalization? September 2020. URL http:///openrrrieennetttetoo/o/t ByXEoo6aKNo
A Hyperparameter tuning
While as noted in the main text, the large value of k = 16 with optimized consistently performs well, increasing k does not always improve performance monotonically. This, however, is to be expected in any real data scenario. Hence in practice, it is often appropriate to search over k. This is not too difficult as in our experiments we found sufficient to only try powers of 2, and performance generally is smoothly varying over a. Several search approaches work well:
Very simple: set a = 1 and search over & (powers of 2). Tt can be seen from our experiments that except for one UCI dataset this approach outperforms or matches the 1-mixup performance at any a.
2. More complex but still relatively simple: search over & for 1-mixup, fix the best such &, and then scarch over k (in powers of 2 up to k: = 16 or 32). This approach always outperforms 1-mixup and does not add much hyperparameter search overhead.
3. Full hyperparameter grid search: Not too expensive for many neural networks, for instance full grid search (with & powers of 2) for ResNet-18 CIFAR-10 requires only a few hours of time on our medium-sized cluster. If the model will be deployed in high-stakes or high-volume applications, full search would be fcasible even for larger networks.
B
Test error on toy datasets.
For completeness, we provide quantitative results for the loy datasets of Figures 1 Ring," " "Four Bars," and "Swiss Roll") in Table g. contimuing the intuition building discussion in each of those figures. We used a fully-connectod 3-layer neural network (130 and 120 hidden units). As the datasets are well-clustered and have no noise, smoothing is not needed for generalization and performance without mixup is typically 100%. Rather than beating FRM, applying mixup to these datasets instead aims to build
2 and 3 (denoted "One
15

---

Published in Transactions on Machine Learning Research (09/2023)
intuition, providing view to the propensity of each variant to oversmooth, damaging performance. For each dataset and each perturbation size &, higher k-mixup outperforms 1-mixup, with k = 16 providing the best. performance in all but one instance. These results quantitatively confirm the intuition built in Figures [ that k-mixup regularization more effectively preserves these structures in data, limiting losses from
and X oversmoothing al a lixed mean perturbation size.
C Additional Image Dataset Parameter Sweeps
In this section, we show the parameter sweep tables for the remaining rows of Table
Table 10 below shows results for CIFAR-10, using the PreAct ResNet-18 architecture as in Zhang et al. (2018). Again, increasing & for fixed & tends to improve gencralization performance, especially in the high-5 regime. The best performance overall achieved at k exceeds that of the best 1-mixup by 0.16%, recall that in this setting, 1-mixup outperforms ERM by only 1.4%
with = 16.7. While the best k-mixup performance
[Zhang al. 2018 so when combined with the low overall error rate, small gains not surprising. Results for DenseNet and WideResNet architectures can be found in Table [1 with the best k-mixup outperforming the best 1-mixup by 0.44% and 0.28% respectively. Note that in the case of DenseNet, k 16 outperforms or (statistically) matches k = for all values of 5.
Table 10: Results for CIFAR-10 with ResNet18 architecture (no mixup (ERM) error: 5.6%), averaged over 20 Monte Carlo trials (+.03 confidence on performance). Difference between best k-mixup and best 1-mixup is 0. 0.16%; for fixed high (a 100), the improvement increases to 1.19%.
a=
05
a=
a=.2
a=.5
a=1
a= 10
a= 100
k
5.6
{ = 7.4
= 10.0
{
13.8
= 16.7
= 24.0
[ 27.2
5.01
4.68
4.41
4.24
4.18
4.95
5.67
4.92
4.69
4.46
4.13
4.03
4.58
5.46
4.88
4.68
4.52
4.13
4.03
4.48
4.91
4.77
4.51
4.21
4.08
4.42
4.92
16
4.92
4.77
4.48
4.23
4.02
4.40
4.75
32
4.98
4.78
4.58
4.24
4.16
4.36
4.48
Table 11: CIFAR-10 test error for DenscNet-BBC-190 and WideRosNet-101 architecturos. For DenseNet, the difference between best k-mixup and best 1-mixup 0.44%, for fixed high & (a = 20.2), the improvement increases to 0.65%. For WideResNet, the difference between best k-mixup and best 1-mixup 0.28%, for fixed high (E 20.2), the improvement increases 1.25%.
.5
a a=2
13.8
16.7
19.0
20.2
1
3.35
3.29
3.42
16
3.38
2.98
2.85
2.92
(a) DenseNet- BaaseNet-BCC-1og BC-190 architecture(f.000 confidence, no mixup (ERM) error: 3.7%)
a
.05
a .2
=
a=2
Q
a
= 5.6
[ = 10.0
{
= 13.8
{
= 16.7
{
19.0
{ = 20.2
11.54
11.53
11.62
11.78
12.25
12.99
4
11.36
11.47
11.27
11.41
11.59
12.16
16
11.53
11.59
11.25
11.34
11.38
11.74
(b) WedeResNNt-1000 101 architecture (+.09
confidence, mixup (ERM) error: 11.6%)
Table 12 shows results for CIFAR-100 and SVHN, with DenseNee-BCC-10 and ResNet-18 architectures respectively. As before, for fixed &, the best performance is achieved for some 1. The improvement the best k-mixup over the best 1-mixup is 2.14% for CIFAR-100 and 0.15% for SVHN. For fixed high a, the k-mixup improvement over 1-mixup rises 2.85% for CIFAR-100 and 0.52% for SVHN, possibly indicating that the OT matches yield better interpolation between classes, aiding generalization.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Table 12: CIFAR-100 (DenseNet-BC-119)) and SVHN (ResNet18), averaged over 20 trials.
1
a= 2
a=
a=
a= 10
a=1
a=2
a=4
k
3.8
{ 5.1
{ 6.9
{ 9.5
{ 11.6
{ 3.4
{ 4.2
{ 4.8
{ = 5.4
3.29
3.22
2.99
2.93
3.89
28.52
27.76
20.45
21.53
2
3.32
3.19
2.96
2.79
3.65
8
18,35
18.78
19.16
19.71
4
3,30
3.25
2.94
2.78
3.55
16
18.33
18.31
18.53
18.85
8
3.28
3.20
3.01
2.86
3.44
16
3.26
3.24
3.04
2.93
3.37
(a) CIFAR-100 error (#.05 confidence,
mixup (ERM) 18.91%)
(b) SVHN error (#.02 confidence, no mixup (ERM) error: 3.37%)
D Proof of Proposition 3.2
sampte-sample convergence results for empirical measures (Theorem 9.1 of Solomon et al. 2020 Weed
Bach (019)) imply that for an arbitrary sampling of points fk, we have
(2 & O(k-2/1)
with 1 1/k² probability. The triangle incquality then implics that the Wasserstein-2 distance between our batches of & samples will tend to 0 at the same asymptotic rale, specifically
W2 ( < W2(u, A + W2(A7 ()) S 10k0(k1//)))
again with 1/k² probability.
Now, recalling the definition of the optimal coupling permutation o(i), we have
& 1
w3 A&)
Σ || - =
Hence,
Σ ||
5 1
11 -
10k-2///
with 1 1/k² probability. Thus, for any I C [1, with ||>
k- for all i e I,
O(k 2/d))
1
Σ
|0
25
11
7 11/
h
k
implying
= k- μ - 0 - K5,
where the last incquality holds for any 8 (0, given & large enough.
In essence, the fraction of matches that are long-di digtanie (i.e. those in Z) is bounded by 8 for large enough k. Under these conditions, the set of short-distance matches (i.e. those in the complement of I) satisfy
I 2 (1 10--1 8)k,
where by definition, ||=] - S l1 < k-1/ for all e İ. Crucially, for any chosen e, have k-1/d for k large enough, so for large enough
| - <e, Vieī.
By definition of k-mixup, for all E I, the corresponding mixup interpolated point will be an interpolation between x. and (0 i.e.
ArZ + (1 2) (())
for X E [0, 1]. Since all x? lie in S and for all i E Σ, ||>; I5 - ll2 < e, the mixup interpolated point will lie in (S). The proposition results.
17

---

Published in Transactions on Machine Learning Research (09/2023)
E
Proof of Lemma 333
Firstly, obscrve that the maximum number of within-cluster matches is
Σ min((r;ss))
by definition, and the total number of matches overall must equal E; 8, Eirr. Hence the number of cross-cluster matchings must be larger than or equal to
En ri Σ min(ri,s;) s:) = Σ max(0, -8:),
equivalently,
Σ 81 - Σ min(ri,88) 8:) = Σ max(s; - ri. 0).
Averaging these implics the number of cross-cluster matchings cannot be smaller than
1 Σ r -sl. 2 Σ (max(r; - s..0) 0) max(0, s, -rr))
2
It remains to show that the number of cross-cluster matchings cannot exceed E, - si/. We argue by contradiction and prove the result for m 2 first. Suppose that the number of cross-cluster matchings exceeds Ir 1--1j s1|- Then by the pigeonhole principle (i.e. via the fact that all points must have exactly one other matched point), there must be least two such matchings. WL.G,, let us say these cross-cluster matches are between P; and qi and P++1 and li+1, where Pt and 4++1 must be in the same cluster, as are qe and p 1 since there are only two clusters. By our assumption on the spacing of the clusters, the cost of matching pt to qi and pi+1 to 9++1 at least
2(2A)²
However, consider the alternative matching of P; to 4-1 and P++1 to q:- These are both intra-cluster matchings, which by our assumption on the radius of the clusters must have total cost smaller than equal to
< 2(22))
This matching has smaller cost than the inter-cluster matching strategy, this contradicts optimality of the inter-cluster pairing.
In the scenario with m clusters, an analogous argument works. As above, Ira - si| is the number of cluster elements that must be matched in a cross-cluster fashion. If there are more than the minimum 1 Srir: - si], by the pigeonhole principle, there must be additional cross-cluster matches that form cycle in the graph over clusters. As above, the cost would be reduced by matching points within the same clusters, so this contradicts optimality.
Since the number of cross-clusted matchings cannot be less than or exceed 1 E. sel, it must equal this number and the Jemma results.
F Proof of Theorem 3.4
By Lemma 3.3 if r: and 8; denote the number of points in cluster from batch 1 and 2, the resulting number of cross-cluster matches is & Exlr - sy). As the samples for our batches are ii.d., these random variables (ra, 9;, where r; is independent of s;) each (marginally) follow a simple binomial distribution B(k, We bound the expectation of this quantity with Jensen's inequality:
[ s=1)² < El(r:
2 2Var(rs) 2kps(p 1).
18

---

Published in Transactions on Machine Learning Research (09/2023)
This implies that
Er - si E.
m 2kpi(1 Ps) = -22) Σ pi(1 p.). 2k
yielding the bound in the theorem. It is also possible to get an exact rate with some hypergeometric identities Katti 1960). but these simply differ by a constant factor, so omit the exact expressions here.
G Proof of Theorem 3.5
By the smooth boundary and positive density assumptions, know that P(As) 0 and P(Bs) >> 0 for any
8 > O. Hence, for fixed 8 and k large enough, we know that with high probability the sets Ag and Bş cach contain more points than the number of cross-cluster identifications.
Now consider A. and B, for = 26 + (max(RA: R8) (B)))/(212)) All cross-cluster matches need to be assigned. The cost of assigning a cross-cluster match to point in As and point in B, is at most (1 + 25)2 (since we are using W2). Furthermore, the cost of assigning cross-cluster match that contains a point in A outside A, and arbitrary point in B is at least (1 c)2 D2. Consider the difference between these two costs:
=(((e- 26) 2 - 482) 2 2D max(RA, RB)² 2 R
(1
2D
Since this difference > and we have shown As contains sufficient points for handling all assignments, this assignment outside of A, will only occur if there is within-cluster pair which benefits from using the available point in A, more than is lost by not giving it to the cross-cluster pair (> R2). The maximum possible benefit gained by the within-cluster pair is the squared radius of A, i.c. R2 Since we have shown that the lost cost for the cross-cluster pair is bigger than R2, we have arrived at a contradiction. The proof is similar for the B side.
We have thus shown that for large enough (depending on 6), with high probability all cross-cluster matches have an endpoint each in A, and B, where e 28 (max(Ra, Rg))) (2D3). Setting 6 (max(Ra, Ra,, RB)2)/(4D²) completes the proof.
H Proof of Theorem 4.1
We mostly follow the notation and argument of Zhang al. (2020) (c.f. Lemma 3.1), modifying it for our setting. There they consider sampling X Beta(a, 3) from an asymmetric Dirichlet distribution. Here, assume symmetric Dirichlet distribution, such that 8, simplifying most of the expressions. The analogous results hold in the asymmetric case with simple modifications.
Consider the probability distribution DA with probability distribution: B(a+ 1k++1,a) Note that this distribution is more heavily weighted towards across all a, and for a < 1, there is an asymptote as you approach 1.
Let us adopt the shorthand notation (()),((()()) == Ar + (1 ))a5 for an interpolated feature point. The manipulations below are abbreviated, as they do not differ much for our generalization.
emiz
(
1
E EANsaaa
Σ
WJ (( - (N0? +(1 - A)b (1)) (1
Ck
(())
7-=1
1
Ex
Ex~M(ao) En-Bern(A)
[lh(f(2ra, (A))) v/ (
k())
1m1
(1
BB) ((
S (7s.os0
(2)
Σ
E Ex Ex (1 (A))) (
k 1())2
19

---

Published in Transactions on Machine Learning Research (09/2023)
For the third equality above, the ordering of sampling for À and B has been swapped via conjugacy: A B(a,a) ~ (Arn(A) is equivalent to B ~ (0, 1), A ~ B(a++ B, B)). This is combined with the fact that (i,,)c(i((1 - X) (,)(()),(A)) to get the last line above.
Now we can swap the sums, grouping over the initial point to express this as the following:
Σ Lx 1 T N
c e
+ (1 -)r)) vs ((A,; + (1 A)r),
where the probability distribution D; is described in the text.
The remainder of the argument performs Taylor expansion of the loss term h(f(Xz; (1- A)r)) y?? f(xr; +
(1 A)r) in terms of 1- A, and is not specific to our setting, so we refer the reader to Appendix A.1 of (Zhang et al. 2020 for the argument.
k-mixup as Mean Reversion followed by Regularization
Theorem I.1. Define (#7, 0)) as
27 x. + 0(27 x.}
0. 07 + Oly? 77).
where x7 = (2) [
x ox(()) and 87 = (c) C= (e) are expectations under the matchings and 0 ~
B (112211(0..). a). Further, denote the zero mean perturbations
87 (0 0)a? (1 0)x² (() (1 00
? (0 f)u? (1 f)yE f)y5 - (1 -0)
Then the k-mioup loss can be written as
cTT @mmizup (f)
e fl07 +871)
&
Es,5
The mean क7 being shifted toward is exactly the mean of the locally-informed distribution D. Moreover, the covariance structure of the perturbations is detailed in the proof (simplified in Section L.1 and is now also derived from the local structure of the distribution, inferred from the optimal transport matchings.
Proof. This argument modelled on a proof of Carratino et al. (2020), so we adopt analogous notation and highlight the differences in our setting and refer the reader to Appendix B.1 that paper for any omitted details. First, let us use shorthand notation for the interpolated loss function:
m = E(f(2> + +11 - A)a5 ( Ao? + (1 - 2)
Then the mixup objective may be written as:
&
E emix (f)
Σ
3xm35(N)
k (M))
md==
As ~ B(a, a), we may leverage the symmetry of the sampling function and use a parameter 0 (1/22,((a,a) write the objective as:
emire (f)
N li
where e,
Egm? (0).
k(N)
20

---

Published in Transactions on Machine Learning Research (09/2023)
To obtain the form of the theorem in the text, we introduce auxiliary variables 27.07 to represent the mean-reverted training points:
/.-- - Es,s Oxx + (1 - 0)0 x((1
07 = Eo.s Ovy? + (1 (1-0)y 0)
and E7 to denote the zero mean perturbations about these points:
8 On? + (1 0)xx oic( - -Bo, Oxx + (1 - 0)25 [((
?? = 0y + (1 - 0)0 Es,s Oyy + +1 + 0)0 (0)
These reduce to the simplified expressions given in the theorem we recall that 0 and ( are independent. random variables. Note that both the mean-reverted points and the perturbations are informed by the local distribution D,
Covariance structure
As in (Carratino et al.j 2020), it is possible come up with some simple expressions for the covariance structure of the local perturbations, hence write out the analogous result below. As the argument is very similar to that in (Carratino et al. 2020), omit it.
Lemma 1.2. Let o² denote the variance of (/1/2.1 a), and v² = 0² + +(1 0)2. Then the following expressions hold for the covariance of the mean perturbations:
o² - -7) - IT)) + v² 1² D3737
1
(E)+ 02 057 1/ (07 T T 12 12
0
d - x7 (1) 8. T + 12 129NT0
Ea 57 (E7)
02
where Eggay, Egpoy Eaps denote empirical covariance matrices.
Note again, that the covariances above are locally-informed, rather than globally determined. Lastly, there is also a quadratic expansion performed about the mean- reverted points 47, 97 with terms that regularize f. but we omit this result the regularization of Thcorem 4.1 more intuitive (c.f. Theorem of (Carratino) et al. 2020
J Vicinal Distribution of k: Nearest-Neighbors
We provide this section for explicit illustration of why k nearest-neighhors is not a sensible alternative for k-mixup. Tn this setting, consider drawing two sets of k samples {(x7 vf ) and {(55 vf )) match each x! with its nearest-neighbor in (= and then interpolate these matches to obtain the vicinal distribution. In Figure 7 we provide example images of the generated matching distributions. Note that general, these matching distributions have very limited cross-cluster matchings. The "Swiss Roll" dataset is exception due to the interwoven arms of the spirals. Additionally, note that the intra-cluster matchings are much more concentrated around the points in question, and do not perform as much smoothing within clusters.
21

---

Published in Transactions on Machine Learning Research (09/2023)
Figure 7: Example of matching distributions generated by k 32- 22-nearest-nighhbors using the same point in Figure
22


Reviews:
[{"review_id": "n6hZGFOhtP", "reviewer": "Reviewer_dnUq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n* The proposed method is theoretically sound and achieves better empirical performance than baselines.\n\nWeakness:\n\n* The empirical performance improvement is not that significant and it requires an additional hyperparameter $k$.\n* The authors may also need to provide the run-time analysis as it is not clear whether calculating the optimal match is time-consuming or not.\n* The motivation that the data point generalized by Mixup should lie in the data manifold needs more justifications. For instance, the authors should also consider performing Mixup for the data pair with the same labels (as the data generated by this type of mixup could be closer to the data manifold than the data generated from two data points with different labels.) \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The major contribution of this paper is that it proposes a new mixup method based on optimal transport, which can lead to improved generalization performance than the standard Mixup. Detailed contributions are summarized as follows:\n\n* The proposed method can give better empirical generalization performance.\n* This paper develops a theoretical analysis to justify that the proposed methods can generate interpolated samples that are closer to the data manifold.\n* This paper also theoretically shows that the proposed k-mixup method can lead to more informed regularizations.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness section.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YJzl9mu8JO", "reviewer": "Reviewer_tNWs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- In my opinion, the paper is well-written and easy to follow in most parts. More involved technical concepts, are illustrated nicely through figures that help conveying the intuition of the paper.\n- The theoretical contributions on preservation of global structure and locally-informed regularization looks solid.  \n- Empirical results consistently demonstrate that generalization using k-Mixup is not worse than standard Mixup after tuning a hyperparameter.\n- Based on the experiments, the generalization improvement over standard Mixup is significant in the presence of additive Gaussian perturbations with large variance.\n\nWeaknesses:\n- Experiments are on datasets that can be considered small/low-dimensional with today's standard. It would be interesting to see how the method performs on ImageNet with a larger ResNet model for instance. As the improvements over standard Mixup are fairly small in some cases, it would be important to demonstrate that the computational overhead introduced by k-Mixup on larger datasets is worthwhile. Otherwise, the practical contribution of the paper is somewhat limited to small datasets.\n- I believe the compute cost analysis should be better highlighted in the paper and more rigorously compared to standard Mixup, as this is the main trade-off when deciding between Mixup and k-Mixup.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an extension of the popular Mixup regularization technique. In particular, instead of simply taking convex combinations of random data points, this work proposes interpolating between k-batches of data, each representing a discrete distribution in itself, where interpolation is performed under the Wasserstein metric. Theoretical results indicate that the proposed k-Mixup technique better preserve the original structure of the data compared to standard Mixup. Numerical experiments on various datasets across different modalities suggest that k-Mixup can further improve generalization and robustness to some types of perturbations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the following additions would strengthen the work's practical contribution significantly:\n- Additional experiments on large-scale vision datasets (see details under Weaknesses above).\n- More rigorous comparison of the compute requirements of Mixup/k-Mixup, potentially moving the analysis to the main paper from the Appendix.\n\nFurthermore, I think it would improve the readability of the paper if the Algorithm summary/implementation from the appendix was moved to the main paper. ", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KvrI7qJC3j", "reviewer": "Reviewer_6SVz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The method is well motivated and justified theoretically and empirically.\n2. The experiments are comprehensive and in different domains / applications.\n\nWeakness:\n\n1. [Proposition 3.2] This proposition indicates that the instances generated by k-mixup will be in the manifold with high probability. However, is this enough to justify the advantages of k-mixup? It is sort of necessary condition instead of a sufficient condition, which should be like: can the data in the manifold be generated by k-mixup? Since even for 1-mixup, the generated data would be highly likely in the manifold when $\\alpha$ is very small.\n\n2. For the experiments, Table 1, since every setting is run for 20 times, it is better to report the performance variance.\n\n3. In Figure 5, what is the definition of \"train accuracy\"? Since the generate data does not have a one-hot label.\n\n4. Based on the motivation and the theoretical analysis, the algorithm should work better with a bigger $k$, but this is not always true based on the results in Table 3. Can you explain the performance degradation when $k$ is too large?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes k-Mixup which is a data augmentation technique mixing k (k is generally bigger than 1) instances based on the optimal transport. The authors theoretically and empirically demonstrate that k-mixup is more likely to generate instances within the data manifold compared with 1-mixup. For data of several clusters, k-mixup generates a higher proportion of within-cluster instances instead of inter-cluster ones with the increase of k. Extensive experiments in various applications demonstrate the effectiveness of k-mixup and its advantages over 1-mixup. In addition, the authors show models trained by k-mixup demonstrate robustness against random noise or simple FGSM perturbation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Generally, this work is well written, well motivated and easy to follow. The revision should address the weakness part pointed in the previous section.", "broader_impact_concerns": "This paper proposed a generic method and there is no concerns on ethics as far as what I know.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all found the method was well-motivated and the experiments were correct. There were some concerns about the lack of large-scale experiments, but the experiments provided were comprehensive and technically correct. Therefore, I recommend acceptance.
Decision: Accept (poster)