C-Mixup: Improving Generalization in Regression
Authors: Huaxiu Yao
Yiping Wang
Linjun Zhang
James Zou
Chelsea Finn
Venue: NeurIPS 2022 Accept (2022)
Keywords: data augmentation
regression
mixup
generalization
deep neural networks

Abstract: Improving the generalization of deep networks is an important open challenge, particularly in domains without plentiful data. The mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. These interpolated examples augment the original training set. Mixup has shown promising results in various classification tasks, but systematic analysis of mixup in regression remains underexplored. Using mixup directly on regression labels can result in arbitrarily incorrect labels. In this paper, we propose a simple yet powerful algorithm, C-Mixup, to improve generalization on regression tasks. In contrast with vanilla mixup, which picks training examples for mixing with uniform probability, C-Mixup adjusts the sampling probability based on the similarity of the labels. Our theoretical analysis confirms that C-Mixup with label similarity obtains a smaller mean square error in supervised regression and meta-regression than vanilla mixup and using feature similarity. Another benefit of C-Mixup is that it can improve out-of-distribution robustness, where the test distribution is different from the training distribution. By selectively interpolating examples with similar labels, it mitigates the effects of domain-associated information and yields domain-invariant representations. We evaluate C-Mixup on eleven datasets, ranging from tabular to video data. Compared to the best prior approach, C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out-of-distribution robustness, respectively. Code is released at https://github.com/huaxiuyao/C-Mixup.

C-Mixup:
Improving Generalization in Regression
Huaxiu Yao" Yiping Wang"', Linjun " Zou', Chelsea Finn
Stanford University, "Zhejiang University, 'Rutgers University
(huaxiu,cbfinn) res.stanford.eduu jamesz stanford.edu yypinngwang616 61 @ @gmail.com, "linun @ @rutgers.edu
Abstract
Improving the generalization of deep networks an important open challenge, particularly in domains without plentiful data. The mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. These interpolated examples augment the original training set. Mixup has shown promising results various classification tasks, but systematic analysis mixup regression remains underexplored. Using mixup directly regression labels can result in arbitrarily incorrect labels. In this paper, we propose a simple yet powerful algorithm, C-Mixup, improve generalization contrast with vanilla mixup, which picks training examples for mixing with uniform probability, C-Mixup adjusts the sampling probability based the similarity of the labels. Our theoretical analysis confirms that C-Mixup with label similarity obtains a smaller mean square error in supervised regression and meta-regression than vanilla mixup and using feature similarity. Another benefit of C-Mixup is that it can improve out-o--distributio robustness, where the test distribution different from the training distribution. By selectively interpolating examples with similar labels, mitigates the effects of Nomain-assooiinn representations. We evaluate C-Mixup on eleven datasets,
regression tasks. In
information and
yields domain-invarias ranging from tabular to video data. Compared the best prior approach, C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out- distriibuutoooooonnnnn robustness, respectively. Code released https///gihbbbboom//uuxxxxxoyyys
Introduction
Deep learning practitioners commonly face the challenge of overfitting, To improve generalization. prior works have proposed number of techniques, including data augmentation [3] 10
12
81 [2]
and explicit regularization [15] 38 60). Representatively. mixup S2 densifies the data distribution and implicitly regularizes the model by linearly interpolating the features of randomly sampled pairs of examples and applying the same interpolation on the corresponding labels. Despite mixup having demonstrated promising results in improving generalization in classification problems, it has rarely been studied in the context of regression with continuous labels, on which we focus in this paper.
In contrast to classification, which formalizes the label as one-hot vector, the goal of regression is to predict continuous label from each input. Directly applying mixup input features and labels in regression tasks may yield arbitrarily incorrect labels. For example, shown in Figure aa), Sh pose prediction [18] aims to predict the current orientation of the object relative to its canonical orientation. We randomly select three mixing pairs and show the mixed images and labels in Figure [b), where only pair exhibits reasonable mixing results. We thus see that sampling mixing pairs uniformly from the dataset introduces number of noisy pairs.
Equal contribution. This work was done when Yiping Wang was remotely co-mentored by Huaxiu Yao and Linjun Zhang.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Training set
Vanila mixup
ProbjP1) Prob))2) Prob(h3)
My: 150
15:1711
Yv: 150"
Xaa 346"
Yaa 171"
Yl 3466"
Optimized model for
maup
Test set
Palnc (PX)
Pair (P2)
Pair (PA)
Prob/P1) = Frob(P3) >o ProbIP2)
C Mixuup (Durs) Prob(P1) as Probij2) Prob(Ps)
5
9:248
3t 250.5
(a] ShapeNet1D Pose
(c) Sampling Probability Comparison
Prediction
(b) Mixing Pairs (2 0.5)
Figure 1: Illustration of C-Mixup on ShapeNet pose prediction. À represents the interpolation ratio. (a) ShapeNetiD pose prediction task, aiming to predict the current orientation of the object relative to its canonical orientation. (b) Three mixing pairs are randomly picked, where the interpolated images visualized and ý represents interpolated labels. (c) Illustration of rough comparison of sampling probabilities among three mixing pairs in (b). The Euclidean distance measures input feature distance and the corresponding results between examples in pairs 1, 2, are 1.51 10 1.82 10 1.50 x 10 respectively. Hence, pairs and 3 have similar results, leading to similar sampling probabilities. C-Mixup is able assign higher sampling probability for more reasonable mixing pairs.
In this paper, we aim to adjust the sampling probability of mixing pairs according to the similarity of examples, resulting in simple training technique named C-Mixup. Specifically, we employ Gaussian kernel to calculate the sampling probability of drawing another example for mixing. where closer examples are more likely to be sampled. Here, the core question is: how to measure the similarity between two examples? The most straightforward solution is to compute input feature similarity. Yet, using input similarity has two major downsides when dealing with high-dimensional data such as images or time-series: substantial computational costs and lack good distance metrics. Specifically, takes considerable time to compute pairwise similarities across all samples, and directly applying classical distance metries (e.g., Euclidean distance, cosine distance) does not reflect the high- level relation between input features. In the ShapeNet rotation prediction example (Figure [a)), pair 1 and 3 have close input similarities, while only pair 1 can reasonably interpolated.
To overcome these drawbacks, C-Mixup instead uses label similarity, which is typically much faster to compute since the label space is usually low dimensional. In addition to the computational advantages, C-Mixup benefits three kinds of regression problems. First, it empirically improves in-distribution generalization in supervised regression compared to using vanilla mixup or using feature similarity. Second, we extend C-Mixup to wradient-based meta-learning by incorporating into MetaMix, a mixup-based task augmentation method [74). Compared to vanilla MetaMix, C-Mixup empirically improves task generalization. Third, C-Mixup is well-suited for improving out-of-distribution robustness without domain information, particularly to covariate shift (see the corresponding example in Appendix A.1). By performing mixup examples with close continuous labels, examples with different domains are mixed. In this way, C-Mixup encourages the model to rely on domain- domain-invarian invariant features to make prediction and ignore unrelated spurious correlations, making the model more robust to covariate shift.
The primary contribution of this paper is C-Mixup, simple and scalable algorithm for improving generalization in regression problems. In linear or monotonic non-linear models, our theoretical analysis shows that C-Mixup improves generalization in multiple settings compared to vanilla mixup or compared to using feature similarities. Moreover, our experiments thoroughly evaluate C-Mixup on eleven datasets, including many large-scale real-world applications like drug-target interaction prediction [28] ejection fraction estimation with echocardiogram videos (50). poverty estimation with satellite imagery (78). Compared to the best prior method, the results demonstrate the promise of C- Mixup with 6.56%, 4. 4.76%, 5.82% improvements in in-distribution generalization, task generalization. and -ut-of-distribution robustness, respectively.
2 Preliminaries
In this section, we define notation and describe the background of ERM and mixup in the supervised learning setting, and MetaMix in the meta meta-learning setting for task generalization.

---

ERM. Assume machine learning model f with parameter space O. In this paper, we consider the setting where one predicts the continuous label y É y according the input feature a E X. Given a loss function f, we train model fo under the empirical training distribution per with the following objective, and get the optimized parameter l" E θ:
0
f arg mui par [8(f((1). w)]- BE
(I)
Typically, we expect the model to perform well on unseen examples drawn from the test distribution pt* We are interested in both in-distribution (P"r = pt*) and out-of-distribution (P" # PP*) settings.
Mixup. The mixup algorithm samples a pair of instances (Izz, Mi)) and (2j, Mj), sampled uniformly random from the training dataset, and generates new examples by performing linear interpolation on the input features and corresponding labels as:
f = A x, + (1 - A) x,, y A y + (1 -1- .),
(2)
where the interpolation ratio À interpolated examples are then used to optimize the model as follows:
[0, 1] is drawn from a Beta distribution, i.e., X Beta(a, a). The
8" + arg min (y ((ff(()) DEG
(3)
Task Generalization and MetaMix. In this paper, we also investigate few-shot task generalization under the gradient-based meta-regression setting. Given task distribution p(T), we assume each task Tm is sampled from pT)) and is associated with a dataset Dm. A support set D = {(X Ymm) = are sampled from D, Representatively, in model-agnostic meta-learning (MAML) [14), given predictive model f with parameter 0, it aims to learn an initialization 0" from meta- tratt-taaning tasks (TT, ) Specifically, at the meta-training phase, MAMI. obtained the task-specific parameter 6m for each task Tm by performing a few gradient steps starting from A. Then, the corresponding query set Dy, is used to evaluate the performance of the task task-specific model and optimize the model initialization as:
[(x".... Vm, i) and a query set D {(X... YD)] {(#9 (mmm.. :. B. ))
I Σ
1 M
== arg min
: where @n = 0 [VoL[(o;
(4)
At the meta-testing phase, for each meta-testing task Tr MAML fine-tunes the learned initialization O" on the support set Dj and evaluates the performance on the corresponding query set Dj.
To improve task generalization, MetaMix 77] adapts mixup (Eqn. (3) to meta-learning, which
linearly interpolates the support set and query set and uses the interpolated set to replace the original query set Dy, in Eqn. { Specifically, the interpolated query set is formulated as:
X1 AX A)x Y
AY + - -
(5)
where A ~ a)).(ar.).
3
Mixup for Regression (C-Mixup)
For continuous labels, the example in Figure bb) illustrates that applying vanilla mixup to the entire distribution is likely to produce arbitrary labels. To resolve this issue, C-Mixup proposes to sample closer pairs of examples with higher probability. Specifically, given an example (I;, mi), C-Mixup introduces a symmetric Gaussian kernel to calculate the sampling probability P((x,) My (1,, ys)) for another (zz, U,) example to be mixed as follows:
d(i.j) 2
P( 85 (x,, v.)) x exp
(6)
where d(i,j) represents the distance between the examples (Ni, vt) and (1z, and a describes the bandwidth. For the example (x,, W:), the set (P((s), (,),, [v)) is then normalized to probability mass function that sums to one.
3

---

Algorithm 1 Training with C-Mixup
Require:
Learning rates y/: Shape parameter a
Require:
Training data D == {x1, Ms) )=)
1: Randomly initialize model parameters 0
2: Calculate pairwise distance matrix P via Eqn. (6)
3: while not converge do
4: Sample batch of examples B 'D
5: for each example (zi, Vi) E B do
6: Sample (xzz Yÿ) from P(- (a)) and X from Bcta(a, (:)
7: Interpolate (Ir,, y:), (z,, M/y to get (ž, 17) according to Egn. 22
8: Use interpolated examples to update the model via Eqn. (3)
One natural way to compute the distance is using the input feature I, i... d(i,j) d( Tz)) However, when dealing with the high digeddimensional data such as images or videos, we lack good distance metrics capture structured feature information and the distances can be easily influenced by feature noise. Additionally, computing feature distances for high-dimensional data is time- consuming. Instead, C-Mixup leverages the labels with f) d(y:: Mj) Ily; ツزlą where y: and yj are vectors with continuous values. The dimension of label is typically much smaller than that of the input feature, therefore reducing computational costs (see more discussions about compuatational efficiency in Appendix A.3). The overall algorithm of C-Mixup is described in Alg. and we detail the difference between C-Mixup and mixup in Appendix A.4. According Alg. C-Mixup assigns higher probabilities to example pairs with closer continuous labels. In addition to its computational benefits, C-Mixup improves generalization on three distinct kinds of regression problems in-distribution generalization, task generalization, and distribiittiittionn robustness, which is theoretically and empirically justified in the following sections.
4
Theoretical Analysis
In this section, we theoretically explain how C-Mixup benefits in-distribution generalization, task generalization, and cul-of-distribution robustness.
4.1 C-Mixup for Improving In-Distribution Generalization
In this section, we show that C-Mixup provably improves in-distribution generalization when the features are observed with noise, and the response depends on small fraction of the features in a monotonic way. Specifically, we consider the following single index model with measurement error,
y ( 2) +G
where 0 Rp and is sub-Gaussian random variable and 9 is a monotonic transformation. Since images are often inaccurately observed in practice, we assume the feature 2 is observed or measured + & with being random vector with
with noise, and denote the observed value by mean O and covariance matrix o² We assume g to be monotonic to model the nearly one-to-one correspondence between causal features (e.g., the car pose in Figure (a) and labels (rotation) in the in-distribution setting. The out-of-distriibttio setting will be discussed in Section 4.3 We would like to also comment that the single index model has been commonly used in econometrics, statistics, and deep learning theory [19
27
47
53 73
Suppose we have {(x, ) N i.i.d. drawn from the above model. We first follow the single index model literature (e.g. (73) and estimate 0 by minimizing the square error Σi (i. 2. 0)², where the (Z., Da are the augmented data by either vanilla mixup, mixup with input feature similarity, and C-Mixup. We denote the solution by 0% fizzupp fatt and 0- Mixup respectively. Given an estimate 0", we estimate g by ý via the standard nonparametric kernel estimator [64] (we specify this in detail in Appendix B.I for completeness) using the augmented data. We consider the mean square error metric as MSE(O) = E[(x 0(0 =))? and then have the following theorem (proof: Appendix B.1);
Theorem 1. Suppose 0 E IRP is sparse with sparsity s = o(min(p, of o(N) and g is smooth with co g < ci. C2 g" ca for some universal constants co, C1, C22 C3 0. There exists

---

distribution on x with a function,, such that when the sample size N is sufficiently large. with probability o(1),
MSB(fc- ((SB(O)-Mieup)) < min(MME((0)..)) MSSE (iiiuup) ).
(8)
The high-level intuition of why C-Mixup helps is that the vanilla mixup imposes linearity regulariza- tion on the relationship between the feature and response. When the relationship is strongly nonlinear and one-to-one, such a regularization hurts the generalization, but could be mitigated by C-Mixup.
4.2 C-Mixup for Improving Task Generalization
The second benefit of C-Mixup is improving task generalization in meta-learning when the data from each task follows the model discussed in the last section. Concretely, we apply C-Mixup to MetaMix [7].. For each query example, the support example with more similar label will have a higher probability of being mixed. The algorithm of C-Mixup MetaMix is summarized in Appendix A.2.
Similar to in-distribution generalization analysis, we consider the following data generative model: for the m-th task (m E [M). we have (I:(7), (y(m) TTm with
(m) = 9m (9 2(1)) te and = + (m))
Here, 0 denotes the globally-shared representation, and 9m 's are the task-specific transformations. Note that, the formulation is close [54] and wildly applied to theoretical analysis of meta- learning [7]. Following similar spirit of [62] and last section, we obtain the estimation of e by 8" M m== (Begtt)), e. [1(m) - T x (.)))) D. Here, D.,. denotes the generic dataset augmented by different approaches, including the vanilla MetaMix, MetaMix with input feature similarity, and C- Mixup. We denote these approaches by Oseeta - feats and 0 Meenn- Mirup respectively. For new task Tr, we again use the standard nonparametric kernel estimator to estimate It via the augmented target data. We then consider the following error metric MSETarget (f*) = E(z.u))T, [(v - G.(8" )))
Based on this metric, we get the following theorem to show the promise of C-Mixup in improving task generalization (see Appendix B.2 for detailed proof). Here, C-Mixup achieves smaller Target compared to vanilla MetaMix and MetaMix with input feature similarity.
Theorem 2. Let N Em Nm and Nm is the number of examples of Tm. Suppose 0g is sparse with sparsity o(minfd, of)) p= o(N) and gm 's are smooth with I'm < C1, C2 < 9mh c3 for some universal constants C1, C2 C3 > 0 and m E [M] U (t). There exists a distribution on x with a kernel function, such that when the sample size N is sufficiently large, with probability 1 1- o11),
MSETarga (fi (CMeta--MMirupp) < min(MSEnngee (eMeta-feat) MSETarga (8 (ett)
(10)
4.3 C-Mixup for Improving Out-of-distribution Robustness
Finally, we show that C-Mixup improves OOD robustness in the covariate shift setting where some unrelated features vary across different domains. In this setting, we regard the entire data distribution consisting of 3 = [1..... E) domains, where each domain is associated with a data distribution Pa for e ε. Given set of training domains er C E, we aim to make the trained model generalize well to an unseen test domain gx that is not necessarily in Erh Here, we focus on covariate shift, i.e., the change of P. among domains is only caused by the change of marginal distribution P. (X),,, while the conditional distribution Pa P is fixed across different domains.
To overcome covariate shift, mixing examples with close labels without considering domain infor- mation can effectively average out coamain-changcable correlations and make the predicted values rely on the invariant causal features. To further understand how C-Mixup improves the robustness to covariate shift, we provide the following thcoretical analysis.
We assume the training data (1,, > follows x; (2) E +P and y: 0T + Gs where 2; E R and a; E F2 are regarded as invariant and dhnnan-changeable unrelated features, respectively, and the last Pa coordinates of 0 E RP are O. Now we consider the case where the training data consists of a pair of domains with almost identical invariant features and opposite domain-changeable features, i.e., I; (Zi, az), ar4
Np, (0, dTT,, 2 = 2, + é, a ~ N2z (0, pp2), al = -a + c" 6,,6%, e," are noise terms with mean O and sub- Gaussian norm bounded by a... We use ridge estimator 0 (k)) arg min, (E, 1/ o 2i1 12 + 101)))) to reflect the
af). where
5

---

implicit regularization effect of deep neural networks [4]] The covariant shift happens when the test domains have different a; distributions compared with training domains. We then have the following theorem to show that C-Mixup can improve robustness to covariant shift (see proof in Appendix B.3):
Theorem 3. Supposed for some max(exp((n'-oti) exp( )) < 1, we have variance con- liope se, > and of S pm Then for any penalty k satisfies
straints: da = 2102, 0x 2 02 max
n5/2
(n!/(tt(l)) < k < min( S+ /pint-O0, n. and bandwidth h satisfies O < h S vlog(n=/P1) c/ PL in C-Mixup, when is sufficiently large, with probability at least o(1), we have MSE(OC-Miase < min Maaeeaaaaeeaa.....aaa MSE((mizup)),,
(11)
where ej 2 L, C2 c3, Ca, cb co > are universal constants, I miney, | V/y and pr <n př.
5 Experiments
In this section, we evaluate the performance of C-Mixup, aiming to answer the following questions: Q1: Compared corresponding prior approaches, can C-Mixup improve the in-distribution, task generalization, and compared to using other distance metrics? Q3: Is C-Mixup sensitive to the choice of bandwidth in the Gaussian kernel of Eqn. ( In our experiments, we apply cross-validation to tune all hyperparameters with grid search.
robustness on regression? Q2: How does C-Mixup perform
5.1 In-Distribution Generalization
Datasets. We use the following five datasets to evaluate the performance of in-distribution gener- alization (see Appendix C.1 for detailed data statistics). (1)&(2) Airfoil Self-Noise (Airfoil) and NO2 (35] are both are tabular datasets, where airfoil contains aerodynamic and acoustic test results of airfoil blade sections and NO2 aims to predict the mount of air pollution at a particular location. (3))(4): Exchange-Rai Rate, and Electricity [40] are two time-series datasets, where Exchange-Rand Rate reports the collection of the daily exchange rates and Electricity is used to predict the hourly electric- ity consumption. (5) Echocardiogram Videos (Echo) [50] is ejection fraction prediction dataset, which consists of series of videos illustrating the heart from different aspects.
Comparisons and Experimental Setups. We compare C-Mixup with mixup and its variants (Man- ifold mixup 688, k-Mixup 20 and Local Mixup (5) that can be easily to adapted to regression tasks. We also compare to MixRL, recent reinforcement learning framework to select mixup pairs in regression. Note that, for k- M-Mixup, Local Mixup, MixRL, and C-Mixup, we apply them to both mixup and Manifold Mixup and report the best-performing combination.
For Airfoil and NO2, we employ three-layer fully connected network as the backbone model. We use LST-Attn 40 for the Exchange-Rate and Electricity, and EchoNet-Dynamic (50) for predicting the ejection fraction. We use Root Mean Square Error (RMSE) and Mean Averaged Percentage Error (MAPE) evaluation metrics. Detailed experimental setups are in Appendix C.2.
Results. We report the results in Table D and have the following observations. First, vanilla mixup and manifold mixup are typically less performant than ERM when they are applied di- rectly to regression tasks. These results support hypothesis that random selection of example pairs for mixing may produce arbitrary inaccu- rate virtual labels. Second, while limiting the scope of interpolation (c.g.. k-Mixup, Manifold k-Mixup, Local Mixup) helps improve gener- alization in most cases, the performance of these approaches is inconsistent across different datasets. As an example, Local Mixup outperforms ERM in NO2 and Exchange-Rate but fails to benefit results in Electricity. Third, even in more complicated datasets, such as Electricity and Echo, MixRL performs worse than Mixup and ERM, indicating that it is non-trivial to train policy network that works well. Finally, C-Mixup consistently outperforms mixup and its variants, ERM, and MixRI., demonstrating its capability to improve in-distribution generalization on regression problems.
x10-1
3.0 X30
ManiMix Mixup
ERM
Gererabzation Gap
2..5
ManilMix
ERM
Test RMBE
1.5
1.0
50
160 Epochs
150
200
50
100 Epochs
15o
200
Figure 2: Overfitting Analysis on Exchange-Rate C-Mixup achieves better test performance and smaller generalization gap.

---

Table Results for in-distribution generalization. We report the average RMSE and MAPE of three seeds. Full results with standard deviation are reported in Appendix C.4. The best results and second best results are bold and underlined, respectively.
Airfoil
NO2
Exchange-Ran
Electricity
Echo
RMSE
MAPE
RMSS
MAPF
RMSF
MAPE
RMSE
MAPF
RMSF
MAPR
ERM
2,901
1.753%
0.537
13.615%
0.0236
2.423%
0.0581
13.861%
5.402
8.700%
mixup
3.730
2.327%
0.528
13.534%
0.0239
2.441
0.0585
14.306%
5.393
8.838%
Mani mixup
3,063
1.842%
0.522
13.382%
0.0242
2.475%
0.0583
14.556%
5.482
8.955%
k-Mixup
2.938
1.769%
0.519
13.173%
0.0236
2.403%
0.0575
14.134%
5.518
9.206%
Local Mixup
3.703
2.290%
0.517
13.202%
0,0236
2.341
0.0582
14.245%
5,652
9.313%
MixRL
3,614
2.163%
0.527
13.298% %
0,0238
2.397%
0.0585
14.417%
5.618
9.165%
C-Mixup (Ours)
2.717
1.610%
0.509
12.998%
0.0203
2.041 %
0.0570
13.372%
5.177
8.435%
Analysis of Overfitting. In Figure 2 visualize the test loss (RMSE) and the generalization gap between training loss and test loss of ERM, Manifold Mixup, and C-Mixup with respect to the training epoch for Exchange-Rate More results are illustrated in Appendix C.3. Compared with ERM and C-Mixup, the better test performance and smaller generalization gap of C-Mixup further demonstrates its ability improve in-distribution generalization and mitigate overfitting.
5.2 Task Generalization
Dataset and Experimental Setups. To evaluate task generalization in meta-learning settings, we two rotation prediction datasets named as ShapeNet [88 and PascalI [71 79], the goal of both datasets is to predict an object's rotation relative the canonical orientation. Each task is rotation regression for one object, where the model takes a 128x128 grey-scale image as the input, and the output an azimuth angle normalized between (0, 10]. We detail the description in Appendix D.1.
Since MetaMix outperforms most other methods in PASCAL3D in Yao et al. [74], we only select two other representative approaches Meta-Aug [55] and MR-MAML [79] for comparison. We further apply k-Mixup and Local Mixup to MetaMix, which are called as k-MetaMix and Local MetaMix, respectively. Following Yin et al. [79]. the base model consists of an encoder with three convolutional blocks and a decoder with four convolutional blocks. Hyperparameters are listed in Appendix D.2.
Results. Table shows the average MSE with 95% confidence interval over 6000 meta-testing tasks. Our results corroborate the findings of Yao et al. [74] that MetaMix improves perfor- mance compared with non-mixup approaches (i... MAML, MR-MAML, Meta-Aug). Sim- ilar to our previous in-distribution generaliza- tion findings, the better performance of Local MetaMix over MetaMix demonstrates the effec- tiveness of interpolating nearby examples. By mixing query examples with support examples that have similar labels, C-Mixup outperforms all of the other approaches on both datasets, verifying its effectiveness of improving task generaliza- tion.
Table 2: Meta-regression performance (MSE +
95% confidence interval) on 6000 meta-test tasks.
Model
ShapeNet1D 4
PASCAL3D 1
MAMI.
4.698 + 0.079
2,370 + 0.072
MR-MAML
4.433 1 0.083
2.276 1 0.075
Meta-Aug
4.312 1 0.086
2.298 1 0.071
MetaMix
4.275 1: 0.082
2.135 1 0,070
k- k-MetaMix
4.268 = 0.078
2.091 + 0.069
Local MetaMix
4.201 = 0.087
2.107 = 0.078
C-Mixup (Ours)
4.024 + 0.081
1.995 1 0.067
5.3 Out-of-Distribution Robustness
Synthetic Dataset with Subpopulation Shift. Similar synthetic datasets with subpopulation shifts in classification, e.g., ColoredMNIST 4 we build a synthetic regression dataset RCFashion- MNIST (CCF-MNIST), which is illustrated in Fig-
120"
240"
Tiiining
Text
Anglet
Angie1
Red
Anglet
Anglet
20%
80%
Rte
ure 3
Built on the MNSST [8], the goal of
RCF-MNIST is to predict the angle of rotation for
Figure 3: Illustration of RCF-MNIST.
each object. As shown in Figure we color each
image with color between red and blue. There is spurious correlation between angle (label) and
7

---

Table 3: Results for out-of-distribution robustness. We report the average and worst-domain (pri- mary metric) performance here and the full results are listed in Appendix E.3. Sub. Shift means Subpopulation Shift. Higher R or lower RMSE represent better performance. mixup and C-Mixup uses the same type of mixup variants reported in Table E.2 For PovertyMap. most results are copied from WILDS benchmark (4 and worst-domain performance is the primary metric. We bold the best results and underline the second best results.
Sub. Shift
Domain Shift
RCF-MNIST
Mav (R)
Crime (RMSE)
SkillCraft (RMSE)
DTI (R)
Avg. (RMSE) 4
Avg. f
Worst
Avg. J
Worst
Avg. 4
Worst L
Avg.
Worst
ERM
0.162
0.80
0.50
0.134
0.173
5.887
10.182
0.464
0.429
IRM
0.153
0.77
0.43
0.127
0.155
5.937
7.849
0.478
0.432
TB-IRM
0.167
0.78
0.40
0.127
0.153
6.055
7.650
0.479
0.435
V-REx
0.154
0.83
0.48
0.129
0.157
6.059
7.444
0.485
0.435
CORAL
0.163
0.78
0.44
0.133
0.166
6.353
8.272
0.483
0.432
GroupDRO
0.232
0.75
0.39
0.138
0.168
6,155
8.131
0.442
0.407
Fish
0.263
0.80
0.30
0.128
0.152
6,356
8.676
0.470
0.443
mixup
0.176
0.81
0.46
0.128
0.154
5.764
9.206
0.465
0.437
C-Mixup (Ours)
0.146
0.81
0.53
0.123
0.146
5.201
7.362
0.498
0.458
color in training set. The larger the angle, the more red the color. In test set, we reverse spurious correlations to simulate distribution shift. We provide detailed description Appendix E.I.
Real-world Datasets with Domain Shifts. In the following, we briefly discuss the real-world datasets with domain shifts; the detailed data descriptions in Appendix E.1: (1) PovertyMap 34 is a satellite image regression dataset, aiming to estimate asset wealth in countries that are not shown in the training set. (2) Communities and Crime (Crime) [56] is tabular dataset, where the problem is to predict total number of violent crimes per 100K population and we aim to generalize the model to unseen states. (3) SkillCraft1 Master Table (SkilICraft)
6 a tabular dataset, aiming to predict the mean latency from the onset of a perception action cycles to their first action in milliseconds. Here, "LeagueIndex" is treated as domain information. (4) Drug- target Interactions (DTI) (28] is aiming to predict aut-of-distributin drug-target interactions in 2019-2020 after training on 2013-2018
Comparisons and Experimental Setups. To evaluate the out-of-distribution robustness C-Mixup, we compare it with nine invariant learning approaches that can be adapted to regression tasks, including ERM, IRM 4 IB-IRM [I]. V-REx (39], CORAL 4T.. DRNN [16), GroupDRO (58). Fish [59]. and mixup. All approaches use the same backbone model, where we adopt ResNet-50, three-layer full connected network, and DeepDTA [51] for Poverty, Crime and SkillCraft, and DTI, respectively. Following the original papers of PovertyMap (78) and DTI (28) we use R value to evaluate the performance. For RCF-MNIST, Crime and SkillCraft, we use RMSE as the evaluation metric. In datasets with domain shifts, we report both average and worst-domain (primary metric for PovertyMap
344 performance. Detailed hyperparameters are listed in Appendix E.2.
According to the results, we can see that the performance of prior invariant learning approaches is not stable across datasets. For example, IRM and CORAL outperform ERM on Camelyon 77, but fail to improve the performance on PovertyMap. C-Mixup instead consistently shows the best performance regardless of the data types, indicating its effeacy in improving robustness to covariate shift.
Results. We report both average and worst-domain performance in Table 3
8.5 x10*
Analysis of Learned Invariance. Following Yao et al. (76). we analyze the domain invariance of the model learned by C-Mixup. For regression, we measure do-
ERR
CORAL
Pairwise Divergence
8.0
c-Mbou
main invariance as pairwise divergence of the last hidden
representation. Since the labels are continuous, we first
evenly split examples into C = {l..... ......) C} bins accord-
ing to their labels, where C 10. After splitting, we perform kernel density estimation to estimate the prob- ability density function P(hş) of the last hidden repre- sentation. We calculate the pairwise divergence as Inv = I Ecec Ee'cee KL(P(gft E = P(ge d(). The results on Crime and CrillCraft are reported in Fig- ure 4 where smaller number denotes stronger invariance.
Crime
SkillCraf
Datasets
Figure 4: Results of invariance analy- sis. Smaller pairwise divergence value represents stronger invariance.
8

---

We observe that C-iixup learns better invariant representation compared to prior invariant learning approaches.
5.4 Analysis of C-Mixup
In this section, conduct three analyses including the compatibility of C-Mixup, alternative distance metrics, and sensitivity of bandwidth. In Appendix F.3.2 and F.4, we analyze the sensitivity of
hyperparameter in Beta distribution and the robustness of C-Mixup to label noise, respectively.
Compatibility of C-Mixup. C-Mixup is comple- mentary approach to vanilla mixup and its variants, where it changes the probabilities of sampling mix- ing pairs instead of changing the way mixing. We further conduct compatibility analysis of C-Mixup by integrating it to three representative mixup vari- ants - PuzzleMix [32], CutMix [8]) AutoMix [45]. We evaluate the performance MNIST, PovertyMap). The reported results in Table indicate the compatibility and efficacy of C-Mixup
Table 4: Compatibility analysis. See Ap- pendix F.I for full results.
RCF-MNIST
PovertyMap
Model
RMSE L
Worst R
0.194
0.46
CutMix
+C-Mixup
0.186
0.53
two datasets (RCF-
0.159
0.47
PuzzleMix
+C-Mixup
0.150
0.50
0,152
0.49
AutoMix
regression.
+C-Mixup
0.146
0.53
II. Analysis of Distance Metrics. Besides the the- oretical analysis, here we empirically analyze the effectiveness of using label distance. Here, d(a, b) to denote the distance between objects a and b, e.g., d(ys: Y/) in our case. We consider four substi- tute distance metrics, including: (1) feature distance: d(ari, x;); (2) feature and label distance: we concate- nate the input feature x and label y to compute the distance, i.e., d(æ; @ Vi. @ yj); (3) representation distance: since using feature distance may fail to capture high-level feature relations, we propose to use the model's hidden representations to measure the distance d(h,, f,)). detailed in Appendix F.2. (4) representation and label distance: d(h; &v hy @Dj))
Table Performance of different distance
metrics. I, y, h represent input feature, label, and hidden representation, respectively. The full results are listed in Appendix F.2.
Ex -ate
ShapelD
DTI
Model
RMSE +
MSE 4
Avg. R1
ERM/MAML mixup/MetaMix
0.0236
4.698
0,464
0.0239
4.275
0.465
d( xy)
0.0212
4.539
0.478
di @ Mn my vs)
0.0212
4.395
0.484
dihu, by)
0.0213
4,202
0.483
d(k, @ Ms. Ay 4y))
0.0208
4.176
0.487
d(y., 25) (C-Mixup) 0.0203
4.024
0.498
The results reported in T We find that (1)
feature distance d(I;; xT) does benefit performance compared with ERM and vanilla mixup in all cases since it selects more reasonable example pairs for mixing: (2) representation distance d(hs, hy) outperforms feature distance d(z;, 2t)) since it captures high-level features; (3) though involving label in representation distance is likely to overwhelm the effect of labels, the better performance of d(h, b yt, fy + yj) over d(h4, h2) indicates the effectiveness of regression labels in measuring example distance, which is further verified by the superiority of C-Mixup. computational efficiency in Appendix A.3 indicates that C-Mixup is much more computationally efficient than other distance measurements.
(4) Our discussion of
III. How does the choice of bandwidth affects the performance? Finally, we analyze the effects of the bandwidth a in Eqn. (6). The performance with respect bandwidth of Exchange-Rate and DTI visualized in Figure respectively. We point out that if the bandwidth is too large. the results are close to vanilla mixup or Manifold Mixup, depending on which one is used. Otherwise, the results are close to ERM if the bandwidth is too small. According to the results, we see that the improvements of C- Mixup is somewhat stable over different bandwidths. Most importantly, C-Mixup yields a good model for a wide range of bandwidths, which reduces the efforts to tune the bandwidth for every specific dataset. Additionally, we provide empirical guidance about how to pick suitable bandwidth in Appendix F.3.I.
0.024
0.49
0.023
0.02
ERM
0.48
Miuu
0.021
0.47
C.
16-9
10-1
161
10-1 Bandwitth
Sandwitth
(a) Exchange-Rate
(b): DTI
Figure 5: Sensitivity analysis of bandwidth. mix represents the better mixing approach be- tween mixup and Manifold Mixup.

---

6
Related Work
Data Augmentation and Mixup. Various data augmentation strategies have been proposed to improve the generalization of deep neural networks, including directly augmenting images with man- ually designed strategies (c.g., whitening, cropping) [37], generating more examples with generative
models 33 7
88 85 and automatically finding augmentation strategies [10 11 43] Mixup [82] and 32
its variants 99 12 20 24
26
33 44 45 [2, 66668 8I 87 propose to improve generalization
by linear interpolating input features of a pair of examples and their corresponding labels. Though mixup and its variants have demonstrated their power in classification [22, sequence labeling [84), and reinforcement learning [70), systematic analysis of mixup on different regression tasks is still underexplored. The recent MixRL. (29) learns a policy network to select nearby example pairs for mixing, which requires substantial computational resources and is not suitable to high-dimensional real-world data. Unlike this method, C-Mixup instead adjusts the sampling probability of mixing pairs based on label similarity, which makes it much more efficient. Furthermore, C-Mixup, which focuses on how to select mixing examples, is a complementary method over mixup and its representative variants (see more discussion Appendix A.4). Empirically, our experiments show the effectiveness and compatibility of C-Mixup on multiple regression tasks in Section 5.1
Task Generalization. Our experiments extended C-Mixup to gradient- meta-learning, aiming to improve task generalization. In the literature, there are two lines of related works. The first line of
research directly imposes regularization on meta-learning algorithms 21 30
63
79). The second
line of approaches introduces task augmentation to produce more tasks for meta-training, including
imposing label noise (55), mixing support and query sets in the outer-loop optimization (49 74, and directly interpolating tasks to densify the entire task distribution [77]. C-Mixup is complimentary to
the latter mixup-based task augmentation methods. Furthermore, Section 5.2 indicates that C-Mixup
empirically outperforms multiple representative prior approaches [79 55
Out-of- cettof-Disttibution Robustness. Many recent methods aim to build machine learning models that are robust to distribution shift, including learning invariant representations with domain alignment [17 80
41 42 46, 65
72
86
using explicit regularizers finding a invariant predictors that performs 33 (5). Recently, LISA [76] cancels out
well over all domains
23
31 36
correlations and learns invariant predictors by mixing examples either with the same label but different domains or with the same domain but different labels. While related, C-Mixup can be considered as a more general version of LISA that can be used for regression tasks, Besides, unlike LISA, C-Mixup do not use domain annotations, which are often expensive to obtain.
7 Conclusion
In this paper, we proposed C-Mixup, simple yet effective variant of mixup that is well-suited to regression tasks deep neural networks. Specifically, C-Mixup adjusts the sampling probability of mixing example pairs by assigning higher probability to pairs with closer label values. Both theoretical and empirical results demonstrate the promise of C-Mixup in improving in in-distribution generalization, task generalization, and out- -ott--iistributionn robustness.
Theoretical future work may relax the assumptions in our theoretical analysis and extend the results to more complicated scenarios. We also plan to analyze more properties of C-Mixup theoretically, e.g., the relation between C-Mixup and manifold intrusion (2 in regression. Empirically, we plan to investigate how C-Mixup performs in more diverse application tasks such as semantic segmentation, natural language understanding, reinforcement learning.
Acknowledgement
We thank Yoonho Lee, Pang Wei Koh, Zhen- Yu Zhang, and members of the IRIS lab for the many insightful discussions and helpful feedback. This research was funded in part by JPMorgan Chase & Co. Any views or opinions expressed herein are solely those of the authors listed, and may differ from the views and opinions expressed by JPMorgan Chase & Co. or its affiliates. This material not a product of the Research Department of J.P. Morgan Securities LLC. This material should not be construed as an individual recommendation for any particular client and is not intended as a recommendation of particular securities, financial instruments or strategies for a particular client. This material does not constitute solicitation or offer in any jurisdiction. The research was also supported by Apple, Intel and Juniper Networks. CF is CIFAR fellow. The research of Linjun Zhang is partially supported by NSF DMS-2015378.
10

---

References
[1] Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. 2021.
[2] Magne Aldrin. Cmu statlib dataset, 2004.
Last accessed on May. 22th, 2022.
[3] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04344 2017.
[4] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization, arXiv preprint arXiv: MXXYv:9007.028800 0000990 2019.
[5] Raphael Baena, Lucas Drumetz, and Vincent Gripon. Preventing manifold intrusion with locality: Local mixup. arXiv preprint arXiv:2201 000368, 2022.
[6]
Mark Blair, Joe Thompson, Andrew Henrey, and Bill Chen. SkillCraftl Master Table Dataset. UCI Machine Learning Repository, 2013.
[7] Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Hammers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueckert. Gan augmentation: Augmenting training data using generative adversarial networks. arXiv preprint arXiv: 1810. 2018
[8] Pope D. & Marcolini Michael Brooks, Thomas. Airfoil Self-Noise. UCI Machine Learning Repository, 2014.
[9] Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12135-1211 12144, 2022.
[10] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113-123, 2019.
[11] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[12] Terrance Vrie and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 045520 2017.
[13] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:// archive. ics uci edu/ml
[14] Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. arXiv preprint 1rXiv:1710.1162.. 2017.
[15] Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In ICMI, pages 1050-1059, 2016.
[16] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[17] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):209-20300 2016.
[18] Ning Gao, Hanna Ziesche, Ngo Anh Vien, Michael Volpp, and Gerhard Neumann. What matters for meta-learning vision regression tasks? 2022.

---

[19] Rong Ge, Rohith Kuditipudi, Zhize Li, and Xiang Wang. Learning two-layer neural networks with symmetric inputs. In International Conference on Learning Representations, 2019.
[20] Kristjan Greenewald, Anming Gu, Mikhail Yurochkin, Justin Solomon, and Edward Chien. k- mixup regularization for deep learning via optimal transport. arXiv preprint N.Ni::206.002333 2021.
[21] Simon Guiroy, Vikas Verma, and Christopher Pal. Towards understanding generalization in gradient-based meta-lcarning. arXiv preprint arXiv:1907, 194,,07777,77877, 2019.
[22] Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3714-3722, 2019.
[23] Ruocheng Guo, Pengchuan Zhang, Hao Liu, and Emre Kiciman. Out-of-distribution pre- diction with invariant risk minimization: The limitation and an effective fix. arXiv preprint arXiv:2101.0773: 2021.
[24] Dan Hendrycks, Norman Mu, Ekin Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method improve robustness and uncertainty. arXiv preprint arXiv: 12.02781, 2019.
[25] Arthur E. Hoerl and Robert W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems.
Technometrics, 12()):55677, 1970. doi: 10.1080/
00401706. 104844486344, 10488634. URL https: wtwp://ana.oooiinmmmmmmmm com/ doi wwwddbsi//bb//ee abs/
1080/004C1700 1970. 1020..1440 10488634
[26] Minui Hong, Jinwoo Choi, and Gunhee Kim. Stylemix: Separating content and style for enhanced data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14862-14870, 2021.
[27] Joel L Horowitz. Semiparametric and nonparametric methods in econometrics, volume 12. Springer, 2009.
[28] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley. Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for therapeutics. arXiv preprint (tt.::2022.095455 2021.
[29] Seong-Hyeon Hwang and Steven Euijong Whang. Mixrl: Data mixing augmentation for regression using reinforcement learning. arXiv preprint arXiv:2/06.03374, 2021.
[30] Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 11719-11727, 2019.
[31] Kia Khezeli, Arno Blaas, Frank Soboczenski, Nicholas Chia, and John Kalantari. On invariance penalties for risk minimization. arXiv preprint arXiv:2106.09771 2021.
[32] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pages 5275-5285. PMLR, 2020.
[33] Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. arXiv preprint arXiv:2102.03000 2021.
[34] Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, al. Wilds: A benchmark in-the wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[35] Charles Kooperberg. Statlib: an archive for statistical software, datasets, and information. The American Statistician, 51())998, 1997.
[36] Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal invariant predictor. arXiv preprint arXiv:2008.01880 2020.
12

---

[37] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Annual Conference on Neural Information Processing Systems, pages 1097-1105, 2012.
[38] Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In NeurlPS, pages 950-957, 1992.
[39] David Krueger, Ethan Caballero, Joem- Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrap- olation (rex). In International Conference Machine Learning, pages 5815-5826. PMLR, 2021.
[40] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 4/ International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 95-104 2018.
[41] Haoliang Li. Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 5400-5409, 2018
[42] Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for distiiiiuitiiiioonnnn generalization. In Advances in Neural Information Processing Systems, 2022.
[43] Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32, 2019.
[44] Jihao Liu, Boxiao Liu, Hang Zhou, Hongsheng Li, and Yu Liu. Tokenmix: Rethinking image mixing for data augmentation vision transformers. arXiv preprint arXiv:2207.0844 084099 2022.
[45] Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Jianzhu Guo, and Stan Z Li. Automix: Unveiling the power of mixup for stronger classifiers. In ECCV, 2022.
[46] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 105. PMLR, 2015.
[47] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. International Conference on Learning Representations, 2019.
[48] Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv:1709.0195: 2017.
[49] Renkun Ni, Micah Goldblum, Amr Sharaf, Kezhi Kong, and Tom Goldstein, Data augmentation for meta-learning. JCML, 2021.
[50] David Ouyang, Bryan Hc, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul A Heidenreich, Robert A Harrington, David H Liang, Euan A Ashley, et al. Video-based ai for beat-to-beat assessment of cardiac function. Nature, 580(7802):252-256 2020.
[51] Hakime Özüürk, Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep drug-target binding affinity prediction. Bioinformatics, 34(17):1821-882 2018.
[52] Joonhyung Park, June Yong Yang, Jinwoo Shin, Sung Ju Hwang, and Funho Yang. Saliency grafting: Innocuous surribttionguuidee mixup with calibrated label mixing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7957-7965, 2022.
[53] James L Powell, James H Stock, and Thomas M Stoker. Semiparametric estimation of index coefficients. Econometrica: Journal of the Econometric Society, pages 1403-1430, 1989.
[54] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representations, 2020.
13

---

[55] Janarthanan Rajendran, Alex Irpan, and Eric Jang. Meta-learning requires meta meta- augmentation. In International Conference on Neural Information Processing Systems, 2020.
[56] Michael Redmond. Communities and Crime. UCI Machine Learning Repository, 2009.
[57] Philippe Rigollet and Jan-Christian Hütter. High dimensional statistics.
[58] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.
[59] Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937, 2021.
[60] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMIR, 15(1):1929-1958 2014.
[61] Charles Stein, Persi Diaconis, Susan Holmes, Gesine Reinert, et al. Use of exchangeable pairs in the analysis of simulations. In Stein's Method, volume 46, pages 1-26. Institute of Mathematical Statistics, 2004.
[62] Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear repre- sentations. In International Conference on Machine Learning, pages 10434-10443, PMLR, 2021.
[63] Hung- Yu Tseng, Yi-Wen Chen, Yi-Hsuan Tsai, Sifei Liu, Yen-Yu Yu Lin, and Ming-Hsuse Yang. Regularizing meta-learning via gradient dropout. In Proceedings of the Asian Conference on Computer Vision, 2020.
[64] Alexandre Tsybakov. Introduction to nonparametric estimation, 2009. URL https://doi. org/10. 10170b/1907//1379 Revised and extended from the, 9(10), 2004.
[65] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell, Deep domain confusion: Maximizing for domain invariance. arXiv preprint
2014.
[66] AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bac, et al. Salien- cymix: A saliency guided data augmentation strategy for better regularization. arXiv preprint arXiv:2006.0179/, 2020.
[67] Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. Alignmixup: Improving representations by interpolating aligned features. In Proceedings of the IÉEE/CVF Conference Computer Vision and Pattern Recognition, pages 19174-19183 2022.
[68] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR, 2019.
[69] Roman Vershynin. High-dimensional probability: An introduction with application in data science
[70] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforce- ment learning with mixture regularization. Advances in Neural Information Processing Systems, 33:7968-7977 2020.
[71] Yu Xiang, Roozbeh Mottaghi. and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In WACV, pages 75-82. IEFE, 2014.
[72] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang Adversarial domain adaptation with domain mixup. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6502-6509, 2020.
14

---

[73] Zhuoran Yang, Krishnakumar Balasubramanian, and Han Liu. High- non gaussian single index models via thresholded score function estimation. In International conference on machine learning, pages 3851-3860. PMLR, 2017.
[74] Huaxiu Yao, Longkai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, and Zhenhui Li. Improving generalization in meta-learning via task augmentation. In Proceeding of the eiihth International Conference on Machine Learning. 2021.
[75] Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. In Thirty-sixhh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[76] Huaxiu Yao. Yu Wang, Sai Li, Linjum Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-o--distriuttion robustness via selective augmentation. arXiv preprint arXiv:2201.00299, 2022.
[77] Huaxiu Yao, Linjun Zhang, and Chelsea Finn. Meta-learning with fewer tasks through task interpolation. In Proceeding of the 10th International Conference on Learning Representations, 2022.
[78] Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature Communications, 2020.
[79] Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta- learning without memorization. In International Conference on Learning Representations, 2020.
[80] Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-real general- ization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2100-2110, 2019.
[81] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023-6032, 2019.
[82] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. 2018.
[83] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani. and James Zou. How does mixup help with robustness and generalization? In ICLR, 2021.
[84] Rongzhi Zhang, Yue Yu, and Chao Zhang. Seqmix: Augmenting active sequence labeling via sequence mixup. arXiv preprint arXiv:2010.02322, 2020.
[85] Allan Zhou, Fahim Tajwar, Alexander Robey, Tom Knowles, George J Pappas, Hamed Hassani, and Chelsea Finn. Do deep networks transfer invariances across classes? arXiv preprint 20X3v:2003.0973, 2022.
[86] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep advain-adversarial image generation for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13025-13032, 2020.
[87] Jianchao Zhu, Liangliang Shi, Junchi Yan, and Hongyuan Zha. Automix: Mixup networks for sample interpolation via cooperative barycenter learning. In European Conference on Computer Vision, pages 6449. Springer, 2020.
[88] Barret Zoph, Ekin D Cubuk, Golnaz Ghiasi, Tsung- Yi Lin, Jonathon Shlens, and Quoe V Le. Learning data augmentation strategies for object detection. In European conference on computer vision, pages 566-583. Springer, 2020.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix
(c) Did you discuss any potential negative societal impacts of your work? [N/A] C-Mixup
a general machine learning method and we believe that has no potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section and Appendix B
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See the code in
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C.2, D.2, and E.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix C, D, E,
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C.2, D.2, and E.2.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [Yes] See the code in https://githb..com/// httpaluhooooouuuuuaooC-C//C
(d) Did you discuss whether and how consent was obtained from people whose data you't 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "fszTYQufrlX", "reviewer": "Reviewer_13Sq", "summary": "This paper proposed a mixup algorithm (mixReg) to improve generalization on regression tasks. Different from mixup methods for classification tasks, the proposed method adjusts the sampling probability based on the similarity of labels. Theoretical analysis shows that the proposed mixReg can achieve a lower mean square error and improve in-distribution generalization, task generalization, and out-of-distribution robustness. Experiments on eleven datasets prove the improvements of the proposed mixReg.", "questions": "Why mixReg work? The proposed mixReg only takes the intra-cluster (semantically similar) samples to perform vanilla mixup. Although the theoretical analysis proves the superior of mixReg upon vanilla mixup (and its variants) and mixup with input-similarity, the compared methods usually yield worse performances than ERM. The cause of bad performances of vanilla mixup might be the unreliable inter-cluster samples, and mixReg improves the mixup performance by removing these samples (similar to AdaMixup [2]). However, mixReg might degenerate to ERM with the large sample size N and the small bandwidth. What if we have a better inter-class mixup policy like PuzzleMix [3] and AutoMix [4]? We might transform regression tasks into classification tasks by splitting the regression targets into C bins and training the classifier to generate more reliable inter-class mixup samples [3, 4].\n\n[1] CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In ICCV, 2019.\n\n[2] MixUp as Locally Linear Out-Of-Manifold Regularization. In AAAI, 2019.\n\n[3] Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup. In ICML, 2020.\n\n[4] AutoMix: Unveiling the Power of Mixup for Stronger Classifiers. In ECCV, 2022.\n\n\n================ Post-rebuttal ================\n\nSince the author has addressed my concerns and updated the revision according to my comments during the rebuttal period, I change my score from 5 to 6.", "limitations": "The author discusses the limitations of this paper in the appendix, which might be solved by further experiments. I do not find any negative social impact of this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n(1) This paper studies an interesting problem that applies mixup augmentations to general regression tasks. Since data augmentations in regression tasks are rarely studied, the proposed mixReg is the first work that introduces the general regression augmentation strategy.\n\n(2) The proposed method in three sceneries is well supported by theoretical analysis and proofs.\n\n(3) Extensive experiments verify the effectiveness of the proposed mixReg from three aspects.\n\n### Weakness\n\n(1) Details of selecting mixup samples according to Eq. (6) are vague. There might be some thresholds to determine the intra-cluster samples based on Eq. (6). Meanwhile, the way to determine the bandwidth hyper-parameter is not clear. It is better to provide an empirical range of bandwidths for easy implementation in various scenarios.\n\n(2) Although the author performs extensive experiments on many datasets, more mixup methods with various mixing policies should be compared. For example, CutMix [1] can be performed on 2D images and 1D time series (replacing the segment). See questions for more details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cJ-oh_4QhQd", "reviewer": "Reviewer_t9o8", "summary": "This paper proposes a simple method for regression tasks. It improves the mixup data augmentation by selectively interpolating examples with similar labels, or re-weighting the probability of mixup pairs. Extensive experiments on various data modalities have been conducted to validate its effectiveness.", "questions": "1. lines 50-56 are hard to be understood. Does Figure 1(a) only present the ShapeNet1D Pose Prediction task? At the same time, why do pair 1 and pair 3 have close input similarities? Besides, why is the regression task more sensitive to noise than the classification task? Is there any literature to support this claim, and what's noise in the whole paper (Can an image have noise?)?\n\n2. The choice of bandwidth. I first appreciate the ablation study of bandwidth sigma in section 4.5, but it has not convinced me of the claims that mixReg reduces the efforts to tune the bandwidth. For example, the label of ShapeNet1D has a range of [0, 360], and the other datasets may have a much different value range. How to deal with this problem to choose the correct sigma? Why is the output normalized to [0, 10] in line 270? The xlabels of Figure 3 are incorrect.\n\n3. An essential property of mixup is the robustness of label noise. As mixReg uses the distances between labels to select the pairs, will the label noise harm the generalization of mixReg?\n\n4. Can mixReg be applied to more complex and real-world natural tasks (eg, counting or pose estimation)? Mixup has shown its effectiveness on various classification tasks such as ImageNet. This paper only presents the experiments on some simple datasets, such as Shape1D and etc.\n\n5. Will the code be publicly available?", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* clear motivation and good presetation;\n* Extensitive expermients;\n\n### Weaknesses\nI have not found the critical drawbacks of this paper. However, I have not carefully checked the correctness of sections 3.2, 3.3, and 3.4. In my opinion, section 3.1 has thoroughly presented the proposed method, and the rest provides little information but harmed the readability.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "TX27y5JcMb2", "reviewer": "Reviewer_VT7i", "summary": "In this work, the authors propose to use mixup to improve generalization in regression tasks. They argue that when mixup is directly using in regression, it may generate arbitrary labels which are incorrect as the linear assumption may not hold. The authors aim to overcome this drawback by adjusting the sampling probability based on the similarity of the labels. They further show that this label similarity obtains smaller mean square error and also improves out-of-distribution robustness. Interpolating examples with similar labels mitigates the effect of domain-specific information and pushes invariant representations. The authors evaluate their work on several benchmarks and show its effectiveness on in-distribution generalization, task generalization and out-of-distribution robustness.", "questions": "#### **Weakness**/ **Questions**\n\nI appreciate the authors efforts in developing an interesting idea. However, I have the following concerns/ queries which I would like the authors to answer:\n- I quite did not understand the difference between sampling probability and the interpolation factor $\\lambda$ typically used in mixup papers.\n- In L133, the authors say that its meaningful to use $d(i,j) = ||y_i - y_j||^2$. However, what is the dimension of $y$? is it a one-hot vector. This is quite unclear again. Also instead of using the $L_2$ distance, how about using cosine similarity in lower-dimensional embeddings instead of $L_2$ distance in the label space?\n- In the original mixup paper [62], the authors show that interpolating examples from semantically similar manifolds leads to sub-optimal performance and the best performance is achieved by randomly mixing samples from the data manifold. However, in MixReg, the authors claim that mixup samples from semantically similar data manifold results in the best performance. Could the authors please explain its significance?\n- Continuing from the above point, interpolating between semantically similar points in the data manifold could lead to a problem of manifold intrusion [1] - Given two examples with different class labels, the interpolated example may actually lie in a region associated with a third class in the feature space. Im curious to know if the authors face this problem in MixReg, if not, what is being done to avoid manifold intrusion.  \n[1] Guo et al., Mixup as locally linear out-of-manifold regularization. In AAAI, 2019.   \n\n- Could the authors evaluate their work on large scale tasks such as semantic segmentation which is also a pixel-wise regression task on Pascal-VOC or MS-COCO? How does one apply mixup to such kind of tasks in general?\n- The authors have missed citing several state-of-the-art mixup works such as PuzzleMix [2], Co-Mixup [3], SaliencyMix [4], AlignMixup [5], StyleMix [6], StyleCutMix [6],  AutoMix [7] etc. It would be nice to discuss these papers in the related work section.    \n\n[2] Kim et al., Puzzle mix: Exploiting saliency and local statistics for optimal mixup. ICML 2020.   \n[3] Kim et al., Co-mixup: Saliency guided joint mixup with supermodular diversity. ICLR 2021.   \n[4] Uddin et al. Saliencymix: A saliency guided data augmentation strategy for better regularization. ICML 2021.   \n[5] Venkataramanan et al., AlignMixup: Improving Representations By Interpolating Aligned Features. CVPR 2022.   \n[6] Hong et al., Stylemix: Separating content and style for enhanced data augmentation. CVPR, 2021.    \n[7]  Zhu et al., Automix: Mixup networks for sample interpolation via cooperative barycenter learning. ECCV, 2020\n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### **Strengths**\nThe following are some strengths of the work\n- The authors argument that when mixup is directly applied to regression task generates incorrect labels due to the failure of the linear assumption rule is quite interesting. Its also quite interesting that mixup in regression tasks has been understudied and the authors aim to bridge this gap.\n- The experimental analysis is quite exhaustive and the authors show the performance of MixReg on many different benchmarks and in-distribution generalization, task generalization and out-of-distribution robustness.\n- The paper is also quite clear to read and is well presented.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept