Anchor Data Augmentation
Authors: Nora Schneider
Shirin Goshtasbpour
Fernando Perez-Cruz
Venue: NeurIPS 2023 poster (2023)
Keywords: Data Augmentation
Regression
Deep Learning
TL;DR: We present a new data augmentation algorithm inspired from the causality and distributional robustness literature to aid model generalization in regression problems. We demonstrate its efficacy on multiple synthetic and real world datasets.

Abstract: We propose a novel algorithm for data augmentation in nonlinear over-parametrized regression. Our data augmentation algorithm borrows from the literature on causality. Contrary to the current state-of-the-art solutions that rely on modifications of Mixup algorithm, we extend the recently proposed distributionally robust Anchor regression (AR) method for data augmentation. Our Anchor Data Augmentation (ADA) uses several replicas of the modified samples in AR to provide more training examples, leading to more robust regression predictions. We apply ADA to linear and nonlinear regression problems using neural networks. ADA is competitive with state-of-the-art C-Mixup solutions.

Anchor Data Augmentation
Nora Schneider'
Shirin Goshtasbpour 1.2
Fernando Perez-Cruz 1,2
CCmppter Science Department, ETH Zurich, Zurich, Switzerland Swiss Data Science Center, Zurich, Switzerland
nschneide@sttudent ethz. ch shirin. goshtaabpoouruinf ethz. ch fernando, perezcruzosdsc ethz. ch
Abstract
We propose a novel algorithm for data augmentation in nonlinear over-parametrized regression. Our data augmentation algorithm borrows from the literature on causal- ity and extends the recently proposed Anchor regression (AR) method for data augmentation, which is in contrast the current state-of-the-at domain-agnostic solutions that rely on the Mixup literature. Our Anchor Data Augmentation (ADA) uses several replicas of the modified samples in AR to provide more training exam- ples, leading to more robust regression predictions. We apply ADA to linear and nonlinear regression problems using neural networks. ADA is competitive with state-of-the-ane C-Mixup solutions.
Introduction
Data augmentation is one of the key ingredients of any successful application of machine learning classifier. The first example that typically comes to mind augmentation in the now-famous Alexnet paper 26 Data augmentation algorithms come in different flavors, and they mostly rely on the expectation that small perturbations, invariances, or symmetries applied to the imput will not change the class label. That way, alterations of the available examples for training. These transformations modify the input distribution make the algorithm more robust for cases where the distribution of the test may differ from that of the training set. We refer the reader to the related work section (Section 2.1 for an overview and description of different data augmentation strategies.
in-depth description the data
present 'fresh new" samples as
The literature for data augmentation regression is slim. The paper on Mixup augmentation S11 proposes simple and general scheme for data augmentation using convex combinations of samples. The authors only apply their data augmentation proposal to classification problems. They conjecture in the discussion that the application to regression is straightforward, however, this is not the case in practice. Mixup is thcoretically analyzed in [5] 52 as regularization technique for classification and regression problems. However, it is only illustrated in classification problems.
The Mixup algorithm has been extended to regression problems in [8 40 in which the authors
explain that Mixup cannot be blindly applied to regression problems. To our knowledge, these are the only two papers in which data augmentation for regression is proposed. RegMix [18] relies on hard-to-train prior neural network controller before augmenting the data using Mixup strategy. 40 a method proposed more recently, solves some of the issues limiting the standard
C-Mixup
Mixup algorithm for regression problems. The authors propose to mix only closeby samples in the
'Our Python implementation of ADA is available http:://ggituub.cco//rrreeeeeeee auchordataaugnentttiion///
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

output space (i.e., samples which have close enough labels). This strategy is only valid when the target variables are monotonic with the input and applied in a transformed space. The authors present comprehensive results in data augmentation for in-distribution generalization, task generalization and -ut-of-distribution robustness.
In this paper, we rely on the causality literature to provide a different avenue for augmenting data in regression problems. Causal discovery finds the causes of a response variable among a given set of observations or helps to recognize the causal relations between a set of variables [39] These causes allow us to understand how these relations will change if we were to intervene in a subset of the (input) variables or what would be the effect on the output. So. in general. the regression model will be robust to perturbations in the imput variables making the prediction less sensitive to changes in the distribution of the test set. For example, the authors in A0 use the invariance property for prediction to perform causal inference. turn, Anchor Regression (AR) builds upon the causality literature to obtain robust regression solutions when the input variables have been perturbed [422] The procedure relies on anchor variables capturing the heterogeneity within dataset and parameter y that measures the deviation with respect the least square solution. Once the values of the anchors are known, AR modifies the data and obtains the least square solution, as detailed Section 2.2
In this paper, we propose Anchor Data Augmentation (ADA) to augment the training dataset with several replicas of the available data. We use a simple clustering of the data to encode homogeneous group of observations and use different values of y to robustify the solution to different strengths of potential distribution shifts. In every minibatch, we sample y from a predetermined range around
= 1. As AR was developed for linear regression, the data augmentation strategy needs to be modified for nonlinear regression accordingly. We validate ADA for in-distribution generalization and out-of -itt----iittiittoon robustness under the same conditions proposed in C-Mixup [ as well as some illustrative linear and nonlinear regression examples. In the replicated experiments, ADA is competitive superior to other augmentation strategies such as C-Mixup, although some datasets the performance gain is marginal.
The rest of the paper is organized as follows: First, we provide background information in Section 2 We give brief overview of related work on data augmentation in Section 2. and summarize the key concepts on Anchor Regression in Section 2.2 Second, Section B shows how we extend Anchor Regression and introduces ADA. Section 4 reports empirical evidence that our approach can improve predictions, especially in over-parameterized settings. We conclude the paper in Section 5
2 Background
2.1 Data Augmentation
Many different data augmentation methods have been proposed in recent years with several appli- cations in mind. Still most augmentations we mention here use human-designed transformations based on domain knowledge which leave the target variable invariant. For instance, Cutout [0 is an image-specific augmentation technique that is successfully used to train models on CIFAR10 and CIFAR100 [25). but was determined to be unsuitable for larger image datasets like ImageNet with higher resolution Φ. Other augmentation methods for images such as random crop, horizontal or vertical mirroring, random rotation, translation
29 [ may similarly apply certain group of image datasets while being inapplicable to others, e.g. datasets of digits and letters.
In an attempt to automate the augmentation process and reduce human involvement, policy or search- based automated augmentation methods were developed. In AutoAugment 7 a neural network is trained with Reinforcement Learning (RL) to combine assortment of transformations in varying strengths to apply on samples of a given dataset and improve the model accuracy. Methods such as RandAugment S Fast AutoAugment 30,, UniformAugment at reducing the cost of the pretraining search phase in automated augmentation with randomized transformations and reduced search space.
3 and TrivialAugment [3 aim
Alternatively, in order to adapt the augmentation policy to the model during training, Population- Based Augmentation [I and Online Hyperparameter Learning [31 use multiple data augmentation workers that are updated using evolutionary strategies and RI., respectively. Adversarial AutoAugment [ and AugMax 47 optimize for the augmentation policy that deteriorates the training accuracy

---

and improves its robustness. DivAug augmented data.
62 finds the policy which maximizes the diversity of the
Having a separate search phase for optimal augmentation policy is computationally expensive and may exceed the required computation to train the downstream model B 45
In addition, these
methods and their online counterparts need to be trained separately on every single dataset. While OngineAugment 44 and DDAS exploit meta-learning to avoid this problem, they still rely on set of predefined class invariant transformations that require domain-specific information.
Generic transformations such Gaussian adversarial noise [00
28
45
and dropout
also effective in expanding the training dataset. Generative models such as Generative Adversarial Networks (GAN) [ and Variational Auto Encoders (VAE) ?4 are trained in [[] 44
to synthesize
samples close to the low dimensional manifold of the data for classification.
Mixup 51 is popular data augmentation using a convex combination of pairs of samples from different classes and their softened labels for augmentation. Mixup is only evaluated on classification problems, even though it is claimed that the application to regression is straightforward. Various extensions of Mixup have been proposed to prevent data manifold intrusion [46, use more complex mixing strategies were predominantly designed to excel in classification tasks. In particular, Mixup for regression was studied in 5 [
33
50 account for saliency in augmented samples 20 P1. These methods
49 52 but it reported to adversely impact the predictions in regression
problems when misleading augmented samples are generated from a pair of faraway samples.
2.2 Anchor Regression
We summarize the key concepts of Anchor Regression (AR) presented in 42 Let X E X and
y E y be the predictors and target variables sampled from distribution (X, y) ~ Pirains C Rd and
O' C R. Traditionally, causal framework models the relation of y and X to accurately predict the value of y under given interventions or arbitrary perturbations on X. A commonly held assumption is that the underlying causal relation among variables remains the same while the sampling distribution Piein is altered by the intervention shift or the applied perturbation. For instance, if the distribution Pirain is induced by unknown linear causal model, then the causally optimal parameters can be expressed as the solution to the optimization problem:
beausal = arg min max max Epl(y Epl(y-x"Pr bb
(1)
PEP
where P is the class of distributions containing all interventions on components of X [41. Therefore, causal parameters provide distributionally robust predictions that optimal under the intervention in P. In comparison, Ordinary Least Squares (OLS):
bols = arg min Ep Ky X bbb],
may lead to arbitrarily large predictive errors on distributions in P. On the other hand, on Pirain- causal parameters beausal lead to conservative predictions, while bols presents optimal least squared performance.
To trade-off predictive accuracy the training distribution with distribution robustness and to enforce stability over statistical parameters, AR proposes to relax the regularization in the optimization problem in to a smaller class of distributions P.
Assume that X and y are centered and have finite variance. We use A Ra (called anchors) to denote the exogenous (random) variables in X which generate heterogeneity in y. We further denote the Lr-preecction on the linear span of the components of A with PA and Id(y) = y. Under linear assumption between A and (X, y), we can write the relaxed optimization problem as:
by.A arg min E..... [(Idd - (V)(V - xTb)) + tEE.... (PA(y b)))),
(3)
where 7 > 0 is hyperparameter. The first term of the AR objective in Equation 3 is the loss after "partialling out" the anchor variable, which refers to first linearly regressing out from X and y and subsequently using OLS on the residuals. The second term is the well-known estimation objective used the Instrumental Variable setting [T]. Therefore, for different values of 7 AR interpolates
between the partialling out objective (y 0) and the IV estimator (y + oo) and coincides with
3

---

OLS y = 1. The authors show that the solution of AR optimizes worst-case risk under shift- interventions on anchors up to given strength. This in turn increases the robustness of the predictions to distribution shifts at the cost of reducing the in-distribution generalization.
In the finite-sample case with observations from Purin. let matrix X E R contain the obser- vations of X and let Y E IR" be the vector of corresponding targets. Similarly, we denote the matrix containing the observations of A with A |2* and we use IIA A (A"A) A as the projection operator on the column space of the anchor matrix A where A denotes the pseudo-inverse of matrix A. Further, denotes the identity matrix. Then, the finite-sample optimization regression problem can be written as
by A arg min (I IIA)YY X6) + AY X6)11
The AR regression estimate A can be obtained by applying the OLS solution modified set of inputs and outputs:
X X 1
(5)
Y (V7- 1)ILAY
(6)
3 Anchor Data Augmentation
In this section, introduce Anchor Data Augmentation (ADA), domain- -immiinnnddeendent data augmen- tation method inspired by AR. ADA does not require previous knowledge about the data invariances nor manually engineered transformations. As opposed to existing domain-agnostic data augmen- tation methods 110 45 2rd. we do not require training of an expensive generative model, and the augmentation only adds marginally the computation complexity of the training. In addition, since ADA originates from causal regression problem, it can be readily applied to regression problems. Even when ADA does not improve performance, its effect on performance remains minimal.
Data augmentation aims to introduce informative data in addition to the original dataset during the training procedure of the model to improve its generalization. Similar to AR, ADA employs linear projection, given by the anchor variables A, to determine the most relevant perturbation directions based on the similarity of the samples. ADA inherits the generalization properties from AR. In (42), the authors recommend that the anchor variable can be set as an indicator of the datasels, where each dataset is homogeneous set of observations. A key insight of our work is that this can be achieved by clustering the data into g clusters. The matrix A E RX is then constructed as an indicator matrix with a one-hot encoding of the assigned cluster index per row. For our experiments, we use k-means clustering [35] to construct A. Further, in AR, only one value for ?y is used, which should be chosen based on the desired strength of perturbations on test datasets, in comparison to the training dataset [2). We suggest that the value of y is sampled from a distribution with density p(7). In our experiments, we use a uniform distribution between 1/a and a, where a is hyperparameter to be tuned.
ADA augments sample (X(), (()) by normalizing the original AR modifications and by 1 1+ 1) (TTA) (x/) (() to unify the noise level across the augmentations independent of the value of while approximately preserving the potentially nonlinear relation between X and y (see also section 3.2
x X ( OX
7 (+()--)),((I)))
( = YO (/T ( 7.A
(8)
1) (TA)))))))
where we denote (M) (M), and (M) denote respectively the i-th row, the j-th column and (i.j) component of some matrix M. As is standard practice, we rely on stochastic gradient descent to optimize our (nonlinear) regressors and apply ADA on each minibatch rather than the entire dataset.
ADA combines samples from the same cluster and generates augmented samples along the data manifold. For a general A, II provides a "collective" mixing approach for the samples in batch

---

groups)
(12 groups)
Figure 1: Comparison ADA augmentations on nonlinear Cosine data model. For a larger partition size, ADA augmentations are more accurate due to the high local variabilityy of the Cosine function. We used k-means clustering to construct A and 7 E (1/2 1.03/3,.000.4... 1.0 2.0).
44 (0004
Andhar Augiwrred Dam
Orgina Dam
by determining center, while 7 controls the extend of contraction expansion of the augmented
sample around this center. In particular, for one-hot encoding matrix A, A (0) X defines the centroid of the cluster to which sample belongs. Then, the modified samples are located on the ray that originates from the centroid and goes through the original data point (X()), Y( ). As increases, the augmented samples move towards their corresponding centroid and specifically, for they coincide with the original samples. Furthermore, the cluster size, regulated by the number of clusters g, directly impacts the number of samples mixed together; with smaller clusters, fewer samples are combined. Applying ADA on each minibatch introduces further diversity and enhances robustness, because the composition of samples being mixed together and the value of y changes in each minibatch. In Appendix A.2 we provide detailed explanation and analysis of the impact of ADA hyperparameters, q controlling the number of clusters and controlling the range of values for In Appendix B.4 we empirically show how regression performance varies with respect to these hyperparameters.
In Figure D we visually illustrate the augmentation effects of ADA. We uniformly sampled 30 data points between =3 (i.e. z4 ~ 14[-3,3) 3) and set the corresponding target variable as yi cos cos((T2) (Tw;;) without added noise. We then clustered this data in = 5 and 12 groups using k-means and applied eq. 7 and eq. ( the 30 samples with 7 (1/2, 2 1, 1,32,, 23 resulting in 150 augmented data points.
3.1 Comparison to C-Mixup
ADA can be interpreted as generalized variant of C-Mixup 49 In C-Mixup samples mixed in pairs, and the combination probability of each sample pair is given by the similarity their labels, measured by Gaussian kernel. Augmented samples between the pair. In contrast, ADA allows mixing multiple samples based on their cluster membership and the resulting augmentations that may reside in the convex hull of the original samples of cluster 7 2 1 or beyond it when < 1. In particular, ADA and C-Mixup augmentations would be similar if the anchor matrix A indicates pairs of samples weighted by the similarity of their labels and
then obtained as the convex combination
1.
3.2 Preserving nonlinear data structure
In the following show that the scaled transformations in eq. and eq. 6 preserve the nonlinear relationship, that we
the modified pair (X,.A, A) augment the dataset (X, Y). Let (X(), (+) be the ith sample from Purin corresponding the ith row of X and ith component of Y. When the data has nonlinear relation,
Y (i) + ((()
given the zero mean noise variable c we can alter the anchor loss accordingly 4.
arg minE bNONLIN,, fT.A
b,f man [((Id - P - - f + ( - (o))))
The AR modification Equations and do not preserve the nonlinear relation between the target and predictors,
( + (x((0))+)
with another zero mean variable ((0) operating as the observation noise in the augmented data. Therefore, we propose to further extend the original AR and perform the data augmentation with
5

---

scaled transformations to get the modified sample (X y.A. Y y.A) which approximately preserves the nonlinear relationship of sample (X ( as shown below.
We can rewrite ( 7.A in Equation 8 as
fx(
+ ( (I
YO A.A
2(1 7.A
+
1 1V)--)),(m ((6))
-(6) where ,, a zero mean noise variable and Fx(X)= = ()((X)), for continuously differentiable function f. we can use the first order Taylor expansion of (
((x [x In Appendix A.1
7.A around
to show that X A
y-0) 7.A
=
+
(0)
7.A
(10)
which approximately has the same nonlinear relation as the original model for small X (1)
{()
ll
(IAA) (X x 1l2
With the one-hot hot partitioning matrix, A (introduced in the previous section), the approximation of the true nonlinear model becomes accurate in partitions with small diameter (where we define partition diameter as the maximum distance of two samples ( and X in the same cluster).
3.3 Algorithm
Finally, in this section, present the ADA al- gorithm step by step (Algorithm/] [T to generate minibatches of data that can be used to train neu- ral networks (or any other nonlinear regressor) by any stochastic gradient descent method. As discussed previously, we propose to repeat the augmentation with different parameter combina- tions for each minibatch.
Algorithm 1 ADA: Minibatch generation
1: Input: L training data points (X, Y); prior distribution for 7: p(7) L q binary matrix A with per row indicating the clustering assignment for each sample.
Given centered training dataset (X, Y). its
2: Output: (X, Y)
clustering assignment A, and prior function p()), the ADA minibatch algorithms takes L random samples from the training set and its cor- responding rows in A and outputs L-sample mini-bath (X,AA Y,,A).
3: Sample y from P(r)
4: Projection matrix: II < A(A
5: for 0 to row of X do
X"+ (() ()((/N-)((T))"> 1+(/5-1),, (
6: 7.A
In order to do so, we first choose y according to the provided criterion p(7) (line 3). 'he cor- responding projection matrix II is computed from A (line 4). Finally, in lines five to seven, the transformation applied according to Equa-
7:
(0)
7.A
8: end for
9: return (X,,A, Y ,,,))
tions Z and g
4 Experiments
We experimentally investigate and compare the performance of ADA. First, we use ADA in an in-distribution setting for linear regression problem (Section 4.1.. in which we show that even in this case, ADA provides improved performance the low data regime. Second, in Section 4.2 we apply ADA and C-Mixup to the California and Boston Housing datasets as we increase the number of training samples. In the last two subsections, we replicate the in-distribution generalization (Section 4.3) and the out-o--distribution Robustness (Section 4.4) from the C-Mixup paper [49]. In
40 the authors further assess a task generalization experiment. However, the corresponding code was not publicly provided, and comparison could not be easily made.

---

4.1 Linear synthetic data
Using synthetie linear data, we investigate ADA can improve model performance an over- parameterized setting compared to C-Mixup, vanilla augmentation, or classical expected risk mini- mization (ERM). Additionally, we analyze the sensitivity of our approach to the choice of y and the number of augmentations.
Data: The generated data follows standard linear structure
Y = (x()) T b+ bo + +(
(11)
with ((), b b R 19 and ( (0),b, e( E R. The parameters are sampled randomly from a Gaussian distribution . NN, 1). We sample 20 different training datasets and one validation set with NO,, 0.13), x NO, I1s). For each training set, we take subsets with increasing number of samples to evaluate the methods on different levels of data availability. The subsets are hierarchically constructed (i.e., meaning smaller set is always subset of larger one). The validation set has 100,000 samples.
Models and Comparisons: We investigate and compare the impact of ADA using two different models with varying complexity: a linear Ridge regression and a multilayer perceptron (MIP) with one hidden layer and 10 units with ReL.U activation. Using an MIP with more hidden layers shows similar results (see Appendix B.I for details).
The ERM models only use the original data. We perform vanilla data augmentation by adding Gaussian noise e' ~ (0, 0.1²) to the output leaving the input unchanged. Next, we apply C-Mixup with bandwidth of 1 and set the a of the Beta- Beta-distribution to 2. Finally, we apply ADA with varying the number of obtained augmentations i {10, 100} and varying range of values for 7. To be precise, we define E {2, 4, 6, 8, 10} and specify B = 1 + e-1 i (with i E {1...,, k/2{) and
k/2
1 1, 31 B2/2-1, a.}. A is constructed using k- means clustering with 8. Y E (a 96/2-1 B1
For the Ridge regression model, we increase the dataset by a factor of 10 by sampling from the respective augmentation methods and subsequently compute the regression estimators. In contrast, for the MLP, we implement the augmentation methods on minibatch level. Specifically, we incorporate vanilla augmentation by adding Gaussian noise to each batch, apply C-Mixup after sampling from the beta distribution in each batch, and finally, apply ADA after sampling from the defined gamma values each batch.
We plot our results in Figure 2 First, as expected, Ridge regression outperforms the MLP model. Second, when there is little data availability, using ADA decreases the test error compared to ERM. The effect diminishes when the training dataset is sufficiently large, and all models converge the noise limit of 0.12. Third, vanilla augmentation achieves similar results ADA and C-Mixup for Ridge regression but not quite for the MLP. This suggests that ADA (and C-Mixup) are more meaningful than randomly adding noise and especially well suited for highly parameterized models as the MLP has almost 20 times more parameters than Ridge regression. In real-world applications, the value of e usually unknown, and choosing e' for vanilla augmentation not trivial. especially when the number of samples is small. Fourth, we conclude that generating more augmentations (100 instead of 10) further improves prediction error in vanilla and anchor augmentation (Appendix B Figure Di and the effectiveness of anchor augmentation is further increased as the range for is wider (Appendix B Figure IO). Finally, C-Mixup and ADA perform similarly with ADA having a tendency to achieve a lower test error.
Results:
In summary, even in the simplest of cases, in which we should not expect gains from ADA (or C-Mixup), these data augmentation strategies provide gains in performance when the number of training examples is not sufficient to achieve the error floor.
4.2 Housing nonlinear regression
We extend the results from the previous section to the California and Boston housing data and compare ADA to C-Mixup 40 datasets. Results are provided Appendix B.2 and also show the superiority of ADA over C-Mixup for data augmentation in the implemented experimental setup.
We repeat the same experiments on three different regression
7

---

Ridge Regression
hidden
(ours)
10
10099023
101
300
160
120
Figure 2: Mean Squared Error for Ridge Regression model and MLP model with varying number of training samples. For Ridge regression, vanilla augmentation and C-Mixup generate k 10 augmented observations per observations. Similarly, Anchor Augmentation generates A = 10 augmented observations per observation with parameter a 10.
Data: We use the California housing dataset 19 and the Boston housing dataset [4] The training
dataset contains up results as a function of the number of training points.
406 samples, and the remaining samples are for validation. We report the
Models and comparisons: We fit ridge regression model (baseline) and train MLP with one hidden layer with a varying number hidden units with sigmoid activation. The baseline models only use only the original data. We train the same models using C-Mixup with Gaussian kernel and bandwidth of 1.75. We compare the previous approaches to models fitted on ADA augmented data. We generate 20 different augmentations per original observation using different values for controlled via a = 4 similar to what was described Section The Anchor matrix is constructed using k-means clustering with q 10.
Results: We report the results in Figure First, we observe that the MLPs outperform Ridge regression suggesting nonlinear data structure. Second, when the number of training samples is low, applying ADA improves the performance of all models compared C-Mixup and the baseline. The performance gap decreases the number of samples increases. When comparing C-Mixup and ADA, we see that using sufficiently many samples both methods achieve similar performance, While on the Boston data, the performance gap between the baseline and ADA persists, on California housing, the non-augmented model fit performs better than the augmented one when data availability increases. This suggests that there is a sweet spot where the addition of original data samples is required for better generalization, and augmented samples cannot contribute any further.
Boston Housing
Califormia Housing
40 hidden units
1x
10 hidden units
4x 161-
Ridge Regression
6x
bogNNS))
4x181
3x
1x118-1
2x
4x
100
700
samp
Base
C-Mooup
ADX
Figure 3: MSE for housing datasets averaged over 10 different sain-validation -eest splits. On California housing Ridge regression performs much worse which is why is not considered further (see Appendix B.2).
4.3 In-distribution Generalization
In this section, we evaluate the performance of ADA and compare it to prior approaches on tasks involving in-distribution generalization. We use the same datasets as 40 and closely follow their experimental setup.
8

---

Data:
We use four of the five in-distribution datasets used in 49]. The validation and test data are expected to follow the same distribution as the training data. Airfoil Self-Noise (Airfoil) and NO2 [24] are both tabular datasets, whereas Exchange-
and Electricity [ are time series datasets. We divide the datasets into train-, validation- and test data randomly, as the authors of C-Mixup did. For Hchocardiogram videos 137 (the 5th dataset in 49 ), we could not replicate their preprocessing.
Local-Mixup [2] Manifold-Mixup [46,, Mixup [il and classical expected risk minimization (ERM), Following the work of 40
Models and comparisons: We compare our approach, ADA, to C-Mixup 49
we use the same model architectures: a three-layer fully connected network for the for the time series.
tabular datasets; and an LST-Attn 27
We follow the setup of 49 and apply C-Mixup,
Manifold-Mixugg Mixup, and ERM with their reported hyperparameters and provided code. For the ADA and Local-Mixup experiments, we use hyperparameter tuning and grid search to find the optimal training (batch size, learning rate, and number of epochs), and Local-Mixup parameters (distance threshold c) and ADA parameters (number of clusters, range of y, and whether to use manifold augmentation). We provide a detailed description in Appendix B.4 The evaluation metrics are Root Mean Squared Error (RMSE) and Mean Averaged Percentage Error (MAPE).
Results: We report the results in Table [ For full transparency. in the last row, we copy the results from 3 We assess that ADA is competitive with C-Mixup and superior to the other data augmentation strategies. ADA consistently improves the regression compared to ERM. Under the same conditions (split of data and Neural network structure), ADA is superior C-Mixup. But, the degree of improvement is marginal on some datasets and as we show in the last row, we could not fully replicate their results. The only data in which ADA is significantly better than C-Mixup and the other strategies is for the Airfoil data, in which ADA reduces the error by around 15% with respect to the ERM solution.
Table Results for in-distribution generalization. We report the average RMSE and MAPE of three
different seeds. Standard deviations reported in Appendix B.4 The best results per column
printed in bold and the second- bestt results are underlined (not applicable to the last row).
Airfoil
NO2
Exchange- Rate
Electricity
RMSE
MАРE
RMSE
MAPE
RMSE
MAPE
RMSE
MАPE
ERM
2.758
1.694
0.529
13,402
0.024
2.437
0.058
13.915
Mixup
3.264
1.964
0.522
13.226
0.025
2.513
0.058
13.839
ManiMixup
3.092
1.871
0.528
13.358
0.025
2.541
0.058
14.031
Local Mixup
3.373
2.043
0.524
13.309
0.021
2.136
0.063
14.238
C-Mixup
2.800
1.629
0.516
13.069
0.024
2.431
0.057
13.512
ADA
2.360
1.373
0.515
13.128
0.021
2.116
0.059
13.464
C-Mixup in
40
2.717
1.610
0.509
12.998
0.020
2.041
0.057
13.372
4.4
Out-of-distribution Robustness
In this section, evaluate the performance of ADA and compare it to prior approaches tasks involving experimental setup.
robustness. We use the same datasets as 49 and closely follow their
Data: We use four of the five out-of-distribution datasets used 49 First, we use RCFashion- MNIST (RCF-MNIST) 4. which is synthetic modification of Fashion-MNIST that models subpopulation shifts. Second, we investigate domain shifts using Communities and Crime (Crime) [12.. SkillCraft Master Table (SkillCraft) [2] and Drug-targe target Interactions (DTI) 17 all of which are tabular datasets. For Crime, we use state identification, in SkillCraft we use "League Index", which corresponds to different levels of competitors, and in DTI we use year, as domain information. We split the datasets into train-, validation- and test data based on the domain information resulting in Coomain-disting datasets. We provide detailed description of datasets in Appendix [B.4] Due to computational complexity, we could not establish a fair comparison on the satellite image regression dataset 2 (the fifth dataset in (49). so we report some exploratory results in Appendix B.4
Models and comparisons: As detailed Section 4.3
Additionally, we use a ResNet- 18 15 for
RCF-MNIST and DeepDTA
3X for DTI, as proposed in 4

---

Results: We report the RMSE and the "worst" domain RMSE. which corresponds to the worst within-domain RMSE for out-of-domain test sets in T Similar to !9), we report the value for the DTI dataset (higher values suggest better fit of the regression model). For full transparency, in the last row, we copy the results from 4 We can assess that ADA competitive with C-Mixup and the other data augmentation strategies. Under the same conditions (split of data and Neural network structure), ADA is superior to C-Mixup. But, the degree of improvement marginal on some datasets and as we show in the last row, could not fully replicate their results. ADA is significantly better than C-Mixup and other strategies on the SkillCraft data, in which ADA reduces the error by around 15% compared to the ERM solution.
Table 2: Results for diittiibtitiitto generalisation. We report the average RMSE across domains in the test data and the "worst -omain RMSE over three different sceds. For the DTI dataset, we report average R and "worst within-domain" R. Standard deviations are reported in Appendix B.4 The best results per column are printed in bold and the second-best results are underlined (not applicable to the last row).
RCF-MNIST
Crimes
SkillCraft
DTI
avg. RMSE
avg.
worst RMSE
avg. RMSE
worst RMSE
avg. R
worst
RMSE
R
ERM
0.164
0.136
0.170
6.147
7.906
0.483
0.439
Mixup
0.159
0.134
0.168
6.460
9.834
0.459
0.424
ManiMixup
0.157
0.128
0.155
5.908
9.264
0.474
0,431
LocalMixup
0.187
0.133
0.1590
7.251
10.996
0,470
0.433
C-Mixup
0.158
0.132
0.165
6.216
8.223
0.474
0.435
ADA
0.175
0.130
0.156
5,301
6.877
0,493
0.448
C-Mixup in
49
0.146
0.123
0.146
5.201
7.362
0.498
0.458
Conclusion
We introduced Anchor Data Augmentation (ADA), extension of Anchor Regression for the purpose of data augmentation. AR is a novel causal approach to increase the robustness in regression problems. In ADA, systematically mix multiple samples based on collective similarity criterion, which is determined via clustering. The augmented samples are modifications of the original samples that are moved towards away from the cluster centroids based the desired degree of robustness in AR. Our empirical evaluations across diverse synthetic and real-world regression problems consistently demonstrate the effectiveness of ADA, especially for limited data availability. ADA is competitive with or outperforms state-of-the-ae data augmentation strategies for regression problems, even though the improvements are marginal on some datasets.
ADA can be applied to any regression setting, and we have not found any case in which the results were detrimental. To apply ADA, only need to cluster data and select distribution for ry We relied on vanilla k-means, and the results are robust with respect to the number of clusters. Other clustering algorithms might be more suitable for different applications. For setting the parameter 7, we used a uniform distribution. We believe a gamma distribution could be equally effective.
Broader Impact
The purpose of data augmentation is compensate for data scarcity in multiple domains where gathering and labeling data accurately by experts is impractical, expensive, or time-consuming. If applied properly, can effectively expand the training dataset, reduce overfitting and improve the model's robustness, as was shown in the paper. However, It is important to note that the choice and combination of the data augmentation technique depends on the specific problem and using the wrong augmentation method may introduce additional bias to the model. More generally, incorrect data augmentation can lead to the following problems: overfitting the augmented data, loss of important information, introduction of unrealistic patterns and imbalanced presentation of the data, Detecting emerging problems due to data augmentation may not be straightforward. In particular, the performance on a test distribution that matches the training data distribution may be misleading and the model's predictions should be used with caution on new data that reflects the potential distribution shifts variations encountered in real-world.
10

---

References
[1] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04344 2017.
[2] Raphael Baena, Lucas Drumetz, and Vincent Gripon. Preventing manifold intrusion with locality: Local mixup, 2022.
[3] Xavier Bouthillier, Kishore Konda, Pascal Vincent, and Roland Memisevic. Dropout as data augmentation. arXiv preprini
2015.
[4] Peter Bühlmann. Invariance, causality and robustness. Statistical Science, 35(3):404-426, 2020.
[5] Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean- Jean-Philippe Vert. On mixup regularization. arXiv preprint arXiv:2006. 06049, 2020.
[6] Clément Chadebec and Stéphanie Allassonnière. Data augmentation with variational autoen- coders and manifold sampling. In Deep Generative Models, and Data Augmentation, Labelling. and Imperfections, pages 184-192. Springer, 2021.
[7] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pullern recognition workshops, pages 113-123, 2019.
[8] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pages 702-703, 2020.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages
leee, 2009.
[10] Terrance Vriei and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1XXx:1700800452 2017.
[11] Vanessa Didelez, Sha Meng, and Nuala A Sheehan. Assumptions of iv methods for observational epidemiology. Statistical Science, 25(1):22 40, 2010.
[12] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[13] Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 141191444,,, 2020.
[14] David Harrison and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management, 5(1):81-10) 8-0122 1978.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[16] Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pages 2731-2741. PMLR, 2019.
[17] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learn- ing datasets and tasks for drug discovery and development. arXiv preprint arXiv:2102.09548. 2021.
[18] Seong-Hyeon Hwang and Steven Euijong Whang. Regmix: Data mixing augmentation for regression. arXiv preprint arXiv:2106.03374 2021.
[19] R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics and Probability Letters, 33(3):291-297. 1997.

---

[20] lang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. arXiv preprint arXiv:2102.03055, 2021.
[21] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pages 17755-285.225 PMLR, 2020.
[22] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 14, 2013.
[23] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie. Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of mn-the-will distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[24] Charles Kooperberg. Statlib: an archive for statistical software, datasets, and information. The American Statistician, 51(1):98, 1997.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Alex Krizhevsky, lya Sutskever, Geoffrey Hinton, et al. Imagenet classification with deep convolutional neural networks. 2012.
[27] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41 international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018
[28] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[29] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):22788--222 1998.
[30] Sungbin Lim, Ildoo Kim. Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32, 2019.
[31] Chen Lin, Minghao Guo, Chuming Li, Xin Yuan, Wei Wu, Junjie Yan, Dahua Lin, and Wanli Ouyang. Online hyper-parameter learning for auto-augmentation strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 6579-6588, 2019.
[32] Tom Ching LingChen, Ava Khonsari, Amirreza Lashkari, Mina Raf Nazari, Jaspreet Singh Sam- bee, and Mario A Nascimento. Uniformaugment: A search-free probabilistic data augmentation approach. arXiv preprint arXiv:2003. 14348, 2020.
[33] Zicheng Liu, Siyuan Li, Di Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, and Stan Z Li. Automix: Unveiling the power of mixup for stronger classifiers. In European Conference on Computer Vision, pages 141-458 Springer, 2022.
[34] Zirui Liu, Haifeng Jin, Ting- Wang, Kaixiong Zhou, and Xia Hu. Divaug: Plug-in automated data augmentation with explicit diversity maximization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4762-4770, 2021.
[35] J MacQueen. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, pages 281-297, 1967.
[36] Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet sate-of-the-art data augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 774-782, 2021.
[37] David Ouyang, Bryan He, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul A Heidenreich, Robert A Harrington, David H Liang, Fuan Ashley, et Video-based ai for beat-to-beat assessment of cardiac function. Nature, 580(7802):252-2566 2020.
12

---

[38] Hakime Öztürk. Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep drug-targe target binding affinity prediction. Bioinformatics, 34(17):1221-122) 2018.
[39]
Jonas Peters, Dominik Janzing, and Bernhard Schoelkopf, editors. Elements of Causal Inference. MIT Press, Cambridge, MA, 2017.
[40] Jonas Peters, Nicolai Meinshausen, and Peter Bühlmann. Causal inference by using invariant prediction: Identification and confidence intervals. Journal of the Royal Statistical Society Series B, (8855:947-1012, 2016.
[41] Mateo Carull,, Bemhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1009-134,, 2018.
[42] Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, Jonas Peters, et al. Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B, 83(2):215-246, 2021.
[43] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 2014.
[44] Zhiqiang Tang, Yunhe Gao, Leonid Karlinsky, Prasanna Sattigeri, Rogerio Feris, and Dimitris Metaxas. Onlineaugment: Online data augmentation with less domain knowledge, In European Conference on Computer Vision, pages 313-329. Springer, 2020.
[45] Luke Taylor and Geoff Nitschke. Improving deep learning with generic data augmentation. In 2018 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1542-1547, IEEE, 2018.
[46] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR, 2019.
[47] Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. Ad- vances in neural information processing systems, 34:237-250, 2021.
[48] Xiaogang Xu, Hengshuang Zhao, and Philip Torr. Universal adaptive data augmentation. arXiv preprint arXiv:2207. 220.066888. 2022.
[49] Huaxiu Yao, Yiping Wang, Linjun Zhang, James Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. In Proceeding of the Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
[50] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[51] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
[52] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? arXiv preprint arXiv:2010.048/1 2020.
[53] Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment. arXiv preprint arXiv: 1912. 2019.
13


Reviews:
[{"review_id": "lqIdedeUvj", "reviewer": "Reviewer_2wsW", "summary": "This paper design a new mixup data augmentation algorithm for regression problems, which is a challenging field for data augmentations. Specifically, the authors extend Anchor Regression (AR) method as Anchor Data Augmentation (ADA), which utilizes several replicas of the modified samples in AR to generate more training samples. Since AR can provide the data in optimal least square errors once the values of the anchors are known, the proposed ADA generates robust augmented data in the homogeneous groups (achieved by clustering). With comprehensive comparison experiments on linear and non-linear regression problems, the proposed ADA achieves competitive performances with previous mixup approaches, especially C-Mixup (a special-designed mixup method for regression).", "questions": "Refer to weaknesses, and I have no more questions.", "limitations": "I have found no more limitations besides my weaknesses and my concerns.\n\n### Post-rebuttal\nThanks for the detailed replies, which addressed my concerns, and I raised the rating to 5. I hope the authors add more experiments and discussions with existing works. Maybe the automatic version of the proposed ADA with fewer hyper-parameters can be the improvement direction.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* **S1**: This paper tackles the data augmentation problem in regression tasks, which is not well-studied in the data augmentation community. Despite the proposed ADA using an existing AR method (refer to W1), this paper still has some valuable contributions and is interesting.\n\n* **S2**: The designed methods are well-motivated and well-written, which is easy to follow. Meanwhile, the experimental settings and hyper-parameters of ADA are provided in the main text or the appendix, which ensures practical usage and reproducibility.", "weaknesses": "* **W1**: In comparison to C-Mixup and AR, the novelty of the proposed ADA is relatively weak. The idea of ADA is straightforward, which generates reliable augmented samples in the local scope of the clustered anchors. From my perspective, ADA seems like an improved version of C-Mixup with the existing AR method, which is less novel than these two works.\n\n* **W2**: The conducted experiments mainly focus on 1D data (e.g., linear synthetic data, time series prediction) in tabular format. More data modalities are essential to verify the data augmentation for a generalization purpose. For example, C-Mixup provides angle regression tasks with real-world images.\n\n* **W3**: The compared mixup baselines are not representative enough. For example, C-Mixup compares the learnable mixup methods (e.g., PuzzleMix [1] and AutoMix [2]) in addition to the vanilla mixup and Manifold Mixup. Meanwhile, more lecture review of published mixup augmentation methods is required in Sec. 2.\n\nConsidering the pros and cons, I decided to borderline reject this manuscript and look forward to the authors’ feedback during the rebuttal period.\n\n#### Reference\n[1] Jang-Hyun Kim, et al. Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup. ICML, 2020.\n\n[2] Zicheng Liu, et al. AutoMix: Unveiling the Power of Mixup for Stronger Classifiers. ECCV, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your balanced review and your suggestions for improvement. \n\nThe literature on Mixup methods is vast and especially interesting for image classification. There are only two valid approaches for regression (reviewed in our paper). C-Mixup is a solid paper, and it brought some of the Mixup gains to regression problems. Anchor regression (AR) is a method to robustify regression methods in a narrow set of cases. It builds on the causality literature and is not a method for data augmentation per se. ADA (our proposal) has several relevant contributions:\n1 We reinterpret AR as a data augmentation. AR and MixUp come from different sets of assumptions and are almost different research fields in ML altogether. Noticing that AR applies to data augmentation is not straightforward and, in our opinion, a significant contribution. \n\n2 Our work empirically shows that using $\\gamma<1$ helps with generalization too. In MixUp, the augmentation interpolates samples between the original samples. ADA does that for $\\gamma > 1$ and pushes them further apart for $\\gamma < 1$. It is an added benefit when using the AR framework for data augmentation in regression. \n\nWe repeat all the experiments from Sections 5.1 and 5.3 from the C-mixup paper. There are tabular, time series, and image data in that set of experiments. We report the results for the missing datasets in our original submission in the added Pdf file (Table 2). The experiments in Section 5.2 are not reproducible with their publicly available code (comparisons were not possible). Except for the toy cosine data, all our experiments are multi-dimensional.\n\nPuzzleMix and AutoMix are solid contributions to the MixUp literature for image classification. We will add them to our literature review. Both methods are not directly applicable to regression problems. In any case, we have also tried to run the code, it was not straightforward, and we cannot present preliminary results here. In the attached PDF, we have also added Local MixUp for comparison (Tables 1 and 2)."}, {"review_id": "rWWNcZQQAt", "reviewer": "Reviewer_zYBi", "summary": "The authors propose a new approach for automatic data augmentation in nonlinear regression, which requires no specific domain knowledge and the added computation cost is low. The idea is borrowed from Anchor regression and Mixups. The idea is to first run a k-means on the data, use the cluster memberships to construct anchor variables and then perform a non-linear anchor regression. They illustrate the performance of their methods on linear and non-linear regression problems.", "questions": "1. Why do the transformation on the variables but not directly modify the loss function as a nonlinear version of (4)?\n\n2. Any rules on selecting the number of clusters or parameter k? Are the results sensitive to different options?", "limitations": "The authors have discussed some limitations of their work in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors did a comprehensive literature review, and it borrows the strength of mixup augmentation and anchor regression. Intuitively, by performing a k-means, it allows the method to borrow strength across samples that are \"similar\", and the anchor regression further improve the robustness. The experiments also show that the method is especially useful when the number of observations is small.", "weaknesses": "1. It is not very intuitive why Anchor regression will help generalization from Section 2.2. I understand the first paragraph, but it is not straightforward from the equation in (3). Why do partial pooling and IV help?\n\n2. There is no theoretical analysis of the performance.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and questions about our paper.\n\nADA relies on AR to augment data samples, and it inherits the generalization properties of AR. The main objective of AR is prediction robustness concerning some directions of perturbations and interventions. Therefore, it is natural to expect augmentation with ADA to improve the out-of-distributions accuracy for some distribution shifts at the cost of reducing the in-distribution generalization. We understand AR helps with generalization by looking at Equations 5 and 6 instead of 3 because these equations show the transformation of the data before applying OLS. The original data X and Y are projected on the anchor variables. If the Anchor variables are binary, representing a clustering assignment, $\\Pi_A*X$ is the cluster centroid. The modified samples pull the original data X (and Y) toward the cluster center if $\\gamma>1$, and repel them if $\\gamma<1$. AR only uses one value for $\\gamma$, and we use a multitude of gamma in ADA to be robust in different ways. In that way, ADA does something similar to C-MixUp, but with more diversity while mixing the samples, as in each mini-batch, the number of samples from the same cluster changes.\n\nThe AR paper has a theoretical analysis in which they prove the benefit of using AR to solve regression problems if there is a shift in the distributions of the anchor variables. But we did not carry out an additional analysis of why ADA provides robustness.\n\nWe could have modified the loss functions instead of the samples. But we thought it was more natural to change the samples. It is what AR is already doing, i.e., the AR says that solving Eqn. 4 is equivalent to solving OLS with the transformed data (Eqns. 5 and 6).\n\nWe used a held-out dataset to set the number of clusters, as we have not developed a good rule for setting them. In the attached Pdf, we added a figure to show how the number of clusters affects the solution, and the results only change slightly."}, {"review_id": "Mo3Kzm4Ofm", "reviewer": "Reviewer_K137", "summary": "This paper introduces a new data augmentation method, designed for regression problems. The method is based on anchor regression, where new samples are generated via their projection to the normal subspace spanned by the anchors. The method is evaluated on several datasets and compared with ERM and C-Mixup, showing on par performance.\n", "questions": "See above", "limitations": "See above", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors address the problem of data augmentation for regression which is an emerging topic in ML, and thus the significance is high. The authors motivate their approach using anchor regression, although I doubt the proposed method and AR are at all related. Thus, originality is questionable, however, I might have missed something crucial.\n", "weaknesses": "In general, the paper seems to be technically sound. However, given Alg. 1 and lines 6-7 in it, it is not clear why Eqs. 3 and 4 are even mentioned in the paper, and what is their relation to the method in practice. Essentially, the authors project the data to the normal subspace spanned by $A$. They motivate it via anchor regression (AR), however, it is clear that results totally depend on the choice of $A$, which is somewhat independent of AR to begin with. Also, the authors use indices $i,j$ around equations 7 and 8, but do not specify the indices roles. Thus, one of the main shortcomings of this work is the justification of the approach and its relation to anchor regression is unclear and should be improved. \n\nAnother shortcoming is the evaluation section. In particular, it is unclear to me how you select the ADA model among the various ones you trained. In addition, it seems like grid search was more extensive in ADA in comparison to other baselines. Also, why noise (vanilla DA) is added only to output? I would like to see the results of adding noise to input and input&output. There are no standard deviation results. The results of ADA are often inferior to the baselines. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your direct review and comments about the clarity of the connection between AR and ADA and the derivation of augmentation equations 7 and 8. We will rewrite these sections to improve the fluency and notations. In particular, ADA uses a linear projection as in AR to select perturbation directions considered more relevant based on the similarity of samples in the batch. The augmentation equations (7-8) are row-wise scaled versions of the modified X, Y in AR (5-6), where the scale preserves the (non)linear relation between X and Y after augmentation. The subindices $i$ and $j$ denote the row and column indices of the corresponding matrices as detailed on line 154 of the paper.\n\nWe understand from your review that the connection between AR and ADA is murky. The variables in A are the main point of AR, as they are the ones that allow AR to robustify the regression solution, and they make the connection between AR and ADA. The variables in A are typically not available. We decided to use clustering to infer A, inspired by the solution in the first example in the AR paper. It is a design choice. We use locality consistency to create our A for data augmentation (C-MixUp does something similar). Other choices might be possible, but we believe clustering is a solid universal choice. We will improve this description in the paper. \n\nWe did the same hyper-parameter search for all the models and did not go the extra mile to make ours better. The hyperparameters used for the benchmarks were the proposed ones in the C-MixUp paper, where they employ cross-validation with grid-search. Furthermore, we report the results of all the models trained with the mentioned hyperparameters for comparison. \n\nWe also report the results from the original C-MixUp paper because we could not replicate their results. In other comparison methods, we got better results than C-MixUp reported in [1].  It is important to notice those differences are small and can be due to factors like initialization, random splitting of the data, or the selected optimization procedure. It is not worrisome, but it is notable.\n\nWe also added noise to the X variable in vanilla DA, and the results were significantly worse. We decided not to add it to the attached PDF because it did not add anything. We can send it to the Area Chair if you want to see the results. \n\nThe standard deviations of RMSE and MAPE results are in Tables 1 and 2 in Appendix B.3.\n\nYou mentioned that ADA is sometimes not better than the baseline in your last comment. Data augmentation for regression is hard. There are very few papers, and the gains are typically small. We (as a community) have not been able to find a data augmentation for regression that is as good as for classification. When the gains are small, we should not expect them to be always better than the baseline. But in some cases, we are significantly better, and ADA never hurts the results. \n\n[1] Yao, H., Wang, Y., Zhang, L., Zou, J. Y., & Finn, C. (2022). C-mixup: Improving generalization in regression. Advances in Neural Information Processing Systems, 35, 3361-3376."}, {"review_id": "Rrl1fa3HGC", "reviewer": "Reviewer_SE5E", "summary": "In this paper, the authors proposed anchor data augmentation, which borrows from the recently proposed Anchor regression method. The anchor data augmentation uses several replicas of the samples, generated according to the anchor matrix. The proposed augmentation is empirically evaluated both for linear and non-linear models.", "questions": "When using k-means clustering to generate the anchor matrix, can it be intuitively understood as the augmentation is trying to construct local interpolations to create augmented samples (where the locality is defined by the clusters)? If so, does this method generalize to high-dimensional data well?", "limitations": "Yes. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple and easy to implement. The anchor matrix can be generated by k-means clustering without prior knowledge of the data distribution.\n\n2. The illustrative examples of augmenting the data generated from a cosine model show that the proposed method can generate augmented samples that better represent the underlying data distribution.\n\n3. The proposed method is empirically evaluated on both linear and non-linear models. The results on the linear synthetic data with MLP show that the proposed augmentation can significantly improve the generalization performance.", "weaknesses": "1. The standard deviation of empirical evaluations is not reported for sections 4.2 and 4.3. It is hard to verify the statistical significance of the results.\n\n2. It seems the choice of $\\gamma$ and design of the anchor matrix $A$ largely remain as hyper-parameters and requires tuning. It would be interesting to see some theoretical justification (or intuition) for choosing $\\gamma$ and designing $A$.\n\n3. The improvement over vanilla augmentation is somewhat marginal. It seems that for linear synthetic data with the linear model, California Housing dataset, it is not clear whether the proposed method is significantly better than vanilla augmentation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and discussion about scalability and intuition on hyperparameter selection. \n\nThe standard deviations of RMSE and MAPE results are in Tables 1 and 2 in Appendix B.3.\n\nThe performance of ADA is indeed dependent on the choice of the anchor matrix A. As with AR, we can choose the anchor matrix based on expert opinion about “plausible” or “importance” shifts (invariances) in the distribution. If not available, we can infer that information from the data. One way to select the anchor matrix is to use the one-hot encoding of cluster membership. This method partitions the data into similar clusters to account for locality and contracts or expands the samples around the cluster centroids. We provide example models trained on the nonlinear (cosine) data when A is defined by varying cluster sizes (Appendix A.2 Figure 4). For larger clusters, more samples are “mixed” to produce one augmented instance, which may result in over-smoothing of the underlying X-Y relation. On the other hand, small clusters lead to less “mixing” and in the extreme case where each cluster has only one member the augmented dataset would be the same as the original. In the general form, $\\Pi_A$ provides a “collective” mixing for the samples in a mini-batch to determine a center, and $\\gamma$ controls the amount of contraction or expansion around the center.\n\nTherefore, when data has (approximately) linear structure, with sufficiently large clusters the one-hot-based centroid is close to the underlying hyperplane, and the cluster contraction (expansion) would preserve the underlying hyperplane. On the other hand, augmenting data with uniformly distributed perturbation directions would still preserve the underlying hyperplane. We believe this is the case with the California Housing dataset. However, for a nonlinear data structure, augmenting data samples in arbitrary directions with the same perturbation magnitude is bound to fail, and more complicated schemes are necessary. \n\nData augmentation for regression is hard. There are very few papers, and the gains are typically small. We (as a community) have not been able to find a data augmentation for regression that is as good as for classification. When the gains are small, we should not expect them to be always better than the baseline. But in some cases, we are significantly better, and ADA never hurts the results.\n\nYes, ADA provides a systematic method to “mix” multiple samples based on collective similarity criteria,  which is measured by the anchor matrix A (which we compute in ADA via a clustering approach). When using k-means clustering to generate A, ADA “mixes” the samples which fall into the same clusters. The cluster centroid is computed via the $\\Pi_A$ operator, and augmented samples are moved towards or away from the centroid based on the value of $\\gamma$ (see Appendix A.2 “Anchor Matrices and Locality” for details).\n\nWe have not found limitations when using our procedure in high dimensional data, and the linear example at the beginning of the experimental section can illustrate what happens as we decrease the number of data points. The error is too high, and no method helps when there is little data. We obtain the same solution with all the algorithms when there is a lot of data. ADA helps in the middle range on the number of data points when there is not too much or too little data. We will never be in the too-much-data regime in high dimensions and should expect to be in the middle range. But we can also be in the too-little-data regime. In that case, it will be hard to find a good regressor unless additional information about the invariances is available. "}, {"review_id": "ipHjbp11gw", "reviewer": "Reviewer_KQFt", "summary": "The paper proposes a new data augmentation method called Anchor Data Augmentation (ADA) that can improve the performance of machine learning classifiers, especially in over-parameterized settings.\n\nNovelty:\n\nThe work presents the Anchor Data Augmentation (ADA) which is based on the concept of Anchor Regression. This is a unique approach, but its departure from existing methods is somewhat moderate. The main novelty lies in its application to machine learning classifiers using the concept of linear regression model fitting. However, one could argue that using regression for data augmentation is not entirely groundbreaking.\n\nSignificance:\n\nThe significance of the paper is somewhat moderate. The idea of using Anchor Regression to improve machine learning classifiers is promising, especially when it comes to over-parameterized settings. However, the paper seems to be more of a proof-of-concept, as it primarily focuses on housing datasets, leaving other domains unexplored. The lack of significant improvement over other existing methods also somewhat diminishes its overall impact.\n\nSoundness:\n\nThe methodology presented is fairly sound, with ADA being applied to various types of regression models. However, there are concerns related to the many hyperparameters introduced and the lack of study on their impact, which could be a potential flaw in the work. Additionally, The paper performs a replication-like operation on the data, so isn't there more homogeneous data? The paper does not seem to address potential issues of overfitting in the presence of homogenous data, which is a common concern in machine learning.\n\nClarity:\n\nThe paper is written in a comprehensible manner. It provides a clear explanation of the proposed ADA method and its underlying concept of Anchor Regression. The experiments are also described clearly, though there is room for improvement in detailing the parameter settings and choices made during experimentation. It might benefit from restructuring certain sections to make the presentation even clearer, especially when discussing the results.\n\nLiterature:\n\nThe paper discusses some existing works, mainly focusing on regression models. However, it would have been beneficial if it drew more comparisons with other data augmentation methods and their results. It feels like there could be a more comprehensive discussion on how ADA fits in the broader landscape of data augmentation techniques, especially in terms of strengths and weaknesses.\n\nConclusion:\n\nAfter taking the Area Chair's feedback into consideration, the paper's novelty is evident, but not groundbreaking. Its significance is moderate, given the specific domain application and the moderate improvement over existing methods. Soundness has potential issues due to hyperparameters and the risk of overfitting. The clarity is fairly good but could be improved upon, and there is room for a more in-depth literature review. Based on these updates, my recommendation change to 'Borderline accept' at this time.", "questions": "How does ADA compare to other data augmentation methods in terms of computational efficiency?\n\nHave you considered applying ADA to other types of classifiers beyond regression models, such as decision trees or support vector machines?\n\nHave you evaluate the robustness of ADA to different types of data distributions and noise levels？\n\nCan it be used for augmentation performance testing on other visual datasets or point cloud datasets?\n\nWhy are the hyperparameters set to a uniform distribution in the experiments, and although it's claimed that this is also effective for a gamma distribution, why are no experimental results provided? Is this also effective for other distributions such as chi-square, Rayleigh, sub-Gaussian, and Super-Gaussian ?", "limitations": "The paper only explores the application of ADA to housing datasets and does not investigate its use in other domains such as healthcare or finance. \n\nCompared to other methods, the results of this paper's approach did not show significant improvements.\n\nThe proposed method might struggle to address the issue of overfitting that tends to occur in scenarios with an abundance of homogenous data.\n\nThis is an extremely heuristic paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "ADA is based on the concept of Anchor Regression, which involves fitting a linear regression model to a subset of the training data and using the model to predict the target variable for the remaining data. \n\nADA can be applied to various types of regression models, including neural networks, and can be used to improve predictions in the low data regime. \n\nADA can be applied to real-world datasets, such as the California and Boston Housing datasets, and can provide improved performance as the number of samples is increased.", "weaknesses": "The paper includes many hyperparameters, but does not study the impact of different hyperparameters on the performance of ADA or develop methods for automatically tuning them.\n\nWhile proposing the ADA method, the paper does not further consider new data augmentation methods that combine ADA with other techniques such as generative adversarial networks or transfer learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and discussion about scalability and intuition on hyperparameter selection. \n\nThe standard deviations of RMSE and MAPE results are in Tables 1 and 2 in Appendix B.3.\n\nThe performance of ADA is indeed dependent on the choice of the anchor matrix A. As with AR, we can choose the anchor matrix based on expert opinion about “plausible” or “importance” shifts (invariances) in the distribution. If not available, we can infer that information from the data. One way to select the anchor matrix is to use the one-hot encoding of cluster membership. This method partitions the data into similar clusters to account for locality and contracts or expands the samples around the cluster centroids. We provide example models trained on the nonlinear (cosine) data when A is defined by varying cluster sizes (Appendix A.2 Figure 4). For larger clusters, more samples are “mixed” to produce one augmented instance, which may result in over-smoothing of the underlying X-Y relation. On the other hand, small clusters lead to less “mixing” and in the extreme case where each cluster has only one member the augmented dataset would be the same as the original. In the general form, $\\Pi_A$ provides a “collective” mixing for the samples in a mini-batch to determine a center, and $\\gamma$ controls the amount of contraction or expansion around the center.\n\nTherefore, when data has (approximately) linear structure, with sufficiently large clusters the one-hot-based centroid is close to the underlying hyperplane, and the cluster contraction (expansion) would preserve the underlying hyperplane. On the other hand, augmenting data with uniformly distributed perturbation directions would still preserve the underlying hyperplane. We believe this is the case with the California Housing dataset. However, for a nonlinear data structure, augmenting data samples in arbitrary directions with the same perturbation magnitude is bound to fail, and more complicated schemes are necessary. \n\nData augmentation for regression is hard. There are very few papers, and the gains are typically small. We (as a community) have not been able to find a data augmentation for regression that is as good as for classification. When the gains are small, we should not expect them to be always better than the baseline. But in some cases, we are significantly better, and ADA never hurts the results.\n\nYes, ADA provides a systematic method to “mix” multiple samples based on collective similarity criteria,  which is measured by the anchor matrix A (which we compute in ADA via a clustering approach). When using k-means clustering to generate A, ADA “mixes” the samples which fall into the same clusters. The cluster centroid is computed via the $\\Pi_A$ operator, and augmented samples are moved towards or away from the centroid based on the value of $\\gamma$ (see Appendix A.2 “Anchor Matrices and Locality” for details).\n\nWe have not found limitations when using our procedure in high dimensional data, and the linear example at the beginning of the experimental section can illustrate what happens as we decrease the number of data points. The error is too high, and no method helps when there is little data. We obtain the same solution with all the algorithms when there is a lot of data. ADA helps in the middle range on the number of data points when there is not too much or too little data. We will never be in the too-much-data regime in high dimensions and should expect to be in the middle range. But we can also be in the too-little-data regime. In that case, it will be hard to find a good regressor unless additional information about the invariances is available. "}, {"review_id": "QZH0FFoZ6t", "reviewer": "Reviewer_dYVk", "summary": "The authors proposed a new data augmentation algorithm for regression datasets. They borrow ideas from Anchor regression model which captures the heterogeneity of the dataset using anchor variables to create additional augmented data. They show that applying this type of augmentation improves results compared to naive mixup augmentation on many datasets.", "questions": "Will your model generalize to datasets with only categorical variables?\nDo you have ablations on different values of \"k\" and \"alpha\"? ", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-written paper.\n- The augmentation proposed is theoretically motivated. \n- Evaluated on many datasets and scenarios (ID, OOD datasets etc).\n\n", "weaknesses": "- The results are not significantly better than the baselines in low and high data regimes.\n- The theoretical motivation from Anchor Regression is not clearly explained or justified.\n- The comparison is limited only to C-Mixup as a prior method, could be broader.\n- More analysis could be provided on why and how ADA improves robustness.\n- (As far as I understand) ADA has more hyperparameters to tune compared to some other simpler data augmentation techniques.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your direct review and questions about our procedure. We will address them in the following paragraphs.\n\nData augmentation for regression is hard. As we mentioned in our paper, there are two papers on the topic. The results sometimes are much better than the baseline, but in most cases, they provide modest improvements. Given the effectiveness of data augmentation in classification, we should explore the potential solutions for regression.\n\nIn the final version of the paper, we will expand on how a method for robust regression method that builds on the causality literature is reinterpreted as a data augmentation strategy. The overall goal is to get better regressors. ADA relies on AR to augment data samples. As in AR, it uses a linear projection to select perturbation directions that are considered more relevant based on the similarity in the batches. Hence, it inherits the generalization properties and prediction robustness (concerning those directions) from AR. Combining the original data with perturbed augmentations in the prescribed directions leads to improved predictions on the mixture of distributions, and they improve generalization.\n\nIn Appendix Section A.2, we provide further explanations and figures to illustrate how ADA augments the samples and how each parameter influences the augmented samples. By constructing the Anchor matrix A from a one-hot encoding, ADA “mixes” the samples which fall into the same clusters, and the augmented samples are either moved towards or away from the centroid based on the value of $\\gamma$. We control the range of values for $\\gamma$ via the hyperparameter $\\alpha$.\n\nWe made a comparison to the methods proposed in the C-MixUp paper. We also compare ADA with the original MixUp and Mani-MixUp. Additionally, we have added the comparison to the Local MixUp in the attached PDF. Also, the other methods are for classification problems, and their application to regression is not straightforward. \n\nOur proposed method has three hyperparameters: the number of clusters $k$, $\\alpha$, and whether to use manifold augmentation. For reference, C-Mixup also has three hyperparameters: bandwidth, $\\alpha$ in Beta distribution, and whether to use manifold mixing.\n\nCategorical variables are treated as reals, which makes them meaningless. But small values of gamma might still provide robustness to the solutions. We have not tried to use categorical variables only and wouldn’t expect improvements.\n\nWe show how the results vary with $\\alpha$ and $k$ for three representative datasets (see the attached PDF). The value of $k$ does not affect the solution significantly, and large $\\alpha$ penalized the results. \n"}]

Author rebuttal:
We thank all reviewers for their constructive feedback and suggestions for improving our contribution to NeurIPS, and we hope to have a fruitful discussion in the coming days. We also want to thank the Area Chair for leading this paper review and the discussion. We address each question in our direct responses to the respective review. This global response contains a PDF attachment, in which we present some additional experimental results, which we reference to throughout our responses. First, we show how the ADA results change with different values for $\alpha$ and $k$. Second, we compared the performance of Local Mixup [1] to our baselines and ADA. Last, we added results for two additional datasets to complete the comparison with all the data used by C-MixUp. We are looking forward to the coming discussion with all the reviewers.

[1] Baena, R., Drumetz, L., & Gripon, V. (2022). Preventing manifold intrusion with locality: Local mixup. arXiv preprint arXiv:2201.04368.

Meta-review: I have read the reviews and discussions. I currently tends to agree with the majority of the reviewers that the contribution is valuable since designing good data augmentations for regression is still an open and promising topic.

There were some points raised by reviewer K137 about the impact of the matrix A on the results and on the connection between the paper and AR. From the authors response, I infer that the proposed approach for choosing A by clustering seems to work well in practice and the authors further clarify the connection between AR and ADA. This should be made clearer in the paper. 

The experiments appear to be in favor of the method, although the improvements is sometimes marginal compared to other data augmentation methods. Hence, more the conclusions on the method should be more nuanced.  

In addition, there were several clarifications that were made during the discussion phase and that must be included in the final version of the paper.
Decision: Accept (poster)