SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With the one-hot partitioning matrix,  $\mathbf{A}$  (introduced in the previous section), the approximation of the true nonlinear model becomes accurate in partitions with small diameter (where we define partition diameter as the maximum distance of two samples  $\mathbf{X}^{(i)}$  and  $\mathbf{X}^{(j)}$  in the same cluster).
3.3 Algorithm
Finally, in this section, we present the ADA algorithm step by step (Algorithm 1) to generate minibatches of data that can be used to train neural networks (or any other nonlinear regressor) by any stochastic gradient descent method. As discussed previously, we propose to repeat the augmentation with different parameter combinations for each minibatch.
Given a centered training dataset  $(\mathbf{X},\mathbf{Y})$ , its clustering assignment  $\mathbf{A}$ , and prior function  $p(\gamma)$ , the ADA minibatch algorithms take  $L$  random samples from the training set and its corresponding rows in  $\mathbf{A}$  and outputs an  $L$ -sample mini-bath  $(\tilde{\mathbf{X}}_{\gamma,\mathbf{A}},\tilde{\mathbf{Y}}_{\gamma,\mathbf{A}})$ .
In order to do so, we first choose  $\gamma$  according to the provided criterion  $p(\gamma)$  (line 3). The corresponding projection matrix  $\Pi_A$  is computed from A (line 4). Finally, in lines five to seven, the transformation is applied according to Equations and
Algorithm 1 ADA: Minibatch generation
1: Input:  $L$  training data points  $(\mathbf{X},\mathbf{Y})$ ; prior distribution for  $\gamma$ :  $p(\gamma)$
$L \times q$  binary matrix  $\mathbf{A}$  with a one per row indicating the clustering assignment for each sample.
2: Output:  $(\tilde{\mathbf{X}},\tilde{\mathbf{Y}})$
3: Sample  $\gamma$  from  $p_{(\gamma)}$
4: Projection matrix:  $\mathbf{\Pi}_{\mathbf{A}}\gets \mathbf{A}(\mathbf{A}^{T}\mathbf{A})^{\dagger}\mathbf{A}^{T}$
5: for  $i = 0$  to row of  $\mathbf{X}$  do
6:  $\tilde{\mathbf{X}}_{\gamma ,\mathbf{A}}^{(i)}\gets \frac{\mathbf{X}^{(i)} + (\sqrt{\gamma} - 1)(\boldsymbol{\Pi}_{\mathbf{A}})^{(i)}\mathbf{X}}{1 + (\sqrt{\gamma} - 1)\sum_{j}(\boldsymbol{\Pi}_{\mathbf{A}})^{(ij)}}$
7:  $\tilde{\mathbf{Y}}_{\gamma ,\mathbf{A}}^{(i)}\gets \frac{\mathbf{Y}^{(i)} + (\sqrt{\gamma} - 1)(\boldsymbol{\Pi}_{\mathbf{A}})^{(i)}\mathbf{Y}}{1 + (\sqrt{\gamma} - 1)\sum_{j}(\boldsymbol{\Pi}_{\mathbf{A}})^{(ij)}}$
8: end for
9: return  $(\tilde{\mathbf{X}}_{\gamma ,\mathbf{A}},\hat{\mathbf{Y}}_{\gamma ,\mathbf{A}})$
We experimentally investigate and compare the performance of ADA. First, we use ADA in an in-distribution setting for a linear regression problem (Section 4.1), in which we show that even in this case, ADA provides improved performance in the low data regime. Second, in Section 4.2 we apply ADA and C-Mixup to the California and Boston Housing datasets as we increase the number of training samples. In the last two subsections, we replicate the in-distribution generalization (Section 4.3) and the out-of-distribution Robustness (Section 4.4) from the C-Mixup paper [49]. In [49] the authors further assess a task generalization experiment. However, the corresponding code was not publicly provided, and a comparison could not be easily made.
4.1 Linear synthetic data
Using synthetic linear data, we investigate if ADA can improve model performance in an overparameterized setting compared to C-Mixup, vanilla augmentation, or classical expected risk minimization (ERM). Additionally, we analyze the sensitivity of our approach to the choice of  $\gamma$  and the number of augmentations.
Data: The generated data follows a standard linear structure
$$
\mathbf {Y} ^ {(i)} = \left(\mathbf {X} ^ {(i)}\right) ^ {T} b + b _ {0} + \epsilon^ {(i)} \tag {11}
$$
with  $\mathbf{X}^{(i)}, b \in \mathbb{R}^{19}$  and  $\mathbf{Y}^{(i)}, b_0, \epsilon^{(i)} \in \mathbb{R}$ . The parameters are sampled randomly from a Gaussian distribution  $N(0,1)$ . We sample 20 different training datasets and one validation set with  $\epsilon \sim \mathcal{N}(0,0.1^2)$ ,  $\mathbf{X}^{(i)} \sim \mathcal{N}(\mathbf{0},\mathbf{I}_{19})$ . For each training set, we take subsets with an increasing number of samples to evaluate the methods on different levels of data availability. The subsets are hierarchically constructed (i.e., meaning a smaller set is always a subset of a larger one). The validation set has 100,000 samples.
Models and Comparisons: We investigate and compare the impact of ADA using two different models with varying complexity: a linear Ridge regression and a multilayer perceptron (MLP) with one hidden layer and 10 units with ReLU activation. Using an MLP with more hidden layers shows similar results (see Appendix B.1 for details).
The ERM models only use the original data. We perform vanilla data augmentation by adding Gaussian noise  $\epsilon' \sim N(0, 0.1^2)$  to the output leaving the input unchanged. Next, we apply C-Mixup with a bandwidth of 1 and set the  $\alpha$  of the Beta-distribution to 2. Finally, we apply ADA with varying the number of obtained augmentations  $k = \{10, 100\}$  and varying range of values for  $\gamma$ . To be precise, we define  $\alpha \in \{2, 4, 6, 8, 10\}$  and specify  $\beta_i = 1 + \frac{\alpha - 1}{k / 2} \cdot i$  (with  $i \in \{1, \dots, k / 2\}$ ) and  $\gamma \in \{\frac{1}{\alpha}, \frac{1}{\beta_{k / 2 - 1}}, \dots, \frac{1}{\beta_1}, 1, \beta_1, \dots, \beta_{k / 2 - 1}, \alpha\}$ . A is constructed using k-means clustering with  $q = 8$ .
For the Ridge regression model, we increase the dataset by a factor of 10 by sampling from the respective augmentation methods and subsequently compute the regression estimators. In contrast, for the MLP, we implement the augmentation methods on a minibatch level. Specifically, we incorporate vanilla augmentation by adding Gaussian noise to each batch, apply C-Mixup after sampling from the beta distribution in each batch, and finally, apply ADA after sampling from the defined gamma values in each batch.
Results: We plot our results in Figure 2. First, as expected, Ridge regression outperforms the MLP model. Second, when there is little data availability, using ADA decreases the test error compared to ERM. The effect diminishes when the training dataset is sufficiently large, and all models converge to the noise limit of  $0.1^2$ . Third, vanilla augmentation achieves similar results as ADA and C-Mixup for Ridge regression but not quite for the MLP. This suggests that ADA (and C-Mixup) are more meaningful than randomly adding noise and especially well suited for highly parameterized models as the MLP has almost 20 times more parameters than Ridge regression. In real-world applications, the value of  $\epsilon$  is usually unknown, and choosing  $\epsilon'$  for vanilla augmentation is not trivial, especially when the number of samples is small. Fourth, we conclude that generating more augmentations (100 instead of 10) further improves prediction error in vanilla and anchor augmentation (Appendix B Figure 9) and the effectiveness of anchor augmentation is further increased as the range for  $\gamma$  is wider (Appendix B Figure 10). Finally, C-Mixup and ADA perform similarly with ADA having a tendency to achieve a lower test error.
In summary, even in the simplest of cases, in which we should not expect gains from ADA (or C-Mixup), these data augmentation strategies provide gains in performance when the number of training examples is not sufficient to achieve the error floor.
4.2 Housing nonlinear regression
We extend the results from the previous section to the California and Boston housing data and compare ADA to C-Mixup [49]. We repeat the same experiments on three different regression datasets. Results are provided in Appendix B.2 and also show the superiority of ADA over C-Mixup for data augmentation in the implemented experimental setup.
Figure 2: Mean Squared Error for Ridge Regression model and MLP model with varying number of training samples. For Ridge regression, vanilla augmentation and C-Mixup generate  $k = 10$  augmented observations per observations. Similarly, Anchor Augmentation generates  $k = 10$  augmented observations per observation with parameter  $\alpha = 10$ .
Data: We use the California housing dataset [19] and the Boston housing dataset [14]. The training dataset contains up to  $n = 406$  samples, and the remaining samples are for validation. We report the results as a function of the number of training points.
Models and comparisons: We fit a ridge regression model (baseline) and train a MLP with one hidden layer with a varying number of hidden units with sigmoid activation. The baseline models only use only the original data. We train the same models using C-Mixup with a Gaussian kernel and bandwidth of 1.75. We compare the previous approaches to models fitted on ADA augmented data. We generate 20 different augmentations per original observation using different values for  $\gamma$  controlled via  $\alpha = 4$  similar to what was described in Section 4.1. The Anchor matrix is constructed using k-means clustering with  $q = 10$ .
Results: We report the results in Figure 3. First, we observe that the MLPs outperform Ridge regression suggesting a nonlinear data structure. Second, when the number of training samples is low, applying ADA improves the performance of all models compared to C-Mixup and the baseline. The performance gap decreases as the number of samples increases. When comparing C-Mixup and ADA, we see that using sufficiently many samples both methods achieve similar performance. While on the Boston data, the performance gap between the baseline and ADA persists, on California housing, the non-augmented model fit performs better than the augmented one when data availability increases. This suggests that there is a sweet spot where the addition of original data samples is required for better generalization, and augmented samples cannot contribute any further.
Figure 3: MSE for housing datasets averaged over 10 different train-validation-test splits. On California housing Ridge regression performs much worse which is why it is not considered further (see Appendix B.2).
4.3 In-distribution Generalization
In this section, we evaluate the performance of ADA and compare it to prior approaches on tasks involving in-distribution generalization. We use the same datasets as [49] and closely follow their experimental setup.
Data: We use four of the five in-distribution datasets used in [49]. The validation and test data are expected to follow the same distribution as the training data. Airfoil Self-Noise (Airfoil) and NO2 [24] are both tabular datasets, whereas Exchange-Rate and Electricity [27] are time series datasets. We divide the datasets into train-, validation- and test data randomly, as the authors of C-Mixup did. For Echocardiogram videos [37] (the 5th dataset in [49]), we could not replicate their preprocessing.
Models and comparisons: We compare our approach, ADA, to C-Mixup [49], Local-Mixup [2], Manifold-Mixup [46], Mixup [51] and classical expected risk minimization (ERM). Following the work of [49], we use the same model architectures: a three-layer fully connected network for the tabular datasets; and an LST-Attn [27] for the time series.
We follow the setup of [49] and apply C-Mixup, Manifold-Mixup, Mixup, and ERM with their reported hyperparameters and provided code. For the ADA and Local-Mixup experiments, we use hyperparameter tuning and grid search to find the optimal training (batch size, learning rate, and number of epochs), and Local-Mixup parameters (distance threshold  $\epsilon$ ) and ADA parameters (number of clusters, range of  $\gamma$ , and whether to use manifold augmentation). We provide a detailed description in Appendix B.4. The evaluation metrics are Root Mean Squared Error (RMSE) and Mean Averaged Percentage Error (MAPE).
Results: We report the results in Table [1]. For full transparency, in the last row, we copy the results from [49]. We can assess that ADA is competitive with C-Mixup and superior to the other data augmentation strategies. ADA consistently improves the regression fit compared to ERM. Under the same conditions (split of data and Neural network structure), ADA is superior to C-Mixup. But, the degree of improvement is marginal on some datasets and as we show in the last row, we could not fully replicate their results. The only data in which ADA is significantly better than C-Mixup and the other strategies is for the Airfoil data, in which ADA reduces the error by around  $15\%$  with respect to the ERM solution.
Table 1: Results for in-distribution generalization. We report the average RMSE and MAPE of three different seeds. Standard deviations are reported in Appendix B.4. The best results per column are printed in bold and the second-best results are underlined (not applicable to the last row).
Airfoil	NO2	Exchange-Rate	Electricity
RMSE	MAPE	RMSE	MAPE	RMSE	MAPE	RMSE	MAPE
ERM	2.758	1.694	0.529	13.402	0.024	2.437	0.058	13.915
Mixup	3.264	1.964	0.522	13.226	0.025	2.513	0.058	13.839
ManiMixup	3.092	1.871	0.528	13.358	0.025	2.541	0.058	14.031
Local Mixup	3.373	2.043	0.524	13.309	0.021	2.136	0.063	14.238
C-Mixup	2.800	1.629	0.516	13.069	0.024	2.431	0.057	13.512
ADA	2.360	1.373	0.515	13.128	0.021	2.116	0.059	13.464
C-Mixup in [49]	2.717	1.610	0.509	12.998	0.020	2.041	0.057	13.372
4.4 Out-of-distribution Robustness
In this section, we evaluate the performance of ADA and compare it to prior approaches on tasks involving out-of-distribution robustness. We use the same datasets as [49] and closely follow their experimental setup.
Data: We use four of the five out-of-distribution datasets used in [49]. First, we use RCFashion-MNIST (RCF-MNIST) [49], which is a synthetic modification of Fashion-MNIST that models subpopulation shifts. Second, we investigate domain shifts using Communities and Crime (Crime) [12], SkillCraft1 Master Table (SkillCraft) [12] and Drug-target Interactions (DTI) [17] all of which are tabular datasets. For Crime, we use state identification, in SkillCraft we use "League Index", which corresponds to different levels of competitors, and in DTI we use year, as domain information. We split the datasets into train-, validation- and test data based on the domain information resulting in domain-distinct datasets. We provide a detailed description of datasets in Appendix B.4. Due to computational complexity, we could not establish a fair comparison on the satellite image regression dataset [23] (the fifth dataset in [49]), so we report some exploratory results in Appendix B.4.
Models and comparisons: As detailed in Section 4.3. Additionally, we use a ResNet-18 [15] for RCF-MNIST and DeepDTA [38] for DTI, as proposed in [49].
Results: We report the RMSE and the "worst" domain RMSE, which corresponds to the worst within-domain RMSE for out-of-domain test sets in Table 2. Similar to [49], we report the  $R$  value for the DTI dataset (higher values suggest a better fit of the regression model). For full transparency, in the last row, we copy the results from [49]. We can assess that ADA is competitive with C-Mixup and the other data augmentation strategies. Under the same conditions (split of data and Neural network structure), ADA is superior to C-Mixup. But, the degree of improvement is marginal on some datasets and as we show in the last row, we could not fully replicate their results. ADA is significantly better than C-Mixup and other strategies on the SkillCraft data, in which ADA reduces the error by around  $15\%$  compared to the ERM solution.
Table 2: Results for out-of-distribution generalisation. We report the average RMSE across domains in the test data and the "worst within-domain RMSE over three different seeds. For the DTI dataset, we report average R and "worst within-domain" R. Standard deviations are reported in Appendix B.4. The best results per column are printed in bold and the second-best results are underlined (not applicable to the last row).
RCF-MNIST	Crimes	SkillCraft	DTI
avg. RMSE	avg. RMSE	worst RMSE	avg. RMSE	worst RMSE	avg. R	worst R
ERM	0.164	0.136	0.170	6.147	7.906	0.483	0.439
Mixup	0.159	0.134	0.168	6.460	9.834	0.459	0.424
ManiMixup	0.157	0.128	0.155	5.908	9.264	0.474	0.431
LocalMixup	0.187	0.133	0.1590	7.251	10.996	0.470	0.433
C-Mixup	0.158	0.132	0.165	6.216	8.223	0.474	0.435
ADA	0.175	0.130	0.156	5.301	6.877	0.493	0.448
C-Mixup in [49]	0.146	0.123	0.146	5.201	7.362	0.498	0.458
We introduced Anchor Data Augmentation (ADA), an extension of Anchor Regression for the purpose of data augmentation. AR is a novel causal approach to increase the robustness in regression problems. In ADA, we systematically mix multiple samples based on a collective similarity criterion, which is determined via clustering. The augmented samples are modifications of the original samples that are moved towards or away from the cluster centroids based on the desired degree of robustness in AR. Our empirical evaluations across diverse synthetic and real-world regression problems consistently demonstrate the effectiveness of ADA, especially for limited data availability. ADA is competitive with or outperforms state-of-the-art data augmentation strategies for regression problems, even though the improvements are marginal on some datasets.
ADA can be applied to any regression setting, and we have not found any case in which the results were detrimental. To apply ADA, we only need to cluster our data and select a distribution for  $\gamma$ . We relied on vanilla  $k$ -means, and the results are robust with respect to the number of clusters. Other clustering algorithms might be more suitable for different applications. For setting the parameter  $\gamma$ , we used a uniform distribution. We believe a gamma distribution could be equally effective.
The purpose of data augmentation is to compensate for data scarcity in multiple domains where gathering and labeling data accurately by experts is impractical, expensive, or time-consuming. If applied properly, it can effectively expand the training dataset, reduce overfitting and improve the model's robustness, as was shown in the paper. However, It is important to note that the choice and combination of the data augmentation technique depends on the specific problem and using the wrong augmentation method may introduce additional bias to the model. More generally, incorrect data augmentation can lead to the following problems: overfitting the augmented data, loss of important information, introduction of unrealistic patterns and imbalanced presentation of the data. Detecting emerging problems due to data augmentation may not be straightforward. In particular, the performance on a test distribution that matches the training data distribution may be misleading and the model's predictions should be used with caution on new data that reflects the potential distribution shifts or variations encountered in real-world.
[38] Hakime Öztürk, Arzucan Özgür, and Elif Ozkirimli. Deepdata: deep drug-target binding affinity prediction. Bioinformatics, 34(17):i821-i829, 2018.
[39] Jonas Peters, Dominik Janzing, and Bernhard Schoelkopf, editors. Elements of Causal Inference. MIT Press, Cambridge, MA, 2017.
[40] Jonas Peters, Nicolai Meinshausen, and Peter Buhlmann. Causal inference by using invariant prediction: Identification and confidence intervals. Journal of the Royal Statistical Society Series B, 78(5):947-1012, 2016.
[41] Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309-1342, 2018.
[42] Dominik Rothenhausler, Nicolai Meinshausen, Peter Buhlmann, Jonas Peters, et al. Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B, 83(2):215-246, 2021.
[43] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[44] Zhiqiang Tang, Yunhe Gao, Leonid Karlinsky, Prasanna Sattigeri, Rogerio Feris, and Dimitris Metaxas. Online augmentation: Online data augmentation with less domain knowledge. In European Conference on Computer Vision, pages 313-329. Springer, 2020.
[45] Luke Taylor and Geoff Nitschke. Improving deep learning with generic data augmentation. In 2018 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1542-1547. IEEE, 2018.
[46] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR, 2019.
[47] Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. Advances in neural information processing systems, 34:237-250, 2021.
[48] Xiaogang Xu, Hengshuang Zhao, and Philip Torr. Universal adaptive data augmentation. arXiv preprint arXiv:2207.06658, 2022.
[49] Huaxiu Yao, Yiping Wang, Linjun Zhang, James Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. In Proceeding of the Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
[50] Sangwoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[51] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
[52] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? arXiv preprint arXiv:2010.04819, 2020.
[53] Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment. arXiv preprint arXiv:1912.11188, 2019.