Curvature Enhanced Data Augmentation for Regression
Authors: Ilya Kaufman
Omri Azencot
Venue: ICML 2025 poster (2025)
Keywords: Manifold learning
Data augmentation
Regression
Primary area: deep_learning->everything_else

Abstract: Deep learning models with a large number of parameters, often referred to as over-parameterized models, have achieved exceptional performance across various tasks. Despite concerns about overfitting, these models frequently generalize well to unseen data, thanks to effective regularization techniques, with data augmentation being among the most widely used. While data augmentation has shown great success in classification tasks using label-preserving transformations, its application in regression problems has received less attention. Recently, a novel manifold learning approach for generating synthetic data was proposed, utilizing a first-order approximation of the data manifold. Building on this foundation, we present a theoretical framework and practical tools for approximating and sampling general data manifolds. Furthermore, we introduce the Curvature-Enhanced Manifold Sampling (CEMS) method for regression tasks. CEMS leverages a second-order representation of the data manifold to enable efficient sampling and reconstruction of new data points. Extensive evaluations across multiple datasets and comparisons with state-of-the-art methods demonstrate that CEMS delivers superior performance in both in-distribution and out-of-distribution scenarios, while introducing only minimal computational overhead. Code is available at https://github.com/azencot-group/CEMS.

Curvature Enhanced Data Augmentation for Regression
Iyya Kaufman Omri Azencot
Abstract
et al., 2016; Nam & Han, 2016; Wu al., 2016). De- spite their success, these models are often significantly over- parameterized, meaning they possess more parameters than the number of training examples. As a result, deep neural networks are prone to overfitting, whereby they "memorize" the training set rather than learning generalizable patterns, thus compromising their performance on unseen data. Reg- ularization techniques are crucial to address this issue, as they modify the learning process to prevent overfitting by reducing the variance and increasing the bias of the under- lying model (Goodfellow, 2016). Classical regularization methods, such as weight decay, dropout (Srivastava et al., 2014), and normalization techniques like Batch Normaliza- tion (loffe & Szegedy, 2015) and Layer Normalization (Ba et al., 2016). have been effective in many scenarios. In ad- dition to these methods, recent research has explored the potential of data augmentation (DA) as form of regulariza- tion. In this paper, we focus on the problem of regularizing regression models via data augmentation. That is, we ex- plore how to artificially expand the train set (DA) for models that predict continuous values (regressors) to improve gen- eralization and robustness.
Deep learning models with large number of pa- rameters, often referred to parameterride models, have achieved exceptional performance across various tasks. Despite concerns about overfitting, these models frequently generalize well to unseen data, thanks to effective regular- ization techniques, with data augmentation be- ing among the most widely used. While data augmentation has shown great success in clas- sification tasks using label-preserving transfor- mations, its application in regression problems has received less attention. Recently, a novel manifold learning approach for generating syn- thetic data was proposed, utilizing first-order approximation of the data manifold. Building on this foundation, we present theoretical frame- work and practical tools for approximating and sampling general data manifolds. Furthermore, we introduce the Curvature-Enhancee Manifold Sampling (CEMS) method for regression tasks. CEMS leverages a second-order representation of the data manifold to enable efficient sampling and reconstruction of new data points. Exten- sive evaluations across multiple datasets and com- parisons with state-of-the-art methods demon- strate that CEMS delivers superior performance in both in-distribution and out-of-distribution sce- narios, while introducing only minimal compu- tational overhead. Code is available at https: //github. wrwpeennooooooocceee.e.
Early work in modern computer vision revealed the effec- tiveness of basic image transformations such as translation and rotation (Krizhevsky et al., 2012), promoting data aug- mentation to become one of the key components in design- ing generalizable learning models (Shorten & Khoshgoftaar, 2019). In particular, classification tasks, whose goal is to predict discrete label, benefited notably from the rapid de- velopment of DA (Simonyan & Zisserman, 2015; Vries, 2017; Zhang et al., 2018; Zhong al., 2020). The discrete and categorical nature of classification labels makes it easier to define label-preserving transformations and apply inter- polations without compromising data integrity. In contrast, regression tasks, where the outputs are continuous, face unique challenges in ensuring that transformations produce valid input-output pairs and that interpolations maintain the underlying functional relationships. While certain regres- sion challenges have adopted standard data augmentation approaches successfully (Redmon et al., 2016; Nochumsohn & Azencot, 2025), existing DA methods are generally less effective for regression problems (Yao et al., 2022). For this reason, developing data augmentation tools for gen- eral regression problems is an emerging field of interest
1. Introduction
Deep neural networks have demonstrated remarkable per- formance across a wide range of applications in various fields (Krizhevsky et al., 2012; Long et al., 2015; Mnih et al., 2015; Noh et al., 2015; Vinyals et al., 2015; He
'Department of Computer Science, Ben-Gurion University of the Negev, Beer-Shor ShevaS Israel. Correspondence to: Ilya Kaufman <ilyakau @ post. bgu.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Curvature Enhanced Data Augmentation for Regression
with a relatively small number of available effective tech- niques. One of the recent state-of-the-ar (SOTA) works introduced FOMA, a data-driven and Nomain-independen approach based on the theory and practice of manifold learn- ing (Kaufman& & Azencot, 2024b). Our work is also inspired by manifold learning, where we consider DA as a manifold approximation and sampling challenge.
distribution and out-of-distribution tasks. The contributions of our work can be summarized as follows:
1.
We consider data augmentation for regression as a manifold learning problem, extending and generalizing prior approaches through providing the foundational theory and practice.
Manifold learning is fundamental to modern machine learn- ing primarily through the manifold hypothesis (Belkin & Niyogi, 2003; Goodfellow, 2).6), where complex and high- dimensional data is assumed to lie close or on an associated low-dimensional manifold. Multiple works leveraged the re- lation between data and manifolds (Zhu et al., 2018; Ansuini et al., 2019; Kaufman & Azencot, 2023), and particularly, FOMA (Kaufman & Azencot, 2024b) can be viewed as method for generating new examples by sampling from the tangent space of the data manifold, approximated using the training distribution. The tangent space at a point is linear approximation of the manifold at that point (Lee, 2012), and thus, FOMA is a first-order approach. However, while first-order approximations work well for relatively simple or well-behaved data, they often fall short when dealing with complex, curved real-world data. We demonstrate this effect in Fig. IB-C, where first-order approximations of points with high curvature fail to capture the structure of the manifold. While it is natural to consider higher-order approximations for improving FOMA, their computational burden may be too limiting. In this work, we advocate that second- order manifold representations offer a compelling trade-off between effectiveness and compute requirements for data augmentation for regression problems.
2. We introduce CEMS, a novel bully-differemtiabl. data- driven and domain-independent data augmentation technique that based on a second-order approxima- tion of the data manifold.
3. Across nine datasets, featuring several large-scale, real- world in-distribution and out-o--distribttion tasks, we demonstrate that CEMS performs competitively or sur- passes other augmentation strategies.
2. Related work
The theoretical foundation for data augmentation (DA) is related to the study of Empirical Risk Minimization (ERM) (Vapnik, 1991) vs. Vicinal Risk Minimization (VRM) (Chapelle et al., 2000). In VRM, one considers extended distribution train on, in comparison to ERM, where the train distribution is used. Although related to generative modeling (Kingma & Welling, 2014; Goodfel- low et al., 2014; Sohl-Dickstein et al., 2015; Naiman et al., 2024b;a), data augmentation is a distinct approach com- monly used to expand data distributions by generating syn- thetic samples. With the increased dependence of deep models large volumes of data, DA has become corner- stone in enhancing the performance and generalization of neural networks VDeVries, 2017: Chen et al., 2020b; Feng et al., 2021: Yang et al.. 2022). Early work focused on Nomainn-deenndee augmentations for image, audio, and nat- ural language data (Krizhevsky et al., 2012; He et al., 6:16; Huang et al., 2017; Kobayashi, 2018; Park et al., 2019; Zhong al., 2020). Later, automatic augmentation tools have been proposed (Cubuk et al., 2019; Lim et al., 2019), including domain-dependent search spaces for transforma- tions. Still, adapting these methods to new data domains remains a challenge. This has sparked interest in devel- oping domain iittuiinnnddeeeddent approaches that make minimal assumptions about the data domain, and it is the focus of our research.
We propose the curvature enhanced manifold sampling (CEMS) approach, which generates new examples by draw- ing from a second-order representation of the data manifold. Particularly, CEMS randomly generates points in the tan- gent space of the manifold, whereas FOMA different as it scales down the orthogonal complement of the tangent space which captures how the manifold deviates from the linear approximation. FOMA and CEMS are data-driven and domain-indeeenddee underlying data distribution, whose domain can be arbitrary, e.g., time series, tabular, images, and other data forms. The inclusion of curvature information allows CEMS better capture the intrinsic geometry of the data manifold, as it accounts for the non-linearities and complex structures that first-order methods might miss. In general, second-order methods are infeasible in modern machine learning due to computational costs incurred by high- high-dimensional vectors. Nevertheless, our analysis shows that CEMS is governed by the intrinsic dimension of the manifold, and its value is much smaller than the data dimension D, i.e., d & D. We extensively evaluate CEMS and show it is competitive in comparison to SOTA data augmentation approaches on in-
i.c., their samples are based on the
DA for Classification. Zhang et al. (2018) introduced mixup, well-known domain-indepenaa DA method that convexly combines pairs of input samples and their one- hot label representations during training. Following their work, a plethora of mixup- based techniques have been sug- gested such as manifold mixup (Verma et al., 2019) which extends the idea of mixing examples to the latent space.
2

---

Curvature Enhanced Data Augmentation for Regression
Original data
FOMA
CEMS (Ist order)
CEMS
1.0
D
-0.5-
1.0
0
4
0
2
0
2
6
0
6
Figure 1. We demonstrate the effect of sampling from a one-dimensional manifold embedded two-dimensional space. A) The original data representing sine wave where the color of each point represents the curvature at that point (brighter means higher curvature). B) Sampling using FOMA. C) Sampling using a first-order approximation. D) Sampling using CEMS (our approach).
CutMix (Yun et al., 2019) implants a random rectangular region of the input into another and many others (Guo et al., 2019; Hendrycks et al., 2020; Berthelot et al., 2019; Gree- newald et al., 2023; Lim et al., 2022). Recently, (Erichson al., 2024) extended the work (Lim et al., 2022) via stable training and further noise injections. While the family of mixup techniques have been shown consistently improve classification learning systems (Cao et al., 2022), its efficacy is inconsistent on regression tasks (Yao et al., 2022).
Manifold Learning, Manifold learning has been a funda- mental research area in machine learning, aiming to discover the intrinsic low-dimensional structure of high- nigh-dimensional data. While early work focused on dimensionality reduction of points and preserving their geometric features (Tenen- baum et al., 2000; Roweis & Saul, 2000; Belkin & Niyogi, 2003; Weinberger & Saul, 2004; Zhang & Zha, 2005; Coif- man & Lafon, 2006), modern approaches also considered regularization (Ma et al., 2018; Zhu et al., 2018), explain- able artificial intelligence (Ansuini et al., 2019; Kaufman & Azencot, 2023; 2024a), and autoencoding (Chen et al., 2020a), among other applications.
DA for Regression. Unfortunately, there has been consid- erably less focus on developing data augmentation methods for regression tasks in comparison to the classification set- ting. Due the simplicity and effectiveness of mixup-based tools in classification, growing body of literature is drawn to adapting and extending the mixing process for regression. For instance, RegMix (Hwang & Whang, 2021) learns the optimal number of nearest neighbors to mix per sample. C-mixup (Yao et al., 2022) employs a Gaussian kernel to create a sampling probability distribution for each sample, taking label distances into account, and selecting samples for mixing according this distribution. Anchor Data Aug- mentation (Schneider et al., 2023) clusters data points and adjusts the original points either towards or away from the cluster centroids. R-Mixup (Kan et al., 2023) focuses on enhancing model performance specifically for biological networks, whereas RC-Mixup (Hwang et al., 2024) extends C-mixup be more robust against noise. Perhaps clos- our work the recent FOMA method (Kaufman & Azencot, 2024b) that does not rely mixing samples, but rather, it samples from first-order approximation of the data manifold. Still, to the best of our knowledge, our work first suggesting fundamental manifold learning theory and tools for DA, accompanied by an effective second-order augmentation technique.
Second-order methods have also been investigated in this domain. Notably, Donoho & Grimes (2003) introduced Hessian Eigenmaps, a technique that incorporates second- order information to more accurately preserve local curva- ture during nonlinear dimensionality reduction. Although their approach is designed for unsupervised embedding, our method extends the use of second-order local approxima- tions the supervised setting. Specifically, we utilize this geometric insight to guide data augmentation, enabling the generation of new training samples in manner that is both geometry-aware and differentiable.
Recent work from the statistical learning community has also proposed more sophisticated local chart models. These include Gaussian process-based manifold inference (Dun- son & Wu, 2021), spherelet-basee approximations (Li et al., 2022), and manifold denoising via generalized L, medi- ans (Faigenbaum-Gokke Golovvi & Levin, 2020). While these methods aim to infer or reconstruct manifold structure with statistical guarantees, our focus is on utilizing local geo- metrie information specifically for data augmentation in supervised learning.
Recent advancements in manifold learning have enhanced
3

---

Curvature Enhanced Data Augmentation for Regression
anomaly detection and out-of-distribution (OOD) recog- nition. Li al. (2024) leveraged submanifold geome- try, estimating tangent spaces and curvatures to define in- distribution regions for OOD detection. Gao et al. (2022) proposed a hyperbolic feature augmentation method, using the Poincaré ball model for distribution estimation and in- finite sampling, improving few-shot learning performance. Humayun al. (2022) proposed MaGNET, a framework that enables uniform sampling on data manifolds derived from generative adversarial networks providing a retraining- free solution for data augmentation. Similarly, Chadebec & Allassonnière (2021) introduced a geometry-aware varia- tional autoencoder that leverages second-order Runge-Kung Kutta schemes for effective data generation in low-sample-size scenarios. Extending these ideas, Cui et al. (2023) presented trajectory-wware principal manifold framework for image generation and data augmentation, which aligns sampled data with learned projection indices to improve representa- tion and synthesis quality.
where U = = u.,... _ " representations of Z. In practice, CAML parameterizes f by projecting E Z to its tangent and normal spaces at u E M,, where the tangent space is obtained by a lin- ear transformation and the normal space is provided via a second-order local approximation. Specifically, given a point = E Z, we find close points N₂ = { forming the neighborhood of z. Next, we construct an orthonormal basis B, == [By, E Dxddd By,, TR. Dx( E RRD for the tangent space T.M and the normal space N at where the operation [::]] denotes column-wise con- catenation. We then project N₂ and onto BT, and BN., } yielding U_- = {u,} u and G. = {9; 15-1 1- respec- tively. To allow arbitrary sampling from M, we assume that g is a map from the tangent space the normal space, i.c. g : TuM - M. The second-order Taylor expansion of g around a point . is given by
Rd are low-dimensional
g(u,) g(u) + (ug - v))"(((())
+ 2 (u, - u)) uu + 0(u,)))) (2)
3. Background
where the linear (gradient) and quadratic (Hessian) terms are unknown. To compute Vg(u) and H()), one needs to collect the linear coefficients and constants arising from Eq. 2 into matrices v and G. respectively, solve a linear system of equations (Eq. 6). and extract the numerical esti- mates of the gradient and Hessian. Finally, we can map the pair (u, g) back into its original space = E Z by computing 2 := f(u) = Bu [u, g(u)]. See also App. A.
3.1. Manifold learning
manifold M C [Rd is mathematical structure that locally resembles an Euclidean space near each of its points (Lee, 2012). A ubiquitous assumption in machine learning states that high-dimensional point clouds Z C TR. P satisfy the manifold hypothesis. Namely, the data Z lie on a mani- fold M whose intrinsic dimension d is significantly lower than the extrinsic dimension of the ambient space D. i.e., < D (Goodfellow, 2016). Manifold learning is field in machine learning that develops theory and tools for analyz- ing and processing high- -igeedimensional data under the lens of geometric manifolds.
4. Curvature Enhanced Manifold Sampling
We assume to be given a regression training set D := ((xt, (od.y') y') where x² E R is the data sample and y' E R is its corresponding prediction. Following the approach used in FOMA (Kaufman & Azencot, 2024b), we denote by =' = [x² yy' E DR the concatenation of the input x² and its corresponding label y² along the columns, treating the pair as point on a joint input-output manifold. During training, given a mini-batch Z = {2',-- ... 22)) where b is the batch size, perform the following procedure for each E Z to create new sample ă, omitting the superscript to simplify notation. To account for the discrepancies in scale between X and Y, we normalize Y to match the range of X, i.e., [0, 1]. Our curvature enhanced manifold sam- pling (CEMS) augmentation approach consists of four main steps: 1) Extract a neighborhood N₂ from D for every point 2; 2) Construct basis B., := [BT.. By..] for the tangent space T.M and the normal space N,,M and project the neighborhood onto it; 3) Form and solve the linear system of equations in Eq. 6 to obtain the parameterization g; 4) Sample a new point from T,,, evaluate its via Eq. 2, and un-project it onto the ambient space using f. Below, we de- tail how we perform each step, and we motivate our design
3.2. Curvature-aware manifold learning
Our curvature enhanced manifold sampling (CEMS) data augmentation method based on second-order approxima- tion of the data manifold. There are several existing practical approaches for parameterizing manifold given a collection of data points Z = {2', 22. ................... ,2N3 } C focus on the curvature-aware manifold learning (CAML) algorithm (Li, 2018), since it scales to high-dimensional problems, is numerically stable, and it is easy to code.
PR.. Here, we
Below, we include the necessary details for describing our approach, and we refer the reader to (Lee, 2012; 2018; Li, 2018 8) for additional details on Riemannian geometry and its realization in machine learning.
Following our discussion above, we assume Z satisfies the manifold hypothesis. Formally, it means that there exists an embedding map f : :M- + RD such that
²= =(()), i= i N,
4

---

Curvature Enhanced Data Augmentation for Regression
Neighborhood of
Tangent space Tu
BF.-
y N
U.
M
f(r) +
Figure 2. CEMS forms a neighborhood for every point = (left), computes basis Br, for the tangent space via SVD (middle) while obtaining estimate for the embedding f, samples new point 17 close (right), and un-projects it back to R. using f (red arrow).
choices. Pseudo-code and illustration of CEMS are given in Alg. and Fig. 2.
3j E Ng This improves efficiency, though the cost of accuracy, since every 2j E N₂ is assumed to share the same neighborhood. Finally, the batch size determines the number of neighbors for each point, providing a balance between computational efficiency and accuracy.
Neighborhood extraction. The approach we consider in this work is local in the sense that we represent the manifold structure in the vicinity of a specific point 2 E 'D or within its neighborhood N₂. High-quality approximations of local properties of the manifold Vg(u) and H (u) depend directly A straightfor-
Basis construction and projection. To find an orthonor- mal basis for the tangent space T,M and the normal space Mg we follow standard upproaches (Singer & Wu. 2012: Li, 2018) that utilize the singular value decomposition (SVD). Specifically,
on the proximity of the elements in N₂ ward approach to extracting N₂ is to compute the k-nearest neighbors (kNN) (Cover & Hart, 1967) of z. Specifically, we construct neighborhoods in the joint input-output space X x y to align with the manifold hypothesis, preserving the local continuity of the data and avoiding the artificial separations introduced by clustering- based methods. For each point 2; Z, N₂, is defined the ls closest points in feature space, which ensures the local geometry is captured reliably while minimizing diversity within the neighbor- hoods. Furthermore, our reliance on the Euclidenn prior assumes that the manifold is sufficiently smooth at small scales, justifying the validity of linear approximations such as those employed by our method. This assumption un- derpins the extraction of neighborhood sets and their utility in manifold analysis, it guarantees that the neighborhoods respect the underlying manifold structure.
compute SVD on the centered points {-, = マトا USVT while keeping our pipeline to be fully differentiable (Ionescu et al., 2015). Importantly, SVD is calculated once for every batch, as discussed above. The first d columns U determine the basis for the tangent space, i.e., BT, == U[: 1:d] and the last D d columns de- termine the basis for the normal space Bx_ == U|: 1: such that B, bases. While the intrinsic dimension d can be viewed as hyper-parameter of CEMS, we estimate it in practice using a robust estimator (Facco et al., 2017). The centered neigh- bors 2ÿ - are projected to the tangent space and the normal space via uj == BT: (z, 2). g(u,) := BT: (2y - 2), respectively, where AT is the transposed matrix of A, and A a is a matrix- multiplication. Centering the points around map the point u to the zero vector, and thus, Eq. is transformed the following approximation:
[Br.. By., is the concatenation of the
Our analysis below and in App. D shows that the com- putational complexity of CEMS is governed by singular value decomposition (SVD) calculations, required for basis construction. To reduce runtime, D can be pre-processed, storing (y( and H (u) for every z := fu) on the disk, as these properties remain unchanged during training. While this pre-computation reduces runtime significantly, it incurs high memory complexity, O(2((D- - d)), and the choice of neighbors is fixed during training. To address these limita- tions, we use the same neighborhood for all points in N,, re-using neighborhoods and basis computations for every
g(u,) = uz Vg(u) 2f H(w)uz
(3)
Linear system of equations. Under the change of basis B,,, we form the matrices v and G as described in App. A, containing {u, 3/- and {g(u))}f respectively. We then solve Eq. 6 via differentiable least squares, and we obtain an estimation of Vg(u) and H Hu)) allowing to map new points in the vicinity of by computing g. Note that while N;
5

---

Curvature Enhanced Data Augmentation for Regression
Algorithm 1 CEMS,* Sample generation
sample point 7 near ui 1) M (u, 0 cla). Step 8 changes to g(m) = ((4) + (1) -4) =
Require: Training data Z (= [x+, w''l
(mn w)² H H(m w) and Step 9 changes to 2m :- f(m) = B, [m, mmm]] + UN., A full description of the algorithm appears in Alg. 2.
Require: A sample 2 E Z
1: Find k-nearest neighbors N₂ of 2
2: Construct an orthonormal basis B,, spanning N₂-
3: Project 2j - to local orthonormal coordinates:
Ug f +- BT (zj - =2)
Complexity analysis. There are two computationally demanding calculations used by CEMS, SVD and least squares. Given a data mini-batch Z E R where is the batch size. Then, SVD requires O D)P)) op- erations, whereas the solution of under-determined least squares costs (b2dd). Using the manifold hypothesis, we assume that d < D therefore d (DD and thus, the over- all time complexity of CEMS is given by O D) which is proportional to the ambient dimension D. See also App. D for a more detailed analysis.
4: Construct G and V/ as in Eq. (6)
5: Solve V A = G
6: Extract Vg(z) and (z) from A
7: Sample noise 1) ola)
8: Calculate g(n) from:
g(n) c 77 Vg + - Hnj.
9: Un-project 7 back to the original coordinates:
Zy - [(n) = B,, (.(.gg())) +
Return Zn
and B. are shared across the batch, the linear solve is still computed separately per point.
Memory analysis. The memory requirements of CEMS are primarily dictated by the computation of the SVD. No- tably, the SVD is computed independently for each batch rather than for the entire dataset. In our PyTorch imple- mentation, we leverage the economy/reduced SVD variant, which significantly reduces memory usage compared to the full SVD. For batch matrix of size b x D (where b is the batch size and D is the ambient dimension), the space complexity is O(D + min(b, D) + D)). This is sub- stantially more efficient than the full SVD, which requires O(bD+b2 memory. In practice, CEMS is particularly effective in scenarios where the batch size b is much smaller (4)
Sampling and un-projecting. To generate new examples using the above machinery, we need sample a point 1) E Rd from the neighborhood of u. and un-project it to the ambient space Rd (through the parameterization y and map f). While various sophisticated sampling techniques could be devised, we opted for a simple sampler with a single hyperparameter. In practice, we draw 1) ~ (O, aIa). To un-project vi back to the original space, we compute
Eqp == f(n) = Bu (1))( +
than the ambient dimension D (common in deep learning), resulting in a memory complexity that is proportional to D.
Adaptation to batches. For completeness, we also de- scribe briefly the adaptation of CEMS to the training setting where we utilize mini-batches. We provide a comparison between the two methods in App. G. In contrast to CEMS,,, where neighborhood close points is constructed for each individual sample to generate a corresponding synthetic point, CEMS requires a batch of mutually close points to estimate shared tangent space and produce new batch of samples. As mentioned above, given z and its neighborhood Nz. we re-use the same neighborhood and subsequent basis computations for every 2/ Nz. This adaptation requires a small modification to the method. 1) We include the point z : 20 in the neighborhood N; = {z, / 2) We find an orthonormal basis B, that spans N₂- μ. where μ is the mean of Nz. 3) After projecting to the coordinates of B,, we get U.= = {u, 1 and G_ {9, 1 1 For every point Uj. we gather set of close points and their embed- dings via g. 4) In contrast to the point-wise basis estimation, where u served as the origin (zero vector), in the batch-wise computation we need to account for uy and g(u;) in Eq. 2. While Steps 5-7 in Alg. 1 remain unchanged, at Step we
Comparison with FOMA. FOMA (Kaufman & Azen- cot, 2024b) can be interpreted as special case of CEMS. Specifically, we can describe FOMA using our notations as follows: given a sample 2 and its neighborhood N_, FOMA constructs a basis B, := =[BT.. Bx,] for the tangent space T.M and the normal space N, and projects the neighbor- hood onto it, yielding U_ [w²) and G- {g:} respectively. Rather than estimating the gradient Vg(u;) and Hessian H(u,) at each point uj E U,, and then sam- pling using the Taylor expansion as performed in CEMS, FOMA generates new samples by scaling down G.. That is, for each 2j E NN. the corresponding 9J = >g; is scaled where E 1). To complete the sampling process, every uy is un-projected back to the original coordinates by com- puting 2 == f(u,) B. [uy, Ag( Therefore, FOMA does not use the embedding map y as detailed in Eq. 2, but it samples random points instead. Unfortunately, this sam- pling technique may yield new points that are not on the data manifold, especially on highly curved locations, as is also illustrated in Fig. 1.
6

---

Curvature Enhanced Data Augmentation for Regression
Table 1. Results for in-distribution generalization. Values in bold indicate the hest results, while underlined values represent the second best. We present the average RMSE and MAPE across three seeds. Detailed results, including standard deviation, are available in App K.
Airfoil
NO2
Exchange- Rate
Electricity
RMSE
MAPE
RMSE
MAPE
RMSE
MAPE
RMSE
MAPE
ERM
2.901
1.753
0.537
13.615
0.024
2.423
0.058
13.861
Mixup
3.730
2.327
0.528
13.534
0.024
2.441
0.058
14.306
Mani Mixup
3.063
1.842
0.522
13.382
0.024
2.475
0.058
14.556
C-Mixup
2.717
1.610
0.509
12.998
0.020
2.041
0.057
13.372
ADA
2.360
1.373
0.515
13.128
0.021
2.116
0.059
13,464
FOMA
1.471
0.816
0.512
12.894
0.013
1.262
0.058
14.614
CEMS
1.455
0.809
0.507
12.807
0.014
1.293
0.058
13.353
5. Experiments
Datasets. We evaluate in-distribution generalization us- ing four datasets. Two of these are tabular datasets: Airfoil Self-Noise (Airfoil) (Brooks et al., 2014). containing aerody- namic and acoustic measurements of airfoil blade sections, and NO2 (Aldrin, 2004), which predicts air pollution levels at specific locations. We also use two time series datasets: Exchange- Rate and Electricity (Lai et al., 2018), where Exchange- Bachange-Rate includes daily exchange rates of several cur- rencies and Electricity contains measurements of electric power consumption in private households. For detailed description of these datasets, see App.
5.1. Sine example
Real-world data is often complex and curved, exhibiting in- tricate patterns that cannot be adequately captured by linear or simplistic models. By employing higher-order approx- imations of the manifold, we can generate samples that align with the true nature of woal-world data. In Fig. 1, we demonstrate a toy sine example, highlighting the differences between first-order and second-order approaches. Specif- ically, we generated two-dimensional point cloud of sine wave whose intrinsic dimension is one (Fig. 1, left). Then, sampled points from this distribution using mini- batches from the train set and various data augmentation techniques. The first-order method, FOMA (Kaufman & Azencot, 2024b), struggles to adhere to the curvature of the manifold in highly-curved points, as can be seen in Fig. 1, middle left. Similarly, restricting CEMS to a first-order approximation presents similar behavior (Fig. 1, middle right). Finally, our second-order CEMS method samples the manifold well, even near high curvature areas (Fig. 1. right). See App. A.1 for a theoretical justification for the sampling error of CEMS in comparison to first-order approaches.
Experimental Settings. We perform comparative anal- ysis between our method, CEMS, and several established baseline approaches, including the standard empirical risk minimization (ERM) training, Mixup (Zhang et al., 2018 1), Manifold- (Verma et al., 2019), C-Mixup (Yao et al., 2022), Anchor Data Augmentation (ADA) (Schneider et al., 2023), and FOMA (Kaufman Azencot, 2024b). The neu- ral networks trained are the same considered in (Yao et al., 2022), where fully connected three layer model was used for tabular datasets, and an LST-Attn (Lai et al., 2018) is utilized for time series data. The evaluation met- rics include the root mean square error (RMSE) and mean absolute percentage error (MAPE). Additional details and hyperparameters are available in App.
5.2. In-Distribution Generalization
In what follows, we consider the in-distribution benchmark that was introduced in (Yao et al., 2022). This benchmark evaluates the performance of various data augmentation techniques in the setting of training on a train set and its augmentations, while testing on test set that was sampled from the same distribution the train set. Thus, strong performance in this benchmark implies that the underlying DA method mimics the train distribution well. Below, we compare CEMS to other recent SOTA approaches, while using the same datasets that were studied in (Yao et al., 2022) and closely replicating their experimental setup.
Results. We present the in-distribution generalization benchmark results in Tab. 1. The results of all previous methods are reported as they appear in the corresponding original papers. Lower values are preferred either in RMSE or in MAPE. Boldface and underline denote the best and sec- ond best approaches, respectively. Remarkably, across all datasets and metrics, CEMS attains the best second best error measures. In particular, CEMS outperforms other data augmentation strategies on Airfoil and NO2 while being comparable with FOMA on Electricity and Exchange-Rate We also note that when CEMS is second best, its result is relatively close to the best result. We present the full results
7

---

Curvature Enhanced Data Augmentation for Regression
Table Comparison of out-of-distribution rohustness. Bold values indicate the best results, while underlined values represent the second
best. We present the average RMSE across domains as well as the "worst within- domain" RMSE from three different seeds. For the DTI
Poverty datasets, we provide the average A and the "worst within-domain" R. Complete results, including standard deviation, can be
found in App K.
RCF (RMSE)
Crimes (RMSE)
SkillCraft (RMSE)
DTI (R)
Poverty (R)
Avg.
Avg. +
Worst 4
Avg. +
Worst
Avg. 1
Worst t
Avg. t
Worst 1
ERM
0.164
0.136
0.170
6.147
7.906
0.483
0.439
0.80
0.50
Mixup
0.159
0.134
0.168
6,460
9.834
0.459
0.424
0.81
0.46
ManiMixup
0.157
0.128
0.155
5,908
9.264
0.474
0.431
C-Mixup
0.146
0.123
0.146
5.201
7.362
0.498
0.458
0.81
0.53
ADA
0.175
0.130
0.156
5,301
6.877
0.493
0.448
0.79
0.52
FOMA
0.159
0.128
0.158
0.503
0.459
0.78
0.49
CEMS
0.146
0.128
0.159
5.142
6.322
0.511
0.465
0.81
0.50
including standard deviation measures in App. K.
Experimental Settings. Similar to Sec. 5.2, we consider the same baseline DA approaches. For metrics, we report the RMSE (lower values are preferable) for RCF, Crimes, and SkillCraft. addition, we use (higher values are preferable) as the evaluation metric for Poverty and DTI, as was originally proposed in their corresponding papers (Koh et al., 2021; Huang et al., 2021). For fair comparison, we follow the methodology in (Yao et al., 2022), and we train ResNet-18 on the RCF and Poverty datasets, three-layer fully connected networks on Crimes and SkillCraft, and DeepDTA (Öztürk et al., 2018) on DTI. We provide further details on hyperparameters and experiments App.
5.3. Out-of-distributio
To extend our in-distribution evaluation, we also consider an out-of-distribution benchmark, as was proposed in (Yao et al., 2022). Unlike the in-distribution case, here the test set is sampled from a distribution different from that of the train set. Therefore, excelling in this scenario provides valuable information regarding the generalization capabilities of data augmentation tools. In what follows, perform compari- son between CEMS and several SOTA methods, while using the same datasets that were studied in (Yao et al., 2022) and closely replicating their experimental setup.
Results. We detail our out-of-distribution benchmark re- sults in Tab. 2. Similarly to the in-distribution setting, the error measures of previous SOTA approaches were taken from the related original papers. We include both the aver- age (Avg.) and worst domain performance metrics. Lower values are preferred in RMSE, and higher values are opted for R. We denote in bold and underline the best and second best results, respectively. Our results indicate that CEMS attains strong performance measures, achieving the best re- sults in 6/9 tests. Further, the rest of the error measures of CEMS are either second best or very close to the second best. We particularly note the SkillCraft test where CEMS improves the second best results by relative 1% and 8% for the average and worst metrics. The relative improve- ment is computed via Grel 100, where Grel = (e ecEMs)/e, with eCEMS and e denoting the errors of CEMS and the sec- ond best approach, respectively. We present the full results including standard deviation measures in App. K.
Datasets. We leverage five datasets to evaluate the perfor- mance of aut-of-distribunn
robustness. 1) RCFashionM- NIST (RCF) (Yao et al., 2022) is a synthetic variation of Bushion-MNISS designed to model sub-population distribu- tion shifts, with the aim of predicting the rotation angle for each object. 2) Communities and Crime (Crime) (Redmond, 2009) is tabular dataset focused on predicting the total number of violent crimes per 100,000 population, aiming to create a model that generalizes to states not included in the training data. 3) SkillCraftI Master Table (Skill- Craft) (Blair et al., 2013) is a tabular dataset designed to predict the average latency in milliseconds from the onset of perception- aatiionnn-tiiinnnnn cycles to the first action where "LeagueIn- dex" considered domain information. 4) Grug-Targed Interactions (DTI) (Huang al., 2021) seeks to predict drug- target interactions that are -itfo---distibutionn using the year as domain data. 5) PovertyMap (Poverty) (Koh et al., 2021) is a satellite image regression dataset created to estimate asset wealth in countries that were not part of the training set. For more details, please refer App.
6. Conclusions
This work introduces CEMS, a novel data augmentation method tailored specifically for regression problems, framed within the context of manifold learning. By leveraging
8

---

Curvature Enhanced Data Augmentation for Regression
second-order manifold approximations, CEMS captures the underlying curvature and structure of the data more accu- rately than previous first-order methods. Our extensive eval- uation across nine diverse benchmark datasets. spanning both in-distribution and out-of-distributte tasks, shows that CEMS achieves competitive performance compared to SOTA techniques, often surpassing them in challenging settings. The main contributions of this work are threefold: (1) extend the view of DA for regression learning problem, thereby providing principled foundation and practical tools; (2) proposing CEMS, fully differen- tiable, data-driven, and domain-independert second-order augmentation method; and (3) empirically validating CEMS across a variety of regression scenarios, showing its poten- robust and effective regularization tech- nique for models predicting continuous values. Our results suggest that higher-order manifold sampling approaches hold promise for improving the generalization of regression models, especially in scenarios with limited data.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
manifold
tial to serve
One limitation of CEMS is that the linear system in Eq. 6 might be underdetermined for data sets with large intrinsic dimension d. In practice, the number of neighbors has to be (dd), for overdetermined system. Our implementation sets the number of neighbors to be constant size and thus it is independent of d. While this requirement is reasonable for low values, it can become expensive for large d. This can be resolved by regularizing the linear system via, c.g., ridge regression. Another limitation related to the SVD computation, where CEMS needs at least d columns. This may require full SVD calculation, demanding O(DD²) memory, where b is the batch size and D is the extrinsic di- mension, which may be impractical for datasets with many features. A potential solution is to consider a different intrin- sic dimension d, such that d < d. In future work, we plan to investigate these ideas, and, in addition, we plan to explore extensions of CEMS to include adaptive strategies for dy- namically selecting the appropriate order of approximation based on local data properties. By pushing the boundaries of data augmentation for regression, we hope to pave the way for more robust and versatile learning systems capable of tackling complex, real-world prediction tasks.
Acknowledgments
This research was partially supported by the Lynn and William Frankel Center of the Computer Science Depart- ment, Ben-Gurion University of the Negev, an ISF grant 668/21, an ISF equipment grant, and by the Israeli Council for Higher Education (CHE) via the Data Science Research Center, Ben-Gurion University of the Negev, Isracl.

---

Curvature Enhanced Data Augmentation for Regression
References
Cover, T. and Hart, P. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21-27, 1967.
Aldrin, M.
CMU statlib
dataset.
http://iibssat..mmu. http:/bbsstttmuu...dddd/atttttt/ 2004.
Cubuk, D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. AutoAugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 113- 123, 2019.
Ansuini, A., Laio, A., Macke, J. H., and Zoccolan, D. In- trinsic dimension of data representations in deep neural networks, Advances in Neural Information Processing Systems, 32, 2019.
J. I.., Kiros, J. R., and Hinton, G. E. Layer normalization. In NIP,, 2016.
Cui, E., Li, B. Y., Wong, W., and Wang, D. Trajectory-aware principal manifold framework for data augmentation and image generation. arXiv preprint arXiv:2310.07801, 2023.
Belkin, M. and Niyogi, P. Laplacian eigenmaps for di- mensionality reduction and data representation. Neural computation, 15(6):1373-1366 2003.
Deieiss, T. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: (eXXiv1708.0455:: 2017.
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. A. MixMatch: A holistic ap- proach to semi-supervised learning, Advances in Neural Information Processing Systems, 32, 2019.
Donoho, D. L.. and Grimes, C. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data, Proceedings of the National Academy of Sciences, 100 (10):5591-5596 2003.
Birdal, T., Lou, A., Guibas, L. J., and Simsekli, U. Intrin- sic dimension, persistent homology and generalization in neural networks. Advances in neural information pro- cessing systems, 34:6776-6789, 2021.
Dunson, D. B. and Wu, N. Inferring manifolds from noisy data using gaussian processes. arXiv preprint arXiv:2110.07478, 2021.
Blair, M., Thompson, J., Henrey, A., and Chen, B. Skill- craftl master table dataset. UCI Machine Learning Repos- itory, 2013.
Erichson, B., Lim, S. H., Xu, W., Utrera, F., Cao, Z. and Mahoney, M. NoisyMix: Boosting model robustness to common corruptions. In International Conference on Artificial Intelligence and Statistics, pp. 4033-4041. PMLR, 2024.
Brooks, T., Pope, D., and Marcolini, M. Airfoil Self-Noise. UCI Machine Learning Repository, 2014.
Cao, C., Zhou, F., Dai, Y., and Wang, A survey of mix-based data augmentation: Taxonomy, meth- ods, applications, and explainability. arXiv preprint arXiv:2212.10888. 2022.
Facco, E.,, d' Errico, M., Rodriguez, A., and Laio, A. Esti- mating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):12140, 2017.
Chadebec, C. and Allassonnière, S. Data generation in low sample size setting using manifold sampling and geometry-aware vae. CoRR, 2021.
Paigenbaum-Golovie S. and Levin, D. Manifold recon- struction and denoising from scattered data in high di- mension via a generalization of 11-median. arXiv preprint arXiv:2012. 12244 2020.
Chapelle, O., Weston, J., Bottou, L.,, and Vapnik, V. Vici- nal risk minimization. Advances in neural information processing systems, 13, 2000.
Feng, S. Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., and Hovy. E. H. A survey of data augmentation approaches for NLP. In Findings of the Association for Computationnal Linguistics: ACL/IJCNLP, volume ACL/UCNLPP 2021 of Findings of ACL, pp. 968- 988. Association for Computational Linguistics, 2021.
Chen, N., Klushyn, A., Ferroni, F., Bayer, J., and van der Smagt, P. Learning flat latent manifolds with VAEs. In Proceedings of the 37th International Conference on Ma- chine Learning, ICML, volume 119 of Proceedings of Ma- chine Learning Research, pp. 1587-1596. PMLR, 2020a.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Fowkes, M., Gould, N. I,, and Farmer, C. I.. branch and bound algorithm for the global optimization of hes- sian lipschitz continuous functions. Journal of Global Optimization, 56:1791-1815, 2013.
Coifman, R. R. and Lafon, S. Diffusson maps. Applied and Computational Harmonic Analysis, 21:5-30, July 2006.
Gao, Z., Wu, Y., Jia, Y., and Harandi, M. Hyperbolic fea- ture augmentation via distribution estimation and infinite
10

---

Curvature Enhanced Data Augmentation for Regression
sampling on manifolds. Advances in neural information processing systems, 15:33421-34433 2022.
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Goodfellow, I. Deep learning, 2016.
Ionescu, C., Vantzos, O., and Sminchisescu, C. Matrix backpropagation for deep networks with structured layers. In Proceedings of the IEEE international conference on computer vision, pp. 2965-2965999 0965-2973, 2015.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio. Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Kan, X., Li, Z.., Cui, H., Yu, Y., Xu, R., Yu, S., Zhang, Z.,, Guo, Y., and Yang, C. R-mixup: Riemannian mixup for biological networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1073- 2023.
Grassberger, P. and Procaccia, I. Characterization of strange attractors. Physical review letters, 50(5):346, 1983.
Greenewald, K. H., Gu, A., Yurochkin, M., Solomon, J., and Chien, E. k-Mixup regularization for deep learning via optimal transport. Trans. Mach. Learn. Res., 2023.
Kaufman, I. and Azencot, O. Data representations' study of latent image manifolds. In International Conference on Machine Learning, pp. 15928-1555 159445. PMLR, 2023.
Guo, H., Mao, Y., and Zhang, R. Mixup as locally linear out-of-manifole regularization. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 33,
Kaufman, I. and Azencot, O. Analyzing deep transformer models for time series forecasting via manifold learning, Trans. Mach. Learn. Res., 2024a.
pp. 3714-3722, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Kaufman, I. and Azencot, O. First-order manifold data augmentation for regression learning. In Forty-first Inter- national Conference on Machine Learning, ICMI., 2024b.
Hendryeks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. AugMix: A simple data processing method to improve robustness and uncertainty. In 8th International Conference on Learning Representa- tions, ICLR, 2020.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR, 2014.
Kobayashi, S. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Short Papers), pp. 452 457, 2018.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEFE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Huang, K., Fu, T., Gao, W., Zhao, Y., Roohani, Y., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., and Zitnik, M. Ther- apeutics data commons: Machine learning datasets and tasks for drug discovery and development. In Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021.
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, L., Gao, I., et al. Wilds: A benchmark of in-the- wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664, PMLR, 2021.
Humayun, A. I., Balestriero, R., and Baraniuk, R. Magnet: Uniform sampling from deep generative network mani- folds without retraining. In The International Conference on Learning Representations (ICIR) 2022, 2022.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. NIPS, 2012.
Lai, G., Chang, W.-C., Yang, Y., and Liu, H. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR confer- ence on research & development in information retrieval, pp. 95-104, 2018
Hwang, S.HH. and Whang, S. E. Regmix: Data mixing augmentation for regression. arXiv preprint 13X3:22006.03344, 2021.
Hwang, S.-H., Kim, M., and Whang, S. F. RC-Mixup: A data augmentation strategy against noisy data for regres- sion tasks. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1155-1165, 2024.
Lee, J. M. Smooth manifolds. Springer, 2012.
Lee, J. M. Introduction to Riemannian manifolds, volume 2. Springer, 2018

---

Curvature Enhanced Data Augmentation for Regression
Levina, E. and Bickel, Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, 2004.
Noh, H., Hong, S., and Han, B. Learning deconvolution network for semantic segmentation. In ICCV, 2015.
Öztürk, H., Özgür, A., and Ozkirimli, DeepDTA: deep drug-target binding affinity prediction. Bioinformatics, 34( 34(17):821-1829 2018.
Li, D., Mukhopadhyay, M., and Dunson, D. B. Efficient manifold approximation with spherelets. Journal of the Royal Statistical Sociely: Series B (Statistical Methodol- ogy), (M44):119-111 2022.
Park, S. Chan, W. Zhang, Y., Chiu, C., Zoph, B., Cubuk, E. D., and Le, Q. V. SpecAugment: A simple data augmentation method for automatic speech recognition. In 20th Annual Conference of the International Speech Communication Association, Interspeech, pp. 2613-2617. ISCA, 2019.
Li, X., Fang, Z., Zhang, Y., Ma, N., Bu, J., Han, B., and Wang, H. Characterizing submanifold region for out-of- distribution detection. IEEE Transactions on Knowledge and Data Engineering. 2024.
Li, Y. aurvature-aware manifold learning. Pattern Recogni- fion, 83:273-286, 2018.
Redmon, J., Divvala, S. K., Girshick, R. B., and Farhadi, A. You only look once: Unified, real-time object detec- tion. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 779-788, 2016.
Lim, S., Kim, I., Kim, T., Kim, C., and Kim, S. Fast Au- toAugment. Advances in neural information processing systems, 32, 2019.
Redmond, M. Communities and Crime. UCI Machine Learning Repository, 2009.
Lim, S. H., Erichson, N. B., Utrera, F., Xu, W., and Ma- honey, M. W. Noisy feature mixup. In The Tenth Inter- national Conference on Learning Representations, ICLR, 2022.
Roweis, S. and Saul, L. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323- 2326, 2000.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. CVPR, 2015.
Schneider, N., Goshtasbpour, S., and Perez-Cruz, F. An- chor data augmentation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Ma, X., Wang, Y., Houle, M. E., Zhou, S., Erfani, S., Xia, S., Wijewickrema, S., and Bailey, J. Dimensionality-drive. learning with noisy labels. In International Conference on Machine Learning, pp. 3355 3364. PMLR, 2018
Shorten, and Khoshgoftaar, M. survey on image data augmentation for deep learning. Journal of big data, 6(1): 6(1):1-48, 2019.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fid- jeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, King, H., Kumaran, D.,, Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. nature, 518:5188888 318:52995533 2015.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In 3rd Inter- national Conference Learning Representations, ICLR, 2015.
Naiman, I., Berman, N., Pemper, I., Arbiv, I., Fadlon, G., and Azencot, O. Utilizing image transforms and diffusion models for generative modeling of short and long time series. In Advances in Neural Information Processing Systems 38: NeurlPS, 2024a.
Singer, A. and Wu, H.-T. Vector diffusion maps and the con- nection laplacian. Communications pure and applied mathematics, 6568) 2012.
Sahl-Dickstten, J., Weiss, E., Maheswaranathan. N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265. pmlr, 2015.
Naiman, I., Erichson, N. B., Ren, P., Mahoney, M. W., and Azencot, O. Generative modeling of regular and irregular time series data via Koopman VAES. In The Twelfth International Conference on Learning Representations, ICLR, 2024b.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
Nam, H. and Han, B. Learning multi-domain convolutional neural networks for visual tracking. In CVPR, 2016.
Nochumsohn, L. and Azencot, O. Data augmentation policy search for long-term forecasting. Trans. Mach. Learn. Res., 2025.
Tenenbaum, J., de Silva, V., and Langford, J. global geo- metric framework for nonlinear dimensionality reduction. Science, 290(5500) 2000.
12

---

Curvature Enhanced Data Augmentation for Regression
Vapnik, V. Principles of risk minimization for learning the- ory. Advances in neural information processing systems, 4, 1991.
conference on computer vision and pattern recognition, pp. 2743-2751, 2018.
Verma, V., Lamb, A., Beckham, C., Najañi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio. Y. Manifold mixup: Better representations by interpolating hidden states. In Interna- tional Conference on Machine Learning (ICML), 2019.
Vinyals, O., Toshev, A., Bengio, S., and Erhan, Show and tell: neural image caption generator. In CVPR, 2015.
Weinberger, K. and Saul, L. Unsupervised learning of image manifolds by semidefinite programming. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pp. 988-995, 2004.
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 0609.080111100 011990084444 144, 2016.
Yang, S., Xiao, W., Zhang, M., Guo, S., Zhao, J., and Shen, F. Image data augmentation for deep learning: survey. 2022.
arXiv preprint arXiv:2204.088
Yao, H., Wang. Y., Zhang, L.,, Zou, J., and Finn, C. C-mixup: Improving generalization in regression. In Proceeding of the Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong clas- sifiers with localizable features. In Proceedings of the IEEF/CVF international conference computer vision, pp. 6023 6023-6032, 2019.
Zhang, H., Cissé, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. 6th Inter- national Conference on Learning Representations, ICLR, 2018.
Zhang, Z. and Zha, H. Principal manifolds and nonlinear dimension reduction via local tangent space alignment. SIAM Journal on Scientific Computing. 26(1):313-33 2005.
Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Ran- dom erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001-13008. 2020.
Zhu, W., Qiu, Q. Huang, J., Calderbank, R., Sapiro, G., and Daubechies, I. LDMNet: Low dimensional manifold regularized neural networks, Proceedings of the IEEE
13

---

Curvature Enhanced Data Augmentation for Regression
A. Curvature-aware manifold learning
Given a train set = {zt,.... zx J. we parameterize the data manifold around point 2 E that we map to u =
resulting in the following truncated Maclaurin series for a nearby point uy:
1 g ("())= 22 u + O(1s))), a= a=l...., d.
(5)
In order to estimate the gradient and Hessian of the embedding mapping 9", we build set of linear equations that solves Eq. 5. Particularly, approximate g" by solving the system G = WXX, where X holds the unknown elements of the gradients Vg and the Hessians H for every a. We define gd = 0 (w1), g (ux))* E TR., where nj are points in the neighborhood of := fu), and G = -.' ..g..-----p _. The point and points {u,} } are associated with the train set Z under a natural orthogonal transformation. The local natural orthogonal coordinates are a set of coordinates that are defined specific point u of the manifold. They are constructed by finding a basis for the tangent space and normal space at a point a by applying principal component analysis on the neighborhood N, = {z, J== Namely, the first d coordinates (associated with the most significant modes, i.e., largest singular values) represent the tangent space, and the rest represent the normal space. Then, we define L
V 1.....
Mx stacking v in rows, where , is given via
, / u (u))² ....
(uf) 2 (u) x u)),
(us x u8}]
and
X Vgal Vgad H .... Hd,d H Had-l,d?
with = _x2,... x DDddj The set of linear equations
G V rwhere G E ( and vp e (x ddd+))
(6)
is solved by using the least square estimation given X VfG. In practice, estimate only the upper triangular part of Ha since it is symmetric matrix. We refer the reader for a more comprehensive and detailed treatment in (Li, 2018).
A.1. Approximation Error Bounds
In what follows, provide theoretical justification for the sampling error of CEMS in comparison to first-order approaches. Let f Rd >> D be a twice-differentiable function, we can express the Taylor expansion around point up to first and second order as follows,
(w)) = f(uo) V (u - up)
(2) = f(uo) (f(oo) (u up)
2 (u- uo) H (uo) - uo)
(8)
Under standard smoothness assumptions, the approximation errors can be bounded follows:
Theorem A.1 (Error Bounds). (Fowkes et al., 2013) For a twice- difceedifereettibbee function f with Lipschitz continuous Hessian neighborhood of ug, we have that
M
Ilf(w) - (w))
<
uu - uol?
(9)
2
L
11f(v) - (1)) ||u uo 1ol
(10)
6
where M bounds the spectral norm of (u) and L is the Lipschitz constant of Hu)u)) in the neighborhood of un-
The second-orded error decreases as O(Iu - =) compared to O(I - uoll?) - convergence rate ensures more accurate sampling in the vicinity of training points.
for first-order methods. This faster
14

---

Curvature Enhanced Data Augmentation for Regression
B. Intrinsic Dimension
Many techniques have been proposed for estimating a dataset's intrinsic dimension (ID). Global linear methods like principal component analysis (PCA) provide simple baseline: one examines the eigenvalue spectrum to find the number of principal components needed to explain most variance. This works well if the data lie near a linear subspace, but it can overestimate ID for nonlinear manifolds noisy data. In contrast, fractal geometric approaches such as the correlation dimension (Grassberger & Procaccia, 1983) measure how the number of point pairs grows with distance. The classic Grassberger-Procaceee algorithm computes the count of pairs within a radius r and uses the slope of the log-log plot of this count versus as the estimated dimension. While the correlation dimension can capture non-integer (fractal) structure, it typically requires large samples and is sensitive to noise, limiting its robustness in practice. Overall, these early global methods can struggle with high curvature or sparse, high-dimensional data.
Local neighbor statistics have inspired more robust ID estimators. A prominent example is the maximum likelihood estimator (MLE) (Levina& & Bickel, 2004), which uses distances to each point's k-nearest neighbors. By modeling the distribution of neighbor distances (under a Poisson process assumption), this method finds the ID that maximizes the likelihood of the observed spacing of points. The MLE approach showed improved accuracy over earlier techniques and is relatively robust across different data distributions. However, it requires choosing the neighborhood size k and can be biased if k: is too small or large. Building on this idea, Facco et al. introduced the Two-Ncarest Neighbors (TwoNN) estimator (Facco et al., 2017), which uses only the first and second nearest-neighbor distances for each point. TwoNN analytically derives an ID estimate from the ratio of these two distances, minimalistic approach that greatly reduces bias from curved manifolds or non-uniform densities. This method is computationally lightweight and has been shown to produce consistent ID estimates even when data lic on a twisted manifold or are embedded in nigh-dimensional noise, highlighting its robustness.
Recently, topologial data analysis methods, specifically persistent homology, have emerged as robust tools for ID estimation. Birdal et al. (Birdal et al., 2021) introduced a persistent nomology-based estimator that leverages stable topological features, such as connected components and loops, which persist across multiple scales. The intuition behind persistent homology is that true geometric and topological features of a dataset tend to persist as one examines data at varying resolutions, while noise-induced features disappear quickly. This persistence-bant approach provides reliable dimension estimates even in highly noisy, nonlinear, or sparse settings.
In summary, contemporary ID estimation techniques such as TwoNN, persistent homology-based measures, and MLE represent the state of the art, offering significantly improved robustness to noise, curvature, and sampling heterogeneity compared to classical PCA or correlation dimension methods.
C. Batch Selection
In our framework, for any data point (z. v) E X' x V, we seek to generate training samples that lie near the manifold M, which represents the true data distribution P. Since directly sampling from M is impossible without knowing its structure, we instead approximate it locally using the tangent plane T(x, y) at point (z, y).
To construct this tangent plane approximation, we require a neighborhood set N₂ around z. We compute the tangent plane using Singular Value Decomposition (SVD) on these neighboring points. The accuracy of this approximation heavily depends on how close the points in N₂ are to 2.
To implement this approach, we explored three distinct batch construction methods:
1. Random batch selection (random)
2. k-Nearest Neighbors batch selection (knn), where points grouped based on their Euclidean distances in Z
3. k-Nearest Neighbors Probability batch selection (knnp), where points are sampled with probabilities inversely propor- tional to their distance from the original point, ensuring closer points have higher sampling probabilities
Our hypothesis was that both proximity-based and probability-based batch selection methods would generate samples closer to the data manifold, thereby improving model performance. We first trained models using the proximity-based batch selection method and selected the best-performing model based on the validation set. Using the optimal parameters found from this model, we then trained two additional models using random batch selection and knnp methods for fair comparison. The experimental results, presented in Table 3, demonstrate the relative performance of these three approaches
15

---

Curvature Enhanced Data Augmentation for Regression
under identical parameter settings. The results reveal interesting patterns across different batch selection methods. While single method dominates across all datasets, each approach shows strengths in specific scenarios. The knnp method demonstrates superior performance on the SkillCraft dataset and matches the best performance on NO2. The knn approach excels in both RCF and DTI datasets, showing particular strength in structured data scenarios. Interestingly, random batch selection remains competitive, achieving the best performance on Airfoil and matching the best result on NO2. These results suggest that the effectiveness of batch selection methods may be dataset-dependent, highlighting the importance of considering data characteristies when choosing batch selection strategy.
C.1. Neighborhood Construction
The validity of CEMS relies on three key theoretical foundations. First, our neighborhood construction approach balances computational efficiency with geometric fidelity by sharing neighborhoods across points in close proximity. While this might appear to reduce sampling diversity, it actually preserves manifold structure because points that are close in the normalized input-output space typically share similar geometric properties. The shared neighhorhood assumption is particularly valid because we normalize both input X and output Y features to the same range, ensuring that proximity in the combined space meaningfully reflects similarity in both domains.
The reliability of our construction is maintained even in regions of high output diversity through our careful treatment of the input-output space. For a point 2; = [ Vs]. its neighborhood NV, is constructed considering distances in both input and output spaces simultaneously, naturally limiting the diversity of outputs within each neighborhood. This approach ensures that points sharing neighborhood basis have similar geometric properties, maintaining the validity of our second-order approximation.
The stochastic nature of mini-batch training provides an additional beneficial property for CEMS. As different batches are sampled each epoch, the method naturally explores varying neighborhoods and their associated tangent spaces. This dynamic sampling process enables CEMS to build a comprehensive representation of the manifold's local geometry, adapting to variations in data density across different regions. The continuous exploration of diverse local structures throughout training enhances the method's s ability to capture the full geometric complexity of the underlying manifold.
The
structure of CEMS is supported by two complementary mechanisms. First, the second-order ap- proximation naturally captures local curvature through the Hessian term, enabling accurate representation of nonlinear geometries. Second, our stochastic batch sampling strategy ensures exposure to different neighborhoods and tangent spaces throughout training. The projection of points onto the tangent space at μ (the neighborhood mean) maintains validity through the second-order terms our Taylor expansion, which account for the primary nonlincarities within each neighborhood. This approach is particularly robust because the neighborhood size adapts with the batch size, preserving accurate local approximations even in regions of high curvature.
The empirical success of this construction is demonstrated in our ablation studies (Table 5), where compare point-wise basis computation (CEMS,) with our more efficient shared neighborhood approach (CEMS). The comparable performance across multiple datasets validates our theoretical assumptions about the effectiveness of shared geometric information within local neighborhoods.
C.2. Local vs. Global Sampling
It is important to note that CEMS focuses on local sampling within neighborhoods where the second-order approximation is valid. While alternative approaches based on geodesics can enable global sampling along the manifold, they typically incur significantly higher computational costs. Our local approach strikes a balance between sampling accuracy and computational efficiency, making it particularly well-suited for data augmentation during training.
Table 3, Results for different batch selection methods CEMS.
Dataset
Airfoil $
NO2 L
SkillCraft |
RCF 4
DTI
CEMS knn
1.455
0.507
5.142
0.146
0.511
CEMS- knnp
1.441
0.506
4,941
0.173
0.509
CEMS- random
1.435
0.506
5.155
0.162
0.491
16

---

Curvature Enhanced Data Augmentation for Regression
The theoretical guarantees provided by Theorem I hold within the neighborhood where our Taylor approximations are valid. This aligns with the manifold hypothesis, which posits that real data typically lies on or near lower-dimensional manifold with locally Euclidean structure (Goodfellow, 2016; Belkin & Niyogi, 2003). By focusing accurate local sampling, CEMS can effectively augment the training data while maintaining the essential geometric structure of the underlying manifold.
D. Computational cost
General analysis of n-th order embedding maps: To estimate the n-th order Taylor approximation of the embedding g. need to find all the partial derivatives up and including order n. The gradient V vector is composed of d first-order partial derivatives, the Hessian matrix is composed of d² second-order partial derivatives. Third-order and higher partial derivatives are represented using mathematical objects called tensors. In general, the number of partial derivatives of a multi-input function grows exponentially with the order.
The order n of the partial derivatives determines the number of unknowns X and the size of the matrix Vi which used to solve Eq. 6 via least squares. The number of columns in the matrix is Σi= di Odd"), and thus, the dimensions of the matrix V are k x d", It is preferred to solve an over-determined set of equations such that the number of neighbors k d". which is memory and computationally expensive. For small values of d and it is feasible to solve Eq. 6 in time complexity of Okkaa)) ). For larger values of n, it becomes extremely computationally and memory expensive to achieve k 2 d", therefore, we can assume that k < d".
There are two computationally expensive operations used to estimate the n-th order approximation of the manifold, SVD and least squares. The matrix on which we perform SVD is of shape N₂ E R where represents the number of neighbors and D is the extrinsic dimension. Thus, the run time complexity of the SVD operation per batch is (mmin(k²D) D, DD*)). The complexity of least squares is determined by the dimensions of the matrix V R=xx((d)) resulting in ((kdP)). If we wish to solve over-determined system of equations, we need to set k > d" e.g., k = 2dd resulting in run time of O O D. d D²)) for SVD and Odd Oddm)) for least squares.
Computational analysis of CEMS. Given a batch of data A E RAxDD where b is the batch size and D is the ambient dimension, our analysis considers the point-wise and batch-wise settings.
In the point-wise case, we construct a matrix No E R neighbors, where the number of neighbors is fixed as the batch size. This practical choice decouples the computational complexity from the intrinsic dimension d. On each matrix Na. we perform SVD the complexity of O Db*)). We then solve the set of b equations with | = d x (d + 1)/2 variables (representing the unknowns in the gradient V and Hessian H) at complexity of (b x 12) = (b x d²t). In practice, the batch size is small and constant which leads to an underdetermined system that can be solved used Bidgr-Regreesse
D for every sample a E IR in the batch, containing its b closest
at a complexity of O(b² x 1) O x dd). The total complexity per point is therefore O(min(bD² D)2) + Ob2 x d²). Since the batch size b is usually smaller than the ambient dimension D, the total complexity can be revised as ( + d))). Under the manifold hypothesis, we assume that d << D and thus d E O(D²), resulting in the following complexity O D) for a single point and O D) for the entire batch. Our analysis reveals that under our assumptions, the complexity is proportional the ambient dimension D.
In the batch-wise setting, the entire batch A RbxD; is processed collectively. We compute the SVD of A at complexity of (mmin(DD2,DDb
The subsequent step involves solving b equations with / = d x (d + 1)/2 variables at a complexity of O(b x ()). As in the point-wise case The total computational complexity of CEMS for the batch-wise setting for single batch is (122 D)
Run time comparison In Table 4, we compare the total run time of the training process in seconds to provide an estimate for the empirical computational cost of CEMS and competing methods. The results are obtained with a single RTX3090 GPU. For each data set, all the methods were estimated using the same parameters (e.g., batch size, number of epochs) for fair comparison. It is evident from the results that the empirical run time of CEMS is o par with competing methods and does not require large overhead.
E. Adaptation to batches
Below we provide the algorithm for the batched version as described in Sec,
17

---

Curvature Enhanced Data Augmentation for Regression
Table 4. Training times comparison (in seconds).
AiRFOn.
NO2
RCF
DTT
ERM
3.84
1.01
172
653
C-MIXUP
11.64
2.04
1700
1064
ADA
8.72
3.22
465
3519
FOMA
7.11
1.85
364
1095
CEMS
12.2
3.04
445
1317
Algorithm 2 CEMS: Batch generation
Require: Training data = {2' [x'', 1c'.y'Il y'
Require: A sample 2 E Z. batch size B. Set = B
Find K-nearest neighbors N₂ < (z,
U 20) of
2: Find orthonormal basis Bu, that spans N₂- JNN.
3: Project every 2, MNN to the local orthonormal coordinates:
4: "j f BT, (zz JN.). 9j + BN. (2) MN.)
5: For each 1 = 1,...,k construct:
6: U. < {uj- - " and G²2 + {ag 1
7:
Construct G and v as in Eq. 6
8:
Solve V G
9:
Extract Vg(2) and (2) from
10:
Sample point near ur: 7 (Wu, Ta)
11:
Calculate g(n)) < g(w) + (ıR - u)) " Vg y + (1 u) H(n - w))
12:
Un-project back to the original coordinates:
Zqp f() B, [m.g(n)] μN.
Return Zn (2m
F. Application in Data Space vs. Latent Space
Our method can be applied in either the data space (i.e., raw input features) or a learned latent space (e.g., the output of a hidden layer a neural network). Each setting presents different trade-offs and use cases:
Data Space: Applying CEMS directly in the input space offers interpretability and preserves direct relationships between augmented samples and original features. This approach is model-agnostic and particularly suitable when the input space has meaningful geometric structure (e.g., tabular data). However, nigh-dimensional or noisy input spaces may not exhibit a well-defined manifold, limiting the effectiveness of local approximations.
In contrast, applying CEMS learned latent space-typically a lower-dimensional and semantically structured representation- lead to more compact and smoother manifolds. Latent representations often disentangle underlying factors of variation, making them more amenable to local geometrie modeling. Moreover, when CEMS is implemented as differentiable module in latent space, gradient signals can flow through the augmentation step. This enables the network to shape the latent space with respect to the augmented samples during training, potentially improving the alignment between the learned representation and the underlying manifold geometry.
Latent Space:
While latent space augmentation benefits from smoother geometry and end-to-end optimization, it relies on the quality of the learned representations and is tied to specific model architecture. Data space augmentation, the other hand, is more general and interpretable but may struggle in high-dimensional or non-smooth input spaces.
Trade-offs:
In our experiments, we found both variants to be viable, and the choice between them depends on the dataset characteristics and model architecture. We view deeper investigation of this trade-off as important direction for future work.
18

---

Curvature Enhanced Data Augmentation for Regression
Table Ablation results for estimating the basis per point in the neighborhood (CEMS,) vs. estimating it once and re-using the hasis for every point N, (CEMS).
Airfoil
NO2
Crimes (RMSE)
SkillCraft (RMSE)
RMSE
MAPE
RMSE
MAPE
Avg. +
Worst +
Avg.
Worst +
CEMS, CEMS
1.462
0.783
0.503
12.759
0.130
0.157
5.026
8.063
1.455
0.809
0.507
12.807
0.128
0.159
5.142
6.322
G. Ablation Study
G.1. Ablation: basis computation per point vs. per neighborhood
As mentioned in Sec. 4, given a data point 2, we construct a neighborhood N, which can be used to 1) sample a point 5 near 2 2) sample points N₂ near Ng. The first option requires estimating a basis for the tangent space for each point in the dataset, whereas the second option estimates single basis for the the entire batch of points Ng. In practice, the first option requires significantly more SVD calculations, determined by the batch size b. For large datasets, using the first option becomes very time consuming. In Tab. 5, we compare between Option 1 (CEMS,), where p stands for point and Option 2 (CEMS), considering specifically the smaller datasets. Based on these results, we find that CEMS, achieves error measures similar to CEMS. However, the computational complexity of CEMS, is much higher, and thus we advocate the batch-wise computation as suggested in CEMS.
G.2. Ablation: Intrinsic dimension sensitivity
To evaluate the sensitivity of our method to intrinsic dimension estimation, we conduct a systematic analysis by perturbing the estimated values obtained from the TwoNN estimator. Table 6 presents results for in-distribution generalization (Airfoil and NO2) and out-of-distribution generalization (Crimes and SkillCraft), where the parameter id denotes the baseline dimension estimated by TwoNN, and i 1 id#2 correspond to configurations where the dimension is manually adjusted up down. We find that performance is relatively stable under moderate changes in the intrinsic dimension. In many cases, the baseline value (id) yields the best second-best results. For example, in the NO2 dataset, the lowest RMSE is achieved exactly at the TwoNN-estimated dimension (i.), and nearby offsets produce slightly worse but comparable performance. Notably, in this dataset, the results for id+1 and i 1d+2 are identical because the intrinsic dimension is capped at the extrinsic dimension minus one (i.e., 7), given the total number of features is 8. Similar robustness patterns are observed across other datasets. For instance, SkillCraft achieves the best average RMSE at 1,+1, while Crimes shows minimal variation across offsets, with the best worst-case RMSE appearing at d--2 and d++2. Additionally, Table 6 (bottom) compares intrinsic dimension estimates produced by different estimators (TwoNN, PH (Birdal al., 2021), MLE (Levina & Bickel, 2004)). The strong agreement among estimators e.g., all three return 10 for Cr Crimes supports the reliability of TwoNN and validates its use as our default choice. Overall, these findings suggest that our method is not overly sensitive to the intrinsic dimension, and that TwoNN provides stable and empirically effective estimate.
Table 6. Comparison of intrinsic dimension estimates across different estimators: TwoNN, PH, and MLE, The TwoNN values are used in our method as the baseline intrinsic dimensions. Consistency between methods (e.g., all return 10 for Crimes) supports the robustness the chosen estimates.
Airfoil
NO2
Crimes
SkillCraft
TwoNN
3
6
10
12
PH
3
10
11
MLE
3
4
10
11
G.3. Ablation: Noise sensitivity
To assess the robustness of our method to the choice of the noise scale parameter a, we conducted a sensitivity analysis by varying a across a wide range of values, from a/10 to 10- o. Table 7 summarizes the performance across four datasets in terms of average RMSE and MAPE (or worst-case RMSE) over three seeds. We observe that performance remains stable
19

---

Curvature Enhanced Data Augmentation for Regression
Table Sensitivity analysis of performance to changes in the estimated intrinsic dimension. The row i corresponds the baseline intrinsic dimension estimated by the TWONN method. Rows id + and id + 2 represent perturbations of the estimated dimension. Results show average RMSE and MAPE across three seeds. Bold and underlined values indicate best and second-bext results per column, respectively.
Airfoil
NO2
Crimes
SkillCraft
RMSE
MAPE
RMSE
MАРE
Avg.
Worst
Avg.
Worst
id -2
1.466
0.818
0.510
12.688
0.129
0.158
5.183
6.541
id 11
1.444
0.810
0.511
12.762
0.128
0.161
5.211
6.705
id
1.454
0.809
0.507
12.807
0.128
0.159
5.142
6.322
id +1
1.473
0.820
0.523
13.375
0.133
0.172
5.070
6.192
id +2
1.433
0.798
0.523
13.375
0.129
0.158
5.196
6.599
Table Sensitivity of performance variations in the parameter. Each row corresponds different scaling factor applied to the default a. Results show average RMSE and MAPE across three seeds. Bold and underlined values indicate the best and second- bestt results per column, respectively.
Airfoil
NO2
Crimes
SkillCraft
RMSE
MAPE
RMSE
MAPE
Avg.
Worst
Avg.
Worst
/ 10
1,534
0.843
0.521
13.280
0.133
0.176
5.640
8.730
/5
1.510
0.836
0.515
13.049
0.132
0.174
5.367
8.034
a 1
1,518
0.865
0.513
12.927
0.128
0.156
5.297
6.948
1.454
0.809
0.507
12.807
0.128
0.159
5.142
6.322
0 2
1.526
0.853
0.507
12.824
0.133
0.159
5.173
6.244
5
1,494
0.831
0.520
12.961
0.137
0.164
5,849
7.613
10
1.494
0.826
0.524
12.953
0.138
0.168
5.971
7.370
within a reasonable range around the default value o, with minimal degradation at extreme low or high values. Notably, the best performance on NO2 is achieved exactly at the default while SkillCraft achieves optimal or near-optimal results at both a and 2 a. The variation in metrics is relatively small across the range, indicating that the method is not overly sensitive to the exact value of a. These findings suggest that our approach is robust to noise scaling and does not require fine-graiee fine-grained tuning of this parameter.
G.4. Ablation: Batch sensitivity (neighborhood)
To assess the robustness of our method to the neighborhood size parameter B, which determines the size of the local batch used in tangent space estimation, we conduct sensitivity analysis by scaling B across wide range-from from B/4 to 4B. As shown in Table 9, our method demonstrates stable performance across all datasets for both in-distribution (Airfoil, NO2) and out-of-distribbution (Crimes, SkillCraft) settings. The default value B achieves the best second- best results in most metrics, notably yielding the best RMSE and worst-case RMSE in SkillCraft, and the lowest RMSE in NO2. Interestingly, while smaller neighborhoods (e.g., B/2 or B/4) occasionally improve MAPE in Airfoil or average RMSE in Crimes, overly small or large values (e.g., B/4 4 or 4B) tend to degrade performance in other metrics. These results indicate that the method is not overly sensitive to the exact batch size, and that the default setting strikes a good balance between local geometric fidelity and stability. Overall, this analysis provides empirical support for the robustness of our hyperparameter selection strategy and clarifies the trade-offs involved in adjusting B, thereby addressing concerns regarding parameter sensitivity and general applicability of the method.
H. Geometric Properties Effect on CEMS
To evaluate the impact of curvature on CEMS for regression tasks, we generated a synthetic dataset where features lie on a manifold of constant scalar curvature. The manifold is defined as a hypersphere with a constant scalar curvature. Data points are sampled uniformly on the hypersphere using normalized random directions and embedded into higher-dimensional
20

---

Curvature Enhanced Data Augmentation for Regression
Table Sensitivity analysis of performance the batch size B. Each row corresponds different scale of the batch size. Results show average RMSE and MAPE (or worst RMSE) across three seeds. Bold and underlined values indicate the best and second-bes results per column, respectively.
Airfoil
NO2
Crimes
SkillCraft
B
RMSE
MAPE
RMSE
MAPE
Avg.
Worst
Avg.
Worst
B/4
1,435
0.792
0.544
14.021
0.131
0.166
5.148
6.667
B/2
1,427
0.800
0.517
13.120
0.131
0.162
5.178
6.829
B
1,454
0.809
0.507
12.807
0.128
0.159
5.142
6.322
x 2
1.499
0.831
0.513
12.714
0.134
0.167
5.571
7.502
1.597
0.895
0,518
13.080
0.132
0.162
5.871
8.420
ambient space through a deterministic projection matrix to preserve the manifold structure. The regression target Y is computed as non-linear function of the intrinsic coordinates, Y sin (E Xinrinsie), introducing a smooth dependency on the features. The features and targets are normalized to lie within the range [0, 1] using min-max scaling. This setup enables the systematic study of the effects of curvature of CEMS on regression model performance. In Fig. the graph illustrate the relative improvement in RMSE between the CEMS and baseline ERM for regression tasks across varying scalar curvatures of the data manifold. The graph plots relative improvement against scalar curvature, highlighting that CEMS provides minimal advantage for nearly flat manifolds with low curvature but exhibits increasing improvement the curvature grows. At higher curvatures (e.g.. 16-64), CEMS demonstrates substantial gains, reflecting its ability to exploit geometric information in highly curved spaces. The hyperparameters of CEMS were not fine-tuned for this experiment and remained consistent across all intrinsic dimensions and curvature values. This likely explains why, in some cases, CEMS does not achieve better performance than ERM.
Hyperparameters
We present the hyperparameters for each dataset in Table 10. In our main results, we apply our method to the input space or the latent space, and we report the configuration with the best performance. All hyperparameters were selected through cross-validation and evaluated on the validation set. Some hyperparameters, such as architecture and optimizer, are not included in tables since they remained unchanged and were used as specified in previous works (Yao et al., 2022: Schneider al., 2023).
0.20
Relative Improvement
0.15
0.10
0.05
0.00
0
10
20
30
40
50
60
Curvature
Figure 3. Illustration of the relative improvement in RMSE of CEMS over ERM. The graph demonstrates the effect of curvature, indicating minimal gains for nearly flat manifolds but substantial improvements for highly curved manifolds. These results emphasize the influence of data geometry on the performance of CEMS relative to ERM.
21

---

Curvature Enhanced Data Augmentation for Regression
Table 10. Hyperparameter choices for the experiments using CEMS.
Dataset
Airfoil
NO2
Exchange-Rate Electricity
RCF
Crimes
SkillCraft
PovertyMap
DTI
Learning rate
le-3
le-3
1e-3
5e-4
le-1
le-3
le-3
5e-3
le-1
Batch size
16
32
32
32
64
16
32
32
Input/Manifold
manifold
input
input
manifold
manifold
manifold
input
input
input
Epochs
700
100
200
100
50
100
100
50
60
le
0.2
0.1
1e-3
0.01
0.3
0.2
0.1
1e-3
J. Dataset Description
This section provides detailed descriptions of the datasets used in our experiments.
Airfoil Self-Noise (Brooks et al., 2014). This dataset contains aerodynamic and acoustic test data for various NACA 0012 airfoils, recorded at different wind tunnel speeds and angles of attack. Each data point includes five features: frequency, angle of attack, chord length, free-stream velocity, and suction side displacement thickness, with the label representing the scaled sound pressure level. Input features are normalized using min-max normalization. The dataset is divided into 1003 training examples, 300 validation examples, and 200 test examples as noted in (Hwang & Whang, 2021).
NO2 (Aldrin, 2004). The NO2 dataset examines the relationship between air pollution near a road and traffic volume along with meteorological variables. Each input consists of seven features: the logarithm of the number of cars per hour, temperature at 2 meters above ground, wind speed, temperature difference between 25 and 2 meters above ground, wind direction, hour of the day, and the day number since October 2001. The response variable is the hourly logarithm of NO2 concentration measured in Oslo from October 2001 to August 2003. The dataset is split into 200 training examples, 200 validation examples, and 100 test examples as in (Hwang & Whang, 2021).
Exchange- (Lai et al., 2018). This time series dataset includes daily exchange rates for eight countries (Australia, Britain, Canada, Switzerland, China, Japan, New Zealand, and Singapore) from 1990 to 2016, totaling 7,588 observations with daily frequency. A sliding window size of 168 days is applied, resulting in an input dimension of 168 x 8 and label dimension of 8 data points. The dataset is partitioned into training (60%), validation (20%), and test (20%) sets in chronological order as described in (Lai et al., 2018).
Electricity (Lai et al., 2018). This dataset contains hourly electricity consumption data from 321 clients, recorded every 15 minutes from 2012 to 2014, totaling 26,304 observations. A sliding window size of 168 is used, resulting in an input dimension of 168 321 and label dimension of x 321. The dataset is divided into training, validation, and test sets following methodology similar to that used for the Exchange-Rate dataset.
Sosatedd-Cooorr-Fashhiiii dataset features images with specific color and rotation attributes. Images are colored using RGB vectors based the rotation angle g E [0, 1). In the training set, 80% of images are colored with [g, 0, 1 g). and 20% with [1 g, 0, g). creating spurious correlation between color and label.
RCF (Yao et al., 2022). The RCF-MNIST
Poverty (Koh et al., 2021). Part of the WILDS benchmark (Koh et al., 2021). this dataset consists of satellite images from 23 African countries used to predict village-level asset wealth. Each input is 224 x 22 multispectral LandSat image with channels, and the label is the real-valued asset wealth index. The dataset is divided into 5 cross-validation folds with disjoint countries to facilitate the distriibutton setting, following the methodology in (Koh et al., 2021).
Crime (Redmond, 2009). The Communities And Crimes dataset merges socio-economic data from the 1990 US Census, law enforcement data from the 1990 US LEMAS survey, and crime data from the 1995 FBI UCR. It includes 122 attributes related to crime, such as median family income and percentage of officers in drug units. The target is the per capita violent crime rate. Numeric features are normalized to range of 0.00 to 1.00, and missing values are imputed. The dataset is divided into training (1,390), validation (231), and test (373) sets, with 31, 6, and disjoint domains, respectively.
SkillCraft (Blair et al., 2013). The SkillCraft dataset from UCI consists of video game telemetry data from real-timo strategy (RTS) games, focusing on player expertise development. Each input includes 17 player-related parameters, such as
22

---

Curvature Enhanced Data Augmentation for Regression
cognition-action-cyyle variables and hotkey usage, while the label is the action lateney. Missing data filled by mean padding. The dataset is divided into training (1,878), validation (806), and test (711) sets with 4, 1, and 3 disjoint domains, respectively.
DTI (Huang et al., 2021). The Drug-Targe Interactions dataset aims to predict the binding activity score between small molecules and target proteins. Input features include one-ho hot vectors for drugs and target proteins, and the output is the binding activity seore. Training and validation data are from 2013 to 2018, while the test data spans 2019 to 2020. The "Year" attribute serves as domain information.
K. Results with Standard Deviation
In Table 1I we report the full results of in-distribution generalization and in Table 12 report the full results of out-of- distribution robustness.
Table 11. Full results for in- in-distribution generalization. Standard deviations calculated seeds.
Airfoil
NO2
RMSE
MAPE
RMSE
MAPE
ERM
2.901 = 0.067
1.753 + 0.078
0.537 + 0.005
13.615 + 0.165
Mixup
3.730 = 0.190
2.327 = 0.159
0.528 0.005
13.534 0.125
Mani Mixup
3.063 0.113
1.842 + 0.114
0.522 0.008
13.357 0.214
C-Mixup
2.717 0.067
1.610 + 0.085
0.509 0.006
12.998 0.271
ADA
2.360 0.133
1.373 = 0.056
0.515 + 0.007
13.128 = 0.147
FOMA
1.471 0.047
0.816 + 0.008
0.512 + 0.008
12.894 = 0.217
CEMS
1.455 0.119
0.809 + 0.050
0.507 0.003
12.807
0.044
Exchange-Rai
Electricity
RMSE
MAPE
RMSE
MAPE
ERM
0.023 0.003
2.423 +: 0.365
0.058 + 0.001
13.861 + 0.152
Mixup
0.023 0.002
2.441 + 0.286
0.058 + 0.000
14.306 + 0.048
Mani Mixup
0,024
0.004
2.475 + 0.346
0.058 + 0.000
14.556 + 0.057
C-Mixup
0.020 = 0.001
2.041 = 0.134
0.057 0.001
13.372 = 0.106
ADA
0.021 0.006
2.116 + 0.689
0.059
0.001
13.464
0.296
FOMA
0.013 0.000
1.262 + 0.037
0.058 0.000
14.653
0.166
CEMS
0.014 = 0.001
1.269 + 0.062
0.058 + 0.000
13.353 + 0.217
23

---

Curvature Enhanced Data Augmentation for Regression
Table 12. Full results for out-of-distribution robustness. Standard deviations derived from 5-fold data split in Poverty and calculated over seeds for other datasets.
RCF (RMSE)
Crimes (RMSE)
SkillCraft (RMSE)
Avg- 4
Avg.
Worst +
Avg.
Worst
ERM
0.164 + 0.007
0.136 + 0.006
0.170 + 0.007
6.147 + 0.407
7.906 + 0.322
Mixup
0.159 + 0.005
0.134 + 0.003
0.168 + 0.017
6,461 = 0,426
9.834 + 0.942
ManiMixup
0.157 = 0.021
0.128 + 0.003
0.155 = 0.009
5.908 = 0.344
9.264 1.012
C-Mixup
0.146 0.005
0.123 + 0.000
0.146 0.002
5.201 + 0.059
7.362
0.244
ADA
0.163 0.014
0.130 + 0.003
0.156 0.007
5.301 = 0.182
6.877
1.267
FOMA
0.159 + 0.010
0.128 + 0.004
0.158 0.002
CEMS
0.146 + 0.002
0.128 + 0.001
0.159 + 0.004
5.142 + 0.143
6.322
0.191
DTI (R)
Poverty (R)
Avg.
Worst 1
Avg. f
Worst t
ERM
0.483 + 0.008
0.439 0.016
0.80
0.04
0.50 0.07
Mixup
0.459 + 0.013
0.424 0.003
0.81 0.04
0.46 + 0.03
ManiMixup
0.474 + 0.004
0.431 + 0.009
C-Mixup
0.498 + 0.008
0.458 = 0.004
0.81 + 0.03
0.53 + 0.07
ADA
0.493 + 0.010
0.448 + 0.009
0.79 + 0.03
0.52 + 0.06
FOMA
0.503 + 0.008
0.459 + 0.010
0.77 + 0.03
0.49 + 0.05
CEMS
5.110 +: 0.005
0.465 0.004
0.81 0.05
0.50 + 0.07
24


Reviews:
[{"review_id": "MwQAK3ITSN", "reviewer": "Reviewer_JBBQ", "summary": "Presents a data-augmentation method for regression problems, taking advantage of the manifold structure of the data. Defined on the concatenation of data and labels, the local neighbourhood of all points defines the assumed manifold structure, and the two first moments (mean and Hessian) are used to sample points for data-augmentation. The resulting method is evaluated on several datasets and performs favourably against SOTA.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and encouraging feedback. We appreciate the recognition of our method’s integration of manifold learning and regression, as well as the clarity of our experiments and supplementary material. Below, we address the comments and describe the corresponding revisions.\n\n[Additional tables](https://jmp.sh/GDqh987h)\n\n1. **The method is well-justified when the Hessian is evaluated around each point...**  \n   Thank you for this observation. In our method, the Hessian itself is not shared. Instead, we reuse the local neighborhood to compute a shared orthonormal basis, which each point then uses to estimate its Hessian independently.\n\n   Specifically:\n   - **Shared Neighborhood:**  \n     A *k*-nearest neighbor set is constructed for each mini-batch point and reused for nearby points, based on the manifold hypothesis assuming local smoothness.\n   - **Shared Local Basis:**  \n     A basis for the tangent and normal spaces is computed via Singular Value Decomposition (SVD) from the shared neighborhood.\n   - **Point-wise Hessian Estimation:**  \n     Each point solves a linear system in the shared basis to obtain its own Hessian. Thus, only the basis is shared; curvature information remains point-specific.\n\n   We compared this method (CEMS) with a fully point-wise variant (CEMSp) and found negligible performance differences (see Table 5), confirming that the shared-neighborhood approximation is both efficient and accurate.\n\n2. **Where did the idea of creating a manifold by concatenating data x and label y come from?**  \n   We thank the reviewer for highlighting this point. The idea of creating a manifold by concatenating input data *x* with its corresponding label *y* is not unique to our paper. Relevant prior work has been added and cited in the revised manuscript.\n\n3. **Are local neighborhoods disjoint sets or overlapping sets?**  \n   We thank the reviewer for raising this important point. The local neighborhoods we construct are overlapping sets rather than disjoint sets. Specifically, each point forms its own neighborhood based on its *k*-nearest neighbors. Therefore, it is natural that neighborhoods of close points may partially overlap.\n\n   To clarify, the Hessian itself is not computed only once per neighborhood; rather, it is computed once per point. What we re-use across overlapping neighborhoods is only the local orthonormal basis, which is computed from the SVD of points in the shared neighborhood. Once this basis is established, each individual point solves its own linear system separately to estimate its Hessian within that shared basis.\n\n   Hence, overlap between neighborhoods is not problematic in our setting. On the contrary, overlap can be seen as beneficial, as it can encourage consistent local geometry estimates across adjacent regions of the manifold, ensuring smooth transitions and coherent geometric structure.\n\n4. **Is the intrinsic dimension of the manifold an important hyperparameter?**  \n   We thank the reviewer for bringing up this important clarification. In principle, the intrinsic dimension *d* of the manifold could be treated as an important hyperparameter. However, in practice, we opt for an automatic estimation using a robust intrinsic dimension estimator TwoNN (Facco et al., 2017) to reduce the complexity of hyperparameter tuning. Thus, the intrinsic dimension is inferred from the data rather than manually set.\n\n   To address the reviewer’s concern regarding the robustness to intrinsic dimension, we performed a sensitivity analysis by perturbing the TwoNN estimated dimension by ±1 and ±2. As shown in Table 2 in the anonymous pdf file, our method exhibits stable performance across all datasets, with the best or second best results frequently aligning with the baseline value. Additionally, in Table 1 we compare TwoNN with alternative estimators [a, b] and observe strong agreement, e.g., all three methods estimate the same dimension for Crimes. These results support both the robustness of our method to this hyperparameter and the empirical reliability of using `TwoNN` as the default.\n\n   **[a]** Levina, E., & Bickel, P. (2004). Maximum likelihood estimation of intrinsic dimension. *Advances in Neural Information Processing Systems*, 17.  \n   **[b]** Birdal, T., Lou, A., Guibas, L. J., & Simsekli, U. (2021). Intrinsic dimension, persistent homology and generalization in neural networks. *Advances in Neural Information Processing Systems*, 34.", "claims_and_evidence": "The claims made in the submission seem to be supported by clear and convincing evidence. The application of manifold learning techniques to data regression seems both straight-forward but novel.", "methods_and_evaluation_criteria": "Both in-distribution and out-of-distribution problems are evaluated, and strong baselines are used.", "theoretical_claims": "No theoretical claims are made. The theoretical justifications and derivations for the regression and the resulting error bounds (appendix A1) look good to me.", "experimental_designs_or_analyses": "The experimental design is sound and exceptionally detailed in terms of practical considerations, such as batch size and intrinsic dimension estimation.", "relation_to_broader_scientific_literature": "The manuscript is candid in presenting both the regression problem literature and the manifold learning literature, and the contribution to previous studies is very clear.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "Strengths \n * beautiful synthesis of manifold learning tools and the regression problem\n * detailed discussion of practical considerations\n * great evaluation on various regression datasets, good results\n\nWeaknesses\n * The method is well-justified when the hessian is evaluated around each point, but this seems computationally expensive, so un-justified approximations are made (sharing Hessian across a local neighbourhood).", "other_comments_or_suggestions": "None.", "questions_for_authors": "* Where did the idea of creating a manifold by concatenating data x and label y come from? If it is your contribution, clarify it. If it is well-known, cite previous literature.\n * Are local neighborhoods disjoint sets or overlapping sets? As the Hessian be calculated once per each (as in \"re-using neighborhoods and basis computations\"), what happens if those overlap?\n * Is the intrinsic dimension of the manifold an important hyperparameter (\"CEMS is governed by the intrinsic dimension of the manifold\") or an inferred value which is part of the algorithm (\"While the intrinsic dimension d can be viewed as a hyper-parameter of CEMS, we estimate it in practice using a robust estimator\")? How dependant is the performance on the validity of this estimator?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NRZRsjEhFc", "reviewer": "Reviewer_PF8v", "summary": "This paper proposes a data augmentation method tailored for regression problems, leveraging the manifold hypothesis in the joint input-output space. The method approximates the data manifold up to the second order and samples new data points that adhere to this approximation. The effectiveness of the approach is demonstrated across multiple in-distribution and out-of-distribution benchmarks, where it achieves comparable or superior performance to existing state-of-the-art methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. We're glad the method’s empirical strength and relevance were recognized. We addressed the comments on hyperparameters, differentiability, and implementation through added clarifications and revisions. Below are our detailed responses.\n\n[Additional tables](https://jmp.sh/GDqh987h)\n\n1. **Essential References Not Discussed.**  \n   We added the reference and discussion to the Related Work section in the revised manuscript.\n\n2. **The paper does not provide a comprehensive discussion for key hyperparameters.**  \n   While we did not include a detailed discussion in the paper, we selected key hyperparameters via standard cross-validation, following the approach used in prior works (e.g., Yao et al., 2022; Schneider et al., 2023).\n\n3. **Sensitivity analysis is missing.**  \n   As shown in the additional tables uploaded, Our method shows robustness to hyperparameter choices. Perturbing the intrinsic dimension estimated by TwoNN yields stable performance, with the baseline or nearby values often achieving the best results. Sensitivity analyses on the neighborhood size parameter $B$ and the noise scale $\\sigma$ also show consistent performance across a broad range of values. These findings demonstrate that our method is not overly sensitive to intrinsic dimension, neighborhood size, or noise level.\n\n\n4. **Application to different data representations.**  \n   Our method is compatible with both data and latent space representations. We added a discussion in the appendix covering the trade-offs and practical considerations of each setting, including when gradient flow through the augmentation module is beneficial.\n\n5. **Differentiability and its impact.**  \n   Differentiability is relevant for latent-space augmentation, enabling backpropagation through the augmentation module and potentially improving representations. For input-space augmentation, differentiability is unnecessary.\n\n6. **Implications of ignoring gradients.**  \n   Even if latent-space augmentation is non-differentiable, it can still diversify training data. However, the lack of gradient flow may reduce the benefits of adaptively optimizing the augmentation process.\n\n7. **Lack of clarity in the batch vs. mini-batch implementation**  \n   We clarify the distinction between the two variants: `CEMS` and `CEMS_p`.\n\n- `CEMS_p` (point-wise) samples mini-batches randomly. For each point in the batch, a neighborhood of size $k$ is drawn from the full dataset to estimate a local tangent space and Hessian, producing one synthetic sample.\n- `CEMS` (batch-wise) samples a point $z_0$ and builds a batch $N_z$ of $B$ nearby points. A shared, mean-centered tangent space is computed, and each point uses it to estimate its own Hessian and generate one sample.\n\nWe have revised the manuscript to explicitly define $B$ in `CEMS_p`, clarify neighborhood construction, and highlight that both variants generate one sample per point using different geometric setups.\n\n\n8. **The dimensions of vectors and matrices should be explicitly stated in the text.**  \n   We added the dimension in the revised manuscript.\n\n9. **Inconsistencies in notation should be addressed**  \n   - **$[u, g(u)] \\rightarrow [u^{\\top}, g(u)^{T}]^{T}$?**  \n     Clarified that $[\\cdot, \\cdot]$ denotes column-wise concatenation.\n   - **Is only $Y$ normalized?**  \n     Yes. Since $X \\in [0, 1]$, we normalize $Y$ to balance the concatenation.\n   - **Why not use the Echo dataset?**  \n     Due to time constraints and its size, we did not complete evaluation on Echo. We plan to include it in future work.\n   - **Figure 2: Red arrow meaning is unclear.**  \n     We clarified that it represents un-projecting $\\eta$ from the tangent space to $\\mathbb{R}^D$ via $f$.\n   - **Transpose notation inconsistencies.**  \n     We revised the manuscript to ensure consistent use of transpose notation.\n   - **Equation: Solve $A \\Psi = G$.**  \n     The correct equation is $\\Psi A = G$. This has been corrected.\n   - **Table 2: DTI performance seems off.**  \n     The correct value is \"0.511\", now fixed in the revision.\n\n10. **Why is the linear system solved per point?**  \n    To estimate the gradient and Hessian at each point $u$, we fit a second-order Taylor expansion using neighboring points $u_j$. Since this expansion is specific to $u$, we solve a separate linear system per point, allowing us to capture local geometric variations across the manifold.\n\n11. **What types of datasets benefit most from this method, and in which scenarios might it be less effective?**  \n    Our method performs best on datasets where input-output pairs lie near a smooth, low-dimensional manifold with meaningful curvature, common in structured data like images and audio signals. It is especially effective in sparse or low-data regimes. However, it may be less effective when the manifold assumption breaks down, such as with high-dimensional noise or unstructured data.", "claims_and_evidence": "The authors substantiate their claims with extensive empirical evaluations on several real-world datasets, comparing their method against state-of-the-art techniques.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria appear appropriate for addressing data augmentation in regression. \n\nHowever, key hyperparameters such as the number of neighbors $k$, the intrinsic manifold dimensionality $d$, the choice of $\\sigma$, batch size, the choice of the space (data or latent space), would influence performance, but their robustness is not systematically analyzed. \n\nFurther discussion is needed to assess the sensitivity of the method to these hyperparameters.", "theoretical_claims": "This paper does not introduce novel theoretical contributions beyond the second-order approximation framework for data augmentation.", "experimental_designs_or_analyses": "The experimental design is mostly sound and well-structured. \n\nHowever, some aspects require further clarification, particularly regarding the robustness of hyperparameter choices and the applicability of the method in different settings (e.g., data space vs. latent space).", "relation_to_broader_scientific_literature": "Effective data augmentation techniques are essential for improving regression model performance across various applications. This work aligns with broader research in manifold learning and data augmentation.", "essential_references_not_discussed": "The Hessian eigenmaps paper **[a]** explores similar second-order manifold approximations and should be referenced in the related work section.\n\n**[a]**  Donoho, D. L., & Grimes, C. (2003). \"Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data.\" Proceedings of the National Academy of Sciences, 100(10), 5591-5596.", "other_strengths_and_weaknesses": "**Strengths:**\n- The paper introduces a novel data augmentation method rooted in manifold learning.\n- Demonstrates strong empirical performance across multiple datasets, often surpassing existing methods.\n\n**Weaknesses:**\n- Lack of clarity in hyperparameter selection:\n  - The paper does not provide a comprehensive discussion on determining optimal values for key hyperparameters.\n  - Sensitivity analysis is missing, leaving open questions about the method’s robustness to parameter choices.\n  - Application to different data representations: the method appears applicable in both data space and latent space, but no explicit discussion clarifies the trade-offs or performance implications.\n- Lack of clarity in the role of differentiability:\n  - The paper briefly mentions differentiability but does not explain its necessity or its impact on performance.\n  -  Why is differentiability necessary for the data augmentation module, and how does it impact performance? How does the performance change if gradient information is ignored and the generated data is only used for augmentation during standard training?\n- Lack of clarity in the batch vs. mini-batch implementation: \n  - The description of the batch-wise and mini-batch implementations is unclear.\n  - If using mini-batch training, is the neighborhood constructed solely from the mini-batch? What is $z_0$? \n  - Since $CEMS$ (rather than $CEMS_p$) appears to be the primary approach, a clearer explanation of $CEMS$ in the main text would be beneficial.", "other_comments_or_suggestions": "**Minor Comments:**\n- The dimensions of vectors and matrices should be explicitly stated in the text, e.g., for $B_u$ (line 199, col. 1, page 4), $x_i, y_i$ (line 168, col. 2, page 4),  $\\Psi$ and $G$ (line 271, col. 1, page 5), to name a few.\n\n- Inconsistencies in notation should be addressed:\n\n  - Line 218, col. 1, page 4: $[u, g(u)]$ -> $[u^\\top, g(u)^\\top]^\\top$?\n\n  - Line 175, col. 2, page 4: Is only $Y$ normalized?\n\n  - Line 355, col. 1, page 7: Yao et al. (2022) includes the Echo dataset, which is not used here—why?\n\n  - Figure 2: The meaning of the red arrow is unclear.\n\n  - Transpose notation is inconsistently represented as $T$ and $\\top$.\n\n**Typos:**\n\n- Algorithm 1, Line 5: Solve $\\Psi A = G$ for $A$?\n\n- Table 2: CEMS performance for DTI should be checked.", "questions_for_authors": "Please refer to the questions in **Other strengths and weaknesses** section.\n\n**Other questions:**\n1. Why is the linear system solved per point (line 258, col. 2, page 5)?\n2. What types of datasets benefit most from this method, and in which scenarios might it be less effective?\n3. How does CEMS compare when applied in data space vs. latent space, and what are the trade-offs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iBwb51CJCT", "reviewer": "Reviewer_a6ac", "summary": "This paper targets the problem of data augmentation for regression tasks where data has some intrinsic manifold structure. Specifically, the goal is to capture this manifold structure and generate new data on this manifold. Local neighborhoods are formed through nearest neighbor algorithms. The tangent space is formed by taking SVD within the locla neighborhood, and the local chart functions are quadraticly approximated, with gradient and hessian empirically estimated by linear systems. New samples are obtained by drawing from normal distributions over the tangent space and transforming back to ambient space. Simulation examples and numerical applications are conducted.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. We are glad that the core methodology and experimental evaluations were found to be clear and well-executed. We appreciate the suggestions to improve the discussion of intrinsic dimension estimation and to better situate our work within the broader landscape of manifold learning literature, particularly recent advances from the statistical perspective. In the revised manuscript, we have addressed these points through additional discussion, new references, and clarification of our experimental setup. We respond in detail to each comment below.\n\n1. **It would be great if methods for determining the intrinsic dimension of manifold can be discussed.**  \n   We thank the reviewer for the suggestion. A discussion has been added to the appendix, reviewing classical and modern intrinsic dimension (ID) estimation methods, including both statistical and geometric approaches. Due to space constraints and the inability to upload additional text, we are unable to include it here.\n\n2. **The numerical experiments all make sense. However, they all seem to have relatively low intrinsic dimensions and ambient dimensions.**  \n   While some datasets in our experiments such as Airfoil and NO2 have low ambient and intrinsic dimensions (6/3 and 8/6, respectively), others involve significantly higher dimensions. For instance, Exchange-Rate has an ambient dimension of 1352 and intrinsic dimension of 234, while Electricity reaches 54,249 ambient and 20 intrinsic dimensions. This range highlights the diversity of our benchmarks and demonstrates the scalability and robustness of our method across both low- and high-dimensional regimes, supporting the manifold hypothesis.\n\n3. **Also, for the real world applications, are the intrinsic dimensions considered known, or are they determined by some preprocessing techniques such as the elbows of cumulative singular values?**  \n   In real-world applications, the intrinsic dimension is generally not known a priori and must be estimated. In our experiments, we employ the TwoNN estimator based on the method of Facco et al. (2017), which leverages minimal neighborhood statistics to robustly estimate local intrinsic dimensionality. This approach is parameter-free and can be applied consistently across datasets. While alternative techniques such as analyzing the elbow of cumulative singular values can also be used, we chose a method that aligns well with our second-order manifold framework and scales effectively across diverse data regimes.\n\n4. **There are many latest work on manifold learning, especially from the statistical side, that utilizes more sophisticated modeling for the local charts using e.g. Gaussian processes, spherelets, etc.**  \n   We thank the reviewer for their thoughtful observation. Our approach is general and can incorporate any module for estimating the local chart. In this work, we used a simple and widely accepted method based on PCA to demonstrate the core ideas. Importantly, other, more advanced tools can be used. We have added a discussion in the Related Work section to acknowledge this flexibility and to better situate our method within the broader literature.\n\n5. **Overall the paper is well-written and clear. The method makes sense and is straightforward, but may lack novelty/originality compared to the state-of-the-art methods in this field.**  \n   We appreciate the reviewer's overall positive assessment and the opportunity to clarify the novelty of our contribution. While second-order approximations of manifolds have indeed been explored previously, our primary contributions are as follows:\n\n   - **Differentiable Second-Order Augmentation:**  \n     To our knowledge, ours is the first method explicitly designed to provide a differentiable, second-order manifold approximation tailored specifically toward data augmentation for regression tasks. Previous second-order methods typically focus on manifold embedding and dimensionality reduction rather than augmentation aimed at improved neural network generalization.\n\n   - **Practical and Efficient Implementation:**  \n     Our method introduces a practical mini-batch-based strategy for second-order local manifold approximation, significantly improving computational efficiency and scalability. This practicality is crucial for widespread adoption in neural network training contexts, distinguishing our work from more computationally demanding prior methods.\n\n   - **Empirical Demonstration of Effectiveness:**  \n     We empirically demonstrate the significant benefits of explicitly incorporating curvature information into data augmentation, showing substantial improvements over state-of-the-art methods in several regression settings.", "claims_and_evidence": "The method is straightforward and all makes sense.\n\nIt would be great if methods for determining the intrinsic dimension of manifold can be discussed.", "methods_and_evaluation_criteria": "Out-of-distribution evaluation is included, which is great.", "theoretical_claims": "No theoreical claim present.", "experimental_designs_or_analyses": "The numerical experiments all make sense. However, they all seem to have relatively low instrinsic dimensions and ambient dimensions.\n\nAlso, for the real world applications, are the intrinsic dimensions considered known, or are they determined by some preprocessing techniques such as the elbos of cumulative singular values?", "relation_to_broader_scientific_literature": "There are many latest work on manifold learning, especially from the statistical side, that utilizes more sophisticated modeling for the local charts using e.g. Gaussian processes, spherelets, etc.", "essential_references_not_discussed": "Here are a few examples mentioned above:\n\nFaigenbaum-Golovin, Shira, and David Levin. \"Manifold Reconstruction and Denoising from Scattered Data in High Dimension via a Generalization of L1-Median.\" arXiv preprint arXiv:2012.12546 (2020).\n\nDunson, David B., and Nan Wu. \"Inferring manifolds from noisy data using gaussian processes.\" arXiv preprint arXiv:2110.07478 (2021).\n\nLi, Didong, Minerva Mukhopadhyay, and David B. Dunson. \"Efficient manifold approximation with spherelets.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 84.4 (2022): 1129-1149.", "other_strengths_and_weaknesses": "Overall the paper is well-written and clear. The method makes sense and is straightforward, but may lack novelty/originality compared to the state-of-the-art methods in this field.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lE0eOWy8F7", "reviewer": "Reviewer_fTyJ", "summary": "The paper proposed a new data augmentation approach called Curvature-Enhanced Manifold Sampling (CEMS) specifically for the regression task. It moved one step further by utilizing the second-order representation instead of a first-order approximation of the data manifold. Experiments are conducted on both in-distribution and out-of-distribution scenarios. Overall, the paper is well-written and interesting. However, there seem to be several important issues remaining unsolved.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback. We’re glad the paper was found clear and the core contribution appreciated. Your comments on method design and evaluation helped improve the revised manuscript. Below, we address each point.\n\n[Additional tables](https://jmp.sh/GDqh987h)\n\n1. **The contributions (theoretical and empirical) seem incremental and limited.**\n\n   We appreciate the reviewer raising this critical point, and we welcome the opportunity to clarify the significance, context, and complexity of our contributions compared to first-order methods, such as FOMA.\n\n   - **Theoretical Significance and Motivation of Second-Order Approximation:**  \n   Although the shift from first-order (FOMA) to second-order (CEMS) approximations might appear incremental, the explicit incorporation of curvature information significantly enriches geometric modeling. First-order methods like FOMA rely solely on linear approximations (via SVD), assuming local linearity and potentially overlooking key structure in curved regions. In contrast, our method introduces differential-geometric elements (e.g., tangent spaces, normal coordinates, Hessian estimates) to better capture nonlinear local geometry.\n   \n   - **Empirical Results and Conditions for Improvement:**  \n   While second-order methods may not always outperform first-order ones, CEMS performs better than FOMA on all but one dataset in Table 1 and shows consistent gains across architectures in Table 2. Second-order methods are especially effective on data with pronounced curvature, while first-order baselines suffice in flatter regions.\n\n   - **It will be useful to show some augmented images for the image datasets.**  \n   We thank the reviewer for this suggestion. While Figure 1 (1D sine wave) offers the clearest geometric illustration, we now include visualizations from RCF-MNIST [Figure a](https://imgur.com/a/0PXdS8P) and [Figure b](https://imgur.com/a/LY5fjwn), showing original, augmented, and difference images. These confirm that CEMS introduces smooth, semantically consistent perturbations.\n\n2. **For regression, the choice of loss functions is important for real-world datasets.**\n\n   We agree. While we used RMSE to match standard benchmarks, CEMS is agnostic to the loss function. The augmentation process is independent of the training objective, and alternative losses like Huber or quantile loss could interact with the data differently. Evaluating these is a promising direction for future work.\n\n3. **The model architectures seem too simple.**\n\n   - **Model Architecture Choice:**  \n   We followed C-Mixup, ADA and FOMA to ensure fair comparisons. This isolates the contribution of the DA method itself.\n   - **Applicability to Advanced Models:**  \n   CEMS is model-agnostic and compatible with advanced models like TabTransformer or TabR. We now mention this as future work.\n   - **Is DA Always Crucial?**  \n   Geometry-aware DA is particularly valuable in low-data or noisy regimes. Gains may vary with model complexity.\n   - **Authenticity of Augmented Data:**  \n   CEMS constrains augmentation using second-order geometry, ensuring samples remain close to the true manifold.\n\n4. **Although 3 different batch selection methods are discussed...**\n\n   - In **CEMS**, $k$ is determined by the mini-batch size and not tuned separately. We added a sensitivity analysis on batch size in the appendix. Table 4 in the additional tables file shows that our method is not overly sensitive to this parameter.\n   - In **CEMS_p**, neighborhoods are constructed from the full dataset and $k$ is selected via cross-validation.\n   - Optimal $k$ varies by dataset. In sparse regions, too large a $k$ may hurt performance, so we recommend validation-based tuning.\n\n5. **For data augmentation, it is important to determine how many augmented samples to generate.**\n\n   - **Number of Augmented Samples:**  \n   We generate one sample per point for consistency with baselines, but our method supports generating more via resampling.\n   - **Noise and Sample Quality:**  \n   Augmented samples are constrained by local curvature, but not all are equally useful.\n   - **Relation to C-Mixup:**  \n   Incorporating sampling probabilities based on geometric uncertainty could improve robustness—an avenue for future work.\n\n6. **In Table 2, why is it \"5.11\" for DTI which is different from Table 3?**\n\n   Thank you for catching this. We corrected the value to be 0.511 in the revised manuscript.\n\n7. **Since the generation process is based on a single sample z and its neighborhood...**\n\n   CEMS was developed for regression, where DA methods from classification (e.g., mixup) do not translate directly due to continuous targets. We model a joint manifold over $(X, Y)$, enabling smooth label-aware augmentation. However, CEMS is not limited to regression, it can be applied to classification or segmentation by operating on $X$ only.", "claims_and_evidence": "The contributions (theoretical and empirical) seem incremental and limited - changing from first-order to second-order manifold approximations. And it is not consistently better than the first-order (see Table 1). When and why the second-order is better (or worse) than the first-order?", "methods_and_evaluation_criteria": "It will be useful to show some augmented images for the image datasets so that we can have a straightforward and visual understanding of the generation. Furthermore, it would be useful to show some generated images for some widely used benchmarking datasets (e.g., MNIST) - although they are not originally for regression. For instance, on the MNIST data, there could be (at least) two ways to construct a regression dataset: <1> transforming the digit 0, 1, 2, ..., to continuous values, and <2> predicting the number of digits in one image, similar to https://github.com/shaohua0116/MultiDigitMNIST. Since Figure 1 is only a simple 1D example, in the above ways, it is possible to show the generated images from different DA approaches on real-world data, which should be useful for understanding different DA methods.", "theoretical_claims": "For regression, the choice of loss functions is important for real-world datasets, some losses beyond RMSE, such as the Huber loss and the quantile loss, can be particularly useful. Will the choice of loss functions affect the result (theoretical or empirical)?", "experimental_designs_or_analyses": "Furthermore, the model architectures seem too simple - simply a three-layer MLP. There are many aspects that can affect the result of a regression model. Hence, is DA always crucial, even when more advanced models/losses are used (e.g., TabTransformer and TabR)? Essentially the generated data is actually fake, and it may potentially do harm to the training if the data augmentation is not authentic enough.", "relation_to_broader_scientific_literature": "The prior works are cited.", "essential_references_not_discussed": "The prior works have been cited.", "other_strengths_and_weaknesses": "For data augmentation, it is important to determine how many augmented samples to generate. From Algorithm 1, it seems a new point is generated for each sample. Since the generated samples should contain more noise w.r.t. the original data, I think it's important to discuss how many generations to include for training. Moreover, not all generations are created equal, so maybe it would be better to consider sampling probability as in C-Mixup.", "other_comments_or_suggestions": "In Table 2, why is it \"5.11\" for DTI which is different from Table 3?", "questions_for_authors": "Since the generation process is based on a single sample z and its neighborhood, I was wondering how is the method motivated specifically for the regression problem instead of a general-purpose data augmentation approach? That is to say, what specific characteristics/constraints of regression make this method only suitable for regression, not for other tasks such as classification, segmentation, and more?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submissions introduces a new data augmentation algorithm, which addresses regression problems. Making use of manifold learning on local neighbourhoods, a curvature-enhanced approach is used for the sampling procedure. Reviewer unanimously agreed upon the relevance of the paper and particularly appreciated the clear writing and the excellent experimental setup. During the rebuttal, some minor issues of the submission were discussed and could be addressed. I trust the authors to implement their proposed changes, but otherwise, this paper more than deserves acceptance.
Decision: Accept (poster)