Comfort Zone: A Vicinal Distribution for Regression Problems
Authors: Roee Weiss Lifshitz
Omri Azencot
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Learning Regularization
Data Augmentation
Regression Learning

Abstract: Domain-dependent data augmentation methods generate artificial samples using transformations suited for the underlying data domain, for example rotations on images and time warping on time series data. However, domain-independent approaches, e.g. mixup, are applicable to various data modalities, and as such they are general and versatile. While mixup-based techniques are used extensively in classification problems, their effect on regression tasks is somewhat less explored. To bridge this gap, we study the problem of domain-independent augmentation for regression, and we introduce comfort-zone: a new data-driven, domain-independent data augmentation method. Essentially, our approach samples new examples from the tangent planes of the train distribution. Augmenting data in this way aligns with the network tendency towards capturing the dominant features of its input signals. Evaluating comfort-zone on regression and time series forecasting benchmarks, we show that it improves the generalization of several neural architectures. We also find that mixup and noise injection are less effective in comparison to comfort-zone.

Under review as a conference paper at ICLR 2023
COMFORT ZONE: A VICINAL DISTRIBUTION FOR REGRESSION PROBLEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Domain- dependent data augmentation methods generate artificial samples using transformations suited for the underlying data domain, such as rotations on images and time warping on time series data. However, innnainnnddeendeent approaches, e.g. mup, are applicable to various data modalities, and as such they are general and versatile. While mi mixu-bbased techniques are used extensively classification problems, their effect regression tasks is somewhat less explored. To bridge this gap, we study the problem of domain-indeppedent augmentation for regression, and we introduce comfort- zone: new data-driven domain- independent data augmentation method. Essentially, our approach samples new examples from the tangent planes of the train distribution. Augmenting data in this way aligns with the network tendency towards capturing the dominant features of its input signals. Evaluating comfort contt-zomn zone on regression and time series forecasting benchmarks, we show that it improves the generalization of several neural architectures. We also find that mi mipup and noise injection are less effective in comparison to comfort-zons
INTRODUCTION
Classification and regression problems primarily differ in their output's domain. In classification, we have a finite set of labels, whereas in regression, the range is an infinite set of quantities
either discrete continuous (Goodfellow et al. 2016). In classical work (Devroye et al. 2013).
classification is argued to be "easier" than regression, but more generally, it is agreed by many that classification and regression problems should be treated differently (Muthukumar et al. [021]] Particularly, the differences between classification and regression are actively explored in the context of regularization. Regularizing neural networks to improve their performance on new samples has received a lot of attention in the past few years. One of the main reasons for this increased interest that most of the recent successful neural models are overparameterized. Namely, the amount of learnable parameters is significantly larger than the number of available training samples (Allen-Zhu et al.) 2019 9a b), and thus regularization is often necessary to alleviate overfitting issues. Recent studies overparameterized linear models identify conditions under which overfitting is "benign" in regression (Bartlett et al. 2020), and uncover the relationship between the choice of loss functions in classification and regression tasks (Muthukumar et al. 2021. Still, the regularization of deep neural regression networks is not well understood.
In this work, focus on common regularization approach known as Data Augmentation (DA) in which data samples are artificially generated and used during training. In general, DA techniques can be categorized into domain-dependent (DD) methods and domain-indeependen (DI) approaches. The former techniques are specific for certain data modality such as images, whereas the latter methods typically do not depend on the data modality. Numerous DD- and DI-DA approaches are available for classification tasks (Shorten & Khoshgoftaar 2019 Shorten et al.) 2021). I). and many of them consistently improve over non-augmented models. Unfortunately, DI-DA for regression problems is significantly less explored topic. Recent works on linear models study the connection between the DA policy and optimization (Hanin & Sun 202I). as well as the generalization effects of linear DA transformations (Wu al.) 2020). We contribute to this line of work by proposing and analyzing a new gomain-indeeendem data augmentation method for nonlinear deep regression, and by extensively evaluating our approach in comparison to existing baselines.
Many strong data augmentation methods were proposed in the past few years. Particularly relevant to study is the family of mi xup-baeed techniques that are commonly used in classification

---

Under review as a conference paper at ICLR 2023
applications. The original method, mixup (Zang al. 2017 produces convex combinations of training samples, promoting linear behavior for in- in-between samples. The method is domain- independent and data-agnostic, and it shown to solve the Vicinal Risk Minimization (VRM) problem instead of the usual Empirical Risk Minimization (ERM) problem. In comparison, our approach is domain-independent and data-driven, and it can also be viewed as solving a VRM problem. Through extensive evaluations, we will show that mixup and noise injection are less effective for regression.
Contribution. Challenged by the differences between classification and regression and motivated by the success of domain--nddeenddnt methods such as mi xup, we propose simple, domain- independent and data-driven DA routine, termed comfort- - (Sec. B Let X, Y be the input and output mini-batch tensors, respectively, and let Z g (X) be the hidden representation at layer l (Verma et al. 2019). Essentially, our method produces new training samples (A), Y (A) from the given ones by scaling their small singular values by random À E [o, 1]. At its core, comf comfort-- zone incorporates into training the assumption that data with similar dominant components of the train set should be treated as true samples. We offer two simple implementations of comfort zone; non-differentiable approach that can be used for input-level application, and a fully differentiable pipeline which is applicable any layer (App. A).
We analyze comf cooffort-zoon risk minimization (Sec. 4. Our experimental evaluation focuses on benchmark regression tasks (Sec. 5.1), and on time series forecasting tasks with small and large datasets (Sec. 5.2). The results show that comfort- zone improves the test error on several neural architectures and datasets, and in comparison to other DA baselines. We offer a potential explanation to the success of our method (Sec. App. B. Finally, an ablation study is performed, justifying our design choices (App.
using perturbation theory, and we introduce its associated vicinal
2 RELATED WORK
Deep neural networks regularization is an established research topic with several existing works (Goodřellow et al. 2016). Common regularization approaches include weight decay, dropout (Sri vastava et al. 20114). batch normalization (loffe & Szegedy] 2015), and data augmentation (DA). In what follows, we categorize DA techniques to be either domain-dependen or commin-indeeendeen Domain-dependent DA was shown to be effective for, e.g., image data (LeCun et al. 1998) and audio signals (Park et al., 2019), among other domains. However, adapting these methods to new data domains is typically challenging and often infeasible. In the past few years, an increased interest has been drawn to domain-inde iinreindenndeeeeennt DA methods, allowing to regularize neural networks when only basic data assumptions are allowed. We focus in what follows on domain nomain-independent techniques that were proposed in the context of classification and regression problems.
DA for classification. Recently, Zhang et al. (017) proposed to convex mixing of input samples as well as one-hot output labels during training. The training procedure, named mi xup, minimizes the Vicinal Risk Minimization (VRM) problem instead of the typical Empirical Risk Minimization (ERM). Many extensions of mixup were proposed, including mixing latent features (Verma et al. 2019). same-class mixing VDees & Taylor 2017). among others (Guo et al. 2019: Hendrycks
ct al. 2019 Yun al. 2019 Berthelot al. 2019 Greenewald al. 2021 Lim et al. 2021
DA for regression. Significantly less attention has been drawn to designing domain- -emain-independent data augmentation for regression tasks. A recent survey (Wen et al. (020) on DA for time series data lists a few basic augmentation methods including noise injection. Incorporating noise in the data can be used for regression tasks, and it can also be incorporated into other DA methods such ours. Dubost et al. (2019) propose to recombine samples for regression tasks with countable outputs, and thus their method can not be directly extended to the uncountable regime. Recently, mi xRL (Hwang & Whang 202T) developed a meta learning framework based on reinforcement learning for mixing samples in their neighborhood.
2

---

Under review as a conference paper at ICLR 2023
def scale_ down (2, lem, :ho):
Data
additive noise
J, S, Vt numpy.linales www.lgg.aaiiii............ avd cumpexo - numpy.cumauni cumaum (s) numpy. sum (s) numpy. where (cungere > zho, 1am, 1.0)
1000
(1ag
0.50
X, Y aze in barch x reats
000
for (x, Y) in loader:
mixub
comfort-2om
numpy. 1am www.atenateeeeeecceccccee......e (2, exis-1)
alpha)
1.00
scalecaaeeeco (2, 1am, rho)
X, Y 2(t, :m] B[t, m:]
030
op=imizer.zeto_geeee 2ero
loss (net (X), YJ bscckarrdd op:imizer.atepl.
We show the pseudocode for comfort- zone at the input level, = (left). We demonstrate the effect of a few DA methods on 2D data whose intrinsic dimension one (right).
Figure 1:
COMFORT ZONE
A learning task is typically described as function which maps inputs to outputs. In this view, learning model is approximating that function using e.g., neural network, and formulated via f: X + y. denoting the input and output domains by X' and V, respectively. A regression problem is such that the output domain is (un)countable, e.g., y c N or y C R" For simplicity, we consider the setting X, y C R but method is applicable to other cases. During training, the learning model is provided with a training set DD = { y4)) {=)) sampled from (I;, ys) ~ P. Our method extends the training distribution by producing new training
describe below.
To generate new samples, we consider the singular value decomposition (SVD) of a matrix A E Raxr, q which is given by A USVT, The matrices U, V are orthogonal, and S is diagonal matrix whose main diagonal consists of the singular values ordered by σ1 2 02 2 ... 2 a, 2 0. SVD is intimately related to principal component analysis (PCA) which in turn is heavily studied in manifold learning and dimensionality reduction (Ma & Fu 2012). It is well-known that the best rank approximation of is given by omitting the last (r -) singular values, i.e., A = Σj=1 uz (Eckart & Young 1936). The matrix A preserves the (k) dominant components A, and it discards the rest. The key insight in our approach that scaling down the small singular values should yield training samples that in close proximity to the true distribution of the data P.
Let the input and output mini-batch tensors E hxxm and Y E Rdmm respectively, where w.l.o.g b > 2m is the batch size. We denote the network by f(X)
(X)), Zi (x(X) where @ maps inputs to latent representations Z, at layer E [0, LJ. and fi maps latent vectors to outputs (Vermal al. 2019). Let Beta(a, a) for (0, oo), and & E [1, 2ml be the index of the singular value after w which we scale down. Then, the new artificial samples (A, k), Y ((A, k) are defined via
A = 1, Y = USVT E 2x2m
A(A, k) = (A, A), Y(,, })] = US(A USA,
where . concatenates along columns, and S(X, k) is the diagonal matrix of scaled down singular values. Namely, we compute S(x, k) diag( (011 01.....07 ok Ao Ad The value & depends on the hyper-parameter p E [0, 1] that represents the "amount" of signal to keep unchanged, i.c.,
k arg max Σ o. Σo,sp.
Similar to mixup (Zhang et al. 2017 our method recovers the original dataset as a - 0, Vp.
The loss function associated with comfort-zon is
C()) Ex,y) Ex E c ((Ji, (x(x s.t. (X,Y) D, X Beta a), IN 10, L].
3

---

Under review as a conference paper at ICLR 2023
1.0
Air Passengers (normalized)
0.8
MSE
0.0015
0.6
baseline confoott zore
0.5
1.0
0.0010
0.00
0.25
0.50
0.75
1.00
10
15
A
Time
Figure 2: Evaluating a non-augmented model and model trained with comfort ceffottzoon on train data whose small singular values are scaled down for different values of À (left). We show on the right panel an example of time series sample (black), and its modifications using À = 0.5 (bluc), 0.3 (orange), and = 0.0 (gray).
where c Rbxm x 2 - [0, oc) is a cost function, typically mean squared error (MSE). The transformation X takes pair of tensors gi (X), Y. and it scales down the last (2m 22m-k) k) singular values of their concatenation by X. A key attribute comfort- coonfort-zoooo zzone is that it is fully differentiable since the singular value decomposition comfort- zone is applied only at the input level (I = 0), a straightforward non-differentiable imple- mentation is sufficient. We provide example py Torch pseudocode for input-level comfort- zone in Fig. (left), and we discuss in App. A potential differentiable implementation. The computational complexity of comfort- zone is governed by SVD calculation which has complexity of SVD is (mmin(qr², (gr²,rg²)) for an X r matrix. In comparison, mi xup samples scalar from random distribution, and linearly blends two samples, whereas additive noise samples points from a random distribution. Thus, mi xup and additive noise have a complexity of O(gr).
be backpropagated (Ionescu al. 2015). Importantly, when
Design choices. For certain À values, the new sample Zi k). Y). k) may be too far from P. With this in mind, we explored the option of scaling down the loss function c(. by parameter (A(A) in addition to modifying the singular values. However, we tested various profiles p(A) and discovered the best models are obtained when scaling of loss occurs, App. Importantly, this means that our approach adopts a different ansatz in comparison to mi xup. While mi mixup incorporates uncertainty into the model training using "in-between" samples and labels, our method uses the new data as if it was sampled from the true distribution. An alternative option which would be conceptually closer mi xup is to scale down the large singular values well the loss term. We show in App. C that this choice is also inferior to comfort-zome
The effect of comfort intrinsic dimension is one (shown in blue, Fig. []. and we applied different DA methods on this data. The three panels in the figure show in orange the augmented data when using additive noise, mi mixup, and comfort-ze zone with a = 1.0 over the original point cloud colored in light blue. Injecting noise alters each point in its neighborhood, whereas mi mixup draws the points towards the center of their convex hull. In contrast, comf ort zone aligns the new samples along the dominant component of the original data. Notably, our approach may increase the span training data, and thus it can improve estimation in regression as was recently shown in (Wu et al. 2020).
data and learning. We generated 2D point cloud whose
We argue that training on samples created with our method encourages the inherent tendency of the network to model the dominant parts of the data better. To demonstrate this phenomenon, we trained an N-BEATS (Oreshkin et al. 2019) architecture with and without com comfort-2pe fort-zone on the Air Passengers dataset provided in DARTS (Herzen et al.] 2022). The trained models are evaluated on the dataset modified using a 100 varying singular values of every batch in the dataset using different X, and feed the resulting data for inference. Surprisingly, the non-au con-augmented model (blue curve) performs better on the unseen modified samples, yielding the minimum at À 0.5. In comparison, the regularized network attains qualitatively similar plot, but the MSE is lower for all and the minimum is obtained for a lower value at À ss 0.3. This behavior found to be consistent across several architectures and datasets, see App.
[0. values, see Fig. 2 (left). Namely, we modify the
4

---

Under review as a conference paper at ICLR 2023
Inspecting an example of the data and its modifications, reveals the differences between the samples when select À values are used. The original sample (black) shown in Fig. (right) exhibits primary extrema Sp times t 4, 7, 15 and secondary extrema S, at times dots. The blue curve (A = 0.5) for which the non-augmented model attained the minimum loss, maintains the primary extrema while significantly "flatenning" the secondary extrema. In comparison, the orange curve (A = 0.3) for which the augmented model achieves the minimum loss, flattens Sy completely. Finally, we observe that the S, data points are qualitatively different when À 0.0. From the analysis above, we conclude the following. First, the network prefers data with less small scale features; this finding is consistent with similar results on e.g., autoencoder models (Jain et al.) 2021). Second, our regularization encourages this tendency by providing the model with such data, leading to improved MSE profiles. To the best of our knowledge, the above analysis is novel on deep regression models.
9, 13 marked with black
Notably, while may argued that the behavior in Fig. (left) is natural and intuitive as the model "simply" performs better denoised signals, argue differently. In particular, this plot somewhat contradicts our understanding of overfitting which occurs in high probability for tiny datasets such as Air Passengers (a single time series with 144 entries) using multiple weights network such as N-BEATS. Specifically, since the data is highly likely to be overfit by the network, we expect the MSE value be lowest for À = 1, and MSE value equal higher for any À < 1. Thus, we advocate that the above analysis may reveal a characteristic feature of regression neural networks. Our analysis reinforced further as other datasets and architectures follow a similar pattern (App. B). Importantly, we are unaware of similar experiment in the literature of deep regression neural networks.
4 ANALYSIS
Relation to additive noise. In what follows, we would like to answer the following question: Does applying comfort- zone is merely a variant of injecting additive noise? To this end, we analyze comfort zone from perturbation theory viewpoint. Specifically, we would like to understand how random data perturbation affects the singular values of the data matrix A E R q 2 r. We denote by 01 2 d2 2 2 or the singular values of A. The perturbed matrix and its singular values set are denoted by À A+ E and { respectively. We write (nf2(A) and |Al2 to denote the smallest and largest singular values of any matrix A. The following classical result provides an estimated bound for the perturbed singular values (Stewart 1979: 1998).
Theorem Let be the orthogonal projection onto the column space of A. Let P1 =I I P. Then
a} = (a, + 71)² + nn j j= j=1....,r,
where 17s| S |P E12 and inf < 1|1
Following Stewart (1979), we make two observations with respect Thm. First, if a, >> EE|2 then it dominates the bound and we have ,j _ aj + 7 Second and more important to our setting, when o, of order E|2, the term 17, will tend to dominate. Indeed, in these cases the term n7 increases the singular value Tj. We conclude that random perturbations to A tend to increase its small singular values. In contrast, comfort- zone typically decreases the small singular values, while leaving the large o, unchanged. Thus, com comforte foonfortteeeeeee is in effect a complementary approach to injecting additive noise, allowing a finer control over the resulting new samples. Finally, we note that for a certain choice of hyper-parameters,
approach can be viewed as injecting noise per the above analysis. For example, taking p = 0.0 and Uniform(1.0 a) for a > 1.0 will increase all the singular values of A by a factor of X E [1.0, a)), where Uniform is the random uniform distribution.
Vicinal Risk Minimization (VRM). Given cost function e: yxy' - I the learning problem aims at minimizing the expectation of the loss c( c(f(x), y) over the distribution P(x,2) v). x E X E y. A fundamental challenge, shared by most real-world scenarios, is that the true distribution of the data is unfortunately unknown. The alternative is to minimize over the empirical distribution of train set (11,, y1) given by
comfort-zone
d Pa emp (I, 4)
Σ Ses (1)6)
5

---

Under review as a conference paper at ICLR 2023
The resulting scheme is the common training procedure of modern neural networks, formally known as the Empirical Risk Minimization (ERM) (Vapnik 1991).
While Pemp provides a basic approximation of the true P. it suggested (Chapelle et al. 2001 that other density estimates d Pess that take into account the vicinity of (z) should be constdered. The recent mi xup approach (Guo et al.) 2019) exploits this idea by proposing Vicinal Risk Minimization
(VRM) procedure that is based on the vicinal distribution estimate 1 Dig 0ä,, (A) (A) defined using convex combinations Zig (A) = λz + (1 2) for 2 E {z, y and X ~ Beta( aa a). In this context, the main difference between comfort - zone and mi mixup in the definition of vicinity as we describe below.
We denote by T(x, y) the tangent plane of the data manifold M at the point (1,, y) E EMM x V. Namely, T(r, y) is the linear approximation of M at (z, y). For every pair (x, y). we define new density distribution Penn which considers all pairs (a, b) in the tangent plane of (u, v) E .M. Formally,
d P v) = -N (a())66(y) dobdue.
Then, comfort-zee zone approximates the latter expression by generating an estimate of the tangent plane Teat via SVD, yielding the following vicinal estimate
d Pe v) -
Ki Σ de, (x)d,, (v),
(x), TealI v1). Kr (Ta y:)l
EXPERIMENTS
5.1
REGRESSION BENCHMARK DATASETS
While there is extensive work deep regression in the vision community for e.g., object detection (Szegeyy et al. 2013) and human pose estimation (Li & Chan 2014), we aimed for an evaluation setting where data modalities different from images and text are being considered. To this end, we evaluate com -one on regression benchmark datasets that frequently appear in the literature, see e.g., Hemmaaddd-eezoobbttttt & Adams (2015). The datasets include Diabetes listing 442 patients with 10 feature variables (Efron et al. 2004); Concrete describes 1030 instances of the actual concrete strength using features Yeh| 1998); Energy details the energy efficiency of 768 building shapes using 8 variables (Tsanas & Xifara 2012); and Wine which consists of 1599 red wine instances with 11 features (Cortez et al. 2009). The output of Diabetes, Concrete and Wine has one feature, whereas Energy has two features. We perform min-max normalization to all datasets, and we remove during model testing.
The baseline architecture consider is residual network (ResNet) (He al., 2016). ResNet models are typically overparameterizee, and thus they serve as a good baseline to explore DA effects. We use fully connected layers in the residual block instead of convolutions, employing 18 (ResNet 8) and 34 (ResNet 34) residual layers followed by a linear layer. During training, data is split to 70%, 10% and 20% for the train, validation and test sets, respectively. Each model trained 20 times with random splits, and it is trained for 40 epochs using a hidden size of 100 and batch size of 16. We employ an Adam optimizer, with 0.0001 weight decay and an initial learning rate of 0.001, and we reduce it by half with patience of 3 based on the validation loss. The loss is MSE, and we infer over the models which yield the best loss on the validation set when averaged over 20 runs.
We compare the baseline (ERM) to mixup (Guo et al. 2019). additive uniform noise (UN), and comfort-zone (CZ). The results are detailed in Tab. uusing the metrics: root MSE (RMSE), mean absolute percentage error (MAPE), and R2 (Makridakis & Hibon 2000). In m. xup, we follow the authors guidelines and use a E [0.1, 0.4), and with the additive noise we use a scale of 0. We apply comfort- zone at the input level, and perform grid search over E {0.5, 1.0, 2.0), and E {0.97, 0.98). Our results show that more depth yields inferior results, which may be related to the network size w.r.t dataset size. Injecting uniform noise to the data generally under-performs w.nt the baseline. In contrast, mi nixup improves ResNet18 generalization, and its results on ResNet 34 not as effective. Our approach always yields improved test errors, except for Concrete with

---

Under review as a conference paper at ICLR 2023
Table 1: Test errors regression benchmarks using ResNet18 and ResNet architectures.
Diabetes, ResNet18
Diabetes, ResNet
RMSEL 4
MAPE
R21
R
MAPE
R21
ERM
76.00
9.28
56,00 + 7.64
-0.11
82.24 + 9.26
61.16 10.64
-0.29
0.29
m xup
73.76
8.88
53.55 + 8.71
+0.04+0.25
81.12 + 8.63
58.53 + 7.61
-0.26
0.27
UN
76.88
- 6.53
58.19
1 4.75
86.14 0.19
18.69
61.14 = 8.26
-0.47 = 0.79
C2
71.79
5.26
51.80 = 7.09
0.02
77.69 + 5.58
56.63 1 6.23
0..15 1 0.17
Concrete, ResNec18
Concrete, ResNet34
ERM
10.58 + 1.72
2
0.55 + 0.17
11.87 + 0.74
36.65 + 3.50
0.45 + 0.07
mixup
10.21 = 0.77
31.14
0.59 0.06
12.41 + 1.27
36.68 + 4.27
0.40 + 0.12
UN
11.19 0188
35.65 + 4.07
0.51 0.08
14.33 + 4.00
45.25 + 11.87
0.14 0.60
C2
9.87 0.61
28.35
3.47
0.62
1 0.05
11.91 + 0.74
35.85 + 3.64
0.45 0.07
Energy. ResNet18
Energy, ResNet
ERM
3.70 0.29
12.93 + 1.33
0.86 + 0.02
4.35 + 0.30
15.75 + 1.68
0.81 + 0.03
mixup
3.78 = 0.27
12.88 + 0.95
0.86 + 0.02
4.72 + 0.39
17.08 + 1.69
0.78 + 0.04
UN
3.83 0.24
13.69
+ 1.03
0.85 + 0.02
5.98 + 3.09
22.85 + 12.41
0.55 + 0.58
CR
3.61 0.16
12.25
0.70
0.87 + 0.01
4.29 + 0.33
15.23 + 1.50
0.81 + 0.03
Wine, ResNet18
Wine, ResNet34
ERM
0.68 0.03
9.89 0.49
0.28 + 0.05
0.75 + 0.09
10.66 + 0.82
0.13 0.26
mixup
0.67 0.04
9.52 0.53
0.31 0.08
0.74 +
10.48
0.16 0.12
UN
0.69 0.03
0.28 + 0.05
0.73 0.03
10.58 0.55
0.18 0.07
C2
0.65 0.03
0.34 9.33 0.42
0.06
0.72 0.05
10.27 + 0.71
0.20 + 0.11
34 layers by small RMSE margin. Further, our method achieves the best results on all datasets in comparison to all other DA baselines. We also find that comfort-2e zone reduces the standard deviation for almost all datasets and metrics.
5.2 TIME SERIES FORECASTING (TSF)
Small-scale TSE. Forecasting time series data is one of the fundamental regression tasks in machine learning. We test comfort-zons using the DARTS (Herzen et al., 2022) time series forecast- ing framework, which supports several TSF methods and datasets. Specifically, we consider the DARTS implementations of RNN, TCN (Bai al. 2018), TRANSFORMER (Vaswani al. 2017). and N-BEATS (Oreshkin et al., 2019). The datasets are mostly univariate, i.e., the time series samples are one-dimensional. In our experiments, we perform a min-max normalization to the data, and we convert it to single-precision floating point representation. Importantly, DARTS datasets are small, ranging from & 100 samples to R 3000 samples in total. This regime of small training sets expected to benefit the most from DA techniques such ours.
The data is split to approximately 80% for training and 20% for testing. Unless otherwise noted, covariates such hour-of-day are not used (Salinas et al. 2020p. We train for 300 epochs, using batch size of 32 and an Adam optimizer with a learning rate 0.001 and no scheduling. In all cases, the training loss is mean squared error (MSE). The specific input and output tensor sizes depend on the dataset, and we provide this information in App. D For reproducibility and to reduce variability, we train each model the same hundred seeds (0, ..... .9). During inference, evaluate the trained models using the root mean square error (RMSE), mean absolute percentage error (MAPE), and R2 measures, see c.g., (Makridakis & Hibon 2000). We report the average measures and their standard deviation over the seed set. In our experiments, we compared the effect of comfort- -oonerttttnneeee in relation to the baseline model (ERM), and to the baseline augmented with the DA approaches m xup, additive noise (UN), and com commorte orttoroo rone (CZ). Following the evaluation protocol proposed in the original mi xup paper (Zhang et al. 2017). we evaluate the dependence different DA methods on the choice of hyper-parameters. To this end, we fix the parameerssssrrss for all DA baselines. We used a = 0.4 for mixup, a scale of 0.1 for UN, and a = 0.2 and = 0.9 for cZ in all cases. For comfort- zone. take the best result out of the original data and noise-injected data. The parammers were chosen using a basic grid test, taking the parameters which yield the best average across DARTS datasets.
7

---

Under review as a conference paper at ICLR 2023
Table 2:
Test errors of small-scale time series forecasting datasets from DARTS. Each dataset is trained on generic N BEATS, S, and it is augmented using comfort - zone and other DA approaches.
Air Passengers
Australia Beer MAPE !
RMSE x
MAPE
R21
RMSE xx100
R2 1
ERM
5.66 + 1.2
8.30 + 1.8
0.49 + 0.2
3.27 + 0.8
4.79 + 1.2
0.91 + 0.0
mixup
5.24 1.1
7.73 + 1.6
0.57 0.2
2.97 0.6
4.60 = 1.0
0.92 - 0.0
3.64 1.0
1.6
0.78 0.1
3.72 0.9
5.24
+ 1.4
0.88 0.1
CZ
3.56 + 0.9
5.43 = 1.4
0.79 + 0.1
2.96 0.6
4.51 0.9
0.92 + 0.0
US Gasoline MAPE
Sunspots MAPE
RMSE
10
R21
RMSE 1
R21
ERM
6.33 + 0.5
8.64:..5
-0.11
6.17 = 0.9
27.45 - 3.2
-0.68 + 0.5
mixup
0.5
6.73
0.2
26.17
-0.61 = 0.4
UN
+.6
6.68 + 0.6
-0.16
6.02 0.7
27.59 +.5
-0.59 + 0.4
cz
6.25 + 0.6
6.59 = 0.6
-0.08 0.2
6.09 0.9
27.20 #34
-0.64 + 0.5
Tab. 2 shows the statistics and results for the univariate datasets Air Passengers, Australia Beer, US Gasoline and Sunspots provided in DARTS. These datasets trained on the baseline N-BEATS architecture whose time series forecasting capabilities are considered state-of-the-art (Oreshkin et al. 2019), and then trained again on baseline with DA. We observe consistent behavior where confort-2one improves the generalization error compared to ERM and usually reduces the standard deviation for all datasets. Further, comfort zone beats all other methods, except on Sunspots where the best results for MAPE are attained by mi xup, and for RMSE and R2 by additive noise. In addition Tab. we show in Tab. an extended evaluation, showing the results on DARTS datasets on the baseline architectures RNN. TCN and TRANSF ORMER, and on their DA augmented versions. In this extended setting, we observe that TCN and TRANSFORMER benefit from our DA for all datasets, whereas RNN yields mixed results with com fonfortt-oo fort-zone. Furthermore, the standard deviation typically becomes smaller with our DA in comparison to the baseline. Additive noise and mixup somewhat depend on the architecture and dataset, where in some cases the generalization improves and in others, deteriorates. Notably, the best overall results (marked blue) for each dataset were almost always obtained with comfort-zone metrics for US Gasoline, where comfort- zone yields the best MAPE results, and second best RMSE and R2 (marked in red). Finally, we note that while our C2 on Sunspot with N-BEATS in Tab. 2 did not yield the best estimates in comparison to other DA techniques, the setting of TCN with C7 attains the best overall results for Sunspots, see Tab. 9
The only exception was for RMSE and R2
Large-scale TSE. To further evaluate our approach in the context of forecasting, we consider larger-scale benchmark datasets. Electricity contains the hourly electricity consumption of 370 customers for a total of x 9.7M samples, and Traffic includes the hourly occupancy rate of 963 lanes of San Francisco bay freeways for total of =s 10.1M samples. Both datasets appear frequently in the forecasting literature, e.g., (Salinas et al.] 2020 Oreshkin et al., 2019). We incorporate our confort- zone into the N-BEATS framework which includes ensembling during inference. Following Oreshkin et al. 20119) we train the generic and interpretable models with and without DA, and we use total of 180 models for evaluation. These models arise from using different metrics, different horizon lengths, and different initialization. Finally, different data splits are considered. Using the original code repository of the authors, we approximately recover their results. We refer to (Oreshkin et al. 2019) for the full details regarding the evaluation protocol and testing setup. We perform grid search over a [0.1..... 1.0) and p E (0.8.0.85, 0.9) yielding a total of 30 models per dataset, architecture and split. Many hyper-paramede improvement in the normalized deviation (ND) test error. 3 shows the ensemble median results of the generic (N-BEATS-OQ and interpretable (N-BEA TS-I) baselines, well as results. While comfort- zone improves both datasets, we observe that the generic net benefits relatively more from DA in comparison its interpretable version.
combinations lead to an
We further extend our evaluation on large-scale TSF tasks where we consider the datasets ETTm2, Exchange, weather and ILI in the challenging setting of long horizon forecasting benchmark including
8

---

Under review as a conference paper at ICLR 2023
Table 3: Test errors for N- N-BEATS architectures generic (G) and interpretable (T) trained with and without comfort on Electricity and Traffic datasets for different train-test splits.
Electricity
Traffic
2014-09-01
2014-03-31
last 7 days
2008-06-15
2008-01-14
last 7 days
TT-BEAT--O
0.065
0.066
0.179
0.129
0.231
0.115
N H-BEATS-C
+ CZ
0.062
0.062
0.166
0.118
0.231
0.116
N BEATS- BEATS
0,072
0.071
0.179
0.122
0.234
0.118
0.070
0.068
0.174
0.12
0.231
0.116
forecasts of lengths 96, 192 and 336. We refer the reader to Wu et al. (2021) where the datasets are described as well as the benchmark setting. For baseline, we consider the generic version of N-BEATS, trained in ERM and augmented with Cz. We performed a grid search for -rreeore using p E (0.85, 0.90, 0.95], and a {O.I,.... 1.0]. Many
combinations lead to improved results over the baseline, and we report the best results of our approach per horizon. Tab. - reports the MSE and mean absolute error (MAE) metrics of this experiment. In all cases except for ETTma with horizon 96, CZ improves generalization and yields better enror metrics.
Table 4: Long horizon time series forecasting results.
ETTma
Exchange
Weather
ILI
96
192
336
96
192
336
96
192
336
96
192
336
MSE
0.173
0.250
0.301
0.101
0.157
0.320
0.169
0.220
0.284
1.830
2.074
2.329
MAE
0.257
0.312
0.354
0.216
0.288
0.423
0.205
0.250
0.302
0.883
0.961
1.062
MSE
0.175
0.242
0.297
0.078
0.156
0.298
0.164
0.217
0.275
1.798
1.997
2.306
MAE
0.259
0.310
0.349
0.195
0.287
0.401
0.201
0.247
0.293
0.854
0.939
1.039
6 DISCUSSION
We have proposed comfort-zone, a data-driven method for data augmentation of regression tasks. We showed that comfort zone supports the network tendency of representing dominant components of its input signals by creating virtual examples sampled from the tangent planes of the original train set. Implementing comfort-ze zone is straightforward, and admits fully differentiable as well as simpler non-differentiable versions. Throughout extensive evaluation. we have shown that comfort- confoorttnnee zone improves the generalization error of neural models on time series forecasting datasets and regression benchmarks. In addition, com fort-zene obtains better results when compared to a few data augmentation baselines, while reducing the standard deviation of the model ensemble.
When inspecting the effect of the hyper-parameters a and we observe that for small datasets increases, and for medium datasets the results are stable deteriorate for
the results improve increasing a. Further, larger neural models (ResNet 3 were less affected by changes in a in comparison to smaller models (ResNet18). We identify that choosing the value of p depends on the intrinsic features of the dataset. In general, higher p preferable when the intrinsic dimension of the data higher. However, our understanding of the interplay between the super-parameters and model behavior is still somewhat limited. The time complexity of comfort-zone is governed by the SVD calculation, which may be restrictive for large train batches.
There are several exciting avenues for future exploration. First, is there fundamental link between the vicinal distribution employed and the learned representation? While several existing works suggest that linearity yields better models, the model dependency on the specific definition of vicinity still not well understood. Second, can similar methods be useful in classification tasks? The adaptation of confort zone classification is straightforward, however, several design choices which were tuned for regression may require change in a classification setting.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances neural information processing systems, 32, 2019a.
Zeyuan Allen-Zhu Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In International Conference Machine Learning, pp- 242-252. PMLR, 2019b.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 2018.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48))30063 2020.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raf- fel. Mixmatch: A holistic approach to semi-supervised learning. Advances Neural Information Processing Systems, 32, 2019.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. Advances in neural information processing systems, pp. 416-422, 2001.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision support systems, 47(4): 547-553, 2009.
Terrance VeVriss and Graham W Taylor. Dataset augmentation feature space. arXiv preprint arXiv: 1702.00500 2017.
Lue Devroye, László Györfi, and Gábor Lugosi. A probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013.
Florian Dubost, Gerda Bortsova, Hieab Adams, M Arfan Ikram, Wiro Niessen, Meike Vernooij, and Marleen de Bruijne. Hydranet: data augmentation for regression neural networks. In International Conference on Medical Image Computing and Computer-Asiisted Intervention, pp. Springer, 2019.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, 1(3):211-218, 1936.
Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. Least angle regression. The Annals of statistics, 32(2):407-45) 499. 2004.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Kristjan Greenewald, Anming Gu, Mikhail Yurochkin, Justin Solomon, and Edward Chien. k-mixup regularization for deep learning via optimal transport. arXiv preprint wwXiv:210..029:: 2021.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regulariza- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3714-3722, 2019.
Boris Hanin and Sun. How data augmentation affects optimization for linear regression. Advances in Neural Information Processing Systems, 34, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 1912.0278) 2019.
10

---

Under review as a conference paper at ICLR 2023
José Miguel Heemández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International conference on machine learning. pp. 1861-1869.
PMLR. 2015.
Julien Herzen, Francesco Lässig, Samuele Giuliano Piazzetta, Thomas Neuer, Léo Tafti, Guillaume Raille, Tomas Van Pottelbergh, Marek Pasieka, Andrzej Skrodzki, Nicolas Huguenin, et al. Darts: User-friendly modern machine learning for time series. Journal of Machine Learning Research, 23 (124))1-6, 2022.
Seong- Haynnn Hwang and Steven Euijong Whang. MixRL: Data mixing augmentation for regression using reinforcement learning. arXiv preprint urXiv:2106.03374, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Catalin lonescu, Orestis Vantzos, and Cristian Sminchisescu. Matrix backpropagation for deep networks with structured layers. In Proceedings the IEEE International Conference on Computer Vision, pp. 2965-2973, 2015.
Saachi Jain, Adityanarayanan Radhakrishnan, and Caroline Uhler. A mechanism for producing aligned latent spaces with autoencoders. arXiv preprint arXiv:2106. 15456, 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11): (16(1))227722333 1998.
Sijin Li and Antoni B Chan. 3d human pose estimation from monocular images with deep con- volutional neural network. In Asian Conference on Computer Vision, pp. 332-347. Springer, 2014.
Soon Hoe Lim, N Benjamin Erichson, Francisco Utrera, Winnie Xu, and Michael W Mahoney. Noisy feature mixup. arXiv preprint arXiv:2110.02180, 2021.
Yungian Ma and Yun Fu. Manifold learning theory and applications, volume 434. CRC press Boca Raton, FL, 2012.
Spyros Makridakis and Michele Hibon. The M3-competition: results, conclusions and implications. International journal of forecasting, 16(4):451-476, 2000.
Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. Classification vs regression in overparameterized regimes: Does the loss function matter? Journal of Machine Learning Research. 22(222):1-69, 2021.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 1rXiv:1905.10453 2019.
Daniel S Park, William Chan, Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. 2019.
arXiv preprint
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
Connor Shorten, Taghi Khoshgoftaar, and Borko Furht. Text data augmentation for deep learning. Journal of big Data, 8(1):1-34, 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1) 2014.
11

---

Under review as a conference paper at ICLR 2023
Gilbert W Stewart. Perturbation theory for the singular value decomposition. Technical report, 1998.
GW Stewart. A note the perturbation of singular values. Linear Algebra and Its Applications, 28: 213-216, 1979.
Christian Szegedy, Alexander Toshev, and Dumitru Erhan. Deep neural networks for object detection. Advances in neural information processing systems, 26, 2013.
Athanasios Tsanas and Angeliki Xifara. Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy and Buildings. 49:560-567, 2012.
Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 1138-6447. PMLR, 2019.
Qingsong Wen, Liang Sun, Fan Yang. Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey, arXiv preprint arXiv:2002. 2020.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with nuto-correlation for long-term series forecasting. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Sen Wu, Hongyang Zhang, Gregory Valiant, and Christopher Ré. On the generalization effects of linear transformations in data augmentation. In International Conference Machine Learning, pp. 10410-10420. PMLR, 2020.
I-C Yeh. Modeling of strength of gegh-performanc concrete using artificial neural networks. Cement and Concrete research, 28( (2): 1797-1808, 1998.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
12

---

Under review as a conference paper at ICLR 2023
A A FULLY DIFFERENTIABLE COMFORT-ZONE
In Sec. 3 Fig. T we discuss a potential implementation of our method at the input level. i.c., for I = 0. However, this approach is not suitable for the latent version. Indeed, identifying the indices of singular values which should be scaled by À
proposed in Sec. is not differentiable action. Specifically, the use of numpy where () does not allow for end-to-end learning, and it should be replaced. Fortunately, PyTorch allows for differentiable index selecting from tensor, thus by using this feature we can separate the singular values we wish to scale from those we wish to keep as is. We separate the s, the singular values vector, into two vectors, scale the desired singular values and then concatenate the vectors, a differentiable operation in itself, to recreate s. For completeness, we provide the pseudocode for the fully differentiable sca e down function in Fig.
def scale down (2, 1am, rho) :
U, s, Vt torch. lineh..inngg.... sasnalgg.di(22 (2)
cumpere = torch. cumsum (s) / torch
Jts torch. nonzero (cumperc rho, True) [0]
Jbs nonzero (cumpere <- rho, True) [0] ts s Lidex____ee_ seleessseecctteeeeeeeeeeeeeeeeee (dim=C, index=Jts)
s. i index selecteeeeeeeee (dim-C, index-Jbs) s torch, cat ((ts, bs lam) )
a = U torch.dian (a) @ Vt
Figure 3: A differentiable version of the scal seale_downn function.
B SEQUENTIAL MODELS CAPTURE DOMINANT COMPONENTS OF DATA BETTER
Following the discussion in Sec. 3 verify empirically that neural networks model the dominant parts of their data better. We repeat the experiment Fig. in the main text using several datasets and architectures. Every pair of dataset and architecture are evaluated on the dataset whose singular values are modified using varying values of X. The results are presented in Fig. 4 where solid lines represent the non-regularized version, and dashed lines are associated with models trained with our DA. In all cases we observe a similar qualitative behavior as we reported in Sec. S In particular, the highest MSE values are obtained for both the baseline and regularized models for À == i.e., when the data is unchanged. Further, the model attain improved eгror measures as À decreases, where the error profile is similar for the baseline and regularized models. Based on these results, deduce that sequential models prefer to represent and compute the dominant components of data.
Air Paseergger NASEATS
0.010
Australia Beer N N-BEATS
0.0075
0.008
MSE
MSE
0.0050
0.006
0.0025
Passengers
Australa
Casoline Tansffrrme
0.004
Surspets RNN
0.25
0.50
0.50
1.00
A
Figure 4: We reproduce Fig. 2 for several architectures and datasets. In all cases the models achieve better error measures for the modified data, whether it appeared during training or not.
13

---

Under review as a conference paper at ICLR 2023
C ABLATION STUDY
To motivate the specific design choices in comfort-zond we run an ablation study over different design settings. The first hyper-parameter
consider is μu(A) which is used scale the cost function during training. Our experiments show that µч(A) = 1, i.e., no scaling, leads to the best results, and we report for profiles μ*(A= E (1, )²). The second hyper-parameter marks whether to scale down the small or large singular values. In comfort-zee zone we always scale the small singular values. The third eyper-parameter deals with modifying the samples the input level or in the latent space. The results are given in Tab. [ The ablation study is performed on the Concrete regression dataset using ResNet18 architecture, and Australia Beer dataset using N-BEATS architecture. For both datasets, scaling down the singular values at the input level and with no scaling to the loss function leads the best test measures. Further, the latent version of comfo porotottoooo yields the second best results. Finally, scaling down the large singular values and the loss function was beneficial for Australia Beer, but resulted poor measures on Concrete.
Table 5: Ablation study of comfort -etoorttzonne zone over different loss scaling profiles μ (A),, scaling down the small or large singular values, and modifying data the input or latent levels. Data
#samples #feats mode (()) scale RMSE
MAPE4
R21
input small
9.87 + 0.61
28.35 + 3.47
0.62
0.05
input À
small
10.60 + 1.04
31.35 + 3.20
0.56 0.09
Concrete
1030
input X²
small
11.29 + 0.80
33.17 3.05
0.50 + 0.07
input À large 22.89 + 4.33
61.62 + 13.97
-1.10 0.78
latent small
10.20 0.71
30.56 3.56
0.59 + 0.06
input small
0.030 + 0.006
4.510 + 0.927
0.924 + 0.031
input small 0.034 + 0.009
4.989 + 1.427
0.899 0.050
Australia Beer 176
input 2 small 0.035 + 0.008
5.150 + 1.314
0.890 + 0.051
input large 0.031 0.005
4.799
0.892
0.918
0.028
latent small 0.032 + 0.009
4.732 = 1.324
0.908 = 0.050
D TIME SERIES FORECASTING EVALUATION DETAILS
Tab. 6 shows several training parameters related to the time series forecasting experiments on DARTS datasets reported in Sec. 5 The split column specifies the point in time from which we split the data to train and test sets. Then #in and #out represent the series length for the input and output, respectively. Finally, #pred is the length of series predicted during model evaluation.
Table 6: Training parameters of DARTS datasets.
Data
split
#in
#out
#pree
Air Passengers
1958-
12-01
12
Australia Beer
2002-06-01
12
6
6
US Gasoline
2015-01-01
125
18
18
Sunspots
1940-10-01
125
18
18
E EFFECTS OF ADDITIVE NOISE IN COMBINATION WITH COMFCRT-ZONE
In our experiments, we tested DARTS time series datasets using comfort-zon (CZ) and additive noise UN, together and separately. We reported the results achieving the best metries with or without UN. For the sake of completeness, we add in Tab. 7 comparison of the results for each of our DARTS datasets, when training with CZ with and without UN. Typically, there is some improvement when using UN alongside cz, but this is not always the case, e.g., for Australia Beer and N N-BEATS where CZ alone had better results. For each dataset and architecture we also add the baseline without any DA. which shows that generally CZ improves the baseline with our without UN. We believe additive noise is helpful in this test scenario since the DARTS datasets are extremely small, and thus data augmentation does not necessarily span wide enough regime.
14

---

Under review as a conference paper at ICLR 2023
Table 7: Test errors several sequential neural architectures the small-scale time series forecasting datasets from DARTS, Each architecture is trained with comfort zone and either with (C2+ (C2+UM) or without additive uniform noise (CZ).
Data
#samples Method
RMSE | x x10
MAPE $
R2
N-BEATS
5.66 1.2
8.30 + 1.8
0.49 + 0.2
N-BEATS CZ N-BEATS
4.78 1.0
7.18 1.7
0.64 0.1
UN
3.63 - 0.9
5.57
0.79
Air Passengers
144
TRANSFORMER
5.23 1.0
1.25
0.59:00
TRANSFORMER C2
5.02 1.2
6.96
0.60 16000.2 0.2
TRANSFORMER + C2 + UN
4.35 + 1.0
1.20
0.70 0.2
N-BEATS N-BEATS C7 N-BEATS TRANSFORMER
3.27 + 0.8
4.2
0.91
0.03
2.90 - 0.7
0.92 0.03 1.4
UN
3.21 0.7
4.66
0.91
Australia Beer
176
0.9
6.51
0.00
TRANSFORMER CZ
1.27
7.07
0.76 0.11
TRANSFORMER + CZ + UN
4.55 1.0
5.94
0.82 0.08
N-BEATS N-BEATS CZ
6.33 0.5
6.64
-0.11 0.2
-0.0 6.6
0.2
N-BEATS TRANSFORMER
UN
6.35 0.6
6.61
0.2
US Gasoline
1578
6.07
6.48
0..01
TRANSF FOMMR
CZ
6.15
6.52
0.2
TRANSFOR ORMER CZ + UN
6.1
6.44
0.2
N-BEATS
6.17
27.45
0.66
N BEATS CZ
6.61
27.51
0.55
N-BEATS CZ TRANSFORMER
UN
6.03
27.71
0..60
Sunspots
2820
0.9
27.16
-0.81
TRANSFORMER CZ
6.38 0.6
27.07
-0.79
TRANSFORMER CZ + UN
5.88 + 0.6
26.86
- 0.51 0.3
F TIME CONSUMPTION OF APPLYING COMFORT-ZONE
We add in Tab. R the timings of applying cz on batch of the different time series datasets. We measured these timings by applying CZ on the batch 1000 times, measuring the entire time length then dividing the total time by 1000 get the average time of single application.
Table 8: Running times of single application of our method for several datasets.
Data
Data Size
Method timing x 10 -4 (ms)
Air Passengers
32 X 18
5.168
Australia Beer
32 x 18
5.122
US Gasoline
32 x 143
7.210
Sunspots
32 x 143
7.426
G ADDITIONAL. TIME SERIES RESULTS
In addition to the results in Tab. we evaluate our method on three architectures (RNN, TCN and TRANSFORMER), using ERM and mi xup, UN and C2 for DA approaches. We report the results in Tab. 9 where we highlight in blue the best method and in red the second-best. Overall, cZ achieves the best results all cases and RMSE, MAPE and R2 metrics, except of US Gasoline where TRANSFORMER yields better RMSE and R2 estimates.
15

---

Under review as a conference paper at ICLR 2023
Table 9: Test errors of several sequential neural models the time series forecasting datasets from DARTS, Each architecture is also trained with: mixup, additive noise. and comfort zone.
Data
#samples Method
RMSE x10
MAPE
R21
RNN
9.117 + 1.64
14.403
2.84
-0.044 + 0.49
RNN + mixup
9.269 1.81
14.643
3.18
-0.355 + 0.53
RNN + UN
8.805 1.60
14.000 2.80
-0.217 +.41
RNN
8.773 1.28
13.918
-.31
TCN
9.710 4.43
1.304:7.7
-0.731
TCN mixup
9.634
14.901
-0.620
Air Passengers 144
TCN + UN
8.857 = 2.43
13.686 =
-0.281
TCN + CZ
9.141 #366
14.225 6.32
-0.469
2.00
TRANSFORMER
5.234 + 1.02
1.245=1.7
0.568
0.17
TRANSFORMBR
mixup
6.321 1.64
8.876
+ 0.673 0.673 0.17
0.352 0.34
TRANSFORMER
UN
4.507 = 1.10
6.456 - 1.72
TRANSFORMEN
CZ
4.352 0.99
6.197 = 1.52
0.697 + 0.15
RNN
4.392 + 1.46
6.169 + 2.08
0.822 + 0.16
RNN + mixup
5.682 2.38
8.096 3
0.684 + 0.28
RNN + UN
6.416 + 1.77
9.109 2.67
0.631 + 0.23
RNN + CZ
4.522 1.71
6.385 2.48
0.806 + 0.20
TCN
4.058 + 1.85
5.939 2.51
0.834 0.21
TCN + mixup
4.040 2.05
+.775
+.25
Australia Beer
176
TCN + UN
1.798=2.0
0.775
TCN + CZ
3.757
5.522
2.34
0.858 + 0.19
TRANSFORMER TRANSFORMER mixup
4.939 0.89
6.507 1.44
0.790=0.00
4.113 0.99
5.776 1.53
0.851 0.07
TRANSFORMER
UN
5.914 + 1.47
7.963
2.25
0.691 0.15
TRANSFORMER CZ
4.431 + 0.88
5.730 1.26
0.830 + 0.07
RNN
7.146 + 0.48
7.996 + 0.67
0.406 + 0.19
RNN + mixu mixup
7.170 0.50
8.014 0.74
0..16 0.20
RNN + UN
7.322 0.47
8.230 0.62
-0.755
0.19
RNN + CZ
7.329
8.244
00.81
+ 0.23
TCN
6.943
7.649 0.44
00.25 + 0.15
TCN + mixup
6.903
7.611 =
-0.300
+ 0.15
US Gasoline
1578
TCN + UN
6.606
7.233
= 0.37
-0.199
0.12
TCN + CZ
4.337
7.399 0.43
-0.248
+.14 0.14
TRANSFORMER
6.066
6.478 0.36
-0.012
0.13
TRANSF TRANSFORMET UN 6.187 0.43
FRANSFORMER ORMER + mixup
0.46
6.517 0.42
-0.078 0.17
6.461 0.40
-0.054 0.15
TRANSFORMER + C2
6.153 + 0.44
6.443 + 0.38
0.043 0.15
RNN
5.773 0.24
28.259 1.45
0.443 + 0.12
RNN + mixup
5.797=0..29
28.842 1.44
0..45 - 0.13
RNN UN
5.953+0.3
30.169
-0.537 0.17
RNN + CZ
3.759:0..20
28.262 +.65
-0.43 0.13
TCN
5.901 0.66
25.455 2.06
-0.224 0.35
TCN + UN
6.277 0.66
28.336
2.83
-0.722
Sunspots
2820
TCN mixup
5.936 0.52
26.062 #
-0.535
TCN + CZ
5.785 + 0.63
25.287 + 2.35
-0.464
TRANSFORMER
6.399 0.94
27.161 2.57
-0.808 +.55
TRANSF ORMER
mixup
6.230 0.83
26.653 + 2.54
-0.707
TRANSFORMER
UN
5.899 0.61
26.121
2.31
-0.520
TRANSFORMER
CZ
5.902 0.47
26.890 + 2.44
0..25
16

---

Under review as a conference paper at ICLR 2023
H RESULTS OF CZ ON CIFAR DATASETS
In Tab. TO we demonstrate the results of applying CZ to image datasets, as well as a comparison to application mi xup and baseline with no DA. The results were produced using the aforementioned DA methods in their manifold setup on preact rreecttresnet_. 1 model and CIFAR datasets, each run with three different seeds then averaged. For the manifold-mi xup and baseline code we used the repository in (Lim et al. (20T)) and we added the manifold CZ version mentioned in Sec.Bon on top of When using CZ, we applied on the data alone, and did not incorporate the labels into augmentation. That is, after applying CZ to a sample, its target stayed the same. 'hee results are unfavorable towards CZ, but it's worth pointing out that mi xup was designed with classification in mind, and it augments the data using the targets well as the input data. In contrast, C2 was designed originally for regression tasks, and even though it incorporates the targets into the DA in those setups, it is less obvious realize how do so with the type of targets used in classification. As mentioned, the naive way we tried did not use the targets as part of the augmentation, and we think this is the main reason for the deficit in the results. We leave further exploration of this research direction to future work.
Table 10: Accuracy results image (CIFAR) datasets.
Data
Method
Mean Accuracy (%)
Baseline
94.673 + 0.071
CIFARIO
mixup (a 1.0) Cz (a = 1.0, 0.95)
95.507 + 0.076
94.623 + 0.119
Baseline
76.130 + 0.164
CIFAR100
mixup (a 1.0)
77.943 + 0.039
C2 (a 1.0, p 0.99)
75.747 + 0.135
17


Reviews:
[{"review_id": "zPQJbYqgen", "reviewer": "Reviewer_6Zsi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a data-driven method called comfort-zone for data augmentation of regression tasks. By producing new samples from the given ones by scaling their small singular values by random values, it incorporates the assumption that dominant components of the train set can also be viewed as true samples. Author[s] provide both non-differentiable input-level and differentiable pipeline which is applicable to any layer. Results on small and medium regression tasks and time-series forecasting show its effectiveness. ", "summary_of_the_review": "A simple drop-in approach for many data augmentation applications with comprehensive analysis. Results on regression tasks look good but need to see its performance on classification tasks and domain adaptation applications to verify its potential.  ", "strength_and_weaknesses": "[Strength]\n\n1. A nice idea of applying dominant components to mixup so that the new data points can mostly preserve the statistics of true samples.\n2. Author[s] provide a comprehensive analysis of its relation to additive noise and VRMs.\n3. Results on small/medium regression datasets are strong. \n\n[Weakness]\n\n1. Could author[s] provide any insights on why it cannot be used as a drop-in replacement to standard mixup (for classification)?\n2. Isn't comfort-zone encouraging stronger local manifolds around data points? Fig. 1 sounds like an overfit to the training data. How about the results used in domain adaptation, e.g. works like https://arxiv.org/abs/2001.00677? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CFqW8uu41f", "reviewer": "Reviewer_5t1j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a domain-independent data augmentation operator for regression tasks: $\\texttt{comfort-zone}$. It extracts the \"noise\" components in the data and scales them (by scaling small singular values of inputs/features). Apparently this is a new augmentation operation, and its effectiveness has been experimentally verified on some datasets.", "summary_of_the_review": "It is necessary to explore domain-independent data augmentation as a general-purpose regularization tool. Its value is further highlighted by the applicability to regression tasks.\nThe proposed augmentation ($\\texttt{comfort-zone}$) is well-motivated and innovative, with the only drawback that it brings very marginal improvements on large datasets.\n\nMy preliminary recommendation is to accept the article.", "strength_and_weaknesses": "**[Strength]**\n\n- At a high level, this work has far-reaching implications for our community because: &nbsp; **(i)** Designing domain-independent data augmentation can facilitate the development of deep learning in different areas, considering that many data modalities **do not** have well-defined data augmentation operations (e.g., rotation, panning) as images do. &nbsp; **(ii)** We don't have many options for regularizing deep models on regression tasks (e.g., label smoothing can only be used for classification).\n- The proposed operator ($\\texttt{comfort-zone}$) is flexible because it can be applied in both input space and intermediate hidden space. Also, using SVD and scaling the \"noise\" or \"trivial\" components to generate new data is intuitive.\n- $\\texttt{Comfort-zone}$ yields better regularization effect than its counterpart (noise injection and mixup) on some benchmarks.\n\n**[Weaknesses]**\n\n- $\\texttt{Comfort-zone}$ seems to be the opposite of noise injection, as it is trying to reduce the unimportant components in the data ($\\lambda \\in [0, 1]$). This seems to be contrary to the original purpose of data augmentation (to increase the noise in the data). What do the authors think about this?\n- Performance improvement from $\\texttt{comfort-zone}$ on large-scale datasets (in Sec. 5.2) seems marginal.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7MGJpRPJUK", "reviewer": "Reviewer_tecT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a domain-independent approach inspired by mixup, which they call comfort zone. Comfort zone works by taking the SVD of the matrix formed by the concatenation of batch inputs and outputs, and then generating augmented date by reducing the scale of some subset of the singular values (the scale and number of values are parameters).\n\nThey test this approach on regression and time series forecasting benchmarks and achieve good performance relative to the baselines they have chosen.", "summary_of_the_review": "The paper shows that attenuating some of the singular values can result in improved performance on some specific datasets. More experiments would be needed as discussed above to make those conclusions stronger. The novelty is debatable: SVD is designed so that removing singular values is natural. ", "strength_and_weaknesses": "Strenghts:\n- The paper is easy to follow, and explains the method well.\n- Compared to the baselines, in the settings the authors evaluate on, their method performs well.\n\nWeaknesses:\n- The empirical evaluation is somewhat short. For the forecasting experiments, only N-BEATS is used. Does the method also work for other datasets (the commonly used ETT, Electricity, Weather, ...), models (N-HITS, Transformer models, TCNs, LSTMs...)? What is the impact of the method on different approaches? More experiments would be needed to make the conclusions more credible.\n- While the approach is ingenious, taking the SVD of a batch and making modifications to some of the singular values is part of the set of transforms certain fields will naturally apply to their input data as part of pre-processing (e.g. signal processing). There is in this sense actual, but limited novelty.\n- The method claims to be domain independent but would not actually work (without modifications) for a non-regression task, contrary to Mixup.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0CJxoPhwhl", "reviewer": "Reviewer_CWda", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this manuscript, the authors proposed a method named comfort zone which utilize the SVD to augment data for regression problems. To demonstrate the effectiveness of proposed approach, experiments across different datasets and network architecture are conducted.  \n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n[Updates after rebuttals]\n\nReally appreciate the authors’ efforts for addressing my concerns. After discussion with other reviewers, I finalized my score as a marginal acceptance as the proposed comfort zone is flexible and very interesting for independent domain augmentation. I also have some advises for the authors as follows. \nFirstly, as replied by the authors, the proposed comfort zone has higher complexity than mixup and additive noise. Applying the SVD on large scale datasets would lead to long training time and the experimental results also show that the proposed approach can only achieve marginal improvement as pointed out by other reviewers. Secondly, I slightly agree with authors’ claim that other data augmentation methods on mentioned time series references are not applicable to the regression settings they consider. However, I would still suggest applying the proposed approach to time series regression tasks as they are very important in many real-world applications. Lastly, there is still room for improvement in terms of novelty and evaluation as mentioned by other reviewers. \n\n\n", "summary_of_the_review": "Some issues need to be further clarified by the authors:\n\n1. It is better provide the complexity comparison between the proposed one and other approaches.\n2.  Besides the mix-up and UN, other advanced data augmentation approaches in regression problems should be compared. For instance, the permutation-and-jitter strategy in reference [1], decomposition-based methods [2,3] or generative models [4]. \n\n[1] Time-Series Representation Learning via Temporal and Contextual Contrasting\n[2] Robusttad: Robust time series anomaly detection via decomposition and convolutional neural networks.\n[3] Fast RobustSTL: Efficient and robust seasonal-trend decomposition for time series with complex patterns\n[4] Time-series generative adversarial networks\n\n3. The experiments of using proposed CZ in latent space is insufficient if the authors want to claim their approach suitable for latent space. ", "strength_and_weaknesses": "Strength\n+ the paper is clearly written and easy to follow. The proposed method is technically sound. \n+ Extensive experiments are conducted to verify the effectiveness of proposed method.\n\nWeakness\n- Lack of comparison with SOTA data augmentation approaches \n- Lack of comparison with other augmentation approaches in terms of complexity", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject