Data Augmentation Policy Search for Long-Term Forecasting
Authors: Liran Nochumsohn
Omri Azencot
Venue: Accepted by TMLR ()

Abstract: Data augmentation serves as a popular regularization technique to combat overfitting challenges in neural networks. While automatic augmentation has demonstrated success in image classification tasks, its application to time-series problems, particularly in long-term forecasting, has received comparatively less attention. To address this gap, we introduce a time-series automatic augmentation approach named TSAA, which is both efficient and easy to implement. The solution involves tackling the associated bilevel optimization problem through a two-step process: initially training a non-augmented model for a limited number of epochs, followed by an iterative split procedure. During this iterative process, we alternate between identifying a robust augmentation policy through Bayesian optimization and refining the model while discarding suboptimal runs. Extensive evaluations on challenging univariate and multivariate forecasting benchmark problems demonstrate that TSAA consistently outperforms several robust baselines, suggesting its potential integration into prediction pipelines. Code is available at this repository: \href{https://github.com/azencot-group/TSAA}{https://github.com/azencot-group/TSAA}.

Published in Transactions on Machine Learning Research (02/2025)
Data Augmentation Policy Search for Forecasting
Liran Nochumsohn
kraamnoc@ppos..bg.cco
Deportment of Computer Science Ben-Gurion University of the Negev
Omri Azencot
azencot@bgu..cc ac.
Department of Computer Science Ben- Gurion University of the Negev
Reviewed on OpenReview: https: // openreview. net/ forum? id- www.aaboooouunnnoteeed-0000000an
Abstract
Data augmentation serves as a popular regularization technique to combat overfitting challenges in neural networks. While automatic augmentation has demonstrated success in image classification tasks, its application time-series problems, particularly in long-term forecasting, has received comparatively less attention. To address this gap, we introduce time-series automatic augmentation approach named TSAA, which is both efficient and easy to implement. The solution involves tackling the associated bilevel optimization problem through a two-step process: initially training a non-augmented model for a limited number of epochs, followed by an iterative split procedure. During this iterative process, we alternate between identifying a robust augmentation policy through Bayesian optimization and refining the model while discarding suboptimal runs. Extensive evaluations on challenging univariate and multivariate forecasting benchmark problems demonstrate that TSAA consistently outperforms several robust baselines, suggesting its potential integration into prediction pipelines. Code is available at this repository:
1 Introduction
Modern machine learning tools require large volumes of data to elfectively solve challenging tasks. However, high-quality labeled data is difficult to obtain as manual labeling is costly and it may require human expertise (Shorten & Khoshgoftaar, 2019). Small datasots may lead to overfitting in overparameterized models, a phenomenon in which the model struggles with examples it has not seen before (Allen-Zhe et al., 2019). One of the effective methods alleviate poor generalization issues is via data augmentation (DA). Data augmentation aims to generate artificial new examples whose statistical features match the true distribution of the data (Simard et al., 1998). In practice, DA has been shown to achieve state-of the-art ofth--ffthe-art (SOTA) results in e.g. vision (Krizhevsky et al., 2012) and natural language (Wei & Zou, 2019) tasks.
Unfortumately, DA is not free from challenges. For instance, Tian et al. (2020b) showed that the effectivity of augmented samples depends on the downstream task. To this end, recent approaches explored automatic augmentation tools, where good DA policy is searched for (Temley al., 2017: Cubuk et al., 2019). While automatic frameworks achieved impressive results on image classification tasks (Zheng al., 2022) and other data modalities, problems with time-series data received significantly less attention Kaufman & Azencot. (2024a); Nochumsohn et al. (2024). Toward bridging this gap, propose in this work new automatic data augmentation method, designed for time-series forecasting problems.
Time-series forecasting is long-standing lask in numerous scientific and engineering fields (Chatfield, 2000). While deep learning techniques achieved grouddbreaiing results on vision and NLP problems already a decade ago, time-scries forccasting (TSF) was considered by many to be too challenging for deep models, up until recently (Oreshkin et al., 2020). While recent linear approaches showed interesting forecast results (Zeng et. al., 2023; Nochumsohn et al., 2025), existing SOTA approaches for TSF based deep learning

---

Published in Transactions on Machine Learning Research (02/2025)
architectures that are structurally similar to vision models. In particular, current TSF deep models are overparameterized Kaufman & Azencot. (2024b); and thus they may benefit from similar regularization techniques which were found effective for vision models, such as (automatic) data augmentation. Ultimately, our work is motivated by the limited availability of DA tools for time-scrics tasks (Wen et al., 2020).
The main contributions of our work can be summarized as follows: 1) We develop a novel automatic data augmentation approach for long-term time-series forecasting tasks. Our approach is based on a carefully designed dictionary of time-serics transformations, Bayesian optimization for policy search, and pruning tools that enforce early stopping of ineffective networks. While these components appear in existing work, their combination and adaptation to time-scries forccasting was not done before, to the best of our knowledge. 2) We analyze the optimal policies our approach finds. Our analysis sheds light into the most effective transformations, and it may inspire others in designing effective data augmentation techniques for time-series data. 3) Our approach augments existing time-serics forecasting baselines, and we extensively evaluated it on long-term forecasting univariate and multivariate TSF benchmarks with respoct to several strong bascline architectures. We find that our framework enhances performance in most long-term forecast seltings and across most datasets and baseline architectures.
2 Related Work
Time-series forecasting. Recently, several neural network approaches for TSF have been proposed. Based on recurrent neural networks, DeepAR (Salinas et al., 2020) produced probabilistic forecasts with uncertainty quantification. The N-BEATS (Oreshkin et al., 2020) model employs fully connected layers with skip connections, and subsequent work (Challu et al., 2022) improved long-tene oon-terrm forecasting via pooling and interpolation. Another line of works based on the transformer architecture (Vaswani et al., 2017) used sparse encoder and a generative decoder in the Informer (Zhou et al., 2021), srenn-seasonality decomposition in the Auloformer (Wu et al., 2021), and Fourier and Wavelet trausfornations in the FEDformer (Zhou et al., 2022). Recently, Pyraformer (Liu et al., 2021) significantly reduced the complexity bottleneck of the attention mechanism, PSt (Nie et al., 2023) exchanges the point-wise attention input with a tokenized sub-series representation. Finally, Zeng et al. (2023) propose a single-layer MLP with a larger input lookback.
Data augmentation. DA techniques have appeared since the early rise of modern deep learning to promole labeled image invariance to certain transformations (Krizhevsky et al., 2012). Typical image augmentations include rotation, scaling, crop, and color manipulations. Recent methods focused on modality- agnostic methods which blend lincarly the inputs and labels (Zhang ct al., 2018) or utilize manifold learning approaches Kaufman & Azencot. (2023; 2024b;a). Other works produce augmented views in the feature space (DeVries & Taylor, 2017; Verma et al., 2019). In contrast to image and text data, angmenting arbitrary time-series (TS) data have received less attention in the literature (Wen et al., 2020; Iwana & Uchida, 2021). In the review (Wen et al., 2020), the authors consider three different tasks: TS classification, TS anomaly detection, and TS forecasting. Their analysis is based on common time-series augmentation approaches such as scaling, adding noise (Um et al., 2017), window cropping or slicing, and stretching of time intervals (Le Guennee ct al., 2016), dynamic time warping (Ismail Fawaz et al., 2019), perturbations of the frequency domain (Gao al., 2020; Chen et al., 2023), and utilizing surrogate data (Lee et al., 2019). In (Smyl& & Kuber, 2016), the authors discuss additional TS augmentation approaches including generating new TS using the residuals of statistical TS (Bergmeir et al., 2016). Another technique would be to sub-sample the parameters, residuals, and forecasting from MCMC Bayesian models. The survey (Iwana & Uchida, 2021) further details a large list of TS augmentations such as jittering, rotation, time warping, time masking, interpolation and others in the context of time-scries classification. The authors in (Wen et al., 2020) propose the selection and combination of augmentations using automatic approaches as a promising avenue for future research, which is the focus of the current work. Finally, we also mention the large body of work generative modeling of time scries (Yoon et al., 2019; Naiman et al., 2024b;a) which is related to data augmentation. We show in Fig. 1A two examples of DA policies.
Automatic DA. To avoid hand-tailored DA, recent efforts aimed for automatic tools, motivated by similar advances in neural architecture search (NAS) approaches (Zoph & Te, 2017). AutoAugment (Cubuk et al.,
2

---

Published in Transactions on Machine Learning Research (02/2025)
Electricity
1
Input h*
Upscale Smooth
rung 0
Baseline
Validation Loss
-1-
50
100
Time
rung
Electricity
Input
jittering Downscale
A
50
100
Time
B
Epochs
Figure 1: A) Two examples of sub-policies applied on Electricity data.
B) The above plot demonstrates the behavior of ASHA with respect to the baseline model (blue). Some of the poorly performing runs are discontinued at the end of rungs, whereas the other runs train to completion.
2019) used recurrent controller along with reinforcement learning for the search process, yielding highly effective but computationally intensive framework. Following works such Fast AutoAugment employed Bayesian optimization and density matching (Lim et al., 2019). RandAugment (Cubuk et al., 2020) reduces the search space significantly by introducing stochasticity. Tian et al. (2020a) suggested partial training using augmentation-win wise weight sharing (AWS). Further, recent approaches utilize gradients for the search problem, including the diffcrentiable automatic DA (DADA) (Li et al., 2020b) and Deep AutoAngment (Zheng ot al., 2022). Cheung & Yeung (2020) developed automatic DA that does not depend on the data-modality as it exploits latent transformations. In (Fons et al., 2021), the authors propose adaptive-weightin strategies which favor subset of time-series DA for classification, based on their effect on the training loss.
3 Background
Below, we briefly describe background information Bayesian optimization and pruning approaches which use to find the best augmentation policy and improve model training efficiency, respectively.
Tree-structured Parzen Estimators and the Expected Improvement. Bayesian optimization relates to family of techniques where an objective function f(x): Rd + R+ is minimized, i.e.,
min f(z)
(1)
In the typical setting, f is costly to evaluate, its gradients are not available, and d < 20. For instance, finding the hyperparameters (r) of a neural network (f) is common use case for Bayesian optimization (Bergstra et al., 2013). Unlike grid/random search, Bayesian optimization methods utilize past evaluations of f to maintain a surrogate model p(y]<)) for the objective function y == f(x). Thus, Bayesian optimization solves Eq. while limiting the costly evaluations of f to a minimum.
A practical realization of Bayesian optimization is given by Sequential Model-Based Optimization (SMBO) (Hutter et al., 2011)- SMBO iterates between model fitting with the existing parameters (ex- ploitation) to parameter selection using the current model (exploration). SMBO constructs surrogate model p(yla), finds set of parameters æ that performs best on the p(y]ar)) using an acquisition function, applies the objective function f on x to obtain the score updates the surrogate model, and repeats the last three steps until convergence. Most SMBO techniques differ in their choice of the surrogate model and acquisition function. We will focus on Tree-structured Parzen Estimator (TPE) for the surrogate model, combined with Expected Improvement for the acquisition function. The main idea behind TPE is to model the surrogate via two distributions, ((x) and g(z), corresponding to model evaluations that yield positive, and negative improvement. Formally,
p(zly) = ((xx) y < y g(x) y 2 y" 5
(2)
3

---

Published in Transactions on Machine Learning Research (02/2025)
where y" is threshold score, and the surrogate model is obtained via Bayes rule. It can be shown that maximizing (x)//((1) leads to an optimal Expected Improvement, (EI) (Bergstra et al,, 2011).
Asynchronous Successive Halving. While Bayesian optimization uses a minimal number of evaluations of f, the overall minimization is computationally demanding due to the high cost of f, e.g.; if f is a neural network that needs to be trained. To alleviate some of these costs, Asynchronous Successive Halving (ASHA) (Jamieson & Talwalkar, 2016; Li et al., 2020a) enforces early stopping of poorly performing parameters 2, whereas parameters with low [(x), are trained to the fullest. In fixed budget system, given maximum resource R, minimum resource r, and reduction factor 7, ASHA works follows. One creates model chockpoints during the training process at epochs r where j = 1,.... [log,, R/rl. Each checkpoint is referred to a rung, and at the end of each rung, one keeps only the best runs. To avoid waiting for all runs to reach the next rung, ASHA performs asynchronous evaluations to promote or halt runs on the go. We illustrate in Fig. 1B an example of bascline model with multiple different runs, administered by ASHA.
4 SimeeSeries AutoAugment (TSAA)
Automatic augmentation via bi-level optimization. The task of finding data augmentations auto- matically during the training of a deep neural network model can be formulated as a bi-level optimization problem, see e.g., (Li et al., 2020b). Namely,
min (w, 0
(3)
subject to min Epg [Lr((w,))
(4)
At the top level, Eq. 3, the optimization aims find the optimal augmentation policy 0 ~ po, where po is some distribution of augmentation policies, e.g., additive noise. Importantly, Eq. 3 minimizes the validation loss, Lval, that is parameterized both by the augmentation policy and by the network weights w. The latter weights are obtained in the bottom level, Eq. 4, describing an optimization problem that is similar to standard training of a neural network as it minimizes the training loss, Lur- The main two differences Eq. 4 from standard training is the dependence on 0 and the expectation over all possible augmentation policy distributions pe, leading overall Unfortunately, the above problem is difficult to solve in practice, and therefore, relax it as detailed next.
optimization problems, or, bi-level optimization.
TSAA overview. Our approach, which call time-serics automatic augmentation (TSAA), consists of two main steps, as illustrated in Fig. 2 and summarized in Alg. 1. In the first step, we partially train the model for a few epochs and construct a set of shared weights. The second step iterates between solving Eq. 3 in search of an augmentation policy using TPE and EI to solving Eq. with fino-tuning and ASHA for an optimal model. A complexity analysis given in App. D.
compute shared weights. Solving Eq. 4 iteratively requires repeated trainings of the deep model, which is computationally prohibitive. To reduce these costs, we propose to partially train the baseline model and generate shared set of weights Wahare Doing so, Step 2 is reduced to an iterative process of fine-tuning models for a small number of epochs, where Wahare are shared across all augmentations policics. Beyond efficiency aspects, applying DA in the later stages of training is assumed to be more influential (Tian el al., 2020a). In practice, we partially train for [BK] epochs, where 3 = 0.5 is a hyperparameter and is the active number of train epochs. In our tests, K S 10, and it may be strictly less due to an early stopping schoduler. K is found by training the bascline model with no augmentation to completion and saving the weights after every epoch. Then, we define
Step 1:
Wahare := w([SK)), :: K [3K],
where R is the maximum resource parameter, and r = 1 is the minimum resource, see Sec.
Step 2: iterative split optimization. Given Wahares remains solve Eqs. and 4 to find the best augmentation policy 0* and final weights w*. In TSAA, we propose to split this problem to an iterative
4

---

Published in Transactions on Machine Learning Research (02/2025)
Step 2
TPE
fine-tune + ASHA
Seasonality Uperale
input
Step
Wshare
P#
Jitter
partial train
fine-tune
find policy
find weights
w
Our time-series automatic augmentation (TSAA) approach is based on a partial train of the model (Step 1), followed by an iterative process (Step 2) where we alternate between improving the augmentation policy 9 training the model weights w. We find w" by fine-tuning over pg
Figure 2:
process, where we alternate between exploring augmentation policies @ via Eq. 3 to exploiting the current policy and produce model weights w via Eq. 4. Namely, for fixed set weights w, the upper minimization finds the next policy to try by evaluating the validation set. Then, we fine-tune the model using a fixed 0 with early stopping for a maximum of epochs to produce the next w. This procedure repeated until predefined number of trials Tmax is reached. The k best-performing policies define pa. from which 0* is sampled, where we only allow policies that improve the baseline validation loss. Finally, fine-tune the model again to obtain w".
Solving Eq. 3. Existing work solved the upper problem using reinforcement learning (Cubuk al., 2019; Tian et al., 2020a), grid search (Cubuk et al., 2020; Fons et al., 2021), and one-pass optimization (Li et al., 2020b; Zheng et al., 2022). Inspired by (Lim et al., 2019), we propose to use Tree-structured Parzen Estimator (TPE) with Expected Improvement (EI), see Sec. 3. the context of TSAA, the parameters x in Eq. represent the policy 0 and f L The Bayesian optimization conducted over the policy search space and time-scries augmentations we describe below.
Policy search space. The augmentation policies we consider are drawn from a distribution po over k sub-policies O ......, 02). where 8j O. Thus, pe allows to sample augmentation 0 from each of the considered p(0y), i.c.,
That is, the distribution pe aggregates several independent distributions p(0,),
& ~ Pa = II ((,)) 02 00 0x)
(5)
Each sub-policy 0j is composed of transformations Ty applied sequentially the output data ø;-1 of the previous transformation with zo being the input data and mja being the magnitude of the transformation. For instance, if the sub-policy 0, consists of trend downscale in T₃₁ and flip in T3, with magnitudes mj,1 = 1, m2,2 = 1, respectively, then, 0z = Ty2,.77 ( $), J.. For the general case of n transformations, 0, is definod via
0g = (Fa o ... (2. m2.)
(6)
Time-series data augmentations. While natural images are invariant to geometric transformations as translation and rotation, arbitrary time-series data need not be invariant to a certain type of transformations. Morcover, capturing the invariance regression problems such as TSF may be more challenging than classification tasks including images (Kaufman & Azencot, 2024a). Finally, time-series data may include slow and fast phenomena such as bursts of electricity usage and seasonal peaks, for which some DA may be

---

Published in Transactions on Machine Learning Research (02/2025)
inapplicable. Thus, we propose exploit DA that manipulate some features of the data and leave some features unchanged. For example, adjusting the trend while keeping the seasonality and noise components unaffected, or diversifying the time intervals in a way that the series mean and variance still stay the same. In particular, we suggest the following timo-serics transformations: identity, jittering, trend scaling, scasonality scaling, scaling, smoothing, noise scaling, flip, permutation, reverse, senamic-time-steteee warping, and mixup. The magnitude of the augmentations can be controlled using a single parameter. The transformations are further elaborated in App. B and Tab. 4 in the appendix.
(DTS), window
Solving Eq. 4. Finally, solving the bottom minimization may be achieved in a straightforward fashion via fine-tuiig. However, as motivated in Sec. 4, doing so iteratively is costly. To prune runs, we augment our approach with Asynchronous Successive Halving (ASHA). Our choice to use ASHA over other techniques such as Bayesian Optimization HyperBand (BOHB) (Falkner et al., 2018) is motivated by the following reasons. First, BOHB has shown to be slightly inferior to ASHA (Li et al., 2020a). Second, In our setling R. E 2., 5) and 1 is set to be more aggresive. As result, only two SHA brackets at most can be exploited in the HyperBand, thus limiting its effectiveness.
Algorithm 1 Time-Series AutoAugment (TSAA)
Inputs: partial train factor 3, resources R,r, max trials Timax, reduction factor 1) and k best DA sub-policies
A 0
{emply set into A}
Waahare, R + partial solve Eq. 4 with 8 (initialize the weights we using the partial solution)
STATE Wg f Washare for = to Tmax do
0, solve Eq. with TPE(0,
w; + tune Eq. with Wahare and ASHA(r, Rt n)
A +- AU {10:, (wr,
(add found policy and loss to A}
end for
pa- best sub-policies 6, from A
w + fine-tune Lur ~ P6-) return pa p0 ,,''", (return the optimal DA distribution, policy, and network weights}
5
Results
In what follows, we provide a comprehensive overview our experimental setup, including models, datasets, and implementation details followed by evaluations of our approach. Tn the supplementary material, we offer additional information on hyperparameters (App. C), and extended results (App. E.1).
5.1 Models and datasets
We extensively evaluate the performance of our Time-Serie AutoAngment (TSAA) framework. To this end, we selected some of the most recent prominent time-series forecasting models. We consider the baseline architectures: N-BEATS (Oreshkin et al., 2020), a deep neural architecture based on backward and forward residual links and a very deep stack of fully-connected layers. Informer (Zhou et al., 2021) adapts the Transformer (Vaswani et al., 2017) architecture to time-series forecasting tasks, with a new attention mechanism. Autoformer (Wu et al., 2021) exchanges the self-attention module for an auto- correlation mechanism and introduces time-series decomposition part of the model's encoding. Finally, FEDformer (Zhou et al., 2022) enables capturing more important details in time-series through frequency domain mapping.
For each of the given baseline models, we apply TSAA on six commonly-used datasets in the literalure of long-term time-series forecasting: (1) ETTm2 (Zhou et al., 2021) contains electricity transformer oil temperature data alongside power load features. (2) Electricity (Zhou et al., 2021) is a collection of hourly electricity consumption data over the span of 2 years. (3) Exchange (Lai al., 2018) consists of 17 years of daily foreign exchange rate records representing different currency pairs. (4) Traffic (Zhou et al., 2021) is

---

Published in Transactions on Machine Learning Research (02/2025)
hourly reported sensor data containing information about road occupancy rates. (5) Weather (Zhon et al., 2021) contains 21 different meteorological measurements, recorded every 10 minutes for an entire year. (6) ILI (Wu et al., 2021) includes weekly recordings of inftuenza-like illness patients.
We summarize in Tab. 5.1 the different datasets and their attributes such as their sampling frequency, variates which determine the number of channels each example, the total number of timesteps in each dataset, the different horizon lengths used for forecasting, and lastly the lookback period which the input length used for the prediction.
Datasel Sammary
datases
frequency
variates
total Siunesteps
horizoo
lookbackk period
ETTm2
15 minutes
7
69,680
96, 192, 336, 720
96
Electricity
baurly
321
26,304
96, 192, 336, 720
96
Exchongge
daily
7,588
96, 192, 336, 720
Traffic
bourky
862
17,541
96, 192, 336, 720
Weather
10 minutes
52,096
96, 192, 336, 720
ILI
weekly
966
24, 36, 48, 60
5.2 Implementation details
Baselines. We train all models based on the implementation and architecture details they appear in (Oreshkin et al., 2020) for N-BEATS and (Zhou et al., 2021; Wu et al., 2021; Zhou et al., 2022) for the Transformer- based models. The model weights are optimized with respect to the mean squared error (MSE) using the ADAM optimizer (Kingma & Ba, 2015) with an initial learning rate of 10-3 for N-BEATS and 10-4 for Transformer- based models. The maximum number of epochs is set to 10 allowing early-stopping with a patience parameter of 3. The reported baseline results are obtained using our environment and hardware, and they may slightly differ from the reported values for the respective methods. Every experiment is run on three different seed numbers, and the results are averaged over the runs. The Pytorch library (Paszke et al., 2019) is used for all model implementationss and executed with NVIDIA GeForce RTX 3090 24GB.
Method. We use Optuna (Akiba et al., 2019) for the implementations of TPE and ASHA. The number of trials Tmax is set to 100. For TPE, In order to guarantee aggressive exploration at the beginning, we run the first 30% of trials with random search. For ASHA, r and 17 are set globally to 1 and 3 respectively. The maximum resource parameter R, representing the epochs, is set differently for each experiment, due to the bascline's sarly-stopping.
After the augmentation policy search is finalized, a maximum of k best policies selected to obtain po-; where & = 3, and the final model is fine-tured with 0 ~ Pa- using the shared weights Wshare- We opt to fine-tune the model and not re-train from random weights so that the final model training matches our optimization process as close as possible. Indeed, Cubuk et al. (2020) discuss the potential differences between the final model behavior in comparison to the performance of the intermediate proxy tasks, ie., the models obtained during optimization. As the similarity in performance between these models and the final model is not guaranteed, a natural choice is to similarly train the proxy tasks and the final model, as we propose.
Augmentations. Each transformation includes different increasing or decreasing magnitude range which are all mapped to the range [0, 1]. This way, = 0 implies the identity and m is the maximum scale. To climinate cases of the identity being repcatedly chosen, we replace the lower bound in the range with an >>0 O such that for all transformations in the search space only m > 0 is possible. The transformations Trend scale and Seasonality scale require computing the seasonality and trend components; we pre-compute these factors using the decomposition STL (Cleveland et al., 1990) and treat it as part of the input data. Each augmentation is applied before the input is fed to the model, namely, on the input x and the target y of the train data batches.
7

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Multivariate long-term time-series forecasting results on six datasets in comparison to five baseline models. Low MSE and MAE values are better, and high relative improvement MSE% and MAE% scores are better. Boldface text highlights the best performing models.
Informer MSE
Autoformer MSE
FEDtormer-w
FEDformer-F
TSAA
MAE
MAE
MSE
MAE
MSE
MAE
MSEL
MAEL
MSEX1
MA0%
96
0.545
0.588
0.231
0.310
0.205
0.290
0.189
0.252
0.187
0.274
1.058
2.837
ETTn2
192
1.054
0.808
0.289
0.346
0.270
0.329
0.258
0.326
0.255
0.314
1.163
3.681
336
1.523
0.948
0.341
0.375
0.328
0.364
0.323
0.363
0.304
0.350
5.882
3.581
720
3.878
1.174
0.411
0.431
0.433
0.425
0.125
0.121
0.398
0.403
6.353
4.276
Electiciity
96
0.336
0.416
0.200
0.316
0.196
0.310
0.185
0.300
0.183
0.297
1.081
1.000
192
0.360
0.441
0.217
0.326
0.199
0.310
0.201
0.316
0.195
0.309
2.010
0.323
336
0.356
0.439
0.258
0.356
0.217
0.334
0.214
0.329
0.208
0.323
2.804
1.824
720
0.386
0. 452
0.261
0.363
0.248
0.357
0.246
0.353
0.238
0.348
3.252
1.416
Exchnnge
96
1.029
0.809
0.150
0.281
0.151
0.282
0.142
0.271
0.143
0.272
0.704
0.369
192
1.155
0.867
0.318
0.409
0.284
0.391
0.278
0.383
0.270
0.378
2.878
1.305
336
L.589
1.011
0.713
0
0.442
0.493
0.450
0.497
0.459
0.504
-3.846
-2.231
720
3.011
1.431
1.246
0.872
1:227
0.868
1.181
0.841
1.213
0.842
-2.710
-0.119
96
0.741
0,420
0.615
0,384
0.584
0.368
0.577
0.361
0.565
0.352
2.080
2.493
Weather Taffee
192
0.753
0.426
0.670
0.421
0.396
0.375
0.610
0.379
0.571
0.351
4.195
6.400
336
0.876
0.495
0.635
0.392
0.390
0.365
0.623
0.385
0.584
0.359
1.017
1.644
720
1.011
0.578
0.658
0.402
0.613
0.375
0.632
0.388
0.607
0.368
0.979
1.867
96
0.315
0,382
0.259
0.332
0.2G9
0.347
0.236
0.316
0.180
0.256
23.729
18.987
192
0.428
0,4419
0.298
0.356
0.357
0.412
0.273
0.333
0.252
0.311
7.692
6.607
3:56
0.620
0.554
0.357
0.394
0.422
0.456
0.332
0.371
0.296
0.355
10.843
4.313
720
0.975
0, 0.722
0.422
0.431
0.629
0.570
0.408
0.418
0.382
0.395
6.373
5.502
21
5.3:19
1,582
3.549
1.305
2.752
1.125
3.268
1.257
2,760
1.123
-0.291
0.178
35
5.203
1.572
2.834
1.094
2.318
0.980
2.648
1.068
2.362
0.984
-1.898
-0.408
48
5,286
1.594
2.889
1.122
2.328
1.006
2.615
1.072
2.264
0.988
2.749
1.789
60
5.419
1.620
2.818
1.118
2.574
1.081
2.866
1.158
2.520
1.062
2.098
1.758
5.3 Main results
In our experiments, we employ similar setup to (Wu et al., 2021; Zhou et al., 2022), where the input length is 96 and the evaluated forecast horizon corresponds to 96, 192, 336, 720. For IIII we use input length 36 and horizons 24, 36, 48, 60. For a fair comparison, we re-produce all baseline results on our system, and the augmentations are applicd on the same gencrated batches as the bascline. Our main results are summarized in Tab. and Tab. 2 including all the baseline results and TSAA. For TSAA, we include the best performing model trained on all baseline architectures. The full results for every architecture with and without TSAA are provided in the appendix spanning tables 6-14. We detail the mean absolute error (MAE) and mean squared error (MSE) (Oreshkin et al., 2020). Lower values are better, and boldface text highlights the best performing model for each dataset and metric. For TSAA, we also include the relative improvement percentage, i.e., 100 (es - -e,)ee,)//ee eete/e, where e is the best basclinc error and en is our result. We denote by MSE% and MAE% the relative improvement of MSE and MAE, respectively. A higher improvement is better.
Multivariate time-series forecasting results. Based on the results in Tab. 1, we observe that most datasets benefit from automatic angmentation, where in the vast majority of cases, TSAA improves the bascline scores. It is apparent that TSAA yields stronger performance in particular in the long-horizon settings with 6.35% (0.425 + 0.398) reduction in ETTm2, 3.25% (0.246 - 0.238) reduction in Electricity, and 2.1% (2.328- + 2.264) reduction in ILI. One of the more prominent results appears for Weather 96 and 336 with reductions in MSE of 23.73% (0.236 + 0.180) and 10.84% (0.332 + 0.296), respectively. For the Exchange dataset, TSAA obtains slightly higher errors with respect to the FEDformer-w w bascline. Overall, TSAA achieves the best results in 39 error metrics, in comparison to FRDformer-f and FEDformer-w with 4 and 5 best models, respectively.
Univariate timc-scries forecasting results. Similar to the multivariate results, most long horizon settings benefit from TSAA. With a 21.74% average reduction across all datasets with a horizon of 720. Furthermore, the results that stand out the most are the MSE and MAE reductions in Weather, with a
8

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Univariate long-term time-series forecasting results on five datasets in comparison five baseline models. Low MSE and MAE values are better, and high relative improvement MSE% and MAE% scores are better. Boldface text highlights the best performing models.
Informer
Autoformer
FEDformer-f
N-BEATS-I MSE
N-BEATS-G
TSAA
MSE
MAE
MSE
MAE
MSE
MAE
MAE
MSE
MAE
MSEL
MAE]
MSE%
MAE%
96
0.085
0.225
0.123
0.270
0,068
0.198
0.080
0.213
0,080
0.210
0.068
0,192
0.000
3.030
ETTm2
192
0.130
0.282
0.141
0.289
0.096
0.238
0.103
0.240
0.110
0:250
0.096
0.237
0.000
0.420
336
0.1G1
0.314
0.170
0.319
0.138
0.286
0.162
0.312
0.172
0.320
0.139
0.290
-0.725
-1.399
720
0.221
0.373
0.206
0.353
0.189
0.335
0.199
0.3-17
0.201
0.353
0.187
0,336
1.058
-0.299
Electrictt
96
0.261
0.367
0.454
0.508
0.244
0.364
0.326
0.402
0.324
0.397
0.244
0.354
0.000
2.747
192
0.285
0.386
0.511
0.532
0.276
0.382
0.350
0.417
0.363
0.420
0:277
0.368
-0.362
3.665
336
0.324
0.417
0.739
0.651
0.347
0.432
0.393
0.440
0.392
0.443
0.310
0.394
4.321
5.516
720
0.632
0.612
0.673
0.610
0.408
0.473
0.458
0.490
0,489
0.502
0.378
0.447
7.353
5.497
Weather Trafic Exchange
96
0.490
0.354
0.149
0.308
0.133
0.284
0.210
0.344
0.223
0.351
0.093
0.236
30.075
16,901
192
0.790
0.721
0.290
0.415
0.292
0.419
1.130
0.840
0.783
0.675
0.215
0.352
25.862
15.181
2.146
1.223
0.708
0.662
0.477
0.532
1.587
1.047
2.622
1.266
0.532
0.572
-11.30
-7.519
720
1.447
1.008
1.324
0.892
1.304
0.882
0.870
0.747
2.588
1.303
0.527
0.594
39.425
20.482
0.262
0.348
0.266
0.372
0.210
0.318
0.181
0,268
0.159
0.240
0.158
0.239
0.629
0.417
0.294
0.376
0.272
0.379
0.2006
0.177
0.263
0.181
0.264
0.160
0.243
9.605
7.605
336
0.308
0.390
0.261
0.374
0.217
0.322
0.180
0.271
0.155
0.239
0.156
0.244
-0.645
22.92
720
0.361
0.110
0.269
0.372
0.243
0.3-12
0.226
0.316
0.212
0.301
0.189
0.279
10.849
8.221
96
0.005
0.048
0.009
0.078
0.009
0.073
0.003
0.044
0.003
0.043
0.001
0.024
66.667
44.186
192
0.001
0.051
0.009
0.068
0.007
0.067
0.001
0.0:1
0.001
0.017
0.001
0.027
75.000
41.304
336
0.003
0.043
0.006
0.058
0.006
0.062
0.004
0.048
0.005
0.054
0.002
0.035
33.333
18.605
720
0.004
0.049
0.007
0.063
0.006
0,060
0.004
0.049
0.004
0.048
0.002
0.034
50.000
29.167
Multivariate
60
86005
40
30
30
20-
20
20
20
20
10
10-
10
Traffic
Weather
ETTm2
ECL
Exchange
Univarlate
30
9kops
15-
20
10-
10
20
10-
5-
5
Traffic
Weather
ETTmZ
ECL
Exchange
Jittering
Smooth
Flip
Permutation
SeasonalityUpscale
Identity
Downscale
TrendDownscale
TrendUpscale
DTS
Mixup
NoiseScale
Upscale
WindowUpsize
SeasonalityDonnscale
Figure The best five performing transformations per dataset attained with TSAA, measured with the percentage proportion of the selected operations (%ops). Each colored bar represents a transformation and the y-axis represents the percentage proportion the given transformation accounts for.
66%, 75%, 33, 50%%, and respectively 44.2%, 41.3%, 17.6%, 29.2% performance improvements corresponding to the 96, 192, 336 and 720 horizons. Further, it is evident in Tabs. 10-14, that the improvements in the Weather dataset are not limited to specific baseline architecture. Tn contrast to the multivariate setting, TSAA achieves significantly better scores on the Exchange dataset with average improvements of 21% and 11.27% for the MSE and MAE metries. Notably, the results in the univariate case are slightly more involved than the multivariate setting such that that only Weather always benefits from TSAA, whereas the results for other datasets are mixed. Still, TSAA shows a positive advantage over all bascline models. In particular, TSAA obtained the best models for 32 error metrics, whereas FEDformer-f and N-BEATS-G are better in 9 and 2 measures.
Policy analysis. The most noticeable selected transformations are illustrated Fig. is evident that the transformations Trend Downscale, Jittering, Mixup, and Smoothing are some of the prominent selections in the overall setup. Trend Downscale accounts for more than 30% of the operations in ETTm2, Weather
9

---

Published in Transactions on Machine Learning Research (02/2025)
and Electricity; this may indicate that the deep models tend to overestimate the trend, and thus it requires downscaling. Jittering and Smoothing the other hand, do not violate time-series characteristies such trend or seasonality but still promote diversity within the given dataset, where Smoothing is approximately the opposite of Jittering. Notably, Mixup appeared as one of the five most important transformations for lour and three datasets the multivariate and univariate settings, respectively. We believe that Mixup is beneficial to TSF since it samples from a vicinal distribution whose variability higher than the original train set. We show in Fig. 5 the outcome with and without TSAA compared to the ground truth, showing that employing custom policies per signal may significantly improve forecasting.
6 Ablation and Analysis
6.1 Parameter selection
Choice of B. what follows, we motivate our choice for the 3 hyperparameter which dictates for how many epochs we pre-train the baseline architecture obtain Wshare. To this end, we investigate the effect of utilizing different values of B.. We consider four different settings: 1) full training with augmentation, i.e,, 3 = 0.0, 2) half training with augmentation, ie., B 0.5, 3) augmentation applied only in the last epoch, 4) baseline training with no augmentation, i.e., 3 1.0. We used TSAA on the ILI with respect to N-BEATS-G in the univariate setting, and Informer, Autoformer and FEDformer-f in the multivariate case, as well as on multivariate ETTm2 with Autoformer and FEDformer-f. We plot the averaged results of these architectures in Fig. 4A, showing four colored curves corresponding to the various forecasting horizons 24/96, 36/192, 48/336 and 60/720 with colors blue, orange, green and red, respectively. The best. models are obtained for 3 = 0.0 and 3 = 0.5, that is, full- and half-augmented training. Somewhat surprisingly, two of the four best forccasting horizons (36/192 48/336) obtained for B 0.5. Overall, the fully augmented model (i.e., B = 1.0) attains 5.1% average improvement over the baseline, whereas using B = 0.5 yields a 5.3% average improvement. Thus, fully training with augmentation achieves similar performance to half training, while requiring significantly more resources. Indeed, Tian et al. (2020a) employs a similar strategy, and thus, we propose to generate Wshare after training for half of the active epochs, and we fine-tune the model using optimal augmentation policies.
Horizon
0.210
Madel
0.26
24/96
Informer
36/192
Autoformer
0.205
FEDromerr
48/336
NBEATS-G NBEATS-I
60/720
0.25
0.200
0.195
A full
half
last epoch
none
B
100
150
200
250
Figure 4: A) The normalized average performance measures as function of different 3 values. Our results indicate that B = 0.5 (half) attains the best computational resources to performance gain ratio. B) We plot the normalized average performance measures as a function of different Tiaxx values. As the number of trials grows, we observe better overall performance.
Reduction factor and linked operations. In Sec. 3, we introduced the roduction factor 7 that controls the number of kept runs in ASHA. Additionally, we discuss in Sec. that every sub-policy is composed would like to empirically justify choices of linked operations of time-series angmentations. Here, for these two hyperparameters. Our ablation study uses the ILII dataset on the 36, 48, 60 forecasting tasks, with N-BEATS-G for the univariate case, and Informer, Autoformer, and FEDformer-f for the multivariate configuration. We test. the values v E (2,3) and n E [1,2]. Every experiment repeated three times, and we analyze the average results.
Overall, we propose to use the values 1 and n 2 due to the following observations arising from our experiments. The improvement difference between r and 7) 3 is only 0.12% in favor of ') 2, thus
10

---

Published in Transactions on Machine Learning Research (02/2025)
true
prediction
prediction + TSAA
Informer
Autoformer
FEDformer-f
NBEATS-G
0.0
0.1-
0.1
0
-0.5
0.0-
0.0
100
200
0
100
200
100
200
0
100
200
WMM
Policy
([TD: 0.84, M: 0.85], [TD: 0.99, 0.88), [TD: 0.99, P: 0.94]}}
[IP: 0.87, 0.07],
([SD, 0.2, D, 0.971,
{[F, 1.0, SD, 0.24],
[P: 0.85, M: 0.26],
[NS, 0.06, D, 0.91, [SU, 0.2, D, 0.79])
[F, 1.0, SD, 0.3], [F. 1.0, SD, 0.23]}
[P: 0.31, 0.84]}
P Permutation M Mixup SD Seasonality Downscale
TD Trend Downscale SU Seasonality Upscale
NS Noise Scale
D Downscale
F Flip
NS Noise Scale
Figure 5: The ground truth, prediction, and prediction with augmentation attained with TSAA applied to the same forecast target in ETTm2 with Informer, Autoformer multivariate, and Weather with FEDformer-f and NBEATS-G univariate. It is shown that augmentation can assist the different models to achieve more accurate predictions with better alignment, reduce excessive noise, and curtail outliers. The attained policies are given underneath each plot.
suggesting that neither exhibits a statisticalll--ominss performance advantage. Nevertheless, 1/ 3 is resource efficient as it reduces the amount of kept runs 1/n by 16.67%. Moreover, a single operation attains 6.4% average improvement compared to the baseline, whereas two linked operations = 2 yield a 7.4% average improvement.
Convergence of TSAA. In our experiments, we look for good augmentation policies for Tmax 100 iterations. Here, explore the effect of this value on the performance of the resulting models. We evaluate our framework on the ILI dataset with the architectures Informer, Autoformer, FEDformer-f, N-BEATS-G and N-BEATS-I using varying values for Tmax E {100, 150, 200, 2503. Intuitively, greater Tmax values may result in an improved convergence and a better overall performance as the framework can explore and exploit. a larger varicty of configurations from the scarch space. Indoed, we show in Fig. 4B the normalized average MSE values obtained for the various tests. We observe an MSE reduction of 1% for the transformer- models when increasing Tiax 100 to Tmax 250. The N-BEATS architecture benefited more and achieved 7.25% reduction. In conclusion, the hyperparameter THax presents natural trade-off to the practitioner: higher Taax values generally lead to better performance at higher computational cost, whereas lower values are less demanding computationally but present inferior performance.
6.2 AutoAugment Method Comparison
We compare TSAA to other efficient AutoAugment methods via an experiment that shows that both Fast AutoAugment (Lim et al., 2019) and
(Cubuk et al., 2020) are inconsistent and thus inferior to TSAA. In this experiment, we tested the performance of deploying Fast AutoAugment and RandAngment with the same scarch space, with the exception of discrctized magnitude ranges so Randnngment falls line with the original method. The given methods were tested on Autoformer and FEDformer-f multivariate together with datasets: ETTm2, Traffic, Weather, and ILI. The results are provided in Tab. 3. For Fast AutoAugment.
11

---

Published in Transactions on Machine Learning Research (02/2025)
Table 3: Comparison of automatic augmentation approaches including TSAA, Fast AutoAugment and
RandAugment. We denote by %AI the average improvement in percentage.
Best Baseline
Fast AA
Rugmentntt
TSAA
Metrie
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.189
0.282
0.197
0.279
0.192
0.282
0.187
0.274
ETTm2
192
0.258
0.326
0.262
0.319
0.257
0.323
0.255
0.314
336
0.323
0.363
0.322
0.356
0.326
0.364
0.311
0.350
720
0.425
0.421
0.415
0.407
0.427
0.420
0.406
0.403
96
0.577
0.361
0.655
0.410
0.590
0.376
0.577
0.362
Traffic
192
0.610
0.379
0,652
0,408
0.622
0.390
0.601
0.371
336
0.623
0.385
0.674
0.421
0.626
0.392
0.619
0.383
720
0.632
0.388
0.705
0.427
0.643
0.396
0.632
0.388
96
0.236
0.316
0.191
0.252
0.203
0.275
0.207
0,285
Weather
192
0.273
0.333
0.240
0.290
0.267
0.332
0.252
0.311
336
0.332
0.371
0.290
0.321
0.328
0.364
0.313
0.355
720
0.408
0,418
0.363
0.367
0.398
0.413
0.382
0.395
24
3.268
1.257
4.671
1.603
3.160
1.234
3.150
1.219
36
2.648
1.068
3.835
1.375
2.457
1.019
2.578
1.049
48
2.615
1.072
3.694
1.359
2.558
1.045
2.609
1.069
60
2.866
1.158
3.855
1,410
2.775
1.108
2.805
1.140
%A1
0.0
-9.5
-1.88
1.67
1.22
3.33
3.1
we set K- 3 folds which control the number of subsets, each subset explores and exploits 100 augmentation discretize the magnitude range to 8 bins and utilize the partial train scheme
trials. For RandAngment, with the same 3 as in TSAA, to allow RandAngment to benefit from the same approach. While it is shown that Fast AutoAugment and RandAugment are superior on Weather and ILI respectively, they attain inferior results on the other datasets. Nevertheless, TSAA is shown to be most effective on Traffic and ETTm2 and second-best on ILI and Weather. Additionally, TSAA maintains a consistent improvement across all datasets providing a 3.33% average MSE reduction as opposed to RandAngment which offers approximately half of that, or Fast AutoAugment with a negative average MSE reduction. conclusion, TSAA is more consistent with better overall performance when compared to prominent AutoAugment methods in the time-series domain.
7 Conclusion
In this work, we study the task of data augmentation in the setting of time-scries forccasting. While recent approaches based on aulomatic augmentation achieved state-of-the-arl results in image classification lasks, problems involving arbitrary time-series information received less attention. Thus, we propose novel time-serics automatic augmentation (TSAA) method that relaxes a difficult bilevel optimization. In practice, our framework performs a partial training of the baseline architecture, followed by an iterative split process. Our iterations alternate between finding the best DA policy for a given set of model weights, to fine-tuning the model based on a specific policy, In comparison several strong methods on multiple univariate and multivariate benchmarks,
framework improves the bascline results in the majority of prediction settings.
In the future, we would like to explore better ways for relaxing the bilevel optimization, allowing to train an end-to-end model (Li et al., 2020b; Zheng et al., 2022). Further, we believe that our approach would benefit from stronger time-series augmentation transformations. Thus, one possible direction forward is to incorporate learnable DA modules, similar in spirit to filters of convolutional models.
Acknowledgments
This rescarch was partially supported by an ISF grant 668/21, an ISF equipment grant, and by the Isracli Council for Higher Education (CHE) via the Data Science Research Center, Ben-Gurion University of the Negev, Isracl.
12

---

Published in Transactions on Machine Learning Research (02/2025)
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparamcter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & dala mining, pp. 2623 2631, 2019.
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in newral information processing systems, 32, 2019.
Christoph Bergmeir, Rob J Hyndman, and José M Benítez. Bagging exponential smoothing methods using stl decomposition and box cox transformation. International journal of forecasting, 32(2):303 312, 2016.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimiza- tion. Advances in neural information processing systems, 24, 2011.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. International conference on machine learning, pp. 115 123. PMLR, 2013.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-HiTS: Neural hicrarchical interpolation for time scries forccasting. arxiv preprint arXiv:2201.12886, 2022.
Chris Chatfield. Time-series forecasting. Chapman and Hal//CRC, 2000.
Muxi Chen, Zhijian Xu, Ailing Zeng, and Qiang Xu. FrAug: Frequency domain augmentation for time scries Torecasting, Niv preprint ar arXiv:2302.0922 2023.
Tsz-Him Cheung and Dit-Yan Yeung. Modals: Monslity-agnostic automated data augmentation in the latent space. In International Conference on Learning Representations, 2020.
Robert B Cleveland, William S Cleveland, Jean E McRae, and Irma Terpenning. STL: A seasonal-trend decomposition. J. Off. Stat, 6(1):3 73, 1990.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. AutoAugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pottern Recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. RandAugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CV /CV conference compuler vision and pattern recognition workshops, pp. 702-703, 2020.
Terrance DeVrics and Graham W. Taylor. Datasct augmentation feature space. 5th International Conference on Learning Representations, ICLR, 2017.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning, pp. 1437 1446. PMLR, 2018.
Eugene F Fama. The behavior of stock-market prices. The journal of Business, 38(1):34 105, 1965.
Elizabeth Fons, Paula Dawson, Xiao-jun Zeng, John Keane, and Alexandros Iosifidis. Adaptive weighting scheme for automatic time-series data augmentation. orXiv preprint arXiv:2102.088 08310, 2021.
Jingkun Gao, Xiaomin Song, Qingsong Wen, Pichao Wang, Liang Sun, and Huan Xu. Robusttad: Robust. time series anomaly detection via decomposition and convolutional neural networks. arXiv preprint arXin:2002.09545, 2020.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-basod optimization for general algorithm configuration. International conference on learning and intelligent optimization, pp. 507-523. Springer, 2011.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: review. Dala mining and knowledgee discovery, 33(4):917-96; 2019.
Brian Kenji Twana and Seiichi Uchida. An empirical survey of data augmentation for time series classification with neural networks. Plas one, 16(7)xc0254841, 2021.
Kevin Jamieson and Ameet Talwalkar. Non-stochastie best arm identification and hyperparameter optimiza- tion. In Artificial intelligence and statistics, pp. 240-248. PMLR, 2016.
Nya Kaufman and Omri Azencot. Data representations' study of latent image manifolds. In International Conference on Machine Learning, pp. 15928 15945. PMLR, 2023.
Tya Kaufman and Omri Azencot. First-order manifold data augmentation for regression learning. In Forty-first International Conference on Machine Learning, ICML, 2024a.
Ilya Kaufman and Omri Azencot. Analyzing deep transformer models for time series forecasting via manifold learning. Trunsactions Machine Learning Research, TMLR, 2024b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR, 2015.
Alex Krizhevsky, Tya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2012.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 95 45-104, 2018.
Arthur Le Guennec, Simon Malinowski, and Romain Tavenard. Data augmentation for time series classification using convolutional neural networks. In ECML/PKDD workshop on advanced analytics and learning on temporal data, 2016.
Tracey Fileen KM Lee, YL. Kuah, Kee-Hao Leo, Saeid Sanei, Elfie Chew, and Ling Zhao. Surrogate rehabilitative time series data for image-basee based deep learning. In 2019 27th European Signal Processing Conference (EUSIPCO), pp. 1-5. IEEE, 2019.
Joseph Lemley, Shabab Bazrafkan, and Peter Corcoran. Smart augmentation learning an optimal data augmentation strategy. leee Access, 5:5858 5869, 2017.
Liam Li, Kevin Jamicson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan BaurTaur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. system for massively parallel hyperparameter tuning. Proceedings of Machine Learning and Systems, 2:230 246, 2020a.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M Robertson, and Yongxin Yang. DADA: Differentiable automatic data augmentation. arXiv preprint 2:Xiv:2003.03755 2020b.
Sungbin Lim, Iodoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast AutoAugment. Advances in Neural Information Processing Systems, 32, 2019.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Tow-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations, 2021.
Iaа Naiman, Nimrod Berman, Itai Pemper, Idan Arbiv, Gal Fadlon, and Omri Azencot. Utilizing image transforms and diffusion models for generative modeling of short and long time scries. Advances Neural Information Processing Systems, 2024a.
Ilan Naiman, N. Benjamin Erichson, Pu Ren, Michael W. Mahoney, and Omri Azencot. Generative modcling of regular and irregular time series data via koopman vaes. In The Twelfth International Conference on Learning Representations, 2024b.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Thai-Son Nguyen, Sebastian Stucker, Jan Nichues, and Alex Waibel. Improving sequence-to-sequence speech recognition training with on-the-fly data augmentation. In ICASSP 9020-2028 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7689 7693. LEEE, 2020.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Represenlations, 2023.
Liran Nochumsohn, Michal Moshkovitz, Orly Avner, Dotan Di Castro, and Omri Azencot. Beyond data scarcily: A frequency-driven framework for zero-shot forecasting. arXiv preprinl
2024.
Liran Nochumsohn, Hedi Zisling, and Omri Azencot. A multi-task learning approach to lincar multivariate forecasting, 2025. URL https://arxis: wwwp:////..bb.......////// httpps///eaatt..orrrraa/////..// 03571.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: neural basis ex- pansion analysis for interpretable time series forecasting. In 8th International Conference on Learning Representations, ICLR, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in newral information processing systems, 32, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181 1191, 2020.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Patrice Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition langent distance and langent propagation. In Neural networks: tricks of the trade, pp. 239 274. Springer, 1998.
Slawek Smyl and Karthik Kuber. Data preprocessing and augmentation for multiple short time series forecasting with recurrent neural networks. In 36th international symposium on forecasting, 2016.
Keyu Tian, Chen Lin, Ming Sun, Luping Zhou, Junjie Yan, and Wanli Ouyang. Improving aulo-augment via augmentation-we weight sharing. Advances in Neural Information Processing Systems, 33:19088-1908 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive lcarning? Advances in Neural Information Processing Systems, 33:6827-6839, 2020b.
Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. Data augmentation of wearable sensor data for parkinson's disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM international conference on multimodal interaction, pp. 216-220, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6438 6447. PMLR, 2019.
Jason W. Wei and Kai Zou. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, FMNI.P-IJCNIP,, pp.
2019.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkum Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. In Zhi-Hua Zhou (ed.), Proceedings of the Thirlieth International Joint Conference on Artificial Intelligence, IJCAI 2021, pp. 4653 4660, 2020.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decompasition transformers with auto-correlation for long-term scries forccasting. Advances in Neural Information Processing Systems, 34: 22419 22430,2230022 2021.
Jinsung Yoon, Daniel Jarrett, and Mihacla Van der Schaar. Time-series generative adversarial networks. Advances in neural informalion processing systems, 32, 2019.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, 2023.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR, 2018.
Yu Zheng, Zhi Zhang, Shen Yan, and Mi Zhang. Deep AutoAugment. In The Tenth International Conference Learning Representations, ICLR, 2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11106 11115, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. International Conference on Machine Learning, ICML, volume 162, pp. 27268 27286. PMLR, 2022.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR, 2017.
16

---

Published in Transactions on Machine Learning Research (02/2025)
A
Appendix
In what follows, we present additional details the transformations used in our method (App. B), hyperparameter values (App. C), complexity analysis of TSAA (App. D), and lastly, we provide extended result tables for every baseline alongside TSAA (App. E.1).
B Time-series transformations
In this section, we offer a detailed description of the different time-series transformations, which can be found Tab. 4 and depicted in Fig. with m = 0.85 compared to the original signal. For each of the transformations Trend scale, Scale, Seusonality scale, Window warping, we use two separate and independent. transformations demonstrate an increase decrease of the given effect.
Table 4: Our search space is composed of the following time-series transformations and their associated magnitude range.
Transformation
Description
Range of magni- tudes
Jittering
Adds white noise with controlled m (Um al., [0.0.1] 2017).
Trend scale
Multiplies the trend component by m.
[1,10], [0,1]
Scale
Multiplies the entire series by (Um et al., 2017).
[1,3,, [0.3,1]
Seasonality scale
Multiplies the seasonality component by m.
[1,3], [o.1]
Smooth
Performs low-pass filtering with a convolution kernel, where m controls the kernel size.
[0.11]
Noise scale
Performs high-pass filtering with a sccond-order con- volution kernel to extract the difference, which is then multiplied by m and added back to the original series.
[0.1]
Permulation
Exchanges two non-overlapping time intervals, such that the interval size is controlled by m. (Um et al., 2017).
[0.0.3]
Dynamic time stretching
Manipulates the length of different non-overlapping [1.5] time intervals, where m controls the manipulation magnitude. (Nguyen et al., 2020).
Window warping
Manipulates the length of the entire window (Um et al., 2017).
[1,1.5],
[0.5,1]
Mixup
Linearly interpolates between two series, m controls the contribution of each series (Zhang et al., 2018),
[0.0.5]
Idcutity
Returns the original series.
Noue
Flip
Flip series relative the value location by multi-
(0,1)
plying by (-1) (Iwana & Uchida, 2021).
Reverse
Change the relative location of the time steps to span {0.1) from start.
* marks a transformation implemented with min-max scaling to ensure equal relative changes.
17

---

Published in Transactions on Machine Learning Research (02/2025)
original jittoring
original
Trend Upscala
5
0
25
75
100
125
150
200
50
125
175
200
originall
original
Trend Downscate
Upscale
2.5
0.0
-2.5
25
75
100
125
150
175
200
50
75
1000
125
1.50
175
2.00
original
original Upscale
Seasonality Downscale
0-
25
75
100
125
150
175
200
25
50
75
100
125
150
175
2000
original
original
3
Smooth
0-
25
75
100
125
150
175
200
25
50
75
100
125
150
175
2.00
original
original
Noise
0-
25
75
100
125
150
175
200
50
75
100
125
150
175
2.00
original
original Reverse
Permutation
0-
25
75
100
125
150
175
200
50
75
1.00
125
1.50
175
2000
ori giinal
original
Window Upsize
0-
50
100
150
200
250
300
50
100
150
200
250
originall Mixcup
2
original2
original Window Downsize
25
75
100
125
150
175
200
50
75
100
125
150
175
200
Figure 6: We demonstrate the effect of different transformations when applied to the same example from the Electricity dataset. Blue and orange represent the original signal and its transformed version, respectively.
18

---

Published in Transactions on Machine Learning Research (02/2025)
C
TSAA hyperparameters
We detail in Tab. 5 the hyperparameter values we used in the evaluation of TSAA.
Table 5: Hyperparameter values used in the evaluation of TSAA.
Model Parameters TSAA
exploration
T maa
trials fraction
range
7
resource type 8
k
100
0.3
(0.1
2
3 1
epoch
0.5
Complexity
A straightforward upper bound of our method is given by O((1- 3)KT evaluated in epochs, where K, (1 - 9), and Tmaz correspond to the number active epochs the model trains for, the fraction of K to be used for fine-tuning, and the maximum number of trials, respectively. However, for TSAA practically reach such an upper bound when K 2 would require each trial outperform the preceding trials, thus avoiding being pruned by ASHA. This setup is very unlikely for the following reasons: (1) fraction size 0.3 of Traz of starting trials are manually dedicated to random search to promote aggressive exploration at the start. (2) To the best of our knowledge Bayesian Optimization does not guarantee monotonic improvement and inherently promoles exploration (Bergstra et al., 2011). (3) (0, w²) is not promised to be convex with respect to 0, making even more difficult to attain a monotonic improvement. In our empirical evaluations, we have observed different operations with different transformations being selected, as opposed to a single policy being repeatedly selected, which strengthent hens our claim. Further, we would like share solid example from our empirical experiments with FEDformrer-w applied to Electricity with horizon of 336. This setup single RTX3090. In the
is considered resource expensive given setup, K was set to 8 and 3 0.5 as defined in the hyperparameter table Tab. 5. Therefore, according to the bound mentioned above, the total number of epochs would be 400 epochs. However, the use of ASHA with Bayesian optimization reduced it to 170, proving the computational impact. For comparison, applying a naive approach without ASHA and will result in a total number of 800 training epochs.
epoch may take long 20 minutes
E Extended results
In this section, we provide the performance of TSAA for each individual model featured in Tab. and Tab. in the main text. We can observe from the multivariate results that Informer benefits the most from TSAA, with the 720 horizon in particular. On the other end, TSAA struggles achieve significant improvements in FEDformer-f, especially for the Exchange dataset. In the univariate setting, unlkke the multivariate setting, the models NBEATS-G and NBEATS-T gain large improvement rates in Exchange thanks to TSAA. Overall, we can conclude that all models and datasets exhibit a degree of improvement with the support of TSAA.
19

---

Published in Transactions on Machine Learning Research (02/2025)
E.1 TSF results
Tabs. 6, 7, 8, 9, 10, 11, 12, 13, 14 detail the extended results for every baseline architecture and dalaset we considered in the main text.
Table 6: Informer multivariate
Table 7: Autoformer multivariate
Informer MSE
TSAA
Autoformer MSE
TSAA
MAE
MSEI
MAE]
MSE%1
MAE%
MAE
MSET
MAE]
MSEX1
MAE
Exchaango Elccticity ETTo2
192
1.054
0.545
0.588
0.808
0.224
0.270
0.355
0.321
58.899
74.383
56,064
45.108
E7T
96
192
0.289
0.231
0.346
0.310
0.269
0.211
0.327
0.293
6,920
8.658
5.491
5.484
336
1.523
0.948
0.304
0.374
80.039
60.549
336
0.341
0.375
0.322
0.359
5.572
4.267
720
3.878
1.474
0.398
0.435
89.737
70.488
0.444
0.434
0.410
0.407
7.058
6.221
96
0:336
0.416
0.324
0.407
3.571
2.163
Ebetticity
96
0.200
0.316
0.188
0.302
6.000
4,430
192
0.300
0:441
0.336
0.419
6.667
4.989
192
0.217
0.326
0.221
0.328
-1.843
-0.613
336
0.356
0.139
0.347
0.429
2.526
2.278
336
0.258
0.356
0.252
0.352
2.326
1.124
720
0.386
0.462
0.381
0.4488
1.295
0.886
720
0:261
0.363
0.248
0.351
4.981
3.306
1.029
0.569
50. 50.243
Exchrape
96
0.150
0.261
0.143
0.272
4.667
3.203
192
1.155
0.867
0.791
0.696
31.515
19.723
192
0.318
0.409
0.344
0.425
-8176
-3.912
336
1.589
1.011
1.040
0.763
34.550
24.530
336
0.713
0.616
0.742
0.641
4.067
4.058
720
3.011
1.431
1,213
0.842
59.714
41.160
720
1.246
0.872
1.997
1.170
60.277
-34.174
96
0.744
0.420
0.723
0.408
2.823
2.857
96
0.615
0.384
0.602
0.376
2.114
2.344
Theflle
192
0.753
0.414
2.390
2.817
Weather Traflie
0,670
0:121
0.663
0.416
1,045
333
0.876
0.495
0.811
0.462
7.420
6.667
336
0.633
0.392
0.627
0.387
1.260
1.276
720
1.011
0.578
0.985
0.566
2.572
2.076
0.658
0.402
0.662
0.405
-0.608
-0.746
Woather
96
0.315
0.382
0.180
0,256
42.857
32.984
96
0.259
0.332
0.216
0,292
16.602
12.048
192
0.428
0.440
0.253
0.331
401.888
26,281
192
0.298
0.356
0.278
0.336
6.711
5.618
336
0.620
0.296
52.-558
34.888
0.357
0.394
0.341
4.482
3.299
720
0.975
0.722
0.392
0.426
59.795
40.997
720
0.122
0.431
0.397
0.410
5.924
4.872
5.319
1.582
5.313
1.559
0.673
1.154
24
3.5.1
1.305
3.565
1.302
-0.151
0.230
5.203
1.572
5,280
1.581
-1.096
-0,573
36
2.834
1 1.094
2.754
1.068
2.823
2.377
5.286
1.594
5.415
1.623
-2 22.40
-1.819
48
2,889
1.122
2.856
1.114
1.142
0.713
6.419
1.620
1.593
2.196
1.667
60
1.118
2,896
1.118
-0.284
0.000
Table 8: FEDformer-f multivariate
Table 9: FEDformer-w W multivariate
FEDformer-F
TSAA
FEDfarmer-w
TSAA
MSE
MAB
MSEL
MAEI
MSE%
MAE%
MSE
MAE
MSEJ
MAE]
MSEXt
MAEX
ETTo2
0.189
0.282
0.187
0.274
1.058
2.837
E77m2
96
0:205
0:290
0.199
0.280
2.9/7
3.448
192
0.258
0.328
0.255
0.314
1.163
S.681
192
0.270
0.329
0.255
0.311
5.556
4.559
336
0.325
0.363
0.311
0,350
3.715
5.581
336
0.328
0.364
0.316
0.352
3.659
3.297
720
0.121
0.406
0.403
1.471
4.276
0.133
0.125
0.410
0.401
5.312
1.911
1 Exchaange Electriiity
96
0.185
0.300
0.185
0.300
0.000
0,000
Electriitty
96
0.196
0.310
0.183
0.297
6.633
4.194
192
0.201
0.316
0.201
0.316
0.000
0.000
192
0.199
0.310
0.195
0.309
2.010
0.323
336
0.214
0.329
0.214
0.329
0.000
0,000
336
0.217
0.331
0.208
0.323
4.147
3.293
720
0.246
0.353
0.246
0.353
0.000
0.000
720
0.248
0.357
0.238
0.348
4.032
2.521
0.142
0.271
0.149
-1.930
-2.211
Exchange
0.151
0.282
0.141
0.272
4.636
3.546
192
0.278
0.383
0.273
0.380
1.799
0.763
192
0,284
0.391
0,270
0.378
4.930
3.325
336
0.450
0.497
0.517
0.540
-14.859
-8.652
336
0.442
0.493
0.450
0.504
-3.846
-2,231
720
1.161
0,841
2.440
1.343
-106.606
-39.691
720
1.227
0.865
1.952
1.167
-59.087
-34.44
96
0.577
0.361
0.577
0.362
(1.000
-0.277
96
0.584
0.368
0.565
0.352
3.253
4.348
192
0.379
0.371
1.475
2.111
Weather Trifle
0.506
0.375
0.571
0.351
4.195
336
0.623
0.385
0.619
0.383
0.612
0.519
336
0,590
0.365
0.584
0.359
1.017
1.644
720
0.632
0.388
0.632
0.388
U.000
0.000
720
0.613
0.375
0.607
0.368
0.979
1.867
Weother
0.236
0.316
0.207
0.285
12.288
9.510
96
0.269
0.347
0.213
0,291
20,818
16.138
192
0.273
0.333
0.252
0.311
7.692
6.607
192
0.357
0.412
0.274
0.339
23.249
17.718
336
0.332
0.371
0.313
0.355
5.728
4.313
0.422
0.456
0.335
0.379
20.616
16,886
720
0.108
0.418
0.382
0.395
6.373
5.502
720
0.629
0.570
0.406
0.423
35.453
25.789
3.208
1.257
3.150
1.219
3.611
3.023
24
2.752
1.125
2.700
1.128
-0.291
0.178
36
2.618
1.068
2.578
1.049
2.611
1.779
36
2.318
0.950
2,362
0.964
-1.898
-0.408
2.615
1.072
2.609
1.069
0.229
0.280
48
2.328
1.006
2,264
0.988
2.749
1.789
2.866
1.158
2.805
1.140
2.128
1.554
60
2.574
1.081
2.520
1.062
2.098
1.758
20

---

Published in Transactions on Machine Learning Research (02/2025)
Table 10: Informer univariate
Table 11: Autoformer univariate
Informer
TSAA
Autoformer MSR
TSAA
MSE
MAE
MSEL
MAEL
MSEX1
MAEX
MAE
MSEI
MAEI
MSEX1
MAE%
Thelli Exohargge Flortriikt ETTm22
0.085
0.225
0,085
0.226
-0.444
ETTm2
0.123
0:270
0.106
0.250
13.821
7.407
192
0.130
0.282
0.128
0.279
1.538
1.064
192
0.141
0.289
0.126
0.273
10.638
5.536
336
0.161
0.314
0.164
0.317
-0.955
336
0.170
0.119
0.139
0.290
16.235
9.091
720
0.221
0.373
0.221
0.371
0.000
0.536
720
0.206
0.353
0.187
0.338
9.223
4.249
96
0.261
0.367
0.245
0.356
6.130
2.997
Exchange Eleetrioity
96
0.454
0.508
0.388
0.456
14.537
10.236
192
0.285
0.386
0.297
0:401
-4.211
-3.886
192
0.511
0.532
0.463
0.509
9.393
4.323
336
0.324
0.417
0:433
0-485
-33.642
-16.307
336
0.739
0.651
0.493
0.517
33. 288
20.584
720
0.632
0.612
0.707
0.611
-5,229
720
0.673
0.532
0.549
20.951
96
0.490
0.554
0.149
0.307
69.592
44.585
96
0.149
0.308
0.146
0.295
2.013
4.221
192
11.90
0.721
0.260
0.102
67.089
44.244
192
0.290
0.415
0.298
0,116
-2.759
-0.211
336
2.146
1.223
0.744
0.671
65.331
45.
336
0.708
0.862
0.712
0.643
-0.565
2.870
720
1.447
1.008
0.527
0.5944
63.580
41.071
720
1.322
0.892
2,703
1,300
-104.155
-55.830
0.262
0.318
0.254
0.3112
3.053
1.724
0.266
0.372
0.248
0.351
6.767
5.645
192
0:294
0.376
0.284
0.366
3.401
2.394
Weathher Troffte
192
0.272
0.379
0.244
0.353
8.456
6,860
336
0.3088
0.390
0.300
0.351
2.597
1.338
336
0.261
0.374
0.239
0.347
8,129
7.219
720
0.364
0.430
0.326
0.409
10.440
7.045
720
0.269
0.372
0.237
0.343
11.896
7.796
96
0.005
0.048
0.000
01.44
40.000
10.417
96
0.009
0.078
0.003
0.039
66.667
50.000
Woather
192
0.004
0.051
0.003
0.038
25.000
25.490
192
0.009
0.068
0.003
0.040
66.667
41.176
336
0.003
0.043
0.004
0:047
-33.333
-9.302
396
0.006
0.058
0.004
0.046
33.333
20,690
720
0.004
0.049
0.003
0.043
25.000
12.245
720
0.007
0.063
0.004
0.046
42.857
26.984
Table 12: FEDformer-f f univariate
Table 13: NBEATS-I univariate
FEDformer-f
TSAA
NBEATS-I
TSAA
MSE
MAB
MSEI
MAEI
MSE%
MAE%
MSE
MAE
MSEL
MAEL
MSB%
MAEX
ETTu2
96
0.068
0.198
0.068
0.198
0.000
0,000
Electricity ETTm2
96
0.080
0.213
0.075
0.202
6.250
5.164
192
0.096
0.238
0.096
0.238
0.000
0,000
192
0.103
0.102
0.237
0.971
1.250
336
0.138
0.286
0.140
0.290
-1.449
-1.399
336
0.162
0.312
0.171
0.321
-5.556
-2.885
720
0.189
0.335
0.190
0.336
-0.529
-0.299
720
0.199
0.3417
0.231
0.376
-16.080
-8.357
Electricitt
0.244
0.364
0.244
0.356
0.000
2.198
0.326
0.402
0,270
0.360
17.178
10.448
192
0.276
0.362
0.277
0.381
-0.362
0,262
192
0.350
0.417
0.277
0.368
20.867
11.751
336
0.317
0.432
0.347
0.432
336
0.393
0.140
0.310
0.394
21.120
10.455
720
0.408
0.473
0.408
0.473
0.000
0.000
720
0.458
0.419
0.394
0.453
13.974
7.551
Trafle Exchange
0.133
0,284
0.145
0,293
-9.023
-3.169
Exchauge
0.210
0.311
0.093
0,238
55.714
30.814
192
0.292
0.419
0.319
0.434
-7.192
-3,580
192
1.130
0.840
0.215
0.352
80.973
$8.095
336
0.477
0.532
0.525
0.505
-20.545
-11.842
336
1.587
1.047
0.582
0.572
66.478
45.368
720
1.304
0.882
2,852
1.453
-128.712
-64.739
720
0.870
0.747
0.744
0.665
14.483
10.977
96
0.210
0.318
0.191
0.292
9.048
8.176
96
0.181
0.268
0.183
0.270
-1.105
-0.746
192
0.206
0.311
0.197
0.301
4.369
3.215
Teafli
192
0.177
0.263
0.176
0.263
0.565
0.000
336
0.217
0.322
0.204
0.309
5,991
4.037
336
0,150
(1,271
0,180
0.270
0,000
0.369
720
0.243
0.342
0.219
0.318
9.877
7.018
720
0.226
0.316
0.224
0.311
0.885
1.582
0.009
0.073
0.033
77.778
54.795
Wee Lheeh Lieer
0.003
0.001
0.024
66.667
45.455
West
192
0.007
0.067
0.002
0.034
71.429
49.254
192
0.004
0.046
0.001
0.027
75.000
41.304
336
0.006
0.062
0.002
0.036
66.667
41.935
336
0.004
0.048
0.004
0.035
0.000
27.083
720
0.006
0.060
0.002
0.039
66.667
35.000
720
0.001
0.019
0.002
0.034
50.000
30.612
Table 14: NBEATS-G G univariate
NBEATS-G
TSAA
MSE
MAE
MSEL
MAEL
MSE3%
MARE%
ETim2
0.219
0.071
0.192
11.250
8.571
192
0.110
0.250
0.109
0.246
0.909
1.600
333
0.172
0.320
0.176
0.323
-2.326
-0.938
720
0.201
0.353
0.218
0.366
-8.458
-3.683
Electricity
SKG
0.324
0.397
0.263
0.354
18.827
10.831
192
0.120
0.278
0.368
23.416
12.381
336
0.392
0.445
0.350
0.422
10.714
4.740
720
0.489
0.502
0.378
0.447
22.699
10.956
Exchange
0,223
0.351
0.083
0.236
58.296
32.764
192
0.783
0.675
0.2155
0.352
72.542
47.852
336
24622
1.266
1.167
0.861
55-492
31.991
720
2,588
1.303
1.687
1.033
34.815
20.721
0.159
0.240
0.158
0.239
0.629
0.417
Tinific
192
0,181
0,261
0,160
0.243
11.602
7.955
333
0.155
0.239
0.156
0.244
-0.645
-2.092
720
0.212
0.304
0.189
0.279
10.849
8.224
Weasber
96
0.003
0.043
0.002
0.031
33.333
27.907
192
0.004
0.047
0.002
0.028
50.000
40..26
333
0.005
0.051
0.004
0.037
20.000
311
720
0.004
0.018
0.002
0.036
50.000
25.000
21

---

Published in Transactions on Machine Learning Research (02/2025)
E.2 Main results with standard deviation
The following tables Tabs. 15, 16 augment the tables presented in Sec. 5 in the main text with standard deviation measures computed over three different seed numbers.
Table 15: Multivariate long-term time-series forecasting results with the standard deviation.
Informer
Autoformer
PEDformer-w
FEDformer-f
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.545 + 0.024
0.588 + 0.014
0.231 0.004
0.31 0.0
0.205 0.001
0.29
0.0
0.189
0.001
0.282
0.001
ETTm2
192
1.054 = 0.044
0.808 + 0.018
0.289
0.013
0.346 + 0.01
0.27 = 0.0
0.329
0.0
0.258
0.002
0.326 = 0.001
336
1.523
1 0.036
0.948 1 0.014
0.341 1 0.008
0.375 1: 0.004
0.328 1 0.0
0.364
1 0.0
0.323
0.004
0.363 = 0.004
720
3,878
1 0.072
1.474 1 0.004
0.444 1 0.005
0.434 1 0.005
0.433 1 0.005
0.425 1 0.005
0.425
0.008
0.421 = 0.001
Electricity
96
0.336 = 0.013
0.416 + 0.007
0.2 4 0.007
0.316 + 0.007
0.196 +: 0.002
0.31 + 0.002
0.185
0.001
0.3 = 0,001
192
0.36 + 0.015
0.441 + 0.012
0.217 + 0.0
0.326 + 0.002
0.199 + 0.001
0.31 + 0.002
0.201
0.007
0.316
0.008
336
0.356 + 0.007
0.439 1 0.006
0.258 1: 0.024
0.356 1 0.013
0.217 0.001
0.334 1 0.001
0.214
0.002
0.329
0.002
720
0.386
0.012
0.452 + 0.01
0.261 0.002
0.363 = 0.004
0.248 0.004
0.357 = 0.004
0.246 0.002
0.333 0.0
Exchange
96
1.029
0.038
0.809 + 0.02
0.15 + 0.005
0.281 + 0.005
0.151 + 0.004
0.282 + 0.004
0.142
0.004
0.271 = 0.004
192
1.155 = 0,038
0.867 1 0.015
0.318 + 0.012
0.409 = 0.008
0.284 + 0.003
0.391 + 0,002
0.278 = U. 0.003
0.383 = 0.001
336
1.589 = 0.048
1.011 = 0.011
0.713 + 0.516
0.616 = 0.243
0.442 + 0.003
0.493 = 0.001
0.45 = 0.003
0.497 = 0.001
720
3.011 + 0.302
1.431 + 0.067
1.246 +: 0.007
0.872 + 0.002
1.227 +: 0.024
0.868 + 0.011
1.181 0.018
0.841 = 0.002
96
0.744 + 0.006
0.42 1 0.006
0.615 0.015
0.384 1 0.009
0.584 0.005
0.368 1 0,005
0.577
0.001
0.361
0.002
Traffic
0.753
0.01
0.426 + 0.011
0.67 0.068
0.421 + 0.045
0.594 0.007
0.372 + 0.007
0.61 0.003
0.379
0.004
336
0.876
0.024
0.495 + 0.013
0.635 0.027
0.392 + 0.014
0.59 0.001
0.365 + 0.002
0.623 0.005
0.385
0.004
720
1.011 = 0.032
0.578 = 0.018
0.658 + 0.011
0.402 = 0.01
0.613 + 0.006
0.375 + 0.003
0.632 = 0.006
0.388 = 0.007
Weather
96
0.315 = 0.004
0.382 + 0.004
0.259 + 0.014
0.332 + 0.011
0.269 + 0.011
0.347 + 0.008
0.236 = 0.024
0.316 = 0.028
192
0.428 1 0.005
0,449 T 0.012
0.298 1 0.012
0.356 1 0.012
0.357 1 0.001
0.412 1 0,002
0.273 0.013
0.333 = 0.012
336
0.62 = 0.027
0.554 + 0.009
0.357 +: 0.01
0.394 + 0.011
0,422 + 0.008
0.456 1 0,006
0.332
0.016
0.371 = 0.016
720
0.975 + 0.035
0.722 + 0.009
0.422 + 0.011
0.431 + 0.01
0.629 + 0.008
0.57 + 0.004
0.408
0.009
0.418 - 0.008
24
5.349
0.229
1.582 1 0.052
3.549 0.33
1.305 1 0.068
2.752
0.023
1.125 1 0.005
3.268 1 0.04
1.257 - 0.014
I1.I
30
5.203
= 0.129
1.572 + 0.029
2.834
0.166
1.094 + 0.032
2.318
0.017
0.98 + 0.006
2.648 = 0.034
1.0G8 = 0.009
48
5.286
0.049
1.594
+ 0.018
2.889
0.178
1.122 + 0.033
2.328
0.043
1.006 + 0.013
2.615 0.019
1.072 = 0.004
60
5.419 + 0.103
1.62 1 0.019
2.818 + 0.157
1.118 1 0.039
2.574 + 0.052
1.081 1 0.016
2.866 = 0.031
1.158 = 0.008
TSAA
MSE
MAE
96
0.187 1 0.000
0.274 1 0.000
ETTm2
0.255
0.001
0.314 = 0.001
336
0.304
0.009
0.350 + 0.002
720
0.308 = 0.007
0.403 1 0.002
Electricity
96
0.183= = 0.001
0.297 # 0.001
192
0.195 = 0.001
0.309+ # 0.002
336
0. 0.208 1 0.002
0.323 T 0.002
720
0.238 = 0.003
0.348 + 0.003
Exchange
96
0.143
0.012
0.272 + 0.012
0.27
0.002
0.378 1 0.002
336
0.459
= 0.007
0.504 + 0.007
720
1.213 1 0.056
0.842 1 0.008
96
0.565 + 0,005
0.352 + 0.004
Traffic
192
0.572 = 0,002
0.351 + 0.001
336
0.584 + 0.004
0.359 1 0.005
720
0.607
1 0.002
0.368 T 0.002
96
0.18
0.024
0.256 + 0.024
Weather
192
0.252- + 0.001
0.311 + 0.002
336
0.296 + 0.001
0.355 # 0.005
720
0.382 = 0.006
0.395 # 0.007
24
2.76 + 0.063
1.123 T 0.016
36
2.362
0.024
0.984 1 0.008
48
2.264 0.074
0.988
+ 0.012
60
2.52
0.064
1.062 + 0.022
22

---

Published in Transactions on Machine Learning Research (02/2025)
Table 16: Univariate long-term time-series forecasting results with the standard deviation.
Informer
Autoformer
FEDformer-f
N-BEATS-I
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTm2
96
0.085 = 0.004
0.225 + 0.006
0.123 + 0.017
0.27 + 0.018
0.068 + 0.001
0.198 + 0.002
0,08
0.003
0.213 = 0,006
192
0.13 = 0.007
0.282 = 0.008
0.141 + 0.01
0.289 = 0.01
0.096 + 0.001
0.238 = 0.001
0.103
0.004
0.24 0.006
336
0.161 1 0.008
0.314 1 0.006
0.17 0.046
0.319 1 0.042
0.138 0.013
0.286 1 0.014
0.162
0.009
0.312
0.009
720
0.221
1 0.006
0.373 1 0.007
0.206 0.02
0.353 1 0.017
0.189 0.002
0.335 1 0.002
0.199
0.007
0.347 = 0.007
Electricity
96
0.261
0.005
0.367 + 0.002
0.454 0.014
0.508 + 0.014
0.244 0.001
0.364 + 0.002
0.326
0.006
0.402 + 0.004
192
0.285 + 0.006
0.386 + 0.003
0.511 + 0.05
0.532 + 0.027
0.276 +: 0.004
0.382 + 0.004
0.35 0.008
0.417 = 0.005
336
0.324 = 0.004
0.417 # 0.004
0.739 + 0.086
0.651 = 0.012
0.347 + 0.007
0.432 = 0.006
0.393 = 0. 0.008
0.44 = 0.003
720
0.632 = 0.049
0.612 = 0.028
0.673 + 0.082
0.61 = 0.037
0.408 + 0.025
0.473 = 0.015
0.458 = 0.008
0.49 = 0.002
Exchange
96
0.49 + 0.065
0.554 + 0.034
0.149 + 0.004
0.308 + 0.006
0.133 + 0.015
0.284 + 0.018
0.21
0.047
0.344 - 0.036
192
0.79
0.039
0.721 + 0.016
0.29 0.005
0.415 = 0.004
0.292 0.002
0.419 + 0,003
1.13 0.392
0.84 = 0.153
336
2.146
0.25
1.223 + 0.084
0.708 0.108
0.662 + 0.053
0.477 0.002
0.532 + 0.002
1.587
0.219
1.047 = 0.077
720
1.447
1 0.105
1. 1.008 1 0.038
1.324 0.005
0.892 1: 0.002
1.304 0.003
0.882 0.0
0.87
0.088
0.747 = 0.015
96
0.262 + 0,006
0.348 1 0.006
0.266 +: 0.005
0.372 1 0.01
0.21 1: 0.006
0.318 + 0.009
0.181 = 0.004
0.268 = 0.005
Traffic
192
0.294 + 0.004
0.376 + 0.006
0.272 + 0.014
0.379 + 0.01
0.206 + 0.01
0.311 + 0.006
0.177 - 0.001
0.263 = 0.001
336
0.308 = 0,007
0.39 + 0.002
0.261 + 0.016
0.374 + 0.016
0.217 + 0.005
0.322 + 0.0
0.18 = 0.005
0.271 = 0,006
720
0.364 = 0.018
0.44 = 0.017
0.269 + 0.012
0.372 = 0.005
0,243 + 0.021
0.342 = 0.021
0.226 0.003
0.316 = 0.004
96
0.005
0.001
0.048 + 0.005
0.009 0.002
0.078 + 0.009
0.009 0.004
0.073 + 0.014
0.003
0.001
0.044 = 0.006
Weather
192
0.004 1: 0.0
0.051 1 0.001
0.009
0.002
0.068 1 0.001
0.007
0.002
0.067 1 0.008
0.004 T 0.0
0.046 = 0.003
336
0.003
= 0.001
0.043 + 0.004
0.006 0.001
0.058 + 0.006
0.006
0.001
0.062 + 0.007
0.004
0.001
0.048 = 0.004
720
0.004
0.002
0.049 + 0.007
0.007 + 0.001
0.063 + 0.004
0.001 0.006
0.06 + 0.007
0.004 + 0.0
0.049
0.003
N-BEATS-G
TSAA
MSE
MAE
MSE
MAE
96
0.08 = 0.005
0.21 + 0.007
0.068 + 0.001
0.192 + 0.002
ETTm2
192
0.11 + 0.004
0.25 1 0.005
0.096 1: 0.001
0.237 1 0.004
336
0.172 + 0.007
0.32 1 0.007
0.139 0.005
0.29 1 0.005
720
0.201
0.008
0.353 + 0.008
0.187 0.008
0.336 + 0.001
Electricity
96
0.324 1 0.005
0.397 1 0.002
0.244 1 0.006
0.354 1 0.012
192
0.363 = 0.005
0.42 # 0.003
0.277 + 0.003
0.368 + 0.003
336
0.392 + 0.002
0.443 + 0.006
0.31 + 0.006
0.394 + 0.005
720
0.489 = 0.013
0.502 + 0.005
0.378 + 0.026
0.447 + 0.012
Exchange
96
0.223 = 0.046
0.351 = 0.045
0.093 0.008
0.236 = 0.007
192
0.783
0.203
0.675 1 0.085
0.215 0.035
0.352 T 0.017
336
2.622
1.07
1.266 T 0.23
0.532
0.045
0.572 1 0.006
720
2.588
0.11
1.303 + 0.019
0.527 + 0.047
0.594 + 0.019
96
0.159 + 0.001
0.24 + 0.002
0.158 + 0.001
0.239 + 0.001
Traffic
192
0,181 = 0.005
0.264 = 0.001
0.16 = 0.0
0.243 = 0.001
336
0.155 = 0.001
0.239 + 0.001
0.156 + 0.004
0.244 + 0.006
720
0.212 1 0.003
0.304 T 0.003
0.189 0.002
0.279 T 0.003
96
0.003 + 0.0
0.043 + 0.002
0.001 + 0.0
0.024 0.0
Weather
192
0.004 = 0.001
0.047 = 0.004
0.001 + 0.0
0.027 = 0.001
336
0.005
1 0.001
0.054 1 0.004
0.002
1 0.0
0.035 1 0.004
720
0.004 + 0.0
0.048 + 0.002
0.002 + 0.0
0.034 = 0.002
E.3
Comparing with Crossformer, PatchTST, and iTransformer models
Multivariate results for TSAA using the baselines: Crossformer, PatchTST, and iTrranffrmee. Each result reports an average of three random seeds, with a lookback of 96. The results are presented in Tabs. 17, 18, and 19.
E.4 Autoformer and FEDformer augmented with the best augmentations
We show in Tabs. 20 and 21 a comparison between TSAA and Autoformer and FEDformer, respectively. Specifically, we train from scratch Autoformer and FEDformer only using the best augmentations found by TSAA. Overall, the results are inconsistent, highlighting that in some cases, these augmentations yield strong
23

---

Published in Transactions on Machine Learning Research (02/2025)
Table 17: Crossformer multivariate
Table 18: PatchTST multivariate
Crasaformer
TSAA
PSt
TSAA
MSE
MAE
MSEL
MAEL
MSE%1
MAEX
MSE
MAE
MSEL
MAE]
MSEXT
MAE
FTTn2
96
0.263
0.35
0.197
0.283
25.095
19.143
96
0.175
0.269
0.176
0.256
0.0
1.158
192
0.476
0.492
0.273
0.336
42.847
31.707
0.212
0.303
0.24
0.301
0.826
0.86
336
0.895
0.638
0.335
0.378
60.819
40.752
336
0.303
0.341
0.302
0.339
0.33
0.587
720
3.293
1,26
0.397
0.1
87.944
68.254
720
0:101
0.399
0.1
0.396
0.249
0.752
0.144
0.243
0.141
0.24
2.083
1.235
96
0.187
0,269
0.187
0.269
0.0
0.0
ECL.
192
0.165
0.264
0.157
0.2654
4.848
3.788
ECL
192
0.191
0.274
0.191
0.274
0.0
0.0
0.287
0.175
4.181
0.206
0.345
720
0.249
0.332
0.208
013302
16.466
9.036
720
0.248
0.323
0.248
0.324
0.0
-0.31
Traffie Exchangs
96
0.187
0.339
0.307
0:416
-61.171
-22.711
Taeffee Exchange
96
0.08
0.197
0.088
0.205
-10.0
-1.061
192
0.61
0.623
1.022
0.757
-67.541
-21509
0.168
0.291
0.17
0.292
-1.19
-0.341
336
1.023
0.803
0.941
0.724
8.016
9.838
336
0.313
0.403
0.312
0.403
0.319
0.0
720
1.295
1.621
-25.174
0.818
0.679
0.816
0.679
96
0.684
0.399
0.68
0.4
0.585
-0,251
96
0.523
0.330
0.521
0.337
0.382
0.59
192
0.94
0.546
0.935
0.547
0.532
-0.183
192
0.519
0.333
0.517
0.332
0.385
0.3
336
1.127
0.681
1.059
0.635
6.034
3.933
336
0.529
0.337
0.528
0.336
0,189
0.297
720
1.349
0.792
1.345
0.796
0.297
-0.505
720
0.563
0.355
0.561
0.354
0.355
0.282
0.164
0.236
0.226
4.237
West heer
0.187
0.225
0.184
0.222
1.604
L.333
Wost
192
0.217
0.301
0.228
0.279
-5.069
7.309
192
0.233
0.264
0.232
0.263
0.429
0.379
336
0:266
0.325
0,258
01.315
4.887
3.077
336
0.284
0.3
0.283
0.299
0.352
0.333
720
0.383
0.396
0.31
0.35
19.06
11.616
720
0
0.317
0.355
0.346
0,281
0.288
24
4.324
1.413
4.267
1.401
1.318
0.849
24
2.333
0.975
2.079
0.938
10.9664
3.795
11
1.565
L/122
4.261
6.659
3.98
36
2,259
0.962
1.781
0.878
21.027
8.732
48
4.603
1.477
3.741
1.271
18.727
11.552
48
2.41
0.992
2.157
0.949
10.498
4.335
1.435
4.193
1.362
8.27
5.087
60
2.297
0:965
1.947
0.918
15.237
4.87
Table 19: iTTransformer multivariate
iTrnnformmr
TSAA
MSE
MAE
MSEL
MAE]
MSE%%
M
F77m2
96
0.184
0.27
0.18
0.259
2.174
4.074
192
0.312
0.246
0.302
1.992
3.205
336
0.315
0.351
0.311
0.345
1.27
1.709
720
0:41
0.405
0.403
0.397
1.707
1.975
0.147
0.147
0.0
TCC.
192
0.163
0.255
0.163
0.255
0.0
0.0
336
0.176
0.269
0.738
720
0.208
0.298
0.209
0.297
-0.481
0.336
Exchhange
96
0.088
0.208
0.09
0.21
-2.273
-0.822
192
0.18
0.303
0.181
0.304
-0.556
-0.33
336
0.387
0.42
0.34
0-424
-0.89
-0.952
0.564
0.906
0.723
-2.699
96
0.393
0.267
0.393
0.267
0.0
0.0
Traffic
192
0.412
0.277
0.412
0.275
0.0
0.722
336
0.423
0.282
0.423
0.281
0.0
0.355
720
0.459
0.3
0L.559
0,299
0.0
0.333
Weather
0.215
0.173
0.212
1.395
192
0.227
0:261
0.226
0.259
0.441
0.766
336
0.282
0.3
0.282
0.3
0.0
0.0
720
0.357
0.349
0.36
0.352
-0.84
-0.86
24
2.775
1.11
2.565
1.065
7.568
4.054
2.668
L.061
2.348
0.996
11.994
6.126
48
2.663
1.082
2.294
0.997
13.857
7,856
2.921
1.159
2.651
1.098
9.243
5.263
results, but in other cases, results are inferior. In comparison, TSAA is consistent across all models and datasets.
E.5 Limitations of TSAA
TSAA shows less favorable results when applied to the Exchange dataset. While several factors may contribute to this underperformance, we would like to focus on what we believe is the key distinction of this dataset compared to others, supported by additional experiments. The Exchange dataset captures foreign exchange prices between major currency pairs, categorizing it as financial data similar to stock market data. It is widely accepted that financial data often exhibits random fluctuations, characteristic of a random walk (RW)
24

---

Published in Transactions on Machine Learning Research (02/2025)
Table 20:
Autoformer: TSAA compared to the direct application of the best-performing transformations
trend downscaling (TD), jittering, mixup, and smoothing. Each transformation was deployed with fixed magnitude of 0.5.
TD MSE
TD MAB
Jittering MSE
Jittering MAE
Mixap MSE
Mixup MAB
Smooth MSE
Smooth MAE
TSAA MSE
TSAA MAE
pro__een
96
0.253
0.324
0.265
0.33
0.236
0.317
0.278
0.337
0.211
0.293
0.29
0.344
0,293
0.269
336
0.353
0.384
0.419
0.418
0.341
0.371
0.134
0.37
0.322
0.359
0.473
0.452
0.433
Avg.
0.317
0.377
0.362
0,586
0.32
0.30
0.334
0.37
0.303
0,346
0.632
0.402
10623
0.385
0.633
0.4011
0.634
0.399
0,602
0.316
192
9.630
0.415
0/667
0:416
0.742
0.463
0.633
0.401
0.063
0-416
330
0.615
0.402
0.631
0.398
0.683
0.129
0.654
0:208
0,627
0.387
720
0.679
0.416
0.605
0-407
0.744
0.455
0.661
D-406
0.602
0.405
Avg.
0.634
0.409
0.647
0.401
0.437
0.645
0.403
0.639
0,396
96
0.212
0.317
0.259
0.328
0.222
0.297
0.216
0.32
0.216
0,292
192
0.295
0.355
0.124
0.375
0,256
0.346
0.331
0.382
0.278
0.336
3396
0.367
0.358
0.381
0.327
0.367
0.379
0.408
0.341
0.181
720
0.436
0.443
0.428
0.43
0.396
0.411
0.449
0.440
0.107
0.41
Avg.
0.335
0.379
0,312
0,381
0,308
8.898
1.284
3.056
1.311
3:4653
T,280
3.565
1.302
36
2,897
1.117
2,878
1.0966
3.024
1,135
2.812
1 1.087
2.754
1.068
2.94
2.856
1.114
60
2.85
1.135
2,849
1.118
3,106
1,198
2.805
1.114
2.826
1.118
Ang.
3.081
1.644
Table 21: FEDformer: TSAA compared to the direct application of the best-performing transformations
trend downscaling (TD), jittering, mixup, and smoothing. Each transformation was deployed with a fixed magnitude of 0.5.
TD
TD Jittering MSE
Jistering MAE
Mixcp MSE
Mixup
Smooth MSE
Smooth
MSE
TSAA MAE
pred len
96i
0.187
0.274
0.187
0.279
0.185
0:276
0.102
D:285
0.187
0.274
192
0.225
013115
0.251
0.521
0,252
0.318
0.261
0.33
0.255
0.314
339
0.313
0.354
0.322
0.366
0.32
0.338
0.324
0.365
0.311
0.35
720
0.409
0.409
0.127
0.125
0.12
0.414
0.124
0.42
0.408
0.403
Ang.
0.29
0.338
0.298
0.348
0.264
0.342
0.301
0.35
0.29
0.335
96
0.506
0.382
0.375
0.36
0.502
0.379
0.502
0.368
0.522
0.382
192
0.629
0.397
0.604
0.372
0.300
0.612
0.378
0.601
0.371
338
0.635
0.401
0.82
0.384
0.635
0.403
0.829
0.386
0.619
0.383
720
0.646
0.629
0.385
0.639
0.387
0.632
0.388
Avy.
0.626
0.395
0.607
0.376
0.628
0.3088
0.618
0.38
0.607
0.376
Woxter
96
0.2228
0.307
0.283
0.358
0.199
0.271
0.217
0:296
0.207
0.285
0.314
0.964
339
0.318
0.357
0.368
0.398
0.505
0.345
0.359
0.394
0.313
0.865
0.395
Ang.
0.3001
0.347
0.346
0.385
0.282
0.327
0.33
0.371
0.288
0.356
24
5,306
1.222
3.272
1,254
3.544
1,320
3.258
125
3.15
1.219
2.682
1.002
2.638
1.063
2,877
2.653
2.578
1.049
48
2.656
LO88
2.61
1.067
2,844
1.135
2,618
1.073
2,609
1,060
2.92
1.178
2.87
1.158
3.072
1,213
2,832
1.148
2.805
1.14
Avg. 2,891
1.155
2,818
1.136
3,084
1,201
2.81
1.156
2.786
1.119
behavior, as suggested in (Fama, 1965). We hypothesize that the random walk component is dominant in the
Exchange dataset, which may explain why TSAA encounters difficulties when applied to datasets with a strong random walk influence.
To empirically validate our hypothesis, we evaluated the performance of TSAA on two different types of datasets: (1) seasonal dataset (w/o RW), characterized by strong seasonal components without any random walk inffuence, and (2) A seasonal + random walk dataset (+RW), where random walk component added to the seasonal data. To further remove stationarity, both series were multiplied by an additional random walk vector, formally represented as: (1) Fg Zrw, and (2) (x, + Tre + &rw, where I, is the scasonal component, and Trw and &rw are two independent random walk vectors, with the operation * applied element-wise. Examples of these datasets are illustrated in Fig. 7. The results indicate that, for both Autoformer and FEDformer, performance on the dataset containing a random walk component (+RW) is notably lower compared to the scasonal dataset (w/o RW), despite both sharing the same 2, and Frw components. Specifically, the median and mean performance the (w/o RW) dataset consistently surpass those of the (+RW) dataset. across all
25

---

Published in Transactions on Machine Learning Research (02/2025)
Autoformer 96
Autoformer 720
FE FEDformer 96
FEDformer 720
80.0
80.0
MSE reduction
60.0
40.0
100.0
40.0
20.0
60.0
20.0
0.0
-2000
20.0
-40.0
-20.0
0.0
-20.0
-60.0
20.0
w/o
RW
w/o
+ RW
w/o
+ RW
w/o
+ RW
Synth: Seasonal (w/o RW)
Synth: Seasonal
RW (+RW)
10.0
0.0
20.0
Mww
10.00
0.0
-20.0
250
500
750
10000
1250
1500
1750
2000
250
500
750
1000
1250
1500
1750
2000
Electricity
Exchange rate
400.0
300.0
0.7
100.0
250
5.00
750
1000
1250
1500
1750
2000
250
500
750
1000
1250
1500
1750
2000
Figure 7: Top: MSE % reduction with TSAA for two synthetic datasets characterized by scasonal component only (w/o RW), and seasonal component with a random walk (+RW), respectively. The red dashed line and the orange line represent the mean and median respectively. The reported results represent the performance of five different setups. Bottom: Samples of the synthetic datasets used in the given experiments, and real datasets with similar corresponding characteristics.
configurations, as shown in Fig. 7. our experiment, five different setups using generated datasets were tested, each maintaining the same Vas Trws and &rw components.
26

---

Published in Transactions on Machine Learning Research (02/2025)
E.6
Further results
In this section provide bar comparisons of the results presented in E.1.
Multivariate
Informer
Autoformer
FEDformer-f
FEDforer-w
2.128
%
7.06%
3.01%
4.53%
ETTm2
1.631
0.302
0,288
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
0.259
0.218
Electricity
3.61%
2.99%
0.0%
4.19%
0.371
0.245
0.215
0.216
0.354
0.231
0.212
0.217
0.209
0.203
0.202
Baseline
Baseline
TSAA
0.947
47.58%
-32.78%
-64.72%
-34.22%
Exchange
1.730
0.964
0.826
0.744
1.353
0.721
0.704
0.630
0.976
0.479
0.582
0.515
0.599
0.401
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
3.442
2,608
-0.15%
0.73%
2.21%
0.68%
2,822
5.189
2.798
2.743
2.424
5.058
2.833
2.663
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
0.892
0.714
3.78%
0.93%
0.49%
2.18%
Traffic
0.862
0.670
0.614
0.599
0.609
0.604
0.500
0.571
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
0.363
0.457
52.05% %
7.78%
7.69%
26.73%
Weather
0.570
0.343
0.325
0.414
0.455
0.323
0.302
0.371
0.340
0.303
0.278
0.328
0.283
0.255
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Figure 8: Multivariate comparison between TSAA and the baseline models per dataset. Rows represent the datasets and columns represent the models. Each score represents the average MSE across four horizons, namely, 96, 19,,33,3 and 720.
27

---

Published in Transactions on Machine Learning Research (02/2025)
Univariate
Informer
Autoformer
FEDformer-f
NBEATS
NBEATS G
0.172
0210
0.132
-0.77%
12.5%
-0.81%
-6.62%
-2.13%
0.161
0.181
6.128
0.165
0.158
ETTm2
0.150
0.152
0.141
0.143
0.122
0.128
0.093
0.094
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Basell
TSAA
-11.7%
21.04%
0.0%
18.06%
19.13%
Electricity
0.401
0.541
6.319
0.35
0.300
0.323
0.466
6.309
0.32
0.318
0.244
0.347
0,29
0.294
Bascline
TSAA
Baseline
TSAA
Baseline
TSAA
Bascline
TSAA
Baseline
TSAA
1240
65.52%
-56.15%
-75.91%
58.27%
49.16%
Exchange
0.514
0.606
0.585
0.358
0. 0.634
0.395
0.000
0.038
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Baseline
Baseline ne
TSAA
0.328
0.250
0.199
5.21%
8.99%
7.31%
0.0%
6.21%
0.314
0.996
0.155
0.178
Traffic
0.191
Bascline
TSAA
Baseline
TSAA
Baseline
Baseline
TSAA
Basell
TSAA
25.0%
50.0%
0.012
71.43%
0.006
50.0%
0.008
50.0%
Weather
0.010
6.009
0.005
8.000
0.006
0.003
0.002
0.002
0.001
Baseline
TSAA
Baseline
TSAA
Baseline
TSAA
Baseline
TSM
Baseline
TSAA
Figure 9: Univariate comparison between TSAA and the baseline models per dataset. Rows represent the datascts and columns represent the models. Each score represents the average MSE across all four horizons, namely, 192,33,, and 720.
28

---

Published in Transactions on Machine Learning Research (02/2025)
Table 22: Informer: Comparison of automatic augmentation approaches including TSAA, Fast AutoAngment and Rand Rugdengment.
Dotreet
Horizon
Base line MSE
Broeline MAP
Rand-Aggaon of MSP
RaureAiggeett MAR
Past AA MSR
Faot AA MAR
TSAA MSE
TSAA MAR
E1
96
0.545
0.585
0:417
0.458
0:443
0.488
0.224
0.321
0.583
0.355
330
1.523
019918
0.561
0.702
1.565
0,974
0.304
0.374
7200
3,878
1,474
260
0.386
3.630
1,409
0.398
01.435
0.416
0.325
0.445
0.407
BCL
192
0.35
0.441
0.341
0.435
0.388
0.463
0.336
0.419
330
0.336
1,420
0.304
0.441
0
0.776
0.347
01.499
0.448
56
0.744
0.42
0.704
0.426
0.544
0.529
0.728
0.408
Westber Thutk
192
773
0.424
0.78
0.434
1.446
0.777
0.735
0.414
0,495
7200
1.111
0.578
1.12
0.6312
1.511
0,819
0.985
0,366
96
0.315
0.382
0.488
0.485
0.258
0.339
0.18
0.256
330
0.82
01.554
0.585
0.551
0.542
0.519
0.296
0.361
730
0.375
0.722
1.0
0.741
0.818
0.665
0.392
0.422
6.349
4.82
1.505
1,579
36
5.200
1.572
4.326
1.346
5.264
L.:00
6.26
1.581
48
T 1,285
1,784
4,656
1.456
6.440
1.506
6.415
1.025
Table 23: Autoformer: Comparison of automatic augmentation approaches including TSAA, Fast AutoAug-
ment and RandAugment.
Datasct
Horizon
Baseline MSE
Baseline MAE
RandAngment MSE
RandArgmeet MAE
Fast AA MSE
Fast AA MAE
TSAA MSE
TSAA MAE
EOL ETT1Z
96
0.251
0.31
0.2013
0.293
0.259
0.346
0.286
0.282
0.331
0.280
0.527
338
0.311
0.375
0355
0.371
0.352
0,377
0.322
0.359
720
0.644
0.454
0.127
0.12
0.422
0,413
0.41
0.407
0.316
0.102
0.321
0.302
192
0.217
0.326
0.217
0.326
0.240
0,3
0,221
0.525
0.258
01336
0.232
0.254
0.352
0,261
0.363
0.279
0.37
56
0.815
0.554
0.631
0.301
0.065
0.318
0.802
0.375
Weather Staftic
0.67
01421
0.636
0,764
0.48
0.418
0.835
0.392
0.632
0.443
0.387
720
0.658
0.402
0.663
0.400
0.672
0.526
0.662
0.405
192
208
0.556
0.288
0.344
0.255
0.304
0.228
0.535
330
0.357
(13994
0,328
0.364
0.307
0.338
0.:44
01381
0.422
0.41
24
5.549
1.305
3.774
1.354
4.737
1.605
3.565
1.302
36
2.831
1,094
2,80
1.100
3.564
1,375
2.754
1.068
48
2,889
1.122
2.676
1.045
1.368
1.114
2,818
1.118
2.775
1.412
1.118
Table 24: FEDformer-f: Comparison of automatic augmentation approaches including TSAA, Fast AutoAng-
ment and RandAugment.
Datawet
Horisom
Baseline MSE
Baeeline MAE
Rand-Agrrer MSE
RanciArggmeet MAE
Fost AAA. MSE
Fost AAA MAE
TSAA MSE
TSAA MAE
EThnd
56
0.189
192
0.258
0.325
0.257
0.322
0.282
0.519
0.255
0.314
336
0.323
01303
0.326
0.364
0.322
0.311
0.36
56
0.185
0.3
0.11
0.304
0.201
0.315
0.185
0.3
BCL
1922
0.201
(L31G
0.206
0.318
0.201
0.315
0.201
1L316
0.529
0.927
0.529
720
0.216
0.353
0.291
0.385
0.254
0.357
0.246
0.353
96
0.577
0.36
0.30
0.376
0.665
0.41
0.577
0.362
1111 Wonrhan Thilte
0.372
336
01:255
01.855
0,026
0.20 0.3022
0.674
0.421
0.619
0.583
720
0.632
0.388
0.644
0 100
0. 7.706
0-427
0.632
0.388
192
012775
0.335
0.267
0.3011
0.24
0.29
0,262
11:11
336
0.332
0.371
0.328
0. 16368
0.22
0.321
0.313
0.355
24
3.368
1.257
3.16
1 1.234
4.671
1.612
3.15
1.219
36
2,645
1,063
2.457
1.019
3.835
1.377
2,528
L042
48
2.555
1,350
60
2.966
1.158
2,522
1.146
3.855
1.41
2.805
1.14
29


Reviews:
[{"review_id": "RhaOuQsTib", "reviewer": "Reviewer_91K2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe research problem addressed in this paper, namely data augmentation for time series forecasting and its automatic selection, is highly significant.\n\n\n\n### Weaknesses\n\n1. The presentation needs improvement, for example:\n     1) the basic formulation of the problem, i.e. Eq (3) and (4), requires a more detailed explanation, instead of just citing another work.\n     2) Equations used in this work are mostly highly abstract and generalized, making it difficult to understand the details of the algorithms.\n2. I have concerns about the novelty of the proposed algorithm. The problem formulation is proposed by a previous work. The core algorithms TPE and ASHA are well-established methods in the field of Bayesian optimization. Moreover, the proposed algorithms appear generic, and there is no specific design for time series forecasting.\n3. The selected baseline models are out-of-date: recently the community has tend to patch-based models, e.g. PatchTST, Crossformer, iTransformer, but models used in experiments are all old-style step-wise ones.\n4. The work lacks a true ablation study as the corresponding section is mainly hyper-parameter analysis. The following ablation study should be performed:\n     1) Ablation study showing the proposed automatic augmentation is better than pre-defined ones, i.e., directly using data augmentations in page 5;\n     2) Ablation study showing fine-tuning is better than re-training from random weights.\n5. As the proposed algorithm involves expensive searching processes, an efficiency or complexity analysis, either through theoretical analysis or experiments, is needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an automatic data augmentation approach specifically designed for time-series forecasting tasks named TSAA (Time-Series AutoAugment), a two-step method that combines Bayesian optimization with early stopping to efficiently find optimal augmentation policies. In the first step of TSAA, the model undergoes partial training for a limited number of epochs to compute shared weights. In the second step, TSAA iteratively alternates between two tasks: 1) optimizing augmentation policies using Bayesian optimization and 2) fine-tuning the model's weights based on the current best policy. For augmentation policy optimization, TSAA uses Tree-structured Parzen Estimator (TPE) with Expected Improvement (EI); for network parameters fin-tuning, Asynchronous Successive Halving is used to stop poorly performing rungs earlier. The model's performance is evaluated on multiple univariate and multivariate time-series forecasting benchmarks, showing some improvements over baseline models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Improve presentation to make the work self-contained and use detailed equations for description instead of too generic ones;\n2. Add recent patch-based models (e.g. PatchTST, Crossformer, iTransformer...) as baselines;\n3. Perform the ablation study mentioned in Weaknesses #4;\n4. Perform an efficiency or complexity analysis mentioned in Weaknesses #5;", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ETE95SMg9D", "reviewer": "Reviewer_1m6A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper proposes a new method of automatic augmentation for time-series forecasting. \n- The effectiveness is impressive.\n\nWeaknesses\n- The bilevel optimization is just like basic HPO for training, which does not introduce novelty. The process of Algorithm 1 may be wrong since the optimization of Eq. (3) is based on the optimization of Eq. (4).\n- The policy space is confusing. Augmentations are generally applied to original data, but Eq. (6) means that a sequence of augmentations may be applied on $x_0$ simultaneously. This would make the space explode, which is not discussed in complexity analysis.\n- Some important issues are not discussed. Augmentations may introduce bias in the original data and how to constrain the influence. Augmentations can also be introduced from scratch without optimization, which is not discussed or compared.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method of automatic augmentation for time-series forecasting. It introduces bilevel optimization to search for augmentation hyperparameters and model parameters. The effectiveness is impressive.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes\n- The process of Algorithm 1 may be wrong since the optimization of Eq. (3) is based on the optimization of Eq. (4).\n- The policy space is confusing. Augmentations are generally applied to original data, but Eq. (6) means that a sequence of augmentations may be applied on $x_0$ simultaneously. This would make the space explode, which is not discussed in complexity analysis.\n- Some important issues are not discussed. Augmentations may introduce bias in the original data and how to constrain the influence. Augmentations can also be introduced from scratch without optimization, which is not discussed or compared.\n\nAll concerns have been resolved.", "broader_impact_concerns": "No ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yhHvflyEd6", "reviewer": "Reviewer_irFR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Pros\n\n1. The paper conducts extensive evaluations across various well-known datasets and compares the proposed method against several state-of-the-art baselines.\n\n2. The combination of Tree-structured Parzen Estimator (TPE) and Asynchronous Successive Halving (ASHA) is new and leverages state-of-the-art techniques for efficient optimization.\n\n- Cons\n\n1. The paper reports cases where TSAA does not outperform the baselines (e.g., Exchange dataset), but the discussion on why TSAA underperforms in these scenarios is minimal. A more detailed analysis of these cases would strengthen the paper.\n\n2. The paper focuses solely on time-series forecasting. While this is the main objective, it would be beneficial to discuss how the proposed method could be generalized or adapted to other time-series tasks, such as classification or anomaly detection.\n\n3. The choice of augmentations could be better justified. While the paper lists various transformations, it does not provide a rigorous theoretical basis for why these specific augmentations are expected to improve forecasting performance. A more detailed discussion on the theoretical underpinnings of the selected augmentations would strengthen the methodology.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary\n\nThe work presents a method noted Time-Series AutoAugment (TSAA) that addresses a gap in the existing literature on data augmentation specifically for long-term time-series forecasting.  It consists of two main components: (1) a partial training step to obtain shared weights and (2) an iterative process that alternates between searching for an optimal augmentation policy using Bayesian optimization and fine-tuning the model with that policy.\n\n- The use of partial training to obtain shared weights and the incorporation of Asynchronous Successive Halving (ASHA) to reduce computational costs are motivated and add value to the proposed approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In some of the more detailed figures, consider using insets to zoom in on critical areas of the plots. This can help highlight the differences between TSAA and baseline methods, which might be subtle in the main figure.\n\n2. When presenting comparative results (e.g., with or without TSAA), consider using paired bar charts or connected scatter plots to make the improvements (or lack thereof) more visually apparent.\n\nQuestions\n\n- Some results are less compelling, particularly on the Exchange dataset, where TSAA underperforms. The paper would benefit from a deeper analysis of these cases to understand the limitations of the approach. For instance, does TSAA struggle with certain types of time-series patterns or noise levels?", "broader_impact_concerns": "Not major broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Exploring data augmentation for long-term forecasting is interesting in the time series analysis field, which has been done by this work. Evaluation shows decent performance gains via the proposed bi-level optimization method, but new results on more recent baselines show marginal improvement. In addition, reviewers are concerning the novelty of using TPE and ASHA, which are existing methods and are of much more complexity than well-established time series models. The bi-level optimization approach also lacks sufficient and convincing explanations. In conclusion, the merits delivered by the paper may slightly outweigh the weaknesses, because it can draw more attention to data augmentation for time series models. Therefore, the AE recommends Weak Accept.
Decision: Accept (poster)