Optimizing Time Series Forecasting Architectures: A Hierarchical Neural Architecture Search Approach
Authors: Difan Deng
Marius Lindauer
Venue: Accepted by TMLR ()

Abstract: The rapid development of time series forecasting research has brought many deep learning-based modules to this field. However, despite the increasing number of new forecasting architectures, it is still unclear if we have leveraged the full potential of these existing modules within a properly designed architecture. In this work, we propose a novel hierarchical neural architecture search space for time series forecasting tasks. With the design of a hierarchical search space, we incorporate many architecture types designed for forecasting tasks and allow for the efficient combination of different forecasting architecture modules. Results on long-term time series forecasting tasks show that our approach can search for lightweight, high-performing forecasting architectures across different forecasting tasks.

Published in Transactions on Machine Learning Research (09/2025)
Optimizing Time Series Forecasting Architectures: A Hierarchical Neural Architecture Search Approach
Difan Deng Institute of Artificial Intelligence Leibniz Universily Hannover
d.dengGai, uni-hannover.rumm de
Marius Lindauer
m. Windadddd@aaas. www.-aiiennn---aaeeee.e.e..
Instibute of Artificial Intellgence
Leibniz University Hannover
L3S Research Center
Reviewed on OpenReview: /ttps://openreese net/ forum? id= Ym2uqq j jm4e
Abstract
The rapid development of time series forecasting research has brought many deep learning- based modules to this field. However, despite the increasing number of new forecasting architectures, it is still unclear we have leveraged the full potential of these existing modules within a properly designed architecture. In this work, we propose a novel hierarchical neural architecture search space for time series forecasting tasks. With the design of hierarchical search space, we incorporate many architecture types designed for forecasting tasks and allow for the efficient combination of different forecasting architecture modules. Results long-term time series forecasting tasks show that our approach can search for lightweight, nigh-performing forecasting architectures across different forecasting tasks.
1
Introduction
Time series forecasting techniques are widely applied in different fields, e.g., energy consumplion Trindade
2015), business [Makridakis et al.] 2022), traffic planning (Lana et al. 2018). However, unlike computer
vision (CV) and natural language processing (NLP) tasks that are dominated by the CNN (He ct al. 2016
Liu et al. 2018 Zoph et al. 2018) and Transformer (Brown al., 2020 Devlin el al. 2019 Liu et al.
2021
Vaswani et al. 2017) families, there no clearly dominating architecture time series forecasting
tasks.
Although there are lots of haansfommrr-bsee approaches applied forecasting models (Ansari et al,,
2024
Das et al.] 2023 Wu et al.) 2021 Liu al., 2022 Zhou et al. 2021). many of them might even be
outperformed by simple linear baseline (Zeng al. 2023
The success of transformer models in CV and NLP tasks is based on their ability to capture long-term dependencies with the help tokens with fruitful semantic information, i.c., each word embedding already contains lots of information, while an image patch with many pixels can already tell us a lot of information. All this information relaxes the requirement for the models grab the temporal information within the input series. However, this information is usually crucial in time series forecasting tasks, given that a single value at each time step only contains limited amount of information. The strong ability of the transformer family to capture long-term dependencies might not compensate for its poor ability to build temporal connections within the input series. This finding is evident in Zeng et al. 2023) where a simple linear layer could outperform many state-of-the-art. transformer models.
Some recent works show the efficiency of transformers (Liu et al. 2024b Nie et al. 2023 on long-time forecasting tasks. However, their approach could still not overcome the temporal dependencies issues for transformers, e.g., PatchTST Nie et al. 2023) augments the information within each token by constructing

---

Published in Transactions on Machine Learning Research (09/2025)
patch with the data from multiple time steps. While iTransformer (Liu et al 2024b simply encodes the entire input sequence into a token and tries to construct the connections among different variables.
On the other hand, many architectures were constructed for mining the local dependencies, such as CNNs (Bai
et al. 2018 Luo and Wang 2024 and RNNs (Hevamalage et al. 2021 Hochreiter et al., 2001 These architectures might not work well on long-term dependencies due to the limited receptive field (for CNN), latent bottlenecks (Didolkar et al. 2022), or vanishing gradients (for RNN families). Recent work, such ModernTCN (Luo and Wangi 2024), showed that an increased convolutional kernel size and improved microarchitecture could lead to more accurate model. However, this approach results in huge memory and computation consumption. Here we provide another perspective: one could combine these architectures with other operations, such as transformer layers, to develop a new architecture that combines the best of two
worlds (Didolkar et al., 2022 Lai et al. 2018 Lim et al. 2021
Nevertheless, it is still unclear (i) which type of architocture we would like to construct, given the variability of different forecasting models (Deng et al. 2022
Oreshkin al. 2020 Salinas al. 2020 Zeng et al. 2023 and (ii) how to connect different operations to form new architecture. Designing a new architecture from scratch for each task might take a lot of human expert effort and tedious trial-and-error. Neural architecture search (NAS) is a technique that automatically searches for the optimal architecture given new task.
Previous NAS frameworks mainly focused on single network backbone types such as CNN or Transformer networks (Chen et al. 2021b Liu et al. 2018 Zoph et al. 2018). It still unclear how to optimize the forecasting architectures due to their internal complexity. For instance, an encoder-decoder architecture (Wul et al. 2021 Zhou et al., 2021 might. work well some tasks, while the other tasks might. prefer encoder-only
architectures (Liu et al. (024b) even MLP-only architcctures (Oreshkin ct al. 2020 Zeng et al., 2023 This provides another challenge for designing a search space for time series forecasting tasks. In this work, we will address this challenge by designing unified search space for time series forecasting tasks.
Our contributions are summarized as follows:
1. We design a hierarchical search space that contains most forecasting architecture design decisions and allows any sort of architect ure layers to be combined to form new architect.rre.
2. We show that by applying DARTS-PT (Wang et al. 2021 differentiable neural architecture search approach, to our search space. The resulting architectures, dubbed DARTS- RS, are comparable to the state-of-the-art models with much less computational resource requirements.
3. We provide an analysis of our search space, showing that our search space has different properties compared to the existing CNN-based NAS search space, and provides further challenges for the NAS rescarch.
2 Related Work
Although many different deep learning architectures are proposed solve time series tasks, there is little work that applics neural architecture scarch to scarch for new architecture within the cxisting frameworks. In this section, we will provide brief overview of deep learning-basee forecasting frameworks and neural architecture search techniques.
2.1 Deep Learning- Leasdd Time Series Forecasting
Time series forecasting aims to predict the future values of target variables given their historical data. Because of its importance, much work has been investigated for more accurate forecasting model in this rescarch field. Previous work mainly focused on traditional istiaal local approaches that train an individual model for each series (Athanasopoulos 2021 Box et al., 2015). However, these approaches might not fit well in the era of big data, where a datasct could contain thousands of scries. On the other side, the machine learning-basod model trains a single global model across all the series and uses this model to predict all the series in the More recently, deep learning-baser forecasting
dataset (Godahewa et al. 2021 Makridakis et al. 2020 2022
2

---

Published in Transactions on Machine Learning Research (09/2025)
models Bai et al. 2018 Hewamalage et al. 2021 Lai et al. 2018 Salinas et al 2020
Shi et al.
2015 Wen have gradually
et al. 2017), or even the wero-shot foundation models (Ansari et al. 2024 become mainstream in this research field.
Das et al.
2023)
Time series forecasting models need to work with sequential inputs (Alexandrov et al. 2020 Beitner 2020
Overall, these networks can be categorized into two families: Seg Net and Flat Net (Deng et al. 2022). Given
a batch of sequences with shape 1, NV where B is the batch size, L is the sequence length, and N the
number of time series variables. A Seq Net, such as RNNs (Cho et al., 2014 Hochreiter and Schmidhuber
1997.. TCNs (Bai et al.) 2018 Oord et al. 2016 and Transformers (Li et al., 2019: Liu et al.) 2022 Vaswani
ct al. 2017 Wu et al. 2021 Zhou et al.) 2021), computes the correlations across different time steps without
breaking the structure of the input sequence. On the contrary, a Flat Net, such as MLP (Zeng et al. 2023
and N- N-BEATS (Oreshkin et al. 2020), decomposes the variables into independent single-variable series: [B x N, [ After that, this variable is passed to another network. This strategy was previously introducod such that the time scries can be handlod by the machine learning framework designed for tabular datasets, such as MLP layers ÚZeng et. al.
2023) and Lig (Ke et al. 2017 Makridakis et al.] 2022). Some
recent works, such as PatchTST Nie et al. 2023). release the correlation between different variables within multi-variable series and consider it as collection of independent scries, and make predictions for cach series independently, which also belongs to this type of architecture. However, these approaches might be too expensive for datasets with many variables since a forward pass is required for each series. Hence, in this work, we mainly focus on the MLP-based Flat Net to only search for lightweight architectures.
Diving deeper into the Seq Net architecture families, we find another two main branches: encoder-decoder architectures and encoder-only architecture Encoder-decoder architectures (Sutskever et al., 2014 Vaswani et al. 2017 maintain two individual networks that embed the information from the past and future correspondingly. These architectures have shown great success in time series forecasting tasks (Wu et al., 2021 Zhou et al., 2021). On the other hand, encoder-only architectures only apply Seg encoder that maps the past information into a latent fcature map and utilizes another lincar layer to provide the prediction with the latent feature map (Liu el al., 2024b Nie el al. 2023 Salinas et al.
2020
Many Seq Net models, especially the transformer family (Lim et al.) 2021
Wu et al. 2021 Zhou et al. 2021). )
are designed for solving series input. However, study by Zeng et al. 2023) showed that these transformers might even be outperformed by linear model. This inspires us to seek other uncovered modules that might perform well within properly designed architecture, esp. architectures with more than one type of operation.
Many forccasting architectures are homogenous and only contain one type of operation layer (Bai et al.) 2018 et al. 2019 Liu et al. 2022 Luo and Wang. 2024 Oord et al. 2016) and stack this layer repeatedly to construct new architecture. However, some recent work has also shown the efficiency of combining architectures from different model families. LSTNet (Lai et al. 2018) stacks an RNN on top of a CNN layer. ComvLSTM (Shi et al.)
2015) constructs convolutionary operation within an LSTM cell. Temporal fusion transformer (Lim et al. 2021 stacks an explainable multi-head attention layer on top of an LSTM encoder- decoder model. All these works suggest the efficiency of combining operations from different architecture lamilies. However, it is often ledious to find these combinations manually. Neural architecture search (N AS) is a technique that automatically searches for the optimal architecture for a given task. In the following soction, we will briefly overview the NAS framework.
2.2 Neural Architecture Search
Previous NAS research mainly considered network training as a black- process, training every configuration from scratch until convergence (Deng et al. 2002 Jin et al. 2019 White ct al. 2021 Zimmer et al. 2021 Zoph and Le 2017). However, training a network is very expensive and requires lots of resources. To overcome
this problem, the One-Shot NAS (Pham et al. 2018 Zoph et al. 2018) approach defines a supernetwork where all the child architectures' weights are inherited from the supernet. DARTS [Liu et al. 2019a) further
Howeverr in practice, would still like to preserve the correlations between multi-variant series with, for instance, batch normalization [Toffe and Szegedy 2015). In this case, the input series becomes [B, N, L] 2Sometimes we might also have delyder-only architectures; however, for the sake of simplicity, we consider both as part of encoder-only architectures.
3

---

Published in Transactions on Machine Learning Research (09/2025)
relaxes the discrete operation search space to continuous parameters and optimizes these values with model weights jointly with gradient descent. Finally, the optimal operations and paths are selected based on the architecture parameter values.
DARTS may be unstable during the search process: the architectures suggested by the DARTS might be dominated by skip connections (Chu et al. 2020; Jiang et al. 2023 Wang al., 2021 Zela et al. 2020 Robust DARTS (Zela et al. 2020) showed that this instability is caused by the high validation loss curvature in the scarch space, while operation-level early stopping DARTS (Jiang et al.) 2023) found that the instability arises when the network weights are overfitted to the training sets. DARTS-PT (Wang et al. 2021 provides perturbation-basel approach to mcasure the importance of each operation and use it to replace the architecture parameter-based approach introduced in Liu et al. (2019a).
ONE-NAS Lyu et al. 2023) is an online architecture search framework that applies evolutionary algorithms to search for the optimal RNN networks on online forccasting tasks. On the other hand, SNAS4MTF (Chen) et al. 2021a) proposes to use archilecture search to form an end-to-end forecasting architecture framework. Furthermore, Auto-PyTorch TS (Deng et al. 2022 provides uniform search space that includes many forecasting modules. The optimizers can freely assemble these modules to form new architectures. However, these works still focus on homogeneous architecture designs, where the type of Seg decoders is restricted by the decision of Seq Encoders. For instance, an RNN encoder only allows RNN decoders (Seg decoder) or MLP decoders (Flat decoder). As the current neural architecture tends to focus more on the one-shot weight-sharing approaches (Jiang et al.) 2023 Liu et al. 2019a; Wang et al.) 2021 Zoph et al. 2018), in this work, we will show how to design a general one-shot model for time series forecasting that fits most of the forecasting models, making automated deep learning for time series forecasting substantially faster than before.
Evaluating an architecture could take lots of resources. One-shot NAS approaches alleviate this by constructing a supernet where the weights of all the candidate architectures are inherited from this super network. However, training a supernet still requires lots of resources. Zero-Cost (ZC) proxies (Chen et al. 2022 Krishnakumar et al. 2022) attempt to address this problem by estimating the performance of the target model without
updating the model weights, thereby greatly accelerating the search process. However, nearly all existing
zero-cost proxies (Abdelfattah al. 2021 Lee et al.] 2019 Lin et al. 2021 Ning et al.) 2021 Tanaka et al. 2020 Turner 2019: Wang el al.] 2019) are developed lor computer vision NAS tasks that only contain one architecture type (convolutional operations). It is still unclear if the ZC proxies can be generalized to compare the performances between different operation types.
3 Problem Setting
Time series forecasting tasks aim to predict the values of the target variables for number of iterations after certain time step, with the observed same variables and several other feature variables. Formally, given dataset that is composed of multiple scries D = {D:) where each scries is composed of the past observed largels Ys,IT, past observed features x,1:;;; and known future features & Given a required forecasting horizon H, the model is asked to predict the target variables プEл+ITTH with all the available information:
9UTHLITHHH = f(YLIT, XI,T, (.T+IT
(1)
Two model families can mainly handle multi-horizon forecasting tasks: the autoregressive approach and the non-autoregressive approach. Autoregressive approaches (Box et al. 2015 Salinas et al. 2020) only predict one step within one forward pass and iteratively use the prodicted value as the known feature that can be further fed to the model. On the other hand, non-autoregressive models (Nie et al.) 2023 Zeng et al. 2023 Zhou al., 2021) directly generate multiple forecasting values within one forward pass. In this work, we mainly focus searching for non-autoregreesive architectures.
Neural architecture search aims at finding the optimal architecture a, for given task:
min Loat(f" o.) s.t. 0* E arg min Ltrain (0,a)
(2)
4

---

Published in Transactions on Machine Learning Research (09/2025)
The search space in previous NAS work still focused on the homogeneous search space, where all the operations belong to the same architecture families. This work presents a unified heterogeneous search space containing most of the potential architectures applied for time series forecasting tasks.
4 Search Space Design
Tt is a common observation and a foundational assumption of NAS that no single architecture family always outperforms the others; based on the results of Deng et al. [0022] we believe that this also holds for time series forecasting tasks. We are unlikely to define a single type of model that works for all datasets. Additionally, some of the architecture might be dependent on the other decision choices. For instance, if we decide to have an encoder-only Seq Net, there is no need for us to search for a Seq decoder. Here, we propose a hicrarchical search space that incorporates most of the forecasting architecture families described in Section 2.1 We will start from the most basic operation level and gradually decrease the granularity until our search space contains all the required components.
4.1 Operation Level
The first level, the operation level, describes the oper- ations that can be used in the network. As shown in Figure I our search space follows the DARTS Liu et al. 2019a) search space, a cell-based architecture where each cell is a directed acyelic graph that con- tains N nodes, including Nin input nodes. Each node represents latent feature map, and the edges that connect the nodes are the operations applied to the latent feature maps. DARTS defines two types of cells: normal cells and reduction cells. These two cell types share the same form of input feature maps. Therefore, they can be easily concatenated to form an architecture. However, this is not the case for fore- casting tasks. Seq Net and Flat Net transform the input features differently. We cannot casily connect Seg cell after Flat cell and vice versa. Hence, provide a search space for each of the model families. Detailed information about each operation can be found in the appendix A
Figure 1: Operation level Search Space. The nodes and 1 are input nodes that receive the network inputs or the outputs from the other cells. Node 4 is the output node. Each colored edge represents an operation. The Search space (Left) is fully connected directed acyclic graph. Once we have finished the search, we get the final architecture (Right).
For the Seq Net families, we cousider the following operations: 1. MLPMixer (Chen el al. 2023), an alI-MLP architecture that applies linear layer to feature and time dimensionss, respectively, 2. LSTM (Hochreiter and Schmidhuber 1997), 3. GRU (Cho
2014), Transformer (Vaswani al. 2017), TCN (Bai al., 2018), 6. TeN Luo and Wang 2024]. and 7. skip connections. These operations spread to encoder and decoder architectures which will be discussed in the following section.
For the Flat Net families, we have: 1. Linear (Zng et al. 2023), a single linear layer that encodes the past information to the future variable, 2. N-BEATS (Oreshkin et al. 2020), a repeatedly stacked MLP block, where each block contains a set of fully connected (FC) backbone layers, a forecast, and a backcast head, and 3. skip connections. There are several variants in N-BEATS modules: generic model, trend model, and seasonality model. We incorporate all these modules into our search space, providing another two operations. Since the only difference between different N-BEATS modules is their forecast and backcast head, ask these N-BEATS modules to share the same FC layers backbones. In total, we have five operations for each edge in the Plat Net cell.
Unlike the previous search space that focused on one single architecture type (Klyuchnikov et al. 2020 Krishnakumar al.) 2022 Mehta et al., 2022: Ying et al.] 2019). the operations contained in DARTS-TS TS come from different families. Therefore, the outputs from each operation might have completely different distributions even different output formats for the same input. The zero-cost. proxies that are proven to
5

---

Published in Transactions on Machine Learning Research (09/2025)
be efficient on other benchmarks, such flops or number of parameters Krishnakumar et al. 2022
Wan
et al.
2022 might no longer work well within this search space.
4.2
Micro Network Level
Flat operations only receives the past information y.,.T;- Therefore, we stack several Flat cells as a Flat Net. As shown in Figure 2a] the past targets are first transposed and then fed to the encoder layers. The transposed target, Le., the backcast part, and zero tensor whose length is equal to the forecasting horizon, i.e., the forecast part, are fed to the Flat encoders. Finally, the forecast output is fed to the forecasting head to predict the target values 3
Different from the Flat Nel, we decompose the Seg architectures into two parts: encoders and decoders. The encoders encode the past observed values into an embedding and feed them to the decoder networks. The design of the Seq encoder similar the Flat Encoder, and we stack the encoder cells to form the encoder network. However, as discussed in Section 2.1 two polential ways exist. to transform the encoder latent. features to the forecasting heads. Therefore, we design the following two types of decoder networks: Seg decoder and Flat decoder for Seg encoder.
The design of the Seg Encoder and Seq Decoder architecture has been widely applied in previous architecture works [Lim et al. 2021 Sutskever et al. 2014 Vaswani et al. 2017). However, architecture decoders might require different information from the encoder network. For instance, a Transformer decoder (Vaswani et al. 2017) only requires the output from the last layer of the corresponding encoder; an RNN decoder would need the hidden states from the corresponding encoder layer. We record the following information stored by each edge:
1. The output hidden feature map of this edge. It will then be contacted with the corresponding decoder feature maps and fed to the TCN decoder network.
2. The last step's 's feature map. This value is considered hidden state that can be fed the corresponding GRU and LSTM layers to initialize their states.
3. The cell gate state that is applied to initialize the hidden cell states of the corresponding LSTM layer. If the encoder is not an LSTM network, following the idea of stitchable network (Pan et al. 2023) that uses linear layer to stitch two networks with different shapes, we use a linear layer that transforms the hidden states into the cell gate states.
Hence, we record all the related information as intermediate states during the forward pass. This information is then used to inform the decoder networks of the information provided by the encoder.
On the other hand, we might not want a complex decoder architecture since the information provided to the decoder and most of the features presented in the past are no longer available in the future. Hence, here we define another type of decoder for Seq network: the linear (or flat) decoder for Seq Net. We apply one linear layer across the time scries dimension (Chen et al. 2023 Zeng et al.) 2023) that maps the encoder output and the available future information to the decoder output feature. This feature then fed to the forecasting head to generate the final prediction result. The overall Sey Nel architecture is presented in Figure 2b
We also consider the choice of the Seq Net decoder as part of the architecture search procedure. Hence, we assign another set of architecture paramcters to the output of the two decoder architectures. This architecture is jointly optimized with the her operations architecture weights introduced in Section 4.1
4.3 Macro Architecture Level
The Flat Net and Seq Net families defined above encode the input data from different perspectives: Flat Net families encode the input series along the time series dimension, while Seg Net families encode the input series
We note that the head here does not necessarily need to network module since Flat net only predicts one variable each time.
6

---

Published in Transactions on Machine Learning Research (09/2025)
Forerasting
Flnt Decoder
Pornessting Head]
Remessting
Tiunsposee
Badicost Output Layer N
Forecast Output Lagee N
Forecasting Hood
Output Encoder Block N
Ontipe Decader Block N
Emnader Layer
Fncoder Layer N Drigrnt Layer
Dacadlrr Block N
Barisasst
Fonreze Maps
Festure Maps
for
Baricast Output Layer
Forocast Ourput Lager
Dooder
Formssa Foature Maps
Dutput Encoder Layer
Outpe: Decoder Block
Ruiue Maps for Thardome
Layre
Encoder
Dmigit Layer
Layne
Transpooed Post Torgrte
Future Placebolder
Raiuze Hags for TUS
Trippppood
Port Targets
Pood Ingot
Krown Future Ingedi
Decaders
(a) Search Space for Flat cells
(b) Search Space for Seg cells
Figure 2: Micro-level search space design for Flat and Seq cells.
across different variations. Hence, the two architectures can complement each other, and we concatenate the two architectures sequentially.
Frewwetng
As shown in Figure 3 the past target val- ues are first fed to the Flal Nel, which re- sults in a backeast and forecasting feature maps. The forecast feature maps are then concatenated with the known future fea- tures and fed to the Seq decoder. Finally, the final forecasting result is the weighted sum of both Flat Nel and Seq N
Sog Net
Knotn Futur lepett
Bickeast Ostyut Pli Noc
Rowesst Outgat Piat Not
Mit Wed
Port Trrgtt
Figure 3: Marco Search Space.
Smodes(ye.1zg (ve.fee((iinnn.faaaayy.. + faat (y
33)
These weights are considered an architecture parameter that can be jointly learned with the other architecture parameters described in the aforementioned scctions.
5 Searching for the Optimal Architectures
Here, provide an exemplary searching strategy, i.e., DARTS (Liu et al. 2019b Wang et al. 2021 to search within our search space. DARTS assigns weight for each of the operations within its search space and optimizes these architecture weights jointly with the model weights using gradient descent. Some of the modules, such as Dropout (Srivastava et al., 2014) and Batch Normalization (Toffe and Szogedy) 2015)), behave differently during training and inference time. We thus switch these modules to the eval mode when we update the architecture weights with validation losses to simulate the evaluation process.
During the scarch phase, we divide the official training/validationn validation splits of the datasct into training and validation sets of the same size, such that they do not overlap with the test sets. The training set and validation sets are then used to optimize the architecture weights and architecture parameters, respectively. We follow the common practice for the vaiiinn-validattonn split in time series forecasting tasks: the validation set is located at the tail of the training set. The first half of the dataset is considered the training set that is used optimize the weights of the supernet, while the second part of the dataset is the validation set, which is used optimize the architecture parameters. We apply RevIN (Kim et al. 2222)) during both the searching and training phases to ensure that the input features fed to the networks stay in the same distribution.
4Frr the sake simplicity; we omit the feature variables K,I-T; and ReTT411:T44 that are fed the smostet and Jseq
7

---

Published in Transactions on Machine Learning Research (09/2025)
DARIS-TS
TTanntormer
ModeraTCN
PatchTST
TSMixer
DLibear
TimesNet
Autoformer
MSE
MAR
MSE
MAT
MSE
MAE
MSF:
MAF.
MSF.
MAE
MAE
MSE.
MAF
ETTml
0.311
0.368
0.368
0.395
0.362
0.386
0.352
0.381
0.382
0.408
0.360
0.382
0.402
0.115
0.619
0.539
ETTm2
0.253
0.300
0,273
01.33
(1:261
03.319
0.257
0.315
0.146
0.477
0.2667
0.3529
0.2290
0.339
0.125
0.411
ETTbl
0.413
0.423
0.473
0.168
0.104
0.421
0.415
0.429
0.515
0.503
0.447
0.456
0.486
0.482
0.559
0.529
ET
0.351
0.388
0.387
0.415
0.133
0.385
0:330
0.379
0.571
0.548
0.422
0.439
0.399
0.435
0.739
0.621
ECL
0.156
0.245
0,166
0:261
0.163
0.257
D.161
0.254
0.168
0.272
0.166
0,264
0.203
0.302
0.221
0.334
Exchannge
0.378
0.409
0.411
0.440
0.525
0.505
0.385
0.417
0.366
0.452
0.382
0.419
0.540
0.524
1.010
0.775
Weather
0.230
0.262
0.239
0..274
0.231
0.269
0.229
0.265
0,222
0. 0.288
0.214
0.297
0.249
0.287
0.398
0.431
Traffic
0.394
0.357
0.273
0.421
0.287
0.398
0.267
0.541
0.413
0.434
0.295
0.624
0.336
0.671
0.412
Table 1: The results of the mean performance over all the long-term forecasting datasets. The best models are marked in red, while the socond-best model is marked with underlines.
DARTS-TS
iTransformer MSE
MoN
PatchTST MAE
TSMixer MAE
DLinear
TimesNet
Autoformer
MSE
MAE
MAE
MSE
MAE
MSE
MSE
MSE
MAE
MSE
MAE
MSE
MAE
PEMS03
0.137
0.241
0.458
0.408
0.408
0.419
0.199
0.291
0.162
0.281
0.264
0.358
0.151
0.248
0.554
0.541
PEMS04
0.112
0.221
0.127
0.237
0.488
0.471
0.266
0.338
0.136
0.254
0.264
0.355
0.127
0.239
0.763
0.669
PEMS07
0.102
0.204
0.504
0.477
0.304
0.374
0.209
0.293
0.150
0.251
0.311
0.373
0.132
0.233
0.361
0.438
PEMSUS
0.169
0.256
0.202
0.269
0,510
0.479
0.231
0.304
0.225
0.305
0.331
0.376
0.191
0.267
0.739
0.627
Table 2: The results of the mean performance over all the PEMS datasets. The best models are marked in red, while the second-best model is marked with underlines.
5.1 Hierarchical Pruning of the One-Shot Model
Vanilla DARTS are shown to be unstable during the search process and might prefer to select architectures that are dominated by skip connections (Chu et al., 2020 Jiang et al.] 2023 Wang et al.] 2021 Zela et al., 2020). Hence, once the weights and architecture parameters are trained, as the last step, we select the optimal operations and edges using the perturbation-based approach (Wang et al. 2021).
Given that our scarch space is a hicrarchical scarch space, some of the operations might be dependent on others, and we have to consider that for the pruning phase. For instance, if we select a Linear Decoder, then we do not need to further select the operations in the Seq decoder. However, on the other hand, this also indicates that our estimate will be biased if we select the choice of decoder before selecting any edge operations in the Seq Decoder. Hence, we propose pruning our network from the lowest granularity level and gradually increasing the granularity level until we prune the operations in our search space. Once all the operations are selected, we further prune the edges of our network using the same perturbation-base based approach. Finally, only two edges are preserved for each node, including the cell output node
6 Experiments
In this section, first demonstrate that DARTS-TS can identify the optimal architecture, which is comparable to many other handcrafted architectures on various datasets. We then provide brief analysis of our search space to show the potential challenges in our scarch space. After that, we provide an analysis of the latency of the optimal architecture. Finally, we show some optimal archilectures as examples.
6.1 Time series forecasting tasks
We evaluate our architecture search framework on the popular long-term forecasting datasets introduced by Wu et al. (2021) and Zeng et al. (2023): Weather, Traffic, Exchange Rate (Exchange). Electricity (ECL), and four ETT datasets. Additionally, we evaluate our approach on the four PEMS datascts (Chen et al. 2001 that record the public traffic network data in California.
Following the experiment setup from the other works, we set the forecasting horizon H for ETT, ECL, Exchange Rate, Weather, and Traffic datasets as (96, 192, 336, 720]. For the four PEMS datasets, we set.
We note that this setting is different from the traditional NAS framework, where all the edges towards the output nodes are preserved. This approach helps to reduce the architecture size and required latency further.
8

---

Published in Transactions on Machine Learning Research (09/2025)
these values {12, 24, 48, 96). We compare our results with the following baselines: PathTST (Nie al. 2023), ModernTCN (Luo and Wang 2024). DLinear (Zeng et. al.
2023). TSMixer (Chen et al. 2023 2021) and TimesNet (Wu et al. 2023). For the sake
Trranforrer (Liu et a 2024b), Autoformer (Wu et al.
2023 and sct the input sequence length of fair comparison, we follow the sctup from PStchTST (Nic et al.
for all models to 336. The training/ver training/validatiinn hatp://vliaaaiioonooooo ion ioe/tsst splits of each dataset follow the rules defined in the Time Series Library where a fixed ratio of the series described as test sets, i.c., each test set might contain more time steps than the required forecasting horizons. i.e,, we perform rolling origin evaluation by continuously moving the forecasting origin until we iterate through the entire test set. We ran each experiment times with different seeds and recorded their mean and standard deviation, respectively[] Further experimental details, including the datasets information, searching and evaluation costs, hyperparameter settings, and GFlops changes after each stage, can be found under the appendix B
The results for long-term forecasting tasks are shown in Table The best results are marked in rod. The full results can be found in the appendix C where we also provide a statistical analysis against the best baseline models each dataset. Our network achieves the best comparable results on the ECT, ETTm, Traffic, and Weather Datasets. Overall, show that DARTS-IS automatically found architectures that are comparable to or better than many other hand-crafted architectures specifically designed for forccasting tasks with only the vanilla series modules.
While on another benchmark, the PEMS dataset, DARTS-TS outperforms all the other baselines for all the tasks, as shown in Table [ This shows that DARTS-TS could adapt to different tasks and suggests optimal architectures for different time series distributions.
6.2 Search Space Analysis
Unlike the search space in other tasks, which contains only a single operation type (Klycchnikov et al. 2020
Krishnakumar et al. 2022 Mehta et al. 2022 Ying el al. 2019); ; our search space encorn encompasses operations from different architectures. This differs from the previous architecture search space and may present additional challenges to the optimizers. To provide a basic understanding of our search space, we randomly sample 3, 000 configurations from our search space and evaluate them on the ECL and Traffic datasets.
Speermen Comrelation
200
g0000
Dersity
200
150
100
100
0.44
0.14
0.16
0:18
0.20
0.35
0.40 0.45 0.50 0.55 0.60 0.85
ECL 95 Test MSE Loss
Traffic 96 Test MSE Loss
Texe MGE Cectriiiyy 92
Figure 4: Random Search Evaluation Results on the ECL. and Traffic datasets. (Left), MSE loss distributions on the ECL dataset. (Middle), MSE loss distributions on the Traffic datasct. (Right), loss distributions on the Traffic ECL dataset from the same architecture configurations
The random configuration performances shown in Figure 4 For the same set of hyperparameter configurations, the MSE losses these two tasks follow different distributions: losses the ECL datasets are more skewed towards the lower bounds, while the losses the Traffic datasets are more uniformly distributed. This indicates different difficulty levels for the optimizers to search for the optimal architectures on different datasots. To chock whether the performance of the same hyperparameter configurations could transfer to different tasks, we plot the performance of the same configurations from the two tasks in the right part of Figure 4 The overall trend shows the consistent performance of the worst-performing configurations on both datasets. However, the performance diverges as we move towards the near-optimal configurations. Hence, the optimal configuration for one task may not work equally well for another task. There is still a need to further search for the optimal model on the target datasel. However, since the worst configurations
httpp: //gptt/ www.t//ee//tteeeeeeeeeecc wttpthhhnnntnnneeeeeeee http:/iineeSSrri---Lbba
"Our code can lound http::://giitib..o///aatt/////// http://ttt....aaatoo//////oaaoat

---

Published in Transactions on Machine Learning Research (09/2025)
0.2
Traffic
0.084
0.0094
-0.17
-0.078
-0.085
0.096
-0.013
0.025
0.062
0.021
0.26
0.09
0.1
Spearman Cor
0.0
-0.1
ECL
-0.26
-0.27
0.29
0.04
0.12
0.22
0.15
0.11
-0.25
-0.24
0.12
0.2
-0.2
fisher
grad norm
grasp
jacov
12 norm
flops
params
nwot
plain
snip
synflow
zen
Figure Spearman correlation between different ZC metrics and the ovaluation test MSE losses
perform consistently on the two tasks, we could incorporate the runs from other tasks as prior Hvarfner
et al. 2022 Mallik et al.) 2023).
Operations in our search space come from different architecture types. This provides further challenge for the optimizers the performance difference among the operations the scarch space can no longer be describod with simple proxies. For instance, in the computer vision NAS benchmark search space, layer could, in most cases, achieve better performance than 1 layer, the number of parameters of 3 x 3 convolutional layer is larger than that of 1 x 1 convolutional layer. However, it is questionable if Transformer layer (with 12d parameters) will outperform an LSTM layer (with Sd parameters) since these two operations follow different computational rules. Hence, the widely applied zero-cost proxies applied the computer vision NAS benchmarks might no longer work in our search space.
3 convolutional
We computed the zero-cost (ZC) metrics for the sampled architectures. Following the settings from NAS- Bench-Suite-Zerro (Krishnakumar et al. 2022), evaluate the following
cost proxies: fisher (urner 2019 flops, number of parameters (Ning et al. 2021), grad-norm, 12-norm, plain (Abdelfattah et al. 2021 grasp (Wang el al.] 2019), jacov, nwol (Mellor et al. 2020 snip (Lee et al. 2019). synflow (Tanaka et al. 2020), and zen-score (Lin et al.] 2021). We evaluate the ZC scores of all the sampled models and compute the Spearman correlation between the ZC scores and their test performance. The results are shown Figure 5 Supervisingly, even though many zero-cost proxies do not limit their application to pure CNN or image classification tasks, nearly all of the zero-cost, proxies fail our search space. This indicates that the existing zero-cost proxies might not generalize well more complex search spaces. Additionally, shown in Figure even the same configuration might perform differently different datasets. This further shows the necessity of defining new data-dependent vero-cost proxies that work well across different architecture types.
To provide further insight into the design of ZC proxies for our search space, we check the performance of the architectures that contain at least one operation within our search space (Lopes et al. 2023 This provides a preliminary estimattin of the strength of each operation on the architecture performance. The result is shown in Figure 6 Although the sequential network with scquential dccoder familics performs similarly to the sequential network with linear decoder the ECL dataset, the gap becomes larger when the same architecture is evaluated on the Traffic dataset. However, the linear decoders might also lead to much worse model, as the worst-performing linear decoders' loss is much higher than that of the sequential decoders.
Among the sequence operations, the TCN families (TCN and separated TCN models) achieve better median and 25th quantile performance on both datasets, which indicates that the dataset requires the models to focus more on local dependencics. While transformer families dominate the other benchmark, their modian and 25th quantile losses are higher than those of the other operations. Although the MLP mixer encoders quite close (and sometimes
optimal) to the other operations, ML. Mixer decoders perform
10

---

Published in Transactions on Machine Learning Research (09/2025)
worse than the other sequential decoding operations. This highlights the importance of incorporating various encoder-decoder architectures into forecasting architecture designs.
For flat operations, MLP Flat and SEBEATS-Genert achieve better performance compared to N-BEATS- Seasonal and N-BEATS-Trent models. However, the top-performing T-BBAAS--eassne models achieve a lower loss compared to the top-performing MLP flat layers. This might indicate that the inductive bias contained in the S-BEATTS-SeaOon models, i.e. the forecasting results are periodic, would help the model achieve better performance.
Operation Scores ECL
Operation Scores Traffic
Skip HBEATS
Skip Fitt
NBEATS
NBCA1S.G
NOCATS
Seo.
Sep. TSN D6
L5TH
GBU
Skip Con Sun
Skip Con Sat. En
Transfortee
"randorma" 2%
MLP Mover
MLP Miser
Test MSE
Test MSE
Figure 6: Performance of the randomly sampled models that contain least one operation
We further show the scores of the architectures that do not contain the target operations in Figure / In this case, a higher score indicates that the target function is more important for the target task. Results show that many operations have different impacts on the searched architectures. For instance, missing N-BEATS-Tremt results in a poorer optimal architecture for the ECL. dataset, while a missing separated TCN results poorer merf-performing architecture on the same dataset. For the traffic dataset, removing the same operation could even yield a better-performing set of architectures. This shows that even the same operation might have completely different influence on the architectures in the search space.
6.3 Model Efficiency Analysis
The growing demand for forecasting models has posed more challenges to the forecasting networks: the network should be fast, so that it can quickly predict the following trend. Additionally, networks need to contain fewer parameters and consume less memory so we can deploy them on embedded systems.
To further show that our network could find an efficient and strong network, we ask all the networks to do single forward pass and backpropagation with the series within the Traffic and ECL dataset, where each series contains 862 and 321 variables, respectively. We set the batch size of the series to 32 and the look-back window size to 96. The networks are then asked to predict the future series with a forecasting horizon of 96. This experiment is executed on one single Nvidia 2080 TI GPU with 11 GB GPU RAM Due to this
Howwevrr scarch the onc-shot model on an Nvidia A100 CPU with 40 CB CPU RAM.
Traffic
All OP:
All 0%
XT2 T
MBEATS
NSSAAS
NLP Bon
MLP Fat
Skip Con.
Skip Con
Sep. 2.N
Transfrme
Cit De
HIP
Skip
Skip
Sepp 11:34
Sep 10% En
1.114
USIM
MLP Miver
NUP Kiner
0.14
0.16
0.18
0.28
0.35
0.40
0.45
0.50
0.55
0.60
8.5
MSE
Test MSE
Figure 7: Performance of the randomly sampled models that do not contain the specific operation
11

---

Published in Transactions on Machine Learning Research (09/2023)
Network Latency Information on the Traffic Dataset
Network Latency Information on the ECL Dacaset
GPU Nemory
Inference
3500
GPU Memory
Inference Time Imsiteri Modell (M)
s443
(Train)
Time irnaliter)
350
(Train)
120
GRU Momery Usage (his)
8000
Memory [Inference]
Model Size
300
Inferrene Time Dong
GRU Memory Usoge (MB)
2500
3000
GPU Memory (inferenca)
100
Inference Time (mociaa
Time (ms/tter)
250
Time (msrittri)
200
4000
150
1500
CL
60
1000
1000
40
2000
500
776
144
14.2
72 40
Dlinear
DARTS-TS
Auto Former
Dlinear
DARTSSTS
Trrnnffrmer
former
TST
Fermer
TST
Figure 8: Latency information of different networks on the Traflic (left) and ECL (right) datasel. We set the look- back window size and the forecast horizon to 96. The batch size for both training and inference is set as 32.
memory constraint, some of the networks, such as ModernTCN, cannot into this GPU with our setup and do not appear in this comparison.
As shown in Figure g while having comparable performance with iTrrnsformer and PatchTST the traffic dataset, approach requires around 2x less GPU memory and is faster than the i'Transformer and 3x less GPU memory and speed up compared to PatchTST during the training phases. During the test phases, the required GPU memory is even further reduced to around 300 MB, which is 4x less than the iTrransfrmer and 6x less than the PatchTST, and only requires 3x more memory compared to a linear layer. A similar trend can be observed on the ECL dataset: DARTS-TS requires 7x less GPU memory and is 2.8x faster than PStchTST to achieve better performance.
Although we do not optimize for the latency operations already allows us to achieve similar performance with much less compulational power required. e.g., to cover the same receptive field, we might need to stack multiple TCN layers or increase their kernel size. Both approaches result in an increased amount of computational time and memory consumption. However, if we instead apply a transformer or RNN layer on top of a TCN layer, the model could still learn the global and local information with less computational power required. Additionally, models such as PatchTST (Nie let al., 2023) decompose one single multi-variant series instance into multiple uni-variant series instances and therefore, need to run the forward pass multiple times for single multi-variant time series instance. DARTS-TS only performs one forward pass to get the forecasting results, which could significantly reduce the required computational power and the memory requirements. Hence, DARTS-TS could scarch for a more lightweight architecture, even if latency information is not the optimization target. However, we could also apply other optimization approaches, such as multi-objective differentiable architecture search (Sukthanker et al. 2024); , to search for the architecture that concerns both latency and accuracy.
optimization objects explicitly, the combination of different
tcm
nbeuts
(in)
sep Iem
tem
(out)
mlp mix
(in)
abcats
nbcats. S
mlp
(oal)
sep_ien
nheats
Figure 9: One optimal architecture on the ECL dataset, this is an encoder-only architecture and will be trained with an MSE loss
6.4 The optimal architectures
We show one of the optimal architectures found on the ECL datasct Figure Its Seg Net an encoder-only architecture that composed of MLPMixer, TCN, and Separate TCN modules. TCN modules are still preferable over the other components, showing that the ECL. dataset might prefer a model that focuses on the
12

---

Published in Transactions on Machine Learning Research (09/2025)
1stm
mlp mix
(in) 0
gru
2
sep_ten
(in) 0
axip_connection
Istm
connectinnn
sep_
(our)
transformer
Istm
(out)
(in)
(in)
mlp mix
mlp mix
skip_ connection
mlp
mlp
mlp
(in)
skip connection
nbeats
mlp
(out) 4
3
Figure 10: One optimal architecture on the ETTm2 dataset, this is an encoder-decoder architecture and will be trained with a quantile loss
local correlation. While the Flat Net contains lots of N-BEATS-SSasouul modules, this indicates the strong seasonal and little trend signal that the datasct contains.
We present another encoder-decoder archilecture in Figure 10
This architecture optimized on the ETTm2 dataset. applies two MLP-Mixer layers to the input node to first collect the global information from the input sequence and then apply two Separate TCN modules top of that. Our optimizer also seleets many LSTM modules for the decoder architectures, even if the corresponding encoder edges do not provide any hidden states. Additionally, no TCN module in the decoder layer. This is different from the optimal encoder architecture, where lots of TCN family components are selected. This indicates that the decoder networks would require modules that provide a global perspective to utilize all the information from the encoder networks, and therefore, the priority of TCN modules might decrease. More optimized architectures can be found the appendix
7 Discussion and Future Work
This work proposes a general search space for time series forecasting lasks. Our search space allows the components in the search space to freely connect to each other and form new network. This search space contains most of the forecasting architectures and can be easily extended to other frameworks. For instance, Trransformer (Liu et al.) 2024b) can considered as a special case the Flat Net and searched jointly with the other modules from this family. Decomposing multi-variant series into single variant series and applying special kernel result in PatchTST (Nie et al. 2023). and then we can search for the optimal architecture jointly with the other Seq Nel.
In Section e we showed that DARTS-TS can search for lightweight architecture while keeping strong performance various datascts. However, unlike traditional supervised problems where all the sample instances are i.i.d., time series data might have the problem of distribution shift. The optimal model searched on the validation set might no longer work well on the test set. This provides a future challenge for the AutoML forecasting frameworks from the meta-level, i.c., they need to be able to pre-determine the optimal approach to evaluate the generalization ability.
8 Acknowledgement
The authors gratefully acknowledge the computing time provided to them on the high-performance computers Noctua2 at the NHR Center PC2 under the project hpe-prf-intexml. These are funded by the Federal Ministry of Education and Research and the state governments participating on the basis of the resolutions of the GWK for the national high performance computing at universities (www. wwwar-eeriin.dddusneeeppaaee
Difan Deng was supported by the Federal Ministry of Education and Research (BMBF) under the project AI service center KISSKI (grant no.01IS22093C).
13

---

Published in Transactions on Machine Learning Research (09/2025)
References
M. Abdelfattah, A. Mchrotra, L. Dudziak, and N. Lanc. Zero-cost proxies for lightweight NAS. In The Ninth International Conference on Learning Representalions (ICLR 21) ICL 2021
A. Alexandrov, K. Benidis, M. Sohlke-Schneider, V. Flunkert, J. Gasthaus, T. Januschowski, D. Maddix, S. Rangapuram, D. Salinas, J. Schulz, L. Stella, A. Türkmen, and Y. Wang. Gluonts: Probabilistic and ncural time series modoling in python. Journal of Machine Learning Research, 21:11611-116:; 2020.
Ansari, L. Stella, A. Türkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. Rangapuram, S. Pineda- Arango, S. Kapoor, J. Zschiegner, D. Maddix, M. Mahoney, K. Torkkola, A. Wilson, M. Schneier, and Y. Wang. Chronos: Learning the language of time series. arXiv:2403.07811 [cs.LG], 2024.
R. Hyndmanand G. Athanasopoulos. Forecasting: principles and practice. OTexts, 3. edition, 2021.
J. Ba, J. Kiros, and G. Hinton. Layer normalization, 2016.
S. Bai, J. Kolter, and V. Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.0127 [cs.LG], 2018.
J. Beitner. PyTorch Forecasting: Time series forecasting with PyTorch. gi github.com/jdbll com/jdb78/
pytorch- forecastire 2020.
Y. Bian, Ju, J. Li, Z. Xu, D. Cheng, and Q. Xu. Multi-patch prediction: Adapting Iims for time series representation learning. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 4 1st International Conference on Machine Learning (ICML 24), volume 251 of Proceedings of Machine Learning Research. PMLR, 2024.
G. Box, G. Jenkins, G. Reinsel, and G. Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish,
A. Radford, I. Sutskever, and Amodei. Language models are few-shot learners. In Larochelle al.
(2020), pages 1877-1901.
C. Chen, K. Petty, A. Skabardonis, P. Varaiya, and Z. Jia. Freeway performance measurement system: Mining loop detector data. Transportation Research Record, 2001. URL
31t////0.11//774-11
D. Chen, L. Chen, Z. Shang, Y. Zhang, B. Wen, and C. Yang. Scale-aware neural architecture search for multivariate time series forecasting, 2021a.
H. Chen, M. Lin, X. Sun, and H. Li. NAS-bench-zero: A large scale dataset for understanding zero-shot neural architecture search, 2022. URL
M. Chen, H. Peng, J. Fu, and H. Ling. Autoformer: Scarching transformers for visual recognition. In Proceedings of the 24nd IEEE/CVF International Conference on Compuler Vision (ICCV'21) Com 2021 pages 12270 12280.
S. Chen, C. Li, S. Arik, N. Yoder, and T. Pfister. TSMixer: An all-MLP architecture for time series forecast- ing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:://peerreiier net/ forum?id=wbpx/uXxmm
K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In A. Moschitti, B. Pang, and W. Daclemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1724 1734. Association for Computational Linguistics, 2014.
14

---

Published in Transactions on Machine Learning Research (09/2025)
X. Chu, T. Zhou, B. Zhang, and J. Li. Fair darts: Eliminating unfair advantages in differentiable architecture search. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors, 16th European Conference on Computer Vision (ECCV'20), pages 465 480. Springer, Springer, 2020.
J. Chung, K. Kastner, L. Dinh, K. Goel, A. Courville, and Y. Bengio. recurrent latent variable model for sequential data. Xrx:::1506. 02216x6 [cs, 2015.
Proceedings of the 24nd IEEE/CVF International Conference on Computer Vision (ICCV211) 2021. Computer Vision Foundation and IEEE Compuler Sociely, IEEE.
A. Das, W. Kong, R. Sen, and Y. Zhou. A decoder-only foundation model for time-series forecasting. arxiv:2300. 10688.08.888.00 [cs. 2023.
D. Deng, F. Karl, F. Hutter, B. Bischl, and M. Lindauer. Efficient automated deep learning for time series forecasting. In M.R. Amini, S. Canu, A. Fischer, T. Guns K. Novak, and G. Tsoumakas, editors, Machine Leurning and Knowledgge Discovery in Dalabases. Research Track Europeam Conference, ECML PKDD. ACM, 2022.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pages 4171-4186. Association for Computational Linguistics, 2019.
A. Didolkar, K. Gupta, A. Goyal, N. Gundavarapu, A. Lamb, N. Rosemary Ke, and Y. Bengio. Temporal latent bottleneck: Synthesis of fast and slow processing mechanisms in sequence learning. In Advances in Newral Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS, 2022.
R. Godahewa, C. Bergmeir, G. Webb, R. Hyndman, and P. Montero-Manso. Monash time series forecasting archive. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks. Curran Associates, 2021.
I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors. Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS171) 2017. Curran Associates.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR'16), pages 770-778. Computer Vision Foundation and IEEE Computer Society, IEEE, 2016.
H. Hewamalage, C. Bergmeir, and K. Bandara. Recurrent neural networks for time scrics forecasting: Current status and future directions. International Journal of Forecasting, pages 388 427, 2021.
S. Hochreiter and J. Schmidhuber. Long Short-Tern Memory. Neural Computation, 98)):1735-1780, 1997. Based on TR FKT-207-95, TUM (1995).
S. Hochreiter, A. Younger, and P. Conwell. Learning to learn using gradient descent. In G. Dorffner, H. Bischof, and K. Hornik, editors, Proceedings of the 11th International Conference Artificial Neural Networks (ICANN'01), pages 87 94. Springer, 2001.
C. Hvarfner, D. Stoll, A. Souza, L. Nardi, M. Lindauer, and F. Huttor. nBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization. The Tenth International Conference on Learning Representations (ICLR 22) ICL (2022).
Proceedings of the International Conference on Learning Representations (TCLR'17), 2017. ICLR.
Proceedings of the International Conference on Learning Representations (ICLR'19), 2019. ICLR.
Proceedings of the International Conference on Learning Representations (I)),'20), 2020. ICLR.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Proceedings of the International Conference on Learning Representations (ICLR'21), 2021. ICLR.
Proceedings of the International Conference on Learning Representations (ICLR 22), 2022. ICLR.
Proceedings of the International Conference on Learning Representations (TCLR 23), 2023. ICLR.
Proceedings of the International Conference on Learning Representations (CCLR'24), 2024. ICLR.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning (ICML'IN) 15), volume 37. Omnipress, 2013.
S. Jiang, Z. Ji, G. Zhu, C. Yuan, and Y. Huang. Operation-level early stopping for robustifying differentiable NAS. In Proceedings of the 36th International Conference on Advances in Newral Information Processing Systems (NeurIPS 23), 2023. URL
H. Jin, Q. Song, and X. Hu. Auto-Keras: An efficient neural architecture search system. Tn A. Teredesai, V. Kumar, Y. Li, R. Rosales, E. Terzi, and G. Karypis, editors, Proceediiggs of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD'19) '9), pages 1946-1956. ACM Press, 2019.
G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q Ye, and T.-Y. Liu. Lightgbm: A highly efficient. gradient boosting decision tree. In Guyon et al. 20177)
T. Kim, J. Kim, Y. Tae, C. Park, J. Choi, and J. Choo. Reversible instance normalization for accurate timo-sorics forecasting against distribution shift. In The Tenth International Conference on Learning Representations (ICLR 22) ICL (2022).
N. Klyuchnikov, I. Trofimov, E. Artemova, M. Salníkov, M. Fedorov, and E. Burnaev. NAS-Bench-NLP. Neural Architecture Search benchmark for Natural Language Processing. orXie:2006.071 071166l [cs. 2020.
A. Krishnakumar, C. White, A. Zela, Renbo R. Tu, M. Safari, and F. Hutter. Nas-bench--uuit-zere Accelerating research on zero cost proxies. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Proceedings of the 35th International Conference on Advances in Neural Information Processing Systems (NeurIPS 22). Curran Associates, 2022.
G. Lai, W. Chang, Y. Yang, and H. Liu. Modeling long- and short-term temporal patterns with deep neural networks. In K. Thompson, Q. Mei, B.Dviison, Y. Liu, and E. Yilmaz, editors, International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 95-104. ACM, 2018. URL https: org/ 10.
I. Lana, J. Del Ser, M. Vélez, and E. Vlahogianni. Road traffic forecasting: Recent advances and new challenges. IEEE Intell. Transp. Syst. Mag., 10(2):93- 10093---,,,000 2018
H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, editors. Proceedings of the 33rd International Conference Advonces in Neural Information Processing Systems (NewrIPS120), 2020. Curran Associates.
N. Lee, T. Ajanthan, and P. Torr. Snip: single-shot network pruning based on connection sensitivity. In The Seventh International Conference on Learning Representations (ICLR' TCL. 2019
S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y. Wang, and X. Yan. Enhancing the locality and breaking the memory bottlenock of transformer on time series forecasting. In Wallach et al. (2019) pages 52544.254.
B. Lim, S. Ari,, N. Loeff, and T. Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4):1748 1764, 2021.
M. Lin, Wang, Z. Sun, H. Chen, X. Sun, Q. Qian, H. Li, and R. Jin. Zen-nas: A zero-shot nas for perforace image recognition. In Proceedings of the 24nd IEEE/CVF International Conference on Computer Vision (ICCV'21) Com 2022)) ) pages 347-356:
16

---

Published in Transactions on Machine Learning Research (09/2025)
H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukeuoglu. Hicrarchical representations for efficient architecture search. In The Sirth International Conference on Learning Representations (ICLR'18 (18). TCLR, 2018.
H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. In The Seventh International Conference on Learning Representations (ICLR'19) ICL 2019))
S. Liu, H. Yu, C. Liao, J. Li, W. Lin, A. Liu, and S. Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In The Tenth International Conference on Learning Representations (ICLRR 22) TCL. (2022). URI. htt.::///pperrreeeeeeeee///oodds
X. Liu, J. Liu, G. Woo, T. Aksu, Y. Liang, R. Zimmermann, C. Liu, S. Savarese, C. Xiong, and D. Sahoo. Moirai-moe: Empowering time series foundation models with sparse mixture of experts. 2024a.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXic:1907.1169 [cs. CLJ, 2019b.
Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, and M. Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations (ICLR 24) ICL (2024).
Z. Liu, Y. Lin, Y. Cao, H. HU, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the 24nd IEEE/CVE International Conference on Computer Vision (ICCV'21) Com 2021] 3 pages 10012-10022.
V. Lopes, B. Degardin, and L. Alexandre. Are neural archttecure search benchmarks well designed? deeper look into operation importance. arXiv:2303. w:Xi::2303.00038/es//. 16938/cs..1.0600000 I.G.) 2023.
Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In The Fifth International Conference on Learning Representations (ICLR'17 TCL. (017)).
D. Luo and X. Wang. ModernTCN: A modern pure convolution structure for general time series analysis. In The Twelfth International Conference on Learning Representations (ICLR 24) [CL (0024). URL
sttps://openreeven net/ fond//oorummddddpllleerttt
Z. Lyu, A. Ororbia, and T. Desell. Online evolutionary neural architecture search for multivariate non- stationary time series forecasting. Appl. Soft Comput., 2023. URL ftttp:://doi.org/10...00 asoc
2023. 110522
S. Makridakis, E. Spiliotis, and V. Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecusting, pages 54 74, 2020.
S. Makridakis, E. Spiliotis, and V. Assimakopoulos. M5 accuracy competition: Results, findings, and conclu- sions. International Journal of Forecasting, pages 1346 1364, 2022. URL http://ciieeee/rrccceee////////t
N. Mallik, E. Bergman, C. Hvarfner, D. Stoll, M. Janowski, M. Lindauer, L. Nardi, and F. Hutter. Priorband: practical hyperparamcter optimization in the age of deep learning. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine, editors, Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS'2) 23). Curran Associates, 2023.
Y. Mehta, C. While, A. Zela, A. Krishnakumar,, G. Zabergia, S. Moradian, M. Safari, K. Yu, and F. Hutter. NAS JAS-Bench-Suite NAS evaluation is (now) surprisingly easy. In The Tenth International Conference Learning Representations (ICLR '22) TCL 2022).
J. Mellor, Turner, A. Storkey, and E. Crowley. Neural Architecture Scarch without training. In H. Daume III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning (ICML '20), volume 98. Proceedings of Machine Learning Research, 2020.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Y. Nie, N. Nguyen, P. Sinthong, and J. Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Fleventh International Conference on Learning Representations [ 23) TCL (002)). URL https://openreeiee,
X. Ning, C. Tang, W. Li, Z. Zhou, S. Liang, H. Yang, and Y. Wang. Evaluating efficient performance estimators of neural architectures. In Ranzato al. (2021).
A. Oord, S. Dicleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Scnior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. In the 9th ISCA Speech Synthesis Workshop,
page 125, 2016.
B. Oreshkin, D. Carpov, N. Chapados, and Y. Bengio. N-BEATS: neural basis expansion analysis for interpretable time series forecasting, In The Ligth International Conference on Learning Representations (ICLR 20) ICL 22020).
Z. Pan, J. Cai, and B. Zhuang. Stitchable neural networks. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR 23), pages 16102-16112. Computer Vision Foundation and EEEE Computer Society, LEEE, 2023. URL https://doii org/10. httgg//..../////P////e L109/CVPR52729 2023.01545
A. Paszke, S. Gross, F. Massa, A. Lerer, et al. Py'oorch: An imperative style, high-performance deep learning library. In Wallach et al. 2019 pages 8024 8035.
H. Pham, M. Guan, B. Zoph, Q Le, and J. Dean. Efficient Neural Architecture Search via parameter sharing. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning (ICML'IS) 18), volume 80. Proceedings of Machine Learning Research, 2018.
M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vanghan, and Y. Dauphin, editors. Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NewrIPS'21), 2021. Curran Associates.
O. Ronneberger, P. Fischer, and T. Brox. U-Net convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Assisted Intervention, pages 234-241, 2015.
D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski. Deepar: Probabilistic forecasting with autore- gressive recurrent networks. International Journal of Forecasting, pages 1181 1191, 2020.
X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting, In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Proceedings of the 29th International Conference on Advances in Neural Information Processing Systems (NeurIPS115 15), pages 802 810. Curran Associates, 2015.
N. Srivastava, G. Hinton, A. Krizhevsky, 1. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929 1958, 2014.
R. Sukthanker, A. Zela, B. Staffler, S. Dooley, J. Grabocka, and Frank Hutter. Multi-objective differentiable neural architecture search. ar arXiv:2402. 1e0.0...11110 18213fcs. 2024.
I. Sutskever, O. Vinyals, and Q. Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, editors, Proceedings of the 28th International Conference on Advances in Neural Information Processing Systems (NeurIPS'1A) Curran Associates, 2014.
H. Tanaka, D. Kunin, D. I.. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Larochelle al. 2020)).
A. Trindade. Electraity<oodDDigraas2011111111 hutps://do..org/10244////// /C58888
UCI Machinc Learning Repository, 2015. DOL:
R. Turner. The bayes opt benchmark documentation. https://aayesnaa..reaattdll 2019.
18

---

Published in Transactions on Machine Learning Research (09/2025)
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Guyon et al. 2017).
H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche Buc, E. Fox, and R. Garnett, editors. Proceedings of the 32nd International Conference on Advances in Neural Information Processing Systems (NeurIPS 2019. Curran Associates.
X. Wan, B. Ru, P. Esperança, and Z. Li. On redundancy and diversity in cell-based neural architecture search. In The Tenth International Conference on Learning Representations (ICLR' 22) ICL (2022).
C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Represenlations [CL (2019).
R. Wang, M. Cheng, X. Chen, X. Tang, and C. Hsich. Rethinking architecture selection in differentiable nas. In International Conference on Learning Represenlation ICL (202T)
R. Wen, K. Torkkola, B. Narayanaswamy, and D. Madeka. multi-horizon quantile recurrent forecaster. In 31st Conference on Neural Information Processing Systems, Time Series Workshop, 2017.
C. White, W. Neiswanger, and Y. Savani. BANANAS: Bayesian optimization with neural architectures for neural architecture search. In Yang et al. (2021). pages 11293-10301.
H. Wu, J. Xu, J. Wang, and M. Long. Autoformer: Decomposition transformers with Auto-Correlas for long-term series forecasting. In Ranzato et al. 20021
H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations (ICLR 223 ICL (023). URL httts:// hutp:/rw.eeet//affaaoouuuuouppp
Q. Yang, K. Leyton-Bronn, and Mausam, editors. Proceedings of the Thirly-Fifth Conference on Arlificial Intelligence (AAAI21), 2021. Association for the Advancement of Artificial Intelligence, AAAI Press.
C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter. BAS-Bench-101: Towards reproducible Neural Architecture Search. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning (ICML '1)) volume 97, pages 7105 7114. Proceedings of Machine Learning Research, 2019.
A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, T. Brox, and F. Hutter. Understanding and robustifying differentiable architecture search. In The Figth International Conference on Learning Representations (ICLR 200) ICL (2020). URL httpps://peereevier wttp:/eteeeotootttdddtteeeeeeeee
A. Zeng, M. Chen, L.. Zhang, and Q. Xu. Are transformers effective for time series forecasting? In B. Williams, S. Bernardini, Y. Chen, and J. Neville, editors, Proceedings of the Thirty-Seventh Conference on Artificial Intelligence (AAAT pages 11121 11128. Association for the Advancement of Artificial Intelligence, AAAI Press, 2023. URL battps://dat http:///ooo.or//////// 111111......1110 10099160aaa www.9.799.. v3719.26317
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Yang et al. (2021) URL
https://doi http://bo..rr/111.166///aaaaaaa
1609/69aiaa... 2t..://aaaaivvv11........2 v35112.17325
L. Zimmer, M. Lindauer, and F. Hutter. Auto-Pytorch: Multi-fidclity metalcarning for efficient and robust AutoDL. IEEE Transactions Pallern Analysis and Machine Intelligence, 43:3079 3090, 2021.
B. Zoph and Q V. Le. Neural Architecture Search with reinforcement learning. In The Fifth International Conference on Learning Representations (ICLR' ICL 2017).
B. Zoph, V. Vasudevan, J. Shlens, and Q. Le. Learning transferable architectures for scalable image recognition. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR'18) Computer Vision Foundation and JEEE Computer Society, IEEE, 2018.
19

---

Published in Transactions on Machine Learning Research (09/2025)
A Operations Deatails
In section 4.1 we briefty introduced the operations within our scarch space. Here, we will provide the details of these operations.
A.1 Seq Net
For Seq net, encoders and decoders share the same operation sets. Overall, we have the following operations:
TSMixer (Chen et al. 2023), a full MLP-basod Soquential operation. Each TSMixer operation is constructed by the time and feature mixing blocks. The time mixing blocks use fully connected (FC) layer to mix the information across different time steps. In contrast, the feature mixing block uses another set of FC layers to enhance the information within each channel. Our TSMixer encoders follow the design from Chen et al. 2023), and the size of the feature mixing layer is set as 2 x dmodde-- For TSMixer decoders, we first construct the input feature map with the encoder network outputs. This concatenated feature is then provided to the time mixing modules to recover its size and return the forccasting horizon. Additionally, we use LayerNorm (Ba et al.] 2016) instead of BatchNorm ÇToffe and Szegedy, 2015) in TSMixer to ensure that the operations within edge generate the feature maps that follow the same distribution (since all other components in our module used LayerNorm to normalize the feature maps).
LSTM (Hochreiter and Schmidhuber 1997 is an RNN model. It maintains set of cell gale states to control the amount of information passed to the next time steps. Therefore, it suffers less from the known gradient explosion problems in the RNN families. However, the introduction of the gates brings lots of additional paramcters to the modules. As described in Section 4.2 the hidden states of the LSTM decoder are initialized by the last time step of the encoder feature from the corresponding layer and another feature generated with linear embedding (or directly from the corresponding LSTM encoder).
GRU (Chung al. 2015) is yet another type of RNN network. It only maintains one hidden state and therefore requires a much smaller number of parameters and computations compared to the LSTM. The setting of the GRU Encoder/Decoder is nearly the same as the LSTM families. The only difference is that we do not maintain an additional state to initialize the GRU decoders.
Transformer (Vaswani et al. 2017) has attracted lots of attetion from different research fields and is therefore widely applied in time series forecasting tasks. Here, we use the vanilla Transformer implemented in PyTorch (aazkke et al. 2019) with the hidden size to be x dmodel-
TCN (Bai et al., 2018) is a lype of CNN network that can capture the local correlations among different time steps. However, the receptive field of the TCN network is restricted by its kernel size. To efficiently increase the receptive field without introducing too much computation overhead with a larger kernel, TCN implemented dilated convolution operations. Here we implement a similar approach, for edge i i j that starts from node j to node i as cell k, set its dilation 2(/+k-nin), where Rin is the number of inputs of the current cell. Hence, the deeper convolutional layers will have a larger receptive field. For the TCN decoders, we concatenate the feature maps from the corresponding encoder layer with our input feature and feed them together to our TCN network. This idea is similar to U-Net (Ronneberger et al. 2015), where features with similar levels should be gathered together.
SepTCN (uo and Wang 2024 is a variation of the vanilla TCN. We replace the full convolutional operations TCN with a combination of a separated TCN model and another linear layer.
Skip Connection, an identity layer that passes input to the next level. However, for the skip connection encoder, we still have linear layer to provide initial cell gate states to the corresponding LSTM decoder layers.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Another type of Seq decoder is linear decoder. We apply a linear layer that transforms the encoder's output feature map with size RIB.L to RlB.H.N) and feed it further to the forecasting heads.
A.2 Flat Net
We only consider the MLP families in our Flat Nel to minimiie the computational overhead when applying our approaches to problems with higher series amounts. Given an imput feature series with shape [B,N, L]. we first concatenate it with a zero tensor with shape N H] that represents the prediction results. Then this concatenated tensor is fed to the Flat encoder.
This architecture family includes:
a simple Lincar model (Zeng et al. 2023). This lincar layer maps its input features with shape [B, N, L, H] to feature map whose size is equal to the forecasting horizon: [B, H]. Then the output feature is concatenated with the first part of the imput feature maps. If the network only contains skip connections for all but the last layer, then this network becomes a DLinear model (Zeng et al. 2023). If the operation is not the output layer, we attach an activation and normalization layer to introduce some non-linearity.
N-BEATS (Oreshkin et al. 2020) modules. N-BEATS hicrarchical modulc where each model is composed of multiple stacks. Each stack contains multiple blocks. Each block has FC stack with multiple FC layers, forecasting head, and backcasting head. In our search space, each N-BEATS edge corresponds to an N-BEATS block. N-BEATS provides three variations: generic, trend, and seasonal. We include them all in our search space. Since the only difference between these variations their prediction heads. We ask the models to share the same FC layer backbones and only diverge at the forecasting heads.
Skip Connection, skip connection layer.
A.3 Forecasting Heads
We also consider the forecasting heads part of the operations within our graph. Each of these heads is composed of one or multiple linear layers that map the Seq Nel | output feature maps to the desired multiple-variable target values. These linear layers then trained with set of specific training losses. Let's assume that the target value is and prediction value is y
Quantile loss [Lim et al.; 2021 Wen et al.) 2017) predicts the percentiles of the target values. A quantile head can be composed of multiple heads, and each of the heads is asked to predict quantile. Given a required quantile value q, the quantile loss is computed by by = ma (1 - (5- - y)). In our network, we used the following quantile values: 0.51, 0.9.) The final prediction given by the 0.5 quantile values. quantile head then a set of linear layers whose size the number of quantile values
0.5,
MSE loss is yet another popular choice in time series forecasting tasks. It is computed by LMSE (y - 2) An MSE head is a single lincar layer whose weights are updated with MSE loss.
MAE loss is similar MSE loss. However, instead of computing the mean square error from MSE, computes the mean absolute error: LMAE 1 - yl. Similar to the MSE layer, an MAE head is also composed of one linear layer, but its weights are updated with MAE losses.
We stack these forecasting heads top of the Seq Net decoders and optimize their architecture weights with validation losses.
For Flat Net, there need have additional head if the loss only requires one output
21

---

Published in Transactions on Machine Learning Research (09/2025)
B
Experiment Details
We show detailed information on the dataset that we applied in Table 4 Given the great discrepancy variable size between different datasets, we divide the datascts into two groups: we assign a smaller model to the datasets with lewer variables, such as Weather, Exchange, and ETTs, We then atlach normalization layer within the linear decoder. For the other datasets, design an architecture search space with relatively large model. Additionally, larget variables is larger than our model size, and the final forecasting head does not need to recover the distribution of the target variable from the normalized feature maps.
removed the normalization layer in the linear decoder since the number of
We run our experiments on a Cluster equipped with Nvidia A100 40 GB GPUs and AMD Milan 7763 CPUs. For each dataset, we perform the search over the smallest forecasting horizons and evaluate the same optimal model on all the other forecasting horizons. Each task is repeated times. The resources spent on each evaluation depend on the size of the dataset. The exact GPU hours spent for each task and stage are presented in Table 3 The search is divided into three stages. The first stage is to jointly optimize the network weights and parameters, and the second stage is to select the optimal operations within each edge. Finally, the third stage is applicd to only preserve two edges toward cach node. Overall, it takes roughly 4 hours to evaluate smaller datasets such as ETThs and Exchange. Other tasks might require up to 10 GPU hours. The ablation study requires another 500 GPU hours. Overall, takes roughly 1200 GPU hours to finish the experiments.
ECL
ETThi
ETTh2
ETTml
ETTm2
Traffic
Exchange
Weather
Optimization
3.665
0.845
0.845
3.512
3.594
3.187
0.818
3.273
Operation Prune
2.672
0.730
0.738
2.984
3.024
2.445
0.584
2.316
Topology Prune
0,088
0.024
0.030
0.106
0.106
0.107
0.022
0.078
Search
6.424
1.599
1.612
6.602
6.724
5.738
1.424
5.666
Table 3: GPU hours used for the search stage
For all the baselines, use their official implementation from PST Modern TCN while for the other baselines, we take the implementation from Time-Serre Serie--eees---e Seaeeseerr---brrrr 13
11
and TSMixe 12
Datasct
# Time steps
# Variables
Datasot
# Time steps
#
Variables
ECL
26304
321
ETTml
69680
7
Traffic
17544
862
ETTm2
69680
7
Weather
62696
21
PEMS03
26208
358
Exchange
7588
PEMS04
16992
307
ETTh1
17420
PEMS07
28224
883
ETTh2
17420
PEMS08
17856
170
Table Data set information
Our architecture search framework is two-stage approach. In the first stage, search for the optimal architecture, while in the second stage, we train the proposed network from scratch. We preserve most of the searching hyperparameters from Liu et al. (2018). However, we apply ADAM instead of SGD during the test phases to optimize the network weights. All the optimizers are applied with CosineAnnealinng- WarmRestarts (Loshchilov and Hutter (017)) that restarts learning rates every 20 epochs. We also apply a smaller learning rate and stronger weight decay the smaller datasets to avoid overfitting. The concrete hyperparameter settings during search and evaluation phases are presented in Table 5
All the look-back window sizes set to 336 (for long-term forecasting tasks) and 96 (for PEMS tasks) for both the scarching and testing phases. We divide the datascts into two groups based on their number of
16 htttps///gthuu..cc//ggii..//////
httpe///ghhbbbcccooeeeeeererrrre http:://gthubb.cmm/bb///////
22

---

Published in Transactions on Machine Learning Research (09/2025)
HP Names
Search HP Values
Evaluation HP Values
Weights Optimizers
Architecture Optimizers
big datasets
small datasets
Epochs
40
100
Gradient Clip
0.1
0.1
0.1
Type Learning Rate
SGD
Adam
Adam 0.001
AdamW
0.025
0,001
0.0002 0.1
Weight Decay
0.001
Momentum
Betas
(0.5, 0.999)
(0.9, 0.999)
(0.9, 0.999)
Table 5: Training Hyparameters (HP) during searching and training phases
The PEMSs, traffic, and ECL datasct belongs the big datasot 2. The remaining datascts, including Weather and ETTs, are small dalasets.
variables: 1.
Both datasets share nearly the same architecture, which is mixed network introduced in Section 4.3 The number of Seg and Flat cells is both set to 2, and they could receive 2 and 1 input variables, respectively. However, for the big dataset, we set the number of Seq Net hidden dimensions as 32, while this value for the small dataset is 8. This approach is also applied to the hidden NBATS dimension of Flat Net: we set this value as 256 for the big datasets and 96 for the small datasets. The hyperparameters applied during training and validation are listed in Table
Since the Seq encoder has two imput nodes, instead of feeding the raw input value to both input nodes, decompose the input nodes into trend-cyclical components by using average moving [w et al. 2021
Zeng
ct al. 2023 to ask the edges in the cell to focus on different levels of information.
ECL
ETThl
ETTh2
ETTml
ETTm2
Traffic
Exchange
Weather
Supernet
26.241
0.750
0.750
0.750
0.750
61.841
0.778
1.149
After OP Prune
4.991
0.078
0.079
0.061
0.068
15.237
0.087
0.158
After Topology Prune
4.358
0.042
0.060
0.031
0.040
11.911
0.052
0.079
Table 6: GFLOPS changes after different stages
We now study how FLOPS change within each Prune stage. DARTS-TS search process involves three stages: weights optimization, operation pruning, and topology pruning. We show how the flops change after each stage in Table 5 Most of the FLOPS are substantially reduced after the operation pruning stage, where only one operation is preserved for each edge. This value is further reduced with the Topology Prune stage, where each node only preserves at most two edges inputs.
C Results on all the forecasting horizons
Table and Table show the results w.r.t. each forecasting horizon. While DARTS-TS requires a much smaller amount of resources and parameters, it still shows comparable performances on many datasets.
C.1 Statstical Test
Table 9 and Table 10 show the statistical test against the best non-NAS baseline for each dataset. For the LTSF datascts, the optimal bascline models differed across different datascts. With significant level of 0.05, DARTS-TS is significantly better than the baseline models on most of the benchmarks for ECL, ETTml, and ETTm2 datasets. While for the other benchmarks, DARTS- TS is many times comparable to the best basclines, with the forccasting losses slightly higher or lower than the basclincs. Additionally, the optimal basclines differ across different datasets, which also shows that the architectures need adapt to different. fasks, further highlighting the importance of heterogeneous forecasting architectures.
23

---

Published in Transactions on Machine Learning Research (09/2025)
DARTSTS
Elaest
MAA
08
1:22 3061
133 (M68)
18112 11.00
1-28 13:96
8.23 0061
0323
80
14317 18:35
6.281 (UM1
1348 (M6)
6537 14.05
18.0
11:96
0.2% (1:41
6:22
180
(16)
183 (668)
0.33 80
0089 (3.91
0.30 11.03
8.38 9981
8.68 1149
0.89
68 80
03333
6.31 366
158 (M4)
6587 08:30
0.05 (8.50
0289 02.96
6205 [100
8.33 (M02
15
133
10141
0000
0100 10.00
11097799 11
1330
9334 0061
1.3 1000
03.00
0037
1303 300
2000
033 33311
01033 1130
0333
130 330
11
1-46 (016
13.00
11
E 11:01
8.07 9181
0.03
0.39
04330
311
648 mm
6477 01
(8:11
032) 1100
596 0U00)
130
140 104
1.0 (440)
0.36
15.02
06200 (1:11
0.6 10.00
801 9961
2.001
5.60
64.8 106
130 (06
0221
10:311
0.50 18.00
060-
6490 (114
856 0006
31714
1331 1061
11.00
833 0961
1000
00200
6.6 100
120 (046
03300 (8.00
6534 (00
(1:0
19:
E377 (00)
194 (100)
0.03 (2.00
0420 (101)
0300
837 0061
038 (3.0)
0.28
033 (m
168 (60
0439 01:11
0.204.000 10:00
0455 m1
0331 (11
1336
0.3
15 00
1000
11:
135 9061
1000
038
80
31
035 100
130 1000
0.53 (610
60031130
191
1.2)) (1681
1.5 (660)
0.50 15 03)
11
652
83% 0161
0.0
0.00
0.26
0086
m1
1914 (01-9001
0308 00
0:30 (8.5)
0.0911.00
(400 110
65m (100)
18)
141 (001
1:02 (AD
11.00
0.39
3.67 1,00
0456
0122
0.53 (220
114511.0
136
6TT
1PS.
(810)
6.70 (1:
17% 12L-0061
0241 P-
0:37 (8:1)
00:311 m.
130
E334 1041
0J95
E36T
0.550
55x
23
LEW 1041
130 (033
11.00
1022 11.91
133
111-3 15.30
122 2001
0.00 (621)
(033 11
(341
29:
1.30 (001
81)
0106 (1)
0:00 11:0
18.00
6319 (10)
0455 90
0.007 (8:77)
0338 m)
1790 11:1
880
(1)
19:05
15
E34 1041
1025
133
1351
0232
123911191
106)
11
1724 (010)
0.273:0000 00
0.45 11:0
9 (16)
130 (1:0)
1223 18
0720 311
172-006)
0547 XX
0015 (1:00
0:2211111
1109) (10)
6:11 (600)
0.333:103 03
0:35 ()
130
EAH J066
1244
6.42
(146)
0215
(841)
111111111 m1
1777711 11
110
115 (780
110
E31
130
124 2007
0,55 (1.00
6.5
132
(34m
6.3
(Ue)
0314 19:30
4.73 (8.1)
(1.8)
141
100 1.23
1066
1:50 (11))
0.23 60
6:27 (I
133 (06)
0.09 (1)0)
1325 18 B0
00/1 31
6:30 (101
178 (016)
0585 XC
0.04 (8.00
0381 0g
0.48) (10]
6: (W/0)
Table 7: The full evaluation results on long-term forecasting datasets. We evaluate each model five times and take the mcan of the final results. The optimal models are marked in red, and we underline the models that are not significantly worse than the optimal.
DARES 75
PIALTST
AutiGorace
MAAR
$445
MAR
3/A
MAVE
PED15
32
GURG 13 DC
F4L72 11.01
1,00 20/6
0.175 (RAX)
ILL12 18.80
D1221 13:96
5400 [UX
8:07 DUR)
0,75 (1,00
0,085 18
13.30
1:200 SUR1
1.8G (MAR)
14.02 (8.00)
(A277 11.00
E.3R7 TUNL
(1-113
3.111
(8.00)
13:91
8:15
0.15
19.33
JURI
3.58
(U865 13.00)
(3.44
18
60201 31-01
1.63-00 16-35
1/91
0.779
11:0)
028. 31:0
8:0 [U8]
0:88 68
01200 18:91
0.000 19.95
633 3160
13:3 (916)
0.770
0.63: 11m
04200 11.01
1.234
3.20 (8.00
0.308 (2.00
0J888
13:06
6:341 (UX
8.37 (08)
0.113 (846)
0.58 11:4
0.00 10.32
6.30 3U61
1 (0/6)
0.336 (8:63)
06077 13-29
6467 1U0
0.02111
1558
53
01276
11:01
1.30
(610)
13.00
11:91
6397 100
855
8.58 (888)
81
15.32
1.461 31:1
0.25 08:27)
0185-13.55 51)
11.0
12 16000
01100 21.00
1.84
1480 10.80
0.19)
11:05
0400
8.4
8:23 88.0
0.172
6311 1661
180
1486 12:02)
(330) 1105
46
11133 18 311
12216 31.11
180 11101
0,27 (2.1)
13:1
028 H0X
812 (0R)
8100 (8
0,85 80
0575 11.32
0.4 TUR)
153
14.33 (8:17)
1322213:0
0472 100
11115111
152
T10
311
1101 706
1181
11:17
31
1.23 (848)
0,29
1.36
0.34
(885)
0U87
6138
15.
4.24
13:07
6257 11.02
1.27
(2.00)
11:91
(UR)
0,58
19.37
SUR1
IU84
6.22) (111)
94.
0508
18 00
00200 (1
1.2 (u0)
733781
1103 60
0484 (1:8)
0.40 (1X]
840 (68)
9.88 (1:55
0135
18
0686 19 XC 6390 (u0)
172 m007
1:01 11 10)
(196 (1K]
Table 8: The full evaluation results on PEMS datasets. The optimal models are marked in red, and we underline the models that are not significantly worse than the optimal.
While for the PEMS datasets (Table 10 the optimal baselines concentrates mostly on the iTrrasformer (Liu ct al. 2024b) and TimesNet (Wu et al. 2023). However, DARTS-TS is still significantly better than the baselinn models on the PEMS04 and PEMS07 datasets.
D Comparing with zero-shot forecasting foundation models
Time series forecasting foundation models (Ansari al. 2024 Bian et al. 2024 Das al. 2023 Liu et al., 2024a have become the trend for recent forecasting tasks. Instead of training multiple networks for different tasks separately, forecasting foundation models train one single model across multiple datasets and directly test it on the target sequence without further training the model on that dataset.
Here, we compare DARTS-TS TS with another two forecasting foundation models: TimesFM (Das et al., 2023 (with 200M parameters) and Moirai-MOF, (Liu et al., 20244a (moirai-moe- 1.tmmoo--1------- R-S-Rl with 117M parameters and 11M active parameters). Both approaches consider the input scries independently and patchify cach single variant time series by integrating neighboring data points into one single patch (Nie et al. 2023). Since further training process is required for these two models, and the evaluation costs can be expensive, we only evaluate these two models with one random seed.
The result is shown in Table [1 and Table 12 We run all the foundation models on Nvidia H100 GPUs with 80GB GPU memory for at most 24 hours. Any run that fails to finish the prediction (either by memory out or out of time) will be marked with NaN. DARTS-TS achieves the optimal performance on the ETT datascts and is only slightly worse than TimesFM on the ECL datasets with forecasting horizons of 96 and 192. FM achieves a better performance on the traffic, weather, and ECI. datasets. However, these
24

---

Published in Transactions on Machine Learning Research (09/2025)
DARTS-TS
Baoeline
statlatle
pevalue
DARTS-TS
Baseline
statletlic
p-ville
Name
Valuo
Nome
Value
ECL
96
MSE
0.129
(0.00)
PatchTST
0.130 (0.00)
-2.114
0.067
ETTm2
06
MSE
0,102
(0.00)
Pat
0.165 (0.00)
-3.616
0.007
0.217
(0.00)
PatcTTST
0,223 (0.00)
-3,954
0.004
MAE
0,244
(0.00)
PurchTST
0,254 (0.00)
-14.951
0,000
192
MSE
0.347
(0.00)
PateETST
0.148 (0.00)
-3.985
0.064
192
MSE:
0.223
(0.00)
PatchTST
0.222 (0:00)
0.735
0.484
MAE
0.254
(0.00)
PatchTST
0.241 (0.00)
-7.053
0.000
MAE
0.285
(0.00)
Par
0.203 (0.00)
-12.251
0.000
MISE
0.104
(0.00)
PattIST
0.105 (000)
1,4455
0.186
0:274
(0.00)
FucchTS7
0.27 (0.00)
2:276
0,052
MAE
0.253
(0.00)
PatcTTST
0.259 (0.00)
-2.992
0.017
MAE
0.320
(0.00)
PatchTST
0.329 (0.00)
-32.523
0.000
720
MSE
0. 186
10.00)
PatchTST
0.202 (0.00)
-6.015
0.000
T20
MSE:
0.354
(0.00)
Pat
0.364 (0.00)
-7.163
0.000
MAF:
PSt
(0.00)
-15,057
0.000
MAR
PaccTTST
0.383
-10.913
0.000
ETTTI
58
MSB
0.365
(0.00
Modern TCN
0.380 (0:00)
-1.735
0.121
Exchungge
06
MSE
0,085
(0:00)
DLincor
0.084 (0.00)
2:000
0.076
MAE
0.085
(0.00)
MCN
0,384
(0.00)
-5,077
0.001
MAE
0.210
(0.00)
DLincat
0,203 (0.00)
3.046
0.016
TCN
0.407 (0000)
-1.129
0.251
MSE:
BLison
0.154 (0:01)
6,302
0:000
MAEE
1
(0.00)
MoNenTTCN
0.415 (0.00)
-2.168
0.062
MAB
0.308
(0.00)
Dlincar
0.203 (0.00)
5.440
0.001
336
MSE
0:457
[0.01]
MC
0.382
(0.0)
11.610
0.000
MSE
0.350
(0.01)
PS
0,350 (C.00)
0.010
0,292
MAE
0.433
(0.01)
Mo
0.413
(0.00)
6.234
0.000
MAE
PS
0.421
-0.724
0.490
720
VISR
0.446
(0.01)
PATETTT
(0000)
0.220
0:385
MSSE
0.355
(0.03)
TS
0.071 (0:11)
3.502
0.0005
MAE
0.462
(0.00)
MCNenTTCN
0.461
(0.00)
0.331
0.749
MAE
0.669
(0.01)
TSMaer
0,653 (0.D4)
1.634
0.141
ETTH2
16
MSE
0,276
Modern TCN
0.264
(0:00)
26.684
0.000
Tratie
36
MSE:
0.368
(0.00)
Iiaasdorust
0.356 (0.00)
1.327
0:221
MAF
0,332
(0.00)
Modemn/C TCN
0.333 (0.00)
-2,500
0.034
MAR
0.240
(0.01)
ParchTST
0,250 (0.00)
-3.321
0.011
102
MSE
0:344
(0.00)
Modern ICN
0.322
(0000
18.238
0.000
192
MSE
0.356
(0:01)
0,376 (0.00)
2.652
0.020
MAE
MCN
0.377 (0.00)
1.436
0.189
MAE
0,256
PatchTST
0.258 (0.00)
-1.174
0.274
336
MRB
0.377
(0.0)
Modern NCN
0.315
(0.00)
0.000
0.305
(0.01)
Waasooomer
0.389 (0.00)
3.337
0.010
MAP
0.406
(0.00)
MoNennTCN
0.377 (0.00)
23.442
0.000
MAE
0.262
(0.00)
PatchTST
0,267 (0.00)
-2.100
0.079
720
MES
0.406
(0:01
PALETST
0.375 (0000)
T0.4401
0.000
730
MSF:
0:434
(0100
Tarneooreer
0.32% (0:00)
1.503
0.10Y
0.455
(0.00)
PatcTTST
(0:00)
7,311
0.000
MAE
PatcTTT
0.292 (0.01)
-1488
0.175
ETTml
16
MSE
0.283
(0.00)
PatEETST
0.250 (0:00)
-2.776
0.024
Weather
966
MSE
0.148
(0.00)
ISMaxr
0.136 (0.00)
1.586
0.151
MAR
0.330
(0.00)
PatcTTST
0.341 (0.00)
-6.752
0.000
MAR
0.194
(0.01)
Par
0.180 (0.00)
-0.890
0.404
MSE
0.520
(0.00)
PARETST
0.355 (0.00)
-6.075
0.105
(0.00)
TSMaxr
0.102 (0.00)
1.331
MAE
0.354
(0.00)
DLinsw
0.368 (0.00)
-9.211
0.000
MAE
0.230
(0.01)
PSt
0,242 (0.00)
-0.482
0.048
336
MSE
0.357
(0.00)
P
0.367 (0.00)
-5,200
0.001
396
MSE:
0,262
(0.01
TSMase
0.240 (C.00)
2.741
0.025
0.377
(0.00)
PatekTST
(0:00)
-20,55
0.000
MAE
(0.01)
PS
0.283 (0.00)
0.921
720
MSE
0.418
(0.01)
PaRETST
0.417 (0.00)
0.207
0.797
720
MSE
0,123 (0.00)
TSMaxe
0.311 (0.01)
2.537
0.019
MAE
0.422 (0.00)
MadennTCN
0.421 (0.00)
-9.198
0.000
MAE
0.331 (0.01)
ParchTST
0,335 (0.00)
-0.941
0:374
Table 9: Statistical Test against the non-NAS architectures on the LTSF datasets
DARTS-TS
Readine
statistic
p-oadoe
DARTS-TS
Boseline
statistic
p-osiue
Name
Value
Name
Value
PEMSO
12 MSE
0.066
(0.00)
iTranfornee
0.069 (0.00)
-2:400
0.010
PEMS07
MSE
0,060
(0.00)
iTunsfommer
0.086 (0.00)
-6.855
0.000
MAE
0.171
(0.00)
(Tranfornee
(0.00)
-2.651
0.029
MAE
0.155
(0.00)
Tausdormer
0.161 (0.01)
-4.457
0,0022
24
MIS
0.067
(0.00)
Tronddemes
0.088 (0.00)
-0.679
0.516
24
MSE
0.081
(0.00)
Mausfarmer
0.087 (0.00)
-4.175
0.003
MAE
0.206
(0.00)
iranddenner
0.200 (0.00)
-1.589
0.151
MAE
0.180
(0.00)
Trunsfrmer
0.190 (0.00)
-4.539
0. 0.002
MSW
0.152
[0.01]
TomonNot
-2216
XISF:
(0.01)
TimmNet
0.133 (0.00)
MAE
0.257
(0.01)
TomeNet
0.263 (0.00)
-1.186
0.277
MAE
0.218
(0.01)
TinesNet
0.235 (0.00)
-3.299
0.011
96
MSE
0.234
(0.02)
ThmesNet
0.242 (001)
-0,588
0.373
96
MSE
0.158
(0.12)
TimesNet
0.211 (0L06)
-1.755
1,110
MAF:
TimesNet
(0.00)
1.170
0.276
MAR
0.262
TimosNet
-1.555
0.158
PEMSO
12
MIS
0.033
(0.00
TTranooueer
0181 (0.00)
-15.078
0,000
PEXISOS
XISIE
0.074
70.00)
Taundrmer
0.087 (0.00)
-11.011
0.000
MAE
0.176
(0.00)
Trandfonace
0.188 (0.00)
116,158
0.000
MAE
0.175
(0.00)
Taundormer
0.193 (0.00)
-7.833
ILBK0
XISR
TimesNel
-8,80
24
XISE:
0.107
Taussarmer
0.135
-0.255
0.D0K
MAE
0.198
(0.00)
TomeNet
0.216 (0.00)
-7,884
0.000
MAE
0.213
(0.01)
TimoNNet
0.234 (0. (0.00)
-4. -1.576
0.002
J8
MISE
0.120
(0.00)
Hrandooasee
0.135 (000)
8,544
0.000
48
MISE
0.118
(1.12)
Timeseet
0.191 (0.100)
1.710
1.126
MAE:
0.232
Troadoomee
(000)
-7.388
MAE
0.277
(0.02)
Tinssarmer
0.277 (0.01)
-0.06
11988
MSR
0.165
(0.00)
(ronddrmerr
0.160 (0.00)
-2,519
0.006
96
MSF
0.318
(0.04)
TimesNet
0.322 (0.01)
-0.160
0.877
MAE
0.278 (0.00)
iTrnnfornce
0.260 (0.00)
1.429
0.188
MAE
0.360 (0.04)
TimesNet
0.569 (0.01)
0.643
0.538
Table 10: Statistical Test against the non-NAS architectures on the PEMS datasets
three datasets are also part of the training set from TimesFM, and therefore, the forecasting results on these datasots can no longer be considered as a zero-shot forccasting sctup. While for the other datasot, where TimesPM is not trained, including the four PEMS datasets, DARTS-TS still outperforms TimesFM's zero-shot forecasting results.
We also provide a comparison of the latency cost during inference time between DARTS-TS, TimesFM, and Moirai-MOE. Due to the memory limitation, we only evaluate with a batch size of 1 and set the number of samples for Moirai-MoE to 10 instead of 100 random samples in its original setup. The result is shown in Figure I1 DARTS-TS provides a 67 times speed up on Traffic and a 27 times speed up on ECL compared to TimesFM. This speed-up increases to 322 times on Traffic and 123 times on ECL compared to Moirai-MoE: Showing the efficiency of DARTS-TS compared to the L.MM-based approaches.
25

---

Published in Transactions on Machine Learning Research (09/2025)
DARTS-TS
TimnsFM
Moini-MoB
DARTS-TS
TTimsFM
MairsiMMaR
MSE
MAE
MSE
MAE
MSE
MAE
MIS
MAE
MSE
MAE
MSE
MAE
ECL
96
0.129
0.217
0.125
0.217
0.215
0,280
ETTm2
96
0.162
0.244
0.208
0.273
0.256
0.515
192
0.347
0.230
0:222
0.295
0.222
0.323
0.319
336
0.364
0:203
0.168
0.225
NoN
NoN
336
0.274
0.320
0.361
0.369
0.968
0. 0.382
720
0.186
0.275
0.214
0.295
NoN
NaN
720
0:354
0.373
0,460
0.434
0.964
0.452
0.383
0,055
0,412
0:372
0,415
Exchange
50
0,109
0.201
0.083
192
0,405
0.411
01.00
01430
0.564
0.450
1922
0.186
0.308
0.228
0.340
0.184
0.300
336
0.437
0.433
0.518
0.477
0.612
0.479
336
0:350
0.428
0.412
0.465
0.550
0.424
0.400
0.764
ETTE2
95
0.276
0.522
0.325
0.353
0:345
0.355
Traffic
55
0.355
0.230
0.310
0.225
NcN
NaN
192
0.344
0.277
0.392
0.338
11.13
0.204
192
0.388
10255
0.368
0.241
N&N
NaN
0.423
0.429
0.254
N&N
N&N
720
0.406
0:436
0.462
0.462
0.461
0.659
720
0.434
0.284
0.432
0.279
NcN
NaN
ETTml
95
0.283
0,330
0.363
0.371
1,097
0.805
Weather
565
0.148
0,199
0.136
0.309
0.244
0.250
1.009
0.200
336
0.357
0:377
0.482
0.477
0.940
0.502
336
0.252
0.982
0.236
0.257
0.351
0.530
720
0,418
0.412
0.544
0.487
0,006
0,594
720
0.323
0.351
0.340
0.337
0.418
0.376
Table 11: Results against forccasting foundation models in Long Term Forecasting datasots
DARTS-TS
TameFFM
Moirni-Moo
DARTS-IS
TimmsF MAE
Moltai-Maa:
MSE
MAE
MSE
MAE
MSB
MAA
MSE
MAB
MSE
MSE
MAE
PEMS0O
12
0.066
0.171
0.158
PEMS07
0.250
0.110
24
0.007
0.206
0.325
0.391
0.250
0.330
24
0.081
0.150
0.340
0.397
0.232
0.316
48
0.162
0.267
0.707
0.632
0,580
0.726
48
0.118
0.218
0.821
0.647
0.588
0.521
0.331
0.867
1.009
1.445
PEMSON
12
0.013
0.176
0.176
0.282
0.131
0.211
PEXIOSS
0.04
0.175
0.180
0.270
0.128
0.251
24
0.0011
0.198
0.363
0.416
0.251
0.340
24
0.307
0.213
0.333
0.307
112447
0.331
48
0.
0.507
56
0.166
0.278
1.824
1.055
1.476
0.303
0.318
0.300
1,855
1,050
1.500
0.902
Table 12: Results against forecasting foundation models in PEMS dataset
E
Ablation Study
E.1
The impact of window size
In our experiments, we fixed our window size to 336 and applied this window size to predict different. forecasting horizons. However, since the look-back window size is an important hyperparameter in forecasting tasks, it is interesting to see if the model should always stick to the window where it is trained. To answer this, we ask the optimizer to search for an architecture that requires the input window size (96, 192, 336, 7203. We then evaluate each of these found architectures with the window sizes mentioned above.
We run this task on the ECL-96 dataset. The result is shown in Figure 12
We see that our model will perform better with the increase in search window size in general. While the search window size also influences the final evaluations, a model scarched with a window size of 720 performs the worst when the architecture suggested by this optimizer is asked to make a prediction with a window size of 96 and vice versa. However, this gap becomes smaller if the search window size is closer to the eval window size. This indicates that the window size we used to search for the network should not be too far away from the actual window size used to evaluate the model.
E.2 Forecasting Horizon
In experiments, we only searched with the forecasting horizon of 96 and applied them to the other forecasting horizon tasks. Here, we check how the optimal architecture searched for one target forecasting horizon can be transferred to another forccasting component. Similar to Section E. we ask the optimizer to search for the models with forecasting horizon of size {96, 192, 336, 720). We then evaluate each of these optimal architectures with the other forecasting horizons mentioned above. The sliding window size is set 96 for all tasks this scenario. The result is shown in Figure 13 The search forccasting horizons could also be transferred to the other forecasting horizons during evaluation, as different searching forecasting horizons provide similar performance under the same evaluation forecasting horizon.
26

---

Published in Transactions on Machine Learning Research (09/2025)
Network Latency Information on the Traffic Dataset
Network Latency Information on the ECL Dataset
GPU Memory
3371
200
GPU Memory
199
200
3000
(Inference)
(nference)
GRU Nemory (68))
Time (mallter)
GPU Nemory Usage (MB)
1200
Inference
Time (ma/tter]
Inference Time Conse
2000
Size
2000
800
100
100
1500
600
1000
EP6
200
1.21
DARTS-15
TimesFM
Motra-Mott
DARTS-TS
Timest M4
MairalMoo
Figure 11: Latency information of different networks the Traflic (left) and ECL (right) datasel. We set the window size and the forecast horizon as 96 and the batch size as 1.
0.14107
0.14752
0.14298
0.15794
0.150
192
0.13467
0.13439
0.13467
0.13454
Eval Window
0.140
0.13008
0.13121
0.12921
0.12865
720
0.13025
0.12788
0.12763
0.12810
0.130
96
192
336
720
Figure 12: The impact of window size during searching and evaluations on the ECL-96 task
E.3
Forecasting components
We constructed a hierarchical search space in Section Here, we will show the efficiency of each component. We provide the following variations:
DARTS-TS CV. This variation has the same architecture as DARTS-TS. However, the training validation dataset splits strategy during its scarch phase is 5-fold cross-validation instead of the holdout strategy described in Section
Flat Only. This variation only contains the Flat Net in the search space.
Seq Only. This variation only contains the Seq Net the scarch space.
Parallel. This variation is similar to our approach. However, the Seg Net receives only feature variables instead of the output of the Flat Net
Concat Seq First. This variation first feeds the input data to the Seg Net and then the output of Seg Net is fed to the Flat Net
No Weights. This variation removed the weighted sum approach described in Section 4.3
The result is shown Table 13 Despite that DARTS-TS CV achieves nearly the same performance on some datasets, such as ECL, ETTm2, Exchange, and Weather, there still a performance gap between DARTS-TS Holdout (i.e., DARTS-TS in the table) and DARTS-TS CV on the other datasets. The Parallel
27

---

Published in Transactions on Machine Learning Research (09/2025)
0.20
96
0,14162
0.14202
0.14160
0.14121
Evall Forecasting Horizon
192
0.15902
0.15870
0.15856
0.15841
0.18
0.17
336
0.17110
0.17361
0.17347
0.17246
0.16
720
0,20372
0.20262
0.20009
0.20326
0.15
96
192
336
720
Search Forecasting Horizon
Figure 13: The impact of forecasting horizon during searching and evaluations on the ECL-96 task
HAIISS 135
HARIS-SS
Porild
Dayy
Okty HRE
Cauoor Plst
MSB
MAE
3082
MAB
SIBB
MAB
MBB
MAE
XISE
MAE
MAX
MBS
MAB
SIRE
MAB
Elint
1290 (031
1.50 001
1.20 (022)
132 1072
12302 (23))
10039 10R)
1:30 (ED)
L360 01.31
9346 (1001
1.34 (0231)
11583
11358 (ET0))
1130
E1102
76
1220 10.90
1.000 001
7:24 10.00
1330 1030)
1133 1000
0331 12760
1032 7000
0:901 m161
1207 10-20
1381 1039)
1.288 1039)
01339 1230
12299
7670)
11.35 (2170
1.27 0011
12300
015)
10333 1330
1100 (1:0)
9924 000)
190
(000)
0331
0.00
0700
00000000
0900 (000
1118
0994
(0191
(6000)
16
108
1200000
198 00
198 (109)
0883
1600
0399
1710
0337
1100
033
0779
000 1010
000 m
100 00
0907 1001
3100
1009)
0300
160
0107
1:30 00:0
1200100
1.00 10110
750 1001
02.9
1009
831
022
12
1-0000000
8300 1000
8117
1000
0327
110000000
Table 13: Ablation over the components in DARTS-TS
approach is slightly worse than DARTS-IS many datasets. However, there is a huge gap between Parallel and DARTS-TS on the Traffic dataset. While the Flat Only approach is generally worse than DARTS-TS on the ETT datasets. Overall, we show that the architectural design of DARTS-TS generally provides us with architectures that are robust across many datasets.
E.4 Optimizazion Epochs
Table we optimize the supernet for 40 epochs and then prune the network with the pre-trained architectures. Here, we study the impact of the number of optimization epochs.
The result is shown in Figure 14 Each model is optimized for f0, 20, 40, 60, 80, 100) epochs, respectively. Since we will also update the architecture weights and parameters during the pruming stage, even optimizing the models for zero epochs might also return well-performing model. However, training the model for too long might overfit the validation set (Zela et al. 2020). Overall, the optimal number of epochs might differ across different datasets, whilc optimizing the super net for 40 or 60 epochs might result in a better average performance.
E.5 Operations
As described in Section A our scarch space contains many different operations. To verify the importance of each operation, here we remove each operation from the search space and check its performance on each of the target datasets.
The result is shown in Table 14
Despite that, removing certain operations from the scarch space might provide better results on certain lasks. For instance, removing Separated TCN on ECL datasets results in a model with 0.128 MSE loss instead of the 0.129 MSE loss found by the Full search space. However, removing the same operation on the ETTh1 dataset incrcases the loss from 0.365 to 0.381. This degencration also appears for other operations, for instance, removing MLP increases the loss on the Traffic dataset from 0.358 to 0.379, while removing E.BEATS-Trent could further increase this value to 0.402. Hence, the search space
28

---

Published in Transactions on Machine Learning Research (09/2023)
Performance Over Time
ECL
ETTh1
ETTh2
ETTm1
0.134
0.31
0.288
0.286
0.1
0.30
0.40
0.204
0.130
0.282
0.38
0.280
Test MSE Loss
0.28
0.278
40
80
100
40
80
100
40
100
0
60
100
ETTm2
Traffic
Exchange Rate
Weather
0.39
0.168
0.160
0.166
0.38
0.155
0.164
0.162
1,090
0.150
0.36
0.160
40
80
100
40
80
100
0.5
40
60
100
60
100
Optimization Epochs
Figure 14: Performance Over Number of Supernet Optimization Epochs
Oge
N-BFATS
K-BEATS
GRC
Sep. MSE
MSB
MAB
MSE
MAE
MISE
31AE
3ISE
MAE
MSE
MAE
MSE
31AE
MLAE
3ESE
MAE
MAE
MSE
3AE
MSE
MAE
ECL.
0.120
0.217
0.151
0.225
0.139
0.222
0.135
0.227
0.196
0.233
0.132
6.224
0.131
0.221
0.129
0.220
0.125
0.216
0.151
0.220
0.129
0.219
Th
0.365
0.385
(.214)
0.408
0.355
0.401
0.375
0.308
0.375
0.314
0.172
0.313
0.263
0.309
0.571
0.391
0-381
0.358
0.38
0.387
0.384
0.403
ET T12
I1276
0.332
0.280
0.341
0.256
0.539
0.286
0.235
0.291
0.847
3,292
0.345
0.292
0.345
0.286
0.340
0.285
0.346
0.257
0.340
0.257
0.341
RT Tm
0:983
6:330
0.261
0:335
6.278
0.327
0.27%
0:327
0.281
0.33)
0.280
0:578
0.365
0.352
0.285
0.333
0.381
0.35
0.29
0.330
0.983
0.399
BTTu2
Exchange
0.210
0.211
0.214
Westher
0.188
0.191
0.198
0.307
0.154
0.198
0.132
0.195
0.149
0.102
0.301
0.182
0.152
Table 14: Ablation over the Operations in our search space
could benefit from all the operations described in Section A depending the datasets to which the model is applied. This further highlights the importance of exploring various architectures for different target tasks.
E.6 ReviN
In Section applied RevIN (Kim et al. 2022 in DARTS-TS during both the scarching and testing phases; here, present ablation study on the RevIN for DARTS-TS.
ECL
ETTM
ETT12
ETTml
BTTh2
Trefic
Exchange Rate
Weather
w/ RevIN
MSE
0.129(0.00L)
0.365(0.004)
0.276(0.001)
0.283/0.005)
0.162/0.001)
0.355(0.003)
0.088(0.002)
0.1d8(0.003)
0.217(0.008)
0.385(0.003)
0.332(0.000)
0.21400.001 0.330)0.008)
0.210/0.006)
0.216(0.002)
0.184/0.011)
w/o RevIN
MSE
0.130(0.004)
0.000(0.022)
0.287
(228/0.002)
0.175(0.020)
0.375(0.009)
0.093(0.004)
0.147:0.008)
0.225(0.009)
0.121(0.018)
(..00)
0.331(0.002)
0.267(0.030)
0.216(0.008)
0.218(0.003)
0.183/0.007)
Table 15: Ablation for the RevIN in our search space
Table 15 shows the impact of RevIN. Except for the Weather dataset, where the mean MSE loss increases from 0.147 to 0.148 by applying RevIN, the performance of all the other datasets can be improved with RevIN, especially the ETTs and traffic datasets. Overall, RevIN enables more accurate forecasting models within our search space.
E.7
Separated Incoder-Decoder Search Space
In Section 4.2 we design a search space that allows the encoder-decode pairs from the same edge to have different architecture types. This design decision enlarges the search space. Here, demonstrate that the
29

---

Published in Transactions on Machine Learning Research (09/2023)
ECL 96
Traffic 96
0.18
Text NSE Loss
0.53
0.16
W55
0.50
Mixed
Identity
Mixed
Identity
Figure 15: Performance of the architectures with mixed and identical encoder-decoder architectures
enlarged search space encompasses architectures that may outperform those with the same encoder-decoder architecture.
Hence, we check the random configurations from Section 6.2 and select all the configurations that contain Seq decoders. Then, for each decoder edge, instead of randomly sampling a new decoder component, we initialize the decoders with operations that are the same as those of the corresponding encoder nodes. We then reevaluate those architectures with the identical operations and compare them with the corresponding mixed architectures.
The result is shown in Figure 15 On both datasets, architectures with mixed operations achieve lower minimum test loss (0.365 es 0.369 on the Traffic dataset and 0.125 us 0.126 on the ECL dataset). This shows that applying different operations to the encoders and decoders provides a potentially better model compared to the architectures that apply the same encoder and decoder operation within the search space.
F Optimal Architectures on other datasets
(in)
(cut)
mlp
icm
skip_cunnae connectinnm
nbcuts
nbeats_s
mly
(out)
2
in) 1
(stm
(in)
nbetts
Figure 16: An optimal architecture the ECL datasct. This is encoder-only architecture.
(in)
skip connectinnn
obeats
cip_connection
ten
docats
abeats
mlp
(out) 4
mlp
(in)
tcn
(out) 5
(in)
nicats.
nbeais_1
Figure 17: An optimal architecture on the Traffic dataset. This is encoder-only architecture.
mip
transtiomer
skip connection
tranffrmne'
mlp mix
gru
trunsformer
skip
tandforne
(ouz)
Gn)
skip connection
(ous)
mlp mlp
abeats.s
mlp
(ini
mlp
skip connection
(out)
Figure 18: An optimal architecture the ETTml dataset. This is an encoder-decoder architecture.
30

---

Published in Transactions on Machine Learning Research (09/2025)
210
(in)
A20
mlp
gru
sep, Jen
mip_mix
mlp
mlp
nheats
mlp
(eal)
IransCormer
(out)
(in)
mlp
Figure 19: An optimal architocture on the ETTm2 dataset. This an encoder-only architecture.
Hdp
cundrrmer
Insademeer
somasoniiionn
mlp
skip stp-stentttoon
EAEE
abeats
nbeats_
mlp
(oul) 4
(in)
mbeats
Figure 20: An optimal architecture on the Weather dataset. This is an encodcr-only architecture.
we showed one of the optimal architectures in the electricity dataset. In this section, we will provide more searched architectures.
In Section 6.4
Figure 16 shows yet another optimal architecture the ECL dataset. We that TCN and N- BEATS-Seasonal modules are still contained in the optimal modules, further confirming our conclusions in Scction e
We also show some of the optimal architectures in Figure 17 operation dominates the optimal architecture, which shows the necessity of performing an architecture search for the optimal architecture.
18
and 20
We can see that no single
31


Reviews:
[{"review_id": "dnaNS7WOmz", "reviewer": "Reviewer_SEc1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary\n\nThe authors introduce DARTS-TS, a hierarchical neural-architecture-search (NAS) framework that automatically assembles lightweight yet accurate deep-learning models for multivariate time-series forecasting. By nesting three levels of design choices—operations, micro-networks and macro topologies—then pruning them with a differentiable NAS algorithm (DARTS-PT), the method discovers hybrid CNN/RNN/Transformer/MLP combinations that outperform or match hand-crafted baselines while using far less memory and compute.\n\n### Strengths\n\n1. **Search efficiency** – By adopting the differentiable DARTS-PT algorithm with weight-sharing and perturbation-based pruning, the authors find high-performing architectures in just a single super-net run, cutting GPU time and memory compared with both classic RL/EA NAS and large “train-from-scratch” Transformers.\n2. **Effective sparsification** – The perturbation step quantitatively measures each candidate operation’s contribution and prunes redundant edges, avoiding the skip-connection collapse that plagues vanilla DARTS and yielding compact models with 2 – 6 × lower inference cost.\n3. **Well-rationalized three-level hierarchy** – The operation → cell → macro design cleanly separates *what* building blocks to use, *how* to assemble them inside Flat/Seq cells, and *how* to fuse the two branches. This heterogeneous space lets the search mix CNN, RNN, Transformer and MLP biases and empirically produces state-of-the-art results on 11 forecasting benchmarks.\n\n### Weaknesses\n\n1. **No mechanism for distribution shift** – The search ignores test-time adaptation techniques (e.g., RevIN, TENT) and therefore may still over-fit the validation slice when real-world time-series drift.\n2. **Limited validation protocol** – Experiments rely on a single hold-out split; they omit rolling-origin or expanding-window cross-validation, making it hard to gauge robustness across forecast horizons and start dates.\n3. **Fixed macro topology** – Only the Flat → Seq + weighted-sum wiring is allowed. Alternative macro layouts (Seq-first, purely parallel branches, or learned ordering) are not in the search space, so the conclusions about hierarchy optimality remain provisional.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper supplies (i) clear architectural diagrams, (ii) detailed algorithmic pseudo-code, and (iii) comprehensive benchmarks on eight long-term forecasting datasets plus four PEMS traffic sets. Metrics (MSE/MAE) are reported over five independent runs with mean ± s.d., and ablations isolate the impact of each design choice (hierarchy levels, perturbation pruning, branch removal). Runtime/VRAM statistics are collected on identical hardware. While the experimental protocol could be stronger (see weaknesses), the evidence presented adequately supports the headline claims of accuracy and efficiency.", "audience_yn": "Yes", "audience_explanation": "Time-series forecasting is a prominent subfield in machine learning, and NAS for resource-efficient model discovery remains highly relevant to both academia and industry. TMLR readers who work on AutoML, architecture design, or practical forecasting systems would benefit from the paper’s hierarchical search-space design and its empirical insights on hybrid CNN/RNN/Transformer architectures.", "requested_changes": "1. Add **rolling-origin or expanding-window cross-validation** to quantify robustness over multiple forecast start dates.\n2. Incorporate (or at least discuss) **test-time adaptation mechanisms** such as RevIN, TENT, or simple batch-norm recalibration to evaluate resilience to distribution shift.\n3. Extend the macro search space to include **Seq-first and parallel-only variants**, or justify why Flat-first is universally preferable.", "broader_impact_concerns": "I do not see immediate ethical red flags.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YLEJz0FJ8f", "reviewer": "Reviewer_cyWs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces DARTS-TS, a hierarchical neural-architecture-search (NAS) framework tailored to time-series forecasting.\nInstead of committing to a single model family, the authors build a three-level search space (operation -> micro-network -> macro-architecture). Seq Net cells (e.g., Transformer, TCN, LSTM) and Flat Net cells (e.g., N-BEATS, Linear) are first optimised independently and then stitched together through a learned weighting scheme. The differentiable search relies on DARTS-PT with hierarchical pruning . Experiments on eight long-term-forecasting benchmarks and four PEMS traffic sets  show that the searched models often outperform hand-crafted architectures such as PatchTST and iTransformer while using less GPU memory.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors benchmark DARTS-TS on eight long-term-forecasting datasets and four PEMS traffic sets; the method attains the best or second-best MSE/MAE on most tasks", "audience_yn": "Yes", "audience_explanation": "Time-series forecasting is a core application area, and Neural-Architecture Search remains an interesting research frontier.", "requested_changes": "1. Add wall-clock search time on a single A100 and the average GPU-hours required per dataset, so practitioners can budget realistically.\n2. Section B mentions “ADAM instead of SGD” but not learning-rate schedule or weight-decay; please include a hyper-parameter table in the main text.\n3. Specify whether baselines were re-trained with identical data splits and th input window\n4. Provide statistical test / significance results against the best non-NAS baseline for each dataset\n5. In Section 5.1 you “prune from lowest granularity”; include a comparison of FLOPs before/after pruning\n6. Clarify whether the validation set used for architecture gradients overlaps with the official test split\n7. Right now the architecture weights are optimised solely to minimise validation MSE/MAE (Eq. 2, Section 5) but the narrative of the paper stresses latency and memory. How are they optimized?\n8.  The authors search on the shortest horizon of each dataset and reuse the architecture for longer horizons. Has any analysis on the more general setting on length generalization? (more datasets than in Appendix D, more horizons, etc.)", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9HimXsWEHN", "reviewer": "Reviewer_jYEq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of finding optimal deep learning architectures for time series forecasting, where no single handcrafted architecture dominated others. The authors introduce DARTS-TS, a hierarchical Neural Architecture Search (NAS) framework. Unlike previous NAS methods, DARTS-TS allows combining diverse model types, such as Flat Net (Linear, NBEATs, etc.) and Seq Net (MLP, GRU and Transformer, etc.). Using an efficient search technique known as DARTS-PT, the framework discovers highly effective and computationally efficient forecasting models. Extensive experiments demonstrate that architectures identified by DARTS-TS either outperform or match state-of-the-art methods on popular long-term forecasting datasets. The paper also highlights unique challenges posed by the heterogeneous search space and analyzes the limitations of zero-cost performance proxies in mixed-model search scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Pros\n\n- Novel Search Space Design: Introduces a hierarchical heterogeneous search space that flexibly integrates Flat Net and Seq Net families. This gives a more flexible search space than homogenous NAS benchmarks.\n\n- Strong Empirical Results: Achieves SOTA or competitive accuracy on multiple benchmarks, while reducing memory usage and improving speed (up to 6× less GPU memory than PatchTST).\n\n- Insightful Analysis of NAS Challenges: Demonstrate empirically that classical zero-cost proxies (FLOPs, #params, etc.) fail in mixed search spaces.", "audience_yn": "Yes", "audience_explanation": "Refer to the comments above.", "requested_changes": "Cons\n\n- Limited Comparison to Other NAS Methods and Foundation Models: The paper’s baselines are all hand-crafted forecasting models; it lacks head-to-head comparisons with other AutoML/NAS frameworks (e.g., Auto-PyTorch TS, SNAS4MTF), leaving DARTS-TS’s relative search efficiency and accuracy unquantified. Moreover, the time-series community is rapidly adopting large pre-trained foundation models for zero-shot or few-shot forecasting—please evaluate DARTS-TS in both accuracy and inference cost against representative examples such as Moirai-MoE (Liu et al., 2024), the decoder-only FM (Das et al., 2023), and aLLM4TS (Bian et al., 2024).\n\n- Lack of Robustness Under Distribution Shift: The paper identifies non-stationarity as a key challenge for time-series forecasting but presents no experiments or methods (e.g., rolling-origin evaluation, meta-learning fine-tuning) to quantify or address it. Please clarify or extend the work by: Performing a rolling-origin evaluation to measure how DARTS-TS architectures perform under realistic distribution shifts; Applying simple strategies—such as MAML-style fine-tuning—to improve out-of-distribution performance.\n\n- High Search Cost & Scalability Concerns: Although inference is efficient (on an NVIDIA 2080 Ti), the search still requires large-scale resources (e.g., an A100 GPU), and the paper provides no analysis of search time versus performance trade-offs.\n\n- Operator-Family Ablation: While high-level ablations exist (Parallel, Seq Only, Flat Only, and No Weights), the paper should remove each entire operator family (Transformers, TCNs, RNNs, N-BEATS) separately to quantify each family’s contribution.", "broader_impact_concerns": "No any concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9zgHl9vlEO", "reviewer": "Reviewer_pcKL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes DARTS-TS, a hierarchical NAS approach for time series forecasting that addresses the challenge of finding optimal architectures in a field where no single model family consistently dominates. The authors design a comprehensive search space that incorporates multiple forecasting architecture types including transformers, CNNs, RNNs, and MLP-based models, organized in a three-level hierarchy: operation level, micro network level, and macro architecture level. Using DARTS-PT as the search strategy, the method can automatically discover lightweight, high-performing architectures that combine different forecasting modules. Experiments on several forecasting datasets demonstrate that DARTS-TS achieves comparable or better performance than state-of-the-art hand-crafted models while requiring significantly less computational resources.\n\nAdvantages:\n1. The hierarchical design intelligently incorporates diverse forecasting architectures (Seq Net and Flat Net families) in a unified framework, addressing the unique challenge that no single architecture dominates time series forecasting.\n2. DARTS-TS achieves competitive or superior performance compared to state-of-the-art models across multiple datasets while being significantly more efficient (2-6x less GPU memory, 2-3x faster training).\n3. The paper provides extensive analysis including search space properties, zero-cost proxy limitations, efficiency comparisons, and ablation studies that demonstrate the necessity of each component.\n\nMy concerns:\n1. The core contribution is primarily an application of existing NAS techniques (DARTS-PT) to time series forecasting rather than fundamental algorithmic innovations. Could the authors clarify what is the technical contribution in terms of NAS technique?\n2. The hierarchical design, while comprehensive, introduces significant complexity that may limit reproducibility and practical adoption. The interdependencies between levels could make the search process unstable.\n3. In terms of evaluation, results show only marginal improvements over existing methods in many cases. Besides, there is limited evaluation on very long sequence forecasting tasks. No comparison with more recent NAS methods beyond DARTS variants.\n4. The comprehensive search space may not scale well to larger datasets or longer sequences, and the computational cost of the search phase itself is not thoroughly analyzed.\n5. For Figure 9, this architecture shows much more diversity in operations (mlp_mix, gru, skip_connection, lstm, transformer) compared to the ECL architecture. What underlying differences between ECL and ETTm2 datasets might explain this architectural diversity?", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims about performance and efficiency are adequately supported with clear experimental evidence.", "audience_yn": "Yes", "audience_explanation": "Yes, researchers working on time series forecasting, neural architecture search, and automated machine learning would be interested in this work.", "requested_changes": "Please refer to the Summary Of Contributions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)