Bi-Level One-Shot Architecture Search for Probabilistic Time Series Forecasting

Authors: jonas  seng
fabian  kalter
zhongjie  yu
fabrizio  ventola
kristian  kersting
Venue: automl_cc_AutoML_2024_Conference
Published: 2024-02-25 20:49:13

Abstract: Time series forecasting is ubiquitous in many disciplines. A recent hybrid architecture named predictive Whittle networks (PWNs) tackles this task by employing two distinct modules, a tractable probabilistic model and a neural forecaster, with the former guiding the latter by providing likelihoods about predictions during training. Although PWNs achieve state-of-the-art accuracy, finding the optimal type of probabilistic model and neural forecaster (macro-architecture search) and the architecture of each module (micro-architecture search) of such hybrid models remains difficult and time-consuming. Current one-shot neural architecture search (NAS) methods approach this challenge by focusing on either the micro or the macro aspect, overlooking mutual impact, and could attain the overall optimization only sequentially. To overcome these limitations, we introduce a bi-level one-shot NAS method that optimizes such hybrid architectures simultaneously, leveraging the relationships between the micro and the macro architectural levels. We empirically demonstrate that the hybrid architectures found by our method outperform human-designed and overparameterized ones on various challenging datasets. Furthermore, we unveil insights about underlying connections between architectural choices and temporal features.

Main paper: 
 introduction: 
 Time series forecasting is a predictive task of primary importance that finds applications in several disciplines such as energy management, finance, and healthcare. Recently, next to well-known statistical models like ARIMA, deep neural methods gained popularity due to their representational and predictive power. While achieving good performance across many tasks, most deep neural approaches only provide point estimates without giving insights regarding the uncertainty of their predictions. However, uncertainty about predictions is crucial in many critical applications since it provides additional information that can tell us if the predictions might be erratic. For example, a model predicting that a patient will benefit from a certain treatment should not be trusted if the prediction comes with high uncertainty. Hybrid architectures are a prominent way to obtain high predictive accuracy with useful uncertainty estimations. These architectures generally consist of a neural forecaster, e.g., an RNN, and a probabilistic model, e.g., isotropic Gaussian distributions (Salinas et al., 2020) , normalizing flows (Rasul et al., 2021b) , or probabilistic circuits (Yu et al., 2022) . In the context of time series, most of these methods operate in the time domain (Salinas et al., 2020; Rasul et al., 2021a,b) while recent works have demonstrated how to leverage the Whittle approximation (Whittle, 1953) to operate in the spectral domain achieving better scaling and taming complexity (Yu et al., 2021a) . A recent hybrid architecture named predictive Whittle networks (PWNs) (Yu et al., 2022) exploits predictive uncertainty to further improve forecasting accuracy, outperforming state-of-the-art neural architectures. Thus, PWNs turn out to be a promising venue for probabilistic time series forecasting. However, to date, PWNs have been employed only with standard neural network architectures, thus, it is unclear whether these hand-crafted architectures are optimally designed for time series forecasting tasks.Designing a neural architecture from scratch requires often expertise and computational resources. Thus, neural architecture search (NAS) targets the challenge of automatically discovering valuable neural architectures with the goal of substantially reducing the required resources. In the last years, NAS has made significant progress that led to the proliferation of many approaches based, e.g., on genetic algorithms, reinforcement learning, and gradient descent. Despite most NAS algorithms being generally evaluated on well-established tasks such as image classification, recent works have changed the trend and devised NAS approaches tailored to the challenging task of time series prediction (Rakhshani et al., 2020; Chen et al., 2021; Deng et al., 2022) . However, existing NAS approaches for time series forecasting only optimize certain parts of an architecture (often referred to as micro-and macro-architecture), therefore, they fail to optimize the entire architecture which is crucial to model performance (Santra et al., 2021) .To tackle this shortcoming, we propose a novel differentiable one-shot method that optimizes both the micro-and macro-architectures of PWNs. Our method frames optimization of the microand macro-architecture as a bi-level optimization problem and updates the hybrid architecture using gradients from the loss w.r.t. the micro-and macro-architecture in an alternating fashion. Furthermore, we provide an extensive empirical evaluation demonstrating that our bi-level architecture search algorithm significantly improves the prediction accuracy of PWNs by optimizing both its micro-and macro-architecture. 1 To summarize, we make the following contributions: 1. To optimize the entire architecture, we propose a novel bi-level neural architecture search algorithm that optimizes both the macro-and micro-architectures of a hybrid model.2. We empirically show that optimizing the hybrid PWN architecture using the proposed bi-level architecture search algorithm outperforms hand-crafted baseline architectures to show the effectiveness of our method.3. To rule out confounding factors, we empirically demonstrate that the performance improvement indeed stems from the optimization, and not from the overparameterization of our models during the search for an optimal architecture.The rest of this paper is structured as follows. First, we touch upon related work before we formally describe our method. Then, we provide an extensive empirical evaluation before concluding our work. 
related work: 
 Our work combines two lines of research, namely probabilistic time series modeling and neural architecture search (NAS). We briefly review both in the following.Probabilistic Time Series Modeling. Conventional time series forecasting models such as long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) , gated recurrent unit (GRU) (Cho et al., 2014) , N-BEATS (Oreshkin et al., 2019) , spectral RNN (Wolter et al., 2020) , and Informer (Zhou et al., 2021) have shown their great power in computing the prediction 𝑦 given the context 𝑥 of a time series. However, these neural forecasters do not naturally provide uncertainty estimates to the predictions, and as a result, users will have difficulty telling how much to trust their predictions.More recently, based on the development of probabilistic time series models (eg. Gaussian processes (Rasmussen and Williams, 2006; Bruinsma et al., 2020) , time series graphical models (Tank et al., 2015) , probabilistic circuit based models (Kalra et al., 2018; Trapp et al., 2020; Yu et al., 2021b) ), several probabilistic time series forecasting models have been proposed. Neural auto-regressive models and normalizing flows have been shown to improve predictions (Salinas et al., 2020; Rasul et al., 2021a,b) , and modeling the time series in the spectral domain with tractable probabilistic models further improves the forecasting accuracy and provides useful uncertainty estimates (Yu et al., 2022) . Although the above-mentioned time series models have shown their predicting power on benchmark datasets, users still need to carefully tune the hyperparameters of the models when given new data sets or new tasks.Neural Architecture Search. As the success of deep learning techniques emerged, the problem of finding optimal architectures for the task at hand became important to solve. Pioneering approaches include the utilization of reinforcement learning (RL) (Pham et al., 2018) , evolutionary algorithms (EA) (Real et al., 2017) and the introduction of differentiable neural architectures that allows optimization of architectures using Stochastic Gradient Descent (SGD) (Liu et al., 2019) . While RL and EA-based methods find state-of-the-art architectures, they come with high resource consumption, hence often rendering them infeasible, especially for large problems. In contrast, gradient-based approaches are significantly more efficient. Their efficacy relies on approximations of the first-order solution of the bi-level optimization problem these approaches have to solve. Since the major task in evaluating NAS methods on is image classification, recently several works considered NAS specifically in the context of time series data. Deng et al. (2022) designed an AutoML pipeline that optimizes hyperparameters and neural architectures to obtain high-performance time series forecasting models. Chen et al. (2021) and Rakhshani et al. (2020) add time series specific operations in the search space or as pre-processing steps to improve the performance of existing NAS algorithms. The works mentioned above were designed to optimize only one aspect of neural architectures, i.e. the micro-or macro-architecture. Recently, Sun et al. (2022) proposed AGNAS to tackle the joint optimization of micro-and macro-architecture exploiting attention mechanisms to weight operations/blocks of an architecture based on the computed feature representations. This renders AGNAS inapplicable in hybrid architectures that contain components that do not compute feature representations directly (e.g., probabilistic circuits). Thus, we propose a method capable of optimizing both the micro-and macro-architecture jointly, solely requiring a differentiable loss w.r.t. the architecture components rather than relying on feature representations. 
bi-level differentiable architecture search: 
 We aim to optimize both, the macro-and micro-architecture of hybrid (neural) models at the same time using a bi-level optimization approach, leading us to a bi-leve architecture search. Therefore, we first briefly describe the hybrid model employed in our paper -predictive Whittle networks (PWNs) -and proceed with our bi-level architecture search algorithm.We focus on the task of time series forecasting with hybrid models such as PWNs, where we are given the context 𝑥 and need to predict the future 𝑦 𝑃𝑟𝑒𝑑 without knowing the ground truth 𝑦 𝐺𝑇 . PWNs are a recent hybrid deep learning architectures that model time series in the spectral domain, and aim to both improve prediction accuracy and provide prediction uncertainty estimates. This is done by simultaneously training a neural forecasting network in combination with a Whittle probabilistic circuit (Whittle PC) . The Whittle PC allows the tractable computation of inference queries, in our case the Whittle PC computes the log-likelihood of the predictions from the neural forecaster given the forecaster's context. This information is then used as feedback to the neural forecaster, to further improve the prediction accuracy. This is achieved by using the Whittle forecasting loss (L WF ), which can be described as a prediction error (L Pred , e.g. mean squared error) (re-)weighted by the Whittle likelihood (L WLL )L WF = L Pred (𝑦 𝑃𝑟𝑒𝑑 , 𝑦 𝐺𝑇 ) • L WLL (𝑥, 𝑦 𝑃𝑟𝑒𝑑 , 𝑦 𝐺𝑇 ).(1) While the prediction error L Pred , measures the deviation between the prediction and the ground truth, the Whittle likelihood indicates how likely the prediction fits its distribution. The Whittle likelihood models time series in the spectral domain. More specifically, the Fourier coefficients of a time series are assumed to follow a multivariate Gaussian distribution, and modeled with a Whittle PC for computing the Whittle likelihood. We refer to Yu et al. (2021a Yu et al. ( , 2022) ) for a detailed discussion. Furthermore, in order to accelerate the training process and extract the valuable likelihood feedback efficiently, a warm-up phase is utilized, resulting in the following loss functionEQUATIONwhere the parameter 𝛽 is linearly increased during training. We refer to Yu et al. (2022) for more details of the PWN model. The architecture of the PWN in Yu et al. ( 2022) is hand-crafted. In the following we show how to automatically search for 1) the combination of the neural components (the macro-architecture) and 2) the architecture of each given neural component itself (the micro-architecture).Inspired by Liu et al. (2019) we view both, the micro-and macro-architecture as directed acyclic graphs (DAGs) G macro and G micro respectively, each containing any number of nodes in addition to an input and an output node. In both DAGs, each node is connected to all of its predecessors. The edges represent possible operations from a set of operations O macro and O micro respectively. In the case of the micro-architecture, operations usually represent small building blocks such as convolutions while in the case of the macro-architecture, operations represent cells from the micro-level. The general overview of our algorithm is visualized in Figure 1 . In the case of PWN, the macro-architecture includes the PWN building blocks, i.e. different neural forecasters and different choices of PCs.Macro-Architecture Search. The set of possible macro-architectures of PWNs is represented as a DAG G macro . The set of operations O macro consists of modules parameterized as neural networks.The input of a cell is defined as the weighted sum of the outputs of its predecessors, with an macro-architecture weight 𝜔 𝑚 for an edge with moduleEQUATIONThe architecture weights indicate which module is expected to best optimize the loss w.r.t. the macro-architecture. We relax the categorical search to a continuous space by computing the softmax over all possible modules. The objective of PWN in eq. ( 2) then becomesEQUATIONwhere m𝑃𝑟𝑒𝑑 (𝑥) represents the weighted predictions from the neural forecaster building blocks, 𝑥 is the original input (context) in the time domain, and 𝑦 𝐺𝑇 is the ground truth prediction in the time domain. We define the objective of the macro-architecture search for modules parameterized by 𝜔 as minEQUATIONwhere 𝑤 * (𝜔) = arg min 𝑤 L 𝑡𝑟𝑎𝑖𝑛 (𝑤, 𝜔). This two-step optimization loop is computationally expensive and inefficient as one single step for the macro-architecture weights requires a full optimization of the entire network. To tackle this, we follow Liu et al. (2019) and approximate the optimization task with one-step gradient descent.EQUATIONIn this way, we are able to efficiently obtain the macro-architecture weights that indicate which modules to use for the PWN model.Micro-Architecture Search. After discussing the macro-architecture search, we now complement our method by the optimization of the micro-architecture, leading to our bi-level architecture search. Similar to the macro-architecture we represent the set of possible micro-architectures a DAG G micro . Each module 𝑚 ∈ O macro has its own associated DAG G 𝑚 micro , inducing a hierarchy of graphs. Let us denote the architecture weights of the neural spectral forecaster and Whittle PC as 𝜔 𝑁 and 𝜔 𝑃 respectively. For better understanding and readability the level of a corresponding weight is indicated by 1 for the macro-level and 2 for the micro-level.We can now rewrite Eq. 4 in terms of the micro-architectures of all modules since these fully define the architecture of each moduleEQUATIONThe spectral forecaster weights affect both parts of the loss as the time series prediction is part of calculating the Whittle likelihood, while the Whittle PC only influences said likelihood. We can now also extend the formulation of our optimization objective to the bi-level problem and optimize both levels in union minEQUATIONAs in the macro search space, each node in G micro is defined as a weighted sum over operations:EQUATIONWith that, we can calculate the final output of G micro by feeding the weighted operations m(𝑥) as input into all following nodes of G micro . The final output is then calculated as the mean of all outputs m𝑚𝑖𝑐𝑟𝑜 (𝑥) = 1 𝑛 𝑛 𝑖=1 m𝑖 (𝑥), and can be used to replace a corresponding 𝑚(𝑥) in equation 3. This allows us to include the set of micro-architecture weights in our optimization of the macroarchitecture weights to enable our algorithm to update both levels in union.We now can state our optimization strategy to optimize the architecture on both, the macro-and micro-level as stated in Eq. 9. As all aspects of the architecture are relaxed to a continuous space, gradients can be computed w.r.t. the macro-and micro-architecture. The model-and architecture parameters are updated in an alternating fashion. Model parameters are updated using standard SGD with the architecture parameters kept fixed. To update the architecture, we extend Eq. 6:∇ 𝜔 1 ,𝜔 2 L 𝑣𝑎𝑙 (𝑤 * (𝜔 1 , 𝜔 2 ), 𝜔 1 , 𝜔 2 ) ≈ ∇ 𝜔 1 ,𝜔 2 L 𝑣𝑎𝑙 (𝑤 -𝜉∇ 𝑤 L 𝑡𝑟𝑎𝑖𝑛 (𝑤, 𝜔 1 , 𝜔 2 ), 𝜔 1 , 𝜔 2 ) (11)The architecture parameters are updated with SGD on validation data to foster the discovery of well-generalizing architectures. 
experiments: 
 We now examine how the differentiable bi-level architecture search performs on various time series forecasting datasets. The following research questions are tackled:(Q1) How do PWN architectures found by our algorithm perform compared to manually designed architectures?(Q2) How reliably does our algorithm choose the optimal PWN macro-architecture across datasets?(Q3) How do different datasets influence the optimized micro-architectures?Datasets. To evaluate the effectiveness of the proposed algorithm, we use 5 real-world time series data sets for empirical experiments. The well-known M4 competition dataset (Makridakis et al., 2020) consists of multiple time series from different domains such as finance and demographics. Moreover, the time series in M4 cover a large range of sampling frequencies, from hourly to yearly, making it a challenging dataset for time series forecasting. The Power dataset contains power consumption data from different EU power grids sampled with a frequency of 15 minutes, from 2011 to 2019 (Wolter et al., 2020) . The Exchange dataset is a collection of daily exchange rates from 8 countries from 1990 to 2016 (Lai et al., 2018) . The Wiki dataset reports the daily hits of 2000 Wikipedia pages (Gasthaus et al., 2019; Rasul et al., 2021b) . The Solar dataset, depicts the solar energy production of 137 plants in the state of Alabama in 2006 (Lai et al., 2018) . A comprehensive overview over different properties of the data sets can be found in Appendix A.Search Space. The search space in NAS tasks is crucial and thus needs to be carefully designed.In our experiments, the macro-architecture search space consists of the PWN building blocks: SRNN, STrans, CWSPN, and WEin. The micro-architecture search space for the SRNN component consists of 8 cells and 5 operations, where each cell is a GRU cell implemented in Yu et al. (2022) .The operations include a linear transformation followed by sigmoid, tanh or ReLU activation functions, as well as none and identity connections. The CWSPN search space, on the other hand, is designed with 4 nodes, as this network tends to be smaller than the SRNN. The 6 operations contain convolutions and dilated convolutions with kernel sizes 3 and 5, as well as identity and none connections. We now are ready to quantitatively compare the time series forecasting results with the baselines.For our evaluation, we first applied our bi-level architecture search to obtain an optimized PWN architecture denoted by PWN ★ . Then, the found architecture was trained from scratch and evaluated on the test set against the baseline models. As baseline models, we trained all possible combinations of the macro-search space and fixed the micro-architecture to the same architecture used in (Yu et al., 2022) . Additionally, Exponential Smoothing (ETS) (Gardner, 2006) , Deep AR (Salinas et al., 2020) and Temporal Fusion Transformer (TFT) (Lim et al., 2020 ) served as baselines. The time series forecasting results from PWN ★ together with our baselines -the 4 handcrafted PWN structures -are depicted in table 2. On each dataset, we repeated the experiment 10 times and report the average and the standard deviation. We find that the architecture PWN ★ found by our bi-level architecture search outperforms the baselines that are guided by uncertainty during training on M4, Power, Wiki and Solar datasets. Also, PWN ★ is very competitive on the Exchange dataset as runner-up. Further, we observe a slight increase in standard deviation for PWN ★ , likely originating from random effects during architecture search. Our results indicate that a bi-level architecture search algorithm can indeed improve the performance of hybrid models, by finding the best modules as well as the suitable structures of each module.In order to evaluate the influence of the increased number of parameters in PWN ★ during the search, we conduct another experiment by employing a larger SRNN s.t. the number of parameters is close to the model used during the search. The larger model has an SRNN with 12 GRU cells, denoted as SRNN + & CWSPN, shown in the second last row in table 2. The results confirm that, generally, simply enlarging the model size does not yield an improvement in performance, as all the prediction results from SRNN + & CWSPN are worse than our bi-level searched PWN ★ . Therefore, we can conclude that the improvement of PWN ★ is indeed from architecture search, rather than by only increasing the model size.To conclude, our results show that we outperform the handcrafted models with our bi-level optimization algorithm on all but one data set. Additionally, we have shown that this increase is not solely based on larger models but the result of a more optimal architecture.We now evaluate the module selection performance of the search over the macro-architecture search space. The results from all combinations of modules given the original PWN are shown in table 2, and the best modules are summarised in table 3 labeled 'best'. The proposed macroarchitecture search algorithm successfully selects the module of the spectral neural forecaster, as they mostly match the best modules. However, the macro-architecture search could not find the best module of the Whittle PC in most of the cases. One possible explanation is that due to their own structural properties, CWSPN converges faster during training than WEin using gradient descent. It is well known that gradient-based architecture search tends to select architectures with higher convergence speed rather than selecting architectures minimizing the objective. This could lead the macro-architecture search to select faster converging modules.To conclude, our method selects the best neural forecasting module in most cases, the selection of the best-performing PC remains challenging and is one potential future research direction. Additionally, table 2 clearly shows that choosing the correct macro-architecture is crucial to obtaining high-performing models. Our approach consistently identifies the best neural forecaster module, allowing it to find state-of-the-art model architectures. After investigating the performance of our bi-level architecture search to find the best macroarchitecture, we proceed by analyzing its behavior in terms of finding micro-architectures. We analyze the frequency at which each operation is selected across different datasets. This allows us to draw conclusions about how sensitive the architecture selection of our algorithm is w.r.t. the data presented. For our analysis, we ran the bi-level architecture search for 5 runs with different random seeds, and illustrate the average number of operations for SRNN (in table 4) and CWSPN (in table 5 ). We refer to Appendix G for better visualization of the tables. We analyze the micro-architectures of the SRNN and CWSPN separately. The micro-architecture search results on SRNN, shown in table 4, demonstrate that the preferred operations differ between data sets. First of all, identity connections are chosen vastly more than other operations, on most of the data sets. This behavior coincides with the behavior of DARTS and has already been discussed in Heuillet et al. (2023) . Identity connections fundamentally represent residual connections, providing an initial performance boost. This initial advantage is often enough to choose this operation over others, even though its long-term performance might be subpar. On the other hand, none connections have been chosen very rarely on all the data sets, which results in much larger model sizes compared to the standard SRNN structure. Secondly, when looking at the activation function operations, one can see that ReLU and sigmoid are the most selected operations on all the data sets. Interestingly, we observe that ReLU operations are chosen more frequently on subsets of the M4 data set with higher sampling rates (weekly, daily and hourly), as well as on Power (15min) and Solar (1h). Similarly, sigmoid operations are more preferred by the data sets with a relatively lower sampling rate, such as Exchange and Wiki (1 day) and M4 (yearly and quarterly).We leave further investigation of this behavior for future work. Table 5 : CWSPN architecture results. Average number of times an operation is present in the optimized CWSPN architecture. We can observe that data sets, with the exception of Exchange, either prefer a combination of 3x1 dilated and 5x1 dilated convolutions or 3x1 and 5x1 dilated convolutions.Similarly, the micro-architecture search results on CWSPN are shown in table 5 . The most notable observation is that the convolution operations are much less frequently selected on the smallest data set Exchange, which might indicate that for data sets with smaller sizes, the fully connected layers are already sufficient. Furthermore, the zero operations are likely to be selected on M4 and Power datasets, whose size are both relatively larger than the rest, resulting in smaller model sizes. Lastly, we also observe that the sampling rate and the length of the context time series also influence the selection of operations. More specifically, the M4 subsets Yearly and Quarterly, which both have a low sampling rate and short context length, tend to select more operations of 3× and 5× dilated convolution, while the rest subsets of the M4 data set are given more operations of 3× and 5× convolution.To summarize, we find that the data presented has a higher impact on the CWSPN microarchitectures than on the SRNN micro-architectures since in the case of SRNN the operation count is quite similar across datasets. This is not the case for CWPSN architectures. Further, we found that our method seems to favor identity connections during the search, similar to the findings in (Heuillet et al., 2023) . Finally, we find that the sampling frequency has a notable effect on the choice of convolution operations in CWSPNs. 
conclusion: 
 In this work, we proposed a novel differentiable neural architecture search capable of optimizing both, the micro-and macro-architecture of hybrid models. We evaluated our algorithm across various time series forecasting datasets using a search space derived from a recent state-of-the-art hybrid model (PWNs) for probabilistic time series forecasting. Empirical results show that our bi-level architecture search algorithm finds suitable micro-and macro-architectures, and in turn, outperforms our baselines on time series forecasting.Limitations & Future Work. One of the limitations of this work is that we did not prove that our bi-level architecture search favors operations with faster convergence properties over operations minimizing the objective function. Probing this conjecture is therefore a good direction for future work as well as exploring how to balance the module selection with their convergence in training if our conjecture holds true. Also, besides inheriting the efficiency of DARTS, we also inherit possible unstable behaviors in some cases. An analysis of the sensitivity of our approach w.r.t. the choice of hyperparameters is an interesting venue for future work as well as exploring other, more robust extensions of DARTS in our framework. Another natural extension is to enable micro-architecture search for other modules, e.g. STrans and WEin, as well as to explore other modules that are even not evaluated in the original PWN. 
broader impact statement: 
 After careful reflection, the authors have determined that this work presents no notable negative impacts to society or the environment. optimization method, and since we apply it on two levels of the architecture, our approach inherits the time requirements of DARTS. Most experiments were conducted on machines with an Nvidia RTX 2080 Ti GPU and an AMD Ryzen 7 3800X CPU. Ablation experiments have been conducted on Nvidia DGX machines with A100 GPUs. The total compute time for all experiments (including baselines) is approximately 5000 GPU hours. 
 
 Appendix: On A100 GPUs, the runtime of one search is approximately 7 hours on average while the training of the found PWN architecture takes about 2.5 GPU days. The maximum memory usage during the experiments was 5GB of RAM (on GPU).In an additional ablation study we only search the macro-and then the micro-architecture. In this setting, the total running time to perform the search on both levels is about 13-15 hours (6.5-7.5 hours for each level), almost double the time required by our bi-level search. Therefore, compared to sequentially first searching for a macro-architecture and then for a micro-architecture (given the selected macro-architecture), by searching both simultaneously with our bi-level approach, we nearly halved the search time. The main computational load is still the full training of the found PWN architecture, which needs 2.5 GPU days.The symmetric mean absolute percentage error (SMAPE) can be calculated asEQUATIONwhere ŷ are the predicted values and 𝑦 refers to the ground truth.In this section, we demonstrate the graph for bi-level architecture search space for PWN. Algorithm 2 Bi-Level Architecture Search. We alternate between updating the model-and architecture weights during optimization (steps 2 and 4). In steps 1 and 2, model weights 𝑤 are updated with a fixed architecture. Therefore, we compute the output of the network by weighting the output of the neural forecasters and Whittle PCs according to the current macro-architecture weights 𝜎 (𝜔 1 ) where 𝜎 is the softmax function. Each forecaster network 𝑁 itself is a supernet (representing the micro-architecture via weights 𝜔 2 ), here the same mechanism is applied. After that the training loss is computed and model weights 𝑤 are updated. In steps 3 and 4 the same procedure is applied. However, the validation loss is computed and architecture weights are updated. 𝑁 , 𝑤 𝑁 ); 𝑜𝑢𝑡 𝑃 = 𝑃 ∈Whittle PCs 𝜎 (𝜔 1 ) 𝑃 • 𝑃 (𝑥, 𝑜𝑢𝑡 𝑁 , 𝑤 𝑃 ); L 𝑣𝑎𝑙 ← L 𝑊 𝐹 (𝑥, 𝑜𝑢𝑡 𝑁 , 𝑜𝑢𝑡 𝑃 , 𝑦) 4. Update micro-and macro-level architecture weights of all micro-level networks with (𝑤 1 , 𝑤 2 ) = (𝑤 1 , 𝑤 2 ) -𝛾 ∇ 𝜔 1 ,𝜔 2 L 𝑣𝑎𝑙 (𝑤 -𝜉∇ 𝑤 L 𝑡𝑟𝑎𝑖𝑛 (𝑤, 𝜔 1 , 𝜔 2 ), 𝜔 1 , 𝜔 2 ) + 𝜌 ||(𝜔 1 , 𝜔 2 )|| where 𝑤 is defined as in step 2.InputThe Example Appendix can be removed. AutoML 2024© 2024 the authors, released under CC BY 4.0Code available at https://github.com/J0nasSeng/Bi-level-optim.git 


Decision: The paper proposes an architecture search approach for probabilistic time-series forecasting. In particular, the method uses an approach similar to DARTS to search for the micro and macro levels operations. 

The reviews are mixed, generally the reviewers saw some quality of the paper in the novel application of Darts to the probabilistic time series forecasting use-case. However, some reviewers also complained about the novelty of the method, issues with empirical results and potential issue with the computational cost.

While most issues related to the computational cost has been addressed by the authors, issues remains on the novelty of the method and its empirical assessment. In particular, the paper seems to be lacking in this aspect as the only baseline are PWN variants but no deep-learning, statistical or AutoML methods are considered making the comparison a bit odd. When looking at the PWN paper, other baselines are compared including deep-learning methods but the results seems inconsistent with the current paper as the accuracy numbers of PWN baselines widely differ. 

One reviewer also noted that the results may not be statistically significant, the authors answered that their methods outperforms all other baseline when taking the the mean of their methods against the mean of a baseline minus the std of the baseline. However, this argument is dubious given that their method exhibits much larger variance (often twice larger), in fact if one takes the mean of their method plus the std of their method, then the number of wins is quite small.

For those reasons, the paper seems to require strenthening in the empirical results, I would encourage the authors to consider Autogluon timeseries [2] and ensemble of statistical forecast methods [3] and take care to analysis the statistical strength of their results, possibly using critical diagrams [4] to make statistical comparisons of multiple methods over multiple datasets.

[1] Predictive Whittle Networks for Time Series. Yu et al.
[2] https://auto.gluon.ai/stable/tutorials/timeseries/index.html
[3] https://github.com/Nixtla/statsforecast
[4] https://www.jmlr.org/papers/volume7/demsar06a/demsar06a.pdf

Reviews: review: paper_summary: paper_summary: The paper presents a bi-level optimization to automatically tune predictive Whittle networks, where the first level is the macro architecture of the nodes in the graph, and the second level is the actual weights (micro architecture) between each connected node. Experiment results demonstrate the effectiveness of the proposed approach.
main_review: main_review: Overall, despite some limited evaluation of the overhead of the approach, I feel that the work is technically sound and the result promising. The only minor point is to provide more justification about the additional overhead and whether any other mechanisms have been applied to tackle such.
review_summary: review_summary: I believe that this paper can have a good impact on the community and promote the bi-level optimization of AutoML. The issue in the evaluation is mostly fixable in the camera-ready version.
paper_summary: paper_summary: This paper proposes to search for the neural architecture of Predictive Whittle Networks (PWN) [1], that is used for time series forecasting. To do that, the authors parameterize both the macro and micro structure of the PWN architecture and deploy standard gradient-based bi-level optimization methods such as DARTS to search for the optimal architecture. Results show that the found architecture is slightly better than the baselines on a couple of standard time series forecasting datasets.

-- References --

[1] Yu et al. Predictive Whittle Networks for Time Series. In UAI 2022
main_review: main_review: While the application of Neural Architecture Search (NAS) to the time series forecasting domain presents an intriguing avenue for improving predictive models, the impact of this approach appears limited. The paper introduces Predictive Whittle Networks (PWN) as the subject of study, proposing to explore their neural architecture using gradient-based bi-level optimization methods like DARTS. However, despite the novelty in applying NAS techniques to time series forecasting, the contribution seems somewhat overshadowed by existing approaches. For instance, prior works such as AGNAS have already delved into the exploration of both macro and micro structures in neural architectures.

Furthermore, while the paper reports marginal improvements over baseline models on standard time series forecasting datasets, the significance of these gains remains questionable. The improvements as shown in Table 2 are statistically not significant, which raises concerns about the practical relevance and generalizability of the proposed architecture. Without substantial performance enhancements, the broader impact of this research on real-world time series forecasting applications may be limited. Moreover, it is not clear to me from the paper what is the computational cost of the NAS search and final training of the found model. It would be great if the authors emphasize this better.

Moreover, the choice of DARTS as the optimization method for searching the optimal architecture raises concerns about the robustness and scalability of the proposed approach. While DARTS is a popular choice for architecture search due to its efficiency, it often suffers from issues such as instability and sensitivity to hyperparameters. Without thorough experimentation and comparison with alternative optimization techniques, the reliability and effectiveness of the proposed method remain uncertain.
review_summary: review_summary: Overall, while the paper presents an interesting exploration into the neural architecture of Predictive Whittle Networks for time series forecasting, its contribution appears somewhat incremental and overshadowed by existing approaches. To make a more substantial impact, future research efforts could focus on addressing the limitations of the proposed approach and exploring alternative methods for architecture search in the time series domain.

-- Updated score --

There seem to be technical flaws in the experimental evaluation that need to be addressed before accepting this paper.
paper_summary: paper_summary: The paper aims to bring neural architecture search (NAS) to the design of predictive Whittle networks (PWNs) used for probabilistic time-series forecasting, which is currently manually designed. To do so, the authors introduce a bi-level optimization framework: at macro- and micro-level, the authors used DARTS-like procedure to both determine both the operations (in the micro level) and the arrangement of the cells resulting from the micro-level search (in the macro level). Across several tasks, the authors show the superiority of the NAS-designed PWNs compared to the state of the art.

-- Post rebuttal -- 

I thank the authors for responding to my comments, which partially addressed my concerns. While the new experimental results resolve my concerns somewhat, what remains unknown to me is how much the performance improvement roots from the addiitonal compute. While it's true that bi-level search outperforms one-level search, it also seems to require more compute and I believe the paper will benefit further to discuss on a compute-matched basis for better understanding of the performance-cost tradeoff. However, I do commend authors for their effort and I've increased my rating.
main_review: main_review: Overall, I think this paper presents a sound idea by straightforwardly adapting NAS methods into a specific problem (probabilistic time series forecasting), and the results seem promising and overall, consistent. As such, I think this paper will be of interest to the people in that relevant community and is of interest to AutoML conference as well. On the flip side, I have concerns on the necessity of the design choices, and I encourage the authors to better justify them through more thorough ablation studies, as this is the core contribution of the paper. Also, as mentioned, methodologically, the paper mainly uses existing ideas from NAS, so I do not deem the methods to be very novel overall (although this is not necessarily a weakness).

Since my concerns about the validity of the design choices touch upon the core contribution of this paper, I am not ready to recommend acceptance yet until I see more concrete evidence supporting the authors' arguments.
review_summary: review_summary: Please see overall review.
paper_summary: paper_summary: This paper presents a new approach for automatically choosing a good design for an approach in probabilistic time series forecasting known as "predictive Whittle networks" (PWNs). PWNs are a hybrid structure, consisting of a macro- and a micro-architecture. The proposed approach is capable of optimizing both architectures in conjunction. The results show a benefit of this new approach over hand-crafted baselines.
main_review: main_review: This paper presents a new approach that outperforms hand-crafted approaches. The authors also analyze the performance of their approach with respect to different features and show that despite the overall top performance, there is still room for improvement on an individual level. I appreciate that. The downsides of this paper are that the main algorithmic approach as well as its cost are not very clearly discussed.
review_summary: review_summary: The approach proposed in this paper seems novel and good to me. It is evaluated fairly and in a diverse manner, making this paper generally very good. However, as some parts are lacking a proper discussion, the paper loses a bit of its value.

## Edit

The author’s updates to the paper convinced me to increase my overall score.