Title: Adaptive Normalization for Non-stationary Time Series Forecasting: A Temporal Slice Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: Time series forecasting

deep learning

normalization
Abstract: Deep learning models have progressively advanced time series forecasting due to their powerful capacity in capturing sequence dependence. Nevertheless, it is still challenging to make accurate predictions due to the existence of non-stationarity in real-world data, denoting the data distribution rapidly changes over time. To mitigate such a dilemma, several efforts have been conducted by reducing the non-stationarity with normalization operation. However, these methods typically overlook the distribution discrepancy between the input series and the horizon series, and assume that all time points within the same instance share the same statistical properties, which is too ideal and may lead to suboptimal relative improvements. To this end, we propose a novel slice-level adaptive normalization, referred to \textbf{SAN}, which is a novel scheme for empowering time series forecasting with more flexible normalization and denormalization. SAN includes two crucial designs. First, SAN tries to eliminate the non-stationarity of time series in units of a local temporal slice (i.e., sub-series) rather than a global instance. Second, SAN employs a slight network module to independently model the evolving trends of statistical properties of raw time series. Consequently, SAN could serve as a general model-agnostic plugin and better alleviate the impact of the non-stationary nature of time series data. We instantiate the proposed SAN on four widely used forecasting models and test their prediction results on benchmark datasets to evaluate its effectiveness. Also, we report some insightful findings to deeply analyze and understand our proposed SAN. We make our codes publicly available.

Adaptive Normalization for Non-stationary Time Series Forecasting: A Temporal Slice Perspective

Abstract

Deep learning models have progressively advanced time series forecasting due to their powerful capacity in capturing sequence dependence. Nevertheless, it is still challenging to make accurate predictions due to the existence of non-stationarity in real-world data, denoting the data distribution rapidly changes over time. To mitigate such a dilemma, several efforts have been conducted by reducing the non-stationarity with normalization operation. However, these methods typically overlook the distribution discrepancy between the input series and the horizon series, and assume that all time points within the same instance share the same statistical properties, which is too ideal and may lead to suboptimal relative improvements. To this end, we propose a novel slice-level adaptive normalization, referred to SAN , which is a novel scheme for empowering time series forecasting with more flexible normalization and denormalization. SAN includes two crucial designs. First, SAN tries to eliminate the non-stationarity of time series in units of a local temporal slice (i.e., sub-series) rather than a global instance. Second, SAN employs a slight network module to independently model the evolving trends of statistical properties of raw time series. Consequently, SAN could serve as a general model-agnostic plugin and better alleviate the impact of the non-stationary nature of time series data. We instantiate the proposed SAN on four widely used forecasting models and test their prediction results on benchmark datasets to evaluate its effectiveness. Also, we report some insightful findings to deeply analyze and understand our proposed SAN. We make our codes publicly available 2 .

1 Introduction

Time series forecasting is becoming increasingly prevalent in real-world scenarios. Various applications have been facilitated by the advancement of forecasting, such as energy consumption planning [33], clinical healthcare analysis [16], financial risk assessment [15] and cloud resource allocation [2]. Recently, deep learning-based methods have largely advanced forecasting and other tasks due to their powerful capacity to capture sequence dependence [23, 27, 3, 42].

Nevertheless, it is still challenging to make accurate predictions for time series forecasting due to the rapid evolution of time series points over time (a.k.a. non-stationarity of time series) [31]. Such

non-stationarity can lead to discrepancies between different time spans and hinder the generalization of deep learning models. To alleviate the impact of the non-stationary nature, removing these dynamic factors from the original data through normalization has been proposed as a feasible solution [28].

Recently, some pioneering efforts have been devoted to this research topic [17, 25, 10]. Although these normalization approaches have significantly improved the prediction performance, we identify two limitations in existing solutions. On the one hand, most existing methods overlook the distribution discrepancy between the input series and the horizon series, and simply adopt the statistical properties of the input series to denormalize the output results. Furthermore, previous studies assume that all time points within the same instance share the same statistical properties during the normalization processing, and a global instance normalization is widely applied. Such coarse-grained settings are not appropriate since time series points rapidly change over time [6, 20], particularly in long-term forecasting scenarios where both input and horizon series may span a considerable duration. For example, there may be sudden changes in data distribution due to events like holidays or temperature spikes for electricity consumption data.

We plot a forecasting sample in Fig. 1 to better illustrate our opinion. Though temporally related, the input series' mean differs from the horizon's significantly (from 0.75 to 1.5), indicating a potentially universal distribution discrepancy. Moreover, such a distribution shift can happen rapidly at a more fine-grained slice level, violating the basic assumption of existing normalization methods. Therefore, these approaches risk damaging instinct patterns of each slice of the input sequence by normalizing with improper statistics while also causing a prediction shift in final forecasting results due to poor estimation of future statistics .

To overcome these limitations, we propose a general normalization framework for nonstationary time series forecasting named S licing A daptive N ormalization (SAN). SAN models the non-stationarity in the

Input Series' Mean

Horizon Series' Mean

Sliced

Distribution

Shift

fine-grained temporal slices , or patches[26], which are more informative than single data points and can be regarded as fundamental units of the time series data[4, 14]. To be specific, the input sequence is first split into non-overlap equally-sized slices, which are then normalized according to their statistics and fed into the forecasting model. Meanwhile, we use a statistics prediction module to predict the distributions of future slices based on the statistics of the input. Finally, the non-stationary information is restored to the output of the forecasting model with well-estimated statistics. By modeling the slice-level characteristic, SAN is able to remove the non-stationarity in a local region. Besides, with the statistics prediction module independently modeling the evolving trends of statistical properties, SAN adopts more precise statistics for adaptive denormalization. Consequently, the non-stationary forecasting task is actually simplified by being split into statistic prediction and stationary forecasting. Moreover, SAN is a model-agnostic framework and can be applied to arbitrary forecasting models. Sufficient experiments have been conducted in a widely used benchmark dataset. The results demonstrate that SAN surpasses advanced normalization approaches by boosting the performance of various kinds of mainstream forecasting models by a large margin.

In summary, our main contributions are as follows:

We propose SAN, a general normalization framework for non-stationary time series forecasting tasks which distinguishes by modeling the non-stationary nature from a temporal slice perspective. In this way, SAN can better remove the non-stationary factors in input sequences while keeping their distinct patterns.

We design a flexible statistics prediction module for SAN which independently models the evolving trends of statistical properties. By explicitly learning to estimate future distributions, SAN can simplify the non-stationary forecasting task through divide and conquer.

We conduct sufficient experiments on nine real-world datasets. Results show that SAN can be applied to various mainstream forecasting models and boost the performance by a large margin. Moreover, the comparison between SAN and state-of-the-art normalization methods demonstrates the superiority of our proposed framework.

2 Related Works

2.1 Time Series Forecasting

Time series forecasting has been extensively studied in recent decades. Originally, ARIMA [1, 40] builds an auto-regressive model and forecasts in a moving average fashion. Though theoretical guarantees are achieved, such traditional methods usually require data with ideal properties, which is inconsistent with the real-world scenario. With the increasing data availability and computing power, numerous deep-learning-based models have emerged, which always follow a sequence-to-sequence paradigm. Recurrent neural networks (RNNs) are first utilized to capture the temporal dependence by summarizing the past information in time series [30, 32, 37]. Such architectures naturally suffer from a limited reception field and an error accumulation issue caused by the recursive inference schema[43], both dragging down forecasting precision. To further boost the performance of the final prediction, many advanced architectures have been introduced to capture the long-range dependencies, such as the self-attention mechanism and the convolutional networks [19, 21, 24]. Besides, to leverage the characteristics of the time series data, recent works also integrate traditional analysis methods like trend-seasonal decomposition and time-frequency conversion into neural networks [36, 44]. In addition, a recent study points out that a simple linear network enhanced with decomposition also achieves competitive performance[39]. Furthermore, slice-based methods demonstrate superior accuracy in the long time series forecasting tasks [26, 41].

2.2 Non-stationary Time Series Forecasting

Most time series forecasting methods prioritize designing powerful architectures that can effectively capture temporal dependencies, but often overlook the non-stationary nature of the data. Considering the basic assumption of deep-learning-based models that the data in both training and test sets follow the same distribution, such a discrepancy will definitely drag down the precision of the model for future time prediction. Moreover, the distribution differences among instances in the training set may introduce noise, making the learning task harder to converge. To address these challenges, various stationarization methods have been explored.

In detail, DDG-DA[22] predicts the evolving data distribution in a domain adaptation fashion. Du et al.[8] propose an adaptive RNN to alleviate the impact of non-stationary factors by distribution characterization and distribution matching. Besides, normalization-based approaches have also gained popularity as they aim to remove non-stationary factors from original data and normalize all data to a consistent distribution. DAIN [28] introduces a non-linear network to learn how to normalize each input instance adaptively and ST-norm [7] proposes two normalization modules from both temporal and spatial perspectives. Later researchers point out that non-stationary factors are essential in accurate forecasting and simply removing them may result in poor prediction. Therefore, they propose RevIN [17], a symmetric normalization method that first normalizes the input sequences and then denormalizes the model output sequences through instance normalization [34]. Based on the similar structure, Non-stationary Transformers [25] presents de-stationary attention that incorporates the non-stationary factors in self-attention, resulting in significant improvements over Transformerbased models. Moreover, a recent study [10] identifies the intra- and inter-space distribution shift in time series, and proposes to relieve these issues by learning the distribution coefficients.

Despite the effectiveness of existing normalization methods, they inappropriately assume that all time points within the same instance share the same statistical properties during the normalization processing. Different from them, our proposed approach focuses on further thinking of the nature of the data, i.e., the distribution is inconsistent across compact time slices and such inconsistency is not just on a per-instance basis.

3 Proposed Method

We propose a general model-agnostic normalization framework for time series forecasting called Slicing Adaptive Normalization (SAN) to address the inconsistency mentioned above. Considering an input set of time series X = { x i } N i =1 and their horizon series Y = { y i } N i =1 , SAN is expected to remove the non-stationary factors and assist the forecasting models to predict more accurately based on the observed input series. In this section, we will present the detailed workflow of the entire framework and explain how it works with non-stationary time series data. To provide better clarity, we summarize the key notations in Table 1 and the whole framework can be referred to in Fig. 2.

3.1 Normalization

Similar to existing normalization methods for non-stationary time series forecasting [17], SAN first normalizes the input sequence to remove the non-stationary factors and later restores them to the output sequence by denormalization. Differently, SAN applies such operation on a per-slice basis instead of the whole input sequence. Such a localized operation can better maintain the instinct pattern of each slice than global instance normalization. The framework first splits the input x i into M non-overlapping slices { x i j } j M =1 based on T . Then the mean and standard deviation for each slice is computed as:

where µ i j , σ i j ∈ R V ∗ 1 and x i j,t is the value of slice x i j at t -th time step. Later, SAN normalizes every slice of the original input sequence by their individual statistics as:

Here we use · to denote the element-wise product and ϵ is a small constant. Finally, SAN restores all the slices in their original chronological order and lets the processed series without non-stationary factors ¯ x i be the new input of the forecasting models.

3.2 Statistics Prediction

As illustrated in Fig. 2, SAN introduces a unique statistics prediction module, f ϕ ( ∗ ) , to better estimate future distributions in addition to the backbone forecasting model, g θ ( ∗ ) . Unlike existing works that denormalize the entire output of backbone models with the statistics of original input sequences, SAN faces a natural challenge of per-slice normalization: how to estimate the evolving distributions for each future slice. To simplify and improve efficiency, we use a two-layer perceptron network with an appropriate activation function (e.g., Relu() for standard deviation to ensure non-negativity) that learns to predict future distributions based on input statistics and stationarized sequence.

The quality of statistics predictions determines the overall performance of SAN since we depend on an accurate estimation of future distribution to restore the non-stationary nature of each instance. In

Normalization

Slicing

Input

Forecasting model

Statistics prediction

module

𝝁 𝑖

, 𝝈

𝑖

Slicing & Denormalization

ෝ

𝝁 𝑖

, ෝ

Output

ෝ

𝒚 𝑖

Normalization

Forecasting

&

Statistics Prediction

Denormalization

𝑓 𝜙 (∗)

𝑔𝜃(∗)

stop-gradient

Horizon

𝒚 𝑖

𝑙

𝑓𝑐

𝑙

𝑠𝑝

ഥ

𝒚 𝒊

ഥ

our modeling of distribution, the mean determines the approximate scale of a given slice and the standard deviation represents the degree of dispersion, where the scale of a small slice may be more important in the task of forecasting. Therefore, we aim to further refine the modeling approach for the mean component on the basis of analyzing its properties.

In detail, we believe that the overall mean of the input sequence ρ i = 1 L in ∑ L in 1 x i ∈ R V ∗ 1 is a maximum likelihood estimation of the target sequence's mean ˆ ρ i = 1 L out ∑ L out 1 y i ∈ R V ∗ 1 since they are temporally related. That is, ρ i ≈ ˆ ρ i . Such property is widely accepted in existing works [17, 25] as they denormalize the output using the statistics of the whole input sequence. Based on the above assumption, we introduce a residual learning [13] technique in our method, letting the module learn the difference between the future slice mean ˆ µ i and the overall input mean ρ i , instead of predicting the specific values. This approach reduces the difficulty in modeling means with prior knowledge about future trends. Additionally, to account for different variables exhibiting distinct patterns in scale changes, we further use two learnable vectors W 1 , W 2 ∈ R V initialized as ones-vector to present the individual preference weights for each variable, making the prediction computed in a weighted-sum manner. The statistics prediction procedure can be formulated as:

Here µ i = [ µ i 1 , µ i 2 ...µ i M ] ∈ R V ∗ M denotes all the mean values of M slices of the input, and ˆ µ i ∈ R V ∗ K stands for the predicted mean of future K slices. The same notation works for the standard deviation. The mean squared error (MSE) between predicted statistics and ground truth is utilized as the loss function ( l sp ) to train the network through backpropagation.

In this work, we mainly focus on proposing and modeling the non-stationary nature of time series from a slicing perspective. The challenge of how to design powerful deep models for statistics prediction is left for future explorations.

3.3 Denormalization

Simultaneous with statistic prediction, SAN feeds the normalized sequence into the forecasting model, which is responsible for producing internal output ¯ y i . Finally, SAN denormalizes the output given by the backbone, restoring the non-stationary factors for an accurate forecasting result.

Symmetrically, SAN performs on a per-slice basis as illustrated in Fig. 2. For the internal output ¯ y i , we first split it into K non-overlapping slices { ¯ y i j } K j =1 . Then the denormalization operation for an arbitrary slice based on our predicted statistics can be defined as the following formula:

Finally, by restoring all the slices in their chronological order, we can get the final prediction ˆ y i of the whole framework, which will be later used for loss computation ( l fc ) and performance evaluation.

3.4 Two-stage Training Schema

Though the overall framework is simple and clear, we find that the training process needs to be carefully deliberated. Since the normalization flow of SAN functions as a constraint to the backbone model, the overall learning procedure is actually a bi-level optimization problem [12]. The goal of the upper level is the performance of time series forecasting while the goal of the lower level is the distribution similarity between denormalized output and ground truth. Formally, the original overall training process can be described as:

Here we omit the transformation process of the data and only keep the original input required for the calculation for brevity.

We propose a two-stage training paradigm for SAN by simplifying the lower-level optimization objective so that it can focus on estimating the future distribution, instead of reducing the distribution discrepancy between the denormalized output of a certain model and the ground truth. Specifically, we optimize ϕ ∗ = arg min ϕ ∑ ( x i , y i ) l sp ( ϕ, ( x i , y i )) using stochastic gradient descent. This decouples the original non-stationary forecasting task into a statistic prediction task and a stationary forecasting task. In practice, the statistics prediction module is first trained into convergence, which is then frozen and treated as a plugin during the second stage of training the forecasting model. The training algorithm is provided in the Appendix C.2.

Such a solution has some desirable qualities: The first is simplicity. The two-stage schema allows for a concise and easy-to-follow design of the model architecture and training process. The second is effectiveness. The statistics prediction module is expected to produce reliable predictions on future distribution since it is optimized on the whole training set into convergence. Therefore the forecasting model can handle the simpler task of learning the scale-free pattern in the normalized data. These two advantages greatly meet our ultimate goal of designing a concise yet effective framework for non-stationary time series forecasting tasks. The third and the most important is flexibility. Though there exist many advanced methods for the bi-level optimization problem [5, 11], their upper and lower objectives are always highly related. In contrast, our proposal completely decouples these parts, making SAN a model-agnostic framework that can migrate to various scenarios without special design and further tuning.

4 Experiments

In this section, we conduct sufficient experiments within a widely used benchmark dataset compared to state-of-the-art methods to evidence the effectiveness of our proposed SAN framework.

4.1 Experimental Setup

Datasets We use nine datasets in our experiment and here are brief descriptions of them. (1) ETT 3 [43] records the oil temperature and load features of the electricity transformers from July 2016 to July 2018. It is made up of 4 sub datasets where ETThs are sampled per hour and ETTms are

sampled every 15 minutes. (2) Electricity 4 contains the electricity consumption data of 321 clients from July 2016 to July 2019. (3) Exchange 5 [19] collects the daily exchange rates of 8 countries from 1990 to 2016. (4) Traffic 6 includes the hourly traffic load of San Francisco freeways recorded by 862 sensors from 2015 to 2016. (5) Weather 7 is made up of 21 indicators of weather, including air temperature and humidity collected every 10 minutes in 2021. (6) ILI 8 records the weekly ratio of influenza-like illness patients versus the total patients by the Centers for Disease Control and Prevention of the United States from 2002 to 2021. The detailed information about these datasets are listed in the Table. 2. We also report the ADF test (Augmented Dickey-Fuller Test) [9] results in the table, which evaluate the stationarity of a time series. Following the standard protocol, we split each dataset into training, validation and testing sets according to the chronological order. The split ratio is 6:2:2 for ETT dataset and 7:1:2 for the other datasets [38]. Also, we apply a z-score normalization on them based on the statistics of training data as preprocessing to measure different variables on the same scale. Note that z-score normalization is unable to handle non-stationary time series since the statistics are fixed during normalization [28].

∗ A smaller ADF test result indicates a more stationary time series data

Backbone models SAN is a model-agnostic framework that can be applied to arbitrary time series forecasting models. To evidence the effectiveness of the framework, we select some mainstream models based on different architectures and evaluate their performance under both multivariate and univariate settings: Linear model based DLinear [39], Transformer based Autoformer [38] and FEDformer [45], and dilated convolution based SCINet [24]. We follow the implementation and settings provided in the official code of DLinear 9 and SCINet 10 to implement these models.

Experiments details We use ADAM [18] as the default optimizer across all the experiments and report the mean squared error (MSE) and mean absolute error (MAE) as the evaluation metrics. A lower MSE/MAE indicates a better performance. For the statistics prediction module in SAN, we use a simple two-layer perceptron network with a hidden size the same as the embedding size of the backbone model for simplicity. The detailed implementation of the statistics prediction module can be referred to in Appendix C.1. All the experiments are implemented by PyTorch [29] and are conducted for three runs with a fixed random seed on a single NVIDIA RTX 3090 24GB GPU.

Slicing length Regarding the selection of slicing length for each dataset, we adopt a heuristic idea that real-world time series data exhibit similar changing patterns within artificially defined or actual periods (daily, weekly, etc.). Combing the frequencies of benchmark datasets, we establish a range of { 6 , 12 , 24 , 48 } as slicing lengths such that most settings cover a meaningful time span. For example, we selected a slicing length of 24 for datasets such as ETTh1, Electricity and Traffic with a frequency of 1 hour. This ensures that each time slice contained data within a day and guarantees optimal performance among candidates. Here we admit that one limitation of our method is that the current design cannot handle indivisible length such that we set the slicing length to 6 which approximately represents a weekly period instead of 7 in the Exchange dataset. We present the ablation study on the effect of slicing length in Appendix B.5.

4.2 Main Results

We report the multivariate forecasting results in Table 3. The ILI dataset has a forecasting horizon of L out ∈ { 24 , 36 , 48 , 60 } while the others have a forecasting horizon of L out ∈ { 96 , 192 , 336 , 720 } . As for the input sequence length, we follow the traditional protocol and fix L in = 96 for Autoformer, FEDformer and SCINet with respect to all datasets ( L in = 36 for ILI dataset) and extend it to 336 (96 for ILI dataset) for DLinear. Full benchmarks of ETT datasets and univariate results are provided in the Appendix.

As shown in the table, we clearly find that our proposed SAN framework can boost these models by a large margin in most cases of the benchmark dataset. We attribute this improvement to two aspects. Firstly, SAN mitigates the impact of non-stationary factors, as demonstrated by the performance on three typical non-stationary datasets (Exchange, ILI and ETTh2, determined by ADF test results). Specifically, under all experimental forecasting lengths with DLinear, SAN achieves an average MSE reduction of 7.67% in the Exchange dataset, 11.13% in the ILI dataset and 21.29% in the ETTh2 dataset. This conclusion applies to other backbone models as well and the enhancement is even more pronounced. Secondly, even in long-term forecasting scenarios where the difficulty of forecasting increases significantly with the length of the forecast, SAN imposes constraints on

backbone models to produce more reliable results using a novel statistical prediction module. For instance, when predicting for a length of 720 time steps, SCINet accompanied by SAN achieves a 70.37% reduction in MSE on the ETTh2 dataset and a 20.77% reduction on the Electricity dataset. These improvements make SCINet comparable to other forecasting models and suggest that SAN can help stabilize outputs in long-term forecasting scenarios.

(a) SAN

(b) RevIN

(c) NST

(d) Dish-TS

4.3 Comparison With Normalization Methods

In this section, we compare SAN with three state-of-the-art normalization methods for non-stationary time series forecasting: RevIN [17], Non-Stationary Transformers ( NST ) [25] and Dish-TS [10]. Following the same experimental settings in Section 4.2, we report the average MSE evaluation of Autoformer and FEDformer over all the forecasting lengths for each dataset and the relative improvements in Table 4. Other models are not as involved backbone since NST can only suit Transformer-based models, while the rest methods are much more flexible to be applied for arbitrary forecasting models on the contrary.

It can be concluded that SAN achieves the best performance among existing normalization methods. The improvement is significant with an average MSE decrease of 10.71% by FEDformer. SAN consistently performs better than baseline models except for Weather and the improvement is more pronounced in typical non-stationary datasets like Exchange and ILI (determined by the ADF test). The comparison reveals that SAN may be more effective at removing non-stationary factors from a time-slicing perspective rather than considering the entire instance. Additionally, the proposed two-stage training schema is crucial as it enables SAN to outperform Dish-TS by a large margin, which ignores the bi-level optimization nature. However, this exceptional ability of SAN may lead to an over-stationarization problem [25], resulting in decreased performance on the Weather dataset. The detailed results of all cases and further discussions are provided in Appendix B.6.

4.4 Qualitative Evaluation

The quality of prediction results in time series forecasting is crucial, in addition to the accuracy of metrics. Figure 3 displays a sample forecast on the ETTm2 dataset using FEDformer as the backbone with SAN, RevIN, NST or Dish-TS enhancements. The input length is 96 and the forecasting length is set to 336. It's evident that SAN produces more realistic predictions while its counterparts even fail to capture the scale of future data. We guess the poor quality of RevIN and NST is caused by their coarse way of denormalizing. Although the mean value of an input sequence can be considered a maximum likelihood estimation for future data, it's likely that non-stationary datasets' distribution will change significantly in comparison to inputs. Therefore, simply denormalizing output from backbone models with input sequence statistics may lead to mismatches like those seen in RevIN and NST forecasts where both scales are similar. As for Dish-TS, though the method tries to learn future distribution, it ignores the bi-level optimization nature and its entangled learning schema limits the estimation accuracy of statistics and finally leads to poor performance. On the opposite, SAN models the dynamic nature of time series from a slicing perspective and introduces an independent statistics prediction module to learn to predict the future distribution for denormalizing by a two-stage training schema. In this way, we adaptively adjust the scale and bias of forecasting results based on the statistic predictions, capturing the tendency of future data. As a result, though the average value of the input is rather low, SAN still produces higher predictions that are consistent with ground truth.

5 Conclusion

In this study, we focused on alleviating the non-stationary property of time series data using a novel slice view. We proposed the SAN framework for time series forecasting, which is a modelagnostic approach that normalizes the input by removing non-stationary factors and restores them to the output through denormalization on a per-slice basis. Additionally, with the help of a novel statistics prediction module, SAN simplifies non-stationary forecasting by dividing it into two subtasks to improve forecasting model performance. To demonstrate the superiority of SAN, we conducted experiments on a widely used benchmark dataset and found that SAN significantly improves mainstream forecasting models and outperforms state-of-the-art normalization methods. We hope that SAN can serve as a foundation component for time series forecasting, and stimulate further research on modeling time series from a slice perspective.

References

George EP Box and Gwilym M Jenkins. 1968. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics) 17, 2 (1968), 91-109.

Hongjie Chen, Ryan A Rossi, Kanak Mahadik, Sungchul Kim, and Hoda Eldardiry. 2021. Graph deep factors for forecasting with applications to cloud resource allocation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining . 106-116.

Mingyue Cheng, Qi Liu, Zhiding Liu, Zhi Li, Yucong Luo, and Enhong Chen. 2023. FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification. In Proceedings of the ACM Web Conference 2023 . 1437-1445.

Mingyue Cheng, Qi Liu, Zhiding Liu, Hao Zhang, Rujiao Zhang, and Enhong Chen. 2023. TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders. arXiv preprint arXiv:2303.00320 (2023).

Benoît Colson, Patrice Marcotte, and Gilles Savard. 2007. An overview of bilevel optimization. Annals of operations research 153, 1 (2007), 235-256.

Shohreh Deldari, Daniel V Smith, Hao Xue, and Flora D Salim. 2021. Time series change point detection with self-supervised contrastive predictive coding. In Proceedings of the Web Conference 2021 . 3124-3135.

Jinliang Deng, Xiusi Chen, Renhe Jiang, Xuan Song, and Ivor W Tsang. 2021. St-norm: Spatial and temporal normalization for multi-variate time series forecasting. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining . 269-278.

Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. 2021. Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management . 402-411.

Graham Elliott, Thomas J Rothenberg, and James H Stock. 1992. Efficient tests for an autoregressive unit root.

Wei Fan, Pengyang Wang, Dongkun Wang, Dongjie Wang, Yuanchun Zhou, and Yanjie Fu. 2023. DishTS: A General Paradigm for Alleviating Distribution Shift in Time Series Forecasting. arXiv preprint arXiv:2302.14829 (2023).

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. 2018. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning . PMLR, 1568-1577.

Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. 2016. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05447 (2016).

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770-778.

Wenqiang He, Mingyue Cheng, Qi Liu, and Zhi Li. 2023. ShapeWordNet: An Interpretable Shapelet Neural Network for Physiological Signal Classification. In International Conference on Database Systems for Advanced Applications . Springer, 353-369.

Min Hou, Chang Xu, Zhi Li, Yang Liu, Weiqing Liu, Enhong Chen, and Jiang Bian. 2022. MultiGranularity Residual Learning with Confidence Estimation for Time Series Prediction. In Proceedings of the ACM Web Conference 2022 . 112-121.

Shruti Kaushik, Abhinav Choudhury, Pankaj Kumar Sheron, Nataraj Dasgupta, Sayee Natarajan, Larry A Pickett, and Varun Dutt. 2020. AI in healthcare: time-series forecasting using statistical, neural, and ensemble architectures. Frontiers in big data 3 (2020), 4.

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. 2021. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations .

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research &development in information retrieval . 95-104.

Kwei-Herng Lai, Daochen Zha, Junjie Xu, Yue Zhao, Guanchu Wang, and Xia Hu. 2021. Revisiting time series outlier detection: Definitions and benchmarks. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 1) .

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems 32 (2019).

Wendi Li, Xiao Yang, Weiqing Liu, Yingce Xia, and Jiang Bian. 2022. DDG-DA: Data Distribution Generation for Predictable Concept Drift Adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 36. 4092-4100.

Bryan Lim and Stefan Zohren. 2021. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A 379, 2194 (2021), 20200209.

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. 2022. SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction. Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022 (2022).

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. 2022. Non-stationary Transformers: Rethinking the Stationarity in Time Series Forecasting. arXiv preprint arXiv:2205.14415 (2022).

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2022. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In The Eleventh International Conference on Learning Representations .

Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2020. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations .

Nikolaos Passalis, Anastasios Tefas, Juho Kanniainen, Moncef Gabbouj, and Alexandros Iosifidis. 2019. Deep adaptive input normalization for time series forecasting. IEEE transactions on neural networks and learning systems 31, 9 (2019), 3760-3765.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).

Gábor Petneházi. 2019. Recurrent neural networks for time series forecasting. arXiv preprint arXiv:1901.00069 (2019).

Fotios Petropoulos, Daniele Apiletti, Vassilios Assimakopoulos, Mohamed Zied Babai, Devon K Barrow, Souhaib Ben Taieb, Christoph Bergmeir, Ricardo J Bessa, Jakub Bijak, John E Boylan, et al. 2022. Forecasting: theory and practice. International Journal of Forecasting (2022).

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. 2020. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36, 3 (2020), 1181-1191.

Arunesh Kumar Singh, S Khatoon Ibraheem, Md Muazzam, and DK Chaturvedi. 2013. An overview of electricity demand forecasting techniques. Network and complex systems 3, 3 (2013), 38-48.

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2016. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016).

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).

Qingsong Wen, Zhe Zhang, Yan Li, and Liang Sun. 2020. Fast RobustSTL: Efficient and robust seasonaltrend decomposition for time series with complex patterns. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 2203-2213.

Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. 2017. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053 (2017).

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems 34 (2021), 22419-22430.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are Transformers Effective for Time Series Forecasting? Proceedings of the AAAI Conference on Artificial Intelligence .

G Peter Zhang. 2003. Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing 50 (2003), 159-175.

Yunhao Zhang and Junchi Yan. 2022. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations .

Yi Zheng, Qi Liu, Enhong Chen, Yong Ge, and J Leon Zhao. 2014. Time series classification using multi-channels deep convolutional neural networks. In International conference on web-age information management . Springer, 298-310.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. 11106-11115.

Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. 2022. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems 35 (2022), 12677-12690.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022) (Baltimore, Maryland).

A Effects of SAN on Non-stationary Time Series Forecasting

A.1 Discussions

As illustrated in the main paper, our proposed SAN is a compact plug-and-play framework. We will first give a brief discussion in this section on how SAN can be effective.

It is of utmost importance that SAN can well alleviate the impact of the non-stationary nature of time series data. Forecasting models may encounter a non-i.i.d problem with non-stationary data, that is, the marginal distribution of each input instance can be different, which may lead to a huge difference between the distribution of the training set and the test set. Thus the models can not generalize well in future predictions. However, SAN will normalize all the input instances into a standard normal distribution and force the mean and variance of the training and test data distributions to be identical. In this way, all the data instances are from the same distribution, therefore the forecasting task is simplified as the models can get rid of the noises caused by non-stationary factors and only focus on mining the time-invariant patterns. Moreover, compared to existing normalization methods for forecasting, our modeling of the non-stationary property in a time slice view is more in-depth and realistic, so SAN can better remove the non-stationary factors in input sequences while keeping their instinct information in the normalization phase. Hence, SAN is theoretically expected to perform better in non-stationary time series forecasting.

Another part that contributes to the effectiveness of SAN is the statistics prediction module and the two-stage training schema. With the statistics prediction module independently modeling the evolving trends of statistical properties, SAN adopts more precise statistics for adaptive denormalization than existing solutions. Moreover, the proposed two-stage strategy actually simplifies the original forecasting task by divide and conquer: In the first stage we try to learn the general direction and dispersion of the future data, which is easy to fit and is conducted by the light statistics prediction module. Next, we utilize the powerful backbone model to discover the scale-free periodic-like features to estimate future values under the guidance of the well-trained statistics prediction module. Therefore, backbone models in SAN are actually responsible for an easier subtask. Considering that SAN can usually give reliable estimations on future distributions, SAN is expected to perform well on non-stationary time series forecasting by splitting the task into two simpler subtasks.

A.2 Theoretical Analysis

Using the same notation in the paper, we prove that all the inputs after SAN's normalization follow a standard normal distribution, validating SAN's capability to remove the non-stationary factors theoretically.

In detail, for arbitrary input sequence x i , SAN first split it into M non-overlapping slices { x i j } j M =1 and normalizes them according to their statistics. Therefore we will get:

And as for the statistics of normalized input ¯ x i , it satisfies the following equations:

Here ¯ x i : ,t ∈ R V ∗ 1 denotes all the normalized variables in time step t . From the above equations, we can learn that any input sequence follows a standard normal distribution after the normalization operation of SAN, which meets our expectations.

B Supplementary Experiments

B.1 Full Benchmark on the ETT Dataset

We provide the full multivariate forecasting results on the ETT dataset in Table 5, which includes the hourly datasets ETTh1&ETTh2 and the 15-minutes datasets ETTm1&ETTm2. It is obvious that SAN also achieves significant improvements on these datasets on various backbone models.

B.2 Univariate Forecasting Results

Following the same settings of our main experiment, we provide the univariate forecasting results in Table 6. Similar to the results of multivariate forecasting, SAN can boost the performance of mainstream forecasting models in most cases. On average of all the benchmark settings, DLinear enhanced by SAN reduces MSE by 6.04% (from 0.230 to 0.214). The improvements for FEDformer, Autoformer and SCINet are 15.40% , 29.27% and 36.29% respectively.

B.3 Validation on Various Input Lengths

The input length plays an essential role in time series forecasting tasks as it determines how much historical temporal information the model can mine. One may hope that for powerful deep models, the longer the input length, the better the forecasting results. However, a recent study on this question reveals that deep Transformer-based models are not capable of capturing temporal dependencies in the long-term input sequences [39]. That is, the performance of these deep models stays stable or even degrades when the input length increases.

Apart from the design of these deep models, we hold that such a phenomenon can be raised by the non-stationary property of time series. As the input length increases, the variance among input sequences grows larger and ultimately makes it harder for deep models to discover the time-invariant patterns. Therefore, by removing the non-stationary factors in the input by SAN, deep models are expected to exhibit a steady decline in metrics with longer input lengths.

To evidence our thoughts, we conduct long-term forecasting experiments, i.e., L out = 720 , with various input lengths L in ∈ { 24 , 48 , 72 , 96 , 120 , 144 , 168 , 192 , 336 , 504 , 672 , 720 } on the Transformerbased models. Here we choose Transformer [35], Informer [43], Autoformer [38] and FEDformer [45] as the backbone models. The MSE evaluations are plotted in Fig. 4. Note that we omit large values in the line chart to better demonstrate the trend of the overall results. From the figure, we can see that with the assistance of SAN, the performance of deep models with long sequence input is

0.18

0.23

0.28

0.33

0.38

0.43

24

48

72

96

120

144

168

192

336

504

672

720

Transformer

Transformer+SAN

Autoformer

Autoformer+SAN

FEDformer

FEDformer+SAN

Informer

Informer+SAN

(a) 720 steps-Electricity

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

1.10

24

48

72

96

120

144

168

192

336

504

672

720

(b) 720 steps-Weather

largely improved. When the input length is set to 720 on the Electricity dataset, the performance of Informer has been boosted by 77.83% (from 0.9426 to 0.2090), and the average improvement on four backbones under the same setting is 52.55% . Moreover, all of the backbones enhanced by SAN tend to produce more accurate forecasting as the length increases. To be specific, on the Weather dataset, Transformer achieves a reduction on MSE of 29.40% when prolonging input from 24 steps to 720 steps, and the average improvement on four backbones is 33.11% . These results greatly meet our expectations and also validate the effectiveness of SAN on various input lengths.

B.4 Additional Prediction Showcases

We provide the additional comparison between SAN and other normalization methods in Fig. 5 with FEDformer [45] on various datasets. Clearly, SAN can better estimate the future distribution so as to help the backbone model to achieve superior performance, where the forecasting results are better aligned with the groundtruth.

(a) SAN

(b) RevIN

(c) NST

(d) Dish-TS

ILI

ETTh2

Exchange

B.5 Ablation Study

Statistic Prediction Module In this section, we aim to analyze the effectiveness of our designs in the statistic prediction module. We instantiate our method and its variants on Autoformer and test their performance on two typical non-stationary datasets: Exchange and ETTh2. Similarly, we repeat the experiments three times with fixed seed and report the evaluations with standard deviation in Table 7.

Obviously, with the proposed two techniques combined, the statistic prediction module can achieve the best accuracy, leading to optimal forecasting performance. Besides, both residual learning and individual preference contribute positive effects and the former one is much more important, without which SAN can even bring negative effects to the backbone model. These results validate the rationality of our thoughts about the characteristics of the mean value and also reveal the importance of accurate modeling of future statistics to SAN. Besides, SAN without individual

modeling performs well on the ETT2 dataset but performs poorly on the Exchange dataset. Such a phenomenon reveals that the evolving trends of different scenarios vary, and it is required to model the complex relationships among multiple variables individually. Moreover, since we only incorporate the properties of mean values into a simple MLP network, how to design a proper mechanism or network architecture for statistics modeling is a promising direction for optimizing our method, and we leave such explorations for future work.

Slicing Length The slicing length is a key parameter of SAN. We aim to study the effect of different slicing lengths on our method. Ablation experiments are conducted by using SCINet as the backbone model under the long-term forecasting setting ( L out = 60 for the ILI dataset and L out = 720 for the rest datasets). Each experiment is conducted three times with a fixed random seed. The forecasting errors and the corresponding standard deviation are presented in Table 8.

Our heuristic selection of slicing length appears to be effective among the candidates, indicating that both artificially defined and actual periods are useful in selecting the optimal setting. Additionally, there were no significant performance differences observed under various settings, suggesting that SAN is resilient to changes in slicing length.

B.6 Detailed Results of the Comparison between SAN and Normalization Methods

In Table 9, we provide the detailed experimental results of the comparison between SAN and state-ofthe-art normalization methods for non-stationary time series forecasting: RevIN [17], NST [25] and Dish-TS [10]. We re-implement the former two methods and Dish-TS is implemented by its official code 11 .

The table clearly shows that SAN outperforms existing approaches in most cases, except for the Weather dataset. Considering that the Weather dataset is the most stationary dataset, the results suggest that SAN can better remove the non-stationary factors in the raw data, even leading to an over-stationary issue that degrades the performance.

Besides, Dish-TS performs poorly in the benchmark. While it addresses the distribution shift between input and horizon series, it fails to optimize both the coefficient network and backbone network for overlooking the intrinsic bi-level optimization target of distribution estimation and forecasting tasks. By adopting a joint training schema, Dish-TS disturbs both networks and results in poor performance in certain cases. On the opposite, SAN benefits from the proposed two-stage schema which decouples the two tasks. This allows for proper optimization of each component and leads to improved overall performance.

B.7 SAN for Slice-based Forecasting Methods

In this section, we turn to investigate the generalizability of SAN towards recently emerged slicebased forecasting methods: PatchTST [26] and Crossformer [41]. We build forecasting models using their official codes and hyper-parameter settings (if available) 1213 . For PatchTST, we replace the RevIN[17] layer with our SAN. We report the experimental results on 5 datasets in Table 10.

The results demonstrate that SAN can improve the forecasting performance of both PatchTST and CrossFormer to some extent in most cases. The improvement for PatchTST is not significant due to two main reasons: 1) RevIN has already been introduced in the model to mitigate the impact of

non-stationary time series; 2) Since both are slice-based methods that split series into slices (nonoverlapping slices for SAN and overlapping patches for PatchTST), the parameter settings may have a greater impact on performance and how to determine proper settings require further explorations. Besides, The official code of CrossFormer provides parameter settings for Electricity, Weather, and ETTh1 datasets. For Exchange and ETTm2 datasets, we extracted common and reasonable settings. Without SAN, CrossFormer performs poorly compared to PatchTST on these latter two datasets due to unsuitable parameters. However, when enhanced with SAN under the same setting, CrossFormer achieves competitive or even superior performance. This phenomenon reveals that SAN can potentially reduce reliance on parameter settings for backbone models while also reducing costs associated with parameter adjustment in real-world forecasting applications, further validating our attempt to divide complex non-stationary forecasting task into two easier subtasks through the two-stage training schema.

C Implementation Details

C.1 Architecture of Statistic Prediction Module

The computation of MLP ( x 1 , x 2 ) in our paper can be summarized as follows:

Here, the symbol [*;*] represents the concatenate operation. We set act 1 () , act 2 () = Relu () , Relu () for standard deviation and the activate function of the mean is set to Tanh () , Identity () respectively. W 1 , W 2 , W 3 are learnable transformation matrices with hidden sizes of {512,512,1024}.

C.2 Algorithm of The Two-stage Training Schema

To apply SAN to backbone forecasting models, we propose a two-stage training schema to tackle the challenge of the bi-level optimization target. The statistics prediction module is first trained into

convergence, which is then frozen and treated as a plugin during the second stage of training the forecasting model. We provide the pseudo-code of such a procedure in Alg. 1.

D Limitations

Though SAN shows promising performance on the benchmark dataset, there are still some limitations of this method. First is that we mainly select the slicing length heuristically or search in predefined candidates and the current design cannot handle indivisible length or the multi-period characteristic of time series. Such a solution works for the experiments but lacks generality in real-world applications. Second is that SAN may lead to an over-stationary issue, leading to sub-optimal performance. Moreover, when applied to similar slice-based methods (especially overlapping-slice-based ones), determining how to adjust SAN's parameters for them is not a straightforward task. Therefore, a more flexible solution with automatic slicing length selection and normalization intensity control will be our exploring direction.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a normalization technique that works on sliced time-series with the main goal to remove non-stationary behavior of the inputs (and outputs). The paper computes mean and standard deviation of the slides inputs and then normalizes the inputs by them. Additionally, the paper proposes to estimate output normalization for y from the input mean and standard devation.\n\nSTRENGTHS:\nProblem statement is well presented.\n\nWEAKNESSES:\nThe paper proposes to normalize the slice of a time series by its mean and standard deviation of its inputs.\n\nThere is a critical issue with that: this is a **non-causal** normalization, i.e., inputs at the start of the time series are normalized by values observed later in the slice. This makes the proposed approach **unsuitable for forecasting**. It is very likely that the good empirical results presented in the paper are due to this leakage of future values of the inputs **x**.\n\nSecondly, assuming that the first problem can be fixed, there are already very similar works presented in the literature. The authors do  mention them but do not compare. Given the practical nature of the paper, these other approaches should be compared as the difference seems quite minor from a general theoretical view point. For example, DAIN (https://arxiv.org/pdf/1902.07892.pdf) is very similar and uses normalization by mean and standard deviations, mixed with neural networks. Please explain why the proposed method is novel compared to DAIN and provide experimental support that it actually improves upon DAIN.\n\nQUESTIONS:\n* Why should the proposed method perform better than DAIN, which also uses similar normalization ideas (mean and stdev).\n* What is the variable that equation 1 is summing over?\n\nLIMITATIONS:\nNo limitations were mentioned in the paper. But clearly the choice of the length of the slicing window is a problem for the proposed method, compared to other methods. The authors should either clearly state that this is a limitation or make experiments showing that the method performance is not affected by the slice length.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach called Slicing Adaptive Normalization (SAN) for non-stationary time series forecasting. The proposed method addresses the challenge of accurate predictions in the presence of non-stationarity in real-world data. It overcomes limitations in existing normalization techniques by considering the distribution discrepancy between input and horizon series and by modeling the evolving trends of statistical properties at a fine-grained temporal slice level. SAN is a model-agnostic framework that can be applied to various forecasting models, and experiments on benchmark datasets demonstrate its effectiveness.\n\nSTRENGTHS:\nAddressing non-stationarity: The paper tackles the problem of non-stationarity in time series forecasting, which is a significant challenge in real-world scenarios. By considering the distribution discrepancy between input and horizon series, SAN provides a mechanism to mitigate the impact of non-stationary nature on predictions.\n\nFine-grained normalization: SAN introduces a slice-level adaptive normalization approach, which operates on local temporal slices (sub-series) rather than the entire instance. This fine-grained normalization allows for a more accurate representation of the statistical properties within each slice, preserving distinct patterns and avoiding suboptimal improvements.\n\nEvolving trends modeling: SAN incorporates a statistics prediction module to independently model the evolving trends of statistical properties in raw time series. This module improves the estimation of future distributions, enabling adaptive denormalization and enhancing the accuracy of predictions.\n\nModel-agnostic framework: SAN is designed to be a general framework that can be applied to arbitrary forecasting models. It can serve as a plugin to existing models, making it flexible and adaptable to different forecasting scenarios.\n\nWEAKNESSES:\nThe overall method is simple and easy to understand, however, I have the following concerns or questions.\n\n[Major] Experiments. Most of my concerns are from the experimental section. (1) The baselines are not sufficient. The advanced models such as PatchTST [1], and crossformer[2] are not compared. If the method can also improve the advanced backbones, I will increase my rating. \n\n[1] Nie, Yuqi, et al. \"A Time Series is Worth 64 Words: Long-term Forecasting with Transformers.\" arXiv preprint arXiv:2211.14730 (2022).\n[2] Zhang, Yunhao, and Junchi Yan. \"Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.\" The Eleventh International Conference on Learning Representations. 2023.\n\nLimited discussion of limitations: The paper does not extensively discuss the limitations or potential challenges of the proposed method. Providing a more thorough analysis of the limitations and potential drawbacks would give a clearer understanding of the scope and applicability of SAN.\n\nQUESTIONS:\nsee Weaknesses\n\nLIMITATIONS:\nsee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNon-stationary time series forecasting is a challenging problem, and recent research has focused on using normalization techniques to address non-stationarity. However, these methods have limitations when it comes to handling the distribution discrepancy between the input and the forecasted horizon. This discrepancy arises from the assumption that all time points within the same instance share the same statistical properties. To overcome this limitation, the authors propose a method called SAN (sliced-level adaptive normalization) that empowers time series forecasting with more flexible normalization and denormalization.\nSAN addresses this issue in several ways. First, it utilizes local temporal slices instead of considering the entire global instance, allowing for more localized analysis and adaptation. Second, SAN incorporates a slight network module that independently models the evolving trends of the statistical properties of the raw time series. This approach enables the model to adapt to changing statistical characteristics over time. Finally, SAN is a general model-agnostic plugin, which means it can be integrated into various existing forecasting models to enhance their performance.\n\nSTRENGTHS:\n- The authors emphasize the importance of adopting a local perspective for adaptive normalization. By considering local temporal slices instead of the entire global instance, the model gains a more granular understanding of the data, allowing for better adaptation to each slice's specific characteristics and trends. This localized approach enables more accurate and flexible normalization, leading to improved forecasting performance.\n- The experimental results presented in the paper demonstrate promising performance in time series forecasting. By incorporating the SAN (sliced-level adaptive normalization) technique into existing forecasting models, the authors achieve notable improvements in mse/mae and predictive capabilities.\n\nWEAKNESSES:\nA. Insufficient justification for the problem definition.\n    i. The main contribution of this paper is to perform normalization based on locally slicing the time series data, instead of assuming the entire time series follows the same distribution.\n    ii. However, there is a lack of explanation regarding why global normalization should not be used. It would be beneficial to provide a detailed explanation, for example, by referencing papers on CPD (change point detection) or OOD (Out-of-Distribution) analysis within the same time series data, to clearly define the reasons for adopting a local perspective.\nB.  Citation in Section 2.1. – Detailed weakness and gentle recommendation\n    i. In section 2.1, discussing the RNN sentence, it would be better to cite papers on RNN, LSTM, and GRU to provide more comprehensive coverage (96-97).\n    ii. Furthermore, in the following sentence, while Informer is cited for pointing out issues with RNN, it might be more suitable in the context to cite papers addressing RNN’s gradient exploding or vanishing problems.\n    iii. In the subsequent sentence, when discussing self-attention and convolution for time series forecasting, it would be more appropriate to cite Scinet alongside Informer.\n    iv. Following that, the mention of Fedformer and Autoformer, which incorporate decomposition characteristics from Transformer-based architecture, could be supplemented by referencing Dlinear.\nC. Section 2.2 contents\n    i. Section 2.2 introduces existing methodologies that employ adaptive normalization for non-stationary time series forecasting.\n    ii. However, there is a lack of discussion on the limitations of previous research and the rationale behind the need for this study.\nD. Section 3 – the methodology for slicing\n    i.\tThe methodology of slicing the data, which is the most crucial aspect of this study, is not well understood.\n    ii.\tIf equally sized segments were used, how were the discrepancies in length handled at the end of the slices?\n    iii.\tAdditionally, it seems fitting to explore the application of methods such as DTW or CPD for segmentation based on similarity or probability in this study. Were they considered and applied?\nE. The data example of Figure 2\n    i.\tThe illustration in Figure 2 appears to represent variations in the magnitude of a sine wave, which does not seem suitable as an example of non-stationary data.\n    ii.\tIt is suggested to replace the illustration in Figure 2 with a more non-stationary dataset and provide a detailed diagram depicting the slicing process and the resulting changes after normalization.\nF.  Experiments\n    i.\tHow do authors perceive the increase in learning complexity and inefficiency due to SAN being trained separately from the forecasting model as a module?\n    ii.\tThe backbone models used in the experiments leverage decomposition and excel in handling stationary data. Is there a specific reason for applying SAN, designed for non-stationary learning, to these models?\n    iii.\tPatchTST also involves patching time series sub-sequences and applying them to a transformer-based architecture. Was SAN also applied to PatchTST?\n\nQUESTIONS:\nPlease address my concerns in the above weakness\n\nLIMITATIONS:\nAdaptive normalization is one of the commonly used methodologies for handling non-stationary time series data. The application of slicing in this context is a contribution of this paper. However, the motivation behind this approach is not clearly explained.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a normalization approach for predicting non-stationary timeseries. While previous work on this topic assumes that output timeseries roughly share the same statistics (mean and variance) as the input timeseries and adjusts predictions accordingly, the authors propose two adjustments. Firstly, they predict the statistics of the target timeseries instead of relying solely on those of the input. Secondly, they utilize local time statistics from slices of the timeseries data, rather than global statistics from the entire timeseries. This approach outperforms alternative methods numerically.\n\nSTRENGTHS:\n- The proposed approach, SAN, is model-agnostic and can be applied to any timeseries regression model.\n- SAN is simple but effective. The improvement of SAN is consistent in the reported benchmark.\n\nWEAKNESSES:\nThe proposed model generalizes existing methods [14, 22], but it reverts to previous approaches when slide number = 1 and an identity statistics prediction model are chosen. While the method performs well empirically, conducting ablation experiments can enhance its validation. For example, testing with a single slice and solely learning statistics prediction would confirm the necessity of slicing.\n\nMoreover, the slicing process requires tuning several important hyperparameters, such as the number of input slices (M) and the number of output slices (K). It would be beneficial if the authors could address the outcome's sensitivity to these hyperparameters.\n\nQUESTIONS:\nThe SAN results were slightly worse than the normalization alternatives on the weather dataset. In the paper, the authors attributed this to an over-stationarization issue. Can we detect this problem during training without knowing the generalization performance?\n\nAdditionally, does SAN enhance baseline results with a single slice and a learnable statistics prediction module? (See the suggested ablation experiment in the \"weakness\" section).\n\nLIMITATIONS:\nI recommend conducting ablation studies on slicing and hyperparameter choices (e.g., $M$ and $K$) as mentioned in the \"Weakness\" and \"Questions\" sections. Indeed, these are the two new adjustments in the paper compared to earlier work. While these adjustments seem to be useful, the approach's effectiveness will depend on how easily appropriate hyperparameters can be found. The authors could also comment on empirical guidelines for finding new hyperparameters for an unseen task.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}