Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting
Authors: Yong Liu
Haixu Wu
Jianmin Wang
Mingsheng Long
Venue: NeurIPS 2022 Accept (2022)
Keywords: Time series forecasting
Transformers
Deep learning

Abstract: Transformers have shown great power in time series forecasting due to their global-range modeling ability. However, their performance can degenerate terribly on non-stationary real-world data in which the joint distribution changes over time. Previous studies primarily adopt stationarization to attenuate the non-stationarity of original series for better predictability. But the stationarized series deprived of inherent non-stationarity can be less instructive for real-world bursty events forecasting. This problem, termed over-stationarization in this paper, leads Transformers to generate indistinguishable temporal attentions for different series and impedes the predictive capability of deep models. To tackle the dilemma between series predictability and model capability, we propose Non-stationary Transformers as a generic framework with two interdependent modules: Series Stationarization and De-stationary Attention. Concretely, Series Stationarization unifies the statistics of each input and converts the output with restored statistics for better predictability. To address the over-stationarization problem, De-stationary Attention is devised to recover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from raw series. Our Non-stationary Transformers framework consistently boosts mainstream Transformers by a large margin, which reduces MSE by 49.43% on Transformer, 47.34% on Informer, and 46.89% on Reformer, making them the state-of-the-art in time series forecasting. Code is available at this repository: https://github.com/thuml/Nonstationary_Transformers.

Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting
Yong Liu, Haixu Wu, Jianmin Wang, Mingsheng Long
School of Software, BNRist, Tsinghua University, China
[liuyong21, whx20] tsinghua. edu. en, (jimwang, mingggggiiinnhgghhgiinguuuu ediig cn
Abstract
Transformers have shown great power time series forecasting due to their global-range modeling ability. However, their performance degenerate ter- ribly on non-stationary real-world data in which the joint distribution changes over time. Previous studies primarily adopt stationarization to attenuate the non- stationarity of original series for better predictability. But the stationarized series deprived of inherent non-stationarity can be less instructive for real-world bursty events forecasting. This problem, termed over-stationaaization in this paper, leads Transformers to generate indistinguishable temporal attentions for different series and impedes the predictive capability of deep models. To tackle the dilemma between series predictability and model capability, we propose Non-stationary Transformers as a generic framework with two interdependent modules: Series Stationarization and De- reastationary Attention. Concretely, Series Stationarization unifies the statistics of each input and converts the output with restored statis- ties for better predictability. To address the over-stationarization problem, De- stationary Attention is devised to recover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from raw series. Our Non-stationary Transformers framework consistently boosts mainstream Transformers by large margin, which reduces MSE by 49.43% on Transformer, 47.34% on Informer, and 46.89% Reformer, making them the state-of-the-art in time series forecasting. Code available at this repository: bettp://gitthut www.auu..oomeeee comm/hhamll/oonsttiiioonyyy TrrarrrooorrryTaaaaafferrrrss
Introduction
Time series forecasting has become increasingly ubiquitous in real-world applications, such weather forecasting, energy consumption planning, and financial risk assessment. Recently, Transformers [32) have achieved progressive breakthrough on extensive areas [II] series forecasting, credited to their stacked structure and the capability of attention mechanisms, Transformers can naturally capture the temporal dependencies from deep multi-level features 337
I0 22]. Especially in time
20 35),
thereby fitting the series forecasting task perfectly.
Despite the remarkable architectural design, it is still challenging for Transformers to predict real- world time series because of the con-stationarity of data. Non-stationary time series characterized by the continuous change of statistical properties and joint distribution over time, which makes the time series less predictable 66, [4]. Besides, it is a fundamental problem to make deep models generalize well varying distribution 266 19 [5] In previous work, it is generally acknowledged to pre-process the time series by stationarization 124 27 15], which can attenuate the non-stationarity of raw time series for better predictability and provide more stable data distribution for deep models.
Equal Contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Pl, Oi
Learned Atteation
Hoy Og
Non stationury Varied 1r.
Ma. 03
Vanilla
(b) Trmmffrmer with Series Stationriizaionn
(6) Non-stationary
Transformer
Figure 1: Visualization of learned temporal attentions for different series with varied mean μ and standard deviation σ. (a) is from the vanilla Transformer [32 trained on raw series. (b) is from the Transformer trained on stationarized series, which presents similar attentions. (c) is from Non- stationary Transformers, which involves De-stationary Attention avoid over-stationarization.
However, non-stationarity is the inherent property of real-world time series and also good guidance for discovering temporal dependencies for forecasting. Experimentally, we observe that training on the stationarized series will undermine the distinction of attentions learned by Transformers. While vanilla Transformers
[32] can capture distinct temporal dependencies from different series in Figure [a). Transformers trained on the stationarized series tend to generate indistinguishable attentions in Figure Tb). This problem, named by the wver-statonnarzzatinn, will bring unexpected side-effect that makes Transformers fail to capture eventful temporal dependencies, limit the model's predictive ability, and even induce the model to generate outputs with huge non-statin con-stationarity deviation from the ground truth. Thus, how to attenuate time series non-stationarity towards better predictability and mitigate the over-stationurizution problem for model capability simultaneously is the key problem to further improve the performance of forecasting.
In this paper, we explore the effect of stationarization in time series forecasting and propose Non- stationary Transformers as general framework, which empowers Transformer [32) and its efficient variants 17 37 35)
with great predictive ability for real- world time series. The proposed framework involves two interdependent modules: Series Stationarization to increase the predictability of non- stationary series and De-stationary Attention to alleviate strrssttiioaaizatinn. Technically, Series Stationarization adopts simple but effective normalization strategy to unify the key statistics of each series without extra parameters. And De-stationary Attention approximates the attention of unstationarized data and compensates the intrinsic non-stationarity of raw series. Benefiting from above designs, Non-stationary Transformers can take advantage the great predictability of stationarized series and crucial temporal dependencies discovered from original non-stationary data. Our method achieves state-of-the-ee performance on six real-world benchmarks and can generalize to various Transformers for further improvement. The contributions lie in three folds:
We refine that the predictive capability of non-stationary series is essential in real-world forecasting. By detailed analysis, we find out that current stationarization approaches will lead the over-stationarization problem, limiting the predictive capability of Transformers.
We propose Non-stationary Transformers
generic framework, including Series Sta- tionarization to make the series more predictable and De-stationary Attention to avoid the ver-stationarizztion problem by re-incorporating the non-stationarity of original series.
-ta-stationary Transformers consistently boosts four mainstream Transformers by large margin and achieves sate--o-the--ee art performance on six real-world benchmarks.
2
Related Work
2.1 Deep Models for Time Series Forecasting
In recent years, deep models with elaboratively designed architectures have achieved great progress in time series forecasting. RNN-based models 33
36
23
29
30 proposed for application in an

---

autoregressive manner for sequence modeling, but the recurrent structure can suffer from modeling long-term dependency. Soon afterward, Transformer [32 emerges and shows great power in sequence modeling. To overcome the quadratic computation growth on sequence length, subsequent works
aim to reduce Self-Attention's complexity. Especially time series forecasting, Informer [37 extends Self-Attention with Kl-divergence criterion to select dominant queries. Reformer [1 introduces local-sensie sensitiveiiive hashing (LSH) to approximate attention by allocated similar queries. Not only improved by reduced complexity, the following models further develop delicate building blocks for time series forecasting. Autoformer [35] fuses the decomposition blocks into a canonical structure and develops Auto-Corne Cotto-Correlttin to discover series-wisc connections. Pyraformer 21 designs pyramid attention module (PAM) to capture temporal dependencies with different hierarchies. Other deep but Transformer-free free models also achieve remarkable performance. N-BEATS [25] proposes the
explicit decomposition of trend and seasonal terms with strong interpretability. N-HiTS 9 introduces hierarchical layout and multi-rate sampling for tackling time series with respective frequency bands. In this paper, different from previous works focusing on architectural design, we analyze the series forecasting task from the basic view of stationarity, which is essential property of time series 14). It is also notable that as a general framework, our proposed Non-stationary Transformers can be easily applied various Transformer-based models.
2.2 Stationarization for Time Series Forecasting
While stationarity is important to the predictability of time series 6 14, real-world series always present non-stationarity. To tackle this problem, the classical statistical method ARIMA 77 8 stationarizes the time series through differencing. As for deep models, since the distribution-varying problem accompanied by non-stationarity makes deep forecasting even more intractable, stationariza- tion methods are widely explored and always adopted as the pre-processing for deep model inputs. Adaptive Norm 24 applies z-score normalization for each series fragment by global statistics of sampled set. DAIN (27 employs a nonlinear neural network to adaptively stationarize time series with observed training distribution. RevIN [15] introduces two-stage instance normalization [3I that transforms model input and output respectively to reduce the discrepancy of each series. In contrast, find out that directly stationarizing time series will damage the model's capability of modeling specific temporal dependency. Therefore, unlike previous methods, in addition to the stationarization, Non-stationary Transformers further develops De-stationary Attention to bring the intrinsic non-stationarity of the raw series back to attention.
3 Non-stationary Transformers
As aforementioned, stationarity is an important element of time series predictability. Previous "direct stationarization" designs can attenuate non-stationarity of series for better predictability, but they obviously neglect inherent properties of real-world series, which will result in the over-stationarization problem stated in Figure I To deal with the dilemma, go beyond previous works and propose Von-stationary Transformers as a generic framework. Our model involves two complementary parts: Series Stationarization to attenuate time series non-stationarity and De-stationary Attention to re-incorporate non non-stationary information of raw series. Empowered by these designs, Non-stationary Transformers improve data predictability and maintain model capability simultaneously.
3.1 Series Stationarization
Non-stationary time series make the forecasting task intractable for deep models because it is hard for them to generalize well on series with changed statistics during inference. typically varied mean and standard deviation. The pilot work, RevIN [15] applies instance normalization with learnable affine parameters to each input and restores the statistics to the corresponding output, which makes each series follow a similar distribution. Experimentally, we find that this design also works well without learnable parameters. Thus, we propose more straightforward but effective design to wrap Transformers as the base model without extra parameters, naming by Series Stationarization. As is shown in Figure 2 it contains two corresponding operations: Normalization module at first to deal with the non-stationary series caused by varied mean and standard deviation, and De-normalization module at the end to transform the model outputs back with original statistics. Here are the details.
3

---

y
Fxx 0x
De-nomalization
Add Norm
MatMul
ty/
Softmax
Feed
Forward
Encader
Decoder
Projector
T,
Rescale
Add Norm
MatMul
A
De-stationary
Embedding
Attention
H.
Normalization
x. Uxx: Ox
non-stationary Transformers. Series Stationarization is adopted as wrapper the base model to normalize each incoming series and de-normalize the output. De-stationary Attention replaces the original Attention mechanism approximate attention learned from unstationarized series, which rescales current temporal dependency weights with learned de-stationary factors т, A.
Figure 2:
Normalization module To attenuate the non-stationarity of each input series, conduct nor- malization on the temporal dimension by a sliding window over time. For each input series [1 22: E RSC transform by translation and scaling operations and ob- tain x 22,, d's] RXC where and C denote the sequence length and variable number respectively. The Normalization module can be formulated as follows:
s Σ 1=1
1 5
Px
Dis 0x =
px)
x = 2 (2+ uxx;
where Px: E Rx means the element-wise division and @ is the element-wise product. Note that Normalization module decreases the distributional discrepancy among each input time series, making the distribution of the model input more stable.
De-normalization module As shown in Figure 2 after the base model H. predicting the future value with length-O, we adopt De-normalization transform the model output y' [V] v'o" ROxC with σx and Hx and obtain ý [01, $2..... ... 0o] as the eventual forecasting results. The De-normalization module can be formulated as follows:
y' H(x)), 0: = ax (1/ (x))
By means of the two-stage transformation, the base models will receive stationarized inputs, which follow a stable distribution and are easier to generalize. This design also makes the model equivariant to translational and scaling perturbance of time series, thereby benefiting real-world series forecasting.
3.2 De-stationary Attention
While the statistics of each time series are explicitly restored to the corresponding prediction, the non-stationarity of the original series cannot fully recovered only by De-normalization. For instance, Series Stationarization can generate the same stationarized input x' from distinct time series X2 (i.c. X2 = ox1 + 3), and the base model will get identical attention that fails to capture crucial temporal dependencies entangled with non-stationarity (Figure T.. In other words, the undermined effects caused by over-stationarization happen inside the deep model, especially in the calculation of attention. Furthermore, non-stationary time series are fragmented and normalized into several series chunks with the same mean and variance, which follow more similar distributions than the raw data before stationarization. Thus, the model is more likely to generate over-stationary and uneventful outputs, which is irreconcilable with the natural non non-stationarity of the original series.
To tackle the over-stationarization problem caused by Series Stationarization, we propose novel De-stationary Attention mechanism, which can approximate the attention that obtained without stationarization and discover the particular temporal dependencies from original non-stationary data.

---

Analysis of the plain model As mentioned above, the over-stattonariiation problem is caused by the vanishment of inherent stationariit information, which will make the base model fail to capture eventful temporal dependencies for forecasting. Therefore, we try approximate the attention learned from the original non-stationary series. We start from the formula of Self-Attention [2]
QK
At K, V) SSftmax
V,
Vdl.
where Q, K, V RSdx are length-S queries, keys and values of de-dimension respectively, and Softmax(() is conducted row by row. To simplify the analysis, we assume the embedding and feed-forward layers f to hold the linear properties and is conducted separately on each time point, that is, each query token in Q = (0-92,.... ... can be calculated as
f(x;) with respect to the input series x = [21, Z2, ... xs| Since it is a convention to conduct normalization on each time series variable to avoid certain variable that dominates the scale, we can further assume each variable of series x shares the same variance, and thus original 0x E Cx is reduced to a scalar. After Normalization module, the model receives the stationarized input x' (x 1 //x where 1 E R an all-ones vector. Based on the linear property assumption, it can be proved that (Q [/Q)/ax where ptq E Rd,x1 x
the Attention layer will receive Q' [f(x)), [(x's] is the mean of Q along the temporal dimension (See Appendix for detailed proof). And so is the corresponding transformed K', V'. Without Series Stationarization, the input of Softmaxx()) in Self-Attention should be QK" /vda, while now the attention is calculated based on Q', K':
Q'K'T
(QK - - ) - (QęK) T + I(QQ(K)1))
ox
QK
ox Q + +1(gKT (QΙK)I 1((2
(4)
Softmax
Sofumax
Vdl.
Vde
We find that Q/k E SX1 and E R, and they are repeatedly operated on each column and element of Q'K' E RRS respectively. Since Softmax(() is invariant to the same translation on the row dimension of imput, we have the following equation:
QK
(02 Q'K' +1NOK
Softmax
= Softmax
(5)
Vdz.
ddk
deduces a direct expression of the attention Softmax (QK /Vdz)) learned from raw series x. Except for the current Q', K' from stationarized series , this expression also requires the non-stationary information xx /qq; K that are eliminated by Series Stationarization.
Equation 5
De-stationary Attention To recover the original attention on non-stationary series, we attempt to bring the vanished non-stationary information back to its calculation. Based on Equation 5 the key approximate the positive scaling scalar r = of R and shifting vector A KjQ R which are defined as de-stationary factors. Since the strict linear property hardly holds for deep model, other than estimating and utilizing real factors with great effort, we try learn de-stationary factors directly from the statistics of unstationarized x, Q and K by simple but effective multi- layer perceptron layer. As we can only discover limited non non-stationary information from current Q', K', the unique and reasonable source to compensate non-stationarity is the original without being normalized. Thus, as a direct deep learning implementation of Equation we apply multi- layer perceptron the projector to leam de de-stationary factors т, A from the statistics fxx 0x of unstationarized x individually. And the De-stationary Attention is calculated as follows:
log MLP((x, x), A -MLP(μx, x),
T Q'K'
+ +IAT
(6)
Attn Q'KK', V A) Softmax
V
Vdz
where the de-stationary factors 7 and A are shared by De-stationary Attention of all layers (Figure De-stationary Attention mechanism learns the temporal dependencies from both stationarized series
Function f has the linear property if it satisties that f(ax + by) = af af(x) + by(.), where b are scalar constants and x, y vector variables.
5

---

Q', K' and non non-stationary series x, Px. xx, and multiplies by the stationarized values V Therefore, can benefit from the predictability of stationarized series and maintain the inherent temporal dependencies of raw series simultaneously.
Overall architecture Following the prior use of Transformers 35| in time series forecasting, we adopt the standard -eeederr structure (Figure 2) where the encoder is to extract information from past observations, and the decoder is to aggregate past information and refine the prediction from simple initialization. The canonical Non-stationary Transformer is wrapped by Series Stationarization to both the input and output of vanilla Transformer [32], and replacing the Self-Attention by our proposed De-stationary Attention, which can boost the non-stationary series predictive capability of the base model. For the Transformer variants [17 37 (3), we transform the terms inside Softmax(() with the de-stationary factors т, to re-integrate the non-stationary information (See Appendix for the implementation details).
Experiments
We conduct extensive experiments to evaluate the performance of Non-stationary Transformers on six real-world time series forecasting benchmarks and further validate the generality of the proposed framework on various mainstream Transformer variants.
Datasets Here are the descriptions of the datasets: (1) Electricity [ records the hourly electricity consumption of 321 clients from 2012 to 2014. (2) ETT [37] contains the time series of oil de- stationary factors and power load collected by electricity transformers from July 2016 to July 2018. ETTml /ETTm2 are recorded every 15 minutes, and ETThI/ETTh2 are recorded every hour. (3) Exchange 18 collects the panel data of daily exchange rates from 8 countries from 1990 to 2016. (4) III collects the ratio of influenza-like illness patients versus the total patients one week, which is reported weekly by Centers for Disease Control and Prevention of the United States from 2002 and 2021. (5) Traffic [2] contains hourly road occupancy rates measured by 862 sensors on San Francisco Bay area freeways from January 2015 to December 2016. (6) Weather 4 includes meteorological time series with 21 weather indicators collected every 10 minutes from the Weather Station of the Max Planck Biogeochemistry Institute in 2020.
Especially, in this paper, we adopt the Augmented Dick-Fuller (ADF) test statistic [3] as the metric to quantitatively measure the degree of stationarity. A smaller ADF test statistic indicates higher degree of stationarity, which means the distribution is more stable. Table summarizes the overall statistics of the datasets and lists them in ascending order by degree of stationarity. We follow the standard protocol that divides each dataset into the training, validation, and testing subsets according to the chronological order. The split ratio is 6:2:2 for the ETT dataset and 7:1:2 for others.
Table 1: Summary of datasets. Smaller ADF test statistic indicates more stationary dataset.
Dataset
Variable Number
Sampling Frequency
Total Observations
ADF Test Statistic
Exchange
Day
7,588
-1.889
IIII ETTm2
7
Week
966
-5.406
7
15 Minutes
69,680
-6.225
Electricity
321
1 Hour
26,304
-8.483
Traffic
862
Hour
17.544
-15.046
Weather
21
10 Minutes
52,695
-26.661
Baselines We evaluate the vanilla Transformer b2 equipped by the Non-stationary Transform-
ers framework in both multivariate and univariate settings to demonstrate its effectiveness. For multivariate forecasting, we include six tfet-----thh-rtttt deep forecasting models: Autoformer [35).
Pyraformer [21), Informer (37), LogTrans 20). Reformer [17] and LSTNet [18 For univariate
forecasting, include seven competitive baselines: N-HITS 9. N-BEATS 25). Autoformer [35).
Pyraformer [21). Informer 33), Reformer [17 and ARIMA 77]. In addition, we adopt the pro-
posed framework on both the canonical and efficient variants of Transformers: Transformer (32),
Informer 377. Reformer 17 and Autoformer [35] to validate the generality of our framework.
Implementation details All the experiments are implemented with PyTorch 28 and conducted on single NVIDIA TITAN V 12GB GPU. Each model is trained by ADAM 16 using L2 loss with

---

the initial learning rate of 10 and batch size of 32. Each Transformer- based model contains two encoder layers and one decoder layer. Considering the efficiency of hyperparameters search, we use two-layer perceptron projector with the hidden dimension varying in (64, 128, 256} in De-stationary Attention. We repeat each experiment three times with different random seeds and report the test MSE/MAE under different prediction lengths, and the standard deviations are also provided in Appendix. A lower MSE/MAE indicates better performance.
4.1 Main Results
Forecasting results As for multivariate forecasting results, the vanilla Transformer equipped with our framework consistently achieves state-of tart performance in all benchmarks and prediction lengths (Table Notably, statiinat Transformer outperforms other deep models impressively on datasets characterized by high non-stationarity: under the prediction length of 336, we achieve 17% MSE reduction (0.509 - 0.421) on Exchange and 25% % (2.669 + 2.010) on ILI compared to previous state-of-the-art results, which indicates that the potential of deep model is still constrained on -ta-stationary data. We also list the univariate results of two typical datasets with different stationarity in Table 3 Non-stationary Transformer still realizes remarkable forecasting performance.
Table 2: Forecasting results comparison under different prediction lengths 0 E {96, 192, 336, 720). The input sequence length is set to 36 for ILI and 96 for the others. Additional results (ETTml, ETTh1, ETTh2) can be found in Appendix.
Models
Ours
Autoformer
35
Pyraformer
11 Informer
37
LogTrans
20
Reformer
LSTNet
18
Metric
MSE MAE MSE MAE
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
96 0.111 0.237 0.197 0.323 0.852 0.780 0.847
0.752
0.968
0.812
1.065
0.829
1.551
1.058
192
0.219 0.335 0.300 0.369
0.993 0.858 1.204
0.895
1.040
0.851
1.188
0.906
1.477
1.028
336
0.421
0.476 0.509
0.524
1,240
0.958
1.672
1.036
1.659
1.081
1.357
0.976
1.507
1.031
720
1.092 0.769 1.447
0.941
1,711
1.093
2.478
1.310
1.941
1.127
1,510
1.016
2.285
1.243
24
2.294
0.945 3.483
1,287
5,800
1.693
5.764
1.677
4,480
1.444
4.400
1.382
6.026
1.770
36
1.825
0.848 3.103
1.148
6.043
1.733
4.755
1.467
4,799
1.467
4.783
1.448
5.340
1.668
48
2.010
0.900 2.669
1.085
6.213
1.763
4.763
1.469
4,800
1.468
4.832
1.465
6.080
1.787
2.178
0.963 2.770
1.125
6.531
1.814
5.264
1.564
5.278
1,560
4.882
1.483
5,548
1.720
ET7m2
0.192
0.274 0.255
0.339
0.409
0.488
0.365
0,453
0.768
0.642
0.658
0.619
3.142
1.365
192
0.280
0.339 0.281 0.340
0.673
0.641 0.533
0.563
0.989
0.757
1.078
0.827
3.154
1.369
336
0.334
0.361 0.339
0.372
1,210
0.846 1.363
0.887
1,334
0,872
1.549
0.972
3.160
1,369
720
0.417 0.413 0.422
0.419
4.044
1.526 3.379
1.388
3.048
1.328
2.631
1.242
3.171
1.368
Electricity
96
0.169 0.273 0.201
0.317
0.498
0.299
0.274
0.368
0.258
0.357
0.312
0,402
0,680
0,645
192
0.182
0.286 0.222
0.334
0.828
0.312 0.296
0.386
0.266
0.368
0.348
0.433
0.725
0.676
336
0.200
0.304 0.231
0.338
1.476
0.326 0.300
0.394
0.280
0.380
0.350
0.433
0.828
0.727
720
0.222 0.321 0.254
0.361
4.090
0.372 0.373
0.439
0.283
0.376
0.340
0.420
0.957
0.811
0.612 0.338 0.613 0.388
0.684
0.393 0.719
0.391
0.684
0.384
0.732
0.423
1.107
0.685
Traffic
192
0.613
0.340 0.616 0,382
0.692
0.394 0.696
0.379
0.685
0,390
0.733
0,420
1.157
0.706
336
0.618
0.328 0.622 0.337
0.699 0.396 0.777
0.420
0,733
0,408
0.742
0,420
1.216
0,730
720
0.653
0.355 0.660 0.408
0.712 0.404 0.864
0.472
0.717
0.396
0.755
0.423
1.481
0.805
Weather
96
0.173 0.223 0.266 0.336
0.354
0.392 0.300
0.384
0.458
0.490
0.689
0.596
0.594
0.587
192
0.245
0.285 0.307 0.367 0.673 0.597 0.598 0.544
0.658
0.589
0.752
0.638
0.560 0.565
336
0.321
0.338 0.359 0.395 0.634 0.592 0.578
0.523
0.797
0.652
0.639
0.596
0.597
0.587
720
0.414
0.410 0.419 0.428 0.942 0.723 1.059 0.741
0.869
0.675
1.130
0.792
0.618 0.599
Framework generality We apply our framework to four mainstream Transformers and report the performance promotion of each model (Table Our method consistently improves the forecasting ability of different models. Overall, it achieves averaged 49.43% % promotion on Transformer, 47.34 % on Informer, 46.89 on Reformer and 10.57% on Autoformer, making each of them surpass previous state-of tfte--of-the-ar.. Compared to native blocks of the models, there is hardly any parameter and computation increase by applying our framework (See Appendix for details), and thereby their computational complexities can be preserved. It validates that Non-stationary Transformer is an effective and lightweight framework that can be widely applied to Transformer-based models and enhances their non -oa--sttionary predictability to achieve state-of-the-art performance.
7

---

Table 3: Univariate results under different prediction lengths O E {96, 192, 336, 720} on two typical datasets with strong non-stationary. The imput sequence length is set to 96.
Models
Ours
N-IITS
N-BEATS
25
Autoformer
3
Pyrafommrr
21
Informer
37
Reformer
M7
ARIMA
6
Metric
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
96
0.104 0.235 0.114 0.248 0.156 0.299 0.241
0.387
0.290
0.439
0.591
0.615
1.327
0.944
0.112 0.245
192
0.230 0.375 0,250 0.387 0.669
0.665 0.273
0,403
0.594
0,644
1.183
0.912
1.258
0.924
0.304 0,404
336
0.432
0.509 0.434 0.516 0.611 0.605
0.508
0.539
0,962
0.824
1.367
0.984
2.179
1.296
0.736 0.598
720
0.782
0.682 1.061 0.773 1.111
0.860
0.991
0.768
1.285
0.958
1.872
1.072
1.280
0.953
1.871 0.935
96
10.099
0.193 0.092 0.232 0.082 0.219
0.065
0.189
0.074
0.208
0.088
0.225
0.131
0.288
0.211 0.362
F11m2
192
0.109
0.249 0.128 0.276 0.120 0.268
0.118 0.256
0.116
0.252 0.132
0.283
0.186
0.354
0.261 0.406
336
0.139
0.286 0.165 0.314 0.226 0.370 0.154 0,305
0.143 0.295 0.180 0.336
0.220
0.381
0.317 0.448
720
0.180 0.331 0.243 0.397 0,188 0.338 0.182 0.335 0.197 0.338 0.300 0.435 0.267
0.430
0.366 0.487
Table 4: Performance promotion by applying the proposed framework to Transformer and its variants. We report the averaged MSE/MAE of all prediction lengths (stated in Table 2 and the relative MSE reduction ratios (Promotion) by our framework. Full results (under all prediction lengths and promotion on ETSformer
34 FEDformer
(38 can be found in Appendix.
Dataset
Exchange
ILI
ETTm2
Electricity
Traffic
Weather
Model
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Transformer
1.425
0.915
4.864
1,460
1.501
0.869
0.277
0.372
0.665
0.363
0.657
0.573
Ours
0.457
0.449
2.077
0.914
0.306
0.347
0.193
0.296
0.628
0.345
0.288
0.314
Promotion
67.93%
57.30%
79.61 %
30.32%
5.56%
56.16%
Informer
1.550
0.998
5.137
1.544
1,410
0.823
0.311
0.397
0.764
0.416
0.634
0.548
Ours
0.496
0.460
2.125
0.928
0.460
0.434
0.226
0.330
0.719
0.409
0.275
0.302
Promotion
68.00%
58.63%
67.38%
27.33%
5.89%
56.78%
Reformer
1.280
0.932
4.724
1.443
1,479
0.915
0.338
0.429
0.741
0.423
0.803
0.656
Ours
0.462
0.468
2.865
1.065
0.493
0.441
0.206
0.308
0.682
0.372
0.286
0.308
Promotion
63.91%
39.35%
66.67%
39.05%
7.96%
64.38%
Autoformer
0.613
0.539
3.006
1.161
0.324
0.368
0.227
0.338
0.628
0.379
0.338
0.382
Ours
0.487
0.491
2.545
1.039
0.305
0.345
0.216
0.315
0.619
0.364
0.286
0.310
Promotion
20.55%
15.34%
5.86%
4.85%
1.43%
15.38%
4.2
Ablation Study
Quality evaluation To explore the role of each module in our proposed framework, we compare the prediction results on ETTm2 obtained by three models: vanilla Transformer, Transformer with only Series Stationarization, and our Non-stationary Transformer. In Figure we find out that the two modules strengthen the non-stationary forecasting ability of Transformer from different perspectives. Series Stationarization focuses on the alignment of statistical properties among each series input that benefits Transformer a lot to generalize on out-of out-of-distribune distriibutioon data. However, as is shown in Figure 3 3b), the over- stetsnaaton circumstance for training makes the deep model more likely to output uneventful series with significant high stationarity and neglect the nature of non non-stationary real-world data. With the aid of De-stationary Attention, the model gives concern back the inherent non-stationarity of real-world time series. It is beneficial for an accurate prediction of the detailed series variation, which is vital in real-world time series forecasting.
(a) Vanilla Transformer
(b) Transformer Series Stationarization
(e) Non-stationary Transformer
4.44
Greand Truis
Gooard Touh
Gooand Tadb
Predition
Prefecies
Prodeiie
-0.8
Volaa
Wile
-1.2
-1.6
-3.6
.1.6
100
200
Time
Time
Time
Figure 3: Visualization ETTm2 predictions given by different models.
8

---

Table 5: Forecasting results obtained by applying different methods to Transformer and Reformer. We report the averaged MSE/MAE of all prediction lengths (stated in Table 2 for comparison. Complete results can be found in Appendix.
Base Models
Transformer
Reformer
Series Stationarization
+ Series Stationarization
Methods
RevIN
15
Ours
RevIN
15
Ours
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
0.567
0.487
0.569
0,488
0.461
0.454
0.469
0.472
0,470
0.473
0.462
0.468
ILI
2.205
0.934
2.206
0.934
2.077
0.914
3.024
1.096
3.023
1.096
2.865
1.065
ETTm2
0,460
0.416
0,461
0,416
0.306
0.347
0.542
0.459
0.537
0.459
0.493
0.441
Electricity
0.197
0,298
0.197
0.298
0.193
0.296
0.208
0.309
0.207
0.309
0.206
0.308
Traffic
0.643
0.352
0.641
0,352
0.628
0.345
0.687
0.378
0.691
0.380
0.682
0.372
Weather
0.301
0.316 0.304
0.317
0.288
0.314
0.291
0.309
0.292
0.309
0.286
0.308
Quantitative performance In addition to the above case study, we also provide quantitative forecasting performance comparison with stationarization methods: deep method RevIN [15] and Series Stationarization (Section 3.1). As is shown Table [5] the forecasting results assisted by RevIN and Series Stationarization are basically the same, which indicates that the parameter-free version of normalization in our framework performs sufficiently to stationarize time series. Besides, the proposed De-stationary Attention in Non-stationary Transformers further boosts the performance and achieves the best all six benchmarks. The MSE reduction brought by De-stationary Attention becomes significant, especially when the dataset is highly non-stationary (Exchange: 0.569 - 0,461, ETTm2: 0.461 - 0.306). The comparison reveals that simply stationarizing time series still limits the predictive capability of Transformers, and the complementary mechanisms in Non-stationary Transformers can properly release the models' potential for non-stationary series forecasting.
4.3 Model Analysis
Over-stationarizztion problem To verify the over-stationarization problem from statistical view, we train Transformers with the aforementioned methods respectively, arrange all predicted time series in chronological order and compare the degree of stationarity with the ground truth (Figure While models solely equipped with stationarization methods tend to output series with unexpected high degree of stationarity, the results assisted by De-stationary Attention are close the actual value (relative stationarity E [97%, 103%)). Besides, as the degree of series stationarity increases, the over- stationarization problem becomes more significant. The huge discrepancy of the degree of stationarity can account for the inferior performance of Transformer with only stationarization. And it also demonstrates that De-stationary Attention as an internal renovation alleviates over-stationarization
Relative Stationarity %
Our
115
Transformer RevIN
Transformer Series Suationarization
110
105
100
Ground-truth
95
Weather
Traffic
Electricity
ETTm2
ILI
Exchange
Figure 4: Relative stationarity is calculated as the ratio of ADF test statistics between the model predictions and ground truth. From left to right, the dataset is increasingly non-stationary. While models equipped with only stationarization tend to output highly stationary series, method gives predictions with stationarity closer to ground truth.

---

Exploring of Non-stationary Information Re-ineorporation It is notable that by specifying over-stationarization as less distinguishable attention, we narrow down our design space into the attention calculation mechanism. To explore other approaches to retrieve non-stationary information, we conduct experiments by re-incorporating the 1r and into feed-forward layers (DeFF), which the left part of the Transformer architecture. In detail, we feed learned ju and a into each feed-forward layer iteratively. As is shown in T re-incorporating non non-stationarity is necessary only when the inputs are stationarized (Stationary), which is beneficial for forecasting but will lead to stationarity discrepancy of model outputs. And our proposed design (Stat + DeAttn) makes further promotion and achieves the best in most cases (77%). In addition to the theoretical analysis, experimental results further validate the effectiveness of our design in re- veeicorpporaiing non- non-stationarity on attention.
Table 6: Ablation of framework design. Baseline means vanilla Transformer, Stationary means adding Series Stationarization, DeFF means re- re-incorporating non-station non-stationarity on ferd-forward layers, DeAtm means re- incorporating by De-stationary Attention, Stat + DeFF means adding Series Stationarization and re- re-incorporating on feed-forward layers. Stat + DeAttn means our proposed framework.
Models
Baseline
Stationary
DeFF
DeAttn
Stat + DeFF
Stat + DeAttn
Metric
MSE
MAE
MSE
MAE
MSR
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
96
0.567
0.591
0.136
0.258
0.784
0.696
0.611
0.613
0.116
0.243
0.111
0.237
192
1.150
0,825
0.239
0.348
1.162
0.866
1.202
0.840
0.280
0.383
0.219
0.335
336
1.792
1.084
0,425
0.479
1.346
0.963
1,516
0.981
0.371
0.452
0.421
0.476
720
2.191
1.159
1.475
0.865
2.042
1.163
2.894
1.377
0.934
0.704
1.092
0.769
24
4.748
1.430
2.573
0.980
4,850
1.445
4.734
1.424
2.404
0.985
2.294
0.945
ILI
36
4.671
1.430
1.955
0.870
4.848
1.452
4.927
1.482
2.585
0.983
1.825
0.848
48
4,994
1.482
2.057
0.902
4,903
1.466
4,996
1.483
2.496
0.991
2.010
0.900
60
5.041
1.499
2.238
0.982
5.196
1.524
5.184
1.519
2.667
1.059
2.178
0.963
96
0.572
0.552
0.253
0.311
0.767
0.635
0.304
0.406
0.275
0.329
0.192
0.274
ETTm2
192
1.161
0.793
0.453
0.404
0.960
0.717
0.820
0.652
0.406
0.403
0.280
0.339
336
1,209
0.842
0.546
0.461
1.159
0.811
1.406
0.883
0.502
0.465
0.334
0.361
720
3.061
1.289
0.593
0.489
3.187
1.308
2.858
1.108
0.694
0.575
0.417
0.413
Electricity
96
0.260
0.358
0.171
0.275
0.260
0.356
0.253
0.351
0.170
0.274
0.169
0.273
192
0.266
0,367
0.192
0.296
0.264
0.365
0.257
0.358
0.188
0.293
0.182
0.286
336
0.280
0.375
0.208
0.306
0.277
0.374
0.270
0.365
0.206
0.309
0.200
0.304
720
0.302
0.386
0.216
0.315
0.299
0.384
0.295
0.380
0.223
0.323
0.222
0.321
96
0.647
0.357
0.614
0.337
0,646
0.353
0,650
0.358
0.605
0.333
0,612
0.338
Traffic
192
0.649
0.356
0.637
0.351
0.645
0.352
0,655
0.358
0,617
0.342
0.613
0.340
336
0.667
0.364
0.653
0.359
0.672
0.360
0.656
0.355
0.635
0.349
0.618
0.328
720
0.697
0.376
0.661
0.360
0.695
0.376
0.681
0.366
0.649
0.351
0.653
0.355
Weather
0.395
0.427
0.175
0.225
0,417
0.445
0.296
0.364
0.178
0.226
0.173
0.223
192
0.619
0.560
0.273
0.297
0.699
0.604
0.480
0.464
0.256
0.295
0.245
0.285
336
0.689
0.594
0.333
0.325
0.773
0.620
0.581
0.519
0.338
0.351
0.321
0.338
720
0.926
0.710
0.436
0.420
1.008
0.718
0.795
0.642
0.417
0.412
0.414
0.410
5
Conclusion
This paper addresses time series forecasting from the view of stationarity. Unlike previous studies that simply attenuate non-stationarity leading to sversstationarization,, we propose an efficient way to increase series stationarity and renovate the internal mechanism re-incorporate non-stationary information, thus boosting data predictability and model predictive capability simultancously. Ex- perimentally, our method shows great generality and performance on six real-world benchmarks. And detailed derivations and ablations are provided to testify the effectiveness of each component in our proposed Non-stationary Transformers framework. In the future, we will explore a more model-agnostic solution for the over-stationaaiiztion problem.
Acknowledgments
This work was supported by the National Key Research and Development Plan (2021 YFC3009C)))) National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program 120010000682004 and BNRist Innovation Fund (BNR202 RC01002).
10

---

References
Illness Dataset. httpg///.......aaaaa///////////r
Traffic Dataset. http://pems.dott
UCI Electricity Load Time Series Dataset.
[4] Weather Dataset. httpp:///wwt bww-bggeanee mpg. de/netter/
[5] Kartik Ahuja, Ethan Caballero, Dinghuai Zhang. Jean Christophe Gagnon- Audet, Yoshua Bengio, loannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of disttribbtttonn generalization. NeurIPS, 2021.
O. Anderson and M. Kendall. Time-series. 2nd edn. J. R. Stat. Soc. (Series D). 1976.
G. E. Box and Gwilym M. Jenkins. Time series analysis, forecasting and control. 1970.
George EP Box and Gwilym M Jenkins, Some advances in forecasting and control. J. R. Stat. Soc. (Series-C), 1968.
19] Cristian Challu, Kin G Olivares, Boris Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint 12Xiv:2201.12886 2022.
[10] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. NeurlPS, 2021.
[11] Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HILT, 2019.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kalesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. ICLR, 2021.
[13] Graham Elliott, Thomas J. Rothenberg, and James H. Stock, Efficient tests for an autoregressive unit root. Econometrica, 1996.
[14] Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
[15] Taesung Kim, Jinhee Kim, Yunwon Tac, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In ICLR, 2022.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[17] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
[18] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. SIGIR, 2018
[19] Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017.
[20] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forccasting, In NeurIPS, 2019.
[21] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. ICLR, 2021.
[22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[23] Danielle C Maddix. Yuyang Wang, and Alex Smola. Deep factors with gaussian processes for forecasting. arXiv preprint 182.188800000 2018.

---

[24] Eduardo Ogasawara. Leonardo C. Martinez, Daniel de Oliveira, Geraldo Zimbrão, Gisele L. Pappa, and Marta Mattoso. Adaptive normalization: A novel data normalization approach for non-stationary time series. IJMN, 2010.
[25] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. ICLR, 2019.
[26] Sinno Jialin Pan and Qiang Yang. A survey transfer learning. TKDE, 2009.
[27] Nikolaos Passalis, Anastasios Tefas, Juho Kanniainen, Moncef Gabbouj, and Alexandros losifidis. Deep adaptive input normalization for time series forecasting. TNNLS, 2019.
[28] Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen,
Z. Lin, N. Gimelshein, L.. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style. high perfformremme deep learning library. In NeurlPS, 2019.
[29] Syama Sundar Rangapuram, Matthias Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In NeurlPS, 2018.
[30] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. Int. J. Forecast., 2020.
[31] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint 1xXx::607700022 2016.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrIPS, 2017.
[33] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. NeurlPS, 2017.
[34] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint 1406 2022.
[35] Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting, In NewrlPS, 2021.
[36] Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Long-term forecasting using tensor-train mnss. arXiv preprint a7Xiv: 1.0007,, 2017.
[37] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
[38] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICMI, 2022.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes] See Section
(b) Did you describe the limitations your work? [Yes] Section of the Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section of the
Appendix.
(d) Have you read the cthics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all thcoretical results? [Yes] See Section 1 of the
Appendix.
(b) Did you include complete proofs of theoretical results? [Yes] See Section I of the Appendix.
3. If you experiments...
(a) Did you include code, data, and instructions needed to reproduce main experimental results (cither the supplemental material or as a URL)? Yes] We provide the data link and code in https: htt///githhbbbb com/ /ttmm//ull/////m Nonstationaaaoao oonroonrryyy.o Transformert
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)?
Yes]] See Section 5.1 of the Appendix.
(c) Did you report error bars (c.g.. with respect to the random seed after running experiments multiple times)? [Yes] See Table 4 of the Appendix.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] The data source is described in Scction
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes] We provide the code in pestpa://githan wtw/toomtaateeeece tttml/tumll////// wwwaetttatioaaa_arnaanaaaaee
(d) Did you discuss whether and how consent obtained from people whose data you're us-
ing/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "khBkzzwZWh", "reviewer": "Reviewer_PhZA", "summary": "The authors note that the stationarisation of time series removes information from the time series that can be used to aid in prediction. They call this problem the \"over stationarisation of time series\". Given that we want stationarity to improve model performance, but do not want to lose the information from the stationarisation transform, motivates the \"Non-stationary transformer\" introduced in the paper. \n\nThe model consists of a stationarisation component, and an additional component to reincorporate the stationarised information back into the time series. \n\nThey key component introduced is the De-stationary attention module. This transforms the dat a", "questions": "Why does the De-stationary attention module not just operate on the existing, non-stationarised features directly? ", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is certainly important to reincorporate the information removed by the stationarity transform into the prediction model. The proposed method introduces some method of doing this and demonstrates that it improves performance accross a wide range of benchmarks.\n\nStationarisation losing information and thus potential predictive power is intuitively obvious. The novelty here (to me at least) is the destationary attention mechanism. This is another attention block that has some scaling taking into account non-stationary information. \n\nIt's not clear to me why it has to be done in this way. Why can I not just incorporate mu sigma into the MLP applied to the output of the self attention? My biggest issue with the paper is the motivation behind it. Stationarity loses information - fine. Adding information back in some way should improve performance - fine. This is known and not novel. We add some additional transformer to the existing transformer that accounts for some of this information - fine, but I'm not sure why I should be so interested in this architecture? There are lots of ways we could add information back in, why should I choose this one?\n\nThe experiments do not compare other mechanisms that incorporate such information. I havent been given a reason to be particularly excited about this mechanism over some other method of sticking in mu and sigma. Is there some motivation behind this mechanism that I am missing?\n\nIt would be nice if you had compared with non transformer benchmarks, e.g. LSSL/GRU/\n\n\"We refine that the predictive capability of non-stationary series is essential in real-world forecasting. By detailed analysis, we find out that current stationarization approaches will lead to the over-stationarization problem, limiting the predictive capability of Transformers.\" - From what I can tell, this 'refinement' is a single image showing different attention weights in different scenarios. I dont think there is enough done on this point in this paper to define this as a contribution. \n\n\nThe writing is not great, I found many parts quite difficult to read. A couple of examples of difficult-to-parse sentences:\n\n\"However, non-stationarity is the constitutional property of real-world time series that can be entangled with essential temporal dependencies for forecasting\"\n\n\"Considering the deep model scenario under the real case, de-stationary factors should be disentangled from the statistics of unstationarized x, Q and K.\"\n\nIn general I think a lot of the wording is overcomplicated and could be significantly simplified. \n\nYou say transformers perform well \"credited to their stacked structure\". Most DL models for time series are stacked in some way. I dont really see how this is a plus for transformers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cRsEi9uUaEf", "reviewer": "Reviewer_ZUaG", "summary": "This paper focuses on forecasting and introduces a new method for scaling attention.\n* The input raw series is chunk-normalized to yield the input of the backbone transformer model\n* the chunk-wise scaling parameters are processed through MLPs to scale the rows of the attention matrix.\n\nThis procedure is showed to consistently improve performance of transformer-based forecasting models.  ", "questions": "* You should not use the words \"stationary\", \"stationarity\", \"de-stationary\" etc unless this is actually what you mean. \n* From a broad perspective, I see your proposed method as applying some chunk-wise transform, say x', \\theta = F(x), that returns some modified version for a chunk along with some parameters, and then using some MLP1(\\theta) and MLP2(\\theta) for scaling the attention matrix. At the output, you apply F^{-1}(out, \\theta). I genuinely think this is a great idea, and I think it is ok to motivate this by the following story:\n   - Having all chunks have the same first two moments (mean and variance) is a first step in enforcing them to have the same distribution and this is what we do with our series normalization. In this case, our chunkwise transform is a simple normalization, and \\theta={\\mu, \\sigma}.\n   - Imagining that we do applied just some linear transform on the input, here's what the attention matrix would be like. So we propose some scaling scheme for the attention inspired by this, that looks like.....\n- at the output, we can transform back.\n   The advantage of this  way of writing things is that you leave room for further research (notably more sophisticated chunkwise methods) and you don't wrongly pretend that normalizing means enforcing stationarity... because it's not the case.\n\nI think that this paper really brings great things on the table, but that it should undergo some strong modifications to be ok. The good news in my view is that I am not suggesting any change to the actual model, which is good, but just to the story that is told regarding the theory, so that I believe this is feasible (although it requires some heavy work).\n\n\nBelow, please find a list of comments on the go.\nTitle\n* I personally don't like all these \"rethinking *\" titles, that sound very pretentious in my view, somehow suggesting that things were not correctly thought about before and that everything will change from now on. Please note that your view of stationarity is not rigorous, adding a bit to my frustration.\n\nIntroduction:\n* \"From the aspect of data\": awkward. Furthermore, this statement about predictability is not very well motivated in my opinion.\n* \"From the view of deep learning\": why would that be limited to deep learning ?\n* \"a hot topic\": this is an exagerated statement, provided you give 3 references, among which one is almost 15 years old.\n* Figure 1 is not clear enough in my opinion. I don't really understand what the input data to the transformer is: are these the I, II, III chunks ? What is that the second column ? Just zooms for the data ? And finally, what are these \"attention matrices\" ? what's the number of rows, columns. I suspect some interpolation for these images ? Are you using some `interpolation='nearest'` for your plot ?\n\nRelated work\n* \"which is the essential property\": an essential\n* \"is the key to predictability\": a key. I am not sure this statement is supported by such generic references.\n* I am a bit annoyed by the fact that you constantly talk of stationarity and non-stationarity, but you don't define the terms.\n\n\nNon-stationary Transformers\n* \"the key to time series predictability\": again, too strong. \"an important element\" ?\n* The reference to figure 1 is not clear. As mentioned already I don't really see what Figure 1 brings on the table exactly, maybe just because it is poorly explained and I don't understand it.\n* \"To deal with the dilemna [...] simultaneously\": you wrote that already\n* although this is classical to use \"T\" for transposition, please note that using \\top instead gives a better rendering.\n\n* I really must protest against your statement that a simple normalization is enough to \"transform it into standard Gaussian distribution\". I don't see any connection between having x be centered and unit variance and x being N(0,1). Of course, the first two moments are matching, but this is not at all sufficient to make x Gaussian. You should remove this statement and simply mention that you are centering and normalizing it.\n* I understand the use of \"Hadamard product\", but the fact is that it can be confusing to some readers. If you like, you could be using a statement like \"\\frac{a}{b} and a\\circ b are element-wise division and product, respectively\".\n* I don't understand how these \\mu_x and \\sigma_x vectors are computed. From a single S-dimensional chunk, how do you end up computing a S-dimensional mean and a S-dimensional variance statistic ? Checking out the followup, it turns out you are simply normalizing each chunk, so that \\mu_x and \\sigma_x are just scalars. You should not be introducing this useless notation.\n* Note that Normalization module eliminates the [...] statistics\": I don't see at all how this statement is supported. The only thing you enforce is that every chunk is normalized. You may not at all write that this leads to all of them having the same \"statistics\". As you know, you can't seriously write that mean and standard deviation are the only statistics for a time series... What about all the auto-correlation structure, higher-order statistics, etc. I am fine with your normalization, but please don't write that it results in chunks having the same distribution ! You are simply matching the first two moments.\n\n* You didn't explain how the \"sliding window\" is achieved: is there some overlap between your chunks ? Or do you obtain your chunks with a simple folding operation (window size=hop size)\n* I don't see the point in using uppercase \"Series Stationarization\" if you're not going to use the (unfortunate) acronym SS. Maybe just use lower case then.\n* \"the base models will receive stationarized inputs\": again, I feel uncomfortable with accepting the statement that a series of normalized chunks is a stationary time series. Just imagining a dataset with outliers, your procedure will obviously change the content of a chunk containing the outlier in a undesired way, so that future research might focus on some other transformation than mean-variance normalization, like quantiles or such. Don't misread me: the method is nice, but in my view you are claiming something that is not exactly there.\n* \"for an instance\": for instance\n* again, the reference to figure 1 is unclear to me, because I didn't get what you want exactly to be conveying with this figure\n* \"following the same distribution\": it is not because they are both centered and unit variance that two vectors have the same distribution !\n* \"to tackle the trivial attention\": awkward. to tackle this over-stationarization problem ?\n* \"To simplify the analysis [...] input series\": this assumption is completely unreasonable and it is useless in my view to rely on it seriously for anything. This said, I understand that assuming f to be linear allows you to derive the *general form* of your \"de-stationary attention\", as what would be happening in the linear case. The fact is that you are actually not using this assumption in the followup since your attention module don't use these \\mu and \\sigma as such, but instead through some MLPs. I think that the story would be more readable if you introduced it as you do in the linear case, but then simply propose this attention scaling procedure ase a way to reinject the normalization parameters back into attention in a way that generalizes the linear case. This would read much better to me.\n* \"Since it is a convention to conduct [...] is reduced to a scalar\". This sentence makes no sense to me. Just take \\mu_x and \\sigma_x as scalars right from the start and be good with this.\n* As stated above, it is not necessary to write that your \\tau and \\Delta are there to \"approximate \\sigma_x^2 and K\\mu_q\" (you know them already !) but rather that you introduce them as scaling factors in analogy to what happens in the linear case. Don't forget that your derivations assume a linearity that is not there.\n\n\nExperiments\n* What do you call \"stationarity\" and \"relative stationarity\" for the \"over-stationarization problem\" section ? This is not defined.\n\n\nConclusion\n* \"The impressive generality and performance of the proposed framework\": please change this pretentious statement.\n", "limitations": "The authors are a bit pretentious at times, but other than that it is all right", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The proposed procedure is quite simple to implement\n* In my view, introducing this attention scaling is actually a nice and elegant thing to do.\n* The proposed scaled-attention mechanism obviously helps a lot in practice to improve performance\n\nWeaknesses\n* overall, I would say that the theoretical part is not very rigorous.\n* The paper focuses its whole discussion on the concept of \"stationarity\", which is a pretty well defined word with a strong historical background. basically, a time series x is stationary if time shifts do not change its distribution, but the use that is made here is. Here, the authors call a time series stationary if all (vector) samples are normalized. This is not related.\n\n-----\nEDIT after reviews and discussions\n\nI believe the authors did an amazing job during this review round and I actually think this paper has some big potential impact, even if I still have many things to say regarding its rigour, but I think this is not a fundamental flow that should prevent the paper from being published.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M1uwH5kVmmD", "reviewer": "Reviewer_q3Tk", "summary": "This paper study the non-stationary problem for transformer based forecasting model. They find that the nomarlization step in pre-process will lead transformers to generate indistinguishable temporal attention, which harms the prediction capability. To address this issue, the authors propose the de-stationary attention mechanism which considers the mean and variance statistics when computing the attention score. Empirical studies show consistent improvements over different type of transformers.", "questions": "see above", "limitations": "see above", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The paper is easy to read and the structure is well organized. \n2. The proposed strategy can be easily integrate with different transformer backbone.\n3. Empirical results show consistent improvement.\n\nCons:\n1. The proposed method is not very convincing. I appreciate the author for presenting the analysis for vanilla self-attention in section 3.2. However, I am confused for the motivation of using normalization layer. If the normalized data lead to the vanishment of so-called non-stationary information, why not remove the normalization before embedding layer and use the raw data for computation of attention score?\n2. My major concern is the experiment setting is unfair. The standard preprocess protocol for long sequence forecasting includes the zero-mean normalized, but the reported RMSE, MAE has not inverted the forecasting results. It is unclear the coupled effect for the proposed normalization strategy and the normalization in preprocessing steps. It would be more convincing if the authors could compare the prediction on the original space with the scale-based evaluation metric, like sMAPE, MASE, etc.\n3. It would be better to incorporate more advanced transformed based forecasting model, like ETSformer, FEDformer ( since these models has beaten autoformer with a margin), and justify the new normalization can also bring performance boost.\n4. There exist some open question for the experiments on long sequence forecasting setting. It would be more convincing if the author can present the hyper-parameter selection strategy for the baselines to guarantee the fair comparison, for example, the hyper-parameters for N-Beats and LSTNet.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0mKB71pMTsN", "reviewer": "Reviewer_8mJi", "summary": "This paper proposes Non-stationary Transformers as a generic framework to tackle over-stationarization problem. It includes Series Stationarization and De-stationary Attention module, where Series Stationarization converts raw time series into more stationary ones for better predictability and De-stationary Attention is devised to recover the intrinsic non-stationary information in raw time series during self-station stage. Results show that Non-stationary Transformers are generally applicable with various Transformers and significantly improve results compared to the existing state-of-the-art. ", "questions": "What is the standard deviation of results listed in the paper? It will be good to know how statistically significant the proposed method is.", "limitations": "Figure 4 requires readers to re-read Table 1 \"ADF Test Statistics\" column. Authors can consider adding information in figure 4 to clarify that datasets are sorted according to ADF Test Statistics so that readers are easier to follow.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The motivation is clear and the writing is easy to follow. The De-stationary Attention module is quite simple and clearly deduced in Equation 5, making its implementation straightforward and easy to understand.\n* The results are quite impressive and simple techniques can significantly improve over various state-of-the-art baselines.\n\n**Weaknesses**\n* The proposed Series Stationarization technique share lots of similarity with section 3.3 Scale handling in [1]. The authors need clearly discuss their difference.\n* $\\tau$ and $\\triangle$ are learned in the paper. However, if  $\\tau$ and $\\triangle$ are used to approximate the variance and mean term, authors need to show if directly utilizing pre-computed statistics will work. Such an ablation study will make this work more solid.\n\nReference\n[1] David Salinas, Valentin Flunkert, Jan Gasthaus. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept