Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors note that the stationarisation of time series removes information from the time series that can be used to aid in prediction. They call this problem the "over stationarisation of time series". Given that we want stationarity to improve model performance, but do not want to lose the information from the stationarisation transform, motivates the "Non-stationary transformer" introduced in the paper. 

The model consists of a stationarisation component, and an additional component to reincorporate the stationarised information back into the time series. 

They key component introduced is the De-stationary attention module. This transforms the dat a
strengths_and_weaknesses: It is certainly important to reincorporate the information removed by the stationarity transform into the prediction model. The proposed method introduces some method of doing this and demonstrates that it improves performance accross a wide range of benchmarks.

Stationarisation losing information and thus potential predictive power is intuitively obvious. The novelty here (to me at least) is the destationary attention mechanism. This is another attention block that has some scaling taking into account non-stationary information. 

It's not clear to me why it has to be done in this way. Why can I not just incorporate mu sigma into the MLP applied to the output of the self attention? My biggest issue with the paper is the motivation behind it. Stationarity loses information - fine. Adding information back in some way should improve performance - fine. This is known and not novel. We add some additional transformer to the existing transformer that accounts for some of this information - fine, but I'm not sure why I should be so interested in this architecture? There are lots of ways we could add information back in, why should I choose this one?

The experiments do not compare other mechanisms that incorporate such information. I havent been given a reason to be particularly excited about this mechanism over some other method of sticking in mu and sigma. Is there some motivation behind this mechanism that I am missing?

It would be nice if you had compared with non transformer benchmarks, e.g. LSSL/GRU/

"We refine that the predictive capability of non-stationary series is essential in real-world forecasting. By detailed analysis, we find out that current stationarization approaches will lead to the over-stationarization problem, limiting the predictive capability of Transformers." - From what I can tell, this 'refinement' is a single image showing different attention weights in different scenarios. I dont think there is enough done on this point in this paper to define this as a contribution. 


The writing is not great, I found many parts quite difficult to read. A couple of examples of difficult-to-parse sentences:

"However, non-stationarity is the constitutional property of real-world time series that can be entangled with essential temporal dependencies for forecasting"

"Considering the deep model scenario under the real case, de-stationary factors should be disentangled from the statistics of unstationarized x, Q and K."

In general I think a lot of the wording is overcomplicated and could be significantly simplified. 

You say transformers perform well "credited to their stacked structure". Most DL models for time series are stacked in some way. I dont really see how this is a plus for transformers.
questions: Why does the De-stationary attention module not just operate on the existing, non-stationarised features directly? 
limitations: Yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper study the non-stationary problem for transformer based forecasting model. They find that the nomarlization step in pre-process will lead transformers to generate indistinguishable temporal attention, which harms the prediction capability. To address this issue, the authors propose the de-stationary attention mechanism which considers the mean and variance statistics when computing the attention score. Empirical studies show consistent improvements over different type of transformers.
strengths_and_weaknesses: Pros:
1. The paper is easy to read and the structure is well organized. 
2. The proposed strategy can be easily integrate with different transformer backbone.
3. Empirical results show consistent improvement.

Cons:
1. The proposed method is not very convincing. I appreciate the author for presenting the analysis for vanilla self-attention in section 3.2. However, I am confused for the motivation of using normalization layer. If the normalized data lead to the vanishment of so-called non-stationary information, why not remove the normalization before embedding layer and use the raw data for computation of attention score?
2. My major concern is the experiment setting is unfair. The standard preprocess protocol for long sequence forecasting includes the zero-mean normalized, but the reported RMSE, MAE has not inverted the forecasting results. It is unclear the coupled effect for the proposed normalization strategy and the normalization in preprocessing steps. It would be more convincing if the authors could compare the prediction on the original space with the scale-based evaluation metric, like sMAPE, MASE, etc.
3. It would be better to incorporate more advanced transformed based forecasting model, like ETSformer, FEDformer ( since these models has beaten autoformer with a margin), and justify the new normalization can also bring performance boost.
4. There exist some open question for the experiments on long sequence forecasting setting. It would be more convincing if the author can present the hyper-parameter selection strategy for the baselines to guarantee the fair comparison, for example, the hyper-parameters for N-Beats and LSTNet.
questions: see above
limitations: see above
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Non-stationary Transformers as a generic framework to tackle over-stationarization problem. It includes Series Stationarization and De-stationary Attention module, where Series Stationarization converts raw time series into more stationary ones for better predictability and De-stationary Attention is devised to recover the intrinsic non-stationary information in raw time series during self-station stage. Results show that Non-stationary Transformers are generally applicable with various Transformers and significantly improve results compared to the existing state-of-the-art. 
strengths_and_weaknesses: **Strengths**
* The motivation is clear and the writing is easy to follow. The De-stationary Attention module is quite simple and clearly deduced in Equation 5, making its implementation straightforward and easy to understand.
* The results are quite impressive and simple techniques can significantly improve over various state-of-the-art baselines.

**Weaknesses**
* The proposed Series Stationarization technique share lots of similarity with section 3.3 Scale handling in [1]. The authors need clearly discuss their difference.
* $\tau$ and $\triangle$ are learned in the paper. However, if  $\tau$ and $\triangle$ are used to approximate the variance and mean term, authors need to show if directly utilizing pre-computed statistics will work. Such an ablation study will make this work more solid.

Reference
[1] David Salinas, Valentin Flunkert, Jan Gasthaus. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks.
questions: What is the standard deviation of results listed in the paper? It will be good to know how statistically significant the proposed method is.
limitations: Figure 4 requires readers to re-read Table 1 "ADF Test Statistics" column. Authors can consider adding information in figure 4 to clarify that datasets are sorted according to ADF Test Statistics so that readers are easier to follow.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on forecasting and introduces a new method for scaling attention.
* The input raw series is chunk-normalized to yield the input of the backbone transformer model
* the chunk-wise scaling parameters are processed through MLPs to scale the rows of the attention matrix.

This procedure is showed to consistently improve performance of transformer-based forecasting models.  
strengths_and_weaknesses: Strengths
* The proposed procedure is quite simple to implement
* In my view, introducing this attention scaling is actually a nice and elegant thing to do.
* The proposed scaled-attention mechanism obviously helps a lot in practice to improve performance

Weaknesses
* overall, I would say that the theoretical part is not very rigorous.
* The paper focuses its whole discussion on the concept of "stationarity", which is a pretty well defined word with a strong historical background. basically, a time series x is stationary if time shifts do not change its distribution, but the use that is made here is. Here, the authors call a time series stationary if all (vector) samples are normalized. This is not related.

-----
EDIT after reviews and discussions

I believe the authors did an amazing job during this review round and I actually think this paper has some big potential impact, even if I still have many things to say regarding its rigour, but I think this is not a fundamental flow that should prevent the paper from being published.
questions: * You should not use the words "stationary", "stationarity", "de-stationary" etc unless this is actually what you mean. 
* From a broad perspective, I see your proposed method as applying some chunk-wise transform, say x', \theta = F(x), that returns some modified version for a chunk along with some parameters, and then using some MLP1(\theta) and MLP2(\theta) for scaling the attention matrix. At the output, you apply F^{-1}(out, \theta). I genuinely think this is a great idea, and I think it is ok to motivate this by the following story:
   - Having all chunks have the same first two moments (mean and variance) is a first step in enforcing them to have the same distribution and this is what we do with our series normalization. In this case, our chunkwise transform is a simple normalization, and \theta={\mu, \sigma}.
   - Imagining that we do applied just some linear transform on the input, here's what the attention matrix would be like. So we propose some scaling scheme for the attention inspired by this, that looks like.....
- at the output, we can transform back.
   The advantage of this  way of writing things is that you leave room for further research (notably more sophisticated chunkwise methods) and you don't wrongly pretend that normalizing means enforcing stationarity... because it's not the case.

I think that this paper really brings great things on the table, but that it should undergo some strong modifications to be ok. The good news in my view is that I am not suggesting any change to the actual model, which is good, but just to the story that is told regarding the theory, so that I believe this is feasible (although it requires some heavy work).


Below, please find a list of comments on the go.
Title
* I personally don't like all these "rethinking *" titles, that sound very pretentious in my view, somehow suggesting that things were not correctly thought about before and that everything will change from now on. Please note that your view of stationarity is not rigorous, adding a bit to my frustration.

Introduction:
* "From the aspect of data": awkward. Furthermore, this statement about predictability is not very well motivated in my opinion.
* "From the view of deep learning": why would that be limited to deep learning ?
* "a hot topic": this is an exagerated statement, provided you give 3 references, among which one is almost 15 years old.
* Figure 1 is not clear enough in my opinion. I don't really understand what the input data to the transformer is: are these the I, II, III chunks ? What is that the second column ? Just zooms for the data ? And finally, what are these "attention matrices" ? what's the number of rows, columns. I suspect some interpolation for these images ? Are you using some `interpolation='nearest'` for your plot ?

Related work
* "which is the essential property": an essential
* "is the key to predictability": a key. I am not sure this statement is supported by such generic references.
* I am a bit annoyed by the fact that you constantly talk of stationarity and non-stationarity, but you don't define the terms.


Non-stationary Transformers
* "the key to time series predictability": again, too strong. "an important element" ?
* The reference to figure 1 is not clear. As mentioned already I don't really see what Figure 1 brings on the table exactly, maybe just because it is poorly explained and I don't understand it.
* "To deal with the dilemna [...] simultaneously": you wrote that already
* although this is classical to use "T" for transposition, please note that using \top instead gives a better rendering.

* I really must protest against your statement that a simple normalization is enough to "transform it into standard Gaussian distribution". I don't see any connection between having x be centered and unit variance and x being N(0,1). Of course, the first two moments are matching, but this is not at all sufficient to make x Gaussian. You should remove this statement and simply mention that you are centering and normalizing it.
* I understand the use of "Hadamard product", but the fact is that it can be confusing to some readers. If you like, you could be using a statement like "\frac{a}{b} and a\circ b are element-wise division and product, respectively".
* I don't understand how these \mu_x and \sigma_x vectors are computed. From a single S-dimensional chunk, how do you end up computing a S-dimensional mean and a S-dimensional variance statistic ? Checking out the followup, it turns out you are simply normalizing each chunk, so that \mu_x and \sigma_x are just scalars. You should not be introducing this useless notation.
* Note that Normalization module eliminates the [...] statistics": I don't see at all how this statement is supported. The only thing you enforce is that every chunk is normalized. You may not at all write that this leads to all of them having the same "statistics". As you know, you can't seriously write that mean and standard deviation are the only statistics for a time series... What about all the auto-correlation structure, higher-order statistics, etc. I am fine with your normalization, but please don't write that it results in chunks having the same distribution ! You are simply matching the first two moments.

* You didn't explain how the "sliding window" is achieved: is there some overlap between your chunks ? Or do you obtain your chunks with a simple folding operation (window size=hop size)
* I don't see the point in using uppercase "Series Stationarization" if you're not going to use the (unfortunate) acronym SS. Maybe just use lower case then.
* "the base models will receive stationarized inputs": again, I feel uncomfortable with accepting the statement that a series of normalized chunks is a stationary time series. Just imagining a dataset with outliers, your procedure will obviously change the content of a chunk containing the outlier in a undesired way, so that future research might focus on some other transformation than mean-variance normalization, like quantiles or such. Don't misread me: the method is nice, but in my view you are claiming something that is not exactly there.
* "for an instance": for instance
* again, the reference to figure 1 is unclear to me, because I didn't get what you want exactly to be conveying with this figure
* "following the same distribution": it is not because they are both centered and unit variance that two vectors have the same distribution !
* "to tackle the trivial attention": awkward. to tackle this over-stationarization problem ?
* "To simplify the analysis [...] input series": this assumption is completely unreasonable and it is useless in my view to rely on it seriously for anything. This said, I understand that assuming f to be linear allows you to derive the *general form* of your "de-stationary attention", as what would be happening in the linear case. The fact is that you are actually not using this assumption in the followup since your attention module don't use these \mu and \sigma as such, but instead through some MLPs. I think that the story would be more readable if you introduced it as you do in the linear case, but then simply propose this attention scaling procedure ase a way to reinject the normalization parameters back into attention in a way that generalizes the linear case. This would read much better to me.
* "Since it is a convention to conduct [...] is reduced to a scalar". This sentence makes no sense to me. Just take \mu_x and \sigma_x as scalars right from the start and be good with this.
* As stated above, it is not necessary to write that your \tau and \Delta are there to "approximate \sigma_x^2 and K\mu_q" (you know them already !) but rather that you introduce them as scaling factors in analogy to what happens in the linear case. Don't forget that your derivations assume a linearity that is not there.


Experiments
* What do you call "stationarity" and "relative stationarity" for the "over-stationarization problem" section ? This is not defined.


Conclusion
* "The impressive generality and performance of the proposed framework": please change this pretentious statement.

limitations: The authors are a bit pretentious at times, but other than that it is all right
review_title: null

Meta-review: metareview: The paper introduces a transformer-based method for non-stationary time series forecasting. 
This research addresses a clear need, as acknowledged by the reviewers. Also, most reviewers found the method clearly described and the experiments compelling, demonstrating an improvement of the state of the art.

The reviewers asked questions about the baselines, evaluation methods and ablation studies. They also made requests related to clarifying the wording and some of the theory. The authors put in significant effort in addressing the comments, offering detailed responses to every reviewer. Only one of the reviewers responded during the discussion period, and the response came very late in the discussion period. However, I read the authors' response and concluded that they adequately addressed most issues raised by the reviewers.

As the model is in the Transformer space, and transformers have previously been shown to be state of the art on a number of tasks, I do not find it necessary to compare against other 'families' of methods. So I will consider that issue addressed as well.
confidence: Certain
Decision: Accept