Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes to use a normalization method to address temporal distribution shift in time-series forecasting. The proposed approach, *RevIN*, consists of two steps: instance normalization on input sequences and "de-normalization" of output sequences by re-using statistics (mean and variance) computed during the normalization step. Experiments are conducted on two time-series datasets (ETT and ECL) with varying splits and prediction windows lengths. Results show that RevIN used on top of deep-learning based methods for time-series forecasting (Informer, N-BEATS and SCINet) improves prediction performances, in particular for long sequence prediction. Finally, they conduct an empirical comparison with other normalization methods, such as batch normalization and min-max normalization, to evaluate the adequateness of instance normalization and of the denormalization step. 
main_review: **Strengths:**
- Addressing distribution shift is a relevant task to the ICLR community and of crucial importance to deploy deep-learning based model time-series forecasting;
- Despite being simple, the proposed approach is technically sound, easy to implement, and empirically validated on various datasets, split and prediction windows length;
- The experimental evaluation is thorough and allow one's to evaluate the importance of the type of normalization and the effect de-normalization module.

**Weaknesses:**
- In the recent literature [1], complementary metrics are used to evaluate the shape error (DTW) and temporal error (TDI) separately. Here, the authors only evaluate their method with MSE/MAE metric.

&nbsp;

Small typos:
- introduction of Section 3 : RevIN acronyme used two times in a row;
- page  8: "we the batch normalization" => "we use the batch normalization".

&nbsp;

-----------
[1] V. Le Guen & N. Thome. *Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models*. In NeurIPS 2019.
summary_of_the_review: -----
**Post-rebuttal comments**
The authors have addressed my concerns and engaged with other reviewers concerns. Their rebuttal includes several complementary analysis and experiments which helps to better validate the quality of the approach. Even though the proposed approach is straightforward, the idea behind RevIN is technically sound and the thorough experimental evaluation is conclusive. Thus, I maintain my rate in favor of an acceptance. 
------

While instance normalization is a simple and well-known method in the distribution shift literature, its application for time-series forecasting is sound and empirically validated in this paper. Overall, I tend to vote for accepting but authors should provide complementary metrics raised in weaknesses to further analyze the effect of their approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a "reversible instance normalization" as an input pre and post processing procedure to improve the forecasting of any given base model - targeted at addressing the distribution shift that is common in time series data - e.g., time series are typically non-stationary.

This works by normalizing each time window input to a (deep learning) forecast model, applying the model on the normalized data, then unnormalizing the predictions to get the final predictions.   The normalization is done by subtracting the mean and dividing by the std. dev. in a current (input instance) window, followed by scaling and shifting by learnable, shared cross-instance (global), scaling and shifting parameters per input feature / variable.  

The authors perform extensive experiments to show the proposed approach significantly improves the base metric score results for 3 recent, state-of-the-art forecasting algorithms across several datasets, and further that it significantly improves over other normalization approaches, and helps align distributions between train and test windows.
main_review: Some strengths:
* The approach is interesting and well motivated, and the experiment results useful, striking, and very extensive.  
* It is a good data point to have to see the benefit of this type of normalization across modern state-of-the-art models, and point out this method in the community, which has typically just used global per-series normalization.  
* The results conclusively show the benefit and the method provides a simple modification that can be broadly used in many cases, and the detailed results in the appendix that also include original reported scores and std. dev. are really great to have.  
* The thorough comparisons to different normalization approaches are also helpful.  
* Also the appendix details and code provided are very helpful for reproducibility


Weaknesses:
1. Novelty:

However, the biggest issue is that the proposed work is not very novel, and leaves out the related work on "sliding window normalization" doing essentially the same thing.  I.e., "sliding window normalization" is commonly used in practice for time series forecasting and has been used in the literature on time series forecasting as well.  Sliding window normalization typically just takes the current instance time window mean and std. dev. to normalize the inputs and de-normalize the outputs after getting the predictions.  For example, see the following, and references:

Ogasawara, Eduardo, et al. "Adaptive normalization: A novel data normalization approach for non-stationary time series." The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE, 2010.

Overall it is a simple and straight-forward method, and, the only difference from the classical sliding window normalization approach is the additional learnable parameters (gamma and beta).  However, such learnable parameters are not new on their own either and already used in other general normalization approaches as well as forecasting specific - including instance normalization for forecasting, like DAIN (albeit in that case the difference is it doesn't use the built-in denormalization).

I think the key main missing experiment and result needed is to demonstrate the impact of the learnable parameters, gamma and beta.  I.e., in each of the experiments, if the learnable parameters are excluded what is the impact?  This amounts to an additional ablation study - the same proposed method is used, but with gamma and beta excluded (or fixed to 1 and 0 respectively so they have no effect).

Additionally, I think the hybrid class of models are very similar conceptually and should ideally be compared with and called out as an alternative approach to dynamic normalization.  I.e., the hybrid models fit a simple model essentially based on the recent history of each series combined with a more complex deep learning model - and one of the key motivations for this was to dynamically normalize each series using this simple model.  For example the author of ES-RNN explicitly said that the idea of using the exponential smoothing model was to enable per-series normalization before applying the deep learning part of the model - and it fits an exponential smoothing model to set the level and seasonal pattern (and this is inherently based on limited recent history which is controlled by the smoothing parameters that are learned as part of modeling).  Similarly LSTNet and other methods before and after it simultaneously fit a linear auto regressive model and add it to the neural net part of the model to get the final output.  Both of these kind of approaches are essentially normalizing the time series for the deep learning part, as is done here, and this is done dynamically and potentially capturing nonstationarity of the model as well in the normalization, which isn't done with the proposed approach.  
ES-RNN is Smyl 2020 in the cited references and LSTNet is: Lai, Guokun, et al. "Modeling long-and short-term temporal patterns with deep neural networks." SIGIR 2018.   
It would be best to compare with these (including the distributions) / compare this approach in combination with the baseline models as well - e.g., simultaneously fitting a simple baseline AR model + or * the deep learning model to get the output, as this is an alternate way to perform this kind of dynamic normalization, so it would be important to see how it compares to the proposed one.


2. 
It seems a key hyper parameter is the window length, as clearly if the window length is too short, the instance normalization would hardly be stable (and enable accurate prediction with denormalization to the full horizon).  However, this is not discussed and related experiments not performed.  

I.e., it would therefore be best to show the impact of changing window lengths - i.e., hyper parameter sensitivity as this is varied from a very short window to a longer and longer window (additionally at some point too long windows would also be problematic as it would not be adaptive enough and also limit the number of data points)

It's also important to make clear how the window length was selected for the experiments.


3. Additionally issues / open questions around the formulation and the experiments:
* The mathematical formulation is not precise / changes at different places. E.g., at first instances are referred to as x^(i), then suddenly referred to by x_k.  
* Alg. 1 definition directly contradicts the description of the method and definitions in the main paper.  Namely, the main paper states that there are N time windows, each with K variables / time series (in the case of multivariate series), and that the scaling and shift parameters, gamma and beta, are in R^K, i.e., one shared value for each variable / dimension K - shared across input windows.  However, in Alg. 1 they are said to be in R^NxK - which implies a unique value per window and feature/variable is learned, which doesn't make sense (as then one wouldn't be able to apply it for future forecasts, e.g., series N+1 where we don't have a parameter).  
* It would be best to make clear the dimensions of the different parameters, the dimensions of the inputs and outputs, and data 
* It is also not explained how the hyper-parameters tuned, and it would be 
* It would also best to show the results on a larger variety of datasets, even those used in prior work by the base methods - as the benefit might only be for this common type of data used here.


summary_of_the_review: My main hesitance to accept the paper is the lack of novelty - as sliding window normalization is already commonly used, and the method is a very minor change. I feel this could possibly be improved with the ablation study comparing the results of the proposed method without the learnable parameters gamma and beta.  

I would also like to see more analyses and discussion around the impact of the window length used, and fixes / additional details around method and experiment details.   It would also be best to see the performance on a larger variety of datasets - in case the method is only beneficial for the type of data used here - and there are more datasets from the base method papers that could be used for this, that are left out here.  Additionally it would also be useful to see the results of a hybrid approach with the baselines as a commonly used alternative to the proposed dynamic normalization approach (e.g., fit a linear AR model + the baseline model).

Overall, I feel the paper provides an extensive study showcasing the benefit of the proposed normalization approach / sliding window normalization so it also provides value to the community even if lack of novelty - and highlights this approach to the community as an alternative to the global normalization that has become the standard in recent work in ML-based forecasting.  The method is simple and effective, and this brings attention to a potentially useful tool to have in the forecasting toolbox.  


***Update:***
I feel the authors adequately addressed most of my comments, and provided a lot of useful follow-on study and analyses and additional experiments which are much appreciated and really provide a lot more useful information.  Some of it is useful even for providing more info. on the performance of the base models such as varying the input window length to see the impact on the forecast errors.  It would have been good to see even smaller input window lengths however, as I would expect a behavior where at some point if the input window is too small the method is not beneficial, and if the window becomes too big, it should not be much different from the baselines.

Overall, I feel the set of results are impressive and useful.  I agree with the authors that it's useful to bring the approach to the attention of the community (as there are some similar approaches used in some communities but not well known or used in this community), and also the motivation and perspective comes from very different places compared to the similar approaches, and the idea of how it affects the data distribution and showing results along that direction is also new compared to the prior similar approaches.  It provides a widely applicable and effective method, with thorough experiment results and study.  I also feel it could motivate future work in a direction not too well considered before.  Therefore I think it is worthy of acceptance despite the actual algorithmic approach contributed itself being incremental and simple, and given the responses and revision, decide to change my rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes RevIN that performs (a) instance normalization on input time series and (b) reversible instance normalization on output time series using normalization statistics of the input. By doing this, the paper claims to handle distribution shift in time series and hence yield better forecasting results.  
main_review: Strengths:
The idea behind RevIN is straightforward but seems to work well, outperforming min-max normalization, batch normalization, layer normalization and instance normalization. RevIN is also cheap to compute and model-agnostic.

Weaknesses:
W1: The method seems ad-hoc with no theoretical justification. Is is possible to provide a proof for a simple case (e.g. 1-dimensional time series) that RevIN indeed handles distribution shift over time?

W2: Experiments are on 4 data sets only. To validate the merit of RevIN, I would like to see experimental results on more real-world data sets, for instance those taken from UCI Repository.

Other remarks:
R1: In the first paragraph of Section 3.1, it looks like output sequence $Y$ needs to have the same length as the input sequence $X$. Does this have to be the case?

R2: ECL data set originally has a lot of dimensions and a few instances. After preprocessing, what are the length and dimensionality of the new time series?
summary_of_the_review: The paper is easy to follow as the main idea is straightforward. The technical contribution however seems ad-hoc and incremental. A proof for a simple case would make the paper more convincing.
review_title: null

Meta-review: This paper introduces the "reversible instance normalization" (RevIN), a method for addressing temporal distribution shift in time-series forecasting. RevIN consists in normalizing (subtracting the mean and dividing by the standard deviation) each layer of of deep neural network in a given temporal window for a given instance, and de-normalizing by introducing learnable shifting and scaling parameters. 

The paper initially received one weak accept and two weak reject recommendations. The main limitations pointed out by reviewers relate to the limited novelty of the approach, the positioning with window normalization methods and hybrid methods in times series, and clarifications on experiments. The authors' rebuttal did a good job in answering the main concerns: rV5fo increased its grade from weak reject to clear accept, and RuPmn maintained its weak acceptance recommendation.

The AC carefully read the submission. The AC considers that the idea is simple yet meaningful. The large set of experiments are well conducted and conclusive. The rebuttal successfully answers to relevant issues raised by reviewers, regarding ablation studies (for highlighting the importance of the learnable de-normalization), the impact of the temporal window, the comparison to hybrid approaches and the difference with respect to Adaptive normalization. The AC thus acknowledge that this submission draws important take-home messages for the community, and therefore recommends acceptance.
Decision: Accept (Poster)