Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift

1 INTRODUCTION . Time-series forecasting plays a significant role in various daily problems such as health care , economics , and traffic engineering ( Che et al. , 2018 ; Bauer et al. , 2016 ; Zhang et al. , 2017 ) . Recently , time-series forecasting models have achieved comparative performance on these problems by overcoming several challenges , e.g. , long-term forecasting ( Zhou et al. , 2021 ) and missing value im- putation ( Tang et al. , 2020 ) . However , these time-series forecasting models often severely suffer from statistical properties that change over time , which is also widely known as a distribution shift problem ; for example , the mean and variance of the time-series data distribution vary over time . Accordingly , the input sequences to the forecasting models have different underlying distributions from each other . In the case of the train and test data , they are usually divided into data before and after a specific point in time . Thus , the distributions of the two data often hardly overlap , which is commonly known as the reason for model performance degradation . Thus , we assume that if we normalize every sequence in the data to have identical mean and variance and provide the transformed data to the model , the discrepancy in the distribution will be reduced , thereby improving the model performance . However , if the input to the model is shifted and scaled , the prediction of the model can be shifted and scaled as much as the input ; since this will differ from the groundtruth distribution , the accuracy of the model will decrease . Also , we further hypothesize that if the model output , shifted and scaled , can be returned to its original distribution explicitly , we can alleviate the discrepancy in the distribution of the data during the forward pass of the model as well as allow its prediction to obey the original distribution . Here , reconstructing the original distribution can be achieved by applying denormalization on the output of the model as a reversed function of the normalization applied on the input data . Inspired by these , we propose a simple yet effective normalization method , reversible instance normalization ( RevIN ) , which first normalizes the input sequences and then denormalizes the model output sequences to solve the time-series forecasting task against the distribution shift problem . RevIN is symmetrically structured to return the model output to the original distribution in the denormalization layer by scaling and shifting the model output as much as the input data is shifted and scaled in the normalization layer . Adding only the normalization and denormalization layers on the input and output of the model , respectively , RevIN can be applied to any model without modification to the model architecture . We empirically found that RevIN effectively alleviates the distribution discrepancy within the data , as well as significantly improves the model performance when applied to various state-of-the-art time-series forecasting methods . To verify the effectiveness of RevIN , we conduct extensive quantitative evaluations with various baselines : Informer ( Zhou et al. , 2021 ) , N-BEATS ( Oreshkin et al. , 2020 ) , and SCINet ( Liu et al. , 2021 ) . We also present an in-depth analysis of the behavior of the model , including the verification of the assumptions on reversible instance normalization ( See Section 3.2 ) . RevIN is a flexible , end-to-end trainable layer that can be applied to any arbitrarily chosen layers , effectively suppressing non-stationary information ( mean and standard deviation of the instance ) from one layer and restoring it on the other layer at a virtually symmetric position . Despite its remarkable performance , there has been no work on generalizing and expanding the instance-wise normalization-and-denormalization as a flexibly applicable , trainable layer in the time-series domain . Recently , deep learning-based time-series forecasting approaches , such as Informer ( Zhou et al. , 2021 ) and N-BEATS ( Oreshkin et al. , 2020 ) , have shown outstanding performance in timeseries forecasting . However , they overlooked the importance of normalization , merely using simple global preprocessing to the model input without further exploration and expecting their endto-end deep learning model to replace the role . Despite the simplicity of our method , there have been no cases of using such techniques in modern deep-learning-based time-series forecasting approaches ( Zhou et al. , 2021 ; Liu et al. , 2021 ; Oreshkin et al. , 2020 ) . In this sense , we enlight the importance of the appropriate normalization method in deep-learning-based time-series approaches ; we propose a carefully designed , deep-learning-friendly module , for time-series forecasting , by combining the learnable affine transformation with the method , which has been widely accepted in recent deep-learning-based normalization work ( Ulyanov et al. , 2016 ) . In summary , our contributions are as follows : • We propose a simple and effective normalization-and-denormalization method for timeseries , called RevIN , symmetrically structured to remove and restore the statistical information of a time-series instance ; it is a generally-applicable layer to arbitrary deep neural networks with negligible cost and significantly improves the performance by reducing the distribution discrepancy within the data . • By adding RevIN to the baseline , we achieve the state-of-the-art performance on four largescale real-world datasets with a significant margin . • We conduct extensive evaluations on RevIN with quantitative analysis and qualitative visualizations to verify its effectiveness , addressing the distribution shift problem . 2 RELATED WORK . Time-series forecasting . Time-series forecasting methods are mainly categorized into three distinct approaches : ( 1 ) statistical methods , ( 2 ) hybrid methods , and ( 3 ) deep learning-based methods . As an example of the statistical models , exponential smoothing forecasting ( Holt , 2004 ; Winters , 1960 ) is a well-established benchmark to predict future values . Statistical models have several advantages , e.g. , interpretability and theoretically well guaranteed . In order to boost the performance , recent work proposed a hybrid model ( Smyl , 2020 ) which incorporates a deep learning module with a statistical model . It achieved better performance over the statistical methods in the M4 timeseries forecasting competition . The deep learning-based method basically follows the sequenceto-sequence framework to model the time-series forecasting . Initially , deep learning-based models utilized variations of recurrent neural networks ( RNNs ) . However , to overcome the limitation of the limited receptive field , several studies utilize advanced techniques , such as the dilatation and attention module . For instance , SCINet ( Liu et al. , 2021 ) and Informer ( Zhou et al. , 2021 ) modified the sequence-to-sequence-based model to improve the performance for long sequences . However , most previous deep learning-based models are hard to interpret compared to statistical models . Thus , inspired by statistical models , N-BEATS ( Oreshkin et al. , 2020 ) designed an interpretable layer for time-series forecasting by encouraging the model to explicitly learn trend , seasonality , and residual components . This model shows superior performance on M4 competition datasets . Distribution shift . Although there are various models for time-series forecasting , they often suffer from non-stationary time-series , where the data distribution changes over time . Domain adaptation ( DA ) ( Tzeng et al. , 2017 ; Ganin et al. , 2016 ; Wang et al. , 2018 ) and domain generalization ( DG ) ( Wang et al. , 2021 ; Li et al. , 2018 ; Muandet et al. , 2013 ) are common solutions for alleviating the distribution shift . A domain adaptation algorithm attempts to reduce the distribution gap between source and target domains . A domain generalization algorithm only relies on the source domain and hopes to generalize on the target domain . Both DA and DG have a common objective that bridges the gap between source and target distributions . However , in non-stationary time-series , defining a domain is not as straightforward since the data distribution shifts over time . Recently , Du et al . ( Du et al. , 2021 ) proposed Adaptive RNNs ( AdaRNNs ) to handle the distribution shift problems for non-stationary time-series data . It first characterizes the distribution information by splitting the training data into periods . Then , it matches distributions of the discovered periods to generalize the model . However , unlike AdaRNNs that are costly expensive , RevIN is simple yet effective and model-agnostic since the method can be adopted easily to any deep-learning model . 3 PROPOSED METHOD . In this section , we propose reversible instance normalization ( RevIN ) for alleviating the distribution shift problem in time-series , which causes the discrepancy among the input sequences of the training data , as well as the discrepancy between train and test data distributions . Section 3.1 describes RevIN in detail and Section 3.2 discusses how RevIN alleviates the distribution discrepancy in time-series throughout qualitative visualization . 3.1 REVERSIBLE INSTANCE NORMALIZATION . We consider the multivariate time-series forecasting task in discrete time , with a sliding window size of Tx for input data . Here , N , K , Tx , and Ty indicate the number of the sequences , the number of variables , the input sequence length , and the prediction length . Let X = { x ( i ) } Ni=1 and Y = { y ( i ) } Ni=1 denote the input data and corresponding target , respectively . In RevIN , the input sequence length Tx and the prediction length Ty can be different since the observations are normalized and denormalized across the temporal dimension , as will be explained below . Given an input sequence x ( i ) ∈ RK×Tx , we aim to solve the time-series forecasting problem , which is to predict the subsequent values y ( i ) ∈ RK×Ty . Our proposed method , RevIN , symmetrically transforms the input data x ( i ) and the prediction output ỹ ( i ) of the network , as illustrated in Fig . 2 . First , we normalize the input data x ( i ) with the instance-specific mean and standard deviation , which is widely accepted as instance normalization ( Ulyanov et al. , 2016 ) . The mean and standard deviation are computed for every instance x ( i ) k· of the input data ( Fig . 2 ( a-3 ) ) as Et [ x ( i ) kt ] = 1 Tx Tx∑ j=1 x ( i ) kj and Var [ x ( i ) kt ] = 1 Tx Tx∑ j=1 ( x ( i ) kj − Et [ x ( i ) kt ] ) 2 . ( 1 ) Using these statistics , we normalize the input data x ( i ) as ( Fig . 2 ( a-1 ) ) x̂ ( i ) kt = γk ( x ( i ) kt − Et [ x ( i ) kt ] √ Var [ x ( i ) kt ] + ) + βk , ( 2 ) where γ , β ∈ RK are learnable affine parameter vectors . The normalized sequences have the same mean and variance , where the non-stationary information is reduced . As a result , the denormalization layer allows the model to accurately predict the local dynamics within the sequence while receiving inputs of consistent distributions in terms of the mean and variance . The model then receives the transformed data x̂ ( i ) as input and forecasts their future values . However , the input data have different statistics from the original distribution , and only observing the normalized input x̂ ( i ) , it is difficult to capture the original distribution of the input x ( i ) . Thus , to make this easier for the model , we explicitly restore the non-stationary properties removed from the input data to the model output by reversing the normalization step ( Fig . 2 ( a-3 ) ) . Such denormalization step can return the model output to the original time-series value as well ( Ogasawara et al. , 2010 ) . Accordingly , we denormalize the model output ỹ ( i ) by applying the reciprocal of the normalization in Eq . 2 ( Fig . 2 ( a-2 ) ) as ŷ ( i ) kt = √ Var [ x ( i ) kt ] + · ( ỹ ( i ) kt − β γ ) + Et [ x ( i ) kt ] , ( 3 ) where the same statistics used in the normalization step in Eq . ( 2 ) are used for the scaling and shifting . Now , ŷ ( i ) is the final prediction of the model instead of ỹ ( i ) . Simply added to the input and output of a network , RevIN can be easily applied to any network with negligible cost while effectively improving time-series forecasting performance . Additionally , each step of RevIN , normalization and denormalization , can be applied to any activations of the intermediate layers in the network . However , our method is most effective when applied to the layers of symmetric encoder-decoder structure . In a typical time-series forecasting model , the boundary between the encoder and the decoder is often unclear . Therefore , we apply normalization and denormalization to the input and output of a forecasting network , which can be interpreted as an encoderdecoder structure , generating subsequent values given input data . Additional analysis on selecting layers to apply RevIN is provided in Section 4.2.3 .

Summary: The authors propose a "reversible instance normalization" as an input pre and post processing procedure to improve the forecasting of any given base model - targeted at addressing the distribution shift that is common in time series data - e.g., time series are typically non-stationary.  This works by normalizing each time window input to a (deep learning) forecast model, applying the model on the normalized data, then unnormalizing the predictions to get the final predictions.   The normalization is done by subtracting the mean and dividing by the std. dev. in a current (input instance) window, followed by scaling and shifting by learnable, shared cross-instance (global), scaling and shifting parameters per input feature / variable.    The authors perform extensive experiments to show the proposed approach significantly improves the base metric score results for 3 recent, state-of-the-art forecasting algorithms across several datasets, and further that it significantly improves over other normalization approaches, and helps align distributions between train and test windows.