Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift
Authors: Taesung Kim
Jinhee Kim
Yunwon Tae
Cheonbok Park
Jang-Ho Choi
Jaegul Choo
Venue: ICLR 2022 Poster (2022)
Keywords: Time-series forecasting
Normalization
Distribution shift

Abstract: Statistical properties such as mean and variance often change over time in time series, i.e., time-series data suffer from a distribution shift problem. This change in temporal distribution is one of the main challenges that prevent accurate time-series forecasting. To address this issue, we propose a simple yet effective normalization method called reversible instance normalization (RevIN), a generally-applicable normalization-and-denormalization method with learnable affine transformation. The proposed method is symmetrically structured to remove and restore the statistical information of a time-series instance, leading to significant performance improvements in time-series forecasting, as shown in Fig. 1. We demonstrate the effectiveness of RevIN via extensive quantitative and qualitative analyses on various real-world datasets, addressing the distribution shift problem.

Published as a conference paper at ICLR 2022
REVERSIBLE INSTANCE NORMALIZATION FOR ACCURATE TIME-SERIES FORECASTING AGAINST
DISTRIBUTION SHIFT
Taesung Kim KAIST AI 2k11989 @kaist.
Jinhee Kim KAIST AI
Yunwon Tae VUNO yunwon. tae @vuno.co
seharanul17 @kaist. kr
Cheonbok Park NAVER Corp. cbok.park cnavercorp.com
Jang-Ho Choi ETRI janghochoi @etri... re.kr
Jaegul Choo KAIST AI jchoo @kaist ac kr
1.0
Informer ReviN
BEAATS
ReviNi
SCINet ReviN
0.5
0.0
(b)
Ground Truth
Ground Truth
Ground Truth
0.5
Informer
N-BRRATS N-BEATS BaulN
SCiNiet SCINet Reeot
Informer Beuth
0.0
-0.5
day
Figure 1: Multivariate time-series forecasting results comparing our method with the state-of-the-art baselines, i.e., Informer (Zhou al. (021), N-BEATS (Oreshkin al. (020), and SCINet (Liu al.) 2021). The analysis is conducted on electricity consuming load (ECL) dataset, with prediction length of seven days. The predictions of the baselines are inaccurately (a) shifted and (b) scaled. When adopted to the baselines, our method significantly improves their forecasting performance and better aligns the distribution of the prediction results with the groundtruth values.
ABSTRACT
Statistical properties such as mean and variance often change over time in time series, i.e., time-series data suffer from distribution shift problem. This change in temporal distribution is one of the main challenges that prevent accurate time- series forecasting. To address this issue, we propose simple yet effective nor- malization method called reversible instance normalization (RevIN), a generally- applicable normalization-am wwwrraatiinnnnddeennoooo uund-enormalizztion method with learnable affine trans- formation. The proposed method is symmetrically structured to remove and re- store the statistical information of a time-series instance, leading to significant performance improvements in time-series forecasting, as shown in Fig. 1. We demonstrate the effectiveness of RevIN via extensive quantitative and qualitative analyses various real-world datasets, addressing the distribution shift problem.
Both authors contributed equally. The order of the first authors determined by coin flip.

---

Published as a conference paper at ICLR 2022
INTRODUCTION
Time-series forecasting plays a significant role in addressing various daily problems, including health care, economics, and traffic data analyses (Kim et al. 2021a Ahmadi et al.] 2019; Park
et al. 2020). Recently, time-series forecasting models have achieved outstanding performance on these problems, overcoming several challenges. such as long-term forecasting (Zhou et 202T
Liu et al. [222]) and missing value imputation (Zhang et al. [022]] Kim et al.] (021bb. However, the time-series forecasting models often suffer badly from a unique characteristie time-series data: their statistical properties, e.g., mean and variance, can change over time. This is widely known as the distribution shift problem, and can yield discrepancies between the distributions of the training and test data of the forecasting models. In time-series forecasting tasks, the training and test data are usually divided from the original data based on a specific point in time. Accordingly, they often hardly overlap, which is common reason for model performance degradation. Furthermore, the in- put sequences to the model can have different underlying distributions as well. We can assume that the discrepancy between different input sequences can significantly degrade the model performance.
Under this assumption, if we remove non-stationary information from the input sequences, specifi- cally, the mean and standard deviation of the instances, the discrepancy in the data distributions will be reduced, thereby improving model performance. However, applying such normalization to the model input can cause another problem since can prevent the model from capturing the original data distribution. It removes non-stationary information that can be important to predict future values in the forecasting task. The model would need to reconstruct the original distribution only using the normalized input, which degrades its forecasting performance due to the inherent limitation. Thus, we explicitly return the information removed by input normalization back to the model, the model will not have to rebuild the original distribution by itself while keeping the advantage of normalizing the input. To accomplish this, we propose to reverse the normalization applied to the input data in the output layer, i.e., to denormalize the model output using the normalization statistics.
Inspired by this, we propose a simple yet effective versible instance normalization (RevIN), which first normalizes the input sequences and then de- normalizes the model output sequences to solve the time-series forecasting problems against distri- bution shift. RevIN is symmetrically structured to return the original distribution information to the model output by scaling and shifting the output in the denormalization layer in an amount equivalent to the shifting and scaling of the input data in the normalization layer. To verify the effectiveness of RevIN, we conduct extensive quantitative evaluations using several state-of-the-art time-series fore- casting methods as the basclines: Informer (Zhou et al. 202I), N-BEATS (Oreshkin et al.) 2020). and SCINet (Liu et al., (022). We also provide an de-depth analysis of the behavior of the proposed approach, including verification of the assumptions on reversible instance normalization.
method, re-
RevIN is a flexible, end-to-end trainable layer that can be applied to any arbitrarily chosen layers, effectively suppressing non-stationary information (mean and variance of the instance) in one layer and restoring it in another layer at a virtually symmetric position, e.g., input and output layers. De- spite its remarkable performance, there has been no work on generalizing and expanding instance- wise normalization-ande www.romm-nnddddddomaiiitatteo as flexibly applicable, trainable layer in the time-series domain. Recently, deep learning- based time-series forecasting approaches, such as Informer (Zhou et al. 2021) and N-BEATS (Oreshkin et al. 2020). have shown outstanding performance in time- series forecasting. However, they have overlooked the importance of normalization, merely using simple global preprocessing of the model input without further exploration and expecting their end- to-end deep learning model to replace the role. Despite the simplicity of our method, there have been no cases of using such techniques in modern deep-learning-bee based time-series forecasting ap- proaches (Zhou et al. 2021 Liu et al. [021] Oreshkin et al. 22020). In this sense, we introduce the importance of an appropriate normalization method for ddep-learning-bart based time-series approaches. We propose a carefully designed, deep-leat leapl-nan-nn---iieeee -freinlll module for time-series forecasting by com- bining the method with the learnable affine transformation, which has been widely accepted in recent deep-learning-basse normalization work (Ulyanov et al.) 2016).
In summary, our contributions are as follows:
We propose a simple yet effective normalization wwwpritii--andddddnmmmiiiiii denommaaomaiiinoonn method for time- series, called RevIN, which is symmetrically structured to remove and restore the statisti-
2

---

Published as a conference paper at ICLR 2022
cal information of time-series instance. The proposed method is generally applicable to arbitrary deep neural networks with negligible cost.
By adding RevIN to the baseline, we achieve state-of-the-an performance on seven large- scale real-world datasets by a significant margin.
We conduct extensive evaluations of RevIN using quantitative analysis and qualitative vi- sualizations verify its effectiveness, addressing the distribution shift problem.
2
RELATED WORK
Time-series forecasting. Time-series forecasting methods are mainly categorized into three dis- tinct approaches: (1) statistical methods, (2) hybrid methods, and (3) deep learning-based methods. Statistical models are theoretically well guaranteed and have several advantages, including inter- pretability, As an example of the statistical models, exponential smoothing forecasting (Holi) 2004 Winters 1960) is a well-established benchmark for predicting future values. To further boost perfor- mance, recent work proposed hybrid model (Smyl] 2020) that incorporates a deep learning module with a statistical model. achieved better performance than statistical methods in the M4 time- series forecasting competition. The deep -earning-based method basically follows the sequence- to-sequence framework to model the time-series forecasting. Initially, deep learning-based models utilized variations of recurrent neural networks (RNNs). However, overcome the limitation of the limited receptive field, several studies utilized advanced techniques, such as the dilatation and at- tention module. For instance, SCINet (Liu et al.) (021) and Informer (Zhou et al., (021)) modified model to improve performance for long sequences. However, most previous deep learning- based models are hard to interpret compared statistical models. Thus, in- spired by statistical models, N-BEATS (Oreshkin et al.) 2020) designed an interpretable layer for time-series forecasting by encouraging the model learn trend, seasonality explicitly, and residual components. This model shows superior performance on the M4 competition dataset.
the sequence-to-seee too-seeuence-baaee
Distribution shift. Although there are various models for time-series forecasting, they often suf- fer from non-stationary time-series, where the data distribution changes over time. Domain adapta- tion (Tzeng et al., 2017: Ganin al. 2016 Wang al., 2018) and domain generalization (Wang) et al. 2021: Li et al.) | Muandet et al. 2013) are common ways to alleviate the distribution shift. A domain adaptation algorithm attempts reduce the distribution gap between source and target domains. A domain generalization algorithm only relies on the source domain and hopes to generalize on the target domain. Both domain adaptation and generalization have a common ob- jective, which bridges the gap between source and target distributions. However, defining a domain is not straightforward in non-stationary time series since the data distribution shifts over time. Re- cently, Du et al. (Du et al. 2021) proposed Adaptive RNNs to handle the distribution shift problems of non-stationary time-series data. It first characterizes the distribution information by splitting the training data into periods. Then, it matches the distributions of the discovered periods general- ize the model. However, unlike Adaptive RNNs, which is costly, RevIN is simple yet effective and model-agnostic. The method can be easily adopted to any deep neural network.
3 PROPOSED METHOD
This section proposes reversible instance normalization to alleviate the distribution shift problem in time-series, which is known to cause a substantial discrepancy between the training and test data distributions. Section 3.1 describes the proposed method in detail, and Section 3.2 discusses how our approach mitigates the distribution discrepancy in time-series data.
3.1 REVERSIBLE INSTANCE NORMALIZATION
Given set of input X' (,
1== and the corresponding target =²= ( |= we consider mul- tivariate time-series forecasting task in discrete time, where N denotes the number of sequences. Let K, Tu, and T, denote the number of variables, the input sequence length, and the model predic- tion length, respectively. Given an input sequence ((() E KT,, we aim solve the time-serie forecasting problem, which is to predict the subsequent values (()) RKTT, In RevIN, the input
3

---

The model then receives the transformed data (6) as input and forecasts their future values. How- ever, the imput data have different statistics than the original distribution, and by observing only the normalized input a(0) it is difficult to capture the original distribution the input (x(() Thus, make this easier for the model, we explicitly return the non-stationary properties removed from Published as a conference paper at ICLR 2022
(b-1)
(b-4)
Source distribution
Target distribution
Pa
Pa y
(a-1)
(a-3) Non-stationary information
(a-2)
Instance normalization
ReviN
μl, O, B, Y
ReviN
Denormalization
(b-2)
(b-3)
distribution
Target distribution
Ps
Figure 2: Overview the proposed method. We illustrate an example a univariate case, where 2(()) E R IXx:; the input data x²(0) is actually multivariate (See Section 3.1 In RevIN, the (a-1) instance normalization and (a-2) denornalization are symmetrically structured to remove (a-3) non- stationary information from one layer and restore on the other layer. Here, RevIN is applied to the input and output layers. The (a-3) non-stationary information includes statistical properties from the input data: mean μr, variance o², and learnable affine parameters y, 3. The normalization layer transforms the (b-1) original data distribution into (b-2) mean-centered distribution, where the distribution discrepancy between different instances is reduced. Using f, the model predicts the future values ý following the (b-3) distribution where non-stationary information is eliminated. To restore it (b-4), RevIN reverses the instance normalization in the output layer.
sequence length T₂ and the prediction length Tg be different since the observations normal- ized and denormalized across the temporal dimension, as will be explained below. Our proposed method, RevIN, consists of symmetrically structured
layers, as
illustrated in Fig. 2 First, normalize the input data (x()) using its instance- -peecicc mean and stan- dard deviation, which is widely accepted as instance normalization (Ulyanov et al. 2016). The mean
and standard deviation are computed for every instance x E To of the input data (Fig 2 a-3)) as
(t)
Tc
Th
and
Var
(0)
Σ
E.
(1)
T
Tx J=1
kJ
Using these statistics, we normalize the input data z(() (Fig. 2 a- aa-l)) as
E. (x
kt
7
+ 8k.
(2)
Varlz
where 8,8 RK are learnable affine parameter vectors. The normalized sequences can have more consistent mean and variance, where the non-stationary information is reduced. As result, the normalization layer allows the model to accurately predict the local dynamics within the sequence while receiving inputs of consistent distributions in terms the mean and variance.
the input data to the model output by reversing the normalization step at symmetric position, the output layer. denormalization step can return the model output the original time-series value as well (Ogasawara al. 2010). Accordingly, denormalize the model output (()) by applying the
4

---

Published as a conference paper at ICLR 2022
E
Density
ETTm1 Density
dria
87
dalo
dara
-33 -50
par 1bt Iest dat
Trsi Tool data
test dita
tsst data
ECL Density
(a) original input (b) RevIN-norrmaized
25
-1
(c) model output - (d) ReviN-denormalized
Figure 3: Effect of RevIN on distribution discrepancy between training and test data. From left to right columns, we compare the training and test data distributions of a variable on each step of the sequential process in RevIN: (a) the original input 2, (b) the input è normalized by RevIN, (c) the model prediction output ÿ. and (d) the output ý denormalized by RevIN, the final prediction. The analysis is conducted on the ETT and ECL datasets using SCINet (Liu et al. 2021 the baseline.
reciprocal of the normalization in Eq Fig. 2
(a)3)) as
Tkt
(6)
3k
Varl 1ar[
E,
7k
The same statistics used in the normalization step in Eq. g(t) is the final prediction of the model instead of g(().
2 are used for the scaling and shifting. Now,
Simply added to virtually symmetric positions in network, RevIN can effectively alleviate dis- tribution discrepancy in time-series data, as cenerally-applicall trainable normalization layer to arbitrary deep neural networks. Indeed, the proposed method is a flexible, end-to-end trainable layer that can be applied to any arbitrarily chosen layers, even to several layers. We verify its effectiveness as a flexible layer by adding it to the intermediate layers in the model in Table 7 in Appendix [A.4] Nevertheless, RevIN is most effective when applied to virtually symmetric layers of encoder-decoder structure. In a typical time-series forecasting model, the boundary between the encoder and the de- coder is often unclear. Thus, we apply RevIN to the input and output layers of model they can be interpreted as an encoder-decoder structure, generating subsequent values, given input data.
3.2 EFFECT OF REVERSIBLE INSTANCE NORMALIZATION ON DISTRIBUTION SHIFT
This section verifies that RevIN can alleviate the distribution discrepancy problem by removing non-stationary information in the input layer and then restoring it in the output layer. We analyze the distributions of the training and test data each step of the proposed approach, shown in Fig. 3
When comparing the distribution of training and test data in each example (Fig. 3(ab)),, we can observe that RevIN significantly reduces their discrepancy. To be specific, in the original input (Fig. B(),, the training and test data distributions hardly overlap (especially ETTm,). which is caused by the distribution shift problem. Also, each data distribution has multiple peaks (especially the test data of ETTh; and HCL), implying that sequences in the data might have severe discrepan- cies in their distributions. However, in the proposed approach, the normalization step transforms each data distribution into mean-centered distributions (Fig. 3 b)). This result supports that the original multimodal distributions (Fig. (() are caused by discrepancies in distributions between different sequences in the data. Even more, the proposed approach makes training and test data dis- tributions overlapped. This verifies that the normalization step of RevIN can alleviate the distribution shift problem, reducing the distribution discrepancy between training and test data.
5

---

Published as a conference paper at ICLR 2022
Taking the normalized data the imput, the model can retain aligned training and test data distri-
butions in the prediction output (Fig. C)). As expected, these are then returned back to the orig-
inal distribution by the denormalization step of RevIN (Fig. d)). Without denormalization, the model needs to reconstruct the values that follow the original distributions (Fig. a() using only the normalized input that follows the transformed distributions where non-stationary information is removed (Fig. Bjb)). Additionally, we hypothesize that the distribution discrepancy will be reduced in the intermediate layers of the model as well. when RevIN is applied the input and output layers only, which will be discussed in Section 4.2.3 As result, this RevIN procedure can be considered to first make problems easier, and then restore them back to the original state, rather than directly solving the challenging problem where the distribution shift problem exists.
4 EXPERIMENTS
This section describes the experimental setup and provides extensive experimental results of RevIN.
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate methods mainly four large-scale real-world time-series datasets. Ad- ditionally, we provide experimental results on three more datasets, including the air quality and Nas- daq datasets taken from the UCI repository and M4 competition dataset (Makridakis et al. 2020) in Appendix A (i) Electricity transformer temperature (ETT) data consists of seven features, including power load features and oil temperature. is collected from two different regions in China for two years. Following the same protocol as Informer (Zhou et al.) 202I), we split the data into three datasets: ETTh,, ETTh,, and ETTm1. The ETTh; and ETTh datasets are hourly data ob- tained from different regions. The ETTm; dataset has value every 15 minutes. For each dataset, we split the first months, the middle four months, and the last four months as training, validation, and test data, respectively. (ii) Electricity Consuming Load (ECL data contains the electricity consumption (kWh) collected from 321 clients. Following the prior work (Zhou et al.) 2021), data from each client is used variable on an hourly basis in the multivariate forecasting setting. For the ECL dataset, we use 15, 3, and 4 months as training. validation, and test data, respectively.
Experimental details. We set the prediction lengths to be one day (1d), 2d, 7d, 14d, 30d, and 40d for the hourly-basis datasets, ETTH,, ETTh,, and ECL. For the ETTm, dataset, we chose six hours (6h). 12h, 3d, 7d, and 14d as the prediction window lengths. We evaluate the time-series forecasting performance on the mean squared error (MSE) and mean absolute error (MAF). Following the same evaluation procedure used in the previous study (Zhou et al., [202I). we compute the MSE and MAE on z-score normalized data to measure different variables on the same scale. More details on exper- imental settings, including training details and byperparameters, are provided in Appendix A1
Baselines compared. RevIN is a model agnostic method, generally applicable to any deep neural network. In this paper. we verify the effectiveness of RevIN by adopting to three state-of-the-ar time-series forecasting models: Informer (Zhou et al. 2021), N-BEATS (Oreshkin et al., 2020), and SCINet (Liu et al. 2021). These are non-autoregresssve forecasting models. The reproduction details for the baselines are provided in Appendix AA12]] Unless stated otherwise, we compare RevIN and the basclines under the same hyperparameter settings, including the input and prediction lengths.
4.2 RESULTS AND ANALYSES
This section provides the quantitative analysis and qualitative visualization results of RevIN in com- parison with the state-of-the-art time-series forecasting baselines.
4.2.1 EFFECTIVENESS OF REVERSIBLE INSTANCE NORMALIZATION ON VARIOUS TIME-SERIES FORECASTING MODELS
Table [ compares the forecasting accuracy of the baselines and RevIN. The results show that RevIN consistently outperforms all three baselines, Informer, N-BEATS, and SCINet, by large margin,
http::/iieiiie......ccdc/////aar http://wwww.accccooooooooaoaaaan 12014

---

Published as a conference paper at ICLR 2022
Table 1: Comparison forecasting errors between the baselines and RevIN. The analysis on the four datasets, ETTh, ETThg. ETTm,, and ECL, is conducted by increasing the prediction length from 24 to 960/1344. We report the average errors for five runs. The complete results are provided in Appendix A. including standard deviation and the originally reported values for the baselines.
Method
Informer
RevIN
N-BEATS
RevIN
SCINet
RevIN
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0,550
0.536
0.504
0.472
0.478
0.505
0.330
0.373
0.338
0.373
0.308
0.347
0.772
0.668
0.646
0.547
0.536
0.542
0.372
0.400
0.436
0.459
0.365
0.389
ETTh
168
1.138
0.853
0.655
0.561
1.005
0.782
0.466
0.452
0.459
0.461
0.406
0.416
336
1.278
0.909
1.058
0.758
0.932
0.743
0.515
0,483
0.527
0.513
0.467
0.471
720
1.357
0,945
0.926
0.717
1.389
0.926
0.576
0,534
0.596
0.571
0.507
0.505
960
1,470
0,990
0.902
0.715
1.383
0.932
0.678
0.575
0.604
0.574
0,545
0,526
24
0,450
0.520
0.238
0.325
0.403
0.472
0.192
0.276
0.199
0.295
0.180
0.263
48
2.171
1,200
0.361
0.404
1,330
0.918
0.254
0.320
0.350
0,422
0.231
0.302
ElTh2
168
8.157
2.558
0.859
0.649
7.174
2.329
0.410
0.418
0,559
0.518
0.337
0.378
336
4.746
1.844
0.890
0.673
4.859
1,863
0.449
0.447
0.664
0.583
0.357
0,403
720
3,190
1.529
0.576
0,546
5.656
2.012
0.496
0.482
1.546
0,944
0.411
0.445
960
2.972
1.441
0.600
0.570
6,408
2.077
0.471
0.481
1.862
1 1.066
0.438
0.462
24
0.330
0,382
0,309
0.352
0.443
0,437
0,403
0.392
0,130
0.231
0.106
0.196
48
0.499
0.486
0.390
0.391
0.453
0.472
0.328
0.371
0.155
0.262
0.135
0.222
ETTm,
96
0.605
0.554
0.405
0.411
0.603
0.581
0.379
0.406
0.195
0,291
0.162
0.247
288
0.906
0,738
0.563
0.502
0.849
0.702
0.451
0.445
0.361
0.419
0.265
0.321
0,943
0,760
0.663
0.550
0.860
0.726
0.555
0.511
1,020
0,756
0.357
0.380
1344
1.095
0.823
0.824
0.632
14.613
1.948
0.631
0.556
1.841
1.044
0.412
0.422
24
0.250
0.358
0.148
0.257
0.279
0.372
0.176
0.285
0.138
0.246
0.112
0.207
48
0.300
0.386
0.171
0.279
0.309
0.388
0.194
0.301
0.163
0.265
0.126
0.222
ECI.
168
0,345
0,423
0.261
0.354
0.333
0.410
0,218
0.320
0.177
0,281
0,153
0.249
336
0.429
0.473
0.356
0.414
0.326
0.406
0,241
0.337
0.202
0.308
0.162
0.262
720
0.851
0.719
0.834
0.700
0.420
0.467
0.303
0.383
0.234
0.333
0.183
0.281
960
0.930
0.750
0.894
0.741
0.399
0.455
0.325
0.398
0.235
0.330
0.200
0.292
Table 2: Comparison of long sequence forecasting performance. We analyze the forecasting error of the baselines and RevIN by increasing the prediction length from 48 to 960 while the input length fixed to 48. The experiment is conducted on ETTh, The average errors for five runs are reported, and the complete results, including standard deviation, are provided in Appendix A.14
Prediction length
48
168
336
720
960
Metrie
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Informer
0.687
0.628
0.982
0.795
1.212
0.893
1.157
0.863
1.203
0.888
+ RevIN
0.540
0.481
0.680
0.574
0.939
0.696
1.021
0.752
1.061
0.775
N-BEATS
0.512
0.523
0.804
0.690
1.001
0.773
1.022
0.765
0.901
0.728
+ RevIN
0.365
0.389
0.454
0.438
0.526
0.477
0.568
0.514
0.638
0.544
SCINet
0.376
0.396
0.600
0.556
0.841
0.695
0.875
0.721
0.900
0.737
+ RevIN
0.349
0.370
0.445
0.426
0.509
0.461
0.533
0.494
0.557
0.510
achieving state-of-the-art performance on the four datasets. Moreover, the effectiveness of RevIN is more evident for the long sequence prediction, where it remarkably reduces the errors of the baselines. RevIN shows a stable performance in contrast to the baselines, which show a high increase in error as prolonging the prediction length. For example, when the prediction length increases from 24 to 960 on the ETThy dataset, the forecasting error of N-BEATS significantly increases from 0.403 to 6.408. In contrast, RevIN shows much slight increase in error, i.c., from 0.192 to 0.471. A similar tendency appears with the other prediction lengths, datasets, and bascline models as well. These results demonstrate that RevIN makes the baseline model more robust to prediction length.
7

---

Published as a conference paper at ICLR 2022
10.0
10.0
5.0
Mean squared error
Informer Informer+ReviNN
N-BEATS
SCINet
R-BEA
SCINet
6.0
6.0
3.0
4.0
4.0
2.0
2.0
2.0
0.0
40
0
10
20 day
30
35
40
5
10
15
20
25
30
35
40
Figure 4: Forecasting error for each time step. We compare the error of predicting 1~960 steps
ahead between the baselines and RevIN on ETTh; when the prediction length is 960 (40 days).
Table 3: Comparison with classical and state-of-the-art normalization methods. The mean squared errors are compared on the four datasets, using N-BEATS as the baseline for all experiments. Here, every normalization method is applied the input data. DAIN, deep adaptive input normal- ization (Passalis et al. 2219);; RevBN, the reversible batch normalization, i.e., the modified version
of RevIN. Complete results, including different prediction lengths, are provided in Appendix A.7
Dataset
ETTh;
ETTh;
ETTm,
ECL
Prediction length
168
960
168
960
96
1344
168
960
Min-max norm
1.074
1.224
2.987
3.308
1.035
1.320
0.374
0.387
z-score norm
0.953
1.043
3.329
3.087
1.016
1,274
0.335
0.377
Layer norm
0.871
1.303
4.092
5.822
0.502
2.488
0.343
0.379
DAIN
0.996
1.032
1.982
2.802
0.672
1.348
0.347
0.381
Batch norm RevBN
0.851
1.691
6.206
7.755
0.505
1.147
0.320
0.422
0.717
0.779
0.729
2.148
0.601
0.991
0.327
0.414
Instance
0.946
1.090
3.240
3.145
1.021
1.329
0.333
0.386
RevIN (ours)
0.515
0.697
0.419
0.465
0.388
0.602
0.220
0.329
We further quantitatively analyze the effect of RevIN on long sequence prediction in Table When the model prediction length is increased from 48 to 960. RevIN reduces the prediction eтror com- pared to the baseline, showing robust performance against the prediction length. The difference in the forecasting eтror between SCINet and RevIN is relatively small when the prediction length is short (e.g., 48), but RevIN remarkably surpasses SCINet by a significant margin when the prediction length is long (e.g., 336, 720, and 960). These results substantiate that adopting RevIN can make model robust to the prediction length.
Additionally, to study how RevIN can perform well in long sequence prediction, we visualize the forecasting error for each time step in Fig. The error at the 1-th time step computed as ykt (2) Overall, RevIN shows superior performance compared
MSE-t = EN EE (() to the baselines; the performance degradation significantly slower, showing low emror even when forecasting 960 steps ahead. Specifically, for N-BEATS and SCINet, the error extremely increases as predicting the distant future values. RevIN alleviates this significant increase error, showing remarkable performance compared to the baselines. Informer also shows unstable performance for the different time steps. The error is substantial in the early steps and becomes relatively small in the distant steps. RevIN allows the models to have consistently minor emrors in every time step, where the baselines originally show high error (early steps for Informer, distant steps for N-BEATS and SCINet). The results demonstrate the effectiveness of RevIN when forecasting long sequences.
2k=1
4.2.2 COMPARISON WITH EXISTING NORMALIZATION METHODS
We compare RevIN with classical and state-of-the-an normalization methods, including min- max normalization, z-score normalization, layer normalization (Ba et al. (016). batch normaliza- tion (Toffe & Szegedy 2015), instance normalization (Ulyanov al. 2016). and deep adaptive input
8

---

Published as a conference paper at ICLR 2022
Informer ReviN
informer ReviN
46 48
Informer
2.0
nnfoer
RcviN
ReviN
3,03
1.5
1.28
Train test featrue
1.07
4.33
1.0
20
10.28
3.23
10-
0.5
0.3
0.25
0.56
0.14
Layer-1
Layer-2
Layer-1
0.18
Layer-2
0.29
Layen
Layer-2
0.0
Layen-1
Layer-2
(a) 1
(b) ETTh2
(c) ETTm1
(d) ECL
Figure 5: Feature divergence between the training and test data in the intermediate layers of the model The feature divergences computed on the ETTh;, ETTh,, ETTm, and ECL datasets using the features obtained from the first (Layer-1) and the second (Layer-2) encoder layers in Informer.
normalization (DAIN) (Passalis et al. 2019) in Table Here, we compute the statistics for min-max and z-score normalization methods for every input instance, not for the entire data. Additionally, we attempt to use batch normalization as the input normalization method in RevIN, named reversible batch normalization (RevBN). Layer normalization cannot be used in similar manner since it is not reversible when the input and prediction lengths are different, as in our experimental settings.
As a result, RevIN shows outstanding performance compared to the other normalization methods, especially on ETTh, and ETTm; datasets. Also, RevBN improves forecasting performance of batch normalization. Specifically, the error considerably decreases in the long sequence prediction, such as 960 and 1344. This result supports that the denormalization step of RevIN is essential as a key com- ponent the proposed method for improving long sequence forecasting. However, batch normal- ization applies identical normalization to all the input sequences, using the global statistics obtained from the entire training data; can not reduce the discrepancy between the training and test data distributions. Consequently, RevIN, which transforms the data in the instance level, outperforms RevBN by a significant margin, demonstrating that successfully reducing the discrepancy between distributions of different input sequences can effectively improve performance. Moreover, RevIN not only shows the best performance but also has the advantage of being lightweight compared to the baselines. For example, when K the number of variables, DAIN requires at least 3k²2 additional parameters, whereas RevIN only requires 2K additional parameters.
4.2.3 ANALYSIS OF DISTRIBUTION SHIFT IN THE INTERMEDIATE LAYERS
In Fig. [ we analyze the feature divergence between the training and test data to verify that RevIN reduce the distribution shift at the intermediate feature level as well. We conduct the experiment using Informer as the baseline; it comprises two encoder layers and one decoder layer. Thus, we an- alyze the features of the first (Layer-l) I) and the second (Layer-2) encoder layers. Following the prior work (Pan et al.] 2018), we compute the average feature divergence using symmetric KL divergence (See Appendix 10) The results show that RevIN significantly reduces the feature divergence be- tween the training and test data in both layers, demonstrating that the proposed approach, when added only the input and output layers, successfully alleviates the distribution shift problem in the intermediate layers. Moreover, this strengthens RevIN as generally-applicable flexible layer. An arbitrary model can adopt RevIN by adding to input and output layers without any architectural modifications. Note that our approach still can be added to any arbitrarily chosen layers, significantly improving the model performance, as shown in Appendix A.4
CONCLUSION
This paper aims to address the distribution shift problem in time series, proposing a simple yet fective wanprattionnnnddddemmiiiizioooii method, reversible instance normalization (RevIN). The proposed approach effectively alleviates the discrepancy between training and test data distribu- tions, leading to significant performance improvements in time-series forecasting. As generally- applicable layer to arbitrary deep neural networks, the proposed approach achieves ofte--f--theaa art performance seven real-world time-series datasets by significant margin. The extensive quanti- tative and qualitative experiments with in-depth analysis demonstrate the effectiveness of RevIN for accurate time-series forecasting against the distribution shift problem.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide the source code of our method publicly, including the pre- trained model weights. In the main manuscript, Section 4.1 describes how we conduct data prepro- cessing on the datasets used in the experiments. Appendix A.11 explains the experimental details, including random seed values for the experiments. Appendix 12 provides detailed explanation of hyperparameter configurations with the reproduction details of the baselines.
ACKNOWLEDGMENT
This work was supported by the Institute Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. time complex artificial intelligence blue-green algae prediction technology based on direct-readable water quality complex sensor and hyperspectral image, and No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).
Space-
REFERENCES
Mohsen Ahmadi, Saeid Ghaaushhhhhh Rahim Taghizadeh, and Abbas Sharifi. Presentation of new hybrid approach for forecasting economic growth using artificial intelligence approaches. Newral Computing and Applications, 31(12):66118868) 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1Xiv:1607.06455 2016.
Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adarn:: Adaptive learning and forecasting of time series. Proc. the International Conference on Information and Knowledge Management (CIKM), 2021.
Laura Frías-Paredes, Fermín Mallor, Martín Gastón-Romeo, and Teresa León. Assessing energy forecasting inaccuracy by simultaneously considering temporal and absolute errors. Energy Con- version and Management, 142:533-546, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research (JMLR), 17(1):2096-0300, 2016.
Charles Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting, 20(1):5-10, 2004.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. the International Conference on Machine Learning (ICML), pp. 448-456. PMLR, 2015.
Jinhee Kim, Taesung Kim, Jang- Ho Choi, and Jaegul Choo. End-to-end multi-task learning of miss- ing value imputation and forecasting in time-series data. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 1849-88566 IEEE, 2021a.
Taesung Kim, Jinhee Kim, Wonho Yang, Hunjoo Lee, and Jacgul Choo. Missing value imputation of time-series air-quality data via deep neural networks. International journal of environmental research and public health, 18(22):12213, 2021b.
Guokun Lai, Wei-Cheng Chang. Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. CoRR, 2017.
Vincent Le Guen and Nicolas Thome. Shape and time distortion loss for training deep time series forecasting models. In Proc. the Advances Neural Information Processing Systems (NeurIPS), volume 4191, 2019.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
10

---

Published as a conference paper at ICLR 2022
Minhao Liu, Ailing Zeng, Qiuxia Lai, and Qiang Xu. Time series is a special sequence: Forecasting with sample convolution and interaction. arXiv preprint arXiv:2106.09305. 2021.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1): 54-74, 2020.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invari- ant feature representation. In Proc. the International Conference on Machine Learning (ICML). PMLR, 2013.
Eduardo Ogasawara, Leonardo C Martinez, Daniel De Oliveira, Geraldo Zimbrão, Gisele L Pappa, and Marta Mattoso. Adaptive normalization: novel data normalization approach for non- stationary time series. In The International Joint Conference on Neural Networks (IJCNN) pp- 1-8. IEEE, 2010.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. Proc. the International Conference on Learning Representations (ICLR), 2020.
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two once: Enhancing learning and generalization capacities via ibn-net. In Proc. of the European Conference on Computer Vision (ECCV), 2018.
Cheonbok Park, Chunggi Lee, Hyojin Bahng, Yunwon Tae, Seungmin Jin, Kihwan Kim, Sungahn Ko, and Jaegul Choo. St-grat: A novel spatio-temporal graph attention network for accurately forecasting dynamically changing road speed. In Pruc. the ACM Conference on Information and Knowledge Management (CIKM), 2020.
Nikolaos Passalis, Anastasios Tefas, Juho Kanniainen, Moncef Gabbouj, and Alexandros Iosifidis. Deep adaptive input normalization for time series forecasting. IEEE transactions on neural net- works and learning systems, 31(9):3760-377 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Proc. the Advances in Neural Information Processing Systems (NeuriPS), 32, 2019.
Slawek Smyl. hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36(1):75-5)) 2020.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 2016.
Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu. Visual do- main adaptation with manifold embedded distribution alignment. Proc. the ACM international conference on Multimedia, pp. 402-410, 2018.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: survey domain generalization. arXiv preprint arX1v:2103.03097, 2021.
Peter R Winters. Forecasting sales by exponentially weighted moving averages. Management sci- ence, 6(3):324-34, 1960.
Ying Zhang, Baohang Zhou, Xiangrui Cai, Wenya Guo, Xiaoke Ding, and Xiaojie Yuan. Miss- ing value imputation in multivariate time series with end-to-end generative adversarial networks. Information Sciences, 551:67-82, 2021.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proc. the AAAI Conference Artificial Intelligence (AAAI), 2021.
11

---

Published as a conference paper at ICLR 2022
A APPENDIX
This section provides additional information, visualizations, and experimental results that support the main manuscript. Section A.1 shows the experimental results additional real-world datasets along with a qualitative analysis on one of the datasets to verify the effectiveness of our method on obvious non-stationary time series. Section [.2 addresses the potential of RevIN solving the cross-domain time-series forecasting task. Section A.33 and Section AA.4 present the hyperparame- ter sensitivity analysis and the ablation study on the proposed method, respectively. Section [A.5] evaluates our method using complementary metrics, which measure the similarity between two se- quences. Section A.6 and Section A.7 compare the forecasting results using the proposed method and existing normalization methods. Section A.8 Section [A.9] and Section [A.10] provide the algo- rithm for RevIN, the theoretical justification of RevIN, and the calculation details of the feature divergence used in Section 4.2.3 respectively. Section 11 and Section A. describe additional implementation details and the reproduction details of the baselines where RevIN is applied, respec- tively. Section A.13 illustrates additional quantitative results for RevIN and the baselines. Lastly, Section A. 1.14 provides complete quantitative results, including the standard deviation values for five experiments, which can not be included in the main manuscript due to the lack of space.
A.1 EXPERIMENTAL RESULTS ON ADDITIONAL REAL-WORLD TIME- DATASETS
Table 4: Forecasting performance on the air quality, Nasdaq, and M4 competition datasets. The results on the M4 dataset (*) are multiplied by ten for readability. The average value and the standard deviation value for five runs are reported.
Methods
Informer
+ RevIN
N-BEATS
RevIN
SCINet
+ RevINN
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0,802
0,671
0,585
0,539
0.698
0,626
0.527
0.498
0,512
0.514
0.490
0,474
0.1778
0.084
11133
0.123
0.004
0.027
00005
0,0013
-0029
0.0199
0.006
+ 0.004
Air quality
48
+: 0.0554
0.966
+ 0.023
0.761
: IJ1866
0.859
0.668
1,044
+ 0,106
0.955
0.740
0.035
0.705
0,019
0.600
0,009
0.712
0.099
+ 0.0047
0.627
0.659
0.013
4: 1,007
0.566
0.923
0.761
1.079
0,818
0.789
0,660
0.957
0.794
1: 0.100
0.010
:: 0.056
0,020
1 0,108
0.046
0,008
0.005
0.067
1: 0,031
0.025
: 0.014
336
1,278
0.901
1.145
0.801
1,105
0.835
0.660
0.685
0.989
0,760
0.854
0.676
1: 0.074
0.032
0.032
0,010
1 0.052
0.021
0.017
0,006
00.111
1. 0.04
0.029
1. 0,010
1.104
0,810
0,968
0.686
1.226
0,858
1: 0.216
0.061
+: 0.028
0.009
+ 0.419
0.110
1 0.015
0.008
= 0.048
+: 0.021
0.024
+ 0.013
30
+ 0.052
5,318
1,093
0.017
1.273
0.078
0,630
0.009
5,500
0.647
1,254
0.086
1,023
0.034
0.577
0.007
1.742
-0.111
+ 0,028
0,739
0.985
0.018
+ 0.005
0,564
Nasdaq
60
+ +.0022
5.525
+ 0.066
1.098
+ 110M
1,573
0.666
0.111
+ 0.424
5,226
1,236
0.092
:: (L004
1.207
0.617
0.001
2.304
0.062
+ +.00
0.790
1.161
0.021
+ 00,03
0.601
120
0.144
5,793
1.090
0.012
2.648
0.762
0,016
6.023
1.197
0.034
1.959
0,714
3.227
0,853
1.869
0,697
M4'
average
+ 0.0002
0.099
0.258
0.020
+ IJIM5
0.00K
0,174
11005
+ 0.037
2.241
2,065
0.029
+ 0.014
2,082
1.974
0.006
2.180
1.943
+ 0.011
1.943
2.079
0.011
+ 0.004
1.892
We evaluate the proposed method on the four large-scale worl-world time-series datasets, the ETTh, ETTh,, ETTm;, and ECL. datasets in the main manuscript. Additionally, this section provides ex- perimental results on three more datasets, including two real-world datasets taken from the UCI repository. the air quality dataset and the Nasdaq dataset, and the M4 competition dataset (Makri- dakis et al.) 2020). In total, our proposed method is evaluated on seven datasets in this paper.
Air quality dataset contains hourly averaged responses collected from five metal oxide chemical sensors located in Italy. The data consist of 13 variables of length 9537. We set the prediction length as (24, 48, 168, 336, 720) and the corresponding input length as (48, 96, 168, 168, 360} so that their ratios become (2x, 2x, 1x, 0.5x, 0.5x).
Nasda dataset consists of 82 variables, including important indices of markets around the world, the price of major companies in the U.S. market, treasury bill rates, etc. It is measured daily, having a total of 1984 data samples for each variable. We prolong the prediction length as [30, 60, 120] and set the corresponding input length as 60 for all.
"htpss//archive.ics.. www.:eeecssiiiiiiieec. tcd.ccclllmm/aaaass////AAaAalla
12

---

Published as a conference paper at ICLR 2022
Groundtruth
N-BEATS
N-BEATS ReviN
Train
Test
2
Close
-1
-2
-3
15
DTB6
10
DE1
-2
2010.01.11
2010.10.25
2011.08.09
2012.05.23
2013.03.12
2013.12.23
2014.10.08
04.29
2016.02.11
2016.08.30
2017 2017.06.15
Figure 6: Prediction results the Nasdaq dataset. The results three variables in the data, Close, DTB6, and DE1, are shown. The prediction length is 60 days, and the seventh value is illus- trated to show the results on the entire set. We compare RevIN with N-BEATS.
M4 competition dataset consists six different hourly, daily, weekly, monthly, quarterly, and yearly sets, containing 100,000 test data. We follow the original experimental protocol of the M4 competition (Makridakis et al. 2020). For evaluation, we measure the micro-averaged mean ab- solute and mean squared error: first compute the metric independently for each set, i.e., hourly, daily, weekly, monthly, quarterly, and yearly sets, and then calculate the weighted average of the metrics using the contributions of each set the weights.
As shown in Table 4 RevIN significantly improves the forecasting performance of the baselines on all three datasets. Notably. RevIN shows outstanding performance on the Nasdaq dataset, reducing the prediction errors by more than half compared to the baselines.
Additionally, conduct qualitative analysis on the Nasdaq dataset verify the effectiveness of our method on obvious non-stationary time series. As shown in Fig. 6 the Nasdaq index (the variable 'Close') has steadily increased since 2010. Accordingly, when data are divided into the training and test data based on specific point in time (vertical dashed line in Fig. ( the test data values tend to be higher than the training data values. other words, the data severely suffers from the distribution shift problem, where the training and test data show discrepancy in their distribution. As a result, even existing ofate-of-the-ait models often cannot predict the future values appropriately, shown in Fig. 66) The baseline fails keep up with the trend in data, whose mean value continues to increase, and thus the prediction results are shifted. However, ReviN mitigates this distribution discrepancy and remarkably increases the prediction performance of the baseline.
hhtps:/mmocc.uiicccccymym
13

---

Published as a conference paper at ICLR 2022
Similarly, RevIN shows superior performance on the other rapidly increasing data ('DTB6' in Fig. O) and the decreasing test data ('DE1' in Fig. 6 accurately predicting the changing mean the data.
A.2 CROSS-DOMAIN TIME-SERIES FORECASTING
Table Cross-domain time-series forecasting results. We conduct a cross-domain evaluation on the ETT datasets, ETTh, ETTh2, and ETTm; We train RevIN using SCINet as the baseline. We report the average errors and the standard deviation values for five runs.
Train
FTTh;
ETTh;
ETTm;
Test
ETThy
ETTM,
ETTh;
ETTm,
ETTh;
ETThy
Prodiction length
0,471
0,741
0471
0.765
0.614
0.671
0.608
16668
0.659
0.654
0.518
1811
0.004
0,056
0.043
0.02
0.020
0.049
0..10
MAE
0,450
0.024
0.640
0.027
= 0.032
0,27
= 0.013
0.666
0.507
0.014
0.607
0.035
+ 0.031
0.877
1.044
0.046
01562
0.015
0.688
0.016
: 0.004
0.509
1.027
0.078
0.350
0.419
0.346
0,452
0,501
0.586
0.321
0.441
0,478
0.542
0.387
0.506
0.010
0.008
0.0004
0.006
0.001
0.003
0,010
0.383
0.451
0.331
0.449
0.449
0.542
0.331
0.445
0.477
0.531
0.417
0,5(L3
MAR 0.006
0.004
0,005
0,001
0.004
0,007
:: 0,004
0,001
0.002
0,006
:: 0.002
0,006
As RevIN can alleviate the distribution discrepancy between the training and test data, we investigate the ability of RevIN in mitigating distribution discrepancy between different domains through cross- domain time-series forecasting task. In time series, domain can be a location of sensor where the data is collected. We use the ETT datasets for the experiment since they have the same feature categories. However, their distributions ETTh₂ datasets are collected from different locations, and the ETTh and ETTm; datasets have dif- ferent measurement time intervals. Thus, we alternately use each ETT dataset source domain for training and target domain for testing. The goal of cross-domain time-series forecasting is to alleviate the data distribution discrepancy between the source and target domains, e.g., between the ETTh; and ETTh, datasets.
exhibit significant discrepancy because the ETTh, and
In Table despite the difference in the data distributions, RevIN shows remarkable performance in cross-domain time-series forecasting. In particular, RevIN outperforms SCINet by a large margin when the model needs to reduce the discrepancy between the ETTh, and ETTm; datasets. The results demonstrate that RevIN successfully solves the distribution shift problem by alleviating data distribution discrepancy between different domains, leading to better generalization performance.
A.3 HYPERPARAMETE SENSITIVITY ANALYSIS
We analyze the hyperparameter sensitivity of the proposed method compared with the baseline mod- els. Input sequence length can be crucial hyperparameter to RevIN since the method computes the mean and the standard deviation across the entire input sequence and then uses the statistics its
Informer MSE
- Informer MAE
SCINet MSE
SCINeet MAE
RReiNN MSE
- ReviN MAF
RRviN
-RVVV MAF
25.955
1.3
+ReviN MIEE - -RvvN MAE
3.5
3.2
3.0
1.3
2.5
1.0
11
0.9
0.8
0.7
0.7
0.5
0.5
0.5
480
168
336
480
720
960
720
Inout sequence longth
Input sequence length
Input sequence length
Figure 7: Impacts of the input sequence length on RevIN compared with the baselines. We prolong the input length from 48 (two days) to 960 (40 days) when the prediction length is set as 960 (40 days) on the ETTh, dataset. The average errors for five runs, with standard deviation values, are reported. In N- N-BEATS, the standard deviation value for the mean squared error is too large to visualize when the prediction length is 960; we write the value as "#4.955" instead.
14

---

Published as a conference paper at ICLR 2022
normalization and denormalization steps. Thus, the imput sequence length would play crucial role in the prediction accuracy and stability of the training process. Accordingly, we prolong the model input sequence length to see its impact on the forecasting performance of RevIN, as shown in Fig. 7 As a result, RevIN consistently outperforms the baseline for various input sequence lengths. More importantly, RevIN makes the baseline models more robust the input length. In other words, RevIN shows stable performance for various input lengths in contrast to the baseline models, which shows the high variance in their performance according to the input length. Notably, in N-BEATS, the average еттоr, as well as the standard deviation of the enror, significantly increase as prolonging the input length. This is because the trend in data is expressed As the input length becomes prolonged, there is a chance that the variance (or non-stationarity) in the time- series values will become higher, decreasing the accuracy of the linear trend to fit the data unless the data is monotonically increasing or decreasing. Also, the mispredicted trends linearly increase the error in future values. However, when N-BEATS adopts the RevIN layer, it removes the non-stationary statistics from the input, and thus, the model shows robust performance against the input length. Removing the variability, i.e., normalizing its mean and standard deviation, before feeding it to the model and returning it to the output makes the model learning stable.
a linear function in N-BEATS.
A.4 ABLATION STUDY
Table 6: Ablation study results. We ablate the affine transformation (affine.) from RevIN and evalu- ate forecasting performance the six datasets. N-BEATS is used as the baseline for all experiments. We report the average and the standard deviation values for the five runs.
Method
RevIN w/o affine.
+ RevIN w/ affine. (ours)
Metric
MSE
MAE
MSE
MAE
48
0,370 + 0.006
0.393 + 0.003
0.363 + 0.005
0.389 + 0.003
ETTh:
960
0.675 + 0.038
0.576 + 0.014
0.638 + 0.035
0.559 + 0.017
48
0.257 + 0.003
0.322 + 0.002
0.255 + 0.008
0.321 + 0.005
ETTha
960
0.483 = 0.012
0.487 + 0.007
0.471 = 0.015
0.481 + 0.008
96
0.384 + 0.013
0.408 + 0.009
0.378 + 0.011
0.406 + 0.007
ETTm:
1344
0.664 + 0.085
0.567 + 0.039
0.631 + 0.061
0.556 + 0.020
48
0.197 + 0.002
0.302 + 0.002
0.195 + 0.002
0.301 + 0.001
ECL
960
0.347 = 0.034
0.415 + 0.026
0.325 + 0.019
0.398 1 0.015
Air
48
0.707 + 0.009
0.601 + 0.002
0.705 + 0.019
0.600 1 0.009
quality
720
0.852 = 0.025
0.689 - 0.013
0.842 = 0.015
0.686 + 0.008
0.983 = 0.017
0.565 + 0.005
0.981
0.017
0.564
0.005
Nasdaq
60
1.163 + 0.010
0.610 + 0.002
1.155 + 0.020
0.608 + 0.005
We ablate the affine transformation from RevIN to analyze its impact on forecasting performance. We conduct the analysis on the ETTh; ETT,,, ETTm; ECL, Nasdaq, and air quality datasets using N-BEATS as the baseline. The results in Table 6) show that the affine transformation consistently contributes to performance improvement can add RevIN in an intermediate layer, even to several layers. While existing approaches are a wwwp:eesssinn-andd--ppppeeeeoeee
variety of datasets. As mentioned earlier, model
method applied outside of the main prediction model, RevIN is an end-to-end trainable layer that can be added to any layer in the model as batch normalization (Toffe) & Szegedy 2015) and instance normalization (Ulannov et al. (016)). which are recently proposed deep Saaming-basee normalization layers. Thus, we verify that adopting RevIN in the intermediate layers instead of the input and output layers can improve the forecasting performance as well. We add RevIN to the first stack of N-BEATS and SCINet and evaluate their performance on the six datasets. The results in Table demoonsrate that even when added to the intermediate layers, RevIN improves the performance of the baselines, adding RevIN to the input and output of a model since it shows robust performance on average. Nevertheless, the model adopting RevIN in the intermediate layers consistently outperforms the baseline without RevIN, frequently achieving the best performance among all. This performance is even better than the dynamic normalization methods, LSTNNet and ES-RNN", when they
learnable normalization layer. We mainly focus on
15

---

Published as a conference paper at ICLR 2022
adopted to N-BEATS as well (See Table 9 in Appendix A.6). For example, when the prediction length is 960, the mean squared errors of LSTNet", ES-RNN', RevIN (inter.), and RevIN (1/o) are 5.627, 1.338, 0.523, and 0.471, on average. In conclusion, RevIN is flexible, end-to-end trainable
Table 7: Effectiveness of RevIN when added to the intermediate layers in the model. We add RevIN to the first stack of N-BEATS and SCINet and evaluate their performance on the six datasets. We report the average value and standard deviation of five experiments. RevIN (inter.) indicates the model where RevIN is added to the intermediate layers of the baseline network. RevIN (i/o)
indicates the model where RevIN is added to the input and output layer of the baseline network.
Method
N-BEATS
RRyIN (inter.)
RevIN (ilo
SCINet
RRevIN tinter)
+RevIN (3lo
Matric
MSE
MAE
MSR
MAR
MSE
MAF
MSE
MAR
MSF
MAR
MSR
MAR
0.478
0,505
0.347
0,389
0.330
0.373
0.338
0.373
0,306
0.347
0.308
0.347
+: 0.022
+: +.0012
+: 0.000
0.004
4. 0.006
0.004
+: 0.012
4 10.009
+. 0.0004
1: 0.004
+ 0.000
0.002
0.536
0.512
0.375
0,107
0,372
0,400
0:136
0.159
0,363
0,394
0.365
0,389
0.0000
+0.041
0.005
0.001
0.002
0.025
0.021
0.004
0.004
0.005
ET1b:
168
1,005
000146
0.7782
0.064
+ 0.086
0.495
0.481
0.062
+ 1.0030
0.466
0.452
0.014
0,499
0.015
+ 0.013
01461
+ 0.0001
0.415
+ 0.002
0.424
0.486
0.003
0.416
0.003
0,992
0.743
0.508
0.515
0,483
0,527
0.552
0.516
0.467
0.079
0.012
1 0,043
0.027
+ 0.013
0.008
0.010
1 0.000
0.000
1: 0,001
0,005
0,003
1,389
0.926
0.608
0.572
0,576
0.534
0.596
0,571
0,560
0.550
0.507
0.505
720
4.2.30
0,066
0.016
0.009
E 0,035
0.018
0.0L5
0.013
0.007
1. 0.005
0,006
0,004
960
+ 0.380
1,383
0.120
0.932
+ 0.033
0.664
0.604
0.015
+ 0.019
0.678
0.575
0.009
0,604
0.017
+ 0.014
0,574
+ 0.005
0.619
+ 0.002
0.582
0.545
0,010
0.526
0.005
0,403
0.472
0.199
0.291
0.192
0.276
0.199
0.295
0.186
0.272
0.180
0,263
0.188
0.101
0.002
D.002
0.083
0.002
0.026
+ D027
0000
+ 0.001
0.004
0.002
48
0.240
1,330
+ 0.073
0,918
+ 0,007
0.263
0.335
0,005
+ 10.11
0,254
0,320
0
0.550
0,025
+ 0.077
0,423
0005
0,313
+ 0030
0,373
0.231
0.0066
0.302
0,006
ETThu
168
7.174
0.449
2.329
0.049
+ 0.015
0.425
0,131
0.010
+ 0.000
0,410
0.418
0,005
+ 0.044
0,559
+ 0.025
0.518
+ 00000
0.338
+ 10000
0.337
0,007
0.378
0.003
336
4,859
0.268
+0044
1,863
0.446
0,007
0.456
0.005
0.449
0.01
0.447
0.006
0.664
0.073
0,58.
0.030
0.422
0.000
0.443
0.000
0.357
0,000
0.403
0.002
720
5.656
1.005
0.186
20122
+ 0,022
0.505
0,501
1.013
+ 1.008
0.496
0.482
0.002
+ 0.378
1,546
+ 1.141
0,944
+ +.00d
0.64
+ 0.005
0.564
0,11
0.003
0.445
0.002
6,108
2,077
0,523
0.522
0,471
0,481
1,862
1,066
0.734
0.603
0.438
0,462
2.039
0.242
0.040
0.025
0.015
0.008
0.153
0.055
0.014
+ 0.005
0.007
0.004
0.443
0.437
0.387
0.391
0,403
D.392
0.130
0,231
0.108
0,203
0.106
0002
0.196
0.043
1 0.035
1 0,018
0.012
1 0,006
0,005
0.003
1 0.003
0.002
1 0.000
0,002
0,001
48
D.453
0.0334
0.472
0.018
A 0.008
0:40
0,388
0.007
E 0.010
0.328
0.371
0,007
+ 0.004
0,156
1 1.0004
0,262
11000
0220 1:41
Ef E100 0
0
0,000
0.222
0,002
ET7ah
0.603
0.581
0401
0.428
0.379
0.496
0.195
0.291
0.192
12285
0.160
0.247
0.05
0.027
0,00
0,004
0,011
0,047
0.012
+ 0013
0000
+ 0017
0.000
0,001
288
+ 0.095
0,860
0.849
0.726
0,702
0.0511
1 0,032
03002
0.553
0,512
0,463
0.018
E 0.016
0.555
0.451
0.445
0,511
0,008
= 0.008
0,361
1.020
1 10004
0.756
0.119
1 0.000
0.663
0,264
1 1,000
0.223
05833
0000
-
000000
0.265
0,380
0.321
0.002
0.57
+ 00026
0,200
0009
D.11
0.008
0,040
+ 1002
+0081
+ 0.33
0,004
0.002
14.613
1.948
0.723
0.594
0.631
0.556
1.84
1.044
0.989
0.717
0.412
0.422
1344
+ 26.
1.655
+ 0.064
0.027
10061
0,020
0.242
+ 0,000
0.211
+ 0,000
0.008
0,103
24
0.279
0.007
0.372
0.003
+ 0.001
0,182
0,300
0.00
+ 0.002
0.176
0.285
0
0.138
0.004
+ A000S
0,246
+ 0.000
01111
+ 10000
0,207
0.112
0.001
0.207
0.001
1.007
0,309
0,388
0.004
0.207
0,003
0,318
0,002
0,194
0.001
0.301
1000
0,163
0.007
0,265
0.007
10000
E 1000
0.178
100000
0,222
0.001
0.333
0.410
0237
0,340
0,218
0,320
0.177
0,281
0,10
02248
0.153
0.249
DCL
168
+: 0.016
+ 0.012
+ 0.007
0.004
+ 0.002
1. 0,001
+ 0.003
+ 0.005
4 0.0022
T 10000
0,003
0.002
0.326
0.406
0.245
0,348
0,241
0.337
0,202
0,308
0.161
0,261
0.162
0.262
0.004
0,001
* 1,00
0.007
0.005
0.002
0.004
+ 0.004
0.002
+ 0,002
0.061
0.001
0.420
D.467
0.300
0.393
0.303
0.383
0.234
0.333
0.184
11283
0.1K3 1
0.281
720
0.094
0.058
1 0.009
0,016
1 0,012
0.011
0.006
1 0.004
0.033
00003
1 0.003
0,003
0,002
0.399
0.455
0.335
0,413
0.325
0,398
0,236
0,330
0.196
0.295
0.200
0.292
960
0.002
0.017
A 0,018
0.0L5
1. 0.019
0.015
0.0L1
1 0.008
0.005
1 0,005
0.003
0.002
0.698
0.626
0.558
0.537
0,527
0.498
0.512
0,514
0.485
0
0.490
0,474
0.064
0.029
+ 1
0.008
0.005
0
0.029
+ D.019
0.006
0.009
0.006
0,064
0.955
0.740
0.722
0.629
0.705
0.600
0.712
0.627
0.651
0.578
0.659
0.566
Air quality
48
+.0.06
0.03
+ 0,013
0.005
+ 1.019
0
0.099
+ 0.047
0,032
+ 0.023
+ 0.013
0J/077
1,079
0,818
0.199
0.691
0.789
0.660
0.957
0.737
0,717
0.648
0,794
0.108
0.046
+ 0,07
0000
0.008
0JNIS
0.067
+ 0O01
0.020
+ 0.012
0.025
0.014
336
L1005
0.052
0.835
0.021
+ 0,018
0,902
0.721
0.0LO
E 0.017
0,860
0.685
0.006
0.989
0,111
1 0.046
0,760
0,870
0.022
+ 0,011
0.695
0.854
0.029
0.676
0,010
720
1,538
0.419
+ 0.110
0.968
+ 0.030
0.445
0,757
0.012
+ 0.015
0.842
0.686
0
+ 0.048
1.228
+ 0.021
0.858
+ 0.064
0.939
+ 0.025
0.730
0,839
0.024
0.013
5.500
1.254
0,940
0.581
1,023
0.577
1.742
0.739
1.111
0.599
0.985
0,564
Nasdag
60
0.647
5.226
0.086
1,236
0.989
0.055
0.578
0.027
1.207
0.034
0.617
0.067
2,504
0.111
0.790
0.095
1,280
0,630
0.020
1.161
0.018
0.601
0.005
+.424
0.032
+ 0.025
0.008
+ 0.044
0.009
+ 0.062
+ 0,010
0.023
+ 0,004
0.021
0,003
120
6.023
0.382
1.197
0.034
1 0.014
0.615
0.003
1 0.062
1,959
0.714
0.006
5.227
0.236
1 0.007
0,853
2,585
0.374
1 0.031
1.869
0.037
0.003
16

---

Published as a conference paper at ICLR 2022
layer that can significantly increase the performance of a model in time series forecasting, applied to any arbitrarily chosen layers.
A.5 PERFORMANCE EVALUATION ON SIMILARITY METRICS FOR TIME SERIES
We evaluate the forecasting performance of RevIN mainly on the mean squared error and the mean absolute error. Additionally, we use complementary metrics that measure the similarity between two sequences. dynamic time warping (DTW) and temporal distortion index (TDI) (Le Guen & Thome,
2019 Frfas-Paredes et al. 2017). Table [ shows that approach significantly improves the base- line models across all datasets in terms of the DTW and TDL. Notably, RevIN exhibits outstanding performance by large margin compared the baselines for long prediction length. For example, when RevIN is added, the average DTW decreases from 38.348 to 15.240 for Informer, from 53.148 to 12.766 for N-BEATS, and from 20.498 to 11.080 for SCINet when the prediction length is 960 on ETTh,. There are a few cases where the proposed method predicts a less similar sequence than the baseline. However, the margin is minimal in terms of either DTW or TDI compared the significant margin found when our method outperforms the baseline. These results demonstrate that adopting RevIN can generate a sequence more similar to the groundtruth than the baseline, especially showing better prediction accuracy on the longer sequences.
Table 8: Comparison results on similarity metries for time series. We assess the model forecast- ing results in terms of the shape and temporal errors using the DTW and the TDI, respectively (the lower, the better). The experiments are conducted for the four datasets, using the three baselines. We report the average value and standard deviation of five experiments.
Method
Informer
RevIN
N-BEATS
+ RRvIN
SCINe:
RevIN
Metric
TDI
DIW
TD1
DTW
TDI
DTW
TDI
DIW
TDI
DTW
TDI
DTW
24
1,612
0.093
2,515
=001121
:: -0.037
1.207
2.2016
0.0997
#0.147
1.309
2,361
0.0448
1.831
-0.021
2: 200011
1,830
= =0.113
1.136
# +.0037
1,784
:: =0.017
0.966
+ 0.007
1.672
48
3,932
1.0.96
0.333
4.178
2.762
1..190
3,401
0.063
1.0.88
2.430
1.2230
3.564
1.592
2.745
0.297
2.418
1.0117
2.873
1,456
2.508
0.0122
168
+.6,522
31.560
10.384
0.420
+ 1.173
9.459
6.532
0.227
+ 3.679
19,369
+ 0.4490
8,937
6.190
1,033
+ 0.161
5.756
4.913
11085
0.170
5,344
+ 0.0075
3.791
5.017
0.00N
333
72,959
6,769
14,850
0.514
45.215
17.098
11,6155
1,321
47.961
8.500
11,782
0.531
14_302
2.127
8.532
0.162
6.120
0.329
7.723
0.087
6,973
0,292
7.319
0.025
720
1 13.008
167.254
0.2
98.648
18.571
18.483
0,346
1. 28.984
143.832
20.694
0.952
26.265
8,439
12,383
0.458
3.558
1.2.4
11,550
1. 0,296
10,511
0.056
9660
+ 16.076
182.008
28.848
1.217
: 9.174
128.152
22.018
0.204
+ 48 669
148.671
23,810
2,418
40.774
13,955
15.107
0.744
24.067
3.383
+ 0,270
13.551
+: 0,461
14.148
12,508
0,063
24
2,016
0.152
2.481
0.353
1,601
0.029
1,476
0.230
2,507
0.488
0,084
1.409
0.007
1,210
0.128
1,350
0.095
1J0NK
0,022
1,250
0,015
48
4,462
0.567
8.194
0.399
+ 0.342
4802
2.806
0,081
4.244
0.420
6.131
0,542
E 0.165
2.529
2.207
0,010
3.368
0.350
+ 0,183
2.362
+ 0,058
2.374
1.906
0.019
ETTh:
168
24,016
3.355
32.196
1.563
24.516
6.950
0,313
42,270
1,708
28.487
0.642
14.759
5.242
0.104
12.307
1.637
4.899
0,229
10.205
4.225
0,083
336
74.168
9.015
79.549
2.455
+ 3,180
85.817
9.572
0.270
95.645
6.432
+ 1.115
30.837
42.654
3,450
8.063
0.155
28.833
2 681
+ 0.350
7.447
+ 0.141
15,831
5.975
0J112
720
120 440
22,426
35,008
2.230
11L342
18,634
13,109
0.391
+ 17.061
185.321
47,461
6.122
92.521
33.126
11.953
0.917
142 118
524524
16.428
2.748
26.577
0,987
S. $,989
0,035
960
177.336
18.126
38.348
1.665
170.218
14.187
15.240
0.246
1. 75.629
281.720
53.148
9,108
29.742
12.766
0.734
218.477
27.365
1 1.999
20.198
1. 2.007
43,028
11,000
0.108
244
2.343
0,130
1,730
0.098
1: 0.094
2.178
1,590
0,040
1. 0.135
3.109
2,062
0,178
2: 0.067
3.368
1882
0.026
0.099
2.618
:: 0.023
L,991
1: 0,034
2.134
0.995
0,0077
4.818
3,133
2.457
5.223
3,036
4.400
1,470
0.122
0.097
0.093
0.038
0.340
0.083
0,084
0.061
00335
0.077
0.045
0.015
ETTm:
96
1. 0.671
8,550
4.813
0.228
+ 0.098
5.837
1. 0,085
3.586
9.696
L.050
+: 0.162
5.220
E: 0.271
6.503
3,803
0.090
+: 0.349
5.479
+: 0.093
2.409
+ 0,045
4.782
2.150
0,009
2888
1.
32,918
11.054
17.231
0.759
7,403
0.122
3,710
42,809
1. 0.513
10.706
15.942
0.531
7.285
0.064
22.650
2.733
5.365
0.072
15.457
4,436
0,033
672
+ 17.789
80.546
16.365
0.749
+ 6.209
38.265
+ 0.455
12,030
+ 13.075
111,531
+ 0.649
16,601
+ 5.569
45.064
+ 0.311
12,462
= 10.180
149.483
+ 0.350
13,980
+ 1,867
44.721
K,040
0.126
1344
+: 16.808
161.539
23,822
1.203
77,844
7.682
17.293
0.606
444.199
59,765
69.125
63.858
161.725
99.382
19.490
3.115
397, 100
39.675
+ 2.824
27.319
98,890
4,495
12.779
0.156
24
0.517
0.007
1.600
0,015
1 0.095
0,006
0.506
0.004
1,702
0.017
0.002
1,368
0.010
0.017
1. 1.016
P 0,001
0.281
1,05S
0.053
48
0.677
0.029
2.375
0.038
: 0.006
0.383
1806
0,012
0.676
0.025
2.444
0,028
0.446
0.M88
1.972
0.011
0004
0-166
: 0.038
L.752
: 0,006
0,318
1.527
0,004
168
1,338
4.364
0.745
3.709
1,738
4.576
0,339
3,828
I.730
3.122
0.017
0.071
0.024
0.054
+ 0.166
0.087
0.013
0.016
0.077
0.033
0.029
0,026
336
1. 0.046
2.276
# 0.148
6.358
L. 0.044
1.495
1. 0.126
5.640
1. 0.5662
3.555
1. 0.049
6,448
1. 0.683
1.661
1. 0.038
5.702
+: 0.209
2.336
+ 0.003
5.103
+. 0.034
1.194
1L 0.012
4,591
720
23/481
15.963
16.561
22.117
3.790
17.227
4.169
12,379
10.841
1.204
5,583
1.060
9.239
4.576
0.102
7.913
2,830
7.162
0.156
960
+ 18.405
32,548
21,065
3.649
+ 8.781
28.328
+ 3.002
21.717
+ 3.309
15,866
+ 0.377
12.128
+ 1.400
9.198
11.140
0.347
6,972
0.678
+ 0.108
9,255
+ 0.233
4.502
8,560
0.064
17

---

Published as a conference paper at ICLR 2022
Table 9: Forecasting performance of RevIN in comparison with existing dynamic normaliza- tion methods. LSTNet" indicates the model where the autoregressive linear bypass module of LST- Net is added the baseline network. ES-RNN" indicates the model where the exponential smooth- ing of ES-RNN is added to the baseline network. The experiments are conducted on the ETTh, ETTh,, E ECL, and the M4 datasets using N-BEATS as the baseline. The missing perfor- mances in the table are where the model fails converge.
Methods
N-BEATS
ISSNNtt
ESRNN
RevIN
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.478
0.505
0.462
0.497
0.547
0.515
0.330
0.373
24
+ 0.022
0.012
0.047
0..35
+ 0.031
+ 0.021
0.006
+ 0.004
0.536
0.542
0.587
0.576
0.662
0.567
0.372
0,400
48
0.060
0.041
0.063
0.043
+ 0.027
+ 0.012
0.001
+ 0.002
1.005
0.782
1.031
0.795
0.698
0.600
0.466
0,452
ETTh
168
+ 0.146
0.064
+ 0.099
+ 0.054
+ 0.044
0.018
+ 0.030
+ 0.014
0.932
0,743
0.964
0.760
0.768
0.640
0,515
0.483
336
+ 0.079
0.042
+ 0.047
+ 0.023
+ 0.041
+ 0.022
+ 0.013
+ 0.008
1.389
0.926
1.549
0.994
0.966
0.742
0.576
0.534
720
+ 0.230
0.066
0.061
0.022
+ 0.084
+ 0.022
0.035
+ 0.018
1.383
0.932
1.293
0.897
0.678
0.575
960
0.380
0.120
0.059
0.026
0.019
+ 0.009
0.403
0.472
0.394
0.485
0.614
0.522
0.192
0.276
24
0.185
0.101
0.099
0.068
0.010
+ 0.004
0.003
0.002
1.330
0.918
1.261
0.907
0.654
0.343
0.254
0.320
48
+ 0.240
0.073
0.211
0.075
+ 0.009
+ 0.007
+ 0.011
+ 0.008
ETTh2
168
7.174
2.329
7.053
2,290
0.962
0.696
0,410
0,418
+ 0.449
0.049
+ 0.428
+ 0.095
+ 0.129
+ 0.054
+ 0.010
+ 0.005
4.859
1.863
5.070
1.914
1.204
0.789
0.449
0.447
336
+ 0.268
0.043
+ 0.336
- 0.083
+ 0.158
0.050
- 0.011
+ 0.006
5.656
2.012
6.311
2.049
1.284
0.810
0.496
0.482
720
+ 1.053
0.186
2.057
0.225
+ 0.145
+ 0.033
0.008
+ 0.002
6.408
2.077
5.627
1.965
1.338
0.809
0.471
0.481
960
+ 2.039
0.242
+ 1.670
0.314
+ 0.535
+ 0.127
+ 0.015
+ 0.008
0.443
0.437
0.412
0.426
0.564
0.477
0.403
0.392
24
0.043
0.035
0.026
0.024
+ 0.015
+ 0.009
0.006
+ 0.005
0.453
0.472
0.420
0.455
0.615
0.531
0.328
0.371
48
+ 0.034
0.018
+ 0.028
- 0.018
+ 0.093
+ 0.049
+ 0.010
+ 0.007
El1m
96
0.603
0,581
0.572
0.553
0.668
0.555
0.379
0.406
+ 0.051
0.027
+ 0.039
+ 0.030
+ 0.031
+
+ 0.011
+ 0.007
0.849
0.702
0.789
0.677
0.795
0.623
0.451
0.445
288
+ 0.095
0.051
+ 0.069
+ 0.039
+ 0.070
0.032
+ 0.016
+ 0.008
0.860
0.726
0.958
0.758
1.657
0.890
0.555
0.511
672
+ 0.057
0.026
0.183
0.076
1.116
+ 0.290
0.011
+ 0.008
14.613
1,948
5.592
1.497
0.556
1344
+ 26.108
1.655
+ 7.032
- 0.671
- 0.061
+ 0.020
0.279
0.372
0.198
0.310
0.242
0.332
0.176
0.285
24
0.007
0.003
0.005
0.003
0.005
+ 0.006
+ 0.002
0.001
0.309
0.388
0.245
0.343
0.275
0.352
48
+ 0.007
0.004
+ 0.009
+ 0.007
+ 0.007
+ 0.006
+ 0.001
+ 0.001
0,333
0,410
0.285
0.375
0.218
0.320
168
ECL
+ 0.016
+ 0.012
+ 0.006
+ 0.004
+ 0.002
+ 0.001
0.326
0.406
0.304
0.393
0.241
0.337
336
+ 0.004
0.001
0.019
+ 0.013
+ 0.005
+ 0.002
0.420
0.467
0.378
0.443
0.303
0.383
720
0.094
0.058
0.083
0.056
+ 0.012
+ 0.011
0.399
0.455
0.360
0.433
960
+ 0.022
0.017
+ 0.037
+ 0.027
+ 0.019
+ 0.015
0.224
0.207
0.223
0.206
0.223
0.204
0.208
0.197
M4
average
0.004
0.004
0.003
+ 0.001
+ 0.001
0.001
0.001
18

---

Published as a conference paper at ICLR 2022
ESRNN*
trai dats Uoi MMd
leak Vii
1331 2313
Teal: 0810
train dhts
Loii Vide
Density
34
24-
LSTNet* Density
tram date
11a7 deta
T1E7 681a
18
Tram cots
04
ReviN Densily
R S. 2 F.R.
05-
(a) model input
(b) normalized input
(c) model output
(d) denormalized output
Figure 8: Effect of RevIN on distribution discrepancy on training and test data compared to
existing dynamic normalization methods. We compare RevIN with LSTNet", which adds the autoregressive linear bypass module of LSTNet to the baseline and ES-RNN", which adds the expo- nential smoothing of ES-RNN to the baseline. The analysis is conducted on the ETTh, dataset with a prediction length of 960 using N-BEATS as the baseline. From left to right, the columns compare the training and test data distributions of each step of the sequential process in each method.
A.6 COMPARISON WITH EXISTING DYNAMIC NORMALIZATION METHODS
We compare RevIN with the dynamic normalization methods proposed in LSTNet LLai et al. 2017) and ES-RNN (Smyl, 2020p. Similar to adding RevIN to the baseline model, we add the autoregres- sive linear bypass module of LSTNet and the modified Wolt-Winterrs exponential smoothing of ES- RNN to the baseline model, respectively. As shown in Table RevIN consistently achieves the best performance among the normalization methods adopted on N-BEATS by significant margin. When we replace RevIN with the other normalization methods, the autoregressive linear bypass module of LSTNet (LSTNet") ) also consistently reduces the prediction error compared to the baseline. How- ever, the performance improvement is smaller than our method. For example, when the prediction length is 960 on the ETTh, dataset, N-BEATS shows an average error of 6.408, and LSTNet" duces the error to 5.627. But this is still much worse than RevIN, which reduces the error to 0.471. Similarly, when the prediction length is 1344 on the ETTm; dataset, the baseline shows an average error of 14.613 and LSTNet" largely decreases the error to 5.592, but RevIN more significantly de- creases the error to 0.631. In the case of ES-RNN', the training of the model is unstable, failing to converge for several cases. Also, ES- ES- often degrades the baseline performance, e.g., when the prediction length either 24 or 48 on the ETTh; and ETTm; datasets. It significantly reduces the enror for long prediction length much better than LSTNet", but still worse than RevIN, for example, when the prediction length is 960 on the ETTh; datasct.
Additionally, we further analyze the data distributions of the dynamic normalization methods on the ETTh; dataset, as shown in Fig. 8 We compare the training and test data distributions of each step of the sequential process in each method.
ES-RNN' shows the distributions of (a) the original model input, (b) the normalized input where the level and scasonality removed by its proposed method, (c) the model prediction output, (d) the denormalized output where the level and seasonality is multiplied back to the original distribution.
LSTNet" shows the distributions of (a) the original model input, (b) the same original input since the method does not transform the input data before feeding them to the main prediction model, the model prediction output (c) before, and (d) after adding the output of the autoregressive network.
RevIN shows the distributions of (a) the original model input, (b) the normalized input by RevIN, (c) the model prediction output, and (d) the denormalized output by RevIN.
19

---

Published as conference paper at ICLR 2022
In Fig. a). the original training and test data show discrepancy in their distributions. Also, they have several peaks, not being centered on the mean. This implies that sequences the data will have different mean values. In Fig. 8()), both RevIN and ES-RNN" transform data distributions into mean-centered distributions. Particularly, ES-RNN" extremely concentrates the distribution on the mean, with only small variance. Both RevIN and ES-RNN' result in data sequences with similar statistics, thereby alleviating the distribution shift problem in the input data. This leads to outstanding performance long prediction sequences, contrast LSTNet". LSTNet" cannot resolve the distribution discrepancy because it does not have any module that can change the input statistics. Also, in LSTNet'. the distributions the model output (Fig. g c))) completely differ from the input data distributions (Fig. (())). The method cannot make the input and output distribution
be consistent. In addition, proposed autoregressive model barely affects the model output distributions, shown in Fig. [ (ccd);; there is almost no difference between the model output and the final output distributions.
Most importantly, the distributions of the final output (Fig. (()) significantly differ from the original data (Fig. a(a)) LSTNet'. Similarly, although ES-RNN alleviates the distribution discrepancy in the input data, it fails to return the model output (Fig. 8 d)) back to the original distribution (Fig. Bla)), especially with the test data. These results imply that LSTNet" and ES "S-NNN" fail to learn the appropriate data distribution, and this could be the main reason why their prediction error higher than RevIN. On the other hand, in RevIN, the distributions the final output (Fig. - 8 d)) are successfully returned the original distributions (Fig. a)). Also, with RevIN, the input and output of the model maintain consistent distributions, as well as the training and test data be overlapped. As result, RevIN shows superior performance than the other dynamic normalization methods.
Table 10: Additional results on the comparison with classical and state-of-the--ar normaliza- tion methods in Table 3 in the main manuseript. The mean squared eтror measured on the ETTh, ETTh,, ETTm; and ECL datasets. T, indicates the prediction length. RevBN is the modi- fied version of RevIN, where input normalization replaced by batch normalization.
Dataset
T
Min-max
z-score norm
Layer norm
DAIN
Batch norm
RevBN
Instance norm
RevIN (Ours)
norm
0.885
0.959
0.472
0.652
0.451
0.574
0.989
0.322
48
1.010
0.898
0.741
1.389
0.557
0.649
0.999
0.373
ETTh:
168
1.074
0.953
0.871
0.996
0.851
0.717
0.946
0.515
336
1.083
0.969
0.827
0.979
0.828
0.775
1.078
0.509
720
1.226
0.978
1.184
1.014
0.916
0.705
0.986
0.567
960
1.224
1.043
1.303
1.032
1.691
0.779
1.090
0.697
24
2.659
3.152
0.478
1.437
0.336
0.550
2.976
0.192
48
2.772
3.232
1.335
1.476
1.018
1.058
3.175
0.244
ETThg
168
2.987
3.329
4.092
1.982
6.206
0.729
3,240
0,419
336
2.914
3.288
4,207
2.631
5.422
0.546
3.186
0.452
720
3.092
3.031
5.822
2.954
7.062
1.552
3.079
0.492
960
3.308
3.087
5.204
2.802
7.755
2.148
3.145
0.465
24
0.981
0.930
0.515
0.431
0.477
0.680
0.926
0.395
0.998
1.005
0.555
0.747
0.489
0,531
1.005
0.337
ET1m;
96
1.035
1.016
0.502
0.672
0.505
0.601
1.021
0.388
288
0.974
0.988
0.773
0.877
0.677
0.656
1.056
0.444
672
1.157
1.029
0.795
1.043
0,620
0,670
1.157
0.549
1344
1.320
1.274
2.488
1.348
1.147
0.991
1.329
0.602
0,370
0.313
0.294
0.348
0.301
0.304
0.307
0.174
48
0.334
0.326
0.310
0.387
0.319
0.331
0.313
0.194
ECL
168
0.374
0.335
0.343
0.347
0.320
0.327
0.333
0.220
336
0.378
0.338
0.358
0.357
0.337
0.369
0.335
0.244
720
0.746
0.417
0.371
0.366
0.374
0.440
0.378
0.294
960
0.387
0.377
0.379
0.381
0.422
0.414
0.386
0.329
20

---

Published as a conference paper at ICLR 2022
A.7 ADDITIONAL RESULTS ON COMPARISON WITH EXISTING NORMALIZATION METHODS
This section provides complete results that compare with existing normalization methods, which are not included in the main manuscript due lack of space. The forecasting error of RevIN and existing normalization methods are evaluated on the ETTh; ETThy, ETTm1, and ECL datasets in Table TO] RevIN consistently outperforms the other normalization methods across all datasets, Interestingly, when the denormalization step is added to batch normalization (RevBN) as RevIN, the model better forecasts long sequences than batch normalization (Batch norm), e.g., when the prediction length is 960. The denormalization step of RevIN plays a critical role in improving model performance by restoring the model prediction to the original distribution. However, a denormalization step cannot be added to DAIN since it has the Hadamard multiplication operation in the last step, which is not reversible when the input and prediction sequence lengths are different. These differences could be the reason for its worse performance compared to RevIN despite that DAIN requires higher computational costs and a larger amount of model parameters.
A.8 ALGORITHM OF REVERSIBLE INSTANCE NORMALIZATION
Algorithm summarizes the procedure of the proposed approach. Reversible instance normalization consists of the normalization (line 3-4) and denormalization layers (line 6-7). It transforms the input and output of a model using identical statisties. As RevIN is generally applicable, go Algorithm (line 5) can be any arbitrary deep neural network.
Algorithm 1: RevIN, applied to input x and output y of module in the model.
Input T2 IR,, the input sequence length; Tet R!, the k-th feature time step of the i-th item in a mini-batch; 7, 8 E RK learnable parameters for RevIN; ge, module the model parameterized by Output: 8,8,0 0.
(7)
T2 ( HT T. 1
Compute
instance mean
Compute
o7 - + Eラ=1 (xk) - 1T)²
b instance variance
NNrmalize
1kt + #et -PT
b normalization
Vattc
Transform
1 7k-- -(()
(6)
+ 5x = RevIN sx (x(1))
scale and shift
Predict
ÿ + go (z)
b forward propagation
(4)
yi -3
(c)
Retransform
Vkt
reverse scale and shift
7
Denormalize 1 + + Diet
o
+ RevIN dn 7.9
>> denormalization
A.9 THEORETICAL JUSTIFICATION OF REVIN AGAINST DISTRIBUTION SHIFT
Let (1) RR denote time series comprising K variables of length Tи. Consider univariate case where K = 1 without the loss of generality. Then, (3) RT denotes the i-th time series in the data. Consider training and test data, whose distributions are denoted as Pira and Pttt. re- spectively. We consider distribution shift problem where the training and test data have different distributions (Du et al. 2021). That is,
(ira(x) * Prec(z).
(4)
In our work, we consider the distribution shift problem in terms of the mean and the variance. Then, the distribution shift problem can be redefined as
E[ A E[Fest] Var[ A Var[srar], for Ttro Peras Test
Ptst-
Let's ' assume that the given training and test data suffer from the distribution shift problem in terms of the mean and variance. In order solve this problem, RevIN first normalizes training sample
21

---

Published as a conference paper at ICLR 2022
Pira- Mathematically, training sample is transformed
5 x1 E[
+ 8.
/arlx
By the laws of expectation and variance,
B[ = 3 and Var[s!0] =72 for all i.
By symmetry, this also holds for x(01 Pist, for all i. Therefore, the mean and variance of the training and test data distributions become identical. Thus. by definition (Eq. 5 the distribution shift problem for the training and test data solved by the first step of RevIN.
Given the normalized time-series data, the forecasting model parameterized by 8, : RT., predicts the corresponding subsequent future values, ý = So (()) Then, the denormalization step of RevIN returns the stattttttonary information of the original data, i.e., [E(x(() and Var(a:(0 in Eq to the prediction output so that model does not have to reconstruct them from the normalized input. In summary, the model prediction ÿ(() is denormalized as
5()) ?
g0
Var[x(0] B[r(
By the laws of expectation and variance, the mean and variance of (() can be expressed as
B = A + E[x and Varlg")] À Var[x(").
The denormalization step allows the mean and variance of the final prediction values to be expressed the difference from the input statistics. Here, since the input data z(() and the groundtruth future values are consecutive sequences, we can assume that their difference in the mean and variance be expressed Eq. Eq.9aa as well. Under this assumption, the model adopting RevIN only needs to capture the difference from the input statistics, A and X, to accurately predict the statistics of the future values. In conclusion, through the normalization and denormalization steps of RevIN, a model can focus on learning the offset from the input distribution to the output distribution by removing their common non- stationary statisties.
A.10 CALCULATION DETAILS ON FEATURE DIVERGENCE
This section explains how the feature divergence is computed in Section vious work (Pan et al. 2018), we calculate the average feature divergence between the training and test data using symmetric KL divergence, assuming that the output features of the model layer will follow a Gaussian distribution with mean and variance o². Then, the equation for the feature divergence of the k-th feature f& can be expressed as
Following the pre-
D(Stiin (1stt
K L
(OS) ( KL 10SE ( [
(10)
12 + (HL
where KLU log k
(11)
ok
B
B2
2
A.11 ADDITIONAL EXPERIMENTAL DETAILS
We train and evaluate the models using the following seeds: 12, 22, 32, 42, and 52. The experiments using N-BEATS and Informer as the baseline are performed on NVIDIA TITAN RTX. and the experiments using SCINet are conducted on NVIDIA TITAN Xp. Following the multivariate time- series forecasting settings of the previous studies (Zhou al. 202T Liu et al. 2021). we select input sequence length from two days (2d), 4d, 7d, 14d, 15d, 20d, 28d, 30d for the hourly datasets, i... the ETTh, ETTh2, and ECL datasets, and from half day, 1d. 3.5d, 7d for the ETTm, dataset. Particularly, we set the ratio of input length to prediction length to be smaller from 2.0 to 0.35 as the prediction length becomes longer. In the case SCINet, when the prediction length is 720, we set its input length to be 736, unlike the other baselines. It is because the original paper of the method requires its input sequence length to meet a specific condition. To be specific, the input length needs to be multiple of 32 due its hierarchical architecture (Liu et al. 2021).
22

---

Published as a conference paper at ICLR 2022
A.12 REPRODUCTION DETAILS FOR BASELINE MODELS
This section describes the implementation details of the baselines, Informer, N-BEATS, and SCINet. Note that compare each baseline model and RevIN using the same hyperparameters except for the presence of RevIN. We exactly follow the experimental settings of the baseline models by using their official code to conduct experiments, except for N- N-BEATS that have no officially released code. We reproduce the model and set hyperparameters as stated in the original N-BEATS paper.
Informer. We use the official open-source code of Informer If provided, we follow the same hy- perparameter settings in training the network, e.g., hidden dimension of the network or the learning rate. For the ECL dataset, detailed hyperparameter settings are not officially provided in Informer; we use the same settings with the ETTh, dataset.
N-BEATS. We reproduce N-BEATS using the PyTorch framework (Paszke et al. 2019). We follow the same hyperparameter settings of the N-BEATS-I model in the original paper. We train N-BEATS to minimize the mean squared error between the model prediction and groundtruth values. For fair comparison with the other baselines, we use single model instead of using the ensemble method originally proposed in the N-BEATS paper. Since N-BEATS is a model tailored to univariate time- series forecasting,
flatten each multivariate input sequence into a univariate sequence having single dimension for the feature before feeding it to the model. Additionally, we conduct a grid search for the learning rate of N-BEATS with the range of [le-5, 1e-3] and train the model using the weight decay with the factor of 0.001 to stabilize training.
SCINet. We follow the experimental settings provided in the official code P of SCINet.
A.13
ADDITIONAL QUALITATIVE RESULTS
In Fig. 9. we illustrate the additional results comparing the predictions of RevIN and the baselines. Overall, the prediction results of the baselines are inaccurately sealed and shifted. However, RevIN shows remarkable performance, consistently improving the baselines predict more precise results. With RevIN, the forecasting results are better aligned with the groundtruth.
A.14 COMPLETE QUANTITATIVE RESULTS
Table II provides the standard deviation values of five independent experiments to compare long sequence forecasting performance in Table 2 in the main manuscript. Table 12 shows the complete results of the comparison of the forecasting errors between the baselines and RevIN in Table I the main manuscript. They include the standard deviation of five runs and the performance reported in the original papers of the baselines. RevIN shows significant performance improvement compared to the state-of-the-arr forecasting baselines.
Table 11: Standard deviation values of the five runs for the comparison of long sequence fore- casting performance Table in the main manuseript.
Prediction length
48
168
336
720
960
Metric
MSF
MAE
MSE
MAE
MSF
MAE
MSE
MAE
MSE
MAE
Informer
0.056
0.035
0.052
0.024
0.085
0.031
0.037
0.024
0.034
0.022
+ RevIN
0.030
0.008
0.051
0.030
0.073
0.026
0.067
0.033
0.041
0.022
N-BEATS
0.042
0,030
0.056
0.027
0.081
0.037
0,072
0.029
0.067
0.030
+ RevIN
0.006
0.003
0.014
0.007
0.011
0.006
0.041
0.020
0.027
0.012
SCINet
0.008
0,007
0.063
0.042
0.115
0.064
0,033
0.022
0.049
0.029
+ -RevIN
0.002
0.001
0.013
0.007
0.022
0.010
0.028
0.017
0.015
0.008
23

---

Published as a conference paper at ICLR 2022
Ground Intermer
Giound
Giound fect Now
N-BEATS
Informe Hexit
Wit4ls Newit
ETTh1
BEATS MOEATS Beatl
DevP)
Dev
-23
-12
Ground
Grzund
Dirund
ETTh2
Ground Buth
Ground Truth
Cround Truth
Informer
4-EAAT
Deafi
Rextl
155
156
125
122
150
175
25
135
159
115
Ground Buth
Ground Truth
Ground Truth
Intorme
4OCATS
SC Sciaee
Informa Hoxtt
NeEArS Hextt
5C Norhh
ETTm1
Ground
Ground
Ground
Informer
SCRit
Informe Revil
4BGATS Revil
Sci Revcl
Ground Bith Informe
Ground Trlhh
Dround
SC Bitt Revc
Inferme- - Bevil
4OEATS ReviN
N
ECL
SCreet
Informer Bevii
N-BEATS ReviN
SCRiet Bevh
15
1.5
-19
-24
-10
Figure 9: Additional multivariate time-series forecasting results comparing RevIN and state- of-the-art baselines. The analysis is conducted on the ETTh, ETTh,, ETTm;, and ECL datasets. We set the prediction length 168 for the hour datasets and 288 for the ETTm, dataset.
24

---

maolished a as a Cauterenee a paper
a at ICLR 2022 2022
SCINcI+
MAE
0.347
1: 0.002
0.389
0.000
0.416
= +
0.447
+ 0.000
0.505
+ 1.0004
0.526
+ 0.005
0.263
1 1.0000
0.302
+ +..006
0.378
+ 0.003
0.403
0.002
0..45
+ 0.000
0.462
+ 0.004
0.196
1 1.001
0.222
+ 0.002
0.247
+ 0.001
0.321
+ 100.002
0.380
+ 0.000
0.422
+ 1.0003
0.207
+ 1.0.01
0.222
+ 0.001
0.249
+ +.000
0.262
4: 1.0001
0.281
1.0000
0.292
# 0.002
Table 12: Complete results for the comparison of the forecasting errors in Table Tin the main manuscript. The mean and standard deviation of the five independent experiments are recorded. 1 denotes the reported performances for the baselines in their original paper.
MSE
0.308
1 0.003
0.365
- 0.000
0.406
1 0.003
0.467
+ 0,000
0.507
+ 0.006
0.545
+ 0.010
0.180
1 0.004
0.231
+ 0.006
0.337
+ 0.007
0.357
+ 0.003
0.411
1 0.003
0.438
1 0.007
0.106
1 0.002
0.135
F 0.003
0.162
+ 0.001
0.265
+. 0.003
0.357
+ 0.004
0.412
1.000
0.
1 0.001
0.126
+ 0.001
0.153
+ 0.003
0.162
+ 0.001
0.183
+ 0,003
0.200
f 0.003
MAE
0.373
1 0.000
0.459
0.021
0461
0.513
#0006
0.571
0.013
0.574
10.014
0.295
0.027
0.422
= 00027
0.518
0.025
0.583
+ 0.030
0.944
00144
1.066
#0055
0.231
10.00
0.262
#000
0.291
0.013
0.419.
+ 0.004
0.756
0.025
1.044
10100
0.24
1.005
0.265
0.007
0.281
+ 0.005
0.308
= 0.00
0.333
0.004
0.330
0.008
SCINet
MSE
0.338
10012
0.436
#0.02
0459
#
0.527
+
0.5%
1.015
0.604
#
0.199
1.02
0.350
0.559
0.044
0.664
4.0.73
1.546
0.377
1.862
#0.55
0.130
10000
0.155
#0004
0.195
#
0.361
0.000
1.020
0.040
1.841
#.242
0.138
+ 0.004
0.163
#
0.177
# 1.0003
0222
# 0.00
0.234
1,000
0.255
10011
MAE
0.311 0.348
0.364 0.388
0.491
0.494
0.582
0.271
0.341
0.509
0.608
0.761
0.226
0.261
0.291
0.462
0.527
SCINet
| MSE
0.497
0.491
0.612
0.183
0.259
0.528
0.648
1.074
0.127
0.150
0.190
0.417
0.554
N X-BEATS+RevIN
MAE
0.373
1 0.004
0.400
- 0.002
0.452
1 0.014
0.483
+ 0.008
0.534
1 0.018
0.575
- 0.000
0.276
1 0.002
0.320
= 0.008
0.418
- 0.005
0.447
+ 0.006
0.482
+ 01.002
0.481
= 0,008
0.392
1 0.005
0.371
1 0.007
0.406
+ 0.007
0.445
+ 0.008
0.511
+ 0.008
0.556
- 0.020
0.285
1 0.001
0.301
- 0.001
0.320
+ 0.001
0.337
+: 10,002
0.383
+ 0.011
0.398
+ 0.015
MSE
0.330
1 0.006
0.372
+ 0.001
0,466
= 0.030
0.515
+ 0.013
0.576
+ 0.035
0.678
+ 0.019
0.192
1 0.003
0.254
= 0.011
0.410
+ 0.010
0.449
+ 0.011
0.49%
+ 0.008
0.471
0.015
0.403
1 0.006
0.328
1
0.379
0.011
0.451
+ 0.016
0.555
+ 0.011
0,631
+ 0.061
0.176
1 0.002
0.194
+ 0.001
0.218
+ 0.000
0.241
4: 0.005
0.303
0.012
0.325
+ 0.019
MAE
0.505
1 0,012
0.542
- 0.041
0.782
1 0.064
0.743
+ 0.042
0.926
+ 0.066
0.932
1 0.120
0.472
0.101
0.918
0.03
2.329
1 0,049
1.863
+ 0.043
2.012
1 0.186
2.077
1 0.242
0.437
1 0.035
0.422
F 0,018
0.581
4 0.027
0.702
4 0.051
0.726
+ 0.026
1.948
1.65
0.372
1 0.00
0.388
+ 0.004
0.410
+ 0012
0.406
+ 0.001
0.467
+ 0.058
0.455
+ 0017
N-BEATS
MSE
0.478
1 0.022
0.536
- 0.060
1.005
1 0.146
0.932
+ 0.079
1.389
1 0.230
1.383
1 0.380
0.403
1 0.185
1.330
+ 0.240
7.174
# 0.449
4.859
+ 0.2668
5.656
1 1.053
6.408
- 2.039
0.443
1 0.043
0.453
1 0.034
0.603
+ 0.051
0.849
+ 0.095
0.860
+ 0.057
14.613
26.108
0.279
1 0.007
0.309
+ 0.007
0.333
+ 0.016
0.326
4 0.004
0.420
+ 0,094
0.399
+ 0.022
Informer #RevIN
MAE
0.472
1 0.025
0.547
0.015
0.561
1 0.024
0.758
+ 0.059
0.717
+ 0.036
0.715
= 0.025
0.325
1 0.006
0.404
= 0.014
0.649
1 0.026
0.673
0.023
0.546
+ 0.025
0.570
0.018
0.352
0.010
0.391
1 0.006
0.411
= 0.006
0.502
+ 0.015
0.550
= 0.031
0.632
= 0.019
0.257
1 0.001
0.279
= 0.003
0.354
0.007
0.414
+ 0.015
0.700
0.076
0.741
+ 0.031
MSE
0.504
1 0.049
0.646
+ 0.009
0.655
1 0.055
1.058
+ 0.119
0.926
+ 0,057
0.902
+ 0,033
0.238
1 0,010
0.361
+ 0,023
0.859
+ 0.072
0.890
+ 0.057
0.576
+ 0.044
0.600
+ 0,033
0.309
0,020
0,390
1 0.008
0.405
+ 0.013
0.563
+ 0.022
0.663
+ 0,082
0.824
+ 0,039
0.148
1. 0.001
0.171
+ 0.004
0.261
+ 0,010
0.356
+ 0.026
0.834
0.122
0,894
+ 0.047
MAE
0.536
1 0.025
0.668
0.055
0.853
- 0.045
0.909
0.055
0.945
+ 0.009
0.990
0.052
0.520
0.071
1.200
0.048
2.558
0.113
1.844
0.102
1.529
0.085
1.411
0.033
0.382
0.017
0.486
0001
0.554
0.027
0.738
4 0.028
0.760
0.034
0.823
0.040
0.358
0.005
0.386
0.005
0.423
0.010
0.473
+
0.719
0.07
0.750
0.042
Informer
MSE
0.350
+ 0.041
0.772
0.122
1.138
+
1.278
0.129
1.357
+ 0.056
1.470
# 0.124
0.450
1 0.099
2.171
= 0.094
8.157
- 0.631
4.746
0.455
3.190
# 0.326
2.972
0.183
0.330
1 0.021
0.499
0.605
+ 0.033
0,906
+ 0.039
0.943
+ 0.062
1.095
0.065
0.250
+ 0.005
0.300
- 0.010
0.345
= 0.012
0.429
+ 0.041
0.851
+ 0.088
0.930
+ 0.075
MAE
0.349
0.625
0.752
0.873
0.896
0.665
1.001
1,515
1.340
1.473
0.369
0.503
0.614
0.786
0.926
0.393
0.424
0.431
0.443
0.548
Informer
MSE
0.577
0.685
0.931
1.128
1.215
0.720
1.457
3.489
2.723
3.467
0.323
0.494
0.678
1.056
1.192
0.344
0.368
0.381
0.406
0.460
Methods
Metric
24
48
168
336
720
960
24
48
168
336
720
960
24
48
96
288
672
1344
24
48
168
336
720
960
ECL
25


Reviews:
[{"review_id": "hzc2um10xE1", "reviewer": "Reviewer_J9jm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe idea behind RevIN is straightforward but seems to work well, outperforming min-max normalization, batch normalization, layer normalization and instance normalization. RevIN is also cheap to compute and model-agnostic.\n\nWeaknesses:\nW1: The method seems ad-hoc with no theoretical justification. Is is possible to provide a proof for a simple case (e.g. 1-dimensional time series) that RevIN indeed handles distribution shift over time?\n\nW2: Experiments are on 4 data sets only. To validate the merit of RevIN, I would like to see experimental results on more real-world data sets, for instance those taken from UCI Repository.\n\nOther remarks:\nR1: In the first paragraph of Section 3.1, it looks like output sequence $Y$ needs to have the same length as the input sequence $X$. Does this have to be the case?\n\nR2: ECL data set originally has a lot of dimensions and a few instances. After preprocessing, what are the length and dimensionality of the new time series?", "summary_of_the_paper": "The paper proposes RevIN that performs (a) instance normalization on input time series and (b) reversible instance normalization on output time series using normalization statistics of the input. By doing this, the paper claims to handle distribution shift in time series and hence yield better forecasting results.  ", "summary_of_the_review": "The paper is easy to follow as the main idea is straightforward. The technical contribution however seems ad-hoc and incremental. A proof for a simple case would make the paper more convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MlvogQVnOYA", "reviewer": "Reviewer_uPmn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- Addressing distribution shift is a relevant task to the ICLR community and of crucial importance to deploy deep-learning based model time-series forecasting;\n- Despite being simple, the proposed approach is technically sound, easy to implement, and empirically validated on various datasets, split and prediction windows length;\n- The experimental evaluation is thorough and allow one's to evaluate the importance of the type of normalization and the effect de-normalization module.\n\n**Weaknesses:**\n- In the recent literature [1], complementary metrics are used to evaluate the shape error (DTW) and temporal error (TDI) separately. Here, the authors only evaluate their method with MSE/MAE metric.\n\n&nbsp;\n\nSmall typos:\n- introduction of Section 3 : RevIN acronyme used two times in a row;\n- page  8: \"we the batch normalization\" => \"we use the batch normalization\".\n\n&nbsp;\n\n-----------\n[1] V. Le Guen & N. Thome. *Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models*. In NeurIPS 2019.", "summary_of_the_paper": "This work proposes to use a normalization method to address temporal distribution shift in time-series forecasting. The proposed approach, *RevIN*, consists of two steps: instance normalization on input sequences and \"de-normalization\" of output sequences by re-using statistics (mean and variance) computed during the normalization step. Experiments are conducted on two time-series datasets (ETT and ECL) with varying splits and prediction windows lengths. Results show that RevIN used on top of deep-learning based methods for time-series forecasting (Informer, N-BEATS and SCINet) improves prediction performances, in particular for long sequence prediction. Finally, they conduct an empirical comparison with other normalization methods, such as batch normalization and min-max normalization, to evaluate the adequateness of instance normalization and of the denormalization step. ", "summary_of_the_review": "-----\n**Post-rebuttal comments**\nThe authors have addressed my concerns and engaged with other reviewers concerns. Their rebuttal includes several complementary analysis and experiments which helps to better validate the quality of the approach. Even though the proposed approach is straightforward, the idea behind RevIN is technically sound and the thorough experimental evaluation is conclusive. Thus, I maintain my rate in favor of an acceptance. \n------\n\nWhile instance normalization is a simple and well-known method in the distribution shift literature, its application for time-series forecasting is sound and empirically validated in this paper. Overall, I tend to vote for accepting but authors should provide complementary metrics raised in weaknesses to further analyze the effect of their approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4sXWNIJwF6z", "reviewer": "Reviewer_V5fo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some strengths:\n* The approach is interesting and well motivated, and the experiment results useful, striking, and very extensive.  \n* It is a good data point to have to see the benefit of this type of normalization across modern state-of-the-art models, and point out this method in the community, which has typically just used global per-series normalization.  \n* The results conclusively show the benefit and the method provides a simple modification that can be broadly used in many cases, and the detailed results in the appendix that also include original reported scores and std. dev. are really great to have.  \n* The thorough comparisons to different normalization approaches are also helpful.  \n* Also the appendix details and code provided are very helpful for reproducibility\n\n\nWeaknesses:\n1. Novelty:\n\nHowever, the biggest issue is that the proposed work is not very novel, and leaves out the related work on \"sliding window normalization\" doing essentially the same thing.  I.e., \"sliding window normalization\" is commonly used in practice for time series forecasting and has been used in the literature on time series forecasting as well.  Sliding window normalization typically just takes the current instance time window mean and std. dev. to normalize the inputs and de-normalize the outputs after getting the predictions.  For example, see the following, and references:\n\nOgasawara, Eduardo, et al. \"Adaptive normalization: A novel data normalization approach for non-stationary time series.\" The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE, 2010.\n\nOverall it is a simple and straight-forward method, and, the only difference from the classical sliding window normalization approach is the additional learnable parameters (gamma and beta).  However, such learnable parameters are not new on their own either and already used in other general normalization approaches as well as forecasting specific - including instance normalization for forecasting, like DAIN (albeit in that case the difference is it doesn't use the built-in denormalization).\n\nI think the key main missing experiment and result needed is to demonstrate the impact of the learnable parameters, gamma and beta.  I.e., in each of the experiments, if the learnable parameters are excluded what is the impact?  This amounts to an additional ablation study - the same proposed method is used, but with gamma and beta excluded (or fixed to 1 and 0 respectively so they have no effect).\n\nAdditionally, I think the hybrid class of models are very similar conceptually and should ideally be compared with and called out as an alternative approach to dynamic normalization.  I.e., the hybrid models fit a simple model essentially based on the recent history of each series combined with a more complex deep learning model - and one of the key motivations for this was to dynamically normalize each series using this simple model.  For example the author of ES-RNN explicitly said that the idea of using the exponential smoothing model was to enable per-series normalization before applying the deep learning part of the model - and it fits an exponential smoothing model to set the level and seasonal pattern (and this is inherently based on limited recent history which is controlled by the smoothing parameters that are learned as part of modeling).  Similarly LSTNet and other methods before and after it simultaneously fit a linear auto regressive model and add it to the neural net part of the model to get the final output.  Both of these kind of approaches are essentially normalizing the time series for the deep learning part, as is done here, and this is done dynamically and potentially capturing nonstationarity of the model as well in the normalization, which isn't done with the proposed approach.  \nES-RNN is Smyl 2020 in the cited references and LSTNet is: Lai, Guokun, et al. \"Modeling long-and short-term temporal patterns with deep neural networks.\" SIGIR 2018.   \nIt would be best to compare with these (including the distributions) / compare this approach in combination with the baseline models as well - e.g., simultaneously fitting a simple baseline AR model + or * the deep learning model to get the output, as this is an alternate way to perform this kind of dynamic normalization, so it would be important to see how it compares to the proposed one.\n\n\n2. \nIt seems a key hyper parameter is the window length, as clearly if the window length is too short, the instance normalization would hardly be stable (and enable accurate prediction with denormalization to the full horizon).  However, this is not discussed and related experiments not performed.  \n\nI.e., it would therefore be best to show the impact of changing window lengths - i.e., hyper parameter sensitivity as this is varied from a very short window to a longer and longer window (additionally at some point too long windows would also be problematic as it would not be adaptive enough and also limit the number of data points)\n\nIt's also important to make clear how the window length was selected for the experiments.\n\n\n3. Additionally issues / open questions around the formulation and the experiments:\n* The mathematical formulation is not precise / changes at different places. E.g., at first instances are referred to as x^(i), then suddenly referred to by x_k.  \n* Alg. 1 definition directly contradicts the description of the method and definitions in the main paper.  Namely, the main paper states that there are N time windows, each with K variables / time series (in the case of multivariate series), and that the scaling and shift parameters, gamma and beta, are in R^K, i.e., one shared value for each variable / dimension K - shared across input windows.  However, in Alg. 1 they are said to be in R^NxK - which implies a unique value per window and feature/variable is learned, which doesn't make sense (as then one wouldn't be able to apply it for future forecasts, e.g., series N+1 where we don't have a parameter).  \n* It would be best to make clear the dimensions of the different parameters, the dimensions of the inputs and outputs, and data \n* It is also not explained how the hyper-parameters tuned, and it would be \n* It would also best to show the results on a larger variety of datasets, even those used in prior work by the base methods - as the benefit might only be for this common type of data used here.\n\n", "summary_of_the_paper": "The authors propose a \"reversible instance normalization\" as an input pre and post processing procedure to improve the forecasting of any given base model - targeted at addressing the distribution shift that is common in time series data - e.g., time series are typically non-stationary.\n\nThis works by normalizing each time window input to a (deep learning) forecast model, applying the model on the normalized data, then unnormalizing the predictions to get the final predictions.   The normalization is done by subtracting the mean and dividing by the std. dev. in a current (input instance) window, followed by scaling and shifting by learnable, shared cross-instance (global), scaling and shifting parameters per input feature / variable.  \n\nThe authors perform extensive experiments to show the proposed approach significantly improves the base metric score results for 3 recent, state-of-the-art forecasting algorithms across several datasets, and further that it significantly improves over other normalization approaches, and helps align distributions between train and test windows.", "summary_of_the_review": "My main hesitance to accept the paper is the lack of novelty - as sliding window normalization is already commonly used, and the method is a very minor change. I feel this could possibly be improved with the ablation study comparing the results of the proposed method without the learnable parameters gamma and beta.  \n\nI would also like to see more analyses and discussion around the impact of the window length used, and fixes / additional details around method and experiment details.   It would also be best to see the performance on a larger variety of datasets - in case the method is only beneficial for the type of data used here - and there are more datasets from the base method papers that could be used for this, that are left out here.  Additionally it would also be useful to see the results of a hybrid approach with the baselines as a commonly used alternative to the proposed dynamic normalization approach (e.g., fit a linear AR model + the baseline model).\n\nOverall, I feel the paper provides an extensive study showcasing the benefit of the proposed normalization approach / sliding window normalization so it also provides value to the community even if lack of novelty - and highlights this approach to the community as an alternative to the global normalization that has become the standard in recent work in ML-based forecasting.  The method is simple and effective, and this brings attention to a potentially useful tool to have in the forecasting toolbox.  \n\n\n***Update:***\nI feel the authors adequately addressed most of my comments, and provided a lot of useful follow-on study and analyses and additional experiments which are much appreciated and really provide a lot more useful information.  Some of it is useful even for providing more info. on the performance of the base models such as varying the input window length to see the impact on the forecast errors.  It would have been good to see even smaller input window lengths however, as I would expect a behavior where at some point if the input window is too small the method is not beneficial, and if the window becomes too big, it should not be much different from the baselines.\n\nOverall, I feel the set of results are impressive and useful.  I agree with the authors that it's useful to bring the approach to the attention of the community (as there are some similar approaches used in some communities but not well known or used in this community), and also the motivation and perspective comes from very different places compared to the similar approaches, and the idea of how it affects the data distribution and showing results along that direction is also new compared to the prior similar approaches.  It provides a widely applicable and effective method, with thorough experiment results and study.  I also feel it could motivate future work in a direction not too well considered before.  Therefore I think it is worthy of acceptance despite the actual algorithmic approach contributed itself being incremental and simple, and given the responses and revision, decide to change my rating.\n\n \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the "reversible instance normalization" (RevIN), a method for addressing temporal distribution shift in time-series forecasting. RevIN consists in normalizing (subtracting the mean and dividing by the standard deviation) each layer of of deep neural network in a given temporal window for a given instance, and de-normalizing by introducing learnable shifting and scaling parameters. 

The paper initially received one weak accept and two weak reject recommendations. The main limitations pointed out by reviewers relate to the limited novelty of the approach, the positioning with window normalization methods and hybrid methods in times series, and clarifications on experiments. The authors' rebuttal did a good job in answering the main concerns: rV5fo increased its grade from weak reject to clear accept, and RuPmn maintained its weak acceptance recommendation.

The AC carefully read the submission. The AC considers that the idea is simple yet meaningful. The large set of experiments are well conducted and conclusive. The rebuttal successfully answers to relevant issues raised by reviewers, regarding ablation studies (for highlighting the importance of the learnable de-normalization), the impact of the temporal window, the comparison to hybrid approaches and the difference with respect to Adaptive normalization. The AC thus acknowledge that this submission draws important take-home messages for the community, and therefore recommends acceptance.
Decision: Accept (Poster)