Evolving Multi-Scale Normalization for Time Series Forecasting under Distribution Shifts

## Abstract

Abstract Complex distribution shifts are the main obstacle to achieving accurate long-term time series forecasting. Several efforts have been conducted to capture the distribution characteristics and propose adaptive normalization techniques to alleviate the influence of distribution shifts. However, these methods neglect the intricate distribution dynamics observed from various scales and the evolving functions of distribution dynamics and normalized mapping relationships. To this end, we propose a novel model-agnostic Evo lving M ulti- S cale N ormalization (EvoMSN) framework to tackle the distribution shift problem. Flexible normalization and denormalization are proposed based on the multi-scale statistics prediction module and adaptive ensembling. An evolving optimization strategy is designed to update the forecasting model and statistics prediction module collaboratively to track the shifting distributions. We evaluate the effectiveness of EvoMSN in improving the performance of five mainstream forecasting methods on benchmark datasets and also show its superiority compared to existing advanced normalization and online learning approaches. The code is publicly available at https://github.com/qindalin/EvoMSN .

## 1 Introduction

Time series forecasting can provide valuable future information and thus plays an important role in many real-world applications (Lim & Zohren, ; Sezer et al., ; Qin et al., ; Karevan & Suykens, ). Abundant studies have been carried out for accurate time series forecasting, ranging from traditional statistics-based methods (Contreras et al., ; Holt, ; Theodosiou, ; Qiu et al., ) to recent deep learning-based methods (Wen et al., ). However, accurate forecasting of time series under distribution shifts is still a challenging and under-resolved problem.

Figure: Figure 1: The marginal distribution $P(Y)$ of a time series viewed from different scales will show diverse dynamics, while the conditional distribution $P(Y|X)$ also evolves across time.
Refer to caption: extracted/5887719/problem_statement.jpg

Recently, several pioneering studies have been proposed for time series forecasting under distribution shifts by adaptive normalization approaches (Kim et al., ; Fan et al., ; Liu et al., ). These approaches first remove the dynamic distribution information from the original series, enabling the model to learn from normalized data, and then predict the future distribution and recover this information to the model’s output by denormalization. However, two limitations of these approaches can be identified. Firstly, existing methods model the dynamic of distribution $P(Y)$ merely from an instance level, neglecting the intricate dynamics spanning various scales. Real-world time series present diverse variations at different temporal scales, such as the electricity load exhibits unique patterns spanning hourly, daily, and weekly scales. The dynamic distribution from a larger scale will show the general trend of the series, while the distribution on a smaller scale presents the local variations. The visualization of distribution dynamics from a multi-scale perspective is shown in Fig. 5 in the appendix. To this end, the modeling of statistical distribution dynamics from a multi-scale perspective is needed but still lacking. Secondly, even though current normalization methods can help alleviate the influence of non-stationarity, they are incapable of handling changing input-output relationships caused by gradual distribution shifts, where the conditional distribution $P(Y|X)$ is not static. In this circumstance, an online approach is required for model updates with continuous data to adapt to the changing distribution over time. The problem can be illustrated by Fig. 1, where the distribution of a time series that is viewed from different scales will show diverse dynamics, and the input-output mapping function will also evolve across time.

To overcome the above limitations, we propose an Evolving Multi-Scale Normalization (EvoMSN) framework for time series forecasting under complex distribution shifts. Specifically, we first propose to divide the input sequence into slices of different sizes according to their periodicity characteristics, forming views of diverse temporal scales to be normalized by the slice statistics. The backbone forecasting model can thus process the normalized series that are viewed from different scales to generate multiple outputs. Then, we propose a multi-scale statistic prediction module to capture the dynamics of statistics and predict the distributions of future slices. Consequently, we denormalize the outputs of the backbone forecasting model with the predicted distribution statistics and propose an adaptive ensemble method to aggregate the multi-scale outputs. With the proposed MSN approach, the dynamics of the distribution can be well modeled to address the non-stationarity issue. Finally, we tackle the challenge of gradual distribution shifts by proposing an evolving bi-level optimization strategy to update the statistics prediction module and the backbone forecasting model online.

In summary, our contributions are as follows:

- •
We propose EvoMSN, a general online normalization framework that is model-agnostic to enhance arbitrary backbone forecasting models under distribution shifts by adaptively removing and recovering the dynamical distribution information.
- •
We propose a multi-scale statistics prediction module to model the complex distribution dynamics and enable the estimation of future distributions. An adaptive ensembling strategy is designed in the denormalization stage to aggregate the multiple forecasting outputs and realize multi-scale modeling of time series.
- •
We propose an evolving bi-level optimization strategy, including offline two-stage pretraining and online alternating learning to update the statistics prediction module and the backbone forecasting model collaboratively.
- •
We conduct comprehensive experiments on widely used real-world benchmark datasets with various advanced forecasting methods as the backbone. Experimental results demonstrate the effectiveness of the proposed method in boosting the forecasting performance under distribution shifts and the superiority compared to other state-of-the-art normalization methods and online learning strategies.

## 2 Related Works

### 2.1 Time Series Forecasting

Traditionally, time series forecasting is carried out by utilizing statistical methods, such as the auto-regressive moving average model (Contreras et al., ), the exponential smoothing method (Holt, ), and decomposition-based methods (Theodosiou, ; Qiu et al., ). Recent decades have witnessed the fast development of deep learning-based methods to achieve accurate time series forecasting. The family of transformer models has shown great effectiveness in capturing both temporal dependency and variable dependency by applying the self-attention mechanism (Wen et al., ), where successful examples include Informer (Zhou et al., ), Autoformer (Wu et al., ), FEDformer (Zhou et al., ), Pyraformer (Liu et al., ), and so on. However, some recent research questions the effectiveness of transformers in time series forecasting and shows a simple linear model can achieve comparable or even superior performance (Zeng et al., ; Li et al., ; ). Such concerns are responded by the latest studies on transformers, which propose constructing transformers with patching approaches and showcase that transformer architecture still has its advantages in time series forecasting (Nie et al., ; Zhang & Yan, ; Chen et al., ). In addition to transformers, convolutional neural networks also achieve state-of-the-art performance in time series forecasting by extracting variation information along the temporal dimension (Bai et al., ). Multi-scale isometric convolution network (Wang et al., ) is proposed to learn both the local features and global correlations from time series. TimesNet (Wu et al., ) and PDF (Dai et al., ) propose to transform the 1D time series into 2D tensors based on the intrinsic periodicity characteristics, which enables the convolutional neural network to model both the intra-period and inter-period variations from a multi-scale perspective.

### 2.2 Time Series Forecasting under Distribution Shifts

Even though the above-mentioned methods are well-designed, they may still suffer from the complex distribution shifts problem. The inevitable temporal non-stationarity and the discrepancy between the training and testing data caused by long-term continuous distribution shifts will largely influence the forecasting model’s performance and generalizability. Adaptive normalization approaches have been investigated in recent research to account for the non-stationarity issue. The deep input normalization method is proposed for time series forecasting with adaptive shifting, scaling, and shifting (Passalis et al., ). Kim et al. suggest an instance-level normalization and design a symmetric structure to remove and recover the distribution information according to the statistics of the instance’s input window (Kim et al., ). Fan et al. summarize the distribution shifts problem in time series forecasting into two categories, namely intra-space shift and inter-space shift (Fan et al., ). They propose Dish-TS with learnable distribution coefficients for normalization and denormalization. Liu et al. further point out that the distribution shift may also happen within the window, especially in long-term tasks, and propose a finer-grained slice-based normalization strategy (Liu et al., ). Liu et al. investigate the non-stationarity issue, especially for transformer models, and propose a de-stationary attention mechanism to alleviate the over-stationarization problem (Liu et al., ). Another class of studies mainly focuses on online learning to address the evolving distribution problem (Anava et al., ; Liu et al., ; Aydore et al., ), where the model will be updated with the continuous streaming data. Most recent advanced online learning approaches enhance the model adaptability under distribution shifts by complementary continual learning (Pham et al., ) and combining strategy (Zhang et al., ).

However, the above studies can only tackle one of the forms of distribution shifts but neglect scenarios where non-stationarity and evolving distribution shifts happen simultaneously, especially for online long-term time series forecasting. Besides, most existing methods incomprehensively model distribution dynamics, which only considers the statistics of the input window for normalization and denormalization. Even Liu et al. (Liu et al., ) predict the distribution and carry out normalization from a finer-grained slice perspective, which is most relevant to our approach, it merely considers a static scale to model the distribution dynamics and the effectiveness is subjected to the predefined slice length. Unlike the existing methods, we propose a more comprehensive normalization framework to tackle the complex distribution shifts problem by modeling distribution dynamics in a multi-scale perspective.

## 3 Proposed Method

Figure: Figure 2: The proposed evolving multi-scale normalization framework. The overall structure includes periodicity extraction, multi-scale distribution dynamics modeling, and normalization-denormalization with adaptive ensembling. The learning strategy is designed with offline two-stage pretraining and online alternate updating to facilitate models to capture the evolving distribution.
Refer to caption: extracted/5887719/framework.jpg

Given a set of $N$ multivariate time series samples with looking back windows $\bm{X}=\left\{\bm{x}_{n}\right\}_{n=1}^{N}$ and corresponding target horizon windows $\bm{Y}=\left\{\bm{y}_{n}\right\}_{n=1}^{N}$, we aim to forecast $\bm{y}_{n}\in\mathbb{R}^{H\times C}$ given input $\bm{x}_{n}\in\mathbb{R}^{L\times C}$, where $C$ is the number of channels, $L$ and $H$ is the length of input and output window respectively.

The proposed overall framework is given as Fig. 2. Firstly, we model the distribution dynamics by periodicity extraction, multi-scale slice-based statistics calculation, and prediction. Secondly, we integrate distribution dynamics for normalization, denormalization, and adaptive ensembling of model forecasting outputs. Besides, we consider the learning procedure of the statistic prediction module and the backbone forecasting model as a bi-level optimization problem and propose a training strategy for the overall framework. Details of the proposed method are given in the following subsections.

### 3.1 Multi-Scale Distribution Dynamics Modeling

The dynamics of distribution may exhibit diverse patterns according to different scales. For example, the daily electricity load data distribution will evolve differently from that of hourly data, where the latter distribution reflects short-term fluctuations and presents a more transient dynamic than the previous one. Considering that multi-scale modeling is usually intertwined with multi-periodicity modeling (Wu et al., ), we model the dynamics of distribution according to the time series periodicity properties.

#### Global Periodicity Extraction

We first analyze the time series in the frequency domain and discover the global periodicity by Fast Fourier Transform (FFT) as follows:

$$ $\mathbf{A}=\operatorname{Avg}\left(\operatorname{Amp}\left(\operatorname{FFT} \left(\bm{X}\right)\right)\right),\left\{f_{1},\cdots,f_{k}\right\}=\underset{ f_{*}\in\left\{1,\cdots,\left[\frac{t}{2}\right]\right\}}{\arg\operatorname{ Topk}}(\mathbf{A}),p_{i}=\left\lceil\frac{t}{f_{i}}\right\rceil,i\in\{1,\cdots ,k\},$ (1) $$

where $\operatorname{Amp}(\cdot)$ denotes the calculation of amplitude. The operation $\operatorname{Avg}(\cdot)$ calculates the averaged amplitude over all $M$ channels and $N$ samples to obtain $\mathbf{A}\in\mathbb{R}^{t}$, where $\mathbf{A}_{j}$ represents the intensity of the frequency-$j$ periodic basis function. ${\arg\operatorname{Topk}(\cdot)}$ is there to select the most prominent frequencies $\left\{f_{1},\cdots,f_{k}\right\}$ with top-$k$ amplitudes $\left\{\mathbf{A}_{f_{1}},\cdots,\mathbf{A}_{f_{k}}\right\}$ to represent the global periodicity with period lengths $\left\{p_{1},\cdots,p_{k}\right\}(p_{i}=\left\lceil\frac{t}{f_{i}}\right\rceil)$.

#### Multi-Scale Slice-based Statistics Calculation and Prediction

Based on the discovered global periodicity, we propose to split looking back windows and horizon windows into non-overlapping slices with respect to different period lengths $\left\{p_{1},\cdots,p_{k}\right\}$ as follows:

$$ ${\bm{X}_{(i)}}=\operatorname{Slicing}_{{{p}_{i}}}\left(\operatorname{Padding} \left(\bm{X}\right)\right),{\bm{Y}_{(i)}}=\operatorname{Slicing}_{{{p}_{i}}} \left(\operatorname{Padding}\left(\bm{Y}\right)\right),i\in\left\{1,\cdots,k \right\},$ (2) $$

where $\operatorname{Padding(\cdot)}$ is to first extend the time series by copying a segment of itself from the end to make it compatible for $\operatorname{Slicing}_{p_{i}}(\cdot)$ with length $p_{i}$. Then, the slicing process will transform the original windows into a set of sliced windows $\left\{\bm{X}_{(1)},\cdots,\bm{X}_{(k)}\right\}$ and $\left\{\bm{Y}_{(1)},\cdots,\bm{Y}_{(k)}\right\}$ with respect to different periods, where ${\bm{X}_{(i)}}=\left\{\left\{\bm{x}_{(i),n}^{j}\right\}_{j=1}^{\frac{L}{{{p}_{
i}}}}\right\}_{n=1}^{N}$, ${\bm{X}_{(i)}}\in\mathbb{R}^{N\times{\frac{L}{{{p}_{i}}}}\times{p_{i}}\times C}$ and ${\bm{Y}_{(i)}}=\left\{\left\{\bm{y}_{(i),n}^{j}\right\}_{j=1}^{\frac{H}{{{p}_{
i}}}}\right\}_{n=1}^{N}$, ${\bm{Y}_{(i)}}\in\mathbb{R}^{N\times{\frac{H}{{{p}_{i}}}}\times{p_{i}}\times C}$. Then, the mean and standard deviation for each slice are computed as:

$$ $\mu_{(i),n}^{j}=\frac{1}{p_{i}}\sum_{t=1}^{p_{i}}\bm{x}_{(i),n}^{j,t},\quad \left(\sigma_{(i),n}^{j}\right)^{2}=\frac{1}{p_{i}}\sum_{t=1}^{p_{i}}\left(\bm {x}_{(i),n}^{j,t}-\mu_{(i),n}^{j}\right)^{2},$ (3) $$

$$ $\phi_{(i),n}^{j}=\frac{1}{p_{i}}\sum_{t=1}^{p_{i}}\bm{y}_{(i),n}^{j,t},\quad \left(\xi_{(i),n}^{j}\right)^{2}=\frac{1}{p_{i}}\sum_{t=1}^{p_{i}}\left(\bm{y} _{(i),n}^{j,t}-\phi_{(i),n}^{j}\right)^{2},$ (4) $$

where $\mu_{(i),n}^{j},\sigma_{(i),n}^{j},\phi_{(i),n}^{j},\xi_{(i),n}^{j}\in\mathbb{
R}^{1\times M}$, $\mu_{(i),n}^{j}$, $\sigma_{(i),n}^{j}$ are mean and standard deviation of slice $\bm{x}_{(i),n}^{j}$, and $\phi_{(i),n}^{j}$, $\xi_{(i),n}^{j}$ are statistics of $\bm{y}_{(i),n}^{j}$.

Considering the multi-scale characteristics of time series data with corresponding evolving distribution, we propose a multi-scale prediction module to capture the dynamics of statistics and predict the distributions of future slices. Concretely, the dynamics of mean and standard deviation are modeled separately, and specific models for each periodicity are constructed:

$$ ${{\hat{\bm{\phi}}}_{(i)}}={{f}_{{{\omega}_{i}}}}\left({\bm{\mu}_{(i)}},\bm{x} \right),\quad{{\hat{\bm{\xi}}}_{(i)}}={{f}_{{{\theta}_{i}}}}\left({\bm{\sigma} _{(i)}},\bm{x}\right),$ (5) $$

where ${{f}_{{{\omega}_{i}}}}\left(\cdot\right)$ and ${{f}_{{{\theta}_{i}}}}\left(\cdot\right)$ are prediction models parameterized by $\omega_{i}$ and $\theta_{i}$ for modeling mean and standard deviation statistics under periodicity-$i$. Each prediction model is designed to be lightweight and consists of a two-layer perceptron network with $\operatorname{Relu}()$ and $\operatorname{Identity}()$ as final activations for standard deviation and mean statistics, respectively.

### 3.2 Normalization with Adaptive Ensemble

Our proposed method alleviates the influence of non-stationarity on the forecasting model by following the logic of ”normalization - backbone model forecasting - denormalization” as existing normalization methods (Kim et al., ; Fan et al., ; Liu et al., ). Moreover, the proposed multi-scale statistics prediction module can provide more comprehensive information about how data statistics are evolving. To this end, our method will tackle the distribution shift challenge from a multi-scale perspective and further improve forecasting by adaptive ensembling.

#### Normalization

Firstly, we consider the periodicity with intricate distribution shifts of a given series $\bm{x}_{n}$ and normalize it with slice statistics:

$$ $\tilde{\bm{x}}_{(i),n}^{j}=\frac{1}{\sigma_{(i),n}^{j}+\varepsilon}*\left(\bm{ x}_{(i),n}^{j}-\mu_{(i),n}^{j}\right),$ (6) $$

where $\tilde{\bm{x}}_{(i),n}^{j}$ is the normalized series corresponding to slicing with periodicity-$i$, $\varepsilon$ is a small constant value to prevent calculation instability. The operation * denotes the per-element product. To this end, a set of $k$ normalized series can be obtained ${{\tilde{\bm{x}}}_{n}}=\left\{{{{\tilde{\bm{x}}}}_{(i),n}}\right\}_{i=1}^{k}$, which ensures each period of series has a similar statistics and thus to be easier analyzed by the backbone forecasting model.

#### Denormalization

Subsequently, the set of normalized series will be processed by an arbitrary backbone forecasting model ${{f}_{\psi}}\left(\cdot\right)$:

$$ ${{\tilde{\bm{y}}}_{n}}=\left\{{{{\tilde{\bm{y}}}}_{(i),n}}\right\}_{i=1}^{k}={ {f}_{\psi}}\left({{{\tilde{\bm{x}}}}_{n}}\right),$ (7) $$

where ${{\tilde{\bm{y}}}_{n}}$ is a set of $k$ normalized output generated by the backbone model with the same set of parameter $\psi$. These outputs will then be split into slices and denormalized by the predicted slice statistics given by the multi-period statistics prediction module to recover the non-stationary information of different periodicities:

$$ $\hat{\bm{y}}_{(i),n}^{j}=\tilde{\bm{y}}_{(i),n}^{j}*\left(\hat{\xi}_{(i),n}^{j }+\varepsilon\right)+\hat{\phi}_{(i),n}^{j}.$ (8) $$

By concatenating the denormalized slices in their chronological order, we can obtain a set of output series $\left\{{{{\hat{\bm{y}}}}_{(i),n}}\right\}_{i=1}^{k}$.

#### Multi-Scale Adaptive Ensemble

Even though the periodicity is analyzed from a global perspective where the dominant periodicities are determined by the average amplitude as equation 1, each individual series and channel may show different degrees of these periodicities. With this consideration, we analyze the periodicity from a local perspective by calculating the local amplitude of the global periodicity for a given input series $\bm{x}_{n}$ as:

$$ ${\bm{A}_{n}}=\underset{\left\{{{f}_{1}},\cdots,{{f}_{k}}\right\}}{ \operatorname{Amp}}\,\left(\operatorname{FFT}\left({\bm{x}_{n}}\right)\right),$ (9) $$

where $\bm{A}_{n}\in\mathbb{R}^{k\times C}$. As the amplitudes can reflect the relative importance of the periodicity, we adaptively ensemble the denormalized outputs based on the weights of the local amplitude as:

$$ ${{\hat{\bm{y}}}_{n}}=\sum{{{\bm{w}}_{(i),n}}*{{{\hat{\bm{y}}}}_{(i),n}}},\quad {{w}_{(i),n}}=\frac{{{A}_{(i),n}}}{\sum\nolimits_{i=1}^{k}{{{A}_{(i),n}}}}$ (10) $$

### 3.3 Evolving Bi-Level Optimization

The performance of the backbone forecasting model is subject to the output given by the statistics prediction module. To this end, the training of these models is formulated as a bi-level optimization problem as follows:

$$ $\displaystyle\underset{\psi}{\mathop{\min}}\,\sum\nolimits_{n=1}^{N}{\mathcal{ L}\left(\left({\bm{x}_{n}},{\bm{y}_{n}}\right),\psi|{\bm{\omega}^{*}},{\bm{ \theta}^{*}}\right)}$ (11) $\displaystyle\text{s.t. }{\bm{\omega}^{*}},{\bm{\theta}^{*}}=\arg{{\min}_{\bm{ \omega},\bm{\theta}}}\sum\nolimits_{n=1}^{N}{\sum\nolimits_{i=1}^{k}{\mathcal{ L}_{i}^{stat}\left(\left({\bm{x}_{n}},{\bm{y}_{n}}\right),\psi,{\bm{\omega}_{i }},{\bm{\theta}_{i}}\right)}},$ $$

where the upper objective is to optimize the backbone forecasting model with the normal MSE loss function $\mathcal{L}$ and the lower objective is to minimize the distribution discrepancy between the denormalized output $\hat{\bm{y}}_{(i)}=f(\bm{x}_{n},\psi,\omega_{i},\theta_{i})$ and true distribution of $\bm{y}_{n}$ in views of different scales.

To solve such a bi-level optimization problem, we design a training strategy including offline decoupled pretraining and online alternate updating. In the offline pretraining stage, we first decoupled the optimization into two separate processes to enable the multi-scale statistics prediction module to focus on estimating the future distributions and let the backbone model focus on learning from normalized series generated by statistics of different scales. Concretely, the statistics prediction module is trained by optimizing $\bm{\omega}^{*},\bm{\theta}^{*}=\arg\min_{\bm{\omega},\bm{\theta}}\sum_{n=1}^{
N}\sum_{i=1}^{k}\mathcal{L}_{i}^{\text{stat}}\left(\left(\bm{x}_{n},\bm{y}_{n}
\right),\omega_{i},\theta_{i}\right)$, which mainly focuses on estimating future statistics information regardless of the backbone forecasting model. With such simplification, the original challenging task of minimizing the distribution discrepancy between two series is transformed to minimize the difference between the estimated slice-based statistics and the corresponding ground truth. To this end, The statistics prediction module corresponding to periodicity-$i$ can be trained with the loss function calculated by the mean squared error $\ell(({{\hat{\bm{\phi}}}_{(i)}},{{\hat{\bm{\xi}}}_{(i)}}),({{\bm{\phi}}_{(i)}}
,{{\bm{\xi}}_{(i)}}))$. After the statistics prediction module is trained to converge, the backbone forecasting model is optimized to minimize the loss between the ensembled multi-scale output and the ground truth value $\ell({{\hat{\bm{y}}}_{n}},\bm{y}_{n})$.

Considering the evolving characteristics of both distribution dynamics and the forecasting model’s input-output relationship, an alternate updating strategy is then designed to enable the model to learn from continuous data samples online. In the online learning stage, the forecasting loss between the denormalized series and the ground truth is set as the overall optimization target for both the backbone model and the multi-scale statistics prediction module, which enables a tighter collaboration between these components. We first freeze the backbone forecasting model and update the multi-scale statistics prediction module by descending $\nabla_{\bm{\omega},\bm{\theta}}\mathcal{L}\left(\left(\bm{x}_{n},\bm{y}_{n}
\right),\psi,\bm{\omega},\bm{\theta}\right)$. For the next coming data, we freeze the statistics prediction module as the condition for updating the backbone forecasting model with gradient $\nabla_{\psi}\mathcal{L}\left(\left(\bm{x}_{n},\bm{y}_{n}\right),\psi,\bm{
\omega},\bm{\theta}\right)$. We repeat such alternating updates with online streaming data to enable both the statistics prediction module and the backbone model to track the evolving distribution.

## 4 Experiments

### 4.1 Experimental Setup

#### Datasets

We evaluate our methods on five large-scale real-world time-series datasets: (1) Electricity transformer temperature (ETT) (^1^11[https://github.com/zhouhaoyi/ETDataset](https://github.com/zhouhaoyi/ETDataset)) records of oil temperature and electricity transformers’ power load from July 2016 to July 2018. We choose the hourly ETTh1 data and 15-minute-resolution Ettm1 data for our experiments. (2) Electricity (^2^22[https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014)) contains hourly electricity load data of 321 clients from July 2016 to July 2019. (3) Exchange (^3^33[https://github.com/laiguokun/multivariate-time-series-data](https://github.com/laiguokun/multivariate-time-series-data)) collects the panel data of daily exchange rates from 8 countries from 1990 to 2016. (4) Traffic(^4^44[http://pems.dot.ca.gov](http://pems.dot.ca.gov)) contains hourly traffic load recorded by 862 sensors from January 2015 to December 2016. (5) Weather(^5^55[https://www.bgc-jena.mpg.de/wetter/](https://www.bgc-jena.mpg.de/wetter/)) contains meteorological time series with 21 weather indicators collected every 10 minutes in 2020.

#### Backbone Models

We evaluate the proposed model-agnostic EvoMSN framework with various mainstream time series models as the backbone under both online and offline multivariate forecasting settings. We selected backbone models including linear model DLinear (Zeng et al., ), transformers Autoformer (Wu et al., ), FEDformer (Zhou et al., ), PatchTST (Nie et al., ), and convolutional model TimesNet (Wu et al., ).

#### Implementation Settings

We evaluate the effectiveness of the proposed method in tackling the distribution shifts problem under two scenarios, including online and offline forecasting. For online forecasting, we split the data into warm-up pretraining and online learning phases by the ratio of 20:75. We follow the optimization details in (Pham et al., ) by optimizing the $l_{2}$ (MSE) loss with the AdamW optimizer (Loshchilov & Hutter, ). According to the widely applied online setting, the forecasts will be made as each test data sample arrives, and the model will be updated by one epoch according to the online forecasting loss. For offline forecasting, we split the training and testing data with the ratio of 70:20 and follow the implementation details in (Liu et al., ). The hyperparameter tuning is conducted based on the performance of a separate validation set. We utilize the mean squared error (MSE) and mean absolute error (MAE) as the metrics to evaluate forecasting performance. All the experiments are conducted with PyTorch (Paszke et al., ) on a single NVIDIA GeForce RTX 3080 Ti 12GB GPU.

### 4.2 Main Results

Table: Table 1: Online multivariate forecasting results. The bold values indicate better performance when the backbone model is equipped with the proposed EvoMSN method.

We report the online multivariate forecasting results in Table 1. The length of the online long-term forecasting window is set as $L_{\text{out}}\in\{96,192,336\}$. We set the length of the input sequence according to the original setting of the backbone forecasting model as $L_{\text{in}}=96$ for Autoformer, FEDformer, TimesNet, and $L_{\text{in}}=336$ for DLinear and PatchTST.

For the online forecasting, we set the baseline as letting the backbone models pretrain on the training data set and adopt simple online learning strategies to train and test continuously on the testing data. The proposed EvoMSN framework enables the fine-grained modeling of distribution dynamics and time series evolving patterns, which greatly enhances the forecasting performance of most benchmark models across various forecasting horizons. Taking TimesNet as an example, EvoMSN can achieve an average reduction in MSE by 11.30% on the Exchange dataset, 12.77% on the Traffic dataset, 31.74% on the Electricity dataset, 55.09% on the Weather dataset, 28.41% on the ETTh1 dataset, and 12.12% on the ETTm1 dataset. The results also demonstrate that the proposed method can improve the performance of transformers and linear model across different forecasting horizons. It should be noted that even though PatchTST and TimesNet involve similar concepts of slicing and multi-scale modeling in their model design, these methods still benefit from the proposed normalization framework because the decoupled and explicit modeling of distribution dynamics enables these backbones to better learn from a stationarized series. For a better visualization, we show the online long-term forecasting results with the output window length of 336 in Figure 3. The results are given by setting DLinear as the backbone model, where the vanilla online learning strategy and the proposed EvoMSN method are compared. We can observe that the model is struggling to track the complex evolving distributions with the vanilla online learning strategy, while the proposed EvoMSN method adaptively adjusts the distribution of forecasting outputs to align with the ground truth. More visualization results can be found in Appendix A.1.

To further validate the efficacy of the proposed method, we have analyzed its performance on large-scale data, low-frequency data, and computation efficiency, where detailed results can be found in Appendix A.2. We have also conducted comprehensive experiments to showcase the effectiveness of the proposed method in enhancing offline long-term non-stationary time series forecasting, where the results are provided in Appendix A.5. The results show that the proposed method can also help tackle the distribution shift problem without the online learning process.

Figure: (a) Electricity
Refer to caption: extracted/5887719/Electricity.png

### 4.3 Comparison with Online Learning Strategies

In order to showcase the effectiveness of the proposed method, we also compare it with other advanced online learning strategies that all use Temporal Convolution Network (TCN) (Bai et al., ) as the backbone. First, the Online-TCN adopts the vanilla online learning strategy to train continuously with the streaming data. Then, we consider the Experience Replay (ER) strategy (Chaudhry et al., ) that employs a buffer to store previous data and update with new coming data. DER++ (Buzzega et al., ) is the variant of ER, which augments the standard ER with a knowledge distillation strategy (Hinton et al., ). Lastly, we consider the previous state-of-the-art online learning method FSNet (Pham et al., ), which adopts a complementary continual learning strategy. We use the same setting for these methods for a fair comparison, where the input window length is 96, and the learning rate is 0.001. We equip the proposed EvoMSN method to TCN (denoted as EvoMSN-TCN) and report the comparison results with existing online strategies in Table 2.

Table: Table 2: Comparison between EvoMSN and existing online learning strategies. The best and second best results are in bold and underline.

We can observe that ER, DER++, and FSNet are strong competitors and can effectively improve the online forecasting performance compared to the vanilla method. However, such methods do not model the dynamics of distribution and cannot fully address the complex distribution shift problem in the presence of both non-stationarity and gradual distribution shifts. Especially for FSNet, which performs well in the original work that focuses on short-term forecasting but degrades with a longer forecasting horizon. In comparison, the proposed EvoMSN method largely improves the capability of the TCN model in the online setting and shows promising results on all datasets that outperform the strong baselines. The EvoMSN-TCN can achieve a performance improvement that is measured on all datasets across all forecasting horizons by 65.90% compared to Online-TCN, 63.89% compared to ER, 64.12% compared to DER++, and 63.01% compared to FSNet. To investigate the model performance during the online forecasting process, we plot the cumulative average MSE loss of different online methods in Fig 4. With such visualization, the distribution shifts are likely to happen when the cumulative loss increases or shows peaks. The results clearly show the EvoMSN-TCN has a much lower and smoother cumulative loss curve, especially for challenging Electricity, Exchange, and Traffic datasets that are highly non-stationary. The above observations validate that the proposed EvoMSN method can effectively enhance the model for online forecasting, where comprehensive results are provided in Appendix A.3.

Figure: (a) Electricity
Refer to caption: extracted/5887719/electricity_cum.png

### 4.4 Comparison with Normalization Methods

Table: Table 3: Comparison between EvoMSN and existing normalization approaches for online forecasting.

Subsequently, we compare EvoMSN with three state-of-the-art normalization methods for time series forecasting under distribution shifts: SAN (Liu et al., ), RevIN (Kim et al., ), and Dish-TS (Fan et al., ). As these normalization methods were originally proposed for the offline forecasting setting, we apply them to online forecasting by updating the backbone model with streaming data. The comparison of normalization approaches for online forecasting based on Autoformer and TimesNet backbones are presented in Table 3. It can be concluded that EvoMSN achieves the best performance among existing normalization methods. The RevIN and Dish-TS mainly utilize statistics of the whole input window for normalization and denormalization; such a coarse way causes them a relatively inferior performance. SAN adopts a finer slice-based approach for normalization and explicitly models the distribution dynamics to predict future distribution for denormalization. To this end, SAN achieves a better performance than RevIN and Dish-TS. However, SAN only considers a single scale to model the distribution dynamics but neglects the difference in distribution dynamics across various scales. Besides, SAN only considers the offline two-stage training scheme but is not tailored for online forecasting. Distinct from these methods, the proposed EvoMSN adopts a more comprehensive multi-scale approach to capture the evolving distribution and consequently achieves the best performance, where detailed results across various forecasting horizons are provided in Appendix A.4. In addition, we compare forecasting performance when different normalization approaches are combined with the SOTA continual learning approach (FSNet) and show the consistent advantage of EvoMSN in Appendix A.4. The superiority of the proposed method versus other normalization methods is also validated for offline long-term forecasting, where results are provided in Appendix A.6.

### 4.5 Abaltion Study

This section investigates the impact of the multi-scale modeling approach and the proposed evolving optimization strategy, where the DLinear model is utilized as the backbone. First, we vary the number of scales in modeling multi-scale distribution dynamics and report results in Table 4. We find an overall trend that the model performance is better when more scales are considered to model the distribution dynamics, where $K=4$ achieves the best performance and is set as default in our experiments. Compared to considering a single scale, multi-scale modeling approaches can reduce the average MSE loss by 23.82% on Exchange and 12.64% on ETTm1, which validates the necessity of the proposed multi-scale modeling strategy.

Table: Table 4: Sensitivity study. The online prediction accuracy varies with the number of multi-scales $K$.

Table: Table 5: Ablation study. W/O online, W/O stat, and W/O backbone represent forecasting without online updating of both the backbone forecasting model and the statistics prediction module, without online updating of the statistics prediction module, and without online updating of the backbone forecasting model, respectively.

Then, we investigate the proposed evolving updation strategy with different ablation settings in Table 5: W/O online means only carrying out offline two-stage pretraining without an online updating of both the backbone model and statistics prediction module. W/O stat and W/O backbone mean online learning without an update of the backbone model and statistics prediction module, respectively. We can find that only modeling the distribution dynamics in the offline pretraining process is inadequate to tackle the online forecasting challenges, where the functions of both distribution dynamics and normalized input-output relationship will evolve over time. To this end, the online updation of the statistics prediction module and backbone forecasting model are both important to address the distribution shifts problem, where the proposed online alternating updation strategy provides a promising solution.

## 5 Conclusion

In this paper, we propose an evolving multi-scale normalization framework for time series forecasting under complex distribution shifts, which enables a more comprehensive way to model the distribution dynamics and facilitates the forecasting model to better track the evolving distributions. As a model-agnostic framework with arbitrary backbones, the proposed method effectively boosts mainstream forecasting techniques to achieve state-of-the-art performance on real-world time-series datasets by a significant margin. Extensive experiments have been conducted to examine the superiority of the proposed method in tackling distribution shift challenges compared to other advanced normalization and online learning strategies. As our future direction, we will further investigate a more general normalization framework that takes into account more detailed distribution information beyond mean and variance.

## Appendix A Supplementary Experiments

### A.1 Full Visualization Results

The methodology designed in this work is motivated by the fact that time series data distribution shows diverse dynamics across different scales. We plot the statistics (mean and standard deviation) of windows that have different scales in Fig. 5. We can see from the figure that a larger scale shows the general trend of how the distribution is evolving while a smaller scale presents more detailed local variations.

Figure: (a) Window length 96
Refer to caption: extracted/5887719/stat_96.png

The visualization results in comparing the forecasting with and without the proposed EvoMSN method are shown in Fig. 6, which reports the performance of the Dlinear backbone on Electricity, Exchange, Traffic, Weather, ETTh1, and ETTm1 data. It shows clearly that the vanilla online learning strategy is inadequate to address the complex distribution shift problem, where the proposed EvoMSN approach is of great necessity to enable the model to generate outputs with a more reasonable distribution.

Figure: (a) Electricity
Refer to caption: extracted/5887719/Electricity.png

### A.2 Extended Experiments Results

We conduct experiments on a larger traffic dataset (Liu et al., ) to online forecast hourly traffic flow with 716 sensors in San Diego county in 2019. The proposed EvoMSN method is applied to three competing backbone forecasting models in the reference, including LSTM (Hochreiter, ), STGCN (Yu et al., ), and GWNET (Wu et al., ). The results are shown in Table 6, where EvoMSN can achieve an average reduction in RMSE by 18.94% with LSTM, 10.08% with STGCN, and 7.3% with GWNET compared to the vanilla online approach.

Table: Table 6: Online forecasting results on large traffic dataset.

In order to evaluate the proposed method on low-frequency data, we have included M5 competition data (Makridakis et al., ) for the experiment. Specifically, we conduct multivariate forecasting of aggregated daily sales of each state, each product category, each store, and each department (a total of 23 variables). We utilize the DLinear model as the forecasting backbone for both online and offline forecasting (all experiments are repeated 3 times). We compare the offline forecasting performance with/without the proposed MSN method and online forecasting performance with/without the proposed EvoMSN method in Table 7. The results show the proposed method can boost the performance on low-frequency data for both online forecasting (average reduction in MSE by 19.12%) and offline forecasting (average reduction in MSE by 21.81%).

Table: Table 7: Online and Offline forecasting results on M5 dataset.

To evaluate the computational efficiency of the proposed method, we have measured the computation time in seconds for each training epoch and testing batch, the time of each process in online forecasting and offline forecasting is reported in Table 8 and Table 9, respectively. The proposed method will cause extra computational complexity and running time from two perspectives. First, the proposed method requires training multi-scale statistics prediction modules, where the complexity is the number of considered periodicities times the complexity of an individual prediction module for a single periodicity. The computational complexity of this part is independent of the complexity of the forecasting backbone model and we have designed this part to be lightweight with two-layer perception networks to avoid a large computation burden. Second, the backbone forecasting model will handle the time series from multi-scale perspectives, where the complexity is the number of considered periodicities times the complexity of the backbone forecasting model. This part becomes the main computation burden when the backbone model is much more complex than the statistics prediction module. There are two possible approaches to accelerate the computation process, one is to let the multi-scale analysis in parallel (our experiment is conducted in series); another approach is to consider an attentive updation strategy in the online learning process, which is to update the model only when server distribution shift is detected instead of updating for every sample. Overall, the computational complexity of the proposed method is acceptable in many real-world applications, especially for low-frequency forecasting scenarios, such as day-ahead electricity load forecasting.

Table: Table 8: Computation Time for Online Forecasting with EvoMSN.

Table: Table 9: Computation Time for Offline Forecasting with MSN.

### A.3 Full Results of Comparison of Online Learning Strategies

This section provides comprehensive results of comparison between the proposed EvoMSN method with other online learning strategies, where the performance of forecasting horizons {96, 192, 336} are reported in Table 10. It shows the advantage of EvoMSN in enhancing online forecasting performance across all horizons. The full visualization results of online cumulative average MSE loss are shown in Fig. 7, which showcases the efficacy of EvoMSN to enable backbone models to better adapt to shifting distribution.

Table: Table 10: Comparison between EvoMSN and existing online learning strategies. The best and second best results are in bold and underline.

Figure: (a) Electricity
Refer to caption: extracted/5887719/electricity_cum.png

### A.4 Full Results of Normalization Comparison in Online Forecasting

We provide comprehensive results of comparison between the proposed EvoMSN method with other normalization strategies for online forecasting in Table 11. It can be observed that EvoMSN can achieve the best performance in most cases and its effectiveness is prominent when the forecasting horizon is long. The slice-based SAN method also shows a better performance than the instance-level normalization approaches (RevIN and DishTS), which showcases the necessity of modeling the distribution dynamics in a finer-grained manner. However, better performance can also be achieved by the instance-level normalization approaches in some cases, which shows that a coarser-grained modeling of distribution dynamics that helps models to capture a general trend is also important. To this end, the proposed multi-scale modeling approach is a more comprehensive solution that integrates the advantages of both approaches.

Table: Table 11: Comparison between EvoMSN and existing normalization approaches for online forecasting. The best and second best results are in bold and underline.

We conduct experiments to investigate combining continual learning with different normalization methods. Specifically, we utilize FSNet (Pham et al., ) as the backbone and compare the vanilla FSNet, FSNet equipped with the proposed EvoMSN, with RevIN, and with DishTS in Table 12. As shown in Table 10 the proposed EvoMSN can already enable the TCN model to outperform FSNet. Table 12 shows that the combination of EvoMSN and the continual learning strategy can further improve online forecasting performance. This is because the backbone model with a continual learning strategy can better deal with the stability-plasticity dilemma. The above results demonstrate the compatibility of the proposed method with the continual learning strategy, and superior performance compared to other normalization methods also validates the strength of the proposed method.

Table: Table 12: Performance comparison when different normalization methods combine with continual learning strategy.

### A.5 The Effect of Multi-Scale Normalization (MSN) for Offline Long-Term Forecasting

We investigate the traditional offline long-term forecasting setting with the train-test ratio of 70:20. This is a non-trivial task since the model will not be updated online and will suffer from the distribution discrepancy between training data and testing data. We investigate backbone models including DLinear (Zeng et al., ), Autoformer (Wu et al., ), FEDformer (Zhou et al., ), PatchTST (Nie et al., ), and TimesNet (Wu et al., ) on six benchmark datasets: (1) Electricity, (2) Exchange, (3) Traffic, (4) Weather, (5) Etth2, and (6) ILI (^6^66https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html), which collects the ratio of illness patients versus the total patients in one week that reported weekly from 2002 and 2021. We follow the same configuration as (Liu et al., ) for the length of the input window and output window. In this setting, we carry out the proposed multi-scale normalization (MSN) method without online updation but only with offline two-stage training, where the results are reported in Table 13. By applying MSN to the backbone models, we observe an average performance improvement on all datasets across all forecasting ranges by 15.03% with DLinear, 25.60% with Autoformer, 22.62% with FEDformer, 4.05% with PatchTST, and 7.77% with TimesNet. Such great improvements validate the effectiveness of the proposed method in explicitly modeling the distribution dynamics and removing the non-stationary factors for time series forecasting, which also shows that the proposed method can help alleviate the effects of distribution discrepancy between training and testing data to some extent. The efficacy of the proposed method is especially prominent on data that are highly non-stationary, such as Exchange, ETTh2, and ILI.

Table: Table 13: Offline multivariate forecasting results. The bold values indicate better performance when the backbone model is equipped with the proposed MSN method.

### A.6 The Comparison of MSN with Advanced Normalization Benchmarks for Offline Long-Term Forecasting

We compare the proposed MSN with other stat-of-the-art normalization methods for offline long-term forecasting, including SAN (Liu et al., ), RevIN (Kim et al., ), and Dish-TS (Fan et al., ). In addition, we also include Non-Stationary Transformers (NST) (Liu et al., ) as a benchmark, which is designed especially for transformers to forecast non-stationary time series. We utilize FEDformer and Autoformer as the backbones and report results in Table 14. We can find that SAN performs better than the instance-level normalization methods (RevIN and Dish-TS) due to its slice-based modeling of distribution dynamics. NST shows more promising results on Weather data where other methods suffer from the over-stationarization problem. In comparison, the proposed method achieves the best performance thanks to the multi-scale modeling and ensembling.

Table: Table 14: Comparison between MSN and existing normalization approaches for offline forecasting. The best and second best results are in bold and underline.

## Appendix B Limitation

Even though the proposed method can greatly improve the overall forecasting accuracy, it may sometimes cause slight performance degradation. We conclude potential reasons for method failure as follows: 1) In the online forecasting setting, the multi-scale statistics prediction module is updated only once for each incoming new data, which may challenge the module in capturing some fast-changing statistics and will result in bad performance. Besides, the global dominant periodicity is determined according to the training data, where the relatively small training data split ratio in the online setting may cause an inappropriate periodicity extraction and thus affect the effectiveness of multi-scale slice-based analysis. 2) In the offline forecasting setting, the multi-scale statistics prediction module is only updated on the training data, where the distinct statistics evolving dynamics of training data and testing data may cause a worse performance in the evaluation stage. Despite the potential shortcomings identified above, it is essential to emphasize the overall strengths of the proposed method.

Moreover, we point out two directions to further improve the proposed method. First, we only model the distribution dynamics by investigating slice statistics including mean and deviation, where more comprehensive characteristics of distribution, such as minimum and maximum value, are important but neglected in the proposed framework. We will propose a more general approach to model data distribution dynamics in our future work. Second, the proposed EvoMSN may suffer from the stability-plasticity dilemma when the backbone model and statistics prediction module are updated online, where incremental learning techniques could be further investigated and integrated into the proposed framework to improve the online forecasting performance.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: Refer to the weakness.
soundness: 2
strengths: 1. The EvoMSN framework introduces the concept of multi-scale statistics prediction, extending existing normalization methods. By combining this with adaptive ensembling and online learning strategies, the framework better captures and responds to data distribution changes. This integration of multiple techniques provides a new approach to addressing distribution shifts in time series forecasting.
2. The authors present robust experimental results that demonstrate the effectiveness and superiority of EvoMSN. These experiments span a variety of real-world datasets and utilize multiple advanced forecasting methods as backbones, ensuring the reliability and generalizability of the findings.
3. The paper is well-structured and logically coherent.
summary: This paper introduces the Evolving Multi-Scale Normalization framework, a model-agnostic approach designed to address complex distribution shifts in time series forecasting. EvoMSN combines multi-scale statistics prediction, adaptive ensembling, and an evolving bi-level optimization strategy to improve the accuracy of long-term forecasts. The framework is evaluated on several real-world datasets and shown to significantly enhance the performance of five mainstream forecasting methods.
weaknesses: 1. The paper aims to tackle the complex distribution shifts problem, but the description of what constitutes these distribution shifts is somewhat vague. It would be beneficial to provide a more specific description of how the distributions change. For instance, among the distributions \(P(X)\), \(P(Y)\), \(P(X|Y)\), and \(P(Y|X)\), which ones are changing and which ones remain constant? A clearer delineation of these changes would enhance the understanding of the problem being addressed.

2. The paper lacks a detailed explanation of why normalization and denormalization can effectively address distribution shifts in time series forecasting. Providing a theoretical or empirical justification for this approach would strengthen the paper's arguments and help readers understand the underlying mechanisms better.

3. The paper states that the EvoMSN framework can be combined with existing time series forecasting methods to improve their performance. However, it does not quantify the increase in computational complexity during training and testing when incorporating the EvoMSN framework. Understanding the trade-off between performance improvement and computational cost is crucial for practical applications.

4. Section 3.3 introduces an offline two-stage pretraining and online alternate updating method to solve the bi-level optimization problem. The paper does not adequately explain why this particular method was chosen and whether there are theoretical guarantees for the optimization results obtained using this approach. A more detailed discussion of the advantages and potential limitations of this method would be beneficial.

5. The experimental results show that the EvoMSN framework does not consistently improve the performance of the models. The paper does not provide a detailed analysis of why this inconsistency occurs. A thorough investigation of the conditions under which the framework performs well or poorly would help in understanding the robustness of the proposed method and guide future improvements.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The multi-scale statistics prediction module is a two-layer perceptron network. Have you ever attempted to employ any
other different architecture and analyze the sensitivity?

2. Windowing functions are often coupled with FFT. Have you ever attempted to use different windowing functions and
analyze the sensitivity?

3. FFT often assumes the time-series data is stationary. When this assumption is violated, the resulting frequency-amplitude
information might be not meaningful. Have you consider using any other approaches, and if so, how does the performance
vary?
soundness: 3
strengths: The proposed novel framework (EvoMSN) is unique since it combines multi-scale statistics prediction, adaptive ensembling, and
an evolving bi-level optimization strategy to tackle the distribution shift problem in time-series forecasting. It differs from existing
implementations that lack of the capability of capturing the distribution dynamics from various scales and modeling the evolving
normalized input-output mapping functions caused by gradual distribution shifts.

The research is based on sufficient analysis of related existing works. Each component of the proposed framework is well-
introduced. The effectiveness is demonstrated using five large-scale real-world time-series benchmark datasets with appropriate
experiment setup.

The paper is well-organized clear sections for introduction, related works, proposed framework, experiments and conclusion.
The introduction outlines the significance of distribution shift problem in time-series forecasting, the limitations of existing
methods, and the proposed solution.

This paper addresses the distribution shift problem more comprehensively compared to existing implementations. The proposed
framework improves the accuracy of the time-series forecasting. The proposed EvoMSN is model-agnostic that makes it
possible to enhance any arbitrary backbone forecasting models.
summary: This paper introduces a model-agnostic Evolving Multi-Scale Normalization (EvoMSN) framework to tackle the forecasting of
time series with distribution shifts since existing implementations lack the capability of capturing the distribution dynamics
from various scales and modeling the evolving normalized input-output mapping functions caused by gradual distribution
shifts.

The contributions are as follows:

1. EvoMSN is a model-agnostic online normalization framework that enhances any arbitrary backbone forecasting models by
adaptively removing and recovering dynamical distribution information.

2. A multi-scale statistics prediction module is introduced to estimate the statistics of future distributions. An adaptive
ensemble strategy is designed to ensemble the denormalized outputs based on the weights of the local amplitude.

3. An evolving bi-level optimization strategy, including offline two-stage pretraining and online alternating learning, is
proposed to update the statistics prediction module and the backbone forecasting model collaboratively.

4. The effectiveness of the proposed method in boosting forecasting performance under distribution shifts is evaluated on
five large-scale real-world time-series benchmark datasets.
weaknesses: Online forecasting limitations
1. Module update frequency: The parameters of multi-scale statistics prediction module are updated only once for each
incoming new data which might lead to difficulties in extracting some fast-changing statistics, resulting in poor performance.

2. Periodicity extraction: The global dominant periodicity is determined based on the training data. In the online setting, a
relatively small training data split ratio might cause inappropriate periodicity extraction and will affect the effectiveness of
multi-scale slice-based analysis.

Offline forecasting limitations

1. Module update scopes: In the offline pretraining stage, the multi-scale statistics prediction module is only updated on the
training data. The distinct statistics evolving dynamic of training data and testing data may cause worse performance in the
evaluation stage.

Modeling limitations

1. Incomplete distribution characteristics: The multi-scale statistics prediction module only produces the mean and the
deviation of each slice. More comprehensive characteristics of distribution, such as minimum and maximum value, are
significant but neglected in the proposed framework.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: As is commented in Weakness.
soundness: 4
strengths: 1. The proposal to use multi-scale statistics prediction and adaptive ensembling for normalization is innovative and addresses a critical gap in handling distribution shifts in time series data. And the framework is model-agnostic, increasing its versatility and applicability across various forecasting models.
2. The paper provides a thorough experimental evaluation, showcasing the effectiveness of EvoMSN in improving forecasting performance under distribution shifts across different datasets and forecasting models as well as other frameworks. The results demonstrate substantial improvements over state-of-the-art methods.
summary: The paper presents the EvoMSN framework, a novel approach to time series forecasting under distribution shifts. It introduces a model-agnostic methodology that utilizes multi-scale statistics prediction and adaptive ensembling to normalize and denormalize data, addressing the complexities of distribution dynamics and non-stationarity. The framework is evaluated through comprehensive experiments on real-world datasets, demonstrating significant improvements over the original models.
weaknesses: 1. The paper could benefit from a more detailed discussion on the limitations of the EvoMSN framework, such as assumptions made, potential scenarios where the framework may underperform, and its scalability challenges.

2. While the framework is innovative, the paper lacks a detailed analysis of its computational complexity and scalability, which are crucial for practical applications, especially with large datasets. Even more, the training samples are sampled from a look back windows, which means that the effectiveness of the module relies heavily on the training data, potentially affecting the framework's performance in real world data. And how is it able to learn the preicise multiscale statistics predictions without sampling from a multiscale look back window.

3. The paper needs a much more detailed visualization on the learned statistics predictions to prove that the module actually works.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See Weaknesses.
soundness: 2
strengths: Time series prediction is very important.

The author proposes an effective model.
summary: The author proposed a new model-agnostic Evolutionary Multi-Scale Normalization (EvoMSN) framework to address the distribution shift issue. It includes a multi-scale statistical prediction module and flexible normalization and denormalization methods with adaptive integration. The experiments demonstrated the effectiveness of the model.
weaknesses: Q1. The relevant experimental details are confusing. What does the statement 'According to the widely applied online setting, the forecasts will be made as each test data sample arrives, and the model will be updated by one epoch according to the online forecasting loss' mean? Do you update the model once for a test sample or once for a batch sample? Please further explain the specific settings of online learning in detail.

Q2.  To validate the model's effectiveness across a broader spectrum, larger-scale datasets spanning longer time periods should be incorporated.  This will help ascertain the model's robustness and applicability in scenarios with more significant temporal variations.

Q3. Could authors add more baselines in the experiment, such as PatchMixer, Crossgcn, iTransformer, TimeMixer, Koopa, and other models.

Q4. The author's contribution is incremental. The author's online learning strategy mainly involves patch normalization and two-stage training strategies, which have already been discussed in SAN and [2]. The contributions of SAN and [2] were overlooked in the related work and methods section. Additionally, what are the differences between the evolutionary training strategy and [2]? In Table 14, I did not observe significant performance improvements of the proposed model compared to SAN. The idea of multi-scale modeling originates from TimeMixer. It is recommended that the author further discuss these works in detail to clearly demonstrate their contribution. Furthermore, I suggest rewriting the recommendation section to highlight the technical gaps in existing works more clearly to clarify the motivation.

Q5. In Table 5, we can see that the primary performance of the proposed method comes from the Online-strategy based on BI-LEVEL OPTIMIZATION, as discussed in [1] and [2]. Additionally, why did the author not evaluate the performance variability of multi-scale modeling variables? What findings are observed when K is greater than 4? Why were only ablation experiments conducted on DLinear? As far as I know, DLinear is a small model with unstable training, where even a small difference in a epoch could lead to significant performance variations. Therefore, it is recommended to add more benchmarks for a comprehensive evaluation.

Q5. Time series datasets are typically offline, and the running time of various models does not exceed one day.  Could you elaborate on the practical applications and advantages of online time series learning? 

Q6. DLinear reported a performance of 0.081 in the Exchange but 0.131 in this paper, and the performance of other benchmarks also dropped significantly in this paper. What could be the reason for this? A significant decline in baseline performance also occurs in Table 13. Please explain potential reasons for the discrepancy in results.

Q7. Can the author provide a comparison of other models in Table 11?

Q8. The author's motivation for the paper is the non-stationarity of time series, and the contribution is to address this problem from an online learning perspective. However, by comparing Table 1 and Table 8, I find that online learning does not actually have any benefits, and the performance of most models has shown a huge decline. The potential reason is that a test sample will lead to overfitting after updating the model. We do not need this update, and the original model can perform very well (at least better than after online learning).


Q9. Unfair comparison. As mentioned above, updating the model with data from one batch or one sample can lead to overfitting. The baseline model was updated twice, while the author's model was actually updated only once, resulting in a delayed decrease in model performance.

Ref:
[1] Adaptive Normalization for Non-stationary Time Series Forecasting: A Temporal Slice Perspective.
[2] Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. 2016. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05447 (2016).