SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiding Liu $^{1,2}$ , Mingyue Cheng $^{1,2}$ , Zhi Li $^{3}$ , Zhenya Huang $^{1,2}$ , Qi Liu $^{1,2}$ , Yanhu Xie $^{4}$ , Enhong Chen $^{1,2*}$
<sup>1</sup>Anhui Province Key Laboratory of Big Data Analysis and Application, University of Science and Technology of China
$^{2}$ State Key Laboratory of Cognitive Intelligence
<sup>3</sup>Shenzhen International Graduate School, Tsinghua University
<sup>4</sup>The First Affiliated Hospital of University of Science and Technology of China zhiding@mail.ustc.edu.cn,{mycheng,huangzhy,qiliuql,cheneh}@ustc.edu.cn zhilizl@sz.tsinghua.edu.cn, xyh200701@sina.cn
Deep learning models have progressively advanced time series forecasting due to their powerful capacity in capturing sequence dependence. Nevertheless, it is still challenging to make accurate predictions due to the existence of non-stationarity in real-world data, denoting the data distribution rapidly changes over time. To mitigate such a dilemma, several efforts have been conducted by reducing the non-stationarity with normalization operation. However, these methods typically overlook the distribution discrepancy between the input series and the horizon series, and assume that all time points within the same instance share the same statistical properties, which is too ideal and may lead to suboptimal relative improvements. To this end, we propose a novel slice-level adaptive normalization, referred to SAN, which is a novel scheme for empowering time series forecasting with more flexible normalization and denormalization. SAN includes two crucial designs. First, SAN tries to eliminate the non-stationarity of time series in units of a local temporal slice (i.e., sub-series) rather than a global instance. Second, SAN employs a slight network module to independently model the evolving trends of statistical properties of raw time series. Consequently, SAN could serve as a general model-agnostic plugin and better alleviate the impact of the non-stationary nature of time series data. We instantiate the proposed SAN on four widely used forecasting models and test their prediction results on benchmark datasets to evaluate its effectiveness. Also, we report some insightful findings to deeply analyze and understand our proposed SAN. We make our codes publicly available<sup>2</sup>.
Time series forecasting is becoming increasingly prevalent in real-world scenarios. Various applications have been facilitated by the advancement of forecasting, such as energy consumption planning [33], clinical healthcare analysis [16], financial risk assessment [15] and cloud resource allocation [2]. Recently, deep learning-based methods have largely advanced forecasting and other tasks due to their powerful capacity to capture sequence dependence [23, 27, 3, 42].
Nevertheless, it is still challenging to make accurate predictions for time series forecasting due to the rapid evolution of time series points over time (a.k.a. non-stationarity of time series) [31]. Such
*Enhong Chen is the corresponding author.
$^{2}$ https://github.com/icantnamemyself/SAN
non-stationarity can lead to discrepancies between different time spans and hinder the generalization of deep learning models. To alleviate the impact of the non-stationary nature, removing these dynamic factors from the original data through normalization has been proposed as a feasible solution [28].
Recently, some pioneering efforts have been devoted to this research topic [17, 25, 10]. Although these normalization approaches have significantly improved the prediction performance, we identify two limitations in existing solutions. On the one hand, most existing methods overlook the distribution discrepancy between the input series and the horizon series, and simply adopt the statistical properties of the input series to denormalize the output results. Furthermore, previous studies assume that all time points within the same instance share the same statistical properties during the normalization processing, and a global instance normalization is widely applied. Such coarse-grained settings are not appropriate since time series points rapidly change over time [6, 20], particularly in long-term forecasting scenarios where both input and horizon series may span a considerable duration. For example, there may be sudden changes in data distribution due to events like holidays or temperature spikes for electricity consumption data.
We plot a forecasting sample in Fig. 1 to better illustrate our opinion. Though temporally related, the input series' mean differs from the horizon's significantly (from 0.75 to 1.5), indicating a potentially universal distribution discrepancy. Moreover, such a distribution shift can happen rapidly at a more fine-grained slice level, violating the basic assumption of existing normalization methods. Therefore, these approaches risk damaging instinct patterns of each slice of the input sequence by normalizing with improper statistics while also causing a prediction shift in final forecasting results due to poor estimation of future statistics.
To overcome these limitations, we propose a general normalization framework for nonstationary time series forecasting named Slicing Adaptive Normalization (SAN).
SAN models the non-stationarity in the
Figure 1: An illustration of a forecasting instance in energy consumption along with its daily mean (Mean-ByDay). We also plot the input series' mean and the horizon series' mean in the figure.
fine-grained temporal slices, or patches[26], which are more informative than single data points and can be regarded as fundamental units of the time series data[4, 14]. To be specific, the input sequence is first split into non-overlap equally-sized slices, which are then normalized according to their statistics and fed into the forecasting model. Meanwhile, we use a statistics prediction module to predict the distributions of future slices based on the statistics of the input. Finally, the non-stationary information is restored to the output of the forecasting model with well-estimated statistics. By modeling the slice-level characteristic, SAN is able to remove the non-stationarity in a local region. Besides, with the statistics prediction module independently modeling the evolving trends of statistical properties, SAN adopts more precise statistics for adaptive denormalization. Consequently, the non-stationary forecasting task is actually simplified by being split into statistic prediction and stationary forecasting. Moreover, SAN is a model-agnostic framework and can be applied to arbitrary forecasting models. Sufficient experiments have been conducted in a widely used benchmark dataset. The results demonstrate that SAN surpasses advanced normalization approaches by boosting the performance of various kinds of mainstream forecasting models by a large margin.
In summary, our main contributions are as follows:
- We propose SAN, a general normalization framework for non-stationary time series forecasting tasks which distinguishes by modeling the non-stationary nature from a temporal slice perspective. In this way, SAN can better remove the non-stationary factors in input sequences while keeping their distinct patterns.
- We design a flexible statistics prediction module for SAN which independently models the evolving trends of statistical properties. By explicitly learning to estimate future distributions, SAN can simplify the non-stationary forecasting task through divide and conquer.
- We conduct sufficient experiments on nine real-world datasets. Results show that SAN can be applied to various mainstream forecasting models and boost the performance by a large margin. Moreover, the comparison between SAN and state-of-the-art normalization methods demonstrates the superiority of our proposed framework.
2.1 Time Series Forecasting
Time series forecasting has been extensively studied in recent decades. Originally, ARIMA [1, 40] builds an auto-regressive model and forecasts in a moving average fashion. Though theoretical guarantees are achieved, such traditional methods usually require data with ideal properties, which is inconsistent with the real-world scenario. With the increasing data availability and computing power, numerous deep-learning-based models have emerged, which always follow a sequence-to-sequence paradigm. Recurrent neural networks (RNNs) are first utilized to capture the temporal dependence by summarizing the past information in time series [30, 32, 37]. Such architectures naturally suffer from a limited reception field and an error accumulation issue caused by the recursive inference schema[43], both dragging down forecasting precision. To further boost the performance of the final prediction, many advanced architectures have been introduced to capture the long-range dependencies, such as the self-attention mechanism and the convolutional networks [19, 21, 24]. Besides, to leverage the characteristics of the time series data, recent works also integrate traditional analysis methods like trend-seasonal decomposition and time-frequency conversion into neural networks [36, 44]. In addition, a recent study points out that a simple linear network enhanced with decomposition also achieves competitive performance[39]. Furthermore, slice-based methods demonstrate superior accuracy in the long time series forecasting tasks [26, 41].
2.2 Non-stationary Time Series Forecasting
Most time series forecasting methods prioritize designing powerful architectures that can effectively capture temporal dependencies, but often overlook the non-stationary nature of the data. Considering the basic assumption of deep-learning-based models that the data in both training and test sets follow the same distribution, such a discrepancy will definitely drag down the precision of the model for future time prediction. Moreover, the distribution differences among instances in the training set may introduce noise, making the learning task harder to converge. To address these challenges, various stationarization methods have been explored.
In detail, DDG-DA[22] predicts the evolving data distribution in a domain adaptation fashion. Du et al.[8] propose an adaptive RNN to alleviate the impact of non-stationary factors by distribution characterization and distribution matching. Besides, normalization-based approaches have also gained popularity as they aim to remove non-stationary factors from original data and normalize all data to a consistent distribution. DAIN [28] introduces a non-linear network to learn how to normalize each input instance adaptively and ST-norm [7] proposes two normalization modules from both temporal and spatial perspectives. Later researchers point out that non-stationary factors are essential in accurate forecasting and simply removing them may result in poor prediction. Therefore, they propose RevIN [17], a symmetric normalization method that first normalizes the input sequences and then denormalizes the model output sequences through instance normalization [34]. Based on the similar structure, Non-stationary Transformers [25] presents de-stationary attention that incorporates the non-stationary factors in self-attention, resulting in significant improvements over Transformer-based models. Moreover, a recent study [10] identifies the intra- and inter-space distribution shift in time series, and proposes to relieve these issues by learning the distribution coefficients.
Despite the effectiveness of existing normalization methods, they inappropriately assume that all time points within the same instance share the same statistical properties during the normalization processing. Different from them, our proposed approach focuses on further thinking of the nature of the data, i.e., the distribution is inconsistent across compact time slices and such inconsistency is not just on a per-instance basis.
We propose a general model-agnostic normalization framework for time series forecasting called Slicing Adaptive Normalization (SAN) to address the inconsistency mentioned above. Considering an input set of time series  $X = \{\pmb{x}^i\}_{i=1}^N$  and their horizon series  $Y = \{\pmb{y}^i\}_{i=1}^N$ , SAN is expected to remove the non-stationary factors and assist the forecasting models to predict more accurately based on the observed input series. In this section, we will present the detailed workflow of the entire framework and explain how it works with non-stationary time series data. To provide better clarity, we summarize the key notations in Table 1 and the whole framework can be referred to in Fig. 2.
Table 1: The Key Mathematical Notations.
Notation	Description
N	the number of instances
V	the number of variables
T	the given slicing time span
Lin,Lout	the sequence length of input/target sequence
M,K	the number of slices of input/target sequence, M = Lin/T, K = Lout/T
xi,yi	the i-th input/target series for the whole framework
x̂i,yi	the i-th input/target series for forecasting models
*ij	the property of j-th slice in the i-th series, determined by *
μ,σ	the mean and the standard deviation value
*	the predicted value, determined by *
3.1 Normalization
Similar to existing normalization methods for non-stationary time series forecasting [17], SAN first normalizes the input sequence to remove the non-stationary factors and later restores them to the output sequence by denormalization. Differently, SAN applies such operation on a per-slice basis instead of the whole input sequence. Such a localized operation can better maintain the instinct pattern of each slice than global instance normalization. The framework first splits the input  $\boldsymbol{x}^i$  into  $M$  non-overlapping slices  $\{\boldsymbol{x}_j^i\}_{j=1}^M$  based on  $T$ . Then the mean and standard deviation for each slice is computed as:
$$
\mu_ {j} ^ {i} = \frac {1}{T} \sum_ {t = 1} ^ {T} \boldsymbol {x} _ {j, t} ^ {i}, \left(\sigma_ {j} ^ {i}\right) ^ {2} = \frac {1}{T} \sum_ {t = 1} ^ {T} \left(\boldsymbol {x} _ {j, t} ^ {i} - \mu_ {j} ^ {i}\right) ^ {2}, \tag {1}
$$
where  $\mu_j^i,\sigma_j^i\in R^{V*1}$  and  $\pmb{x}_{j,t}^{i}$  is the value of slice  $\pmb {x}_j^i$  at  $t$ -th time step. Later, SAN normalizes every slice of the original input sequence by their individual statistics as:
$$
\bar {\boldsymbol {x}} _ {j} ^ {i} = \frac {1}{\sigma_ {j} ^ {i} + \epsilon} \cdot \left(\boldsymbol {x} _ {j} ^ {i} - \mu_ {j} ^ {i}\right). \tag {2}
$$
Here we use  $\cdot$  to denote the element-wise product and  $\epsilon$  is a small constant. Finally, SAN restores all the slices in their original chronological order and lets the processed series without non-stationary factors  $\bar{x}^i$  be the new input of the forecasting models.
3.2 Statistics Prediction
As illustrated in Fig. 2, SAN introduces a unique statistics prediction module,  $f_{\phi}(*)$ , to better estimate future distributions in addition to the backbone forecasting model,  $g_{\theta}(*)$ . Unlike existing works that denormalize the entire output of backbone models with the statistics of original input sequences, SAN faces a natural challenge of per-slice normalization: how to estimate the evolving distributions for each future slice. To simplify and improve efficiency, we use a two-layer perceptron network with an appropriate activation function (e.g., Relu() for standard deviation to ensure non-negativity) that learns to predict future distributions based on input statistics and stationarized sequence.
The quality of statistics predictions determines the overall performance of SAN since we depend on an accurate estimation of future distribution to restore the non-stationary nature of each instance. In
Figure 2: The illustration of the proposed SAN framework. SAN is a model-agnostic symmetrical normalization framework that removes and restores the non-stationary factors in the time series data from the perspective of slicing. SAN trains in a two-stage manner. It first optimizes the statistics prediction module into convergence  $(l_{sp})$ , which learns to predict future statistics based on sliced input mean  $\pmb{\mu}^i$  and standard deviation  $\pmb{\sigma}^i$ . The second stage is the traditional procedure of training forecasting models  $(l_{fc})$ , where the statistics prediction module is frozen and functions as a plugin.
our modeling of distribution, the mean determines the approximate scale of a given slice and the standard deviation represents the degree of dispersion, where the scale of a small slice may be more important in the task of forecasting. Therefore, we aim to further refine the modeling approach for the mean component on the basis of analyzing its properties.
In detail, we believe that the overall mean of the input sequence  $\rho^i = \frac{1}{L_{in}}\sum_{1}^{L_{in}}\pmb{x}^i\in R^{V*1}$  is a maximum likelihood estimation of the target sequence's mean  $\hat{\rho}^i = \frac{1}{L_{out}}\sum_{1}^{L_{out}}\pmb{y}^i\in R^{V*1}$  since they are temporally related. That is,  $\rho^i\approx \hat{\rho}^i$ . Such property is widely accepted in existing works [17, 25] as they denominate the output using the statistics of the whole input sequence. Based on the above assumption, we introduce a residual learning [13] technique in our method, letting the module learn the difference between the future slice mean  $\hat{\mu}^i$  and the overall input mean  $\rho^i$ , instead of predicting the specific values. This approach reduces the difficulty in modeling means with prior knowledge about future trends. Additionally, to account for different variables exhibiting distinct patterns in scale changes, we further use two learnable vectors  $\mathbf{W}_1,\mathbf{W}_2\in R^V$  initialized as ones-vector to present the individual preference weights for each variable, making the prediction computed in a weighted-sum manner. The statistics prediction procedure can be formulated as:
$$
\hat {\boldsymbol {\mu}} ^ {i} = \boldsymbol {W} _ {1} * \operatorname {M L P} \left(\boldsymbol {\mu} ^ {i} - \rho^ {i}, \bar {\boldsymbol {x}} ^ {i} - \rho^ {i}\right) + \boldsymbol {W} _ {2} * \rho^ {i}, \hat {\boldsymbol {\sigma}} ^ {i} = \operatorname {M L P} \left(\boldsymbol {\sigma} ^ {i}, \bar {\boldsymbol {x}} ^ {i}\right). \tag {3}
$$
Here  $\pmb{\mu}^{i} = [\mu_{1}^{i},\mu_{2}^{i}\dots \mu_{M}^{i}]\in R^{V*M}$  denotes all the mean values of  $M$  slices of the input, and  $\hat{\pmb{\mu}}^i\in R^{V*K}$  stands for the predicted mean of future  $K$  slices. The same notation works for the standard deviation. The mean squared error (MSE) between predicted statistics and ground truth is utilized as the loss function  $(l_{sp})$  to train the network through backpropagation.
In this work, we mainly focus on proposing and modeling the non-stationary nature of time series from a slicing perspective. The challenge of how to design powerful deep models for statistics prediction is left for future explorations.
3.3 Denormalization
Simultaneous with statistic prediction, SAN feeds the normalized sequence into the forecasting model, which is responsible for producing internal output  $\bar{\pmb{y}}^i$ . Finally, SAN denormalizes the output given by the backbone, restoring the non-stationary factors for an accurate forecasting result.
Symmetrically, SAN performs on a per-slice basis as illustrated in Fig. 2. For the internal output  $\bar{\mathbf{y}}^i$ , we first split it into  $K$  non-overlapping slices  $\{\bar{\mathbf{y}}_j^i\}_{j=1}^K$ . Then the denormalization operation for an arbitrary slice based on our predicted statistics can be defined as the following formula:
$$
\hat {\boldsymbol {y}} _ {j} ^ {i} = \bar {\boldsymbol {y}} _ {j} ^ {i} * \left(\hat {\boldsymbol {\sigma}} _ {j} ^ {i} + \epsilon\right) + \hat {\boldsymbol {\mu}} _ {j} ^ {i}. \tag {4}
$$
Finally, by restoring all the slices in their chronological order, we can get the final prediction  $\hat{\pmb{y}}^i$  of the whole framework, which will be later used for loss computation  $(l_{fc})$  and performance evaluation.
3.4 Two-stage Training Schema
Though the overall framework is simple and clear, we find that the training process needs to be carefully deliberated. Since the normalization flow of SAN functions as a constraint to the backbone model, the overall learning procedure is actually a bi-level optimization problem [12]. The goal of the upper level is the performance of time series forecasting while the goal of the lower level is the distribution similarity between denormalized output and ground truth. Formally, the original overall training process can be described as:
$$
\begin{array}{l} \arg \min  _ {\theta} \sum_ {i = 1} ^ {n} l _ {f c} (\theta , \phi^ {*}, (\boldsymbol {x} ^ {i}, \boldsymbol {y} ^ {i})), \\ \text {s . t .} \phi^ {*} = \arg \min  _ {\phi} \sum_ {\left(\boldsymbol {x} ^ {i}, \boldsymbol {y} ^ {i}\right)} l _ {s p} \left(\theta , \phi , \left(\boldsymbol {x} ^ {i}, \boldsymbol {y} ^ {i}\right)\right). \tag {5} \\ \end{array}
$$
Here we omit the transformation process of the data and only keep the original input required for the calculation for brevity.
We propose a two-stage training paradigm for SAN by simplifying the lower-level optimization objective so that it can focus on estimating the future distribution, instead of reducing the distribution discrepancy between the denormalized output of a certain model and the ground truth. Specifically, we optimize  $\phi^{*} = \arg \min_{\phi}\sum_{(\pmb{x}^{i},\pmb{y}^{i})}l_{sp}(\phi ,(\pmb{x}^{i},\pmb{y}^{i}))$  using stochastic gradient descent. This decouples the original non-stationary forecasting task into a statistic prediction task and a stationary forecasting task. In practice, the statistics prediction module is first trained into convergence, which is then frozen and treated as a plugin during the second stage of training the forecasting model. The training algorithm is provided in the Appendix C.2.
Such a solution has some desirable qualities: The first is simplicity. The two-stage schema allows for a concise and easy-to-follow design of the model architecture and training process. The second is effectiveness. The statistics prediction module is expected to produce reliable predictions on future distribution since it is optimized on the whole training set into convergence. Therefore the forecasting model can handle the simpler task of learning the scale-free pattern in the normalized data. These two advantages greatly meet our ultimate goal of designing a concise yet effective framework for non-stationary time series forecasting tasks. The third and the most important is flexibility. Though there exist many advanced methods for the bi-level optimization problem [5, 11], their upper and lower objectives are always highly related. In contrast, our proposal completely decouples these parts, making SAN a model-agnostic framework that can migrate to various scenarios without special design and further tuning.
In this section, we conduct sufficient experiments within a widely used benchmark dataset compared to state-of-the-art methods to evidence the effectiveness of our proposed SAN framework.
Datasets We use nine datasets in our experiment and here are brief descriptions of them. (1)  $\mathbf{ETT}^3$  [43] records the oil temperature and load features of the electricity transformers from July 2016 to July 2018. It is made up of 4 sub datasets where ETThs are sampled per hour and ETTms are
<sup>3</sup>https://github.com/zhouhaoyi/ETDataset