SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yi-Fan Zhang $^{1,2}$ , Qingsong Wen $^{3*}$ , Xue Wang $^{3}$ , Weiqi Chen $^{3}$ , Liang Sun $^{3}$ , Zhang Zhang $^{1,2}$ , Liang Wang $^{1,2}$ , Rong Jin $^{3\dagger}$ , Tieniu Tan $^{1,2}$
$^{1}$ School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)
$^{2}$  State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation
3Alibaba Group
Online updating of time series forecasting models aims to address the concept drifting problem by efficiently updating forecasting models based on streaming data. Many algorithms are designed for online time series forecasting, with some exploiting cross-variable dependency while others assume independence among variables. Given every data assumption has its own pros and cons in online time series modeling, we propose Online ensembling Network (OneNet). It dynamically updates and combines two models, with one focusing on modeling the dependency across the time dimension and the other on cross-variate dependency. Our method incorporates a reinforcement learning-based approach into the traditional online convex programming framework, allowing for the linear combination of the two models with dynamically adjusted weights. OneNet addresses the main shortcoming of classical online learning methods that tend to be slow in adapting to the concept drift. Empirical results show that OneNet reduces online forecasting error by more than  $50\%$  compared to the State-Of-The-Art (SOTA) method. The code is available at https://github.com/yfzhang114/OneNet.
In recent years, we have witnessed a significant increase in research efforts that apply deep learning to time series forecasting [Lim and Zohren, 2021, Wen et al., 2022]. Deep models have proven to perform exceptionally well not only in forecasting tasks, but also in representation learning, enabling the extraction of abstract representations that can be effectively transferred to downstream tasks such as classification and anomaly detection. However, existing studies have focused mainly on the batch learning setting, assuming that the entire training dataset is available beforehand, and the relationship between the input and output variables remains constant throughout the learning process. These approaches fall short in real-world applications where concepts are often not stable but change over time, known as concept drift [Tsymbal, 2004], where future data exhibit patterns different from those observed in the past. In such cases, re-training the model from scratch could be time-consuming. Therefore, it is desirable to train the deep forecaster online, incrementally updating the forecasting model with new samples to capture the changing dynamics in the environment.
The real world setting, termed online forecasting, poses challenges such as high noisy gradients compared to offline mini-batch training [Aljundi et al., 2019a], and continuous distribution shifts which can make the model learned from historical data less effective for the current prediction. While some studies have attempted to address the issues by designing advanced updating structures or learning objectives [Pham et al., 2023, You et al., 2021], they all rely on TCN backbones [Bai et al.,
$^{\dagger}$ Work done at Alibaba Group, and now affiliated with Meta.
Table 1: A motivating example for online ensembling, where the reported metric is MSE and the forecast horizon length is set as 48. Cells are colored on the basis of the MSE value, from low (red) to medium (white) to high (blue). Columns titled cross-variable refer to methods that focus on modeling cross-variable dependence, and columns titled cross-time refer to methods that only exploit the temporal dependence and assume independence among covariates. All methods use the same training and online adaptation strategy.
Dataset	#Variables	Cross-Variable	Cross-Time	Both	Ours
TCN	FSNet	Time-TCN	DLinear	PatchTST	CrossFormer	TS-Mixer	Fedformer
ETTh2	7	0.910	0.846	1.307	6.910	2.716	5.772	3.060	1.620	0.609
ETTm1	7	0.250	0.127	0.308	1.120	0.553	0.370	0.660	0.516	0.108
WTH	21	0.348	0.223	0.308	0.541	0.465	0.317	0.482	0.372	0.200
ECL	321	10.800	7.034	5.230	7.388	5.030	94.790	5.764	27.640	2.201
(a) ETTh2.
(b) ETTm1.
Figure 1: A motivating example for online ensembling, where the reported metric is MSE and forecast horizon length is set to 48 during online adaptation. Cross-Time refers to a TCN backbone that assumes independence among covariates and only models the temporal dependence, and cross-variable refers to a TCN backbone that takes into cross-variable dependence.
(c) WTH.
(d) ECL.
2018], which do not take advantage of more advanced network structures, such as transformer [Nie et al., 2023, Zhou et al., 2022b]. Our studies show that the current transformer-based model, PatchTST [Nie et al., 2023], without any advanced adaption method of online learning, performs better than the SOTA online adaptation model FSNet [Pham et al., 2023], particularly for the challenging ECL task (Table.1). Furthermore, we find that variable independence is crucial for the robustness of PatchTST. Specifically, PatchTST focuses on modeling temporal dependency (cross-time dependency) and predicting each variable independently. To validate the effectiveness of the variable independence assumption, we designed Time-TCN, which convolves only on the temporal dimension. Time-TCN is better than FSNet, a state-of-the-art approach for online forecasting, and achieves significant gains compared to the commonly used TCN structure that convolves on variable dimensions.
Although variable independence enhances model robustness, the cross-variable dependency is also critical for forecasting, i.e. for a specific variable, information from associated series in other variables may improve forecasting results. As shown in Table 1 for datasets ETTm1 and ETTh2, cross-time forecasters tend to yield lower performance for datasets with a small number of variables. Surprisingly, existing models that are designed to leverage both cross-variable and cross-time dependencies such as CrossFormer [Zhang and Yan, 2023] and TS-Mixer [Chen et al., 2023], tend to perform worse than a native TCN. To investigate this phenomenon, we visualized the MSE at different time steps during the entire online adaptation process for both a Cross-Time model (Time-TCN) and a Cross-Variable model (TCN) in Figure 1. We observe a large fluctuation in MSE over online adaption, indicating a significant concept drift over time. We also observe that neither of these two methods performs consistently better than the other, indicating that neither of the two data assumptions holds true for the entire time series. This is why relying on a single model like CrossFormer cannot solve this problem. Existing work depends on a simple model, but for online time series forecasting, data preferences for model bias will continuously change with online concept drifts. Therefore, we need a data-dependent strategy to continuously change the model selection policy. In other words, online time series forecasting should go beyond parameter updating.
In this paper, we address the limitation of a single model for online time series forecasting by introducing an ensemble of models that share different data biases. We then learn to dynamically combine the forecasts from individual models for better prediction. By allowing each model to be trained and online updated independently, we can take the best out of each online model; by dynamically
adjusting the combination of different models, we can take the best out of the entire model ensemble. We refer to our approach as Online Ensembling Network or OneNet for short. More concretely, OneNet maintains two online forecasting models, one focused on modeling temporal correlation and one focused on modeling cross-variable dependency. Each model is trained independently using the same set of training data. During testing, a reinforcement learning (RL) based approach is developed to dynamically adjust the weights used to combine the predictions of the two models. Compared to classical online learning methods such as Exponentiated Gradient Descent, our RL-based approach is more efficient in adapting to the changes/drifts in concepts, leading to better performance. The contributions of this paper are:
1. We introduce OneNet, a two-stream architecture for online time series forecasting that integrates the outputs of two models using online convex programming. OneNet leverages the robustness of the variable-independent model in handling concept drift, while also capturing the inter-dependencies among different variables to enhance forecasting accuracy. Furthermore, we propose an RL-based online learning approach to mitigate the limitations of traditional OCP algorithms and demonstrate its efficacy through empirical and theoretical analyses.
2. Our empirical studies with four datasets show that compared with state-of-the-art methods, OneNet reduces the average cumulative mean-squared errors (MSE) by  $53.1\%$  and mean-absolute errors (MAE) by  $34.5\%$ . In particular, the performance gain on challenging dataset ECL is superior, where the MSE is reduced by  $59.2\%$  and MAE is reduced by  $63.0\%$ .
3. We conducted comprehensive empirical studies to investigate how commonly used design choices for forecasting models, such as instance normalization, variable independence, seasonal-trend decomposition, and frequency domain augmentation, impact the model's robustness. In addition, we systematically compared the robustness of existing Transformer-based models, TCN-based models, and MLP-based models when faced with concept drift.
Concept drift. Concepts in the real world are often dynamic and can change over time, which is especially true for scenarios like weather prediction and customer preferences. Because of unknown changes in the underlying data distribution, models learned from historical data may become inconsistent with new data, thus requiring regular updates to maintain accuracy. This phenomenon, known as concept drift [Tsymbal, 2004], adds complexity to the process of learning a model from data. In this paper, we focus on online learning for time series forecasting. Unlike most existing studies for online time series forecasting [Li et al., 2022, Qin et al., 2022, Pham et al., 2023] that only focus on how to online update their models, this work goes beyond parameter updating and introduces multiple models and a learnable ensembling weight, yielding rich and flexible hypothesis space. Due to the space limit, more related works about time series forecasting and reinforcement learning are left in the appendix.
Online time series forecasting: streaming data. Traditional time series forecasting tasks have a collection of multivariate time series with a look-back window  $L$ :  $(\mathbf{x}_i)_{i=1}^L$ , where each  $\mathbf{x}_i$  is  $M$ -channel vector  $\mathbf{x}_i = (x_i^j)_{j=1}^M$ . Given a forecast horizon  $H$ , the target is to forecast  $H$  future values  $(\mathbf{x}_i)_{i=L+1}^{L+H}$ . In real-world applications, the model builds on the historical data needs to forecast the future data, that is, given time offset  $K' > L$ , and  $(\mathbf{x}_i)_{i=K'-L+1}^{K'}$ , the model needs to forecast  $(\mathbf{x})_{i=K'+1}^{K'+H}$ . Online time series forecasting [Anava et al., 2013, Liu et al., 2016, Pham et al., 2023] is a widely used technique in real-world due to the sequential nature of the data and the frequent drift of concepts. In this approach, the learning process takes place over a sequence of rounds, where the model receives a look-back window and predicts the forecast window. The true values are then revealed to improve the model's performance in the next rounds. When we perform online adaptation, the model is retrained using the online data stream with the MSE loss over each channel:
$$
\mathcal {L} = \frac {1}{M} \sum_ {j = 1} ^ {M} \parallel \hat {x} _ {K ^ {\prime} + 1: K ^ {\prime} + H} ^ {j} - x _ {K ^ {\prime} + 1: K ^ {\prime} + H} ^ {j} \parallel .
$$
Variable-independent time series forecasting. The traditional cross-variable strategy used in most structures takes the vector of all time series features as input and projects it into the embedding space to mix the information. On the contrary, PatchTST [Nie et al., 2023] adopts a variable-independent approach, where each input token only contains information from a single channel/variable. Our research demonstrates that variable independence is crucial for boosting model robustness under
(a) The overall OneNet architecture.
(b) OCP block.
Figure 2: (a) OneNet processes multivariate data through cross-time and cross-variable branches, each responsible for capturing different aspects. The weights of these two branches are generated by the OCP block, and only the black arrows require execution during training. (b) The OCP block produces ensembling weights by utilizing both the long-term history of exponential gradient descent (EGD) and the short-term history of offline reinforcement learning (RL).
concept drift. For multivariate time series samples  $(x_{i}^{j})_{i = 1}^{L}$ , each channel  $j$  is fed into the model independently, and the forecaster produces prediction results  $(x_{i}^{j})_{i = L + 1}^{L + H}$  accordingly. As shown in Table 1, cross-variable methods tend to overfit when the dataset has a large number of variables, resulting in poor performance. This is evident in the poor performance of the SOTA online adaptation model FSNet [Pham et al., 2023] in the ECL dataset. However, models that lack cross-variable information perform worse on datasets with a small number of variables where cross-variable dependency can be essential. Although some existing work has attempted to incorporate both cross-variable interaction and temporal dependency into a single framework, our experiments show that these models are fragile under concept drift and perform no better than the proposed simple baseline, Time-TCN. To address this, we propose a novel approach that trains two separate branches, each focusing on modeling temporal and cross-variable dependencies, respectively. We then combine the results of these branches to achieve better forecasting performance under concept drift. We first introduce the OCP block for coherence.
3 OneNet: Ensemble Learning for Online Time Series Forecasting
We first examine online learning methods to dynamically adjust combination weights used by ensemble learning. We then present OneNet, an ensemble learning framework for online time series forecasting.
3.1 Learning the best expert by Online Convex Programming (OCP)
For notation clarity, here we denote  $\mathbf{x} \in \mathbb{R}^{L \times M}$  as the historical data,  $\mathbf{y} \in \mathbb{R}^{H \times M}$  as the forecast target. Our current method involves the integration of multiple complementary models. Therefore, how to better integrate model predictions in the online learning setting is an important issue. Exponentiated Gradient Descent (EGD) [Hill and Williamson, 2001] is a commonly used method. Specifically, the decision space  $\triangle$  is a  $d$ -dimensional simplex, i.e.  $\triangle = \{\mathbf{w}_t | w_{t,i} \geq 0$  and  $\| \mathbf{w}_t \|_1 = 1\}$ , where  $t$  is the time step indicator and we omit the subscript  $t$  for simplicity when it's not confusing. Given the online data stream  $\mathbf{x}$ , its forecasting target  $\mathbf{y}$ , and  $d$  forecasting experts with different parameters  $\mathbf{f} = [\tilde{\mathbf{y}}_i = f_i(\mathbf{x})]_{i=1}^d$ , the player's goal is to minimize the forecasting error as
$$
\min  _ {\mathbf {w}} \mathcal {L} (\mathbf {w}) := \| \sum_ {i = 1} ^ {d} w _ {i} f _ {i} (\mathbf {x}) - \mathbf {y} \| ^ {2}; \quad s. t. \quad \mathbf {w} \in \triangle . \tag {1}
$$
According to EGD, choosing  $\mathbf{w}_1 = [w_{1,i} = 1 / d]_{i = 1}^d$  as the center point of the simplex and denoting  $\ell_{t,i}$  as the loss of  $f_{i}$  at time step  $t$ , the updating rule for each  $w_{i}$  will be
$$
w _ {t + 1, i} = \frac {w _ {t , i} \exp (- \eta \| f _ {i} (\mathbf {x}) - \mathbf {y} \| ^ {2})}{Z _ {t}} = \frac {w _ {t , i} \exp (- \eta \ell_ {t , i})}{Z _ {t}} \tag {2}
$$
where  $Z_{t} = \sum_{i = 1}^{d}w_{t,i}\exp (-\eta l_{t,i})$  is the normalizer, and the algorithm has a regret bound:
Proposition 1. (Online Convex Programming Bound) For  $T > 2\log(d)$ , denote the regret for time step  $t = 1, \ldots, T$  as  $R(T)$ , set  $\eta = \sqrt{2\log(d)/T}$ , the OCP updating policy has an External regret (See appendix B.1 for proof and analysis.)
$$
\sum_ {t = 1} ^ {T} \mathcal {L} (\mathbf {w} _ {t}) - \inf  _ {\mathbf {u}} \sum_ {t = 1} ^ {T} \mathcal {L} (\mathbf {u}) \leq \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {d} w _ {t, i} \| f _ {i} (\mathbf {x}) - \mathbf {y} \| ^ {2} - \inf  _ {\mathbf {u}} \sum_ {t = 1} ^ {T} \mathcal {L} (\mathbf {u}) \leq \sqrt {2 T \log (d)} \tag {3}
$$
That is, the exponentially weighted average forecaster guarantees that the forecaster's cumulative expected loss is not much larger than the cumulative loss of the best decision. However, an exponentially weighted average forecaster is widely known to respond very slowly to drastic changes in the distribution [Cesa-Bianchi and Lugosi, 2006]. This phenomenon is sometimes referred to as the "slow switch phenomenon" in online learning literature, and is further illustrated in Figure 3 where the loss for  $f_{1}$  is 0 for the first 50 trials and 1 for the next 50 trials. The performance of  $f_{2}$  is the opposite. When the step size  $\eta$  is small (e.g.,  $\eta = 0.01$ ), small changes are made to the weights and no clear adaptation takes place. When a large step size  $\eta$  is applied (e.g.,  $\eta = 1$ ), we observe that the EGD algorithm quickly adapts to the environment change for the first 50 trials by increasing weight  $w_{1}$  to almost 1 in the first few iterations. But it takes many iterations for the EGD algorithm to adapt to the change in the next 50 iterations, where  $f_{2}$  works much better than  $f_{1}$ . We finally note that no matter how we adjust the step size  $\eta$ , the EGD algorithm has to suffer from the trade-off between speed of switching and overall good performance throughout the horizon.
Although few algorithms have been developed to address this issue in online learning [Stoltz and Lugosi, 2005, Cesa-Bianchi and Lugosi, 2003, Blum and Mansour, 2007, Foster and Vohra, 1998], the key idea is to find an activation function that maps the original policy  $\mathbf{w}_t$  to a new one based on the recent loss of all experts. Despite the efforts, very limited successes have been achieved, either empirically or theoretically. In this work, we observe in our experiments that the combination weights  $\mathbf{w}$  generated by the EGD algorithm are based on historical performance over a long period of time and thus cannot adapt quickly to transient environment changes. Hence, it is better to effectively incorporate both long-term historical information and more recent changes in the environment. A straightforward idea is to re-initialize the weight  $\mathbf{w}$  per  $K$  steps. We show that such a simple algorithm can achieve a tighter bound:
Figure 3: The evolution of the weight assigned to  $f_{1}$  where the losses for forecasters vary across the first regime [0, 50] and the second regime [50, 100].
Proposition 2. (Informal) Denote  $I = [l, \dots, r] \in [1, \dots, T]$  as any period of time. We then have the  $K$ -step re-initialize algorithm has a tighter regret bound compared to EGD at any small interval  $I$ , where  $|I| < T^{\frac{3}{4}}$ . (See appendix B.2 for proof.)
Proposition 2 stresses that, by considering short-term information, we can attain lower regret in short time intervals. Such a simple strategy still struggles with the hyper-parameter choice of  $K$ . Besides, discarding long-term information makes the algorithm inferior to EGD for a long period of the online learning process. In this work, we address this challenge of online learning by exploiting offline reinforcement learning [Levine et al., 2020]. At first, we use EGD to maintain long-term weight  $\mathbf{w}$ . Besides, we introduce a different set of weights  $\mathbf{b}$  that can better capture the recent performance of individual models. By combining  $\mathbf{w}$  and  $\mathbf{b}$ , our approach can effectively incorporate both long-term historical information and more recent changes in the environment.
Specifically, we adopt the RvS [Emmons et al., 2022] framework, which formulates reinforcement learning through supervised learning, as shown in Figure 2(b). At time step  $t$ , our target is to learn
a short-term weight conditioned on the long-term weight  $\mathbf{w}$  and experts' performances during a short period of history  $I = [l,t]$ . For simplicity and computation efficiency, we just let  $l = t - 1$ . The agent then chooses actions using a policy  $\pi_{\theta_{rl}}\left(\mathbf{b}_t|\{\{w_{t,i}\tilde{y}_i\}_{i = 1}^d\}_{t\in I};\mathbf{y}\right)$  parameterized by  $\theta_{rl}$ . During training, we concatenate the product between each prediction and expert weight  $(w_{t,i}*\tilde{\mathbf{y}}_i)$  with the outcome  $\mathbf{y}$  as the conditional input. We follow RvS [Emmons et al., 2022] to implement the policy network as a two-layer MLPs  $f_{rl}:\mathbb{R}^{H\times M\times (d + 1)}\to \mathbb{R}^d$ . Then the short-term weight and final ensembling weight will be:
$$
\mathbf {b} _ {t} = f _ {r l} \left(w _ {t, 1} \tilde {\mathbf {y}} _ {1} \otimes \dots \otimes w _ {t, d} \tilde {\mathbf {y}} _ {d} \otimes \mathbf {y}\right) \text {a n d} \tilde {w} _ {t, i} = \left(w _ {t, i} + b _ {t, i}\right) / \left(\sum_ {i = 1} ^ {d} \left(w _ {t, i} + b _ {t, i}\right)\right) \tag {4}
$$
However, unlike in RvS, we cannot train the decision network through simple classification tasks since the ground truth target action is inaccessible. Instead, we propose to train the network by minimizing the forecasting error incurred by the new weight, that is,  $\min_{\theta_{rl}}\mathcal{L}(\tilde{\mathbf{w}})\coloneqq \parallel \sum_{i = 1}^{d}\tilde{w}_{t,i}f_{i}(\mathbf{x}) - \mathbf{y}\parallel^{2}$ . During inference, as concept drift changes gradually, we use  $\mathbf{w}_{t - 1} + \mathbf{b}_{t - 1}$  to generate the prediction and train the networks after the ground truth outcome is observed. We theoretically and empirically verify the effectiveness of the proposed OCP block in appendix B.4.
3.2 OneNet: utilizing the advantages of both structures
The model structure is shown in Figure 2(a) and we introduce the components as follows: