SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1 Introduction 1
2 Preliminary and Related Work 3
3 OneNet: Ensemble Learning for Online Time Series Forecasting 4
3.1 Learning the best expert by Online Convex Programming (OCP) 4
3.2 OneNet: utilizing the advantages of both structures 6
4.1 Experimental setting 7
4.2 Online forecasting results 7
4.3 Ablation studies and analysis 8
5 Conclusion and Future Work 10
A Extended Related Work 15
B Proofs of Theoretical Statements 16
B.1 Online Convex Programming Regret Bound 16
B.2 Theoretical guarantee for the  $K$  -step re-initialize algorithm. 16
B.3 Necessary definitions and assumptions for evaluating model adaptation speed to environment changes. 17
B.4 Existing theoretical intuition and empirical comparison to the proposed OCP block. 18
C Additional Experimental Results 20
C.1 Datasets 20
C.2 Implementation Details 21
C.3 Baseline details 21
C.4 Hyper-parameters 21
C.5 Additional Numerical Results 22
C.6 Ensembling more than two networks. 23
C.7 More visualization results and Convergence Analysis of Different Structures 23
C.8 Online forecasting results with delayed feedback 23
C.9 The effect of variable independence and frequency domain augmentation 25
C.10 Comparison of existing forecasting structures. 25
Concept Drift Concepts in the real world are often dynamic and can change over time, which is especially true for scenarios like weather prediction and customer preferences. Because of unknown changes in the underlying data distribution, models learned from historical data may become inconsistent with new data, thus requiring regular updates to maintain accuracy. This phenomenon, known as concept drift [Tsymbal, 2004], adds complexity to the process of learning a model from data. Concept drift poses several challenging subproblems, ranging from fast learning under concept drift [Pham et al., 2023, Zhang et al., 2020, Gama et al., 2014], which involves adjusting the offline model with new observations to recognize recent patterns, to forecasting future data distributions [Li et al., 2022, Qin et al., 2022], which predicts the data distribution of the next time-step sequentially, enabling the model of the downstream learning task to be trained on the data sample from the predicted distribution. In this paper, we focus on the first problem, i.e. online learning for time series forecasting. Unlike most existing studies for online time series forecasting [Li et al., 2022, Qin et al., 2022, Pham et al., 2023] that only focus on how to online update their models, this work goes beyond parameter updating and introduces multiple models and a learnable assembling weight, yielding rich and flexible hypothesis space.
Test-Time adaptive methods is similar to online time series forecasting but mainly focuses on domain generalization [Zhang et al., 2022c,a, Zhou et al., 2022a], domain adaptation [Zhang et al., 2023b] and other tasks [Liang et al., 2023]. These recently proposed to utilize target samples. Test-Time Training methods design proxy tasks during tests such as self-consistence [Zhang et al., 2021], rotation prediction [Sun et al., 2020] and need extra models; Test-Time Adaptation methods adjust model parameters based on unsupervised objectives such as entropy minimization [Wang et al., 2020] or update a prototype for each class [Iwasawa and Matsuo, 2021]. Domain adaptive method [Dubey et al., 2021] needs additional models to adapt to target domains. There are also some methods that do not need test-time tunning, for example, [Zhang et al., 2022b] introduces specific classifiers for different domains and adapts the voting weight for test samples dynamically and [Zhang et al., 2023a] apply non-parametric test-time adaptation.
Time Series Modeling Time series models have been developed for decades and are fundamental in various fields. While autoregressive models like ARIMA [Box and Pierce, 1970] were the first data-driven approaches, they struggle with nonlinearity and non-stationarity. Recurrent neural networks (RNNs) were designed to handle sequential data, with LSTM [Graves and Graves, 2012] and GRU [Chung et al., 2014] using gated structures to address gradient problems. Attention-based RNNs [Qin et al., 2017] use temporal attention to capture long-range dependencies, but are not parallelizable and struggle with long dependencies. Temporal convolutional networks [Sen et al., 2019] are efficient, but have limited reception fields and struggle with long-term dependencies. Recently, transformer-based [Vaswani et al., 2017, Wen et al., 2022] models have been renovated and applied in time series forecasting. Although a large body of work aims to make Transformer models more efficient and powerful [Zhou et al., 2022b, 2021, Nie et al., 2023], we are the first to evaluate the robustness of advanced forecasting models under concept drift, making them more adaptable to new distributions.
Reinforcement learning and offline reinforcement learning. Reinforcement learning is a mathematical framework for learning-based control, which allows us to automatically acquire policies that represent near-optimal behavioral skills to optimize user-defined reward functions [Hafner and Riedmiller, 2011, Silver et al., 2017]. The reward function specifies the objective of the agent, and the reinforcement learning algorithm determines the actions necessary to achieve it. However, the online learning paradigm of reinforcement learning is a major obstacle to its widespread adoption. The iterative process of collecting experience by interacting with the environment is expensive or dangerous in many settings, making offline reinforcement learning a more feasible alternative. Offline RL [Prudencio et al., 2023, Levine et al., 2020] learns exclusively from static datasets of previously collected interactions, enabling the extraction of policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL. Although there are many types of offline RL algorithms, such as those using value functions [Fujimoto et al., 2019], dynamics estimation [Kidambi et al., 2020], or uncertainty quantification [Agarwal et al., 2020], RvS [Emmons et al., 2022] has shown that simple conditioning with standard feedforward networks can achieve state-of-the-art results. In this work, we draw inspiration from RvS and learn an additional bias term for the OCP block for simplicity.
B Proofs of Theoretical Statements
B.1 Online Convex Programming Regret Bound
Proposition 1. For  $T > 2\log(d)$ , denote the regret for time step  $t = 1, \ldots, T$  as  $R(T)$ , set  $\eta = \sqrt{2\log(d) / T}$ , and the EGD update policy has regret.
$$
R (T) = \sum_ {t = 1} ^ {T} \mathcal {L} (\mathbf {w} _ {t}) - \inf  _ {\mathbf {u}} \sum_ {t = 1} ^ {T} \mathcal {L} (\mathbf {u}) \leq \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {d} w _ {t, i} \| f _ {i} (\mathbf {x}) - \mathbf {y} \| ^ {2} - \sum_ {t = 1} ^ {T} \inf  _ {\mathbf {u}} \mathcal {L} (\mathbf {u}) \leq \sqrt {2 T \log (d)} \tag {5}
$$
Proof. The proof of Exponentiated Gradient Descent is well-studied [Ghai et al., 2020] and here we provide a simple Regret bound for both OCP.
Denote  $\ell_{t,i} = \| f_i(\mathbf{x}) - \mathbf{y}\|^2$ , recall that the normalizer for OCP-U is  $Z_{t + 1} = \sum_{i = 1}^{d}w_{t,i}\exp (-\eta \ell_{t,i})$ , then we have
$$
\log \frac {Z _ {t + 1}}{Z _ {t}} = \log \frac {\sum_ {i = 1} ^ {d} w _ {t , i} \exp (- \eta \ell_ {t , i})}{Z _ {t}} = \log \sum_ {i = 1} ^ {d} p _ {t, i} \exp (- \eta \ell_ {t, i}), \tag {6}
$$
where  $p_{t,i} = w_{t,i} / Z_t \leq 1$ . For clarity, we let  $w_{t,i}$  be the unnormalized weight and  $p_{t,i}$  be the normalized weight. Now, we assume  $\eta \ell_{t,i} \in [0,1]$ . Although it is not guaranteed that  $\ell_{t,i}$  will be small under concept shift, it is generally safe to assume that the concept will shift gradually and will not lead to a drastic change in the loss. As a result, the loss will not become arbitrarily large, and we can divide some large constant such that the loss is bounded in a small range. Based on the assumption, we can use the second Taylor expansion of  $e^{-x} \leq 1 - x + x^2 / 2$  and the inequation  $\log (1 - x) \leq -x$  for  $x \in [0,1]$ . Then we have
$$
\begin{array}{l} \log \sum_ {i = 1} ^ {d} p _ {t, i} \exp (- \eta \ell_ {t, i}) \leq \log \left(1 - \eta \sum_ {i = 1} ^ {d} p _ {t, i} \ell_ {t, i} + \frac {\eta^ {2}}{2} \sum_ {i = 1} ^ {d} p _ {t, i} \ell_ {t, i} ^ {2}\right) (7) \\ \leq - \eta \sum_ {i = 1} ^ {d} p _ {t, i} \ell_ {t, i} + \frac {\eta^ {2}}{2} \sum_ {i = 1} ^ {d} p _ {t, i} \ell_ {t, i} ^ {2} \leq - \eta \sum_ {i = 1} ^ {d} p _ {t, i} \ell_ {t, i} + \frac {\eta^ {2}}{2} (8) \\ \end{array}
$$
Note that  $w_{t,i}$  is the unnormalized weight, and then  $w_{1,i} = 1$  and  $Z_{1} = d$ . Then we can get the lower bound and upper bound of  $\log Z_{T + 1}$ :
$$
\log Z _ {T + 1} = \sum_ {t = 1} ^ {T} \log \frac {Z _ {t + 1}}{Z _ {t}} + \log \left(Z _ {1}\right) \leq - \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {d} p _ {t, i} \ell_ {t, i} + \frac {T \eta^ {2}}{2} + \log (d) \tag {9}
$$
$$
\log Z _ {T + 1} = \log \sum_ {i = 1} ^ {d} w _ {t, i} \exp (- \eta \ell_ {t, i}) \geq - \eta \sum_ {i = 1} ^ {d} \ell_ {t, i} \tag {10}
$$
Finally, recall that we set  $\eta = \sqrt{2\log(d) / T}$ , then we have
$$
\eta \left(\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {d} p _ {t, i} \ell_ {t, i} - \sum_ {i = 1} ^ {d} \ell_ {t, i}\right) \leq \eta \left(\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {d} p _ {t, i} \ell_ {t, i} - \inf  _ {\mathbf {u}} \sum_ {t = 1} ^ {T} \mathcal {L} (\mathbf {u})\right) \leq \frac {T \eta^ {2}}{2} + \log (d), \tag {11}
$$
B.2 Theoretical guarantee for the  $K$ -step re-initialize algorithm.
The proposed  $K$ -step re-initialize algorithm is detailed as follows: at the beginning of the algorithm, we choose  $\mathbf{w}_1 = [w_{1,i} = 1 / d]_{i = 1}^d$  as the center point of the simplex and denote  $\ell_{t,i}$  as the loss for  $f_{i}$  at time step  $t$ , the updating rule for each  $w_{i}$  will be  $w_{t + 1,i} = \frac{w_{t,i}\exp(-\eta[\partial\mathcal{L}_{U}(\mathbf{w}_{t})]_{i})}{Z_{t}} = \frac{w_{t,i}\exp(-\eta\|f_{i}(\mathbf{x}) - \mathbf{y}\|^{2})}{Z_{t}} = \frac{w_{t,i}\exp(-\eta\ell_{t,i})}{Z_{t}}$ , where  $Z_{t} = \sum_{i = 1}^{d}w_{t,i}\exp (-\eta l_{t,i})$  is the normalizer.
Different from the native EGD algorithm, we re-initialize the weight  $\mathbf{w}_{K + 1} = [w_{K + 1,i} = 1 / d]_{i = 1}^{d}$  per  $K$  time steps. We call each  $K$  step one round. This simple strategy interrupts the influence of the historical information of length  $K$  steps on the ensembling weights, which helps the model to quickly adapt to the upcoming environment.
Proposition 2. For  $T > 2\log(d)$ , denote  $I = [l, l + 1, \dots, r]$  as any period of time of length  $r - l + 1$  where  $l > 1$  and  $r \leq T$ . Denote the length of  $I$  as a sublinear sequence of  $T$ , namely,  $|I| = T^n$ , where  $0 \leq n \leq 1$ . We choose  $K = T^{\frac{2n}{3}}$ . We then have, the  $K$ -step re-initialize algorithm has an regret bound  $R(I) \leq \mathcal{O}(T^{2n/3})$  at any  $I = [l, l + 1, \dots, r]$ . Namely, for any small internal  $n < \frac{3}{4}$ , we have  $R([l, l + 1, \dots, r]) < \mathcal{O}(T^{1/2})$
Proof. We discuss regret in three cases:
- At first, according to Proposition 1, if all  $K$  steps fall into  $[l, l + 1, \dots, r]$ , then we have  $R(K) \leq 2\sqrt{K \ln d(d - 1)}$ . There exist  $L / K$  rounds that are all contained in  $[l, \dots, r]$  and the regret of these rounds will be  $\mathcal{O}(L / K * 2\sqrt{K})$ .
- For the first round, we do not know when the weights are reinitialized ( $l$  may or may not be a multiple of  $K$ ) and the performance of the algorithm in historical time. Let's think about the worst case, where regret will be less than  $\mathcal{O}(K)$ .
- For the last round, we know when the last round begins and the weights are reinitialized. However, some of the future time steps are not in  $[l, l + 1, \dots, r]$  and we can only treat the case as the first round, which has a regret  $\mathcal{O}(K)$ .
Considering all cases, we have an internal regret that is bounded by
$$
R (I) \leq \mathcal {O} (L / K \times \sqrt {K} + K) = \mathcal {O} (L / K \sqrt {K} + K) \tag {12}
$$
When we choose an small internal length  $L = T^n$  and  $K = T^m$  where  $m \leq n$ . To minimize the upper bound, we should choose  $m = \frac{2n}{3}$  and the bound will be  $\mathcal{O}(T^{2n/3})$ . Namely, for any small internal  $n < \frac{3}{4}$ , we have  $R([l, l + 1, \dots, r]) < \mathcal{O}(T^{1/2})$ , which is tighter than the regret bound of the EGD algorithm under the whole online sequence. Specifically, when we choose  $L = T^{2/3}$  and  $K = T^{4/9}$ , we have  $R([l, l + 1, \dots, r]) < \mathcal{O}(T^{4/9}) < \mathcal{O}(T^{1/2})$ . When we choose  $L = T^{1/4}$  and  $K = T^{1/6}$ , then  $R([l, l + 1, \dots, r]) < \mathcal{O}(T^{1/6}) < \mathcal{O}(T^{1/2})$ .
In other words, the algorithm focuses on short-term information and leads to a better regret bound in any small time interval. However, with increasing length of  $I$ , the bound of the simple algorithm will become worse. Consider an extreme case where  $n = 1$ , the bound will be  $R([l,l + 1,\dots ,r]) < \mathcal{O}(T^{2 / 3})$ , which is inferior to the native EGD algorithm.
B.3 Necessary definitions and assumptions for evaluating model adaptation speed to environment changes.
To complete the proofs, we begin by introducing some necessary definitions and assumptions. Given the online data stream  $\mathbf{x}_t$  and its forecasting target  $\mathbf{y}_t$  at time  $t$ . Given  $d$  forecasting experts with different parameters  $\mathbf{f}_t = \{f_{t,i}\}$ , denote  $\ell$  as a nonnegative loss function and  $\ell_{t,i} := \ell(f_{t,i}(\mathbf{x}_t), \mathbf{y}_t)$  as the loss incurred by  $f_{t,i}$  at time  $t$ , we define the following notions.
Definition 1. (Weighted average forecaster). A weighted average forecaster makes predictions by
$$
\tilde {\mathbf {y}} _ {t} = \frac {\sum_ {i = 1} ^ {d} w _ {t - 1 , i} f _ {t , i}}{\sum_ {i = 1} ^ {d} w _ {t - 1 , i}}, \tag {13}
$$
where  $w_{t,i}$  is the weight for expert  $f_i$  at time  $t$  and  $f_{t,i}$  is the prediction of  $f_i$  at time  $t$ .
Definition 2. (Cumulative regret and instantaneous regret). For expert  $f_{i}$ , the cumulative regret (or simply regret) on the  $T$  steps is defined by
$$
R _ {T, i} = \sum_ {t = 1} ^ {T} r _ {t, i} = \sum_ {t = 1} ^ {T} \left(\ell \left(\tilde {\mathbf {y}} _ {t}, \mathbf {y} _ {t}\right) - \ell_ {t, i}\right) = \hat {L} _ {T} - L _ {T, i} \tag {14}
$$
where  $r_{t,i}$  is the instantaneous regret of the expert  $f_i$  at time  $t$ , which is the regret that the forecaster feels of not having listened to the advice of the expert  $f_i$  right after the  $t$ th outcome that has been revealed.  $\hat{L}_T = \sum_{t=1}^{T} \ell(\tilde{\mathbf{y}}_t, \mathbf{y}_t)$  is the cumulative loss of the forecaster and  $L_{T,i}$  is the cumulative loss of the expert  $f_i$ .
Definition 3. (Potential function). We can interpret the weighted average forecaster in an interesting way which allows us to analyze the theoretical properties easier. To do this, we denote  $\mathbf{r}_t = (r_{t,1},\ldots ,r_{t,d})\in \mathbb{R}^d$  as the instantaneous regret vector, and  $\mathbf{R}_T = \sum_{t = 1}^T\mathbf{r}_t$  is the corresponding regret vector. Now, we can introduce the potential function  $\Phi :\mathbb{R}^d\to \mathbb{R}$  of the form
$$
\Phi (\mathbf {u}) = \psi \left(\sum_ {i = 1} ^ {d} \phi \left(u _ {i}\right)\right) \tag {15}
$$
where  $\phi : \mathbb{R} \to \mathbb{R}$  is any nonnegative, increasing, and twice differentiable function, and  $\psi : \mathbb{R} \to \mathbb{R}$  is nonnegative, strictly increasing, concave, and twice differentiable auxiliary function. With the notion of potential function, the prediction  $\tilde{\mathbf{y}}_t$  will be
$$
\tilde {\mathbf {y}} _ {t} = \frac {\sum_ {i = 1} ^ {d} \nabla \Phi \left(\mathbf {R} _ {t - 1}\right) _ {i} f _ {t , i}}{\sum_ {i = 1} ^ {d} \nabla \Phi \left(\mathbf {R} _ {t - 1}\right) _ {i}}, \tag {16}
$$
where  $\nabla \Phi (\mathbf{R}_{t - 1})_i = \partial \Phi (\mathbf{R}_{t - 1}) / \partial R_{t - 1,i}$ . It is easy to prove that the exponentially weighted average forecaster used in Eq.(2) is based on the potential  $\Phi_{\eta}(\mathbf{u}) = \frac{1}{\eta}\ln \left(\sum_{i = 1}^{d}e^{\eta u_i}\right)$ .
Theorem 1. (Blackwell condition, Lemma 2.1. in [Cesa-Bianchi and Lugosi, 2006].) If the loss function  $\ell$  is convex in its first argument and we use  $\mathbf{x}_1 \cdot \mathbf{x}_2$  denote the inner product of two vectors, then
$$
\sup  _ {\mathbf {y} _ {t}} \mathbf {r} _ {t} \cdot \nabla \Phi \left(\mathbf {R} _ {t - 1}\right) \leq 0 \tag {17}
$$
The following theorem is applicable to any forecaster that satisfies the Blackwell condition, not limited to weighted average forecasters. Nevertheless, this theorem will lead to several interesting bounds for various variations of the weighted average forecaster.
Theorem 2. (Theorem2.1 in [Cesa-Bianchi and Lugosi, 2006].) Assume that a forecaster satisfies the Blackwell condition for a potential  $\Phi$ , then for all  $i = 1, \dots$ ,
$$
\Phi \left(\mathbf {R} _ {T}\right) \leq \Phi (0) + \frac {1}{2} \sum_ {t = 1} ^ {T} C \left(\mathbf {r} _ {t}\right), \tag {18}
$$
$$
C \left(\mathbf {r} _ {t}\right) = \sup  _ {\mathbf {u} \in \mathbb {R} ^ {d}} \psi^ {\prime} \left(\sum_ {i = 1} ^ {d} \phi (u _ {i})\right) \sum_ {i = 1} ^ {d} \phi^ {\prime \prime} (u _ {i}) r _ {t, i} ^ {2}. \tag {19}
$$
B.4 Existing theoretical intuition and empirical comparison to the proposed OCP block.
With the help of the two theorems in Section B.3, we now recall that the Internal Regret  $R_{in}(t,\mathbf{w})$  [Blum and Mansour, 2007] that measures forecaster's expected regret of having taken an action  $\mathbf{w}$  at step  $t$ :
$$
R _ {i n} (T, \mathbf {w}) = \max  _ {i, j = 1, \dots , d} \sum_ {t = 1} ^ {T} r _ {t, (i, j)} = \max  _ {i, j = 1, \dots , d} \sum_ {t = 1} ^ {T} w _ {t, i} \left(\ell_ {t, i} - \ell_ {t, j}\right). \tag {20}
$$
While Proposition 1 ensures that a small external regret can be achieved, ensuring a small internal regret is a more challenging task. This is because any algorithm with a small internal regret also
has small external regret but the opposite is not true, as demonstrated in [Stoltz and Lugosi, 2005]. The key question now is whether it is possible to define a policy  $\mathbf{w}$  that attains small (i.e., sublinear in  $T$ ) internal regret. For simplicity, we use  $R$  as internal regret in this subsection. To develop a forecasting strategy that can guarantee a small internal regret. We define the exponential potential function  $\Phi : \mathbb{R}^M \to \mathbb{R}$  with  $\eta > 0$  by
$$
\Phi (\mathbf {u}) = \frac {1}{\eta} \ln \left(\sum_ {i = 1} ^ {M} e ^ {\eta u _ {i}}\right), \tag {21}
$$
where  $M = d(d - 1)$ . Here, we denote  $\mathbf{r}_t = (r_{t,(1,1)}, r_{t,(1,2)}, \dots, r_{t,(d,d - 1)}) \in \mathbb{R}^{d(d - 1)}$  as the instantaneous regret vector and  $\mathbf{R}_T = \sum_{t=1}^T \mathbf{r}_t$  is the corresponding regret vector. Then, any forecaster satisfying Blackwell's condition will have a bounded internal regret (Corollary 8 in [Cesa-Bianchi and Lugosi, 2003]) by choosing a proper parameter  $\eta$ :
$$
\max  _ {i, j} R _ {t, (i, j)} \leq 2 \sqrt {t \ln d (d - 1)} \tag {22}
$$
$$
R _ {i n} (t, \mathbf {w}) = \max  _ {i, j = 1, \dots , d} \sum_ {t = 1} ^ {T} r _ {t, (i, j)} = \max  _ {i, j = 1, \dots , d} \sum_ {t = 1} ^ {T} w _ {t, i} \left(\ell_ {t, i} - \ell_ {t, j}\right). \tag {23}
$$
For simplicity, we use  $R$  as internal regret in this subsection. To conduct a forecasting strategy that can guarantee a small internal regret. We define the exponential potential function  $\Phi :\mathbb{R}^{M}\to \mathbb{R}$  with  $\eta >0$  by
$$
\Phi (\mathbf {u}) = \frac {1}{\eta} \ln \left(\sum_ {i = 1} ^ {M} e ^ {\eta u _ {i}}\right), \tag {24}
$$
$$
\max  _ {i, j} R _ {t, (i, j)} \leq 2 \sqrt {t \ln d (d - 1)} \tag {25}
$$
Now our target is to find a new policy that makes the forecaster satisfy the Blackwell condition.
$$
\begin{array}{l} \nabla \Phi (\mathbf {R} _ {t - 1}) \cdot \mathbf {r} _ {t} = \sum_ {i, j = 1} ^ {d} \nabla_ {(i, j)} \Phi (\mathbf {R} _ {t - 1}) w _ {t, i} (\ell_ {t, i} - \ell_ {t, j}) \\ = \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \nabla_ {(i, j)} \Phi (\mathbf {R} _ {t - 1}) w _ {t, i} \ell_ {t, i} - \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \nabla_ {(i, j)} \Phi (\mathbf {R} _ {t - 1}) w _ {t, i} \ell_ {t, j} \tag {26} \\ = \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \nabla_ {(i, j)} \Phi (\mathbf {R} _ {t - 1}) w _ {t, i} \ell_ {t, i} - \sum_ {j = 1} ^ {d} \sum_ {i = 1} ^ {d} \nabla_ {(j, i)} \Phi (\mathbf {R} _ {t - 1}) w _ {t, j} \ell_ {t, i} \\ = \sum_ {i = 1} ^ {d} \ell_ {t, i} \left(\sum_ {j = 1} ^ {d} \nabla_ {(i, j)} \Phi (\mathbf {R} _ {t - 1}) w _ {t, i} - \sum_ {k = 1} ^ {d} \nabla_ {(k, i)} \Phi (\mathbf {R} _ {t - 1}) w _ {t, k}\right) \\ \end{array}
$$
To ensure that this value is negative or zero, it is sufficient to demand that.
$$
\sum_ {i = 1} ^ {d} \ell_ {t, i} \left(\sum_ {j = 1} ^ {d} \nabla_ {(i, j)} \Phi (\mathbf {R} _ {t - 1}) w _ {t, i} - \sum_ {k = 1} ^ {d} \nabla_ {(k, i)} \Phi (\mathbf {R} _ {t - 1}) w _ {t, k}\right) = 0, \forall i = 1, \dots , d \tag {27}
$$
(b) ETTm1.
(d) ECL.
Figure 5: Empirical verification of the proposed OCP block can significantly reduce the internal regret compared to vanilla EGD, where the forecasting window  $H = 48$ .
That is, we need to find a new policy vector  $\mathbf{w}_t$  that satisfies  $\mathbf{w}_t^T A = 0$ , where  $A = \left\{ \begin{array}{c} -\nabla_{k,i}\Phi (\mathbf{R}_{t - 1}) \\ \sum_{j\neq i}\nabla_{k,j}\Phi (\mathbf{R}_{t - 1}) \end{array} \right.$  if  $i\neq k$ , is an  $d\times d$  matrix. However, determining the existence of a new policy vector and efficiently calculating its values can be challenging. Even if we assume the new vector exists, the time complexity of calculating the new policy by the Gaussian elimination method[Foster and Vohra, 1999] is  $O(d^{3})$ , which is expensive, particularly for datasets with a large number of variables such as the ECL dataset with 321 variables and 321*2 policies. To address this issue, we propose the OCP block which utilizes an additional offline reinforcement learning block  $f_{rl}$  with parameter  $\theta_{rl}$  to learn a bias vector  $\mathbf{b}_t$  for the original policy  $\mathbf{w}_t$ . The new policy vector is then defined as  $\tilde{\mathbf{w}}_t = \mathbf{w}_t + \mathbf{b}_t$ . The learned bias pushes the predicted outcomes closer to the ground truth values, that is, we minimize  $\min_{\theta_{rl}}\mathcal{L}(\tilde{\mathbf{w}}):= \| \sum_{i = 1}^{d}\tilde{w}_{i}f_{i}(\mathbf{x}) - \mathbf{y}\|^{2}; s.t.$ $\tilde{\mathbf{w}}\in \triangle$  to train  $\theta_{rl}$ . We measure the internal regret  $\max_{i,j = 1,\dots,d}w_{t,i}(\ell_{t,i} - \ell_{t,j})$  at each time step empirically. As shown in Figure 5, the proposed method significantly reduces internal regret without the need for constructing and computing a large matrix.
We investigate a diverse set of datasets for time series forecasting. ETT [Zhou et al., 2021]<sup>3</sup> logs the target variable of the "oil temperature" and six features of the power load over a two-year period. We also analyze the hourly recorded observations of ETTh2 and the 15-minute intervals of ETTm1 benchmarks. Additionally, we study ECL<sup>4</sup> (Electricity Consuming Load), which gathers electricity
<sup>4</sup>https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
consumption data from 321 clients between 2012 and 2014. The Weather (WTH) dataset contains hourly records of 11 climate features from almost 1,600 locations across the United States.