SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Two-stream forecasters. The input multivariate time series data is fed into two separate forecasters, a cross-time forecaster  $f_{1}$  and a cross-variable forecaster  $f_{2}$ . Each forecaster contains an encoder and a prediction head. Assuming that the hidden dimension of the models are all  $d_{m}$ , the encoder of  $f_{1}$  projects the input series to representation  $z_{1} \in \mathbb{R}^{M \times d_{m}}$ , and the prediction head generates the final forecasting results:  $\tilde{\mathbf{y}}_{1} \in \mathbb{R}^{M \times H}$ . For the cross-variable forecaster  $f_{2}$ , the encoder projects  $\mathbf{x}$  to  $\mathbf{z}_{2} \in \mathbb{R}^{L \times d_{m}}$ . Then, the representation of the last time step  $\mathbf{z}_{2,L} \in \mathbb{R}^{d_{m}}$  is selected and fed into the prediction head to generate the final forecasting results  $\tilde{\mathbf{y}}_{2} \in \mathbb{R}^{M \times H}$ . Compared to  $f_{1}$ , whose projection head has a parameter of  $d_{m} \times H$ , the projection head of  $f_{2}$  has a parameter of  $d_{m} \times M \times H$  which is heavier, especially when  $M$  is large. Additionally, while  $f_{1}$  ignores variable dependency,  $f_{2}$  simply selects the representation of the last time step time series, ignoring temporal dependency. These two modules yield different but complementary inductive biases for forecasting tasks. OCP block is then used for learning the best combination weights. Specifically, we use EGD to update a weight  $w_{i}$  for each forecaster and use offline-reinforcement learning to learn an additional short-term weight  $b_{i}$ , the final combination weight for one forecaster will be  $w_{i} \gets w_{i} + b_{i}$ . Considering the difference between variables, we further construct different weights for each variable, namely, we will have  $\mathbf{w} \in \mathbb{R}^{M \times 2}$  combination weights.
Decoupled training strategy. A straightforward training strategy for OneNet is to minimize  $\mathcal{L}(w_1 * \tilde{\mathbf{y}}_1 + w_2 * \tilde{\mathbf{y}}_2, \mathbf{y})$  for both the OCP block and the two forecasters, where  $w_i$  here denotes the weight with the additional bias term. However, the coupled training strategy has a fatal flaw: considering an extreme case where  $f_1$  always performs much better than  $f_2$ , then  $w_1$  will be close to 1 and  $w_2 \to 0$ . In this case,  $\nabla_{\tilde{\mathbf{y}}_2} \mathcal{L}(w_1 * \tilde{\mathbf{y}}_1 + w_2 * \tilde{\mathbf{y}}_2, \mathbf{y}) \approx 0$ , that is,  $f_2$  is probably not trained for a long time. Under the context of concept drift, if retraining is not applied, as time goes on, the performance of  $f_2$  will become much inferior. In this paper, therefore, we decouple the training process of the OCP block and the two forecasters. Specifically, the two forecasters is trained by  $\mathcal{L}(\tilde{\mathbf{y}}_1, \mathbf{y}) + \mathcal{L}(\tilde{\mathbf{y}}_2, \mathbf{y})$  and the OCP block is trained by  $\mathcal{L}(w_1 * \tilde{\mathbf{y}}_1 + w_2 * \tilde{\mathbf{y}}_2, \mathbf{y})$ .
Remark Note that OneNet is complementary to advanced architectures for time series forecasting and online adaption methods under concept drift. A stronger backbone or better adaptation strategies/structure can both enhance performance.
In this section, we will show that (1) the proposed OneNet attains superior forecasting performances with only a simple retraining strategy (reduce more than  $50\%$  MSE compared to the previous SOTA model); (2) OneNet achieves faster and better convergence than other methods; (3) we conduct thorough ablation studies and analysis to reveal the importance of each of design choices of current advanced forecasting models. Finally, we introduce a variant of OneNet, called OneNet-, which has significantly fewer parameters but still outperforms the previous SOTA model by a large margin. Due to space limitations, some experimental settings and results are provided in the appendix.
Table 2: MSE of various adaptation methods. H: forecast horizon. OneNet-TCN is the mixture of TCN and Time-TCN, and OneNet is the mixture of FSNet and Time-FSNet.
Method / H	ETTH2	ETTm1	WTH	ECL	Avg
1	24	48	1	24	48	1	24	48	1	24	48
Informer	7.571	4.629	5.692	0.456	0.478	0.388	0.426	0.380	0.367	-	-	-	2.265
OnlineTCN	0.502	0.830	1.183	0.214	0.258	0.283	0.206	0.308	0.302	3.309	11.339	11.534	2.522
TFCL	0.557	0.846	1.208	0.087	0.211	0.236	0.177	0.301	0.323	2.732	12.094	12.110	2.574
ER	0.508	0.808	1.136	0.086	0.202	0.220	0.180	0.293	0.297	2.579	9.327	9.685	2.110
MIR	0.486	0.812	1.103	0.085	0.192	0.210	0.179	0.291	0.297	2.575	9.265	9.411	2.076
DER++	0.508	0.828	1.157	0.083	0.196	0.208	0.174	0.287	0.294	2.657	8.996	9.009	2.033
FSNet	0.466	0.687	0.846	0.085	0.115	0.127	0.162	0.188	0.223	3.143	6.051	7.034	1.594
Time-TCN	0.491	0.779	1.307	0.093	0.281	0.308	0.158	0.311	0.308	4.060	5.260	5.230	1.549
PatchTST	0.362	1.622	2.716	0.083	0.427	0.553	0.162	0.372	0.465	2.022	4.325	5.030	1.512
OneNet-TCN	0.411	0.772	0.806	0.082	0.212	0.223	0.171	0.293	0.310	2.470	4.713	4.567	1.253
OneNet	0.380	0.532	0.609	0.082	0.098	0.108	0.156	0.175	0.200	2.351	2.074	2.201	0.747
4.1 Experimental setting
Baselines of adaptation methods We evaluate several baselines for our experiments, including methods for continual learning, time series forecasting, and online learning. Our first baseline is OnlineTCN [Zinkevich, 2003], which continuously trains the model without any specific strategy. The second baseline is Experience Replay (ER) [Chaudhry et al., 2019], where previous data is stored in a buffer and interleaved with newer samples during learning. Additionally, we consider three advanced variants of ER: TFCL [Aljundi et al., 2019b], which uses a task-boundary detection mechanism and a knowledge consolidation strategy; MIR [Aljundi et al., 2019a], which selects samples that cause the most forgetting; and DER++ [Buzzega et al., 2020], which incorporates a knowledge distillation strategy. It is worth noting that ER and its variants are strong baselines in the online setting, as we leverage mini-batches during training to reduce noise from single samples and achieve faster and better convergence. Finally, we compare our method to FSNet [Pham et al., 2023], which is the previous state-of-the-art online adaptation method. Considering different model structures, we compare the performance under concept drift of various structures, including TCN [Bai et al., 2018], Informer [Zhou et al., 2021], FEDformer [Zhou et al., 2022b], PatchTST [Nie et al., 2023], Dlinear [Zeng et al., 2023], Nlinear [Zeng et al., 2023], TS-Mixer [Chen et al., 2023].
Strong ensembling baselines. To verify the effectiveness of the proposed OCP block, we compare it with several ensembling baselines. Given the online inputs  $\mathbf{x}$ , predictions of each expert  $\tilde{\mathbf{y}}_1$ ,  $\tilde{\mathbf{y}}_2$ , and the ground truth outcome  $\mathbf{y}$ , the final outcome  $\tilde{\mathbf{y}}$  of different baselines will be as follows: (1) Simple averaging: we simply average the predictions of both experts to get the final prediction, i.e.,  $\tilde{\mathbf{y}} = \frac{1}{2} (\tilde{\mathbf{y}}_1 + \tilde{\mathbf{y}}_2)$ . (2) Gating mechanismLiu et al. [2021]: we learn weights to the output of each forecaster, that is,  $h = \mathbf{W} \text{Concat}(\tilde{\mathbf{y}}_1, \tilde{\mathbf{y}}_2) + \mathbf{b}$ ;  $w_1, w_2 = \text{softmax}(h)$ , and the final result is given by  $\tilde{\mathbf{y}} = w_1 * \tilde{\mathbf{y}}_1 + w_2 * \tilde{\mathbf{y}}_2$ . (3) Mixture-of-expertsJacobs et al. [1991], Shazeer et al. [2017]: we use the mixture of experts approach, where we first learn the weights  $w_1$  and  $w_2$  by applying a softmax function on a linear combination of the input, i.e.,  $h = \mathbf{W} \mathbf{x} + \mathbf{b}$ ;  $w_1, w_2 = \text{softmax}(h)$ , and then we obtain the final prediction by combining the predictions of both experts as  $\tilde{\mathbf{y}} = w_1 * \tilde{\mathbf{y}}_1 + w_2 * \tilde{\mathbf{y}}_2$ . (4) Linear Regression (LR): we use a simple linear regression model to obtain the optimal weights, i.e.,  $[w_1, w_2] = (X^T X)^{-1} X^T y$ , where  $X = [\tilde{\mathbf{y}}_1, \tilde{\mathbf{y}}_2]$  and  $y$  is the ground truth outcome. (5) Exponentiated Gradient Descent (EGD): we use EGD to update the weights  $w_1$  and  $w_2$  separately without the additional bias. (6) Reinforcement learning to learn the weight directly (RL-W): we use the bias term in the OCP block to update the weights based on the predictions of both experts and the ground truth outcome, i.e., the weight is only dependent on  $\tilde{\mathbf{y}}_1$ ,  $\tilde{\mathbf{y}}_2$ , and  $\mathbf{y}$ , but not on the historical performance of each expert. For all baselines with trainable parameters, the training procedure is just the same as the proposed OCP block.
4.2 Online forecasting results
Cumulative performance Table.2 and Table.3 present the cumulative performance of different baselines in terms of mean-squared errors (MSE) and mean-absolute errors (MAE). In particular, Time-TCN and PatchTST exhibit strong performance and outperform the previous state-of-the-art model, FSNet [Pham et al., 2023]. The proposed OneNet-TCN (online ensembling of TCN and Time
Table 3: MAE of various adaptation methods. H: forecast horizon. OneNet-TCN is the mixture of TCN and Time-TCN and OneNet is the mixture of FSNet and Time-FSNet.
Method / H	ETTH2	ETTm1	WTH	ECL
1	24	48	1	24	48	1	24	48	1	24	48
Informer	0.850	0.668	0.752	0.512	0.525	0.460	0.458	0.417	0.419	-	-	-
OnlineTCN	0.436	0.547	0.589	0.085	0.381	0.403	0.276	0.367	0.362	0.635	1.196	1.235
TFCL	0.472	0.548	0.592	0.198	0.341	0.363	0.240	0.363	0.382	0.524	1.256	1.303
ER	0.376	0.543	0.571	0.197	0.333	0.351	0.244	0.356	0.363	0.506	1.057	1.074
MIR	0.410	0.541	0.565	0.197	0.325	0.342	0.244	0.355	0.361	0.504	1.066	1.079
DER++	0.375	0.540	0.577	0.192	0.326	0.340	0.235	0.351	0.359	0.421	1.035	1.048
FSNet	0.368	0.467	0.515	0.191	0.249	0.263	0.216	0.276	0.301	0.472	0.997	1.061
Time-TCN	0.425	0.544	0.636	0.211	0.395	0.421	0.204	0.378	0.378	0.332	0.420	0.438
PatchTST	0.341	0.577	0.672	0.186	0.471	0.549	0.200	0.393	0.459	0.224	0.341	0.375
OneNet-TCN	0.374	0.511	0.543	0.191	0.319	0.371	0.221	0.345	0.356	0.411	0.513	0.534
OneNet	0.348	0.407	0.436	0.187	0.225	0.238	0.201	0.255	0.279	0.254	0.333	0.348
Table 4: Ablation studies of ensembling methods (MSE results).
Method	ETTH2	ETTm1	WTH	ECL	Avg
1	24	48	1	24	48	1	24	48	1	24	48
Baseline	FSNet	0.466	0.687	0.846	0.085	0.115	0.127	0.162	0.188	0.223	3.143	6.051	7.034	1.594
Average	0.381	0.607	0.595	0.088	0.105	0.111	0.154	0.176	0.197	2.458	2.833	3.309	0.918
Gating	0.476	0.678	0.782	0.089	0.121	0.135	0.161	0.207	0.232	2.474	2.181	2.301	0.820
Ensembling methods	MOE	0.488	0.565	1.238	0.084	0.107	0.133	0.155	0.179	0.213	3.312	3.086	2.497	1.005
LR	0.741	0.634	0.589	0.153	0.107	0.113	0.229	0.179	0.198	4.376	2.235	2.478	1.003
EGD	0.383	0.614	0.682	0.081	0.113	0.117	0.153	0.183	0.213	2.546	2.102	2.373	0.797
RL-W	0.374	0.634	0.735	0.091	0.099	0.109	0.157	0.174	0.202	2.457	2.115	2.192	0.778
Ours	OneNet-OneNet	0.381	0.732	0.932	0.084	0.154	0.153	0.162	0.222	0.208	2.351	2.322	3.833	0.961
0.380	0.532	0.609	0.082	0.098	0.108	0.156	0.175	0.200	2.351	2.074	2.201	0.747
TCN) surpasses most of the competing baselines across various forecasting horizons. Interestingly, if the combined branches are stronger, for example, OneNet combined FSNet and Time-FSNet, achieving much better performance than OneNet-TCN. Namely, OneNet can integrate any advanced online forecasting methods or representation learning structures to enhance the robustness of the model. The average MSE and MAE of OneNet are significantly better than using either branch (FSNet or Time-TCN) alone, which underscores the significance of incorporating online ensembling.
Comparison with strong ensembling baselines is shown in Table 4. The two-branch framework greatly improves performance compared to FSNet with just simple ensembling methods such as averaging. The MOE approach that learns the weight from the input  $\mathbf{x}$  performs poorly and is even inferior to simply averaging the prediction results. On the other hand, learning the weight from the prediction results in  $\tilde{\mathbf{y}}_1$  and  $\tilde{\mathbf{y}}_2$  (Gating) performing much better than MOE. This indicates that the combination weights should be dependent on the model prediction. However, formulating the learning problem as linear regression and using the closed-form solution is not a good idea due to the scarce nature of the online data stream and the high noise in the learned weight. EGD provides significant benefits compared to the averaging method, which highlights the importance of the cumulative historical performance of each expert. Additionally, we observe that RL-W achieves performance comparable or even better than EGD on some datasets. Therefore, we propose the OCP block that uses EGD to update the long-term weight and offline RL to learn the short-term weight. This design leads to superior performance compared to all the other baselines.
Forecasting results are visualized in Figure 4. Compared to baselines that struggle to adapt to new concepts and produce poor forecasting results, OneNet can successfully capture the patterns of time series. More visualization results and convergence analysis are presented in Appendix C.7.
4.3 Ablation studies and analysis
The effect of instances normalization and seasonal-trend decomposition is shown in Table. 5. The results show that removing the seasonal-trend decomposition component from PatchTST has limited
(b)
Figure 4: Visualizing the model's prediction and parameters during online learning. (a) Number of parameters for different models on the ECL dataset with different forecast horizons. We concentrate on a short 50-time step horizon, starting from  $t = 2500$ . (b), and (c) depict the model's prediction results for the first and second channels of the ECL dataset.
(c) Channel 2.
Table 5: Ablation studies of the instances normalization (inv) and seasonal-trend decomposition (Decomp) of non-adapted PatchTST and online adapted PatchTST, where the metric is MSE.
Dataset	ETTH2	ETTm1	WTH	ECL	Avg
Online	Inv	Decomp	1	24	48	1	24	48	1	24	48	1	24	48
✓	✓	✗	0.360	1.625	2.670	0.083	0.436	0.555	0.161	0.370	0.464	1.988	4.345	5.060	1.510
✗	✓	0.380	1.492	3.060	0.084	0.427	0.463	0.164	0.358	0.421	2.510	5.320	6.280	1.747
✓	✓	0.362	1.622	2.716	0.083	0.427	0.553	0.162	0.372	0.465	2.022	4.325	5.030	1.512
✗	✗	0.392	1.450	2.630	0.084	0.416	0.487	0.163	0.357	0.431	2.617	5.557	5.655	1.687
✗	✓	✗	0.397	2.090	3.156	0.084	0.448	0.553	0.161	0.372	0.467	2.000	4.398	5.100	1.602
✗	✓	0.674	3.100	4.510	0.086	0.462	0.686	0.165	0.362	0.443	3.900	11.340	21.540	3.939
✓	✓	0.427	2.090	3.290	0.083	0.433	0.570	0.163	0.375	0.467	2.030	4.395	5.101	1.619
✗	✗	0.723	3.030	6.300	0.085	0.451	0.559	0.164	0.361	0.439	3.540	14.170	18.680	4.042
effect, regardless of whether the model is adapted online or not. Instances normalization is commonly used to mitigate the distribution shift between training and testing data, which is crucial for model robustness when online adaptation is impossible. However, when online adaptation is performed, the influence of instance normalization is reduced. Interestingly, our experiments reveal that instance normalization impedes the model adaptation process in ETTH2, ETTm1, and WTH datasets when the forecast horizon is long (24 or 48). Thus, simply normalizing time series with zero mean and unit standard deviation may not be the optimal approach under concept drift. Ablation studies of the variable independence and frequency domain augmentation are detailed in the appendix.
Delve deep into parameter-efficient online adaptation. Although OneNet significantly reduces the forecasting error, it also increases the number of parameters and inference time due to its two-stream framework. We also design a variant of OneNet that may have slightly lower performance than OneNet, but with fewer parameters, making it more suitable for lightweight applications, denoted by OneNet-. Specifically, we ensemble PatchTST and Time-FSNet, which are both variable-independent. In this case, denote  $\mathbf{z}_1, \mathbf{z}_2$  as the generated features for one variable from two branches, we concatenate the two features and feed them into the projection head, which further avoids the offline reinforcement learning block for ensembling weight learning and reduces the parameters. For example, in the ECL dataset, the hidden dimension FSNet [Pham et al., 2023] is 320, and the sequences have 321 channels. When the forecast horizon is 48, the projection head consists of just one linear layer with  $320 \times 321 \times 48 = 4,930,560$  parameters. On the contrary, the concatenated features of OneNet- are always less than 1024 dimension, resulting in a final projection head with less than  $1024 \times 48 = 49,152$  parameters. Figure 4(a) shows a detailed comparison of different methods on the ECL dataset. For small forecast horizons, all methods have a comparable number of parameters. As the forecast horizon increases, the number of parameters of existing adaptation methods increases rapidly. On the contrary, the number of parameters of OneNet- remains insensitive to the forecast horizon and is always less than all baselines. The performance of OneNet- is shown in Table 12, which is much better than FSNet but achieves fewer parameters.
See the appendix for the comparison of different forecasting models and more numerical results such as detailed ablation studies of different hyper-parameters and adaptation results under more settings.
Through our investigation into the behavior of advanced forecasting models with concept drift, we discover that cross-time models exhibit greater robustness when the number of variables is large, but are inferior to models that can model variable dependency when the number of variables is small. In addition, this problem becomes more challenging due to the occurrence of concept drift, as the data preferences for both model biases are dynamically changing throughout the entire online forecasting process, making it difficult for a single model to overcome. To this end, we propose the OneNet model, which takes advantage of the strengths of both models through OCP. In addition, we propose to learn an additional short-term weight through offline reinforcement learning to mitigate the slow switch phenomenon commonly observed in traditional policy learning algorithms. Our extensive experiments demonstrate that OneNet is able to effectively deal with various types of concept drifts and outperforms previous methods in terms of forecasting performance.
We also discover that instance normalization enhances model robustness under concept drift, but can impede the model's ability to quickly adapt to new distributions in certain scenarios. This prompts further exploration of whether there exists a normalization technique that can mitigate distribution shifts while enabling rapid adaptation to changing concepts. In addition, although we design a lightened version of OneNet to address the problem of introducing additional parameters and inference time, there is potential for more efficient adaptation methods, such as utilizing prompts and efficient tuning methods from the NLP/CV community, to avoid retraining the full model.
This work was supported by the National Key R&D Program of China (2022ZD0117901) and National Natural Science Foundation of China (Grant No. 62373355 and 62236010), and also supported by Alibaba Group through Alibaba Research Intern Program.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems 32, pages 11849-11860. 2019a.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11254-11263, 2019b.
Oren Anava, Elad Hazan, Shie Mannor, and Ohad Shamir. Online learning for time series prediction. In Conference on learning theory, pages 172-184. PMLR, 2013.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.
Avrim Blum and Yishay Mansour. From external to internal regret. Journal of Machine Learning Research, 8(6), 2007.
George EP Box and David A Pierce. Distribution of residual autocorrelations in autoregressive-integrated moving average time series models. Journal of the American statistical Association, 65 (332):1509-1526, 1970.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920-15930, 2020.
Xiaoyu You, Mi Zhang, Daizong Ding, Fuli Feng, and Yuanmin Huang. Learning to learn the future: Modeling concept drifts in time series prediction. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2434-2443, 2021.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? 2023.
Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. Towards principled disentanglement for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8024-8034, 2022a.
Yang Zhang, Fuli Feng, Chenxu Wang, Xiangnan He, Meng Wang, Yan Li, and Yongdong Zhang. How to retrain recommender system? a sequential meta-learning method. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1479-1488, 2020.
Yi-Fan Zhang, Jindong Wang, Jian Liang, Zhang Zhang, Baosheng Yu, Liang Wang, Dacheng Tao, and Xing Xie. Domain-specific risk minimization for out-of-distribution generalization, 2022b.
Yi-Fan Zhang, Zhang Zhang, Da Li, Zhen Jia, Liang Wang, and Tieniu Tan. Learning domain invariant representations for generalizable person re-identification. IEEE Transactions on Image Processing, 32:509-523, 2022c.
Yi-Fan Zhang, Xue Wang, Kexin Jin, Kun Yuan, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Adanpc: Exploring non-parametric classifier for test-time adaptation. ICML, 2023a.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision. arXiv preprint arXiv:2107.09249, 2021.
YiFan Zhang, Xue Wang, Jian Liang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Free lunch for domain adversarial training: Environment label smoothing. arXiv preprint arXiv:2302.00194, 2023b.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations, 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022a.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pages 27268-27286. PMLR, 2022b.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03), pages 928-936, 2003.