ARM: Refining Multivariate Forecasting with Adaptive Temporal-Contextual Learning
Authors: Jiecheng Lu
Xu Han
Shihao Yang
Venue: ICLR 2024 poster (2024)
Keywords: Time Series Forecasting
Transformer
Multivariate Time Series
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Long-term time series forecasting (LTSF) is important for various domains but is confronted by challenges in handling the complex temporal-contextual relationships. As multivariate input models underperforming some recent univariate counterparts, we posit that the issue lies in the inefficiency of existing multivariate LTSF Transformers to model series-wise relationships: the characteristic differences between series are often captured incorrectly. To address this, we introduce ARM: a multivariate temporal-contextual adaptive learning method, which is an enhanced architecture specifically designed for multivariate LTSF modelling. ARM employs Adaptive Univariate Effect Learning (**A**UEL), Random Dropping (**R**D) training strategy, and Multi-kernel Local Smoothing (**M**KLS), to better handle individual series temporal patterns and correctly learn inter-series dependencies. ARM demonstrates superior performance on multiple benchmarks without significantly increasing computational costs compared to vanilla Transformer, thereby advancing the state-of-the-art in LTSF. ARM is also generally applicable to other LTSF architecture beyond vanilla Transformer.

Published as a conference paper at ICLR 2024
ARM: REFINING MULTIVARIATE FORECASTING WITH ADAPTIVE TEMPORAL-CONTEXTUAL LEARNING
Jiecheng Lu', Xu Han', Shihao Yang'
GGorrgia Institute of Technology *Amazon Web Services
14 suuu444gateech.c. edu, amazon com, shihao yanooyaag@@iiy wwwggisye.gttecc.eee.. gatech.
ABSTRACT
Long-term time series forecasting (LTSF) is important for various domains but is confronted by challenges in handling the complex temporal-contextual rela- tionships. As multivariate input models underperforming some recent univariate counterparts, we posit that the issue lies in the inefficiency of existing multivariate LTSF Transformers to model series- wise relationships: the characteristic differ- ences between series are often captured incorrectly. To address this, we intro- duce ARM: a multivariate temporal-contextu adaptive learning method, which is an enhanced architecture specifically designed for multivariate LTSF modelling. ARM employs Adaptive Univariate Effect Learning (AUEL). Random Dropping (RD) training strategy, and Multi-kernel Local Smoothing (MKLS), to better han- dle individual series temporal patterns and effectively learn inter-series depen- dencies. ARM demonstrates superior performance on multiple benchmarks with- out significantly increasing computational costs compared to vanilla Transformer, thereby advancing the state-of-the-art in LTSF. ARM is also generally applicable to other LTSF architecture beyond vanilla Transformer.
INTRODUCTION
Long-term time series forecasting (LTSF) is critical task across various fields such finance, epi- demiology, electricity, and traffic, aiming to predict future values over an extended horizon, thereby facilitating optimal decision-makinn
resource allocation, and strategic planning (Martinez-Alvare/ et al. 2015 Lana al.) 2018 Kim 2003 Yang et al., 2015 Ma al. (022). However, modeling LTSF poses numerous challenges due to the complex and entangled characteristics of multivariate time series data, often leading overfitting and erroneous pattern learning, thereby compromising
model performance (Peng & Nagata 2020 Cao & Tay 2003 [Sorjamaa et al. (007))
Recently, Transformers (Vaswani et al. 2017) have significantly outperformed other structures in sequential modeling. They have showcased advancements in LTSF modeling (Zhou et al.) 2022] Wu et al. 2021 Zhou al., 2021 Li al.) 2019). While models with multivariate time series inputs are generally considered effective for capturing both temporal and contextual relationship, recent studies have shown that LTSF models with univariate input can surprisingly outperform their multivariate counterparts (Zeng et al. 2023 Nie et al. 2022). This is counter-intuitive, as univariate models are limited capturing relationships across multiple series, which is crucial for LTSF.
In this study, we argue that existing multivariate LTSF Transformers fall short in properly mod- eling series-wise relationships due suboptimal training and data processing methods, leading to subpar performance compared to univariate models. These models struggle to handle significant differences of characteristic across various input series, such as the differences in temporal depen- dencies, differences in local temporal patterns, and differences in series wise dependencies beyond intra-series relationships, as exemplified in Figure [ crucial components of the Transformer blocks, such as temporal attention and input embedding, undermines the model's capacity to differentiate them effectively during forecasting tasks.
Improper mixing of these distinct series within
To address these issues, introduce ARM: multivariate temporal-contest
adaptive learning method. ARM incorporates improved training and processing methods to effectively manage data with pronounced series-wise characteristic differences. This leads to better handling of contextual

---

Published as a conference paper at ICLR 2024
Sy rihesrred ThA Duraee Aundbrree
May o MMMM
MM
bow Man Mwm
the Obvious Inten- Dependencies by Previcus Muttvcriate LTSP Tranaformer
pe Ccampion of Adaativve Untvariane Effect Loarning Output Meen
(h) An Exampis of Ressonacie Mustuurtart Terporrs Reproserttaion Idiaa Scerarto
Figure I: Three intuitive problems arising from wrongly handling input series with characteristic differences (further explained in {A.3). (a) Adaptive estimation of output mean necessity for se- ries with diverse characteristics (see $3.T). The Adaptive EMA (green) in our method outperforms previous approaches like RevIN (yellow) and NLinear (red) by adapting to varied temporal depen- dencies. (b) The necessity of building reasonable multivariate temporal representation (see $3.3). Our multi-window local convolutional module addresses the inadequacy of former models in mod- eling different local patterns across series. (c) The Inability of Previous Models to Learn Obvious Inter-Series Dependencies: Demonstrated using the "Multi" dataset generated with simple shifting (see A.2 existing models like Autoformer fail to learn this simple "copy-paste" operations. See Figure for the visualization to show our ARM's performance boost in this dataset.
information and more accurate learning of inter-series dependencies. Our approaches can be easily integrated into other LTSF models, significantly enhancing the forecasting of multivariate time series with only a modest increase in computational complexity. The key contribution of ARM comprise:
(a) We introduce Adaptive Univariate Effect Learning (AUEL), designed to independently learn the univariate effects including optimal output distribution and temporal patterns for each series before the encoder-decoder. This facilitates balanced learning of both intra- and inter-series dependencies;
(b) We implement a Random Dropping (RD) strategy to help the model identify accurate inter-series forecasting contributions and avoid overfitting from learning incorrect series-wise relationships.
(c) We propose the Multi-kernel Local Smoothing (MKLS) module, aiding Transformer blocks in adapting to multivariate input with series-wise characteristic differences. This is achieved by structing reasonable temporal representations and enhancing locality for the Transformer blocks;
RELATED WORKS
Time series forecasting has long been a popular research topic. Traditional approaches, such the
ARIMA model (Box et al.] 1974) and the Holt- Winters seasonal method (Holt) 2004), provided the- oretical guarantees but
limited for complex time series data. Deep learning models (Oreshkin et al. 2020 Sen et al. 2019) introduced a new era in this field. RNNs (Hochreiter & Schmidhuberi
Wen al., 2017 7 Rangapuram al., 2018 Shih al., 2019 Salinas al. [2200] Qin al. 2017 allowed for the summary of past information in compact internal memory states, updated re- cursively with new inputs at each time step. CNNs and temporal convolutional networks (TCNs)
1997
(Lai et al. 2018 Borovykh et al. 2017 van den Oord et al. (016) further advanced the field, cap- turing local temporal features effectively, though with limitations on long-term dependencies.
Recently, Transformer Vaswaani et al.) 2017) succeed in sequential modeling tasks with the effec- tive attention mechanism. Transformer derivatives like LogTrans (Li et 2019 introduced local convolution, reducing complexity for LTSE Similarly, Informer (Zhou et al. (022) and Autoformer (Wu et al. 2021) extended Transformer with efficient attention and auto-correlation mechanisms respectively, while FedFormer (Zhou et al. 2022) and PyraFormer (Liu et al.] 2022) utilized im- proved attention to achieve lower complexity. However, a recent study (Zeng et al. 2023) argued that Transformer may fail to correctly understand multivariate time series structure. PS (Nie et al., 2022) address the issue with independent input but fell short in modeling multivariate corre- lations. CrossFormer (Zhang & Yan 2023) tried address with 2D attention, but its hierarchical
2

---

Published as a conference paper at ICLR 2024
Duppod Xp
LO00
Mathorrala rod
Input
AUEL (Ditributin)
Xyn O
Ap
Xi XG
Random Dropping
AUEL (Pattens)
Transforrar Encadars
AUEL (Preprocesaing)
200
Enootedd Feetures CM
Outpul Projection
Dropping
1p-d
Transformer
EncxdesDDaoder
Transfomer Decoders
Post Poat-AIKLS
Post-MKLS
Transfomer
MKLS
AUEL (Dlstnbution)
Encodlrr
Pre-MKLS
Pre-MKLS
gu... 2 AUEL (Pattern)
AUEL (inverse Procassing)
Imput Emmbedstngs
Multivariate Forecasting
Preprceesed
best Entecdingooiee Dimansion d
Figure 2: Overall Architecture of ARM with Vanilla Transformer as encoder-decoder. The left side depicts the global workflow and the right side illustrates the specific process in the model training.
segmentation failed adapt length per series. To address the challenges, we propose ARM to enhance the accuracy and adaptability of LTSF with proper training and data processing methods.
METHOD
We first define the notation for LTSF. For multivariate time series X E RL our goal is to provide the best predictor Xp of its latter part Xp E L based on its previous input part X, E TR L where T. I. + Lp denotes the overall length of the time series, and represents the total number of series. Let (1); denote the value of the t-th step in the i-th sequence of X, where E (1...... and f1.... C}. To address the shortcomings inherent in the training of existing multivariate LTSF models-namelt their inability to effectively manage multivariate inputs with characteris- tic differences-we we introduce the multivariate temporal-contextual adaptive method, ARM, which incorporates Adaptive Univariate Effect Learning ($3.T), Random Dropping strategy ($3.2), and Multi- Local Smoothing ($3.) on the basis of vanilla Transformer encoder-decoder structure to handle such complexities. The overall architecture of our method is illustrated in Figure
3.1 MODULE A: ADAPTIVE UNIVARIATE EFFECT LEARNING
Determining the most likely output distribution and temporal patterns Lp based on the input at L, for different series is crucial for the accurate training of multivariate LTSF models, especially when the distribution and patterns among the series vary significantly. We refer to this operation as "Univariate Effect Learning" and introduce Adaptive Univariate Effect Learning (AUEL), specif- ically designed determine the most appropriate methods disentangle the univariate effects of different series with varying characteristics before the multivaraite encoder-decoder structure.
In previous LTSF Transformers such as Informer and Autoformer (Wu et al. 202 Zhou et al. 2021), the Lp region of the decoder input is filled with 0 values as default outputs (models without an Lp part input also be considered to use 0 default). However, the actual meaning of 0 can vary significantly across series due to their distributional differences, making it difficult for multivariate models determine the output level for each series. As the output level predominantly influences the loss like MSE during training, the model tends to use the majority of parameters to learn the output level, making challenging to capture finer-grained temporal patterns and inter-series dependencies.
Recent studies tried to mitigate the aforementioned issues by focusing on "eliminating distribution shifts between input and output". These approaches attempt to estimate the mean and variance, eliminating them from the model input, and restoring these values the output. RevIN (Kim al.)
2022) calculates the mean and variance for each input series while NLinear (Zeng et al. (023)) sim- ply employs the last value of each series at L₁ the output mean. However, as illustrated in Figure (a), for series with differing statistical properties, the lookback lengths required to determine their output distribution tend not to be constant. Relying solely on last or all input values will weaken per-
3

---

Published as a conference paper at ICLR 2024
LIBF
LISF
Model
Mode's
191
Model
LTBF
Model
Figure 3: This figure illustrates six LTSF model types, categorized as univariate (Methods a-c) and multivariate (Methods d-f). The figure's parallel arrows show independent series processing, and converging arrows indicate series mixing. Univariate models (a-c) process series separately: (a) employs individual models like DLinear for each series, (b) uses
models such as PatchTST for all series, and (c) dynamically chooses best models per series, in $3.1 Multi- variate models (d-f) capture inter-series dependencies: (d) uses standard structures as Informer and Autoformer, outperformed by univariate methods; (e) combines univariate models with multivariate factors to better build inter-series dependencies but adding computational complexity; (f) introduces our Random Dropping strategy (see $3.2) for efficient learning of inter-series relationships.
formance. Thus, we propose an adaptive method based on a learnable exponential moving average (EMA), which dynamically adjusts the output mean and variance for different series.
Previous methods still employ constant value to fill the Lp part of each series without building temporal patterns. In multivariate LTSF, handling the univariate temporal patterns prior to the multi- variate modeling can help to learn inter- relationships more accurately. In multivariate datasets, suppose given series has no relationships with other series. If we utilize univariate model to initialize X part based on X before other operations, this preliminary step will allow the model to focus more on capturing other inter-series relationships. Previous works like LSTNet (Lai al., 2018) and ES-RNN (Smyl) 2020) have used classic autoregressive and exponential smoothing mod- els for this purpose. However, both methods are limited in capturing basic patterns and suffer from poor generalization. Therefore,
an adaptive method based on the Mixture of Experts (MoE) for temporal patterns to better handle these complexities, as illustrated in Figure (c).
AUEL of Mean and Standard Deviation AUEL uses an adaptive EMA method to learn the output mean. can adjust the lookback length for different series using trainable EMA parameter a² for each series i. To calculate the output standard deviation for each series, A multi-window weighting method is used. We assign & windows of different lengths Wy and separately compute the standard deviation of the data covered by each window. We assign trainable window weights P} for each series to calculate the weighted average of standard deviations across windows, resulting in S The EMA mean E for the i-th input series X3 and its S" are computed:
E p sta(x -fr(x
(1-al a=a") eexp(( 111 - t)
xj
S
(l)
1 exp([Lrl]: tc)
2==12)
where denotes element-wise multiplication, t 11,2, Lr] vector representing the timesteps, and Hi indicates the j-th element of the vector enclosed in brackets; x x x(kr-w,bbz) is vector composed of the part of input series X that goes back wy steps from the last value. When the a² of channel approaches 1, Pylobal = 1, and Plocal = 0, the AUEL of output distribution becomes RevIN. When of approaches O, the AUEL will be similar to NLinear which uses last values.
AUEL of Temporal Patterns Figure parts (a) and (b) illustrate two types of univariate input LTSF models: (b) with fully-shared parameters fits datasets with similar series characteristics while (a) with independent channels is suited data with diverse series. Typically, a dataset may contain multiple clusters of similar series. Thus, (c), an intermediary approach between (a) and (b) is prefer- able. We use MoE, simillar to the module in Switch Transformer (Fedus et al. (2022). to dynamically select the predictor based series characteristics to initialize the temporal patterns for each series (see A. for more details on the implementation of MoE for temporal pattern learning).
Preprocessing and Inverse Processing After extracting E and S based on Xj. we perform the AUEL of output distribution upon the input series X [x] 0f], where 0g is the 0-filled Xp. and [A B] denotes the concatenation of multivariate time series A and B along the time dimension,
4

---

Published as a conference paper at ICLR 2024
yielding the preprocessed X Simillar instance normalization (Ulaanv al. 2016). we further apply a trainable channel affine transformation each channel. Further, we utilize MoE obtain the final input X In this context, the MoE takes an input of length L, and produces an output of length Lp. We use y², 33 for trainable affine parameters, and for small value avoid overflow.
8" 8"=0' (X'-
+ (-=+(()++)++
X
[XT
MoE (xxr XF
(2)
We denote the forecasting for series i from the encoder-decoder X pp- In the inverse process- ing stage, we again employ MoE to assist in balancing intra-series and inter-series dependencies, yielding the MoE output Xi- Subsequently, we reinstate Ft and 5t into the final output Xp.
X MoE ((x** {ko)), 8%
( -3) (8' +c)+
3.2 MODULE R: RANDOM DROPPING AND INTER-SERIES DEPENDENCY LEARNING
In LTSF, accurately modeling inter-series dependencies is often challenging due to the simultane- ous handling of complex temporal and contextual information. [ provides an illustrative example where multivariate Transformer fails to capture some very evident inter-series dependen- cies.
Multivariate LTSF models often struggle to achieve disentanglement between series. Transformers with both multivariate inputs and outputs, shown in Figure (d), are prone to fitting wrong series- wise relationships. In recent research, such architectures were outperformed on datasets with com- monly weaker inter-series dependencies by the two univariate model structures depicted in Figure (a) and (b). To mitigate overfitting, some models combine the advantages of univariate structures, employing the architecture shown in Figure 3 (c). These models make predictions for each series individually, treating other series as context to aid the forecasting. However, when the number of series is large, the approach in (c) significantly increases computational complexity compared to multivariate LTSF models in (d), especially for the complex Transformer. It is also hard for existing structures to discern which series in the context contribute most the forecasting of current series.
We introduce Random Dropping to train multivariate LTSF models effectively. This strategy fa- cilitates the learning of inter-series dependencies by efficiently decoupling relationships between different series, without increasing computational complexity. Applied during the training phase, Random Dropping simultaneously sets a random subset of series to zero in both the input and train- ing target, as illustrated in Figure 3 (f). The model thus learns the contributions of the current subset to the forecasting of series within that subset. Through this random selection, the model incrementally discerns which series significantly influence the forecasting of others, thereby effee- tively mitigating the risk of overfitting. Conceptually, Random Dropping can be viewed as a form of model ensemble that constructs pool of forecasting models for every possible subset-to-subset relationship, then enabling the more effective models among them. Utilizing Random Dropping can straightforwardly enhance the learning ability of multivariate LTSF models in terms of inter-series dependency, achieving disentangiement between series.
3.3 MODULE M: MULTI-KERNEL LOCAL SMOOTHING
In NLP tasks with Transformers, each input vector represents an individual word on each time step. Attention is used to compute the temporal relationships among these words. However, in the con- text of multivariate LTSF, the temporal dependencies for forecasting different series are typically not identical. Directly calculating attention for each input (t) along the temporal dimension may lead to incorrect, blended result. This necessitates method to construct reasonable temporal represen- tations for multivariate input. Moreover, attention assumes equal relevance between different time steps, overlooking the significance of local patterns different series. These issues exemplified in Figure [ (b), which discusses the reasonable representation of multiple series. Here, building multivariate temporal representation may require different local views adjusted for different series.
In order to enhance the understanding of multivariate temporal structure, We propse the Multi-kernel Local Smoothing (MKLS) block which is used in conjunction with the Transformer blocks, as shown in Figure 4 MKLS uses multiple different ID convolutional kernels and a channel-wise attention to learn and extract local information with adjustable view length. To address the aforementioned
5

---

Published as a conference paper at ICLR 2024
Mutii felik-keelll Local Smoothhin
(WNLLS)
Pre-IKLS
Sencshed Oarpu &
Risidbal
PoabiKLL
Iodea
Axd
Output X
Transformer Block
Heig
Add Nornalzation
Wogle Wa
Scaine-te Phodect
of Migg
Duipex X
local Raprsseraaion
Mua Mullkome
Tranaformer
Postionnl Enitedding -Py
Addi
Dropout Mean
MAlll- window Informttor
Pou
kott- Local Smoothing
Infommntion
Proshoona og
Inputt
Cinr
Cory
Gore
Inpur.Un Effect))
Tranehrme Block
Inpul
Figure 4:
Structure of Multi-kernel Local Smoothing (MKLS). The central part of the figure illus- trates the computation of MKLS, incorporating multiple ID convolutions and channel attention. The left side and right side presents the application method of Pre-MKLS and Post-MKLS, respectively.
challenges of building reasonable temporal representations and enhancing locality, we provide two usage of MKLS. (i) Pre-MKLS is applied before feeding data into the Transformer blocks, Ahelp- ing build meaningful local representations for multivariate input. (ii) Post-MKI.S processes data in parallel with the Transformer blocks, playing role like local attention with adjustable local win- dows. Through the incorporation of channel-wise attention, MKLS learns different kernel weights, providing robust adaptability for series with strong characteristic differences.
MKLS block Let the input for the MKLS block be X E Rixdd where I is the length of the temporal dimension and d is the number of channels. Suppose we set n ID convolutional layers with kernel sizes 8j (.1,... 88)) First, obtain the computation results X, from the j-th convolutional layer by: X, = Convid, (X). To acquire the weights mý. for kernel j on channel i, we first perform kernel dropout and average-pooling on X,, and then calculate the cross atten- tion between the averaged output and the original X. For kernel dropout, we randomly drop certain proportion T3 of kernel outputs, which enable independent learning of local patterns thus prevent overfitting. Let the tensor X E gixdxn contain the results of each X₃ E Rddd, We now
Ea X for the attention calcu- lation besides original X. Then, we use NAvg and X as Q and K in the attention, and obtain the kernel weights M E Rdxx for each channel using a trainable latent The specific calculation is: M = softmax [Wo(XT+PP) +Px)(Wx<X +P)) T Wr, where Wq E Raxd and WK E Bdxd are
perform average-pooling to obtain another input Xang = I-rr
va
the weights for the Q and K in the attention, initialized as identity matrices to retain the original channel structure before training; Wy E Rdxa designed to be a latent matrix, which projects the cross-channel attention map to the dimension of n, obtaining the kernel weights M; PE is the trainable positional embedding matrix. Now,
caleulate the weighted kernel output $ + Xi,s, where represents the element-wise multiplication per- as: S., = Σラー (Kis., * M..). t,i.j
formed on the channel and multi-kernel dimensions; the output Št, is the local smoothed output after weighting local kernels. A residual shortcut of the input is added and denoted by X.,i-
Pre-MKLSS and Post-MKLSS MKLS module, used alongside Transformer blocks, effectively manages local shape extraction and alignment in multivariate input. Pre-MKLS constructs local smoothed representations, offering more self-adjustment based on input compared to the pure ID convolutional approach like (Li et al. 2019). Post-MKLS, processing data in parallel with the Trans- former, enables the latter to focus more on learning long- term connections, with its logic similar to the local attention methods such as (Child et al. 2019 Roy et al., (021) and parallel local convolu- tional layer in (Wu et al.) 2020), but with adjustable local view. The using methods of Pre-MKLS and Post-MKI.S are shown the left side and right side of Figure respectively.

---

Published as a conference paper at ICLR 2024
Table 1: Multivariate results with prediction horizon Lp E {96, 192, 336, 720} (and Lp {24, 36, 48, 60) for ILI). The best results are highlighted bold and the second best are underlined. Results for baseline models are sourced from previous literature or derived from additional exper- iments conducted for unreported datasets. Baselines were run with L1 values of 96, 192, 336, 720, and best results among them are reported. In contrast, We use a consistent L, of 720 for ARM.
Modell
ARM (Vamilla)
PacchTST
DLineur
FEDiormer
Autofformer
Informer
Repeat
Metric
MSF
MAF
MSF
MAR
MSF
MAF
MSF
MAF
MSF
MAR
MSF
MAF
MSF
MAF
0.308
Elecmeiity (192)
0.143
0.239
0.240
0.153
0.249
0.201
0.222
0.334
0.296
0.386
1.595
0.950
Flectricity (336)
0.154
0.251
00163
0730
0.169
0.267
0.214
0.320
0:251
0.138
0.300
0.304
1/617
0.961
Flectiiity (720)
0.179
0.275
0.197
0.290
0.203
0.301
0.346
0,355
0.254
0.361
0.375
0.459
1.647
0.975
ETTml (96)
0.287
0.340
0.293
0.346
0.343
0.326
0.340
0.510
0.492
0.626
0.540
1,214
0.665
0.335
ETEml (336)
0.364
0.384
03309
0,392
0.369
0.385
0.392
0.425
0.510
0.492
1.005
0.741
1.283
0.202
ETI 31 (720)
0.411
0.412
0110
0.420
0.425
0422
0.446
0.458
0.527
0.493
1.133
0.845
1.319
0,729
BT1m2 1963
0.163
0.254
0,106
0.256
0.167
0.260
0.203
0.287
0.255
0.339
0.365
0.453
0.266
0.328
ETT (192)
0.240
03355 10
03006
0:234
0.309
0.32%
0.281
0.340
0.533
0.563
ETEm2
0.357
0.382
0.397
0.421
0.421
0.415
0.422
0.419
3.379
1,388
0.465
ETThl (96)
0.366
0,391
0.500
0.000
0.375
0.399
0.376
0.415
0.435
0.146
0.941
0.769
1.295
0.713
ETTbL (192)
0,402
0F11
0.0
0.423
0,466
0.456
0.457
L,077
0.786
1.525
0.753
ETThl (336)
0.421
0423
0.462
0,4%6
0.487
1,038
(720)
0,459
0,490
0.469
1.339
ETTA2 (336)
ETT12 (192)
ETTh2 (96)
0.264
0.127
0.356
FLT74 4
0,293
0.327
113
0274
0329
0.33
0182
8384
0.348
0.385
0.289
04655
0418
0,353
0.332
0.400
D-407
0.374
0.417
0.446
0.332
0.466
0/77
0.17%
0.434
0.368
1,215
1.792
1.589
0.952
1,642
1,542
0.534
0.591
0.433
0.423
0.508
0.473
ETTh2 (720)
0.379
0.122
0.605
0.55)
0.112
0.169
0/453
0.490
3.656
1,619
Weather (96)
0.144
0.193
0.149
0.198
0.176
0.237
0.212
0.296
0.266
0.336
0.300
0.384
0.259
0.254
Weather<195
0.189
0.240
0211
0.220
0.252
0.276
0.336
0.307
0.367
0,598
0.544
0.309
0.292
Weather (336)
0.2332
f1200
0:265
0.319
0.139
0.380
0.359
0 0.395
0.578
0.523
0.377
0.338
Weather (720)
0.296
0,332
033
033
0.325
0.362
0.103
0.128
0.419
0.428
1,059
0.741
0.465
0.391
Traffic (96)
0.282
0.587
0.366
0.613
1079
Traficc (192)
0.373
0.255
0.399
6.286
0.425
0.287
0.604
0.375
0.616
0.382
0.696
0.379
2.756
LO87
Trafic (336)
0.383
0.274
11392
6264
0.436
0.296
0.621
0.383
0.822
0.337
0.777
0.420
2.791
LO99
Traffic (720)
0.425
11.34
0442
0.286
D.465
0.315
0.626
0.382
0.660
0.40%
0.864
0.472
2.611
1.097
Exchange (96)
0,057
0.207
0.081
0.148
0.278
0.197
0.81
0.196
0.280
0.293
1.204
Exchange (336)
1.252
0.367
0.351
0.432
0.305
0.414
0.460
0,500
0.50P
0.524
1.672
1.036
0.305
0. 196
Eachange (720)
0.486
0.535
0.967
0.697
0.643
0.601
1.195
0,841
1.447
0.941
2.478
1.310
0.823
0.681
(2.4)
1.148
0.699
1.319
0.754
2.215
1.08
1.260
3,483
.287
5.764
1,677
6.587
1.00
11.1 (36)
1.599
0.700
0.963
2.679
1,080
3.103
1.148
4.255
1,467
7.130
1788
11 (60)
1.378
0,771
1,470
0.788
2.368
1.096
2.857
1.157
2.770
1.125
5.764
1.564
5803
1677
Multi (96)
0.032
0.125
0,072
0.202
0.067
0.190
0.117
0.261
0.102
0.633
0/992
0.219
0008
2000
0.89
Molli (192)
0.063
0,173
0.167
0.294
0.304
0.300
0.356
0.499
0.007
0.358
0140
0223
(336)
0.238
0.313
9.572
0.414
Mulii 0.450
0.503
0.588
0.544
0.621
0.921
0.677
4 EXPERIMENTS AND RESULTS
We have conducted extensive experiments on 9 widely-used datasets, including the ETT et al.,
2021. Traffic, Electricity, Weather, IIII, and Exchange Rate[Lai et al. 2018) datasets (details in SA.Ip. We also generated a dataset "Multi," in which we employed simple shifting operation to construct significant inter-series relationships (see Figure [ (c). Figure 6 and $A.2). Our base- line comparison included Transformer-based models such as PatchTST et al. 2022). FED- former(Zou et al., 2022). Autoformer Wu et al., (02T), and Informer(Zhou et al., 2021 and a linear model DLinear from (Zeng al. 2023). Additionally. we included a last-value "Repeat" method for comparison. We report the baseline results by referring to their original papers. For models that did not report their performance on specific datasets, we conducted the experiments cor- respondingly. The performance was evaluated using mean squared eиror (MSE) and mean absolute error (MAE). The hyperparameter settings and implementation are detailed in $A.5 and $A.6
In Table D we evaluate the performance of ARM across 10 datasets. Note that the ARM here employs a vanilla Transformer encoder-decoder, as illustrated in Figure 2 which is referred as "Vanilla". The results demonstrate that ARM consistently outperforms previous models across 10 benchmarks. Its accurately modelling of multivariate relationships leads to enhanced performance on datasets with strong inter-series relationships and larger series scales, like Electricity, and Traffic. Also, extracting univariate effects allows better performance on datasets with short-term distribution changes like Exchange and ILL. Building on the vanilla Transformer and only slightly adding the computational costs (see $A.7). ARM offers adaptation across diverse multivariate datasets.
The three core modules of ARM can be easily integrated into existing LTSF models without chang- ing their structure. Replacing the encoder- predictor in Figure 2 with previous models to apply ARM can effectively help them handle multivariate characteristic differences (see $A.6.2 for more details on applying ARM to existing models). The second part of Table shbows the improve- ment ARM offers to both univariate (PatchTST, DLinear) and multivariate ( (Vanilla, Autoformer,
7

---

Published as a conference paper at ICLR 2024
Table 2: The results of applying A/R/ to more LTSF models. We run all the experiments with fixed L, 720. Using percentage comparison, We demonstrated the enhancements in average performance of the original models with the addition of A/R/M. Note that for "+R", since the sole use of Random Dropping does not impact univariate models, these results are not provided.
Medels
Vanilla
Vonill=ARM
VanillaAA
Vanilat8
VanillatM
VanillaRRM
Metrics
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Electriiity 196)
0.300
0.434
0.125
0.223
0.130
0.228
0.34
0.416
0.37%
0.45.
0.246
0.384
Hlcuuictty (336)
0.359
0.430
0.154
0.251
0.166
0.264
0.348
0.419
0.605
0.468
0.383
0.374
Electiintty (720)
0.388
0.435
0.179
0.275
0.187
0.294
0.344
0.407
0.365
0:428
0.246
0.397
Erimi (96)
0.917
0.210
0.257
0.340
0.298
0.357
0.744
0.611
0.834
0.649
0.713
0.579
53 Fm (316)
1.000
0.739
0.364
0.184
D. 1.369
0 1888
0.872
0.674
1,005
0.113
0.758
D6610
FTTml (720))
1.211
0.876
0441
0.412
0.427
0.423
0.873
0.608
1,087
0.700
0.815
D
Average
0.220
0.607
0228
0,311
01257
0.321
0,000
0.550
0.775
0.589
0,530
0,199
Averuge's
100.0%
100.0%
34.6%
$1.2%
35.7%
$2.9%
84.2%
90.6%
94.2%
97.0%
73.1%
$2.2%
Mudels
PaichTST
PachTST+AA
PIRE-TTT-R
PORETST-M
THchTSTTRMM
Metnis
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
Electricity (96)
0.137
0.236
0.130
0.225
0.151
0.228
0.142
0.352
0.151
0.233
Electricity (192)
0.152
0.249
0.115
0.211
0.147
0.212
0.163
0.269
0.147
0.218
Electricaty (336)
0.168
0.161
0.257
0.163
0.258
0.175
0.283
0.161
0.361
(720))
0.298
ETiml (96)
0.347
0.291
0.343
0.295
0.344
0.283
ETTml (192)
0.353
0.370
0.328
0.365
0.352
0.367
0.359
0.375
0.352
0.366
51 ml (336)
0.367
0.389
0.363
0.387
0.363
0.387
0.570
0.391
0.364
0.418
0.260
0.322
0.255
0.266
0.255
0.320
Average
100.0%
100.0%
96.9%
98.1%
98.1%
98.4%
101.5%
102.5%
98.1%
99.4%
Medels
DLDssI-ARM
DLincurtA
DLincirt
DLincar+M
DLincar-MM
Metrics
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Elactriciy (96)
0.140
0:237
0.170
0.227
0.133
0.232
0.147
0.256
0.131
Electriatt (336)
0.169
0.267
0.262
0.165
0.262
0.201
0.317
0.164
0.262
Electicatt (720)
0.204
0.301
0.157
0.291
0.203
0.261
0.228
0.358
0.192
0.293
Erimi (56)
0.307
0.251
0.299
0.341
0.300
0.348
0.305
0.348
0.301
0.346
mm (336)
0.360
0.366
0 0.365
8.567
D 183
0.376
0.704
0.364
0.3K
E11ml 7/20)
0.425
0.421
0.413
0.412
0.423
0.419
0.435
0.428
0.422
0.420
Average
0.261
0.325
0.251
0,315
0,260
0.319
0,275
0.342
0.258
0,319
Averuge/s
100.0%
100.0%
95.4%
97.5%
98.9%
98.8%
104.%
105.9%
98.1%
98.8%
Mudels
Autoformee
AutesormreeRR
Ausoformer
Autofeema--R
Auteformer-M
AutesommeeRMM
Metris
MISE
MAE
MSE
MAE
MSE
MAE
MSE
MAS
MSE
MAE
MISE
MAE
Electricity (96)
0.337
0.425
0.132
0.231
0.131
0.231
0.207
0.330
0.22]
0.332
0.200
0.315
Eletticitty (192)
0.310
0.399
0.147
6.211
0.145
0.266
0.209
0.319
0.265
0.353
0.199
0.313
Blctricttt (336)
0.329
0.417
0.156
0.256
0.158
0.258
0.228
0.337
0.257
0.362
0.214
0.327
ETTm/ (96)
0.511
0.497
0.346
0.304
0.355
0.431
0.458
0.563
0.188
0.272
0.356
81 mm (192)
0.591
0.517
0.320
0.370
0.338
0.374
0.485
0.168
0.636
0.513
0.289
0.371
ETTml (336)
0.370
0.470
0.367
0.503
0.554
0.519
0.423
0.443
0:435
0.439
0,450
112550
0,429
0.284
0.362
Average
100.0%
100.0%
57.6%
69.3%
59.8%
70.2%
82.7%
89.8%
97.7%
95.2%
64.7%
78.99
Models
Informer
Metrics
MSB
MAB
MAB
MAB
Electricit (96)
0.923
0.791
0.238
0.159
0.242
0,402
0.483
01832
0.224
0.272
0.386
Electricity (336)
0.954
0.796
0.158
0.261
0. 0.159
0.362
0,410
0.488
0.730
0.643
0.301
0.377
Electriiity (720)
0.94
0.795
0.770
0.276
0.185
0.203
0.366
0.447
0.93%
0.767
0.351
0.43
Erimi (96)
0.848
0.665
0.291
0.342
3.302
0.354
0.488
0.460
0.809
0.023
0.397
0.454
ETTml (336)
0.977
0.735
0.370
0.301
0.576
0.307
0.581
0.540
0.993
0.732
0.541
0.538
Eriml (/20)
1.067
0.716
0.421
0.420
0.432
0.425
0.897
0,613
1,049
0.775
0.602
0.003
Average
0.946
0.756
0.255
0.319
0.260
0.325
0.475
0.501
0.908
0.721
0.405
0.453
Average
100.0%
100.0%
27.0%
42.2%
27.5%
43.0%
50.2%
66.3%
95.7%
95.4%
42.8%
59.9%
Informer) models. We used 2 datasets for experiments: the large-scale Electricity dataset with pro- nounced inter-series relationships and regular temporal patterns, and the smaller ETTml dataset with less evident inter-series links and irregular patterns. With fixed L, 720, our experiments show ARM's consistent performance boost without the need of tuning L, like previous models. For datasets with weaker inter-series dependencies, like ETTml, models employing univariate pre- dictors remain slightly superior. Conversely, for datasets with strong dependencies like Electricity, multivariate predictors have slight edge. The balanced structure of the vanilla Transformer, without over-design, performs consistently well comparing to other models after integrating ARM.
employing
4.1 ABLATION STUDIES
We conduct ablation studies on the three primary modules within ARM to showcase their specific mechanisms for performance enhancement. We use same setting for comparing different models and fix T., = 720. Implementation details for using and transferring the modules are provided in AA.6) Additionally, we conducted further ablation studies the details within the modules in $A.4
Effeets of AUEL In the third part of Table 2, we showcase the effects of adding the AUEL module to various models. Firstly, AUEL consistently enhances performance, especially for multivariate models: existing multivariate models mixed multiple input series directly without modeling indi- vidual univariate effect. In contrast, for univariate models, AUEL's improvements arise from its adaptive distribution handling. Comparing results from two multivariate models and two univari- ate models, the Electricity dataset, where inter-series dependencies are more evident and
8

---

Published as a conference paper at ICLR 2024
ries follow similar temporal patterns, AUEL-enhanced multivariate models outperform univariate ones. This is attributed to the better recogmition of the multivariate relationship after extracting the univariate effect. However, for ETTml, where series discrepancies more substantial, making inter-series dependencies learning more challenging, univariate models remain advantageous.
Remark 1. AUEL offers foundational univariate forecast for each series, allowing models allo- cate primary parameters towards modeling the more intricate inter-series dependencies.
Remark 2. AUEL's results also clarify why past univariate models surpassed multivariate counter- parts in LTSF: given that many benchmarks featured weak or complex inter-series relationships, previous multivariate models that blended varying series struggled with accurate series-wise pattern recognition. In such scenarios, opting to neglect Mard-to-identee inter-series dependencies and focus solely on modeling univariate dependencies prevents model overfitting and improves performance.
Effects of Random Dropping In the fourth of Table 2 we demonstrate the performance gains of multivariate models from the Random Dropping strategy. Since univariate models train each series separately, they are not affected by this strategy, thus not included. Integrating Random Dropping considerably improves the performance of predictors without noticeably increasing computational costs. The improvements are more pronounced in datasets like ETT with weaker inter-series depen- dencies due to more overfitting reduction. As these relationships strengthen, evident datasets like Electricity and even more in Multi, the improvement of Random Dropping tend to be smaller.
Remark 3. Random Dropping aids multivariate LTSF models by building subset-to-subset forecast- ing ensemble models, effectively identifying the right combination of inter-series dependencies, thus reducing the risk of overfitting from misinterpreted series wise relationships.
Effects of MKLS In the fifth part of Table 2 we show the effects of integrating MKLS into various models. For multivariate input models, MKLS helps Iearn local patterns across series, particularly evident on datasets with pronounced inter-series relationships like Electricity. For univariate models, adding MKLS is to attempt building multivariate relationship at the token representation level, which can lead to overfitting when series-wise relationships are weak, as shown in the results.
Remark 4. MKLS addresses challenges in handling series with diverse local patterns by building suitable multivariate representations and enhancing locality. Due to its dynamic local window adapt- ability and fitting capacity, pairing MKLS with Random Dropping is recommended to prevent po- tential pitfalls from wrongly learned local patterns.
Effects of MKLS with Random Dropping In the sixth part of Table 2 we highlight the benefits of jointly employing MKLS and Random Dropping. Using both significantly enhances performance compared to their individual applications. Adding Random Dropping reduces potential overfitting during muidiw convolutional layer training and channel wwse attention calculations in MKLS. This synergy allows the MKLS MKLS+predictor combination to be trained more effectively for multivariate input. Coupled with AUEL,, ARM consistently tackle diverse multivariate forecasting challenges.
Overall Remark. Within ARM, we employ AUEL to adaptively learn the univariate effect for dis- tinct series, followed by utilizing MKLS and Random Dropping to guide the predictor in discerning correct inter-series connections. Integrating all modules equips the model to handle varied datasets, ensuring correct fitting even when faced with pronounced differences temporal dependencies and lo- cal patterns, subdued inter-series connections, leading substantial performance improvements.
5 CONCLUSION
In this study, we introduce ARM, methodology designed for effectively training multivariate LTSS models. Based on three primary modules: AUEL, Random Dropping, and MKLS, ARM addresses the challenge of handling time series with significant characteristic differences. It achieves this by extracting univariate effects, building reasonable multivariate representations, and discerning effec- tive series combinations for forecasting contribution. Remarkably, with only a minor computational increase, ARM empowers vanilla Transformer to achieve SOTA performance in LTSF tasks. Each module within ARM can be easily integrated into other LTSF models to enhance their performance, underscoring its practical utility and potential to influence future methodology research in the field.

---

Published as a conference paper at ICLR 2024
REFERENCES
Anastasia Borovykh, Sander Bohte, and Cornelis W Oosterlee. Conditional time series forecasting with convolutional neural networks. ArXiv preprint, abs/1703.04691, 2017. URL https://
Ttxiv.org/abs/1100 04691
George EP Box, Gwilym M Jenkins, and John MacGregor. Some recent advances in forecasting and control. Journal of the Royal Statistical Society: Series C (Applied Statistics), 23(2):158-179, 1974.
Li-Juan Cao and Francis Eng Hock Tay. Support vector machine with adaptive parameters in finan- cial time series forecasting. IEEE Transactions neurul networks, 14(6):1506-1518, 2003.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. ArXiv preprint, abs/1904.10509 2019. URL https::/arxivv.org
abs/1904, 15509
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 5232-5270, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting, 20(1):5-10, 2004.
Kyoung-jae Kim. Financial time series forecasting using support vector machines. Neurocomputing, 55(1-2):307-31, 2003.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In The Tenth International Conference on Learning Representattinns ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: /ttp:::://ppeeeeieeee net forum?id- cGDAK0olCOp,
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short- term temporal patterns with deep neural networks. In Kevyn Collins-Thompson Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (eds.). The 41st International ACM SIGIR Conference Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08- 12- 2018, pp. 95-104. ACM, 2018. doi: 00.1145/2099788.2210000 URL 3210006
https: /ttp:::/doi.oor/10..
wwg. 1tt.:/1...111522219999
Ibai Lana, Javier Del Ser, Manuel Velez, and Eleni Vlahogianni. Road traffic forecasting: Recent advances and new challenges. IEEE Intelligent Transportation Systems Magazine, 10(2):93-11) 109, 2018.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buuc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5244-5254, 2019. URL htttt::://procceeeiie 67 $752063c3022222233332
9d8 6be0 neurips. CC cc/ppaper/219//aa...
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, In The Tenth International Conference on Learning Representations, ICLR 2022, Vir- tual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: http:///ppeeeviiieee .ne.ie.ww.ee/eeeee.............. hollatttppp::peeeeeeeee
forum? d= QFXm 5T
10

---

Published as a conference paper at ICLR 2024
Simin Ma, Yan Sun, and Shihao Yang. Using internet search data forecast covid-19 trends: A sys- tematic review. Analytics, ( 1(2):210 2022. ISSNN 2813-2203. doi: 10.3390/amalyties 1020014 com. http://cccmmm11122221111111
UtRtts:::
Francisco Martinez-Alvarez, Alicia Troncoso, Gualberto Asencio- Cortés, and José C Riquelme. A survey on data mining techniques applied to electricity-related time series forecasting. Energies, 8(11::3312-13113, 2015.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-tern tengtterm forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2022.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: neural ba- sis expansion analysis for interpretable time series forecasting. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020. URL
net www.nneeoooodddreeeeeeeeaaa.
Yaohao Peng and Mateus Hiro Nagata. An empirical overview of nonlinearity and overfitting in machine learning using covid-19 data. Chaos, Solitons & Fractals, 139: 10055, 2020.
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison W. Cottrell. A dual-stage attention-based recurrent neural network for time series prediction. In Carles Sierra (ed.), Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 2627-2633, ijcai.org, 2017, doi: 10. 2..49993/jccii.20177366. 7/366.
http./////.....933111111 http:///aaaaaaa111111. 2017/366
Syama Sundar Rangapuram, Matthias W. Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural In- formation Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 2018 URL http:s: 6cf68969fb67es 7 (11 Abst
nenrss wwwpaper/211//heeeeeeeeepee 8/hahh/
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Lin- guistics, 9:53-68, 68, 2021. doi: 10.1162//al URL https:://aaaatttool/////// orr
2021.tac1-1..
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting. 36(3):1181 1191, 2020.
Rajat Sen, Hsiang-Fu Yu, and Inderjit S. Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- fion Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 4838 4847, 2019. URL htt.ttss:///rrcceedddng neurin wttp:/ee.rrrppp.p.peepeeeeeeee 2a044ceeffcfff6ee77eeeeeeeee 3a344feeeedt/eeec11eaaa03333-3-3
hccppappe//211/aaaas .tmal
Shun- Yao Shih, Fan- Fan-Keng Sun, and Hung-yi Lee. Temporal pattern attention for multivariate time series forecasting. Machine Learning, 108: 108:14211144) 2019.
Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36(1):75-85, 2020.
Antti Sorjamaa, Jin Hao, Nima Reyhani, Yongnan Ji, and Amaury Lendasse. Methodology for long-term prediction of time series. Neurocomputing, 70(16-18):2861-28688 2007.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in-
gredient for fast stylization. ArXiv preprint, abs/1607.08022. 2016. URL https: : //arxiv.
org//bb/ 607. 08022

---

Published as a conference paper at ICLR 2024
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In 9th ISCA Speech Synthesis Workshop, pp. 125-125, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL ht httpttppe//poocc..ggg...... neur ps. CC pawppr/2c1//aasss.. Abtraa html
Abstract
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamm, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. ArXiv preprint, abs/1711. 2017. URL https://arx/:
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Marc' Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 22419-22430, 2021. URL https::: ptttps::/proceeedinge wwwrips.pp... pccpppper///21// hash/bccodd 33dddd wttpoaddddcceaaaaaacocc 9aBac0c2b Abst:e22bbbbttttt.. cetsstratttttml.
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nnt, 2020. URL httthhhtt:///pppeee httphtsts/:/opeeeeveeee
net/forum?idee dddoouuummddyeeePP ННPH
Shihao Yang. Mauricio Santillana, and S. C. Kou. Accurate estimation of influenza epidemics using google search data via argo. Proceedings of the National Academy of Sciences, 112(47):14473- 14478, 2015. doi: 10. (0373//pas.11153331111 10. 10.1073/pnas 1515373112
htt.ttts:: www pnas
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 11121-11128, 2023.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Edu- cational Advances in Artificial Intelligence, EAAI 2021, Viriual Event, February 2-9, 2021, pp- 11106-11115, AAAI Press, 2021. URL https: /ots////s///../ aaai. org/
aaai.org/index. ii//xxndexx.///// php//AAI/ AAAI/
Article/view/Iiee 11322
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 27268-27286. PMLR, 2022. vttp/vl2////// 62wo22//hoo22o html URL htttps:/procceecingg mlr wtt.l/ppeeessseeeeoooooooe
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 DATA SOURCES
We evaluate the performance various long-term series forecasting algorithms using a diverse set of 10 datasets. Details of these datasets are listed below:
The ETT lataset(Zhon et al. 2021 is collection of load and oil temperature data from electricity transformers, captured 15-minute intervals between July 2016 and July 2018. The dataset comprises four sub-datasets, namely ETTml, ETTm2, ETTh1, and ETTh2, which correspond to two different transformers(labeeed with 1 and 2) and two different resolutions(15 minutes and I hour). Each sub-dataset includes seven oil and load features electricity transformers.
The Electricity datase includes hourly electricity consumption data for 321 clients from 2012 to 2014. has been used in various studies related to energy consumption analysis and forecasting.
The Exchange Lataset et al. 2018 is a collection of daily foreign exchange rates for eight countries, covering the period from 1990 to 2016.
The Traffic datase is a collection hourly road occupancy rates obtained from sensors located on freeways in the San Francisco Bay area, as provided by the California Depart- ment of Transportation. The dataset spans the period from 2015 to 2016 and has been used in various studies related to traffic forecasting and analysis.
The Weather datase comprises 21 meteorological indicators, such as air temperature and humidity, recorded every 10 minutes throughout the entire year of 2020.
The III datase is a collection of weekly data on the ratio of patients exhibiting influenza- like symptoms to the total number of patients, reported by the Centers for Disease Con- trol and Prevention of the United States. The dataset spans the period from 2002 to 2021 and has been used in various studies related influenza surveillance and analysis.
A.2 To WHAT EXTENT Do WE NEED INDEPENDENCE LTSF: MULTIVARIATE DATASET WITH SIGNIFICANT DEPENDENCIES
Additionally, we generate a multivariate dataset, termed as "Multi", with strong dependencies among series illustrate scenarios where channel-independen approaches may fail in the presence of inter- series causal relationships. This dataset comprises eight series that individually lack long-term pre- dictability, yet when considered in relation each other, significant dependencies become appar- ent. As shown in Table I ARM (Vanilla) outperforms univariate models and previous multivariate models in modelling inter-series dependencies in the Multi dataset, with the visualization of the forecasting results shown in Figure 6
To build the Multi dataset, we first generate 20,000 steps random noise e N 1) and build random walk process X where X(t),1 (t-1)), + +c(9) ((1).1 c()), We then take 18,000 steps from the interval between 2,000 and 20,000 as first time series (0).1 The remaining seven
http://pems.dot.ca.gpp
13

---

Published as a conference paper at ICLR 2024
series generated as follows:
x(1.2 = X y(1-96),1
(4)
X(1).33 X (c-1992),
(5)
= X(1,,
336-336),1
X = (((
1 X(1,33
X x
2
2
x
1 x
x(033
1
(())4
x
X(188 = x
x(1,3
(10)
It can be seen that when each series is considered separately, due to their sampling from the random walk process, effective long-term forecasting unattainable. However, with knowledge of the past trajectory of (0).) we can accurately infer the values of the remaining seven series through (
In this dataset, we can observe that there is only one actual generative sequence, making the mix- ing and compressing of all channels into lower dimension feasible. However, as seen from Table [. although bbannel-independent univariate models are clearly unsuitable for this dataset, neither Autoformer nor Informer outperform the univariate models. This verifies our previous assertion to some extent: previous multivariate LTSF models likely have not learned the causal relationships between different series because of its inefficiency usage of parameters. from Figure 6 ARM effectively learns the causal relationships between series, performing rather perfect for shifting and linear, non-linear combinations of the generative series. In scenarios with strong multivariate dependencies, ARM's forecasting results significantly exceed previous multi- variate univariate LTSF models.
contrast, as we can see
A.3 ADDITIONAL EXPLAINATION OF THE THREE EXAMPLES IN FIGURE
The necessity of adaptive estimation of output mean for series with different characteristics (see 1): We illustrate how different methods estimate the output mean for five example time series. The blue lines denote the original series, with the input and output sections separated by purple line. Using yellow and red lines, we represent methods that employ the input mean (RevIN) and the last input value (NLinear) respectively as the output mean. These previous methods are suboptimal as they don't t account for varying lookback lengths required for different series. We highlight the ideal lookback areas with green boxes. The Adaptive EMA in our Adaptive Univariate Effect Learning method, represented by green line, effectively learns the optimal output mean, thereby handling series with diverse temporal dependencies more accurately.
(a)
(b) The necessity of building reasonable multivariate temporal representation (see $3.3 which is demonstrated by five series from the Electricity dataset. A black vertical line used to indicate the current time step t. Their associated temporal relationships and local patterns vary for different series. Thus, adopting the values from (*) as the representation for time step t, as done in previous models, wrongly blends the modeling across series with distinct characteristics. We use green, red, and yellow boxes to depict the optimal scope of local patterns that should be considered for different series at time To accommodate these distinct patterns, we introduce miltiw local convolutional module construct reasonable representation for multivariate inputs.
(c) The Inability of Previous Models to Learn Obvious Inter-Series Dependencies: We illustrate the ineffectiveness of existing multivariate LTSF Transformers using a dataset "Multi", where the sub- sequent three series are simple shifts of the first (details in section A.2p. A "copy-paste" operation should suffice for forecasting, yet previous multivariate models, such Autoformer, fall short complish this task efficiently. In Figure we visually demonstrate ARM's enhancement in handling these strong inter-series relationships for LTSF models.
14

---

Published as a conference paper at ICLR 2024
A.4 ADDITONAL ABLATION STUDIES
In Table 3
we conducted ablation studies on finer details within ARM. The last row of the table presents the optimal results for ARM (Vanilla), with other detailing variations this vanilla ARM to observe result changes. Some more granular analyses of ARM's effects are also provided.
A.4.1 SUB-MODULES OF AUEL
The first section of Table 3 displays the impact of modifications in the AUEL module. Initially, we attempted to omit modules for learning distribution and temporal patterns separately; both deletions led to performance dips, especially for the latter. Such declines were more pronounced on datasets with weaker inter-series dependencies, like ETTml. Subsequently, we replaced the MoE module in learning temporal patterns (see Figure 3 (c)) with independent linears (see Figure 3 (a)) and single linear layer (see Figure 3 (b)). As analyzed in the main text, these substitutions diminished the effectiveness of temporal pattern learning.
A.4.2 SUB-MODULES OF MKLS
The second section of Table presents ablation studies on MKLS sub-modules. When either Pre- MKLS or Post-MKLS was removed, there was a noticeable performance degradation. Trying to eliminate the channel-wise attention MKLS, where outputs from different convolution kernels are averaged with equal weights for each series, also affected performance.
We further examined optimal kernel size choices within MKLS. It's worth noting that for other experiments, we utilized [49 145 385]. Results indicate that, for Lı = 720, this size selec- tion outperforms other combinations. Especially in datasets like Electricity, with numerous series and longer temporal dependencies, proper use of larger convolutional kernels somewhat improved performance.
A.4.3 OTHER COMBINATIONS OF ARM
Considering eight possible combinations of A/R/M modules, six were addressed in the main text. In the third section of Table evaluate the remaining two combinations. Given that AUEL offers baseline univariate solution for forecasting, combinations including the A module yield results close the SOTA. From AM and AR combinations, it's evident that combinations solely containing M are more susceptible to overfitting. Integrating all ARM modules resulted in a performance boost compared to just using AM or AR.
Table 3: Additional ablation studies of the submodules included in AUEL and MKLS. Other two combinations ARM modules that are not included in the result table the main text is also presented. The experiments are conducted on the basis of the Vanilla ARM setting.
(Predict (.))
Electricity (1.p Electricity (J.p 336)
EfTml (l.p
FTTml (l.p 336)
MAE
MAE
AUEL
w/o learning distribution
0.132
0.232
0.159
0.262
0.293
0.349
0.370
0.391
AUEL
w/o learning temporal patterns
0.142
0.235
0.185
0.287
0.379
0.525
0.403
0.439
AUEL
independent linears
0.133
0,231
0.165
0.264
0.301
0.346
0.379
0.393
AUEL
single linear
0.128
0.226
0.159
0.261
0.307
0.353
0.384
0.399
MKLS
w/o Pre-MKLS
0.127
0.225
0.161
0.262
0.292
0.348
0.369
0.391
MKLS
w/o Post- MKLS
0.130
0.229
0.163
0.261
0.295
0.349
0.370
0.390
MKLS
attention
0.129
0.227
0.158
0.255
0.294
0.347
0.367
0.389
MKLS
kemel [49)
0.129
0.226
0.163
0.260
0.287
0,347
0.367
0.389
MKLS
kernel a 1:15)
0.129
0,230
0.164
0.262
0.292
0.345
0.366
0.387
MKLS
kemel
[49 145 385 673]
0.127
0.224
0.156
0.255
0.297
0.349
0.365
0.389
A/R/
ARR
0.132
0.232
0.165
0.263
0.298
0.349
0.372
0.393
A/R/M
Vinilla+AM
0.136
0.238
0.167
0.266
0.321
0.368
0.386
0.406
Full
ARM (Vanilla)
0.125
0,222
0.154
0.251
0.287
0.340
01.64
0.384
15

---

Published as a conference paper at ICLR 2024
Table 4: Evaluation metries of previous LTSF models with only the adaptive distribution learning module in AUEL (denoted as A(D)). Forecasting input lengths L, are set to 720 for all the experi- ments.
Mudids Mesic
Autufiensr
Autrfomo+AA())
DLincar
Diins++A(D)
PaciTST
PuchITTTAA/D)
M5E
MAE
MSE
MSE
MAE
Exclanee
96
1.130
0.832
1,012
0.765
0.141
0.280
0.104
0.229
0.087
0.210
0.081
0.197
192
1.203
0.850
1.082
0.795
0.364
0.393
0.240
0.346
0.182
0.306
0.170
0.291
316
0.912
1.181
0.81
1,606
1,095
0.465
0.478
0.377
0.446
0.315
0-402
230
1.445
2,066
1.087
6,486
1.314
1.350
0.3%
1.087
0.791
0,877
0.683
0.362
0.142
0.100
0.145
0.195
0.337
0.372
0.379
0.305
0.325
0.309
0.328
PatchTST previously utilized RevIN, which is replaced with the adaptive distribution learning in AUEL
Table 5: The enhancement of the efficiency of parameter utilization provided by learning temporal patterns (MoE) in AUEL. We conducted experiments using the Vanilla model settings. We specif- ically evaluated the performance when the model dimension d Transformer blocks is set to 16, 64, and 256 respectively. The left three experiments employed the MoE from AUEL, while the right three did not. The overall best results are highlighted bold, and the best results within each group underlined.
Model
MoE (d 16)
MoE (d 64)
MoE (d
256)
wlo MoE
16)
w/o MoE (d 64)
w/o MoE (d 256)
Metric
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
Electricity
96
0.128
0.228
0.126
0.225
0.129
0.229
0,313
0.392
0.261
0.348
0.256
0.341
Electriitty
192
0.146
0.242
0.142
0.239
0.145
0.245
0.299
0.378
0.255
0.349
0.251
0.352
ETTml
0.287
0.348
0.300
0.467
HTTm
192
0.325
0.371
0.342
0.376
0.346
0.385
0.531
0.499
0.498
0.482
0.574
0,512
A.4.4 EFFECT OF AUEL DISTRIBUTION LEARNING
This section displays the impact of solely employing adaptive distribution learning from AUEL on existing LTSF models, as illustrated in Table 4 Adaptive learning for level and variance proves effective in scenarios with distinct series temporal dependencies and varying local distributions, as seen in the Exchange and Weather datasets. Results affirm the significant role of adaptive distribution learning in enhancing performance across irregular datasets.
A.4.5 REDUCTION AND ROBUSTNESS IMPROVEMENT OF PARAMETER SIZE WITH ARM
In this section, we demonstrate the efficacy of AUEL in enhancing model parameter efficiency and reducing the optimal parameter size required in the main predictor, as shown in Table []] Owing to AUEL's ability to disentangle univariate effects, the majority of parameters in the main predictor are primarily used for modeling inter-series dependencies, resulting a significant reduction in the number of parameters needed. From the results, it is evident that on both the larger-scale Electricity dataset and the smaller ETTml dataset, the introduction of MoE significantly reduces the optimal model dimension d.
Also, with the existence of ARM, the model become insensitive to the choice of d: after the extrac- tion of univariate effect by AUEL, the model parameters of other parts, whose size mainly controlled by model dimension d, will focus on modelling the inter-series dependencies. d typically needs to be tuned to accommodate different inter-series relationship strength of datasets. avoid overfitting in datasets with weak inter-series relationships, whereas a larger d is necessary for datasets with stronger inter-series dependencies. Incorporating Random Dropping mitigates the need for tuning d across various datasets. Random Dropping can be regarded as an adaptive scaler for d, allowing the model to automatically adjust to the strength of these relationships. Consequently, can be set solely based on the number of input series C' (we use d = 16 for C' < 20 and d = 64 for C' > 20 for all the experiments), with Random Dropping modulating the learning of inter-series relationships. On datasets Exchange (weak inter-series dependencies) and Multi (strong inter-series dependencies) with both of their C' = 8, the results in Table 6 demonstrate the effectiveness of "R": using consistent d 16 that potentially fits stronger inter-series dependencies (like Multi), models with Random Dropping (+ARM) perform equally well on both types of datasets, without the overfitting on Exchange dataset observed in models without Random Dropping (+AM).
smaller d helps
16

---

Published as a conference paper at ICLR 2024
Table 6: The insensitivity of the performance regarding the model dimension with the existence of Random Dropping. We conducted the experiments with a consistent d = 16 on two datasets with the same number of series C = 8 but different strength of inter-series relationship. Exchange dataset have weak inter-series link which inherently requires less parameters d < 16 to model it, while Multi dataset have stronger inter-series dependencies and fits the scenario of = 16. With ARM comparing to AM, models perform equally well on datasets with both strong and weak inter-series dependencies. In other words, random dropping can provide more improvement on the datasets with weaker inter-series dependencies when d is high enough based on C.
Modek
Astoformar+AA
Astoformer-ARM
Informr+AMM
Infremor-ARM
Vanilla-AM
Vanilla+ARM
Metric
MSF
MAF
MSR
MAF
MSF
MAF
MSF
MAR
MSF
MAR
MSF
MAE
Exchange (96)
0.089
0.21
0.081
0.201
0.090
0.212
0.086
0.207
0.085
0.205
0.078
0.197
Exchange (336)
0.352
0.417
0.298
0.395
0.338
0.420
0.312
0.404
0.304
0.398
0.252
0.367
Average
100.0%
100.0%
90.0%
95.1%
100.0%
100.0%
93.0%
96.7%
100.0%
100.0%
94.8%
93.4%
Muhi
0.051
0.034
Mulii (336)
0.181
0.302
0.175
0.300
0.182
0.305
0.183
0.304
0.172
0.291
0.161
0.286
Average'
100.0%
100.0%
97.7%
99.9%
100.0%
100.0%
100.4%
100,25
100.0%
100.0%
95.1%%
97.9%
A.5 MODEL TRANING AND SAPPER-PARAMTERS
The ARM (Vanilla) model is trained using the Adam optimizer and MSE loss in Pytorch, with a learning rate of 0.00005 over 100 epochs on each dataset with a carly-stopping patience being 30 steps. The first 10% of epochs are for warm-up, followed by a linear decay of learning rate. Owing to ARM's adaptability to long sequences, unlike baseline models, we only use a lookback window of length 720 for training (and 104 for the ILI dataset). The melti-kermel size s of MKLS is set to [25 145 385]. which is also used as the multi-window size in the AUEL's adaptive learning of standard deviation.
We run our model on single Nvidia RTX 3090 GPU. We use a batch size of 32 for most datasets. For some datasets with larger number of series, if a longer Lp renders the running unfeasible, we try to reduce the batch size to 16 or 8, respectively. For the dimension of the main part of the model, need adjust it accordingly based consider the scale of dataset if we are not familiar with the dataset. We employ a Transformer model dimension d = 16 for most of the small datasets (C < 20) like ETTs, Exchange, ILI, and Multi with less series-wise relationship needed to be learned by our model. For the datasets like Weather, Electricity, and Traffic (C > 20), which have more series and potentially more series-wise relationship to learn, we raise the dimension d to 64 (you can also refer to Table for more intuition of the setting of d). You can basically use these two dimension settings for most of the datasets. As for the number of heads in multi-head attention, we set it to 8. For the number of layers in the Transformer encoder-decoder structure, we use two encoder layers and one decoder layer. We do not apply dropout in the Transformer Encoder; in the MKLS, we set the dropout rate to 0.25; and in the MoE, we set the dropout rate to 0.75. We use a high dropout rate for MoE to avoid overfitting because we set a relatively large 4(L, + Lp) as the hidden dimension of the MoE predictors. As for the number of experts in the MoE module, we use 2 experts for the small datasets stated above and 4 experts for the larger datasets. We set the random seed for the main experiments as 2024.
the strength of series-wise relationship. We can mainly
For model selection, we partition each dataset into training, validation, and test sets with proportions of 70%, 10%, and 20%, respectively. The models are trained on the training set, and the best model is selected based on its MSE on the validation set. The MSE and MAE of this model on the test set are reported.
A.6 ADDITIONAL IMPLEMENTATION DETAILS OF MODULES
A. ADDITIONAL IMPLEMENTATION DETAILS OF AUEL
In the adaptive learning of distribution, we employ a' 0.9 as the initialized EMA alpha param- eter for each series. For the window of adaptive standard deviation, we initialize with equal weights. Clipping operations are used to prevent the EMA alpha and milti-window weights from becoming negative.
For adaptive learning of temporal patterns, we utilize the MoE predictor structure as described in (Fedus al. 2022). We adjusted the output dimension of its final layer cater to the in-
17

---

Published as a conference paper at ICLR 2024
put length of L1 + Lp, output length of Lp, with hidden dimension of 4(L, Lp). Given the discrepancy in input and output lengths, the MoE's residual shortcut needs specific modifications. We establish a residual connection using the Lp section of the input before the AUEL preprocessing and before the AUEL inverse processing. Specifically, in the calculation of MoE in AUEL preprocessing, where X
Xy
we actually compute MOE (x7] x
MoE (x*+ x selected ([x3+ X
+ Ojp. Here, MLP Malectee denotes the MLP predic-
tor selected by the routing part of MoE based on the input, and 0% is the Lp part before AUEL preprocessing, i.e., the default 0 values before AUEL. During inverse processing, we compute Xir MoE [ {ED MLP seleecedd
([x;* { + +Kin, where the added X is the
Lp part before the AUEL block.
A.6.2 ADDITIONAL. IMPLEMENTATION DETAILS OF MKLS AND THE TRANSFERING OF MKLS TO UNIVARIATE MODELS
When migrating ARM to other LTSF models, both AUEL and Random Dropping can be directly applied at the input and output ends. However, as MKLS requires integration with Transformer blocks, its application becomes challenging if the main predictor lacks a Transformer structure. Hence, for MKLS transfer, we employ approach that remains unaffected by the architecture of the main predictor. We treat the main predictor as whole: after data undergoes AUEL preprocessing and before entering the main predictor, we use Pre-MKLS. Once we obtain the result from the main predictor and before using AUEL inverse processing, we utilize Post-MKLS. This approach, which treats the predictor holistically as Transformer block, effectively enhances its ability to handle multivariate imputs and amplifies its locality learning capability.
In ARM (Vanilla), MKLS is applied to latent model dimension with d channels. For multivariate CTSFTransformes like Autoformer and Informer, they project input data to this latent model dimen- sion, making their MKLS application consistent with Vanilla. However, for univariate models like DLinear and PSt X not projected onto the latent dimension, but maintaining the original C' channels of input series when processing the input series. This inconsistency might skew results in model comparisons, impeding the full potential of MKLS in univariate models. Consequently, for these univariate models, we devised mixed input and output processing method. After AUEL preprocessing, we project the C series to a dimension of d - C and concatenate with these series to get Pre-MKLS input of dimension d. Thus, within this d-dimensional input, channels from the original series (C) coexist with channels representing a mix of multiple series (d- (d-C). After receiv- ing the predictor's output, it is fed into Post-MKLSS, then projected back dimension C, merged with the previous result, and subjected to AUEL inverse processing. This strategy seamlessly inte- grates MKLS's capacity for handling inter-series relationships into these univariate models, while preserving the univariate models' proficiency in processing intra-series information independently, providing fair comparison between the univariate methods and multivariate methods when imple- menting MKLS module on them.
A.6.3 ADDITIONAL IMPLEMENTATION DETAILS OF RANDOM DROPPING
Random Dropping can be regarded as a channel dropout module acting simultaneously on both the input and training target, with the dropout rate adjusted randomly each training step. In every training iteration, we randomly generate dropping rate ra and fill r&C of the C' input series with zeros. Through this method, we can explore all possible series combinations during training. By identifying groups or clusters of series with useful inter-series causal relationships in them, their forecasting contribution is gradually reinforced within the model parameters via gradient updates.
A.6.4 OTHER IMPLEMENTATION DETAILS
We use some types of widely-used embedding matrices adopted from in previous LTSF research. Firstly, a trainable position embedding matrix is added after performing the input projection. We also add two trainable task embedding matrices for both the L, part and Lp of the input. Timestep / Date Embedding is also optional to use our model architecture. Since these embedding matrices have very limited effects our model results, which is similar to the observation in previous research
18

---

Published as a conference paper at ICLR 2024
Table 7: Comparative Analysis of Computational Costs in LTSF Models. This comparison utilizes
the ETTml data format for constructing model inputs.
Vanilla+ARM (FLOPS)
Vanill+ARRM (Parums)
Vanilla (FLOPs)
Vanilla (Params)
Autoformer (FLOPs)
Autoformer
(Params)
Xp = 96
426M
7.89M
244M
14.9M
10.9G
15.5M
Xp = 192
515M
10.4M
273M
17.5M
11.6G
15,5M
Xp 336
664M
14.9M
316M
219M
12.7G
15.5M
Xp 720
1.15G
30.6M
431M
37.8M
15.5G
15.5M
Informer (FLOP)
Informer (Params)
PatchTST (FLOPs)
PatchTST (Params)
DLincar (FLOPs)
DLinear
(Params)
Xp = 96
9.41G
11.3M
5,26G
4.87M
3.06M
138K
Xp 192
10.1G
11.3M
5.31G
7.08M
6.10M
277K
XP 336
11.2G
11.3M
5,38G
10.4M
10,7M
485K
XP 720
14.0G
11.3M
5,58G
19.2M
22.8M
1.04M
like DLinear and PatchTST, we just simply enable these embeddings in our architecture without particularly emphasizing them.
A.7 COMPUTATIONAL COSTS
We present the comparison of computational costs between our Vanilla+ARM method and the isting LTSF SOTAs, as shown in Table 7 The results are calculated using the "ptflops" package. We conduct the experiments with the same input format the data input in ETTml dataset. For existing models, we build them with the best hyper-parameter settings stated in their original papers. We set the token dimension of Vanilla 64 and d of Vanilla+ARM 16 in order to emphasize the efficiently using of predictor parameters provided by our ARM architecture: the optimal dimension required by the encoder-decoder is reduced after applying ARM, as presented in Table
A.8 PREDICTABILITY ANALYSIS
To illustrate the variability in predictability across datasets, we conduct a simple analysis in this section, as shown in Figure 5 In each of the eight datasets shown in the figure, randomly se- lect subseries length 432, using the first 336 steps as the training set and the final steps as the testing set to fit traditional univariate time series models. The models we employ include naive repeat, ARMA(p. q), Simple Moving Average with different lookback windows (SMA), Exponen- tial Smoothing with different alpha values (ES), Random Forest with different lookback windows (RFR), and MLP with various hidden layer dimensions and lookback windows. We calculate the RMSE on the testing set as the performance metric for each model. To compare the relative perfor- mance of different models on various series, we normalize and invert the RMSE for each series, then standardize across the model dimension. This process ensures that a higher value corresponds bet- ter relative model performance. We visualize the transformed performance evaluations of different models on various datasets using boxplots.
As shown in the figure, in datasets where time series with regular patterns dominate, such as Electric- ity, more complex models like MLP and Random Forest demonstrate superior performance for these regularities, and longer lookback lengths yield better results. In contrast, in datasets with weaker regularity and sudden changes, like Exchange and ILI, methods that either focus on a shorter look- back view or assign greater weight recent datapoints, such SMA and ES, perform better. This predictability analysis indicates the necessity of adaptively adjusting the lookback view and com- plexity of model for time series with different characteristics. In ARM, and Random Dropping to accommodate these characteristics changes, effectively handling the sig- nificant differences observed across various datasets.
AUEL, MKLS,
19

---

Published as a conference paper at ICLR 2024
Nove Mideo
CRIC.31
STRIS2)
LE40
NLP138,501
NLH1200
ETT1
ETTIZ
3TInd
ETTm2
Enchange
Seurue
Figure 5: Predictability Analysis of Datasets (Elec. means Electricity)
A.9 VISUALIZATION ANALYSIS
A.9.1 VISUALIZATION OF THE FORECASTING FOR MULTI DATASET
In figure 6 we show the visualization of the best forecasting results for Vanilla, PatchTST, DLinear, and Autoformer models with and without ARM the Multi dataset, with a prediction length (LP ) of 96 steps. ARM effectively equips the LTSF models with enhanced ability of modelling inter- series shape connection.
A.9.2 RANDOMNESS OF TRAINING
Figure 7 illustrates the impact of adjusting the random seed the model performance. ARM integrates random training techniques, which might make the model training sensitive to the setting of random seed in small datasets. Thus, we conduct experiments use different random seeds for the training on the small datasets with irregular patterns like Exchange, ETTh1, ETTh2, Weather. Figure 7 demonstrates that in most cases, using fixed random seed for the training of ARM is enough to surpass previous best best- pestt-erffrming models.
A.10 FUTURE WORKS
For future research, potential applications of ARM modules in other time series tasks can ex- plored. Given that MKLS and Random Dropping decouple relationships between series, would be worth investigating their viability in time series classification and anomaly detection. Addition- ally, as AUEL effectively extracts series' intrinsic effects, enhancing the extraction of inter-series relationships, its application tasks like time series clustering can be further explored.
20

---

Published as a conference paper at ICLR 2024
(a) Vanilla
(b) PatchTST
(c) DLinear
(d) Autoformer
(e) Vanilla+ARM
(I) PatchTST+ARM
DRinear+ARM
(h) ARM
Figure 6: Visualization the best forecasting results for Vanilla, PatchTST, DLinear, and Auto- former models with and without ARM on the Multi dataset, with prediction length (Lp) of 96 steps. The blue lines show the ground truth time series data and the orange lines denote the 96-step forecasting provided by specific model. We have randomly selected one sample from the testing set for the visualization of each model. With an input length (Lr) of 720 steps, for the 2nd, 3rd, 4th, 5th, 6th, 7th, and 8th time series, on the input of X Existing channcl-independent univariate models like PatchTST and DLincar fail to model multivariate dependencies, leading subpar forecasting performance. Furthermore, traditional multivariate LTSF models, like Autoformer, as described earlier, are unable to effectively model the detailed causal relationships between series, leading similarly unsatisfactory perfor- mance. The visualization results show that models with ARM capture the shape of these time series significantly better compared the models without ARM. Note that the evaluation metrics displayed at the top of each subfigure represent the performance on the entire testing set, not on the current datapoint.
effeetively infer their subsequent 96-step values based
21

---

Published as a conference paper at ICLR 2024
Daadine Vanile-Oft
025
0.0
tschaоpele)) Sechnnge1300)
ETThL (90)
ETT1 (336)
STT 196)
ETTH2 230
Weather (PO)
Weather (330)
Figure 7:
Randomness of Training. The blue line represents the MSE of the combined best baseline results over these eight datasets. The red line, accompanied by corresponding error bars, depicts the best, worst, and average results of ARM (Vanilla) under five settings with random seeds 2021, 2022, 2023, 2024, 2025. On most datasets, the worse results of ARM varying random seeds can still surpass the previously best results from combined baslines.
22


Reviews:
[{"review_id": "oI2jYf0hEu", "reviewer": "Reviewer_PUmd", "summary": "The paper proposes an enhanced architecture for multivariate time series forecasting using Transformers. The proposed ARM approach incorporates three innovations - Adaptive Univariate Effect Learning (AUEL), Random Dropping (RD), and Multi-kernel Local Smoothing (MKLS). AUEL component introduces learnable exponential moving average instead of classic autoregressive and exponential smoothing for initiating prediction part into the encoder. Random Dropping is almost like an ensemble model which models selected subsets of time series, with aim to reduce spurious patterns among the timeseries. And MKLS which uses one-dimensional convolutional kernels and a channel-wise attention to encapsulate local information. The approach is evaluated on multiple datasets against several competitor approaches.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Consistently overperforms multiple strong competitor approaches on several datasets.\nAblation study is assessing effects of each of the three presented components.", "weaknesses": "Some of the performances in the Table one are reported from their respective papers, so I am wondering if it is likely to assure the same experimental setup. \nCertain statements are presented without appropriate evidence to support the claim. For example 'we introduce ARM, a methodology designed for correctly training multivariate LTSF models.', which is strong claim, and moreover implies that other approaches are 'incorrectly training'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a97sQu9wVu", "reviewer": "Reviewer_Eni6", "summary": "The manuscript tackles the multivariate time series forecasting problem, and proposes a solution called ARM.\nThe proposed method consists of 3 modules, and can be employed to many existing Transformer based time series forecasting models.\nEmpirical evaluation show that ARM, as well as its 3 modules individually, can improve the forecasting accuracy of the base Transformer model.", "questions": "It is not clear to me how MoE is operated from the manuscript.\n\nWhy Vanilla+ARM performs better than applying ARM to other Transformer based models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "The idea of applying the 3 modules to existing Transformer models for time series forecasting is novel, to the best of my knowledge. \n\nThe text of the manuscript is clear and not difficult to understand.\n\nThe proposed model has the potential to contribute to the time series modelling community.", "weaknesses": "As mentioned in the Strength part, the text of the manuscript is clear and easy to follow.\nHowever, the figures and tables in the manuscript are very hard to read.\n - Font size of the figures and tables are very small\n - The captions are super long and have a very small font size, I assume this violates the submission guideline.\n - The figures are arranged in an order which is different from how the text refers. Readers have to jump back and forth to find the corresponding (sub-)figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "30oLyWFlYu", "reviewer": "Reviewer_X2pk", "summary": "The paper introduces ARM : a multivariate temporal-contextual adaptive learning method for long-term time series forecasting, which is an enhanced architecture specifically designed for multivariate LTSF modelling. ARM consists 3 modules: Adaptive Univariate Effect Learning (AUEL), Random Dropping (RD) training strategy, and Multi-kernel Local Smoothing (MKLS).  AUEL is for adaptively estimate mean and variance and capture temporal pattern for each series; RD is for robust learning and avoid overfitting when series are interdependent; Multi-kernel Local Smoothing (MKLS) for capturing various temporal dependency among series.  ARM demonstrates superior performance on multiple benchmarks without significantly increasing computational costs compared to vanilla Transformer.", "questions": "1.\tHow does MoE better capture temporal patterns compared to autoregressive and exponential smoothing models? \n2.\tWhat is inverse processing stage? \n3.\tThe paper mentions causal dependency/ relationship among different series. Could the authors explain in what sense the notion of causality is embedded? \n4.\tI am confused about the how Multi-kernel local smoothing helps capture temporal dependencies among different series due to notations. For example, What is X_j. Could the authors elaborate more? \n5.\tThe paper demonstrates superior empirical performance for forecasting task.  It is hard to see how significant the improvement is compared to other recent models such as PatchTST and DLinear. Do the authors have some insights on the sensitivity of hyperparameters in ARM module in the tuning/training process since many learnable hyperparameters are introduced. \n6.\tWhat are the complexity/runtime compute/memory to incorporate these modules in the vanilla model? How does that compare to PatchTST and DLinear?  \n7.\tA few discrepancies regarding Table 1. I found in the original PatchTST paper on traffic datasets Table 3: for MAE,  0.249 for L_P = 96, 0.256 for L_P = 192, different from 0.239 and 0.246 in this paper. Could the authors double check if they are consistent?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Originality: this paper is fairly novel. It attempts to address previous inferior performance of transformer models compared to a simple feed forward neural network DLinear(Zeng et al. 2022) in long term forecasting task. It proposes unique insights on temporal pattern learning of the output sequence distribution and interdependency among univariate series.\n\nQuality: The quality of the paper is high. The proposal of the three core modules are well motivated. AUEL is for adaptively estimate mean and variance and capture temporal pattern for each series; RD is for robust learning and avoid overfitting when series are similar; Multi-kernel Local Smoothing (MKLS) for capturing dependency among series. The authors have performed experiments on 10 benchmarks which are diverse enough to contain different aspect of multivariate time series. The authors also incorporate different ablations by incorporate one or two modules of ARM. \n\nClarity: the overall clarity of presentation is good. It is motivated by the drawbacks of current sota models and address issues one by one by proposing the modules of ARM.\n\nSignificance: the paper can be important to time series learning community.", "weaknesses": "I think quality and clarity can be further improved. \n\nQuality: while the empirical results (Table 1 and 2) show superior performance of the proposed model ARM, it will be better to also demonstrate how significant these results are. The authors claim the proposed approach will not significantly increase computational costs. But such discussion/analysis is not included in the paper. \n\nClarity: I found MKLS block in section 3.3 difficult to follow due to notations. The authors introduced many (subscripts and subscripts of X for example); it would be nice to refer to somewhere. \nTable 2 contains large amount of results which are replicate of Table 1. Maybe the authors can eliminate some and find a way to better present them? \nFigure 3 & 4. Due to large amount of information in Figures 3&4, when introducing the components such as MKLS block, I think it will be great to introduce pointers to these components in the figures.   For example \\tilde{X}^{*i} need pointers to point to in Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)