CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting
Authors: Xue Wang
Tian Zhou
Qingsong Wen
Jinyang Gao
Bolin Ding
Rong Jin
Venue: ICLR 2024 poster (2024)
Keywords: time series forecasting
transformer
robust learning
token mixing
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Recent studies have demonstrated the great power of Transformer models for time series forecasting. One of the key elements that lead to the transformer's success is the channel-independent (CI) strategy to improve the training robustness. However, the ignorance of the correlation among different channels in CI would limit the model's forecasting capacity. In this work, we design a special Transformer, i.e., **C**hannel **A**ligned **R**obust Blen**d**  Transformer (CARD for short), that addresses key shortcomings of CI type Transformer in time series forecasting. First, CARD introduces a channel-aligned attention structure that allows it to capture both temporal correlations among signals and dynamical dependence among multiple variables over time. Second, in order to efficiently utilize the multi-scale knowledge, we design a token blend module to generate tokens with different resolutions. Third, we introduce a robust loss function for time series forecasting to alleviate the potential overfitting issue. This new loss function weights the importance of forecasting over a finite horizon based on prediction uncertainties. Our evaluation of multiple long-term and short-term forecasting datasets demonstrates that CARD significantly outperforms state-of-the-art time series forecasting methods. The code is available at the following repository: https://github.com/wxie9/CARD.

Published as a conference paper at ICLR 2024
CARD: CHANNEL ALIGNED ROBUST BLEND TRANS- FORMER FOR TIME SERIES FORECASTING
Xue Wang" Tian Zhou' Qingsong Wen
Jinyang Gao Bolin Ding Rong Jin
(xue. w, tian. 2t, gingsong. wen, jinyang giy, bolin.... ding, jinrong. ir a www.aainn......... com
ABSTRACT
Recent studies have demonstrated the great power of Transformer models for time series forecasting. One of the key elements that lead to the transformer's success is the channeel-ndeeendeet (CI) strategy to improve the training robustness. However, the ignorance of the correlation among different channels in CI would limit the model's forecasting capacity. In this work, we design a special Transformer, i.c., Channel Aligned Robust Blend Transformer (CARD for short), that addresses key shortcomings of CI type Transformer time series forecasting. First, CARD introduces channel-aligned attention structure that allows it capture both temporal correlations among signals and dynamical dependence among multiple variables over time. Second, order efficiently utilize the multi-scale knowledge, we design token blend module to generate tokens with different resolutions. Third, we introduce robust loss function for time series forecasting alleviate the potential overfitting issue. This new loss function weights the importance of forecasting over a finite horizon based on prediction uncertainties. Our evaluation of multiple long-term and short-term forecasting datasets demonstrates that CARD significantly outperforms state-of thet--rr time series forecasting methods. The code is available at the following repository: https: htap:://ggthii /gtph/bttbt/t///////// cwm/ /wmni/////////// CARD.
INTRODUCTION
Time series forecasting has emerged as a crucial task in various domains such as cloud computing, air quality forecasting, energy management, and traffic flow estimation(Qian et al., 2022: Liang et al., 2023; Zhu et al., 2023; Wen et al., 2023a). The rapid development of deep learning models has led to significant advancements in time series forecasting techniques, particularly in multivariate time series forecasting, Among various deep learning models developed for time series forecasting, RNN, CNN, MIP, transformer, and LLM-based models have demonstrated great performance thanks to their ability to capture complex long-term temporal dependencies (e.g., Zhou et al., 2021: Challu et al., 2022; Zeng et al., 2023; Zhou al., 2022a; Wu et al., 2023b; Zhou et al., 2023; Jin et al., 2023).
For multivariate time series forecasting, model is expected to yield better performance by exploiting the dependence among different prediction variables, so-called channel-dependent (CD) methods. However, multiple recent works (e.g., Nie al. 2023; Zeng et al. 2023) show that, in general, channel-inde -ineeeeedddeeeneent (CI) forecasting models (i.e., all the time series variables are forecast independently) outperform the CD models. Analysis from (Han et al., 2023) indicates that CI forecasting models are more robust while CD models have higher modeling capacity. Given that time forecasting models
series forecasting usually involves high noise levels, typical transformer-basee with CD design can suffer from the issue of overfitting noises, leading to limited performance. These empirical studies and analyses raised an important question, i.e., how to build an effective transformer to utilize the cross-chande information for time series forecasting.
In this paper, we propose a Channel Aligned Robust Blend Transformer, or CARD for short, that effectively leverages the dependence among channels (i.e.. forecasting variables) and alleviates the issue of overfitting noises in time series forecasting. Unlike typical transformers for time series
Equal contribution.
Work done at Alibaba Group, and now affiliated with Squirrel AL.
Work done A Alibaba Group, and now affiliated with Meta.

---

Published as a conference paper at ICLR 2024
analysis that only capture temporal dependency among signals through attention over tokens, the CARD also takes attention across different channels and hidden dimensions, which captures the correlation among prediction variables and aligns local information within each token. We observe that related approaches have been exploited in computer vision (Ding et al., 2022; Ali et al., 2021). Moreover, it is known that multi-scale information plays important role in time series analysis. We design a token blend module to generate tokens with different resolutions. In particular, we propose combine the adjacent tokens within the same head into the new token instead of merging the same position over different heads in multi-head attention. To improve the robustness and efficiency of the transformer for time series forecast, we further introduce an exponential smoothing layer over queries/keys tokens and a dynamic projection module when dealing with information among different channels. Finally, to alleviate the issue of overfitting noises, a robust loss function is introduced to weight each prediction by its uncertainty in the case of forecasting over finite horizon. The overall model architecture is illustrated in Figure 1. We verify the effectiveness of the proposed model on various numerical benchmarks by comparing it to the state-of-the methods for Transformers and other models. Here we summarized our key contributions as follows:
1. We propose a Channel Aligned Robust Blend Transformer (CARD) which efficiently and robustly aligns the information among different channels and utilizes the multi-scale information.
2. CARD demonstrates superior performance in several benchmark datasets for forecasting and other prediction- based tasks, outperforming the tfate-of-the-an models. Our studies have confirmed the effectiveness of the proposed model.
3. We develop a robust signal decay-based loss function that utilizes signal decay to bolster the model's ability to concentrate on forecasting for the near future. Our empirical assessment has confirmed that this loss function is effective in improving the performance of other benchmark models as well.
The remainder of this paper is structured as follows. In Section 2, we provide a summary of related works relevant our study. Section presents the proposed detailed model architecture. Section describes the loss function design with theoretical explanation via maximum likelihood estimation of Gaussian and Laplacian distributions. In Section 5, we demonstrate the results of the numerical experiments in forecasting benchmarks and conduct a comprehensive analysis to determine the effectiveness of the self-attention scheme for time series forecasting. Additionally, we discuss ablations and other experiments conducted in this study. Finally, in Section 6, the conclusions and future research directions are discussed.
RELATED WORK
2.1 TRANSFORMERS FOR TIME SERIES FORECASTING
There is a large body of work that tries to apply Transformer models to forecast long-term time series in recent years (Wen et al., 2023b). We here summarize some of them. LogTrans (Li et al., 2019a) uses convolutional self-attention layers with LogSparse design to capture local information and reduce space complexity. Informer (Zhou et al., 2021) proposes a ProbSparse self-attention with distilling techniques to extract the most important keys efficiently. Autoformer (Wu al., 2021) borrows the ideas of decomposition and auto-correlation from traditional time series analysis methods. FEDformer (Zhou et al., 2022b) uses Fourier enhanced structure to get linear complexity. Pyraformer (Liu et al., 2022a) applies pyramidal attention module with inter-scale and intra-scale connections which also get linear complexity. LogTram avoids a point- dot product between the key and query, but its value is still based on single time step. Autoformer uses autocorrelation to get patch-level connections, but it a handcrafted design that doesn't 't include all the semantic information within patch. A recent work PatchTST (Nie et al., 2023) studies using a vision transformer type model for long-term forecasting with channel independent design. The work closest to our proposed method is Crossformer (Zhang& & Yan, 2023). This work designs an encoder-decoder model utilizing a hierarchy attention mechanism to leverage cross cross-dimension dependencies and achieves moderate performance in the same benchmark datasets that use in this work. From the model architecture perspective, different from Crossformer, we employ an encoder-only structure, and the information is induced via lightweight token blend module instead of explicitly generating token
2

---

Published as a conference paper at ICLR 2024
Transformer Block
CAS
Attention for Channele
Altention for Tokens MLP decoder Layer
Attention over Channel Dimension
Hissen D nlcons
of Tokens per Channel
CRd
CA2
CAR
Tokenize
concatonate
Forecasting Results
Input Time Series
Patched Token Tensor
Spii
M.F
add norm
M.P
concaterrite
Channel wise Dutput Taken Tensor
resdual
Stack Several Transtormer Block
Figure Illustration of the architecture of CARD.
hierarchies used in Crossformer. The designs significantly enhance the robustness of CARD and result in substantial improvement in numerical performance.
2.2 RNN, MLP AND CNN MODELS FOR TIME SERIES FORECASTING
Besides transformers, other types of networks are also widely explored. For example, (Lai et al., 2018: Lim et al., 2021; Salinas et al., 2020; Smyl. 2020; Wen al., 2017; Rangapuram al., 2018; Zhou et al., 2022a: Gu et al., 2022) study the RNN/state-space models. In particular, (Smyl, 2020) considered equipping RNN with exponential smooth and, for the first time, beat the statistical models in forecasting tasks (Makridakis et al., 2018). (Chen et al., 2023; Oreshkin et al., 2020; Challu et al., 2022; Li et al., 2023; Zeng et al., 2023; Das et al., 2023; Zhang et al., 2022) explored MLP-ttpe structures for time series forecasting. CNN models (e.g.. Wu et al. 2023b; Wen et al. 2017; Sen al. 2019) use the temporal convolution layer to extract the subsequence level information. When dealing with multivariate forecasting tasks, the smoothness in adjacent covariates is assumed or the channel-independent strategy is used.
MODEL ARCHITECTURE
The illustration of the architecture CARD is shown in Figure 1. Let a, R the observation of time series at time / with channel C' > 1. Our objective is to use L recent historical data points (e.g.. at-L+1:.., a2) to forecast the future T steps observations. (e.g.. a(+1:.., ((T)), where L, T.T. 2
3.1 TOKENIZATION
We adopt the idea of patching (e.g., Nie al. 2023; Zhang & Yan 2023) convert the input time series into token tensor. Let's denote = [a.-i+1.....4 ar] E RCxL as the input data matrix, S and as stride and patch length respectively. We unfold the matrix A into the raw token tensor X XxP where N - + 1J. Here, we convert the time series into several P length segments, and each raw token maintains part of the sequence-level semantic information, which makes the attention scheme more efficient compared to the vanilla point-wise counterpart.
We then use a dense MLP layer Fi P + d, extra token To E IR C'd and positional embedding E IR CxN xd to generate the token matrix follows:
X To, F F(X) E
(1)
3

---

Published as a conference paper at ICLR 2024
where X E (Cx((+)) and d is the hidden dimension. Compared (Nie et al., 2023) and (Zhang Yan, 2023), our token construction introduces a extra To token. The To token an analogy to the static covariate encoder in (Lim et al., 2021) and allows us to have a place to inject the features summarized the longer history of the series.
We consider generating Q, K and V via linear projection of the token tensor X:
Q (X),,, K (x))), V Fx(X),
where Q, K, V E RCx(N+1)xd and Fg, FR, Fg are MLP layers.
where Q K, V. E RCx(N+))xdhadd ..., H. H. and dhead are number of heads and head dimension respectively. For each sam- ple, the total number of tokens is C(N + +1). In order to fully utilize all cross-channel information, the ideal attention should be required ( + 1)) computation cost, which can be very time- consuming and potentially can lead to easily over-fitting when training sample size is limited. In this paper, we consider paying attention alternately over each dimension instead.
We next convert Q K, K V into (,)),
3.2 CARD ATTENTIONS OVER TOKENS
When make attention over tokens, we slice the Q. K, and Vi on channel dimension into (2r)
{KF} and {V/S} with Q K V. E (W+1)xxd and 1, 2, ..., ...,C. C. Besides the standard attention in tokens, we also introduce an extra attention structure in hidden dimensions that helps capture the local information within each patch. The attention in both tokens and hidden dimensions is computed as follows:
A softmax
1
EMA(QF) (EMA(K?))
Va
A softmax (QF) KT
(4)
where AT E ((N++))(N+D, A E R Rhoud x duood and EMA denotes the Exponential Moving Average!.
By applying EMA on Q and K each query token will be able to gain higher attention scores on more key tokens and thus the output becomes more robust. Similar techniques also explored in (Ma et al., 2023) and (Woo et al., 2022). Different from those in the literature, we find that using fixed EMA parameter that remains the same for all dimensions enough stabilize the training process. Thus, our EMA doesn't 't contain learnable parameters.
The outputs are computed as:
A
V. V. A
We next apply the proposed token blend module to merge heads and generate tokens capturing multi- scale knowledge and the detailed discussions are deferred to section 3.4. The batch normalization (Ioffe & Szegedy, 2015) to Of and O% is then used to adjust the outputs' scale. Finally, the residual connection structure is used to generate the final output of the attention block.
The total number of tokens is the order of O(L/S) per channel and the complexity in attention along tokens is upper bounded by (2. 122/5²), which is smaller than OCC-d2 1.2) complexity of the vanilla point-wise token construction. practice, one can use efficient attention implementation (c.g., FlashAttention Dao et al. 2022) to further obtain nearly linear computational performance.
3.3 CARD ATTENTION OVER CHANNELS
We first compute (Q:). K, and (V) via Equation (2) and then slice them over token dimension into }. (KT") and (V:") with Q Ki" ViTh e C'dund and n 1,2, 2,,,.... N 1. Due the
Formally, an EMA operator recursively calculates the output sequence {y:} w.r.t. input sequence {æ:} yt + (1 a) where (0. is the EMA parameter representing the degree of weighting decrease.
4

---

Published as a conference paper at ICLR 2024
Add & Norm
Adti Norm
Projection
Projection
Projection
Projection
Norm
MLP
Narm
Blend
Blend
GOLU
Chern
Blend
Softhlix
SaftMax
SoftMa
Trenigo56
EMA
Transpons
DP
DP
MUP
MUP
input
Input Tokken Matrix
Matrix
Standord
Dual Allention Dynamii Projection
Matrik Product
Exponential Maving Average
Dynamic Projection
Bilen
Figure Architecture for the CARD attention block.
potential high-dimensionality issue covariates, the vanilla method may suffer from computation overhead and overfitting. Take traffic dataset (PeMS) as an example, this dataset contains 862 covariates. When setting the lookback window size as 96, the attention over channels will require at least 80 times the computational cost of attention over tokens. The full attention will also merge a of noise patterns into the output token and lead to spurious correlation the final forecasting results. In this paper, we consider using the dynamic projection technique (Zhu et al., 2021) to get "summarized" tokens to the K;" and Vin for n-th token dimension as shown in Figure 2. We first use MLP layers Fpt and Fpe to project head dimensions from dhead some fixed with < C, and then we use softmax to normalized the projected tensors P: and P as follow:
kd softmax (Fp (K;")) p softmax (F (V?" ()"))),
(6)
where pm pmu Cxr, Next the "summarized" tokens are computed by
K: = (PR) T K;", V. = (P VV
(7)
where Ki V.nh E RR Rxddaadd
Finally, the outputs are generated by applying Q K;" and V. to equations from (3) to (5) for n 1,2, ..., + 1. The upper bound of total computational cost is reduced to OLLSS r
which is smaller than OLL/S C d²) cost of the standard attention.
3.4 TOKEN BLEND MODULE
Multi-scale knowledge plays a crucial role in forecasting tasks and has significantly enhanced the performance of diverse models. (e.g., Xu al., 2021; Zeng et al., 2023; Wang et al., 2023b: Zhou et al.. 2022b: Zhang & Yan, 2023). Most of these works initially decompose the time series into seasonal and trend components and then employ separate structures to process the seasonal and trend components individually. However, this approach, despite its simplicity, leads to higher model complexity, which in turn increases computation cost and makes it susceptible to overfitting issue.
In this work, we consider using a specially designed token blend mechanism to utilize the multi- scaling structural knowledge without additional computation costs. The token blend module replaces the standard token reconstruction after the multi-head attention by merging the adjacent token within the same head to produce the token for the next stage. The output token tensor O from the multi-head attention has 4-D with shape C. H (N - 1) dhead- The token blend module will first merge the second and third dimensions and reshape into 3-D tensor with shape C x H(N + 1) x dhead- We then decouple the second dimension into three dimensions, i.e., H(N +1) hn x lt2 hg where
5

---

Published as a conference paper at ICLR 2024
Head
TO
T6
T4
T5
T8
New Tokans TO
12
13
14
16
17
Standard Taken Construction
Taken Blend
Figure 3: Illustration example of token blend block in CARD.
- Hh3. 22 N and hg 2 1. The final output O uses lı3 11 dhead to construct the token dimension. Here we call h3 as blend size. When h3 = 1. the aforementioned operations generate the same outputs in the standard transformer. When hg 2 2, the outputs will first combine the adjacent token within the same head, which would create the token that represents the knowledge over a larger range, i.e., lower resolution. With increasing the blend size h3, more tokens within the same heads are merged and the attention module in the next stage could have more chance to capture long-term knowledge. An illustration example is shown in Figure 3. By consolidating temporally adjacent tokens within the same head, the resulting new tokens encompass knowledge over an extended time period. This enables more effective exploration of low- resouutottion knowledge by increasing attention on these tokens. Our token blend module is also different from the hierarchical adjacent tokens merging procedure in (Zhang Yan, 2023). First, (Zhang Yan, 2023) merges at the token level, the output token sequence at the coarse level has higher hidden dimensions and shorter sequence lengths. We consider merging at the head level instead which maintains the same output token sequence shape. Second, the merging size in (Zhang & Yan, 2023) is fixed as 2, while we allow more flexible configuration. As a result, we achieve an implicit structure that enhances the extraction of multi-scale information without the need for additional explicit signal disentanglement process.
4 SIGNAL DECAY-BASED Loss FUNCTION
In this section, we discuss our loss function design. In literature, the Mean Squared Error (MSE) loss commonly used measure the discrepancy between the forecasting results and the ground truth observations. Let d (A),.... āitL( (A) and (A+1(A),..... ( be the predictions and real obversations from time 1 to + L given historical information A. The overall objective loss becomes:
min
EA
L Σ [á++((A) (c+((A)l)
(8)
One drawback of plain MSE loss for forecasting tasks is that the different time steps' errors equally weighted. In real practice, the correlation of historical information to far-future observations is usually smaller than that to near-future observations, implying that far-future observations have higher variance. Therefore, the near-future loss would contribute more to generalization improvement than the far-future loss. To see this, we assume that our time series follows the first-order Markov process, i.e., ax+1 (G(((a)) I²I), where G is the smooth transition function with Lipschitz constant 1, σ >> and t 1,2, .... Then, we have
var(a)41)= var(G(ar) + + 3 var(a,) + 1²1,
(9)
where var(a) denote the covariance matrix of a. By recursively using Equation (9) from ł + to and for all l [t, + L}. we have
var(at+:) = lo var(a.).
(10)
When a, is already observed, we have var(a:) = and Equation (10) implies var(a:4) 3 loo we use negative log-likelihood estimation over Gaussian distribution, we come up with the following approximated loss function:
(a - a(4((A)) var (ax-1) (a(+((A) - a(H((a))
min
EA
2
Iml
[(x+1(A) (A))
>E
E A
||ār+((A) ac)((A) (I1)
lg2
L
1=1
1=1

---

Published as a conference paper at ICLR 2024
Compared Equation (11) to Equation (8), the far-future loss is scaled down to address the high variance. Since Mean Absolute Error (MAE) is more resilient to outliers than square error, we propose to use the loss function in the following form:
min A A
1/2 [[)=+((A) -a--((A)))
(12)
T.
1=1
where Equation (12) be derived via Equation (11) with replacing the Gaussian distribution by Laplace distribution.
EXPERIMENTS
5.1 LONG TERM FORECASTING
Datasets We conducted experiments seven real-world benchmarks, including four Electricity Transform Temperature (ETT) datasets (Zhou et al., 2021) comprising of two hourly and two 15- minute datasets, one 10-minute weather forecasting dataset (Wetterstattoon) one hourly electricity consumption dataset (UCT), and one hourly traffic road occupancy rate dataset (PeMS).
Baselines and Experimental Settings We the following recent popular models baselines: FED- former (Zhou et al., 2022b), ETSformer (Woo et al., 2022), FilM (Zhou et al., 2022a). LightTSS (Zhang al., 2022), MICN (Wang al., 2023b), TimesNet (Wu et al., 2023b). Dlinear (Zeng al., 2023), Crossformer (Zhang & Yan, 2023), and PatchTST (Nie et al., 2023). We use the experimental settings in (Wu al., 2023b) applying reversible instance normalization (RevIN, Kim al., 2022) to handle data heterogeneity and keeping the lookback length as 96 for fair comparisons. Each setting is repeated 10 times and average MSE/MAE results are reported. The full results are summarized in Table 7 in the Appendix. More details model configurations, model code, and comparison with other early baselines can found Appendix D and Appendix B, respectively.
Results The results are summarized in Table 1. Regarding the average performance across four different output horizons, CARD gains the best performance out of 7 and 7 out of MSE and MAE, respectively. In single-length experiments, CARD achieves the best results in 82 % cases in MSE metric and 100% cases in MAE metric.
For problems with complex covariate structures, the proposed CARD method beats the benchmarks by significant margins. For instance, in Electricity (321 covariates), CARD consistently outperforms the second-best algorithm by reducing MSE/MAE by more than 9.0% average in each forecasting horizon experiment. By leveraging 21 covariates for Weather and 862 covariates for Traffic, we achieve large reduction in MSE/MAE of over 7.5 This highlights CARD's exceptional capability to incorporate extensive covariate information for improved prediction outcomes. Furthermore, Crossformer (Zhang& & Yan, 2023) employs comparable concept of integrating cross-channel data to enhance predictive accuracy. Remarkably, CARD significantly reduces the MSE/MAE by over 20% % on benchmark datasets compared to Crossformer, which shows our attention design is much more effective in utilizing cross-channel information. It's also important to note that while Dlinear shows strong performance those tasks using an ML.P-based model, CARD still consistently reduces MSE/MAE by 5% to 27.5% % across all benchmark datasets.
Recent works, such as Nie al. 2023; Zeng et al. 2023; Zhang Yan 2023) use the input length other than 96 and have shown performance improvement. In our study, we also report the numerical performance CARD with a varying lookback length in Appendix G, and CARD consistently outperforms all baseline models when prolonging input sequence as well, demonstrating significantly lower MSE errors across all benchmark datasets.
5.2 RECONSTRUCTION BASED ANOMALY DETECTION
Reconstruction based anomaly detection can be viewed as task to predict the input itself. In previous works, the reconstruction is a classical task for unsupervised point-wise representation learning. where the reconstruction error is natural anomaly criterion. We follow the experimental settings in (Wu et al., 2023a) and consider five widely used anomaly detection benchmarks. The results are summarized in Table 2. CARD outperforms the existing best result by 3% in F1 score on average. In
7

---

Published as a conference paper at ICLR 2024
Table 1: Long-term forecasting tasks. The lookback length is set as 96. All models evaluated on 4 different prediction horizons (96, 192, 336, } and average MSE/MAE results of ten repeats are reported. The best model is in boldface and the second best is underlined.
Models
CARD
PatchTST
MICN
TimesNet [Crosstormer Dlinear
LighiTS
FilM
ETSSfommer PEDformer
Metric
MSE MAEIMSE MAE MSE MAE/ MSE MAEJMSE MAE MSB MAE MSE MAE MSE MAE MSE MAE|MSE MSE MAE
ETTml
00.383
0.383 0.408 0.387 0:411 (0.400 4.406)0.435 0.417 0.403 0.407 0.435 0.437/0.408 0.399)0.429 0 0.425|0.44B 0.452
ETTm2
00.271
0.316 0.327 0.2884 0.340/0.291 0.333/0.609 0.521 0.350 0.401 0.409 0.436/0.287 287 0.328/0.292 0.342/0.305 0.349
ETThl
[0.433
0.429|0.455 0.444 0.440 0.462 0.450j0.486
0.481 0.456 0.452 0.491 0.479/0.461 0.456|0.452 0.510)0.4400 0.460
ETTh2
(0.367
0.390 0.406 0.402 0.437/0.414
0.42/60.966 0.690 0,559 0.515 0.602 0.543 0.406|0.439 0.452]0.437 0.449
Weather
(0.240 0.262)0.257 0.280 0.243 0. 252 0.287 0.310 0.265 0.317 06.261 0.312 0.339/0.271 0.334|0.309 0.360
Electricity
(0.169 0.256|0.216 0.318 187 0.295 0.225)0.273 0.363 0.212 0.300 0.329/0.223 0.303/0.208 0.323/0.214 0.327
Trallic
(0.450 0.278|0.488 0.327 00.42 0.316/0.620 0.336|0.593 0.332 0.625 0.383 0.622 0.392 0.389/0.621 0.396|0.610 0.376
particular, CARD achieves 14.2% significant improvement in SMAP task. Those facts imply CARD could generate meaningful representation on time series.
Table 2: Anomaly detection. Fl scores are reported. The best model is in boldface and the second best is underlined.
Models|CARR)
[ESooonee| Dlincur FEDformer |Stationary| Autoformer Informer
SMD
0.872
0.866
|0.800|
0.858
0.778
0.831
0,825
0.771
0.851
0.847
0.851
0.855
MSL
0.817
0.823
[0.16]
0.852
0,820
0.850
0.790
0.849
0.786
0.775
0.791
0.841
SMAP
0.857
0.695
[0.656]
0.715
0.674
0.695
0.692
0.693
0.708
0.711
0.711
0.699
SWaT
0.945
0.909
[0.875|
0,921
0.886
0.849
0.933
0.875
0.932
0.799
0.927
0.814
PSM
0.957
0.951
[0.933|
0.975
0.921
0.918
0.972
0.936
0.972
0.973
0.933
0.771
Avg
0.890
0.849
0.816]
0.864
0.816
0.829
0.842
0.825
0.849
0.821
0.843
0.789
5.3 BOOSTING EFFECT OF SIGNAL DECAY-BASED Loss FUNCTION
In this section, we present the boosting effect of our proposed signal decay-based loss function. In contrast to the widely used MSE loss function employed in previous training of long-term sequence forecasting models, our approach yields reduction in MSE ranging from 3% to 12% % across a spectrum of recent state-of-the-ard baseline models, including Transformer, CNN, and MLP architectures as shown in Table 3. Our proposed loss function specifically empowers FEDformer and Autoformer, two algorithms that heavily rely on frequency domain information. This aligns with our signal decay paradigm, which acknowledges that frequency information carries variance/noise across time horizons. Our novel loss function can be considered a preferred choice for this task, owing to its superior performance compared to the plain MSE loss function. More detailed discussions deferred to Section J in Appendix.
Table 3: Influence for signal decay-based loss function. The lookback length is set as 96. All models are evaluated on different predication lengths {96, 192, 336, 720}. The average results are reported, and the full table deferred to Table 19 in the Appendix. The model name with uses the robust loss proposed in this work. The better results are in boldface.
Models| CARD
CARD* MICN regre///C....ergggg. Metric [MSE MAE MSE MAE MSE MAE MIEE MAE |MSE MAEJMSS MAFIMSE MAE MSE MAF MMSE MAF MSE MAE
Timesiet |TimesNet* FFDDfommer FEDffommr") |Autoformer/ Bacankormmr/Auutfrrreeee Autoformer*
[TIDiormer/Auttooommee
ETTm @TTTm|0.3900 |0,390 0,399 0,383 0.383 0,392 0.414 (0,383
0.414]0.383
0.393 |0.400 4.6/0..339 0.395|0.44 0.452 0.413 0.415 (05888 0.528|0.523 0.475
ETTh (0.499 0.440 0.443 0.425 0.559 0.535 (0.527 0.499 |0.458 0.450j0.449 0.438jo.448 0.460 0.436 0.442 (0.49% 0.487|0.514 0.481
5.4 INFLUENCE OF INPUT SEQUENCE LENGTII
Previous research (Zeng et al., 2023; Wen et al., 2023b) has highlighted critical issue with the existing long-term forecasting transformers. They struggle to leverage extended input sequences, resulting in decline in performance as the input length increases. We assert that this is not an inherent drawback of transformers, and CARD demonstrates robustness in handling longer and noisier historical sequence inputs, evidenced by an 8.6% and 8.9% reduction in MSE achieved in
8

---

Published as a conference paper at ICLR 2024
Table 4: Influence of prolonging input sequence. The lookback length is set 16.199.336..20
Inpai Length|
192
336
512
720
[MSE MAE] MAE] MAE MSE MAB |MSE MAEIMSE MAE
ETTml [0.383 0.384|0.363 0.372/0.352 0.367 (4267)0402 0.420)0.349 0.368
ETThi
[0.442 0,429|0.429 0.425)0.415 0.422 (0.352 0.371 0.421
the ETTh1 and ETTml datasets, respectively, when input lengths were extended from 96 to 720, as shown in Table 4.
5.5 INFLUENCE OF TOKEN BLEND SIZE
In this section, we the effect of the token blend module by varying blend size. The results are summarized in Figure 4. When setting the blend size to 1, the token blend module reduces to the standard token mix method in Transformer literature and we observe test errors in both MSE/MAE increase. While using a larger blend size, the multi-scale information is utilized and the errors are reduced in turn. However, in some cases, further increasing the blend size may damage the performance. conjecture it is due to the nature of the dataset that only some scales of knowledge are useful for forecasting. A higher blend size may oversmooth that knowledge.
0.40 MSE Weather dataset
MAE in Weather dataset
MSE Electricity dataset
MAE in Electricity dataset
0.29
0.28
0.18
0.26
0.25
26
0.25
0.15
16
0.20
0.20
0.14
0.15
0.15
0.1
45
116
142
130
95
142
770
0.20
336
Figure 4: Experiments on token blend size. The blend size is varying in 1, 2, 4, 8, and 16.
5.6 OTHER EXPERIMENTS
We conduct a series of experiments, using both ablation and architecture variants, to evaluate each component in our proposed model. Our findings reveal that the channel branch made the greatest contribution to the reduction of MSE errors, as shown in Appendix Q.2. Furthermore, our experiments on sequential/parallel attention mixing design, detailed in Appendix Q.1, show that our model design is the preferred option. Visual aids and attention maps can be found in Appendix A and O, which effectively demonstrate our accurate predictions and utilization covariate information. Another noteworthy experiment, concerning the impact of training data size, is presented in Appendix R.2. This study revealed that using 70% of training samples can significantly improve performance for half datasets affected by distribution shifts. Besides, Appendix L presents error bar statisties table that demonstrates the robustness CARD.More forecasting experiments on M4 (Makridakis et al., 2018) other datasets are presented in Appendix H and I.
CONCLUSION AND FUTURE WORKS
In this paper, we present a novel Transformer model, CARD, for time series forecasting. CARD is a channel-dependent model that aligns information across different variables and hidden dimensions effectively. CARD improves traditional transformers by applying attention to both tokens and channels. The new design the attention mechanism helps explore local information within each token, making it more effective for time series forecasting. We also propose a token blend module to utilize the multi-scale information knowledge in time series. Furthermore, we introduce a robust loss function to alleviate the issue of overfitting noises, important issue in time series analysis. As demonstrated through various numerical benchmarks, our proposed model outperforms state-of-the. art models.

---

Published as a conference paper at ICLR 2024
REFERENCES
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
CDC. llness. URL ctttss://g....cc......... httpss//.....cc..//s///////////t html.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler-Cansee and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12880 2022.
Si-An Chen, Chun-Liang Li, Sercan O Arik, Nathanael Christian Yoder, and Tomas Pfister. TSMixer: An all-MLP architecture for time series forecast-ing. Transactions on Machine Learning Research, 2023.
Weidong Chen, Xiaofen Xing, Xiangmin Xu, Jianxin Pang, and Lan Du. Speechformer: A hierarchical efficient framework incorporating the characteristics of speech. arXiv preprint arX1v:2203.03812. 2022.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with fo-awareness. Advances in Neural Information Processing Systems, 15:1634411635, 2022.
Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprint arXiv:2304008424 2023.
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Mingyu Ding. Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pp. 74-92. Springer, 2022.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010. 21.22000..19999 2020.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022.
Lu Han, Han-Jia Ye, and De-Chuan Zhan. The capacity and robustness trade-off: Revisiting the chan- nel independent strategy for multivariate time series forecasting. arXiv preprint arXiv:2304.05226, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456 pmlr, 2015.
10

---

Published as a conference paper at ICLR 2024
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin- Yu Chen, Yux- uan Liang, Yuan-Fana Li, Shirui Pan, et al. Time-LLM: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728. 2023.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang- Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs], January 2017, arXiv: 1412.6980.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41 international ACM SIGIR conference research & development in information retrieval, pp. 95-104, 2018
Shiyang Li. Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019a.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, arXiv preprini arXiv: 10XXvv/997.002200
2019b.
Zhe Li, Zhongwen Rao, Lujia Pan, and Zenglin Xu. Mis-mixers: Multivariate time series forecasting via factorized temporal and channel mixing. arXiv preprint arXiv:2302.04501, 2023.
Yuxuan Liang, Yutong Xia, Songyu Ke. Yiwei Wang, Qingsong Wen, Junbo Zhang, Yu Zheng, and Roger Zimmermann. Airformer: Predicting nationwide air quality in china with transformers, In Proceedings of the AAAI Conference on Artificial Intellignnce, volume 37, pp. 14329 143377, 2023.
Bryan Lim, Sercan Ô Ank, Nicolas Loeff, and Tomas Plister. Temporal fusion transformers for time series forecasting. International Jourmal of Forecasting, 2021.
interpretable
Shizhan Liu, Hang Yu, Cong Liao. Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2022a.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Xuezhe Ma. Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In The Eleventh International Conference on Learning Representations, 2023.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4):802-800 808. 2018.
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. In the Eleventh International Conference on Learning Representations (ICLR), 2023.
Kin G Olivares, David Luo, Cristian Challu, Stefania La Vattiata, Max Mergenthaler, and Artur Dubrawski. Hint: Hierarchical mixture networks for coherent probabilistic forecasting. arXiv preprint arXiv:2305.07089 2023.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2024
PeMS. Traffic. URL htRp://///pem dot. www.ca.goo...........ooo
Huajie Qian, Qingsong Wen, Liang Sun, Jing Gu, Qiulin Niu, and Zhimin Tang. Robustscaler: Qws-aware autoscaling for complex workloads. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 2762-2775. IEEE, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.0453 04356, 2022.
Syama Sundar Rangapuram, Matthias Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Proceedings of the 32nd international conference neural information processing systems, pp. 7796-7805, 2018.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
Rajat Sen, Hsiang-Fu Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach high-dimensional time series forecasting. Advances in neural information processing systems, 32, 2019.
Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36(1):55-85 2020.
UCI.
URL https: /atee///rrrhccc aataaarchiiieeeeeeeeeeeeaaeeeeee ics .uc..iiss.....................ec www.uu.ww.....eee.......e edw/mmddataseee Eleetrricicyyooaai..ggrem2222221
Electricity.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, volume 30, 2017.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2022a.
Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. MICN:
Multi-scale local and global context modeling for long-term series forecasting, In The Eleventh International Conference on Learning Representations, 2023b.
Haomin Wen, Youfang Lin, Yutong Xia, Huaiyu Wan, Qingsong Wen, Roger Zimmermann, and Yuxuan Liang. DiffSTG: Probabilistic spatio-temporal graph forecasting with denoising diffusion models. In Proceedings of the stt ACM International Conference on Advances in Geographic Information Systems, pp. 1-12, 2023a.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Trans- formers in time series: survey, International Joint Conference on Artificial Intelligence(([AAI), 2023b.
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. arXiv preprint arxiv: 2017.
Wetterstation. Weather. URL hwtpt::t///........... /twp:///wwwww////// http://wwwwwwwj/eotwt//t www.-bg--aaaaa.. mpg. de/ de/wetter/.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential
smoothing transformers for time-series forecasting, arXiv preprint arXiv:2202. 01331, 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with correlattt for long-term series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), pp. 101-112, 2021.
12

---

Published as a conference paper at ICLR 2024
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference Learning Representations, 2023a.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023b.
Jiehui Xu, Jianmin Wang, Mingsheng Long, al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI Conference Artificial Intelligence, 2023.
Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light cumpling-oriented mlp structures. arXiv preprint arXiv:2207.00 00/ 186, 2022.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021.
Tian Zhou, Ziqing Ma, Xue Wang. Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, and Rong Jin. M: Frequency improved legendre memory model for long-term time series forecasting. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022a.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICMI. 2022), 2022b.
Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained Im. arXiv preprint arXiv:2302.111999 2023.
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. 2021.
Advances in Neural Information Processing Systems,
Zhaoyang Zhu, Weiqi Chen, Rui Xia, Tian Zhou, Peisong Niu, Bingqing Peng, Wenwei Wang, Hengbo Liu, Ziqing Ma, Xinyue Gu, al. Energy forecasting with robust, flexible, and explainable machine learning algorithms. Al Magazine, 44(4):377-393, 2023.
13

---

Published as a conference paper at ICLR 2024
A
VISUALIZATION
Gemundttunn Predution
-08
-10
1.4
-14-
-1.0
-1.
25
card
100
125
1.30
200
25
patchtst
100
150
175
Groundmrum
M
Groundthuth
-0.6
-0.6
-1.2
-12-
-16-
dlinear
crossformer
Figure 5: Sample prediction graph for ETTh1 long-term forecasting task
GroundYiuth
GrotndTTutth Predictiion
-3.0
-3.
-0.8
8.0
-1.0
-1.0
0
25
50
75
100
125
150
17%
200
25
50
100
125
150
375
200
card
patchtst
-0.6
-0.0
-0.7
-0.2
-0.9
-1.0
-1.0
25
50
1200
125
150
375
200
25
50
3000
325
150
375
200
dlinear
crossformer
Figure 6: Sample prediction graph for ETTml long-term forecasting task
14

---

Published as a conference paper at ICLR 2024
0,0425
M Mmra
0.0425
M/
0.3375
0.0075
0.0325
0.0023
0.0215
0.0075
25
100
125
150
1.75
200
25
100
125
159
175
200
card
patchtst
8.3425
Groundfruth Preductor
0.04
5,0400
0.02
0,2375
0.0
-0.01
6.3275
8.0250
Groundfuth Prediction
-0.03
25
75
200
125
150
135
200
dlinear
crossformer
Figure Sample prediction graph for Weather long-term forecasting task
Gruundhuuh Prediction
Groundfiruth
Prediction
20
40
60
1.00
120
140
20
40
50
60
100
140
card
patchtst
Groundtruth Prodiction
200
40
120
140
dlinear
crossformer
Figure 8: Sample prediction graph for M4 short-term forecasting task
15

---

Published as a conference paper at ICLR 2024
B CARD'S ARCHITECTURE AND KEY COMPONENT'S SOURCE CODE
Sample code of CARD
class CARD (nn Module):
def inii_ (self, config, Aazgs, :
super (1 init_l se f-..a-th___enn
se se.1.stride contiq.stride
seif.d_model config.d_mace
self.task config.task_nam
patihh num int (eentii....see
selttpptch__len//sell /al.t..sstttttttttrtt....
seif.patchhaum patch_nun
self. seif.N_pos__me p______eeee Pn. wtw.:/tttot...ccptptttc...eeee config.d +1=-2)
se f.total Stotal_token____ _ttokoennnbbeeee config..otal_.oeee_.uaee
self self-patch_nim
self.total_token
emboding layer ralated
self, it__iintt_pppoeeeeee self. s.s.f.npue_ddooe
Linear (self patch__ee,, contig.d moddell
nn.Dropout (config.
self.els - (Eorch (1, config. dee
nlp decode
self.N_out = Lineir Ioumtch www.llialll...............ee aodel ccontig.d_model config. pred_len)
dual attention encoder related
De....ttentions_.oeee...
an.KodaleList ([CARD (I0AR0_Atten:ien
for in range(
lonfig.e_layerss]1
se.1 intenttios_over__e
= nn. Mn.Moduletis st (CARD_Attent cwnettioooooooonoo ceerrtcceeeeecaeeeeee True) for 1 in leontig.e_layetee...
Acteeettiios Moduleliit Lineer (canfig.d cenfig.d_modell...
ModuleList [nn. (config, dropout) for in in
range (cenfig.e_ e______essss_]
sel S...AAttenioons oropput contig.e__ayers)]
s...AActentioos__ne. config.d met..dddmmiiiimmmmttt.et e_layers)
Ma.Melllllist ( ([an, Sequential (Transpose (1, 2), an. Bat.Bheemllld
config. momentum), Transposa 2)1 for i in range (config.
def (self, 2, vargs, vekwargs):
b 2.shape
irputs iomralization
Remea torch.mean mean dim (-1), keepdims True]
2_std - torch. (2, dim = (-1), keepdimm True) 1c-4)
(z z_mean)
tokenization
scube 2.unfolé idimenaion 1, aize-self, aize-gell..atchh.eee
step-self.
2 Lmmedd taken
(self. weief.._innut...... projectior (zcube) sel..__pss_emeeeeeeeeeee innpiu_i__ttt_tojettiie..
repeat (z cmb.em shape [3], [ e_bbbd shape [hapee[]] 11],
emmbed torch.cat [(ols_token, (_embed), dim -2)
dual attention eneoder inputs s_embec
h, inguts.shape
for sattttoosoovve_ammmmeea se sel f.Attent fors_mlp sel w...f..ftttteo 1 self.
drop, norm
AtteellfAttemttcnsooeseee wowpiiiss_oor_eerrreeeeeeeeeeeee self,
Attentions_nozm
output_] a_ cepputs.peruute. 10, 1.1, permute 1C, 2,133
output_2 {_2(putttt_
outputs drop palppalplene pesllleutput__tooeee
+inputs
outputs norm (ostputs. (botpptss.reehaaeeeee.. -1) reshape (o, e,ty
inputs outputs
nlp decode
g_out - Self,W_ouf
(2, 11)
denomrlaization z_out (2_sttdto-44) z_mean
return
16

---

Published as a conference paper at ICLR 2024
Sample code of CARD Smooth Attention
class CARD Attentiinn ian. :
daf liiiini_ (salf, config, over_channel Felse, vargs, veloargd):
super (1 11l.__nni____ se_f.over_chhannel over_chanael
self.n_heads con.i.eeddd.cci.i........eee config.n_ieads
t-merge_aize contig..errg_stto seif.e_in. config.enc_ia
attention zelated f. -n..==nn..nnsee sel f.saatrrrdrdduut
config.d model + 3,, biaseTruc)
se headddimm contig.d_model // contig.n_heads
seif.dropout_mipp Dropot (config..goog self.mip nn.Lineart config.d_mmedt configg._mmdell
f.normppost
bequertial ( (Transpose (1,2), m..aatchkormliiconrii....d mome
um
coatig.momertune
(ranspose(2,2))
seif.norm_ post2 nn, Sequential (Iranspose (1,2), m.Betchhoomlllconfiii... d_model, momentum
coonfig.momerte.
Transpasa (1,
self.norm_atta Sn. SequentiallTransppol (1, an.BatcetNormmldconnttt d_odel,,
ponfig.memeettun) Tramspcse
sel self.ff_l Sequentislllleeooooooo (tn...nni (canff (config.d_madal,, config.d_ff, biag=True), nn. CELU
Dropoot ccontig.droponn
nn, Linear (config. f_ff, cenfig.d_mode bies-True))
2- an. Sequestt iale (mz.nn... Linear (contig.d_medel, contig.d_it, blas-True), Gn. (),,
Dropout (c. iconfig.e_ff. config.d_model, bsag=Truee)
dynamic projection related
sei 4p_rank config.dp....................... dp__rank
sel f.dp_k Lineer (sal..hedddii self.dp_reak)
se 1..t.dp de_t.dp_v nn. bineie (selt. head_dn,,, selt.op_rank)
EMA related
ema size max ama macrix maxiconfig.enc
config. token number, config.dp_rank.
corch.2arcs wweeacaeeeaeaaeeeeeee (lema_sizen sizes, weatsiieeeassaaaeeee sma_sizet)
alpha config.alphe ema [00__aarixx00
for 1 in range 11,
for J in sange(1);
ema_matrix
ema_matrix(1-1] 13] pha)
[ma_matrix(1) alpha
sel self. ref..registez_beee (ral: return torch.einsum ga_
ema cmm__matrix,,pm ema_metrix)
def
:arc.ehh
[-2]1) daf dyramic_projjcctionnae tof
src_dp (arc)
src_dp F. F. (p dim -1)
src_dp torch. bahef, bnhoo__>__bheet" arre,
return sre_da def forward erc, verga, :rkwarge:::
K,K,V
B, nvara, rark///are..raaaeeetteee gkv
sre.shape
H, 3, self.n_heads, C // self.n_headd) .porute (3,
G, 1,4, 2, 5) 9r k, ==gkv if net gkv[0), gkv[l], gkv12]
setn_score_álong_
}1 self torch.einsum bnhed, bnhtd->onhet', selt.ema (g), selt.ema
0.5
sttn_along_tokk token self (F, se. score scopr_ssooorernnn___c alar__accn_gteeen, dimm-1)
output_ales outpor_along_tokes torch.einsum (nsumm'bnhst, bnhrd-2onhed', sen_allnn_taker, v)
else:
dynamic project V and K
v_dp, k_dp wtw.fdddnmmmmprajjeeeee
sel f.dp_") self www.dddnn_noojosseeee (k, self.
ap_k)
pii_soore_allng_toe
forch,einsum bnhed, bnhtd--bbhht',, self, (g), self.ema k_dp) self.hesd_din dim *. 0.5
sstn_along_toke atllfattn_ddooee
softmax aal.an_neeeeeee dimm-1) torch.eingum bnhaf, bnhrd->6ohed', attn_alang_takee v_6p)
attention over hidden dimenalons
attr_score_alont clod___elon_hiideen terch.einsum( ('bnume bahaf-Sbuhef" k, g.shape[-2) -0.5 attr str__slorg_hidde
celf.aatt_dropou: [F, safttaax hatprr__ccrr__longtttoeeeo dins-1;
aetput_along__u token blend
torot. ln bnhet,
attn_sloog_hiidee, vi
outputl rearrange (output_ (autput_along wow.onggtoee.eesaaaae
"bn (h11 a.2, h13), d -2, bn,
hil),c',
hll h.f.._hedd//eeeff.ie self.merge_size
h12 tutput_elong_tooen..hhl
rasshap (Banhaasa@enaass-11111 -1, dimee
autput2 rearrange cattt_alon__hdddee
"ba bba_th11 a___i_11______31______ (1_____ a
hil hel.:..eeds/7see...ee:
hi2 output
hll self.merge_sr
resshape (Ranvars, -1,, htw.eeddeeell....nm.......
post__oom
(outputl) output2 mol reshapa nvars, -1, self.n_hoads fww.aehe_ettt add & norm
outputl self.nora_port (outppp reshepp (B, nvars, -1, self.n_heads self.head dim)
srež self. [ot.ffoootttuttt==ee....... (output2) (output)
sre2
src reshaps (Bevaaa,, -1, self. helf.n_heads sell.head_dim
sre = nom..nom_sttn(see
reshreseppeeB
1, self heads self, head__eeddd
return src
17

---

Published as a conference paper at ICLR 2024
C DATASETS DETAILS FOR LONG TERM FORECASTING
Datasets of Long-term Forecasting ing SStommrr
Table 5 summarizes details of statistics of long-term forecast-
Table 5: Dataset details in long-term forecasting.
Dataset
Length
Dimension
Frequency
ETTml
69680
15 min
ETTm2
69680
7
15 min
ETTh1
17420
hour
ETTh2
17420
hour
Weather
52696
min
Electricity
26304
321
hour
Traffic
17544
862
hour
D MODEL CONFIGURATIONS FOR LONG TERM FORECASTING
For all experiments, we use reversible instance normalization (RevIN, Kim al., 2022) to handle data heterogeneity. As suggested in (Olivares et al., 2023) and (Salinas et al., 2020), other standardization methods are also useful when data enjoys certain patterns. We would like to defer the detailed analysis of them into future study. Moreover, the Adam optimizer (Kingma & Ba, 2017) with cosine learning rate decay after linear warm-up is used as training scheme. We train the proposed models with at most 8 NVIDIA Tesla V100 SXM2- XXM2-16-GBB 16-GB GPUs. For all experiments, we fixed the number of encoder blocks, head dimensions and dynamic projection dimensions being 2, 8, and 8, respectively. The training epoch is set 100. The default batch size is 128 and is adjusted due to GPU memory restriction. Other details of configurations are summarized in Table 6.
Table 6: Model configurations of CARD.
Dataset
patch stride model dim FFN dim dropout blend size learning rate warm unp batch size
ETTml
8
16
32
0.3
lc-4
128
ETTm2
16
8
16
32
0.3
2
lc-4
128
ETThl
16
8
16
32
0.3
2
1e.4
128
ETTh2
16
32
0.3
2
le-4
128
Weather
128
0.2
lc-4
128
Electricity
128
0.2
le.4
32
Traflic
128
256
0.2
16
le-4
20
24
E EXTENDED NUMERICAL RESULTS OF CARD IN LONG-TERM FORECASTING WITH 96 INPUT LENGTH
We the following recent popular models as baselines: FEDformer (Zhou et al., 2022b), ETS- former (Woo et al., 2022), FilM (Zhou et al., 2022a), LightTS (Zhang et al., 2022), MICN (Wang al., 2023b), TimesNet (Wu et al., 2023b), Dlinear (Zeng al., 2023), Crossformer (Zhang Yan, 2023), and PatchTST (Nie et al., 2023). We use the experimental settings in (Wu et al., 2023b) applying reversible instance normalization (RevIN, Kim et al., 2022) handle data heterogeneity and keeping the lookback length as 96 for fair comparisons. Each setting is repeated 10 times and average MSE/MAE results are reported.
In this section. we report the full results of long-term forecasting experiments in section 5.1. The MSE/MAE results are summarized in Table and standard errors are reported Table 8. CARD achieves 23/28 best performance in MSE and all the best results in MAE. It implies CARD can improve the baselines in a broad range of forecasting horizons. The standard deviation of CARD on the order of le-3, which indicates our proposed framework is very robust. More baselines such as autoformer (Xu et al., 2021), nonstationary transformer (Liu et al., 2022b), Pyraformer (Liu et al., 2022a), LogTrans (Li al., 2019b) and Informer (Zhou et al., 2021) can be found in Table and Table of (Wu et al., 2023b). CARD consistently outperforms those models in all forecasting horizons and omit them for brevity.
18

---

Published as a conference paper at ICLR 2024
Table 7: Long-term forecasting tasks. The lookback length is set as 96. All models are evaluated on 4 different prediction horizons {96, 192, 336, 720}. The best model is in boldface and the second best is underlined. For MICN, we report the better result between regre and MICN-mean.
Models)
CARD
PatchTST|
MICN
TimesNet (Crossoormer Dlinear
LighTTS
FIM
ETSformer FEDformer
Metric
MSE MAE MAE[MSE MSE MAE|
MSE MAE
MMS MAE MSE MAE MSE MAEJ MSE MAE
MSE MAE MSE MAE MSE MAB
96 0.316 0.344 0.378
0.316 0.364 30.338 0.375 0..50.3666 0,400 0.345 0.372 0.400 0.348 0.367 0.375 0.398 0.764 0.416
192
0.363 0.3700..37 0.393
0.363 0.390 0.371 0.387 0.396 0.414 0.380 0,489 0.407 0.387 0.385 0.408 0,410 0.426 0.441
336
0.392 0.390 0.402 0.413
0.408
0.426 0.410 0.411 0.439 0.443 0.413 0.413 0388 0.438 0.418 0.405 0.435 0.428 0.445 0.459
720 0.458
0.462 0.449
0.459
0.464 0.4778
0.4500..54
0.509 00.474 0453 (0.527 0.502 0.479 0.440 0.999 0.462 0.543 0.490
avg 0.383 0.384...395
0.408
0.387
0.411 (.400 0.406 0.435
0.417 0.403 0.435 0.437 0.408 0.399 0.429 0.425 0.448 0.452
96 0.169 0.248
0.176
0,258
1..
0.275 80 0.267 0.346
193 O. 2.2 209 0.308 183 26266 0.189 0.280 0.203 0.287
192 0.234 0.292 0.244
0.304
0.262 0.326 0.249 0.309 0.350 0.421 0.284 0.362 0. 0.382 0.247 0.305 0.253 0.319 0.269 0.328
336 0.294 0.339 0,304 0.342
0.305
0.353 0.321 0,351 0.474 0,505 0.369 0.427 042 O 466 0,309 0.243 0.314 0.357 0.325 0,366
720 0,390 0.388 0.408
0,403
0.389
0.407 0.797 0,403 1 1.347 0.812 0.554 0.522 585 407 0.398 0.4 O 0.413 0.421 0.415
avg
(0.272 0.317 0.283
0.527
0.2844
0.340 092 0.333 (0 0.521 00.350 401 0 409 436 O 287 0.328 0.292 0.342 0.305 0.349
96 00.383 0.391 0.426
0.426
00399
0.427 3.3 O. 402 0.339 0.417 3.6
4.0
432
388 0.1 0.494 0.479 0.376 0.419
192 00.435 0.420 0.469 0.452
0,430
0.453 0.436 0.429 0.449 0.452 00,437 0.432 0.2
0.443 0.439 0.538 0.504 0.420 0.448
336 0.479 0.442 0.506 0.473
0.440
0.460 0.491 0.469 0,510 0.489 0.481 0.459 0. 518 0 488 466 0.574 0.521 0.459 0.465
720 0..471 0.461 0.504 0.495
0.491
0.509 0.521 O. 0,500 0,594 0.567 0.519 0.5 516 0 547 0.533 0. 525 5. O. 0.562 0.535 0.506 0.507
avg
(0.442 0.429 0.455
0.444
0.440
0.462 0.458 0.450 0.686 0.481 0..456 45 0.479 061 0.456 0.452 0.510 0.440 0.460
96
(0.281 0.330
0.330/0.292
0.292
0.342
00.299
0.364 00.340 0.374 0.641 0.549
00.333 0.3 0.437 092 0.344 0.340 0.391 0.358 0.397
Fh
192
(0.363 0..88 0.387
0.400
(1.422
0.441 0.,002 0.414 0. 896 0.656 01.477 0.676 0. 520 O. 504 389 402 0.430 0.439 0.429 0.439
336
0,.411
0.426 0.434
0.447
0.474 0.452 0.452 0.936 0.690 0.599 0. 6.626 0.559 018 0.4 485 0.497 0.496 0.487
720
0,416
0,431
0.430
0,446
0.442
0.467 0.462 0,468 1,390 0.863 0.831 0.657 863 0.672 0.3 0.448 0,500 0.497 0.463 0.474
avg
0.368
0.390
0.384
0.406
(0.402
0.437
(0.414 0, 427 (0. 966 0.690 0.559 0.5 51515 0 602 0.343 0 384 O 406 0.439 0.452 0.437 0.449
96
0..50 0.18%
0.188/0..176 176
0.218
(0.
161 0.229
(0.1 72 0. 0.22000.166 1644 0,232 196 O. 25. 182 0.242
193 0.234 0.237 0.312 0.217 0.296
Weather
192
0.202 0.238 0.223 0.259
0,220 0.281 0.219 0.261 0.211 0.276 0..237 296 0. 227 0.287 0.236 0.269 0.237 0.213 0.276 0,336
336
0.260 0.282 00.277 0.297
(1,778 0.331 0.280 0.306 0.269 01327 08 335 0. 282 0.334 08288 0.304 0.29% 0.353 0.339 0.380
720
0.343 0.353 0.353 0.347
0.311 0.356 0.365 0.359 0.355 0,404 0.345 0.381 0.352 0. 0.358 0.350 0.352 0.388 0.403 0.428
avg
0.239 0.261 0.257
0.243
0.243 0.299 0.259 0.287 0.250 0.310 0.266 0.317 062 0.3 269 0.339 0.271 0.334 0.309 0,360
96 0.141 0.233 190
0,296
00 164 0.269
16 0.272 254 0.347
197 28 207 0.307 191 0.276 08187 0,304 0.193 0.308
Electricity
192 0.160 0.250 0.199 0.304
0.
177 0 0,285 844 0.289 0.261 0,353 0. 196 0.285 0.213 1.3 1.8 0.279 199 0.315 0.201 0.315
333
0.173
0.263 0.217 0.319
193
0.19% 0.300 0.364
209 301 2.30 0.333 0.217 0.301 0.212 323 0.214 0,329
720
0,197
0.284 0.258 0.352
0,212 0.321 0.220 0.320 0.303 0.388 0.245 0.333 O. 265 0.360 0.2 0.357 0.233 0.345 0.246 0.355
avg 0.168 0.25% 0.216
0.318
0. 187 0.295
9.1 0.295 0.273 0.363 0.212 3.300 2.2 0.329 0.223 0.303 0.208 0.323 0.2 0.327
96 0..199 0.269
0.462
0.315
10,519
0.309
(0.593
0.321 0.5558 0,320
0.650 O. 0. 0.391 0.649 0.391 0.607 0.392 0.587 0.366
Traffic
192
0.443
0.276 0.473
0.321
0.537
0.315 0.617 0.336 0.569 0.321 0.650 0.396 601 0.382 0 6.6 0.366 621 0.399 0.604 0.373
336
0.460
0.283 0.494
0,331
0.534
0.313 0.629 0.336 0,591 0.328 0.605 0 0,386 016 0,371 622 0.396 0.621 0.383
720
0.490
0.299 0.522
0.342
0.577
0.325 0.6400 O. 0.652 0.359 0.650 O. 0.658 407 0.692 0.427 0.622 0.396 0.626 0.382
av8 0.453 0.282 0.488 0.327
0.542 0.316 0.620 0.336 0.553 0.332 0.625 0.33 0.392 0.639 0.389 0.621 0.396 0.610 0.376
Table Standard error results of CARD in long-term forecasting with 96 input length. Each setting averaged over 10 random seeds.
Tasks
ETTml
ETTm2
ETThl
ETTh2
Weather |llccriccity|
Traffic
Metric
MSE MAE
MSE
MAE
MSE
MAE MSE
MAE
MSE
MAE|MSE MAE MAE
96
le-3
1o-3
1c-3
1c-3
lc-3
le-3
2c-3
1c-3
1c-3
le-3
2o-3 2c-3
2e-3 2--3
192
lc-3
16-3
26-3
26-3
1e:3
le-3
26-3
le-3
36-3
3e-3
3--3
3e-3
2e-3
20-3
336
1e-3
1o-3
3e-3
2e:3
2e-3
1o.3
2e-3
2e-3
4e-3
36-3
3e-3
4e:3
4e:3
3e.3
720
2e-3
1e-3
5e-3
20-3
Se-3
3e-3
4e-3
3e-3
6e-3
5e-3
5e-3
Se-3
6e-3
4e-3
Avg
1c-3
1o-3
3e-3
2o-3
26-3
2e-3
3e-3
2c-3
4o-3
3e-3
4o-3
4o-3
[3-3 3e-3
F COMPARISON TO EARLY BASELINES
In this section, report the comparison of CARD with early baselines including Nlinear, Linear, and Repret in Zeng et al. (2023). We use the experiment settings in subsection 5.1 and fix the input length as 96. The results are summarized in Table 9. Our model consistently outperforms those baselines.
G EXPERIMENTS ON ALL BENCHMARK DATASETS BY VARYING THE INPUT LENGTH TO ACHIEVE THE BEST RESULTS REPORTED IN BASELINE LITERATURE
We report the proposed model with 720 input length in Table 10. We follow the experimental settings used in (Nie et al., 2023). For each benchmark, we report the best results in the literature or conduct grid searches on input length to build strong baselines. In single-length experiments, CARD achieves
19

---

Published as a conference paper at ICLR 2024
Table 9: Comparision to early basclines in long-term forecasting tasks. All models are evaluated on 4 diffcrent predication lengths (96, 192, 336, 720}. The best model is in boldface and the second best is underlined.
Models]
ETTml
ETTm2
ETThi
ETTh2
Weather Electricity
Traffic
Metric
[MSE MSE MAE MSE MAE MSE MAE MSB MAE MSE MAE) MSE MAE
96
10.316 0.347/0, 0.24 0,391 0, 0,330 O. 150 O. 188 O. 141 (.1990 0.269
192
0.363 0.234 0.292 0,435 0.420 0.363 0.381 0.202 0.238 O. 0,160 0.259 0.443 0.276
336
0.392 0.390 0.339 0.479 0.442 0.411 0.418 0.260 0.282 0.173 0.263 (0.460 0.283
720
0.458
0.339 3.388 0.471 0.461 0.416 0.431 0.343 0.353 0. 197 0.284 (0 490 0.299
avg
0.277 0.317 0.442 0.429 0,368 0.390 039 0.353 0. 168 0.258 00.453 0.282 (03883
96
10.368
0.38550.1887 2. (15556 0,494 0.326 0,373 0.201 0.242 0. 216 0.300 (3 663 0.404
Niinear
192
0.446 0 405 0.413 0.415 0.596 0,518 14 422 O. 0.248 O. 0.277 0.217 0.303 0.615 0,382
336
0:439 0,426 0.312 0.348 0.621 0.531 0,453 0.453 3.300 0.314 O 231 0.318 0.623 0.384
720
0.500 060 0.313 4.4 0.636 0.554 0,459 0.467 0.373 O. 0.361 0.274 0.350 0.661 0.404
avg
(04429
0.419 0.333 0.602 0.525 0.413 0.429 08 O. 298 O 234 0.318 00.641 0.394
96
10.381
0.39%
1.717 0,592 0,516 0.433 06262 O. 203 O. 262 0.2 10 0.300 ( 658 0.406
Linenr
192
0..413 0.415 0. 3,305 379 0.602 0,529 0,570 0.534 242 299
0.301 00.607 0,380
336
0,449
0,433 0.400 O, 0,442 0.633 0.550 0.670 08585 O. 082 O 336 O 221 0.315 0.614 0.383
720
0.496
0.467 0.569 0.532 0.673 0.595 0.922 0. 0,0 O. 385 256 0.346 0.655 0,404
avg
0:443 00.42880.334 0.417 0.625 0.547 0.649 0.570 0.271 0.316 O. 224 0.316 (0.633 0.393
96
1.211
0.66550..666
0.328
1.295
0,713 4.4 0.422 0.259 0.254 1. 1.588 0.946 2. 723 1.079
Repeat
1.261 0.340 0 1.325 0.733 0,534 0.473 0.309 0.2 1.595
2.756 1.087
336
1.283 0. 707 0.412 0.410 1.323 0,744 0,591 0,508 0.377 0.338 1.677 0.961 2,791 1.095
720
1.319 0. 720 0.521 0,465 1.339
0.756 0.588 0.517 0.465 0.394| 1.647 0.975 2.811 1. 1.097
avg
1.269
0. 69.
0.385 0.394|
1.321
0.737
0.536
0.480
0.353 0.320
1.612 0.958
0.958 1.090
the best results in % cases in MSE metric and 86% cases in MAE metric. In terms of average performance, CARD reaches the best results in all seven datasets.
Table 10: Long-term forecasting tasks. All models are evaluated on 4 different predication lengths
(96, 192, 336, 720). The best model is in boldface and the sccond best is underlined.
Models|
CARD
PatchTST| | MICN
TimesNet [Crossformer] Dlinear
|
LighTTS |
FIMM |ETSformer FFDfoommr
Metric|
MSE MAE
MSEE MAEMMSE MAE MAE|MSE MAE [MSE MAE| MAE|MSE MAEJMSE MAEI MSE MAE
96 0.288 0.332
0.240
(...3422333110 0. 0.364 0.338 0.375 0.373
0.299 0.343
0,400 348 O. 0.367 0. 0.39 0),416
192
0.332 0.357
0.332
0.369 0. 0.371
0.337
0.372 0.411
0.355 0. 0.365
0.400
0.407 0,387 0,385 0,408 0.385 0.410 0.42 0.441
336 0.364 0.376
0,366
0,392
0,392 0,408
0,410 0.410
0.429
0.441
0.369
0.386 0.438
0.418 0.488 0.405 0.428 0.445 0.459
0.438
720 0.414 0.407
0,416
0 420
0. 0.47
0.450 0.531
0.425 0.421 0.527 0.502 0.479
0.440 0.544 0,490
avg
0.350
0.368
0.351
0.381
00.37
0.411 00400
0.406
0.439
p.362 0.379 0.435 0.437 0.408 0. 0.399 0.429 0.425 00.488 0.452
96 0.159
0.246
00. 165
0. 0.255
179 0. 0,275|0 187
0,267/0.254
0.348 66 0. 0.26000.209 O. 0,30800,166
0.348
0 0.255/0.118 189 0. 28000, 203 0,287
192
00.114 0.285
0,220
0.292
0,262 0.326 0.249 0.300 0.433 0.224 0.303 0.311
0.382
0.296 (1253 0.319 0. 062 00.228
336
0.266
0.319
0.274
0.329 0,305
0.353 0.321 0.351 0.511 0.527
0.281
0,442
0.466 0.333 0.314 0.357
0.35700322 0.325 0,366
720
0.379 0.390
0.362
0.385
0.3850.389
0,389
0. 0.407 0.497 0 0.403 0.689 0.397 0.421 0.675 0.587 0.37 0.398
0.413 0.421 0.423 0.415
avg
0.255
0.310
0.255
0.315 0.315
0.284 0.341 0.339
0.522 p.256 0.331
0.409
0.436 0.436 0.321 0 0.2292 O. 0.342 0..305 0.349
96 0.368
0.396
0.370 0. 0.399/0.338
0.427
0.402 0.419
0.399
0.382 0.401 0.499 0. 0,476 0.419
192
0.406 0.418
(1.413
(1.421 01.40 0 11453 (1.46 0.4 0.429 0.439
01456
00.455
0.475
0,462
01443 0.439 0.533 0.539
0.504 00.40 0.54400 01.448
336
0..15 0.424
0.422 0. 0.436 0.440
0. 0.491
0.469 0.461
0.439 0.443
0.518
0.521
0.488 0. 0.577 0.52] 0.459 0.465
720
0.416 0.448
0.447
0.499
0.491
0.521 0.509 0.524 0.472
0.544 0.533 0.525
0.519 0.535 0.506 0.507
avg
00.401
0.421
[0.413
0.431 0.440 0. 462 01458 0. 0.450 0.461
0.423
0,437
0.491
0,479 00.61
0,456 0,5 0.510)0.4400 0,460
96
00.662
0.327
00.274
0.3 0.336
0. 3,./0.3400 0. 7.770 0.589
(0.289
0.353
0,397 0. 03.43300.29 0. 0,344 0 | 0.397
192 0.332
0.369
0.339
0.379 0.422 0.441 0.402 0.414 0.848 0.657 0.383 0.418 0,520 0. 5050 0.389 0 0.402 0.430 0 0.439 0.429 0.439
336
0.326
0.378
0.329
(1. 38D 0.447
0.474 0.452 0.452 0.859
0.674
0.448
0.465
0.626
0.418 0.430 0.497 0.496 0,.487
0.4970.496
720
0.373
0.419
0.379
0 4.422
0.442
0.467 0.462
0.468 1.221 0.825
0.60S
0.551 0.863
0.672 0 433 0.448 0,500 0,500
0.497 0.463 0.474
avg 0..311
0.373
0.330
0 0.379
0.402
0.437 0.414 0.427 0.
0.446
0.259
0.2259
0.321
0. 0.002
0.543 0 0.3 O. 0.40 0.449 0. 0.452 0.447 0.449
96 0.145 0.186
14.
191
0.229
0.145 0.211
|0.
0.237
0. 182 0.242
199 0.234 0.233 0.312 0.296
Weather
192 0.187 0.227
0.194
0.241
0.220
0.28 0.219
0.261 0.190
0.259
0.220 0.282 0.227 0.28 0.228 0,288 0.237 0.213 0.276 0.276 0.336
336
0.238
0.258
0..25
0. 0.282
0.278 0.331 0.280 0. 306 0.259 0.326
0.265 0.319
0. 0.28 0.334 0. 26267 0.323 (). 292 O. 0..33.333 0.380
720
0.308
0.321
0.314
0.334
0.311 0.356
0.359
0.382
0.323
0.362
0.352 0. 0.386 0.358 0. 0.350 O. 0.388 0.428
avg 0.219
0.248
0.226
264
0.244
0.299
0.232 0.295 p.240 0.300 0.261
0.26
0.299 0.334 0.330 0.360
96
00.199 0.223
00.129
0.
164
0.269
168 0.272 186 0.281
10.153 0.237
207 0,307
0. 0. 00 187 0,304|0.19 0.308
Electricity
192 0.154
0.245
0.147
0.
0.285
0. 084 0,289 0. 0.300 0.152 0.249
0.213
0.3 1. 0.279 1.99 0.315 0.201 0.315
336
0.161
0.257
0.163
0.259
0.193 0.304
0. 198 0.323 0.369
0,169 0. 06267
0.230 0.333 1888 0. 0.283 0.212 0.329 0.3290..214 0.214 0.329
720
0.185
0.278
1.1
(1.12 11.21 (1.20 0. 0.444 0.423
0.233 0.344
0.265
0,360 0,236 0,332 03233 01345 0,246 0.355
avg
0.157
0.251
0.159
0.253 087 0.295
0. 192 0. 0. 0.343
0.177 0.224
0. 2.2 0.
0. 0.290 0. 0.32330.214 0.327
96
00.34
0.229
00,360
0.321 00.511 0.292
|0.410
0.282
0.615 0.391
(1.466 0. (1667 O. 01.99200.587 0.366
Traffic
192 0.367
0.243
0.379
0 0.537
0.336 0.617 0.617
0.311
0.423 0.287 0.601 0.382 0.408 0.288 0.621 0. 0.604 0.373
0.3820.008
336
0.388
0.254
0.392
0. 264
0.534
0.313 0.6 0. 0.336 530 0.300
0.436 0.296 0.613 0.386 0.425 0.298 0.692 0. 0.396 0.383
720
0.427
0.276
0.432
0.286 0.577
0.325
0.350
0.313
0.466
0.315 0.658
0.407 0.520 0.353 0.622 0. 0.396 0.382
avg
0.381
0.251
0.391
0 0.264
0.542 0.2.0016666.0006363
0.304
0.434 0.295
0.622
0.392
0. 0.308
0.621 0. 0.3960.6100.376
20

---

Published as a conference paper at ICLR 2024
H M4 SHORT TERM FORECASTING
We also conduct experiments short forecasting M4 tasks. M4 dataset (Makridakis et al., 2018) consists 100k time series. It covers time sequence data in various domains, including business, financial, and economy, and the sampling frequencies range from hourly to yearly. We follow the test setting suggested in (Wu et al., 2023b). Each experiment is repeated 10 times and average Symmetric Mean Absolute Percentage Error (SMAPE), Mean Absolute Scaled Error (MASE), and Overall Weighted Average (OWA) are reported. We benchmark our model with N-BEATS (Oreshkin et al., 2020), N-HiTS (Challu et al., 2022), Informer (Zhou et al., 2021), Autoformer (Wu et al., 2021) and 7 baselines in long-term forecasting. Details for datasets and training configurations can be found in Table 11 and Table 12 respectively.
The results are summarized in Table 13. in all tasks. Specifically, we outperform the state-of-the-es MLP-based method N-BEATS (Oreshkin et al., 2020) by 1.8 % in SMAPE reduction. We also outperform the best Transformer based method PStchTST (Nie et al., 2023) and the best CNN-based method TimesNet (Wu et al., 2023b) by 1.5% and 2.2% in SMAPE reductions respectively. Since the M4 dataset only contains univariate time series, the attention to channels in our model plays a very limited role here. Thus good numerical performance indicates CARD's design with attention to hidden dimensions and token blend also effective univariate time series scenarios and can significantly boost forecasting performance.
Our proposed model consistently outperforms benchmarks
The standard errors are reported in Table 14. Since the SAMPE score is not normalized, we observe the absolute value is on the order of le-2 while the MASE and OWA remain on the order of le-3 which is the same in long-term forecasting experiments. After normalizing SAMPE with the corresponding mean value, the standard error of SMAPE will also reduce to the order of 1e-3.
Table 11: Datasets and mapping details of M4 dataset.
Dataset
Length
Horizon
M4 Yearly
23000
M4 Quarterly
24000
8
M4 Monthly
48000
18
M4 Weekly
359
13
M4 Daily
4227
14
Hourly
414
Table 12: Model configurations for M4 experiment.
Dataset
patch stride model dim FFN dim dropout|blend size llaarnng rate/warm-up batch size
M4 Hourly
16
128
512
Sc-4
128
M4 Weekly
16
128
512
0.1
2
5c-4
0
128
M4 Daily
16
128
512
0.1
2
5e-4
0
128
M4 Monthly
16
128
512
0.1
2
5e-4
128
Quarterly
128
512
0.1
5e-4
128
Yearly
3
512
128
Table Short-term Forecasting tasks on dataset. The average results of ten repeats are reported. The best model is in boldface and the second best is underlined.
Models
CCARD
PatchTST MMCN TimesNet N. IBIST N. BEATS B LightTS Dlinear ['E:Doormer Autoformer Informer
13.25
13.258
[4.935
13.387
13.418
13.436
18.009
14.247
16.965
13:728
13.974
14.727
Yearly
MASE
1.922
2.985
3.523
2.996
3.045
3.043
4.487
3.109
4.283
3.048
3.134
3.418
OWA
0.781
0.900
0.786
0.794
0.827
1.058
0.881
SMAPE
9.958
10.179
11.452
10.100
10.202
10.124
13.376
11.364
12.145
10.792
11,338
11.360
Quarterly
MASE
1.182
L,699
OWA
0.876
0.904
1.026
0.890
0.899
0.886
1.302
1.000
1.106
0.958
1.012
1.027
12.414
12.641
13.7733
12.670
12.791
666
14.588
14,014
13,514
14.260
13.958
14.062
Monthly
MASE
0.907
0.930
1.076
0.933
0.969
0.937
1.368
1.053
1.037
1.102
1.103
1.141
OWA
0.856
0.867
0.983
0.878
0.899
0.880
1.149
0.981
0.956
1.012
1,002
1.024
SMAPE
4.522
4.851
6.716
4.891
5.061
4.925
7.267
15.880
6.709
4.954
5.458
24.460
Others
MASF
3,021
1.238
4.717
3.302
3,2116
3.391
5.240
11.434
4.953
3.264
3,865
20.960
SMAPE
11.614
11.807
130
11.829
11.927
11.851
14.718
13.252
13.639
12.840
12.909
14.086
Avg
1,585
1.599
2.408
2.718
OWA
0.832
0.834
0.980
0.851
0.861
0.855
1.172
1.051
1,051
0.918
0.939
1,230
21

---

Published as a conference paper at ICLR 2024
Table 14: Standard results of CARD in M4 short-term forecasting. The results normalized with the corresponding mean value are reported parentheses. Each setting averaged over random seeds.
Metric
Yearly
Quarterly
Munthly
Other
Average
SAMPE 0.022 (0.001) 0.008 (0.001) 0.032 (0. (0,002) 0.024 (0.005) 0.018 (0.002)
MASE 0.007 (0.003) 0.003 (0.002) 0.003 (0.003) 0.026 (0.008) 0.003 (0.002)
OWA 0.003 (0.002) 0.001 (0.001) 0.032 (0.037) 0.004 (0.004) 0.001 (0.001)
I OTHER FORECASTING TASKS
In this section, we report the results of Illness and Exchange tasks. The Illness (CDC) and Exchange (Lai et al., 2018) contains the weekly data influenza- illness from Jan-2002 to Jun-2020 and the daily exchange rates of cight foreign countries including Australia, British, Canada. Switzerland, China, Japan, New Zealand, and Singapore ranging from 1990 2016 respectively. We follow the test setting suggested in (Wu et al., 2023b). Each experiment is repeated 10 times and MSE and MAE are reported. We benchmark model with the baselines in long-tern forecasting. Details for datasets and training configurations can be found in Table 15 and Table 16 respectively.
The results summarized in Table 17. Our proposed model outperforms benchmarks in 4/8 cases in MSE and 6/8 cases in MAE. The standard errors are reported Table 18.
Table 15: Datasets and mapping details of Illness and Exchange datasets.
Dataset
Length
Horizon
Frequency
Illness 966
7
Weekly
Exchange
7588
8
Daily
Table 16: Model configurations for Illness and Exchange tasks.
Dutaset
|atcch|stride model dim IFN dim dropout blend size learning rate watm--u batch size (epcchs
Illness
36
16
32
0.3
2.5e-3
128
100
Exchange
16
16
32
0.3
le-4
64
10
J EXTENDED RESULTS OF KIGNAL-BASED LOSS FUNCTION
The full results of experiments section 5.3 are reported in Table 19 and Table 20. Moreover, we also conduct an experiment on switching to the decay function other than the two forms considered in section 4. The results are summarized in Table 21. Table 21, we consider the following decay function: S(L) 1-1/ f(1) = 1-1/3, [(t) 11 (() 1-2 and (()) - In the ETTml task, find that the decay function from f(t) 1-1 and f(t) t-1 gives similar MSE performance and slightly worse (by 0.001) MAE performane on average compared to the squared root decay. In the ETTh1 task, f(t = 1/4 f(t = 1-33 and ft) = -- work the same good as squared root decay. In practice, we believe the function that is not "decaying" faster than [() 1-1 might be the candidate choice when no further information/asumptioon on datasets could be obtained. 1/4 and f(t) = 11133), vert slight performance
For the slow decaying function (c.g., f(t) = 114 improvement is observed in individual tasks when it is getting close the squared root decay. It implies that the proposed loss robustness for slow decaying function.
We also provide illustration example to show the rationality of the proposed signal-based loss function. Let's consider ID autoregressive model Ø(+1 girue + 6s with A 1), gtrue (0, 1) and [ra] 1. And we want use 2, to forecast x++1 and x4+2. The plain loss function would be as follows:
min 8
11 B- z 112 ||0.02 12-2113)
(13)
22

---

Published as a conference paper at ICLR 2024
Table 17: Exchange and Illness tasks, All models are evaluated on 4 different predication lengths
{96, 192, 336, 720). The best model is in boldface and the second best is underlined.
Models
CARD | PatchTST
MICN
TimesNet [Crossformerj Dlinear
LighTTS
FIM
ETSfomer PEDformer
Metric| MSE MAE|MSE MAE MSE MAE MSE MAEJMSE MAE| MSE MAE MSE MAE MSE MAE MSE MAB MAB
96 0.084 0.202
0 00.88 0.205 102 0.235 007 0.234 0. 0.367 0.086 0.218 0.1 0.262 0. 1.14 0.282 0,085 0.204 148 0.278
Exchange
192 0.179 0.298 0.2989.176 9.176 0.299 0.172 0.316 0.226 0.344
0.509
0.176 0.315 0.215 0.359 0.241 0.364 0.428 0.277 0,380
336 0.333 0.418 0.300 0.397 0.272 0.407 0.367 0.448 1,267 0.883
0.313 0.427 0.377 0.466 0. 0.425 0.488 0.348 0.428
0.460 0,500
720 0.851 0.691 0.901 0.713 0.714 0.658 0.964 0.746 1.767 1.068
0.839 0.695 0.831 0.699 0.993 0.747 1.025 0.774 1.195 0.841
avg
0.362 0,402 0.366 0.404 0.315 0.404 0,416 0.44346.44444....0 0.940 0.707
0.354
0.414 0.447 0,473 0.410 0.442 0,500
96
2.043 0.863 22.334 0.891
33,457
1.288
2.317 0,444
1.237
2.39% 1 1.04088.313 2.144 3.589 1420 1. 1.020 1.260
Illeess
192
2.300 0.917 2.316 0.932
2.711
1.123
1.972 0.920
0.920
3.762
2.175
2.646 1.088 6.631 1 1,902 4.009 1.330 2.615 1.007 2.679 1,080
336
1.899
0.846 2 153 0.900
2.775
1.145 2.359
0.972
3.853
1.307
2.614
7.299 1.982 3.785 1.492 2.359 0.972 2.622 1.078
720
1.993
0.876 2.029 0.910
3.024
1.197 2.487 L.16 4.035
1.344
2. 804
1.14 7 28283 1. 1.985 3.722 1. 1.373 2.487 1.016 2.857 1.157
avg
2.058 0.876
2.133
0.908
2. 992
1.173
2.139
0.931
3.778
1,516
2.616
1.090 7.382 2.003 3.776
2.497
2.004
1.144
Table 18: Standard error results of CARD in Illness and Exchange tasks, Each setting is averaged over 10 random seeds.
Tasks
Illness
Exchange
Metrie
MSE MAE MSE MAE
96
0.172 0.029
4c-4 le-3
192
0.173 0.026
7e-3 7e:3
336
0.059
0.019
le-2 7e-3
720
0.018
le-2 Se.3
Avg. [0.119 0.023 7e-3 Se-3
Table 19: Influence for signal decay-based loss function. The lookback length is set as 96. All models are evaluated on 4 different predication lengths {96, 192, 336, 7203. The model name with # uses the robust loss proposed in this work. The better results are in boldface.
Models| CARD
| CARD* MICN Megr TimesNet |TimmeNett* |FED)ormee(/EEDtooeeee (Auto//oooooooootttaoottooao
Metric
MMS MAE| MAE MSB MAEIMSE MAE |MSE MAE[MSE MAE| MAE
MAE |SS MAE[MSE
MAB
96
00.229 0,36 0,347 0.316 0.362 00,33 0.350
0.3750.35..3.
0.356 0,419|0.344 0.380
|0.505 0.475
0.442
192
0,368 0.385 0.363 0.370 0,363 0.390
0.359 0.372
0.374 0.387 0.377
0.426 0.44] 0.390 0.404 0.553 0.537 0.540
0.477
336
0.400 0.405 0.393 0,390 0.408 0.426
00,39 0.399
0.410
0.41 0.401
0.400
0.45%
0..433
0.621 0.537 0.594 0.505
720
0.468 0.444 0.458 0.426 0.481 0.476
0.4440.458
0.466
0.451
0.478
0.450 0.437 0.543 0.490
0.461 0.671 0.56 0.547 0.476
avg
0.391 0.388 0.384 0.392 0.414
00.38
0.393
0.400009
0.395 0.452
0.415 (0.588 0.528 0.475
96
10.387 0. 0,399 0.391 0.421 0.431
[0,403
0.412
0.402
0,400)0.376
0.419/0.371
0.400
(0.449 0.459|0.455 0.445
192
0.438 0.431 0.435 0.420 0.474 0.487
0,471
0.451
0.436
0.429
0.425
0.420 0.419 0.432 0.500 0,544 0.493
336
0.486 0.454 0.479 0.461 0.569 0.551
0.513 0.496
0.491 0.469
0.45
0.461
0.455
0.521 0.495 0.491
720
0.480 0.472
0.471 0.429 0.770 0.672
0.720
0.636
0.521
0.500
0.477
0.506 0.507 0.491 0.482 00551
0.514 0.495
avg
0.448 0.439 0.442 0.425 0.559 0.535
0.527
0.499
0.458
0.450 0.438 0.440 0.46 0.442
0.496 0.487
0.514 0.481
Table 20; Extended results on the signal decay-baser loss function. The model name with uses the robust loss proposed in this work. The better results are in boldface.
LighTTS
LighTTS*
FIM
FIlM
ETSffrmmrr |ETSformer") Stationary Stationary"
Metric MMSB MAE MAE| MAE
MSEE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
96
00366 0.400 00355
0.364
00.374
0.332 0.360 0,348 0,367 0.335 0.359j0.375 0. 0.382 0.391 (0.366 0.398 0.345 0.364
192
0.396 0.414
0,381
0.372
0.400 0.407 0.365 0,385 0,387 0,385 0.371 0.372 0.410
0.404
0.459 0.444 0.423 0.409
336
0.439 0.443
0.431
0.415
0.438
0.438 0.414 0.408 0.418 0.405 0.399 0.445 0.428 0.442
0.431
(0.495
0.464 0.430 0.415
720
0.540
0.509
0,512
0.472
0.527
0.502 0.497 0.463 0.479 0.440 0.485 0.432 0.462 0.472 0.439 0,5585 0.516 0.561 0.500
avg
0.435
0.417
0.419
0.406
0.435
0.437 0.402 0.404 0.408 0.399 0.398 0.39940.429 0.425 0.421
0.416
(0.481 0.456
0.422
96
10.391 0.417
10,888
0.397
00.424
0.432
0.412
0.418 0,388 0.401 0.387 0.389|0..494 0.479|0.499 0.457
(0.513 0.419 0,509 0.394
192
0.499 0.452
0.489
0.430
00.475
0.462 0.459 0.445 0. 0.443 0.439 0,448 0.43000,538 0. 500 0.453 0.491 (0.534 0.504 0.449 0.429
336
0.510 0.489 0.493
0.472
0.518
0.521 0.499 0.502 0.488 0.466 0.490 0.458 0.521 0.589 0.527 00,588 0.535 0.560 0.501
720
0.594 0.567
0.578
0,533
0.547
0,533 0.505 0.504 0.525 0.519 0,527 0.516 0. 0.535 0.497 0.504 0.643 0.616 0.682 0.643
avg
0.499
0.481
0.487
0,458
0..491
0.479
0.446
0.467 0.461 0.456 0.463 0,448
0,500
0.495
00,570
0.537
0,550 0.492
Via standard generalization analysis procedures and Hoeffiding's Inequality, we have with probability
1 - 6,
5 T log (1/0)
|3 glrue|
(14)
x7
23

---

Published as a conference paper at ICLR 2024
Table 21: Influences the delay function the loss function. The best results are in boldface and the second best is underlined.
Function
((0)
(ste)
< 0.20|
((())
0.30
((0)
((C))
f(0) =
Metric
MSE
MAF] MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAEIMSE
MSE MAB
96
0.329 0. 0.319
0.349
0,318
0,149
0.316
0.347
0.317 0.347
0
0,356 O. 345 0.363
192
0.368 0 0.366
0.370
0.361
0.370
0.363
0.570
0.363 0.369
0.379 0.377 0. 0.430 0.416
336
0,400
0:405 0.393
0.391
0.393
0.390
0,393
0.390
0.396
0,391 0:414
0,402 0.605 0.505
720
0.468
0.444 0.459
0.427
0.459
0.427
0.458
0.426
0.466
0.429 0.491
0.449 0.760 0.578
avg
0,391
0,333
0.384
0,383
0.384
0.383 0.383
0.384 0.386
00:05
0.396 0.535 0,491
96
0.387 0. 0.322
0.391
0..82
0.390
0.382
0.390
[
0.391
(0.387
0.396 0.410 0.413
192
0,438
0.431 0.457
0.421
0,436
0.420
0.435
0,420
0.436
0,421 0,439
0,426 0.559
336
0.486 0.454 0.478
0.443
0.478
0.442
0.478
0.442
0.479
0.443
0,485 0.453 0.712 0.566
720
0,480
0.472 0.472
0.462
0.472
0.462
0.471
0.462
0.470
0.460
0.551 0.508 0.786 0.613
avg 0.448
0
0.439
0.442
0.429
0.442
0.429
0.442
0.429
00.42
0.429
0 0.396
0.617 0.522
In this case, our proposed loss becomes:
min
Ell - x + || - -0442122
(15)
Follows the same analysis procedures, have with probability 1-8
2T (1g/0)
| giraej
(16)
E1 2
Here the constant is improved from 2 to 2. which implies the new loss may yield better convergence upper bound.
K EXTENDED RESULTS OF ANOMALY DETECTION
The full results of the anomaly detection experiment in the section 5.2 reported in Table 22, For each setting, we repeat replicates.
Table 22: Full results for the anomaly detection task. The P, R and Fl represent the precision, recall and Fl-score respectively.
Datasets
SMD
MSL
SMAP
SWaT
PSM
Avg F1
Metrics
R
F1
P
R
F1
P
R
F1
R
F1
P
R
F1
CARD
0.883 0.861 0.872 0.896 0.750 0.817
92.93 0.794
0.857
0.928 0.962
0.945
0.982
0.933 0.957
0.890
PatchTST
0.802 0.942 0.866 0.898 0.760
0.823
89.97 0.566
0.695
0.919
0.899 0.909
0.992
0.913 0.951
0.849
MICN
0.765 0.838 0.780 0.892 0.752 0.816 0.895 0.518 0.656
0.913
0.841 0.875
0.987 0.885 0.933
0.816
TimesNet 0.887 0.831 0.858 0.839 0,864 0.852 0.925 0,583 0.715
0.883
0.962 0.921
0.982 0.968 0.975
0.864
Crusdormer 0.722 0.844 0.778 0.907 0.749 0.820 0.895 0.541 0.674
0.919 0.856 0.886 0.971 0.876 0.921
0.816
ETSformer 0.874 0.792 0.831 0.851 0.849 0.850 0.923 0.558 0.695
0.900 0.804 0.849
0.99.3 0.853 0.918
0.829
LighTTS 0.871 0.784 0.825 0.824 0.758 0.790 0.926 0.553 0.692
0.920 0.947 0.933 0.984 0.960 0.972
0.842
DLincar 0.836 0,715 0.771 0.843 0.854 0.849 0.923 0.554 0.693
0.809 0.953 0.875 0.983 0.893 0.936
0.825
FEDforner
0,880 0,824 0.851 0.771 0.801 0.786 0.905 0.581 0.708
0,902
0.964 0.932 0.973
0.972 0.972
0.850
Stationary 0.883 0.812 0.846 0.686 0,891 0.775 0.894 0.590 0.711
0.680
0.968 0.799 0.978
0.968 0.973
0.821
Autoformer
0.824 0.851 0.773 0.809 0.791 0.904 0.586
0.899
0.958 0.927 0.991
0.882 0.933
0.843
Informer
01866 0.773 0.817 0.818 0.865
0.841
0.901 0.571
0.699
0.703
0.968
0.814
0.643
0.963 0.771
0.788
L ROBUSTNESS EXPERIMENTS
L.I INFLUENCE OF DIFFERENT INPUT LENGTII
In this section, we report the robustness test when varying input length. We conduct experiments on ETThl, ETTm1, Weather, and M4 datasets and repeat each setting with 10 random seeds. The robust experiment results are summarized in Figure 9-Figure 17. In general, we observe the longer input length may yield better performance, and the variance is also enlarged slightly.
24

---

Published as a conference paper at ICLR 2024
200 we
330 M11
4790
0.00
3311
440
Figure 9: ETTml experiments with different input lengths.
Figure 10: ETTml experiments with different input lengths.
Acoe
CCR
CCa
340
Figure 11: Weather experiments with different input lengths.
25

---

Published as a conference paper at ICLR 2024
ETNO mM
2017
200
CRO
Xxe
C215-
X31-
Figure 12: The average results of ETTml, ETThl and Weather experiments with different input lengths.
11.55
Figure 13: M4 Yearly experiments with different input lengths. represents the ratio between input length and forecasting length.
The axis "input length ratio"
econrieerts
Figure 14: M4 Quarterly experiments with different input lengths. The x axis "input length ratio" represents the ratio between input length and forecasting length.
Fatultss of Manitiyy repasteetss
Figure 15: M4 Monthly experiments with different input lengths. The x axis "input length ratio" represents the ratio between input length and forecasting length.
26

---

Published as a conference paper at ICLR 2024
Awerage henurs De by, Weedy and Hourty rxpertmenss
Figure 16: M4 average results of Daily, Weekly and Hourly experiments with different input lengths. The x axis "input length ratio" represents the ratio between input length and forecasting length.
Resutt
Figure 17: Average results of all M4 experiments with different input lengths. The x axis "input length ratio" represents the ratio between input length and forecasting length.
L.1.1 INFLUENCE OF DIFFERENT MODEL. SIZES.
In this section, we report the robustness test when varying model size. We conduct experiments on ETThl, ETTml, Weather, and M4 datasETSformer The model size (hidden dimension in attention) changes from to 128 and the MLP layer dimension set to be times the model size. We repeat each setting with 10 random seeds. The robust experiment results Figure 26. terms of average performance (e.g., Figure 21 and Figure 26), the larger model size gives better results. For each individual task, we observe that the model with large hidden dimension (e.g., 128) tends to overfit in low complexity tasks like ETTh1. For the high complexity task, the larger model size enables bigger learning capacity and gives better performance.
summarized in Figure 18-
ETINAEE
ETTYLAE WH
CTTNL X00
CTTN Docm
6-00-
51350
Figure 18: ETTml experiments with different model sizes.
27

---

Published as a conference paper at ICLR 2024
CTAV mEM
CTTC TXU mese
CTINO T30 mOR
ETNV T7x
3M
Figure 19: ETTh1 experiments with different model sizes.
Figure 20: Weather experiments with difffernnt model sizes.
PTTV
Fifee:
camo
UTRL
STTIO
Figure 21: The average results of ETTml, ETThl and Weather experiments with different model sizes.
28

---

Published as a conference paper at ICLR 2024
Remuta merry ecowtrans.
G.a
Figure 22: M4 Yearly experiments with different model sizes.
Cou
12.55
L150
Figure 23: M4 Quarterly experiments with different model sizes.
Sesults Monrthy eigedimeets
Cood
8:05
$3.0
Figure 24: M4 Monthly experiments with different model size.
Average Daity Moartyy
514
Xoan
Figure 25: M4 average results of Daily, Weekly and Hourly experiments with different model sizes.
Euto
1.04
11.00
31.0
Figure 26: Average results of all M4 experiments with different model sizes.
L.2 INFLUENCE OF DIFFERENT LEARNING RATES AND SCHEDULERS
In this section, we report the robustness test when varying learning rates and schedulers. We conduct experiments ETTh1, ETTm], Weather, and M4 datasets and repeat each setting with 10 random
29

---

Published as a conference paper at ICLR 2024
seeds. We consider the fixed learning rate and cosine learning rate scheduler with the initial learning rate from 1c-3 to le-5. The results are summarized Figure 27-Figure 33. We observe slight improvements when changing the fixed learning rate to the cosine learning rate decaying. For the relatively large learning rate, the variance of testing MSE/MAE increases and for the small enough learning rate, the model tends underfit for the given training epochs. In practice, results suggest the learning should be set on the order of le-4.
TM
6.0
ne
720 ma
130
Figure 27: ETTh1 experiments with different learning rates and schedulers.
310
100
720
Figure 28: ETTml experiments with different learning rates and schedulers.
30

---

Published as a conference paper at ICLR 2024
200
PM
100
Cams
5219
Figure 29: Weather experiments with different learning rates and schedulers.
ETTC
VRENG
2047
340
200
2344
3m
2014
300
300
300
300
Figure 30: The average results of ETTm1, ETThl and Weather experiments with different learning rates and schedulers.
Figure 31: M4 Yearly experiments with different learning schemes.
31

---

Published as a conference paper at ICLR 2024
Readla of Quersety experments
Kit
Figure 32: M4 Quarterly experiments with different learning schemes.
66
Figure 33; M4 Monthly experiments with different learning schemes.
Aerrup Poouao Duiy, Weedly
485
Figure 34: M4 average results of Daily, Weekly and Hourly experiments with different learning schemes.
Figure 35: Average results of all M4 experiments with different learning schemes.
M TRAINING SPEED FOR DIFFERENT INPUT SEQUENCE LENGTH AND MODEL SIZE
In this section, we report the training speed differences when varying input sequence lengths and model sizes. For all experiments, we fix the batch size being 2 and use the average step time cost of 50 training epochs as the speed measure. We use the time cost of the experimental setting with 96 input length and 16 model dimensions the base and report the ratio of the time increasing when using the longer input sequence and/or model size. The results are summarized in Table 23. Due to the patchified tokenization, when changing the input sequence length from 96 720 (7.5 times longer), the time increases less than 600% and thus we don't i observe the quadratic time differences. It implies model can efficiently handle long input sequences.
32

---

Published as a conference paper at ICLR 2024
Table 23: Model variants. All models are evaluated on 4 different predication lengths (96, 192, 336, 720). The best results are in boldface.
model dimension
16
32
64
128
96
100.00%
101.66%
109.63%
113,28%
Traffic Flectricity Weather ETTh
192
108.28%
104.49%
1012.58%
168.11%
336
113.39%
109.12%
159.18%
275.57%
720
125.71%
182.27%
299.45%
580.04%
96
100.00%
102.85%
111.58%
200.53%
192
104,90%
113,98%
169.15%
312.11%
336
119.56%
85%.
274.05%
537.06%
720
115.42%
167.38%
298.41%
604.03%
96
100.00%
105.67%
118.21%
125.63% 633%
192
102.13%
108.96%
128,34%
140.41%
336
104,05%
110.66%
133.88%
291.2685
720
111.04%
112.03%
193.18%
481,24%
96
100.00%
101.61%
105.43%
386 98%,
103.02%
113.02%
117.51%
42%)
336
112.68%
134.68%
168. 168.98%
485,92%
720
128.17%
183.90%
307,04%
561.57%
N MODEL COMPLEXITIES AND RUNNING TIME
As we use the patching trick, the order of the total complexity would be the same as PatchTST and Crossformer, which is OL'2 / Some Transformer type models (c.g., FEDformer and Autoformer) may even break the quadratic dependent in L and reach linear nearly linear complexity in L. Other CNN and RNN type models (e.g.. TimesNet and FilM) by nature maintain the (L) complexity. When S is set as a not-very-small number (e.g. S RM O(vT)), our model's complexity can also be nearly linear. The condition S x (VV)) is not very restrictive. Take L = 900 as an example, the length of S = 30 would be enough. For the case L 96, the corresponding S would be around 10.
The results of the experiments on rumning time are reported in Table 24. The input/forecasting lengths are set as 96/96 and we keep the batch size the same for all benchmarks and run the experiments on a single A 100/80G GPU. Our proposed model yields comparable running time to transformer baselines as well as linear complexity baselines except Dlinear, which implies in practice model could also behave like linear time model and won't 't introduce overhead computational cost.
Table 24: The average per step running time in seconds. The input/forecasting lengths are set as 96/96 and we keep the batch size the same for all benchmarks and run the experiments on a single A 1000/80G GPU. oom is short for out of memory.
CARD Autoformer PatcTTT Crossformer FEDfomeer TimesNet MICN Dlinear
ETSFormer
ETThl Hiddon=16, 16, Batch=128 128
Train 0.0197
0,1091
0.0164
0.2512
0.2107
0,0672
0.0423 0.0074 0.0747
0.0714
Inference 0.0046
0.0102
0.0021
0.0127
0.0178
0.0132
0.0030 0,0009 0.0123
0.0061
Train Hidden=128. Batch= 18 Inference
Weather
0.0779
0.1525
0.0785
0.1186
0,2189
0.2457
0.0613
0.0330
com 0,0014
0.1354
0.0048
0.0139
0.0036
0.0123
0.0378
0,0224
0.0038
0.0092
Electricity
Train Inference
0.2156
0.0835
0,3280
oom
0.1903
com
0.0405
0.0163 com
0.1174
Hidden= 128. Batch=32
0.0064
0.0160
0.0052
uom
0.0349
com
0.0045
0.0021
com 0.0103
Traffic
Train Inference 0.0101
0.2271
0.0960
0.1329
0.1649
0.6048
0.0322
0.0139
0.0607
Hidden=125 Batch=12 12
0.0153
0.0052
opm
0.0369
0.0418
0.0074 0.0058
com 0.0223
O ATTENTION PATTERN MAPS
In this section, we report the attention maps of each head in the last attention layers. We use E and ETTh2 tasks with forecasting length 96. The input length set as 96 and we use patch length with stride 8 to convert 96 time steps into 12 tokens, and we use the model with 2 attention heads. In order to highlight the correlation between the attention maps w.r.t. the forecasting sequences. We also report the dynamic time warping (DTW) scores between patches and the forecasting sequences, and the sum of attention scores for each patch. The DTW score can be treated as a rough ground truth to evaluate which input patches are most useful for forecasting. The results are summarized in Figure 36-Figure 37. We observe that attention maps have smooth landscapes and we believe it is due to the usage of EMA module to query and keyword tensors. Moreover, sum of attention scores for each patch is positively correlated with the post-hoc computed DTW scores between the patch and the forecasting sequence. It implies the proposed model can effectively
find that the
33

---

Published as a conference paper at ICLR 2024
capture the relationship between the input sequences and forecasting sequences and lead to good final performance.
hnseer
100
176
30-
19
114
Zores
taker
head
head
Ume series
12-
12-
street on soows
Figure 36: Attention Map Samples of ETThl task.
P RELATED WORKS
Patched Transformers in other Domains Transformer (Vaswani et al., 2017) has demonstrated significant potential different data modalities. Among all applications, patching is an essential part when local semantic information is important. In NLP, BERT (Devlin et al., 2018), GPT (Radford et al., 2019) and their follow-up models consider subword-based tokenization and outperform claracter-based tokenization. In CV. Vision Transformers (e.g., Dosovitskiy et al. 2020; Liu et al. 2021; Bao et al. 2022: Ding et al. 2022; He et al. 2022) split an image into patches and then feed into the Transformer models. Similarly, in speech fields, researchers convolutions to extract information sub-sequence levels from a raw audio input (e.g., Hsu al. 2021; Radford et al. 2022; Chen et al. 2022; Wang al. 2023a).
34

---

Published as a conference paper at ICLR 2024
enory
Scure
scotes
155
12.
Lequeet
1.73
head
head
SCOTES
12
13
attertion Boaes
keywardd Eelers inder
Leqweed bakeniidee
head
head
12
Figure 37: Attention Map Samples of ETTh2 task.
Q OTHERS
Q.1
ARCHITECTURE VARIANTS
The present study encompasses the design of five distinct sequential parallel feature flow archi- tectures, with the aim of integrating both temporal signal and channel-aligned information. Here we consider 5 different settings shown in Figure 38 and the results are summarized Table 25. Following an exhaustive analysis, it is concluded that the architecture featuring the channel branch, complemented by channel/time blend, is the most resilient variant. Consequently, this specific architecture is adopted the default approach in this work.
Q.2 COMPONENT ABLATION EXPERIMENTS
Ablation on attention over hidden dimensions and over channels. We conducted series of ablation experiments by removing the attention over hidden dimensions and channels sequentially. Consistent with our design, as shown in table 26, the channel branch contributes to the reduction of mean squared error (MSE); its removal resulted in a 2%, 7% and 6%increase MSE for ETTml, Weather and Electricity respectively. The attention over hidden dimensions branch contributes approximately 1% to the reduction of MSE in ETTml and Weather tasks and in Electricity tasks dropping the over hidden dimensions attention branch results in 1 % more MSE score average.
35

---

Published as a conference paper at ICLR 2024
Doot bankoo Ioow
Channss
Toeess
Chames
Stan
Chemes
Tolaan
Takae?
Curmeb
kyubs
Iopa
CuT4C
ToC-T
ToC+T
ToC
C
Figure 38: Architecture Variants
Table 25: Model variants. All models are evaluated on 4 different predication lengths (96, 192, 336, 720). The best results are in boldface.
Models
e-st+o (CARD)
1-cH
[+C
e:>1
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.316
0.347
0.318
0.346
0.318
0.346
0.326
0.363
0.334
0.368
ETTml
192
0.363
0.370
0.367
0.370
0.366
0.369
0.366
0.385
0,372
0.387
336
0,393
0.390
0.399
0.391
0.396
0.391
0.400
0.404
0.401
0.407
720
0.458
0.426
01466
0.429
0.463
01.428
0.459
0.440
01.588
0.438
avg
0.383
0.384
0.388
0.384
0.386
0.384
0.388
0.398
0.391
0.400
96
0.150
0.188
0.153
0.193
0.152
0.189
0.152
0.191
0.152
0.192
Weather
0.202
0.238
0.203
0.239
0.201
0.236
0.201
0.239
0.203
0.240
336
0.260
0,282
0.269
0.288
0.261
0,281
0.263
0.284
0,262
0.284
720
0.343
0.335
0.345
0.339
0.344
0.337
0.347
0.339
0,344
0.337
avg
0.239
0.261
0.243
0.265
0.240
0.261
0,241
0.263
0.240
0,263
Table 26: Component Ablation Experments by removing the attention over hidden dimensions (wo. hidden
column) and removing the attention over channels (w.o channel) sequentially. All models are evaluated on 4
different predication lengths (96, 192, 336, 720). The differences in thousandths w.r.t. predecessor models are
reported in parentheses.
Models|
CARD
|
wo, hidden
wo, channel
Metric
MSE MAE
diff MAB
diff
MSE
diff
MAE diff
96
0,3470..3770.322
(-6) 0,345 (2) 0.326
(-4) 0.348 (-3)
0.363 0.370
(-1) 0.370
(0) 0.372 (-8)
(:1)
336
0.393 0.390
(:2) 0,391
(-1) 0.404
(-9) 0.393 (-2)
720
0.458 0.422
(-4) 0.427
(-1) 0.470
0.429
(-2)
avg
0.383 0.33440.3388
(-3.3)
0.383
(0) 0.393
(-7.3) 0.408
(-2)
0.1800.18880.1555
(-1) 0.191
(:3) 0.173 (-22)
(-14)
Weather
192
0.202 0.238
(1) 0,236
(2) 0.220 (-19) 0.247 (-11)
336
0.2822
(-3) 0,282
(0) 0.275 (-12)
(-5)
720
0.333
0.344
(.8) 0,336
(11) 0,354 (.14) 0.339 (:3)
avg
0.239
0.261 0.239
(-2.5)
0.261
(-0.5)
0.256 (-16.8) 0.270 (-8.3)
Eleeriiity
192
96
0.144
0.160
0.250
0.253
(-12) 0.257
(-14) 0.243
(:7) 0.182 (-10) 0.259 (-2)
(-9) 0,175
(-21) 0.250 (-8)
336
0.173
0.263 190
(-17) 0,274
(11) 0.197
(-7) 0.275 (-1)
0.197
0.284
(2) 0,312
(224)
0.237
0 (6)
avg
06168 0.258
0. 186
18.8) 0,271
(-12.)
098
(-)
1.5)
0.276 (-4.2)
Ablation on projected dimension in dynamic projection. We conduct experiments varying pro- jected dimensions in 1,8.16. The results are summarized in Figure 39. We observe the performance of dynamic projection is not very sensitive to the projected dimensions, and only slight performance improvement is achieved when increasing the projected dimensions. As the goal of dynamic projec- tion to control the computation cost, the robustness in performance implies doesn't 't harm the forecasting accuracy by a large margin. In practice, we may start with relatively small projected dimension and adaptively tune the hyperparameter if necessary.
Ablation on EMA The goal of EMA is to improve the CARD's robustness. We conduct an experiment with different EMA parameters and the results are summarized in Figure 40. In this experiment, we consider three cases, heavy smoothing (e.g., 0.1), medium smoothing (e.g.. 0.5), and light smoothing (e.g., 0.9). We observe that, in the majority of cases, the standard deviations of MAE and MSE decrease when the strength of smoothing is increased. It confirms our hypothesis that the EMA can improve model robustness. Moreover, in our experiment, we also find that changing
36

---

Published as a conference paper at ICLR 2024
Near wetater CInteet
HAL weather detnet
olanc MALin Eaccictyy deteet
Paand Mit in TAFE CANE
MNN in tork cREMt
Figure 39: Experiments on dynamic projection dimensions. The projection dimension is varying in 1, 8, and 16.
the parameter of EMA from being fixed to learnable makes very few performance differences but significantly increases the training time. In practice, we would suggest using a fixed EMA parameter.
the
Bosiitee you atout
Figure 40: Experiments on stability of EMA module. Each setting is averaged over random seeds.
R MORE ANALYSIS ON TRANSFORMERS FOR TIME SERIES FORECASTING
R.
EXTENDED RESULTS ON INFLUENCE OF INPUT SEQUENCE LENGTH
The extended results on varying input sequence lengths are shown in Table 27,
Table 27: Influence of prolonging input sequence. The lookback length is set as 96, 12.33362267 CARD(96)
means using lookback length 96.
Models| CARD(96) (ARD(192)CAAD(33A) CARD(720)
Metric MSE MAE MSE MAE MSE MAE
MSE MAE
96 0.383 0,391 0.378 0.390 0,,72 0.390 0.368 0.392
192
0.435 0.420 0.427 0.418 0.413 0,416 0.407 0,416
336
0.479 0.442 0,458 0.434 0.437 0.431
0.428 0.430
720
0.477 0.461 0,452 0.456 0.436
0,453
0.418 0,449
avg
0.442 0.429 0.429 0.425
0.445
0,422
0.405 0.421
96
00.316 0.347 0.296 0.333
0.2884 0.328
0.288 0.332
192
0.363 0.370 0.342 0.359 0.326 0.354
0.332 0.357
336
0.393 0.390 0.375 0.379 0.368 0.377
0.364 0.376
720
0.455 0.426 0.439 0.418 0.428
0.410
0.414 0.407
avg
0.383 0.384 0.363 0.372 0.352 0.367 0.349 0.368
R.2 IS TRAINING DATA SIZE A LIMITING FACTOR FOR EXISTING LONG-TERM FORECASTING TRANSFORMERS?
We have observed a distribution shift phenomenon in fifty percent of the benchmark datasets: Traffic, ETTh2, and ETTm2. The model's performance demonstrates a significant enhancement with the use of only 70% training data samples compared to the standard training setting for long-term forecasting, as illustrated in table 28. While it has been argued that the transformer model exhibits a weakness where more training data fails to improve performance (Zeng et al., 2023), we contend that this issue is an inherent feature of each time series benchmark dataset, wherein changes in data distribution between historical and current data are not related to the transformer model. Nevertheless, further exploration of this phenomenon may lead to improved performance, and we thus leave as a topic for future study.
37

---

Published as a conference paper at ICLR 2024
Table 28:
Less training data experiment.
Tasks|
ETTml
ETTm2
ETTh1
ETTh2
Weather
Electricity
Traffic
Metric
MSE MAE
MSE MAE
MSE MAF]
MSE MAF MSE MAE] MSE MAF|MSH
MAR
70% Samples samples
96 0.316 0. 0.248 0.383 0.391 0.281 0.330 0.150 0.181 0,419 0.269
192 0.363 0.370 0.234 0.292 0.435 0.420 0.363 0.381 0.202 0.238 0.160 0.250
0.250 0.443 0.276
336
0.393 0.390 0.294 0.339
0.479 0.442 0.411 0.418 0.260 0.282 0.173 0.263004400 0.283
720 0.458 0.426 0,390 0.388 0.471 0.461 0,416 0.431 0.343 0.335 0.19
0.253 0.282
avg 00,383 0.384 0.272 0.317 00.44 0.429
(0.368
0.390 0.329
0.261
0.261
06 0.25% 0.25%
00453 0.282
96 00.350 0.431 0.163 0.242 00,425 0,431
0.272
0.325 0.245
0.263 0.239
0.404 0.263
192 0,401 0.403 0.225 0.285 0,,82 0.462
0.350 0.374 0.312
0.310
0.257
00.428 0.273
336 00.40 0.428 0.284 0.324 0.485
0.394
0.411 0.382 0.352 0.7 0.270
00.444 0,471
720 0.514 0.471 0.371 0.378
0..59 0.506
0.403
0.427 0,473 0.5 0.229 0.29%
00.71 0.296
avg 0.426 0.419 0.261 0.307
0.491
0.471
0.355
0.384
0.353 0.333
0.333 0.266
0.437 0.278
R.3 EXPERIMENT ON REPLACING SELF-ATTENTION WITH LINEAR LAYER
(Zeng et al., 2023) suggests that a linear layer can be used as a substitute for the self-attention layer to achieve higher accuracy in transformer-B based models. To highlight the effectiveness of celf-attention in our model, we conduct experiments of replacing self-attention modules (e.g., attention over tokens and channels) with linear layer. The results are summarized in Table 29. Upon replacing channel- branch attention and token attention with a linear layer in CARD, we observe consistent decline in accuracy across all datasets. The deterioration effect is particularly pronounced in the weather dataset, which contains more informative covariates, with significant drop of over 13%. These findings suggest that the self-attention scheme may be more effective in feature extraction than a simple linear layer for time series forecasting.
Table 29:
The effectiveness of the self-attention scheme. The lookback length is set as 96. CARD((MLP) uses an MLP layer to substitute the token attention layer in CARD, CARD(cMLP) uses an MLP layer to substitute the channel attention layer in CARD, CARD(dMLP) attention, and CARD(oMLP) contains only the embedding layer and an MLP layer.
two MLP layers to substitute both token and channel
Models|
CARD
CARD(MMLP)
Metric MSE MAE MSE
MAE
MSS
MAE
MSE
MAE
MSE
MAE
96
0.316 0.347 0.333
0.369
0..24
0.357
0.355
0.376
0.356
0.376
ElTml
192
0.363 0.370 0.375
0.390
0.371
0.381
0.393
0.394
0.393
0.394
336
0.393 0.390 0.405
0.409
0.403
0.402
0.425
0.415
0.424
0.414
720
0.458
0.426 0.467
0.444
0.463
0.436
0.489
0.451
0.467
0.444
avg
00.38 0.384 0.395
0.403
0.339
0.394
0.415
0.409
0.415
0.408
0.150 0.188 0.160
0.207
00.172
0.213
0.195
0.234
0.195
0.234
Weather
192
0.202 0.238 0.211
0.254
0.220
0.255
0.240
0,270
0.240
0.270
336
0.260
0.282 0.270
0.296
0.276
0.296
0.292
0.306
0.292
0.306
720
0.343
0.335 0.358
0.351
0.353
0.346
0.364
0.353
0.364 0.353
avg 0,239 0.261 0.250
0.277
0.255
0.277
0.272
0,291
0.273 0.291
S EVALUATION ON IMPUTATION
We test the proposed model's 's imputation ability. We adopt the experimental settings in (Wu et al., 2023b) and results are reported in Table 30. CARD obtains top 2 ranks in 22/24 MSE scores and all MAE scores. In particular, in the Electricity dataset, CARD significantly reduces the MSE and MAE by 40% and 28% over the previous best results respectively. Those results suggest CARD may also generate good representations and thus can also work in the problem beyond forecasting.
38

---

Published as a conference paper at ICLR 2024
Table 30; Imputation task. The time sequence is randomly masked 12.5%, 25%, 37.5%. and 50% points. MAE and MAE are reported. he best model is in boldface and the second best is underlined.
Models|
CARD
TimesNett
Dlinear
LighTTS
|ETSformer] Stationary FEDromeer /Attoooorr/uutooommmrrr Informer
Metric
MSE MAF] MSF MAR
MSE MAFIMSE MAF| MAF] MSF MAF MSF MAF| MAF] MSE MAF
MSF
12.5%
0.091 0.019 0.092
0.058 0. 0.00755 0.1 0.398/0.026 107 0.764 0.416 .034 0. 0..047 0.155
25%
0.028 0. 106 0.023 0.101
0.058 0.162 0.093 0.096 0.229 0.032 0.131 0.426 0.441 0.046 0.144 0.063 0.180
37.5%
0.032
0.115 0.029 0.111
0,103 0.113 0.231
0.133 0.271 0.039 0.131 0.445 0.459 0.161 0.079 0.200
50%
0.033
0.114 0.036 0.124
0.132 0.248 0.134 0.255
086 0.323 0.047 0.145 0.089 0.218
0.079 0.218
avg
0.028 0.107 0.027 0.107 0.093 0. 0.206 0.218
0. 0.253 0.036 0.6 0.062 0.175 0.150/0.071 0.188
12.5%
0.0760.013 0.080
0.062
16610.034
0.7 108 0.222 0.088 0.056 159/0.023 0.092 0.270
25%
0.021
0.081 0.020 0.085
0.085 O. 0.042 0.143 164 0.294 0.024 0.096 0.080 O. 195 001 0. 135 0.272
37.5%
0.022 0.086 0.023 0.091
0.106 0.222 0.051 0.257 237 0.356 0.027 103 0,
0.231 0.030 O. 1. 155 0.293
50%
0.024
0.090 0.026 0.098
131 0.247 0.059 O. 174 3.323 0.421 0.030 O. 108 156 0.075 O.
1.w.03555.111910.... 2000 0.333
avg
0.002
0.083 0.027 0.088
0.096 O. 208 0.046 0. 151 0 208 0.327 0.026 0.099 0. 0.215
0.029 0.219
0.
0.292
12.5%
0.044
138 0.0S7
0.
151
0.267 0.240 0.34 126 2.263 26
0. 0.070 190 0.
0.12
0.234
25%
0.054
0.154 0.069
0. 78
180 O. 292 065 0.364 0. 169 0. 304 O. 080 0.
0. 106 O 0.090 O. 0. 140 0.262
37.5%
0.069
0.174 0.084
0. 196
0.215
0. 0.347 0.334 0.404 293 0.402 O. 133 0.240
0.318 296 0.382 0. 220 0.347 O. 102 0.212
0.258 109 0. 0.222 0. 0.293
50%
0.085
0.194 00
0.215
0.257
0.248 0.325 0. 0.299
avg
0.063 065 0.078
081
0.201
0. 0.306
0.239 0.094 0.101 0.
0.214 0.279
12.5%
0.040
0.122 0.040
0.130
0.1 0. 0.231
9 0. 0.096 0.21210.044 0.138 0.754
25%
0.041
0.128 0,046
0.
0.127 0.247 0, 0.246 0.279
0.390 0.049 0.147 0.137 0.25
0. 0.149 0.444
37.5%
0,045
0.135 0.052
1,151
0.158 0.5 0.276
0.257
0.400
0.465 0.056
0.158 087 3.330
0.353 0.462
50%
0.051
0.146 0.060
0. 162
0.183
0.299 136 0.268 0.602 0.572 0.065 1.0 0.232 0.341 0.068 O. 0. 0.472
avg
00.04
0.133 0.049
0.146
0.1142 0.259 0.259 1.119 0.367 0.436 0.053 0.152 0,1 0.279 0.166 0.452
12.5%
12.5%/0.027
0.027
0.040 0.025
0.045
0..099
0.084)0.047
O. 101 0.057 O. 141 0.027 0.051
0.041 O. 107 0.066 0.047 0.093
Weather
25%
0.029
0.042 0.029 0.052
0.048 O. 1.103 0.052 O. 0.052
0.065 O. 155 0.029 0.056 0.064 O. 163 0 030 0. 0.052 0.100
163
37.5%
0.033
0.045 0.031 0.057
0.057 0.117 0.058 1.1 0.081
0.033 0.062
007 0.229 0.032 0.060 0.111
50%
0.036
0.048 0.034 0.062
0.066 0.134
131
0.207 0.037 0.068
0.067 0.114 0.012
avg
0.031
0.044 0.030 0.054
0.052 O. 0.065 0.12
0.207 0.037 0.068
0.183 0.312
0.057 0.104
12.5%
0.043
0.131 0.085
0.202
0.092
0.214 0.229
196
0. 0 0.210
0.089
0.210 0.326
Electricity
25%
0.049 0.142 0.089 0.206
0.1 0.247 021 0.252
0. 207 0.332 0.337 0. 0.251 0.096 0.220 0.326
37.5%
0.059
0.1 0.094 0,213
0,4
0. 0.276
0.273
299 0.344 002 0.220 136 0 0.266 104
0.22 0.328
50%
0.069 0.172 0. 10000000000000 2.22
0. 175 0. 305
0.
160 0. 235 0. 357 0. 0.228 158 284 0. 0. 0.228 0.331
avg
0.055
0.151 0.092 0.210
0. 132 0. 260
131 0.262
0.21
0.220.00..2222 0,328
39


Reviews:
[{"review_id": "XiVM3mImuA", "reviewer": "Reviewer_ajLB", "summary": "The paper proposes a transformer architecture whose main feature seems to be the attention applied both across time dimension and across feature dimension.", "questions": "- Please add results on ILI and Exchange to Table 1\n- I flagged a few concerns and will revise my score if they are addressed appropriately", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The architecture design seems novel and interesting\n- Extensive ablation studies and additional results in Appendices\n- I am happy to see, finally, that the transformer based paper discusses and outperforms results on basic baselines, most importantly\n    - Zeng et al. Are transformers effective for time series forecasting? AAAI'23\n- In general the paper seems to be well written and clear. I like the level of detail and clarity of Figures 1,2", "weaknesses": "- The paper makes same methodological mistake as the previous line of this research. We really need to compare against basic baselines! Please include results of Zeng et al. in Table 1. Today, this is perhaps the most important baseline to include in the Table, so I'd rather not see some of the other entries in the table than to see this one missing\n- The results in Table 2 are not as impressive as in Table 1. The model is not performing well compared to the first entry in the M4 competition and is barely performing better than the second entry, which is not reflected in Table 2. Again, same methodological mistake of not comparing comprehensively against basic baselines. Please include them in the table if you want to demonstrate methodological rigour and promote consistency across studies. The N-BEATS results are probably presented in the table without ensembling, but even the basic ensemble of 18 models is way better than the proposed model as follows from Figure 3 in https://arxiv.org/pdf/1905.10437.pdf. I suggest that the authors follow well accepted protocol for M4 benchmark evaluation and present the ensemble-based results, or remove Table 2 from the paper and replace it with the results of their ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Auo08OqO6C", "reviewer": "Reviewer_gMyA", "summary": "This paper presents CARD, a particular Transformer model for time series forecasting.  CARD contains three core designs：\n\n* A channel-aligned attention structure\n* A token blend module\n* A robust loss function for time series forecasting\n  \nThe author provides extensive experiments to verify the effectiveness of the proposed method in multiple long-term, short-term forecasting and anomaly detection datasets.", "questions": "1. In long-term forecasting tasks, I have questions regarding the experimental settings of the current state-of-the-art model PatchTST. The input length for PatchTST/42 is 336, and for PatchTST/64 it is 512. I am curious whether CARD can outperform PatchTST if the authors use the same input length. If you aim to validate your experimental settings, you should use the official code of the SoTA model and adjust the parameters to ensure a fair comparison.\n\n2. Channel independence introduces a significant computational overhead. To mitigate this issue, the authors employ dynamic projection (DP) to project head dimensions from d_head to a fixed $r$ with $r ≪ C$. They demonstrate through ablation experiments involving projected dimensions of 1, 8, and 16 on ETT datasets that DP does not compromise forecasting accuracy. However, it's worth noting that ETT datasets only consist of 7 channels, which may not be sufficient to draw a definitive conclusion. Therefore, it remains unclear how this technique would perform in datasets with a larger number of variables, such as Traffic (862) or ECL (321).\n\n3. In ablation experiments, our primary focus is on evaluating the effectiveness of the new design between CARD and PatchTST. Instead of solely changing parameters, we aim to demonstrate the method's rationality and validity with greater certainty through more detailed ablation studies.\n\n4. In experiments with less training data, using only 70% of the training data samples compared to the standard training setting for long-term forecasting may not be sufficient.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to follow, with good figures for explanation.\n\n2. The proposed method combines various time series forecasting effective design ideas.\n\n3. The proposed method has conducted experiments on multiple time series forecasting tasks.", "weaknesses": "**Method design**\n\nThe proposed approach needs more novelty. Some of the main design ideas are presented and verified by the previous work, such as channel independence, patchify, EMA, and dynamic projection technique.\n   \n**Experimental rationality**\n\n* The main experimental setting needs to be consistent with the SOTA method to be convincing, and the experimental results need more justification.\n\n* The rationality of some analytical experiments needs further verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L4lbLmIkrA", "reviewer": "Reviewer_eTmJ", "summary": "This paper introduces the Channel Aligned Robust Blend Transformer (CARD), a specialized transformer model designed for time series forecasting. It acknowledges the success of transformer models in this field, particularly due to the channel-independent (CI) strategy that improves training robustness. However, the CI approach overlooks the correlation among different channels, limiting the model's forecasting capacity. To address this limitation, CARD incorporates a channel-aligned attention structure to capture temporal correlations among signals and dynamical dependencies among multiple variables over time. Additionally, a token blend module is introduced to efficiently utilize multi-scale knowledge by generating tokens with different resolutions. The authors also present a robust loss function that considers prediction uncertainties to mitigate overfitting. Experimental evaluations on various long-term and short-term forecasting datasets demonstrate that CARD outperforms state-of-the-art methods in time series forecasting. The article concludes by discussing related work, presenting the detailed model architecture, describing the loss function design, and providing the results of numerical experiments and further analysis.", "questions": "1. Presentation Aspect:\n   It would be beneficial for the article to include a schematic diagram illustrating the limitations of previous models at the beginning, to better introduce the motivation and background of the study. This would help readers understand the context more clearly and provide a more coherent structure to the article.\n\n2. Experimental concerns are listed in the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a novel transformer model specifically designed for time series forecasting. It addresses the limitations of the channel-independent (CI) strategy by incorporating a channel-aligned attention structure and a token blend module to capture correlations among different channels and utilize multi-scale information effectively.\n\n2. This paper evaluates the proposed CARD model on multiple long-term and short-term forecasting datasets, comparing it against state-of-the-art methods, and demonstrates its superiority over existing techniques in various prediction-based tasks.\n\n3. This paper introduces a robust loss function specifically designed for time series forecasting. By considering prediction uncertainties and weighting the importance of forecasting over a finite horizon, the proposed loss function addresses the potential issue of overfitting noises. This contribution enhances the model's ability to make accurate predictions.\n\n4. This paper follows a well-organized structure. The inclusion of figures, such as the model architecture figure and experimental result figures, aids in understanding the proposed methods.", "weaknesses": "1. Experimental Concerns:\n\n   (1) Fixed-parameter EMA: The article directly applies a fixed-parameter Exponential Moving Average (EMA) without providing specific explanations regarding the potential drawbacks of using a learnable-parameter EMA. The advantages or disadvantages of employing a learnable-parameter EMA are not thoroughly explored.\n\n   (2) Lack of comparison with different lookback values: The article fails to compare the results of different lookback values (the number of previous time steps considered) on the M4 dataset. Such a comparison would provide a more comprehensive analysis and insights into the impact of lookback on forecasting performance.\n\n   (3) Inconsistency in short-term forecasting results: While the article highlights the third point of short-term forecasting, the visual results presented in the figures do not seem to align well with the data shown in the tables. Additionally, the evaluation of short-term forecasting appears to be limited to only the M4 dataset, which may not be sufficient to fully demonstrate the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ybJPxFuXVe", "reviewer": "Reviewer_fhNE", "summary": "This paper proposes a novel time series forecasting method, called CARD. Compared with previous works, CARD builds an effective transformer to employ cross-channel information. CARD utilizes attention over tokens and attention over channels to robustly align the information among different channels. Furthermore, the token blend module introduces multi-scaling structural information. To bolster the model’s ability to concentrate on forecasting for the near future, this paper develops a robust signal decay-based loss function. The comprehensive experiments demonstrated the superiority of CARD for long-term forecasting, short-term forecasting, and other prediction tasks.", "questions": "The concerns include the comparison with baselines on the complexity, the confusing experimental results, in-depth insight into the ablation study, and the theoretical analysis of how the model mitigates overfitting issues. Please see the weaknesses for details.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The method proposed in this paper is highly effective, taking into consideration the interactions among different channels, thereby mitigating to some extent the overfitting issues commonly associated with previous channel-dependent models.\n\n(2) The experiments are highly comprehensive, assessing the model's performance across long-term forecasting, short-term forecasting, and other prediction tasks. In addition, this paper conducts extensive ablation studies, including hyperparameter sensitivity analysis experiments, loss function analysis experiments, and so forth.\n\n(3) The paper is generally well-written and easy to understand.", "weaknesses": "(1) The paper analyzed the complexity of the proposed model, which is an advantage. However, this paper did not specifically outline the complexities of the baseline models and there were no direct comparisons with the baselines on the running time or efficiency.\n\n(2) Some of the experimental results are somewhat confusing. For example, in Table 18, the average performance of CARD and wo. hidden are identical, but the diffs are not 0.\n\n(3) In the ablation study of the signal decay-based loss function, i.e. Table 12 and Table 13, I observe that the method proposed in this paper does not exhibit a significant advantage compared to the SOTA baseline without the robust loss. Furthermore, it would be better if the authors could provide test results on a broader range of datasets.\n\n(4) The paper claims that the proposed method can effectively mitigate overfitting issues. Although some empirical analysis has been provided, supplementing with appropriate theoretical analysis would be more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)