Self-Supervised Contrastive Learning for Long-term Forecasting
Authors: Junwoo Park
Daehoon Gwak
Jaegul Choo
Edward Choi
Venue: ICLR 2024 poster (2024)
Keywords: Contrastive learning
time-series representation
long-term forecasting
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Long-term forecasting presents unique challenges due to the time and memory
complexity of handling long sequences. Existing methods, which rely on sliding windows to process long sequences, struggle to effectively capture long-term variations that are partially caught within the short window (i.e., outer-window variations). In this paper, we introduce a novel approach that overcomes this limitation by employing contrastive learning and enhanced decomposition architecture,
specifically designed to focus on long-term variations. To this end, our contrastive
loss incorporates global autocorrelation held in the whole time series, which facilitates the construction of positive and negative pairs in a self-supervised manner. When combined with our decomposition networks, our constrative learning significantly improves long-term forecasting performance. Extensive experiments demonstrate that our approach outperforms 14 baseline models on well-established
nine long-term benchmarks, especially in challenging scenarios that require a significantly long output for forecasting. This paper not only presents a novel direction for long-term forecasting but also offers a more reliable method for effectively integrating long-term variations into time-series representation learning.

Published as a conference paper at ICLR 2024
SELF-SUPERVISED CONTRASTIVE LEARNING FOR LONG-TERM FORECASTING
Junwoo Park, Daehoon Gwak, Jaegul Choo, Edward Choi Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea (junwoo. park, daehoon. gwak, jchoo, edwa rdchoi @eaaekkkatttt.eee
ABSTRACT
Long-term forecasting presents unique challenges due to the time and memory complexity of handling long sequences. Existing methods, which rely slid- ing windows to process long sequences, struggle to effectively capture long-term variations that are partially caught within the short window (i.e., outer-window variations). In this paper, we introduce a novel approach that overcomes this limita- tion by employing contrastive learning and enhanced decomposition architecture, specifically designed to focus on long-term variations. To this end, our contrastive loss incorporates global autocorrelation held the whole time series, which facili- tates the construction of positive and negative pairs in self-supervised manner. When combined with our decomposition networks, our contrastive learning sig- nificantly improves long-term forecasting performance. Extensive experiments demonstrate that our approach outperforms 14 baseline models in multiple ex- periments over nine long-term benchmarks, especially in challenging scenarios that require a significantly long output for forecasting. Source code is available at Forecsating
INTRODUCTION
Time-series data presents unique challenge due its potentially infinite length accumulating over time, making it infcasible to process them all at once (Ding et al.] 2015 Hyndman et al., 2015 Rakthanmanon et al., 2013). This requires different strategies compared other sequence data such as natural language sentences. To address this, the sliding window approach (Kohzadi et al.) 1996) is commonly employed partition a single time-series data into shorter sub- see-eeequeeess (i.e., windows) Typically, in time-series forecasting, the sliding window approach enables models to not only process the long-time series but also capture local dependencies between the past and future sequence within the windows, resulting in accurate short-term predictions.
Recently, as the demands in the industry to predict more distant future increases (Ahmad et al. 2014
Variations entire series (ETTh2)
Vlahogianni et al. 2014 Zhou et al.) 2021). }, various studies have gradually increased the window length. Transformer-based models have reduced computa- tional costs of using long windows through improve-
foursr Comporerb
Astoonnian
Autocorrelation
25-033
ments in the attention mechanism (Zhou et al. 2022
0.2
13:00
Wu al. 202T; Liu al. 2022a Also. CNN-based models (Bai et al. 2018 Yue et al.) 2022) have ap- convolution operations
.023
-0:3
10000
learn more distant dependencies while benefiting from their efficient computational cost. Despite the remarkable progress made by these models, their effectiveness in long-term forecasting remains uncertain. Since the extended window is still shorter than the total time series length, these models may not learn the longer temporal patterns than the window length.
plied a dilation
-0.5
2000
4003
6000
0200 10000 12330
34080
Lag / Period
Figure 1: Long-term variations span beyond the conventional window. There are non-zero correlations (Left Y axis) with longer lags, and Fourier components (Right axis) with longer periods than the window size.

---

Published as a conference paper at ICLR 2024
W1
Yearly-long periodicity
W2
W3
Smiuny Bestro
Window Inctee
Ancher Wincow
Input
Output
Ancher indox
PEDtormer
PatchTST
Sim Sim(Wz, W
Sim ( W3
TimesNet
Time (window Indext
Figure 2: (Top) Electricity time series including long-term variation beyond window size. (Bottom) Plotted representation similarities of four models between an anchor window W2 and all other windows including W1 and W3. To clearly highlight long-term correlation, we smoothed fluctuations caused by short-term correlation. The details of the visualization are found in Appendix [.T] Even though W2 have a similar temporal pattern with Wi due to yearly-long periodicity, three models, except for Ours, fail to learn this periodicity as the representation, The three models result nearly identical cosine similarity scores (i.e., Sim(Wz, W1) z Sim (W2, W3)) between two representations of input parts within each window. This contributes to our model showing lower mean squared errors (0. 275) ) in long-term predictions than PStchTST (0.332) and TimesNet (0.417).
In this paper, we first analyze the limitations of existing models trained with sub-sequences (i.e., based on sliding windows) for long-term forecasting tasks. We observed that most time series often contain long-term variations with periods longer than conventional window lengths as shown in Figure [ and Figure 5 If a model successfully captures these long-term variations, we expect the representations of two distant yet correlated windows to be more similar than uncorrelated ones. However, since the previous studies all treat each window independently during training. is challenging for the model to capture such long-term variations across distinct windows. Explicitly, Figure Segure@show shows that the representations of existing models fail to reflect the long-term correlations between two distant windows. Still, recent methods tend to overlook long-term variations by focusing more on learning short-term variations within the window. For example, existing models based on decomposition approaches (Zeng et al. 2023 Wang et al. 2023) often treat the long-term terettermon variations partially caught in the window as simple nor-poriodic trends and employ a linear model to extend the past trend into the prediction. Besides, window-uni normalization methods (Kim et al.) 202T Zeng al., 2023) can hinder long-term prediction by normalizing numerically significant values (e.g., maximum, minimum, domain-specific values in the past) that may have long-term impact on the time series. Since these normalization methods are essential for mitigating distribution shift problems (Kim et al.) 2021 caused by nonstationarity (Liu et al., 2022b), new approach is necessary to learn long-term variations while still keeping the normalization methods.
Therefore, we propose novel contrastive learning to help the model capture long-term dependencies that exist across different windows. Our method builds on the fact that a mini-batch can consist of windows that are temporally far apart. allows the interval between windows to span the entire series length, which is much longer than the window length. Section 3.1 describes the details of our contrastive loss. Moreover, we use our contrastive loss in combination with a decomposition--ass model architecture, which consists two branches, namely a short-term branch and a long-teni term-ttrmmmm branch. Naturally, our loss is applied to the long-term branch. However, as pointed out carlier, the long-term branch in the existing decomposition architecture has been composed of single linear layer, which unsuitable for learning long-term representations. Thus, as explained in Section 3.2] we redesign the decomposition architecture where the long-te branch has sufficient capacity to learn long-term representation from our loss. In summary, the main contributions of our work are as follows:
Our findings reveal that the long-ten term performances of existing models are poor as those models overlooked the long-term variations beyond the window.
We propose AutoCon, a novel contrastive loss function to learn long term representation by constructing positive and negative pairs across distant windows in a self-supervised manner.
Extensive experiments nine datasets demonstrate that the proposed decomposition archi- tecture trained with AutoCon achieves performance improvements of up to 34% compared to a total of 14 concurrent models including three representation methods.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Contrastive Learning for Time-series Forecasting Contrastive learning (Chen et al. 2020 Khosla jet al. 2020: Zha et al. 2022) is a type of self-supervised learning technique that helps models learn useful representations of data without the need for explicit labeling of data. Motivated by the recent success of contrastive learning in computer vision, numerous methods (Tonekaboni et al. 2021 Yue ct al. 2022| Woo et al., 2022a) have been proposed in time-series analysis. In contrastive learning, since how to construct positive pairs has a great influence on the performance, they mainly proposed positive pair construction strategies such as temporal consistency (Tonekaboni al.] 202T). sub- series consistency (Franceschi et al. 2019), and contextual consistency (uee et al., [022]. However, these strategies have limitation in that only temporally close samples are selected as positives, overlooking the periodicity in the time series. Due to the periodicity, there may be more similar negative samples than positively selected samples. Recently, CoST (Woo et al. (022a) tried to learn representation considering periodicity through Frequency Domain Contrastive loss, but it could not consider periodicity beyond the window length because it still uses augmentation for the window. In the time-series learning framework, we focus on the fact that randomly sampled sequences in a batch can be far from each other in time. Therefore, we propose a novel selection strategy to choose not only local positive pairs but also global positive pairs between the windows in the batch.
Decomposition- Models for Long-term Forecasting Time-series decomposition (Cleveland et al. 1990) is well-established technique that involves breaking down time series into its individual components, such as trend, seasonal, and remainder components. By decomposing a time series into these components. it becomes easier to analyze each component's behavior and make more interpretable predictions. Thus, decomposition-basse
models (Wu et al.) [222] Zhou et al. 20226 Wang al., 2023) have gained popularity in time-series forecasting, as they offer robust and interpretable predictions, even when trained on complex time series. Recently, pLinear(Zeng et al. 2023) has demonstrated exceptional performance by using a decomposition block and a single linear layer for each trend and seasonal component. However, our analysis indicates that these linear models are effective in capturing high-frequency components that impact short-term predictions, while they often miss freeuuueeeyy components that significantly affect long-term predictions. Therefore, single linear model may be sufficient for short-term prediction, but it is inadequate for long-term prediction. In light of this limitation, we propose model architecture that includes layers with varying capacities to account for the unique properties of both components.
3 METHOD
Notations We first describe the forecasting task with the sliding window approach (Zhou et al.
2021, Wu et al. 2021: Park et al. 2023), which covers all possible in-output sequence pairs of the entire time series
.s.....,57 } where T denotes the length of the observed time series and 8, E IR is observation with e dimension. For the simplicity in explaining our methodology, we set the dimension e to throughout this paper. By sliding window with a fixed length W on S, we obtain the windows D W.M.M where W; = (X,, Vi) are divided into two parts: input sequence X, , } with the input length I, and output sequence Di = [St+ S(+1+0-1) with the output length O to predict. Also, we denote the global index sequence of W/2 as Te = {t + o
AUTOCORRELAATOOSS AUTOCORRELATION BASED CONTRASTIVE LOss FOR LONG-TERM FORECASTING
3.1
Missing Long-term Dependency in the Window Many real-world time series exhibit diverse long-term and short-term variations (Wu et al.. 2021 2023: Wang et al. 2023). In such cases, a forecasting model may struggle to predict long-term variations, these variations are not captured within the window. We first identify these long-term variations using autocorrelation, inspired by the stochastic process theory (hhttield & Xing 2019 Papoulis & Unnikrishna Pillai 2002). For a real discrete-time process {S.), we can obtain the autocorrelation function Rss (h) using the following equation:
T
1
L S,S,-A
(ss(h) = lim T T
t=1
The autocorrelation measures the correlation between observations at different times (i.e. time lag h). strong correlation close to 1 or -1 indicates that all points separated by h in the series S are linearly related, moving in the same opposite direction for positive or negative signs, respectively. In other words, autocorrelation can be utilized to forecast future variations that are interval away based on current variations. Although recent methods have leveraged autocorrelation discover period-
3

---

Published as a conference paper at ICLR 2024
Value
Long-term variations
Time
0.8
- =212
Global Autocorrelation
0.6
0.4
0.2
RAs (1(3-
0.0
Rss (ss(((1-ta))
Raslle2-
Rss
(122-211) *nhy
Rss (11z(11))>
R85l113-(2)
mby
Mhy
Map Neg
mny
Mul
Anchor
Pos
Neg
Anchor
Pos
Anchor
Pos
Neg
Figure 3: Example of the relative selection strategy in our AutoCon. Three windows are sampled from different times f1, 12, and 13 on the entire series to make up the batch. In this batch, there are a total of three possible positive pairs (i.e., due to three anchors). Each pair calculates global autocorrelation whose lag is the time distance of the two windows constituting the pair. Then, by comparing the autocorrelation with other remaining pairs, the pairs with lower autocorrelation than the anchor positive pair are designated as negative pairs.
based dependencies (Wu et al. 2021 Wang al., 2022). they only apply to capture variations within the window, overlooking long-term variations that span beyond the window. But as shown in Figure [ non-zero correlations exist outside the conventional window length. For the first time, we propose representation learning method via contrastive learning to capture these long-term variations quantified by the global autocorrelation. Note that, to distinguish our method from those that use local autocorrelation within given window, refer to the autocorrelation calculated across the entire time series as the global autocorrelation.
Autocorrelation-bann based Contrastive Loss (AutoCon) We note that mini-batch consist of windows that are temporally very far apart. This time distance can be as long as the entire series length T', which is much longer than the window length W. Based on this fact, we address long-term dependencies that exist throughout the entire series by establishing relationships between windows. Concretely, we define the relationship between the two windows based on the global autocorrelation. Any two windows We, and W', obtained at two different times t and t2 each have W observations with globally indexed time sequence Tr, = (t1 + i)! and Tr, = (t2 + j W 1= Then, we denote time distances between all pairs of two observations in each window matrix D contains time distance as elements Dr,, (tz + j) (t1 + +)]. In the two windows, the time distances between the same phase (i.e., = j) all have the same value |t f2/, and they are represented by the diagonal terms {D.+ W of the matrix. Therefore, based on this representativeness, we leverage the global autocorrelation Rss((t - t2l) to define the relationship between the two windows as follows:
matrix D E RW This
rT,,, Trg) [Rss([4 [2))
where Rss denote the global autocorrelation calculated from train series S.
Now, we design loss to ensure that the similarities between all pairs of window representations follow the global autocorrelation measured in the data space. To achieve this, we define positive and negative samples in a relative manner inspired by SupCR (Zha al., 2022) for regression tasks in the image domain. However, unlike SupCR which uses annotated labels to determine the relationship between images, we use the global autocorrelation Rss to determine the relationship between windows, making our approach
unsupervised method. We feed mini-batch X E IR N consisting of N windows to the encoder to obtain representations v E Nx the windows i, our iittccorellaioonbaseee contrastive loss, called AutoCon, is then computed over the representations (
x /xd where u = Enc (X', T). Indexed by
with the corresponding time sequence (T()) 1
Ji=1
J==1
LAutoCon
Σ N
1
Σ j=1,j1
(1-) log N Ek=1 (171)) exp ( (Sim (v(1), (b))) /T)
exp (Sim (v b())) /T)
(3)
where Sim (:.-) measures the similarity between two representations (e.g., cosine similarity between
max pooled (1) along with the time axis (Yue al. 2022)) and y(((.)) r(T(), TU)) represents
4

---

Published as a conference paper at ICLR 2024
the global correlation between two windows. During training, there are total of N x (N 1) possible pairs indexed by (i.j). Each pair (i.e. as an anchor pair) designates itself as a relatively positive pair by considering any pairs that exhibit the global autocorrelation y((.k) lower than that ((ti)) of the anchor pair as negative pairs. Figure 3 describes sample cases of our selection strategy in the given batch. Since a different set of windows form the batch in each iteration, we expect that the representations reflect the global autocorrelations of all possible distances. The relative selection strategy does not guarantee that the positive window has high correlation close to one; it only requires a higher correlation than other negative windows in the same batch. Consequently, we introduce p(i)) as weights to differentiate between positive pairs with varying degrees of correlation, similar to focal loss (Lin et al. 2017). To minimize AutoCon, the encoder learns representations so that the pairs with high correlation are closer than the pairs with low correlation.
Our AutoCon offers several notable advantages over conventional contrastive- based methods. First, although AutoCon is an unsupervised representation method, it does not rely on data augmentation, which is common in most contrastive-based approaches (Tonekaboni al. 2021 Yue al. 2022 Woo et al. 2022a). The augmentation-hant by the augmentation process and increase the forward-backwand process for the augmented data. Also, existing contrastive learning methods consider only temporally close samples as positive pairs within windows. This ultimately fails to appropriately learn representations of the windows that are distant from each other but are similar due to long-term -ergetrrmmmm periodicity. Consequently. our method is computationally efficient and able learn long-term term representations, enhancing the ability predict long-term variations effectively.
methods require additional computation costs caused
3.2
DECOMPOSITION ARCHITECTURE FOR LONG-TERM REPRESENTATION
3=3
Xnarm
1 Shoet
Lincar
X50
Window Normalization
Window Denoraalizaion
Xaorma
g
Multi-ccale Decoder
Encoder
Yung
Global Autocorrelation Rys
AutoCon
Backward path Forward
Short-term branch Long-term branch
Figure 4: An overview of the redesigned architecture for long-term representation and forecasting
Existing models commonly adopt the decomposition architecture that has a seasonal branch and trend branch to achieve disentangled seasonal and trend prediction. To emphasize that trends are long-term variations partially caught in the window, we regard the trend branch as a long-term branch and the seasonal branch short-term branch. Our AutoCon method is designed to learn long-term representations, making it natural not to use it in the short-term branch to enforce long-term dependencies. However, integrating AutoCon with current decomposition architectures presents a challenge because both branches share the same representation (Wu et al., 2021; Zhou et al. 2022b Liu et al., 2022b), or the long- branch consists of linear layer that not suitable for learning representations (Zeng et al. 2023: Wang et al.) 2023). Moreover, we observe that recent linear- based models (Zeng et al. (023) outperform complicated deep models at short-term predictions, leaving doubts whether deep model is necessary to learn the high-frequenc, variations. Based on these considerations, we redesign a model architecture with well-defined existing blocks to respect temporal locality for short-term and globality for long-term forecasting as shown in Figure 4 Our decomposition architecture has three main features.
Normalization and Denormalization for Nonstationarity First, we window-uni normalization and denormalization methods (Equation
4 (Kim et al. 2021| Zeng et al. 2023) as follows:
Xnorm = X X.
Y = (J'short + Viong) + X
(4)
where X is the mean of the input sequence. These simple methods help to effectively alleviate the distribution shift problem (Kim et al. 2021 by nonstationarity of the real-world time series.
Short-term Branch for Temporal Locality Next, we observe that short-period variations often repeat multiple times within the input sequence and exhibit similar patterns with temporally close
5

---

Published as a conference paper at ICLR 2024
sequences. This locality of short-term variations supports the recent success of ehear-based mod- els (Zeng et al. 2023). which use sequential information of adjacent sequences only. Therefore, we employ the linear layer for the short-term prediction follows:
short = Linear LiinearXnnom (Norm))
(5)
Long-term Branch for Temporal Globality The long-term branch, designed apply the AutoCon method, employs an encoder-decoder architecture. The encoder with sufficient capacity to learn the long-term presentation leverages both sequential information and global information (i.e., timestamp- based features derived from T) as follows:
v EEn((nnorn T).
The choice of network for the encoder is flexible as long as there are no issues in processing long sequences. We opted for temporal convolution networks (Bai et al. 2018) (TCNs), widely used in learning time-series representation (Yue et al. (022), for its computational efficiency. The decoder employs the multi-scale Moving Average (MA) block (Wang et al. 2023), with different kernel sizes {kr ) to capture multiple periods based on the representation as follows:
Vrong Σ
The MA block at the head of the long- term branch smooths out short-term fluctuations, naturally encouraging the branch to focus on long-term information. Our redesigned architecture is optimized by the objective function as follows:
L -MSE + A -AuutoCon
(8)
where the mean square error (MSE) and the AutoCon loss are combined with the weight as hyperparameter. The hyperparamer sensitivity analysis is available in Appendix [A.6] Detailed descriptions of each operation (e.g., Linear, Padding, and LP) can be found in the Appendix A.
4 EXPERIMENTS
To validate our proposed method, conducted extensive experiments nine real- world datasets from six domains: mechanical systems (ETT), energy (Electricity), traffic (Traffic), weather (Weather), economics (Exchange), and disease (IL.). We follow standard protocol (Wu et al. 2021) and split all datasets into training, validation, and test sets in chronological order by the ratio of 6:2:2. We select the latest baseline models with different architectures categorized into linear-based (Zhou et al. 2022a (Zhou 2023 ). Additionally, we compared our model with two
Zeng et al. 2023). CNN-based (Wu et al.,
2023
Wang et al., 2023). and Transformer-
al. 2022b; Liu et al. 2022b Nie et al.
that focus on learning inter-channel dependencies for multivariate forecasting. Appendix provides more detailed information about the datasets and baseline implementations.
models (Challu et al. 2023) Zhang & Yan
20233
4.1 MAIN RESULTS
Extended Long-term Forecasting In our pursuit to better evaluate model's performance in predicting long-term dariations-while tend to have increasing significance as the forecast length extends-wa designed our experiments extend the prediction length O for each dataset. This shift from the conventional benchmark experiments, which typically predict up to 720 steps, allows us to explore the model's capability in more challenging forecast scenarios. For the datasets with longer total lengths, such as ETTh, Electricity, Traffic, and Weather, we tripled the prediction length from 720 2160. Also, for Exchange and ILI datasets with shorter total lengths, we extend the output length up to 1080 and 112, respectively. Overall, T that our model with AutoCon outperformed the ofate-of-the-are art baselines by achieving first place 42 times in the univariate setting. When examining the performance changes according to length, our model showed significant improvement compared to other best models when predicting further into the future (e.g., on average, errors decreased by 5% 96 and 720, and by 12% at 1440 and 2160). These results empirically demonstrate the contribution of our AutoCon in effectively capturing long-term variations that exist beyond the window.
Dataset Analysis Since our goal is learn long-tem variations, the performance improvements of our model can be affected by the magnitude and the number of long-term variations. Figure 5 shows various yearly-long business cycles and natural cycles unique to each dataset. For instance,

---

Published as a conference paper at ICLR 2024
Table 1: Extended long-term forecasting results with various prediction lengths O and the best input
length / E {48, 96, 168, 336} for each model except for Illness dataset with / = 14. Red and blue
numbers denote the best and second-best results, respectively. The full benchmarks with ETThl and ETTm are available in Appendix [
Models
Ours
TinesNet 2023
MICN
2023
PaishTST 2023
DLincar 22033
FILM 2022a
Nonstatioary [022b)
FEDiomeer 20226
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.124
0.260
0.139
0.290
0.122
0.264
0.136
0.292
0.128
0.271
0.129
0.275
0.192
0.343
0.129
0.277
Eleetricity ETT12
720
0.177
0.344
0.2117
0.370
0.313
0.457
0.233
0.392
0.319
0.461
0.256
0.407
0.231
0.394
0.273
0.419
1440
0.176
0.340
0.192
0.358
0.520
0.599
0.351
0.481
0,514
0.597
0.389
0.506
0.211
0.579
0.384
0.487
2160
0.198
0.358
0.263
0.413
0.759
0.734
0.610
0.740
0.728
0.610
0.645
0.399
0.919
0.737
96
0.196
0.313
0.286
0.386
0.241
0.367
0.227
0.336
0.207
0.323
0.394
0.451
0.332
0.426
0.279
0.393
720
0.275
01.386
0.417
0.471
0.336
0.446
0.332
0.426
0.304
0,412
0.467
0.504
0.505
0.533
0.417
0.486
1440
0.338
0.441
0:491
0,419
0.504
0,484
0.625
0.574
0.651
2160
0.380
0.481
0.536
0.547
0.421
0.501
0.768
0.644
0.415
0.496
0.938
0.758
0.642
0.610
0.896
0.714
96
0.132
0.206
0.145
0.219
0.168
0.256
0.192
0.296
0.219
0.327
0.264
0.334
0.247
0.326
0.220
0.312
Trallic
720
0.144
0.225
0.163
0,269
0.304
0.391
0.213
0.318
0,309
0:419
0.247
0.329
0.277
0.360
0.255
0.344
1440
0.174
0.251
0.188
0,292
0.375
0.443
0.246
0.353
0,409
0.311
0,390
0.361
0.297
0.376
2160
0.175
0,252
0.190
0.304
0.360
0,426
0.261
0,383
0.324
0.386
0.988
0.745
0.222
0.317
0.317
D.394
Exchange Weatber
966
0.521
01.22
0.584
0,536
0.569
0,525
0.545
0.539
0.579
01552
0.589
0.533
0.636
0.567
0.703
0.625
720
0.963
0.715
1,060
0.754
1,007
0.706
1.007
0.725
1.114
1440
1.280
0.835
1.547
0.926
1,351
0.863
1.342
0.860
1,299
0.823
1.472
0.900
1.394
0.867
1.435
0.919
1160
1.415
0.857
1.744
0.991
1,544
0.937
1,506
0.924
1.454
0.587
1.712
0.988
1.598
0.944
1.786
1.054
48
D.051
0.172
0.054
0.17%
0.054
0.181
0.068
0.197
0.049
0.170
0.052
0.173
0.054
0.178
0.059
0.184
360
0.448
0.527
0.479
0.459
0.536
0.548
0,531
0.492
0.493
0.528
720
1.067
0.794
1,239
01866
1.383
0.927
1,264
(1859
1.718
1.024
1.291
0.864
1.358
0.894
1.381
0.903
1080
1.004
0.792
1.327
0,900
1.874
1.972
1,255
0.873
4,982
1.973
1.670
1.010
1.774
1.058
1,600
0.980
0.574
1.414
0.815
0.701
1.397
0.901
1.079
1.107
0.698
0.773
ILI1
28
0.887
0.683
1.604
0.854
1660
1.062
1.878
1.110
2,008
1.134
1.315
0.887
1.515
0.767
0.989
0.770
56
0.807
0.725
1.021
0,787
1.757
1,210
1.451
1.022
1,584
1.075
1,080
0.891
0.895
0.742
0.856
0.741
1.499
1.038
1.609
1.072
3,593
1.759
2.846
1.438
3.332
1.572
2,608
1.387
1.724
1.108
1.097
Count
42
0
2
0
0
Electricity
E
Traffic
Weather
0.2
2302
23
0.2
3.03
1.03
-57
395
5.75
6.75
4:53
0.52
6:53
6.50
0.25
6.02
15660
0.02
30600: 12500
0.05
1960e
0.00
33006
42000
Figure 5: The outer-window autocorrelation exists in varying degrees in four datasets.
the ETTh2 and Electricity datasets have strong long-term correlations with peaks at several lags repeated multiple times. Thus, our method on the ETTh2 and Electricity datasets exhibited significant performance gains, which are 34% and 11% reduced error compared to the second-best model, respectively. In contrast, the Weather dataset has relatively lower correlations outside the windows than the aforementioned two datasets. This leads our model to show the least improvement with a 3% reduced error on the Weather dataset. As a result, our method's superiority manifests more strongly for the datasets with stronger long-term correlation, thus empirically validating our contribution.
Extension to Multivariate Forecasting As shown in Table 2 our method is applicable in multi- variate forecasting by calculating autocorrelation per-channel basis and then following a channel independence approach (Nic et al. 2023). Appendix A.2 describes details for multivariate setting.
4.2 MODEL ANALYSIS
Temporal Locality and Globality As mentioned in Section 3.2 we proposed a model architecture that combines the advantages of both linear models for locality and deep models for globality. Figure demonstrates that, for short-term predictions up to 96 units, the linear model (DLinear) achieved lower eиror rate than the deep models such as TimesNet, Nonstationary, and FEDformer. However, the error of DLinear started to diverge as the prediction length extended. Conversely, the TimesNet and Nonstationary maintained a consistent error rate even with the increase in prediction length but didn't perform as well as the linear model for short-term predictions. These observations served as the motivation for our decomposition architecture that is proficient in both short-term and long-term predictions (blue line in Figure a))).
Ablation Studies Here, we conducted ablation studies to validate each component of our method. Figure b(b) shows the results of the ablation study conducted on the full model, and when the
7

---

Published as a conference paper at ICLR 2024
(a) Comparison with baselines
(b) Ablations in our method
Dure
Long
A/D AucoCen Veeg tern hrdistion
(ESifamer (3220)
Weni lone Pridk le
0:2
Vo
ratio
0.05
1/0
620
1:0
3000
30
201
6.30 500
3300
Prediction Length O
Prediction Length O
Comparison of forecasting error (MSE) according to prediction length O. (a) comparison Figure 6:
with baseline models and (b) comparison between ablations of our method on the ETTh2 dataset.
Table 2: Multivariate forecasting results ETT datasets with different prediction lengths O {96, 192, 336, }20} and the input length I = 96. Due to lack of space. we report the averaged performance of four length settings. The full benchmark is available at Appendix D
Models
Ours
TimesNet
2023
MICN 2023
Crossfomer
2023
X-ITSS
2023
DLincar
2023
EtSformer*
2022h
LighTTS
2023
Dataect
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAB
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTh
0.442
0.431
0.458
0,450
0.559
0.535
0.591
0.550
0.502
0.490
0.456
0.452
0.542
0,510
0.491
0.479
ETT2
0.372
0.401
0.427
0.588
0.885
0.673
0.491
0.559
0.515
0.439
0.602
0.543
ETTml
0.390
0.400
0.400
0.406
0.392
0.414
0.503
0.489
0.428
0.436
0.403
0.407
0.429
0.425
0:435
0.437
ET1m2
0.281
0.325
0:291
0.333
0.328
0.382
0.593
0.535
0.346
0.383
0.350
0.401
0.293
0.342
0.409
0.436
denotes the results, which are taken from TimesNet (Wu et al.]
2023))
short-term branch was removed, it showed significant error for short-term predictions. When the long-term branch was removed, it showed a significant error for long-term predictions. Also, without the integration of our AutoCon, the long-term performance was degraded. As demonstrated in Table 3 these trends were consistent across a variety of datasets.
4.3
COMPARISON WITH REPRESENTATION LEARNING METHOOS
29-
-5
-10
-35
19
-25 60
2.5
6. 75
12.
-10
20 -10
AutoCon
LasT
CoST
T52Vec
Figure 7: The figure displays UMAP (ccInnes et al., 2018) visualization results over the representa- tions of different four methods on the ETTh2 dataset. Our AutoCon demonstrates clear continuity and clustering between adjacent months, indicating an understanding of long-term variation. In contrast, the other models appear to lack this perceptible one-year long-term structure, possibly due to the limited representation learning within the window.
We also demonstrate the effectiveness of our method in capturing long-term representations beyond the window compared to existing time-series representation learning methods. TS2 (Yue et al. 2022) and CoST [Woo et al. 2022a) are both unsupervised contrastive learning methods, with TS2 Vec only considering the augmented data of the same time index as positive pair and CoST using loss that takes into account periodicity, but both have the limitation of only being effective within
Table 3: Ablation of the short-term, long-term branch, and AutoCon in Ours.
Datasets
ETThi
ETTh2
ETTm2
Prediction length
96
720
1440
2160
96
720
1440
2160
192
1440
2880
4320
Ours
MSF
0.055
0.078
0.078
0.074
0.125
0.177
0.176
0.198
0.093
0.214
0.211
0.214
w/o Short-term
MSE
0.071
0.093
0.126
0.094
0.204
0.271
0.263
0.302
0.186
0.313
0.300
0.288
Long-term MSE
0.055
0.084
0.093
0.108
0.126
0.242
0.353
0.592
0.093
0.235
0.258
0.326
w/o AutoCon
MSE
0.061
0.082
0.096
0.130
0.147
0.214
0.212
0.236
0.118
0.302
0.254
0.237
8

---

Published as a conference paper at ICLR 2024
Table 4: Comparison with representation learning methods.
Datasets
ETTml
Exchange
ILI
Prediction length
192
1440
2880
4320
48
360
720
1080
14
28
56
112
AutoCon
MSE
0.041
0.090
0.089
0.082
0.051
0.448
1.067
1,003
0.724
0.886
0.810
1.499
LaST (2022)
MSE
0.053
0,120
0.204
0.274
0.051
0.418
2.022
5.529
1.730
2.712
1.694
3,206
CoST (2022)
(2022)
MSE
0.059
0.130
0.199
0.192
0.054
0.451
1.703
4.470
0.746
1.114
1.490
3.155
TS2Vec (2022)
MSE
0.064
0.150
0.171
0.161
0.059
0.516
1.387
5.337
2.161
2.262
3.043
4.098
Therefore, while they show competitive performance in relatively short lengths, they fail to predict accurately for long-term periods. LaST (Wang et al. (0022) is decomposition based representation learning method and also shows competitive performance in short-term predictions, but fails to predict accurately for long-term periods. Figure 7 shows the learned representation by AutoCon with the other three methods. Appendix C.2 provides experimental protocols and further comparison experiments.
window.
4.4 COMPUTATIONAL EFFICIENCY COMPARISON
Our proposed model shows competitive computational efficiency among other deep models. Specifi- cally, on the ETT dataset, our model without AutoCon exhibits computational times of 31.1 ms/iter, second best after the linear models. Even with the integration of AutoCon during training, the computational cost does not increase significantly (33.2 ms/iter) since there is no augmentation process and the autocorrelation calculation occurs only once during the entire training. Consequently, our model's 's computational efficiency surpasses existing Transformer-basee models (Nonstationary 365.7 ms/iter) and recent state-of- oh-the-art CNN-based models (TimesNet 466.1 ms/iter). Detailed comparisons can be found in Appendix B.4
DISCUSSION & LIMITATIONS
Our proposed method mitigates the constraint of the sliding window approach by learning the long- term variations beyond the window. Nevertheless, we examine whether the limitation of the sliding window we point out can be solved by simply increasing the window length without our method, and also elucidate a limitation of our method.
Can we use very long window to capture long-term variations? Given time series S of length T, the number of windows is T (T + O) + 1. This implies that as the input length 7 (i.e., data complexity) increases, while keeping the output length O fixed, the number of data instances (i.e., windows) available for learning decreases, potentially making the model more susceptible to Consequently, it is challenging for input
overfitting (Park et al. 2023) shown in Figure g
DLinear
TimesNet
MICN
Nonstationary Tranformer
FEDiormer
6335
0.10
M15
train test
222
0.49
6335
132
532
032
2:150
Input Length
The figure shows the training (blue line) and test (orange line) MSE loss trace plots of five models as the input length gradually increases from 192 to 960 the ETThl with the output-720 setting. In the figure, the red horizontal line represents our test loss with the input-96 setting.
Figure 8:
sequences to be long enough to cover all long-term variations present in the data, and models often struggle to capture variations outside the window. Therefore, the limitation we point out regarding the sliding window approach is valid in most situations and worth addressing, Appendix [.. presents comprehensive experiments and empirical findings obtained upon increasing the window length.
Can Autocorrelation capture all ferm variations? While autocorrelation serves valuable tool for capturing certain long-term variations, its linearity assumption limits its effectiveness in dealing with the non-linear patterns and relationships prevalent in real-world time series data. By considering higher-order correlations, non-linear dependencies, and external factors, we are likely to achieve even more accurate and comprehensive long-term forecasting.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation Korea (NRF) grant funded by the (3A200IIDAA2011000000 No.NRF-2022R (000022RRA2BBB022000 (2B5BB20019133) Korea government (MSIT) communications Technology Planning & Evaluation (IITP) grant and Institute of Information & funded by the Korea government (MSIT) (No.2019-0-000) No.2022-0-00984).
REFERENCES
Ahmad S Ahmad, Mohammad Y Hassan, Md Pauzi Abdullah, Hasimah A Rahman, F Hussin, Hayati Abdullah, and Rahman Saidur. review on applications of ann and svm for building electrical energy consumption forecasting, Renewable and Sustainable Energy Reviews, 33:102-109, 2014.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01277 2018.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2023.
Chris Chatfield and Haipeng Xing. The analysis of time series: introduction with R. CRC press, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proc. the International Conference Machine Learning (ICML), pp. 1597-1607. PMLR, 2020.
Robert B Cleveland, William Cleveland, Jean E McRae, and Irma Terpenning. Stl: A seasonal-trent decomposition. Off. Stat, 6(1):3-73, 1990.
Lars Dannecker. Energy time series forecasting: efficient and accurate forecasting of evolving time series from the energy domain. Springer. 2015.
Rui Ding, Qiang Wang. Yingnong Dang, Qiang Fu, Haidong Zhang, and Dongmei Zhang. Yading:
Fast clustering of large-scale time series data. Proceedings of the VLDB Endowment, 8(5::773-444, 2015.
Lean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. 32, 2019.
Rob J Hyndman, Earo Wang, and Nikolay Laptev. Large-scale unusual time series detection. In 2015 IEEE international conference on data mining workshop (ICDMW), pp. 1616-1619. IEEE, 2015.
Muhammad Faisal Iqbal, Muhammad Zahid, Durdana Habib, and Lizy Kurian John. Efficient prediction of network traffic for real-time applications. Journal of Computer Networks and Communications, 2019, 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. 33: 13:18661-18673, 2020.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In Proc. the International Conference on Learning Representations (ICI.R), 2021.
Nowrouz Kohzadi, Milton S. Boyd, Bahman Kermanshahi, and lebeling Kaastra. A com- parison of artificial neural network and time series models for forecasting commodity prices. Neurocomputting, 10(2):169-188 1996. ISSN 0925-2312. doi: https://doi.or// 10.101660922-23122955000000 URL / httpp::/// /tt:://wwww... sci encedi rect. creettccmo..e.ccc.ooo.o.o....oot sciencee Financial Applications, Part I.
Vincent Le Guen and Nicolas Thome. Shape and time distortion loss for training deep time series forecasting models. Proc. the Advances in Neural Information Processing Systems (NeurIPS), pp. 4191 4103,4203, 2019.
10

---

Published as a conference paper at ICLR 2024
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2999-3007, 2017. URL http:/ http::///ppacccsss..ooo thecvf. com/ content ICCV 2017/ papers/Li LLn papers/Lin_Foca Focal Loss_ facass__or_rcc____
paperp pcpe_____cc___0__pppper_- pdd]
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In Proc. the International Conference on Learning Representations (ICLR), 2022a.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long, Non-stationary transformers: Rethinking the stationarity time series forecasting. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2022b.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint / 2018.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In Proc. the International Conference on Learning Representations (ICLR), 2023.
Athanasios Papoulis and S Unnikrishna Pillai. Probubility, random variables and stochastic processes. 2002.
Junwoo Park, Jungsoo Lee, Youngin Cho, Woncheol Shin, Dongmin Kim, Jaegul Choo, and Edward
Choi. Deep imbalanced time-series forecasting via local discrepancy density. In Proc. of Machine
Learning and Knowledge Discovery in Databases: Research Track (ECML/KKDD), pp. 139
155. Springer Nature Switzerland, 2023. doi: 10. 3-1199788---011-444--2222 9. URL https 7/Link. spri nger. www./////ooe////eooooooo
0 007 9788-/9778-1 -331--4433331-4444---0 43421--4444---
Thanawin Rakthanmanon, Bilson Campana, Abdullah Mueen, Gustavo Batista, Brandon Westover, Qiang Zhu, Jesin Zakaria, and Eamonn Keogh. Addressing big data time series: Mining trillions of time series subsequences under dynamic time warping. ACM Transactions on Knowledge Discovery from Data (TKDD), 7(3):1-31, 2013.
Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing, 26(1):43-44 1978.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2/06.0000 2021.
José F Torres, Dalil Hadjout, Abderrazak Sebaa, Francisco Martinez-Alvarez, and Alicia Troncoso. Deep learning for time series forecasting: a survey, Big Data, 91):3-21, 2021.
Eleni Vlahogianni, Matthew G Karlaftis, and John C Golias. Short-term traffic forecasting: Where we are and where we're 're going. Transportation Research Part C: Emerging Technologies. 43:3-19, 2014.
Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Mien: Multi- scale local and global context modeling for long- term series forecasting. In Proc. the International Conference on Learning Representations (ICLR), 2023.
Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2022.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. 2022a.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting, arXiv preprint 2-Xiv:2202.01381 2022b.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto correlaornl for fong-term series forecasting. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2021.
11

---

Published as a conference paper at ICLR 2024
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d variaiationn modeling for general time series analysis. In Proc. the International Conference Learning Representations (ICLR), 2023.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), volume 36. pp. 8980-8987, 2022.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2023.
Kaiwen Zha, Peng Cao, Yuzhe Yang, and Dina Katabi. Supervised contrastive regression. arXiv preprint arXiv:2210001189, 2022.
Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-orienter mip structures. arXiv preprint arXiv:2207.01 186, 2022.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In Proc. the International Conference on Learning Representations (ICLR), 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proc. the AAAI Conference Artificial Intelligence (AAAI). 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, al. Film: Frequency improved legendre memory model for long-term time series forecasting. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. the International Conference on Machine Learning (ICML), 2022b.
A REPRODUCIBILITY
A.1 DETAILS OF OUR METHOD IMPLEMENTATION
In this subsection, we provide a detailed explanation of the operations that were used Section 3.2 Firstly, Linear signifies linear layer along the time dimension. When provided with an input sequence X, the output y is computed
-= Wiime X
(9)
where E RIxe, BOxe and Wiime E BOXI
Next, we describe Paddingg-) and Augpool(.) (.) along with the time axis as follows:
Xpad = Padding(2))
(10)
Xaog Augpool (Xpad)
(11)
Xavg [t.
Xpaalt [1 + c]
(12)
where Xpad E (T++2((-1))xe is padded sequence on both sides with neighboring values to preserve input length after applying Augpool(-). The various kernel sizes of Augpool(-) are selected to handle the multi-ppriodicittes, which are observed in real-world time-series data.
M T.P denotes the use of two linear layers with an activation function g. specifically GELU, Given the input sequence X, the output y is computed as:
5 g @ time X brime) W Wehannel behannet
(13)
12

---

Published as a conference paper at ICLR 2024
where Dtime RR, Dehannel E R", Wime E ROXT and W channel R
To compute AutoCon, global autocorrelation must first be determined. As part of the preprocessing process, calculate autocorrelation for the entire training series, excluding validation and test series. This autocorrelation comprises both long and short-period variations. However, our pursuit of disentangled long-term representation, frequency variations with long periods. Therefore, we smooth out short-period fluctuations in the series before computing the autocorrelation.
intend that the long-term branch address only low-
Our training protocol is identical to conventional training methods, except for the inclusion of AutoCon additional loss, in addition the forecasting loss. To clarify this further, also present the algorithm.
Algorithm 1 AutoCon: Autocorrelation based Constrastive Learning Framework
Require: Entire time series S = (si,..... ST). ..........7... Training set D {(Ae, Vi), Te) AutoCon weight where T denotes total length and M denotes the number windows.
Compute the global autocorrelation Rss(() for all possible lags E [M]
for all number of training iterations do Sample mini-batch {((K,, Di.). Tn) from D Forward ( Tn > and get corresponding representation (u) and predictions (M Compute window relationship matrix r E RNN,, ((,i)) Rss(( (T. -
Compute AutoCon loss LAAutCCon following Equation aas inputs (n(m) J== and Compute forecasting loss Lmse N > > ()), vn,) Do training step using the full loss L Lince X LAutoCon
end for
Our redesigned model and AutoCon were implemented based on the TSlib code repository Our source code can be accessed at zip file in the supplementary.
A.2 DETAILS OF MULTIVARIATE FORECASTING
There are two representative approaches to multivariate forecasting: Channel-mixing and Channel- independence approaches. The channel-mixin approach involves mapping the values of multiple channels at the same step into an embedding space and extracting temporal dependencies from this embedding sequence. This approach has been adopted by various papers (Zhou et al.) 2022b Wu et al.] 2023 Zhang & Yan 2023). The chamneliindependence approach, on the other hand, preserves each channel's information without mixing them and learns temporal patterns within each channel independently. Recently, this method has been adopted in high-perfomme as PatchTST (Challu et al. (023) and Linear models (Zeng al. (003)). demonstrating superior performance on current benchmark datasets. In terms of implementation, each channel is treated as a batch axis for computations. This effectively increases the amount of training data by the number of channels, and the model parameters are shared across multiple channels. Following the channel- independence approach, we first compute autocorrelations for each channel separately in order to calculate AutoCon. We then train the representation tailored to each channel based these autocorrelations.
models such
A.3 DETAILS OF DATASETS
In this paper, we utilized six real-world datasets from diverse domains: mechanical systems (ETT), energy (Electricity), traffic (Traffic), weather (Weather), economics (Exchange), and disease (ILI). The statistics of each dataset are summarized and found in Table As mainstream benchmark, the ETT datasets extensively utilized for assessing long-term forecasting methods Zhou al. (202T):
Wu et al. (2021) Zhou et al. 2022b): Zeng et al. (2023) Wu et al. 2003). ETT consists of critical
https://gthuh..om//umm///// http:///ww.c..../mmm/miiisiiiiia
13

---

Published as a conference paper at ICLR 2024
Table 5: Statistics of nine datasets.
Dataset
Domain
Time series length
The number of variables
Sampling frequency
ETTh1
System Monitoring
17420
Hour
ETTh2
System Monitoring
17420
7
1 Hour
ETTml
System Monitoring
69680
15 minutes
ETTm2
System Monitoring
69680
15 minutes
Electricity
Energy Consumption
26304
321
1 Hour
Traffic
Traffic
17544
862
Hour
Weather
Weather
52695
21
10 Minutes
Exchange
Economic
7588
Day
11.1
Medical
966
7
I Week
indicators (such as oil temperature, load, and others) that are gathered over span of two years from electricity transformers. These datasets are grouped into four distinct sets based on location (ETTI and ETT2) and time interval (15 minutes and one hour). The Electricity dataset captures the hourly electricity consumption of 321 customers from 2012 to 2014. On the other hand, the Traffic dataset compiles hourly data from California Department of Transportation, detailing the occupancy rates of roads measured by different sensors freeways in the San Francisco Bay area. The Weather dataset consists of 21 metcorological indicators, including air temperature, humidity, and others, recorded at 10-minute intervals the course of year. The Exchange dataset chronicles daily exchange rates of eight different countries from 1990 through 2016. Lastly, the IIII dataset includes weekly records of influenza-like illness (ILI) patient data from the Centers for Disease Control and Prevention in the United States, spanning from 2002 to 2021. This dataset illustrates the ratio of patients diagnosed with ILI relative to the total patient count.
A.4 BASELINE MODELS
In the realm of long-term forecasting, numerous models have been proposed since the advent of Informer. These models have demonstrated commendable performance with unique novelties. However, they were compared with underperforming models such as the models based on RNNs, and Transformer- based models, which are known to be susceptible to overfitting. Therefore, our primary focus on
and state-of-the-art models among the most recent proposals. We validate our method against seven forecasting baselines and three representation methodologies.
All models were implemented using PvTorch. For the latest forecasting models, namely TimesNef" DLinear and NLinear) MICN F Nonstationary Transforme and FEDformer we utilized the official code released by the original authors rather than implementing it from scratch.
Similarly, for recent representation methods, such as LaST CoST and TS2Ved official codes provided by the authors instead of implementing models from scratch. We adhered to the unique hyperparameters of each model, tuning within the parameter search range that yielded optimal performance. However, certain configurations, such as input length and output length, were uniformly for ease of comparison. More specific evaluation protocols will be presented in the following section.
utilized the
htanss/github..om//uum///iineee Seriee--eeeeeee-eeeeeeeeeeeee Series-Library
Thttss//jiillbbcccmmmm/////sssnt Transformers
"uapss//gthhbbccmm//tyysss11
14

---

Published as a conference paper at ICLR 2024
A.5 EVALUATION DETAILS
In our experiments, we aim to assess the model's capability to capture long-term variations, so that the output length should be long enough to be affected by these variations. Increasing the output length more than those used in previous experiments, however, involves several considerations. Thus, we delineate the modifications the standard evaluation protocol as follows:
1. The input length 7 is set to 14 (for the III dataset), 48 (for the Exchange dataset), 192 (for the ET'm dataset), and 96 (for the others datasets). These input lengths allow us to increase the output length within the limited window length, in accordance with the total length of each dataset.
2. The standard protocol splits all datasets into training, validation, and test sets in chronological order, with a ratio of 6:2:2 for the ETT dataset and 7:1:2 for the remaining datasets. However, due to the increased window length in the other datasets, the validation set is insufficiently populated. Therefore, we adopt a ratio of 6:2:2 for all datasets.
3. The weather dataset contains negative values for indicators that should logically be non- negative. These erroneous labels, if not corrected, could impede accurate evaluation due to scaling issues. We rectified these errors by filling them with neighboring values.
We adhered standard protocols for all experiments, barring the exceptions outlined above.
A.6 HYPERPARAMETER SENSITIVITY
MSE
MAE
0.04
Reduction Rate
Reduction Rate
0.05
-0.02
-0.44
ETTh1
Weather
-0.0
Exchange
ECL
llness
0.0
16.4
10:3
1c-2
16-1
1.0
0.0
1o-4
10-3
10.2
1o-1
1.0
Lambda
Lambda
Figure 9: Error reduction rates of seven datasets according to the lambda value of AutoCon: (Left) Mean squared error and (Right) Mean absolute error. The error reduction rates were calculated based on the lambda. which is set 0.0 the reference point. The best performances in all datasets were achieved when using our AutoCon (denoted the star markers). is important to note that a high lambda value does not necessarily imply strong long-term variations.
B ADDITIONAL EXPERIMENTS
B.1 EXPERIMENTS ON WINDOW LENGTH
As mentioned in Section we considered simply increasing the length of the window to capture the long-term variations as much as possible. Also, as the window length increases, the number of data windows used for learning decreases. After all, we hypothesize that increasing the window length increases the input complexity of the model, while reducing the number of data points, making the model vulnerable to overfitting.
Figures 10 and II depict the training and testing losses when the input length increase from 192 to 920 for the purpose of predicting 720 steps in ETThl and ETTh2, respectively. We observed that the overall test loss tends to soar or converge, while the training loss persistently decreases when the input size is increased in five models with varying capacities and attributes.
15

---

Published as a conference paper at ICLR 2024
DLinear
TimesNet
MICN
Nonstationary Tansformer
FEDrommer
0.12
2.30
3.38
train
6.9
032
0315
OUTS
6.50
3.53
Input Longth
The figure shows the training (blue line) and test (orange line) loss trace plots of five models as the input length gradually increases on the ETTh1 with the output-720 setting. In the figure, the red horizontal line represents the performance of our model with the input-96 setting.
Figure 10:
DLinear
Timeshett
MICN
Nonstationary. Tansformer
FEDrorerr
0.7
0.500
0122
022
3:2
333
test
0.20
0.15
3:3
OUTS
037
Length
The figure shows the training (blue line) and test (orange line) loss trace plots of five models as the input length gradually increases the ETTh2 with the output-720 setting. In the figure, the red horizontal line represents the performance of our model with the input-96 setting.
Figure 11:
Also, in the case of DLinear in Figure 11 both test and training losses decline in sync due to limited capacity. However, regard it as underfitting problem since test errors higher than method (see red line). Consequently, we empirically substantiate that merely increasing the input length does necessarily enhance the performance of long-term forecasting. Moreover, it is noteworthy that the computational cost for complex models other than linear models increases significantly with increasing sequence length.
B.2 ADDITIONAL FIGURE RESULTS ON OTHER DATASETS
Additionally, we provide results for ETTh1 (Figure 12 and Electricity (Figure 13). which show the long-tern long-term variations. Although there some differences in magnitude, the overall trends are similar across the three datasets.
(a) Comparison with baselines
(b) Ablations in our method
Clane 20331
3.14
3300
snu Stxtee
0.20
Hanokumnnn 12022)
012
Autsionn
5350
Longg dene Pradits
M
1.50
rablo
I/0 ratio
Prediction Length
Length
Figure 12: Comparison of forecasting error (MSE) according to prediction length O. (a) comparison with baseline models and (b) comparison between ablations our method on the ETTh1 dataset.
16

---

Published as a conference paper at ICLR 2024
(a) Comparison with baselines
(b) Ablations in method
0.4
68
0.5
0.0
0.4
83
3050
0.2
Duinee (X01))
0.2
Out
Norata share (3322)
0.4
1/0 ratio
pncioomee (3200)
V/ ratio
deee Prodition
loves Pediation
33
200
400
200
603
2000
230
400
600
300
100S
Prediction Length O
Prediction Length O
Figure 13: Comparison of forecasting error (MSE) according prediction length O. (a) comparison with baseline models and (b) comparison between ablations of our method on the Electricity dataset.
Table 6: Increasing complexity of long-term term branch in DLinear.
KTThl
ETTh2
Electricity
Model
of layers
MSE
MAE
MSE
MAE
MSE
MAE
DLinear
0.1780:0.0054
0.346620000063
0.292940.0140
0.4362+0.0087
0.306740.01544
0.4125+0.0109
2
0.199310.1964
0.2638&0.2191
0.317010.0488
0.458140.0404
0.377510.01574
0.461810.0081
3
0.279320000678
0.459420.0682
0.300820.0046
0.445120.0035
0.305720.09902
0.409720.0065
0,2760=0.0663
0,4564+0.0683
0.3015+0.0031
0,4455+0.0024
0,3165#0.03838
0.4183+0.0279
B.3 ADDITIONAL ABLATION RESULTS FOR LONG-TERM BRANCH
Increasing the complexity of the long-term branch is essential for learning long-term representations, but it is not the sole reason for the superiority of our methodology. In other words, even with the increased complexity, capturing long-term variation is not easy in the current framework that uses only forecasting loss. As the main contribution, using AutoCon is essential for learning long-term variation and leading to performance improvement. To verify this, we additionally present two ablation results: increasing complexity in DLinear and in our model.
First, DLinear utilizes only a single linear layer for both long-term and short-term branches. We increase the complexity of the long-term branch by stacking linear layers with an activation function in the long-term branch. However, as evident in Table even when stacking layers in the long term, performance tends to decrease remain similar. This demonstrates that increasing long- complexity is not effective in the existing decomposition architecture.
Second, the following Table 7 demonstrates the performance changes based the complexity of the long-term branch in our decomposition architecture. Without Autocon, our model may be slightly better or comparable when AutoCon is employed. This further underscores the necessity of the AutoCon we proposed to accurately predict long-term variation.
the second-best model. The highest performance is achieved only
B.4 ANALYSIS OF COMPUTATIONAL COST
Given the real-time nature of most time-series applications, computational efficiency is crucial factor in time-series forecasting (Dannecker) 2015 Iqbal et al.) 2019 Torres al., (021). As forecasting horizons increase, the window length expands, leading to increased computational costs. Therefore, it is imperative evaluate model's computational efficiency. Figure illlstraat the time required to update the parameters by single batch for our model in comparison with the baseline models. The computational cost was measured for four different output lengths, ranging from 96 to 2160, for each model. A batch size of 32 was used, and all measurements were taken independently in the same GPU and server environment. Firstly, the linear model took the least amount of time due to its minimal number of parameters and the simplicity of matrix multiplication operations. On the other hand, TimesNet required the most time as it extracts multiple periods and computes a loop for each period. The Nonstationary model, which is based on the Transforner, has computational complexity of ( in relation to length, which explains the sharp increase in computation time with length. Overall, our model was the second fastest after the linear model, and its computation cost did not
17

---

Published as a conference paper at ICLR 2024
Table 7: Increasing complexity of long-term branch in model.
Model
layers
AutoCon
ETThl
MSE
MAE
TimesNet (baseline best)
0.082410.0024
0.23 0020
Ours
X
0.083720.01885
0.2372+0.022
Ours
0.091010.0188
0.236010.0257
Ours
0.0876+0.0240
0.2351=0.0303
Ours
X
0.0918+0.0194
0.2381=0.0241
Ours (best)
O
0.078720.002
0.2226 10.0023
ETTh2
Model
of layers
AutoCon
MSE
MAE
TimesNet (haseline hest)
0.2074x0.0113
0.370340.115
Ours
X
0.2023=0.0230
0.3594
Ours
X
0.2020+0.0096 0.3525+0.0078
Ours
X
0.203620.0265 0.3573+0.0222
Ours
0.2087a0.0167 0.360510.0130
Ours (best)
0.3441#0.0366
Model
of luyers
AutoCon
Electricity
MSE
MAE
Diinear (buseline best)
0. 0.3067=0.0154
0.4125+0.0109
Ours
0.292840. 1369
0.397840 1009
Ours
2
0.288910.05300
0.392740.0200
Ours
3
0.2975:0004555
0.404940.000
Ours
4
0.3089+0.4777
0.4115+0.0555
Ours (best)
2
0.2753+0.0224
0.3861#0.0166
96 96
720
1440
2160
355-7 malter
Computational Cost (msitee)
200. motter
matter
11.1 mi ber
MLinear
Diimeer
FILM
Timestor
MRON
Honstatioary
FEDiemer
Inu AutoConn
Durs (wib AutoCani
Figure 14: The figure illustrates the comparison of computational costs, measured milliseconds per single batch iteration (ms/iter), of the bascline models and our model. The computational cost for each model was evaluated by increasing the output length from 96 2160.
significantly increase even when training included AutoCon (from 31.1 ms/iter to 33.2 ms/iter). Hence, method manages achieve superior long-term forecasting performance compared to
18

---

Published as a conference paper at ICLR 2024
linear models, while requiring less computational resources than other more complex models. The cost analysis was briefly addressed in Section 4.4 of the main paper.
B.5 VISUALIZATION OF FORECASTING RESULTS
Nilnear (2027), MSSEO 9293. Thend
(2023), 3168
Timmndtien 12003
Monstalorary 12022),
Tradd
Figure 15: The figure displays the visualization of forecasting results for the 96- 1440 setting in ETTh2, showcasing our model along with four other models.
Figure 15 provides qualitative result of the five different models by visualizing the prediction results for 1440 steps in the ETTh2 dataset. In the of linear models, the error increases the prediction distance increases, failing to account for long-term variations. Nonstationary and TimesNet models, although better following long-term variations than the linear model, struggle capture high-freet nigh-frequency patterns effectively. Our model, on the other hand, successfully manages to capture both long-term variations and high-frequency patterns. This can be attributed to our model's structure, which is designed to benefit from both short-term and long-term predictions.
B.6 EVALUATION RESULTS WITH OTHER METRICS
While existing metrics (ie.. MSE and MAE) are standard metrics for long-term forecasting evaluation, they have limitations. Specifically,, they may adequately capture aspects such as the shape and temporal alignment of the time series, which are crucial for comprehensive evaluation of forecasting model's 's performance.
To address these limitations, we introduced two additional metrics based on Dynamic Time Warping (DTW) (Sakoe & Chibal 1978): Shape DTW and Temporal DTW (Le Guen & Thome 2019). Shape DTW focuses on the similarity of the pattern or shape of the predicted sequence to the actual sequence, providing insight into the model's ability to capture the underlying pattern of the time series. Temporal DTW evaluates the alignment of the predicted sequence with the actual sequence, highlighting the model's accuracy in forecasting the timing of events.
These additional metrics offer more nuanced assessment of our model's performance, particularly in areas where MSE and MAE may fall short. Lower values in both Shape DTW and Temporal DTW indicate better performance, signifying lesser distortion between the predicted and actual sequences. As shown in Table S our method demonstrates superior performance not only in MSE and MAE but also in these shape and temporal tiigmment-focuee metrics.
19

---

Published as a conference paper at ICLR 2024
Table 8: Univariate forecasting results, evaluated using Shape and Temporal DTW in 720-output setting across three datasets
Dataset
ETThi
ETTh2
Electricity
Model Metric
Shape DTW
Temporal DTW
Shape DTW
Temporal DTW
Shape DTW
Temporal DTW
Ours
17.14
1..6621.739
42.3840.630
13.47=1.793
80.7347.798
0.09
TimesNet
25.80+4.349
86 88.86.61.5000
62,58+5.390
51,19 834
139 199883111
0,49+0.88 826
PatchTST
22.21
72.23=a4.411
65.45
15.23=00.822
116.50
0.7551.674
MICN
37.08=12.393
65. +5.70+3,588
67.69+8.799
22.67=3.16
123.93+17,543
1.90
DLinear
58.32+ 1955
155.21
82.53+3.627
24. 24.88+2.4403
88.70+3.550
0. 18+000 145
C ANALYSIS OF REPRESENTATION
C.I DETAILS OF REPRESENTATION SIMILARITIES
In Figure 2 we used three baselines with our model and extracted the representations of each baseline either before the final projection layer (TimesNet) or after the encoder layer (ST FEDformer, and Ours). Our main purpose in visualizing the representation long-term correlations. To display more clearly, we applied a filtering method to smooth short-term fluctuations within the given window. We also provide original representation results, which enlarged for each baseline, without smoothing out the short-term fluctuations as shown in Figure Figure shows that the three baseline models learn the short-term correlations within the window, although they do not learn the long-term correlations.
to demonstrate the learning of
One interesting point this finding is that existing models have attempted address the limitations of window length by leveraging time-stapp information. Actually, TimesNet, FEDformer, and our model obtained the representations using timestamps, incorporating them into the input sequences, while PatchTST does not utilize the timestamps. However, not only PatchTST but also both TimesNet and FEDformer do not effectively capture the annual eyclic patterns, despite utilizing timestamps as the same as our model. These failures are noteworthy, particularly considering the Electricity time series, which displays a yearly-long periodicity. These results show that it challenging for the model to learn yearly patterns even when given input sequences and timestamps, solely relying on the existing forecasting loss. Therefore, this result demonstrates the necessity of our AutoCon loss. Furthermore, justify the emergence of long-term representation irrespective of the model's structural aspects, we provide additional results of the representation from an ablation model that does utilize AutoCon in our model. As shown in Figure 17 the model without AutoCon also exhibits a weak periodicity, but similar to other baselines, representation similarity remains relatively flat compared to our full model.
C.2 VISUALIZATION OF REPRESENTATIONS
We benchmarked our AutoCon method against three representation learning methods proposed to enhance forecasting performance. TS2Vec and CoST have a two-stage learning framework in which they utilize a ridge regression model for time-series forecasting and deep learning- based models for representation learning. On the other hand, LaST and our method adopt end-to-end learning framework wherein both representation and time-series forecasting learning occur concurrently.
Figure 7 presents the representation results of four methods over the ETTh2 dataset. To investigate whether each method has learned the representation structure associated with the long-term variations, we extracted representations corresponding to all training time steps and visualized them via UMAP with the month labels derived from the timestamp. Our model clearly displays continuity between adjacent months and demonstrates well-defined clustering, attributes not seen in the other models. It seems that the other models did not learn the structure necessary for recognizing one-year long-term variations beyond the window, their representation learning confined within the window.
20

---

Published as a conference paper at ICLR 2024
W1
Yearly-long periodicity
W2
W3
Windoow Indien Anchor Window Anchor index
Input
Output
1 S Similanty
Sim (wm, W1
Sim (We,
PutchiST
Figure 16: (Top) Electricity time series including long-term variation beyond window size. (Bottom) Plotted representation similarities three baseline models between an anchor window W2 and all other windows including Wi and W3.
W1
Yearly-long periodicity
W2
W3
Electtity
Windon
Window Index
Input
Output
Smiaanty
Fim(wz,We W4
Sim ( W3
Our
Ours AutoCon
Time (window
Figure 17: (Top) Electricity time series including long-term variation beyond window size. (Bottom) Plotted representation similarities of our models (with AutoCon and without AutoCon) between an anchor window Wi and all other windows including W1 and Wg.
C.3 ADDITIONAL. COMPARISON WITH TWO SELF-SUPERVISED LOSS
We designed and provided results for two possible self-supervised objectives based on HierCon (Yuc) al. 2022) and SupCon (Khosla et al. 2020) that can be incorporated into our two-stream model structure. HierCon induces the representations of two partially overlapped windows to be close to each other, while SupCon encourages the encoder learn close representations for the windows with the same month label.
The two SSL objectives were tested with our model architecture, replacing only the AutoCon loss. shown in Table compared to the one without any SSL loss, HierCon shows slight improvement in performance on the ETTh] and ETTh2 for short-term predictions with a length of 96, but performs worse in the long-term prediction, leveraged monthly information beyond the window length, leading to performance improvements in the long-term prediction. However, SupCon only learn single predefined periodicity, unlike AutoCon. Consequently, SupCon shows lower performance than AutoCon learning the periodicities existing in the time series through autocorrelation.
it emphasizes only temporal closeness. SupCon
21

---

Published as a conference paper at ICLR 2024
Table 9: Comparison with different self self-supervised objectives in our redesigned architecture.
Datasets
ETThi
ETTh2
Electricity
Prediction length
96
720
1440
2160
96
720
1440
2160
96
720
1440
2160
MSE
0.061
0,082
0.096
0.130
0.147
0.214
0.213
0.236
0.206
0.289
0.363
0.419
w/o SSL
MAE
0.190
0.226
0.245
0.286
0.285
0.375
0.365
0.372
0.322
0.393
0.460
0.503
MSE
0.059
0.090
0.121
0.145
0.132
0.221
0.221
0.263
0.223
0.313
0.380
0.500
Ours w/ HierCon
MAE
0.187
0.240
0.276
0.306
0.279
0.379
0.378
0.407
0.339
0.407
0.474
0.544
MSE
0,056
0.082
0.092
0.091
0.125
0,185
0.199
0.215
0.209
0.279
0.351
0.408
Ours SupCon
MAE
0.184
0.229
0.242
0.237
0.270
0.349
0.360
0.371
0.326
0.388
0.451
0.489
Ours w/ AutoCon
MSE
0.056
0.079
0.074
0.124
0.177
0.176
0.198
0.196
0.275
0.338
MAE
0.182
0.223
0.225
0.215
0.269
0.344
0.340
0.358
0.313
0,386
0.441
0.481
Table 10: Multivariate forecasting results on ETT datasets with different prediction lengths O (96, 192, 336, 720} with the input length 7 = 96. The full benchmark is available at Appendix D
Models
Ours
TimesNer
(2003)
(2023)
Crasformer
(2023)
N.HITS (2025)
(2023)
Dlinear*
ETSfomeer
(2022)
LighTSS* (2022)
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.387
0.396
0.384
0.402
0.421
0.431
0.407
0.429
0:404
0.424
0.386
0.400
0.494
0.479
0.424
0.432
192
0.137
0.424
0.436
0.129
0.474
0.187
0.505
0.196
0.165
0.466
0.137
0.132
0.538
0.504
0.175
0.462
336
0.476
0.442
0,491
0.469
0.569
0.551
0.620
0.574
0.519
0.501
0.481
0.459
0.574
0.521
0.518
0.488
720
0.468
0.461
0.521
0.500
0.770
0.672
0.830
0.701
0.621
0.569
0.519
0.516
0.562
0.535
0.547
0.533
Average
0.442
0.431
0.458
0.450
0.559
0.535
0.591
0.550
0.502
0.490
0.456
0.452
0.542
0.510
0.491
0.479
01.290
0.341
0.340
0.374
0.299
0.364
0.645
0.562
0.346
0.381
0.333
0.387
0:340
0.391
0.397
0.437
192
0.373
0,398
0,402
0.414
0.441
0.454
0.788
0.636
0.427
0.4-1
0.477
0.176
0.430
0.439
0,520
0.504
336
0.408
0.423
0.452
0.452
0.654
0.567
0.959
0.709
0.518
0.500
0.594
0.541
0.485
0.479
0.626
0.559
720
0.419
0.442
0,462
0.468
0.956
0.716
1.146
0.784
0,888
0.645
0.831
0.657
0.500
0.497
0,863
0.672
Averago
0.372
0.401
0.414
0.117
0.588
0.525
0.885
0.673
0.545
0.491
0.559
0.515
0.439
0.452
0.602
0.543
96
0,330
0.365
0.338
0.375
0.316
0.362
0.364
0.399
0.355
0.389
0.345
0.372
0.378
0.398
0.374
0.400
577ml
192
0.374
0.363
0.390
0.404
0.389
0.400
0.407
336
0.399
0.408
0.410
0.411
0.408
0.426
0.517
0.488
0.452
0.456
0.413
0.413
0.435
0.428
0.438
0.438
720
0.460
0.444
0.478
0.450
0.481
0.476
0.698
0.627
0.500
0.485
0.477
0.453
0.499
0.462
0.527
0.502
Average
0.400
0.400
0.392
0.114
0.503
0.489
0.428
0.407
0.429
0.435
0.437
m17m2
96
0.178
0.260
0.187
0.267
0.179
0.275
0.272
0.357
0.201
0.287
0.193
0.292
0.189
0.280
0.209
0.308
192
0.244
0,303
0.249
0,309
0.307
0.376
0,335
0,414
0.295
0.354
0,284
0.362
0.253
0.319
0.311
0.382
336
0.321
0.325
0.388
0.359
0.369
0.127
0.314
0.112
720
0.398
0.396
0.408
0.403
0.502
0.490
1.203
0.779
0.530
0.498
0.554
0.522
0.414
0.413
0.675
0.587
Average
0,281
0.325
0.291
0.333
0.328
0.382
0.593
0.535
0.346
0.383
0.350
0.401
0.293
0.342
0,409
0.436
denotes the results are taken from TimesNet.
D FULL BENCHMARKS
Table 10 and Table show the full benchmark results including confidence intervals.
22

---

Published as a conference paper at ICLR 2024
Table 11: Extended long-term forecasting full benchmarks with confidence interval. 'S'hon and 'L' ong indicate the length of the conventional experiment and the newly extended experiment setting, respectively. Nonstationary has an out- memory (OOM) problem at the output-4320 setting on
ETTml and ETTm2 datasets.
Modets
Outs
TimesNet (2023)
MICN (2025)
PacNISTT (2025)
DLinear (21125)
FILM
Nonsuaionary (2022)
REDiormer (21122
1-+
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.886
0.182
0.068
0.185
0.062
0.185
0.057
0.184
0.063
0.155
0.057
0.180
0.059
0.197
0.050
0.218
030014
43403
-03000
50807
+400
2000
+1.833
+3086
04.300
44.00
+30014
720
0.879
0.225
0
0.230
0.175
0.342
0.089
0.236
0,180
0.348
0.057
0.245
0.137
0.272
0.130
0.285
20191
331877
20003
200
300833
20005
13/00
20090
2590
23001
25000
200500
15877
230m
1440
0.879
0.008
0.476
0.115
0.275
0.433
0.567
0.125
0.000
0.134
0.309
0.199
0.356
031110
20100
00000
400005
-0000
331014
54.000
25001
0.874
0.143
0.505
0.677
0.183
0.355
0.629
0.187
0.359
0.504
0.356
0
200100
20000
330079
2001
30000
-00103
18011
20175
11.000
53000
10.000
15.99
20.045
18905
13/019
0.124
0.139
0.290
0.364
0.136
0.292
0.125
0.129
0.275
0,343
0.129
730
0.177
0.344
0.207
0.570
0.313
0.457
0.233
0.392
0.319
0.461
0.256
0.40
0.231
0.394
0.73
0.419
002940
45012
1440
20000
0.176
0.340
0.192
0.358
0.520
0.59
0.353
0.481
0.514
25301
0.597
0.589
23000
0.506
0.211
0.379
280m
0.354
0.487
2160
0.198
0.358
0.363
0.418
0.790
0.734
0.610
0.659
0.740
0.728
0.60
0.645
0.240
0.395
0.919
0.717
04181
25:000
01800
5000
-30.3
56.00
1500
$8.355
192
200m
0.042
0.157
23100
0.044
200m
0.161
201011
-0080
0.045
0.160
4019
-1085
0,839
+4/0n
6.150
-10m
0.045
21:00
0.156
0.041
2100
2400
0.14
0.058
21.00
2000
0.181
0.056
0,300
0,090
0,085
0.207
0.09
0.255
0.00
0.27
0.305
0.22
0.68
0.257
0.142
0,20
0.093
0,228
20010
13/04
20009
20208
2000
-000
1307
2000-
11:83
13000
15:04
18004
130813
0.089
0,002
0.60
0150
0.285
0300
0.26
0.125
0.344
0,101
0.50
0151
0,339
0.127
0,083
0,000
0341
0200
0.413
-0094
0.100
0.255
18000
0.371
0.19
10000
0.25
00M
00M
0.167
18012
1371m2
1440
192
20109
2000
0.215
0,043
0.211
0.362
6:370
231001
13180
0.237
12228
12226
0,102
0.400
59000
0:91
22:00
00243
0.085
10100
0:22
5000
0.665
0.23
4000
0388
2000
0:336
0,004
0.233
0.366
0.231
550m
0.300
2008
6:337
0.321
2000
0,013
25.00
0.804
0.384
0.230
0.35
23008
0,202
0.096
5300
51805
0.412
0.386
0.352
259re
0.250
0.208
0.121
2099:
0.444
0,177
0.260
*300
0.002
0.110
0264
0,441
0,257
0.40%
20010
33002
20007
20000
2039
200e
20015
2300
20095
23001
13003
1080
25003
1900
23000
4320
J0836
0.318
6:176
12/02
0236
12025
0.95
12009
6:48
0.555
0.50
-0497
0.630
120ve
0.44
J0082
0.553
45000
0,335
13000
16800
0.688
00M
00M
0.909
16/012
0,518
0.1%%
0,286
0.41
0.507
0.335
0.306
0.207
0.394
Blconnity
720
201m
0.275
0.356
0.417
0.411
0.366
0.446
0.332
0.426
0.304
0.412
0.667
0.504
0.533
0417
0,436
1440
0.338
0.441
0.401
0.533
0.419
0.504
0.482
0.317
0.305
0.454
0.25
0.600
0.577
0.574
0151
0.609
+3000
60800
2160
0,380
0.481
0.536
0.547
0.421
0.501
0.765
0.644
0.415
0.406
0.38
0.758
0.642
0.10
0.896
0.714
20085
20:10
300015
2034
23007
20095
11.094
53009
50801
2630
1811
96
20000
0.1.2
13000
0.306
20009
0,145
0.219
23083
0.104
20013
0.255
20013
0.91
2000
0.256
13023
0.219
20005
210014
0.327
13001
0.264
0.35
2000
0.477
1500m
0.326
20046
1000
0.200
0,312
3300
Tratti
730
0.144
0.225
43/005
0.00
20009
0200
13000
-1110
13304-00
0.096
-0085
0311
0.518
13077
40080
0.309
41.00
0.41
0.247
43001
10854
0.32
0.237
15:00
0,500
14020
1634
0255
0.546
1440
1
0.174
0,195
3220
0273
0,40
0.26
0,31
0.553
0.409
0311
0.300
0.300
0,00
0.97
0.376
J0080
13/014
0.100
0.00
0.300
-000
0.426
2002
-0147
13043
2000
11000
0988
1000
15002
0317
18344
1300
-0000
+3100
-00004
-0000
$1.00
96
0.521
0.522
0.584
0.536
0.500
0,525
0.545
0.519
0,579
0.529
0.589
0.533
0.676
0.567
0.703
0.625
-01.00
Weather
720
20010
0.663
131011
0.115
1,080
0755
-5
1060
30005
0.54
0.987
203155
0.752
20085
1,007
15.00
0.766
23/04
1,005
20901
0.728
25.05
1.007
201177
0.25
1.114
0.222
1440
1.380
10010
0885
13000
1.547
20331
09956
23088
1351
2000P
2000S
0.665
13403
-0150
0.860
13004
J0008
1.200
0883
11.00
13683
1.472
10001
0.000
11.00
1.994
0.86
181240
1.435
0.919
2160
1415
0.887
1.744
0.94
1564
0.97
1.505
0.924
1.454
0.857
1.712
0.98
1.598
0.944
1.756
L,84
-00449
13000
101509
10000
20288
10005
-0005
13000
100000
110055
13002
1000
113544
L00TE
18:30
131.21
0.051
0.172
0.064
0.178
0.054
0.881
0.065
0.197
0.049
0.052
0.173
0.054
0,178
0.059
0,184
20000
Dichagee
360
0.445
0.459
0.522
11450
0.556
0.573
0.485
0.402
03544
0.541
0.528
0.556
720
1.677
0.794
1.239
0.56
1333
0.927
1.264
0.859
1.718
1024
1.291
0.864
1.358
0.594
1.381
0,003
1080
1.804
0.792
1.007
0.90
4.74
1.972
1:355
0.67%
4.982
1973
1.600
1.100
1.714
1.088
1,600
0
14
0.725
0.574
1.414
0.235
03115
0.001
1.558
0.965
1.907
01901
1.079
0.500
1,107
0.698
(L773
0.619
JA0955
43000
JA333
1000N
-0018
1008
-01300
12043
JA113
13084
1081
4E100
28
0.587
0.683
1,004
0.854
1,670
1,002
1.378
1.100
2,005
1.134
1.315
0.357
1,515
0.767
0.959
0.770
J0176
130012
J0184
10000
.0119
104074
-0569
130016
10181
11344
13075
1005
11109
1024
18000
13045
0.807
1,021
0.787
1.757
1.310
1.451
1.028
1.584
1.075
1.080
0.91
0.805
0.43
0.56
0.741
430113
240000
+0000
-6001
-6889)
41.00
+3005
16.12
+31015
112
1,49
1.038
1,699
1,072
3.593
1.759
2.846
1.438
3.332
1.572
2.608
13377
1.724
1.108
1.660
1.097
590/0
20001
25.10
1 CCusi
23


Reviews:
[{"review_id": "tAWQIYW6La", "reviewer": "Reviewer_pLfj", "summary": "The paper is about self-supervised contrastive forecasting in long sequences. Existing methods perform poorly on capturing long-term variations beyond the window (outer-window variations). The proposed method, AutoCon, learns a long-term representation by constructing positive and negative pairs accross distant windows in a self-supervised manner. The authors have perform extensive experiments on 9 datasets, using 14 state of the art models, and MSE and MAE evaluation metrics to show that the models with AutoCon loss outperform the rest in most cases, achieving performance improvements of up to 34%.", "questions": "- There is no discussion about making the code available publicly.\n- The authors can respond on my comments 2-4 in the weaknesses section above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and well-structured. The authors have done a great job with providing the related work and the limitations, giving the details to the proposed methodology and an extensive experimental evaluation. The images and tables are well-designed and show attention to detail.\n- The paper is about an interesting problem, long-term time-series forecasting via contrastive learning. This is topic that has gained a lot of attention recently.\n- The methodology is described with enough details.\n- The extensive experiments, not only capture different cases in a various datasets and with various state of the art comparison methods, but they also show that the proposed methodology outperforms in most cases the other models.", "weaknesses": "- The novelty of the work is incemental. While the authors focus on an interesting problem, the proposed methodology is combination of existing components.\n- It would be useful to add a table with the dataset statistics summarized (in Appendix if it does not fit in the main paper). How does the proposed methodology performs in imbalanced data?\n- It would be interesting and more convincing on the performance if the authors would add results on more evaluation metrics, e.g., Accuracy, AUROC, AUPRC, etc.\n- It would be interesting to show what is the runtime for each method/dataset and time complexities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iEssvD8GKG", "reviewer": "Reviewer_eBaK", "summary": "This paper mainly presents a new contrastive objective in self-supervised representation learning in time series forecasting. Specifically, the paper argues that the trends extracted from a moving window are often long-term seasonal variations that cannot be captured by the window of smaller size. Therefore, the representations of windows within a mini-batch that are more similar are explicitly induced to get closer compared to other samples with lower correlations. The proposed contrastive loss is then added to a decomposition-based forecasting model as a regularization term. Experiments show impressive performance improvements over existing SOTA models.", "questions": "1. How is the timestamp-based features derived and incorporated into the encoder?\n2. Which dataset is used in Figure 6? Do you observe the same result on other datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall the paper is well-written and easy to follow. The idea of global autocorrelation and decomposition is well-motivated and sufficiently grounded, and the argument that short-term trends are long-term variations is quite convincing.\n2. The empirical results on univariate long-term forecasting is very impressive.\n3. Extensive analysis support the effectiveness of the proposed method with abundant evidences", "weaknesses": "1. Most of the ambiguity comes from the global information in the proposed representation. The authors should elaborate on the details of the encoder to highlight how the similarity of representations $v_i, v_j$ differ from the linear autocorrelation $r(i, j)$ in order to justify the contrastive objective (3). See question (1). \n2. The plots of baseline models in figure 2 are not quite convincing. While the baseline models do not consider long-term correlations, it's counter-intuitive that the similarity of moving windows with various patterns stays almost constant as shown in the chart.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3vsrBdpjkf", "reviewer": "Reviewer_MDze", "summary": "This paper points out existing approaches fail to capture the long-term variations that are partially caught within the short window. Based on the finding, this paper presents a novel approach for long-term time series forecasting by employing contrastive learning and an enhanced decomposition architecture. The contrastive loss incorporates global autocorrelation held in the whole time series, which facilitates the construction of positive and negative pairs in a self-supervised manner. The authors conducted extensive experiments on nine benchmarks and achieved superior performance compared to baseline models.", "questions": "Could you provide training protocols and the default value of the contrastive loss weight hyperparameter?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* It is a good observation that long-term forecasting data exists long-term autocorrelations and existing methods fail to take into account.\n\n* By adding autocorrelation constrained contrastive loss, it reaches long-term variation consistency between windows, making it more explainable.\n\n* Extensive experiment results and ablation studies show the effectiveness of the method.", "weaknesses": "* Capturing long-term variation via the contrastive loss part is separated from the forecasting mechanism. Good finding but have no further design for architectures based on this finding. The forecasting module is a simple dual-head forecaster and already achieves good performance. \n\n* Lack of performance comparisons of different self-supervised objectives. \n\n* 14 baselines comparison is claimed in the abstract, only 7 are found.The performance improvement seems marginal. Extending the forecasting window too much has little practical meaning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j0G6qvSjr9", "reviewer": "Reviewer_hiP7", "summary": "This paper addresses challenges of long-term forecasting by utilizes contrastive learning and an enhanced decomposition architecture specifically designed to address long-term variations. The key idea considers the global autocorrelation within the entire time series, enabling the creation of positive and negative pairs in a self-supervised manner. Experiments demonstrate that this approach outperforms baseline models on nine established long-term forecasting benchmarks.", "questions": "Please address my comments on weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Contribution:\n\n- The authors deliver an intuitive and easy to implement technique based on newly defined ‘global autocorrelation’. A plus point for this technique also relates to its fast computing time and no extensive memory requirement.\n- The method is extensible and applicable to both uni and multi-variate datasets.\n- Well-rounded experiments are conducted with multiple datasets of different patterns (different autocorrelation patterns - Fig. 5).\n- The authors are well aware with the weakness of autocorrelation technique - which capture only linear variations.\n\nPresentation:\n\n- Nice visual explainations (e.g. Fig.1, 3).", "weaknesses": "- For circumstance of nearly stationary input sequence, the autocorrelation will produce stationary result and cause the collapse of Constrastive Learning framework.\n- The increasing complexity for long-term branch of decomposition framework might contribute to the better result of the whole proposed pipeline. A potential verification could be an additional ablation case of that architecture with a linear layer like the work of Zeng (2023).\n- While the authors are aware of the linear assumption of autocorrelation and suggest the potential use of different techniques for high-order one, this suggestions might be inapplicable for the lengthy global input sequences with densely use frequency. Any approaches investigating these non-linear correlation can consume much more resources compare to the current autocorrelation.\n- Some grammar and typos (e.g. TiemsNet - page 2)\n\nReference:\n\nAiling Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series\nforecasting? In Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)