CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting
Authors: Gerald Woo
Chenghao Liu
Doyen Sahoo
Akshat Kumar
Steven Hoi
Venue: ICLR 2022 Poster (2022)
Keywords: Time Series
Representation Learning
Forecasting
Self-Supervised Learning

Abstract: Deep learning has been actively studied for time series forecasting, and the mainstream paradigm is based on the end-to-end training of neural network architectures, ranging from classical LSTM/RNNs to more recent TCNs and Transformers. Motivated by the recent success of representation learning in computer vision and natural language processing, we argue that a more promising paradigm for time series forecasting, is to first learn disentangled feature representations, followed by a simple regression fine-tuning step -- we justify such a paradigm from a causal perspective. Following this principle, we propose a new time series representation learning framework for long sequence time series forecasting named CoST, which applies contrastive learning methods to learn disentangled seasonal-trend representations. CoST comprises both time domain and frequency domain contrastive losses to learn discriminative trend and seasonal representations, respectively. Extensive experiments on real-world datasets show that CoST consistently outperforms the state-of-the-art methods by a considerable margin, achieving a 21.3% improvement in MSE on multivariate benchmarks. It is also robust to various choices of backbone encoders, as well as downstream regressors. Code is available at https://github.com/salesforce/CoST.

Published as a conference paper at ICLR 2022
CoST: CONTRASTIVE LEARNING OF DISENTANGLED SEASONAL- TREND REPRESENTATIONS FOR TIME SERIES FORECASTING
Gerald Woo- 2, Chenghao Liu' Doyen Sahoo', Akshat Kumar" Steven Hoi*
*Salesforce Research Asia, 'Singapore Management University
{gwoo, chenghao 1iu, dsahoo, shoij salesforce.com, akshatkunan @ @smu.edu.sg
ABSTRACT
Deep learning has been actively studied for time series forecasting, and the main- stream paradigm is based on the end-to-end training of neural network architec- tures, ranging from classical LSTM/RNNs more recent TCNs and Transform- ers. Motivated by the recent success of representation learning in computer vi- sion and natural language processing, we argue that more promising paradigm for time series forecasting, to first learn disentangled feature representations, followed by simple regression fine-tuning step from a causal perspective. Following this principle, we propose a new time se- ries representation learning framework for long sequence time series forecasting named CoST, which applies contrastive learning methods to learn disentangled geasonal-trent representations. CoST comprises both time domain and frequency domain contrastive losses to learn discriminative trend and seasonal representa- tions, respectively. Extensive experiments on real-world datasets show that CoST consistently outperforms the state-of-the-art methods by considerable margin, achieving a 21.3% improvement in MSE on multivariate benchmarks. robust to various choices of backbone encoders, as well downstream regressors. Code is available at sattps://qile com/ www.//aalsforrr/Coooo
justify such paradigm
also
INTRODUCTION
Time series forecasting has been widely applied to various domains, such electricity pricing (Cuaresma al. (2004), demand forecast- ing (Carbonncau et al.) 2008), capacity planning and management (Kim (003), and anomaly detection (Laptev et al. (017). Recently, there has been a surge of efforts applying deep learning for forecast- ing (Wen al. 2017] Bai et al. 2018 Zhou et al., 2021), and ow- ing to the increase in data availability and computational resources, these approaches have offered promising performance over conven- tional methods in forecasting literature. Compared to conventional approaches, these methods are able jointly learn feature repre- sentations and the prediction function (or forecasting function) by stacking a series of non-linear layers to perform feature extraction, followed by regression layer focused forecasting.
Oborved
Sccsum
Treed
However, jointly learning these layers end-to-end from observed data may lead to the model over-fitting and capturing spurious cor- relations of the unpredictable noise contained The situation is exacerbated when the learned representations
Figure/ I: Time series com- posed of seasonal and trend components.
observed data.
entangled when a single dimension of the feature representation encodes information from mul- tiple local independent modules of the data-generating process and local independent module experiences distribution shift. Figure [ is an example such a case, where the observed time series is generated by a seasonal module and nonlinear trend module. If know that the seasonal
"Corresoondigg author.

---

Published as a conference paper at ICLR 2022
module has experienced distribution shift, could still makes reasonable prediction based on the invariant trend module. However, we learn an entangled feature representation from the ob- served data, it would be challenging for the learned model to handle this distribution shift, even if only happens in a local component of the data-generating process. summary. the learned repre- sentations and prediction associations from the end-to-end training approach unable transfer nor generalize well when the data is generated from a non-stationary environment, a very common scenario in the time series analysis. Therefore, in this work, we take a step back and aim learn disentangled seasonal-trent representations which are more useful for time series forecasting.
To achieve this goal, we leverage the idea of structural time series models (Scott Varian 2015 Qiu et al.) 2018). which formulates time series as a sum of trend, seasonal and error variables, and exploit such prior knowledge to learn time series representations. First, we present the necessity of learning disentangled seasonal-trend representations through a causal lens, and demonstrate that such representations are robust to interventions on the error variable. Then, inspired by |Mitrovic et al. (2020), we propose to simulate interventions the error variable via data augmentations and learn the disentangled reasonal-trend representations via contrastive learning.
Based on the above motivations, we propose a novel contrastive learning framework to learn dis- entangled scasonal-trend representations for the Long Sequence Time-series Forecasting (LSTF) task (Zhou et al., 202T). Specifically, CoST leverages inductive biases the model architecture to learn disentangles ed seasonal-trent representations. CoST efficiently learns trend representations, mitigating the problem of lookback window selection by introducing mixture of auto-regressive experts. It also learns more powerful seasonal representations by leveraging a learnable Fourier layer which enables intra-frequency interactions. Both trend and seasonal representations contrastive loss functions. The trend representations are learned in the time domain, whereas the seasonal representations are learned via a novel frequency domain contrastive loss which encour- ages discriminative seasonal representations and side steps the issue of determining the period of seasonal patterns present the data. The contributions of our work are as follows:
learned via
1. We show via a causal perspective, the benefits of learning disentangled seasonal rep- resentations for time series forecasting via contrastive learning-
2. We propose CoST, time series representation learning approach which leverages inductive biases in the model architecture to learn disentangled seasonal and trend representations, as well incorporating a novel frequency domain contrastive loss encourage discriminative seasonal representations.
3. CoST outperforms existing state-of-the-an approaches by a considerable margin on real- world benchmarks 21.3% improvement in MSE for the multivariate setting. We also analyze the benefits of each proposed module, and establish that CoST is robust to various choices of backbone encoders and downstream regressors via extensive ablation studies.
2 SEASONAL-TREND REPRESENTATION FOR TIME SERIES
Problem Formulation Let (æI,.... FT) E Txxm be time series, where m the dimension of observed signals. Given lookback window h, our goal is to forecast the next A steps, X x(X), where X E 2axm X kxm. and g(-) denotes the prediction mapping function, and X predicts the next & time steps of X.
Data generation
Data anttmmmtmttnni
xn
In this work, instead of jointly learning the representa- tion and prediction association through 9(-). we focus on learning feature representations from observed data, with the goal of improving predictive performance. Formally, we aim to learn a nonlinear feature embedding function V = f(X), where X E / and V E Ig to project m-dimensionaa raw signals into a d d-dimensional la- tent space for each timestamp. Subsequently, the learned representation of the final timestamp Un is used as inputs for the downstream regressor of the forecasting task.
g
Representation learning
Figure 2: Causal graph of the genera- tive process for time series data.
2

---

Published as a conference paper at ICLR 2022
Disentangled Seasonal- Trend Representation Learning and Its Causal Interpretation As dis- cussed in Bengio et al. (2013), complex data arise from the rich interaction of multiple sources a good representation should be able disentangle the various explanatory sources, making it robust to complex and richly structured variations. Not doing so may otherwise lead to capturing spurious features that do not transfer well under non i.i.d. data distribution settings.
To achieve this goal, it is necessary to introduce structural priors for time series. Here, we borrow ideas from Bayesian Structural Time Series models (Scott & Varian 2015] Qiu al.) 2018). As illustrated in the causal graph in Figure we assume that the observed time series data X is gener- ated from the error variable and the error-free latent variable X". X in turn, is generated from the trend variable 'T' and seasonal variable S. As E' is not predictable, the optimal prediction can be achieved if we are able to uncover X which only depends on T and S.
Firstly, we highlight that existing work using end-to-end deep forecasting methods to directly model the time-lagged relationship and the multivariate interactions along the observed data X. Unfortu- nately, each X includes unpredictable noise E, which might lead to capturing spurious correlations. Thus, we aim to learn the arror-free latent variable X*
Secondly, by the independent mechanisms assumption (Peters et al. 2017: Parascandolo et al,, 2018), we can see that the seasonal and trend modules do not influence or inform each other. Therefore, even if one mechanism changes due to a distribution shift, the other remains unchanged. The design of disentangling seasonality and trend leads to better transfer, or generalization in non- stationary environments. Furthermore, independent seasonal and trend mechanisms can be learned independently and be flexibly re-used and re- re-purposed.
We can that interventions E does not influence the conditional distribution P(X*IT, for any c;, Cj in the domain of E. Thus, S and T are invariant under changes in E. Learning representations for S and T allows us to find a stable association with the optimal prediction (of X*) in terms of various types of errors. Since the targets X are unknown, we construct a proxy contrastive learning task inspired by Mitrovic et al. (2220). Specifically, we use data augmentations as interventions representations of T and S via contrastive learning. Since it is impossible to generate all possible variations of errors, we select three typical augmentations: scale, shift and jitter, which can simulate a large and diverse set of errors, beneficial for learning better representations.
i.e. pdo(E=e)(X*(TT)) (X* pdo(E=e))(X*) (T, (X*
the error E and learn invariant
SEASONAL-TREND CONTRASTIVE LEARNING FRAMEWORK
In this section, we introduce proposed CoST framework learn disentangled seasonal-trend representations. We aim to learn representations such that for each time step, we have the disentan- gled representations for seasonal and trend components, i.e., (T) Raxdr and (S) E Rhxds, such that dr + dg.
(V(T) ((S) E R where
Figure illustrates our overall framework. Firstly, we make use of an encoder backbone fo h map the observations latent space. Next, we construct both the trend and seasonal representations from these intermediate representations. Specifically, the Trend Feature Disentangler (TFD), fr= Raxd + R4xdy, extracts the trend representations via a mixture of auto- regressive experts and is learned via time domain contrastive loss Lime- The Seasonal Feature Disentangler (SFD), fs Raxd - A extracts the scasonal representations via a learnable Fourier layer and is learned by frequency domain contrastive loss which includes an amplitude component, Lamp. and a phase component, Lphase- We give a detailed description of both com- ponents in the next section. The model is learned in end-to-end fashion, with the overall loss function being
Rhxm
L L= Ltime + a (Comp + Ephase);
2
where is Finally, we concatenate the outputs of the Trend and Seasonal Feature Disentanglers to obtain our final output representations.
which balances the trade-off between trend and seasonal factors.
3

---

Published as a conference paper at ICLR 2022
Frogercy
F1
CousalCome-2"
Time Domain Contrastive Loss
Amplitude Contrastive Loss
Contrastire Loss
P
CausalConv-24
AvaPool
Vs.
CausalCame-2
Feature
(b) Trend Feature Disentangler
Disentangler
Disentangler
Proquesy
2z:
(Per-
Backbone Encoder
Linear
Layer
(a) Overall Framework
Seasonal Feature Disentangler
Figure 3: (a) Overall Framework. Given intermediate representations from the backbone encoder,
V fa(X (X), the TFD and SFD produce the trend features, V(T) = -T((V, and seasonal features, (8) fV()), respectively. (b) Trend Feature Disentangler. Composition of mixture of auto- regressive experts, instantiated as Id-causal convolutions with kernel size of 2 vi = O..... ......L.. L, where L is hyper-parameter. Followed by average-pond over the L+ representations. (c) Seasonal Feature Disentangler. After transforming the intermediate representations into frequency domain via the FFT, the SFD applies a (complex-valued) linear layer with unique weights for each frequency. Then, an inverse FFT is performed to map the representations back to time domain, to form the seasonal representations, (S).
3.1 TREND FEATURE REPRESENTATIONS
Extracting the underlying trend is crucial for modeling time series. Auto-regressive filtering is one widely used method, it is able to capture time-lagged causal relationships from past observations. One challenging problem is to select the appropriate lookback window smaller window leads to under-fitting, while larger model leads over-fitting and over-parameeerizzaion issues. A straight- forward solution is to optimize this loss (Hyndman & Khandakar 2008), but such an approach is too computationally expensive. Thus, we propose to use mixture of auto-regressive experts which can adaptively select the appropriate lookback window.
by grid search on the training or validation
Trend Feature Disentangler (TFD) As illustrated in Figure 3p. the TFD is mixture of L auto- regressive experts, where L logg (h/2) Each expert is implemented as 1d causal convolution with d input channels and dr output channels, where the kernel size of the i-th expert is 2 Each expert outputs a matrix (T:3) CausalConv (V, (V,2%, 2'). Finally, average-pooling operation is performed over the outputs to obtain the final trend representations.
(T) = = (T.1)
((T.)))
(TT.).
(L+1)
Time Domain Contrastive Loss We employ a contrastive loss in the time domain to learn discrimi- native trend representations. Specifically, we apply the MoCo (He et al. 2020) variant of contrastive learning which makes use of momentum encoder to obtain representations of the positive pair, and dynamic dictionary with queue to obtain negative pairs. We elaborate further the details of contrastive learning in Appendix Then, given N samples and K negative samples, the time domain contrastive loss is
N
exp(q: K2/T)
Lime
log
exp(q: k,/7) + Σj exp(q: k//T)
where given sample (T), first select a random time step for the contrastive loss and apply projection head, which is a one-layer MLP to obtain q. and is respectively the augmented version of the corresponding sample from the momentum encoder/dynamic dictionary.

---

Published as a conference paper at ICLR 2022
3.2 SEASONAL FEATURE REPRESENTATIONS
Spectral analysis in the frequency domain has been widely used in seasonality detection (Shumway al., 2000). Thus, we to the frequency domain to handle the learning of seasonal representa- tions. To do so, we aim to address two issues: i) how can we support intra-frequency interactions (between feature dimensions) which allows the representations to encode periodic information more easily, and, ii) what kind of learning signal is required to learn representations which are able to discriminate between different seasonality patterns? Standard backbone architectures are unable to easily capture fira-ffre level interactions, thus, we introduce the SFD which makes use of a learnable Fourier layer. Then, order to learn these seasonal features without prior knowledge of the periodicity, frequency domain contrastive loss is introduced for each frequency.
Seasonal Feature Disentangler (SFD) As illustrated in Figure the SFD is primarily composed of discrete Fourier transform (DFT) map the intermediate features to frequency domain, followed by a learnable Fourier layer. We include further details and defimitions of the DFT in Appendix [B] The DFT is applied along the temporal dimension and maps the time domain representations into the frequency domain, F(V) Cxd where F |h/2] + 1 is the number of frequencies. Next, the learnable Fourier layer, which enables frequency domain interactions, is implemented via a per- element linear layer. It applies affine transform on each frequency, with unique set of complex- valued parameters for each frequency, since we do not expect this layer to be translation invariant. Finally, we transform the representation back to time domain using an inverse DFT operation.
The final output matrix of this layer is the seasonal representation, ( E Rixds, Formally, we can denote the i, k-th element of the output as
((() 1
- =
(E AL) + B.R). 1=1
where A E Fxdxds B E Fxds are the parameters of the per-element linear layer.
Frequency Domain Contrastive Loss As illustrated in Figure 3L. the inputs to the frequency do- main loss functions are the pre-iFFT representations, denoted by F E CFxds These are complex- valued representations in the frequency domain. To learn representations which able discrim- inate between different seasonal patterns, we introduce a frequency domain loss function. As our data augmentations can be interpreted as interventions on the error variable, the seasonal informa- tion does not change and thus, a contrastive loss in frequency domain corresponds to discriminating between different periodic patterns given a frequency. To overcome the issue of constructing a loss function with complex-vaie valued representations, each frequency can be uniquely represented by its amplitude and phase representations, Fi.l and Φ F...). Then, the loss functions are denoted,
exp (FF!))D)
Lamp = FN
log
exp(
b. (1) (F(()))) + kRt exp([E!!) (i)
F
N
exp(< ((u) <5 ((((F())))
Lphase
log
FN
exp(o(E!!) ( ((F + Dyys exp(o(F!!) of (!!!))
where F(5) is the j-th sample in mini-batch, and (F(!ly is the augmented version of that sample.
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
In this section, we report the results of detailed empirical analysis of CoST and compare it against diverse set of time series representation learning approaches, as well as compare against end-to-end supervised forecasting methods. Appendix contains further results on runtime analysis.
4.1 EXPERIMENTAL SETUP
Datasets We conduct extensive experiments on five real-world public benchmark datasets. ETT (Electricity Transformer Temperature]] (Zhou et al., (021) consists of two hourly-level datasets (ETTh) and one 15-minute-level dataset (ETTm), measuring six power load features and "oil tem- perature". the chosen target value for univariate forecasting. Electricity. measures the electricity consumption of 321 clients, and following popular benchmarks, we convert the dataset into hourly- level measurements and set "MT 320" the target value for univariate forecasting. Weather is an hourly-level dataset containing 11 climate features from nearly 1,600 locations in the U.S., and we take "wet bulb" the target value for univariate forecasting. Finally, we also include the M5 dataset Makkiddakis et al.
2020 in Appendix
Evaluation Setup Following prior work, we perform experiments two settings multivariate and univariate forecasting. The multivariate setting involves multivariate inputs and outputs, con- sidering all dimensions of the dataset. The univariate setting involves univariate imputs and out- puts, which are the target values described above. We use MSE and MAE as evaluation metrics, and perform a 60/20/20 train/validation//esst split. Inputs are zero-mean normalized and evalu- ated over various prediction lengths. Following (Yue et al. [021), self-supervised learning ap- proaches are first trained on the train split, and ridge regression model is trained on top of the learned representations to directly forecast the entire prediction length. The validation set used to choose the appropriate ridge regression regularization term a, over a search space of {0.1, 0.2, 0.5, 1,2, 5, 10, 20, 50, 100, 200, 500, 1000). Evaluation results are reported on the test set.
Implementation Details For CoST and all other representation learning methods, the backbone encoder used is Temporal Convolution Network (following similar practice in TS2Vec (Yue et al. 2021)) unless the approach includes or is an architectural modification (further details in Appendix All methods used have representation dimensionality of 320. We use standard hyper-parameter setting on all datasets a batch size of 256 and learning rate of 1E 3, momentum of 0.9 and weight decay of 1E-4 with SGD optimizer and cosine annealing. The MoCo imple- mentation for time domain contrastive loss uses a queue size of 256, momentum of 0.999, and temperature of 0.07. We train for 200 iterations for datasets with less than 100,000 samples, and 600 iterations otherwise. Details on data augmentations used in CoST can be found in Appendix
4.2 RESULTS
Among the baselines, we report the performance of representation learning techniques including TS2Vec, TNC, and time series adaptation of MoCo in our main results. A more extensive bench- mark of feature-based forecasting approaches can be found in Appendix H due to space limitations. Further details about the baselines can found Appendix [ We include supervised forecasting approaches two Transformer based models, Informer (Zhou et al. (021) and LogTrans{Li et al. 2020), and the backbone TCN trained directly on an end-to-end forecasting loss. A comparison of end-to-end forecasting approaches can be found in Appendix[]
Table I summarizes the results of CoST and top performing baselines for the multivariate setting, and Table (in Appendix[G]dom due to space limitations) for the univariate setting. For end-to-end fore- casting approaches, the TCN generally outperforms the Transformer based approaches, Informer and LogTT At the same time, the representation leaminng methods outperform end-to-end fore- casting approaches, but there are indeed cases, such as in certain datasets for the univariate setting, where the end-to-end TCN performs surprisingly well. While Transformers have been shown to
httts://gthhbb.cm////huaaayy/ "httpss//archive.ic. wwt.://ww....uuum////tttteeeeeee http://wwwwneeeenngg.g//////a/an
www.Diagamms20111222 12014

---

Published as a conference paper at ICLR 2022
Table 1: Multivariate forecasting results. Best results are highlighted in bold.
Representation Learning
End-to-end Forecusting
Methods
CoST
TS2Vee
TNC
MoCo
Informer
Trans
TCN
Metrics
MSF
MAR
MSE
MAR
MSF
MAE
MSE
MAE
MSR
MAE
MSR
MAE
MSE
MAE
24
0.386
0.429
0,590
0,531
0.708
0.592
0.623
0.555
0.577
0.549
0.686
0.604
0.583
0.547
48
0.437
0.464
0.624
0.555
0.749
0.619
0.669
0.586
0.685
0.625
0.766
0.757
0.670
0.606
168
0.643
0.582
0.762
0.639
0.884
0.699
0.820
0.674
0.931
0.752
1.002
0.846
0.811
0.680
336
0.812
0.679
0.931
0.728
1.020
0.768
0.981
0.755
1.128
0.873
1.362
0.952
1.132
0.815
720
0.970
0.771
1.063
0.799
1.157
0,830
1.138
0.831
1,215
0.896
1.397
1.291
1.165
0.813
24
0,447
0.502
0.423
0.489
0.612
0.595
0.444
0.495
0,720
0.665
0.828
0.750
0.935
0.754
ETTh2
48
0.699
0.637
0.619
0.605
0,840
0.716
0.613
0.595
1.457
1.001
1.806
1.034
1.300
0.911
168
1.549
0,982
1.845
1.074
2.359
1.213
1,791
1.034
3,489
1.515
4,070
1.681
4.017
1.579
336
1.749
1.042.
2.194
1.197
2.782
1.349
2.241
1.186
2.723
1.340
3.875
1.763
3.460
1.456
720
1.971
1.092
2.636
1.370
2.753
1,394
2.425
1.292
3.467
1.473
3.913
1.552
3.106
1.381
24
0.246
0.329
0.453
0.444
0.522
0.472
0.458
0.444
0.323
0.369
0.419
0.412
0.363
0.397
ETTml
48
0.331
0.386
0.592
0.521
0.695
0.567
0.594
0.528
0.494
0.503
0.507
0.583
0.542
0.508
96
0.378
0.419
0.635
0,554
0.731
0.595
0,621
0.553
0.678
0.614
0.768
0.792
0.666
0,578
288
0.472
0.486
0.693
0.597
0
0.649
0,700
0.606
1.056
0.786
1.462
1.320
0.991
0.735
672
0.620
0.574
0.782
0.653
0,932
0.712
0.821
0.674
1.192
0.926
1.669
1.461
1.032
0.756
24
0.136
0.242
11,287
01375
0.354
0.423
0,288
0.374
0.312
0.387
0.299
0.374
0.235
0,346
Electricity
48
0.153
0.258
0,309
0.391
0.376
0.438
0,310
0.390
0.392
0.431
0.316
0.389
0.253
0.359
168
0.175
0.275
0.335
0,410
0,402
0.456
0.337
0.410
0.515
0,509
0.426
0.466
0.278
0.372
336
0.196
0.296
0,351
0.422
0.417
0.466
0,353
0.422
0.759
0.625
0.365
0.417
0.287
0,382
0.232
0.327
0.378
0.440
0,380
0.441
0.969
0.788
0.344
0.403
0.287
24
0,298
0.360
0.307
0.363
0.320
0.373
0.311
0.365
0.335
0,381
0.435
0.477
0.321
0,367
Weather
48
0.359
0.411
0.374
01.118
0.380
0.421
0.372
0.416
0.395
0.459
0.426
0.495
0.386
0.423
168
0.464
0.491
0.491
0.506
0.479
0.495
0,482
0.499
0.608
0.567
0.727
0.671
0.491
0.501
336
0.497
0.517
0.525
0.530
0.505
0.514
0.516
0.523
0.702
0.620
0.754
0.670
0.502
0.507
720
0.533
0.542
0.556
0.552
0.519
0.525
0.540
0,540
0.831
0.731
0.885
0.773
0.498
0.508
Avg.
0.590
0.524
0,750
0.607
0.870
0.655
0.753
0.608
1.038
0.735
1.180
0.837
0.972
0.666
be powerful models in other domains like NLP, this suggests that TCN models are still powerful baseline which should still be considered for time series.
Overall, our approach achieves ofte-offtheeart performance, beating the best performing end-to- end forecasting approach by 39.3% and 18.22% (MSE) in the multivariate and univariate settings respectively. CoST also beats next best performing feature-based approach by 21.3% and 4.7 I % (MSE) in the multivariate and univariate settings respectively. This indicates that CoST learns more relevant features by learning composition of trend and seasonal features which are crucial for forecasting tasks.
4.3 PARAMETER SENSITIVITY
Table 2: Parameter sensitivity of a in CoST on the ETT datasets.
IF-01
SE-02
1F-02
5B-03
1F-03
SF-04
1F-04
SE-05
1F-05
Multivariate Univariate
0.810
0.805
0.781
0.781
0.782
0.781
0.780
0.780
0.780
0.120
0.113
0.106
0.104
0.102
0.102
0.103
0.103
0.103
Cases for which larger a is preferred (multivariate).
168
1,509
1.604
1.555
1,550
1,550
1.549
1.544
1.545
1,546
ETTR2
336
1.524
1.646
1.722
1.731
1.744
1.749
1.759
1.762
1.765
a controls the weightage of the seasonal components in the overall loss function, L = Ctime + (Camp + E.hase)) We perform a sensitivity analysis on this hyper-parameter (Table 2 and show that an optimal value can be chosen and is robust across various settings. We choose a 5E-04 for all other experiments since it performs well on both multivariate and univariate settings. We note that the small values of a stems from the frequency domain contrastive loss being generally three orders of magnitude larger than the time domain contrastive loss, rather than being an indicator that the seasonal component has lower importance than the trend component. Further, overall, while choosing a smaller value of a leads to better performance on most datasets, there are certain cases for which larger a might be preferred, as seen in the lower portion of Table 2
highlight that
7

---

Published as a conference paper at ICLR 2022
4.4 ABLATION STUDY
Table 3: Ablation study of various components of CoST on ETT datasets). TFD: Trend Feature Disentangler, MARE: Mixture of Auto-regressive Experts (TFD without MARE refers to the TFD module with a single AR expert with kernel size h/2]), SFD: Seasonal Feature Disentangler, LFL:: Learnable Fourier Layer, FDCL: Frequency Domain Contrastive Loss. indicates model trained end-to-end with supervised forecasting loss. indicates with an additional contrastive loss.
Multivariate
Univariate
TFD
MARE
SFD
LFL
FDCL
MSE
MAE
MSE
MAE
0.882
0.674
0.115
0.243
Trend
0.789
0.630
0.105
0.235
0.905
0.675
0.105
0.237
Seasonal
0.895
0.721
0.103
0.239
0.862
0.668
0.129
0.255
CoSTi
1.376
0.834
0.228
0.366
CoST#
1.477
0.909
0.965
0.883
MoCo
0.996
0.721
0.112
0.248
SimCLR
1,021
0.730
0.113
0.248
CoST
0.781
0.625
0.102
0.233
Table 4: Ablation study of various backbone encoders on the ETT datasets.
Backbones
TCN
LSTM
Transformer
Methods
TS2Vec
CoST
TS2Vec
CoST
TS2Vec
CoST
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Multivariate
0.990
0.717
0,781
0.625
1.415
0.903
0.928
0.706
1.092
0.766
0.863
0.674
Univariate
0.116
0.253
0.102
0.233
0.544
0.596
0.148
0.301
0.172
0.328
0.159
0.320
Table 5: Ablation study of various regressors on the ETT datasets
Regressors
Ridge
Linear
Kernel Ridge
Methods
TS2Vec
CoST
TS2Vec
CoST
TS2Vec
CoST
MSF
MAR
MSE
MAE
MSR
MAE
MSE
MAR
MSR
MAE
MSF
MAR
Multivariate
0.990
0.717
0.781
0.625
1.821
0.944
1.472
0.781
1.045
0.738
0.868
0.686
Univariate
0.116
0.253
0.102
0.233
0.304
0.414
0.182
0.310
0.132
0.273
0.109
0.243
Components of CoST We first perform an ablation study to understand the performance benefits brought by each component in CoST. Table 3 presents the average results over the ETT datasets on all forecast horizon settings (similarly for Tables 4 and 5). We show that both trend and sea- sonal components improve performance over the baselines (SimCLR and MoCo), and further, the composition of trend and seasonal components leads to the optimal performance. We further verify that training our proposed model architecture end-to-end with supervised forecasting loss leads to worse performance.
Backbones Next, we verify that our proposed trend and seasonal components as well as contrastive loss (both time and frequency domain) are robust to various backbone encoders. TCN is the default backbone encoder used in all other experiments and we present results on LSTT and Transformer backbone encoders of equivalent parameter size. While performance using the TCN backbone out- performs LSTM and Transformer, we show that our approach outperforms the competing approach on all three settings.
Regressors Finally, we show that CoST is also robust to various regressors used for forecasting. Apart from ridge regression model, also perform experiments linear regression model and a kernel ridge regression model with RBF kernel. As shown in Table [ we also demonstrate that CoST outperforms the competing baseline all three settings.
8

---

Published as a conference paper at ICLR 2022
4.5 CASE STUDY
We visualize the learned representations on simple synthetic time series with both seasonal and trend components, and show that CoST able learn representations which are able to discriminate between various seasonal and trend patterns. The synthetic dataset is gener- ated by defining two trend and three seasonal patterns, and taking the cross product to form six time series (details in Appendix D). After training the encoders on the synthetic dataset, visualize them via the T-SNE algorithm (Van der Maaten & Hinton 2008). Figure 4 shows that our approach is able learn both the trend and seasonal patterns from the dataset and the learned representations has high clus- terability, whereas TS2 Vec is unable to distin- guish between various seasonal patterns.
Sestonality Treed
Figure 4: T-SNE visualization of learned repre- sentations from CoST and Te2Vec. (Top) Gener- ated by visualizing representations after selecting a single scasonality. Colors represent the two dis- tinet trends. (Bottom) Generated by visualizing representations after selecting a single trend. Col- represent the three distinct seasonal patterns.
5 RELATED WORK
Deep forecasting has typically been tackled as an end-to-end supervised learning task, where early work considered using RNN based models (Lai al. (0188) as natural approach modeling time series data. Recent work have also considered adapting Transformer based models for time series forecasting (Li al., 2020; Zhou et al. 2021). specifically focusing on tackling the quadratic space complexity of Transformer models. Oreshkin et al.] (2020) proposed univariate deep forecasting model and showed that deep models outperform classical time series forecasting techniques.
While recent work in time series representation learning focused on various aspects of representation learning such how to sample contrastive pairs (Franceschi et al.) 2020), Tonekaboni et al.) (02T),
taking a Transformer based approach (Zerveas et al.) 2021). ) exploring complex contrastive learning tasks (Eldele et al. [202]), as well as constructing temporally hierarchical representations (Yue et al.,
2021. none have touched upon learning representations composed of trend and seasonal features.
Whereas existing work have focused exclusively on time series classification tasks, Yue et al. (2021) first showed that time series representations learned via contrastive learning establishes new state- of-the-art performance on deep forecasting benchmarks.
Classical time series decomposition techniques (Hyndman & Athanasopoulos 2018 have been used to decompose time series into seasonal and trend components to attain interpretability. There has been recent work on developing more robust and efficient decomposition approaches (Wen al. 2018 2020 Yang et al., 202T). These methods focus decomposing the raw time series into trend and seasonal components which are still interpreted as time series the original input space rather than learning representations. GGodfrey & Gashler (2017) presents an initial attempt to use neural networks model periodic and non-periodic components time series data, leveraging periodic activation functions to model the periodic components. Different from our work, such a method is only able to model a single time series per model, rather than produce the decomposed seasonal- trend representations given lookback window.
6 CONCLUSION
Our work shows separating the representation learning and downstream forecasting task to be more promising paradigm than the standard end-to-end supervised training approach for time-series forecasting. We show this empirically, and also explain through causal perspective. By fol- lowing this principle, we proposed CoST, contrastive learning framework that learns disentangled seasonal-tres -rend representations for time series forecasting tasks. Extensive empirical analysis shows that CoST outperforms the previous state-of-the-ar approaches by considerable margin and is bust to various choices of backbone encoders and regressors. Future work will extend our framework for other time-series intelligence tasks.

---

Published as a conference paper at ICLR 2022
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: seoi:000311111 20188
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Real Carbonneau, Kevin Laframboise, and Rustam Vahidov. Application of machine learning tech- niques for supply chain demand forecasting. European Journal of Operational Research, 184(3): 1140-1154, 2008.
Jesús Crespo Cuaresma, Jaroslava Hlouskova, Stephan Kossmeier, and Michael Obersteiner. Fore- casting electricity spot-prices using linear univariate time-series models. Applied Energy, 77(1): 87-106, 2004.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2352-2359, 2021.
Jean-Yve Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series, 2020.
Luke B. Godfrey and Michael S. Gashler. Neural decomposition of time-series data for effec- tive generalization. IEEE Transactions Neural Networks and Learming Systems, pp. 1-13, 2017. ISSN 2182-2388. doi: 10.1 (1/9////l..111.2799999...9 11C9/INNLS
CRLhttt://// doi wwtp./oooooog/...o/ooooo///// /1.
Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: fast mixture-of-expert training system, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020.
Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
Rob J Hyndman and Yeasmin Khandakar. Automatic time series forecasting: the forecast package for r. Journal of statistical software, 27(1):1-22, 2008
Kyoung-jae Kim. Financial time series forecasting using support vector machines. Neurocomputing, 55(1-2):307-319 2003.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The s1 International ACM SIGIR Conference on Research Development in Information Retrieval, pp. 904,04, 2018
Nikolay Laptev, Jason Yosinski, Li Erran Li, and Slawek Smyl. Time-series extreme event forecast- ing with neural networks uber. In International conference on machine learning, volume 34, pp. 1-5, 2017.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, 2020.
S Makridakis, Spiliotis, and V Assimakopoulos. The m5 accuracy competition: Results, findings and conclusions. Int J Forecast, 2020.
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Represen- tation learning via invariant causal mechanisms. arXiv preprint arX1v:2010.07922, 2020.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting, 2020.
10

---

Published as a conference paper at ICLR 2022
Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, pp. 4036- 4044. PMLR. 2018.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Jinwen Qiu, S Rao Jammalamadaka, and Ning Ning. Multivariate bayesian structural time series model. Mach. Learn. Res., 19(1):2744 2776, 2018.
Steven L Scott and Hal R Varian. 4. Bayesian Variable Selection for Nowcasting Economic Time Series. University of Chicago Press, 2015.
Robert H Shumway, David S Stoffer, and David Stoffer. Time series analysis and its applications, volume 3. Springer, 2000.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. International Conference on Learning Repre- sentations, 2021. URL https: htttt:::://ppeeeeviieeee wwt.eeewwwwwweeeeeeeeeeeeeeeeeee
fisstfoor forum Fo Bed
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Huan Xu, and Shenghuo Zhu. Robuststl: A robust seasonal-trend decomposition algorithm for long time series, 2018. 8.
Qingsong Wen, Zhe Zhang, Yan Li, and Liang Sun. Fast robuststl: Efficient and robust seasonal-trend decomposition for time series with complex patterns. In Proceedings of the 26th ACM SIGKD) International Conference on Knowledge Discovery & Data Mining, KDD '20, pp. 2203-2213 New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi: 00.11457/3944863433371. URL
3394486.
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11055 2017.
Linxiao Yang, Qingsong Wen, Bo Yang, and Liang Sun. robust and efficient multi-scale seasonal- trend decomposition. In ICASSP 2021 2021 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 5085-5089, 2021. doi: 10.1109/ICASSP3972..
2021.99 991.9911999
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series, 2021.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eick- hoff. transformer- based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining,
KDD '21, pp. 2114-2124, New York, NY, USA, 2021. Association for Computing Machin- ery. ISBN 9781450383325. doi: 10.1145/3447548.444 3467400 URL https:::////mm httpp:://oii.ogg/// 3467401
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of AAAI, 2021.
11

---

Published as a conference paper at ICLR 2022
A CONTRASTIVE LEARNING
Contrastive learning via the instance discrimination task is powerful approach for self-supervised learning. Here, describe the essentials of this method, which underpins our proposed approach. Firstly, a family data augmentations, A. is defined. Given a single sample of data æ; E X, two data augmentation operators sampled, A, ~ A. q: f(a(z;)) is referred to the query representation with encoder f, and k; [(a (x:)) is the positive key representation. Finally, the InfoNCE loss function
exp(q. kt/T)
LInfNNCE
log
exp(q: k./T) + ΣØ= exp(q: k2/T)
where T is the temperature number of negative samples. Standard approaches use an efficient mechanism to obtain negative samples - by simply treating all other samples in the mini-batch as negative samples, i.e. MoCo (He et al. 2020) introduces the idea of a using queue of size K (a hyper-parameten to obtain negative samples. each iteration of training, simply pop N samples from the queue, and push the N representations form the current mini-batch.
k2 are negative key representations, and is the total
N-1.
B DISCRETE FOURIER TRANSFORM
The DFT provides a frequency domain view of given sequence, x = (Zo, Zi,... F x1),, mapping time series with regular intervals into the Fourier coefficients, sequence of complex numbers of equal length. Due to the conjugate symmetry of the DFT of real-valued signals, we can simply consider the first N/22 + Fourier coefficients, c= F(x) E CLN/2J+1,
Ck =(xx)
Σ In exp((i22kkn)) n=0
Each complex component, Ck2 can be represented by the amplitude, |c&l and the phase, d(ck),
H(x)
leal / R + )
<(cx) = Ian
M(cr
where {(cc) and [(c are the real and imaginary components of respectively. Finally, the inverse DFT maps the frequency domain representation back to the time domain,
1 = F = N k=0
Σ ck exp(i2nkn/N).
C DATA AUGMENTATIONS
In experiments, utilize composition of three data augmentations, applied in the following order- scaling, shifting, and jittering, activating with a probability of 0.5.
Scaling The time-series is scaled by single random scalar value, obtained by sampling (O 0.5), and each time step is ā eTg,
Shifting The time-series is shifted by single random scalar value, obtained by sampling e (0 and each time step is H, = Ie +
Jittering LLD. Gaussian noise added each time step, from a distribution & 0.0,0.5). where each time step is now 2N = 24 + Et.
D SYNTHETIC DATA GENERATION
We first construct two trend patterns. The first trend patterns follows a nonlinear, saturating pattern, yt Itepp (t + 6, for Bo 0.2, B1 60, e NO, 0.3). The second pattern is mixture of
12

---

Published as a conference paper at ICLR 2022
ARMA process, ARMA(2. 2) ARMA(3, 3) ARMA(4, 4), where the AR and MA parameters are as follows, (1.9, -.1) {.2, -.5)), (f.1, .3). f.1, .65, -.55), (1.3, .5. -.5 -.3), {.1, .I, -2, .).. Next, we construct three seasonal patterns, consisting of sine waves with the following period, phase, and amplitudes, {20, 0, 3). {50, 22, (100, (5,0,
The final time series are constructed as follows, generate trend pattern g(t) and seasonal pattern s(t), the final time series is y(t) = (t) + st), = o....,9.99 999. We do this for all pairs of trend and seasonal patterns, constructing total of 6 time series.
E DETAILS ON BASELINES
Results for Vec,ec, TNC. MoCo, Triplet, CPC, TST, TCC are based reproduction, while results for Informer, LogTTrans, and LSTNet (Lai et al. 2018) are directly taken from Yue et al. (202T) for the ETT and Electricity datasets, and Zhou al. 2021) for the Weather dataset. For all reproduced approaches, we train for 200 iterations for datasets with less than 100,000 samples, otherwise, and 600 iterations otherwise.
The encoders used in all approaches except Triplet and TST is the causal TCN encoder as proposed in TS2Vec (Yue al.) 2021). A fully connected layer is first used to project each time step from the dimensionality of the multivariate time series to the hidden channel (size 64). Then, there are 10 layers of convolution blocks. Each convolution block is sequence of GELU, DilatedConv, GELU, DilatedConv, with skip connections across each block. The DilatedConvs have dilation of 2' in each layer of convolution block, all have kernel size of 3 and input/output channel size of 64. final convolution block is used to map the hidden channels to the output channel (size 320).
For Triplet, the encoder their proposed causal TCN encoder, while for TST, their proposed Trans- former encoder is used. Further details can be found in their respective papers.
TS2Vec (Yue et al. 2021) TS2Vec was recently proposed universal framework for learning time series representations by performing contrastive learning in a hierarchical manner over aug- mented context views. They propose learn timestamp level representations. We ran the code from their open source repository as is, syper-parameten used are all defaults as suggested in their paper.
TNC (Tonekaboni et al. 2021) TNC proposes self- superviedd framework for learning general- izable representations for non-stationary time series. They make use of the augmented Dickey-Fuller test for stationarity to ensure positive samples come from the a neighborhood of similar signals. We use their open source code setting w = 0.005 in the loss function, soc.samplesizz size = 20, batch size of 8, and learning rate of 1E-3 with Adam optimizer.
MoCo is a popular ceff-super learning baseline computer vision, and we implement time series version of MoCo using their open source code We use a batch size of 128. queue of 256, momentum for the momentum encoder is 0.999, temperature of the loss function is 0.07, learning rate of 1E-3 with SGD optimizer, with cosine annealing. Data augmentations used are described in Appendix
MoCo (He et al., 2020)
Triplet (Franceschi et al. 2020) Triplet proposes a time series self-supervised learning approach by taking positive samples to be substrings of the anchor, and negative samples be randomly sampled from the dataset. We reproduce their approach by adapting their open source code" making use of their proposed causal TCN model architecture. We use batch size of 10, and learning rate 1E 3 with A optimizer.
CPC (van den Oord et al., 2019) space using auto-regressive models. They use a probabilistic contrastive loss which induces latent
CPC learns representations by predicting the future in latent
hotp:///caccooo///oooooossttttnt representation learning
13

---

Published as a conference paper at ICLR 2022
space to capture information maximally useful predict future samples. We reproduce this ap- proach by referencing an unofficial implementation"" We use a GRU for the AR module, use two prediction steps after eight AR steps. We use batch size of 8 and learning rate of 1E-3 with AdamW optimizer.
TST (eereas al., 2021 TST Transformer based approach using reconstruction loss. We use the open source implementation as is.
TCC (Eldele et al. 2021 TCC introduces temopral contrastive module and a tough cross-view prediction task. We use their open source implementation" To We learning rate of 3E-4 with Adam optimizer = 1, A2 = 0.7 as their implementation, and jitter scale ratio or 1.1, maximum segment length of 8, and jitter ratio of 0.8, following their HAR setting.
F RUNTIME ANALYSIS
Table 6: Runtime (seconds) for each method in train and inference phase. For representation meth- ods, we split the runtime into A + B, where refers to the time for encoder, and B is the time for the ridge regressor in downstream phase.
Phase
CoST
TS2Vec
TNC
MoCo
Informer
TCN
262.78 + 7.19
91.9 5.45
1801.58 + 4.78
31.3 + 6.57
331.32
108.36
48
262.78 + 8,41
91.9 + 6.44
1801.58 + 5.79
31.3 8.03 8.03
167.18
109.30
Training
96
262.78 10.0
0.19
1801.58 + 7.23
31.3
325.51
110.02
288
262.78
19.9 262.78 38.3
91.9 16.47
1801.58 14.51
31.3 17.83
449.86
112.08
672
+2.22
1801.58 28.21
3
587.25
112.87
24
23.60
0.04
3.73
3.38 + 0.05
3,67 0.07
10.32
3,20
48
23.60 0.07
3.00
3.38 + 0.09
3,67 0.08
5.78
3.67
Inference
96
23.60
0.08
+
0.10
11.32
4.78
23.60 + 0.24
3.73
3.38 + 0.21
3.67 + 0.22
17.19
7.19
672
23.60 + 0.34
3.73 + 0.30
0.30
3.38 + 0.33
3.67 + 0.33
25.62
11.93
Table 6 shows the runtime in seconds for each phase for various representation learning methods and end-to-end approaches. All experiments are performed on an NVIDIA A100 GPU. Do note that for Informer, follow the hyperparameters (including lookback window length) described by the authors which may vary for different forecasting horizons, thus the runtime may not be strictly increasing the forecasting horizon increases. We want highlight that for all representation learning approaches, the ridge regressor portions should be equal since the dimension size used are the same across all methods, any differences simply due randomness. Despite slightly higher training time compared to some of the baseline approaches, CoST achieves much better results (refer Table | in the main paper). Furthermore, the extra computation time of CoST as compared to VS2Vec is due to the sequential computation of each expert in the mixture of AR expert component, can be further accelerated by parallel methods (He et al. 2021)
httns///ghthhimmmmmccchlllSlS/at Crriereiee--eeiiiiiii--Coonngn Porchh Torch http::///gituuh.cm/ggrrreess transformer
14

---

Published as a conference paper at ICLR 2022
G UNIVARIATE FORECASTING BENCHMARK
Table 7: Univariate forecasting results. Best results are highlighted in bold.
Representation Learning
End-to-end Forecasting
Feature Engineered
Methods
CoST
TS2Voc
TNC
MoCo
Informer
LogTrans Trans
TCN
TSFresh
Metrics
MSB
MAE
MSE
MAB
MSE
MAE
MSE
MAB
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAB
24
48
0.158
0.319
0.167
0.328
0.206
03366
0,092
168
0.097
0.236
0.1412
0.291
0.17]
0.529
0.122
0.268
0.183
0.346
0.207
0.375
0.462
0.586
0.097
0.251
336
0.112
0.258
0.160
0.316
0.192
0.357
0.144
0.297
0.222
0.387
0.250
0.398
0.422
0.564
0.109
0.263
74
0.201
0.091
0.234
0.033
0.240
0.102
0.255
0.109
0.176
0.331
18
0.118
0.259
0.124
0.274
0.13]
0.281
0.130
0.279
0.155
0.314
0.169
0.3418
0.147
0.302
0.202
0.357
168
0.189
0.339
0.198
0.355
0.199
0.354
0.204
0.360
0.232
0.389
0.246
0.422
0.209
0.366
0.273
0,420
1,364
0.267
0.284
720
0.214
0.371
0.208
0.371
0.207
0.570
0.206
0.369
0.277
0.431
0.303
0.493
0.200
0.367
0.339
0.166
24
0.015
0.088
0.016
0.093
0.019
0.103
0.015
0.091
0.030
0.137
0.065
0.202
0.027
0.127
0.027
0.128
HTTmm
48
0.025
0.117
0.028
0,126
0.036
0.142
0.027
0.122
0.069
0.203
0.078
0.220
0,040
0.154
0.043
0.159
0.194
0.246
288
0.077
0,209
0.095
0,235
0.098
0.244
0.083
0.219
0.401
0.551
0.111
0.572
0.305
0:155
0.098
0.245
672
0.113
0.257
0,142
0,250
0.136
0.290
0.122
0,268
0,512
0.644
0.598
0.702
0.445
0.576
0.121
0.274
24
0.243
0.264
0,260
0,288
0.252
0.278
0.254
0,280
0.251
0.275
0.528
0,447
0.243
0.367
Elatneety
0.397
168
0.105
0.375
0,129
0,392
0,112
0.384
0.116
0.391
0.541
0.124
0.959
0.612
0.357
0.119
336
0.566)
0.473
0.565
0,4778
0.548
0.466
0.556
0.482
0.713
0.512
1.079
0.639
0.355
0,446
720
0,559
0.645
0.863
0.651
0.859
0.651
0.858
0.653
1,182
0,806
1.001
0.714
0.387
0.477
Weather
18
0.138
0,262
0,140
0,264
0.139
0.264
0.140
0,264
0.178
0.318
0.206
0.356
0.113
0.269
0.231
0.361
168
0.207
0.334
0,207
0,335
0.198
0.528
0.198
0.326
0.266
0. 1.39%
0. 0.509
0,439
0.188
0.319
0.298
0.415
336
0,230
0,356
0,231
0,360
0115
0347
0,220
0.350
0.297
0,416
0.359
0.484
0.192
0.320
0.314
0.429
0.370
0,233
0,365
0,119
0.359
0,166
0.388
0.196
0.329
0.123
Avg.
0.193
0.283
0,203
0,296
0.207
0.307
0.198
0,294
0.296
0.386
0.352
0.430
0.236
0.367
15

---

Published as a conference paper at ICLR 2022
H RESULTS ON FEATURE-BASER BEATURE-BASED FORECASTING BASELINES
Table 8: Multivariate forecasting results for feature-based approaches.
Representation Leorning
Feature Engineeced
Methods
CaST
VS2Vcc
MrCo
Tripke
CPC
TCC
TSPicsh
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAB
MSE
MAE
MSE
MAA
MSE
MAE
MSC
MAE
MSB
MAG
0.386
0.590
0.531
0.708
0.592
0.623
0.555
0.942
0.729
0.600
0.755
0.766
0.629
3,85%
1.574
kTT
168
0.437
0.643
0.582
0.464
0.762
0.624
0.639
0.355
0.834
0.749
0.619
0.699
0.669
0.520
0.674
0.586
1.135
0.825
0.746
0.920
0.774
0.714
0.629
0.800
0.973
0.768
0.982
0.825
0.751
0.657
3.527
4.246
1.574
1,500
0.812
0.679
0.728
0.581
1.387
0.859
0.779
1.099
0.786
2,905
1.329
0.970
0.799
1.13%
1.285
0.916
1.020
1.267
0.859
2.667
1,253
0.447
0.502
0.423
0.489
0.612
0.595
0.444
1,285
0.911
0.551
0.572
0.994
1.154
0.858
8.720
2,311
Brh2
0.699
0,605
0.840
0,613
0.595
0,966
0.684
1.159
0.350
1.579
0.983
12:771
2.746
1.549
0.982
1.845
1.074
2.359
1211
1.791
1.034
2.175
1,155
1.213
2.609
1.265
3,456
1.459
20.343
3.779
1.749
1.197
3.760
1.349
2.341
1.101
1.504
3.164
1.971
1.370
1.394
2.425
1.159
1.509
2.684
3,538
17:967
0.346
0.444
0.472
0.458
0.444
0.689
0.902
0.471
0.491
0.500
0.47%
0.610
Er
0.521
0.905
0.534
0.634
0.614
0.645
0.559
0.705
0.629
0.37%
0,419
0.554
0.731
0.595
0.621
0.553
0.744
0.623
0.593
0.645
0.675
0.585
0.675
0.606
0.472
0.649
0.308
0.662
0.644
0.749
0.758
0.633
0
0.763
0.653
0.712
0.821
0.730
0.500
0.857
0.689
0.767
Elettiiiyy
0.153
0.375
0.310
0.286
0.564
0.581
0,578
0.326
0.311
0.365
0.489
0.175
0.275
0410
0.337
0.584
0.350
0.456
0.196
0.286
0.351
0.422
0.417
0.465
0.353
0.591
0.591
0.501
0.359
0.407
0.468
0.232
0,327
0.480
0.442
0.463
0.380
0.441
0.603
0.558
0.555
0.383
0,618
0.467
0,360
0.311
0.365
0.522
0.533
0.372
0.292
2.170
Weather
0,411
0.418
0-421
0.372
0.539
0.543
0:435
0.418
0.391
0.439
2:235
0.464
0.506
0.482
0.573
0.565
0.512
0.521
2.514
0.985
336
0.497
0.517
0.530
0.505
0.514
0.516
0.523
0.582
0.572
0.536
0.555
0.523
0.532
2.293
0.969
0.533
0.542
0.556
0.552
0.519
0.525
0.540
0.400
0.397
0.552
0.553
0.575
0.548
0.569
2,468
0,961
Avg.
0.590
0.524
0.750
0.607
0.870
0.655
0.753
0.608
0.969
0.732
0,663
0.893
0.671
1,021
0.701
Table 9: Univariate forecasting results for feature-based approaches
Representaion Learning
Featree Engineeced
Methods
CaST
TS7Vcc
McCo
Triplet
CPC
TOC
TSPicsh
Metrics
MSF
MAF
MSR
MSR
MAF
MAF
MSF
MSF
MAF
MAR
01040
0.151
0.157
0.184
0,040)
0.330
0,281
0.076
0.217
0.127
0.053
0.175
0.0811
0.224
0.060
0.189
0.094
0.063
0.191
0.306
0.250
0.201
0.209
0.342
0.997
0.143
01:91
0.151
0.329
0.122
0.336
0.326
0.491
0.133
0.284
0.253
0.112
0.258
0.316
0.144
0.333
0.526
0.161
0.320
0.148
0.306
0.345
0.405
0.183
0.195
0,368
0.212
0.387
0.717
0.176
0.363
0.30
24
0.079
0.250
0.097
0.238
0.005
0.160
0,316
0.251
0.154
0.131
0.255
0,531
E1122
0.274
0.131
0.130
0.319
0.189
0.355
0.354
0.373
0:302
0.261
0.225
0.306
0.205
0.350
0.260
0.232
0.144
0.371
0.371
0,207
0.370
0.306
0.251
0,406
0.389
0.275
0.420
0.242
0.247
0.339
0,406
0.015
0.093
0,015
0.091
0.150
0.102
0.048
0.006
0.178
ETTml
0.025
0.117
0.025
0.126
0,036
0.142
0:027
0,122
0.084
0.206
0.142
0.064
0.045
6045
0.159
0.038
0.162
0.178
0.041
0.13
0,210
0.188
0.102
0,072
0.11
0004
0.178
0.077
0.235
0,068
0.34
0.083
0.219
0.130
0,256
0.271
0.172
0.316
0.158
0.18
0.06
0.245
0.113
0.257
0.142
0.290
0.136
0.290
0.122
0.268
0.360
0.315
0.177
0.332
0.224
0.366
0,239
0.248
0.121
0.274
0.286
0,252
0.278
0.254
0.355
0.299
0.351
0.266
0.301
Elestrisity
0.202
0,300
0.313
0.321
0,300
0.305
0.304
0.314
0.375
0,390
0.321
0.339
0,398
0.317
0.350
108
0.405
0.375
0.382
0.412
0.364
0.416
0.391
0.482
0.459
0.418
0.531
0.424
0.402
336
0.560
0.473
0.478
0,548
0.466
0.556
0.482
0.633
0.551
0,507
0.656
0.578
0.485
730
0.889
0.645
0.863
0.651
0.859
8.651
0.558
0.653
0.930
0.706
0957
0.679
0.929
0.950
0.667
24
0.096
0.213
0.096
02155
0,102
0.221
0.097
0.16
0.203
0.337
0.105
0:226
0.124
0.107
0.232
0.192
0.330
Weather
48
0.138
0.262
0.140
0.364
0.139
0.264
0.340
0264
0.219
0.351
0.147
0:222
0151
0.280
0.143
0.272
0.231
0.561
168
02117
0.334
0.257
0.335
0.198
0.324
0.198
0.326
0.251
0.379
0213
0:340
0.113
0.342
0.204
0.353
0.298
0.415
336
0.250
0.356
0.231
0.360
0.215
0.347
0.221
0.300
0.262
0.389
0.362
0.233
0.361
0.219
0.250
0.314
0.429
730
0.342
0.370
0.233
0.365
0.219
0.363
0.234
0.357
0.263
0.304
0.566
0.233
0.30
0.230
0.352
0.423
0.499
Aug.
0.193
0.283
0.203
0.298
0.207
0.307
0.198
0.294
0.255
0.360
0.324
0.304
0.396
0.221
0.319
We include hand-crafted features (using the same experiment methodology as representation learn- ing approaches) in our benchmark, by using features from the TSFresh package. We select the same set of features for all datasets and settings, to avoid extensive feature engineering which requires domain expertise. TSFresh generally under performs in the multivariate benchmark due to the high dimensionality of the generated features, since it extracts univariate features and thus feature size increases linearly with input size. On the other hand, it performs relatively well on the univariate setting.
16

---

Published as a conference paper at ICLR 2022
1 RESULTS ON END-TO-END FORECASTING BASELINES COMPARED TO COST
Table 10: Multivariate forecasting results for End-to-end forecasting baselines compared to CoST
Ead-to-end Forecnsting
Methods
CoST
Informer
LogTrans
TCN
I.TTnet
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.429
0.577
0.549
0.604
0.583
0.547
1.293
0.901
48
0.437
0.464
0.685
0.625
0.766
0.757
0.670
0.606
1.456
0.960
0.582
0.931
0.752
0.846
0.811
1.997
336
0.812
0.679
1.128
0.873
1.362
0.952
1.132
0.815
2.655
1.369
0.771
1.215
0.896
1.397
1.291
1.165
0.813
2.143
24
0.447
0,502
0.720
0.665
0.828
0.750
0.935
0.754
2.742
1.457
F1T12
0.699
0.637
1.457
1.001
1,806
1,300
0,911
3.567
1.687
168
1.549
0.982
3.489
1.515
4,070
1.681
4,017
1.579
3.242
2.513
2.723
1,340
1.763
3.460
1,456
2.544
2.591
720
1,971
1.092
3.467
1.473
3,913
1,552
3.106
1,381
4.625
3.709
0.246
0.329
0.323
0,369
0,419
0.412
0.363
0.397
1,968
1.170
ET1ml
48
0.331
0.386
0.494
0.503
0.507
0,583
0.542
0.508
1.999
1.215
0.419
0.678
0.614
0,768
0.792
0.666
0.578
2.762
288
0.472
0.486
1,056
0.786
1.462
1,320
0.991
0.735
1:257
2.076
0.574
1.192
1.669
1.461
1.032
0.756
1.917
2.941
24
0.136
0.242
0.312
0.387
0,297
0,374
0.235
0,346
0.356
0,419
Electiiity
0.258
0.392
0,316
0.389
0.253
0.359
0.456
168
0.175
0.275
0,515
0.509
0,426
0.466
0.278
0.372
0.372
0.425
0.296
0.759
0.625
0.417
0.287
0.382
0.352
0.409
720
0,232
0.327
0.969
0.788
0,344
0.403
0.287
0.381
0.38
0.443
0.298
0.360
0,335
0,381
0,435
0.477
0.321
0,367
0.615
0.545
Weather
48
0.359
0.411
0.395
0.459
0.426
0.495
0.386
0.423
0.66
0.589
0.464
0.491
0.608
0.567
0.727
0.671
0.491
0.501
0.748
0.647
336
0.497
0.517
0.702
0.620
0.754
0.670
0.502
0.507
0.782
0.683
0.831
0.773
0.498
0.508
Avg.
0.590
0.524
1.038
0.735
1,180
0.837
0.972
0,666
1.668
1.284
Table 11: Univariate forecasting results for end-to-end forecasting baselines compared to CoST
End to end Forecasting
Methods
CoST
Informer
LogTrans
TCN
LSTnet
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
24
0.040
0.152
0.098
0.247
0.103
0.259
0.104
0.254
0.108
0.284
0.060
0.186
0.158
0.319
0.167
0.328
0.206
0.366
0.175
0.424
168
0,097
0.236
0.183
0.346
0,207
0.375
0.462
0.586
0.396
0.504
336
0.112
0,258
0,222
0.387
0.230
0.398
0.422
0.564
0.468
0.593
720
0,148
0.306
0.269
0.435
0,273
0.463
0.438
0.578
0.659
0.766
0.079
0.207
0.093
0.240
0.102
0.255
0. 0.109
0.251
3.554
0.445
ETTh2
0,118
0.259
0.155
0.314
0,169
0.348
0.147
0.302
3.190
0.474
168
0.189
0.339
0,232
0.389
0.246
0.422
0,209
0.366
2,800
0.595
336
0,206
0.360
0.263
0.417
0,267
0.437
0.237
0.391
2.753
0.738
720
0.214
0.371
0,277
0.431
0,303
0.493
0,200
0.367
2,878
1.044
24
0.015
0
0.030
0.137
0.065
0.202
0.027
0.127
0.090
0.206
E11ml
0.025
0.117
0.069
0.203
0.078
0.220
0.040
0.154
0.179
0.306
0.038
0.147
0.194
0.372
0.199
0.386
0.097
0.246
0.272
0.399
288
0.077
0.209
0.401
0.554
0.411
0.572
0.305
0.455
0.462
0.558
672
0,113
0.257
0.512
0.644
0,598
0.702
0.445
0.576
0.639
0.697
24
0.243
0.264
0.251
0.275
0.528
0.447
0.243
0.367
0.281
0.287
Electricity
0,292
0.300
0.346
0.339
0.409
0.414
0.283
0.397
0,381
0.366
168
0.405
0.375
0.544
0.424
0.959
0.612
0.357
0.449
0.599
0.500
336
0,560
0.473
0.713
0.512
1.079
0.639
0.355
0.446
0.823
0.624
720
0.889
0.645
1.182
0.806
1.001
0.714
0.387
0.477
1,278
0.906
24
0.096
0.213
0.117
0.251
0.136
0.279
0. 1,109
0.217
Weather
48
0.138
0.262
0.178
0.318
0.206
0.356
0. 143
0.269
168
0.207
0.334
0.266
0.398
0.309
0.439
0.188
0.319
336
0.230
0.356
0.297
0.416
0,359
0.484
0.192
0.320
720
0.242
0.370
0.359
0.466
0,388
0.499
0.198
0.329
Avg.
0.193
0,283
0.296
0.386
0,352
0.430
0.236
0.367
17

---

Published as a conference paper at ICLR 2022
J RESULTS ON M5 DATASETS
Table 12: Results on M5 datasets (Makridakis et al., 2020). M5 is a multivariate dataset, with
forecast horizon is 28, as per M5 competition settings.
Representation Learning
saddtt--nd Forecasting
Methods
CoST
TS2Vec
TNC
MoCo
Informer
TCN
Metries
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
1.1
0,063
0.211
0.299
0.446
0.671
0.627
0.279
0.415
0.836
0.724
1.395
1.020
1.2
0.154
0.311
0.383
0.498
0.521
0.564
0.360
0.482
1.436
0.991
1.310
0.932
L3
0.191
0.340
0.591
0.549
0.621
0.582
0,438
0.496
1.747
1.033
1.847
1.048
L4
0.149
0.293
0.291
0.403
0.393
0.482
0,304
0.408
1.023
0.829
1.388
0.992
L5
0.260
0.390
0.419
0.484
0.506
0.545
0,462
0.508
1.514
0.899
2.039
1.172
M5
L6
0.255
0.386
0,500
0.528
0.586
0.589
0.478
0.510
1.099
0.822
1.309
0.925
L7
0.394
0.482
0.630
0.648
0.618
0.641
0.605
1.565
0,931
1.475
0,953
L8
0.328
0.442
0,589
0.559
0.614
0.584
0.610
0.575
1.969
1.063
0,990
L9
0.702
0.582
2.150
0.748
1.445
0.726
1.293
0.701
4,923
1.313
4.634
1.332
1.10
1.471
0.783
1.677
0.812
1.679
0.830
1.604
0.815
2.474
0.977
2.476
1.025
Avg.
0.397
0.422
0.753
0.563
0.769
0.615
0.647
0.551
1,859
0.958
1.957
1.039
K
CASE STUDY:
DISENTANGLEMENT
TFD Representations
SPD Representationn
TFD Representations
SFD Representations
Rossonall Mon
Trond
(a) Trend Disentanglement.
(b) Scasonality Disentanglement.
Figure 5: T-SNE visualization of seasonal-trend disentanglement in CoST embeddings. TFD Rep- resentations refer to the representations generated by the Trend Feature Disentangler while SFD Representations refer to the representations generated by the Seasonal Feature Disentangler. (a) We select single seasonality and visualize the representations. The two colors represent the two dis- tinct trends. (b) We select single trend and visualize the representations. The three colors represent the three distinct seasonal patterns.
To exhibit CoST's ability to disentangle trend and seasonal components, we plot the T-SNE rep- resentations of the Trend Feature Disentangler (TFD) and Seasonal Feature Disentangler (SFD) separately. From Figure we see that representations from the TFD indeed have better clusterabil- ity and the representations from SFD have a degree of overlap between the two trend patterns. From Figure [bb] we see that the TFD representations have a higher degree overlap between the three seasonality patterns than the SFD representations. Here, to better highlight the stronger capability of seasonality representations in extracting seasonal patterns, we used complex seasonality patterns (described below).
Complex Seasonality Similar to the synthetic data generation process in Appendix D we generate three different seasonality patterns before combining them with the two trend patterns. The first seasonality pattern is no seasonality. The second pattern begins with sine wave of period, phase, and amplitude of {3. 0, 10). thereafter, mask is then applied to the entire pattern, consisting of repeating pattern of three 1s and seven Os. The third pattern begins with sine wave of period, phase, and amplitude of (10, 0.5, 15). thereafter, a mask is then applied to the entire pattern, consisting of repeating pattern of two Is and eight Os.
18


Reviews:
[{"review_id": "mPqwWGPLNp-", "reviewer": "Reviewer_bLmw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading the paper. The method for representation learning is original and clearly described and there's not much for me to argue around. \n\nMy main criticism comes from the experiments and the discussion/awareness of related work. I think that these criticisms can be taken care of by editing/toning down the paper and as part of the review process.\n\nIt's rather surprising that an end-to-end forecast approach can be beaten (by a margin) by a two-step approach. This needs much more and much stronger explanation and the one given here (because of the trend-seasonal-decomposition) does not strike me as strong enough -- a transformer-based model with enough parameters should learn this easily for example and there are plenty of other STL-based approaches in forecasting (e.g., implemented in the R Forecast package) that then should fare well. \n\nIn most of the literature on contrastive learning, the performance gets close to the state of the art but I'm not aware of it beating the state of the art by such a margin. Alas, it's unclear to me how the different representation learning approaches are used to produce forecasts. I may have overread this, but an explanation of how we go from representation to forecast would be needed for the main results (there is an ablation study with Table 5, but I'm not sure what was used for the main results in Table 1). In a typical contrastive learning paper, one would have a simple model (e.g., linear) on top of the complex representations and then get a similar performance to the state of the art. Why is a similar empirical effectiveness not happening here and instead, so much more improvement?\n\nIn the experiments, I'm missing natural baselines (e.g., NBEATS or TFT or DeepAR) and datasets that are a bit more standard, like the M5 data sets for example. Of course, reviewers always want more data sets and more methods, but given the claim that end-to-end SOTA approaches are beaten so convincingly, this leaves me wondering whether indeed SOTA methods were used in comparison or whether the data sets used are representative of the forecasting task. \n\nAlso, using the embeddings as co-variates for an end-to-end-learnt forecasting model would be a natural experiment to explain the accuracy gain better (that would basically be Table 5 with another regressor on top)\n\nFurther questions/comments:\n- overfitting doesn't seem to be a major problem in forecasting where model sizes are comparatively small (when compared with NLP for example), so this motivation doesn't convince me (page 1). A more convincing angle may be greater generality or speed. \n\n- page 2: \"Firstly, we highlight that existing work using end-to-end deep forecasting methods directly model\nthe time-lagged relationship along the observed data X\"  There is a considerable body of literature on probabilistic and deep forecasting models (e.g., https://iclr.cc/virtual/2021/spotlight/3409, but also Deep State Space Models at NeurIPS 20219 and much follow up work) which are in direct contradiction to this statement. In particular for multi-variate probabilistic and deep time series models, there is a growing body of literature which addresses the fact that Figure 2 is an over-simplification in the multi-variate context -- multiple X can have direct causal interaction (e.g., products that cannabilize/substitute/cross-sell)\n\n- experiments: are the embeddings trained on a single data set or across data sets? Over which time horizons do we train the embeddings and how do we forecast (rolling window evaluations), etc? Why is MAE and MSE chosen when there are things like sMAPE and MASE which are more generally accepted?\n\n", "summary_of_the_paper": "This paper proposes a novel way to represent time series learnt with contrastive losses in both the time and frequency domain for the forecasting task. The approach is novel and timely and is clearly described. The authors make a connection to causality which they use also for designing data augmentation schemes.", "summary_of_the_review": "Overall, I'd like to recommend the paper for acceptance given the clear merits of its representation learning approach which I applaud the authors for. However, I have some open questions which I would like to clarify as part of the review process so right now, my evaluation is cautious. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j2h6AbLGTz", "reviewer": "Reviewer_S8vv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The disentanglement of trends and seasonalities is well-motivated. The interpretation of invariance under intervention on E is valid.\n2. The introduced contrastive losses is a good proxy to learn discriminative representations.\n3. The extensive empirical results show the effectiveness of the proposed method and the ability of identifying multiple patterns.\n\n## Weakness\n1. The method of making forecasts from learned representations is not presented, making it incomplete for reproducing the results. In addition, a joint loss with the forecasting error should also be compared with the current contrastive training.\n2.  In case study, besides clustering of multiple patterns, the disentanglement of trends and seasonalities should also be verified. For example, the representation of the same seasonal pattern is supposed to be overlapped even mixed with different trended patterns.\n\n## Comments and Concerns\n1. In ablation study, what is the difference between TFD and MARE? The \"Trend Feature Disentangler\" paragraph in page.4 states TFD is a mixture of AR experts.\n2. In the selected datasets, the seasonalities of samples are highly similar (e.g. daily and weekly) The negative samples in frequency contrastive loss may be indistinguishable from the positive one in the first place. It may hurt the effectiveness of the frequency representations.\n3. Minor: $d$ should be $m$ in the sec 2. \"Problem Formulation\" $\\hat{X}\\in\\mathbb{R}^{k\\times d}$?", "summary_of_the_paper": "This paper introduce a framework of learning disentangled trend and seasonal representations of time series and its application to forecasting tasks. It employs contrastive losses to distill discriminative trend and seasonal representations. DFT is used in obtaining frequency domain information. Empirical results show the proposed framework is able to outperform end-to-end trained forecasters and other representation learning methods. In addition, the learned representations can be clustered to distinguish multiple trends or seasonalities.", "summary_of_the_review": "Overall I think this paper is well-written with clear motivation and technical path. It is acceptable but can be further improved in several aspects.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QA9MPvE8XuB", "reviewer": "Reviewer_KrXv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe idea of this paper seems novel and reasonable. Most parts are easy to follow. The experiments clearly show the improvement against baselines and the effectiveness of each model component.\n\nWeaknesses:\n1. The writing of the paper can be improved, e.g., several places can be more concise.\n2. In the formulas of time domain and frequency domain contrastive losses, it is unclear why only one entry (time step) is selected as the anchor since both the trend and seasonality are concepts containing a series of time steps.\n3. It is still unclear why representation learning methods can outperform end-to-end methods in such a large gap. It would be good if the authors can provide some insights and explanations about this.\n4. When training the regression model on top of the learned backbone, are the parameters of the backbone frozen or not?\n5. What is the specific setting of the TCN method, such as the model architecture, hyper-parameters, etc.\n", "summary_of_the_paper": "The current training paradigm applied in most time series forecasting approaches jointly learns feature representations and the prediction function. This paper argues that this paradigm may lead to several issues, e.g., overfitting problems, obtaining false relations of the unpredictable noise in the data, and entangling representations.\n\nTo tackle these challenges, this paper proposes a new framework (CoST) to first learn error-free feature representations, and then fine-tune the representations through a simple regressor. In the feature learning stage, a pretext task is constructed by using data augmentation and contrastive learning. The goal is to learn disentangled seasonal and trend representations. Moreover, time domain and frequency domain contrastive losses are incorporated to learn discriminative trends and seasonal representations, respectively.\n", "summary_of_the_review": "This paper still has space to improve. I will preserve my score before reading the authors' responses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9PGEeufGbt6", "reviewer": "Reviewer_97uN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The idea of  disentangling trend and seasonal features and using the learnt representations for forecasting is novel.\n- CoST was compared to different representation learning and end-to-end learning appraoches on 5 datasets and showed strong empirical results.\n- The paper preformed ablation studies induding investigating the use of different backbone encoders, the use of different regressors for forecasting and the effect of different componets of CoST on accuracy.\n- Through synthetic datasets the paper showsthat  CoST can actually disentangle seasonal and trend features.\n\nWeakness:\n- Missing simple baselines, like for example linear regression on the raw features rather than the learned representations and linear regression on the features outputted from the backbone encoder.\n- Some issues with paper writting please see clearification questions below.\n- Missing analysis of computational resourses and training time needed by CoST when compared to other methods.\n\nClarification questions:\n\n- In figure3 shouldn't V(l) and V(g) be V(T) and V(S).\n- Is it possible to provide the code to verify the approach and experiments?\n- For the multivariate time series are you forecasting all features or just using them as input and forecasting target features.\n- What does the second half of table 2 show (the detailed breakdown for multivaritate setting).\n- For tables 3-5 is this the mean accross all ETT datasets?", "summary_of_the_paper": "The paper proposes a contrastive learning framework for time series forecasting CoST. This is done by detangling trend and seasonal features than using the learnt feature representations with a simple regression. Trend feature disentangler consists of a mixture of autoregressive experts followed by average pooling. While the seasonal feature is done by changing the features into the frequency domain via FFT and then applying a linear layer with weights to every different frequency. The overall loss is the weighted time-domain contrastive loss from trend feature disentangler and frequency domain contrastive loss which contains both phase and amplitude loss from seasonal feature disentangler. The proposed method was evaluated on 5 datasets in both a multivariate and univariate setting. CoST was compared against different representation learning and end-to-end learning appraoches. ", "summary_of_the_review": "The overall idea is novel and shows strong empirical results. \nThe paper performs ablation studies justifying different design choices.\nThe paper shows the ability of the proposed method to disentangle trends and seasonal features through synthetic datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to learn disentangled trends and seasonal representations of time series for forecasting tasks. It shows separating the representation learning and downstream forecasting task to be a more promising paradigm than the standard end-to-end supervised training approach for time-series forecasting. 

During the post-rebuttal phase, there were interactions from all the reviewers, and reviewer KrXv raised the score. The reviewers think the contrastive learning method is novel and the added experiments have strengthened the paper.  The authors are encouraged to include more standard datasets (M5) in the final version.

Based on the above reasons, I am recommending accepting this paper.
Decision: Accept (Poster)