CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting

1 INTRODUCTION Time series forecasting has been widely applied to various domains , such as electricity pricing ( Cuaresma et al. , 2004 ) , demand forecasting ( Carbonneau et al. , 2008 ) , capacity planning and management ( Kim , 2003 ) , and anomaly detection ( Laptev et al. , 2017 ) . Recently , there has been a surge of efforts applying deep learning for forecasting ( Wen et al. , 2017 ; Bai et al. , 2018 ; Zhou et al. , 2021 ) , and owing to the increase in data availability and computational resources , these approaches have offered promising performance over conventional methods in forecasting literature . Compared to conventional approaches , these methods are able to jointly learn feature representations and the prediction function ( or forecasting function ) by stacking a series of non-linear layers to perform feature extraction , followed by a regression layer focused on forecasting . However , jointly learning these layers end-to-end from observed data may lead to the model over-fitting and capturing spurious correlations of the unpredictable noise contained in the observed data . The situation is exacerbated when the learned representations are entangled – when a single dimension of the feature representation encodes information from multiple local independent modules of the data-generating process – and a local independent module experiences a distribution shift . Figure 1 is an example of such a case , where the observed time series is generated by a seasonal module and nonlinear trend module . If we know that the seasonal module has experienced a distribution shift , we could still makes a reasonable prediction based on the invariant trend module . However , if we learn an entangled feature representation from the observed data , it would be challenging for the learned model to handle this distribution shift , even if it only happens in a local component of the data-generating process . In summary , the learned representations and prediction associations from the end-to-end training approach are unable to transfer nor generalize well when the data is generated from a non-stationary environment , a very common scenario in the time series analysis . Therefore , in this work , we take a step back and aim to learn disentangled seasonal-trend representations which are more useful for time series forecasting . To achieve this goal , we leverage the idea of structural time series models ( Scott & Varian , 2015 ; Qiu et al. , 2018 ) , which formulates time series as a sum of trend , seasonal and error variables , and exploit such prior knowledge to learn time series representations . First , we present the necessity of learning disentangled seasonal-trend representations through a causal lens , and demonstrate that such representations are robust to interventions on the error variable . Then , inspired by Mitrovic et al . ( 2020 ) , we propose to simulate interventions on the error variable via data augmentations and learn the disentangled seasonal-trend representations via contrastive learning . Based on the above motivations , we propose a novel contrastive learning framework to learn disentangled seasonal-trend representations for the Long Sequence Time-series Forecasting ( LSTF ) task ( Zhou et al. , 2021 ) . Specifically , CoST leverages inductive biases in the model architecture to learn disentangled seasonal-trend representations . CoST efficiently learns trend representations , mitigating the problem of lookback window selection by introducing a mixture of auto-regressive experts . It also learns more powerful seasonal representations by leveraging a learnable Fourier layer which enables intra-frequency interactions . Both trend and seasonal representations are learned via contrastive loss functions . The trend representations are learned in the time domain , whereas the seasonal representations are learned via a novel frequency domain contrastive loss which encourages discriminative seasonal representations and side steps the issue of determining the period of seasonal patterns present in the data . The contributions of our work are as follows : 1 . We show via a causal perspective , the benefits of learning disentangled seasonal-trend representations for time series forecasting via contrastive learning . 2 . We propose CoST , a time series representation learning approach which leverages inductive biases in the model architecture to learn disentangled seasonal and trend representations , as well as incorporating a novel frequency domain contrastive loss to encourage discriminative seasonal representations . 3 . CoST outperforms existing state-of-the-art approaches by a considerable margin on realworld benchmarks – 21.3 % improvement in MSE for the multivariate setting . We also analyze the benefits of each proposed module , and establish that CoST is robust to various choices of backbone encoders and downstream regressors via extensive ablation studies . 2 SEASONAL-TREND REPRESENTATIONS FOR TIME SERIES . Problem Formulation Let ( x1 , . . .xT ) ∈ RT×m be a time series , where m is the dimension of observed signals . Given lookback window h , our goal is to forecast the next k steps , X̂ = g ( X ) , where X ∈ Rh×m , X̂ ∈ Rk×m , and g ( · ) denotes the prediction mapping function , and X̂ predicts the next k time steps of X . In this work , instead of jointly learning the representation and prediction association through g ( · ) , we focus on learning feature representations from observed data , with the goal of improving predictive performance . Formally , we aim to learn a nonlinear feature embedding function V = f ( X ) , where X ∈ Rh×m and V ∈ Rh×d , to project m-dimensional raw signals into a d-dimensional latent space for each timestamp . Subsequently , the learned representation of the final timestamp vh is used as inputs for the downstream regressor of the forecasting task . Disentangled Seasonal-Trend Representation Learning and Its Causal Interpretation As discussed in Bengio et al . ( 2013 ) , complex data arise from the rich interaction of multiple sources – a good representation should be able to disentangle the various explanatory sources , making it robust to complex and richly structured variations . Not doing so may otherwise lead to capturing spurious features that do not transfer well under non i.i.d . data distribution settings . To achieve this goal , it is necessary to introduce structural priors for time series . Here , we borrow ideas from Bayesian Structural Time Series models ( Scott & Varian , 2015 ; Qiu et al. , 2018 ) . As illustrated in the causal graph in Figure 2 , we assume that the observed time series data X is generated from the error variable E and the error-free latent variable X ? . X ? in turn , is generated from the trend variable T and seasonal variable S. As E is not predictable , the optimal prediction can be achieved if we are able to uncover X ? which only depends on T and S. Firstly , we highlight that existing work using end-to-end deep forecasting methods to directly model the time-lagged relationship and the multivariate interactions along the observed data X . Unfortunately , each X includes unpredictable noise E , which might lead to capturing spurious correlations . Thus , we aim to learn the error-free latent variable X ? . Secondly , by the independent mechanisms assumption ( Peters et al. , 2017 ; Parascandolo et al. , 2018 ) , we can see that the seasonal and trend modules do not influence or inform each other . Therefore , even if one mechanism changes due to a distribution shift , the other remains unchanged . The design of disentangling seasonality and trend leads to better transfer , or generalization in nonstationary environments . Furthermore , independent seasonal and trend mechanisms can be learned independently and be flexibly re-used and re-purposed . We can see that interventions on E does not influence the conditional distribution P ( X ? |T , S ) , i.e . P do ( E=ei ) ( X ? |T , S ) = P do ( E=ej ) ( X ? |T , S ) , for any ei , ej in the domain of E. Thus , S and T are invariant under changes in E. Learning representations for S and T allows us to find a stable association with the optimal prediction ( of X ? ) in terms of various types of errors . Since the targets X ? are unknown , we construct a proxy contrastive learning task inspired by Mitrovic et al . ( 2020 ) . Specifically , we use data augmentations as interventions on the error E and learn invariant representations of T and S via contrastive learning . Since it is impossible to generate all possible variations of errors , we select three typical augmentations : scale , shift and jitter , which can simulate a large and diverse set of errors , beneficial for learning better representations . 3 SEASONAL-TREND CONTRASTIVE LEARNING FRAMEWORK . In this section , we introduce our proposed CoST framework to learn disentangled seasonal-trend representations . We aim to learn representations such that for each time step , we have the disentangled representations for seasonal and trend components , i.e. , V = [ V ( T ) ; V ( S ) ] ∈ Rh×d , where V ( T ) ∈ Rh×dT and V ( S ) ∈ Rh×dS , such that d = dT + dS . Figure 3a illustrates our overall framework . Firstly , we make use of an encoder backbone fb : Rh×m → Rh×d to map the observations to latent space . Next , we construct both the trend and seasonal representations from these intermediate representations . Specifically , the Trend Feature Disentangler ( TFD ) , fT : Rh×d → Rh×dT , extracts the trend representations via a mixture of autoregressive experts and is learned via a time domain contrastive loss Ltime . The Seasonal Feature Disentangler ( SFD ) , fS : Rh×d → Rh×dS , extracts the seasonal representations via a learnable Fourier layer and is learned by a frequency domain contrastive loss which includes an amplitude component , Lamp , and a phase component , Lphase . We give a detailed description of both components in the next section . The model is learned in an end-to-end fashion , with the overall loss function being L = Ltime + α 2 ( Lamp + Lphase ) , where α is a hyper-parameter which balances the trade-off between trend and seasonal factors . Finally , we concatenate the outputs of the Trend and Seasonal Feature Disentanglers to obtain our final output representations . 3.1 TREND FEATURE REPRESENTATIONS . Extracting the underlying trend is crucial for modeling time series . Auto-regressive filtering is one widely used method , as it is able to capture time-lagged causal relationships from past observations . One challenging problem is to select the appropriate lookback window – a smaller window leads to under-fitting , while a larger model leads to over-fitting and over-parameterization issues . A straightforward solution is to optimize this hyper-parameter by grid search on the training or validation loss ( Hyndman & Khandakar , 2008 ) , but such an approach is too computationally expensive . Thus , we propose to use a mixture of auto-regressive experts which can adaptively select the appropriate lookback window . Trend Feature Disentangler ( TFD ) As illustrated in Figure 3b , the TFD is a mixture of L+1 autoregressive experts , where L = blog2 ( h/2 ) c. Each expert is implemented as a 1d causal convolution with d input channels and dT output channels , where the kernel size of the i-th expert is 2i . Each expert outputs a matrix Ṽ ( T , i ) = CausalConv ( Ṽ , 2i ) . Finally , an average-pooling operation is performed over the outputs to obtain the final trend representations , V ( T ) = AvePool ( Ṽ ( T,0 ) , Ṽ ( T,1 ) , . . . , Ṽ ( T , L ) ) = 1 ( L+ 1 ) L∑ i=0 Ṽ ( T , i ) . Time Domain Contrastive Loss We employ a contrastive loss in the time domain to learn discriminative trend representations . Specifically , we apply the MoCo ( He et al. , 2020 ) variant of contrastive learning which makes use of a momentum encoder to obtain representations of the positive pair , and a dynamic dictionary with a queue to obtain negative pairs . We elaborate further on the details of contrastive learning in Appendix A . Then , given N samples and K negative samples , the time domain contrastive loss is Ltime = N∑ i=1 − log exp ( qi · ki/τ ) exp ( qi · ki/τ ) + ∑K j=1 exp ( qi · kj/τ ) , where given a sample V ( T ) , we first select a random time step t for the contrastive loss and apply a projection head , which is a one-layer MLP to obtain q , and k is respectively the augmented version of the corresponding sample from the momentum encoder/dynamic dictionary .

Summary: The paper proposes a contrastive learning framework for time series forecasting CoST. This is done by detangling trend and seasonal features than using the learnt feature representations with a simple regression. Trend feature disentangler consists of a mixture of autoregressive experts followed by average pooling. While the seasonal feature is done by changing the features into the frequency domain via FFT and then applying a linear layer with weights to every different frequency. The overall loss is the weighted time-domain contrastive loss from trend feature disentangler and frequency domain contrastive loss which contains both phase and amplitude loss from seasonal feature disentangler. The proposed method was evaluated on 5 datasets in both a multivariate and univariate setting. CoST was compared against different representation learning and end-to-end learning appraoches.