CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting

1 INTRODUCTION Time series forecasting has been widely applied to various domains , such as electricity pricing ( Cuaresma et al. , 2004 ) , demand forecasting ( Carbonneau et al. , 2008 ) , capacity planning and management ( Kim , 2003 ) , and anomaly detection ( Laptev et al. , 2017 ) . Recently , there has been a surge of efforts applying deep learning for forecasting ( Wen et al. , 2017 ; Bai et al. , 2018 ; Zhou et al. , 2021 ) , and owing to the increase in data availability and computational resources , these approaches have offered promising performance over conventional methods in forecasting literature . Compared to conventional approaches , these methods are able to jointly learn feature representations and the prediction function ( or forecasting function ) by stacking a series of non-linear layers to perform feature extraction , followed by a regression layer focused on forecasting . However , jointly learning these layers end-to-end from observed data may lead to the model over-fitting and capturing spurious correlations of the unpredictable noise contained in the observed data . The situation is exacerbated when the learned representations are entangled – when a single dimension of the feature representation encodes information from multiple local independent modules of the data-generating process – and a local independent module experiences a distribution shift . Figure 1 is an example of such a case , where the observed time series is generated by a seasonal module and nonlinear trend module . If we know that the seasonal module has experienced a distribution shift , we could still makes a reasonable prediction based on the invariant trend module . However , if we learn an entangled feature representation from the observed data , it would be challenging for the learned model to handle this distribution shift , even if it only happens in a local component of the data-generating process . In summary , the learned representations and prediction associations from the end-to-end training approach are unable to transfer nor generalize well when the data is generated from a non-stationary environment , a very common scenario in the time series analysis . Therefore , in this work , we take a step back and aim to learn disentangled seasonal-trend representations which are more useful for time series forecasting . To achieve this goal , we leverage the idea of structural time series models ( Scott & Varian , 2015 ; Qiu et al. , 2018 ) , which formulates time series as a sum of trend , seasonal and error variables , and exploit such prior knowledge to learn time series representations . First , we present the necessity of learning disentangled seasonal-trend representations through a causal lens , and demonstrate that such representations are robust to interventions on the error variable . Then , inspired by Mitrovic et al . ( 2020 ) , we propose to simulate interventions on the error variable via data augmentations and learn the disentangled seasonal-trend representations via contrastive learning . Based on the above motivations , we propose a novel contrastive learning framework to learn disentangled seasonal-trend representations for the Long Sequence Time-series Forecasting ( LSTF ) task ( Zhou et al. , 2021 ) . Specifically , CoST leverages inductive biases in the model architecture to learn disentangled seasonal-trend representations . CoST efficiently learns trend representations , mitigating the problem of lookback window selection by introducing a mixture of auto-regressive experts . It also learns more powerful seasonal representations by leveraging a learnable Fourier layer which enables intra-frequency interactions . Both trend and seasonal representations are learned via contrastive loss functions . The trend representations are learned in the time domain , whereas the seasonal representations are learned via a novel frequency domain contrastive loss which encourages discriminative seasonal representations and side steps the issue of determining the period of seasonal patterns present in the data . The contributions of our work are as follows : 1 . We show via a causal perspective , the benefits of learning disentangled seasonal-trend representations for time series forecasting via contrastive learning . 2 . We propose CoST , a time series representation learning approach which leverages inductive biases in the model architecture to learn disentangled seasonal and trend representations , as well as incorporating a novel frequency domain contrastive loss to encourage discriminative seasonal representations . 3 . CoST outperforms existing state-of-the-art approaches by a considerable margin on realworld benchmarks – 21.3 % improvement in MSE for the multivariate setting . We also analyze the benefits of each proposed module , and establish that CoST is robust to various choices of backbone encoders and downstream regressors via extensive ablation studies . 2 SEASONAL-TREND REPRESENTATIONS FOR TIME SERIES . Problem Formulation Let ( x1 , . . .xT ) ∈ RT×m be a time series , where m is the dimension of observed signals . Given lookback window h , our goal is to forecast the next k steps , X̂ = g ( X ) , where X ∈ Rh×m , X̂ ∈ Rk×m , and g ( · ) denotes the prediction mapping function , and X̂ predicts the next k time steps of X . In this work , instead of jointly learning the representation and prediction association through g ( · ) , we focus on learning feature representations from observed data , with the goal of improving predictive performance . Formally , we aim to learn a nonlinear feature embedding function V = f ( X ) , where X ∈ Rh×m and V ∈ Rh×d , to project m-dimensional raw signals into a d-dimensional latent space for each timestamp . Subsequently , the learned representation of the final timestamp vh is used as inputs for the downstream regressor of the forecasting task . Disentangled Seasonal-Trend Representation Learning and Its Causal Interpretation As discussed in Bengio et al . ( 2013 ) , complex data arise from the rich interaction of multiple sources – a good representation should be able to disentangle the various explanatory sources , making it robust to complex and richly structured variations . Not doing so may otherwise lead to capturing spurious features that do not transfer well under non i.i.d . data distribution settings . To achieve this goal , it is necessary to introduce structural priors for time series . Here , we borrow ideas from Bayesian Structural Time Series models ( Scott & Varian , 2015 ; Qiu et al. , 2018 ) . As illustrated in the causal graph in Figure 2 , we assume that the observed time series data X is generated from the error variable E and the error-free latent variable X ? . X ? in turn , is generated from the trend variable T and seasonal variable S. As E is not predictable , the optimal prediction can be achieved if we are able to uncover X ? which only depends on T and S. Firstly , we highlight that existing work using end-to-end deep forecasting methods to directly model the time-lagged relationship and the multivariate interactions along the observed data X . Unfortunately , each X includes unpredictable noise E , which might lead to capturing spurious correlations . Thus , we aim to learn the error-free latent variable X ? . Secondly , by the independent mechanisms assumption ( Peters et al. , 2017 ; Parascandolo et al. , 2018 ) , we can see that the seasonal and trend modules do not influence or inform each other . Therefore , even if one mechanism changes due to a distribution shift , the other remains unchanged . The design of disentangling seasonality and trend leads to better transfer , or generalization in nonstationary environments . Furthermore , independent seasonal and trend mechanisms can be learned independently and be flexibly re-used and re-purposed . We can see that interventions on E does not influence the conditional distribution P ( X ? |T , S ) , i.e . P do ( E=ei ) ( X ? |T , S ) = P do ( E=ej ) ( X ? |T , S ) , for any ei , ej in the domain of E. Thus , S and T are invariant under changes in E. Learning representations for S and T allows us to find a stable association with the optimal prediction ( of X ? ) in terms of various types of errors . Since the targets X ? are unknown , we construct a proxy contrastive learning task inspired by Mitrovic et al . ( 2020 ) . Specifically , we use data augmentations as interventions on the error E and learn invariant representations of T and S via contrastive learning . Since it is impossible to generate all possible variations of errors , we select three typical augmentations : scale , shift and jitter , which can simulate a large and diverse set of errors , beneficial for learning better representations . 3 SEASONAL-TREND CONTRASTIVE LEARNING FRAMEWORK . In this section , we introduce our proposed CoST framework to learn disentangled seasonal-trend representations . We aim to learn representations such that for each time step , we have the disentangled representations for seasonal and trend components , i.e. , V = [ V ( T ) ; V ( S ) ] ∈ Rh×d , where V ( T ) ∈ Rh×dT and V ( S ) ∈ Rh×dS , such that d = dT + dS . Figure 3a illustrates our overall framework . Firstly , we make use of an encoder backbone fb : Rh×m → Rh×d to map the observations to latent space . Next , we construct both the trend and seasonal representations from these intermediate representations . Specifically , the Trend Feature Disentangler ( TFD ) , fT : Rh×d → Rh×dT , extracts the trend representations via a mixture of autoregressive experts and is learned via a time domain contrastive loss Ltime . The Seasonal Feature Disentangler ( SFD ) , fS : Rh×d → Rh×dS , extracts the seasonal representations via a learnable Fourier layer and is learned by a frequency domain contrastive loss which includes an amplitude component , Lamp , and a phase component , Lphase . We give a detailed description of both components in the next section . The model is learned in an end-to-end fashion , with the overall loss function being L = Ltime + α 2 ( Lamp + Lphase ) , where α is a hyper-parameter which balances the trade-off between trend and seasonal factors . Finally , we concatenate the outputs of the Trend and Seasonal Feature Disentanglers to obtain our final output representations . 3.1 TREND FEATURE REPRESENTATIONS . Extracting the underlying trend is crucial for modeling time series . Auto-regressive filtering is one widely used method , as it is able to capture time-lagged causal relationships from past observations . One challenging problem is to select the appropriate lookback window – a smaller window leads to under-fitting , while a larger model leads to over-fitting and over-parameterization issues . A straightforward solution is to optimize this hyper-parameter by grid search on the training or validation loss ( Hyndman & Khandakar , 2008 ) , but such an approach is too computationally expensive . Thus , we propose to use a mixture of auto-regressive experts which can adaptively select the appropriate lookback window . Trend Feature Disentangler ( TFD ) As illustrated in Figure 3b , the TFD is a mixture of L+1 autoregressive experts , where L = blog2 ( h/2 ) c. Each expert is implemented as a 1d causal convolution with d input channels and dT output channels , where the kernel size of the i-th expert is 2i . Each expert outputs a matrix Ṽ ( T , i ) = CausalConv ( Ṽ , 2i ) . Finally , an average-pooling operation is performed over the outputs to obtain the final trend representations , V ( T ) = AvePool ( Ṽ ( T,0 ) , Ṽ ( T,1 ) , . . . , Ṽ ( T , L ) ) = 1 ( L+ 1 ) L∑ i=0 Ṽ ( T , i ) . Time Domain Contrastive Loss We employ a contrastive loss in the time domain to learn discriminative trend representations . Specifically , we apply the MoCo ( He et al. , 2020 ) variant of contrastive learning which makes use of a momentum encoder to obtain representations of the positive pair , and a dynamic dictionary with a queue to obtain negative pairs . We elaborate further on the details of contrastive learning in Appendix A . Then , given N samples and K negative samples , the time domain contrastive loss is Ltime = N∑ i=1 − log exp ( qi · ki/τ ) exp ( qi · ki/τ ) + ∑K j=1 exp ( qi · kj/τ ) , where given a sample V ( T ) , we first select a random time step t for the contrastive loss and apply a projection head , which is a one-layer MLP to obtain q , and k is respectively the augmented version of the corresponding sample from the momentum encoder/dynamic dictionary .

Summary: This paper proposes a novel way to represent time series learnt with contrastive losses in both the time and frequency domain for the forecasting task. The approach is novel and timely and is clearly described. The authors make a connection to causality which they use also for designing data augmentation schemes.