MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data

1 Introduction . Time series forecasting has proven to be important to help people manage resources and make decisions [ 20 ] . For example , probabilistic forecasting of product demand and supply in retails [ 9 ] , or the forecasting of loans [ 1 ] in a financial institution can help people do inventory or financing planning to maximize the profit . Most time series of interest are macroscopic time series , e.g. , the sales of an online retail platform , the loans of a financial institution , or the number of infections caused by some pandemic diseases in a state , that are comprised of microscopic time series , e.g. , the sales of a merchant in the online retail , the loans from a customer given the financial institution , or the number of infections in a certain region . That is , the observed macroscopic time series are just the aggregation or sum of microscopic time series . ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Although various time series forecasting models , e.g. , State Space Models ( SSMs ) [ 13 ] , Autoregressive ( AR ) models [ 2 ] , or deep neural networks [ 5 ] , have been widely studied for decades , all of them study the modeling of time series without considering the connections between macroscopic time series of interest and the underlying time series on the microscopic level . In this paper , we study the question whether the forecasting of macroscopic time series can be improved by leveraging the underlying microscopic time series , and the answer is yes . Basically , though accurately modeling each microscopic time series could be challenging due to large variations , we show that by carefully clustering microscopic time series into clusters , i.e. , clustered time series , and using canonical approaches to model each of clusters , finally we can achieve promising results by simply summing over the forecasting results of each cluster . To be more specific , first , we assume that the microscopic time series are generated from a probabilistic mixture model [ 24 ] where there exist K components . The generation of each microscopic time series is by first selecting a component z from { 1 , ... , K } with a prior p ( z ) ( a Discrete distribution ) , then generating the microscopic observation from a probabilistic distribution p ( x ; Φz , z ) parameterized by the corresponding component Φz . We show that as we can identify the ground truth components of the mixture , and the ground truth assignment of each microscopic observation , independent modeling of time series data from each component could be improved due to lower variance , and further benefitting the estimation of macroscopic time series that are of interest . Second , inspired by recent successes of Seq2seq models [ 36 , 10 , 12 ] based on deep neural networks , e.g. , variants of recurrent neural networks ( RNNs ) [ 16 , 41 , 22 ] , convolutional neural networks ( CNNs ) [ 4 , 15 ] , and Transformers [ 19 , 38 ] , we propose Mixture of Seq2seq ( MixSeq ) , a mixture model for time series , where the components come from a family of Seq2seq models parameterized by different parameters . Third , we conduct synthetic experiments to demonstrate the superiority of our approach , and extensive experiments on real-world data to show the power of our approach compared with canonical approaches . Our contributions . We summarize our contributions in two-fold . ( 1 ) We show that by transforming the original macroscopic time series via clustering , the expected variance of each clustered time series could be optimized , thus improving the accuracy and robustness for the estimation of macroscopic time series . ( 2 ) We propose MixSeq which is an end2end mixture model with each component coming from a family of Seq2seq models . Our empirical results based on MixSeq show the superiority compared with canonical approaches . 2 Background . In this section , we first give a formal problem definition . We then review the bases related to this work , and have a discussion of related works . Problem definition . Let us assume a macroscopic time series x1 : t0 = [ x1 , ... , xt0 ] , and xt ∈ R denotes the value of time series at time t. We aim to predict the next τ time steps , i.e. , xt0+1 : t0+τ . We are interested in the following conditional distribution p ( xt0+1 : t0+τ |x1 : t0 ) = t0+τ∏ t=t0+1 p ( xt|x < t ; Θ ) , ( 1 ) where x < t represents x1 : t−1 in interval [ 1 , t ) . To study the above problem , we assume that the macroscopic time series is comprised of m microscopic time series , i.e. , xt = ∑m i=1 xi , t where xi , t ∈ R denotes the value of the i-th microscopic time series at time t. We aim to cluster the m microscopic time series into K clustered time series { x ( z ) 1 : t0 } K z=1 , where x ( z ) t = ∑ { i|zi=z , ∀i } xi , t given the label assignment of the i-th microscopic time series zi ∈ { 1 , ... , K } . This is based on our results in Section 3 that the macroscopic time series forecasting can be improved with optimal clustering . Hence , instead of directly modeling p ( xt0+1 : t0+τ ) , we study the clustering of m microscopic time series in Section 4 , and model the conditional distribution of clustered time series { p ( x ( z ) t0+1 : t0+τ ) } K z=1 with canonical approaches . 2.1 Seq2seq : encoder-decoder architectures for time series . An encoder-decoder based neural network models the conditional distribution Eq . ( 1 ) as a distribution from the exponential families , e.g. , Gaussian , Gamma or Binomial distributions , with sufficient statistics generated from a neural network . The encoder feeds x < t into a neural architecture , e.g. , RNNs , CNNs or self-attentions , to generate the representation of historical time series , denoted as ht , then we use a decoder to yield the result xt . After τ iterations in an autoregressive style , it finally generates the whole time series to be predicted . To instantiate above Seq2seq architecture , we denote o1 : t , where ot ∈ Rd , as covariates that are known a priori , e.g. , dates . We denote Yt = [ x1 : t−1 ‖ o2 : t ] ∈ R ( t−1 ) × ( d+1 ) where we use ‖ for concatenation . The encoder generates the representation ht of x < t via Transformer [ 36 , 19 ] as follows . We first transform Yt by some functions ρ ( · ) , e.g. , causal convolution [ 19 ] to H ( 0 ) = ρ ( Yt ) ∈ R ( t−1 ) ×dk . Transformer then iterates the following self-attention layer L times : H ( l ) = MLP ( l ) ( H ( tmp ) ) , H ( tmp ) = SOFTMAX ( Q ( l ) K ( l ) > √ dq M ) V ( l ) , Q ( l ) = H ( l−1 ) W ( l ) q , K ( l ) = H ( l−1 ) W ( l ) k , V ( l ) = H ( l−1 ) W ( l ) v . ( 2 ) That is , we first transform Y 3 into query , key , and value matrices , i.e. , Q = YWq , K = YWk , and V = YWv respectively , where Wq ∈ Rdk×dq , Wk ∈ Rdk×dq , Wv ∈ Rdk×dv in each layer are learnable parameters . Then we do scaled inner product attention to yield H ( l ) ∈ R ( t−1 ) ×dk where M is a mask matrix to filter out rightward attention by setting all upper triangular elements to −∞ . We denote MLP ( · ) as a multi-layer perceptron function . Afterwards , we can generate the representation ht ∈ Rdp for x < t via ht = ν ( H ( L ) ) where we denote ν ( · ) as a deep set function [ 42 ] that operates on rows of H ( L ) , i.e. , ν ( { H ( L ) 1 , ... , H ( L ) t−1 } ) . We denote the feedforward function to generate H ( L ) as H ( L ) ∼ g ( H ( 0 ) ) , i.e. , Eq ( 2 ) . Given ht , the decoder generates the sufficient statistics and finally yields xt ∼ p ( x ; MLP ( ht ) ) from a distribution in the exponential family . 2.2 Related works . Time series forecasting has been studied for decades . We summarize works related to time series forecasting into two categories . First , many models come from the family of autoregressive integrated moving average ( ARIMA ) [ 7 , 2 ] , where AR indicates that the evolving variable of interest is regressed on its own lagged values , the MA indicates that the regression error is actually a linear combination of error terms , and the “ I ” indicates that the data values have been replaced with the difference between their values and the previous values to handle non-stationary [ 27 ] . The State Space Models ( SSM ) [ 13 ] aim to use state transition function to model the transfer of states and generate observations via a observation function . These statistical approaches typically model time series independently , and most of them only utilize values from history but ignore covariates that are important signals for forecasting . Second , as rapid development of deep neural networks , people started studying many neural networks for the modeling of time series [ 5 , 20 ] . Most successful neural networks are based on the encoder-decoder architectures [ 36 , 10 , 12 , 33 , 3 , 11 , 20 , 21 ] , namely Seq2seq . Basically , various Seq2seq models based on RNNs [ 16 , 32 , 37 , 18 , 41 , 22 ] , CNNs [ 4 , 15 ] , and Transformers ( self-attentions ) [ 19 , 38 ] are proposed to model the non-linearity for time series . No matter models studied in statistics or deep neural networks , these works mainly focus on the forecasting of single or multivariate time series , but ignore the auxiliary information that the time series could be made up of microscopic data . Time series clustering is another topic for exploratory analysis of time series . We summarize the literature into three categories , i.e. , study of distance functions , generative models , and feature extraction for time series . First , Dynamic time wrapping [ 28 ] , similarity metric that measures temporal dynamics [ 40 ] , and specific measures for the shape [ 26 ] of time series are proposed to adapt to various time series characteristics , e.g. , scaling and distortion . Typically these distance functions are mostly manually defined and can not generalize to more general settings . Second , generative 3We ignore the subscript for simplicity in condition that the context is of clarity . model based approaches assume that the observed time series is generated by an underlying model , such as hidden markov model [ 25 ] or mixture of ARMA [ 39 ] . Third , early studies on feature extraction of time series are based on component analysis [ 14 ] , and kernels , e.g. , u-shapelet [ 43 ] . As the development of deep neural networks , several encoder-decoder architectures [ 23 , 21 ] are proposed to learn better representations of time series for clustering . However , the main purpose of works in this line is to conduct exploratory analysis of time series , while their usage for time series forecasting has never been studied . That is , these works define various metrics to evaluate the goodness of the clustering results , but how to learn the optimal clustering for time series forecasting remains an open question .

Summary: Authors model time series as a weighted mixture of a set of time series for forecasting. Proposed a model for Mixture of seq2seq model. Number of components depends on correct clustering of micro time series. The paper also  propose an efficient stochastic auto-encoding variational Bayesian algorithm for the posterior inference and learning for MixSeq