MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data

1 Introduction . Time series forecasting has proven to be important to help people manage resources and make decisions [ 20 ] . For example , probabilistic forecasting of product demand and supply in retails [ 9 ] , or the forecasting of loans [ 1 ] in a financial institution can help people do inventory or financing planning to maximize the profit . Most time series of interest are macroscopic time series , e.g. , the sales of an online retail platform , the loans of a financial institution , or the number of infections caused by some pandemic diseases in a state , that are comprised of microscopic time series , e.g. , the sales of a merchant in the online retail , the loans from a customer given the financial institution , or the number of infections in a certain region . That is , the observed macroscopic time series are just the aggregation or sum of microscopic time series . ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Although various time series forecasting models , e.g. , State Space Models ( SSMs ) [ 13 ] , Autoregressive ( AR ) models [ 2 ] , or deep neural networks [ 5 ] , have been widely studied for decades , all of them study the modeling of time series without considering the connections between macroscopic time series of interest and the underlying time series on the microscopic level . In this paper , we study the question whether the forecasting of macroscopic time series can be improved by leveraging the underlying microscopic time series , and the answer is yes . Basically , though accurately modeling each microscopic time series could be challenging due to large variations , we show that by carefully clustering microscopic time series into clusters , i.e. , clustered time series , and using canonical approaches to model each of clusters , finally we can achieve promising results by simply summing over the forecasting results of each cluster . To be more specific , first , we assume that the microscopic time series are generated from a probabilistic mixture model [ 24 ] where there exist K components . The generation of each microscopic time series is by first selecting a component z from { 1 , ... , K } with a prior p ( z ) ( a Discrete distribution ) , then generating the microscopic observation from a probabilistic distribution p ( x ; Φz , z ) parameterized by the corresponding component Φz . We show that as we can identify the ground truth components of the mixture , and the ground truth assignment of each microscopic observation , independent modeling of time series data from each component could be improved due to lower variance , and further benefitting the estimation of macroscopic time series that are of interest . Second , inspired by recent successes of Seq2seq models [ 36 , 10 , 12 ] based on deep neural networks , e.g. , variants of recurrent neural networks ( RNNs ) [ 16 , 41 , 22 ] , convolutional neural networks ( CNNs ) [ 4 , 15 ] , and Transformers [ 19 , 38 ] , we propose Mixture of Seq2seq ( MixSeq ) , a mixture model for time series , where the components come from a family of Seq2seq models parameterized by different parameters . Third , we conduct synthetic experiments to demonstrate the superiority of our approach , and extensive experiments on real-world data to show the power of our approach compared with canonical approaches . Our contributions . We summarize our contributions in two-fold . ( 1 ) We show that by transforming the original macroscopic time series via clustering , the expected variance of each clustered time series could be optimized , thus improving the accuracy and robustness for the estimation of macroscopic time series . ( 2 ) We propose MixSeq which is an end2end mixture model with each component coming from a family of Seq2seq models . Our empirical results based on MixSeq show the superiority compared with canonical approaches . 2 Background . In this section , we first give a formal problem definition . We then review the bases related to this work , and have a discussion of related works . Problem definition . Let us assume a macroscopic time series x1 : t0 = [ x1 , ... , xt0 ] , and xt ∈ R denotes the value of time series at time t. We aim to predict the next τ time steps , i.e. , xt0+1 : t0+τ . We are interested in the following conditional distribution p ( xt0+1 : t0+τ |x1 : t0 ) = t0+τ∏ t=t0+1 p ( xt|x < t ; Θ ) , ( 1 ) where x < t represents x1 : t−1 in interval [ 1 , t ) . To study the above problem , we assume that the macroscopic time series is comprised of m microscopic time series , i.e. , xt = ∑m i=1 xi , t where xi , t ∈ R denotes the value of the i-th microscopic time series at time t. We aim to cluster the m microscopic time series into K clustered time series { x ( z ) 1 : t0 } K z=1 , where x ( z ) t = ∑ { i|zi=z , ∀i } xi , t given the label assignment of the i-th microscopic time series zi ∈ { 1 , ... , K } . This is based on our results in Section 3 that the macroscopic time series forecasting can be improved with optimal clustering . Hence , instead of directly modeling p ( xt0+1 : t0+τ ) , we study the clustering of m microscopic time series in Section 4 , and model the conditional distribution of clustered time series { p ( x ( z ) t0+1 : t0+τ ) } K z=1 with canonical approaches . 2.1 Seq2seq : encoder-decoder architectures for time series . An encoder-decoder based neural network models the conditional distribution Eq . ( 1 ) as a distribution from the exponential families , e.g. , Gaussian , Gamma or Binomial distributions , with sufficient statistics generated from a neural network . The encoder feeds x < t into a neural architecture , e.g. , RNNs , CNNs or self-attentions , to generate the representation of historical time series , denoted as ht , then we use a decoder to yield the result xt . After τ iterations in an autoregressive style , it finally generates the whole time series to be predicted . To instantiate above Seq2seq architecture , we denote o1 : t , where ot ∈ Rd , as covariates that are known a priori , e.g. , dates . We denote Yt = [ x1 : t−1 ‖ o2 : t ] ∈ R ( t−1 ) × ( d+1 ) where we use ‖ for concatenation . The encoder generates the representation ht of x < t via Transformer [ 36 , 19 ] as follows . We first transform Yt by some functions ρ ( · ) , e.g. , causal convolution [ 19 ] to H ( 0 ) = ρ ( Yt ) ∈ R ( t−1 ) ×dk . Transformer then iterates the following self-attention layer L times : H ( l ) = MLP ( l ) ( H ( tmp ) ) , H ( tmp ) = SOFTMAX ( Q ( l ) K ( l ) > √ dq M ) V ( l ) , Q ( l ) = H ( l−1 ) W ( l ) q , K ( l ) = H ( l−1 ) W ( l ) k , V ( l ) = H ( l−1 ) W ( l ) v . ( 2 ) That is , we first transform Y 3 into query , key , and value matrices , i.e. , Q = YWq , K = YWk , and V = YWv respectively , where Wq ∈ Rdk×dq , Wk ∈ Rdk×dq , Wv ∈ Rdk×dv in each layer are learnable parameters . Then we do scaled inner product attention to yield H ( l ) ∈ R ( t−1 ) ×dk where M is a mask matrix to filter out rightward attention by setting all upper triangular elements to −∞ . We denote MLP ( · ) as a multi-layer perceptron function . Afterwards , we can generate the representation ht ∈ Rdp for x < t via ht = ν ( H ( L ) ) where we denote ν ( · ) as a deep set function [ 42 ] that operates on rows of H ( L ) , i.e. , ν ( { H ( L ) 1 , ... , H ( L ) t−1 } ) . We denote the feedforward function to generate H ( L ) as H ( L ) ∼ g ( H ( 0 ) ) , i.e. , Eq ( 2 ) . Given ht , the decoder generates the sufficient statistics and finally yields xt ∼ p ( x ; MLP ( ht ) ) from a distribution in the exponential family . 2.2 Related works . Time series forecasting has been studied for decades . We summarize works related to time series forecasting into two categories . First , many models come from the family of autoregressive integrated moving average ( ARIMA ) [ 7 , 2 ] , where AR indicates that the evolving variable of interest is regressed on its own lagged values , the MA indicates that the regression error is actually a linear combination of error terms , and the “ I ” indicates that the data values have been replaced with the difference between their values and the previous values to handle non-stationary [ 27 ] . The State Space Models ( SSM ) [ 13 ] aim to use state transition function to model the transfer of states and generate observations via a observation function . These statistical approaches typically model time series independently , and most of them only utilize values from history but ignore covariates that are important signals for forecasting . Second , as rapid development of deep neural networks , people started studying many neural networks for the modeling of time series [ 5 , 20 ] . Most successful neural networks are based on the encoder-decoder architectures [ 36 , 10 , 12 , 33 , 3 , 11 , 20 , 21 ] , namely Seq2seq . Basically , various Seq2seq models based on RNNs [ 16 , 32 , 37 , 18 , 41 , 22 ] , CNNs [ 4 , 15 ] , and Transformers ( self-attentions ) [ 19 , 38 ] are proposed to model the non-linearity for time series . No matter models studied in statistics or deep neural networks , these works mainly focus on the forecasting of single or multivariate time series , but ignore the auxiliary information that the time series could be made up of microscopic data . Time series clustering is another topic for exploratory analysis of time series . We summarize the literature into three categories , i.e. , study of distance functions , generative models , and feature extraction for time series . First , Dynamic time wrapping [ 28 ] , similarity metric that measures temporal dynamics [ 40 ] , and specific measures for the shape [ 26 ] of time series are proposed to adapt to various time series characteristics , e.g. , scaling and distortion . Typically these distance functions are mostly manually defined and can not generalize to more general settings . Second , generative 3We ignore the subscript for simplicity in condition that the context is of clarity . model based approaches assume that the observed time series is generated by an underlying model , such as hidden markov model [ 25 ] or mixture of ARMA [ 39 ] . Third , early studies on feature extraction of time series are based on component analysis [ 14 ] , and kernels , e.g. , u-shapelet [ 43 ] . As the development of deep neural networks , several encoder-decoder architectures [ 23 , 21 ] are proposed to learn better representations of time series for clustering . However , the main purpose of works in this line is to conduct exploratory analysis of time series , while their usage for time series forecasting has never been studied . That is , these works define various metrics to evaluate the goodness of the clustering results , but how to learn the optimal clustering for time series forecasting remains an open question .

Summary: This paper develops methodology for improving time series forecasting by modeling finer-scale structure in the time series; namely, the underlying "microscopic" time series of which the predicted "macroscopic" time series is aggregated. The approach takes advantage of the concept that in many cases the time series of interest are in fact macroscopic aggregates of smaller-scale structure and time series. The work develops a method that models the micro time series as a mixture of latent components and then uses a neutral network based approach to model and parameterize the mixture. Benchmarkins are done with simulated and real data, showing promising performance.