MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data
Year: 2021

Abstract:
Time series forecasting is widely used in business intelligence, e.g., forecast stock market price, sales, and help the analysis of data trend. Most time series of interest are macroscopic time series that are aggregated from microscopic data. However, instead of directly modeling the macroscopic time series, rare literature studied the forecasting of macroscopic time series by leveraging data on the microscopic level. In this paper, we assume that the microscopic time series follow some unknown mixture probabilistic distributions. We theoretically show that as we identify the ground truth latent mixture components, the estimation of time series from each component could be improved because of lower variance, thus benefitting the estimation of macroscopic time series as well. Inspired by the power of Seq2seq and its variants on the modeling of time series data, we propose Mixture of Seq2seq (MixSeq), an end2end mixture model to cluster microscopic time series, where all the components come from a family of Seq2seq models parameterized by different parameters. Extensive experiments on both synthetic and real-world data show the superiority of our approach.

heading: 1 Introduction
text: Time series forecasting has proven to be important to help people manage resources and make decisions [20]. For example, probabilistic forecasting of product demand and supply in retails [9], or the forecasting of loans [1] in a financial institution can help people do inventory or financing planning to maximize the profit. Most time series of interest are macroscopic time series, e.g., the sales of an online retail platform, the loans of a financial institution, or the number of infections caused by some pandemic diseases in a state, that are comprised of microscopic time series, e.g., the sales of a merchant in the online retail, the loans from a customer given the financial institution, or the number of infections in a certain region. That is, the observed macroscopic time series are just the aggregation or sum of microscopic time series.
∗Equal contribution. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Although various time series forecasting models, e.g., State Space Models (SSMs) [13], Autoregressive (AR) models [2], or deep neural networks [5], have been widely studied for decades, all of them study the modeling of time series without considering the connections between macroscopic time series of interest and the underlying time series on the microscopic level.
In this paper, we study the question whether the forecasting of macroscopic time series can be improved by leveraging the underlying microscopic time series, and the answer is yes. Basically, though accurately modeling each microscopic time series could be challenging due to large variations, we show that by carefully clustering microscopic time series into clusters, i.e., clustered time series, and using canonical approaches to model each of clusters, finally we can achieve promising results by simply summing over the forecasting results of each cluster.
To be more specific, first, we assume that the microscopic time series are generated from a probabilistic mixture model [24] where there exist K components. The generation of each microscopic time series is by first selecting a component z from {1, ...,K} with a prior p(z) (a Discrete distribution), then generating the microscopic observation from a probabilistic distribution p(x; Φz, z) parameterized by the corresponding component Φz . We show that as we can identify the ground truth components of the mixture, and the ground truth assignment of each microscopic observation, independent modeling of time series data from each component could be improved due to lower variance, and further benefitting the estimation of macroscopic time series that are of interest. Second, inspired by recent successes of Seq2seq models [36, 10, 12] based on deep neural networks, e.g., variants of recurrent neural networks (RNNs) [16, 41, 22], convolutional neural networks (CNNs) [4, 15], and Transformers [19, 38], we propose Mixture of Seq2seq (MixSeq), a mixture model for time series, where the components come from a family of Seq2seq models parameterized by different parameters. Third, we conduct synthetic experiments to demonstrate the superiority of our approach, and extensive experiments on real-world data to show the power of our approach compared with canonical approaches.
Our contributions. We summarize our contributions in two-fold. (1) We show that by transforming the original macroscopic time series via clustering, the expected variance of each clustered time series could be optimized, thus improving the accuracy and robustness for the estimation of macroscopic time series. (2) We propose MixSeq which is an end2end mixture model with each component coming from a family of Seq2seq models. Our empirical results based on MixSeq show the superiority compared with canonical approaches.

heading: 2 Background
text: In this section, we first give a formal problem definition. We then review the bases related to this work, and have a discussion of related works.
Problem definition. Let us assume a macroscopic time series x1:t0 = [x1, ..., xt0 ], and xt ∈ R denotes the value of time series at time t. We aim to predict the next τ time steps, i.e., xt0+1:t0+τ . We are interested in the following conditional distribution
p(xt0+1:t0+τ |x1:t0) = t0+τ∏ t=t0+1 p(xt|x<t; Θ), (1)
where x<t represents x1:t−1 in interval [1, t). To study the above problem, we assume that the macroscopic time series is comprised of m microscopic time series, i.e., xt = ∑m i=1 xi,t where xi,t ∈ R denotes the value of the i-th microscopic time series at time t. We aim to cluster the m
microscopic time series into K clustered time series { x (z) 1:t0 }K z=1 , where x(z)t = ∑ {i|zi=z,∀i} xi,t given the label assignment of the i-th microscopic time series zi ∈ {1, ...,K}. This is based on our results in Section 3 that the macroscopic time series forecasting can be improved with optimal clustering. Hence, instead of directly modeling p(xt0+1:t0+τ ), we study the clustering of m microscopic time series in Section 4, and model the conditional distribution of clustered time series{ p(x
(z) t0+1:t0+τ ) }K z=1 with canonical approaches.

heading: 2.1 Seq2seq: encoder-decoder architectures for time series
text: An encoder-decoder based neural network models the conditional distribution Eq. (1) as a distribution from the exponential families, e.g., Gaussian, Gamma or Binomial distributions, with sufficient statistics generated from a neural network. The encoder feeds x<t into a neural architecture, e.g., RNNs, CNNs or self-attentions, to generate the representation of historical time series, denoted as ht, then we use a decoder to yield the result xt. After τ iterations in an autoregressive style, it finally generates the whole time series to be predicted.
To instantiate above Seq2seq architecture, we denote o1:t, where ot ∈ Rd, as covariates that are known a priori, e.g., dates. We denote Yt = [x1:t−1 ‖ o2:t] ∈ R(t−1)×(d+1) where we use ‖ for concatenation. The encoder generates the representation ht of x<t via Transformer [36, 19] as follows. We first transform Yt by some functions ρ(·), e.g., causal convolution [19] to H(0) = ρ(Yt) ∈ R(t−1)×dk . Transformer then iterates the following self-attention layer L times:
H(l) = MLP(l)(H(tmp)), H(tmp) = SOFTMAX ( Q(l)K(l)>√
dq M
) V (l),
Q(l) = H(l−1)W (l)q ,K (l) = H(l−1)W (l) k , V (l) = H(l−1)W (l)v .
(2)
That is, we first transform Y 3 into query, key, and value matrices, i.e., Q = YWq, K = YWk, and V = YWv respectively, where Wq ∈ Rdk×dq ,Wk ∈ Rdk×dq ,Wv ∈ Rdk×dv in each layer are learnable parameters. Then we do scaled inner product attention to yield H(l) ∈ R(t−1)×dk where M is a mask matrix to filter out rightward attention by setting all upper triangular elements to −∞. We denote MLP(·) as a multi-layer perceptron function. Afterwards, we can generate the representation ht ∈ Rdp for x<t via ht = ν(H(L)) where we denote ν(·) as a deep set function [42] that operates on rows of H(L), i.e., ν({H(L)1 , ...,H (L) t−1}). We denote the feedforward function to generate H(L) as H(L) ∼ g(H(0)), i.e., Eq (2). Given ht, the decoder generates the sufficient statistics and finally yields xt ∼ p(x; MLP(ht)) from a distribution in the exponential family.

heading: 2.2 Related works
text: Time series forecasting has been studied for decades. We summarize works related to time series forecasting into two categories. First, many models come from the family of autoregressive integrated moving average (ARIMA) [7, 2], where AR indicates that the evolving variable of interest is regressed on its own lagged values, the MA indicates that the regression error is actually a linear combination of error terms, and the “I” indicates that the data values have been replaced with the difference between their values and the previous values to handle non-stationary [27]. The State Space Models (SSM) [13] aim to use state transition function to model the transfer of states and generate observations via a observation function. These statistical approaches typically model time series independently, and most of them only utilize values from history but ignore covariates that are important signals for forecasting. Second, as rapid development of deep neural networks, people started studying many neural networks for the modeling of time series [5, 20]. Most successful neural networks are based on the encoder-decoder architectures [36, 10, 12, 33, 3, 11, 20, 21], namely Seq2seq. Basically, various Seq2seq models based on RNNs [16, 32, 37, 18, 41, 22], CNNs [4, 15], and Transformers (self-attentions) [19, 38] are proposed to model the non-linearity for time series.
No matter models studied in statistics or deep neural networks, these works mainly focus on the forecasting of single or multivariate time series, but ignore the auxiliary information that the time series could be made up of microscopic data.
Time series clustering is another topic for exploratory analysis of time series. We summarize the literature into three categories, i.e., study of distance functions, generative models, and feature extraction for time series. First, Dynamic time wrapping [28], similarity metric that measures temporal dynamics [40], and specific measures for the shape [26] of time series are proposed to adapt to various time series characteristics, e.g., scaling and distortion. Typically these distance functions are mostly manually defined and cannot generalize to more general settings. Second, generative
3We ignore the subscript for simplicity in condition that the context is of clarity.
model based approaches assume that the observed time series is generated by an underlying model, such as hidden markov model [25] or mixture of ARMA [39]. Third, early studies on feature extraction of time series are based on component analysis [14], and kernels, e.g., u-shapelet [43]. As the development of deep neural networks, several encoder-decoder architectures [23, 21] are proposed to learn better representations of time series for clustering.
However, the main purpose of works in this line is to conduct exploratory analysis of time series, while their usage for time series forecasting has never been studied. That is, these works define various metrics to evaluate the goodness of the clustering results, but how to learn the optimal clustering for time series forecasting remains an open question.

heading: 3 Microscopic time series under mixture model
text: We analyze the variance of mixture models, and further verify our results with simple toy examples.

heading: 3.1 Analyses on the variance of mixture model
text: In this part, we analyze the variance of probabilistic mixture models. A mixture model [24] is a probabilistic model for representing the presence of subpopulations within an overall population. Mixture model typically consists of a prior that represents the probability over subpopulations, and components, each of which defines the probability distribution of the corresponding subpopulation. Formally, we can write
f(x) = ∑ i pi · fi(x), (3)
where f(·) denotes the mixture distribution, pi denotes the prior over subpopulations, and fi(·) represents the distribution corresponding to the i-th component.
Proposition 1. Assuming the mixture model with probability density function f(x), and corrsponding components {fi(x)}Ki=1 with constants {pi} K i=1 ({pi} K i=1 lie in a simplex), we have f(x) =∑
i pifi(x). In condition that f(·) and {fi(·)} K i=1 have first and second moments, i.e., µ (1) and µ(2) for f(x), and { µ (1) i }K i=1 and { µ (2) i }K i=1
for components {fi(x)}Ki=1, we have:∑ i pi ·Var(fi) ≤ Var(f). (4)
We use the fact that µ(k) = ∑ i piµ (k) i . By using Jensen’s Inequality on ∑ i pi ( µ (1) i )2 ≥(∑
i piµ (1) i
)2 , we immediately yield the result. See detailed proofs in supplementary.
This proposition states that, if we have limited data samples (always the truth in reality) and in case we know the ground truth data generative process a priori, i.e., the exact generative process of each sample from its corresponding component, the variance on expectation conditioned on the ground truth data assignment should be no larger than the variance of the mixture. Based on the assumption that microscopic data are independent, the variance of the aggregation of clustered data should be at least no larger than the aggregation of all microscopic data, i.e., the macroscopic data. So the modeling of clustered data from separate components could possibly be more accurate and robust compared with the modeling of macroscopic data. This result motivates us to forecast macroscopic time series by clustering the underlying microscopic time series. Essentially, we transform the original macroscopic time series data to clusters with lower variances using a clustering approach, then followed by any time series models to forecast each clustered time series. After that, we sum over all the results from those clusters so as to yield the forecasting of macroscopic time series. We demonstrate this result with toy examples next.

heading: 3.2 Demonstration with toy examples
text: We demonstrate the effectiveness of forecasting macroscopic time series by aggregating the forecasting results from clustered time series.
Simulation setting. We generate microscopic time series from a mixture model, such as Gaussian process (GP) [31] or ARMA [8] with 3 or 5 components. We generate 5 time series for each component, and yield 15 or 25 microscopic time series in total. We sum all the time series as the macroscopic time series. We get clustered time series by simply summing microscopic time series from the same component. Our purpose is to compare the performance between forecasting results directly on macroscopic time series (macro results) and sum of forecasting results of clustered time series (clustered results). We set the length of time series as 360, and use rolling window approach for training and validating our results in the last 120 time steps (i.e., at each time step, we train the model using the time series before current time point, and validate using the following 30 values). We fit the data with either GP or ARMA depending on the generative model. We describe the detailed simulation parameters of mixture models in supplementary.
Simulation results. Table 1 shows the results measured by symmetric mean absolute percentage error (SMAPE)4. It is obvious that no matter time series generated by mixture of GP or mixure of ARMA, the clustered results are superior to macro results. In other words, if we knew the ground truth component of each microscopic time series, modeling on clustered data aggregated by time series from the same component would have better results compared with directly modeling the macroscopic time series.

heading: 4 MixSeq: a mixture model for time series
text: Based on the analysis in Section 3, we assume microscopic time series follow a mixture distribution, and propose a mixture model, MixSeq, to cluster microscopic time series.

heading: 4.1 Our model
text: Our model assumes that each ofmmicroscopic time series follows a mixture probabilistic distribution. To forecast xt0+1:t0+τ given x1:t0 , our approach is to first partition {xi,1:t0} m i=1 into K clusters via MixSeq. Since the distribution is applicable to all microscopic time series, we ignore the subscript i and time interval 1 : t0 for simplicity. We study the following generative probability of x ∈ Rt0 :
p(x) = ∑ z p(x, z) = ∑ z p(z)p(x|z) = ∑ z p(z) t0∏ t=1 p(xt|x<t, z; Φz), (5)
where z ∈ {1, 2, ...,K} is the discrete latent variable, K is the number of components in the mixture model, p(z) is the prior of cluster indexed by z, and p(x|z) is the probability of time series x generated by the corresponding component governed by parameter Φz . Note that we have K parameters Θ = {Φ1,Φ2, . . . ,ΦK} in the mixture model. We use ConvTrans introduced in [19] as our backbone to model the conditional p(xt|x<t, z; Φz). To instantiate, we model the conditional p(xt|x<t, z; Φz) by first generating H(0) ∼ ρ(Yt) via causal convolution [19]. We then generate H(L) ∼ g(H(0)) ∈ R(t−1)×dk given x<t. Finally we generate the representation for x<t as ht = ν(H(L)) = σ(Ws ∑t−1 j=1H (L) j ) ∈ Rdp , where Ws ∈ Rdp×dk and σ as ReLU activation function. Afterwards, we decode ht to form the specific distribution from an exponential family. In particular, we use Gaussian distribution p(xt|x<t, z; Φz) = N (xt;µt, σ2t ), where the mean and variance can be generated by following transformations,
µt = w T µht + bµ, σ 2 t = log(1 + exp(w T σ ht + bσ)), (6)
4Details are in supplementary
where wµ, wσ ∈ Rdp are parameters, and bµ, bσ ∈ R are biases.

heading: 4.2 Posterior inference and learning algorithms
text: We aim to learn the parameter Θ and efficiently infer the posterior distribution of p(z|x) in Eq. (5). However, it is intractable to maximize the marginal likelihood p(x) after taking logarithm, i.e., log p(x), since of the involvement of logarithm of sum. To tackle this non-convex problem, we resort to stochastic auto-encoding variational Bayesian algorithm (AEVB) [17]. Regarding single microscopic time series, the variational lower bound (LB) [17] on the marginal likelihood is as below.
log p(x) = log ∑ z p(x, z) ≥ ∑ z q(z|x) log p(x, z) q(z|x)
= Eq(z|x) log p(x|z)−KL (q(z|x)‖p(z)) = LB, (7)
where q(z|x) is the approximated posterior of the latent variable z given time series x. The benefit of using AEVB is that we can treat q(z|x) as an encoder modeled by a neural network. Hence, we reuse the ConvTrans [19] as our backbone, and model q(z|x) as:
q(z|x) = SOFTMAX(Wa · ν(H(L))), H(L) = g(ρ(Yt0)), (8) where we denote Yt0 = [x1:t0‖o1:t0 ] ∈ Rt0×(d+1), ν(H(L)) = σ(Ws · ∑t0 j=1H (L) j ) with parameter Ws ∈ Rdp×dk is the deep set function, and Wa ∈ RK×dp as parameters to project the encoding to K dimension. After the softmax operator, we derive the posterior distribution that lies in a simplex of K dimension. Note that we use distinct ρ(·)’s, g(·)’s and ν(·)’s with different parameters to model q(z|x) and {p(xt|x<t, z)}Kz=1 respectively. We assign each microscopic xi to cluster zi = arg max
z q(z|xi)
in our experiments.
Mode collapsing. We find that directly optimizing the lower bound in Eq. (7) suffers from the mode collapsing problem. That is, the encoder q(z|x) tends to assign all microscopic time series to one cluster, and does not effectively distinguish the data as expected, thus implying I(x; z) = 0 (I(·) for mutual information). In order to address the above mode collapsing problem, we add I(x; z) to the lower bound in Eq. (7) which expects that the latent variable z can extract discriminative information from different time series [44]. Then, we have
Ex(Eq(z|x) log p(x|z))− Ex(KL(q(z|x)‖p(z))) + I(x; z) = Ex(Eq(z|x) log p(x|z))−KL(q(z)‖p(z)),
(9)
where q(z) = 1m ∑m i=1 q(z|xi) is an average of approximated posteriors over all microscopic data.
We approximate this term by using a mini-batch of m′ samples, i.e., q(z) = 1m′ ∑ i′∈B q(z|xi′).
Annealing tricks. Regarding long-length time series, the reconstruction loss and KL divergence in Eq. (9) are out of proportion. In this situation, the KL divergence has few effects on the optimization objective. So we finally derive the following objective to maximize:
Ex(Eq(z|x) log p(x|z))− α ·KL(q(z)‖p(z))− λ · ‖Θ‖ (10) where α is the trade-off hyperparameter. We use the following annealing strategy α = max(a, b× e(−βn)) to dynamically adjust α in the training process, where β is the parameter controlling the rate of descent. Meanwhile, we also involve the `2-norm regularizers on Seq2seq’s parameters Θ with hyperparameter λ ≥ 0.

heading: 5 Experimental results
text: We conduct extensive experiments to show the advantage of MixSeq. We evaluate the clustering performance of MixSeq on synthetic data, present the results of macroscopic time series forecasting on real-world data, and analyze the sensitivity of the cluster number of MixSeq.

heading: 5.1 Synthetic datasets
text: To demonstrate MixSeq’s capability of clustering microscopic time series that follow various probabilistic mixture distributions, we conduct clustering experiments on synthetic data with ground truth.
We generate two kinds of synthetic time series by ARMA [8] and DeepAR [32] respectively. For each model, we experiment with different number of clusters (2 and 3) generated with components governed by different parameters.
Experiment setting. To generate data from ARMA, we use ARMA(2, 0) and xt = φ1xt−1 + φ2xt−2 + t with t ∼ N(0, 0.27). We set parameters [φ1, φ2] for three components as [−0.25, 0.52], [0.34, 0.27], and [1.5,−0.75] respectively. The synthetic time series from a mixture of 2 components are generated using the first two components. The synthetic time series from a mixture of 3 components are generated using all 3 components. To generate data from DeepAR, we use the DeepAR model with one LSTM layer, and the hidden number of units is 16. Since it is difficult to randomly initialize the parameters of DeepAR, we train a base model on the real-world Wiki dataset [35] (discussed in section 5.2). To build the other two DeepAR components, we respectively add random disturbance N (0, 0.01) to the parameters of the base model. For each cluster, we generate 10, 000 time series with random initialized sequences, and set the length of time series as 100.
We use 1-layer causal convolution Transformer (ConvTrans [19]) as our backbone model in MixSeq. We use the following parameters unless otherwise stated. We set the number of multi-heads as 2, kernel size as 3, the number of kernel for causal convolution dk = 16, dropout rate as 0.1, the penalty weight on the `2-norm regularizer as 1e-5, and dp = dv = 16. Meanwhile, we set the prior p(z) as 1/K, ∀z. For the training parameters, we set the learning rate as 1e-4, batch size as 256 and epochs as 100. Furthermore, the α in MixSeq is annealed using the schedule α = max(5, 20e(−0.03n)), where n denotes the current epoch, and α is updated in the [10, 30, 50]-th epochs. For comparison, we employ MixARMA [39], a mixture of ARMA(2, 0) model optimized by EM algorithm [6], as our baseline. Both methods are evaluated using Rand Index (RI) [29] (more details in supplementary).
Experiment results. We show the clustering performance of MixSeq and MixARMA on the synthetic data in Table 2. The results are given by the average of 5 trials. Regarding the synthetic data from ARMA, both MixSeq and MixARMA perform very well. However, for the synthetic data from DeepAR, MixARMA degrades significantly while MixSeq achieves much better performance. This suggests that MixSeq can capture the complex nonlinear characteristics of time series generated by DeepAR when MixARMA fails to do so. Furthermore, we also generate new time series by the corresponding ARMA and DeepAR models, and infer their clusters with the trained MixSeq model. The performance is comparable with the training performance, which demonstrates that MixSeq actually captures the generation mode of time series.

heading: 5.2 Real-world datasets
text: We further evaluate the effectiveness of our model on the macroscopic time series forecasting task. We compare MixSeq with existing clustering methods and state-of-the-art time series forecasting approaches on several real-world datasets. Specifically, for each dataset, the goal is to forecast the macroscopic time series aggregated by all microscopic data. We cluster microscopic time series into groups, and aggregate the time series in each group to form the clustered time series. Then, we train the forecasting models on the clustered time series separately, and give predictions of each clustered time series. Finally, the estimation of macroscopic time series is obtained by aggregating all the predictions of clustered time series.
We report results on three real-world datasets, including Rossmann5, M56 and Wiki [35]. The Rossmann dataset consists of historical sales data of 1, 115 Rossmann stores recorded every day. Similarly, the M5 dataset consists of 30, 490 microscopic time series as the daily sales of different products in ten Walmart stores in USA. The Wiki dataset contains 309, 765 microscopic time series representing the number of daily views of different Wikipedia articles. The dataset summary is shown in Table 3, together with the setting of data splits.
Experiment setting. We summarize the clustering strategies for macroscopic time series forecasting as follows. (1) “DTCR” [21] is the deep temporal clustering representation method which integrates the temporal reconstruction, K-means objective and auxiliary classification task into a single Seq2seq model. (2) “MixARMA” [39] is the mixture of ARMA model that uses ARMA to capture the characteristics of microscopic time series. (3) “MixSeq” is our model with 1-layer causal convolution Transformer [19]. (4) We also report the results that we directly build forecasting model on the macroscopic time series without leveraging the microscopic data, named as “Macro”.
For time series forecasting, we implement five methods combined with each clustering strategy, including ARMA [8], Prophet [34], DeepAR [32], TCN [4], and ConvTrans [19]. ARMA and Prophet give the prediction of point-wise value for time series, while DeepAR, TCN and ConvTrans are methods based on neural network for probabilistic forecasting with Gaussian distribution. We use the rolling window strategy on the test interval, and compare different methods in terms of the long-term forecasting performance for 30 days. The data of last two months in train interval are used as validation data to find the optimal model.
We do grid search for the following hyperparameters in clustering and forecasting algorithms, i.e., the number of clusters {3, 5, 7}, the learning rate {0.001, 0.0001}, the penalty weight on the `2- norm regularizers {1e− 5, 5e− 5}, and the dropout rate {0, 0.1}. The model with best validation performance is applied for obtaining the results on test interval. Meanwhile, we set batch size as 128, and the number of training epochs as 300 for Rossmann, 50 for M5 and 20 for Wiki. For DTCR, we use the same setting as [21].
Regarding time series forecasting models, we apply the default setting to ARMA and Prophet provided by the Python packages. The architectures of DeepAR, TCN and ConvTrans are as follows.
5https://www.kaggle.com/c/rossmann-store-sales 6https://www.kaggle.com/c/m5-forecasting-accuracy
The number of layers and hidden units are 1 and 16 for DeepAR. The number of multi-heads and kernel size are 2 and 3 for ConvTrans. The kernel size is 3 for TCN with dilations in [1, 2, 4, 8]. We also set batch size as 128 and the number of epochs as 500 for all forecasting methods.
Experiment results. Following [19, 30, 35], we evaluate the experimental methods using SMAPE and ρ-quantile loss Rρ7 with ρ ∈ (0, 1). The SMAPE results of all combination of clustering and forecasting methods are given in Table 5. Table 4 shows the R0.5/R0.9-loss for DeepAR, TCN and ConvTrans which give probabilistic forecasts. All results are run in 5 trials. The best performance is highlighted by bold character. We observe that MixSeq is superior to other three methods, suggesting that clustering microscopic time series by our model is able to improve the estimation of macroscopic time series. Meanwhile, Macro and MixARMA have comparable performance and are better than DTCR, which further demonstrates the effectiveness of our method, i.e., only proper clustering methods are conductive to macroscopic time series forecasting.

heading: 5.3 Sensitivity analysis of cluster number
text: The cluster number K is a critical hyperparameter of MixSeq. To analyze its effect on the forecasting of macroscopic time series, we conduct experiments on both synthetic data and real-world data. The results state the importance of setting a proper number of clusters. We suggest do binary search on this critical hyperparameter. Details are as follows.
Synthetic data. Following the experimental setting in section 5.1, we generate 10, 000 microscopic time series from 3 different parameterized ARMA respectively. That is, the ground truth number of clusters is 3 and there are 30, 000 microscopic samples in total. The aggregation of all samples is the macroscopic time series which is the forecasting target of interest. Then, we compare the forecasting performance between the method that directly forecasts macroscopic time series (denoted as Macro) and our method with different cluster numbers (including 2, 3, 5, denoted as MixSeq_2, MixSeq_3 and MixSeq_5 respectively). We fix the forecasting method as ARMA, and apply the rolling window approach for T+10 forecasting in the last 40 time steps. The average SMAPE of 5 trials are 0.807, 0.774, 0.731 and 0.756 for Macro, MixSeq_2, MixSeq_3 and MixSeq_5 respectively. MixSeq_3 being set with ground truth cluster number shows the best performance, while MixSeq_2 and MixSeq_5 would degenerate though still better than Macro. This result shows the importance of setting a proper number of clusters.
Real-world data. We do the evaluation on three real-world datasets by varying the cluster number K of MixSeq while maintaining the other parameters fixed. For Rossmann and M5 datasets, we set the
7Detailed definition is in supplementary.
cluster number K ∈ {3, 5, 7}, while we explore the cluster number K ∈ {5, 10, 15} on Wiki dataset. The architecture and training parameters of MixSeq are same to section 5.2, except that we set the dropout rate as 0.1, the penalty weight on the `2-norm regularizer as 5e-5, and the learning rate as 1e-4. Meanwhile, we also fix the time series forecasting method as causal convolution Transformer (ConvTrans).
Figure 1 reports the macroscopic time series forecasting performance (testing on R0.5 and R0.9 loss) based on MixSeq with different cluster number K on three real-world datasets. The horizontal dashed lines are the results with K = 1 that directly building ConvTrans model on the macroscopic time series without leveraging the microscopic data (named as “Macro” in section 5.2). It is obvious that each dataset has its own suitable number of clusters, and our method is relatively sensitive to K, especially on the dataset with less microscopic time series, such as Rossmann. Similar to the accurately modeling of each microscopic time series, the larger cluster number K of MixSeq also brings large variance to macroscopic time series forecasting, which degrades the performance of our method.

heading: 6 Conclusion
text: In this paper, we study the problem that whether macroscopic time series forecasting can be improved by leveraging microscopic time series. Under mild assumption of mixture models, we show that appropriately clustering microscopic time series into groups is conductive to the forecasting of macroscopic time series. We propose MixSeq to cluster microscopic time series, where all the components come from a family of Seq2seq models parameterized with different parameters. We also propose an efficient stochastic auto-encoding variational Bayesian algorithm for the posterior inference and learning for MixSeq. Our experiments on both synthetic and real-world data suggest that MixSeq can capture the characteristics of time series in different groups and improve the forecasting performance of macroscopic time series.
Acknowledgments and Disclosure of Funding
This work is supported by Ant Group.

Review summaries:
This paper develops methodology for improving time series forecasting by modeling finer-scale structure in the time series; namely, the underlying "microscopic" time series of which the predicted "macroscopic" time series is aggregated. The approach takes advantage of the concept that in many cases the time series of interest are in fact macroscopic aggregates of smaller-scale structure and time series. The work develops a method that models the micro time series as a mixture of latent components and then uses a neutral network based approach to model and parameterize the mixture. Benchmarkins are done with simulated and real data, showing promising performance.

This paper tackles the problem of predicting a so called macroscopic time series that is the aggregation of multiple microscopic time series. For example, global sales of a chain can be the sum of sales in individual locations. The authors propose to predict the macroscopic time series by building a mixture model on microscopic time series. They provide a theoretical and empirical justification of this approach. They also compare their approach to baselines on synthetic and 3 real-world data sets. For the latter data sets, performance on microscopic time series is also evaluated. The results suggest the proposed approach is better than the competing baselines.

Authors model time series as a weighted mixture of a set of time series for forecasting. Proposed a model for Mixture of seq2seq model. Number of components depends on correct clustering of micro time series. The paper also  propose an efficient stochastic auto-encoding variational Bayesian algorithm for the posterior inference and learning for MixSeq