Probabilistic Transformer For Time Series Analysis
Authors: Binh Tang
David S. Matteson
Venue: NeurIPS 2021 Poster (2021)
Keywords: state space models
transformer
time series forecasting
human motion prediction
video prediction

Abstract: Generative modeling of multivariate time series has remained challenging partly due to the complex, non-deterministic dynamics across long-distance timesteps. In this paper, we propose deep probabilistic methods that combine state-space models (SSMs) with transformer architectures. In contrast to previously proposed SSMs, our approaches use attention mechanism to model non-Markovian dynamics in the latent space and avoid recurrent neural networks entirely. We also extend our models to include several layers of stochastic variables organized in a hierarchy for further expressiveness. Compared to transformer models, ours are probabilistic, non-autoregressive, and capable of generating diverse long-term forecasts with uncertainty estimates. Extensive experiments show that our models consistently outperform competitive baselines on various tasks and datasets, including time series forecasting and human motion prediction.

Probabilistic Transformer for Time Series Analysis
Binh Tang
David S. Matteson
Department of Statistics and Data Science
Department of Statistics and Data Science Cornell University
Cornell University Ithaca, NY 14850 byt5@cornell.cm edu
Ithaca, NY 14850
matteson@cormell..om edu
Abstract
Generative modeling of multivariate time series has remained challenging partly due to the complex, non- -eneemermiiiitic dynamics across long-distance time steps. In this paper, we propose deep probabilistic methods that combine state-space models (SSMs) with transformer architectures. In contrast to previously proposed SSMs, our approaches use attention mechanism to model non-Markovian dynam- ics in the latent space and avoid recurrent neural networks entirely. We also extend our models to include several layers of stochastic variables organized in a hierar- chy for further expressiveness. Compared to transformer models, ours are proba- bilistic, non-autoregressive, and capable of generating diverse long-term forecasts with accounted uncertainty. Extensive experiments show that models consis- tently outperform competitive baselines on various tasks and datasets, including time series forecasting and human motion prediction.
Introduction
Generative modeling of multivariate time series is a challenging problem with wide-ranging appli- cations in demand forecasting [15, 76), autonomous driving [2. 16), robotics [29, 67], and health care [20, 21, 59]. Despite remarkable progress in recent years, models that predict high-dimensional future observations from few past examples have remained intractable, partly due the complex, non-deterministic temporal dynamics across long-distance time steps. Given sequence of human poses, for example, such models must internally figure out the involved dynamics of various body components across space and time while maintaining the inherent uncertainty of multiple plausible futures, even though only one such future is observed.
Among proposed probabilistic approaches, state space models (SSMs) provide a principled frame- work for learning and drawing inference from sequential inputs [27, 66). While autoregressive models feed its predictions back into the dynamics model without any compressed representation of data, SSMs model stochastic transitions between abstract states using latent variables, allowing for efficient state-to-state sampling without the need to render high-dimensional observations. Gaussian linear dynamical systems (LDSs), one of the best known SSMs [92], for example, postulate linear state transitions and enjoy exact inference via the celebrated Kalman filter algorithm.
While early extensions of LDSs focus linearization [46] and unscented transform [88], recent work that marry state space models with deep neural networks offers much more flexibility to model complex dependencies across different time steps. Some approaches retain the Markovian dynamics of LDSs and only replace their linear observation models with feed-forward networks [23, 31, 47, 71], whereas others favor nonlinear state transitions and parametrize such dependencies via recurrent neural networks (RNNs) [22, 23, 30, 39, 51, 75]. Despite differences, both Markovian transitions and RNNs are often not capable of capturing long-ran longerange dependencies in highly structured sequential inputs [36, 100], limiting the capacity of the corresponding SSMs.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

X++1
X++
-
(a) LDS
X
X
X++1
F1
(1)
(b) ProTran (I layer)
(c) Pro Generation (3 layers)
(d) ProTran Inference (3 layers)
Figure I: Graphical model representations of linear dynamical systems (LDSs) in (a), and our proposed models (ProTrran) in (b), (c), and (d). Black arrows denote the generative mechanism and red arrows the inference procedure. The separation of generation and inference in (c) and (d) is for readability. While traditional SSMs such as LDSs are limited to Markovian dynamics and linear dependencies, our models allow for non- non-Markovian and non-linear interactions between time steps via attention mechanism. multi-layer extension of models further increases expressiveness without compromising the tractable inference procedure.
In this work, propose to combine the complementary strengths of SSMs and transformer archi- tectures [85], a powerful mechanism for modeling long-term interactions that enjoys success across variety of sequence modeling tasks [26, 48, 99]. In contrast to most SSMs, extensive use of attention mechanism [5, 85] between latent variables to model non-Markovian dy- namies (see Figure 1). Compared to transformer-hait based methods, models are probabilistic, non- autoregressive in a similar fashion to LDSs, and capable of generating diverse long-term forecasts with uncertainty estimates.
models make
Our main contributions are threefold. First, we propose novel SSMs based on transformer archi- tectures for multivariate time series, which include generative models and inference procedures based on variational inference [49, 74]. Second, we extend our models to include several layers of stochastic latent variables organized in hierarchy for further expressiveness. Third, conduct extensive experiments on time series forecasting and human motion prediction and demonstrate that Probabilistic Transformer (ProTran) performs remarkably well compared various state-of-the- art baselines.
2 Preliminaries
2.1 Variational State Space Models
Let (x Fi 1N consist of N univariate time series where x = (x (0) ,22" ...xx (.)) and x, denotes the vaue of the i-th time series at time t. We consider the multivariate form X1.7 xx.... Xy) where x, (x) ") ....x (")) RN Conditioning on observed values up to time C, we aim produce distributional forecasts into the future p(xc+157) x1.c). For clarity, we refer to X1.0 and XC+1.7 as contexts and targets, respectively.
(2)
(0)
We are interested in probabilistic models parametrized by 0 of the form
po (x x1.c) = ff pe (xL:T (orr)po(z11))
(1)
where 21.7 (z), S2. 2r) denotes the corresponding sequence of latent variables, sometimes referred to states. In other words, we assume generative model that can be decomposed into a transition model po (o x1:c) between the latent variables conditioned on the contexts, and an emission model (x(x 2.r) from the latent variables to observable outputs. In particular, we

---

further impose several assumptions on both models:
Po(Zi:T x1.0) II po(ze 11a-1,
pe (x (I:T)
II pa(x: ze).
As demonstrated in Figure 1(b), the latent variable z++1 depends not only z, but also all of its preceding latent variables, including 24-1, in contrast to linear dynamical systems (LDSs). In addition, the transition and emission models allow for non-linearity via neural network parametriza- tions. These assumptions aim to maximize model capacity for real-world applications with complex emissions or temporal dependencies.
However, neither X1:t-1 nor 21:--1 are included in the emission model p(x: ZL:T, (I.T, Such assumptions are important, it has been argued previously that leakage of information from the latent space in autoregressive models can hinder long-term predictions [23, 47]. While all ground truth observations at test time, making the dependencies on X1 prone to accumulated errors over multiple time steps. By letring the latent variable 2, capture all information needed render x,, also avoid the computational costs associated with repeatedly decoding and encoding x, in multi-step predictions.
available during training, the entire sequence has to be generated sequentially
The inclusion of nonlinear state transitions and observation models necessarily requires approximate inference. We follow the stochastic variational inference framework [49, 74] and assume that the variational posterior parametrized by é can be decomposed nuto-regressively as qz(Z).7 x1:7) II, 40 B1:-11 X1.1). which leads to a lower bound on the log likelihood:
(x 2
(Eg -aa.:////a)0KKKaa/aa//a.......
po 2Ix,,,
where KL is the Kullback-Leibler divergence.
For computational stability, we assume homoscedasticity and choose Laplace distribution with scale parameter B as parametric form for px [z), i.e. optimize for Lı reconstruction loss with a cross-validated factor 3 for the KL term, following similar variational autoencoder (VAE) work [24, 41, 86]. Such assumption does not necessarily limit the capacity of models, powerful stochastic transitions and flexible emission models can theoretically characterize arbitrary noise covariance [66]. Incorporating structured probabilistic outputs such Gaussian copulas [75] or normalizing flows [23] can potentially further improve our model performance.
2.2 Transformer Architectures
Central our models and other transformer- bassed approaches [48, 85] is the notion of attention [5], which allows the models to focus on important parts within a context. Multi-head attention, for example, maps sequence of queries Q xdd of length by to a sequence of outputs O
JO....., Oh] E x&xx of the same size by attending over given lg key-value pairs K E R V R
QuKT Vd
O Attention Qi, Kr, V.. Softmax
Vn:
where Q2 Q.?. h. K, KW VA vwk are projected queries, keys, and values cor- responding to head h [1, H) with learning parameters W h W. WK. respectively. In case Q Q V. we refer to such an attention mechanism self-attention.
Given fully observed sequences of inputs, the mapping can be computed efficiently without any imposed sequential order often seen in recurrent neural networks [19, 42]. More importantly, the direct connections between long-distance time steps are baked into the mechanism as information from previous time steps is casily accessible without being compressed into fixed representation, casing optimization and learning of long-term dependencies [5, 85].
For notational simplicity, assume
21:0 Ø and p(z Xo) p(z).

---

Without recurrence, Transformer [85] encodes information about each time step t with pred efined sinusoidal positional embeddings (osition(t)
(.. E E" where the i-th embedding is given by Px(2) sin(t et/4) for even i and (i) = cos(L et/d) ) for odd i and is some large constant. Empirical results show that such positional embeddings are also important to our models.
Probabilistic Transformer
In this section, first present single-layered model and subsequently its multi-layere ex- tension for hierarchy of stochastic latent variables. As alluded earlier, our model consists a generative model and inference model that share information and parameters extensively.
3.1 Single-Layered Probabilistic Transformer
Generative Model. Given some contexts X1:O; we first apply a linear projection and combine with positional embedding to obtain b1.c E Rd i.e.
h, LayerNorm Layerrorm(MLP((x. + Position(t)),
(5)
where LayerNorm and MLP denote layer normalizations [4] and multi-layer perceptrons, respec- tively. While traditional transformer model often dedicates entire encoder for the same purpose [55, 72]. we find such simple mapping works sufficiently well in conjunction with the context- attention module of the corresponding decoder.
As implied in Equation (2), our latent dynamics decomposes auto-regressively. At each time step, we parametri ze the distribution pa (z, , X1:C) by a Gaussian with parameters resulting from two sequential steps of attention: self-attention over the previously inferred states Zi:t-1 and another attention over the projected contexts hi.c. These two operations mirror those found in the decoder of Transformer [85], with the stochastic latent variables replacing its decoder inputs.
Unfortunately, using stochastic samples of z as attention queries is problematic, as purely stochastic transitions make it difficult for the model to reliably retain information across multiple time steps [17, 30, 39]. We therefore encapsulate the latent variables in hidden representations has deterministic component. Combined with the attention steps, such representations help model long-range temporal dependencies while accounting for the stochasticity of future observations.
that also
Starting with learnable, context-agnostic representation Wo. recursively update w, using stochastic sample from (z, &i X1.c) and th e positional embedding for the current time step The generating process for the time step can be summarized by the following pseudocode:
LayerNorm (W;-1 Attention(w;:11 W1:-1: W1:-1))
W. LayerNorm W; + Attention(w,,. hic: bı:)))
(7)
Z, Sample(N (zz; MLP(W::,, Softplus(MLPP( (L:)))))))))
(8)
W. LayerNorm (w; + MLP + Position(t)),
where Sample and Softplus are the Gaussian sampling and approximating rectifier operators.
Each stochastic sample of Wi.7 is then mapped to a sequence of x via multi-layen perceptron. We emphasize that our generation procedure in the latent space is more efficient than others in the observation space, which requires encoding and decoding high-dimensional inputs repeatedly.
Inference Model. We parametrize the approximate posterior (z( 21:t-1, X1:T) at time step t in simi lar fashion to the prior pz X1.c). Indeed, these parametrizations share most parameters and are done simultancously in the same recursive loop, following the exact same steps in Equation (6) and Equation (7) (see Figure 1). We note that similar sharing techniques between the generative and inference processes have emerged as a common theme among recent successful VAE models [17, 62, 83).
While the prior only has access to the conditioning observations X1.C, the approximate posterior should take into account all observations during training, including the targets IC+1:7 Due to the inherent unidirectional aspect of RNNs, previous work that uses RNNs to parametrize the approx- imate posterior often disregards such property [22, 30, 51] and often resorts to filtering routine

---

p(z, (1:t)1,1;()) In contrast, our inference procedure resembles more of the smoothing process of LDSs, factoring both past and future observations via another application of self-attention:
k Attention (hintionn(hi:..) h hu:))
(10)
2 Sample( MLP([W, k.)), Softplus(MLP((wr, k,))))
(11)
Here, we replace Equation (8) the generative model with Equation (11), where the hidden repre- sentation k, summarizing all information relevant to the current tim estep has been concatenate to the secent-add-oomet---ie aware representation W, preceding the Gaussian parametrization.
The generative model and the inference model are trained end-to-end with single stochastic varia- tional inference objective stated in Equation (3). Such variational bound includes the reconstruc- tion loss for X1:C' and the KL term for 21:C- Alternatively, we can exclude these terms from the objective, which equivalent starting the inference process from C + I instead of
Our models incur time complexity of O(T2d) and memory cost of O(T²d), where T is the total sequence length and d is the dimensionality of the latent space. The recursive latent dynamics also does not allow use the take full advantange of parallelizable attentions. However, we find that our models still efficient in practice, especially for reasonably small values of T.
3.2 Multi-Layered Extension for Probabilistic Transformer
Inspired by recent work on hierarchical VAEs for non-sequential inputs [17, 80, 83, 101], we ex- tend our proposed model to include several layers of latent variables, aiming to further increase its flexibility for modelling sequential data.
We represent each time step with Ma rkov chain of L latent variables (1:L) = (z. (1) (1)
for simplicity (see Figure 1). The generative and inference model also decompose regressggrssively across different time steps and may exhibit non-Markovian dynamics:
po X1:7,
1TT
(1:L)
x1:C)
II po /=1
X
2
(1))
Pa Z
(0) (2)
2i:2-11
21T xx:e)
(8-1)
(12)
f=1t=1
7.
(1:L) 2
(
(6)
(6-1) zi. xL.T)
x1:T)
4z 11:111
(13)
{=1==1
Intuitively, we generate samples X1.7 conditioning on X1.C by following the latent dynamics from the bottom up and using the generative process described earlier within each layer. Analogously, inference proceeds in the same order, resulting variational bound similar to Equation (3):
T
logpo(X1.T) x1.C)= R
By [Log (x((x)
(L) 2.:)
(14)
L
(L(g)(z)) (2)
22 2 |
(1t. XITT || po(z,)) 2 (1
x1i))..
(15)
As before, we parametrize the prior pe (z,
(2) f1:-11 2 1:7 xI:c) using self-attention over the inferred
latent variables from previous time steps w on the same layer and another attention over contexts bnc. In this case, however, we include an additional self-attention over all latent variables from the layer immediately below it (see Equation (16)):
LayerNorm w 1-1 Attention(w!""
(4)
2-11
(2)
w 1:T
(e-1)
WiT ((-!)))
(16)
( LayerNorm w (2)
+ Attention ww
(2)
W w10
(17)
1:t-1
LayerNorm( w (4) + Attention(w) bic. bnc))
(18)
(8) (ample( (8). (LP(W(())) Softplus(MLP(w)) (MLP()
2
(19)
LayerNorm(w MLP (z))) Position(t)),
(20)
5

---

Stacking multiple layers of latent variables increases model expressiveness, but it also result in linear increase in running time and the number of parameters. The time complexity for the L- layers transformer is O(LT'd), while the space complexity remains (TTd due to the Markovian structure of the chain zi (1:2) at each time step f. In our experiments, we restrict the number of layers of our hierachical models to two or three.
4 Related Work
Deep State Space Models. Deep neural networks have been extensively combined with state space models, resulting in flexible, yet principledly motivated latent variable approaches. While some work keep the linear state transition intact leverage the efficient Kalman filer algorithms [23, 31, 47, 71J, more expressive, nonlinear latent dynamics parametrized by neural networks have been proposed [51, 52]. All such models are limited to the Markovian dynamics of LDSs, which hinders learning of long-range dependencies. The limitation is often alleviated by combining the stochastic transitions with a deterministic RNN that enables access to all past states [3, 8, 22, 30, 39, 77]. Our models are similarly non- Non-Markovian, but the dependencies on the past states are done via attention, which allows for easy connections between long-distance time steps. addition, while most existing deep SSMs represent each time step with single latent variable, our models include several layers of hierarchical latent variables with tractable inference mechanism.
Attentive Recurrent Networks. Attention mechanism has also been widely adopted in recent time series work using sequence-to-ssqqeene models [1, 28] or transformer architectures [14, 55, 57, 72, 81, 94]. While our models are equipped with latent variables, these transformer approaches [55, 72] lack inference mechanism and are susceptible feeding back observation noise into the dynamics model at test time. Our work, however, can be considered as an extension of the attentive state space model proposed in [1], with discrete latent states replaced by their continuous analogs. Recent developments in natural language processing [58, 60, 90] also combine transformer and VAE; however, these approaches often use a time-agnostic latent variable, in contrast to our SSM formulation.
Time Series Forecasting. Traditional univariate time series models, such Box-Jenkins methods [12] and exponential smoothing [43], often assume independence between any collection of time series [76]. While multivariate extensions of the classical approaches, including vector autoregres- sion [82] and multivariate GARCH [7], do not require such a strong assumption, they come with many others such as stationarity and homocesdasticity, demand manual selection of covariates and models, and do not scale well to even a moderate number of time series [40, 69].
Deep learning methods for time series forecasting have recently emerged as expressive, scalable framework for industrial applications [10, 68, 79, 91]. While early work focus on point forecasts [53, 70, 96], recent approaches often employ recurrent neural networks with probabilistic forecasts parametrized directly [76], using quantile functions [33]. Gaussian copulas [75], normalizing flows [23], or diffusion models [73]. In contrast, our models entirely devoid of such recurrent archi- tectures and rely on latent variables to output distributional forecasts.
Human Motion Prediction. Despite being almost identical in formulation, human motion pre- diction has often been studied independently from time series forecasting. While some work deter- ministically generate future motions video frames [13, 32, 34, 56], stochastic prediction has also been proposed with deep neural networks often outperforming traditional methods such as hidden Markov models [93] or Gaussian processes [89] complex motion datasets [13, 32, 45, 54, 63). In contrast to earlier work [95, 97] that employ a global latent variable across different time steps via conditional VAE [49], we leverage the principled framework of state space models for learning and inference of hierarchical, time-dependent latent variables.
5 Experiments
We present our experiment results on two tasks, namely, time series forecasting and human motion prediction. These tasks are often studied independently, despite being almost identical as conditional prediction problems.

---

Table I:
set CRPS time series forecasting models (lower is better). The means and standard devia- tions are computed over five runs using different random seeds.
DATASET
SOLAR
ELECTRICITY
TRAFFIC
TAXI
WIKIPEDIA
VES [43]
0,900 + 0.003
0.880
0.004
0.350 +: 0.002
VAR [6]]
0.830 0.006
0.039
0.001
0.290 0.001
VAR-Lasso [61]
0.510 0.006
0.025
0.000
0.150 0.002
3.100 + 0.004
GARCH [84]
0.880 + 0.002
0.190 = 0.001
0.370 0.001
DeepAR [76]
0.336 1.014
0.023 = 0.001
0.055 + 0.003
0.127 + 0.042
LSTM-Copula [75]
0.319 1
0.064 + 0.008
0.103 + 0.006
0.326 + 0.007
0.241 + 0.003
GP-Copula [75]
0.337 + 0.024
0.014 + 0.002
0.078 + 01002
0.208 + 0.183
0.086 + 0.004
KVAE [51]
0.340 + 0.025
0.051 = 0.019
0.100 + 0.005
0.095 + 0.012
NKF [23]
0.320 + 0.020
0.016
0.001
0.100 + 0.002
0.071 + 0.002
Transformer- MARE [72]
0.301 + 0.014
0.021
0.000
0.056 0.001
0.179 = 0.002
0.063 0.003
TimeGrad [73]
0.287 0.020
0.021
0.001
0.044
0.006
0.1 = 0.020
0.049 0.002
ProTran (Ours)
0.194 0.030
0.001 0.016
0.028 0.001
0.084 0.003
0.047 + 0.004
5.1 Time-series Forecasting
Datasets & Covariates. Following the experiment setup in [72, 73, 75], we evaluate our models and multiple competitive baselines on five popular public datasets: SOLAR, ELECTRICITY, TRAF- FIC, TAXI, and WIKIPEDIA. The data is recorded with hourly or daily frequency and shows seasonal patterns of different frequencies (see Appendix A for more dataset details). As in [72, 73], the variates include lagged inputs, fixed time embeddings (e.g. day of week, hour of day), and learnable time-series embeddings. The inputs are scaled using the conditioning examples before being fed into the model, and the predictions are rescaled appropriately afterward.
Metrics. Following [23, 72, 75], we evaluate model and all basclines using continuous ranked probability score (CRPS) [65] summed across time series, denoted by CRPS Given a univariate distribution function F and an observation x, CRPS is defined
CRPS(F, x) (F((t) (F(2) ((25=))
where 1 [(ss2] the indicator function. As argued de Bézenac et al. [23]. CRPS proper scoring rule [35] and can be computed without analytical forecast distributions. We compute the metries in a rolling fashion and use 100 samples for the distributional forecasts, similar to the afore- mentioned work.
Baselines. We benchmark our models against various baselines, including (1) VES [43]. an innova- tion state space model; (2) VAR-Lasso and VAR [61], two multivariate linear autoregressive models with and without Lasso regularization: (3) GARCH [84], multivariate conditional heteroskedastic model; (4) DeepAR [76], an autoregressive recurrent neural network; LSTM-Copula and GP-Copula [75], two RNN- based models that use Gaussian copula to model nonlinearity; (5) KVAE [51], a vari- ational approach based on linear dynamics; (6) NKF [23], normalizing flow model coupled with Kalman filters; (7) Transformer [72], a transformer-be based model based on masked autoregressive flow; and (8) TimeGrad [73], recent autoregressive approach that uses a diffusion model.
Implementations. We use 8-head attentions and 2-layers MI.Ps to parametrize the gener- ative and inference models. The stochastic latent variables z, are 16-dimensional while the hidden representations w, are in 12 Our probabilistic transformers for SOLAR and ELECTRICITY have one stochastic layer while those for the other datasets of higher dimen- sional observations employ two layers. We re- port the numbers of parameters of our models in Table 4 in Appendix C, which are all comparable to those of the state-of-the-an approaches. See Appendix for more details about and training processes.
Table 2: Ablation study on TRAFFIC.
Two Layers
One Layer
Context Attention
X
Deterministic
X
CRPS
0.028
0.031
0.033
0.041
7

---

-10
-10
-10
-10
15
3
M
2
5
2
-10-2
-10-2
-10-2
-10-2
30
10
20
10
15
20
5
10
5
0
0
0
-10 2
-10
-10
-10
15
20
20
10
5
10
10
0
-2
-2
-10-2
-10
30
15
20
10
5
5
5
0
00:00
00:00
00:00
00:00 06-16-08
00:00
00:00 06-16-08
00:00
00:00 06-16-08
06-15-08
06-16-08
06-15-08
06-15-08
06-15-08
Figure 2: Prediction intervals and test set ground-truth from ProTran (our model) for the TRAFFIC dataset of the first 16 of 963 time series.
Accuracy Comparison. Table shows that models perform competitively across all five high- dimensional time series datasets, achieving CRPS comparable to the best methods on ELECTRIC- ITY and WIKIPEDIA while outperforming all baselines, including non-SSM transformer-basel approach [72]. by significant margins on SOLAR, TRAFFIC and TAXI. Further analyses with other metrics, including CRPS and MSE, in Appendíx B also help confirm our findings.
Qualitative Results. Figure 2 shows that the distribution forecasts generated by our model follow closely the ground truths, which is consistent with our accuracy results. In addition, the model ap- pears to capture the uncertainty of future forecasts to some extent; observations of large magnitudes to correctly have higher variance estimates.
and far into the future
Ablation Study. We include a small scale ablation study on the TRAFFIC dataset investigate which components of our models are essential. Table 2 suggests that removing the stochasticity from w, has most impacts on model performance, implying that incorprating latent variables into a transformer is indeed useful. Other aspects such context attention or nultiple layers of stochastic variables do not show dramatic effects this study; however, they do contribute performance gains.
5.2 Human Motion Prediction
Following the experiment setup in [97], we evaluation our models two public motion capture datasets: Human3.6M 36M[444 and HumanEva-I [78]. While Human3.6 is a large-scale dataset with 3.6 million video frames recorded at 50Hz, HumanEva-I is smaller with only subjects and recorded 60Hz. follow the preprocessing steps of previous work [64, 97] and obtain a 17-joint skeleton for H.man3.6 and 15-joint skeleton for HumanEva-L. As in [97], we predict future motion for 2 seconds conditioning on observed motion 0.5 seconds and second conditioning on 0.25 seconds for Human3.6 and HumanEva-I, respectively.
Datasets.

---

Smoking
Walk Together
Phoning
Walking
Dicussion
Walk Dog
Figure 3: Ground-truth pose sequences (first row) and corresponding predictions by ProTran (second row). Solid colors indicate later time-steps and faded older. The body-part movements in the predicted and ground-truth poses resemble similar patterns, while certain variations are retained.
Table 3: Human motion prediction results.
DATASET
HUMAN3.61
HUMANEVA-L
Method
ADE
FDE
ADE
FDE
ERD [32]
0.722
0.969
0.382
0.461
acLSTM [56]
0.789
1.126
0.429
0.541
MT-VAE [95]
0.457
0.595
0.345
0.403
Pase-Knows [87]
0.461
0.560
0.269
0.296
HP-GAN [6]
0.858
0.867
0.772
0.749
Best-Many [11]
0.448
0.533
0.271
0,279
GMVAR [25]
0.461
0.555
0.305
0.345
DeliGAN [38]
0.483
0.534
0.306
0.322
DSP [98]
0.493
0.592
0.273
0.290
DLow [97]
0,425
0.518
0.251
0.268
ProTrrn (Ours)
0.381
0.491
0.258
0.255
Metrics. Following previous work on trajectory forecasting [2, 37]. we adopt two popular metrics, namely, average displacement error (ADE) and final displacement error (FDE). ADE measures the average L2 distance over all time steps between the ground truth motion and the closest sample, while FDE only consider such distance the final pose.
Baselines. We compare our models against 9 models, including ERD [32] and acLSTM [56], two deterministic RNN-based approaches; MT-VAE [95] and Pose-Knows [87], two conditional VAE models; HP-GAN [6], conditional GAN; Best-Many [1 1]. GMVAE [25], DeliGAN [38]. and DSP [98], four approaches optimizing for diversity objectives. The results for these baselines are reported as in [97].
Implementations. Similar the previous experiments, use 8-head attentions and 2-layers MLPs. Since Human3.6M is significantly more complex and multi-modal than the time series fore- casting datasets, make of 3 stochastic layers, opposed to 2 layers for HumanEva-L. For Human3.6M, the context and target observations are significantly longer and set up for long-term predictions, we only infer latent variables for target observations. Appendix C also contains further details about our models and their number of parameters.
Quantitative Results. Table shows that models convincingly outperform all baselines based on both metrics ADE and FDE, with the gains significantly higher for the larger dataset Human3.60 M.MM. We emphasize that our favorable performance evaluated using random samples, while the closest competitor, DLow [97], relies on a separate model for selecting samples to promote diversity, which potentially be combined with probabilistic transformer for further improvements.
Qualitative Results. We show in Figure 3 human pose predictions made by our model that are most similar to the corresponding ground truths among a collection of such stochastic predictions. The similarities between the body-part movements both sequences suggest that our model has been able to capture the temporal dynamics quite well.

---

Conclusion & Discussion
In this work, have introduced generative models for multivariate time series that combines strengths of state space models and transformer architectures. In contrast to previous work, our models do not rely on recurrent neural networks but make extensive use of attention mechanism. We also extend our models to include hierarchical latent variables, inspired by recent developments of VAEs for non-sequential data [17, 83). Empirical experiments show that our models perform remarkably well on time series forecasting and human motion prediction.
Our models do not the reliance attention incurs quadratic time and memory complexity. While we do not find problematic in our experiments, the limitation necessarily hinders applications of our models in tasks characterized by long-term dependencies such as language modelling or music generation [36]. Fortunately, recent work on sparse transformer [9, 18, 50, 55] can potentially address the issue, and we leave such an investigation for future work.
without limitations, however. As in other transformer-based approaches,
Probabilistic time series forecasting is fundamental research problem with wide-ranging applica- tions in society. Although we have not explored healthcare applications of our work, previously proposed methods with similar formulations have demonstrated potentials of forecasting techniques [1, 81] in diagnoses or disease control.
Acknowledgment
Financial support gratefully acknowledged from a Xerox PARC Faculty Research Award, Na- tional Science Foundation Awards 1455172, 1934985, 1940124, and 1940276, USAID, and Cornell University Atkinson Center for a Sustainable Future.
10

---

References
[1] Ahmed Alaa and Mihaela van der Schaar. Attentive state-space modeling of disease progres- sion. 2019.
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016.
[3] Evan Archer, 1l Memming Park, Lars Buesing, John Cunningham, and Liam Paninski. Black box variational inference for state space models. arXiv preprint arXi::1511.07880 2015.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 160.:666070665500 2016.
[5] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 1409.0440904477 2014.
[6] Emad Barsoum, John Kender, and Zicheng Liu. Hp-gan: Probabilistic 3d human motion prediction via gan. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1418-115 4418-142, 2018.
[7] Luc Bauwens, Sébastien Laurent, and Jeroen VK Rombouts. Multivariate garch models: survey. Journal of applied econometrics, 21(1):79-109 2006.
[8] Justin Bayer and Christian Osendorfer: Learning stochastic recurrent networks. NIPS 2014 Workshop on Advances in Variational Inference, 2014.
[9] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150 050000 2020.
[10] Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Bernic Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael BchlkerSchneider, David Salinas, Lorenzo Stella, et al. Neural forecasting: Introduction and literature overview. arXiv preprint arXiv:2004. 10240, 2020.
[11] Apratim Bhattacharyya, Bernt Schicle, and Mario Fritz. Accurate and diverse sampling of se- quences based on a "best of many" sample objective. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8485-8493, 2018.
[12] George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
[13] Judith Butepage. Michael J Black, Danica Kragic, and Hedvig Kjellstrom. Deep represen- tation learning for human motion prediction and classification. In Proceedings of the IEEE conference on computer vision and paltern recogmition, pages 6158-6166, 2017.
[14] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Conguri Huang, Yunhai Tong. Bixiong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecasting. arXiv preprint
2021.
[15] Real Carbonneau, Kevin Laframboise, and Rustam Vahidov. Application of machine learning techniques for supply chain demand forecasting. European Journal of Operational Research, 184(3):1 140-1400-444 1154, 2008.
[16] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8740-8749. IEEE Computer Society, 2019.
[17] Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint 1..vv2001111065/ 2020.

---

[18] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904. 2019.
[19] Kyunghyun Cho, B van Merrienboer, Caglar Gulcehre, F Bougares, H Schwenk, and Yoshua Bengio. Learning phrase representations using ran encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), 2014.
[20] Edward Choi, Mohammad Taha Bahadori, Joshua A Kulas, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in Neural Information Processing Systems, pages 3512-3520, 2016.
[21] Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Doctor ai: Predicting clinical events via recurrent neural networks. In Machine learning for healthcare conference, pages 301-318. PMLR, 2016.
[22] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. Advances in Neural Information Processing Systems, 28:2980-2999 29880 2015.
[23] Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Information Processing Systems, 33, 2020.
[24] Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In Interna- tional Conference on Machine Learning, pages 1174-1183. PMLR, 2018.
[25] Nat Dilokthanakul, Pedro AM Mediano, Marta Gamelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mix- ture variational autoencoders. arXiv preprint arXfvv1611.026443 2016.
[26] Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformen a no-recurrence sequence-to- sequence model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
IEEE, 2018.
[27] James Durbin and Siem Jan Koopman. Time series analysis by state space methods. Oxford university press, 2012.
[28] Chenyou Fan, Yuze Zhang, Yi Pan, Xiaoyue Li, Chi Zhang, Rong Yuan, Di Wu, Wensheng Wang. Jian Pei, and Heng Huang. Multi-horize time series forecasting with temporal at- tention learning. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery Data Mining, pages 2527-2535, 2019.
[29] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical in- teraction through video prediction. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 64-72, 2016.
[30] Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2207-22 2217-2215, 2016.
[31] Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther, A disentangled recogni- tion and nonlinear dynamics model for unsupervised learning. Proceedings of the 31 In- ternational Conference on Neural Information Processing Systems, pages 3604-3613, 2017.
[32] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference Com- puter Vision, pages 4354, 2015.
[33] Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Sali- nas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function rnns. In The 22nd international conference on artificial intelligence and statistics, pages 1901-1910. PMLR, 2019.
12

---

[34] Partha Ghosh, Jie Song, Emre Aksan, and Otmar Hilliges. Learning human motion models for long-term predictions. 2017 International Conference on 3D Vision (3DV), pages 458- 466. IEFE, 2017.
[35] Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estima- tion. Journal of the American statistical Association, 102(477):359-317 -3759,777-- 2007.
[36] Alexander Greaves- -areelll and Zaid Harchaoui. A statistical investigation of long memory in language and music. In International Conference on Machine Learning, pages 2394-244 2403. PMLR, 2019.
[37] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2255-2264, 2018.
[38] Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 166-174, 2017.
[39] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565. PMLR, 2019.
[40] Andrew C Harvey, Forecasting, Structural Time Series Models and the Kalman Filter. Cam- bridge University Press, 1990.
[41] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual con- cepts with constrained variational framework. 2016.
[42] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9 (8):1735-7780, 1997.
[43] Rob Hyndman, Anne B Koehler, Keith Ord, and Ralph Snyder. Forecasting with expo- nential smoothing: the state space approuch. Springer Science & Business Media, 2008.
[44] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions pattern analysis and machine intelligence, 36(7):1325-1339 2013.
[45] Ashesh Jain, Amir R 7amir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on and pattern recognition, pages 5308-5317, 2016.
graphs. In Proceedings the ieee conference on computer vision
[46] Andrew H Jazwinski. Stochastic processes and filtering theory. Courier Corporation, 2007.
[47] Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep vari- ational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint
2016.
[48] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT, pages 4171-4186, 2019.
Ming-Wei
[49] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 2014.
[50] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0444 2020.
[51] Rahul Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlin- ear state space models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[52] Rahul G Krishnan, Uri Shalit, and David Sontag. Deep kalman filters. arXiv preprint arXiv:/511.05122 2015.
13

---

[53] Guokun Lai, Wei-Cheng Chang. Yiming Yang, and Hanxiao Liu. Modeling long- and short- term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 95-104, 2018.
[54] Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. Convolutional sequence to sequence model for human dynamics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5226-5234 2018.
[55] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in Neural Information Processing Systems, 32:5243-555 5253, 2019.
[56] Zimo Li, Yi Zhou, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li. Auto- conditioned recurrent networks for extended complex human motion synthesis. arXiv preprint arXiv:1707.05580 2017.
[57] Bryan Lim, Sercan O Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. arXiv preprini arXiv:19/2.09363, 2019.
[58] Zhaojiang Lin, Genta Indra Winata, Peng Xu, Zihan Liu, and Pascale Fung. Variational transformers for diverse response generation. arXiv preprint 20Xi: 2020.
[59] Zachary C Lipton, David C Kale, Charles Elkan, and Randall Wetzel. Learning to diagnose with Istm recurrent neural networks. In International Conference on Learning Representa- tions, 2016.
[60] Danyang Liu and Gongshen Liu. A transformer-basee variational autoencoder for sentence generation, In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE, 2019.
[61] Helmut Lütkepohl. New introduction to multiple time series analysis. Springer Science & Business Media, 2005.
[62] Lars Maaløe, Marco Fraccaro, Valentin Lievin, and Ole Winther. Biva: A very deep hierar- chy of latent variables for generative modeling. In 33rd Conference on Neural Information Processing Systems, page 8882. Neural Information Processing Systems Foundation, 2019.
[63] Julicta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2891-29000 2017.
[64] Julieta Martinez, Rayat Hossain, Javier Romero, and James Little. A simple yet effective baseline for 3d human pose estimation. In Proceedings of the IEEE International Conference on Computer Vision, pages 2640-2649, 2017.
[65] James E Matheson and Robert Winkler. Scoring rules for continuous probability distribu- tions. Management science, 108-0960 1976.
[66] Kevin Murphy. Machine learning: a probabilistic perspective. 2012.
[67] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action- conditional video prediction using deep networks in atari games. arXiv preprint arXiv:1507.08750, 2015.
[68] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting, In International Conference on Learning Representations, 2019.
[69] Andrew Patton. review of copula models for economic time series. Journal of Multivari- ate Analysis, 110:4-18, 2012.
[70] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual-stage attention based recurrent neural network for time series prediction. arXiv preprint a704v 2017.
14

---

[71] Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting, Advances in neural information processing systems, 31:7785-7794, 2018.
[72] Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Multi-variate probabilistic time series forecasting via conditioned normalizing flows. arXiv preprint arXiv:2002.06 2020.
[73] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denois- ing diffuusin models for multivariate probabilistic time series forecasting. arXiv preprint arXiv:2101.1207,, 2021.
[74] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and upproximate inference deep generative models. In International conference on machine learming, pages 1278-1286 PMLR, 2014.
[75] David Salinas, Michael Bohlke-Sohie Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank gaussian copula pro- cesses. arXiv preprint arXiv: 190.0090,. 2019.
[76] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36 13):1111-111, 2020.
[77] Iulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 31, 2017.
[78] Leonid Sigal and Michael Black. Humaneva: Synchronized video and motion capture dataset for evaluation of articulated human motion. Brown Univertsity TR, 120(2), 2006.
[79] Slawek Smyl, Jai Ranganathan, and Andrea Pasqua. M4 forecasting competition: Introducing new hybrid es-rnn model. URL: https://eng, uber. horp://ffffrrccasiinnccmpppppepo
2018.
[80] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3745-3753, 2016.
[81] Huan Song, Deepta Rajan, Jayaraman Thiagarajan, and Andreas Spanias. Attend and di- agnose: Clinical time series analysis using attention models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[82] Ruey S Tsay. Multivariate time series analysis: with R and financial applications. John Wiley & Sons, 2013.
[83] Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03880 2020.
[84] Roy Van der Weide. Go-garch: multivariate generalized orthogonal garch model. Journal of Applied Econometrics, 17(5):549-564, 2002.
[85] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st In- ternational Conference on Neural Information Processing Systems, pages 6000-6010, 2017.
[86] Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In NeurlPS, 2019.
[87] Jacob Walker, Kenneth Marino, Abhinay Gupta, and Martial Hebert. The pose knows: Video forecasting by generating pose futures. In Proceedings of the IEEE international conference computer vision, pages 3332-3341, 2017.
15

---

[88] Eric A Wan and Rudolph Van Der Merwe. The unscented kalman filter for nonlinear estima- tion. In Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communica- tions, and Control Symposium (Cat. No. 00EX373), pages 153-158. leee, 2000.
[89] Jack M Wang, David J Fleet, and Aaron Hertzmann. Gaussian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence, 30(2): 283-298, 2007.
[90] Tianming Wang and Xiaojun Wan. T-evae: Transformer-based conditioned variational au- toencoder for story completion. In IJAAI, pages $233-5239,201
[91] Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting. In International Conference Machine Learn- ing, pages 6607-6617. PMLR, 2019.
[92] Mike West and Jeff Harrison. Bayesian forecasting and dynamic models. Springer Science
& Business Media, 2006.
[93] Di Wu and Ling Shao. Leveraging hierarchical parametric networks for skeletal joints based action segmentation and recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 724-731, 2014.
[94] Neo Wu, Bradley Green, Xue Ben, and Shawn O' O'Banion. Deep transformer models for time series forecasting: The influenza prevalence case. arXiv preprint arXiv:2001.08317, 2020.
[95] Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, and Honglak Lee. Mt-vae: Learning motion transformations to gener- ate multimodal human dynamics. In Proceedings of the European Conference on Computer Vision (ECCV), pages 265-281. 2018.
[96] Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Long-term forecasting using tensor-train mns. Arxiv, 2017.
[97] Ye Yuan and Kris Kitani. Dlow: Diversifying latent flows for diverse human motion predic- tion. In European Conference on Computer Vision, pages 346-364. Springer, 2020.
[98] Ye Yuan and Kris M Kitani. Diverse trajectory forecasting with determinantal point processes. In International Conference on Learning Representations, 2019.
[99] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention gener- ative adversarial networks. In International conference on machine learning, pages 7354- 7363. PMLR, 2019.
[100] Jingyu Zhao, Feiqing Huang, Jia Lv, Yanjie Duan, Zhen Qin, Guodong Li, and Guangjian Tian. Do mn and Istm have long memory? In International Conference on Machine Learning, pages 1365-1 1375. PMLR, 2020.
[101] Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hicrarchical features from gen- erative models. arXiv preprint 1eX.v:1702.083380 2017.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section I and 5.
(b) Did you describe the limitations of your work? [Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
16

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material URLY? [Yes] See supple-
mental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appdendix
supplemental material.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appdendix D in supplemental material.
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See Section 4. (b) Did you mention the license of the assets? [Yes] See Appendix A in supplemental
material.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See supplemental material. We also commited to open source our code upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [Yes] See Appendix A in supplemental material.
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [Yes] See Appendix A in supplemental mate- rial.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "w0DCpSNUyc", "reviewer": "Reviewer_J2rG", "summary": "The paper proposes a probabilistic forecasting model combining transformers with state space models in a non-trivial fashion. It uses variational inference for inference of the multiple, hierarchically stacked state variables (instead of other related works that assume linear state transitions and can then use kalman filtering). The empirical evaluations show the practical relevance of this approach.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an impressive paper overall on which I can only commend the authors. The proposed model brings together a number of interesting techniques in a non-trivial way: transformers, state-space models (non-linear state transitions), VAEs. The paper discusses relevant work authoritatively, clearly and crisply. The description of the model and inference is clear. However, I am most impressed with the empirical evaluations. The authors picked relevant and multiple methods for comparisons on standard data sets across a spectrum of recent methods. I very much wish that more papers would follow this role-model approach.\n\nI really don't have major comments apart from this. Surely, the authors could have worked more on the well-known short-comings in terms of computational efficiency of transformers (they comment on this in l. 291 thoughtfully), but given the breadth and depth of the contributions of this paper, it's really not necessary. It would have been interesting to see whether the methods in Table 1 are re-implementations or which implementations were chosen and I hope that the authors will release the code in open source. \n\nl. 53. From the notation it is unclear whether x_i \\in \\mathbb{R}^N. It should be (x_1, \\ldots, x_C) \\in \\mathbb{R}^{C \\times N}", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6h"}, "extra_text": {}}, {"review_id": "v--hsjMRdTM", "reviewer": "Reviewer_jdmt", "summary": "The submission presents a probabilistic extension to the Transformer by augmenting the intermediate Transformer representations via stochastic latent variables. The attention mechanism allows for accessing the past and the future steps directly in contrast to the indirect hidden state of RNN-based state-space models (SSMs). The building blocks of the Transformer remain untouched. The attention operation is applied to the deterministic representations as in the Transformer. The random variables are then calculated by using these updated deterministic representations. The authors also propose a hierarchical variant of their model to enable stacking several layers as commonly observed in the Transformer architectures. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Neural state-space models (SSM) have been an active domain and stochastic latent variables have been introduced in a vast range of network architectures including but not limited to fully connected, convolutional, rnn, bi-rnn, tcn. Similarly, this paper combines ideas from two different domains, namely the SSMs and the Transformers. The paper does not present a novel method, yet the proposed hybrid architecture seems effective (based on the provided experiments).\n\nIt is well-written and easy to follow. It provides a good overview. The design choices are motivated. I enjoyed reading the paper. The formulation is correct. The authors also provide experimental details that are enough to reproduce the results as well as the code. \n\nThe authors introduce stochastic latent variables in a straightforward manner. While this simplicity is an advantage, it also impairs the underlying model. For example, the recursive nature of the latent variables takes away the inherent parallelism of the Transformer.\n\nThe discussion about the limitations of the RNN-based SSMs is accurate (#33-#36). However, the authors ignore the convolutional SSMs (see [2] and [3]) which alleviate this problem and have similar motivation with the submission. It would be useful to include a discussion involving them. \n\nI am a bit confused about the hierarchical extension. In equations 12-20, the prior model can access the future information through the lower latent variable in the hierarchy, $z_{1:T}^{(l-1)}$. Can the authors clarify this? Similarly, the prior term condition at #155 and in Eq. (15) should be $z_{1:T}^{(l-1)}$ as in Eq. (12)? The references for hierarchical VAEs (#146) should include [2, 3, 4]. \n\n#130-#132: The statement should be corrected as [30] is not limited to filtering. Similarly, [1] considers future information in the approximate posterior model via a bi-directional RNN.\n\nThe experiment section lacks an in-depth discussion. The authors could explain the motivation for selecting specific baselines. Other than beating several baselines, what does the time-series forecasting (sec 5.1) experiment tell us? The motion prediction experiment provides both quantitative and qualitative analysis. The qualitative results are useful.\nIn the experiments, did the authors use the prior or the approximate posterior model? Both experiments could include the vanilla Transformer baseline.\n\n\nReferences\n[1] Goyal et al., Z-Forcing: Training Stochastic Recurrent Networks\n[2] Aksan et al., STCN: Stochastic Temporal Convolutional Networks\n[3] Lai et al., Stochastic wavenet: A generative latent variable model for sequential data\n[4] Gulrajani et al., PixelVAE: A latent variable model for natural images\n[5] Martin et al., The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction\n\n---------------------\n**Update:**\nI thank the authors for their response. I decided to keep my rating. The submission presents a solid formulation and better results in the experiments. The experimental setup has been my main concern. As the main contribution is a Transformer-based SSM, the baselines should involve the established RNN- and CNN-based SSMs as well as the vanilla Transformer. Only one of the two experiments focuses on this setup, but it also does not compare with some of the SOTA SSMs. I suggest the authors introduce the SSMs into the motion experiment baselines as it could be a new and challenging benchmark for the SSMs.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly mention a potential misuse, namely the surveillance scenario, however, they do not discuss how this risk can be mitigated.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "MfjC2Wqts6s", "reviewer": "Reviewer_KKhX", "summary": "This paper introduces a deep latent variable model for temporal data, that uses a transformer-based architecture with an attention mechanism over the latent states of a SSM.\nThis model can then be seen as a probabilistic transformer, and outperforms a wide range of competing models in time series forecasting and human motion prediction experiments.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this paper to be very interesting. It presents a novel architecture that can be used for time series modelling, which combines the expressiveness of transformer-based architectures with the ability to model uncertainty of deep SSMs. These are both popular research directions that have had many recent successes in a wide range of applications, so I was happy to see a work trying to get the best of both worlds. The paper is well written and presents an extensive literature review.\n\nUnlike several competing methods, this model relaxes the markovian assumption and provides a natural way to compute the smoothed posterior distribution. Also, the transformer allows to better model long term dependencies in the data, that are hard to capture for RNN-like models\n\nMy main concern for the paper is the lack of convincing ablation studies, which I deem a must to really understand what makes such a complex architecture work. The only ablation study is performed on a single dataset, so it is hard to tell if the findings generalize. In particular it would be interesting to understand\n\n1. How important is it to have latent states in the model, as opposed to a fully deterministic architecture? From equations (9) and (20) it is clear that the model could easily learn to ignore all stochasticity. The fact that you pick beta 1e-2 in the Human Eva-I dataset also hints that maybe stochasticity is not that necessary.\n\n2. How important is it to include a latent state at each time step, as opposed to e.g. the models combining transformers and VAEs that you mention in the related work?\n\n3. What is the impact of the number of stochastic layers in different datasets?\n\n4. Why is the laplace distribution a better choice than the more commonly used Gaussian distribution?\n\nMinor comment:\n* The ablation study paragraph in line 248 should go in the previous section (5.1) \n\n\n_____________________________\nReply to author rebuttal:\nThanks for you rebuttal, I have raised my score and will argue for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DMNZUEMcboe", "reviewer": "Reviewer_m6pu", "summary": "## Summary\nThe authors propose a probabilistic generative non-autoregressive transformer based state space model (SSMs) and corresponding inference procedures for them. Starting with a detailed description of the related probabilities within their models. Followed by a description of the architecture split up into Generative Model and Inference Model also serving a Multi-Layered Extension. In related work several alternative forecasting method types are presented. They compare their methods with 11 of these methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n- detailed preliminaries\n- very detailed mathematical declaration of how the different probability distributions form from the given ones\n- showing memory and runtime complexity of different models\n\n## Cons\n- Incomplete  method formalization (parameter initialization, information/processing flow)\n- Partly novel, but limited significance \n- More general discussion and implications would be desirable\n\n\n## Detailed Issues\n\n- Hard to follow preliminaries (fundamentals) section since the authors do neither reference back to previous work nor give a high-level description of the context of their work. Figure 1 is not really helpful since it does not show the parameters of the proposed architecture.\n- The concept of distribution forecast is not properly introduced\n- Some well annotated block diagrams to explain the architecture and the relation between different\nvariables would be desirable\n- misses a proper explanation of how w’s and z’s are initialized\n- Not really using a transformer but multihead attention and positional embedding in some customized way and a encoder decoder structure\n- Maybe the  related work section should be move to the front of the paper\n- table containing models and parameters used for experiments (as a central and clear overview) would be nice\n- unclear when inference model and when generative model is used (one for training one for inference?)\n\n## Minor Flaws\nFigure 1: missing connections to the x states\n53: (x1, x2, …, xC) \\in R^N is wrong because x1 \\in R^N\n58: could use some references?\n77: vague statement with over 1000 page reference.\n91 - 94: c is not really a large constant in (attention is all you need\n(1/10000) and i should be 2i)\n104: not really the transformer architecture (as stated before) rather\nusing pos-embedding and multi-head-attention with a custom MLP as encoder\nEquation 8: z_t depends directly on z_t?\n115: somewhat contradicts with conclusion (line 294)\n142-143: grammar\n167: keeps statt keeps und filter state filer (typos)\n177: transformers listed under Attentive Recurrent Networks\nTable 2: some caption would be nice\n217: different citation style\n251-252: what is the actual statement\n216 (CRPS equation): what are x and z here and what means {x<=z}\n298-301: What is the actual statement/message here?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers advocate strongly for acceptance, one reviewer has been convinced to favor acceptance, and one reviewer recommends rejection. I agree with these first two reviewers about the novelty of the proposed method, and I shared their enthusiasm. I share the concern of reviewer jdmt about the omission of a comparison to convolutional SSMs. Because the authors’ empirical results are already fairly extensive and impressive, I do not favor rejection on these grounds, but I do encourage the authors to include a comparison to convolutional SSMs in the camera-ready version of their manuscript.
Decision: Accept (Poster)