Variational Dynamic Mixtures
Authors: Chen Qiu
Stephan Mandt
Maja Rudolph
Venue:  (2021)
Keywords: sequential latent variable models

Abstract: Deep probabilistic time series forecasting models have become an integral part of machine learning. While several powerful generative models have been proposed, we provide evidence that their associated inference models are oftentimes too limited and cause the generative model to predict mode-averaged dynamics. Mode-averaging is problematic since many real-world sequences are highly multi-modal, and their averaged dynamics are unphysical (e.g., predicted taxi trajectories might run through buildings on the street map). To better capture multi-modality, we develop variational dynamic mixtures (VDM): a new variational family to infer sequential latent variables. The VDM approximate posterior at each time step is a mixture density network, whose parameters come from propagating multiple samples through a recurrent architecture. This results in an expressive  multi-modal posterior approximation. In an empirical study, we show that VDM outperforms competing approaches on highly multi-modal datasets from different domains. 

Under review as a conference paper at ICLR 2021
VARIATIONAL DYNAMIC MIXTURES
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep probabilistic time series forecasting models have become an integral part of machine learning. While several powerful generative models have been proposed, we provide evidence that their associated inference models are oftentimes too lim- ited and cause the generative model to predict mode-averaged dynamics. Mode- averaging is problematic since many real- worldworld sequences are highly multi-modal, and their averaged dynamics are unphysical (e.g., predicted taxi trajectories might run through buildings on the street map). To better capture multi-modality, we develop variational dynamic mixtures (VDM): a new variational family infer sequential latent variables. The VDM approximate posterior at each time step is a mixture density network, whose parameters come from propagating multiple sam- ples through a recurrett architecture. This results in an expressive multi-modal posterior approximation. In an empirical study, we show that VDM outperforms competing approaches on highly multi-modal datasets from different domains.
INTRODUCTION
Making sense of time series data is an important challenge in various domains, including ML for climate change. One important milestone to reach the climate goals is significantly reduce the CO2 emissions from mobility (Rogelj et al. 2016). Accurate forecasting models of typical driving behavior and of typical pollution levels over time can help both lawmakers and automotive engi- neers to develop solutions for cleaner mobility. In these applications, no accurate physical model of the entire dynamic system is known or available. Instead, data-driven models, specifically deep probabilistic time series models, can be used to solve the necessary tasks including forecasting.
The dynamics in such data can be highly multi-modal. At any given part of the observed sequence, there might be multiple distinct continuations of the data that are plausible, but the average of these behaviors is unlikely, or even physically impossible. Consider for example dataset of taxi trajec- torie In each row of Fig. Ia we have selected 50 routes from the dataset with similar starting behavior (blue). Even though these routes are quite similar to each other the first 10 way points, the continuations of the trajectories (red) can exhibit quite distinct behaviors and lead to points on any far edge of the map. The trajectories follow few main traffic arteries, these could be considered the main modes of the data distribution. We would like to learn generative model of the data, that based on some initial way points, can forecast plausible continuations for the trajectories.
Many existing methods make restricting modeling assumptions such Gaussianity to make learn- ing tractable and efficient. But trying to capture the dynamics through unimodal distributions can lead either to "over- swwer-generalizationn, (i.e. putting probability mass in spurious regions) or focus- ing only on the dominant mode and thereby neglecting important structure of the data. Even neural approaches, with very flexible generative models can fail to fully capture this multi-modality be- cause their capacity is often limited through the assumptions of their inference model. To address this, we develop variational dynamic mixtures (VDM). Its generative process is sequential latent variable model. The main novelty is new multi-modal variational family which makes learning and inference multi-modal yet tractable. In summary, our contributions are
A new inference model. We establish new type of variational family for variational inference of sequential latent variables. By successively marginalizing over previous latent states, the pro- cedure can be efficiently carried-out in a single forward pass and induces a multi-modal posterior
http://wwww.agggccccmmmmmiaaaaas

---

Under review as a conference paper at ICLR 2021
(a) Taxi Data
VDM (ours)
AESMC
(d) CF-VAE
(c) VRNN
(1) RKN
Figure 1: Forecasting taxi trajectories is challenging due to the highly multi-modal nature of the data (Fig. Tap. VDM 1b) succeeds generating diverse plausible predictions (red), based the begin- ning of trajectory (blue). The other methods, AESMC [Le et al. 2018), CF- (Bhattacharyya
et al. 2019). FRNNCChung et (2015). RKN Becker et al. (2019). suffer from mode averaging.
approximation. We
in Fig. 1b. that VDM trained dataset of taxi trajectories produces forecasts with the desired multi-modatie while other methods overgeneralize.
An evaluation metric for multi-modal tasks. The negative log-likelihood measures predictive accuracy but neglects important aspect of multi-modal forecasts sample diversity. In Sec- tion 4 we derive a score based on the Wasserstein distance (Villani 2008) which evaluates both sample quality and diversity. This metric complements our evaluation based on log-likelihoods.
An extensive empirical study. in Section 4 we use VDM to study various datasets, including synthetic data with four modes, a stochastic Lorenz aftractor, the taxi trajectories, and a U.S. pollution dataset with the measurements of various pollutants over time. We illustrate [VDM]s ability in modeling multi-modal dynamics, and provide quantitative comparisons to other methods showing that VDM compares favorably to previous work.
RELATED WORK
Neural recurrent models. Recurrent neural networks (RNNs) such as LSTMs (Hochreiter & Schmidhuber 1997) and GRUs (Chung et al.) 2014) have proven successful on many time series modeling tasks. However, as deterministic models they cannot capture uncertainties in their dy- namic predictions. Stochastic RNNs make these sequence models non-deterministic (Chung et al,, 2015| Fraccaro et al.] [016 Gemici et al.] [017:: Li Mandi, 2018). For example, the variational recurrent neural network (VRNN) (Chung et al. 2015) enables multiple stochastic forecasts due to its stochastic transition dynamics. An extension VRNN (Goyal et al. 2017) uses an auxiliary cost to alleviate the KL-vanishing problem. It improves on VRNN
VRNNjnffeennce inference by forcing the latent variables to also be predictive future observations. Another line of related methods rely on par-
ticle filtering (Neesseth al. 2018 Le et al., 2018 Hirt & Dellaportas) 2019) and in particular sequential Monte Carlo (SMC) to improve the evidence lower bound. In contrast, VDM adopts an explicitly multi-modal posterior approximation. Another (SMCbbraS work (Saeedi et 2017)
employs search-based techniques for multi-modality but is limited to models with finite discrete states. Recent works (Schmidt & Hofmann, 2018 [Shhiitt et al., 2019 Ziegler & Rush 2019) use normalizing flows in the latent space to model the transition dynamics. A normalizing flow requires many layers transform its base distribution into a truly multi- modal distribution practice. In contrast, mixture density networks (as used by (DMM achieve multi-modality by mixing only one layer of neural networks. task orthogonal to multi-modal inference learning disentangled repre- sentations. Here too, mixture models are used (Chen et al. 2016 Li et al.) 2017). These papers use discrete variables and a mutual information based term to disentangle different aspects of the data.
VAE-like models [Bhattahhrryya et al. 2018 2019) and GAN-like models (Sadeghian et al. 2019 Kosaraju et al. 2019) only have global, time independent latent variables. Yet, they show good results various tasks, including forecasting, With deterministic decoder, these models focus on average dynamics and don't 't capture local details (including multi-mon modal transitions) very well. Sequential latent variable models are described next.
2

---

Under review as a conference paper at ICLR 2021
Xt-1
X--1
%
At
(a) Generation (Eqs. and (2)
(b) Inference (Eqs.
and
Figure 2: Graphical illustrations of VDM Dashed lines denote deterministic dependencies such as transformations, marginalization, or computing the mean, as explained the main text, while bold lines denote stochastic dependencies. The half-shaded node for indicates that is being marginalized out as opposed to conditioned on.
Deep state-space models. Classical State-space models (SSMs) popular due their tractable inference and interpretable predictions. Similarly, deep (SSM with locally linear transition dynam-
ics enjoy tractable inference (Karl al., 2017 Fraccaro al. 2017 Rangapuram al.) 2018
Becker et al., 2019). However, these models are often not expressive enough to capture comp plpplex (or highly multi-modie
dynamics. Nonlinear dsep/SSMs KKishhaa al. 2017 Zheng al. 2017 Doerr et al. 2018: De Brouwer et al.) 2019) Gedon et al. 2020 are more flexible. Their inference often longer tractable and requires variational approximations. Unfortunately, in order for the inference model to be tractable, the variational approximations are often simplistic and don't don't ap- proximate multi-modal posteriors well with negative effects on the trained models. Multi-modality can be incorporated via additional discrete switching latent variables, such as recurrent switching linear dynamical systems (Linderman et al. 2017; Nassar et al. 2018 Bocker-Ehmede et al. 2019) However, these discrete states make inference more involved.
VARIATIONAL DYNAMIC MIXTURES
We develop[VDNV a new sequential latent variable model for multi-modal dynamics. Given sequen- tial observations X1:7 = (xi....,xx Xx). VDM assumes that the underlying dynamics are governed by latent states 21T (Zi...... Zr). We first present the generative process and the multi-modal inference model of VDM We then derive new variational objective that encourages multi-modal posterior approximations and we explain how is regularized via hybrid-training. Finally, we intro- duce a new sampling method used in the inference procedure.
Generative model. The generative process consists of transition model and an emission model. The transition model p(z: zzr) describes the temporal evolution of the latent states and the emis- sion model p(x; Zg() maps the states to observations. We assume they are parameterized by two separate neural networks, the transition network 6 and the emission network ddee .o give the model the capacity to capture longer range temporal correlations model with a recurrent architecture @GRU (Auger-Méthé et al. 2016 Zheng et al. 2017) such as GRU (Chung al. 2014). The latent states z, are sampled recursively from
parametrize the transition
z, Z<t ~ ((Ho,, T), where [Ma,,, VV,.l = f
tra (h,1), h = & GRU (2- h-22); (1)
and are then decoded such that the observations can be sampled from the emission model,
X ZZr NYHur,,, where [Uz,t, 0 = (ze h,
This generative process is similar to (Chung et al. 2015), though we did not incorporate autoregres- sive feedback due its negative impact on long-term generation (Ranzato et al. 2016 Lamb al.) 2016). The competitive advantage of VDM comes from more expressive inference model.
Inference model. VDM based on a new procedure for multi-modal inference. The main idea is that approximate the posterior at time i, use the posterior approximation of the previous
3

---

Under review as a conference paper at ICLR 2021
time step and exploit the generative model's transition model @GRU. This leads to sequential inference procedure. We first use the forward model to transform the approximate posterior at time samples from the resulting transformed
t - into distribution at time t. In second step, distribution and combine each sample with data evidence X,, where every sample parameterizes Gaussian mixture component. As a result, we obtain a multi-modal posterior distribution that depends on data evidence, but also on the previous time step's posterior.
In more detail, for every Ze, we define its corresponding recurrent state as the transformed random
variable Sy = &GRU (Zr, h--1), i), using a deterministic hidden state h:- = E [s.-)]. The variational family of VDM defined as follows:
q XL.T) = II (Z) xgx) =
II g(zz 11-11, xg()ds:-1-
(3)
Chung et al. (2015) also use a sequential inference procedure, but without considering the distri- bution of 8,- Only a single sample is propagated through the recurrent network and all other in- formation about the distribution of previous latent states Zct lost. In contrast, VDM explicitly maintains By part of the inference model. Through marginalization, the entire distribution is taken into account for inferring the next state 2,. Beyond the factorization assumption and the marginal
consistency constraint of Eq. B the variational family of of[VDM needs two more choices to be fully
specified; First, one has to choose the parametrizations of g(ze Sc-1. x2) and ((s)-1 X<e) and
second, has to choose sampling method approximate the marginalization in Eq. These choices determine the resulting factors (Z: xxx) of the variational family.
We assume that the variational distribution of the recurrent state factorizes g(s: xx+)
X<t), i.e. it is the distribution of the recurrent state given the past observatior +
re-weighted by weighting function ((s xt) which involves only the current observations. For
VDM only need samples from ((s;-1 Xcr). which are obtained by sampling from the previous posterior approximation g( Xct) and transforming the sample with the RNN,
~ q(s xxt) equiv. to s = &GRU (z h;2)), - g(z--1 xet), (4)
where: i indexes the samples. The RNN CCRU has the same parameters as in the generative model.
Augmenting the variational model with the recurrent state has another advantage; approximating the marginalization in Eq. 43 with A: samples from g(s, xgt) and choosing Gaussian parametriza-
tion for qzz S X results in a q-distribution q(z; xx() that resembles a mixture density
network (Bishop 2006), which is a convenient choice to model multi-modal distributions.
q(z, xg() Σ w N 12))
[e 1
(1) 0.
(1) (s
(5)
We assume q(z Xe) to be Gaussian and use an inference network Φinf to model the effect of the observation x; and recurrent state S;-1 on the mean and variance of the mixture components.
The mixture weights w/ = w x///k /k come from the variational distribution g(s:-1 xgg)=
Fl We are free to choose how to parametrize the weights, as long as all variational distributions are properly normalized. Setting
xx1) and importance sampling
w w(s (2,ss x,)/: = 1 arg max
h = s(1))
(6)
achieves this. In Appendix A explain this choice with importance sampling and in Appendix H
we compare the performance of VDM under alternative variational choices for the weights.
In the next time-step. plugging the variational distribution g(z, distribution over recurrent states q(s; xx)). For this, the expected recurrent state h is required.
x<+) into Eq. (4) yields the next
20{ss=1 x<e) is the distribution obtained transforming the previous 2;-1 ~ q(z-11/x<1 through the RNN. It can he expressed analytically using the Kroneeker 5 compare whether the stochastic variable #;-1
cquals the output of the RNN: q(s xx2)) x 5(5)-1 o (RR (z.-1, (G)((,,,1, ht-11,h,2.00 2h. xi-1, de- )dz.-1.
the a adjusts for using samples from j(5:-1 xxe) when marginalizing over w(s-1: xx)((5(-1 xxe)
4

---

Under review as a conference paper at ICLR 2021
We approximate the update using the same & samples (and therefore the same weights) in Eq. (5)
/
h B[s
5c- (( xs:)
S
A schematic view of the generative and inference model of VDM is shown in Fig. 2 In summary, the inference model of[V alternates between Eqs. (4) to (7p. Latent states are sampled from the posterior approximation of the previous time-step and transformed by Eq. 4 into samples of the recurrent state of the RNN. These are then combined with the new observation X to produce the next variational posterior Eq. (5) and the expected recurrent state updated (Eq. (7).. These are then used in Eq. 4 again. Approximating the marginalization in Eq. (3) with single sample, recovers the inference model of VRNN (Chung et al. 2015), and fails in modeling multi-modal dynamics as shown in Fig. 3 In comparison, VDM approximate marginalization over the recurrent states with multiple samples succeeds modeling multi-modal dynamics.
Variational objective.
We develop objective optimize the variational parameters of VDM dsins). J. The evidence lower bound (ELBO) at each time step is
tra gidee G GRGUU
LELBO (x
= k
w 1
Xx)
( xx))
[oog(xe Zz, h 8 8
log p(2: h = {
w(s
(i)
X1 E
(m. (x
g(ze 1/-11
log ((s x.) + c) h Σ w ( xx)
(8)
Claim 1. The ELBO in Eq. ( is a lower bound on the log evidence log p(x, xxt).
log p(x;
Xxt) 2 LELBO
(see proof in Appendix
(9)
In addition the ELBO the objective of VDM has two regularization terms,
(VDM( Bpan |- I-CELBO(Xxs. Φ) - (x 6)] + U2Cado Φ) (10)
In an ablation study Appendix E we compare the effect of including and excluding the regular- ization terms in the objective. VDM is competitive without these terms, but we got the strongest results by setting w1,2 tuned even further.) The first regularization term Lpred, encourages the variational posterior (from the previous time step) to produce samples that maximize the predictive likelihood,
(this is the only nonzero value we tried. This hyperparameter could be
Lyrea Φ) = log (Bg(sr-ixxc [p(xe x, R log Σ
p(x:
S (2))
(1I)
This regularization term is helpful to improve the prediction performance, since it depends on the predictive likelihood of samples, which isn't involved in the EEBOO The second optional regular- ization term Lade (Eq. (12)) is based on ideas from hybrid adversarial-likclioood training (Grover) al. 2018 Lucas al. 2019). These training strategies have been developed for generative mod- els of images to generate sharper samples while avoiding "mode collapse". We adapt these ideas to generative models of dynamics. The adversarial term Lade uses forward KL-divergence, which enables "quality-driven training" to discourage probability mass in spurious areas.
Lado Φ) = (xxx xxt)llp((x xx()) = E p(x; xxx)) - log xxx)] (12)
The expectation is taken w.r.t. p(x; xxx). The true predictive distribution ppl [ unknown. Instead, while assuming an optimal discriminator. As a result, we optimize Eq. (12) in an adversarial manner,
X e) is
train the generator of conditional GAN (Mirza & Osindero 2014),
conditioning Xet each time step. Details about the discriminator are in Appendix
5

---

Under review as a conference paper at ICLR 2021
Stochastic cubature approximation (SCA). The variational family of VDM is defined by number of modeling choices, including the factorization and marginal consistency assumptions of Eq. (3), the parametrization of the transition and inference networks Eqs. 4 and (5), and the choice of weighting function w()). It is also sensitive to the choice of sampling method which we discuss here. In principle, we could use Monte-Carlo methods. However, for relatively small number of samples k. Monte-Carlo methods don't 't have a mechanism to control the quality of samples. We instead develop semi-stochastic approach based on the cubature approximation (Wan & Van Der Merwe 2000; Wu et al. 2006; Arasaratnam & Haykin 2009), which chooses samples more carefully. The cubature approximation proceeds by constructing
2d so-called sigma points, which are optimally spread out on the d-dimensional Gaussian with the same mean and covariance as the distribution we need samples from. In SCA] the deterministic sigma points are infused with Gaussian noise to obtain stochastic sigma variables. A detailed derivation of SCA is in Appendix [
We use SCAAfor various reasons: First, it typically requires fewer samples than Monte-Carlo meth- ods because the sigma points are carefully chosen to capture the first two moments of the underlying distribution. Second, it ensures persistence of the mixture components; when we resample, we sample another nearby point from the mixture component and not entirely new location.
EVALUATION AND EXPERIMENTS
In this empirical study, we evaluate VDM ability to model multi-modal dynamics and show its competitive forecasting performance in various domains. We first introduce the evaluation metrics thereby
and basclines. Experiments on synthetic data demonstrate that VDM is truly supporting the modeling choices of Section especially for the inference model. Then, experiments on real-world datasets with challenging multi-modal dynamics show the benefit Vf over state- of-the art (deep) probabilistic time-series models.
Evaluation metrics. In the experiments, we always create training set, validation set, and test set. During validation and test, each trajectory is split into two parts; initial observations (given the models for inference) and continuations of the trajectories (to be predicted and not accessible to the models). The inference models are used to process the initial observations and to infer latent states. These are then processed by the generative models to produce forecasts.
We use criteria to evaluate these forecasts (i) multi-steps ahead prediction p(x++1=4+ X1:),, (ii) one-step-ahead prediction p(x++1 xI and (iii) empirical Wasserstein distance. As other work (Lee et al. 2017: BBhatacharyya et al.) 2018 2019). (i) and (ii) are reported in terms of negative log-likelihood. While the predictive distribution for one-step-ahead prediction is in closed-form, the long-term forecasts have to be computed using samples. For each ground truth trajectory generate n = 1000 forecasts &; given initial observations from the beginning of the trajectory
NLL - log
Σ n
1
exp
[x, x)²
(13)
2n
2
This evaluates the predictive accuracy but neglects key aspect of multi-modal forecasts - diversity.
We propose a new evaluation metrie, which takes both diversity and accuracy of predictions into account. It relies on computing the Wasserstein distance between two empirical distributions P, Q
W(P,Q) inf inf Σ [x( (xcolla)
(14)
where x and are the discrete samples of P and Q, and denotes all permutations (Villani 2008 To use this as an evaluation measure for multi-modal forecasts, we the following. We select n samples from the test set with similar initial observations. If the dynamics in the data are multi- modal the continuations of those n trajectories will be diverse and this should be reflected in the forecasts. For each of the samples, the model generates 10 forecasts and we get groups of samples. With Eq. (4) the empirical W-distance between the n true samples, and each group of generated samples can be calculated. The averaged empirical W-distance well the generated samples match the ground truth. Repeating this procedure with different initial trajectories evaluates the distance between the modeled distribution and the data distribution.
groups evaluates how

---

Under review as a conference paper at ICLR 2021
VDM(k 9)
VDM 1)
AESMC 9)
D
p(zz(D) p(22)xc1)
D
p(zz(D) p(az (x))
D
p(zz[D) p(za|x<1)
Figure 3: Experiments on 2d synthetic data with 4 modes highlight the multi multi-modality of VDM We train VDM = 9) (left), VDM = 1) (middle), and AESMC &: 9) (right) on training set of trajectories of length 4, and plot generated trajectories D (2 colors for 2 dimensions). We also plot the aggregated posterior p(zz D), and the predictive prior p(zg x<1) (4 colors for 4 clusters, and not related to the colors in the trajectories plot) at the second time step. Only VDM learns a multi-modal predictive prior, which explains its success in modeling multi-modal dynamics.
Table I: Prediction errоr on stochastic Lorenz attractor for three evaluation metrics (details in main text). VDM(k 13) achieves the best performance, and AESMC also gives comparable results.
RKN
VRNN
CF-VAE
AESMC
VDM(k: = 1)
VDM(k: 13)
Multi-steps 104,41
65.8
02.3
25.01=0.22
23.03=0.28
24.46=0.12
One-step
1.88
-1.63
-1.69
-1.81
-1.81
W-distance
16.16
16.14
8.44+0.005
7.29:+0.005
7.31=0.002
7.28=#0.002
Baselines. We choose baselines from three classes of models. Two stochastic recurrent models are variational recurrent neural network (VRNN) (Chung et al. 2013) and auto-encoding sequential Monte Carlo (AESMC) (Le et al. 2018). VRNN has a similar but more powerful generative model than VDM and AESMC uses (SM to achieve tighter lower bound. But compared to VDM both methods have a less powerful inference model which limits their capacity to capture multi multi-mun molalll distributions. The third baseline is deep SSM The recurrent Kalman network (RKN) (Becker) ctt al. 2019) models the latent space with a locally linear SSMM which makes the prediction step and update step analytic (as for Kalman filters (Kalman 1960)). A final baseline is the conditional flow variational autoencoder (AF-) (Bhattacharyya et al., (019), which uses conditional normalizing flows model global prior for the future continuations and achieves tfae--ffthh-- art performances.
To investigate the necessity of taking multiple samples in the VDM inference model, we also com- pared to VDM(K 1) which uses only a single sample in Eq. 55)
VDM(k 1) has a simpler generative model than VRNN (it considers no autoregressive feedback of the observations x), but the same inference model. More ablations for the modeling choices Vr are in Appendix H
For fair comparison, we fix the dimension of the latent variables 2, and h, to be the same for VDM AESMC and VRNN which have the same resulting model size (except for the additional autore- gressive feedback in VRNN). AESMC and
use the same number of particles/samples. RKN does not have recurrent states, so we choose a higher latent dimension to make model size comparable. In contrast, CF-VAE has only one global latent variable which needs more capacity and we make it higher-dimensional than z₁. Details for each experiment are in Appendix G
Synthetic data with multi-modal dynamics. We generate synthetic data with two dimensions and four modes and compare the performance of VDD with 9 samples (Fig. 3, left), VDM with a single sample (Fig. 3 middle), and AESMC using 9 particles (Fig. 3 right). Since variational inference is known to try to match the aggregated posterior with the predictive prior (Tomczak & Welling. 2018), it is instructive to fit all three models and to look at their predictive prior [[Zg|x<1) and the aggregated posterior PDz) Because of the multi-modal nature of the problem, all 3 aggregated posteriors are multi-modal, but only V = 9) learns a multi-modal predictive prior (thanks to its choice of variational family). Although AESMCachievee a good match between the prior and the aggregated posterior, the predictive prior does not clearly separate into different modes. In contrast, the inference model of VDM successfully uses the weights (Eq. (6). which contain information about the incoming observation, to separate the latent states into separate modes.
Stochastic Lorenz attractor. The Lorenz attractor is system governed by ordinary differential equations. We add noise to the transition and emission function to make stochastic (details in Appendix F.I. Under certain parameter settings it is chaotic erable differences in the future. This makes forecasting its dynamics very challenging. All models trained and then tasked to predict 90 future observations given 10 initial observations. Fig. 4
small errors can cause consid-
7

---

Under review as a conference paper at ICLR 2021
(a) True sample (b) VDM (ours)
(c) AESMC
(d) CF-VAE
(e) VRNN
n RKN
Figure 4: Generated samples from VDM and baselines for stochastic Lorenz attractor. The models generate the remaining 990 observations (blue) based on the first 10 observations (red). Due to the chaotic property, the reconstruction is impossible even the model learns the right dynamics. VDM and capture the dynamics very well, while RRNN fails in capturing the stochastic dynamics.
Trajectory
VDM 13)
VDM(k 1)
AESMC 13)
Figure 5: An illustration of predictive priors p(zz)x<<)) of taxi trajectories from VDM 13), VDM(K = 1), and AEdAAMMC k: 13) at 3 forks in the road marked on the map (left). VDM = 13) succeeds in capturing the multi-modal distributions, while the other methods approximate them with uni-modal distributions. For visualization, the distributions have been projected to 2d with KDE.
Table 2: Prediction error on taxi trajectories for three evaluation metrics (details in main text). CF
VAE gives the best result in multi-steps prediction, since uses one global latent variable, while sequential models rely on a sequence of local latent variables. Meanwhile, VDM(k = 13) outper-
forms all sequential models, and performs better in other metrics than CF- VAE
RKN
VRNN
CF-VAE
AESMC
VDM(A: 1)
VDM(K 13)
Multi-stops 4.25
5.502
2.77=00.001
3.31=0.001
3.26+0.001
2.85+0.002
One-step -2.90
-2.77
-2.87 0.66:10.0004
-2.99
-3.62
W-distance
2.07
2.43.20.0002
0.76.10.0003
0.69
0.56.10.0005
illustrates qualitatively that VDM (Fig. 4b) and AESMC (Fig. 4c succeed in modeling the chaotic
dynamics of the stochastic Lorenz attractor, while CF- VAE (Fig. 4d) and and (Fig. [4) miss local details, and RKNN (Fig. (4) which lacks the capacity for stochastic transitions does not work all. VDM achieves the best scores on all metrics (Table D Since the dynamics of the Lorenz attractor governed by ordinary differential equations, the transition dynamics at each time step not ob- viously multi-modal, which explains why all models with stochastic transitions do reasonably well. Next, we will show the advantages of VDMM real-world data with multi-modal dynamics.
Taxi trajectories. The taxi trajectory dataset involves taxi trajectories with variable lengths in Porto, Portugal. Each trajectory is a sequence of two dimensional locations over time. Here, we cut the trajectories to fixed length of 30 to simplify the comparison (details in Appendix F.2 The task to predict the next 20 observations given 10 initial observations. Ideally, the forecasts should follow the street map (though the map is not accessible the models).
The results in Table 2 show that VDM outperforms the other sequential latent variable models in all evaluations. However, it turns out that for multi-step forecasting learning global structure is advantageous, and CC-V which is a global latent variable model, achieves the highest results. However, this value doesn't 't match the qualitative results in Fig. [] Since|CF--AA has encode the entire structure of the trajectory forcast into single latent variable, its predictions seem to average over plausible continuations but are locally neither plausible nor accurate. In comparison, VDM and the other models involve a sequence of latent variables. As the forecasting progresses, the methods update their distribution over latest states, and the impact of the initial observations becomes weaker and weaker. As result, local structure is captured more accurately. While the forecasts are plausible and can be highly diverse, they potentially evolve into other directions than the ground truth. For this reason, their multi-step prediction results are worse in terms of log-likelihood. That's why the empirical W-distance is useful to complement the evaluation of multi-modal tasks. It reflects that the forecasts of VDM are diverse and plausible. Additionally,
illustrate the predictive prior [xc) at different time steps in Fig. 5 VDM = 13) learns a modti-modal predictive prior, with an uni-modal Gaussian.
which VDM 1) and AESMCaaprroxint
8

---

Under review as a conference paper at ICLR 2021
(a) NBA data
VDM (uurs)
AESMC
(d) CF-VAE
(c) VRNN
(
RKN
Figure 6: VDM and CF- VAE generate plausible multi-modal trajectories of basketball plays. Each
model's forecasts (blue) are based on the first observations (red). Ground truth data green.
Table 3: Prediction error on U.S. pollution data for two evaluation metrics (details in main text).
7DMimaker the most accurate multi-step and one-step predictions.
RKN
VRNN
CF-VAE
AESMC
VDM(k 1)
VDM( 17)
Multi-steps One-step 6.98
53.13
49. 19.32-0.13
45.86-00.04
41.44:0.13
42.33-+0.11 7.97
36.72=0.08 6.05
8.69
6.93
Table 4: Prediction error on basketball players' trajectories (details main text). most accurate multi-step and one-step predictions.
VDM makes the
RKN
VRNN
CF-VAE
AESMC
VDM(A 1)
VDM(k- = 13)
Multi-steps
4.88
0.029
3.2400.003
3.74:30.033
3.56:50.005 -4.26
3.18:0.005
One-step
1.55
-2.78
n.a
-3.91
-5.18
U.S. pollution data. In this experiment, we study VDM the U.S. pollution dataset (details in Appendix F3 The data is collected from counties in different states from 2000 to 2016. Each observation has 12 dimensions (mean, max value, and air quality index of NO2, 03, SO2, and O3). The goal is to predict monthly pollution values for the coming 18 months, given observations of the previous months. We ignore the geographical location and time information to treat the development tendency of pollution in different counties and different times as i.i.d. The unknown context information makes the dynamics multi-modal and challenging to predict accurately. Due to the small size and high dimensionality of the dataset, there not enough samples with very similar initial observations. Thus, we cannot evaluate empirical W-distance in this experiment. In multi-step predictions and one-step predictions, VDM outperforms the other methods.
NBA SportVu data. This dataseff of sequences 2D coordinates describes the movements of basketball players and the ball. We extract the trajectories and cut them to fixed length of 30 to simplify the comparisons (details in Appendix F.4. The task is to predict the next 20 observations given 10 initial observations. Players move anywhere the court and hence their movement is less structured than the taxi trajectories which are constrained by the underlying street map. Due this, the initial movement patters are not similar enough each other evaluate empirical W- distance. In multi-step and one-step predictions, VDM outperforms the other baselines (Table 4 Fig. 6 illustrates qualitatively that [VDM (Fig. (6b) and VAE (Fig. 6d) succeed in capturing the multi-modal dynamics. The forecasts of AESMC(FFg 6c) are less plausible (not smooth as data),
and VRNN (Fig. 6e) and RKN (Fig. 6f
capturing the multi-modality.
CONCLUSION
We have presented variational dynamic mixtures VDM,. a sequential latent variable model for multi-modal dynamics. The main contribution new variational family. propagates multiple samples through an RNN to parametrize the posterior approximation with mixture density network. Additionally, we have introduced the empirical Wasserstein distance for the evaluation of multi- modal forecasting tasks, since it accounts for forecast accuracy and diversity. [DMM succeeds in learning challenging multi-modal dynamics and outperforms existing work in various applications.
'A version of the dataset is available https//wwww..aas.com/aaaaaaaaaa

---

Under review as a conference paper at ICLR 2021
REFERENCES
lenkaran Arasaratnam and Simon Haykin. Cubature kalman filters. IEEE Transactions on automatic control, 14(6))1254-1260) 1269, 2009.
Marie Auger-Méthé, Chris Field, Christoffer M Albertsen, Andrew E Derocher, Mark Lewis, Ian D Jonsen, and Joanna Mills Flemming. State-space models' dirty little secrets: even simple linear gaussian models can have estimation problems. Scientific reports, 6(1):1-10, 2016.
Philipp Becker, Harit Pandya, Gregor Gebhardt, Cheng Zhao, James Taylor, and Gerhard Neumann. Recurrent kalman networks: Factorized inference in high-dimensional deep feature spaces. In shitt International Conference Machine Learning. 2019.
Philip Becker-Ehmele Jan Peters, and Patrick Van Der Smagt. Switching linear dynamics for varia- tional bayes filtering. In International Conference Machine Learning, pp. 553-562, 2019.
Apratim Bhattacharyya, Bernt Schiele, and Mario Fritz. Accurate and diverse sampling of sequences based on "best of many" sample objective. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 8485 8493, 2018.
Apratim Bhattacharyya, Michael Hanselmann, Mario Fritz, Bernt Schiele, and Christoph-Nikolas Strachle. Conditional flow variational autoencoders for structured sequence prediction. arXiv preprint arXiv: 1000009908.00008,, 2019.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.555, 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988, 2015.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau, Gru-ode-bayes: Continuous modeling of sporadically-obsere time series. In Advances in Neural Information Processing Systems, pp. 7377-7388, 2019.
Andreas Doerr, Christian Daniel, Martin Schiegg, Duy Couyen-Tuong Stefan Schaal, Mare Tous- saint, and Sebastian Trimpe. Probabilistic recurrent state-space models. In Thirty-fifth Interna- tional Conference on Machine Learning, 2018
Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in neural information processing systems, pp. 2199-2207, 2016.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances in Neural Information Processing Systems, pp. 3601-3610, 2017.
Daniel Gedon, Niklas Wahlström. Thomas B Schön, and Lennart Ljung. Deep state space models for nonlinear system identification. arXiv preprint arXiv:2003. 14162,,1111...... 2020.
Mevlana Gemici, Chia-Chun Hung, Adam Santoro, Greg Wayne, Shakir Mohamed, Danilo Rezende, David Amos, and Timothy Lillicrap. Generative temporal models with memory. arXiv preprint arXiv: 0444004990, 2017.
Anirudh Goyal Alias Parth Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. In Advances in neural information processing systems, pp. 6713-6723,
2017. 2017.
10

---

Under review as a conference paper at ICLR 2021
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. In Thirty-Second AAAI Conference on Artificial Intel- ligence, 2018.
Marcel Hirt and Petros Dellaportas. Scalable bayesian learning for state space models using vari- ational inference with sme samplers. In The 22nd International Conference on Artificial Intelli- gence and Statistics, pp. 76-86,22 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-17800 1780, 1997.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. In 5th International Conference Learning Representations, 2017.
Vineet Kosaraju, Amir Sadeghian, Roberto Martin-Martin, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan gan and graph attention networks. In Advances in Neural Information Processing Systems, pp. 137-146, 2019.
Rahul G Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Thirty-first uaui conference on artificial intelligence, 2017.
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, pp.
2016.
Tuan Anh Le, Maximilian Igl, Tom Rainforth, Tom Jin, and Frank Wood. Auto-encoding sequential monte carlo. In International Conference on Learning Representations, 2018.
Namhoon Lee, Wongun Choi, Paul Vernaaa, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 336-345, 2017.
Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. Thirty-fitth International Conference Machine Learning, 2018.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, pp. 3812-3822, 2017.
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. Artificial Intelligence and Statistics, pp. 914 2017.
Thomas Lucas, Konstantin Shmelkov, Karteek Alahari, Cordelia Schmid, and Jakob Verbeek. Adap- tive density estimation for generative models. In Advances in Neural Information Processing Systems, pp.
2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. 1411 1784, 2014.
arXiv preprint
Christian Nacsseth, Scott Linderman, Rajesh Ranganath, and David Blei. Variational sequential monte carlo. In International Conference on Artificial Intelligence and Statistics, pp. 968-977. PMLR, 2018
Josue Nassar, Scott Linderman, Monica Bugallo, and II Memming Park. Tree-structured recurrent switching linear dynamical systems for multi-scale modeling. International Conference on Learning Representations, 2018.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Jamuschowski. Deep state space models for time series forecasting. In Advances in neural information processing systems, pp. 7785-7794, 2018.

---

Under review as a conference paper at ICLR 2021
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. In 4th International Conference on Learning Representations, 2016.
Joeri Rogelj, Michel Den Elzen, Niklas Höhne, Taryn Fransen, Hanna Fekete, Harald Winkler, Roberto Schaeffer, Fu Sha, Keywan Riahi, and Malte Meinshausen. Paris agreement climate proposals need boost to keep warming well below 2 Nature, 534(7609):631-63) 2016.
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical con- straints. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1349-1358, 2019.
Ardavan Saeedi, Tejas D Kulkarni, Vikash K Mansinghka, and Samuel Gershman. Variational particle approximations. The Journal of Machine Learning Research, 18(1):232822356 2017.
Florian Schmidt and Thomas Hofmann. Deep state space models for unconditional word generation. In Advances Neural Information Processing Systems, pp. 6158-6168 2018.
Florian Schmidt, Stephan Mandt, and Thomas Hofmann.
Autoregressive text generation beyond feedback loops. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.
2019.
Jakub M Tomezak and Max Welling. Vae with a vampprior. In 2 International Conference Artifi-cial Intelligence and Statistics, 2018.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008
Eric A Wan and Rudolph Van Der Merwe. The unscented kalman filter for nonlinear estimation. In Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373), pp. 153-158, 158. Icce, 2000.
Yuanxin Wu, Dewen Hu, Meiping Wu, and Xiaoping Hu. A mumerical-integration perspective on gaussian filters. IEEE Transactions Signal Processing, 54(8):2910-292)) 2006.
Xun Zheng, Manzil Zaheer, Amr Ahmed, Yuan Wang. Eric Xing, and Alexander J Smola. State space Istm models with particle meme inference. arXiv preprint arXiv: 2017.
Zachary M Ziegler and Alexander M Rush. Latent normalizing flows for discrete sequences. In Thitt International Conference on Machine Learning, 2019.
12

---

Under review as a conference paper at ICLR 2021
A SUPPLEMENTARY TO WEIGHTING FUNCTION
In this Appendix we give intuition for our choice of weighting function Eq. 66 Since we approx-
imate the integrals in Eqs. (3) and 7 with samples from @(s Xxt) instead of samples from (s Xge), importance sampling tells us that the weigths should be
(S)-1.X)) = g xge) ((x;
8 xxx) (( Xxi)
@(s xx1)
g(xe)
xxi)
@( Xct)
g(x: s1-1,
& q(x, 81-1.
(15)
g(x1 xx())
This is consistent with out earlier definition of ( xgi) weights are proportional to the likelihood of the variational model q(x; xx(t) We choose to parametrize it using the likelihood of the generative model p(x;) h St-1)) and get
xet). The
w = wss //k/ = [ arg max ma h = (2)
(16)
With this choice of the weighting function, only the mixture component with the highest likelihood selected to be charge of modeling the current observation x, As result, other mixture compo- nents have the capacity to focus on different modes. This helps avoid the effect of mode-averaging. An alternative weight function given Appendix
B
SUPPLEMENTARY TO LOWER BOUND
Claim. The ELBO in Eq. ( is a lower bound on the log evidence log p(x; xxt)).
logp(xe xxt) (FLBO(x((,))
(17)
Proof.
We write the data evidence as the double integral over the latent variables 2,, and zec.
log, xxx) = log /f p(xx aie. Xet)p(Ze Zez,, X<)p((z) x=))dz,dzet
(18)
We multiply the posterior at the previous time step p(zez x<x) with the ratio of the approximated /(a.b) ((Rc)(Xee) and the ratio f(a.b)* where is any suitable function of two variables and b.
posterior q(
The following equality holds, since the ratios equal one.
log xx<))
/ f(a, b) g(Ze)) xex) p(ace x<i) f p(xi ag,, xz),x=))p(zz Zez, X=i, f(a. b) q(z<) xx1)
log
(19)
We move the integral over Zet with respect to f(a. b)g(Zee xxx) out of the log operation with applying the Jensen's 's inequality.
log / p(xx ags,, xct)p(z nic,, xc1)
log xxx)) 2 E (a.baaleec
(20)
log f(a, b) + log g(z<r xxt)
Bs(a.bjo(Realeee)
p(z<l xxt)
We introduce the variational posterior ((z Zeces xxt), and apply Jensen's 's inequality to replace the intractable integral log p(x; 2gc, x (zz Zecee dz, with its lower bound.
p(x;
Zigi, xxe) x<t)p(Z
Zett Xxt)
log p(x; xxxr) 2 (((..b)(eesssee))
E (glealaer1xxr)
log
g(2r
Zice xgx)
gZzze xxt)
(ee))
log (a, b) + log
(21)
p(zce
xxx))
'The ~ just helps visually distinguish the distributions that appear in the main text.
13

---

Under review as a conference paper at ICLR 2021
The expectation with respect to f(a. b)q(zzi x<() is approximated with samples. Instead of re- sampling the entire history, samples from previous time steps are reused (they have been aggregated by the RNN) and sample according to Eq. (4). We plugg in the weighting function w(s X+) for f(a. b). The term log p(zct(xce) treated as a constant.
s(*ct)x<() not affected by the incoming observation X and can be
In this step, we plug our generative model and inference model as they are described in the main text for and q. The conditional independence assumptions can be read of Fig. 2 In the generative model 1 and in the inference model 9 summarize the dependencies of z, on the previous latent variables Zice and observations X In other words, we assume 2, is conditionally independent on Zec and Xxt given | in the inference model (or given h the generative model).
1 b
log p(x; xxi)= 2
Σ
w(s
xx)
log logp(xi Zz, h1-1 = s121)
k
(2.111))) (xxx)
k
p(ze
h 1 = 5/-1)
(8)
+
w(s
(
(1)
E
log g 1/
w(s w( (1)
x1)
logw(s (1)
+ +c]
(22)
C ALGORITHMS OF GENERATIVE MODEL AND INFERENCE MODEL
Algorithm Inference model
Algorithm Generative model
Inputs: X1.r, ho
Inputs: [M2.r, 02.r) h
Outputs: [M,, d2.1r] hr-1
Outputs: 4,+1
[M2.1,02.1 tfnn (ho, X1)
Z- ~N,,( 22.11)
for 2 do
h, GRU (zr, hr-1)
(1)
for = 1:
5 = Φ GRll (z h:-2)
(1)
[Mo.e: Tn = (hy-1)
2, ~ ((o,, od,11)
[u2.t. (0] gins (1 (s
(i)
h, T 1)-1)
(( = 1(i = arg arg max, p(x: hi-1 = 2(21))
[uz,c, (ze h1-1)
(Mr,,,o'. = Σt wi (( (.(21
X; ~ NNNNNnen NNH,,,, (I
h Rd Σ w1 (2)
end for
end for
D SUPPLEMENTARY TO STOCHASTIC CUBATURE APPROXIMATION
Cubature approximation. The cubature approximation is widely used in the engineering com- munity as a deterministic method to numerically integrate a nonlinear function f(-) of Gaussian random variable z ~ ( o2I), with z E Rd. The method proceeds by constructing 2d 1 sigma points (x)) ft + oz5 (t)) The cubature approximation is simply a weighted sum of the sigma points propagated through the nonlinear function f(-).
f [(z)N(Z) Ue, Iz
2d+1
Σ 2020
(23)
Simple analytic formulas determine the computation of weights (0) and the locations of g(1)
14

---

Under review as a conference paper at ICLR 2021
[(++)) , 1, b.... 2n
vn + kC;
, = 1,...,
(10)
,8=0 0
Vn ke;
,1 + ..., 2n
0
, 0,
(24)
where x is a hyperparameter controlling the spread of the sigma points in the n-dimensional sphere. Further e; represents basis in the n-dimensional space, which is choosen to be unit vector in cartesian space, e.g. e Ol.
Stochastic cubature approximation. In SCA
we adopt the computation of (()) in Eq. 24 and
infuse the sigma points with standard Gaussian noise ~ NO, II) to obtain stochastic sigma vari- ables (+) = Pz + o2(((() + e). We choose x = 0.5 to set the weights 700 equally.
E SUPPLEMENTARY TO ABLATION STUDY OF REGULARIZATION TERMS
We investigate the effect of the regularization terms using the synthetic data from Fig. 3 We can see T VDM(k = 9) can be trained successfully with LELBO only, and both regularization terms improve the performance (negative log- log-likelihood of multi- ahead prediction), while VDM(k 1) doesn't 't work whatever the regularization terms. Additionally, we tried train the model only with the regularization terms (each separate or together) but these options diverged during training.
Table 5: Ablation study of the regularization terms for synthetic data from Fig.
LELBO 2.439=0.005
&
CELBO&Lode 2.381 3.743::0.005
LVDM 2.363+10.04
VDM = 9)
2.379:10.008 3.960:10.008
VDM (k 1)
3.75610.003
3.878=0.0007
F SUPPLEMENTARY TO EXPERIMENTS SETUP
F.1 STOCHASTIC LORENZ ATTRACTOR SETUP
Lorenz attractor is system of three ordinary differential equations:
dx dt = o x),
dy = x(p - 2) - dt
dz
xy ,3z,
(25)
de
where p, and are system parameters. We set 10, p 28 and 3 3 8 to make the system chaotic. We simulate the trajectories by RK4 with a step size of 0.01. To make stochastic, we add process noise to the transition, which is mixture of two Gaussians 0.5. P) 0.5.(m,, P), where
0 -1 0
00.06
0.03
0.017
mg
m
p=
0.03
0.03
0.03
(26)
[0.01
0.03
0.05
Besides, add Gaussian noise with zero mean and diagonal standard deviation [0.6, 0.4, 0.8] as the observation noise. Totally, we simulate 5000 sequences as training set, 200 sequences as validation set, and 800 sequences test set. For evaluation of Wasserstein distance, simulate 10 groups of sequences additionally. Each group has 100 sequences with similar initial observations.
F.2 TAXI TRAJECTORIES SETUP
The full dataset is very large and the length of trajectories varies. We select the trajectories inside the Porto city area with length in the range of 30 and 45, and only extract the first 30 coordinates of each trajectory. Thus we obtain dataset with a fixed sequence length of 30. We split it into the training set of size 86386, the validation set of size 200, and the test set of size 10000.
15

---

Under review as a conference paper at ICLR 2021
F.3 U.S. POLLUTION DATA SETUP
The U.S. pollution dataset consists of four pollutants (NO2. O3, SO2 and O3). Each of them has major values (mean, max value, and air quality index). is collected from counties different states for every day from 2000 to 2016. Since the daily measurements are too noisy, we firstly compute the monthly average values of each measurement, and then extract non-overlapped segments with the length of 24 from the dataset. Totally extract 1639 sequences as training set, 25 sequences as validation set, and 300 sequences as test set.
F.4 NBA SPORTVU DATA SETUP
We use a sliding window of the width 30, and the stride 30 to cut the long sequences to short sequences of fixed length 30. We split them into the training set of size 8324, the validation set of size 489, and the test set of size 980.
G IMPLEMENTATION DETAILS
Here, we provide implementation details of VDM models used across the three datasets in the main paper. VDM of
encoder: embed the first observation to the latent space as the initial latent state Zp.
transition network: propagate the latent states
decoder: map the latent states and the recurrent states h, observations X₃-
inference network: update the latent states z, given observations X₃-
latent GRU: summarize the historic latent states Zer in the recurrent states h,,
discriminator: used for adversarial training.
The optimizer is Adam with the learning rate of - 3. In all experiments, the networks have the same architectures but different sizes. The model size depends observation dimension dx. latent state dimension dy, and recurrent state dimension d,- The number of samples used each time step in the training is 2dl + 1. If the model output is variance, its non neg-tvea
the exponential of it to ensure
Encoder: input size is dzi 3 linear layers of size 32, 32 and 2d,, with 2 ReLUs.
Transition network: input size is da; linear layers of size 64, 64, and 2d,, with 3 ReLUs.
Decoder: input size is d, + dz; 3 linear layers of size 32, 32 and 2dx, with 2 ReLUs.
Inference network: input size is dl, dxi 3 linear layers of size 64, 64, and 2dg, with ReLUs.
Latent GRU: one layer GRU of input size d₂ and hidden size dn
Discriminator:
one layer GRU of input size dy and hidden size d, to summarize the pre- vious observations as the condition, and a stack of linear layers size 32, 32 and 1, with 2 ReLUs and one sigmoid as the output activation, whose input size is d,, + dx-
Stochastic Lorenz attractor. Observation dimension dy is 3, latent state dimension d2 is 6, and recurrent state dimension dp, 32.
Taxi trajectories. Observation dimension dy is 2, latent state dimension d2 is 6, and recurrent state dimension dn is 32.
U.S. pollution date
Observation dimension dy is 12, latent state dimension dg is 8, and recurrent
state dimension d, is 48.
16

---

Under review as a conference paper at ICLR 2021
NBA SportVu data. Observation dimension dx is 2, latent state dimension d2 is 6, and recurrent state dimension du is 32.
Here, give the number of parameters for each model in different experiments in Table 6
Table 6: Number of parameters for each model in three experiments.
VDM
AESMC
VRNN
and
RKN have comparable number of parameters.
CF- VAE has much more parameters.
RKN
VRNN
CF-VAE
AESMC
VDM
Lorenz
23170
22506
7497468
22218
22218
Taxi
23118
22248
7491123
22056
22056
Pollution
35774
33192
8162850
31464
31464
Vu
23118
22248
7491123
22056
22056
H ADDITIONAL EVALUATION RESULTS
We evaluate more variants of VDM in the chosen experiments investigate the different choices of sampling methods (Monte Carlo method, and SCC)) and weighting functions (Fqs. (7 and (88
In addition Eq. (27) described in the main text, define other choice in Eq. 28
w ( 1/k = 1 arg max p(x, h (1))
(27)
5
w/"h w x,k/k: 1ii Cat(- [ w)),, w x p(x: h
st-1)
(28)
We define the weighting function as an indicator function, in Eq. (27 we set the non-zero component by selecting the sample that achieves the highest likelihood, and in Eq. (28) the non-zero index is sampled from a categorical distribution with probabilities proportional to the likelihood. The
first choice (Eq. (27 is named with 8-function, and the second choice (Eq. (28) is named with
categorical distribution. Besides, in VDM-Net, we evaluate the performance of replacing the closed-
Table 7: Definition of VDM variants
VDM(k = 1) VDM-MC+0
VDM-SCA+Cat
VDM-SCA+0
Sampling method Weighting function
Monte-Carlo
Monte-Carlo
SCA
SCA
n.a.
ô-function
Categorical distribution
8 6-function
form inference of the weighting function with an additional inference network. In Table the choices in different variants. All models are trained with &
7 we show
H.1 STOCHASTIC LORENZ ATTRACTOR
Table 8: Ablation study of VDM variants stochastic Lorenz attractor for three distance metrics
(see main text). The variants are defined in Table All variants give comparable quantitative results.
VDM(K 1)
VDM-Net
VDM-MC+6
VDM-SCA+Car
VDM-SCA+8
Multi-steps One-step
25.03
20.65
24.67+0.16
F0.16
0.
-1.81
-1.71
-1.84
-1.83
-1.81
W-distance
7.31+0.002
7.68+0.002
7.315
7.30400009
7.29-0.003
H.2 TAXI TRAIECTORIES
H.3 U.S. POLLUTION DATA
17

---

Under review as a conference paper at ICLR 2021
Table Ablation study of VDM| variants on taxi trajectories for three distance metries (see main text). The variants are defined in T7 VDM-SCA+6 outperforms other variants and approaches our default VDM (trained with Lodv additionally).
VDM
VDM-Net
VDM-MC+0
VDM-SCA+Cal
VDM-SCA+0
Multi-steps
3.2010.001
3.682
3.17
3.0910.001
2.88
One-step
-2.99
-2.74
-3.21
-3.24
-3.68
W-distance
0.69:10.0005
0.79-10.0003
0.70=1000008
0.64=0.0005
0,59:0000008
Table 10: Ablation study of VDM variants U.S. pollution data for two distance metrics (see VDM-SCA+6 outperforms other variants.
main text). The variants are defined in Table 7
VDM(k 1)
VDM-Net
VDM-MC+6
VDM-SCA+Cal VDM-SCA+ó
Multi-steps One-step
42.33:10.11
50.44
40.33
39.58-00.09
37.64
7.97
10.70
8.12
7.82
6.91
(a) VDM-SCA+5 (b) VDM-SCA+Cal
(c) VDM-MC+6
(d) VDM-Net
(e) VDM(1: = 1)
Figure 7: Generated trajectories of stochastic Lorenz attractor from VDM variants. The first ten observations (red) are obtained from models given the first 10 true observations. The rest 990 ob- servations (blue) are predicted. We can see, all variants give very good qualitative results. Since the fundamental dynamics is govern by ordinary differential equations, the transition each time step not highly multi-modal. Once the model equipped with a stochastic transition, it is able to model this dynamics.
VDM (ours)
(b) AESMC
(c) CF-VAE
(d) VRNN
(c) RKN
Figure 8: Generated 50 taxi trajectories in different areas from VDM and the baselines. All models are required to predict the future continuations (red), based the beginning of trajectory (blue). generates more plausible trajectories compared with the baselines. While the gener- ated trajectories from VDM follow the street map, the generated trajectories from all baselines are physically impossible. AESMC and CF-VAE can capture the general evolving direction, but suffer from capturing the multi-modality each time step.
18

---

Under review as a conference paper at ICLR 2021
(a) VDM-SCA+5 (b) VDM-SCA+Cat
(c) VDM-MC+6
(d) VDM-Net
(e) VDM(1: = 1)
Figure 9: Generated 50 taxi trajectories from VDM variants. All models are required to predict the future continuations (red), based the beginning of trajectory (blue). VDM-SCA+5 achieves the best qualitative results among all variants. VDM-SCA+6 can generate plausible trajectories, even it is trained without the adversarial term Ladi. We can see, for the weighting function, Eq. (27) is better than Eq. (28). and for the sampling method, SCA is better than Monte-Carlo method.
19


Reviews:
[{"review_id": "dP_WdSl6u87", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Variational dynamic mixtures**\n\nThe paper extends the VRNN to cater for multi-modality in the probability distribution governing a dynamic system. This is particularly important when average trajectories are highly unlikely or even physically impossible. To achieve this, the authors start from VRNN and alter the inference model so that it uses stochastic recurrent states and a mixture variational posterior distribution (with 0/1 weights to trigger only the most likely mixture components encouraging multi-modality).\nAs a minor contribution they propose a new evaluation metric for measuring the diversity of generations based on Wasserstein distance. This is important in their case when the likelihood evaluation may favour generations from a single mode - a situation their model shall prevent.\n\nThe paper is very well motivated (with taxi trajectory prediction as a running use case) and well positioned with respect to the state of the art.\nThe experimental evaluation is convincing, using both synthetic and real experiments to support the claims and shows advantages over baselines. Ablation studies examine the importance of some more ad-hoc choices (showing these help but are not critical).\nThe paper is well written and structured to help the reader follow the main thoughts.\nHowever, there are some points in the mathematical formulation of the model which raise questions and deserve to be explained better - see below.\n\nFor this reason I recommend not accepting the paper for now but I'm am very much willing to improve my score significantly once these will have been clarified.\n\n* From Fig2 it seems that the stochastic states s_t do not exist in the generative model, they only live in the inference model. Right?\n\n* From eq. (1) and Fig2a we have $h_{t} = \\phi^{GRU}(z_{t}, h_{t-1})$ and the generative distribution of $p(z_{t+1} | z_{\\leq t})$ is a function $\\phi^{tra}(h_{t})$. Is it correct to think about this as the prior distribution for $z_{t+1}$?\n\n* My understanding of eq. (3) is that we have $\\widetilde{q}(s_{t-1} | x_{<t}) = 1$ if $s_{t-1} = h_{t-1}$ and zero otherwise. Is that right?\n\n* You say $q(s_{t-1} | x_{\\leq t}) = \\omega(s_{t-1}, x_t) \\, \\widetilde{q}(s_{t-1} | x_{<t})$.\n    * How can you condition $s_{t-1}$ on the future value $x_t$? (Also in comparison to equation (4), where you assume $q(z_t | x_{\\leq T}) = q(z_t | x_{\\leq t})$ for all $t$ thus avoiding dependence on future values of $x$. \n    * If my understanding of eq (3) is correct (see above), this is trivially 1 or 0 irrespective of $\\omega$.\n\n* In eq (4) you say $q(z_{1:T} | x_{1:T}) = \\Pi_{t=1}^T q(z_{t} | x_{\\leq t})$.  Is this correct? I would expect $\\Pi_{t=1}^T q(z_{t} | z_{\\lt t}, x_{\\leq t})$.\n\n* Again in eq (4) you say the final result is $\\Pi_{t=1}^T \\int q(z_{t} | s_{t-1}, x_{t}) q(s_{t-1} | x_{\\leq t}) d s_{t-1}$. Is this correct? I would expect $\\Pi_{t=1}^T \\int q(z_{t} | s_{t-1}, \\mathbf{x_{\\leq t}}) q(s_{t-1} | x_{\\leq t}) d s_{t-1}$. Does this mean you assume the conditional independence $q(z_{t} | s_{t-1}, \\mathbf{x_{\\leq t}}) = q(z_{t} | s_{t-1}, \\mathbf{x_{t}})$?\n\n* samples $s_{t-1}^{(i)}$ in eq (5) are constructed by sampling $z_{t-1}^{(i)} \\sim q(z_{t-1} | x_{<t})$ and passing it through the recurrent net $s_{t-1}^{(i)} \\gets h_{t-1}^{(i)} = \\phi^{GRU}(z_{t-1}^{(i)}, h_{t-2})$, right?\n\n* eq (5) and annex A: How come $\\omega$ is a function of $x_t$ only and not of $x_{<t}$. Given your importance sampling argument, $q$ conditions on $x_{<t}$ as well. Or do you assume the conditional indpepence $q(x_t | s_{t-1}, x_{<t}) = q(x_t | s_{t-1})$?\n\n* what is $p(x_t | s_{t-1})$ used in eq (6)? This looks like a generative distribution ($p$) but has not been defined before and $s_{t-1}$ do not exist in the generative model (figure 1a).\n\n* ELBO proof in annex B: can you please provide details (equations, detailing also the conditional independence assumptions you take) for getting from eq (7) to eq (8)?\n\n* Wasserstein distance in eq (14) - the differences are calculated between the generated trajectories and the corresponding true sample (the *group sample*) or all the n true samples?\n\nAFTER REVIEW UPDATE: I find the revised version much improved explaining the inference model much more clearly. The lack of clarity was for me the main reason for evaluating the paper as below the acceptance threshold despite the fact that otherwise I found the paper to be good and useful for the community. As the lack of clarity has now been, in my view, resolved, I increase my score to 7 - Good paper, accept.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZwanKxEK1eK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presented the variational dynamic mixtures as a deep probabilistic model for time series forecasting. The research issue, called the taxi trajectory prediction problem, is addressed. Some comments are provided.\n\nPros:\nA new solution to mixture density network as a kind of generative model with latent states and multinomial observations was proposed. The detailed experiments were addressed. New evaluation metric was introduced.\n\nCons:\nThere are a number of notations and variables which were not clearly defined. This matter made the reading to be easily confused. A clear algorithm or working flow for complicated system was missing. Some descriptions were not clear.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XDG_O7E77kE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper introduces variational dynamic mixtures (VDM), a new variational family, and demonstrates that using VDM to model the approximate posterior in sequential latent variable models can better capture multi-modality in data. VDM includes a distribution over recurrent states in the inference model, such that a sampling-based marginalization of this distribution reduces the approximate posterior to a mixture model. Setting the weights such that only the most probable mixture component is selected allows other mixture components to capture other modes. The authors validate VDM on both synthetic and real multimodal datasets, which outperform baselines with respect to negative log-likelihood and a new empirical Wasserstein distance.\n\nPositives\n\n+ This paper tackles an important and well-motivated problem: capturing multi-modality in data. This is very practical and I believe will be of interest to the ICLR community. \n\n+ Paper is well-written and easy to follow. I appreciate that the authors highlight their design decisions in the main text, while also providing alternatives and/or ablation studies in the appendix.\n\n+ VDM improves performance while also using a non-autoregressive generative model, compared to baselines with more powerful generative models (e.g. VRNN). This highlights the effectiveness of their inference model, also illustrated in the synthetic experiment in Figure 3. The inference model in VDM is also quite general, as a single-sample approximation in their inference model is equivalent to the inference model in VRNN.\n\nConcerns \n\n- I think this paper would benefit from one additional dataset where the multimodality is inherent in the data. Taxi trajectories are multimodal but also highly structured (trajectories must be on roads), so a different dataset you can consider can be pedestrian or sport trajectory datasets, where the data is also inherently multimodal but also less structured. I think some baseline models can do better in this setting (at least qualitatively), so I’m curious if VDM still convincingly outperforms them. My main concern is that the pollution data is synthetically multimodal (because I think that with contextual information the data is more periodic) and the Lorenz attractor experiment only highlights that VDM can handle stochasticity, which is also present in trajectory data.\n\n- The related work is missing a section about other methods that try to capture multimodality. For instance, VRNNs are known to not capture multimodality well, and there have been extensions along this direction such as in [1]. There’s also another line of work that introduces mutual information between trajectories and latent variables in the objective, such as in [2]. The “sequence forecasting” paragraph can be omitted/combined with “neural recurrent models”.\n\n- The results on the taxi dataset look good. It would be great if you can also provide analysis on the resulting latent space, similar to what was done in Figure 3.\n\n[1] Goyal et al. Z-Forcing: Training Stochastic Recurrent Networks \n\n[2] Li et al. InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations\n\nMinor Comments\n\n- The bold/thin lines in Figure 2 can be hard to distinguish. I recommend using a dotted line instead.\n\n- In Figure 3, how many timesteps are there in total? Are the blue/orange trajectories in the left plots corresponding to the blue/orange clusters in the middle/right plots?\n\n- Tables 1,2 and 3 all cut through paragraphs in the middle, which can be distracting.\n\n- Some quotations on page 5 use close quotations ” on both sides.\n\n---------\n\nPost-rebuttal comments\n\nThank you for adding the additional experiments and analysis. The results with the basketball dataset (Table 4, Figure 6) and the visualization of the latent distribution (Figure 5) address my inital concerns and showcase the versatility of VDM. I've increased my score from 6 to 7.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a variational family of distributions for posterior estimation in sequential latent variable models. The paper does so by extending variational recurrent neural networks so as to use a variational-mixture posterior and capture more realistic multi-modalities in the data. 

During the review process, it was suggested to improve the clarity of the paper, provide results on an additional dataset and a visualization of the latent distributions. I commend the authors for addressing these issues satisfactorily.

Overall, the paper is well-motivated and well-written. However, when considering the novelty of the paper, although none of the reviewers raised this issue, I believe the paper heavily relies on previously proposed ideas and therefore, its contribution can be seen as incremental. Additionally, something important to highlight is in section 2, with regards to deep state-space models (SSMs). The authors make a rather strong claim with regards to assumptions on the variational distribution. However, one can find out-of-the-box implementations where this is not the case, see e.g. https://pyro.ai/examples/dmm.html that implements deep Markov models with posteriors based on inverse autoregressive flows. A comparison with such approaches may be also required. 

Decision: Reject