Conditional Flow Variational Autoencoders for Structured Sequence Prediction
Authors: Apratim Bhattacharyya
Michael Hanselmann
Mario Fritz
Bernt Schiele
Christoph-Nikolas Straehle
Venue:  (2020)
Keywords: Variational Inference
Normalizing Flows
Trajectories

Abstract: Prediction of future states of the environment and interacting agents is a key competence required for autonomous agents to operate successfully in the real world. Prior work for structured sequence prediction based on latent variable models imposes a uni-modal standard Gaussian prior on the latent variables. This induces a strong model bias which makes it challenging to fully capture the multi-modality of the distribution of the future states. In this work, we introduce Conditional Flow Variational Autoencoders (CF-VAE) using our novel conditional normalizing flow based prior to capture complex multi-modal conditional distributions for effective structured sequence prediction. Moreover, we propose two novel regularization schemes which stabilizes training and deals with posterior collapse for stable training and better match to the data distribution. Our experiments on three multi-modal structured sequence prediction datasets -- MNIST Sequences, Stanford Drone and HighD -- show that the proposed method obtains state of art results across different evaluation metrics.

Under review as a conference paper at ICLR 2020
CONDITIONAL FLOW VARIATIONAL AUTOENCODERS FOR STRUCTURED SEQUENCE PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Prediction of future states of the environment and interacting agents is a key competence required for autonomous agents to operate successfully in the real world. Prior work for structured sequence prediction based on latent variable models imposes priors with limited expressiveness are difficult to optimize e.g. determining the number of Gaussian mixture components which makes it challenging fully capture the states. In this work, we introduce Conditional Flow Variational Autoencoders (CF- VAE) using our novel conditional normalizing flow based prior to capture complex multi-modal conditional distributions for effective structured sequence prediction. Moreover, we propose two novel regularization schemes which stabilizes training and deals with posterior collapse for stable training and better fit to the target data distribution. Our experiments on three multi-modal structured sequence prediction datasets MNIST Sequences, Stanford Drone and HighD show that the proposed method obtains state of art results across different evaluation metrics.
of the distribution of the future
INTRODUCTION
Anticipating future states of the environment is a key competence necessary for the success of autonomous agents. In complex real world environments, the future is highly uncertain. Therefore, structured predictions, In many scenarios, these tasks can be cast as sequence prediction problems. Particularly. Conditional Variational Autoencoders (CVAE) (Sohn et al. 2015 Bayer & Osendorfer] [2014] Chung al. 2015) have been very successful - from prediction of pedestrians trajectories (Lee et al. Bhattahcccr
to many mappings of the likely future states of the world, are important.
al.) 2018 Pajouheshgar & Lampert, 2018) to outcomes of robotic actions (Babaeizadeh al. 2018). The distribution of future sequences is diverse and highly multi-modal. CVAEs model diverse futures by factorizing the distribution of future states using a set of latent variables which are mapped to likely future states. However, CVAES assume standard Gaussian prior on the latent variables which induces a strong model bias (Hoffman & Johnson 2016 Tomezak & Welling 2018 which makes it challenging to capture mult-modal distributions. This also leads to missing modes due to
posterior collapse (Bowman et al. 2016 Razavi et al. 2019).
Recent work Tomezak & Welling 2018 Wang al. 2017 Gu et al. 2018 has therefore focused on more complex Gaussian mixture based priors. Gaussian mixtures still have limited expressiveness and optimization suffers from complications e.g. determining the number of mixture components. Normalizing flows are more expressive and enable the modelling of complex multi-modal priors. Recent work on flow based priors (Chen et al.) [0117 Ziegler & Rush 2019), have focused only on the unconditional (plain VAE) case. However, this not sufficient for CVAEs because in the conditional case the complexity of the distributions are highly dependent on the condition.
In this work, 1. We propose Conditional Flow Variational Autoencoders (CF-VAE) based on novel conditional normalizing flow based priors In order model complex multi-modal conditional distributions over sequences. In Iniiuurre I we show example predictions of MNIST handwriting stroke of our CF-. We observe that, given a starting stroke, our CF- VAE model with data dependent normalizing flow based latent prior captures the two main modes of the conditional distribution i... and 8 while CVAES with fixed uni-modal Gaussian prior predictions have limited diversity. 2. We propose regularization scheme that stabilizes the optimization of the evidence lower bound and leads to better fit to the target data distribution. 3. We leverage our conditional flow prior to deal with posterior collapse which causes standard CVAEs to ignore modes in sequence prediction tasks. 4. Finally, our method outperforms the state of the art on three structured sequence prediction tasks handwriting stroke prediction on MNIST, trajectory prediction Stanford Drone and HighD.

---

Under review as a conference paper at ICLR 2020
Latent Prior
Clustered Predictions
Latent Prior
Clustered Predictions
Standard Gaussian Prior - PVAEE
Conditional Flow Prior Our CF-VAE.
Figure 1: Clustered stroke predictions on MNIST sequences. Our multi-modal Conditional Nor- malizing Flow based prior (right) enables regularized CF-VAE to capture the modes of the conditional distribution, while predictions with uni-modal Gaussian prior (left) have limited diversity. Note, our 64D CF-VAE latent distribution is (approximately) projected 2D using tSNE and KDE.
RELATED WORK
Normalizing Flows. Normalizing flows are powerful class of density estimation methods with exact inference. (Dinh et al. 2015) introduced affine normalizing flows with triangular Jacobians. (Dinh et al. 2017) extend flows with masked convolutions which allow for complex (non-autoregessive) dependence between the dimensions. In (Kingma & Dhariwal, 2018),
convolutions were
proposed for improved image generation compared to (Dinh et al. 2017). In (Huang et al. 2018) normalizing flows are auto-regressive and (Behrmann et 2019) extend it to ResNet. (Lu & Huang 2019) extended normalizing flows to model conditional distributions. Here, we propose conditional normalizing flows to learn conditional priors for variational latent models.
Variational Autoencoders. The original variational autoencoder (Kingma & Welling 2014 used uni-modal Gaussian prior and posterior distributions. Thereafter, two lines of work have focused on developing either more expressive prior or posterior distributions, Rezende & Mohamed (2015) propose normalizing flows model complex posterior distributions, Kingma et al. (2016)) Tomezak & Welling (20 Berg et al. (2018) present more complex inverse autoregessive flows, householder and Sylvester normalizing flow based posteriors. Here, we focus on the orthogonal direction of more expressive priors and the above approaches are compatible with our approach.
Recent work which focus more expressive priors include (Nalisnick Smyth (017)) which proposes Dirichlet process prior and (Goyal et al. 2017) which proposes nested Chinese restaurant process prior. However, these methods require sophisticated learning methods. contrast, (Tomezak Welling 2018) proposes mixture of Gaussians based prior (with fixed number of components) which is easier to train and shows promising results on some image generation tasks. (Chen et al.) 2017), proposes inverse autoregressive flow based prior which leads to improvements In complex image generation tasks like CIFAR-10. ZZiegler & Rush 2019) proposes prior for VAE based text generation using complex non-linear flows which allows for complex multi-modal priors. While these works focus on unconditional priors, we aim to develop more expressive conditional priors.
Posterior Collapse. Posterior collapse arises when the latent posterior does encode useful information. Most prior work (Yang et al.) 2017 Dieng et af. 2019 Higgins et al., (017)) concentrate on unconditional VAEs and modify the training objective the KL divergence
annealed to prevent collapse to the prior. Liu et al. (2019) extends KL annealing to CVAEs. However, KL annealing does not optimize
lower bound of the ELBO for most of training. Zhao al. (2017) also modifies the objective to choose the model with the maximal rate. Razavi et al. (2019) propose anti-causal sequential priors for text modelling tasks. (oowman al. (2016); Gulrajani et al.) (2017) proposes to weaken the decoder so that the latent variables cannot be ignored, however only unconditional VAFs are considered, Wang Wang (2019) shows the advantage of normalizing flow based posteriors for preventing posterior collapse. In contrast, we study for the first time posterior collapse in conditional models on datasets with minor modes.
Structured Sequence Prediction. Helbing & Molnar (1995); Robicquet et al. (2016) 6 Alahi et al. (2016); Gupta et al. (2018): Zhao et al. (2019) Sadeghian et al. (2019) consider the problem of traffic participant trajectory prediction in social context. Notably, Guppt et al. 2018, Zhao al. 2019 Sadeghian et al. 2019) use generative adversarial networks to generate socially compliant trajectories. However, the predictions uni-modal. Starting from Bayer & Osendorter (2014): Chung al.
2

---

Under review as a conference paper at ICLR 2020
(2015),
more recently Lee et al. (2017) 7): Bhattacharyya al. (2018); Rhinehart et al. (2018) Deo
Trivedi ) Pajouheshgar & Lampent (018) considers structured (one to many) predictions using- CVAE, improved CVAE training, pushforward policies for vehicle ego- motion prediction, motion planning, spatio-temporal convolutional network respectively. Kuumar et al. (2019) proposes a normalizing flow based model for video sequence prediction, however the sequences considered have very limited diversity compared to the trajectory prediction tasks considered here. Here, we focus on improving structured predictions using conditional normalizing flows based priors.
CONDITIONAL FLOW VARIATIONAL AUTOENCODER (CF-VAE)
Our Conditional Flow Variational Autoencoder is based on the conditional variational autoencoder (Sohn et al.] 2015) which is a deep directed graphical model for modeling conditional data distributions po(y)x). Here, is the sequence up time I, x [x] .... .x']
and y is the sequence to be predicted
up time T. = [v/ ++ ... v²]. CVAEs factorize the conditional distribution using latent variables In detail, po(y|x) = J Po Iz, x) x))(z)x)dz, where p(z) (x)x is the prior on the latent variables. During training, amortized variational inference is used and the posterior distribution qa (2)x, y) is learnt using a recognition network. The ELBO maximized, given by.
log(pe(y|x)) 2 (2)())))) log(pe(y/z x)) DKL
In practice, to simplify learning, simple unconditional standard Gaussian priors are used (Sohn et al. 2015). However, the complexity e.g. the number of modes the target distributions (y x), is highly dependent upon the condition æ. An unconditional prior demands identical latent distributions irrespective complexity of the target conditional distribution - a very strong constraint on the recognition network. Moreover, the latent variables cannot encode any conditioning information and this leaves the burden of learning the dependence on the condition completely on the decoder.
Furthermore, on complex conditional multi-modal data, Gaussian priors have been shown to induce a strong model bias (Tomezak & Welling (2016) Ziegler & Rush 2019). It becomes increasingly difficult to map complex multi-modal distributions to uni-modal Gaussian distributions, further complicated by the sensitivity of the RNNs encoder/decoders to subtle variations in the hidden states (Bowman et al. 2016). Moreover, the standard closed form estimate of the KL-divergence pushes the encoded latent distributions to the mean of the Gaussian leading to latent variable collapse (Wang) ct al. 2017 Gu et al. 2018) while discriminator based approaches (Tolstikhin et al. 2017) lead to underestimates of the KL-divergence (Rosca et al. 2017).
Therefore, we propose conditional priors based on conditional normalizing flows to enable the latent variables to encode conditional information and allow for complex multi-modal latent representations. Next, we introduce our new conditional non- non-linear normalizing flows followed by our regularized Conditional Flow Variational Autoencoder (CF-VAE) formulation.
3.1 CONDITIONAL NORMALIZING FLOWS
Recently, normalizing flow proposed (Chen al., 2017 Ziegler Rush| 2019). Normalizing flows allows for complex priors by transforming simple base density e.g. standard Gaussian to a complex multi-modi density through a series of layers of invertible transformations f.,
Tabak et al. 2010; Dinh et al. 2015) based priors for VAEs have been
h1
h2
However, such flows cannot model conditional priors. In contrast to prior work, we utilize conditional normalizing flows to model complex conditional priors. Conditional normalizing flows also consists of series of n layers of invertible transformations fi (with parameters v/), however we modify the transformations fa such that they are dependent on the condition x,
filx h
falx h2lx
fnly z/x.
c/x
|x
(3)
Further, in contrast prior work (Lu & Huang 2019 Atanov al. 2019 Ardizzone et al. 2019)
which use affine flows (f)), we build upon (Ziegler & Rush 2019) and introduce conditional non- linear normalizing flows with split coupling. Split couplings ensure invertibility by applying flow
3

---

Under review as a conference paper at ICLR 2020
layer fi only half of the dimensions at time. To compute 5 split the dimensions zD of the
latent variable into halfs, z" = {1..... D/2}
and z 19/2.-.. ......... d) at each invertible layer fs. Our
transformation takes the following form for each dimension z' alternatively from z" or z²?
c(zR,x)
(z = e =a(=zz +bzl x
(4)
1+ (d(zR,x) z g(2 x))²
where, z zzh, Details the forward (generating) operation are Appendix A. To ensure that the generated prior distribution is conditioned on in 4 and in the corresponding forward operation fi- the coefficients {a, c, d. 9} are functions of both the other half of the dimensions condition x (unlike Ziegler & Rush (219)). Finally, due to the expressive power of our conditional non-linear normalizing flows, simple spherical Gaussians base distributions were sufficient.
and the
3.2 VARIATIONAL INFERENCE USING CONDITIONAL NORMALIZING FLOWS BASED PRIORS
Here, derive the ELBO for our regularized CF-VAE with our conditional flow based prior. In case of the standard CVAE with the Gaussian prior, the KL divergence term in the ELBO has simple closed form expression. In case of our conditional flow based prior, we can the change of variables formula to compute the KL divergence. In detail, given the base density p(e[x) and the Jacobian J4 of each layer i of the transformation, the log- likeliihoodd of the latent variable z under the prior can be expressed using the change of variables formula,
log(p= (zlx)) log(p(e(x)) - Σ log([des J4l).
(5)
This change of variables allows us to evaluate the likelihood of latent variable over the base distribution instead the complex conditional prior and to express the KL divergence as,
- (z) -Bgg())sy) log(e<(z)x,,)) + (xl))) log(px(zlx))
(6)
=((qg) + (r log(p(c(x)) +
Σ log( J.l).
where, H(gs) is the entropy of the variational distribution. Therefore, the ELBO can be expressed as,
log(po (elx,) log(po(ylz.x)) H(g)) Bgolelas)) log(p(e(x)) log(Idet J:) (7)
To learn complex conditional priors, we alternately opti- mize both the variational posterior distribution go(z)x, y) and the conditional prior Pe x) in (7). This would al- low the variational posterior ge to match the conditional prior and vice-versa so that the ELBO ( is maximized. However, in practice we observe instabilities during train- ing and posterior collapse. Next, we introduce our novel regularization schemes to deal with both these problems.
9 /x.))
Recognition Network
Decoder
Encoder
Conditional
Normalizing
X., X Xp
Flow
Posterior Regularization for Stability (pR). The en- tropy and the log-Jacobian of the CF-VAE objective (7) are odds with each other. The log-Jacobian favours the contraction of the base density. Therefore, log- log-Jacobian the right of (7 is maximized when the conditional flow maps the base distribution (c <> z in Figure 2) to a low entropy conditional prior and thus low en- tropy variational distribution qo (z)x). x. y). Therefore, in practice we observe instabilities during training. We observe that either the entropy the log-Jacobian term dominates and the data log-likelihood fully or partially ignored. Therefore, we regularize the posterior qs (z|x, y) by fixing the variance to C. This leads to constant entropy term which in turn bounds the maximum possible amount of contraction, thus upper bounding the log- JagoJacnn..n. This encourages our model to concentrate on explaining the data and leads better fit to the target data distribution. Note that, although qa (zlx,
Condition
((01)
Figure 2: CF-VAE. The decoder is regu- larized by removing conditioning (grey arrow) to prevent posterior collapse.
4

---

Under review as a conference paper at ICLR 2020
has fixed variance, this does not significantly effect sample quality as the marginal qa (z|x) can be arbitrarily complex due to our conditional flow prior. Moreover, we observe that the LSTM based decoders employed demonstrate robust performance across a wide range of values C [0.05, 0.25].
Condition Regularization for Posterior Collapse (cR). We observe missing modes when the target conditional data distribution has major mode(s) and one more minor modes (corresponding to rare events). This is because the condition x on the decoder is already enough to model the main mode(s). If the cost of ignoring the minor modes is out-weighed by the cost of encoding a more complex latent distribution reflecting all modes, the minor modes and the latent variables are ignored. We propose a regularization scheme by removing the additional conditioning x on the decoder, when the dataset in question has dominating mode(s). This enabled by our conditional flow prior, which ensures that conditioning information encoded in the latent space and po(ylz) can match po(y)x, (/yxxx21 z). I Leading to simpler factorization, Po (y())x) = (y(y) (oo(y//))p p) (z)x x Jdz. Equivalently, this ensures that the latent variable z cannot be ignored by the CF- VAE and thus must encode useful information. Note that this regularization scheme is only possible due our conditional prior, the unconditional Gaussian prior CVAE would always need to condition the decoder.
The parallel work of Klushyn et al. 2019) also proposes similar regularization scheme. However, we employ this regularization to deal with posterior collapse only in case of distributions with dominant modes. We also provide a more detailed analysis of their proposed prior in Appendix E.
Finally, we discuss the integration of diverse sources of contextual information into the conditional prior Py (zlx) for even richer conditional latent distributions of our regularized CF-VAE
3.3 CONDITIONING PRIORS ON CONTEXTUAL INFORMATION
For prediction tasks, it is often crucial to integrate sources of contextual information e.g. past trajectories environmental information for accurate predictions. As these sources are heterogeneous, we employ source specific networks to extract fixed length vectors from each source.
Past Trajectory. We encode the past trajectories using LSTM to an fixed length vector x,. For efficiency we share the condition encoder between the conditional flow and the CF- decoder.
Environmental Map. We use CNN to encode environmental information to a set of region specific feature vectors. We apply attention conditioned on the past trajectory to extract fixed length conditioning vector X,.. such that X, contains information relevant to the future trajectory.
Interacting Agents. To encode information of interacting traffic participants/agents, we build on Deo Trivedi (2018) and propose fully convolutional social pooling layer. We aggregate information of interacting agents using grid overlayed on the environment. This grid is represented using tensor, where the past trajectory information of traffic participants are aggregated into the tensor indexed corresponding to the grid in the environment. In Deo & Trivedi (2018) past trajectory information is aggregated using a LSTM. We aggregate the past trajectory information into the tensor using x 1 convolutions allows for stable learning and is computationally efficient. Finally, we apply several layers of i x l convolutions to capture interaction aware contextual features Xp of traffic participants in the scene.
Due to the expressive power of our conditional non- non-linear normalizing flows, simple concatenation into a single vector (x,, Xm, x was sufficient to learn powerful conditional priors.
4 EXPERIMENTS
We evaluate our CF-VAE on three popular and highly multi-modal sequence prediction datasets. We begin with description of our evaluation metrics and model architecture.
Evaluation Metrics. In line with prior work (Lee et al., 2017 Bhattacharyya et al. 2018 Pa
jouheshgar Lampert 2018 Deo Trivedi 2019 Bhattacharyya et al. 2019). we use the negative conditional log- likeliood (-CLL) and mean Euclidean distances of the oracle Top n% of N predic- tions. The oracle Top n % metric measures not only the coverage of all modes but also discourages random guessing for a reasonably large value of n (e.g. n 10%). This is because, model can only improve this metric by moving randomly guessed samples from an overestimated mode to the correct modes (detailed analysis in Appendix F).
5

---

Under review as a conference paper at ICLR 2020
Condition
BMS-CVAR [Bhattacharyya et al. 2018
Figure 3: Random samples clustered using k-means. The number of clusters is set manually the number of expected digits. The corresponding priors of our CF-VAE pk on the right. Note, our 64D CF-VAE latent distribution is (approximately) projected to 2D using tSNE and KDE.
Conditional Flow Model Architecture, Our conditional flow prior consists of 16 layers of condi- tional non-linear flows with split coupling. Increasing the number of conditional non-linear flows generally led to "over-fitting" on the training latent distribution.
4.1 MNIST SEQUENCES
The MNIST Sequence dataset (D. De Jong 2016) consists of sequences of handwriting strokes of the MNIST digits. The state-of-the-an approach is the Dest-offMany" Many"-CVAE (Bhattacharyya et al., 2018) with Gaussian prior. We follow the evaluation protocol of Bhattacharyya et al. (2018) and predict the complete stroke given the first ten steps. We also compare with, standard CVAE with uni-modal Gaussian prior; 2. A CVAE with data dependent conditional mixture of Gaussians (MoG) prior; 3. A CF-VAE without any regularization 4. A FA-V without the conditional non non-linear flow layers (CF-VAE-Affine, replaced with affine flows (Lu & Huang 2019 Atanov al. 2019)). We also experiment with a conditional MoG prior (see Appendix D and E). We use the same model architecture (Bhattacharyya et al. 2018 across all baselines.
We report the results in Table We see that our CF- VAE with posterior regularization (pR) performs best. It has performance advantage of over 20% against the state of the art EMS-CVAE. We see that without regularization (pR) (C 0.2) there is a 40% drop in performance, highlighting the effectiveness of our pro- posed regularization scheme. We further illustrate the
Method
-CI.I.
CVAE (Sohn et al. 2015
96.4
BMS-CVAE (Bhatacharyya yaryyyyee et al. 2018
95.6
CVAE + increased cupacity (Ours)
94.5
CVAE + conditional prior (Ours)
88.9
MoG-CVAE. M =3
84.6
modes captured and the learnt multi- modal conditional flow priors in Figure We do not use condition reg- ularization here (cR) as we do not observe posterior collapse. In contrast, the BMS-CVAE is unable fully capture all modes - its predictions are pushed to the mean due to the strong model bias induced by the Gaussian prior. The results improve considerably with the multi-modal MoG prior (M 3 components work best). We also experiment with optimiz- ing the standard CVAE architecture. This improves performance only slightly (after increasing LSTM encoder/decoder units to 256 from 48, increasing the number of layers did not help). Moreover, our experiments with a conditional (MoG) AAE/WAE (Gu et al. 2018) based baseline did not improve performance beyond the standard (MoG) CVAE, because the discriminator based KI. estimate tends to be an underestimate (Rosca et al. 2017). This illustrates that in practice it difficult to map highly multi-modal sequences to a Gaussian prior and highlights the need of data-dependent multi-modal priors. Our VAE still significantly outperforms the MoG-CVAE as normalizing flows are better at learning complex multi-modal distributions (Kingma & Dhariwal 2018). We also that affine conditional flow based priors leads to a drop in performance (77.2 VS 74.9 CLL) illustrating the advantage of our non-linear conditional flows.
CF-VAE regularization (Ours)
104.3
CF-VAE Affine + pR, C= (1.2 (Ours)
77.2
CF-VAE + pR, C 0.2 (Ours)
74.9
Table 1: Evaluation on MNIST Sequences.

---

Under review as a conference paper at ICLR 2020
Method
Visual
Errr
Isec
Fimor
2xec
Error & Jsec
Emror 4sec
CLL.
"Shotgun" Pajouheshaar Lampert 2018
None
0.7
4,5
DESIRE-SI-ITA (Top 10mm ze 2017
RGB
1.2
2.3
5.3
STCNN 10%) (Pajouheshyar Lampert 2018
RGB
1.2
2.1
3.3
4,6
BMS-CVAE (Top 10%) [Shutachary ya et 2018
RGB
0.8
1.7
4.6
126.6
MoG-CVAF, M (Top 10%)
None
0.8
1.7
3.9
86.1
CF-VAE regularization (Ouns, Top 10%)
None
0.9
1.9
4.7
96.2
CF-VAE pR. C 0.2 (Ours, 10%)
None
0.7
3.6
84.6
CF CI-VAE + pR. C 0.2 (Ours, Top 10%)
RGB
0.7
1.5
3.5
84.1
Table 2: Five fold cross validation on the Stanford Drone dataset. Euclidean error at (1/5) resolution.
4.2 STANFORD DRONE
Sampled Predictions
Latent Prior
Sampled Predictions
Latent Prior
Sampled Predictions
Latent Prior
Figure 4: Randomly sampled predictions of our CF-VAE + pR model on the Stanford Drone. We observe that prediction are highly multi-modal and is reflected by the Conditional Flow Priors. Note, our 64D CF- VAE latent distribution is (approximatly) projected to 2D using iSNE and KDE.
Figure 5: Comparison of our CF-VAE + pk (Red) and the "Shoutgun" baseline (Yellow) of (Pa. jouheshgar & Lampert 2018). Groundtruth (Blue). Initial conditioning trajectory in white. Our CF-VAE not only learns to capture the correct modes but also generates more fine-grained predictions.
The Stanford Drone dataset (Robicquet et al. 2016) consists of multi-model trajectories traffic
participant e.g. pedestrians, bicyclists, cars captured from a drone. Prior works follow two different evaluation protocols, 1. (Lee et al. 2017 Bhattacharyya et al. [2018] Pajoubeshgar & Lampert 2018) use fold cross validation, 2. (Robicquet al. 2016; Sadeghian et 2018] 2019 9 Deo & Trivedi 2019) use single split. We evaluate using the first protocol in Table 2 and the second in Table
Additionally, Pajouheshgar & Lampert (0088) suggest a "Shotgun" baseline. This baseline extrapolates the trajectory from the last known position and orientation in 10 different ways 5 orientations: (0%, + +15% and velocities: None exponentially weighted over the past with coefficients (0, 0.3, 0.7, 1.0). This base- line obtains results at par with the state-of-the-a be- cause it a good template which covers the most likely possible futures (modes) for traffic participant motion in this dataset. We report the results using 5 fold cross validation in Table 2 compare to a mixture Gaussians prior (Appendix D). We use the same model architecture as in Bhattacharyya et al. (2018) and CNN encoder with attention extract features from the last observed RGB image (Appendix C). These visual features serve as additional conditioning (x..) Conditional Flow model. We see that VR- model with RGB input and posterior
Method
mADE 1
mFDE
SocialGAAN ( Gappla et al
2018
27.2
41.4
GANiZiaa
SaPhie (adeemian man er al 2019
16.2
29.3
Goal Premtemrr 1u8 & Trved 2019
15.7
28.1
pR. (Ours)
Table Evaluation on the Stanford Drone dataset single split (see also Table 2).
We additionally
7

---

Under review as a conference paper at ICLR 2020
by regularization (pR) performs best outperforming the state-of-art "Shotgun" and over 20% (Error @ 4sec). We see that our conditional flows are able to utilize visual scene (RGB) information to improve performance (3.5 vs 3.6 Error @ 4sec). We also see that the MoG-CVAE and our CF-VAE pR outperforms the BMS-CVAF, even without visual scene information. This again reinforces our claim that the standard Gaussian prior induces strong model bias and data dependent multi-modal priors are needed for best performance. The performance advantage of CF-VAE over the MoG-CVAE again illustrates the advantage of normalizing flows at learning complex conditional multi-modal distributions. The performance advantage over the "Shotgun" baseline shows that our CF-VAE + pR not only learns to capture the correct modes but also generates more fine-grained predictions. The qualitative examples in Figure 5] 5hsows that our CF-VAE is better able to capture complex trajectories with sharp turns.
We report results using the single train/test split of (Robicquet et al. 2016 Sadeghian et al., 2018 2019; Deo & Trivedi 2019) in Table 3] We use the minimum Average Displacement Error (mADE) and minimum Final Displacement Error (mFDE) metrics as in (Deo & Trivedi 2019). The minimum is over as set of predictions of size N. Although this metric is less robust to random guessing compared to the Top 12% metric, it avoids rewarding random guessing for a small enough value of N. We choose N = 20 as in (Deo & Trivedi (2019). Similar to the results with 5 fold cross validation, we observe 20% improvement over the state-of-the-art.
4.3 HIGHD
The HighD dataset (Krajewski et al. 2018) consists of vehicle trajectories recorded using drone over highways. In contrast to other vehicle trajectory datasets e.g. NGSIM it contains minimal false positive trajectory collisions or physically improvable velocities.
The HighD) dataset is challenging because
lane changes or interactions are rare ~ 10% of all trajectories. The distribution of future trajectories contain single main mode (lin- ear continuations) along with several minor modes. Thus, approaches which predict single mean trajectory (targeting the main mode) challenging to outperform. In Ta- ble 4 we see that the simple Feed Forward (FF) model performs well and the Graph Convolutional GAT model Diehl et al. (2019), which captures interactions, only narrowly outper- forms the FF model. This dataset is challenging for CVAE based models as they frequently suffer from posterior collapse when a single mode dominates. This is clearly observed with our CVAE baseline in Table 4 To prevent posterior collapse, we use the cyclic KL annealing scheme proposed in Liu et al. (2019) (using a MoG prior did not help). This already leads to significant improvement over the deterministic FF and GAT baselines. We also observe posterior collapse with our CF-VAE model. Therefore, we regularize by removing additional conditioning (cR). Our CF-VAE- {pR,cR} with condition regularization significantly outperforms the CF-VAE pR and CVAE baselines (with cyclic KL annealing), demonstrating the effectiveness of our condition regularization scheme (cR) in preventing posterior collapse. The addition of contextual information of interacting traffic partici- pants using our convolutional social pooling network with 1x1 I convolutions significantly improves performance (also see Appendix G), demonstrating the effectiveness of our conditional normalizing flow based priors.
Method
Context
ADE 1
FDE J
.CLL
Veloxity
FF Dichl al 2019
None
0.45
1.09
x
GAT (Dichl or 2019
Yes
0.47
1.04
x
TVAL
None
0.45
0.96
5.32
CVAE Cyclic (Top 10%)
None
4.80
CF-VAE pR, (Ours, Top 10%)
Nons
0.14
0.91
1.71
CF-VAE (pR.cR). (Ours, Top 10%)
None
0.30
0.57
3.64
CF-VAE+ (pR.cR). (Ours, Top 10%)
342
Table 4: Evaluation on the HighD dataset.
CONCLUSION
In this work, we presented the first variational model for learning multi-modal conditional data distributions with Conditional Flow based priors - the Conditional Flow Variational Autoencoder (CF-VAE). Furthermore, we propose two novel regularization techniques posterior regularization (pR) and condition regularization (cR) which stabilizes training solutions and prevents posterior collapse leading to better fit to the target distribution. This techniques lead to better match to the target distribution. Our experiments on diverse sequence prediction datasets show that our CF-VAE achieves state-of-the-e results across different performance metrics.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In CVPR, 2016.
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf S Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In ICLR, 2019.
Andrei Atanov, Alexandra Volokhova, Arsenii Ashukha, Ivan Sosnovik, and Dmitry Vetrov. Semi- conditional normalizing flows for semi-supervised learning. In ICML. Workshop, 2019.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In ICLR, 2018.
Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv preprint arXiv: 1et::4111.76000 2014.
Jens Behrmann, David Duvenaud, and JömnHenrik Jacobsen. Invertible residual networks. In ICML,, 2019.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. In UAI, 2018.
Apratim Bhattacharyya, Bernt Schiele, and Mario Fritz. Accurate and diverse sampling of sequences based on best of many sample objective. CVPR, 2018.
Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Bayesian prediction future street scenes using synthetic likelihoods. In ICLR, 2019.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. CONLL, 2016.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In ICLR, 2017.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In NIPS, 2015.
Edwin De Jong. The mnist sequence dataset. https: http///wwwinddeeeeeeee aong.giihub.... 1o/ blog/ sew.isst--seueeee-daaaaa. 2016. Accessed: 2019-07-07.
Nachiket Deo and Mohan M Trivedi. Convolutional social pooling for vehicle trajectory prediction. In CVPR Workshop, 2018
Nachiket Deo and Mohan M Trivedi. Scene induced multi-modal trajectory forecasting via planning. In ICRA Workshop, 2019.
Frederik Diehl, Thomas Brunner, Michael Truong Le, and Alois Knoll. Graph neural networks for modelling traffic participant interaction. In ITSC, 2019.
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. AISTATS, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. In ICLR, 2015.
Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio. Density estimation using real nvp. In ICLR, 2017.
Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Eric P Xing. Nonparametric variational auto-encoders for hierarchical representation learning. In ICCV, 2017.
Xiaodong Gu, Kyunghyun Cho, Jung- Woo Ha, and Sunghun Kim. Dialogwae: Multimodal response generation with conditional wasserstein auto-encoder. arXiv preprint arXiv: /2352, 2018.

---

Under review as a conference paper at ICLR 2020
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. In ICLR, 2017.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In CVPR, 2018.
Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review E, 51, 1995.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Matthew Hoffman and Matthew Johnson. Elbo surgery: yet another way carve up the variational evidence lower bound. In NPSS Workshop, 2016.
GC Holmes. The use of hyperbolic cosines in solving cubic polynomials. The Mathematical Gazette.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In ICML, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NeurIPS, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In NIPS, 2016.
Alexej Klushyn, Nutan Chen, Botond Cseke, Justin Bayer, and Patrick van der Smagt. Increasing the generalisaton capacity conditional vacs. In ICANN, 2019.
Robert Krajewski, Julian Bock, Laurent Kloeker, and Lutz Eckstein. The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems. In ITSC, 2018.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv preprint arXiv:1903.01434, 2019.
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In CVPR, 2017.
Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin, al. Cyclical annealing schedule: A simple approach to mitigating kl vanishing, In NAACL, 2019.
You Lu and Bert Huang. Structured output learning with conditional generative flows. In ICML Workshop, 2019.
Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autoencoders. In ICLR, 2017.
Ehsan Pajouheshgar and Christoph H Lampert. Back to square one: probabilistic trajectory forecasting without bells and whistles. In NeurlPs Workshop. 2018.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. In ICLR, 2019.
Danilo Jimenez. Rezende and Shakir Mohamed, Variational inference with normalizing flows. In ICMI., 2015.
Nicholas Rhinehart, Kris M Kitani, and Paul Vernaza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In ECCV, 2018.
10

---

Under review as a conference paper at ICLR 2020
Alexandre Robiequet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In ECCV, 2016.
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. arXiv preprint arXiv::706.. 170.:047777777 2017.
Amir Sadeghian, Ferdinand Legros, Maxime Voisin, Ricky Vesel, Alexandre Alahi, and Silvio Savarese. Car-net: Clairvoyant attentive recurrent network. In ECCV, 2018.
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In 2019.
Kibyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. NIPS, 2015.
Esteban Tabak, Eric Vanden-Eijnden, al. Density estimation by dual ascent of the log-likelihook In Communications in Mathematical Sciences, volume 8, 2010.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv:1711.015: 2017.
Jakub M Tomczak and Max Welling. Improving variational auto-encoders using householder flow. In NIPS Workshop, 2016.
Jakub M Tomezak and Max Welling. Vae with a vampprior. In AISTATS, 2018.
Liwei Wang, Alexander Schwing, and Svetlana Lazebnik. Diverse and accurate image description using variational auto-encoder with additive gaussian encoding space. In Advances in Neural Information Processing Systems, pp. 5756-57666 2017.
Prince Zizhuang Wang and William Yang Wang. Riemannian normalizing flow on variational wasserstein autoencoder for text modeling. In NAACL, 2019.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg- Berg-Kirkpatrice Improved variational autoencoders for text modeling using dilated convolutions. In ICML, 2017.
Shengjia Zhao, Jiaming Song. and Stefano Ermon. Infovae: Information maximizing variational autoencoders. In arXiv preprint arXiv:1706.0226. 2017.
Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In CVPR, 2019.
Zachary M Ziegler and Alexander M Rush. Latent normalizing flows for discrete sequences. In ICML, 2019.

---

Under review as a conference paper at ICLR 2020
APPENDIX A. CONDITIONAL NON-LINEAR NORMALIZING FLOWS
In Subsection 3.1 of the main paper, we describe the inverse operation f:² of our non-linear conditional normalizing flows. Here, we describe the forward operation. Note that while the forward operation is necessary to compute the likelihood (3) (in the main paper) during training, the forward operation is necessary to sample from the latent prior distribution of CF-VAE. The forward operation consists of solving for the roots of the following equation (more details in (Ziegler & Rush 2019
- -od + ((z) aa) 2dgb)(e2)² (2dg(z) a) -bgg² 1))e ((z) a)(s² +)
This equation has one real root which can be found analytically (Holmes). As mentioned in the main paper, note that the coefficients {a, b, c, d, g) also functions of the condition (unlike (Ziegler & Rush 2019)).
APPENDIX B. ADDITIONAL EVALUATION OF CONDITIONAL NON-LINEAR FLOws
Given in,
p(ylx)
Cond Affine Flow
Our Cond NL Flow
Figure 6: Comparison between conditional affine flows of (Atanov et al. 2019 Lu & Huang 2019
and our conditional non-linear (Cond NL) flows. We see that the conditional affine flows cannot fully capture multi-modal distributions ("tails" between modes), while our conditional non-linear flows does not have distinctive "tails".
We compare conditional affine flows of (Atanov et al. 2019 Lu Huang 2019) and our conditional non-linear (Cond NL) flows in Figure 6 and Figure 7 We plot the conditional distribution p(ylx) and the corresponding condition x in the second and first columns. We use and 16 layers flow in case of the densities in Figure 6 and Figure 7 respectively. We see that the estimated density by the conditional affine flows of (Atanov et al. 2019: Lu & Huang 2019) contains distinctive "tails" in case of Figure 6 and discontinuities in of Figure In comparison conditional
12

---

Under review as a conference paper at ICLR 2020
non-linear flows does not have distinctive "tails" discontinuities and is able complex capture the multi-modal distributions better. Note, the "ring" distributions in Figure cannot be well captured by more traditional methods like Mixture of Gaussians. We see in Figure 8 that even with 64 mixture components, the learnt density is not smooth in comparison to our conditional non-linear flows. This again demonstrates the advantage of conditional non-linear flows.
Given in,
p(yx))
Cond Affine Flow
Our Cond NL Flow
Figure 7: Comparison between conditional affine flows of (Atanov al. 2019 Lu & Huang 2019
and our conditional non -inearn (Cond NL) flows. We see that the conditional affine flows cannot fully capture "ring" like conditional distributions (note the discontinuity the top), while our conditional non-linear flows does not have such discontinuities.
p(ylx)
MoG.
MoG.
MoG, 32
MoG, 64
Our Cond Flow
Figure 8: Comparison between our conditional non-linear (Cond NL) flows and a Mixture of Gaussians (MoG) model. We see that even with 64 mixture components, the learnt density is not smooth in comparison to our conditional non-linear flows.
APPENDIX C. ADDITIONAL DETAILS OF OUR MODEL ARCHITECTURES
Here, we provide details of the model architectures used across the three datasets used in the main paper.
MNIST Sequences. We use the same model architecture as in Bhattacharyya et al. (018). The LSTM condition encoder the input sequence the LSTM recognition network To and the decoder LSTM network has 48 hidden neurons each. Also as in Bhattacharyya et al. (2018), we use 64 dimensional latent space.
Stanford Drone.
Again, we use the same model architecture as in Bhattacharyya et al. (2018) except for the CNN encoder. The LSTT condition encoder on the input sequence and the decoder LSTM network has 64 hidden neurons each. The LSTM recognition network qo has 128 hidden neurons. Also as in BBhattacharyya et al. (011)), we use a 64 dimensional latent space, Our CNN encoder has 6 convolutional layers of size 32, 64, 128, 256, 512 and 512. We predict the attention weights on the final feature vectors using the encoding of the LSTM condition encoder. The attention weighted feature vectors passed through final fully connected layer obtain the final CNN encoding. Furthermore, we found it helpful to additionally encode the past trajectory as an image (as in (Pajouheshgar & Lampert 2018) provide this additional channel to the CNN encoder.
13

---

Under review as a conference paper at ICLR 2020
HighD. We the same model architecture with both the CVAE and CF-VAE models. As in the Stanford drone dataset, we use LSTM condition encoder on the input sequence x and the decoder LSTM network with 64 hidden neurons each and the LSTM recognition network go with 128 hidden neurons. The contextual information of interacting traffic participants are encoded into spatial grid tensor of size 13x3 (see Section 3.2 of the main paper). We use CNN with 5 layers of sizes 64, 128, 256, 256 and 256 to extract contextual features.
APPENDIX D. DETAILS OF THE MIXTURE OF GAUSSIANS (MOG) BASELINE
In the main paper, we include results on the MNIST Sequence and Stanford Drone dataset with a Mixture of Gaussians (MoG) prior. In detail, instead of normalizing flow, we the prior MoG form,
M
pe(2|x) Eplcik)i(aamso.oo
We use simple feed forward neural network that takes in the condition x (see Section 3.4 of the main paper) and predicts the parameters of the MoG, & = (C1. d1. -MMMMM a }- Note, to ensure reasonable number of parameters, consider spherical Gaussians. Similar to (5) in the main paper, the ELBO can be expressed as,
lyg(p((y(x)) 2 (xlx,) (yg(:o(y1z, x)) + H(gs) + B log(p: (z)x)).
(10)
Note that we fix the entropy of the posterior distribution do for stability
APPENDIX E. ADDITIONAL EVALUATION ON THE MNIST SEQUENCE DATASET
Here, we perform comprehensive evaluation using the MoG prior with varying mixture components, a CVAE with unconditional non-linear flow based prior (NL-CVAE),, our CF-VAE with Volume- preserving constant Jacobian conditional NICE flows based on Dinh al. (2015), CVAE with the conditional VampPrior (CDV) of (Klushyn et al. 2019), our CF-VAL with varying
C' [0.05, 0.25] of our posterior regularization (pR) scheme and finally analyze the effect of our posterior regularization (pR) scheme in detail. We report the results in Table 5
Method
-CLL
NL-CVAE
107.6+1.2
CVAE (M = 1) Sohn et al. 2015
96.4+0022
MoG-CVAE, M 2
85.3+10.4
MoG-CVAE, M
84.6=0.5
MoG-CVAE, M =
85.7=0.4
MoG-CVAE, M =
86.3++0.6
CDV (Klushyn et al. 2019) M = 12
99.4+00.7
CF-VAE NICE (Ours)
78.9+
CE-VAE + pR, C = 0.05, (Ours)
75.9+
CF-VAE pR, C 0.10. (Ours)
75.4=10.3
CF-VAE + +pR, C = =0.15, (Ours)
75.1
CF-VAE +R,, C -0.20, (Ours)
74.9
CA- +pR, C = 0.25, (Ours)
75.8
Table Evaluation on MNIST Sequences (CL.L: lower better).
MoG Prior. As mentioned in the main paper, CVAE. This again reinforces our claim that the standard Gaussian prior induces a strong model bias. We that using M 3 components with the variance of the posterior distribution fixed
that the MoG-CVAE outperforms the plain
14

---

Under review as a conference paper at ICLR 2020
Condition
CDV Modes
CDV Prior
Figure 9: Random samples using the CDV Prior of (Klushyn al. 2019) clustered using k-means. The number of clusters is set manually to the number of expected digits, The CDV Prior latent distribution on the right. Note, the 64D latent distribution is (approximately projected 2D using tSNE and KDE. In comparison to the samples and latent spaces of our CF-VAE (Figure 3 see that the latent spaces are more simplistic and samples are of poorer quality.
to - leads to the best performance. This is expected as 3 is the most frequent number of possible strokes in the MNIST Sequence dataset. Also note that the results with the MoG prior are learn the variance of the prior (see the section above). Finally, our VAE + pR still significantly outperforms the MoG-CVAE (74.9 vs 84.6). This is expected normalizing flows more powerful compared to MoG learning complex multi-modal distributions (Kingma Dhariwal 2018) (also see Figure 8).
also relatively robust across C [0.05, 0.25]
NL-CVAE. We also that using unconditional non-linear flow based prior actually harms performance (107.6 vs 96.4). This is because the latent distribution is highly dependent upon the condition. Therefore, without conditioning information the non-linear conditional flow learns a global representation of the latent space which leads to out-of distr--distiiuution samples at prediction time.
CF-VAE with conditional NICE flows (Dinh al., 2015). We have added results with the volume preserving NICE flows in Table 5) We observe that even without our posterior regularization scheme (pR) volume preserving NICE flows (Dinh al., 2015) performs well because of the constant Jacobian term. However, our conditional non- flows with posterior regularization still perform significantly better (78.9 vs 74.9 -CLL). This because of the additional expressive power of our conditional non-linear flows combined with the stability offered by our posterior regularization scheme.
Comparison to Klushyn et al. (2019). We also perform additional experiments with the conditional VampPrior (CDV) of Klushyn et al. (2019) using M = 12 components. Using more components makes training/inference significantly slower in comparison to plain CVAEs, Mog-CVAE (M = 3) or our FAE. Furthermore, with M = 12 components we observe that it is outperformed by the simpler MoG-CVAE. This is because the mean and variance parameters of the (M = 12) components are obtained using the recognition network Jo- The recognition network 4s has to learn to both reconstruct the data and maintain latent space representative of full conditional data distribution p(y|x). These objectives are at odds with each other. In practice, we find that this leads to simplistic latent spaces along with lower overall data log- likelihooo in comparison with our CF-VAE (Figure 3).
This can be seen in the samples and corresponding latent spaces in Figure 9
Hypee-paramettrr analysis of our posterior regularization scheme (pR). We provide additional analysis of our posterior regularization scheme in Table 5 We observe that our CF- VAE is relatively robust across C [0.05, 025), with only small variance in performance. This is because our posterior regularization scheme encourages our CF-VAE to focus on explaining the data well. We explain this further in the following paragraph.
Analysis of our posterior regularization scheme (pR). We provide additional analysis of our posterior regularization scheme (pR) in Figure 10. We show each term of our objective (7) in Figure 10 First, we see that with our posterior regularization scheme, our CF-VAE focuses explaining the data well - the data log- ligeeeilloood is best with our posterior regularization (pR) scheme Figure 10a. with C 0.2 having a advantage that without our posterior regularization scheme the Jacobian term dominates while entropy term decreases (Figure 10h Figure 10dp the contraction of the hase density is favoured. Interestingly,
C {0.05, 0.1). Furthermore, see
15

---

Under review as a conference paper at ICLR 2020
1ag (elea))) log (poppyly, x))
-3-
-5"
withaut pR
H((90)
C-0.95
C-0 10
C-0.05
-10
C 0.20
0-1.1
5
C-0.20
50000
100000
150000
200000
50000
100000 150000 200000
Iterations
Itcrations
(a)
log- log-iikelihod.
(b) Entropy the posterior.
(age))))) log (p(()))
log (og(e J+l) J:l)
0.20
-0.01
without
0.15
C=0.05
0.05
without
C=0.10
C-0.05
0.10
C-0.20
C=0.10
<[]]
-0.06
C-0.20
0.05
0
50000
100000 150000 200000
50000
100000 150000 200000
Iterations
Iterations
(c) Likelihood under distribution.
(d) Log determinant the Jacobian.
Figure 10: Analysis of all four terms of our CF-VAE objective 7 training time, with (C (0.05, 0.10, 0.20)) and without our posterior regularization (pR) scheme. We observe better data wigllielihhooss and stable training with our posterior regularization (pR) scheme. Without pR, observe that the Jacobian term dominates at the cost of data log-likelihood.
the likelihood under the prior Figure 10eli similar across methods - with our posterior regularization providing additional stability. We also experimented with re- seiihiht these terms (although its no longer a valid lower bound on the true data log-likelihood), This leads to the opposite behaviour - the entropy term dominates over the Jacobian term at the cost of the data log-likelihood, On the other hand, we observe that all terms of our objective are stable with our posterior regularization scheme, illustrating the advantage of our posterior regularization scheme.
APPENDIX F. EVALUATION OF THE ROBUSTNESS THE TOP N% METRIC
We two simpler uniform "Shotgun" baselines to study the robustness of the Top n% metric against random guessing. In particular, we consider the "Shotgun" Schotgun--u900 and "Shotgun" "ultgunn"u113"" u-13 baselines which: given budget of N predictions, uniformly distributes the predictions between (-90- 90%) and (-135* 135%, 135%) respectively of the original orientation and using the velocity of the last time-step. In Table we compare the Top I (best guess) to Top 10% metric with N= 50, 100, 500 predictions.
We see that in case of both the "Shotgun" u9900 and "Shotgun" "irigun""ul135" u baselines, the Top 1 (best guess) metric improves with increasing number of guesses. This effect even more pronounced in case of the "Shotgun"-ull" = baseline as the random guesses are distributed over larger spatial range. In contrast, the Top 10% metric remains remarkably stable. This is because, in order to improve the Top 10% metric, random guessing not enough the predictions have to be the correct modes. In other words, the only way to improve the Top 10% metric is move random predictions to any of the correct modes.
16

---

Under review as a conference paper at ICLR 2020
Method
K
Error @ ssec
Error 2sec
Error @ 3sec
Error 4sec
Top
(Best Guess)
"Shotgun"" "arigun""u900" -
50
0.9
1.9
3.1
4.4
"Shotgun" -990
100
0.9
1.9
3.0
4.3
"Shotgun" -990
500
0.9
1.9
3.0
4.3
Top 10%
"Shotgun""uue -
50
1.2
2.5
3.9
5.4
Shottunn-u900
100
1.2
2.5
3.9
5.4
"Shotgun"-uu -u90
500
1.2
2.5
3,9
5,4
Top
(Best Guess)
"Shotgun"-u135"
50
0.9
2.0
3.1
4.5
"Shotgun" "ootggnn"-u135" -u135
100
0.9
1.9
3.0
4.3
"Shotgun" -u135
500
0.9
1.9
3.0
4.2
Top 10%
Schotgun"ul133
50
1,4
2.9
4.5
6.2
"Shotgun" -u135
100
1.4
2.9
4.5
6.2
"Shotgum" "ootgum"-u135" -u133
500
1.4
2.9
4.5
6.2
Table 6: Five fold cross validation on the Stanford Drone dataset. Euclidean error at (2/5) resolution.
APPENDIX G. QUALITATIVE EXAMPLES ON THE HIGHD DATASET
Groundrrath samples
samples
Our CP-VAE (pR,ck) samples
Figure 11: Predictions on the HighD dataset. Left: 128 random samples from the HighD set (in yellow). Middle: CVAE predictions (5 samples per test set example). Right: Our CV-VAE + (pR,cR) predictions (5 samples per test set example). While the predictions by the CVAE are linear continuations, our CF-VAE sample predictions are much more diverse and cover events like lane changes e.g. top most sample track from the test set.
We show qualitative examples the HighD dataset in Figure In the left of Figure 11 we show 128 random samples from the HighD test set. In the middle we show predictions on these samples by the CVAE (with cyclic KL annealing (Liu et al.) 2019)). We see that even with cyclic KL annealing, we observe posterior collapse. All samples have been pushed towards the mean and the variance in the 5 samples per test set example is minimal. E.g. note the top most sample track from the test in Figure (left) All CVAE sample predictions are a linear continuation of the trajectory (continuing on the same lane), while there is fact turn (change of lanes). In contrast, our CF-VAE + (pR,cR) sample predictions are much more diverse and cover such eventualities. This also shows that our CF-VAE (pR.cR} does not suffer from such posterior variable collapse.
17


Reviews:
[{"review_id": "SyxVJJ0iKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a combination of conditional VAE wtih normalising flows priors and posterior regularisation strategies to capture the diversity of multi-modal trajectories of complex motion patterns. The paper argues that more flexible priors over the latent space can provide posteriors that more closely resemble the trajectories observed in the training data. To this end, the paper presents a derivation of the evidence lower bound for VAEs with normalising flows and discusses the effect of fixing the variance of the posterior to reduce instability during training. Additionally, it shows that conditioning the regularisation on whether or not the dataset contains a dominating mode leads to more diversity and captures minor modes more effectively. Experiments are reported on sequence datasets of handwritten digits, and two datasets with trajectories of vehicles in traffic. \n\nA central point the paper makes is the importance of prior distributions for the latent space in VAEs such that it can capture diverse modes of trajectories. It is well known that more flexible priors such as MoG lead to better generative power as shown in Tomczak and Welling, 2018. The paper focuses on an extension of the work by Ziegler and Rush, 2019 which proposes normalising flows as priors to capture sequences, conditioned on the initial part of the trajectory. This extension is relatively simple, but does address the specifics of the problem well. \n\nIn general, the paper is well written and clear. The main innovation, in my opinion, is the combination of several ideas applied to the problem of sequence prediction. While none of the ideas, in isolation, are significantly new, the combination can be useful to this particular problem. However, I would like feedback from the authors on the following two main points below which are the main weaknesses of the paper:\n\n1) Posterior regularisation: The posterior regularisation strategies, while intuitive, are very ad-hoc and somewhat contrary to the Bayesian framework. It is difficult to see how the variance in pR and the \"dominant mode\" detector in cR can be estimated automatically. Within a Bayesian framework it would be much more natural to place a prior distribution over the variance and marginalise it out within the variational inference procedure. For the other regularisation (cR), how is the dominant mode detected? \n\n2) Experiments: A major concern reported throughout the paper is the instability of training and the risk for overfitting. I do not think the experiments demonstrate how stable and robust the method is to different initialisations, seeds, training data shuffles, etc. I strongly suggest the authors to run cross validation experiments and report the mean and standard deviation for all methods being compared. Also, how sensitive are the results to different values of C? How to decide whether to use cR or not when don't have access to the ground truth?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BygWcVYatB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper demonstrates how normalising flows can be conditioned. The method is then demonstrated on a set of sequential experiments which show improvements over the considered base lines.\n\nI recommend rejection of the paper, but I can see me changing that assessment if certain improvements are made. The central points are:\n- the paper has errors,\n- the paper does not respect some related work and has been published previously in parts,\n- the paper has a claim that is unsupported in my view,\n- the paper is overcrowded with annoying marketing language; the word \"novel\" appears 16 times according to my pdf viewer.\n\nIn general I like the idea, and the presentation seems solid to a large degree. However, the above points are a show stopper for me personally.\n\nFor one, the statements \n\n- p(y|x) = p(y|x, z) p(z | x) and\n- p(y|x) = p(y|z) p(z|x),\n\nare problematic. I would like the authors to clarify how they arrive at these.\n\nThe paper starts with the claim that \"prior work [...] imposes a uni-modal standard Gaussian prior on the lagent variables\". This is just wrong. The whole literature of stochastic recurrent models does not do this. See  [1, 2] for starting points. Since the authors place their work in the setup of sequential prediction, this is what has to be respected.\n\nFurther, the authors do not seem to be aware of a recently published work [3] that adresses *exactly* this problem. To quote from their abstract: \"To this end, we modify the latent variable model by defining the likelihood as a function of the latent vari- able only and [sic] introduce an expressive multimodal prior to enable the model for capturing semantically meaningful features of the data.\"\n\nI have two more questions with respect to the proposed regularisations.\n\nFirst, I would ask the authors to comment on the relationship of cR and the method proposed in [3]. To me, it appears as if cR is not novel, but has instead been proposed in [3] previously.\n\nSecond, pR fixes the variance of q. The authors claim that the normalising flow of the conditional can undo this fixing by adequately scaling the prior. Hence, so the claim, the expressivity of the model is not reduced.\n\nThis prohibits the posteriors of two distinct data points to share the same mean but not share the same variance. \n\nI request the authors to make a more formal analysis of this, as I do am not convinced how the expressivity of the model is maintained and what influence this has on the ELBO.\n\n\n\nReferences\n[1] Bayer, Justin, and Christian Osendorfer. \"Learning stochastic recurrent networks.\" arXiv preprint arXiv:1411.7610 (2014).\n[2] Chung, Junyoung, et al. \"A recurrent latent variable model for sequential data.\" Advances in neural information processing systems. 2015.\n[3] Klushyn, Alexej, et al. \"Increasing the Generalisaton Capacity of Conditional VAEs.\" International Conference on Artificial Neural Networks. Springer, Cham, 2019.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gBElBTtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work proposes a method to improve conditional VAE with a learnable prior distribution using normalizing flow. The authors also design two regularization methods for the CF-VAE to improve training stability and avoid posterior collapse. The paper is clearly motivated and easy to follow. Experiment results on MNIST, Stanford Drone and HighD datasets show the proposed that the model achieves better results than previous state-of-the-art models by significant margins.\n\nHowever, the reviewer has the following comments on improving the paper:\n\nThe motivation of the conditional normalizing flow design could be made more clear. The posterior regularization originates from the problem that the log Jacobian term encourages contraction of the base distribution. The log Jacobian term would be zero and would not encourage the contraction of the base distribution if the normalizing flow was volume-preserving, like NICE (http://proceedings.mlr.press/v37/rezende15.pdf, https://arxiv.org/pdf/1410.8516.pdf), which could be to convert into a conditional normalizing flow. On the MNIST results, the CF-VAE model with the proposed conditional normalizing flow even has worse performance than the affine flow model without the regularization. Therefore, clarifying the motivation behind this design choice is important.\n\nThe work claims the two regularization methods are used to avoid a low-entropy prior and posterior collapse. But the claims are not fully substantiated in the experimental results. It would be better if the paper explicitly compares the CF-VAE models with and without regularizations in terms of the entropy of prior distribution and KL divergence.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The novelty of the proposed work is a very weak factor, the idea has been explored in various forms in previous work.
Decision: Reject