IN SEQUENTIAL LATENT-VARIABLE MODELS
Year: 2021

Abstract:
Amortised inference enables scalable learning of sequential latent-variable models (LVMs) with the evidence lower bound (ELBO). In this setting, variational posteriors are often only partially conditioned. While the true posteriors depend, e. g., on the entire sequence of observations, approximate posteriors are only informed by past observations. This mimics the Bayesian filter—a mixture of smoothing posteriors. Yet, we show that the ELBO objective forces partially-conditioned amortised posteriors to approximate products of smoothing posteriors instead. Consequently, the learned generative model is compromised. We demonstrate these theoretical findings in three scenarios: traffic flow, handwritten digits, and aerial vehicle dynamics. Using fully-conditioned approximate posteriors, performance improves in terms of generative modelling and multi-step prediction.

heading: 1 INTRODUCTION
text: Variational inference has paved the way towards learning deep latent-variable models (LVMs): maximising the evidence lower bound (ELBO) approximates maximum likelihood learning (Jordan et al., 1999; Beal, 2003; MacKay, 2003). An efficient variant is amortised variational inference where the approximate posteriors are represented by a deep neural network, the inference network (Hinton et al., 1995). It produces the parameters of the variational distribution for each observation by a single forward pass—in contrast to classical variational inference with a full optimisation process per sample. The framework of variational auto-encoders (Kingma & Welling, 2014; Rezende et al., 2014) adds the reparameterisation trick for low-variance gradient estimates of the inference network. Learning of deep generative models is hence both tractable and flexible, as all its parts can be represented by neural networks and fit by stochastic gradient descent.
The quality of the solution is largely determined by how well the true posterior is approximated: the gap between the ELBO and the log marginal likelihood is the KL divergence from the approximate to the true posterior. Recent works have proposed ways of closing this gap, suggesting tighter alternatives to the ELBO (Burda et al., 2016; Mnih & Rezende, 2016) or more expressive variational posteriors (Rezende & Mohamed, 2015; Mescheder et al., 2017). Cremer et al. (2018) provide an analysis of this gap, splitting it in two. The approximation gap is caused by restricting the approximate posterior to a specific parametric form, the variational family. The amortisation gap comes from the inference network failing to produce the optimal parameters within the family.
In this work, we address a previously unstudied aspect of amortised posterior approximations: successful approximate posterior design is not merely about picking a parametric form of a probability density. It is also about carefully deciding which conditions to feed into the inference network. This is a particular problem in sequential LVMs, where it is common practice to feed only a restricted set of inputs into the inference network: not conditioning on latent variables from other time steps can vastly improve efficiency (Bayer & Osendorfer, 2014; Lai et al., 2019). Further, leaving out
∗Equal contribution.
p(z | C, C̄ = 0) p(z | C, C̄ = 1) p(z | C)
ν(z | C) ≈ p(z | C)
ω(z | C)
Figure 1: Illustration of the effect of partial conditioning. Consider a latent-variable model
p
(
C,C
∣∣ z) × p(z), with binary C and arbitrary C. We omit C from the amortised approximate
posterior q(z | C). Shown are two cases of separated (left) and overlapping (right) Gaussian true posteriors. Top row: the full posteriors p ( z ∣∣ C,C = 0) and p(z ∣∣ C,C = 1) as well as their average, the marginal posterior p(z | C). Middle row: Variational Gaussian approximation ν(z | C) to the marginal posterior, which was obtained by stochastic gradient descent on the reverse, modeseeking KL-divergence (Hoffman et al., 2013). Bottom row: The optimal ω(z | C) obtained by optimising the ELBO with a partially-conditioned amortised approximate posterior w. r. t. q. It is located far away from the modes, sharply peaked and shares little mass with the true full posteriors, the marginal posterior as well as the approximate marginal posterior.
observations from the future structurally mimics the Bayesian filter, useful for state estimation in the control of partially observable systems (Karl et al., 2017b; Hafner et al., 2019; Lee et al., 2020).
We analyse the emerging conditioning gap and the resulting suboptimality for sequential amortised inference models. If the variational posterior is only partially conditioned, all those posteriors equal w. r. t. included conditions but different on the excluded ones will need to share an approximate posterior. The result is a suboptimal compromise over many different true posteriors which cannot be mitigated by improving variational family or inference network capacity. We empirically show its effects in an extensive study on three real-world data sets on the common use case of variational state-space models.

heading: 2 AMORTISED VARIATIONAL INFERENCE

heading: 2.1 VARIATIONAL AUTO-ENCODERS
text: Consider the task of fitting the parameters of a latent-variable model pθ(x, z) = pθ(x | z) × pθ(z) to a target distribution p̂(x) by maximum likelihood:
argmax θ Ex∼p̂(x)[log pθ(x)]. (1)
In the case of variational auto-encoders (VAEs), the prior distribution pθ(z) is often simple, such as a standard Gaussian. The likelihood pθ(x | z) on the other hand is mostly represented by a deep neural network producing the likelihood parameters as a function of z.
The log marginal likelihood log pθ(x) = log ∫ pθ(x, z) dz contains a challenging integral within a logarithm. The maximisation of eq. (1) is thus generally intractable. Yet, for a single sample x the log marginal likelihood can be bounded from below by the evidence lower bound (ELBO):
log pθ(x) ≥ log pθ(x)−KL(q(z) || pθ(z | x)) (2) = Ez∼q[log pθ(x | z)]−KL(q(z) || pθ(z)) =: −`(θ, q,x),
where a surrogate distribution q, the variational posterior, is introduced. The bound is tighter the better q(z) approximates pθ(z | x). The gap vanishes with the posterior KL divergence in eq. (2).
Typically, the target distribution is given by a finite data set D = {x(n)}Nn=1. In classical variational inference, equally many variational posteriors {q(n)(z)}Nn=1 are determined by direct optimisation. For instance, each Gaussian q(n)(z) is represented by its mean µ(n) and standard deviation σ(n). The set of all possible distributions of that parametric form is called the variational family Q 3 q(z). Learning the generative model parameters θ can then be performed by an empirical risk minimisation approach, where the average negative ELBO over a data set is minimised:
arg min θ,{q(n)}Nn=1 N∑ n=1 ` ( θ, q(n)(z),x(n) ) .
The situation is different for VAEs. The parameters of the approximate posterior for a specific sample x are produced by another deep neural network, the inference network qφ(z | x). Instead of directly optimising for sample-wise parameters, a global set of weights φ for the inference network is found by stochastic gradient-descent of the expected negative ELBO:
argmin θ,φ
Ex∼p̂(x)[`(θ, qφ(z | x),x)].
This approach tends to scale more gracefully to large data sets.
The Sequential Case The posterior of a sequential latent-variable model
p(x1:T , z1:T ) = T∏ t=1 p(xt | x1:t−1, z1:t)p(zt | z1:t−1,x1:t−1).
of observations (x1, . . . ,xT ) = x1:T and latents (z1, . . . , zT ) = z1:T , with z1:0 = ∅, factorises as
p(z1:T | x1:T ) = T∏ t=1 p(zt | z1:t−1,x1:T ). (3)
The posterior KL divergence is a sum of step-wise divergences (Cover & Thomas, 2006):
KL(q(z1:T | ·) || p(z1:T | x1:T )) = T∑ t=1 Ez1:t−1∼q[KL(q(zt | ·) || p(zt | z1:t−1,x1:T ))] (4)
where we have left out the conditions of q(z1:T | ·) = ∏ t q(zt | ·) for now. This decomposition allows us to focus on the non-sequential case in our theoretical analysis, even though the phenomenon most commonly occurs with sequential LVMs.

heading: 2.2 THE APPROXIMATION AND AMORTISATION GAPS
text: The ELBO and the log marginal likelihood are separated by the posterior Kullback-Leibler divergence KL(q(z) || pθ(z | x)), cf. eq. (2). If the optimal member of the variational family q? ∈ Q is not equal to the true posterior, or p(z | x) /∈ Q, then
KL(q?(z) || pθ(z | x)) > 0,
so that the ELBO is strictly lower than the log marginal likelihood—the approximation gap.
Using an inference network we must expect that even the optimal variational posterior within the variational family is not found in general. The gap widens:
KL(qφ(z | x) || pθ(z | x)) ≥ KL(q?(z) || pθ(z | x)).
This phenomenon was first discussed by Cremer et al. (2018). The additional gap
KL(qφ(z | x) || pθ(z | x))−KL(q?(z) || pθ(z | x)) ≥ 0.
is called the amortisation gap.
The two gaps represent two distinct suboptimality cases when training VAEs. The former is caused by the choice of variational family, the latter by the success of the amortised search for a good candidate q ∈ Q.

heading: 3 PARTIAL CONDITIONING OF INFERENCE NETWORKS
text: VAEs add an additional consideration to approximate posterior design compared to classical variational inference: the choice of inputs to the inference networks. Instead of feeding the entire observation x, one can choose to feed a strict subset C ⊂ x. This is common practice in many sequential variants of the VAE, cf. table 1, typically motivated by efficiency or downstream applications. The discrepancy between inference models inputs and the true posterior conditions has been acknowledged before, e. g. by Krishnan et al. (2017), where they investigate different variants. Their analysis does not go beyond quantitative empirical comparisons. In the following, we show that such design choices lead to a distinct third cause of inference suboptimality, the conditioning gap.

heading: 3.1 MINIMISING EXPECTED KL DIVERGENCES YIELDS PRODUCTS OF POSTERIORS
text: The root cause is that all observations x that share the same included conditions C but differ in the excluded conditions must now share the same approximate posterior by design. This shared approximate posterior optimizes the expected KL divergence
ω := argmin q
EC|C [ KL ( qφ(z | C) ∣∣∣∣ p(z ∣∣ C,C))]. (5) w. r. t. the missing conditions C = x \ C. By rearranging (cf. appendix A.1 for details)
EC|C [ KL ( qφ(z | C) ∣∣∣∣ p(z ∣∣ C,C))] =KL ( qφ(z | C)
∣∣∣∣∣∣ exp(EC|C[log p(z ∣∣ C,C)])/Z)− logZ, where Z denotes a normalising constant, we see that the shared optimal approximate posterior is
ω(z) ∝ exp ( EC|C [ log p ( z ∣∣ C,C)]).
Interestingly, this expression bears superficial similarity with the true partially-conditioned posterior p(z | C) = EC|C [ p ( z ∣∣ C,C)] = exp(log(EC|C[p(z ∣∣ C,C)])).
To understand the difference, consider a uniform discrete C | C for the sake of the argument. In this case, the expectation is an average over all missing conditions. The true posterior p(z | C) is then a mixture distribution of all plausible full posteriors p ( z ∣∣ C,C). The optimal approximate posterior, because of the logarithm inside the sum, is a product of plausible full posteriors. Such distributions occur e. g. in products of experts (Hinton, 2002) or Gaussian sensor fusion (Murphy, 2012). Products behave differently from mixtures: an intuition due to Welling (2007) is that a factor in a product can single-handedly “veto” a sample, while each term in a mixture can only “pass” it.
This intuition is highlighted in fig. 1. We can see that ω is located between the modes and sharply peaked. It shares almost no mass with both the posteriors and the marginal posterior. The approximate marginal posterior ν on the other hand either covers one or two modes, depending on the width of the two full posteriors-a much more reasonable approximation.
In fact, ω(z) will only coincide with either p(z | C) or p ( z ∣∣ C,C) in the the extreme case when
p(z | C) = p ( z ∣∣ C,C) ⇔ p(z | C)p(C ∣∣ C) = p(z, C ∣∣ C), cf. appendix A.2, which implies
statistical independence of C and z given C.
Notice how the suboptimality of ω assumed neither a particular variational family, nor imperfect amortisation—ω is the analytically optimal shared posterior. Both approximation and amortisation gap would only additionally affect the KL divergence inside the expectation in eq. (5). As such, the inference suboptimality described here can occur even if those vanish. We call
EC [ EC|C [ KL ( ω(z | C) ∣∣∣∣ p(z ∣∣ C,C))]] (6) the conditioning gap, which in contrast to approximation and amortisation gap can only be defined as an expectation w. r. t. p(x) = p ( C,C ) .
Notice further that the effects described here assumed the KL divergence, a natural choice due to its duality with the ELBO (cf. eq. (2)). To what extent alternative divergences (Ranganath et al., 2014; Li & Turner, 2016) exhibit similar or more favourable behaviour is left to future research.
Learning Generative Models with Partially-Conditioned Posteriors We find that the optimal partially-conditioned posterior will not correspond to desirable posteriors such as p(z | C) or p(z | C,C), even assuming that the variational family could represent them. This also affects learning the generative model. A simple variational calculus argument (appendix A.3) reveals that, when trained with partially-conditioned approximate posteriors, maximum-likelihood models and ELBOoptimal models are generally not the same. In fact, they coincide only in the restricted cases where p(z | C,C) = q(z | C). As seen before this is the case if and only if C ⊥ z | C.

heading: 3.2 LEARNING A UNIVARIATE GAUSSIAN
text: It is worth highlighting the results of this section in a minimal scalar linear Gaussian example. The target distribution is p̂(x) ∼ N (0, 1). We assume the latent variable model
pa(x, z) = N (x | az, 0.1) · N (z | 0, 1)
with free parameter a ≥ 0. This implies pa(x) = N ( x ∣∣ 0, 0.1 + a2), pa(z | x) = N(z ∣∣∣ a(0.1 + a2)−1x, (1 + 10a2)−1).
With a∗ = √ 0.9, we recover the target distribution with posterior pa∗(z | x) = N ( z ∣∣ √0.9x, 0.1). Next, we introduce the variational approximation q(z) = N (z | µz, σ2z). With the only condition x missing, this is a deliberately extreme case of partial conditioning where C = ∅ and C = {x}. Notice that the true posterior is a member of the variational family, i. e., no approximation gap. We maximise the expected ELBO
ωa(z) = argmax q
Ex∼p̃ [ Ez∼q [ log pa(x, z)
q(z)
]] ,
i. e., all observations from p̂ share the same approximation q. One can show that ωa(z) = N ( z ∣∣∣ 0, (100a2 + 1)−1).
We immediately see that ωa(z) is neither equal to pa∗(z | x) ≡ p ( z ∣∣ C,C) nor to p(z) ≡ p(z | C).
Less obvious is that the variance of the optimal solution is much smaller than that of both p ( z ∣∣ C,C) and p(z | C). This is a consequence of the product of the expert compromise discussed earlier: the (renormalised) product of Gaussian densities will always have lower variance than either of the factors. This simple example highlights how poor shared posterior approximations can become.
Further, inserting ωa back into the expected ELBO and optimising for a reveals that the maximum likelihood model parameter a∗ is not optimal. In other words, pa∗(x, z) does not optimise the expected ELBO in p—despite being the maximum likelihood model.

heading: 3.3 EXTENSION TO THE SEQUENTIAL CASE
text: The conditioning gap may arise in each of the terms of eq. (4) if q is partially conditioned. Yet, it is common to leave out future observations, e. g. q(zt | z1:t−1,x1:t, xt+1:T ). To the best of our knowledge, in the literature only sequential applications of VAEs are potentially affected by the conditioning gap. Still, sequential latent-variable models with amortised under-conditioned posteriors have been applied success fully to, e. g., density estimation, anomaly detection, and sequential decision making (Bayer & Osendorfer, 2014; Soelch et al., 2016; Hafner et al., 2019). This may seem at odds with the previous results. The gap is not severe in all cases. Let us emphasize two “safe” cases where the gap vanishes because Ct ⊥ zt | Ct is approximately true. First, where the partially- and the fully-conditioned posterior correspond to the prior transition, i. e.
p(zt | zt−1) ≈ p(zt | Ct) ≈ p ( zt ∣∣ Ct, Ct).
This is for example the case for deterministic dynamics where the transition is a single point mass. Second, the case where the observations are sufficient to explain the latent state, i. e.
p(zt | xt) ≈ p(zt | Ct) ≈ p ( zt ∣∣ Ct, Ct).
A common case are systems with perfect state information.
Several studies have shown that performance gains of fully-conditioned over partially-conditioned approaches are negligible (Fraccaro et al., 2016; Maddison et al., 2017; Buesing et al., 2018). We conjecture that the mentioned “safe” cases are overrepresented in the studied data sets. For example, environments for reinforcement learning such as the gym or MuJoCo environments (Todorov et al., 2012; Brockman et al., 2016) feature deterministic dynamics. We will address a broader variety of cases in section 5.

heading: 4 RELATED WORK
text: The bias of the ELBO has been studied numerous times (Nowozin, 2018; Huang & Courville, 2019). A remarkable study by Turner & Sahani (2011) contains a series of carefully designed experiments showing how different forms of assumptions on the approximate posterior let different qualities of the solutions emerge. This study predates the introduction of amortised inference (Kingma & Welling, 2014; Rezende et al., 2014) however. Cremer et al. (2018) identify two gaps arising in the context of VAEs: the approximation gap, i. e. inference error resulting from the true posterior not being a member of variational family; and the amortisation gap, i. e. imperfect inference due to an amortised inference procedure, e. g. a single neural network call. Both gaps occur per sample and are independent of the problems in section 3.
Applying stochastic gradient variational Bayes (Kingma & Welling, 2014; Rezende et al., 2014) to sequential models was pioneered by Bayer & Osendorfer (2014); Fabius et al. (2015); Chung et al. (2015). The inclusion of state-space assumptions was henceforth demonstrated by Archer et al. (2015); Krishnan et al. (2015; 2017); Karl et al. (2017a); Fraccaro et al. (2016; 2017); BeckerEhmck et al. (2019). Specific applications to various domains followed, such as video prediction (Denton & Fergus, 2018), tracking (Kosiorek et al., 2018; Hsieh et al., 2018; Akhundov et al., 2019) or simultaneous localisation and mapping (Mirchev et al., 2019). Notable performance in model-based sequential decision making has also been achieved by Gregor et al. (2019); Hafner et al. (2019); Lee et al. (2020); Karl et al. (2017b); Becker-Ehmck et al. (2020). The overall benefit of latent sequential variables was however questioned by Lai et al. (2019). The performance of sequential latent-variable models in comparison to auto-regressive models was studied, arriving at the conclusion that the added stochasticity does not increase, but even decreases performance. We want to point out that their empirical study is restricted to the setting where zt ⊥ z1:t−1 | x1:T , see the appendix of their work. We conjecture that such assumptions lead to a collapse of the model where the latent variables merely help to explain the data local in time, i. e. intra-step correlations.
A related field is that of missing-value imputation. Here, tasks have increasingly been solved with probabilistic models (Ledig et al., 2017; Ivanov et al., 2019). The difference to our work is the explicit nature of the missing conditions. Missing conditions are typically directly considered by learning p ( C ∣∣ C) instead of p(x) and appropriate changes to the loss functions.

heading: 5 STUDY: VARIATIONAL STATE-SPACE MODELS
text: We study the implications of section 3 for the case of variational state-space models (VSSMs). We deliberately avoid the cases of deterministic dynamics and systems with perfect state information: unmanned aerial vehicle (UAV) trajectories with imperfect state information in section 5.2, a sequential version of the MNIST data set section 5.3 and traffic flow in section 5.4. The reason is that the gap is absent in these cases, see section 3.3. In all cases, we not only report the ELBO, but conduct qualitative evaluations as well. We start out with describing the employed model in section 5.1.

heading: 5.1 VARIATIONAL STATE-SPACE MODELS
text: The two Markov assumptions that every observation only depends on the current state and every state only depends on the previous state and condition lead to the following generative model:
p(x1:T , z1:T | u1:T ) = T∏ t=1 p(xt | zt)p(zt | zt−1,ut−1),
where u1:T are additional conditions from the data set (such as control signals) and z0 = u0 := ∅. We consider three different conditionings of the inference model
q(z1:T | x1:T ,u1:T ) = q(z1 | x1:k,u1:k) T∏ t=2 q(zt | zt−1,x1:m,u1:m),
partial with k = 1,m = t, semi with k > 1,m = t, and full with k = m = T . We call k the sneak peek, inspired by Karl et al. (2017a). See appendix B for more details.

heading: 5.2 UAV TRAJECTORIES FROM THE BLACKBIRD DATA SET
text: We apply semi- and fully-conditioned VSSMs to UAV modelling (Antonini et al., 2018). By discarding the rotational information, we create a system with imperfect state information, cf. section 3.3. Each observation xt ∈ R3 is the location of an unmanned aerial vehicle (UAV) in a fixed global frame. The conditions ut ∈ R14, consist of IMU readings, rotor speeds, and pulse-width modulation. The emission model was implemented as a Gaussian with fixed, hand-picked standard deviations, where the mean corresponds to the first three state dimensions (cf. (Akhundov et al., 2019)): p(xt | zt) = N ( µ = zt,1:3,σ 2 = [0.15, 0.15, 0.075] ) . We leave out the partially-conditioned case, as it cannot infer the higher-order derivatives necessary for rigid-body dynamics. A sneak peek of k = 7 for the semi-conditioned model is theoretically sufficient to infer those moments. See appendix C for details.
Fully-conditioned models outperform semi-conditioned ones on the test set ELBO, as can be seen in table 2a. We evaluated the models on prefix-sampling, i. e. the predictive performance of p(xt+1:T | x1:t,u1:T ). To restrict the analysis to the found parameters of the generative model only, we inferred the filter distribution p(zt | x1:t,u1:t) using a bootstrap particle filter (Sarkka, 2013). By not using the respective approximate posteriors, we ensure fairness between the different models. Samples from the predictive distribution were obtained via ancestral sampling of the generative model. Representative samples are shown in fig. 4. We performed a posterior-predictive check for
both models, where we compare the densities of the final observations xT obtained from prefix sampling in fig. 2. Both evaluations qualitatively illustrate that the predictions of the fully-conditioned approach concentrate more around the true values. In particular the partially-conditioned model struggles more with long-term prediction.

heading: 5.3 ROW-WISE MNIST
text: We transformed the MNIST data set into a sequential data set by considering one row in the image plane per time step, from top to bottom. This results in stochastic dynamics: similiar initial rows can result in a 3, 8, 9 or 0, future rows are very informative (cf. section 3.3). Before all experiments, each pixel was binarised by sampling from a Bernoulli with a rate in [0, 1] proportional to the corresponding pixel intensity.
The setup was identical to that of section 5.2, except that a Bernoulli likelihood parameterised by a feed-forward neural network was used: p(xt | zt) = B(FNNθE (zt)). No conditions u1:T and a short sneak-peek (k = 1) were used. The fully-conditioned model outperforms the partiallyconditioned by a large margin, placing it significantly closer to state-of-the-art performance, see table 2b. This is supported by samples from the model, see fig. 3. Cf. appendix D for details.
For qualitative evaluation, we used a state-of-the-art classifier1 to categorise 10,000 samples from each of the models. If the data distribution is learned well, we expect the classifier to behave similarly on both data and generative samples, i. e. yield uniformly distributed class predictions. We report KL-divergences from the uniform distribution of the class prediction distributions in table 2b. A bar plot of the induced class distributions can be found in fig. 3. Only the fully-conditioned model is able to nearly capture a uniform distribution.
1https://github.com/keras-team/keras/blob/2.4.0/examples/mnist_cnn.py

heading: 5.4 TRAFFIC FLOW
text: We consider the Seattle loop data set (Cui et al., 2019; 2020) of average speed measurements of cars at 323 locations on motorways near Seattle, from which we selected a single one (42). The dynamics of this system are highly stochastic, which is of special interest for our study, cf. section 3.3. Even though all days start out very similar, traffic jams can emerge suddenly. In this scenario the emission model was a feed-forward network conditioned on the whole latent state, i. e. p(xt | zt) = N ( FNNθE (zt), σ 2 x ) . We compare partially-, semi- (k = 7) and fully-conditioned models. See appendix E for details. The results are shown in table 2a. While the fully-conditioned posterior emerges as the best choice on the validation set, the semi-conditioned and the fullyconditioned one are on par on the test set. We suspect that the sneak peek is sufficient to fully capture a sensible initial state approximation.
We performed a qualitative evaluation of this scenario as well in the form of prefix sampling. Given the first t = 12 observations, the task is to predict the remaining ones for the day, compare section 5.2. We show the results in fig. 4. The fully-conditioned model clearly shows more concentrated yet multi-modal predictive distributions. The partially-condition model concentrates too much, and the semi-conditioned one too little.

heading: 6 DISCUSSION & CONCLUSION
text: We studied the effect of leaving out conditions of amortised posteriors, presenting strong theoretical findings and empirical evidence that it can impair maximum likelihood solutions and inference. Our work helps with approximate posterior design and provides intuitions for when full conditioning is necessary. We advocate that partially-conditioned approximate inference models should be used with caution in downstream tasks as they may not be adequate for the task, e. g., replacing a Bayesian filter. In general, we recommend conditioning inference models conforming to the true posterior.

heading: A DETAILED DERIVATIONS
text: A.1 DERIVATION OF THE SOLUTION TO AN EXPECTED KL-DIVERGENCE
We want to show that EC|C [ KL ( qφ(z | C) ∣∣∣∣ p(z ∣∣ C,C))] =KL ( qφ(z | C)
∣∣∣∣∣∣ exp(EC|C[log p(z ∣∣ C,C)])/Z)− logZ. This can be seen from
EC|C [ KL ( qφ(z | C) ∣∣∣∣ p(z ∣∣ C,C))] =EC|C [ Eqφ(z|C) [ log qφ(z | C) p ( z ∣∣ C,C)
]] =Eqφ(z|C)[log qφ(z | C)]− Eqφ(z|C) [ log ( exp ( EC|C [ log p ( z ∣∣ C,C)]))]
= KL ( qφ(z | C) ∣∣∣∣∣∣ exp(EC|C[log p(z ∣∣ C,C)])/Z)− logZ, where Z is the normalizing constant of
exp ( EC|C [ log p ( z ∣∣ C,C)]).
A.2 ANALYZING THE OPTIMAL PARTIALLY-CONDITIONED POSTERIOR In this section, we show that assuming either ω(z) = p(z | C) or ω(z) = p ( z ∣∣ C,C) implies
p ( C ∣∣ z, C) = p(C ∣∣ C).
We begin by observing ω(z) ∝ exp ( EC|C [ log p ( z ∣∣ C,C)])
= exp ( EC|C [ log ( p ( z ∣∣ C,C)p(z | C)
p(z | C) )]) = p(z | C) exp ( EC|C [ log p ( z ∣∣ C,C)
p(z | C)
]) .
Now, firstly,
ω(z) = p(z | C)
=⇒ exp ( Ep(C|C) [ log p ( z ∣∣ C,C)
p(z | C)
]) = 1
=⇒ p ( z ∣∣ C,C) p(z | C) = p ( C ∣∣ z, C) p ( C ∣∣ C) = 1
=⇒ p ( C ∣∣ z, C) = p(C ∣∣ C).
Secondly,
ω(z) = p ( z ∣∣ C,C)
=⇒ exp ( Ep(C|C) [ log p ( z ∣∣ C,C)
p(z | C)
]) ∝ p ( z ∣∣ C,C)
p(z | C) =⇒ p ( z ∣∣ C,C) p(z | C) = p ( C ∣∣ z, C) p ( C ∣∣ C) is constant w. r. t. C
=⇒ p ( C ∣∣ z, C) = p(C ∣∣ C).
A.3 PROOF OF SUBOPTIMAL GENERATIVE MODEL
We investigate whether a maximum likelihood solution p? = argminp Ex1:T∼p̂[− log p(x1:T )] is a minimum of the expected negative ELBO. From calculus of variations (Gelfand & Fomin, 2003), we derive necessary optimality conditions for maximum likelihood and the expected negative ELBO as
0 ! = dEx1:T∼p̂[− log p(x1:T )] dp α dG dp , (max. likelihood) (7)
0 ! = dEx1:T∼p̂[− log p(x1:T )] dp + dEx1:T∼p̂[KL] dp + λ dG dp , (ELBO) (8)
respectively. G is a constraint ensuring that p is a valid density, λ and α are Lagrange multipliers. KL refers to the posterior divergence in eq. (2). Equating (7) and (8) and rearranging gives
dEx1:T∼p̂[KL] dp + (α− λ)dG dp = 0. (9)
Equation (9) is a necessary and sufficient condition (van Erven & Harremoës, 2014) that the Kullback-Leibler divergence is minimised as a function of p, which happens when p(zt | Ct, Ct) = q(zt | Ct) for all t.

heading: B MODEL DETAILS
text: We restrict our study to a class of current state-of-the-art variational sequence models: that of residual state-space models. The fact that inference is not performed on the full latent state, but merely the residual, allows more efficient learning through better gradient flow (Karl et al., 2017b; Fraccaro et al., 2016).
Generative Model The two Markov assumptions that every observation only depends on the current state and every state only depends on the previous state and condition lead to the following model:
p(x1:T , z1:T | u1:T ) = T∏ t=1 p(xt | zt)p(zt | zt−1,ut−1),
where u1:T are additional conditions from the data set (such as control signals) and z0 = u0 := ∅. Such models can be represented efficiently by choosing a convenient parametric form (e. g. neural networks) which map the conditions to the parameters of an appropriate distributionD. In this work, we let the emission model Dx be a feed-forward neural network parameterised by θE :
p(xt | zt) = Dx(FNNθE (zt)).
For example, we might use a Gaussian distribution that is parameterised by its mean and variance, which are outputs of a feed-forward neural network (FNN).
Further, we represent the transition model as a deterministic step plus a component-wise scaled residual:
zt = FNNθT (zt−1,ut−1) + FNNθG(zt−1) t, t ∼ D .
Here FNNθG(zt−1) produces a gain with which the residual is multiplied element-wise. The residual distributionD is assumed to be zero-centred and independent across time steps. Gaussian noise is hence a convenient choice.
The initial state z1 needs to have a separate implementation, as it has no predecessor z0 and is hence structurally different. In this work, we consider a base distribution that is transformed into the distribution of interest by a bijective map, ensuring tractability. We use an inverse auto-regressive flow (Kingma et al., 2016) and write pθz1 (z1) to indicate the dependence on trainable parameters, but do not specify the base distribution further for clarity.
Amortised Inference Models This formulation allows for an efficient implementation of inference models. Given the residual, the state is a deterministic function of the preceding state—and hence it is only necessary to infer the residual. Consequently we implement the inference model as such:
q( t | z1:t−1,x1:T ,u1:T ) = D̂ t(FNNφ (z̃t, ft)).
Note that that the state-space assumptions imply p( t | z1:t−1,x1:T ) = p( t | zt−1,xt+1:T ) (Sarkka, 2013). We reuse the deterministic part of the transition z̃t = FNNθT (zt−1,ut−1). Features f1:T = f1:T = (f1, f2, . . . , fT ) define whether the inference is partially-conditioned and are explained below.
Due to the absence of a predecessor, the initial latent state z1 needs special treatment as in the generative case. We employ a separate feed-forward neural network to yield the parameters of the variational posterior at the first time step:
q(z1 | x1:T ,u1:T ) = D̂z1 ( FNNφz1 (f1) ) .
Under-, Semi-, and Fully-Conditioned Posteriors We control whether a variational posterior is partially-, semi- or fully conditioned by the design of the features f1:T . In the case of a fullyconditioned approximate posterior, we employ a bidirectional RNN (Schuster & Paliwal, 1997):
f1:T = BiRNNφft (x1:T ,u1:T ),
For the case of an partially-conditioned model, a unidirectional RNN is an option. Since it is common practice (Karl et al., 2017b) to condition the initial inference model on parts of the future as well, we let the feature at the first time step sneak peek into a chunk of length k of the future:
f1 = FNNφf1 (x1:k,u1:k)
f2:T = RNNφft (x1:T ,u1:T ),
where the recurrent network is unidirectional. We call this approach a semi-conditioned model.

heading: C DETAILS ON THE EXPERIMENTAL SETUP OF UAV MODELLING
text: We go with a typical split into training, validation and test data. Each consists of 10,240 sequences of 25 time steps starting out at the origin. The distribution of the residuals D and approximate posteriors D̂z1 , D̂ is Gaussian. A hyper-parameter search of 128 experiments was conducted. Models were trained by stochastic gradient descent using Adam (Bottou, 2010; Kingma & Ba, 2015) on the negative ELBO. We selected the best model for semi- and fully-conditioned models separately according to the respective ELBOs on the validation set after 30,000 updates, which was sufficient for all models to converge.
C.1 HYPER-PARAMETER SEARCH SPACE
field range
batch size Choice([8, 16, 32, 64, 128]) optimizer.learning rate Choice([0.003, 0.001, 0.0003, 0.0001, 3e-05]) optimizer.beta1 Choice([0.5, 0.8, 0.9]) n latent Range(low=16, high=33) initial.n flows Choice([2, 4, 8, 12]) initial.layer sizes.0 Choice([8, 16, 32, 64]) initial.layer sizes.1 Choice([8, 16, 32, 64]) transition.kind stochastic transition.func kind Choice([’residual’, ’highway’, ’plain’]) transition.layers.0.units Choice([16, 32, 64, 128, 192]) transition.layers.0.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) transition.layers.1.units Choice([16, 32, 64, 128, 192]) transition.layers.1.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) inv initial.layers.0.units Choice([16, 32, 64, 128]) inv initial.layers.0.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) inv initial.layers.1.units Choice([16, 32, 64, 128]) inv initial.layers.1.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) inv disturbance.layers.0.units Choice([16, 32, 64, 128]) inv disturbance.layers.0.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) inv disturbance.layers.1.units Choice([16, 32, 64, 128]) inv disturbance.layers.1.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) feature rnn.n states Choice([32, 64, 128]) feature rnn.n layers Choice([1, 2]) feature rnn.bidirectional True feature rnn.cell type gru feature rnn.initial mlp.prefix length 7 feature rnn.initial mlp.layers.0.units Choice([16, 32, 64, 128]) feature rnn.initial mlp.layers.0.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) feature rnn.initial mlp.layers.1.units Choice([16, 32, 64, 128]) feature rnn.initial mlp.layers.1.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) emission.scale kind fixed emission.scale array([0.15 , 0.15 , 0.075])
C.2 HYPER-PARAMETERS OF SEMI-CONDITIONED MODEL
field value
batch size 16 optimizer.learning rate 0.001 optimizer.beta1 0.9 n latent 22 initial.n flows 12 initial.layer sizes.0 8 initial.layer sizes.1 8 transition.kind stochastic transition.func kind residual transition.layers.0.units 16 transition.layers.0.activation tanh transition.layers.1.units 32 transition.layers.1.activation softplus inv initial.layers.0.units 16 inv initial.layers.0.activation tanh inv initial.layers.1.units 16 inv initial.layers.1.activation softsign inv disturbance.layers.0.units 64 inv disturbance.layers.0.activation tanh inv disturbance.layers.1.units 128 inv disturbance.layers.1.activation relu feature rnn.n states 128 feature rnn.n layers 1 feature rnn.bidirectional True feature rnn.cell type gru feature rnn.initial mlp.prefix length 7 eature rnn.initial mlp.layers.0.units 32 eature rnn.initial mlp.layers.0.activation softplus eature rnn.initial mlp.layers.1.units 32 eature rnn.initial mlp.layers.1.activation softsign emission.scale kind fixed emission.scale array([0.15 , 0.15 , 0.075])
C.3 HYPER-PARAMETERS OF FULLY-CONDITIONED MODEL
field value
batch size 32 optimizer.learning rate 0.003 optimizer.beta1 0.8 n latent 20 initial.n flows 12 initial.layer sizes.0 16 initial.layer sizes.1 8 transition.kind stochastic transition.func kind highway transition.layers.0.units 64 transition.layers.0.activation softplus transition.layers.1.units 16 transition.layers.1.activation softplus inv initial.layers.0.units 32 inv initial.layers.0.activation tanh inv initial.layers.1.units 128 inv initial.layers.1.activation relu inv disturbance.layers.0.units 128 inv disturbance.layers.0.activation tanh inv disturbance.layers.1.units 128 inv disturbance.layers.1.activation relu feature rnn.n states 128 feature rnn.n layers 1 feature rnn.bidirectional True feature rnn.cell type gru feature rnn.initial mlp.prefix length 7 feature rnn.initial mlp.layers.0.units 128 feature rnn.initial mlp.layers.0.activation softsign feature rnn.initial mlp.layers.1.units 32 feature rnn.initial mlp.layers.1.activation softplus emission.scale kind fixed emission.scale array([0.15 , 0.15 , 0.075])

heading: D DETAILS ON THE EXPERIMENTAL SETUP OF MNIST MODELLING
text: We conducted a hyper parameter search of 64 experiments for 15,000 iterations. The 5 best experiments (according to the ELBO at the last iteration) were continued for 85,000 further iterations. Test ELBOs are reported in table 2b.
D.1 HYPER-PARAMETERS OF PARTIALLY-CONDITIONED MODEL
field range
batch size 32 optimizer.learning rate 0.003 optimizer.beta1 0.8 n latent 25 initial.n flows 12 initial.layer sizes.0 64 initial.layer sizes.1 32 emission.scale kind learnable emission.scale 0.2 emission.layers.0.units 16 emission.layers.0.activation relu emission.layers.1.units 32 emission.layers.1.activation softplus transition.kind stochastic transition.func kind highway transition.layers.0.units 32 transition.layers.0.activation softsign transition.layers.1.units 128 transition.layers.1.activation softplus inv initial.layers.0.units 32 inv initial.layers.0.activation softsign inv initial.layers.1.units 64 inv initial.layers.1.activation tanh inv disturbance.layers.0.units 64 inv disturbance.layers.0.activation softplus inv disturbance.layers.1.units 64 inv disturbance.layers.1.activation softsign feature rnn.n states 128 feature rnn.bidirectional True feature rnn.cell type gru feature rnn.initial mlp.prefix length 1 feature rnn.initial mlp.layers.0.units 16 feature rnn.initial mlp.layers.1.units 32 feature rnn.initial mlp.layers.0.activation softplus feature rnn.initial mlp.layers.1.activation softsign
D.2 HYPER-PARAMETERS OF FULLY-CONDITIONED MODEL
field value
batch size 128 optimizer.learning rate 0.003 optimizer.beta1 0.9 n latent 25 initial.n flows 4 initial.layer sizes.0 64 initial.layer sizes.1 32 emission.scale kind learnable emission.scale 0.02 emission.layers.0.units 64 emission.layers.0.activation softsign emission.layers.1.units 64 emission.layers.1.activation softplus transition.kind stochastic transition.func kind highway transition.layers.0.units 192 transition.layers.0.activation relu transition.layers.1.units 192 transition.layers.1.activation tanh inv initial.layers.0.units 64 inv initial.layers.0.activation softsign inv initial.layers.1.units 16 inv initial.layers.1.activation relu inv disturbance.layers.0.units 64 inv disturbance.layers.0.activation relu inv disturbance.layers.1.units 16 inv disturbance.layers.1.activation softsign feature rnn.n states 32 feature rnn.bidirectional True feature rnn.cell type gru feature rnn.initial mlp.prefix length 1 feature rnn.initial mlp.layers.0.units 16 feature rnn.initial mlp.layers.0.activation tanh feature rnn.initial mlp.layers.1.units 128 feature rnn.initial mlp.layers.1.activation softplus

heading: E DETAILS ON THE EXPERIMENTAL SETUP OF TRAFFIC FLOW MODELLING
text: The data is down-sampled to contain average speeds over 30 minute windows from 6:30 to 19:00, resulting in 26 time steps per day. The data was split into training, validation and testing data by months, January up to July for training, July to September for validation and the remainder for testing. The standard deviation σ2x was determined during hyper-parameter optimisation along with the architectural and optimisation parameters. A hyper parameter search of 128 configurations was conducted. After 150,000 weight updates, the model for each partially-, semi- and fully-conditioned with the lowest negative ELBO on the validation set was selected.
E.1 HYPER-PARAMETER SEARCH SPACE
field range
batch size Choice([8, 16, 32, 64, 128]) optimizer.learning rate Choice([0.0003, 0.0001, 3e-05]) optimizer.beta1 Choice([0.5, 0.8, 0.9]) n latent 64 initial.n flows 4 initial.layer sizes.0 Choice([8, 16, 32, 64]) initial.layer sizes.1 Choice([8, 16, 32, 64]) transition.kind stochastic transition.func kind Choice([’residual’, ’highway’, ’plain’]) transition.layers.0.units 64 transition.layers.0.activation softsign transition.layers.0.kernel initia... orthogonal inv initial.layers.0.units 64 inv initial.layers.0.activation softplus inv initial.layers.0.kernel initi... glorot normal inv disturbance.layers.0.units 64 inv disturbance.layers.0.activation softplus inv disturbance.layers.0.kernel i... glorot normal feature rnn.n states 64 feature rnn.n layers 1 feature rnn.bidirectional True feature rnn.cell type gru feature rnn.initial mlp.prefix length 7 feature rnn.initial mlp.layers.0.units Choice([16, 32, 64, 128]) feature rnn.initial mlp.layers.0.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) feature rnn.initial mlp.layers.1.units Choice([16, 32, 64, 128]) feature rnn.initial mlp.layers.1.activation Choice([’softsign’, ’softplus’, ’relu’, ’tanh’]) emission.scale kind fixed emission.scale Choice([0.1, 0.2689655172413793, 0.43793103448... emission.layers.0.units 64 emission.layers.0.activation softplus emission.layers.0.kernel initializer glorot normal n hidden 64
E.2 HYPER-PARAMETERS OF PARTIALLY-CONDITIONED MODEL
field value
batch size 8 optimizer.learning rate 0.0001 optimizer.beta1 0.5 n latent 64 initial.n flows 4 initial.layer sizes.0 32 initial.layer sizes.1 64 transition.kind stochastic transition.func kind plain transition.layers.0.units 64 transition.layers.0.activation softsign transition.layers.0.kernel initia... orthogonal inv initial.layers.0.units 64 inv initial.layers.0.activation softplus inv initial.layers.0.kernel initi... glorot normal inv disturbance.layers.0.units 64 inv disturbance.layers.0.activation softplus inv disturbance.layers.0.kernel i... glorot normal feature rnn.n states 64 feature rnn.n layers 1 feature rnn.bidirectional True feature rnn.cell type gru feature rnn.initial mlp.prefix length 1 feature rnn.initial mlp.layers.0.activation softsign feature rnn.initial mlp.layers.0.units 128 feature rnn.initial mlp.layers.1.activation relu feature rnn.initial mlp.layers.1.units 128 emission.scale kind fixed emission.scale 2.97241 emission.layers.0.units 64 emission.layers.0.activation softplus emission.layers.0.kernel initializer glorot normal n hidden 64
E.3 HYPER-PARAMETERS OF SEMI-CONDITIONED MODEL
field value
batch size 128 optimizer.learning rate 0.0003 optimizer.beta1 0.5 n latent 64 initial.n flows 4 initial.layer sizes.0 8 initial.layer sizes.1 32 transition.kind stochastic transition.func kind plain transition.layers.0.units 64 transition.layers.0.activation softsign transition.layers.0.kernel initia... orthogonal inv initial.layers.0.units 64 inv initial.layers.0.activation softplus inv initial.layers.0.kernel initi... glorot normal inv disturbance.layers.0.units 64 inv disturbance.layers.0.activation softplus inv disturbance.layers.0.kernel i... glorot normal feature rnn.n states 64 feature rnn.n layers 1 feature rnn.bidirectional True feature rnn.cell type gru feature rnn.initial mlp.prefix length 7 feature rnn.initial mlp.layers.0.units 128 feature rnn.initial mlp.layers.1.units 16 feature rnn.initial mlp.layers.0.activation softsign feature rnn.initial mlp.layers.1.activation tanh emission.scale kind fixed emission.scale 0.775862 emission.layers.0.units 64 emission.layers.0.activation softplus emission.layers.0.kernel initializer glorot normal n hidden 64
E.4 HYPER-PARAMETERS OF FULLY-CONDITIONED MODEL
field value
batch size 8 optimizer.learning rate 0.0003 optimizer.beta1 0.8 n latent 64 initial.n flows 4 initial.layer sizes.0 8 initial.layer sizes.1 8 transition.kind stochastic transition.func kind plain transition.layers.0.units 64 transition.layers.0.activation softsign transition.layers.0.kernel initia... orthogonal inv initial.layers.0.units 64 inv initial.layers.0.activation softplus inv initial.layers.0.kernel initi... glorot normal inv disturbance.layers.0.units 64 inv disturbance.layers.0.activation softplus inv disturbance.layers.0.kernel i... glorot normal feature rnn.n states 64 feature rnn.n layers 1 feature rnn.bidirectional True feature rnn.cell type gru feature rnn.initial mlp.prefix length 7 feature rnn.initial mlp.layers.0.units 64 feature rnn.initial mlp.layers.0.activation relu feature rnn.initial mlp.layers.1.units 32 feature rnn.initial mlp.layers.1.activation softsign emission.scale kind fixed emission.scale 1.28276 emission.layers.0.units 64 emission.layers.0.activation softplus emission.layers.0.kernel initializer glorot normal n hidden 64

Review summaries:
This paper investigates the effect of partial conditioning on amortized inference in variational auto-encoders, focusing specifically on sequential data sources where it is common practice to have a posterior that is factorized in such a way that conditioning is partial (usually only conditioning on past signals in the sequence). Given a true posterior that is conditioned on the entire observed datapoint, the authors discuss the effect of having an approximate posterior that is only conditioned on part of the input. As the approximate posterior cannot adapt to the part of the input that is left out of the conditioning, the evidence lower bound becomes less tight, due to the larger KL divergence between the approximate posterior and the true posterior. The authors compare this to the work by Cramer et al. [1], where the distinction was made between having a restricted family of possible distributions for the approximate posterior (approximation gap) and the gap between an amortized approximate posterior with an inference network shared for all datapoints and a non-amortized approximate posterior that is optimized for each datapoint separately (amortisation gap). They argue that partial conditioning leads to a third type of gap which is distinct of the aforementioned inference gaps. Through an example with discrete observations the authors derive that when the true posterior is conditioned on the full data, and the approximate posterior is only partially conditioned, the optimal approximate posterior is something akin to a product of true posteriors over the unconditioned information, and not a mixture where the left out information is marginalized out. Through a 1D example they show that this could lead to overly sharp posteriors that have high densities in regions where the true posterior has very low density. 

The paper considers the problem of Bayesian inference with partially conditioned variational posterior. Namely, this work describes the phenomena of ill-behaved variational posterior for the case of partially observed data. The paper's main theoretical finding is that the partially conditioned variational posterior behaves like a product of experts, resulting in a degenerate solution. Speaking intuitively, the true posterior can be seen as a mixture of distributions: the sum over the unobservable variable. At the same time, the optimal variational posterior mixes as a product of distributions. Clearly, the product of densities hardly depicts features of the mixture since a near-zero value of a single member is enough for zeroing out the product's density.

The paper reviews the issue of partial conditioning of the amortized posterior in sequential latent variable models, typically state-space models trained with a VAE-style loss, but where the posterior used is the filtering rather than smoothing posterior. The author show that training a model with posterior with missing information can lead to a gap in estimating both the posterior and the corresponding model. They show the benefits of using the correct posteriors in simple examples.