<figure><img src="image_25.png" /><figcaption>**Figure 11.2:** Graphical model of DSAE at inference time in developed form (top) and compact form (bottom). In addition to the missing arrows shown in gold, we display in silver the arrows that should not be used, as compared to the structure of the exact posterior distribution. To the best of our knowledge, DSAE is the only model that uses probabilistic dependencies that do not appear in the exact posterior distribution.</figcaption></figure>
for the full inference model (11.11) and obtain

$$ \mathcal{L}(\theta, \phi, \mathbf{x}_{1:T}) = \mathbb{E}_{q_{\phi}(\mathbf{v}, \mathbf{z}_{1:T}|\mathbf{x}_{1:T})} \left[ \log p_{\theta_x}(\mathbf{x}_{1:T}|\mathbf{z}_{1:T}, \mathbf{v}) \right] \\ - D_{KL}(q_{\phi}(\mathbf{v}, \mathbf{z}_{1:T}|\mathbf{x}_{1:T}) \| p_{\theta}(\mathbf{v}, \mathbf{z}_{1:T})) \quad (11.23) $$

$$ = \mathbb{E}_{q_{\phi_{\mathbf{v}}}(\mathbf{v}|\mathbf{x}_{1:T})} \left[ \sum_{t=1}^{T} \mathbb{E}_{q_{\phi_{\mathbf{z}}}(\mathbf{z}_t|\mathbf{v}, \mathbf{x}_{1:T})} \left[ \log p_{\theta_x}(\mathbf{x}_t|\mathbf{z}_t, \mathbf{v}) \right] \right. \\ - \sum_{t=1}^{T} \mathbb{E}_{q_{\phi_{\mathbf{z}}}(\mathbf{z}_{1:t-1}|\mathbf{v}, \mathbf{x}_{1:T})} \left[ D_{KL}(q_{\phi_{\mathbf{z}}}(\mathbf{z}_t|\mathbf{v}, \mathbf{x}_{1:T}) \| p_{\theta_{\mathbf{z}}}(\mathbf{z}_t|\mathbf{z}_{1:t-1})) \right] \\ \left. - D_{KL}(q_{\phi_{\mathbf{v}}}(\mathbf{v}|\mathbf{x}_{1:T}) \| p_{\theta_{\mathbf{v}}}(\mathbf{v})). \right] \quad (11.24) $$

Therefore, one must first compute $q_{\phi_{\mathbf{v}}}(\mathbf{v}|\mathbf{x}_{1:T})$ to then sample from it. Once this is achieved, the parameters of $q_{\phi_{\mathbf{z}}}(\mathbf{z}_t|\mathbf{v}, \mathbf{x}_{1:T})$ for all time-steps $t$ can be computed without sampling from any random variable. Once this is achieved, the samples of $\mathbf{z}_{1:t-1}$ are used to compute the $t$-th KL divergence term in (11.24).
Brief tour of other models

In this section, we briefly present a few other models that have been recently proposed in the literature and can be considered members of the DVAE family. We choose not to present them in a detailed manner, as in the previous sections, because they are either too far from the scope of the review, which focuses on models associating a sequence of observations with a sequence of latent variables, or too close to the already presented models.

12.1 Models related to DKF

Latent LDS and structured VAE: Johnson et al. (2016) considered several models. One of them is a simplified DKF in which the latent variable model (i.e., the dynamical model) is linear-Gaussian; that is, it follows (3.16) (with $u_t$ following a standard Gaussian distribution), while the observation model is a DNN-based nonlinear model similar to the DKF observation model. They called it a latent LDS and extended it to a latent switching LDS model based on a bank of dynamical models (actually the same latent LDS but with different parameters) and an additional *discrete* latent variable that controls the switch between the dynamical models over time to adjust to the observed data dynamics.
This model can be considered an extension of the switching Kalman filter (Murphy, 1998; Fox et al., 2011) to a DNN-based observation model (see also Linderman et al. (2016) for a similar combination). Johnson et al. (2016) did not provide detailed equations for these models. Rather, they showed how the use of structured mean-field approximation in the inference model, combined with the use of an observation model that is conjugate to the latent variable model, can make the inference and training processes particularly efficient. They called the resulting model a structured VAE (SVAE). As they presented these developments in the general framework of probabilistic graphical models (Koller and Friedman, 2009), which is more general than the DVAE framework, they did not provide “temporal equations.” This makes this paper somewhat poorly connected to our review, although they mentioned that the DKF model (Krishnan et al., 2015) is strongly related to their work (they also implied that using RNN models for implementing time dependencies is restrictive in the general framework that they present).

**Black-box deep SSM:** Similarly, Archer *et al.* (2015) also focused on the structure of the approximate posterior distribution to improve the computational efficiency of the inference. They proposed using a multivariate Gaussian approximate posterior with a block tridiagonal inverse covariance matrix. They also proposed a corresponding fast and scalable inference algorithm. This general approach can be applied with different (deep and nondeep) parameterizations of the inference model and is applicable to a large family of SSMs, hence the “black box” denomination in the paper title. The authors mainly focused and experimented on an LG-LDS (to show that their algorithm can efficiently recover the solution of the Kalman filter), an LDS with a linear-Poisson observation model (which has no closed-form inference solution), and a basic one-dimensional nonlinear LDS. This study is well-connected with the DKF model and with deep SSMs in general. Interestingly, in this study, which, again, focuses more on the inference model than on the generative model, only the inference model is deep, whereas the generative models used in the experiments are nondeep. This study was later extended, notably addressing online learning and real-time issues (Zhao and Park, 2017; Zhao *et al.*, 2019).
**Deep variational Bayesian filters:** We have already mentioned this class of models in Section 6.1. DVBFs, which were proposed by (Karl et al., 2017), are an extension of the class of SSM-based DVAE models with dynamical models that depend on stochastic parameters. For example, the transition model at time t (i.e., between $z_t$ and $z_{t+1}$) can be a linear-Gaussian model with matrices and vectors that are a weighted sum of matrices/vectors randomly selected in a predefined set (possibly learned from data) with weights that are provided by a DNN. A similar transition model was applied within the KVAE model in Fraccaro et al.'s (2017) paper (see also Watter et al. (2015)).

**Disentangled SSM:** Miladinović et al. (2019) recently proposed a model called the disentangled state-space model (DSSM), in line with the DSAE model and, more generally, with models that attempt to separate the encoding of the content/object at the sequence level from that of its dynamics at the time-frame level. However, in contrast to DSAE, in which the sequence-level latent variable conditions the observation model, in DSSM, this sequence-level variable conditions the dynamical model. It is assumed to model the fact that the dynamics of an object are dependent on the considered applicative domain (e.g., enzyme kinetics or bouncing ball kinematics). In other words, (11.1) for DSAE can be reshaped in DSSM as<fnref n="1" />

$$p_{\theta}(\mathbf{x}_{1:T}, \mathbf{z}_{1:T}, \mathbf{v}) = p_{\theta_{\vee}}(\mathbf{v})p_{\theta_{z}}(\mathbf{z}_{0}) \prod_{t=1}^{T} p_{\theta_{x}}(\mathbf{x}_{t}|\mathbf{z}_{t})p_{\theta_{z}}(\mathbf{z}_{t}|\mathbf{z}_{t-1}, \mathbf{v}). \quad (12.1)$$

Here, the dynamical model is a (conditioned) first-order model. Miladinović et al. (2019) also proposed a filtering inference model that is implemented in the more general DVBF framework mentioned above.

## 12.2 Models related to STORN, VRNN, and SRNN

**Variational recurrent autoencoder (VRAE):** The VRAE model, proposed by Fabius and Amersfoort (2014), can be considered a simplified version of STORN, from which, according to the authors

<footnote n="1">¹In Miladinović et al.'s (2019) paper, the sequence-level variable is denoted as *D* for "domain." For consistency, we retain the notation *v* used in DSAE.</footnote>
themselves, it took inspiration. Here, the data sequence $\mathbf{x}_{1:T}$ is encoded by a single latent random vector $\mathbf{z}$, instead of the sequence $\mathbf{z}_{1:T}$. The compact form of the generative model is given by

$$p_{\theta}(\mathbf{x}_{1:T}, \mathbf{z}) = p_{\theta_z}(\mathbf{z}) \prod_{t=1}^{T} p_{\theta_x}(\mathbf{x}_t|\mathbf{x}_{1:t-1}, \mathbf{z}), \quad (12.2)$$

Fabius and Amersfoort (2014) provided no information about $p_{\theta_z}(\mathbf{z})$. The generative distribution $p_{\theta_x}(\mathbf{x}_t|\mathbf{x}_{1:t-1}, \mathbf{z})$ is implemented with a forward RNN that uses $\mathbf{z}$ to calculate the first hidden state $\mathbf{h}_1$ and then iteratively takes $\mathbf{x}_{t-1}$ as input to calculate $\mathbf{h}_t$, which provides the parameters of the distribution of $\mathbf{x}_t$. Conversely, the inference model $q_{\phi_z}(\mathbf{z}|\mathbf{x}_{1:T})$ is based on a forward RNN that takes $\mathbf{x}_{1:T}$ as input and delivers a final internal state $\mathbf{g}_T$, from which we obtain the distribution parameters for $\mathbf{z}$. In short, the VRAE generative model can be described by Figure 7.1, where the sequence $\mathbf{z}_{1:T}$ is replaced with a single input $\mathbf{z}$ for $\mathbf{h}_1$, and the VRAE inference model can be described by Figure 7.2, where the sequence $\mathbf{z}_{1:T}$ is replaced with a single output $\mathbf{z}$ for $\mathbf{g}_T$. We thus have a sequence-to-one encoding and a one-to-sequence decoding, which evoke the models designed for text/language processing mentioned in the Introduction. As Fabius and Amersfoort's (2014) paper was published in 2014 and was part of the early papers on DVAEs, it was probably inspiring for the natural language processing (NLP) community.

A similar model was proposed by Babaeizadeh et al. (2018), with a difference being that several vectors $\mathbf{x}_{t:T}$ are predicted from the past context $\mathbf{x}_{1:t-1}$ and from the unique latent vector $\mathbf{z}$. The inference model is also of the form $q_{\phi_z}(\mathbf{z}|\mathbf{x}_{1:T})$, as in Fabius and Amersfoort's (2014) study. Babaeizadeh et al. (2018) compared this model with a “baseline” model in which the latent vector is defined on a frame-by-frame basis (i.e., $\mathbf{z}_t$). The observation model of this baseline model is similar to that of SRNN, and the prior over $\mathbf{z}_t$ is an i.i.d. standard Gaussian distribution.

**Factorized hierarchical variational autoencoder (FHVAE):** The FHVAE model was proposed by Hsu et al. (2017b), to learn disentangled and interpretable latent representations from sequential data without supervision. To achieve this aim, FHVAE explicitly models the multi-
scaled aspect of the temporal information contained in the data. This is
done by splitting each sequence of data vectors into a set of fixed-size
consecutive sub-sequences, called segments, and defining two latent
variables **z** and **v** at the segment level.<fnref n="2" /> The former is dedicated to
capturing data information at the segment level, whereas the latter is
dedicated to capturing data information across segments (i.e., at the
sequence level). This model is particularly appropriate for speech signals:
In this case, 200-ms segments represent the approximate duration of
a syllable, and thus, **z** would typically encode phonetic information,
whereas **v** would typically encode speaker information at the level of a
complete utterance. In essence, FHVAE is strongly related to DSAE,
which also contains a sequence-level latent variable **v** but preserves
a time-frame resolution for the dynamical latent variable **z**<sub>t</sub> (see Sec-
tion 11). In fact, DSAE was published after FHVAE, from which it was
probably inspired.

Even if we do not detail this model, we report a few equations
to help better understand how segmental modeling works. Let here
$t \in [1, T]$ denote the index of a vector within a segment (each segment
has $T$ vectors), and let $n \in [1, N]$ denote the index of a segment within
a sequence. The FHVAE observation model for each individual segment
of data is given by

$$
\mathbf{h}_t^{(n)} = d_h(\mathbf{z}^{(n)}, \mathbf{v}^{(n)}, \mathbf{h}_{t-1}^{(n)}), \quad (12.3)
$$

$$
[\mu_{\theta_x}(\mathbf{h}_t^{(n)}), \sigma_{\theta_x}(\mathbf{h}_t^{(n)})] = d_x(\mathbf{h}_t^{(n)}), \quad (12.4)
$$

$$
p_{\theta_x}(\mathbf{x}_t^{(n)} | \mathbf{h}_t^{(n)}) = \mathcal{N}(\mathbf{x}_t^{(n)}; \mu_{\theta_x}(\mathbf{h}_t^{(n)}), \mathrm{diag}\{\sigma_{\theta_x}^2(\mathbf{h}_t^{(n)})\}). \quad (12.5)
$$

In (12.3), **z**<sup>(n)</sup> and **v**<sup>(n)</sup> respectively denote the latent vectors **z** and **v**
for the considered *n*-th segment. There is a single pair of such vectors
for each segment, and hence, a many-to-one encoding and one-to-many
decoding at the segment level. In practice, these equations are imple-
mented with an LSTM network. The prior distribution of **z**<sup>(n)</sup>, $p_{\theta_x}$(**z**<sup>(n)</sup>),
is a centered isotropic Gaussian that is independent of both the segment
and the sequence. In contrast, the prior distribution of **v**<sup>(n)</sup> depends on
a latent variable **w**, which is defined at the sequence level and whose

<footnote n="2"><sup>2</sup>In Hsu *et al.'s* (2017) paper, **z** and **v** are denoted **z<sub>1</sub>** and **z<sub>2</sub>**, respectively. We changed the notation to avoid confusion between the variable index and time index.</footnote>
prior distribution $p_{\theta_w}(\mathbf{w})$ is also a centered isotropic Gaussian. The distribution of $\mathbf{v}^{(n)}$ is then given by $p_{\theta_v}(\mathbf{v}^{(n)}|\mathbf{w}) = N(\mathbf{v}^{(n)}; \mathbf{w}, \sigma_{\theta_v}^2 \mathbf{I}_{L_v})$. For a given sequence, $p_{\theta_v}(\mathbf{v}^{(n)}|\mathbf{w})$ depends on the value of $\mathbf{w}$ drawn for that particular sequence. In practice, all generated $\mathbf{v}^{(n)}$ vectors within a sequence are close to $\mathbf{w}$. This makes $\mathbf{v}^{(n)}$ a sequence-dependent latent factor, whereas $\mathbf{z}^{(n)}$ behaves as a segment-dependent and sequence-independent latent factor. The joint density of a sequence is given by

$$p_{\theta}(\mathbf{x}_{1:T}^{(1:N)}, \mathbf{z}^{(1:N)}, \mathbf{v}^{(1:N)}, \mathbf{w}) = p_{\theta_w}(\mathbf{w}) \prod_{n=1}^{N} \prod_{t=1}^{T} p_{\theta_x}(\mathbf{x}_t^{(n)}|\mathbf{h}_t(\mathbf{z}^{(n)}, \mathbf{v}^{(n)})) \\ p_{\theta_z}(\mathbf{z}^{(n)})p_{\theta_v}(\mathbf{v}^{(n)}|\mathbf{w}), \quad (12.6)$$

where, as in the previous sections, $\mathbf{h}_t(\mathbf{z}^{(n)}, \mathbf{v}^{(n)})$ is a shortcut for the function that results from unfolding the recurrence in (12.3).

The inference model is a many-to-one encoder that works at the segment level: each segment $\mathbf{x}_{1:T}^{(n)}$ is encoded into a pair $\{\mathbf{z}^{(n)}, \mathbf{v}^{(n)}\}$ (plus an estimate of $\mathbf{w}$ for each whole sequence). As for the variational approximate posterior $q_{\phi}$, Hsu et al. (2017b) proposed the following model:

$$q_{\phi}(\mathbf{z}^{(1:N)}, \mathbf{v}^{(1:N)}, \mathbf{w}|\mathbf{x}_{1:T}^{(1:N)}) = q_{\phi_w}(\mathbf{w}) \prod_{n=1}^{N} q_{\phi_z}(\mathbf{z}^{(n)}|\mathbf{x}_{1:T}^{(n)}, \mathbf{v}^{(n)}) q_{\phi_v}(\mathbf{v}^{(n)}|\mathbf{x}_{1:T}^{(n)}), \quad (12.7)$$

where $q_{\phi_z}$ and $q_{\phi_v}$ are both implemented with a forward LSTM network, whose last state vector is passed to a DNN to provide the distribution parameters. Two encoders are chained here: The first one is used to generate $\mathbf{v}^{(n)}$ (by sampling $q_{\phi_v}(\mathbf{v}^{(n)}|\mathbf{x}_{1:T}^{(n)})$), and then, $\mathbf{v}^{(n)}$ is injected in the second encoder to generate $\mathbf{z}^{(n)}$. As for $q_{\phi_w}(\mathbf{w})$, it is a Gaussian distribution whose mean vector is obtained from a look-up table that is jointly learned with the model parameters (see Hsu et al. (2017b) for details). Cascading the sequence-to-one encoder with the one-to-sequence decoder results in a sequence-to-sequence neural network architecture that is trained by maximizing the VLB (not detailed here). The model can be optimized at the segment level instead of the sequence level; that is, each data segment can be used as a batch dataset.
According to Hsu et al. (2017b), this can solve scalability issues when the training sequences become too long.

**Deep recurrent attentive writer (DRAW):** A somewhat dual model of VRAE was proposed by Gregor et al. (2015) and called DRAW. This model considers a sequence of latent vectors $z_{1:T}$ to encode a single static but highly structured data x (typically a low-resolution image). The generative model is of the general form $p_{\theta_x}(x|z_{1:T})$. It involves the iterative construction of a sequence $\hat{x}_{1:T}$ that can be considered the sequence of images resulting from the “natural” drawing of x over time. The dependency of x on $z_{1:T}$ is implemented by combining the output of a decoder RNN (which takes $z_t$ as the input) and a so-called canvas matrix $c_{t-1}$, which encodes the difference between the final target image x and the current draw $\hat{x}_{t-1}$. Hence, the model combines deep learning and some form of predictive coding (Gersho and Gray, 2012). The inference model is of the form $q_{\phi_z}(z_t|z_{1:t-1}, x)$ and is implemented with an encoder RNN. This network takes as inputs a combination of x, $\hat{x}_{t-1}$, and the decoder output at the previous time step; hence, the predictive coding is implemented in closed-loop mode (Gersho and Gray, 2012). As the name indicates, DRAW includes a selective attention model that enables it to focus on the most relevant parts of the observation. The description of such an attention model is beyond the scope of the present review (see Gregor et al. (2015) and references therein for details).

**Neural adaptive sequential Monte Carlo (NASMC):** Gu et al. (2015) proposed the NASMC generative model, which combines an infinite-order Markovian model on $z_t$ (as that used in DSAE) with the most general observation model, which is that used in STORN and VRNN. In one variant of this model, they replaced the Markovian model on $z_t$ with an i.i.d. model, and thus, this variant is similar to STORN. The inference model has the same general form as that of VRNN; that is, it follows (8.14) and is parametrized by an RNN. The originality of NASMC lies in connecting the DVAE inference framework with sequential Monte Carlo (SMC) sampling. The inference model was used as a proposal distribution for SMC sampling. In fact, a complete framework to learn the parameters of the generative model, the proposal model, and for sampling from the posterior distribution with SMC
was proposed. In other words, Gu et al. (2015) showed that their
sampling-based approach can be used to optimize the observed data
marginal likelihood for estimating the generative model parameters in
the variational framework. For other applications of SMC methods in
the general context of variational approximations, see (Maddison et al.,
2017; Le et al., 2018; Naesseth et al., 2018).

**Recurrent state-space model (RSSM):** Hafner et al. (2018) used the RSSM model, which is very similar, if not identical, to the VRNN model used in the driven mode. In fact, RSSM corresponds to VRNN with an external input **u**<sub>t</sub> (denoted as **a**<sub>t-1</sub> by Hafner et al. (2018)), which is used in place of **x**<sub>t-1</sub> to compute **h**<sub>t</sub>. This is how VRNN was presented in the “Related Works” section of the SRNN paper (Fraccaro et al., 2016) (see in particular Figure 4(b) of this latter paper). Hafner et al. (2018) used this RSSM model for learning the dynamics and planning the actions of a synthetic agent from image sequences in a reinforcement learning framework. They also presented a way to perform multi-step prediction (i.e., prediction several steps ahead).

**Stochastic video generation (SVG):** Denton and Fergus (2018) presented a model very similar to STORN and applied it to stochastic video generation and multiple-frame video prediction, similarly to (Babaeizadeh et al., 2018). The inference model is of the form $q_{\phi_z}(\mathbf{z}_t|\mathbf{x}_{1:t})$. It is thus also similar to the inference model of STORN (see (7.19)). A variant of the generative distribution of $\mathbf{z}_t$, called “learned prior” in the paper, is also proposed. It is of the form $p_{\theta_z}(\mathbf{z}_t|\mathbf{x}_{1:t-1})$ and can be considered a simplification of the generative distribution of $\mathbf{z}_t$ in VRNN or SRNN.

12.3 Other models

**Factorized variational autoencoder (FVAE):** Deng et al. (2017) proposed the FVAE model, which combines a VAE with tensor factorization (Kuleshov et al., 2015; Huang et al., 2016). This latter is applied on the latent vector **z**. As one of the tensor factorization dimension is discrete time, this model implicitly involves data dynamics modeling. However, the temporal patterns are sampled from a standard log-normal
distribution, hence independently over time, and data decoding is also
processed independently at each time frame. It is thus unclear how
temporal dynamics are actually encoded.

**Gaussian process variational autoencoder (GP-VAE):** Fortuin *et al.* (2020) recently combined a VAE for the observed data dimension reduction and a multivariate Gaussian process (GP) (Williams and Rasmussen, 2006) for modeling the dynamics of the resulting latent vector $z_t$. For the GP model, they used a Cauchy kernel, which is appropriate to model data with multiscale time dynamics. They proposed using an approximate posterior distribution that is also a multivariate GP (here, a first-order one). The resulting overall GP-VAE model was trained with the VAE methodology and then used to efficiently recover missing data in test sequences (in videos and medical data). One interesting property of this model is that it provides interpretable uncertainty estimates.
<figure><img src="image_26.png" /></figure>
13

Experiments

In this chapter, we present an experimental benchmark of six of the DVAE models detailed in the previous chapters (DKF, STORN, VRNN, SRNN, RVAE, DSAE). This benchmark is conducted on a dataset of speech signals and a dataset of 3D human motion data. We provide a series of quantitative results on the task of analysis-resynthesis; that is encoding with the DVAE encoder followed by decoding with the DVAE decoder. We also provide qualitative results, in the form of examples of data generated by the models. We first present the models implementation in Section 13.1. Then, we describe the experimental protocol, datasets, model training and testing settings, and evaluation metrics in Section 13.2. Finally, we present and discuss the results in Sections 13.3 and 13.4. We recall that a link to the open-source code and the best-trained models can be found at https://team.inria.fr/robotlearn/dvae/.

**13.1 DVAE architectures**

The architectures of the six DVAE models that we benchmark are summarized in Figure 13.1. For each model, we represent the high-level computational graph corresponding to the encoding, sampling
<figure><img src="image_27.png" /><figcaption>**Figure 13.1:** Model architecture for the six tested DVAE models. For DKF, CF and GTF are a combiner function and a gated transition function, respectively. These functions are described in Appendix B (Section B.1) and in Krishnan et al.'s (2015) paper. For VRNN and SRNN, the gray-shaded boxes are modules shared by the encoder and decoder.</figcaption></figure>
and decoding processes. In particular, we show the types of layers that compose the encoder and decoder networks. Note that none of the DVAEs is used in the driven mode; that is, none of them feature an external input $\mathbf{u}_{1:T}$. The MLPs are generally used to extract high-level features and/or as a combiner function, whereas the recurrent networks (RNNs and BRNNs) are used to accumulate the information over time. All RNNs are instantiated as LSTM networks (and BRNNs are instantiated as bidirectional LSTM networks). DKF includes a specific combiner function (CF) at the end of the encoder and a gated transition function (GTF) to implement the dynamical model. These functions are described in the original DKF paper by Krishnan et al. (2015) and we report them in Appendix B (Sections B.1). We recall that the internal state vector $\mathbf{h}_t$ of VRNN and SRNN is shared between the encoder and decoder (see Sections 8.2 and 9.2). In Figure 13.1, we represent the corresponding shared networks with grey-shaded boxes. We also recall that RVAE was presented in two versions: causal and noncausal (see Section 10.1). Figure 13.1 only shows the architecture of the noncausal RVAE. The schema of the causal RVAE architecture is obtained by replacing the BRNN in the inference and generative models with a backward and a forward RNN, respectively. Finally, for all output variance parameters, we use log-parameterization (i.e., the output of the network corresponding to a variance parameter $\sigma^2$ is actually $\log \sigma^2$).

The general architectures shown in Figure 13.1 are common to the two sets of experiments that we conducted on speech data and on human motion data, although with different layer dimensions. For our experiments with speech data, the observed, latent, and RNN internal state vectors are of dimension 513, 16, and 128, respectively, for all DVAE models. For our experiments with human motion data, they are of dimension 96, 10, and 64, respectively, for all DVAE models (see Sections 13.2.1 and 13.2.2 for the definition of the observed data vectors). Low-level implementation details such as the number of layers and the number of units per layer may also differ between the two sets of experiments on speech and human motion data. These details are provided in Appendices B and C).
## 13.2 Experimental protocol

### 13.2.1 Speech data

For our experiments with speech data, we used the Wall Street Journal dataset (WSJ0; Garofolo et al., 1993), which comprises speech read from WSJ news. We used the speaker-independent, medium vocabulary (5k words) subset of the corpus. More precisely, the *si_tr_s* subset (~25 h) was used for training, the *siDT_05* subset (~2 h) was used for validation, and the *si_et_05* subset (~1.5 h) was used for testing.

The raw speech waveform was sampled at 16 kHz. Analysis-resynthesis was performed with the DVAEs in the time-frequency domain on power spectrograms. Time-domain speech signals were thus preprocessed with the short-time Fourier transform (STFT), using a 64-ms sine window (1,024 samples) with 25%-overlap to obtain sequences of 513-dimensional discrete Fourier spectra (for positive frequencies). Then, we computed the squared magnitude of these STFT spectrograms. For the training dataset, we set $T = 50$, meaning that speech utterances of 0.8 s were extracted from the raw dataset and pre-processed with the STFT. In summary, each training speech sequence is a 513 × 50 STFT power spectrogram. This data preprocessing resulted in a set of $N_{tr} = 46,578$ training sequences (representing about 10.3 hours of speech signal) and $N_{val} = 7,775$ validation sequences (~1.7 h). For testing, we used the STFT spectrogram of each complete test sequence (with the beginning and ending silence portions removed), which can be of variable length, most often larger than 2.4 s.

As discussed by Leglaive et al. (2020) and mentioned in Section 10.1, the complex-valued STFT coefficients are modeled with a zero-mean circular complex Gaussian distribution (see (10.2)), whereas $z_t$ is modeled as usual with a real-valued Gaussian distribution. The data sequence $x_{1:T}$ processed by the DVAE models is the squared magnitude of the STFT spectrogram (i.e., a real-valued nonnegative power spectrogram). The corresponding phase spectrogram is directly combined with the DVAE output magnitude spectrogram to reconstruct the output speech signal using inverse STFT with overlap-add. Modeling the STFT coefficients with a zero-mean circular complex Gaussian distribution with
variance $\sigma^2_{\theta_x, f, t}(\cdot)$ amounts to modeling each entry $x_{f,t}$ of the power spectrogram $\mathbf{x}_{1:T}$ with a Gamma distribution with shape parameter 1 and scale parameter $\sigma^2_{\theta_x, f, t}(\cdot)$ (i.e., $x_{f,t} \sim \mathcal{G}(1, 1/\sigma^2_{\theta_x, f, t}(\cdot))$.<fnref n="1" /> This also amounts to using the Itakura-Saito divergence between $x_{f,t}$ and $\sigma^2_{\theta_x, f, t}(\cdot)$ in the reconstruction term of the VLB (Girin et al., 2019). We recall that all presented DVAE models are versatile regarding the conditional pdf of $\mathbf{x}_t$, and using a Gamma distribution (more appropriate for speech/audio power spectrograms) in place of the Gaussian distribution that was used in the generic presentation of the models does not present any problem. The linear layer estimating the parameters of this distribution has 513 output units corresponding to the log-variance parameters $\{\log \sigma^2_{\theta_x, f, t}(\cdot)\}_{f=1}^F$.

### 13.2.2 3D human motion data

For our experiments with 3D human motion data, we used the H3.6M dataset (Ionescu et al., 2014), which is one of the largest dataset of the kind and has been widely used in video prediction (Finn et al., 2016), human pose and shape estimation (Bogo et al., 2016), and human motion prediction (Martinez et al., 2017). This dataset was obtained from multi-view video recordings of 11 professional actors performing 17 various scenarios (e.g., discussing, smoking, taking a picture, or talking on the phone), using four calibrated cameras with 50 Hz resolution. The 3D {$x, y, z$} coordinates of 32 human skeleton joints were extracted from these multi-view recordings. Each set of coordinates was centered w.r.t. the coordinates of the pelvis joint.

For our experiments with DVAEs, each data frame is organized as a 96-dimensional vector $\mathbf{x}_t$ by concatenating the 3D coordinates of the 32 joints. We used sequences of $T = 50$ consecutive vectors, which represent a duration of 2 s (the data were previously downsampled by a factor 2). These sequences were obtained by applying a 50-frame sliding window on the original H3.6M sequences, with a shift of two frames. In summary, each example in our dataset is thus a matrix of 3D coordinates of skeleton joints of size 96 × 50, which corresponds to

<footnote n="1">¹Here, we do not specify the variables generating the variance, as they depend on the DVAE model. Instead, the subscripts indicate frequency bin *f* and time frame *t*.</footnote>
2 s of human motion.

In H3.6M, 15 scenarios from 7 actors are provided with the ground-
truth annotations. Similarly to Mao et al. (2020), we used the data of
all 15 scenarios from 5 actors (Actors 1, 6, 7, 8, and 9) for training and
from 1 actor (Actor 11) for testing. Applying the sequence extraction
procedure described above led to N<sub>tr</sub> = 88,952 training sequences
(~ 50 h) and N<sub>test</sub> = 13,838 test sequences (~ 8 h). For validation, to
reduce the computation time, we selected 128 sequences for each of the
15 scenarios by Actor 5 (N<sub>val</sub> = 1,920 sequences, ~ 1 h).

Following Bayer and Osendorfer (2014) and Petrovich et al. (2021),
the 3D human motion data vectors x_t are modeled by a Gaussian
generative conditional distribution with a covariance matrix equal to
the identity matrix.

13.2.3 Training and testing

All tested models were implemented in PyTorch (Paszke et al., 2019). To train the models, we used the Adam optimizer (Kingma and Ba, 2014) with mini-batches of size 128. For the speech data, we set the learning rate to 0.002, whereas we used 0.0001 for the human motion data. We also used early stopping on the validation set with a patience of 50 epochs for the speech data and 30 epochs for the human motion data. During the training of the models with the human motion data, we applied warm-up to the KL regularization term in the VLB by multiplying it with a factor $\beta$ and linearly increasing this factor from 0 to 1 after each epoch, during the first 50 epochs (Sønderby et al., 2016b; Vahdat and Kautz, 2020).

Once a model was trained on the training set, with early stopping
on the validation set, its weights were fixed and the model was run on
the test set. We report the average performance obtained on the test
set using the metrics presented in the next subsection.

13.2.4 Evaluation metrics

For the experiments on speech data, we used three metrics to evaluate
the resynthesized speech quality and compare the performance of the
different DVAE models: the scale-invariant signal-to-distortion ratio
(SI-SDR) in dB (Le Roux et al., 2019), the perceptual evaluation of speech quality (PESQ) score in [-0.5, 4.5] (Rix et al., 2001), and the extended short-time objective intelligibility (ESTOI) score in [0, 1] (Taal et al., 2011). For all metrics, the higher the better. Note that these metrics are applied on the time-domain signals (i.e., speech waveforms). We combined the reconstructed magnitude spectrogram with the phase spectrogram of the original signal to obtain the analyzed-resynthesized speech waveform (using the inverse STFT).

For 3D human motion data, we can directly compare each original sequence with the corresponding analyzed-resynthesized sequence. We used the mean per joint position error (MPJPE) proposed by Ionescu et al. (2014), which is an averaged Euclidean distance per joint. We report the results in millimeters (mm). Note that this error corresponds to the log-likelihood term (or reconstruction error term) of the VLB, up to a constant factor that is controlled through the setting of the variance of the data conditional distribution.

# 13.3 Results on speech data

## 13.3.1 Analysis-resynthesis

We first present the results of analysis-resynthesis performed on the speech data. The values of the three metrics described in the previous subsection and averaged over the test dataset are reported in Table 13.1. In this experiment, all three autoregressive models (STORN, VRNN, and SRNN) were trained and tested in the teacher-forcing mode (i.e., using the ground-truth values of past observed vectors $x_{1:t-1}$ when generating $x_t$, see Section 4.4.1). Results of analysis-resynthesis in the generation mode will be discussed in Section 13.3.3.

From Table 13.1, we can draw the following conclusions. First, all tested DVAE models lead to fair signal reconstruction, with an SI-SDR ranging in 6.9–11 dB. This range is in accordance with the fact that we compress each 513-dimensional data vector into a 16-latent vector (using also the 128-dimensional RNN internal state that encodes the past data vectors in the case of autoregressive models). The quality of the reconstructed spectrograms is illustrated in Figures 13.2, 13.3 and 13.4.
<table><caption>**Table 13.1:** Performance of the DVAE models tested in our speech analysis-resynthesis experiment. The SI-SDR, PESQ, and ESTOI scores are averaged over the test subset of the WSJ0 dataset. STORN, SRNN and VRNN were trained and tested in the teacher-forcing mode. SRNN-TF-GM stands for the SRNN model trained in the teacher-forcing mode and tested in the generation mode. SRNN-GM stands for the SRNN model trained and tested in generation mode.</caption><thead><tr><th>DVAE</th><th>SI-SDR (dB)</th><th>PESQ</th><th>ESTOI</th></tr></thead><tbody><tr><td>VAE</td><td>5.3</td><td>2.97</td><td>0.83</td></tr><tr><td>DKF</td><td>9.3</td><td>3.53</td><td>0.91</td></tr><tr><td>STORN</td><td>6.9</td><td>3.42</td><td>0.90</td></tr><tr><td>VRNN</td><td>10.0</td><td>3.61</td><td>0.92</td></tr><tr><td>SRNN</td><td><strong>11.0</strong></td><td><strong>3.68</strong></td><td><strong>0.93</strong></td></tr><tr><td>RVAE-causal</td><td>9.0</td><td>3.49</td><td>0.90</td></tr><tr><td>RVAE-noncausal</td><td>8.9</td><td>3.58</td><td>0.91</td></tr><tr><td>DSAЕ</td><td>9.2</td><td>3.55</td><td>0.91</td></tr><tr><td>SRNN-TF-GM</td><td>-1.0</td><td>1.93</td><td>0.64</td></tr><tr><td>SRNN-GM</td><td>7.8</td><td>3.37</td><td>0.88</td></tr></tbody></table>

Figure 13.2(top) shows the power spectrogram of a speech signal uttered by a female speaker. Figure 13.2(bottom), and Figures 13.3 and 13.4 show the corresponding spectrogram obtained after analysis-resynthesis by the six tested models. Actually, the first 2 s on the left of the red line are obtained with analysis-resynthesis; the following 2 s are obtained by switching the models into generation mode, as presented in the next subsection. As for the analysis-resynthesis part (the first 2 s), we can see in these plots that the reconstructed spectrograms are all quite close to the original spectrogram. They look like a slightly smoothed or blurred version of the original spectrogram, which is typical of a data compression effect, although retaining most of the speech content characteristics. Regarding the perceptual quality of the reconstructed speech signals, Table 13.1 shows the PESQ scores that range from fair to good. The STOI scores, generally higher than 0.90, show good intelligibility.

Second, all DVAE models outperform the standard VAE model by a large margin (except maybe for STORN, which is “only” 1.6 dB SI-SDR
<figure><img src="image_28.png" /><figcaption>**Figure 13.2:** Example of power spectrogram for a speech signal uttered by a female speaker. Top: spectrogram of the original signal. Bottom: spectrogram reconstructed (0-2 s) and generated (2-4 s) with a vanilla VAE (the red line indicates the transition between reconstruction and generation).</figcaption></figure>

higher). The harmonics in the spectrogram reconstructed with the VAE in Figure 13.2(bottom) are slightly noisy and blurrier compared to the harmonics reconstructed with the DVAE models. This demonstrates the merit of including temporal modeling in the VAE framework for modeling sequential data, such as speech signals. SRNN exhibits the best performance and VRNN comes second. By looking at the associated probabilistic models, we can observe that SRNN and VRNN are the most complex models in terms of dependencies between observed and latent variables. We believe that these dependencies allow SRNN and VRNN to better capture the temporal structure of speech signal than the other models. SRNN performs slightly better than VRNN (e.g., it is 1 dB SI-SDR above VRNN), although VRNN has richer variable dependencies. This may be because the inference model of SRNN respects the structure of the exact posterior distribution, whereas that of VRNN (as proposed
<figure><img src="image_29.png" /><figcaption>**Figure 13.3:** Example of speech power spectrogram reconstructed (0-2 s) and generated (2-4 s) by a DVAE model (the original spectrogram is in Figure 13.2 (top)). Top: DKF; middle: DSAE; bottom: noncausal RVAE. The red line indicates the transition between reconstruction and generation.</figcaption></figure>

in the original paper and implemented here) does not. In both cases, the exact posterior of $z_t$ at each time $t$ depends on all observations $x_{1:t}$. However, the inference model of VRNN only takes the causal observations $x_{1:t}$ into account.

The performance scores of DKF and DSAE are very close to each
<figure><img src="image_30.png" /><figcaption>**Figure 13.4:** Example of speech power spectrogram reconstructed (0-2 s) and generated (2-4 s) by a DVAE model (the original spectrogram is in Figure 13.2 (top)). Top: STORN; middle: SRNN; bottom: VRNN. The red line indicates the transition between reconstruction and generation. Note that this figure was obtained with the models being trained and evaluated in a slightly different configuration regarding STFT parameters. Here, we have a window length of 512 points and an overlap of 50%. This is to illustrate the robustness of the results w.r.t. the “audio parameterization.”</figcaption></figure>

other, and slightly below those of VRNN. This is an interesting result, as we recall that DKF and DSAE are SSM-like models, with no explicit
temporal dependency between $x_{t-1}$ and $x_t$, but only between $z_{t-1}$ (or $z_{1:t-1}$ for DSAE) and $z_t$. Thus, even if they are expected to have less "predictive" power compared to SRNN or VRNN, their SSM structure appears quite efficient at encoding the speech dynamics. DSAE can be considered as an improved version of DKF, with an additional sequence-level variable **v** and infinite-order temporal dependency of $z_t$ (as opposed to first-order dependency for DKF). The fact that this more sophisticated structure does not lead to improved performance over DKF might be explained by the structure of the inference model. For DSAE, the inference of $z_t$ depends on $x_{1:T}$, whereas the exact posterior distribution depends on $z_{1:t-1}$ and $x_{t:T}$. Thus, the inference model of DSAE is not only missing some dependencies it should have (previous latent variables) but it is adding dependencies that it should not have (previous observed variables), see Section 11.2. In contrast, the DKF inference model respects the structure of the exact posterior distribution. In the end, all these differences between DKF and DSAE may compensate each other, leading to similar results. One way to improve DSAE may be to design an inference model with the structure of the exact posterior distribution.

The SI-SDR scores obtained by the RVAE model are just below those of DKF and DSAE, whereas the PESQ score of noncausal RVAE is slightly superior to those of DKF and DSAE. This is interesting considering that as DKF and DSAE, RVAE has no predictive link (e.g., no direct dependency between $x_{t-1}$ and $x_t$), and contrary to DKF and DSAE, RVAE does not have a dynamical model on the latent vector $z_t$ (i.e., it is modeled as an i.i.d. variable). However, in the analysis-resynthesis framework, the sequence $z_{1:t-1}$ or $z_{1:T}$ efficiently encodes $x_{1:T}$ owing to an efficient inference model, and then, the generative model is able to exploit this whole sequence to regenerate $x_t$. As expected, the noncausal version of RVAE is slightly better than the causal version.

As for STORN, its performance revealed a bit disappointing in our speech analysis-resynthesis experiment. Here also, this can be explained by the fact that the inference model of STORN does not respect the structure of the exact posterior distribution. In particular it does not use $z_{1:t-1}$ nor $x_{t+1:T}$, which is doubly penalizing compared to if the structure of the exact posterior distribution was considered. Another reason for
the relatively low scores obtained by STORN in these experiments is given in Section 13.3.4.

### 13.3.2 Generation of speech spectrograms

In this subsection, we briefly illustrate the ability of the DVAE models to generate “speech-like” spectrograms with a qualitative example. This example is the “continuation” of the example that we have seen in the previous subsection, provided in Figures 13.2, 13.3 and 13.4. In these figures, the first 2 s of each spectrogram (on the left of the red line) was obtained with analysis-resynthesis; that is, the latent vectors were provided by the inference model, using the ground-truth observed vectors as input, and the output spectrogram was then provided by the generative model using the inferred latent vectors (and the ground-truth past observed vectors for the autoregressive models). After 2 s (on the right of the red line), we turn the models to pure generation mode; that is, the latent vectors and the output spectrogram are now both provided by the generative model, without relying on the inference model and ground-truth past observed vectors anymore. This strategy allows the generation mode to benefit from a good initialization, induced by the analysis-resynthesis part. Indeed, at the time instant corresponding to 2 s, the generation starts with the past latent vectors and current RNN internal state provided by the analysis-resynthesis part, thus encoding the past observed speech data. Therefore, we can expect a smooth transition from the analyzed-resynthesized spectrogram to the generated one.

As expected, we observe in Figure 13.2 that a vanilla VAE is not able to generate a spectrogram with a realistic speech-like structure. In particular, the successive spectrogram “chunks” are too short and with too abrupt transitions to be speech sounds. This is due to the fact that there is no temporal modeling.

Figure 13.3 shows the results obtained with the nonautoregressive DVAE models. We can see that the spectrograms generated by DKF and DSAE, although different, both exhibit a harmonic structure and a variety of different speech-like sounds, which smoothly evolve with time. The smoothness probably comes from the use of a Markov model for
the latent vector, which precisely enforces smoothness. In the original DSAE paper, Li *et al.* (2017) did not provide examples of generated speech spectrograms but they presented good results in voice conversion obtained by exchanging the value of the variable v across two sentences spoken by a different speaker. Regarding RVAE, even though the latent vectors are independently and identically sampled from a standard Gaussian distribution, we observe a generated spectrogram with a temporal structure. This means that the RVAE model is able to “recreate” correlation in the generated data by combining the vectors of an uncorrelated sequence. However, the energy is mostly concentrated in low frequencies, and the harmonic structure, although present, is not as clearly visible as for DKF and DSAE. Moreover, the segments corresponding to the successive speech sounds are shorter than for DKF and DSE and seem shorter than what is expected in natural speech.

Figure 13.4 shows the generation results with autoregressive DVAE models. We see that, as the other models, STORN manages to ensure a smooth transition between the analysis-resynthesis and generation parts, but then the quality of the generated spectrogram becomes much lower than with the other autoregressive DVAE models. SRNN generates a spectrogram with a speech-like structure and a lot of variability. The best result for this example sentence is obtained with VRNN, for which we can observe segments that resemble different phonemes, with smooth transitions between them. The harmonic structure is also clearly visible and the generated data cover the full bandwidth.

### 13.3.3 Training with scheduled sampling

To complement the previous results, we performed an additional analysis-resynthesis experiment with the autoregressive models being trained and tested in the generation mode instead of the teacher-forcing mode; that is, using the previously generated data vectors $\hat{x}_{1:t-1}$ in place of the ground-truth past vectors $x_{1:t-1}$ when generating $x_t$ (see Section 4.4.1). Note that here, the sequence $z_{1:T}$ is still provided by the encoder. For conciseness, we present only the results for SRNN, which was the model performing best in the first experiment above.

Directly training a model in the generation mode was observed to
be difficult, so we adopted a *scheduled sampling* approach (Bengio *et al.*, 2015). We started with the SRNN model trained in the teacher-forcing mode used in the previous experiment. Then, we fine-tuned this model by randomly replacing **x**<sub>1:t−1</sub> with **x**<sub>1:t−1</sub> at the input of the encoder-decoder shared module, when estimating **x**<sub>t</sub>. We replaced 20% of the **x**<sub>1:t−1</sub> vectors for the first 50 epochs, and increased to 40% for the next 50 epochs, and so on, until we completely replaced the ground-truth clean speech signal with generated speech signals. Then we fine-tuned with totally generated speech signals for another 300 epochs. Overall, we fine-tuned the model for 500 epochs. The resulting model is referred to as SRNN-GM hereinafter. Moreover, we also evaluated the initial SRNN model (trained with teacher-forcing) in the generation mode (i.e., we use here **x**<sub>1:t−1</sub> during training and **x**<sub>1:t−1</sub> during testing when generating **x**<sub>t</sub>). We refer to this “hybrid” configuration as SRNN-TF-GM.

We can see from Table 13.1 that SRNN trained in the teacher-forcing mode and tested in the generation mode (SRNN-TF-GM) obtains very poor results. This illustrates the problem of train/test mismatch discussed in Section 4.4.2. The strategy that consists in training SRNN in the generation mode using scheduled sampling (SRNN-GM) is shown to be effective, as the gap between SRNN-TF-GM and SRNN is largely reduced. Nevertheless, SRNN-GM remains a bit below DKF and RVAE in this experiment, showing that it is more difficult to exploit the predictive links in a practical application where the ground-truth values of the observed vectors are not available, compared to the “oracle” configuration of teacher-forcing.

### 13.3.4 Visualization of the latent vector sequence

In this subsection, we illustrate the behavior of the learned latent vector. Figure 13.5 displays the trajectories of the mean vector of $q_{\phi_x}$, the log-variance vector of $q_{\phi_x}$, a vector $z_t$ sampled from $q_{\phi_x}$, and the KL divergence term of the VLB, for an example sentence of the speech test dataset and for the SRNN model trained and applied in the teacher-forcing mode. We observe that some of the dimensions of the latent vector, for example the first one, show a steady profile of the mean (Figure 13.5 (a)) and log-variance (Figure 13.5 (b)) for these dimensions,
<figure><img src="image_31.png" /><figcaption>**Figure 13.5:** Example of the behavior of the latent vector of a speech spectrogram for SRNN (trained with teacher-forcing). (a) mean value of the posterior distribution (i.e., $\mu_{\phi_x}(\cdot)$); (b) log-variance of the posterior distribution (i.e., $\sigma^2_{\phi_x}(\cdot)$); (c) sampled latent vector $z_t$; (d) KL divergence term of the VLB.</figcaption></figure>

with the mean being close to zero and the variance being much lower than 1. As a result, the corresponding entries of the $z_t$ samples shown in Figure 13.5 (c) look like noise with small fluctuations around zero. In short, those dimensions are noninformative, whereas the other “active” dimensions show much larger, and thus informative, fluctuations (note that for these “active” dimensions, because the variance is also small, yet not steady, the sampled latent values are close to the mean). This inactivity of certain dimensions is the sign of the *posterior collapse* problem. The latent vector $z_t$, or at least some dimensions of it, becomes noninformative, as its posterior distribution becomes too close to its prior (generative) distribution, as illustrated by the small (and steady) values of the KL divergence term of the VLB, compared to the other “active” dimensions (Figure 13.5 (d)). This problem has been well identified and largely discussed in the VAE literature (Bowman *et al.*, 2016; Serban *et al.*, 2016; Chen *et al.*, 2017; Lucas *et al.*, 2019; Razavi *et al.*, 2019;
Dai *et al.*, 2020). It remains a largely open topic in the framework of DVAE, as further discussed in Chapter 14.

Overall, the input of the decoder consists of some dimensions con-
taining informative patterns and some other dimensions containing
low-variance stationary noise. In the present example of the SRNN
model, we identified 8 dimensions out of 16 that are active and 8 di-
mensions that seem to have collapsed, suggesting that we would obtain
similar analysis-resynthesis performance by setting the size of the latent
vector to 8. In these experiments with speech signals, we observed that
VRNN has only 2 collapsed dimensions out of 16, whereas STORN had
12 collapsed dimensions out of 16; that is, only 4 active dimensions for
STORN. This latter point is to be related to the fact that STORN is
the less efficient of the three autoregressive models in these experiments,
with performance significantly below that of SRNN and VRNN. In
contrast, we did not observe posterior collapse of any dimension for the
nonautoregressive models (DKF, DSAE, and RVAE) in our experiments;
that is, for the nonautoregressive models, all 16 dimensions are useful to
encode $x_t$. This is consistent with the fact that, for these models, there
is no predictive link and all the information in $x_{1:T}$ must be encoded in
$z_{1:T}$.

We have checked that, for a given model, the active/inactive dimen-
sions remain the same across different examples. In addition, as shown
in the example of Figure 13.5, the dimensions suffering from posterior
collapse are the same over time. In principle, a small KL divergence
at a certain time frame means that the posterior and the generative
distributions of $z_t$ are very close to each other for that time frame.
However, this does not guarantee that the generative distribution on
two consecutive frames remains the same, and the same for the pos-
terior distribution. In practice, we observe that both the generative
and posterior distributions of collapsed dimensions are time-invariant
and noninformative. We believe this is due to the way these models
are implemented in practice, since the architectures used to implement
the DVAEs are time-invariant; that is, the same architecture with the
same weights is used at every time step. Combined with the fact that
noninformative generative distributions are zero-mean and low-variance
Gaussians, this leads us to conjecture that posterior collapse in a given
<figure><img src="image_32.png" /><figcaption>**Figure 13.6:** Example of the behavior of the latent vector of a speech spectrogram for SRNN (trained in the generation mode with scheduled sampling). (a) mean value of the posterior distribution (i.e., $\mu_{\phi_z}(\cdot)$); (b) log-variance of the posterior distribution (i.e., $\sigma_{\phi_z}^2(\cdot)$); (c) sampled latent vector $z_t$; (d) KL divergence term of the VLB.</figcaption></figure>

dimension of $z$ is associated to very small weights to compute the mean and log-variance of that dimension. We have verified this statement by visualising the weights of the output linear layers (not shown here for conciseness).

Figure 13.6 displays the same trajectories as Figure 13.5, for the same example sentence, but for the SRNN model trained in the generation mode (with scheduled sampling) instead of the teacher-forcing mode. By comparing the two figures, we observe that the SRNN model trained in the generation mode exhibits much less posterior collapse, with a larger number of “active” dimensions. By definition, the generated data vectors are approximate values of the ground-truth vectors, hence the model trained in the generation mode uses less accurate and thus less reliable past values of $x_t$ to generate the current value, compared to the same model trained in the teacher-forcing mode. Therefore, the
<table><caption>**Table 13.2:** Performance of the DVAE models tested on 3D human motion data analysis-resynthesis. The MPJPE scores are averaged over the test subset of the H3.6M dataset.</caption><thead><tr><th>DVAE</th><th>MPJPE (mm)</th></tr></thead><tbody><tr><td>VAE</td><td>48.69</td></tr><tr><td>DKF</td><td>42.21</td></tr><tr><td>STORN</td><td>9.47</td></tr><tr><td>VRNN</td><td>9.22</td></tr><tr><td>SRNN</td><td>7.86</td></tr><tr><td>RVAE-Causal</td><td>31.09</td></tr><tr><td>RVAE-NonCausal</td><td>28.59</td></tr><tr><td>DSAE</td><td>28.61</td></tr><tr><td>SRNN-TF-GM</td><td>221.87</td></tr><tr><td>SRNN-GM</td><td>43.98</td></tr></tbody></table>

model trained in the generation mode needs more informative latent dimensions to resynthesize $x_t$.

# 13.4 Results on 3D human motion data

## 13.4.1 Analysis-resynthesis

Table 13.2 shows the results of the analysis-resynthesis experiment with the 3D human motion data. The MPJPE values are approximately within 9–49mm, which is relatively small compared to the average amplitude of the joint coordinates in a human body, and therefore show a fair to good reconstruction for all models. As for the speech analysis-resynthesis experiment, all DVAE models outperform the vanilla VAE model. This confirms the interest of using DVAE models for modeling sequential data.

In this experiment with human motion data, the autoregressive DVAEs largely outperform the nonautoregressive DVAEs. STORN, VRNN, and SRNN have an MPJPE of about 9.5, 9.2 and 7.9 mm, respectively, whereas DKF, RVAE and DSAE (noncausal) obtain about 42.2, 28.6 and 28.6 mm, respectively. Therefore, the performance gap
between the autoregressive models (trained and tested in the teacher-
forcing mode) and the nonautoregressive models is larger than in our
experiment with speech signals. We conjecture that this is because the
3D human motion data has less variability (or, say, smoother trajectories)
compared to speech data. Therefore, for such data, knowing the ground-
truth values of the previous observation(s) (x<sub>t-1</sub> or x<sub>1:t-1</sub>) is a very
strong information for predicting the current observation.

Again, SRNN exhibits the best performance, which is consistent
with the analysis-resynthesis results obtained with the speech data.
In this new experiment with human motion data, STORN is more
efficient than in our experiment with speech data. In contrast, DKF
underperforms compared to the other nonautoregressive models and
exhibits a quite limited improvement over the vanilla VAE.

13.4.2 Generation of 3D human motion sequences

Example videos of a human “skeleton” animated from 3D motion data
sequences generated by the different DVAE models are available at
https://team.inria.fr/robotlearn/dvae/.

13.4.3 Training with scheduled sampling

As for the experiment with speech signals, we have tested the influence of
training and testing the models in the generation mode (using scheduled
sampling for training). Here, we briefly report and comment the results
obtained on the 3D human motion data with SRNN. We can see from
Table 13.2 that, similarly to what we observed in our experiment with
speech signals, SRNN-TF-GM has very poor performance, whereas
training SRNN with scheduled sampling partially addresses this problem,
placing SRNN-GM between the vanilla VAE and the nonautoregressive
DVAEs. However, the gain in performance of SRNN-GM over the vanilla
VAE is here quite limited, and we believe there is room for improvement
when designing the model adaptation method. In other words, in these
experiments, we adopted a simple scheduled sampling strategy and did
not further investigated this issue, but other strategies to bridge the
gap between ground-truth and generated data could be investigated.
13.4.4 Visualization of the latent vector sequence

To conclude this set of experiments, we also provide an example of
visualization of the latent space of the 3D human data, as we did for
speech signals.

Figure 13.7 displays the trajectories of the mean vector of $q_{\phi_z}$, the log-variance vector of $q_{\phi_z}$, a vector $z_t$ sampled from $q_{\phi_z}$, and the KL divergence term of the VLB, for an example sequence of the 3D human motion test dataset and for the SRNN model trained and applied in the teacher-forcing mode. Figure 13.8 displays the corresponding trajectories for the SRNN model trained and applied in the generation mode (trained with scheduled sampling). We can see in Figure 13.7 that the trajectories of the parameters (and of the sampled $z_t$ vector) are (much) smoother than in the case of speech signals, as, again, the 3D human motion data themselves have smoother trajectories compared to speech data. Some dimensions seem more “active” than others, even if, without a deeper investigation, it is quite difficult to interpret the range of values covered by the entries of the latent vector. Such a thorough investigation is beyond the scope of the present paper. We simply note here that there are two dimensions, dimensions 1 and 3, that seem to collapse. For these two dimensions, the mean is steady around zero and the variance has a large (and steady) value, hence the sampled trajectory of the corresponding $z_t$ entries looks like noise. In contrast, some other dimensions seem to have an interesting informative profile. For example, dimensions 4 and 5 have an opposite fluctuation, probably encoding an opposite evolution of the corresponding factors of data variation.

In Figure 13.8, the latent vector generally seems more “active” or
informative than in Figure 13.7. For most dimensions, the ratio between
the range of the mean variation and the range of variance variation is
larger. The range of the KL divergence term values is also larger than
in Figure 13.7, indicating that the approximate posterior and the “prior”
(generative distribution of $z_t$) are less close to each other than in the
teacher-forcing case. These observations are consistent with those made
on the speech signals in Section 13.3.4. Our conjecture that the latent
variable was less important in the teacher-forcing mode than in the
<figure><img src="image_33.png" /><figcaption>**Figure 13.7:** Example of the behavior of the latent vector for a test sequence from the H3.6M dataset and for SRNN (trained with teacher-forcing). (a) mean value of the posterior distribution (i.e., $\mu_{\phi_z}(\cdot)$); (b) log-variance of the posterior distribution (i.e., $\sigma_{\phi_z}^2(\cdot)$); (c) sampled latent vector $z_t$; (d) KL divergence term of the VLB.</figcaption></figure>

generation mode is thus confirmed with the motion data.

## 13.5 Conclusion

In a practical application requiring the modeling of temporal data such as speech spectrograms or 3D human motion data, using either VRNN or SRNN seems a relevant choice, especially if autoregressive models can be used in the teacher-forcing mode. In addition, considering the above results and associated discussion, we suspect that having an inference model that respects the exact variable dependencies at inference time is important for obtaining the optimal performance. However, this is not always possible, as some applications require a causal inference model for online processing. Finally, in a practical application where only $z_{1:T}$ needs to be transmitted, data resynthesis from $z_{1:T}$ with an autoregressive model used in the generation mode was shown to
<figure><img src="image_34.png" /><figcaption>**Figure 13.8:** Example of the behavior of the latent vector for a test sequence from the H3.6M dataset and for SRNN (trained in generation mode with scheduled sampling). (a) mean value of the posterior distribution (i.e., $\mu_{\phi_z}(\cdot)$); (b) log-variance of the posterior distribution (i.e., $\sigma_{\phi_z}^2(\cdot)$); (c) sampled latent vector $z_t$; (d) KL divergence term of the VLB.</figcaption></figure>

be reasonably robust in our experiment with SRNN and speech data, provided that the model is fine-tuned in the generation mode (using here scheduled sampling). For 3D motion data, SRNN-GM was shown in our experiments to perform more poorly (with a limited gain over the vanilla VAE).

The performance of some other models, in particular DKF, also seem to depend on the data type. We thus insist that the above “model ranking” is valid only for the presented experiments, which involve pure analysis-resynthesis of speech spectrograms or 3D human motion data. For data generation, we presented only a limited set of qualitative examples to illustrate the behavior of the models (evaluating the quality of generated data is still a very difficult problem and a hot topic of machine learning with generative model). For other tasks, such as signal/data transformation (with modification of the latent vectors
between analysis and resynthesis), we do not know if our experimental
results would generalize. In particular, it is difficult to know how much of
the information contained in **x**₁:T is encoded into **z**₁:T, or what “features”
of **x**₁:T are encoded. In particular, in the presented experiments we did
not investigate the disentanglement power of each model and how the
disentangled latent dimensions can be interpreted as relevant factors of
variation of the data (from a physical point of view for example). Also,
we illustrated the posterior collapse problem but we did not proceed
to a thorough quantitative investigation of this issue. These points will
be further discussed in the next chapter. However, the experiments
dedicated to illustrating them are beyond the scope of the present
paper.
14

Discussion

In this chapter, we conclude our review of DVAEs with a discussion. First, we recall the fundamental motivation for designing and using DVAEs and then comment on their remarkable flexibility at multiple levels (design of the generation and inference models, high-level and low-level implementation). Then, we return to the crucial point of the disentanglement of latent factors in the present context of sequential data processing. Finally, we present some perspectives on data source coding.

14.1 Fundamental motivation for DVAEs

The fundamental motivation for designing and using DVAEs is to
combine various dynamical models, aimed at modeling the dynamics
of sequential data, and various VAEs, aimed at modeling the latent
factors of data variations. In doing so, we expect to separate the data
dynamics from the other factors of variations (see Section 14.3.3 below
dedicated to this specific point) and use the latter to augment the
expressivity of the models. Another way to express this idea is to point
out the superiority of DVAEs over RNNs: Adding a latent variable z_t
within an RNN adds considerable flexibility and modeling power to the
conditional output density. Let us here quote Chung *et al.* (2015):

> “We show that the introduction of latent random variables can provide significant improvements in modelling highly structured sequences such as natural speech sequences. We empirically show that the inclusion of randomness into high-level latent space can enable the VRNN to model natural speech sequences with a simple Gaussian distribution as the output function. However, the standard RNN model using the same output function fails to generate reasonable samples. An RNN-based model using more powerful output function such as a GMM can generate much better samples, but they contain a large amount of high-frequency noise compared to the samples generated by the VRNN-based models.”

Similarly, we can point out the superiority of DVAE over classical (nondeep) DBNs and SSMs, owing to the deep nonlinear layers of information processing. Again, let us quote Chung *et al.* (2015):

> “Drawing inspiration from simpler dynamic Bayesian networks (DBNs) such as HMMs and Kalman filters, the proposed variational recurrent neural network (VRNN) explicitly models the dependencies between latent random variables across subsequent timesteps. However, unlike these simpler DBN models, the VRNN retains the flexibility to model highly non-linear dynamics.”

Of course, such a statement applies to the entire DVAE family of models.

## 14.2 DVAE outcome: A story of flexibility

### 14.2.1 Flexibility of the generative model(s)

As seen in this review, various generative models can be derived from the general form (4.4) by simplifying variable dependencies. The models we have reviewed (such as STORN, VRNN, and SRNN) are instances of these possible generative models, but there are other possibilities.
Moreover, each model has several variants: Driven/undriven mode, pre-
dictive/nonpredictive mode, and with one or several feature extractors.

When designing a generative model, complexity issues can be con-
sidered. For example, we can quote Bayer and Osendorfer (2014):

"[...] we can restrict ourselves to prior distributions over the latent variables that factorize over time steps, i.e., $p(\mathbf{z}_{1:T}) = \prod_{t=1}^{T} p(\mathbf{z}_t)$. This is much easier to handle in practice, as calculating necessary quantities such as the KL-divergence can be done independently over all time steps and components of $\mathbf{z}_t$."

However, at the same time, the systematic aspect of the VAE method-
ology (and the versatility of the current deep learning toolkits) enables,
in principle, to train a model of arbitrary complexity. Hence, if one
is not limited by computational cost, this offers new possibilities. For
example, we can recall our remark in Section 11.1 about the choice of
the dynamical model in deep SSMs. In the DVAE framework, not only
is it easy to move from a linear dynamical model to a nonlinear one,
but also to move from a first-order temporal model to a (much) higher
order.

14.2.2 Flexibility of the inference model(s)

In DVAEs, as in standard VAEs, the exact posterior distribution is
usually intractable due to nonlinearities, which is why we have to define
an inference model in addition to the generative model (we cannot apply
the Bayes rule analytically). However, a key feature of DVAEs with
respect to standard VAEs is that we must define an inference model over
a sequence of latent vectors. Even though the exact posterior distribution
over this latent sequence is analytically intractable, we can leverage the
chain rule and the D-separation principle to analyze the structure of
the exact posterior distribution induced by the chosen generative model.
It seems quite natural to exploit this knowledge to design the structure
of the inference model so that it is consistent with the structure of
dependencies in the exact intractable posterior. Yet, several seminal
papers on DVAEs have not followed this “consistency principle,” and,
more importantly, not justified the chosen structure of the inference model. Nevertheless, it is not mandatory to follow the structure of the exact posterior distribution to design the inference model. For instance, if the structure of the exact posterior distribution implies a noncausal processing of the observations, the anticausal dependencies can be dropped for the purpose of online applications. Simplifying posterior dependencies can also be motivated by a need to reduce the computational complexity of inference.

Another key difference between DVAEs and VAEs relates to how the VLB (or, actually, its estimate) is computed. The VLB involves intractable expectations, which are usually replaced with empirical averages, using samples drawn from the inference model. The sampling procedure in DVAEs has to be recursive due to the dynamical nature of the model, a constraint that standard VAEs do not have. This recursive sampling is related to the use of RNNs and can be costly. As will be discussed below, other neural network architectures can be more computationally efficient than RNNs.

### 14.2.3 Flexibility of the implementation

As already discussed in Section 4.1.2, various possibilities exist for the high-level implementation of DVAEs. We recall that various developed model representations can correspond to the same compact representation. In fact, the compact form describes all parent-child relationships among random variables, regardless of how these relationships are implemented in practice. Therefore, the compact representation is important to understand the probabilistic dependencies between variables. However, one must be aware that the optimization does not search for all possible models satisfying the relationships of the compact representation but only for a specific model corresponding to the developed representation. This representation allows us to understand how the dependencies are implemented in practice, and therefore, over which parameter space the model is optimized. This representation typically involves a recurrent architecture. While such architectures allow the encoding of high-order temporal dependencies, their developed graphical representations generally do not exhibit dependencies higher than
first-order. Therefore, the developed representation can be “visually misleading.” This duality is important in DVAEs, and we encourage to provide both representations when presenting and discussing DVAEs, as done in this review.

Once the high-level DVAE architecture is chosen, various possibilities exist for the low-level implementation: network type (e.g., LSTM against GRU) and low-level (hyper)parameterization (number of layers in a network, number of units per layer, type of activation function, and classical deep learning modules, such as batch normalization). We choose not to detail these low-level implementation aspects in this review, and instead, considered them a deep learning routine. All these choices (or at least part of them) depend on data nature and datasets and can significantly impact the modeling performance.

### 14.2.4 Other network architectures for sequential data modeling

In this review, we focus on deep generative latent-variable models of sequential data using RNNs (or simple feed-forward fully-connected DNNs for first-order temporal dependencies). However, other neural network architectures can deal with sequential data of arbitrary length, the most popular ones probably being convolutional architectures. While RNNs are (virtually) based on infinite-order temporal modeling, CNNs generally have a fixed-length receptive field, which implies a finite-order temporal modeling. In particular, temporal convolutional networks (TCNs) are becoming increasingly popular due to their competitive performance with RNNs (e.g., in speech separation (Luo and Mesgarani, 2019)) while being more flexible and computationally efficient (Bai et al., 2018). A TCN is based on dilated 1D convolutions, sharing similarities with the Wavenet architecture (Oord et al., 2016a), and just as an RNN, it outputs a sequence of the same length as that of the input sequence. The combination of TCNs with VAEs was explored by Aksan and Hilliges (2019).

Another popular neural network architecture that can deal with sequential data is the transformer (Vaswani et al., 2017), which is based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. However, only a few studies have considered
leveraging transformers for generative modeling in the VAE framework. We found only transformer-based VAEs recently proposed for sentence generation (Liu and Liu, 2019), story completion (Wang and Wan, 2019), and music representation learning (Jiang et al., 2020).

## 14.3 VAE improvements and extensions applicable to DVAEs

Following the seminal VAE papers by Kingma and Welling (2014) and Rezende et al. (2014), many papers have been proposed for VAE improvements and extensions. In this subsection, we mention some of these improvements and extensions and discuss their relation and possible adaptation to DVAEs. This is a nonexhaustive review; the purpose of the present paper is not to deepen this rich part of VAE literature but rather to show that the DVAE development is still largely open, and one way to improve DVAEs is to get inspired from the recent studies on VAEs. The interested readers can refer to Kingma and Welling's (2019) paper for a more detailed review of the "static" VAE improvements and extensions.

### 14.3.1 Improved VAE decoders and the posterior collapse problem

The mathematical formulation of the VAE in the seminal paper by Kingma and Welling (2014) considered a 1D data vector framework; that is, $x$ is a fixed-size $F$-dimensional vector. What happens if we want to apply the VAE to 2D data, such as images, or more generally, to $N$-dimensional data? What happens if one of the dimensions is variable, like variable-length time sequences? By considering variable-size sequences and, in particular, variable-length time sequences, we take a step towards DVAEs. However, we consider models with many-to-one encoding and one-to-many decoding more as VAEs than DVAEs, following the line announced in the Introduction.

Kingma and Welling (2014) considered the application of the VAE to 2D image modeling. However, the correlation between neighboring pixels was poorly exploited, as the conditional generative model (conditioned on the latent variable) was pixelwise independent.<fnref n="1" /> In such a setting, $z$

<footnote n="1">¹Technically, an image is arbitrarily reshaped into a vector $x$, with each pixel</footnote>
encodes both the local statistics of an image (e.g., local texture) and the global structural information of the image (e.g., objects), whereas it is desirable to separate this information, following the essence of latent factors disentanglement.

Subsequent studies (Gulrajani et al., 2016; Gregor et al., 2016; Chen et al., 2017; Lucas and Verbeek, 2018; Shang et al., 2018) considered mixing the VAE latent representation with a more sophisticated decoder exploiting local pixel correlations with either convolutive or autoregressive decoding (Oord et al., 2016b; Oord et al., 2016c), possibly combined with a multilevel or hierarchical latent encoding (see Section 14.3.4). For example, Chen et al. (2017) considered an autoregressive conditional density of the form $p_{\theta_x}(\mathbf{x}|\mathbf{z}) = \prod_i p_{\theta_x}(\mathbf{x}_i|\mathbf{z}, \mathbf{x}_{nb[i]})$ with application to 2D image modeling, where $\mathbf{x}_i$ is the i-th pixel of the image and $\mathbf{x}_{nb[i]}$ are the neighboring pixels. The autoregressive part is typically implemented with an RNN (Oord et al., 2016c). Ideally, the local statistics of an image should be modeled by the autoregressive part, whereas the global structural information of the image should be encoded in $\mathbf{z}$. Another example of a structured VAE for modeling images is VAEs based on convolutional neural networks (CNNs) (Gulrajani et al., 2016; Gregor et al., 2016), which decompose/recompose an image into/from successive feature maps.

Chen et al. (2017) discussed the tendency of the autoregressive part of the model to capture all information on the data structure and let the latent variable remain unused. This problem is referred to as *latent variable vanishing* or as *posterior collapse* in the literature, a term that we have already encountered in the preceding chapter. A general strategy to counter this effect (i.e., controlling the data features encoded by the RNN and the data features encoded in $\mathbf{z}$) is proposed by Chen et al. (2017) at an early level of the model design: The local autoregressive window is constrained to be small, weakening the modeling power of the decoder. This can also be done with a hierarchical structure of the latent space (see the next subsection), possibly combined with the different levels of image feature maps in CNNs (Gulrajani et al., 2016;

<footnote>being an entry of this vector, and the conventional “vector” VAE model, as described in Section 2, is applied. Therefore, each pixel is modeled independently conditioned on $\mathbf{z}$, even though all pixels are not assumed marginally independent.</footnote>
Gregor *et al.*, 2016), or by introducing in the training procedure an auxiliary loss function that controls which information is captured by **z** and what is left to the autoregressive decoder (Lucas and Verbeek, 2018).

The posterior collapse problem has also been observed and discussed in the context of natural language processing (Bowman *et al.*, 2016; Serban *et al.*, 2016). Here, a sequence of words, individually pre-encoded into word embedding vectors, is encoded into and/or decoded from a single latent vector **z**. In Bowman *et al.'s* (2016) paper, both the encoder and decoder are single-layer LSTM RNNs. In this case, the problem is that it is difficult for the latent vector **z** to encode the content of a long input sentence, and again, the RNN internal state vectors tend to encode the whole information, leaving **z** unused. Bowman *et al.* (2016) proposed two strategies to address this problem. The first one is applying annealing to the KL term of the VAE: A weighting factor growing from 0 to 1 is applied progressively to this term during training, first forcing **z** to encode the data information and only then forcing **z** to get disentangled. The second strategy is, as above, a deliberate weakening of the decoder, here by masking a part of the word embedding sequence during training. A more complex strategy was proposed by Yeung *et al.* (2017), where an extra latent variable was added to activate/deactivate certain subvectors of **z**. As only a small part of the latent representation is used at each learning step, the VAE does not need to deactivate some of the dimensions of the latent variable. A heuristic approach was proposed by He *et al.* (2018), where the encoder is aggressively trained (i.e., trained for many iterations) before each training iteration of the decoder. The main intuition behind this approach is that the encoder has difficulties catching up with the changes in the exact posterior distribution and is lagging behind. Aggressively training the encoder allows it to catch up with the evolution of the posterior distribution at each encoder update. Other more recent solutions to the posterior collapse problem in VAEs have been proposed and discussed by Lucas *et al.* (2019), Razavi *et al.* (2019), and Dai *et al.* (2020).

Generally, the solutions to the posterior collapse problem proposed in the literature have yielded a more influential, as well as a more disentangled latent representation. Yet, there is still room for improvement. The
DVAEs focused on in this review do not consider a single latent vector **z** for a data sequence; rather, they consider a latent vector sequence **z**<sub>1:T</sub>, which is generally synchronized with the data sequence **x**<sub>1:T</sub> and with the sequence(s) of internal state vectors of the temporal models. This raises new issues and challenges, compared to the studies conducted on, for example, 2D image or language/text modeling. However, an important remark that is worth mentioning, although quite trivial, is that this DVAE configuration first solves the encoding capacity problem for large data sequences. As mentioned by Li and Mandt (2018),

"[the model] keeps track of the time-varying aspects of $\mathbf{x}_t$ in $\mathbf{z}_t$ for every $t$, making the reconstruction to be time-local and therefore much easier. Therefore, the stochastic model is better suited if the sequences are long and complex."

In short, with DVAEs, it is quite unlikely that a posterior collapse finds its origin in the limited capacity of the latent vector. In fact, we conjecture that it may be the opposite (i.e., a too large capacity of the latent vector sequence $\mathbf{z}_{1:T}$, depending on the size of $\mathbf{z}_t$) that leads to posterior collapse. This is suggested by what we observed in our experiments in Sections 13.3.4 and 13.4.4 with the autoregressive models, where inactive entries of $\mathbf{z}_{1:T}$ might be considered as “superfluous” components. Therefore, adjusting the dimension of $\mathbf{z}_t$ so that it can optimally fit to the content of the observed data sequence (i.e., adjusting the “coding cost” of the latent representation) while limiting the computational complexity is a major issue in DVAEs. All that being said, in autoregressive DVAE models such as SRNN or VRNN, even if we have a “high-capacity” sequence of latent variables, there is no guarantee that the autoregressive part of the model will not capture most of the information, thus ignoring the sequence of latent vectors. This remains an open problem in the DVAE literature.

### 14.3.2 Improved inference models and algorithms

As shown in Chapter 2, learning in the VAE framework relies on amortized variational inference techniques. In fixed-form variational inference (Honkela et al., 2010; Salimans and Knowles, 2013) and, in particular,
stochastic variational inference (Hoffman et al., 2013), the approximate posterior distribution is fixed to a certain parametric form, say Gaussian for instance, and its mean and variance parameters are “freely” optimized through direct maximization of the VLB. In amortized variational inference, the approximate posterior distribution is still Gaussian, but there is an additional constraint imposed by the fact that its parameters are provided by an inference model corresponding to the encoder network in the VAE case. This is an additional constraint in the definition of the variational family, and the resulting amortized approximate posterior distribution is generally less expressive than its counterpart with free parameters. Consequently, the KL divergence between the approximate and exact posterior distributions is generally increased in the amortized variational inference setting, which is referred to as the amortization gap (Cremer et al., 2018; Krishnan et al., 2018). This issue can also limit the performance of the learned generative model, as the amortization gap is directly related to the gap between the VLB and the intractable log-marginal likelihood of the data, which is the criterion that we would ideally like to optimize to learn the generative model parameters (i.e., the parameters of the VAE decoder).

To reduce this gap, several studies have proposed resorting to more sophisticated inference models. Normalizing flow (Rezende and Mohamed, 2015) builds arbitrarily complex approximate posterior distributions with tractable densities by applying a series of invertible transformations to a simple initial distribution. Various normalizing flows have been proposed in the literature, for instance, based on autoregressive models (Kingma et al., 2016; Chen et al., 2017). Because a normalizing flow consists in chaining multiple transformations of an initial latent variable, it can be considered a particular case of a hierarchical model (Kingma and Welling, 2019; Kingma et al., 2016), a type of model that we will discuss in Section 14.3.4. An expressive approximate posterior distribution can also be defined as a mixture of simpler distributions by introducing auxiliary latent variables in the approximate posterior itself and then marginalizing (Maaløe et al., 2016; Ranganath et al., 2016; Salimans et al., 2015).

An alternative to the design of more sophisticated inference models consists in directly modifying the inference algorithm. Marino et al.
(2018a), proposed an iterative amortized inference technique, which consists in iteratively estimating the approximate posterior parameters using a parametric *iterative inference model* that takes as input the current estimate of the parameters, the approximate gradient of the VLB (w.r.t. the approximate posterior parameters), and potentially the observed data. This iterative inference model can, for instance, be defined using a neural network. Similarly to the “learning to learn” principle (Andrychowicz et al., 2016), iterative inference models learn to perform optimization of the VLB for approximate posterior estimation. Moreover, through the encoding of the VLB gradient, iterative inference models naturally account for the top-down information obtained from the data and bottom-up information obtained from the prior to estimate the approximate posterior distribution. This feature complies with the fundamental principle of the Bayes rule, in contrast to the standard inference models, which are purely bottom-up, by simply mapping the observed data to the approximate posterior. With the same objective of overcoming the limitations of standard amortized variational inference, the semi-amortized VAEs proposed by Kim et al. (2018) use a standard inference model (i.e., an encoder network) to provide an initial estimate of the approximate posterior parameters and then run stochastic variational inference (Hoffman et al., 2013) to refine them.

Of particular relevance to DVAEs, the amortized variational filtering algorithm proposed by Marino *et al.* (2018b) generalizes iterative inference models (Marino *et al.*, 2018a) to a general class of dynamical latent variable models for sequential data processing. This algorithm is a general method for performing causal variational inference, using only past and present observed data. When combined with DNNs, the considered general class of dynamical latent variable models corresponds to the DVAE class. The proposed inference method is thus applicable to DVAEs, and the authors conducted experiments using VRNN and SRNN (among other models). An interesting feature of this method is its versatility. In the context of standard amortized variational inference, the DVAE inference model should be designed in accordance with the form of the DVAE generative model, following, for instance, the structure of the exact posterior distribution which can be identified using D-separation (see Chapter 4). In contrast, the
amortized variational filtering algorithm is agnostic of the form of the generative model. Another strength of this algorithm is inherited from iterative inference models (Marino et al., 2018a), which combine information from both the data and the prior to compute the approximate posterior distribution parameters. This principle is also applicable in the context of the amortized variational filtering algorithm, where the “prior” (e.g., $p(\mathbf{z}_t|\mathbf{x}_{1:t-1}, \mathbf{z}_{1:t-1})$) and the approximate posterior (e.g., $q(\mathbf{z}_t|\mathbf{x}_{1:t}, \mathbf{z}_{1:t-1})$) vary in time. The resulting amortized variational filtering algorithm thus resembles classical Bayesian inference filtering methods, such as a Kalman filter, where at a given time instant, the posterior distribution is computed by updating the predictive distribution (involving the prior distribution at the current time instant and the posterior distribution at the previous time instant) using the current observation, as discussed in Section 3.2.2.

### 14.3.3 Disentanglement of latent factors

A common crucial issue for VAEs and DVAEs is how to ensure the disentanglement of latent factors. As stated by Chen et al. (2017),

> “A key goal of representation learning is to identify and disentangle the underlying causal factors of the data, so that it becomes easier to understand the data, to classify it, or to perform other tasks.”

Such disentanglement is not necessarily natural or efficient in the standard VAE; it somehow has to be “encouraged,” either in the model design or in the training procedure (or both).

#### Semi-supervised VAEs

Siddharth et al. (2017) proposed forcing the disentanglement of $\mathbf{z}$ and thus improving its interpretability by using a small amount of supervision during training. This study does not particularly deal with static or dynamical VAEs, and this weak supervision principle can be applied to both.
## Modification of the loss function

A second strategy to improve the disentanglement of latent factors is to modify the loss function (i.e., the VLB). In this line, Higgins et al. (2017) introduced a weighting factor, denoted $\beta$, to weight the regularization term in (2.21), so that the VLB becomes

$$ \mathcal{L}(\theta, \phi, \beta; \mathbf{X}) = \mathbb{E}_{q_{\phi}(\mathbf{Z}|\mathbf{X})}[\log p_{\theta_x}(\mathbf{X}|\mathbf{Z})] - \beta D_{KL}(q_{\phi}(\mathbf{Z}|\mathbf{X}) \| p_{\theta_z}(\mathbf{Z})). \quad (14.1) $$

A value of $\beta$ larger than 1 favors the KL term; hence, it encourages independence/disentanglement of the latent vector entries, although at the price of lower reconstruction/generation quality. For example, the images reconstructed with a $\beta$-VAE can be slightly blurred compared to those reconstructed with a standard VAE. However, the control of the properties of the objects represented in the image from the latent vectors is improved (Higgins et al., 2017).

Chen et al. (2018) went a step ahead. Starting from the VLB (2.21), they introduced the aggregated posterior $q_{\phi}(\mathbf{z}) = \frac{1}{N_{tr}} \sum_{n=1}^{N_{tr}} q_{\phi}(\mathbf{z}|\mathbf{x}_n)$ and then decomposed the KL term of the VLB (summed over the training data) into a sum of three terms. The first one, referred to as the index-code mutual information, quantifies the mutual information between data and latent variables. The second one, referred to as the total correlation, is the KL divergence between the aggregated posterior and the product of its marginals (i.e., $D_{KL}(q_{\phi}(\mathbf{z}) \| \prod_{l=1}^{L} q_{\phi}(z_l))$). It quantifies the independence of the latent vector entries independently of data inputs (i.e., marginal independence as opposed to conditional independence). The third term is the sum over entries of the entry-wise KL divergence between the marginal aggregated posterior $q_{\phi}(z_l)$ and prior $p(z_l)$. The authors noted that minimizing the KL term of the VAE encourages the independence of the latent vector entries through minimization of the total correlation. However, it also penalizes the mutual information between the data and latent vectors, hence decreasing the power of latent components to explain the data. Therefore, they proposed applying a weighting factor (larger than 1) to the total correlation only, leaving the mutual information term unchanged. They experimentally demonstrated the advantage of this strategy over the
β-VAE. A similar idea was proposed independently by Kim and Mnih (2018), with a slightly different decomposition of the VLB KL term and a different implementation (based on an adversarial training of the model). For an extensive discussion and benchmark on disentangled representation learning with VAEs, see Locatello et al. (2020).

Such a general principle of enforcing latent factor disentanglement by modifying the loss function is independent of the issue of “static” against temporal modeling. It can thus, in principle, be applied to the DVAE framework. Finding a relevant decomposition of the loss function in the DVAE framework is still an open topic. Due to the more complex (temporal) intrications of the observed and latent variables, it is difficult to say if terms equivalent to total correlation or mutual information can be evidenced easily. Future studies should consider this aspect to make DVAE models more controllable and interpretable.

### 14.3.4 Hierarchical VAEs and DVAEs

A structured VAE, or hierarchical VAE, is a general subclass of VAEs where the latent space is structured by setting a hierarchical prior distribution on a set of latent variables $z = \{z_0, z_1, ..., z_K\}$ (Kingma and Welling, 2019, Chapter 4) (Salimans, 2016; Sønderby et al., 2016b; Sønderby et al., 2016a). Here, the index denotes different latent variables, not a sample in a training set or a time index in a sequence as before. For example, a hierarchical “multilevel” VAE was proposed by Bouchacourt et al. (2018), with two latent vectors defined at different data scales: One latent vector encodes a common content for a group of data and the other latent vector encodes the style of subgroups of data within a group. In Bouchacourt et al.’s (2018) paper, data grouping involves a certain amount of supervision during training.

Another notable example of a hierarchical VAE was presented by (Salimans, 2016), who proposed to use a (deep) first-order autoregressive prior model:

$$p_{\theta_z}(\mathbf{z}) = p_{\theta_z}(\mathbf{z}_0) \prod_{k=1}^{K} p_{\theta_z}(\mathbf{z}_k|\mathbf{z}_{k-1}). \qquad (14.2)$$

In such an approach, the latent space is structured but not the data
space (we still have a unique observed vector x). In this hierarchical latent variable model, there is no notion of time, but if we think of timely ordered variables, then we move toward Markov models (in the above example, a first-order one). In addition, if we consider both timely ordered latent variables and timely ordered observed variables, we step into the world of DVAEs. In other words, a DVAE can be considered a particular case of structured/hierarchical VAEs, with timely ordered latent and observed variables.

At this point, we can make an interesting parallel among the DVAE papers that we have reviewed and the hierarchical VAE papers (Salimans, 2016; Sønderby et al., 2016b; Sønderby et al., 2016a; Kingma and Welling, 2019) concerning the design of the inference model. We have seen in Section 4.2.1 that the D-separation methodology was not systematically used in the design of DVAE inference models. Interestingly and quite surprisingly, this methodology is not mentioned either in the above hierarchical VAE papers. Yet, we recall that D-separation is a major principled way to guide the design of inference models, including in this more general case. In these papers, the authors rather chose one model among different somewhat intuitive structures (e.g., top-down against bottom-up inference (Kingma and Welling, 2019)). One strategy to make the choice is to favor an inference model with variable dependencies that “mirror” those of the generative model, so that some “module” and parameters can be shared between them. This is a suitable feature that we have discussed for DVAEs in Section 4.2.3 and was applied in the VRNN model for example. In the hierarchical VAE papers, this “module sharing” strategy reportedly led to faster training and better fitting of model and data.

That being said, module sharing is not incompatible with respecting the exact posterior distribution structure. For example, in Salimans’s (2016) paper, the inference model is defined by

$$q_{\phi}(\mathbf{z}|\mathbf{x}) = q_{\phi}(\mathbf{z}_0|\mathbf{x}) \prod_{k=1}^{K} q_{\phi}(\mathbf{z}_k|\mathbf{z}_{k-1}, \mathbf{x}). \quad (14.3)$$

Again, it was chosen by the author because it “mirrors” the generative model. This inference model follows the structure of the exact posterior distribution, even though this fundamental latter point was not men-
tioned by Salimans (2016). In the design of a DVAE, we can apply this
principle: We can look for an inference model that both respects the
structure of the exact posterior and shares some module(s) with the
generative model.

In DVAEs, the problem of disentangling the factors of data variation takes a new flavor, as different factors of variation can have different dynamics. In this context, one way to address the disentanglement challenge is therefore to apply different levels of hierarchical modeling of the latent factors on the time dimension; that is, we can design models with a different time resolution for different latent variables, which is of course not incompatible with other types of hierarchical models. In particular, one general challenge is to separate the data dynamics (i.e., their temporal trajectories) and other factors of variations that are more constant over time (e.g., speaker identity for speech data, or objects present in the scene for videos).<fnref n="2" /> For example, we have seen in this review the DSAE model (Li and Mandt, 2018) and the FHVAE model (Hsu et al., 2017b), which include latent variables defined at the sequence level, segment level (subsequence of consecutive frames), or frame level. For speech signal modeling, this appears as a promising way to separate the modeling and control of phonetic information, which is defined at the segment or frame level, and speaker/session information, which is defined at the sequence level. A generalization of this approach would be to impose a prior distribution of z₁:T that fits the dynamics of the latent factors to extract, which can be significantly different from the data dynamics.

In general, the issue of separating data dynamics and other factors
of variation is still largely open in the literature on DVAE models with
a sequence of latent vectors. For example, we were surprised to notice
that there are very few experiments and information available on the
explainability of the extracted sequence of latent factors. Experiments
involving swapping of the extracted latent factors across two data
sequences before resynthesizing them were reported by, for example, Hsu
et al. (2017b). These experiments show that for speech signals, speaker

<footnote n="2">²Several papers dealing with disentanglement and separate control of content and dynamics in videos have reported impressive results in an adversarial training framework (Denton and Birodkar, 2017; Villegas et al., 2017; Tulyakov et al., 2018).</footnote>
identity can be exchanged between two sentences while preserving the
same phonetic content, which is a very nice result. Yet, the issues
of disentangling and controlling speech production factors separately
remain largely open. Moreover, basic questions such as the impact of
the size of $z_t$ and $h_t$ on modeling quality and the relevance of extracted
latent factors have been poorly considered so far. For example, for
speech processing, what happens if the size of $z_t$ is reduced to a few
entries, while the size of $h_t$ is kept comparable to that of data $x_t$?

Hsu *et al.* (2017b) indicated that

> “to the best of our knowledge, there has not been any
> attempt to learn disentangled and interpretable representa-
> tions without supervision from sequential data.”

Regarding SRNN (Fraccaro *et al.*, 2016), VRNN (Chung *et al.*, 2015), and SVAE (Johnson *et al.*, 2016), Hsu *et al.* (2017b) said

> “[...] it remains unclear whether independent attributes
> are disentangled in the latent space. Moreover, the learned
> latent variables in these models are not interpretable without
> manually inspecting or using labeled data.”

Hence, the models such as STORN, VRNN, and SRNN provide an
elegant and powerful mathematical and methodological framework for
sequential data representation learning; however, there is still a lot of
work to be done on the disentanglement challenge. Solutions for the
disentanglement of $z_t$ in DVAEs, inspired by or combined with existing
structured or hierarchical models such as the ones presented by Salimans
(2016) and Sønderby *et al.* (2016b) and Sønderby *et al.* (2016a), still
have to be developed.

**14.4 Perspectives on source coding**

Although VAE and DVAE are excellent frameworks for extracting effi-
cient and compact data representations, there are relatively few studies
on their practical application to source coding (i.e., data compression
including quantization and bitrate issues for data transmission or stor-
age). We have mentioned above the problem of **z** “vanishing” or “being
ignored” when a powerful deterministic temporal encoder-decoder is
used, and a few papers have related this problem to the need to better
encode **z**, in the source coding sense, with an information-theoretic
interpretation of a VAE as a lossy coder (Kingma *et al.*, 2016; Chen
*et al.*, 2017). Among the few papers on the practical application of
(D)VAE to data coding, we can mention the ConvDRAW model pro-
posed by Gregor *et al.* (2016), which learns and encodes a hierarchy of
latent variables, resulting in an image lossy compression that performs
similarly to JPEG. Other examples include VQ-VAE, which is a mix of
VAE and vector quantization of **z**, applied to speech coding (Oord *et al.*,
2017; Gârbacea *et al.*, 2019) and video compression with rate-distortion
autoencoders (Habibian *et al.*, 2019).

As for a general approach to source coding based on DVAEs with a
sequence of latent variables, we can mention two recent papers: Lom-
bardo et al. (2019) and Yang et al. (2020). Lombardo et al. (2019)
presented a video codec based on the DSAE model (Li and Mandt,
2018), which we reviewed in Chapter 11. The sequence of latent vectors
z_{1:T} extracted by the DSAE encoder is quantized and transformed into
a binary minimum-length sequence by an arithmetic coder, which ex-
ploits the DSAE dynamical model p_{\theta_z}(z_t|z_{1:t-1}) for entropy coding. The
chaining of inverse operations (i.e., arithmetic decoding, inverse quanti-
zation and DSAE decoder) enables to obtain the decoded data sequence
\hat{x}_{1:T}. The global variable v (see Section 11.1) is encoded separately
with a similar scheme. The resulting video codec is shown to exhibit
rate-distortion performance that is comparable to the state-of-the art
video codecs (such as VP9) on generic video sequences while drastically
improving the performance on video sequences with specialized content
(similar to the content of videos used to train the model). Lombardo
et al. (2019) provide no information on the control of the coded data
sequence quality or that of the bitrate. They only mention that the
arithmetic encoding of z_{1:T} requires a number of iterations. Moreover,
the DSAE model is an SSM-like model; that is, x_t is generated from
z_t "alone," not considering the potential of using x_{t-1} (or its quantized
version) for predicting x_t and encoding it more efficiently.

In contrast, Yang *et al.* (2020) proposed different schemes for en-
coding a data sequence $\mathbf{x}_{1:T}$ through the inference and quantization of
the corresponding sequence of latent vectors $z_{1:T}$, with different options for recurrent connections. One of them, called feedback recurrent autoencoder (FRAE), has recurrent connections at both the encoder and decoder, and a feedback connection from the decoder to encoder. The recurrent connections are reminiscent of the *predictive coding* principle that is classical in source coding theory (Gersho and Gray, 2012). In fact, FRAE can be considered a *nonlinear predictive coding* scheme, in which the encoder forms a latent code that encodes only the residual information that is missing when reconstructing a data vector from the deterministic internal state, which depends on past data vectors. This concept of predictive coding is strongly related to that of the predictive mode for the DVAE models that we discussed in general terms in Section 4.1.1 and that we have seen implemented in different (autoregressive) DVAE models. Therefore, from this viewpoint, FRAE is strongly related to STORN, VRNN, and SRNN. The feedback connection from the decoder to encoder is reminiscent of another classical principle of source coding –*closed-loop coding* (Gersho and Gray, 2012)– even though Yang *et al.* (2020) did not refer to it explicitly. In short, closed-loop coding enables the decoder to use the quantized previous data vectors in place of the unquantized ones (not available at the decoder) for predicting the current data vector.

This line of research on nonlinear predictive coders based on DVAEs is quite promising and is only at its infancy. As Yang *et al.* (2020) wrote

> “There is no standard autoencoder architecture for temporally correlated data that has variable-length and long range dependencies such as video, speech, and text. The main challenge lies in the difficulty in capturing correlation information at different time-scales in an online/sequential fashion.”

This agrees with the concluding remark of Chen *et al.* (2017):

> “We believe it’s exciting to extend this principle of learning lossy codes [of the latent variable $z$] to other forms of data, in particular those that have a temporal aspect like audio and video.”
<figure><img src="image_35.png" /></figure>
Acknowledgements

This work was partially funded by the Multidisciplinary Institute in
Artificial Intelligence MIAI@Grenoble-Alpes (ANR-19-P3IA-0003), the
ANR ML3RI project (ANR-19-CE33-0008-01), and the H2020 SPRING
project (under GA #871245).
<figure><img src="image_36.png" /></figure>
# Appendices
<figure><img src="image_37.png" /></figure>
Marginalization of $h_{1:T}$ in STORN

In this appendix, we present how the internal state vector sequence $h_{1:T}$ can be “marginalized” in a DVAE model formulation; that is, how we can express $h_t$ as a deterministic function of the other random variables (and thus move from the developed form of the model to the compact form). This is presented for the STORN model, but a similar derivation can be obtained for the other models as well.

For conciseness, we replace here (7.1) with the generic notation $h_t = f_h(x_{t-1}, z_t, h_{t-1})$. From the dependencies between the different variables, represented by the graphical model in Figure 7.1, the joint distribution between all variables is given by

$$p_{\theta}(\mathbf{x}_{1:T}, \mathbf{h}_{1:T}, \mathbf{z}_{1:T}) = \prod_{t=1}^{T} p_{\theta}(\mathbf{x}_{t}|\mathbf{h}_{t})p_{\theta}(\mathbf{h}_{t}|\mathbf{x}_{t-1}, \mathbf{z}_{t}, \mathbf{h}_{t-1})p(\mathbf{z}_{t}). \quad (A.1)$$

As $h_t$ is a deterministic function of $x_{t-1}$, $z_t$, and $h_{t-1}$, its conditional density is a Dirac distribution with a mode given by $f_h(x_{t-1}, z_t, h_{t-1})$:

$$p_{\theta}(\mathbf{h}_{t}|\mathbf{x}_{t-1}, \mathbf{z}_{t}, \mathbf{h}_{t-1}) = \delta(\mathbf{h}_{t}; f_{h}(\mathbf{x}_{t-1}, \mathbf{z}_{t}, \mathbf{h}_{t-1})). \quad (A.2)$$

Let us denote with $d_{1:T}$ the sequence $h_{1:T}$ considered a (deterministic) function of $x_{1:T}$ and $z_{1:T}$ only; that is, at each time step, we have $d_t = h_t = h_t(x_{1:t-1}, z_{1:t}) = f_h(x_{t-1}, z_t, f_h(x_{t-2}, z_{t-1}, ...))$, with a recursive
injection of the recurrent terms into this latter expression up to the first term $f_h(\mathbf{x}_0, \mathbf{z}_1, \mathbf{h}_0)$. Marginalizing (A.1) with respect to $\mathbf{h}_{1:T}$ leads to

$$p_{\theta}(\mathbf{x}_{1:T}, \mathbf{z}_{1:T}) = \int_{\mathcal{R}^{H \times T}} \prod_{t=1}^{T} p_{\theta}(\mathbf{x}_t | \mathbf{h}_t) \delta(\mathbf{h}_t; f_h(\mathbf{x}_{t-1}, \mathbf{z}_t, \mathbf{h}_{t-1})) p(\mathbf{z}_t) d\mathbf{h}_{1:T} \quad (A.3)$$

$$= \prod_{t=1}^{T} p_{\theta}(\mathbf{x}_t | \mathbf{d}_t) p(\mathbf{z}_t). \quad (A.4)$$

To move from (A.3) to (A.4), one can start by marginalizing over $\mathbf{h}_T$, so that $\mathbf{h}_T$ is replaced with $f_h(\mathbf{x}_{T-1}, \mathbf{z}_T, \mathbf{h}_{T-1})$, and then marginalizing over $\mathbf{h}_{T-1}$, and so on. Hereinafter, for simplification of notations, we identify $\mathbf{d}_t$ with $\mathbf{h}_t$, but we must keep in mind that when doing so, we see $\mathbf{h}_t$ as a deterministic function of $\mathbf{x}_{1:t-1}$ and $\mathbf{z}_{1:t}$ with the recurrence being “unfolded,” and not as a free random variable. Thus, we have

$$p_{\theta}(\mathbf{x}_{1:T}, \mathbf{z}_{1:T}) = \prod_{t=1}^{T} p_{\theta}(\mathbf{x}_t | \mathbf{h}_t) p(\mathbf{z}_t) = \prod_{t=1}^{T} p_{\theta}(\mathbf{x}_t | \mathbf{h}_t(\mathbf{x}_{1:t-1}, \mathbf{z}_{1:t})) p(\mathbf{z}_t). \quad (A.5)$$

From the above equation and (7.7), we deduce the following conditional distribution:

$$p_{\theta}(\mathbf{x}_{1:T} | \mathbf{z}_{1:T}) = \prod_{t=1}^{T} p_{\theta}(\mathbf{x}_t | \mathbf{h}_t) = \prod_{t=1}^{T} p_{\theta}(\mathbf{x}_t | \mathbf{h}_t(\mathbf{x}_{1:t-1}, \mathbf{z}_{1:t})). \quad (A.6)$$

We insist that, in the above equations, $\mathbf{h}_{1:T}$ is to be considered as the set of vectors $\{\mathbf{h}_t(\mathbf{x}_{1:t-1}, \mathbf{z}_{1:t})\}_{t=1}^T$ and not as a free random variable.
DVAE implementation with speech data

In this section, we provide the complete specifications of the DVAE
models used in our experiments with the speech data. For each DVAE
model, we will first present the generation network (decoder) and then
the inference network (encoder). As many of the networks are MLPs, we
define a notation to refer to these architectures concisely: MLP(y, n₁, f₁,
..., nL, fL) refers to an L-layer MLP with input y, and nℓ and fℓ denote
the output dimension and the (element-wise) activation function of the
ℓ-th layer, respectively. The possible activation functions are ReLU,
Sigmoid, hyperbolic tangent (Tanh), and linear I. This latter activation
function is always used for the last layer of the networks computing
the parameters of the random variables xℓ or zt, whether they are the
output of an MLP or of an RNN. Therefore, it will not be made explicit
for conciseness. For example, if we define the generative network for zt
as MLP(ht, 64, Sigmoid, 32, Sigmoid), this means that we use an MLP
with two hidden layers of dimension 64 and 32, respectively, both with
Sigmoid activation function, and an output layer of dimension 2 × 16
(for mean and log-variance vectors, which are both the same size as zt)
with linear activation.
## B.1 DKF

**Generation:** Following Krishnan et al. (2017), we use a gated transition function to implement $d_z$ in (3.9):

$$ \nu_t = \text{MLP}(z_{t-1}, 16, \text{ReLU}, 16, \text{Sigmoid}) \quad (B.1) $$

$$ \mu_t^{\text{nonlin}} = \text{MLP}(z_{t-1}, 16, \text{ReLU}, 16, \mathbb{I}) \quad (B.2) $$

$$ \mu_t^{\text{lin}} = \text{MLP}(z_{t-1}, 16, \mathbb{I}) \quad (B.3) $$

$$ \mu_{\theta_z}(z_{t-1}) = (1 - \nu_t) \odot \mu_t^{\text{lin}} + \nu_t \odot \mu_t^{\text{nonlin}} \quad (B.4) $$

$$ \sigma_{\theta_z}^2(z_{t-1}) = \text{MLP}(\text{ReLU}(\mu_t^{\text{nonlin}}), 16, \text{Softplus}), \quad (B.5) $$

where $\odot$ denotes element-wise multiplication. $\mu_{\theta_z}(z_{t-1})$ is a gated combination of a linear and a nonlinear estimate of the mean vector. The nonlinear estimate is also used to compute the variance $\sigma_{\theta_z}^2(z_{t-1})$.

As for the generation of $x_t$, the function $d_x(z_t)$ in (3.11) is implemented with an MLP($z_t$, 32, Tanh, 64, Tanh, 128, Tanh, 256, Tanh).

**Inference:** We implement the DKS inference model proposed by Krishnan et al. (2017), which follows equations (5.4)–(5.7), where (5.5) is the combiner function shown in Figure 13.1 (a). The function $e_{\bar{z}}$ is implemented using a backward LSTM fed with an MLP($x_t$, 256, Tanh). The affine function of $z$ in (5.5) is implemented with a one-layer MLP($z_{t-1}$, 32, Tanh), and the function $e_z(g_t)$ is implemented with an MLP($g_t$, 32, Tanh).

## B.2 STORN

**Generation:** We first recall that in STORN, the prior distribution of $z_{1:T}$ is an i.i.d. standard Gaussian distribution. Therefore, $z_{1:T}$ can be sampled without requiring a dedicated network. As for $h_t$, the function $d_h(x_{t-1}, z_t, h_{t-1})$ in (7.4) is implemented with the concatenation of an MLP($x_{t-1}$, 256, Tanh) and an MLP($z_t$, 32, Tanh, 64, Tanh) (which can both be considered feature extractors), followed by a forward LSTM network. As for $x_t$, the function $d_x(h_t)$ in (7.5) is implemented with an MLP($h_t$, 256, Tanh).
**Inference:** The function $e_g(\mathbf{x}_t, \mathbf{g}_{t-1})$ in (7.15) is implemented with an MLP($\mathbf{x}_t$, 256, Tanh), followed by a forward LSTM network. The function $e_z(\mathbf{g}_t)$ in (7.16) is implemented with an MLP($\mathbf{g}_t$, 64, Tanh, 32, Tanh).

## B.3 VRNN

We recall that, unlike STORN, VRNN employs a shared RNN for inference and generation with an internal state vector **h**<sub>t</sub>. Furthermore, VRNN explicitly introduces feature extractors for **x**<sub>t</sub> and **z**<sub>t</sub> and uses the extracted features to feed the different encoder and decoder modules.

**Feature extraction:** $\varphi_x(\mathbf{x}_t)$ is an MLP($\mathbf{x}_t$, 256, Tanh) and $\varphi_z(\mathbf{z}_t)$ is an MLP($\mathbf{z}_t$, 32, Tanh, 64, Tanh).

**Generation:** The function $d_h(\varphi_x(\mathbf{x}_{t-1}), \varphi_z(\mathbf{z}_{t-1}), \mathbf{h}_{t-1})$ in (8.1) is implemented with an LSTM network with input $[\varphi_x(\mathbf{x}_{t-1}), \varphi_z(\mathbf{z}_{t-1})]$. The function $d_z(\mathbf{h}_t)$ in (8.4) is implemented by directly mapping **h**<sub>t</sub> to the dimension of latent space (i.e. the output layer mentioned in the beginning of this subsection). The function $d_x(\varphi_z(\mathbf{z}_t), \mathbf{h}_t)$ in (8.2) is also implemented with an simple output layer.

**Inference:** The function $e_z(\varphi_x(\mathbf{x}_t), \mathbf{h}_t)$ in (8.11) is still implemented with an output layer where the input is the concatenation of $\varphi_x(\mathbf{x}_t)$ and $\mathbf{h}_t$.

## B.4 SRNN

We recall that, as with VRNN, SRNN shares an internal recurrent state vector **h**<sub>t</sub> between the generation and inference models.

**Generation:** The function $d_h(\mathbf{x}_{t-1}, \mathbf{h}_{t-1})$ in (9.1) is implemented with an LSTM network with input MLP($\mathbf{x}_{t-1}$, 256, Tanh). The function $d_z(\mathbf{z}_{t-1}, \mathbf{h}_t)$ in (9.4) is implemented with an MLP($[\mathbf{z}_{t-1}, \mathbf{h}_t]$, 64, Tanh, 32, Tanh). The function $d_x(\mathbf{z}_t, \mathbf{h}_t)$ in (9.2) is implemented with an MLP($[\mathbf{z}_t, \mathbf{h}_t]$, 256, Tanh).
**Inference:** The function $e_{\leftarrow g}([\mathbf{h}_t, \mathbf{x}_t], \leftarrow \mathbf{g}_{t+1})$ in (9.10) is a backward LSTM network with input MLP$([\mathbf{h}_t, \mathbf{x}_t]$, 256, Tanh). The function $e_z(\mathbf{z}_{t-1}, \leftarrow \mathbf{g}_t)$ in (9.11) is an MLP$([\mathbf{z}_{t-1}, \leftarrow \mathbf{g}_t]$, 64, Tanh, 32, Tanh).

## B.5 RVAE

As STORN, RVAE assumes an i.i.d. standard Gaussian prior for $\mathbf{z}_{1:T}$, so no network is required to generate $\mathbf{z}_t$. We recall that RVAE has causal and noncausal versions depending on whether the generation of $\mathbf{x}_t$ uses $\mathbf{z}_{t+1:T}$ or not, respectively.

**Generation:** Regarding the causal case, the function $d_h(\mathbf{z}_t, \mathbf{h}_{t-1})$ in (10.4) is a forward LSTM network with input $\mathbf{z}_t$. The function $d_x(\mathbf{h}_t)$ in (10.5) is a single output layer. For the noncausal case, the generation of $\mathbf{h}_t$ in (10.9)–(10.11) is implemented with a bidirectional LSTM with the same input as in the causal case, and $d_x(\mathbf{h}_t)$ is also the same.

**Inference:** In the causal case, the function $e_{\leftarrow g}(\mathbf{z}_{t-1}, \leftarrow \mathbf{g}_{t-1})$ in (10.18) is a forward LSTM network with input $\mathbf{z}_{t-1}$. The function $e_{\leftarrow g}(\mathbf{x}_t, \leftarrow \mathbf{g}_{t+1})$ in (10.19) is a backward LSTM network with input $\mathbf{x}_t$. The function $e_z(\mathbf{g}_t)$ in (10.21) is a single output layer. In the noncausal case, the function $e_{\leftarrow g}^z$ in (10.24) follows the same architecture as $e_{\leftarrow g}$ in the causal inference model, whereas the backward LSTM $e_{\leftarrow g}^x$ is replaced with a bidirectional LSTM. The outputs are indicated as $e_{\leftarrow g}^z$ and $e_{\leftarrow g}^x$ in function (10.25) and (10.26).

## B.6 DSAE

We recall that compared to the other models, DSAE has an extra sequence-level latent variable $\mathbf{v}$. We assume that $\mathbf{v}$ has the same dimension as $\mathbf{z}_t$. As the total dimension of $\mathbf{z}_{1:T}$ is $T$ times that of $\mathbf{z}_t$, introducing this extra latent variable $\mathbf{v}$ will not change the total number of latent variables much. Therefore, we can still consider that it is fair to compare DSAE to the other models in such a configuration. The generation of $\mathbf{v}$ is not detailed in the original paper, and we assume it follows a standard Gaussian distribution.
**Generation:** The function $d_h(\mathbf{z}_{t-1}, \mathbf{h}_{t-1})$ in (11.2) is a forward LSTM network with a hidden layer of dimension 128. The function $d_z(\mathbf{h}_t)$ in (11.3) is a one-layer linear network to project the dimension of $\mathbf{h}_t$ onto the dimension of $\mathbf{z}_t$. The function $d_x(\mathbf{z}_t, \mathbf{v})$ in (11.5) is an MLP($[\mathbf{z}_t, \mathbf{v}]$, 32, Tanh, 64, Tanh, 128, Tanh, 256, Tanh).

**Inference:** As for the inference of $\mathbf{v}$, Eqs. (11.12)–(11.15) are implemented with a bidirectional many-to-one LSTM network with input $\mathbf{x}_t$ and output $\mathbf{g}^v$, followed by a single output layer. As for the inference of $\mathbf{z}_t$, Eqs. (11.17)–(11.19) are implemented with a bidirectional LSTM with input $[\mathbf{v}, \mathbf{x}_t]$. Finally, the function $e_z(\mathbf{g}_t^z)$ in (11.20) is an RNN with a hidden layer of dimension 128.<fnref n="1" />

<footnote n="1">¹In Figure 13.1 (f), we plot an MLP block after $\mathbf{x}_t$ and another one before the BRNN to output $\overrightarrow{\mathbf{g}}_t^z$ and $\overleftarrow{\mathbf{g}}_t^z$, whereas in the implementation we do not use them, or they can be considered as an identity layer. We made this choice because it provides better performance than applying several dense layers in our experiments, but we keep the possibility to use dense layers in our open-source code.</footnote>
<figure><img src="image_38.png" /></figure>
DVAE implementation with 3D human motion
data

In this section, we provide the complete specifications of the DVAE
models used in our experiments with the 3D human motion data. The
notations have been defined in the previous section. We retain the
general same architectures as for the speech data, because the inputs
are still sequences of 1D vectors. Considering that the dimension of
human pose vectors is smaller than that of speech vectors (96 vs 513),
we reduce the dimension of the latent variable z_t (and also v in DSAE)
to 10 and use more lightweight LSTM networks with hidden state of
dimension 64.

C.1 DKF

**Generation:** The gated transition function for motion data is the same as for speech data, except that the hidden dimension for all MLP in (B.1) - (B.5) is reduced to 10. As for the generation of $x_t$, the function $d_x(z_t)$ in (3.11) is implemented with an MLP($z_t$, 32, Tanh, 64, Tanh).

**Inference:** Similar to the implementation for speech data, the function $e_{\bar{z}}$ is implemented using a backward LSTM fed with an MLP($x_t$, 64, Tanh), the affine function of $z$ in (5.5) is replaced with a one-layer
MLP(**z**<sub>*t*−1</sub>, 16, Tanh) and the function *e*<sub>z</sub>(**g**<sub>*t*</sub>) is implemented with an
MLP(**g**<sub>*t*</sub>, 32, Tanh).

C.2 STORN

**Generation:** For **h**<sub>*t*</sub>, the function *d*<sub>**h**</sub>(**x**<sub>*t*−1</sub>, **z**<sub>*t*</sub>, **h**<sub>*t*−1</sub>) in (7.4) is imple-
mented with the concatenation of an MLP(**x**<sub>*t*−1</sub>, 64, Tanh) and an
MLP(**z**<sub>*t*</sub>, 32, Tanh), followed by a forward LSTM network. As for **x**<sub>*t*</sub>,
the function *d*<sub>**x**</sub>(**h**<sub>*t*</sub>) in (7.5) is the output layer.

**Inference:** The function $e_g(\mathbf{x}_t, \mathbf{g}_{t-1})$ in (7.15) is implemented with an MLP($\mathbf{x}_t$, 64, Tanh), followed by a forward LSTM network. The function $e_z(\mathbf{g}_t)$ in (7.16) is implemented with an MLP($\mathbf{g}_t$, 32, Tanh).

C.3 VRNN

**Feature extraction:** $\varphi_x(\mathbf{x}_t)$ is an MLP($\mathbf{x}_t$, 64, Tanh) and $\varphi_z(\mathbf{z}_t)$ is an
MLP($\mathbf{z}_t$, 16, Tanh, 32, Tanh).

**Generation:** Same to speech data, the function $d_h(\varphi_x(\mathbf{x}_{t-1}), \varphi_z(\mathbf{z}_{t-1}), \mathbf{h}_{t-1})$ in (8.1) is implemented with an LSTM network with input $[\varphi_x(\mathbf{x}_{t-1}), \varphi_z(\mathbf{z}_{t-1})]$.
The function $d_z(\mathbf{h}_t)$ in (8.4) is implemented by directly mapping $\mathbf{h}_t$ to
the dimension of latent space (i.e. the output layer mentioned in the
beginning of this subsection). The function $d_x(\varphi_z(\mathbf{z}_t), \mathbf{h}_t)$ in (8.2) is
also implemented with an simple output layer.

**Inference:** Same to speech data, the function $e_z(\varphi_x(\mathbf{x}_t), \mathbf{h}_t)$ in (8.11) is still implemented with an output layer where the input is the concatenation of $\varphi_x(\mathbf{x}_t)$ and $\mathbf{h}_t$.

C.4 SRNN

**Generation:** The function $d_h(\mathbf{x}_{t-1}, \mathbf{h}_{t-1})$ in (9.1) is implemented with an LSTM network with input MLP($\mathbf{x}_{t-1}$, 64, Tanh). The function $d_z(\mathbf{z}_{t-1}, \mathbf{h}_t)$ in (9.4) is implemented with an MLP($[\mathbf{z}_{t-1}, \mathbf{h}_t]$, 32, Tanh). The function $d_x(\mathbf{z}_t, \mathbf{h}_t)$ in (9.2) is implemented with an MLP($[\mathbf{z}_t, \mathbf{h}_t]$, 64, Tanh).
**Inference:** The function $e_{\leftarrow g} ([\mathbf{h}_t, \mathbf{x}_t], \leftarrow g_{t+1})$ in (9.10) is a backward LSTM network with input MLP($[\mathbf{h}_t, \mathbf{x}_t]$, 64, Tanh). The function $e_z(\mathbf{z}_{t-1}, \leftarrow g_t)$ in (9.11) is an MLP($[\mathbf{z}_{t-1}, \leftarrow g_t]$, 32, Tanh).

## C.5 RVAE

**Generation:** The implementation is the same as for speech data for both the causal case and the noncausal case, except the different dimension of the LSTM hidden state vector.

**Inference:** The implementation is the same as for speech data, except the different dimension of the LSTM hidden state vector.

## C.6 DSAE

**Generation:** The function $d_x(\mathbf{z}_t, \mathbf{v})$ in (11.5) is an MLP($[\mathbf{z}_t, \mathbf{v}]$, 32, Tanh, 64, Tanh). The others are the same as for speech data, except different hidden dimension of LSTM.

**Inference:** Same as for speech data, except different hidden dimension of LSTM.
<figure><img src="image_39.png" /></figure>
Abadi, M., A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. (2016). “Tensorflow: Large-scale machine learning on heterogeneous distributed systems”. *arXiv preprint arXiv:1603.04467*.

Aksan, E. and O. Hilliges. (2019). “STCN: Stochastic temporal convolutional networks”. In: *International Conference on Learning Representations (ICLR)*. New Orleans, LA.

Andrychowicz, M., M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas. (2016). “Learning to learn by gradient descent by gradient descent”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Barcelona, Spain.

Archer, E., I. M. Park, L. Buesing, J. Cunningham, and L. Paninski. (2015). “Black box variational inference for state space models”. *arXiv preprint arXiv:1511.07367*.

Babaeizadeh, M., C. Finn, D. Erhan, R. H. Campbell, and S. Levine. (2018). “Stochastic variational video prediction”. In: *International Conference on Learning Representations (ICLR)*. Vancouver, Canada.

Bai, S., Z. Kolter, and V. Koltun. (2018). “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling”. *arXiv preprint arXiv:1803.01271*.
Bando, Y., M. Mimura, K. Itoyama, K. Yoshii, and T. Kawahara. (2018). "Statistical speech enhancement based on probabilistic integration of variational autoencoder and non-negative matrix factorization". In: *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Calgary, Canada.

Bayer, J. and C. Osendorfer. (2014). "Learning stochastic recurrent networks". *arXiv preprint arXiv:1411.7610*.

Bengio, S., O. Vinyals, N. Jaitly, and N. Shazeer. (2015). "Scheduled sampling for sequence prediction with recurrent neural networks". *arXiv preprint arXiv:1506.03099*.

Bie, X., S. Leglaive, X. Alameda-Pineda, and L. Girin. (2021). "Unsupervised speech enhancement using dynamical variational autoencoders". *arXiv preprint arXiv:2106.12271*.

Bishop, C. (2006). *Pattern Recognition and Machine Learning*. Springer.

Bitton, A., P. Esling, and T. Harada. (2020). "Neural granular sound synthesis". *arXiv preprint arXiv:2008.01393*.

Blaauw, M. and J. Bonada. (2016). "Modeling and transforming speech using variational autoencoders". In: *Conference of the International Speech Communication Association (Interspeech)*. San Francisco, CA.

Blei, D. M., A. Kucukelbir, and J. D. McAuliffe. (2017). "Variational inference: A review for statisticians". *Journal of the American Statistical Association*. 112(518): 859–877.

Bogo, F., A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. J. Black. (2016). "Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image". In: *European Conference on Computer Vision (ECCV)*. Amsterdam, The Netherlands.

Bottou, L. (2004). "Stochastic Learning". In: *Advanced Lectures on Machine Learning: ML Summer Schools 2003, Canberra, Australia*. 146–168.

Bouchacourt, D., R. Tomioka, and S. Nowozin. (2018). "Multi-level variational autoencoder: Learning disentangled representations from grouped observations". In: *AAAI Conference on Artificial Intelligence*. New Orleans, LA.
Boulanger-Lewandowski, N., Y. Bengio, P. Vincent, P. Gray, and C. Naguri. (2012). "Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription". In: *International Conference on Machine Learning (ICML)*. Edinburgh, Scotland.

Bowman, S. R., L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. (2016). "Generating sentences from a continuous space". *International Conference on Computational Natural Language Learning (CoNLL)*.

Chen, R. T., X. Li, R. Grosse, and D. Duvenaud. (2018). "Isolating sources of disentanglement in variational autoencoders". *arXiv preprint arXiv:1802.04942*.

Chen, X., D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel. (2017). "Variational lossy autoencoder". In: *International Conference on Learning Representations (ICLR)*. Toulon, France.

Cho, K., B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. (2014). "Learning phrase representations using RNN encoder-decoder for statistical machine translation". *arXiv preprint arXiv:1406.1078*.

Chung, J., K. Kastner, L. Dinh, K. Goel, A. Courville, and Y. Bengio. (2015). "A recurrent latent variable model for sequential data". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Montréal, Canada.

Cremer, C., X. Li, and D. Duvenaud. (2018). "Inference suboptimality in variational autoencoders". In: *International Conference on Machine Learning (ICML)*. Stockholm, Sweden.

Dai, B., Z. Wang, and D. Wipf. (2020). "The usual suspects? Reassessing blame for VAE posterior collapse". In: *International Conference on Machine Learning (ICML)*. Virtual online.

Daum, F. (2005). "Nonlinear filters: Beyond the Kalman filter". *IEEE Aerospace and Electronic Systems Magazine*. 20(8): 57–69.

Dayan, P., G. E. Hinton, R. M. Neal, and R. S. Zemel. (1995). "The Helmholtz machine". *Neural Computation*. 7(5): 889–904.
Dempster, A. P., N. M. Laird, and D. B. Rubin. (1977). "Maximum likelihood from incomplete data via the EM algorithm". *Journal of the Royal Statistical Society. Series B (Methodological)*. 39(1): 1-38.

Deng, Z., R. Navarathna, P. Carr, S. Mandt, Y. Yue, I. Matthews, and G. Mori. (2017). "Factorized variational autoencoders for modeling audience reactions to movies". In: *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*. Honolulu, HW.

Denton, E. and V. Birodkar. (2017). "Unsupervised learning of disentangled representations from video". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Long Beach, CA.

Denton, E. and R. Fergus. (2018). "Stochastic video generation with a learned prior". In: *International Conference on Machine Learning (ICML)*. Stockholm, Sweden.

Diggle, P. J. and R. J. Gratton. (1984). "Monte Carlo methods of inference for implicit statistical models". *Journal of the Royal Statistical Society: Series B (Methodological)*. 46(2): 193-212.

Durbin, J. and S. J. Koopman. (2012). *Time series analysis by state space methods*. Oxford University Press.

Esling, P., A. Chemla-Romeu-Santos, and A. Bitton. (2018). "Bridging audio analysis, perception and synthesis with perceptually-regularized variational timbre space". In: *International Society for Music Information Retrieval Conference (ISMIR)*. Paris, France.

Fabius, O. and J. R. van Amersfoort. (2014). "Variational recurrent auto-encoders". *arXiv preprint arXiv:1412.6581*.

Finn, C., I. Goodfellow, and S. Levine. (2016). "Unsupervised learning for physical interaction through video prediction". *Advances in Neural Information Processing Systems (NeurIPS)*.

Fortuin, V., D. Baranchuk, G. Rätsch, and S. Mandt. (2020). "GP-VAE: Deep probabilistic time series imputation". In: *International Conference on Artificial Intelligence and Statistics (AISTATS)*. Palermo, Italy.

Fox, E., E. B. Sudderth, M. I. Jordan, and A. S. Willsky. (2011). "Bayesian nonparametric inference of switching dynamic linear models". *IEEE Transactions on Signal Processing*. 59(4): 1569-1585.
Fraccaro, M., S. K. Sønderby, U. Paquet, and O. Winther. (2016). "Sequential neural models with stochastic layers". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Barcelona, Spain.

Fraccaro, M., S. Kamronn, U. Paquet, and O. Winther. (2017). "A disentangled recognition and nonlinear dynamics model for unsupervised learning". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Long Beach, CA.

Frey, B. J. (1998). *Graphical models for machine learning and digital communication*. Ed. by T. G. Dietterich. Cambridge, MA: MIT Press.

Gan, Z., C. Li, R. Henao, D. E. Carlson, and L. Carin. (2015). "Deep temporal sigmoid belief networks for sequence modeling". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Montréal, Canada.

Gârbacea, C., A. van den Oord, Y. Li, F. S. Lim, A. Luebs, O. Vinyals, and T. C. Walters. (2019). "Low bit-rate speech coding with VQ-VAE and a WaveNet decoder". In: *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Brighton, UK.

Garofolo, J., D. Graff, D. Paul, and D. Pallett. (1993). "CSR-I (WSJ0) Sennheiser LDC93S6B. https://catalog.ldc.upenn.edu/LDC93S6B". *Philadelphia: Linguistic Data Consortium*.

Geiger, D., T. Verma, and J. Pearl. (1990). "Identifying independence in Bayesian networks". *Networks*. 20(5): 507-534.

Gersho, A. and R. M. Gray. (2012). *Vector quantization and signal compression*. Springer Science & Business Media.

Girin, L., F. Roche, T. Hueber, and S. Leglaive. (2019). "Notes on the use of variational autoencoders for speech and audio spectrogram modeling". In: *Digital Audio Effects Conference (DAFx)*. Birming- ham, UK.

Goodfellow, I., Y. Bengio, and A. Courville. (2016). *Deep Learning*. MIT Press.
Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. (2014). “Generative adversarial nets”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Montréal, Canada.

Goodfellow, I. (2016). “NIPS 2016 tutorial: Generative adversarial networks”. *arXiv preprint arXiv:1701.00160*.

Goyal, A., A. Sordoni, M.-A. Côté, N. R. Ke, and Y. Bengio. (2017). “Z-forcing: Training stochastic recurrent networks”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Long Beach, CA.

Graves, A. (2013). “Generating sequences with recurrent neural networks”. *arXiv preprint arXiv:1308.0850*.

Graves, A., A.-r. Mohamed, and G. Hinton. (2013). “Speech recognition with deep recurrent neural networks”. In: *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Vancouver, Canada.

Gregor, K., F. Besse, D. J. Rezende, I. Danihelka, and D. Wierstra. (2016). “Towards conceptual compression”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Barcelona, Spain.

Gregor, K., I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra. (2015). “DRAW: A recurrent neural network for image generation”. In: *International Conference on Machine Learning (ICML)*. Lille, France.

Gu, S., Z. Ghahramani, and R. E. Turner. (2015). “Neural adaptive sequential Monte Carlo”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Montréal, Canada.

Gulrajani, I., K. Kumar, F. Ahmed, A. A. Taiga, F. Visin, D. Vazquez, and A. Courville. (2016). “PixelVAE: A latent variable model for natural images”. *arXiv preprint arXiv:1611.05013*.

Habibian, A., T. v. Rozendaal, J. M. Tomczak, and T. S. Cohen. (2019). “Video compression with rate-distortion autoencoders”. In: *IEEE International Conference on Computer Vision (ICCV)*. Seoul, Korea.

Hafner, D., T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. (2018). “Learning latent dynamics for planning from pixels”. *arXiv preprint arXiv:1811.04551*.
Hamilton, J. D. (2020). *Time series analysis*. Princeton University Press.

Haykin, S. (2004). *Kalman filtering and neural networks*. John Wiley & Sons.

He, J., D. Spokoyny, G. Neubig, and T. Berg-Kirkpatrick. (2018). “Lagging inference networks and posterior collapse in variational autoencoders”. In: *International Conference on Learning Representations (ICLR)*. Vancouver, Canada.

Higgins, I., L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. (2017). “β-VAE: learning basic visual concepts with a constrained variational framework”. In: *International Conference on Learning Representations (ICLR)*. Toulon, France.

Hinton, G. and R. Salakhutdinov. (2006). “Reducing the dimensionality of data with neural networks”. *Science*. 313(5786): 504–507.

Hinton, G., P. Dayan, B. Frey, and R. Neal. (1995). “The “Wake-Sleep” algorithm for unsupervised neural networks”. *Science*. 268(5214): 1158–1161.

Hochreiter, S. and J. Schmidhuber. (1997). “Long short-term memory”. *Neural Computation*. 9(8): 1735–1780.

Hoffman, M. D., D. M. Blei, C. Wang, and J. Paisley. (2013). “Stochastic variational inference”. *Journal of Machine Learning Research*. 14(1): 1303–1347.

Honkela, A., T. Raiko, M. Kuusela, M. Tornio, and J. Karhunen. (2010). “Approximate Riemannian conjugate gradient learning for fixed-form variational Bayes”. *Journal of Machine Learning Research*. 11(Nov): 3235–3268.

Hsu, W.-N., Y. Zhang, and J. Glass. (2017a). “Learning latent representations for speech generation and transformation”. *arXiv preprint arXiv:1704.04222*.

Hsu, W.-N., Y. Zhang, and J. Glass. (2017b). “Unsupervised learning of disentangled and interpretable representations from sequential data”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Long Beach, CA.

Hu, Z., Z. Yang, X. Liang, R. Salakhutdinov, and E. P. Xing. (2017). “Toward controlled generation of text”. In: *International Conference on Machine Learning (ICML)*. Sydney, Australia.
Huang, K., N. D. Sidiropoulos, and A. P. Liavas. (2016). “A flexible and efficient algorithmic framework for constrained matrix and tensor factorization”. *IEEE Transactions on Signal Processing*. 64(19): 5052–5065.

Ionescu, C., D. Papava, V. Olaru, and C. Sminchisescu. (2014). “Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments”. *IEEE Transactions on Pattern Analysis and Machine Intelligence*. 36(7): 1325–1339.

Jang, M., S. Seo, and P. Kang. (2019). “Recurrent neural network-based semantic variational autoencoder for sequence-to-sequence learning”. *Information Sciences*. 490: 59–73.

Jiang, J., G. Xia, D. Carlton, C. Anderson, and R. Miyakawa. (2020). “Transformer VAE: A hierarchical model for structure-aware and interpretable music representation learning”. In: *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Virtual Barcelona.

Johnson, M. J., D. K. Duvenaud, A. Wiltschko, R. P. Adams, and S. R. Datta. (2016). “Composing graphical models with neural networks for structured representations and fast inference”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Barcelona, Spain.

Jordan, M. I., Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. (1999). “An introduction to variational methods for graphical models”. *Machine Learning*. 37: 183–233.

Kameoka, H., L. Li, S. Inoue, and S. Makino. (2018). “Semi-blind source separation with multichannel variational autoencoder”. *arXiv preprint arXiv:1808.00892*.

Karl, M., M. Soelch, J. Bayer, and P. van der Smagt. (2017). “Deep variational Bayes filters: Unsupervised learning of state space models from raw data”. In: *International Conference on Learning Representations (ICLR)*. Toulon, France.

Kim, H. and A. Mnih. (2018). “Disentangling by factorising”. In: *International Conference on Machine Learning (ICML)*. Stockholm, Sweden.
Kim, Y., S. Wiseman, A. Miller, D. Sontag, and A. Rush. (2018). "Semi-amortized variational autoencoders". In: *International Conference on Machine Learning (ICML)*. Stockholm, Sweden.

Kingma, D. P. and J. Ba. (2014). "Adam: A method for stochastic optimization". *arXiv preprint arXiv:1412.6980*.

Kingma, D. P. and M. Welling. (2014). "Auto-encoding variational Bayes". In: *International Conference on Learning Representations (ICLR)*. Banff, Canada.

Kingma, D. P. and M. Welling. (2019). "An introduction to variational autoencoders". *Foundations and Trends in Machine Learning*. 12(4): 307-392.

Kingma, D. P., T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling. (2016). "Improved variational inference with inverse autoregressive flow". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Barcelona, Spain.

Koller, D. and N. Friedman. (2009). *Probabilistic graphical models: Principles and techniques*. MIT Press.

Krishnan, R., U. Shalit, and D. Sontag. (2015). "Deep Kalman filters". In: *arXiv preprint arXiv:1511.05121*.

Krishnan, R., U. Shalit, and D. Sontag. (2017). "Structured inference networks for nonlinear state space models". In: *AAAI Conference on Artificial Intelligence*. San Francisco, CA.

Krishnan, R., D. Liang, and M. Hoffman. (2018). "On the challenges of learning with inference networks on sparse, high-dimensional data". In: *International Conference on Artificial Intelligence and Statistics*. Lanzarote, Spain.

Kuleshov, V., A. Chaganty, and P. Liang. (2015). "Tensor factorization via matrix factorization". In: *Artificial Intelligence and Statistics*. 507-516.

Le Roux, J., S. Wisdom, H. Erdogan, and J. R. Hershey. (2019). "SDR: Half-baked or well done?" In: *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Brighton, UK.

Le, T. A., M. Igl, T. Rainforth, T. Jin, and F. Wood. (2018). "Auto-encoding sequential Monte Carlo". In: *International Conference on Learning Representations (ICLR)*. Vancouver, Canada.
Lea, C., R. Vidal, A. Reiter, and G. D. Hager. (2016). "Temporal convolutional networks: A unified approach to action segmentation". In: *European Conference on Computer Vision (ECCV)*. Amsterdam, The Netherlands.

Lee, J. Y., S. J. Cheon, B. J. Choi, N. S. Kim, and E. Song. (2018). "Acoustic modeling using adversarially trained variational recurrent neural network for speech synthesis". In: *Conference of the International Speech Communication Association (Interspeech)*. Hyderabad, India.

Leglaive, S., L. Girin, and R. Horaud. (2018). "A variance modeling framework based on variational autoencoders for speech enhancement". In: *IEEE International Workshop on Machine Learning for Signal Processing (MLSP)*. Aalborg, Denmark.

Leglaive, S., L. Girin, and R. Horaud. (2019). "Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization". In: *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Brighton, UK.

Leglaive, S., X. Alameda-Pineda, L. Girin, and R. Horaud. (2020). "A recurrent variational autoencoder for speech enhancement". In: *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Virtual Barcelona.

Li, X., L. Girin, and R. Horaud. (2017). "An EM algorithm for audio source separation based on the convolutive transfer function". In: *IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)*. New Paltz, NJ.

Li, Y. and S. Mandt. (2018). "Disentangled sequential autoencoder". In: *International Conference on Machine Learning (ICML)*. Stockholm, Sweden.

Linderman, S. W., A. C. Miller, R. P. Adams, D. M. Blei, L. Paninski, and M. J. Johnson. (2016). "Recurrent switching linear dynamical systems". *arXiv preprint arXiv:1610.08466*.

Liu, D. and G. Liu. (2019). "A Transformer-based variational autoencoder for sentence generation". In: *IEEE International Joint Conference on Neural Networks (IJCNN)*. Budapest, Hungary.
Locatello, F., S. Bauer, M. Lucic, G. Rätsch, S. Gelly, B. Schölkopf, and O. Bachem. (2020). “A sober look at the unsupervised learning of disentangled representations and their evaluation”. *Journal of Machine Learning Research*. 21: 1–62.

Lombardo, S., J. Han, C. Schroers, and S. Mandt. (2019). “Deep generative video compression”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Vancouver, Canada.

Lucas, J., G. Tucker, R. B. Grosse, and M. Norouzi. (2019). “Don’t blame the ELBO! a linear VAE perspective on posterior collapse”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Vancouver, Canada.

Lucas, T. and J. Verbeek. (2018). “Auxiliary guided autoregressive variational autoencoders”. In: *Joint European Conference on Machine Learning and Knowledge Discovery in Databases*. Dublin, Ireland.

Luo, Y. and N. Mesgarani. (2019). “Conv-Tasnet: Surpassing ideal time-frequency magnitude masking for speech separation”. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*. 27(8): 1256–1266.

Maaløe, L., C. K. Sønderby, S. K. Sønderby, and O. Winther. (2016). “Auxiliary deep generative models”. In: *International Conference on Machine Learning (ICML)*. New York, NY.

Maddison, C. J., J. Lawson, G. Tucker, N. Heess, M. Norouzi, A. Mnih, A. Doucet, and Y. Teh. (2017). “Filtering variational objectives”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Long Beach, CA.

Mao, W., M. Liu, and M. Salzmann. (2020). “History repeats itself: Human motion prediction via motion attention”. In: *European Conference on Computer Vision (ECCV)*. Virtual online.

Marino, J., Y. Yue, and S. Mandt. (2018a). “Iterative amortized inference”. In: *International Conference on Machine Learning (ICML)*. Stockholm, Sweden.

Marino, J., M. Cvitkovic, and Y. Yue. (2018b). “A general method for amortizing variational filtering”. In: *Advances in Neural Information Processing Systems (NeurIPS)*. Montréal, Canada.
Martinez, J., M. J. Black, and J. Romero. (2017). "On human motion prediction using recurrent neural networks". In: *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*. Honolulu, HW.

Mathieu, M. F., J. J. Zhao, J. Zhao, A. Ramesh, P. Sprechmann, and Y. LeCun. (2016). "Disentangling factors of variation in deep representation using adversarial training". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Barcelona, Spain.

Miao, Y., L. Yu, and P. Blunsom. (2016). "Neural variational inference for text processing". In: *International Conference on Machine Learning (ICML)*. New York, NY.

Miladinović, D., M. W. Gondal, B. Schölkopf, J. M. Buhmann, and S. Bauer. (2019). "Disentangled state space representations". *arXiv preprint arXiv:1906.03255*.

Mnih, A. and K. Gregor. (2014). "Neural variational inference and learning in belief networks". *arXiv preprint arXiv:1402.0030*.

Moreno, V. M. and A. Pigazo. (2009). *Kalman filter: Recent advances and applications*. BoD-Books on Demand.

Murphy, K. P. (1998). "Switching Kalman filters". *Unpublished technical report – Available online*.

Murphy, K. P. (2012). *Machine learning: A probabilistic perspective*. MIT Press.

Naesseth, C., S. Linderman, R. Ranganath, and D. Blei. (2018). "Variational sequential Monte Carlo". In: *International Conference on Artificial Intelligence and Statistics*. Lanzarote, Spain.

Neal, R. M. and G. E. Hinton. (1998). "A view of the EM algorithm that justifies incremental, sparse, and other variants". In: *Learning in graphical models*. Springer. 355–368.

Neeser, F. D. and J. L. Massey. (1993). "Proper complex random processes with applications to information theory". *IEEE Transactions on Information Theory*. 39(4): 1293–1302.

Oord, A. van den, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. (2016a). "Wavenet: A generative model for raw audio". *arXiv preprint arXiv:1609.03499*.
Oord, A. van den, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. (2016b). "Conditional image generation with PixelCNN decoders". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Barcelona, Spain.

Oord, A. van den, N. Kalchbrenner, and K. Kavukcuoglu. (2016c). "Pixel recurrent neural networks". *International Conference on Machine Learning (ICML)*.

Oord, A. van den, O. Vinyals, and K. Kavukcuoglu. (2017). "Neural discrete representation learning". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Long Beach, CA.

Papoulis, A. (1977). *Signal analysis*. McGraw-Hill New York.

Pariente, M., A. Deleforge, and E. Vincent. (2019). "A statistically principled and computationally efficient approach to speech enhancement using variational autoencoders". *arXiv preprint arXiv:1905.01209*.

Paszke, A., S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. (2019). "PyTorch: An imperative style, high-performance deep learning library". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Vancouver, Canada.

Pereira, J. and M. Silveira. (2018). "Unsupervised anomaly detection in energy time series data using variational recurrent autoencoders with attention". In: *IEEE International Conference on Machine Learning and Applications (ICMLA)*. Orlando, FL.

Petrovich, M., M. J. Black, and G. Varol. (2021). "Action-conditioned 3D human motion synthesis with Transformer VAE". In: *International Conference on Computer Vision (ICCV)*. Virtual online.

Raiko, T. and M. Tornio. (2009). "Variational Bayesian learning of nonlinear hidden state-space models for model predictive control". *Neurocomputing*. 72(16-18): 3704–3712.

Ranganath, R., D. Tran, and D. Blei. (2016). "Hierarchical variational models". In: *International Conference on Machine Learning (ICML)*. New York, NY.

Razavi, A., A. van den Oord, B. Poole, and O. Vinyals. (2019). "Preventing posterior collapse with delta-VAEs". *arXiv preprint arXiv:1901.03416*.
Rezende, D. J., S. Mohamed, and D. Wierstra. (2014). "Stochastic backpropagation and approximate inference in deep generative models". In: *International Conference on Machine Learning (ICML)*. Beijing, China.

Rezende, D. J. and S. Mohamed. (2015). "Variational inference with normalizing flows". *arXiv preprint arXiv:1505.05770*.

Rix, A., J. Beerends, M. Hollier, and A. Hekstra. (2001). "Perceptual evaluation of speech quality (PESQ): A new method for speech quality assessment of telephone networks and codecs". In: *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Salt Lake City, Utah.

Robbins, H. and S. Monro. (1951). "A stochastic approximation method". *The Annals of Mathematical Statistics*: 400–407.

Roberts, A., J. Engel, C. Raffel, C. Hawthorne, and D. Eck. (2018). "A hierarchical latent vector model for learning long-term structure in music". *arXiv preprint arXiv:1803.05428*.

Roche, F., T. Hueber, S. Limier, and L. Girin. (2019). "Autoencoders for music sound synthesis: A comparison of linear, shallow, deep and variational models". In: *Sound and Music Conference (SMC)*. Malaga, Spain.

Salimans, T. (2016). "A structured variational auto-encoder for learning deep hierarchies of sparse features". *arXiv preprint arXiv:1602.08734*.

Salimans, T., D. Kingma, and M. Welling. (2015). "Markov chain Monte Carlo and variational inference: Bridging the gap". In: *International Conference on Machine Learning (ICML)*. Lille, France.

Salimans, T. and D. A. Knowles. (2013). "Fixed-form variational posterior approximation through stochastic linear regression". *Bayesian Analysis*. 8(4): 837–882.

Saul, L. K. and M. I. Jordan. (1996). "Exploiting tractable substructures in intractable networks". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Denver, CO.

Semeniuta, S., A. Severyn, and E. Barth. (2017). "A hybrid convolutional variational autoencoder for text generation". *arXiv preprint arXiv:1702.02390*.
Serban, I. V., A. G. Ororbia, J. Pineau, and A. Courville. (2016). "Piecewise latent variables for neural variational text processing". *arXiv preprint arXiv:1612.00377*.

Serban, I. V., A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y. Bengio. (2017). "A hierarchical latent variable encoder-decoder model for generating dialogues". In: *AAAI Conference on Artificial Intelligence*. San Francisco, CA.

Shang, W., K. Sohn, and Y. Tian. (2018). "Channel-recurrent autoencoding for image modeling". In: *IEEE Winter Conference on Applications of Computer Vision (WACV)*. Lake Tahoe, NV.

Siddharth, N., B. Paige, J.-W. Van de Meent, A. Desmaison, N. Goodman, P. Kohli, F. Wood, and P. Torr. (2017). "Learning disentangled representations with semi-supervised deep generative models". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Long Beach, CA.

Šmídl, V. and A. Quinn. (2006). *The variational Bayes method in signal processing*. Springer Science & Business Media.

Sohn, K., H. Lee, and X. Yan. (2015). "Learning structured output representation using deep conditional generative models". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Montréal, Canada.

Sønderby, C. K., T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther. (2016a). "How to train deep variational autoencoders and probabilistic ladder networks". In: *International Conference on Machine Learning (ICML)*. New York, NY.

Sønderby, C. K., T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther. (2016b). "Ladder variational autoencoders". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Barcelona, Spain.

Su, J., S. Wu, D. Xiong, Y. Lu, X. Han, and B. Zhang. (2018). "Variational recurrent neural machine translation". In: *AAAI International Conference on Artificial Intelligence*. New Orleans, NV.

Sutskever, I. (2013). *Training recurrent neural networks*. University of Toronto.
Taal, C. H., R. C. Hendriks, R. Heusdens, and J. Jensen. (2011). "An algorithm for intelligibility prediction of time-frequency weighted noisy speech". *IEEE Transactions on Audio, Speech, and Language Processing*. 19(7): 2125-2136.

Tipping, M. E. and C. M. Bishop. (1999). "Probabilistic principal component analysis". *Journal of the Royal Statistical Society: Series B (Statistical Methodology)*. 61(3): 611–622.

Tulyakov, S., M.-Y. Liu, X. Yang, and J. Kautz. (2018). "MoCoGan: Decomposing motion and content for video generation". In: *IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)*. Salt Lake City, Utah.

Vahdat, A. and J. Kautz. (2020). "NVAE: A deep hierarchical variational autoencoder". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Virtual online.

Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. (2017). "Attention is all you need". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Long Beach, CA.

Villegas, R., J. Yang, S. Hong, X. Lin, and H. Lee. (2017). "Decomposing motion and content for natural video sequence prediction". In: *International Conference on Learning Representations (ICLR)*. Toulon, France.

Vincent, P., H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol. (2010). "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion". *Journal of Machine Learning Research*. 11: 3371–3408.

Wan, E. and R. Van Der Merwe. (2000). "The unscented Kalman filter for nonlinear estimation". In: *IEEE Adaptive Systems for Signal Processing, Communications, and Control Symposium*. Lake Louise, Canada.

Wang, T. and X. Wan. (2019). "T-CVAE: Transformer-based conditioned variational autoencoder for story completion". In: *AAAI International Joint Conference on Artificial Intelligence (IJCAI)*. Macao, China.
Watter, M., J. Springenberg, J. Boedecker, and M. Riedmiller. (2015). "Embed to control: A locally linear latent dynamics model for control from raw images". In: *Advances in Neural Information Processing Systems (NeurIPS)*. Montréal, Canada.

Wei, G. C. and M. A. Tanner. (1990). "A Monte Carlo implementation of the EM algorithm and the poor man's data augmentation algorithms". *Journal of the American Statistical Association*. 85(411): 699-704.

Williams, C. and C. E. Rasmussen. (2006). *Gaussian processes for machine learning*. MIT Press.

Williams, R. J. and D. Zipser. (1989). "A learning algorithm for continually running fully recurrent neural networks". *Neural Computation*. 1(2): 270-280.

Winn, J. and C. M. Bishop. (2005). "Variational message passing". *Journal of Machine Learning Research*. 6(Apr): 661-694.

Yang, Y., G. Sautière, J. J. Ryu, and T. S. Cohen. (2020). "Feedback recurrent autoencoder". In: *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. Virtual Barcelona.

Yang, Z., Z. Hu, R. Salakhutdinov, and T. Berg-Kirkpatrick. (2017). "Improved variational autoencoders for text modeling using dilated convolutions". In: *International Conference on Machine Learning (ICML)*. Sydney, Australia.

Yeung, S., A. Kannan, Y. Dauphin, and L. Fei-Fei. (2017). "Tackling over-pruning in variational autoencoders". In: *International Conference on Machine Learning - Workshop on Principled Approaches to Deep Learning*. Sydney, Australia.

Zhao, T., K. Lee, and M. Eskenazi. (2018). "Unsupervised discrete sentence representation learning for interpretable neural dialog generation". *arXiv preprint arXiv:1804.08069*.

Zhao, T., R. Zhao, and M. Eskenazi. (2017). "Learning discourse-level diversity for neural dialog models using conditional variational autoencoders". *arXiv preprint arXiv:1703.10960*.

Zhao, Y., J. Nassar, I. Jordan, M. Bugallo, and I. M. Park. (2019). "Streaming variational Monte Carlo". *arXiv preprint arXiv:1906.01549*.

Zhao, Y. and I. M. Park. (2017). "Variational online learning of neural dynamics". *arXiv preprint arXiv:1707.09049*.