Year: 2018

Abstract:
We study the problem of learning representations of sequence data. Recent work has built on variational autoencoders to develop variational recurrent models for generation. Our main goal is not generation but rather representation learning for downstream prediction tasks. Existing variational recurrent models typically use stochastic recurrent connections to model the dependence among neighboring latent variables, while generation assumes independence of generated data per time step given the latent sequence. In contrast, our models assume independence among all latent variables given non-stochastic hidden states, which speeds up inference, while assuming dependence of observations at each time step on all latent variables, which improves representation quality. In addition, we propose and study extensions for improving downstream performance, including hierarchical auxiliary latent variables and prior updating during training. Experiments show improved performance on several speech and language tasks with different levels of supervision, as well as in a multi-view learning setting.

heading: 1 INTRODUCTION
text: Modeling sequence data is a central problem in domains such as speech and natural language processing. In this work, we study the problem of learning representations (features) of sequence data, with varying degrees of supervision, that can be helpful for downstream sequence labeling tasks. We take a generative approach, inspired by variational autoencoders (VAEs) Kingma & Welling (2014); Rezende et al. (2014); Doersch (2016) for non-sequence data. We assume that the sequence data is generated by a series of latent variables, parameterize the observation likelihood with recurrent neural networks, and maximize (a lower bound on) the observation likelihood. If labels are available during training, a discriminative loss can also be imposed on the latent variables. The learned posterior distribution of the latent variables then provides features for the downstream tasks. This intuitively simple approach, however, requires non-trivial extensions of the deep generative models that have been successful for non-sequence data.
For a (non-sequence) observation x, a VAE assumes a prior distribution, p(z), on the latent variable z which is often simple (e.g., N (0, I)), and consists of a generation network parameterizing p(x|z) with weights θ and an inference network parameterizing the approximate posterior q(z|x) with weights φ. Learning the VAE is done by maximizing an evidence lower bound (ELBO) on the observation log-likelihood log pθ(x) = log ∫ p(z)pθ(x|z)dz, defined as
ELBO := Eqφ(z|x) [log pθ(x|z)]−DKL (qφ(z|x)||p(z)) ≤ log pθ(x). (1)
VAEs can be viewed as autoencoders: The encoder q(z|x) maps the input to a latent variable z according to a (also typically Gaussian) posterior distribution, and the decoder p(x|z) reconstructs the input from samples of z drawn from this posterior. The output of the encoder (most commonly, the mean of q(z|x)) is often used as a learned representation (features) for downstream tasks. Maximizing the ELBO is equivalent to minimizing the reconstruction loss of the output x with respect to the input, plus a regularizer based on the prior distribution of the latent variable. In some variants the importance of the KL term is tuned by weighting the regularizer by a hyperparameter β Higgins et al. (2016); Alemi et al. (2016).
VAEs have been extended to model sequence data in a number of ways. Fabius & van Amersfoort (2014) learn a single representation for the entire sequence, while Hsu et al. (2017) learn both a
whole-sequence representation and a set of representations for pre-defined segments within the sequence. For many tasks, such as the ones we consider here, it is desirable to represent a length-T input sequence x1:T with a corresponding length-T latent sequence z1:T so as to fit directly into typical recurrent network-based prediction models.
Several recent approaches fit this criterion Krishnan et al. (2015); Archer et al. (2015); Chung et al. (2015b); Fraccaro et al. (2016); Goyal et al. (2017a); Chen et al. (2018). Learning is again done by maximizing an ELBO, with the main differences among approaches being the specific forms of the prior p(z1:T ) (typically parameterized so as to capture dynamics in the latent space), the generation distribution pθ(x1:T |z1:T ), and the approximate posterior qφ(z1:T |x1:T ). For example, direct recurrent connections between stochastic variables Fraccaro et al. (2016), as shown in Figure 1(a), or indirect recurrent connections, e.g. ht−1 → zt−1 → ht → zt Chung et al. (2015b); Goyal et al. (2017a), are often introduced in p(z1:T ) to model the dependence between neighboring latent variables. While this is more powerful than a simpler prior for the purpose of generation, it poses challenges for designing the approximate posteriors due to the dependencies among zt’s. On the other hand, given z1:T , the generation model is often fully factorized into T independent terms: pθ(x1:T |z1:T ) = ∏T t=1 pθ(xt|zt).
Most prior work on variational recurrent models has focused on generation quality and likelihood evaluation. It is not clear, however, that the learned representations are useful for downstream tasks. The goal of our work is to fill this gap by developing variational recurrent models that produce high-quality representations for prediction tasks, and to investigate several modeling choices.
We take an approach, similarly to Chen et al. (2018), where the latent distribution p(z1:T ) factors over time conditioned on a deterministic hidden state sequence, while the generation model pθ(x1:T |z1:T ) is factored as ∏T
t=1 pθ(xt|z1:T ), as shown in Figures 1(b,d). Also similarly to Chen et al. (2018), we use a simple Gaussian prior which is updated during training, and find that this can be very helpful. Unlike Chen et al. (2018), however, we consider a wide variety of tasks and levels of supervision. Compared to variational recurrent models with recurrent connections among the stochastic latent variables, this class of models is more efficient and easier to implement, since there are no dependencies between stochastic latent variables and therefore no nested sampling procedure.
One key novelty in this work is that, while we have no recurrent connections among the z1:T , we require each zj to generate each xk with some probability, for all 1 ≤ j, k ≤ T , so that the features are predictive of a large context of observations. We propose a “stochastic generation” model that approximates this idea, and we find that this extension is important for improving downstream performance.
In addition, we study the proposed models on a range of speech and NLP prediction tasks, in a variety of supervision settings, finding improved performance in all cases. We consider fully supervised, semi-supervised, and weakly supervised (using multi-view training data) settings. We believe this is the first attempt at a broad study of recurrent variational representation learning that is applicable to a variety of downstream tasks and levels of supervision.

heading: 2 A VARIATIONAL RECURRENT REPRESENTATION LEARNING MODEL
text: Our variational recurrent representation learning approach, shown in Figure 1(b-d), aims to learn informative representations while keeping inference simple. As in other recurrent variational models, we optimize an ELBO of the same form as in Equation 1 but where the variables are all sequences of length T ; that is, the prior is p(z1:T ), the posterior is qφ(z1:T |x1:T ), and the generation model is pθ(x1:T |z1:T ).

heading: 2.1 INFERENCE
text: Given a length T input x1:T , the model first maps from the input sequence to a sequence of deterministic recurrent hidden state vectors h1:T , produced by one or more stacked recurrent neural network layers. Conditioned on ht (1 ≤ t ≤ T ), the posterior distribution of the latent random variable zt is a spherical Gaussian
zt|ht ∼ qφ(zt|ht) ≡ N (µφt, diag (σφt)), [µφt, log σφt] = Fφ(ht) (2)
Here, qφ is the inference network with parameters φ and Fφ is either a linear transformation or a feedforward neural network with a linear final layer. The full posterior and prior are
qφ(z1:T |h1:T ) := T ∏
t=1
qφ(zt|ht), p(z1:T ) := T ∏
t=1
p(zt) (3)
Finally, a reconstruction of the input sequence is generated from the latent stochastic sequence z1:T . The learned representation of xt, which can be used for downstream tasks, is taken to be the mean µφt of qφ(zt|ht).
Note that no direct dependence is assumed among different time steps of z1:T given h1:T . This greatly simplifies inference over prior work that directly models dependence between the zt Goyal et al. (2017a); Fraccaro et al. (2016); Chung et al. (2015b); Krishnan et al. (2015), since the distribution of each zt (given ht) is a single spherical Gaussian rather than a more complex multimodal distribution that would arise with dependence modeling (e.g., using recurrent connections among stochastic nodes). Consider a model with stochastic recurrent connections as in Figure 1(a), which models the conditional probability of zt given the past z1:t−1. The marginal distribution of each zt is multimodal, and requires nested sampling. In order to obtain good estimates, a large number of samples of z1:t−1 may need to be used. In addition, in a model with multimodal marginal distributions of zt, it is not clear what to consider to be the learned representation for use in a downstream tasks; in particular the mean of zt may have very small probability and thus may not be a good choice.

heading: 2.2 STOCHASTIC GENERATION
text: Whereas our approach simplifies inference, the generation of each xk for 1 ≤ k ≤ T in our model involves samples of all z1:T , with each zt for 1 ≤ t ≤ T having a different effect on the generation of xk , as shown in Figures 1(c,d). Intuitively, this generation approach reintroduces some of the relationships between the zt that may have been lost by having no direct dependence modeling. Given a particular choice of k, t, the generative model for xk given zt is also a spherical Gaussian:
xk|zt ∼ N (µθt, diag(σθt)) = pθ(xk|zt), [µθt, log σθt] = Fθ(zt) (4)
where again Fθ , with parameters θ, can be linear or a feedforward neural network with a linear final layer. Note that the full generative distribution (Figure 1(d)) is multimodal, although we never explicitly evaluate it, as described below.
Figure 1(d) shows our full generation process. However, this generation process is a bit expensive; instead we employ stochastic generation (Figure 1(c)), a technique that aims to approximate the generation process. In stochastic generation, we first randomly select a frame xk from x1:T for time step t. During training we evaluate Eqφ(zt|ht) [ log pθ(xk|zt) ]
for time step t. The way we sample the target to be reconstructed for time step t affects the way zt and ht encode x1:T . The higher the probability that a distant time frame is selected, the more temporal information should be encoded in the latent variables. Next we describe stochastic generation more formally.
Definition 2.1. α (t,k) δ,T Given x1:T and Gaussian distribution p (t)(s) ≡ N (t, δ), where 1 ≤ t ≤ T . Let c0:T = {−∞, 1.5, 2.5, ..., T − 0.5,+∞}. We define α (t,k) δ,T for 1 ≤ t, k ≤ T as
α (t,k) δ,T :=
∫ ck
ck−1
p(t)(s)ds (5)
Definition 2.2. Stochastic Generation Given δ and sequence x1:T , stochastic generation reconstructs xk using samples of zt with probability α (t,k) δ,T . That is, the conditional log-likelihood for time step t is Eqφ(zt|ht) [ log pθ(xk|zt) ] with probability α (t,k) δ,T .
In stochastic generation, each zt contributes to xk differently based on the delay between time steps t and k. If δ is chosen such that only nearby neighbor xks are produced, it becomes a weighted window generation. Remark 1 shows how a particular instance of stochastic generation relates to a full generation model, with graphical model as described in Figure 1(d).
Remark 1. Given z1:T , φ and θ, stochastic generation in expectation is evaluating
T ∑
t=1
{
Eqφ(zt|ht)
[ T ∑
k=1
α (t,k) δ,T log pθ(xk|zt)
]}
(6)
Equation 6 is equivalent to computing the expectation of log pθ(x1:T |z1:T ), where
pθ(x1:T |z1:T ) = T ∏
t=1
pθ(xt|z1:T ) = T ∏
t=1
T ∏
k=1
pθ(xk|zt) α (t,k) δ,T (7)
It is straightforward to show that the ELBO from the two perspectives is identical (see the supplementary material). Through stochastic generation, although we are always using simple Gaussian distributions, we are implicitly creating a more complex generative model. Note that, for purposes of generation and likelihood evaluation, we directly use the generative model described in Equation 7. More detailed derivations can be found in the supplementary material.

heading: 3 EXTENSIONS
text: Thus far we have described a basic recurrent variational representation learning model that uses stochastic generation to account for context instead of recurrent stochastic connections. We next describe a few extensions aimed at improving the learned representations for use in downstream tasks. First, we embed our model in a multitask training approach, in order to utilize different levels and types of supervision (full supervision, semi-supervision, and weak supervision in a multi-view learning setting). Next, similarly to Chen et al. (2018), we consider models with a hierarchy of latent variables per time step and specify a different functionality for each one, which allows some of the latent variables to focus on encoding task-specific/view-invariant information; and we study the use of prior updating in order to improve the quality of inference with our factored posterior.

heading: 3.1 LEARNING WITH SUPERVISION
text: If we have a task-specific prediction model with loss F and labels l, then we can maximize the following objective
(1− α) { Lδ,β(x1:T , z1:T , θ, φ) } − αF(z1:T , l) (8)
where Lδ,β is our representation learning loss (ELBO), with a given stochastic generation parameter δ and KL term weight β, and α > 0 is a trade-off parameter. This is a multitask loss, where the latent variable zt needs to perform well with respect to the unsupervised loss (reconstruction of the input and similarity to the prior) as well as the supervised task-related loss.
In some settings labels are not available, but a second view of the data—that is, a second input sequence u1:T paired with each primary input sequence x1:T —is available at training time but not test time. For example the second view may be from another modality. In such a setting it is often possible to learn a better representation of the primary view by using the second view as a form of weak supervision Ngiam et al. (2011); Srivastava & Salakhutdinov (2012); Sohn et al. (2014); Wang et al. (2015). Here, we extend the variational canonical correlation analysis with private variables (VCCAP) model of Wang et al. (2016) to a variational recurrent multi-view model with stochastic generation, named RecVCCAP for short.
In RecVCCAP, a latent representation y1:T is inferred from x1:T only, but used to reconstruct both modalities. In addition, z1:T conditioned on deterministic hidden state g1:T , and also o1:T conditioned on deterministic n1:T , are also introduced to model information specific (“private”) to x and u respectively, as shown in Figure 2(b). The ELBO of this model is a lower bound on the joint distribution of data in the two views:
log pθ(x1:T , u1:T )
≥ Eqφ(z1:T ,y1:T |g1:T ,h1:T ) [ log pθ(x1:T |z1:T , y1:T ) ] + Eqφ(o1:T ,y1:T |n1:T ,h1:T ) [ log pθ(u1:T |o1:T , y1:T ) ] − DKL ( qφ(o1:T |n1:T )||p(o1:T ) ) −DKL ( qφ(z1:T |g1:T )||p(z1:T ) ) −DKL ( qφ(y1:T |h1:T )||p(y1:T ) )
= ΣTt=1
{
Eqφ(yt|ht)qφ(zt|gt)
[
ΣTk=1 { αt,kδ,T log pθ(xk|zt) } + Eqφ(yt|ht)qφ(ot|nt)Σ T k=1 { αt,kδ,T log pθ(uk|ot) }
]}
− DKL ( qφ(o1:T |n1:T )||p(o1:T ) ) −DKL ( qφ(z1:T |g1:T )||p(z1:T ) ) −DKL ( qφ(y1:T |h1:T )||p(y1:T ) ) (9)
Here the generation model includes pθ(x1:T |z1:T , y1:T )p(z1:T )p(y1:T ) and pθ(u1:T |o1:T , y1:T )p(o1:T )p(y1:T ) for the two modalities respectively. Detailed derivation of the ELBOs can be found in supplementary material.

heading: 3.2 HIERARCHICAL LATENT VARIABLES
text: In supervised training, it may be beneficial to separate the latent variables into task-specific and taskindependent components, and to apply the supervised loss only to the task-specific latent variables. The multitask model of Equation 8 can be easily extended with an auxiliary task-specific latent variable yt, as shown in Figure 2(a). The supervised loss is applied only to yt while the unsupervised loss (reconstruction + prior) is applied to zt. The multitask loss (negative ELBO+supervised loss)
of this hierarchical model is
−(1− α) { Lδ,β(x1:T , z1:T , y1:T , θ, φ) } + αF(y1:T , l)
≡ −(1− α)ΣTt=1
{
Eqφ(yt|ht)qφ(zt|yt,ht)
[
ΣTk=1 { α (t,k) δ,T log pθ(xk|zt) }
]
− β { DKL ( qφ(zt|yt, ht)||pθ(zt) ) +DKL ( qφ(yt|ht)||pθ(yt) )}
}
+ αF(y1:T , l) (10)
Another benefit of such a hierarchical model, over the basic one, is that it increases the complexity of the latent representation—zt ∼ {qφ(zt) = ∫
qφ(zt, yt)dyt} is multi-modal even though yt and zt|yt are assumed to be Gaussian—while keeping inference relatively simple since each of the latent distributions is conditionally a single Gaussian. In this sense the motivation is similar to that of existing work by Rezende & Mohamed (2015) and Maaløe et al. (2016). A final motivation, similarly to Sønderby et al. (2016); Zhao et al. (2017), is that this 2-layer model can directl represent a hierarchical feature space. Unlike prior work, however, a central goal here is to keep the task-specific latent variable uni-modal so that its mean can be used as the learned representation. The multi-view recurrent model of Equation 9 can also be extended with hierarchical latent variables, as shown in Figure 2(c). The resulting ELBO and derivation is given in the supplementary material.

heading: 3.3 PRIOR UPDATING
text: A simple prior p(z) ≡ N (0, I) is most common in vanilla VAEs and other variational approaches (Higgins et al., 2016; Alemi et al., 2016). Some recent work (Serban et al., 2017; Tomczak & Welling, 2017; Goyal et al., 2017b) has considered more complex prior distributions in order to better match the true distribution of the assumed latent space. In models of sequence data, existing variational approaches use priors on zt that are conditioned on various parts of the input and latent variables in other frames (Goyal et al., 2017a; Fraccaro et al., 2016; Chung et al., 2015b).
Motivated by the view of the prior as a regularizer, we propose an approach where the encoder/decoder (φ and θ) and the prior are alternately updated during training. In particular, we set the prior in a given iteration i to be the posterior from the previous iteration: p (i) π (z1:T ) := qφ(i)(z1:T |x1:T ), where p (0) π (z1:T ) ≡ N T (0, I) and φ(i), θ(i) are the solutions maximizing the ELBO with respect to p(i−1)(z1:T ). It is straightforward to show that
max φ,θ
{
Eqφ(z1:T |x1:T ) [ log(pθ(x1:T |z1:T )) ] − βDKL ( qφ(z1:T |x1:T )||p (i) π (z1:T ) )
}
≤ max φ,θ
{
Eqφ(z1:T |x1:T ) [ log(pθ(x1:T |z1:T )) ] − βDKL ( qφ(z1:T |x1:T )||p (i+1) π (z1:T ) )
}
.(11)
The intuition of this approach is that the KL regularization term should have a large effect at the beginning of training, and the effect should progressively diminish during training. More motivations and illustrations are given in the supplementary material.

heading: 4 EXPERIMENTS
text: The main goal of our work is to learn good representations for downstream tasks. We compare models on several tasks: (1) the CoNNL 2003 named entity recognition task Tjong Kim Sang & De Meulder (2003) and (2) the CoNNL 2000 text chunking task Tjong Kim Sang & Buchholz (2000); (3) phonetic recognition on the TIMIT speech corpus Zue et al. (1990) and the University of Wisconsin X-ray Microbeam (XRMB) acousticarticulatory data set Westbury et al. (1990) (the latter used for multi-view representation learning), (4) character-level speech recognition on the Wall Street Journal (WSJ) data set Paul & Baker (1992). The first of these tasks was also studied by Chen et al. (2018), but only in a semi-supervised setting. In order to save space, detailed information on all data sets and hyperparameter tuning is included in the supplementary material.
In the single-view setting, we compare the following models: 1) Variational model with forward stochastic recurrent connections (StocCon, Figure 1(a)), 2) StocCon with prior updating (StocCon+P), 3) StocCon with both forward and backward (with the dependence path zT → zT−1 → · · · )
stochastic recurrent layers (StocCon+B+P), , 4) Our model, variational recurrent model for representation learning, RecRep for short, 5) RecRep with prior updating (RecRep+P), 6) RecRep with hierarchical latent variables (RecRep+H) and 7) RecRep with both hierarchical latent variables and prior updating (RecRep+H+P).
In the multi-view setting, the models are our recurrent extensions of variational canonical correlation analysis (VCCAP) Wang et al. (2016): 1) Recurrent VCCAP with stochastic recurrent connections with prior updating (StocConVCCAP+P), 2) Our model, recurrent VCCAP without stochastic recurrent connections (RecVCCAP), 3) RecVCCAP with prior updating (RecVCCAP+P), 4) RecVCCAP with latent hierarchy (RecVCCAP+H) and 5) RecVCCAP with hierarchy and prior updating (RecVCCA+H+P, Figure 2(b)).

heading: 4.1 OVERVIEW OF RESULTS
text: The results below follow the same general pattern across tasks. Models with (unidirectional) stochastic recurrent connections (StocCon) struggle to produce representations that improve over baselines, even when enhanced with prior updating. Bidirectional stochastic recurrent models (StocCon+B) improves performance, but at the cost of even more complex inference (for this reason we only attempted this model on one task, Table 2). Our RecRep models, without stochastic recurrent connections, always improve over the baselines at least when using prior updating and/or hierarchical latent variables. In some experiments we set δ = 0 in the stochastic generation model (i.e., generating only xt from zt); this always significantly hurts performance, showing the benefit of stochastic generation. Depending on the task, either prior updating or hierarchical latent variables is particularly useful. In all cases the best results are produced by RecRep+H+P models.

heading: 4.2 SUPERVISED SETTING: CONLL 2003 NAMED ENTITY RECOGNITION AND CONLL 2000 CHUNKING
text: For named entity recognition (NER) and text chunking (Table 1) we follow the setting of Peters et al. (2017). The input is a pretrained word embedding (GloVe 100-dimensional embedding Pennington et al. (2014)) concatenated with the output of a character RNN embedding. The baseline prediction model is a 2-layer bidirectional GRU RNN Cho et al. (2014); Chung et al. (2015a). . The decoder distribution pθ(x|z) is a softmax rather than Gaussian, as in Miao et al. (2016). A more detailed generation model description is in the supplementary material.

heading: 4.3 SUPERVISED SETTING: TIMIT PHONETIC RECOGNITION
text: For phonetic recognition on the TIMIT data set, we use the same data processing and train/dev/test split as in Tang et al. (2017).1 The baseline recognizer is a 3-layer stacked bidirectional LSTM Hochreiter & Schmidhuber (1997) network with pyramidal subsampling as in Chan et al. (2016); Lu et al. (2016), where the output of each layer (except the topmost layer) is reduced in length by a factor of two. The supervised phonetic recognition loss is the connectionist temporal classification (CTC) loss Graves et al. (2013). The representation learning loss (ELBO) is applied to the second layer. For all speech models, the decoder distribution pθ(x|z) is modeled as a spherical Gaussian, with diagonal covariance matrix σ2I, where σ is a hyperparameter.

heading: 4.4 SEMI-SUPERVISED SETTING: WSJ CHARACTER-LEVEL SPEECH RECOGNITION
text: For character-level speech recognition on the Wall Stree Journal data set (WSJ), we use the standard dev (503 utterances) and test (330 utterances) sets. For semi-supervised experiments, we use about 5K utterances out of the full WSJ training set as labeled data and an additional subset of 16K utterances as unlabeled data. The recognizers are two-layer bidirectional GRU (BiGRU) networks trained with character-level CTC loss as in Miao et al. (2015). The representation learning loss (ELBO) is applied to the first layer BiGRU output. The results (Table 3) show that our models can improve performance both in the fully supervised setting (with a small amount of labeled data) and further in the semi-supervised setting (with some additional unlabeled data).

heading: 4.5 MULTI-VIEW SETTING: XRMB PHONETIC RECOGNITION
text: We use the same setup as in prior work on XRMB Tang et al. (2018): We use the acoustic+articulatory data from 35 speakers for representation learning and the acoustic-only data from 12 speakers for phonetic recognition experiments in a 6-fold setup (Table 4). The recurrent models are 2-layer bidirectional LSTMs with 256 units per layer. We also include a result from prior work using basic VCCAP with non-recurrent fully connected feedforward networks, which operate on large overlapping fixed-size segments of the input Tang et al. (2018). Interestingly, here all of the representation learning approaches improve significantly over baseline, and the non-recurrent feedforward models perform quite well, although the best performance is again produced by the proposed RecVCCAP models with stochastic generation. In anecdotal experiments, we find that we can further improve phonetic error rates to < 7% with either feedforward or RecVCCAP models by increasing their complexity (e.g., by further increasing the window size in the feedforward models); however, the memory and computation time requirements of feedforward models quickly become prohibitive while the recurrent models remain fairly efficient.

heading: 4.6 VISUALIZATION OF LEARNED REPRESENTATIONS
text: We visualize the features we learned on XRMB in the multi-view setting via t-SNE embeddings Maaten & Hinton (2008). We compare three types of features in Figure 3, computed on 443 randomly selected speech frames each corresponding to one of the 6 vowels [aa,ae,uw,ih,eh,ow]
1We do not, however, use a second validation set as in Tang et al. (2017).
from the 12 test speakers. Qualitatively, these visualizations show that representations learned with the proposed models form tighter label-specific clusters.

heading: 5 CONCLUSION
text: We have proposed an approach for learning sequence representations of sequence inputs via variational recurrent neural models. Unlike most prior work, our approach focuses on the quality of the learned representation for a variety of downstream tasks rather than on generation quality, and the representations are learned in a multitask setting along with a supervised, semi-supervised, or weakly supervised multi-view loss. We use simple, efficient inference networks where the mean of the latent variables naturally serves as a representation, and we introduce stochastic generation as an approach for modeling context without a complex inference model. We have studied the proposed models in the context of several speech and language tasks. We find that the proposed recurrent representations consistently improve task performance, and that hierarchical latent variables and prior updating are useful in a variety of settings.

heading: 1 DERIVATION FOR STOCHASTIC GENERATION
text: As mentioned in the main text, we are not calculating the full generative model shown in Figure 1(d) of the main paper. Instead, we use the process referred to as stochastic generation. Here, we illustrate the connection between using stochastic generation for training and directly evaluating the multi-modal generative model corresponding to Figure 1(d) in the main paper.
Assume we have N sequences in our training set, and assume each sequence has been visited (via stochastic generation)K times. We denote the tth latent variable of sequence x1:T as zt. Suppose zt has tried to reconstruct xk for Mt,k times. So on average, by using stochastic generation in training, the log-likelihood computed for the sequence x1:T given the posterior is as follows:
ΣTt=1
{
Eqφ(zt|ht)
[
ΣTk=1 {Mt,k
Ki log pθ(xk|zt)
}
]}
(1)
with E[ Mt,k K ] = α (t,k) δ,T given the definition of stochastic generation.
If we directly evaluate the generation distribution
pθ(x1:T |z1:T ) = ΠTt=1pθ(xt|z1:T ) = ΠTt=1Π T k=1pθ(xk|zt)α (t,k) δ,T (2)
then we can derive the ELBO as follows:
log pθ(x1:T ) ≥ Eqφ(z1:T |h1:T ) [ log pθ(x1:T |z1:T ) ] −DKL ( qφ(z1:T |h1:T )||p(z1:T ) )
= Eqφ(z1:T |h1:T )
[
ΣTt=1
{
ΣTk=1 { α (k,t) δ,T log pθ(xt|zk) }
}]
−DKL(qφ(z1:T |h1:T )||p(z1:T ))
= Eqφ(z1:T |h1:T )
[
ΣTt=1
{
ΣTk=1 { α (t,k) δ,T log pθ(xk|zt) }
}]
−DKL(qφ(z1:T |h1:T )||p(z1:T ))(3
= ΣTt=1
{
Eqφ(zt|ht)
[
ΣTk=1 { α (t,k) δ,T log pθ(xk|zt) }
] −DKL(qφ(zt|ht)||p(zt)) }
(4)
The log-likelihood of x1:T given the posterior is then as follows:
ΣTt=1
{
Eqφ(zt|ht)
[
ΣTk=1 { α (t,k) δ,T log pθ(xk|zt) }
]
(5)
According to Equation 1, in expectation, the conditional log-likelihood computed using stochastic generation is also equal to Equation 5. This equality shows the connection between stochastic generation and the full generation process.

heading: 2 PROOF AND ILLUSTRATION OF PRIOR UPDATING
text: Given the prior pπ(z1:T ) parameterized by π, we have the ELBO
log pθ,π(x1:T ) = log pθ(x1:T |z1:T )pπ(z1:T )qφ(z1:T |x1:T )
qφ(z1:T |x1:T )pθ(z1:T |x1:T )
= Eqφ(z1:T |x1:T ) log pθ(x1:T |z1:T )pπ(z1:T )qφ(z1:T |x1:T ) qφ(z1:T |x1:T )pθ(z1:T |x1:T ) ≥ Eqφ(z1:T |x1:T ) log pθ(x1:T |z1:T )−DKL ( qφ(z1:T |x1:T )||pπ(z1:T ) ) (6)
As described in the paper, when using prior updating, we simply set p (i) π (z1:T ) := qφ(i)(z1:T |x1:T ). Here, p (i) π (z1:T ) is the prior after i updates, where p (0) π (z1:T ) ≡ N T (0, I); φ(i) and θ(i) are the solutions maximizing the lower bound with respect to p(i−1)(z1:T ), so we have
max φ,θ
{
Eqφ(z1:T |x1:T ) [ log(pθ(x1:T |z1:T )) ] − βDKL ( qφ(z1:T |x1:T )||p(i)π (z1:T ) )
}
= Eq φ(i+1) (z1:T |x1:T )
[ log(pθ(i+1)(x1:T |z1:T )) ] − βDKL ( qφ(i+1)(z1:T |x1:T )||p(i)π (z1:T )
≤ Eq φ(i+1) (z1:T |x1:T )
[ log(pθ(i+1)(x1:T |z1:T )) ]
− { βDKL ( qφ(i+1)(z1:T |x1:T )||p(i+1)π (z1:T ) } = 0
≤ max φ,θ
{
Eqφ(z1:T |x1:T ) [ log(pθ(x1:T |z1:T )) ] − βDKL ( qφ(z1:T |x1:T )||p(i+1)π (z1:T ) )
}
(7)
It is not difficult to show that the equality holds when and only when we have qφ(i+1)(z1:T |x1:T ) = p (i) π (z1:T ) ≡ qφ(i)(z1:T |x1:T ); that is, observing the evidence does not change our understanding of the latent space. An illustration of prior updating from a regularization perspective is given in Figure 1.

heading: 3 ELBO OF RECURRENT MULTI-VIEW (HIERARCHICAL) MODELS
text: We now derive the ELBO for RecVCCAP.
log(pθ(x1:T , u1:T ))
= Eqφ(z1:T ,y1:T ,o1:T |g1:T ,h1:T ,n1:T ) [
log (pθ(x1:T , u1:T |z1:T , y1:T , o1:T )pθ(z1:T , y1:T , o1:T )
pθ(z1:T , y1:T , o1:T |x1:T , u1:T ) )] (8)
Assume the prior distribution is given and factorizes as
pθ(z1:T , y1:T , o1:T ) = p(z1:T , y1:T , o1:T ) = p(z1:T )× p(y1:T )× p(o1:T ) (9)
We then have
log(pθ(x1:T , u1:T ))
= Eqφ(z1:T ,y1:T ,o1:T |g1:T ,h1:T ,n1:T ) [
log (pθ(x1:T , u1:T |z1:T , y1:T , o1:T )p(z1:T , y1:T , o1:T )qφ(z1:T , y1:T , o1:T |g1:T , h1:T , n1:
pθ(z1:T , y1:T , o1:T |x1:T , u1:T )qφ(z1:T , y1:T , o1:T |g1:T , h1:T , n1:T ) = Eqφ(z1:T ,y1:T ,o1:T |g1:T ,h1:T ,n1:T ) [ log ( pθ(x1:T , u1:T |z1:T , y1:T , o1:T ) )]
+ Eqφ(z1:T ,y1:T ,o1:T |g1:T ,h1:T ,n1:T ) [
log (qφ(z1:T , y1:T , o1:T |g1:T , h1:T , n1:T )
pθ(z1:T , y1:T , o1:T |x1:T , u1:T ) )]
+ Eqφ(z1:T ,y1:T ,o1:T |g1:T ,h1:T ,n1:T ) [
log ( p(z1:T , y1:T , o1:T )
qφ(z1:T , y1:T , o1:T |g1:T , h1:T , n1:T ) )]
= Eqφ(z1:T ,y1:T ,o1:T |g1:T ,h1:T ,n1:T ) [ log ( pθ(x1:T , u1:T |z1:T , y1:T , o1:T ) )] + DKL(qφ(z1:T , y1:T , o1:T |g1:T , h1:T , n1:T )||pθ(z1:T , y1:T , o1:T |x1:T , u1:T )) − DKL(qφ(z1:T , y1:T , o1:T |g1:T , h1:T , n1:T )||p(z1:T , y1:T , o1:T )) ≥ Eqφ(z1:T ,y1:T ,o1:T |g1:T ,h1:T ,n1:T ) [ log ( pθ(x1:T , u1:T |z1:T , y1:T , o1:T ) )] − DKL ( qφ(z1:T , y1:T , o1:T |g1:T , h1:T , n1:T )||p(z1:T , y1:T , o1:T ) ) = Eqφ(z1:T ,y1:T |g1:T ,h1:T ) [ log pθ(x1:T |z1:T , y1:T ) ] + Eqφ(o1:T ,y1:T |n1:T ,h1:T ) [ log pθ(u1:T |o1:T , y1:T ) ] − DKL ( qφ(o1:T |n1:T )||p(o1:T ) ) −DKL ( qφ(z1:T |g1:T )||p(z1:T ) ) −DKL ( qφ(y1:T |h1:T )||p(y1:T ) )
= ΣTt=1
{
Eqφ(yt|ht)qφ(zt|gt)
[
ΣTk=1 { α t,k δ,T log pθ(xk|zt) } + Eqφ(yt|ht)qφ(ot|nt)Σ T k=1 { α t,k δ,T log pθ(uk|ot) }
]}
− DKL ( qφ(o1:T |n1:T )||p(o1:T ) ) −DKL ( qφ(z1:T |g1:T )||p(z1:T ) ) −DKL ( qφ(y1:T |h1:T )||p(y1:T ) )
Similarly, we can derive the ELBO for the hierarchical version of RecVCCAP as follows:
log pθ(x1:T , u1:T ) ≥ Eqφ(z1:T ,y1:T |g1:T ,h1:T ) [ logθ(x1:T |z1:T ) ] + Eqφ(o1:T ,y1:T |n1:T ,h1:T ) [ logθ(u1:T |o1:T ) ]
− DKL ( qφ(o1:T |y1:T )||p(o1:T ) ) −DKL ( qφ(z1:T |y1:T )||p(z1:T ) ) −DKL ( qφ(y1:T |h1:T )||p(y1:T ) )
= ΣTt=1
{
Eqφ(yt|ht)qφ(zt|yt,gt)
[
ΣTk=1 { α t,k δ,T log pθ(xk|zt) } + Eqφ(yt|ht)qφ(ot|yt,nt)Σ T k=1 { α t,k δ,T log pθ(uk|ot) }
]}
− DKL ( qφ(o1:T |y1:T )||p(o1:T ) ) −DKL ( qφ(z1:T |y1:T )||p(z1:T ) ) −DKL ( qφ(y1:T |h1:T )||p(y1:T ) )
(11)

heading: 4 DECODER DETAILS
text: For speech tasks we use a Gaussian distribution with diagonal covariance as the decoder distribution, that is
log pθ(x|zt) ≡ N (µ|zt , σ2I) (12) where σ is tuned.
For NLP tasks we use a softmax decoder as described in Equation (6) and (7) of Miao et al. (2016). In our tasks (NER and chunking), the input is a character RNN output concatenated with a pretrained word embedding, and the generation network tries to reconstruct the word corresponding to the given embedding by matching the reconstructed embedding with the (pre-trained) embedding of the target word.

heading: 5 DATA SETS
text: In this section, we give more details for the five data sets we used in the paper: TIMIT, XRMB, WSJ, CoNLL2003 and CoNLL2000.
For TIMIT, there are 3696, 400 and 192 train/dev/test utterances respectively. The per frame input are speaker-normalized 40-dimensional log filter bank features (without energy) and their first and second derivatives. 39 phone labels are used.
For XRMB, there are in total 47 speakers and roughly 2300 acoustic utterances. 35 speakers (roughly 1700 utterances), together with their paired articulatory measurements, are used to train multi-view models. The remaining 12 speakers are used to test the quality of the learned representations by testing on phonetic recognition in a K-fold experimental setup. We use 6-fold validation here, where the 12 speakers (separate from the 35 speakers used for multi-view representation learning) are separated into 6 groups, and each time we pick one group as development set, one group as test set, and the remaining 4 groups (8 speakers) as phonetic recognizer training set. For XRMB, the input features are 13-dimensional MFCCs, also with first and second derivatives, making the feature vectors 39-dimensional.
For WSJ, we start with the standard split, with 37416 train, 503 dev, and 330 test utterances. However, in this paper we don’t use the full training set. For WSJ we again use log filter bank features, but with energy, and also their first and second derivatives. So the per-frame input feature vectors are 123-dimensional. For the purpose of semi-supervised learning, we split the training into 24 parts, and use 3 of them (roughly 5K utterances) as labeled data and treat the other 21 splits as unlabeled data.
For the CoNLL2003 and CoNLL2000 tasks, we refer the reader to Peters et al. (2017), specifically the second and fourth paragraphs of the experimental section.

heading: 6 HYPERPARAMETER TUNING AND TRAINING
text: For all speech models, the decoder distribution pθ(x|z) is a spherical Gaussian, with covariance matrix σ2I. σ is a hyperparameter selected among {0.1, 1.0, 10.0}. The KL term weight β is selected among {0.1, 1.0, 10.0}, α is selected among {0.3, 0.5, 0.7, 0.9, 0.99}, and √ δ is selected among {0, 0.05, 0.1, 0.5, 1, 2, 5, 10, 20}. The tuning strategy applied to all of the tasks in this work.
For learning, we typically use Adam Kingma & Ba (2015), with initial learning rate chosen from {0.001, 0.0005, 0.0001}. For TIMIT, we also use stochastic gradient descent, following the optimization strategy described in Tang et al. (2017).

Review summaries:
This is largely an experimental paper, proposing and evaluating various modifications of variational recurrent models towards obtaining sequence data representations that are effective in downstream tasks. The highlighted contribution is a "stochastic generation" training procedure in which the training objective evaluates the reconstruction of output sequence elements from individual latent variables independently. The main claim is that the resulting model, augmented with prior updating and/or hierarchical latent variables, improves results w.r.t. the baselines.