Title: INFERENCE & INTROSPECTION IN DEEP GENERATIVE MODELS OF SPARSE DATA

Abstract: Deep generative models such as deep latent Gaussian models (DLGMs) are powerful and popular density estimators. However, they have been applied almost exclusively to dense data such as images; DLGMs are rarely applied to sparse, high-dimensional integer data such as word counts or product ratings. One reason is that the standard training procedures find poor local optima when applied to such data. We propose two techniques that alleviate this problem, significantly improving our ability to fit DLGMs to sparse, high-dimensional data. Having fit these models, we are faced with another challenge: how to use and interpret the representation that we have learned? To that end, we propose a method that extracts distributed representations of features via a simple linearization of the model.

1. INTRODUCTION:

Deep latent Gaussian models (DLGMs, a.k.a. variational autoencoders; Rezende et al., 2014; Kingma et al., 2014) have led a resurgence in the use of deep generative models for density estimation. DLGMs assume that observed vectors x are generated by applying a nonlinear transformation (defined by a neural network with parameters θ) to a vector of Gaussian random variables z.

Learning in DLGMs proceeds by approximately maximizing the average marginal likelihood p(x) ≡ z p(z)p(x|z)dz of the observations x. Computing the true marginal likelihood is intractable, so we resort to variational expectation-maximization (Bishop, 2006) , an approximation to maximumlikelihood estimation. To learn the parameters θ of the generative model, the procedure needs to find a distribution q(z|x) that approximates the posterior distribution p(z|x) of the latent vector z given the observations x. In the past, such q distributions were fit using iterative optimization procedures (e.g., Hoffman et al., 2013) . But Rezende et al. (2014) and Kingma et al. (2014) showed that q(z|x) can be parameterized by a feedforward "inference network" with parameters φ, speeding up learning. This inference network is trained jointly with the generative model; as training proceeds, the inference network learns to approximate posterior inference on the generative model, and the generative model improves itself using the output of the inference network.

Embedded within this procedure, however, lies a potential problem: both the inference network and the generative model are initialized randomly. Early on in learning, the inference network's q(z|x) distributions will be poor approximations to the true posterior p(z|x), and the gradients used to update the parameters of the generative model will therefore be poor approximations to the gradients of the true log-likelihood log p(x). Previous stochastic variational inference methods (Hoffman et al., 2013) were slower, but suffered less from this problem since for every data-point, a set of variational parameters was optimized within the inner loop of learning. In this work, we investigate blending the two methodologies for learning models of sparse data. In particular, we use the parameters predicted by the inference network as an initialization and optimize them further during learning. When modeling high-dimensional sparse data, we show that updating the local variational parameters yields generative models with better held-out likelihood, particularly for deeper generative models.

What purpose is served by fitting bigger, deeper, more powerful generative models? Breiman (2001) argues that statistical discriminative modeling falls into two schools of thought: the data modeling culture and the algorithmic modeling culture. The former advocates the use of predictive models that assume interpretable, mechanistic processes while the latter advocates the use of black box techniques with an emphasis on prediction accuracy. Breiman's arguments also ring true about the divide between deep generative models with complex conditional distributions and simpler, more interpretable statistical models. Consider a classic model such as Latent Dirichlet Allocation (Blei et al., 2003) . It is outperformed in held-out likelihood (Miao et al., 2016) by deeper generative models and assumes a simple probabilistic process for data generation that is unlikely to hold in reality. Yet, its generative semantics lend it a distinct advantage: interpretability. The word-topic matrix in the model allows practitioners to read off what the model has learned about the data. Is there a natural way to interpret the generative model when the conditional distributions are parameterized by a deep neural network?

Our second contribution is to introduce a simple, easy to implement method to interpret what is being learned by generative models such as DLGMs whose conditional probabilities are parameterized by deep neural networks. Our hope is to narrow the perceived gulf between a complex generative model's representational power and its interpretability. We use the Jacobian of the conditional distribution with respect to latent variables in the Bayesian network to form embeddings (or Jacobian vectors) of the observations. We investigate the properties of the Jacobian vectors obtained from deeper, more non-linear generative models.

2. BACKGROUND:

Generative Model: We consider learning in generative models of the form shown in Figure 1 . We observe a set of D word count vectors x 1:D , where x dv denotes the number of times that word index v ∈ {1, . . . , V } appears in document d. We assume we are given the total number of words per document N d ≡ v x dv , and that x d was generated via the following generative process:

z d ∼ N (0, I); γ(z d ) ≡ MLP(z d ; θ); µ(z d ) ≡ exp{γ(z d )} v exp{γ(z d ) v } ; x d ∼ Multinomial(µ(z d ), N d ).

(1)

That is, we draw a Gaussian random vector, pass it through a multilayer perceptron (MLP) with parameters θ, pass the resulting vector through the softmax (a.k.a. multinomial logistic) function, and sample N d times from the resulting distribution over the vocabulary.1 

Variational Learning: For ease of exposition notation we drop the subscript on x d to form x referring to a single data point. We need to approximate the intractable posterior distribution p(z|x) during learning. Using the well-known variational principle, we can obtain the lower bound on the log marginal likelihood of the data (or L(x; θ, φ)) in Eq. 2. where the inequality is by Jensen's inequality.

EQUATION

We leverage an inference network or recognition network (Hinton et al., 1995) , a neural network which approximates the intractable posterior, during learning. This is a parametric conditional distribution that is optimized to perform inference. Kingma & Welling (2014); Rezende et al. (2014) use a neural net (with parameters φ) to parameterize q φ (z|x). The challenge in the resulting optimization problem is that the lower bound (2) includes an expectation w.r.t. q φ (z|x), which implicitly depends on the network parameters φ. This difficulty is overcome by using stochastic backpropagation.

With a normal distribution as our variational approximation we have that q φ (z|x) ∼ N (µ φ (x), Σ φ (x)). µ φ (x), Σ φ (x) are functions of the observation x, and we denote by ψ(x) := {µ φ (x), Σ φ (x)} the local variational parameters predicted by the inference network. A simple transformation allows one to obtain unbiased Monte Carlo estimates of the gradients of Eq φ (z|x) [log p θ (x|z))] with respect to φ. If we assume the prior p(z) is also normally distributed, the KL and its gradients may be obtained analytically. Throughout this paper we will use θ to denote the parameters of the generative model, and φ to denote the parameters of the inference network.

3. METHODOLOGY:

Inference with Global Information: Sparse data typically exhibits long tails and learning in the presence of rare features is challenging. Inference networks learn to regress to the optimal posterior parameters for every data point and global information about the relative frequencies of the individual features in the training distribution may present valuable information during learning.

The simplest way to incorporate first order statistics across the training data into the inferential process is to condition on tf-idf (Baeza-Yates et al., 1999) features instead of the raw-counts. tf-idf is one of the most widely used techniques in information retrieval. In the context of building bag-of-words representations for documents, tf-idf re-weight features to increase the influence of rarer words while decreasing the influence of common words appearing in all documents. The tf-idf-transformed word-count vector is xdv ≡ x dv log

D d min{x d v ,1}

. After applying transform, the resulting vector x is normalized by its L2 norm. It's worthwhile to note that leveraging first-order statistics for inference is difficult in the traditional paradigm of tracking variational parameters for each data point but is easy with inference networks.

Optimizing Local Variational Parameters: The inference network initially comprises a randomly initialized neural network. The predictions of the inference network early in optimization are suboptimal variational parameters used to derive gradients of the parameters of the generative model. This induces noise and bias to the gradients used to update the parameters of the generative model; this noise and bias may push the generative model towards a poor local optimum. Previous work has suggested that deep neural networks (which form the conditional probability distributions p θ (x|z)) are sensitive to initialization (Glorot & Bengio, 2010; Larochelle et al., 2009) .

To avoid these issues, we only use the local variational parameters ψ(x) predicted by the inference network to initialize an iterative optimizer that maximizes the ELBO with respect to ψ; we use the optimized variational parameters ψ (x) to derive gradients for the generative model. We then train the inference network using stochastic backpropagation and gradient descent, holding the parameters of the generative model θ fixed. Our procedure is detailed in Algorithm 1.

Algorithm 1 Pseudocode for Learning: We evaluate expectations in L(x) (see Eq. 2) using a single sample from the variational distribution and aggregate gradients across mini-batches. M = 1 corresponds to performing no additional optimization to the variational parameters We update θ, ψ(x), φ using stochastic gradient descent with adaptive learning rates η θ , η ψ(x) , η φ obtained via ADAM (Kingma & Ba, 2015) Inputs:

Dataset D := [x 1 , . . . , x D ], Inference Model: q φ (z|x), Generative Model: p θ (x|z), p(z) while notConverged() do 1. Sample datapoint: x ∼ D 2. Estimate local variational parameters ψ(x) 1 using q φ (z|x) 3. Estimate ψ(x) M ≈ ψ (x) = arg max ψ(x) L(x; θ; ψ(x)) via SGD as: m = 1, . . . , M , ψ(x) m+1 = ψ(x) m + η ψ(x) ∂L(x;θ,ψ(x)m) ∂ψ(x)m 4. Update θ as: θ ← θ + η θ ∇ θ L(x; θ, ψ(x) M ) 5. Update φ as: φ ← φ + η φ ∇ φ L(x; θ, ψ(x)) end while

Introspection: Linear models are inherently interpretable. Consider linear regression, factor analysis (Spearman, 1904) , and latent Dirichlet allocation (LDA; Blei et al., 2003) , which (standardizing notation) assume the following relationships:

Regression: E[y|x] = W x + b; Factor Analysis: x ∼ N (0, I); E[y|x] = W x + b; Latent Dirichlet Allocation: x ∼ Dirichlet(α); E[y|x] = W x.

(3)

In each case, we need only inspect the parameter matrix W to answer the question "what happens to y if we increase x k a little?" The answer is clear-y moves in the direction of the kth row of W . We can ask this question differently and get the same answer: "what is the derivative ∂E[y|x] ∂x ?" The answer is simply the parameter matrix W .

For models as in Fig 1 , the variability in the training data is assumed to be due to the single latent state z. The relationship between latent variables z and observations x cannot be quickly read off of the parameters θ. But we can still ask what happens if we perturb z by some small dz-this is simply the directional derivative ∂E[x|z] ∂z dz. We can interpret this Jacobian matrix in much the same way we would a factor loading matrix, with two main differences. First, the Jacobian matrix

∂E[x|z] ∂z

varies with z-the interpretation of z may change significantly depending on context. Second, DLGMs exhibit rotational symmetry-the prior on z is rotationally symmetric, and the MLP can apply arbitrary rotations to z before applying any nonlinearities, so a priori there is no "natural" set of basis vectors for z. For a given Jacobian matrix, however, we can find the most significant directions via a singular value decomposition (SVD).

Jacobian Vectors: We present our method to generate embeddings from Bayesian networks of the form Figure 1 . We consider three variants of Jacobian embedding vectors, based on the unnormalized potentials from the MLP, logarithmic probabilities, and linear probabilities respectively:

EQUATION

For any z, {J (z) log , J (z) pot , J (z) prob } ∈ R V ×K where K is the latent dimension and V is the dimensionality of the observations. It is this matrix that we use to form embeddings. We denote by u i the Jacobian vector obtained from the row of the Jacobian matrix. When not referring to a particular variant, we use J (z) to denote the Jacobian matrix. J (z) is a function of z leaving open the choice of where to evaluate this function. The semantics of our generative model suggest a natural choice:

J mean := E p(z) [J (z)]

. This set of embeddings captures the variation in the output distribution with respect to the latent state across the prior distribution of the generative model. Additionally, one may also evaluate the Jacobian at the approximate posterior corresponding to an observation x. We study how this may be used to obtain contextual word-vectors.

In frameworks that support automatic differentiation (e.g., Theano; Theano Development Team, 2016), J (z) is readily available and we estimate J mean via Monte-Carlo sampling from the prior.

Deriving Jacobian Vectors: For simplicity, we derive the functional form of the Jacobian in a linear model i.e where γ(z d ) = W z d (c.f Eq 1). We drop the subscript d and denote by γ i (z), the ith element of the vector γ(z).

p(x i = 1|z) = exp(γ i (z)) j exp(γ j (z))

and

γ i (z) = w T i z For linear models, ∇ z γ i (z) = w i directly corresponds to J (z) pot . Noting that ∇ z exp(γ i (z)) = exp(γ i (z))∇ z γ i (z) and ∇ z j exp(γ j (z)) = j exp(γ j (z))∇ z γ j (z)

, we estimate J (z) prob as:

∇ z p(x i = 1|z) = ∇ z exp(γ i (z)) j exp(γ j (z)) = j exp(γ j (z))∇ z exp(γ i (z)) -exp(γ i (z))∇ z j exp(γ j (z)) ( j exp(γ j (z))) 2 = j exp(γ j (z)) exp(γ i (z))w i -exp(γ i (z)) j exp(γ j (z))w j ( j exp(γ j (z))) 2 = p(x i = 1|z)w i -p(x i = 1|z) j p(x j = 1|z)w j = p(x i = 1|z)(w i - j p(x j = 1|z)w j )

Similarly, we may compute J (z) log :

EQUATION

Denoting a word-pair vector as w i -w j , where w i , w j are columns of the matrix W . If we define the set of all word-pair vectors as S, then Eq 5 captures the idea that the vector representation for a word i lies in the convex hull of S. Furthermore, the word vector's location in CONV(S) is determined by the likelihood of the pairing word (x j ) under the model p(x j = 1|z).

When we use a non-linear conditional probability distribution J (z) log becomes:

∇ z log p(x i = 1|z) = j p(x j = 1|z)(∇ z γ i (z) -∇ z γ j (z))

where ∇ z γ i (z) is a non-linear function of z. To the best of our knowledge, Jacobian Vectors and their properties have not been studied.

4. RELATED WORK:

Learning in Deep Generative Models: Salakhutdinov & Larochelle (2010) optimize the local variational parameters obtained from an inference network when learning deep Boltzmann machines. For DLGMs, Hjelm et al. (2016) also consider the optimization of the local variational parameters, though their exposition focuses on deriving an importance-sampling-based bound to use during learning in deep generative models with discrete latent variables. Their experimental results suggest the procedure does not improve performance much on the binarized MNIST dataset. This is consistent with our experience-we found that our secondary optimization procedure helped more when modeling sparse, high-dimensional text data than when modeling MNIST.

Leveraging Gradient Information: The algorithmic procedure for obtaining Jacobian Vectors that we propose resembles that used to derive Fisher Score features. For an data point X under a parameteric distribution p(X; θ), the Fisher scores is defined as U X = ∇ θ log p(X; θ). Jaakkola & Haussler (2007) similarly use U X to form a kernel function for subsequent use in a discriminative classifier.

The intuition behind such methods is to note that the derivative of the log-probability with respect to the parameters of the generative model encodes all the variability in the input under the generative process. We rely on a related intuition, although our motivations are different; we are interested in characterizing isolated features such as words, not vector observations such as documents. Also, we consider Jacobians with respect to per-observation latent variables z, rather than globally shared parameters θ.

In the context of discriminative modeling, (Erhan et al., 2009) use gradient information to study the patterns with which neurons are activated in a deep neural networks while (Wang et al., 2016) use the spectra of the Jacobian to study the complexity of the functions learned by neural networks. Electronic Health Record data. The models above explicitly condition the probability of a word on its nearby context. In contrast, our model models the probability of a word as it appears in the document (or rather, conditioned on its global context). Augmenting the generative model in Figure 1 to incorporate local context is a possible direction for future work.

Miao et al. ( 2016) learn a shallow log-linear model on text data and obtain embeddings for words from the weight matrix that parameterize their generative model. Li et al. (2016) propose a modification to LDA that explicitly models representations for word in addition to modeling the word-topic structure.

5. EVALUATION:

Text Data: We study the effect of further optimization of the variational parameters and inference with tf-idf features on the two datasets of varying size: the smaller 20Newsgroups (Lang, 2008) (train/valid/test: 10768/500/7505, V : 2000) and the larger RCV2 (Lewis et al., 2004 ) dataset (train/valid/test: 789414/5000/10000, V : 10000). We follow the preprocessing procedure defined in (Miao et al., 2016) for both datasets. We also train models on the Wikipedia corpus used in (Huang et al., 2012) . We remove stop words, words appearing less than ten times in the dataset. and select our vocabulary to comprise the union of the top 20000 words in the corpus, the words in the WordSim353 (Finkelstein et al., 2001) Training Procedure: On all datasets, we train shallow log-linear models (γ(z) = W z + b) and deeper three-layer DLGMs (γ(z) = MLP(z; θ)). We vary the number of secondary optimization steps M = 1, 100 (cf. Algorithm 1) to study the effect of optimization on ψ(x) with ADAM (Kingma & Ba, 2015) . We use a mini-batch size of 500, a learning rate of 0.01 for ψ(x) and 0.0008 for θ, φ.

The inference network was fixed to a two-layer MLP whose intermediate hidden layer h(x) was used to parameterize the mean and diagonal log-variance µ(x), log Σ(x). To evaluate the quality of the learned generative models, we report an upper bound on perplexity (Mnih & Gregor, 2014) given by exp(-

1 N i 1 Ni log p(x i ))

where log p(x i ) is replaced by Eq 2. The notation 3-M100-tfidf indicates a model where the MLP parameterizing γ(z) has three hidden layers, the local variational parameters are updated 100 times before an update of θ and tf-idf features were used in the inference network.

Improving Learning: Table 1 depicts our results on 20newsgroups and RCV2. On the smaller dataset, we find that the deeper models overfit quickly and are outperformed by shallow generative models. On the larger datasets, the deeper models' capacity is more readily utilized yielding better generalization. The use of tf-idf features always helps learning on smaller datasets. On larger datasets, the benefits are smaller when we also optimize ψ(x). Finally, the optimization of the local variational parameters appears to help most on the larger datasets. To investigate how this occurs, we plot the held-out likelihood versus epochs. For models trained on the larger RCV2 (Figure 2a ) and Wikipedia (Figure 2b ) datasets, the larger deep generative models converge to better solutions (and in fewer passes through the data) with the additional optimization of ψ(x).

To study the effect of investigate where optimizing ψ(x) is particularly effective on, we train a three layer model on different subsets of the Wikipedia dataset. The subsets are created by selecting the top K most frequently occurring features in the data. Our rationale is that by holding everything fixed and varying the level of sparsity (datasets with smaller values of K are less sparse) in the data, we can begin to understand when our method is most helpful. On held-out data, we compute the difference between the perplexity when the model is trained with M = 1 (denoted P M 1 ) and M = 100 (denoted P M 100 ) and compute the relative decrease in perplexity obtained as P M 1 -P M 100 P M 100 . The results are depicted in Figure 2c where we see that our method improves learning as a function of the dimensionality of the data.

In Table 5 in the supplementary material, we study the effect of varying the parameters of the inference network. There, we perform a small grid search over the hidden dimension and the number of layers in the inference network and find that optimizing the variational parameters continues to produces models with lower overall perplexity.

Jacobian Vectors: Our first avenue for introspection into the learned generative model is using log-singular values of the Jacobian matrix. Since the Jacobian matrix precisely encodes how sensitive the outputs are with respect to the inputs, the log-singular value spectrum of this matrix directly captures the amount of variance in the data explained by the latent space. Said differently, we can read off the number of active units in the DLGM or VAE by counting the number of log-singular values larger than zero. Furthermore, this method of introspection depends only on the parameters of the generative model. In Figure 2d , 2e, we see that for larger models continuing to optimize the variational parameters allows us to learn models that use many more of the available latent dimensions. This suggests that, when fit to text data, DLGMs may be particularly susceptible to the overpruning phenomenon noted by Burda et al. (2015) . In Figure 2 , the lower held-out perplexity and the increased utilization of the latent space suggest that the continued optimization of the variational parameters yields more powerful generative models.

We investigate how the Jacobian matrix may be used for model introspection by studying the qualitative properties of J log mean on DLGMs (of type "3-M100-tfidf") trained on two, diverse sets of data. We form a Monte Carlo estimate of J log mean using 400 samples. The cosine distance is (Blei et al., 2003) , Replicated Softmax (RSM) (Hinton & Salakhutdinov, 2009) , Sigmoid Belief Networks (SBN) and Deep Autoregressive Networks (DARN) (Mnih & Gregor, 2014) , Neural Variational Document Model (Miao et al., 2016) . K denotes the latent dimension in our notation. Right: DLGMs on text data with K = 100.

We vary the features presented to the inference network q φ (z|x) during learning between: normalized count vectors (

x V i=1 x i

, denoted "norm") and normalized tf-idf (denoted "tf-idf") features. The x-axis is not directly comparable on running times since larger values of M take longer during training. We find that learning with M = 100 takes approximately 15 times as long per mini-batch of size 500 on the text datasets. Figure 2c compares relative differences in the final held-out perplexity, denoted P , between models trained using M = 1 and M = 100. On the x-axis, we vary the number of features used in the dataset. Figure 2d , 2e depict the sorted log singular values of J log mean .

Model:

used to define neighbors of words in the embedding space of the Jacobian and spectral clustering (Von Luxburg, 2007) is used to form clusters.

In Table 2a , we visualize some of the nearest neighbors of words using J log mean obtained from models trained on the Wikipedia dataset. The neighbors are semantically sensible. Instead of evaluating the Jacobian at L points z 1:L ∼ p(z), one may instead evaluate it at z 1:L ∼ q(z|x) for some x. In Table 2b , we select three polysemous query words alongside "context words" that disambiguate the query's meaning. For each word-context pair, we create a document comprising a subset of words in the the (a) Word Embeddings (Nearest Neighbors): We visualize nearest neighbors of word embeddings. We exclude plurals of the query and other words in the neighborhood. (Huang et al., 2012) . C&W uses embeddings from the language model of (Collobert & Weston, 2008) . Glove corresponds to embeddings by (Pennington et al., 2014) . The learning algorithm for our embeddings does not use local context. context's Wikipedia page. Then, we use the learned inference network to perform posterior inference to evaluate J log mean at the corresponding q(z|x). This yields a set of contextual Jacobian vectors. We display the nearest neighbors for each word under different contextual Jacobian vectors and find that, while not always perfect, they capture different contextually relevant semantics. The take-away here is that by combining posterior inference in this Bayesian network with our methodology of introspecting the model, one obtains different context-specific representations for the observations despite not having been trained to capture this explicitly.

Query:

In Table 8b (appendix), we visualize clusters formed from the embeddings of medical diagnosis codes to find that they exhibit topical coherence. In Table 2c , the nearest neighbors of drugs include other drugs prescribed in conjunction with or as a replacement for the query drug. For diagnosis codes such as "Asbestosis", the nearest neighbors are symptoms and procedures associated with the Table 4 : Medical Relatedness Measure: Evaluating the quality of embedding using medical (NDF-RT and CCS) ontologies. Each column corresponds to a measure of how well the embedding space is amenable to performing analogical reasoning (NDF-RT) or clusters meaningfully (CCS). A higher number is better. SCUIs corresponds to the application of the method developed by (Choi et al., 2016) on data released by (Finlayson et al., 2014) . The learning algorithm for our embeddings does not use local context. disease. Finally, for a qualitative evaluation of Jacobian vectors obtained from a model trained on movie ratings, we refer the reader to the appendix.

Models:

The Semantics of Embeddings: We evaluate the vector space representations that we obtain from J mean on benchmarks (such as WordSim353 (Finkelstein et al., 2001) and SCWS (Huang et al., 2012) ) that attempt to measure the similarity of words. The algorithmically derived measure of similarity is compared to a human-annotated score (between one and ten) using the Spearman rank correlation. The models that we compare to primarily use local context, which yields a more precise signal about the meanings of particular words. Closest to us in terms of training procedure is (Huang (G)) in Table 3a , whose model we outperform. Finding ways to incorporate local context is fertile ground for future work on models tailor-made for extracting embeddings.

For medical codes, we follow the method in (Choi et al., 2016) . The authors build two kinds of evaluations to estimate whether an embedding space of medical diagnosis codes captures medically related concepts well. MRM NDF-RT (Medical Relatedness Measure under NDF-RT) leverages a database (NDF-RT) to evaluate how good an embedding space is at answering analogical queries between drugs and diseases such as u Diabetes ≈ u Metformin -(u Lung Cancer -u Tarceva ). (Metformin is a diabetic drug and Tarceva is used in the treatment of lung cancer). The evaluation (MRM CCS ) measures if the neighborhood of the diagnosis codes is medically coherent using a predefined medical ontology (CCS) as ground truth. The number computed may be thought of as a measure of precision, where a higher number is better. We refer the reader to the appendix for additional details.

Table 4 details the results on evaluating the medical embeddings. Once again, the baselines we compare (Choi et al., 2016) are variants of Word2Vec that maximize the likelihood of the diagnosis codes conditioned on carefully crafted contexts. Our method performs comparably to the baselines, even though it relies exclusively on global context and was not designed with this task in mind. This setting depicts an instance where Jacobian vectors resulting from a deeper, better-trained model outperform those from a shallow model, highlighting the importance of a method of interpretation agnostic to the structure of the conditional probability functions in the generative model.

Between the three choices of Jacobian vectors, we found that all three perform comparably on the word similarity evaluation with J prob mean slightly outperforming the others. On the medical data, with we found similar results aside from a few cases where J prob mean did not perform well. For deeper models, we found that optimizing ψ(x) improved the quality of the obtained Jacobian vectors on text and medical data. The full versions of Tables 3 and 4 can be found in the appendix.

6. DISCUSSION:

We explored techniques to improve inference and learning in deep generative models of sparse non-negative data. We also developed and explored a novel, simple, yet effective method to interpret the structure of the non-linear generative model via embeddings obtained from the Jacobian matrix relating latent variables to observations. The embeddings are evaluated qualitatively and quantitatively, and were seen to exhibit interesting semantic structure across a variety of domains. Studying the effects of varying the priors on the latent variables, conditioning on context, and varying the neural architectures that parameterize the conditional distributions suggest avenues for blending ideas from generative modeling and Bayesian inference into building more powerful embeddings for data.

:

In keeping with common practice, we neglect the multinomial base measure termN ! x 1 !•••x V ! ,which amounts to assuming that the words are observed in a particular order.

Figures and Tables:

Figure 1: Deep Latent Gaussian Model: The Bayesian network depicted here comprises a single latent variable with the conditional probability p(x|z) defined by a deep neural network with parameter θ. The dotted line represents the inference network parameterized by φ, which is used for posterior inference at train and test time.

Figure 2: Mechanics of Learning: Validation Perplexity and Log-singular Values of J log mean : Best viewed in color. For the RCV2 and Wikipedia (large) datasets, we visualize the validation perplexity as a function of epochs. The solid lines indicate the validation perplexity for M = 1 and the dotted lines indicate M = 100.The x-axis is not directly comparable on running times since larger values of M take longer during training. We find that learning with M = 100 takes approximately 15 times as long per mini-batch of size 500 on the text datasets. Figure2ccompares relative differences in the final held-out perplexity, denoted P , between models trained using M = 1 and M = 100. On the x-axis, we vary the number of features used in the dataset. Figure2d, 2e depict the sorted log singular values of J log mean .

and the words in the Stanford Contextual Word Similarity Dataset (SCWS)(Huang et al., 2012). The resulting dataset is of size train/valid: 1212781/1000 and V :20253.EHR Data: We train shallow and deep generative models on a dataset constructed from Electronic Medical Records. The dataset comprises 185000 patients where each patient's data across time was aggregated to create a bag-of-diagnosis-codes representation of the patient. The vocabulary comprises four different kinds of medical diagnosis codes: ICD9 (diagnosis), LOINC (laboratory tests), NDC (prescription medication), CPT (procedures). For a single patient, we have 51321 diagnosis codes.

Table content: <table/>

Test Perplexity: Left: Baselines Results on the 20newsgroups and RCV1-v2 dataset Legend: LDA

Table content: <table/>

We visualize the nearest neighbors under the Jacobian vector induced by the posterior distribution of a document created based on the context word. Medical Embeddings (Nearest Neighbors): We evaluate nearest neighbors of selected diagnosis, drug and procedure codes (ignoring duplicates and shortening some code names). Metformin, Glimepiride, Pioglitazone and Avandia are diabetic drugs. A contour meter is an instrument to track blood glucose. Advair, Albuterol, Proventil and Spiriva are prescribed to patients with chronic obstructive pulmonary disease (COPD)

Table content: <table><tr><td/><td/><td colspan="2">Neighborhood</td></tr><tr><td>intelligence</td><td/><td colspan="2">espionage, secrecy, interrogation, counterterrorism</td></tr><tr><td>zen</td><td/><td colspan="2">dharma, buddhism, buddhas, meditation,yoga</td></tr><tr><td>artificial</td><td/><td colspan="2">artificially, molecules, synthetic, soluble</td></tr><tr><td>military</td><td/><td colspan="2">civilian, armys, commanders, infantry</td></tr><tr><td colspan="3">(b) Word Embeddings (Polysemy): Word Context</td><td>Neighboring Words</td></tr><tr><td>crane</td><td>construction</td><td/><td>lifting, usaaf, spanned, crushed, lift</td></tr><tr><td/><td>bird</td><td/><td>erected, parkland, locally, farmland, causeway</td></tr><tr><td>bank</td><td>river</td><td/><td>watershed, footpath, confluence, drains, tributary</td></tr><tr><td/><td>money</td><td/><td>banking, government, bankers, comptroller, fiscal</td></tr><tr><td>fires</td><td>burn</td><td/><td>ignition, combustion, engines, fuel, engine</td></tr><tr><td/><td>layoff</td><td/><td>thunderstorm, grassy, surrounded, walkway, burning</td></tr><tr><td>(c) Code</td><td/><td colspan="2">Neighboring Codes</td></tr><tr><td>Metformin</td><td/><td colspan="2">Glimepiride, Avandia, Contour Meter, Pioglitazone</td></tr><tr><td colspan="2">Spiriva (Bronchodilator)</td><td colspan="2">Advair , Albuterol , Proventil</td></tr><tr><td>Asbestosis</td><td/><td colspan="2">Exposure To Asbestos , Coal Workers' Pneumoconiosis, Ct Scan Chest</td></tr></table>

Qualitative evaluation of Jacobian Vectors : In Table2a and 2b, we evaluate the embeddings of words. In Table8band 2c we evaluate embeddings of medical diagnosis codes.

Table content: <table><tr><td>Models</td><td>Spearman ρ ×100</td><td>Models</td><td>Spearman ρ ×100</td></tr><tr><td>Huang(G)</td><td>22.8</td><td>Huang (S)</td><td>58.6</td></tr><tr><td>Huang</td><td>71.3</td><td>Huang (M)</td><td>65.7</td></tr><tr><td>Glove</td><td>75.9</td><td>C&amp;W</td><td>57</td></tr><tr><td>C&amp;W</td><td>55.3</td><td>tf-idf-S</td><td>26.3</td></tr><tr><td>ESA</td><td>75</td><td>Pruned tf-idf-S</td><td>62.5</td></tr><tr><td>Tiered Pruned tf-idf</td><td>76.9</td><td>Pruned tf-idf-M</td><td>60.5</td></tr><tr><td>1-M1 J prob mean</td><td>69.7</td><td>1-M1 J prob mean</td><td>61.7</td></tr><tr><td>3-M100 J prob mean</td><td>59.6</td><td>3-M100 J prob mean</td><td>59.5</td></tr><tr><td colspan="2">(a) WordSim353: "G" denotes the model in</td><td colspan="2">(b) SCWS: (S) denotes a single prototype</td></tr><tr><td colspan="2">Huang et al. learned only with global context</td><td colspan="2">approach versus (M) that denotes a multi-</td></tr><tr><td>in the document.</td><td/><td colspan="2">prototype approach (that leverages context)</td></tr></table>

Semantic Similarity in Words: The baseline results are taken from

Table content: <table/>

Peer review (iclr2017):
The paper claims improved inference for density estimation of sparse data (here text documents) using deep generative Gaussian models (variational auto-encoders), and a method for deriving word embeddings from the model*s generative parameters that allows for a degree of interpretability similar to that of Bayesian generative topic models. To discuss the contributions I will quickly review the generative story in the paper: first a K-dimensional latent representation is sampled from a multivariate Gaussian, then an MLP (with parameters 	heta) predicts unnormalised potentials over a vocabulary of V words, the potentials are exponentiated and normalised to make the parameters of a multinomial from where word observations are repeatedly sampled to make a document. Here intractable inference is replaced by the VAE formulation where an inference network (with parameters phi) independently predicts for each document the mean and variance of a normal distribution (amenable to reparameterised gradient computation). The first, and rather trivial, contribution is to use tf-idf features to inject first order statistics (a global information) into local observations. The authors claim that this is particularly helpful in the case of sparse data such as text. The second contribution is more interesting. In optimising generative parameters (	heta) and variational parameters (phi), the authors turn to a treatment which is reminiscent of the original SVI procedure. That is, they see the variational parameters phi as *global* variational parameters, and the predicted mean mu(x) and covariance Sigma(x) of each observation x are treated as *local* variational parameters. In the original VAE, local parameters are not directly optimised, instead they are indirectly optimised via optimisation of the global parameters utilised in their prediction (shared MLP parameters). Here, local parameters are optimised holding generative parameters fixed (line 3 of Algorithm 1). The optimised local parameters are then used in the gradient step of the generative parameters (line 4 of Algorithm 1). Finally, global variational parameters are also updated (line 5). Whereas indeed other authors have proposed to optimise local parameters, I think that deriving this procedure from the more familiar SVI makes the contribution less of a trick and easier to relate to. Some things aren*t entirely clear to me. I think it would have been nice if the authors had shown the functional form of the gradient used in step 3 of Algorithm 1. The gradient step for global variational parameters (line 5 of Algorithm 1) uses the very first prediction of local parameters (thus ignoring the optimisation in step 3), this is unclear to me. Perhaps I am missing a fundamental reason why that has to be the case (either way, please clarify). The authors argue that this optimisation turns out helpful to modelling sparse data because there is evidence that the generative model p_	heta(x|z) suffers from poor initialisation. Please, discuss why you expect the initialisation problem to be worse in the case of sparse data. The final contribution is a neat procedure to derive word embeddings from the generative model parameters. These embeddings are then used to interpret what the model has learnt. Interestingly, these word embeddings are context-sensitive once that the latent variable models an entire document. About Figures 2a and 2b: the caption says that solid lines indicate validation perplexity for M=1 (no optimisation of local parameters) and dashed lines indicate M=100 (100 iterations of optimisation of local parameters), but the legends of the figures suggest a different reading. If I interpret the figures based on the caption, then it seems that indeed deeper networks exposed to more data benefit from optimisation of local parameters. Are the authors pretty sure that in Figure 2b models with M=1 have reached a plateau (so that longer training would not allow them to catch up with M=100 curves)? As the authors explain in the caption, x-axis is not comparable on running time, thus the question. The analysis of singular values seems like an interesting way to investigate how the model is using its capacity. However, I can barely interpret Figures 2c and 2d, I think the authors could have walked readers through them. As for the word embedding I am missing an evaluation on a predictive task. Also, while illustrative, Table 2b is barely reproducible. The text reads *we create a document comprising a subset of words in the the context’s Wikipedia page.* which is rather vague. I wonder whether this construct needs to be carefully designed in order to get Table 2b. In sum, I have a feeling that the inference technique and the embedding technique are both useful, but perhaps they should have been presented separately so that each could have been explored in greater depth.