Title: Inference and Introspection in Deep Generative Models of Sparse Data
Authors: Rahul G. Krishnan
Matthew Hoffman
Venue: ICLR 2017
Keywords: Unsupervised Learning
Deep learning

Abstract: Deep generative models such as deep latent Gaussian models (DLGMs) are powerful and popular density estimators. However, they have been applied almost exclusively to dense data such as images; DLGMs are rarely applied to sparse, high-dimensional integer data such as word counts or product ratings. One reason is that the standard training procedures find poor local optima when applied to such data. We propose two techniques that alleviate this problem, significantly improving our ability to fit DLGMs to sparse, high-dimensional data. Having fit these models, we are faced with another challenge: how to use and interpret the representation that we have learned? To that end, we propose a method that extracts distributed representations of features via a simple linearization of the model. 

TL;DR: We study two techniques to improve learning in deep generative models on sparse, high-dimensional text data. We also propose an algorithmic tool to visualize and introspect arbitrarily deep learned models. 

Review discussion:
Am I to understand that using tf-idf weighting is really the first contribution? Seems like a good idea, but weird to highlight to that level.
In Algorithm 1, step 8, it's not clear how psi is being reestimated.
Similarly, step 5 could be clearer about what exactly is being sampled and how this is used to compute the gradient.
How much slower is it to iteratively optimize the ELBO compared to just using the inference network?
- Notation clarification: x_{1:D} is a set of D sparse V-dimensional vectors? x_{dv} is the number of times the v^th word in the dictionary appeared in document d? What is x? The same as x_d?
- Experiment: Did you consider using your embeddings to solve a prediction task (text categorization in RCV2) and see if what they capture yields a better result than using simply bag-of-words tf-idf?
[R1] Yes.
Though our point was to highlight that instead of just conditioning on the input directly, thinking about how to incorporate other characteristics of the data the inference network typically has no access to (such as global first order statistics, which in the case of non-negative data corresponds to tf-idf features) helps learning.  We have updated the algorithm box to make clear the role of the additional optimization of the local variational parameters in the revised version.
[R4] The current implementation is about 15x slower (we have emphasized this in the caption for Fig. 2) since we do 100 steps of optimizing the variational parameter. It is possible that this additional optimization is primarily helpful in the first few epochs after which one could resume training normally, though we did not experiment with such variants.
[R3] x is the same as x_d. We have clarified the notation in the revised version. We haven’t tried our embeddings on prediction tasks for text as yet.
This paper presents a small trick to improve the model quality of variational autoencoders (further optimizing the ELBO while initializing it from the predictions of the q network, instead of just using those directly) and the idea of using Jacobian vectors to replace simple embeddings when interpreting variational autoencoders.
The idea of the Jacobian as a natural replacement for embeddings is interesting, as it does seem to cleanly generalize the notion of embeddings from linear models. It'd be interesting to see comparisons with other work seeking to provide context-specific embeddings, either by clustering or by smarter techniques (like Neelakantan et al, Efficient non-parametric estimation of multiple embeddings per word in vector space, or Chen et al A Unified Model for Word Sense Representation and Disambiguation). With the evidence provided in the experimental section of the paper it's hard to be convinced that the Jacobian of VAE-generated embeddings is substantially better at being context-sensitive than prior work.
Similarly, the idea of further optimizing the ELBO is interesting but not fully explored. It's unclear, for example, what is the tradeoff between the complexity of the q network and steps further optimizing the ELBO, in terms of compute versus accuracy.
Overall the ideas in this paper are good but I'd like to see them a little more fleshed out.
Thank you for your review and comments! We are currently looking into extensions of the model that incorporate local context during learning that will aid comparison to work by Neelakantan et. al and Chen et. al.
Regarding tradeoffs in accuracy, please see Appendix A where we have experimented with learning under different architectures of the inference network.
This paper introduces three tricks for training deep latent variable models on sparse discrete data:
1) tf-idf weighting
2) Iteratively optimizing variational parameters after initializing them with an inference network
3) A technique for improving the interpretability of the deep model
The first idea is sensible but rather trivial as a contribution. The second idea is also sensible, but is conceptually not novel. What is new is the finding that it works well for the dataset used in this paper.
The third idea is interesting, and seems to give qualitatively reasonable results. The quantitative semantic similarity results don’t seem that convincing, but I am not very familiar with the relevant literature and therefore cannot make a confident judgement on this issue.
Thank you for your review! We hope you reconsider your score based on some of the additional insights and experiments we have highlighted above.
First I would like to apologize for the delay in reviewing.
Summary : In this paper a variational inference is adapted to deep generative models, showing improvement for non-negative sparse dataset. The authors offer as well a method to interpret the data through the model parameters.
The writing is generally clear. The methods seem correct. The introspection approach appears to be original. I found very interesting the experiment on the polysemic word embedding. I would however have like to see how the obtained embedding would perform with respect to other more common embeddings in solving a supervised task.
Minor :
Eq. 2: too many closing parentheses
Thank you for your review! We’ve fixed the typo and addressed your other question above.
The paper claims improved inference for density estimation of sparse data (here text documents) using deep generative Gaussian models (variational auto-encoders), and a method for deriving word embeddings from the model's generative parameters that allows for a degree of interpretability similar to that of Bayesian generative topic models.
To discuss the contributions I will quickly review the generative story in the paper: first a K-dimensional latent representation is sampled from a multivariate Gaussian, then an MLP (with parameters \theta) predicts unnormalised potentials over a vocabulary of V words, the potentials are exponentiated and normalised to make the parameters of a multinomial from where word observations are repeatedly sampled to make a document. Here intractable inference is replaced by the VAE formulation where an inference network (with parameters \phi) independently predicts for each document the mean and variance of a normal distribution (amenable to reparameterised gradient computation).
The first, and rather trivial, contribution is to use tf-idf features to inject first order statistics (a global information) into local observations. The authors claim that this is particularly helpful in the case of sparse data such as text.
The second contribution is more interesting. In optimising generative parameters (\theta) and variational parameters (\phi), the authors turn to a treatment which is reminiscent of the original SVI procedure.
That is, they see the variational parameters \phi as *global* variational parameters, and the predicted mean \mu(x) and covariance \Sigma(x) of each observation x are treated as *local* variational parameters. In the original VAE, local parameters are not directly optimised, instead they are indirectly optimised via optimisation of the global parameters utilised in their prediction (shared MLP parameters). Here, local parameters are optimised holding generative parameters fixed (line 3 of Algorithm 1). The optimised local parameters are then used in the gradient step of the generative parameters (line 4 of Algorithm 1).
Finally, global variational parameters are also updated (line 5).
Whereas indeed other authors have proposed to optimise local parameters, I think that deriving this procedure from the more familiar SVI makes the contribution less of a trick and easier to relate to.
Some things aren't entirely clear to me. I think it would have been nice if the authors had shown the functional form of the gradient used in step 3 of Algorithm 1. The gradient step for global variational parameters (line 5 of Algorithm 1) uses the very first prediction of local parameters (thus ignoring the optimisation in step 3), this is unclear to me. Perhaps I am missing a fundamental reason why that has to be the case (either way, please clarify).
The authors argue that this optimisation turns out helpful to modelling sparse data because there is evidence that the generative model p_\theta(x|z) suffers from poor initialisation. Please, discuss why you expect the initialisation problem to be worse in the case of sparse data.
The final contribution is a neat procedure to derive word embeddings from the generative model parameters. These embeddings are then used to interpret what the model has learnt. Interestingly, these word embeddings are context-sensitive once that the latent variable models an entire document.
About Figures 2a and 2b: the caption says that solid lines indicate validation perplexity for M=1 (no optimisation of local parameters) and dashed lines indicate M=100 (100 iterations of optimisation of local parameters), but the legends of the figures suggest a different reading. If I interpret the figures based on the caption, then it seems that indeed deeper networks exposed to more data benefit from optimisation of local parameters. Are the authors pretty sure that in Figure 2b models with M=1 have reached a plateau (so that longer training would not allow them to catch up with M=100 curves)? As the authors explain in the caption, x-axis is not comparable on running time, thus the question.
The analysis of singular values seems like an interesting way to investigate how the model is using its capacity.
However, I can barely interpret Figures 2c and 2d, I think the authors could have walked readers through them.
As for the word embedding I am missing an evaluation on a predictive task.
Also, while illustrative, Table 2b is barely reproducible. The text reads "we create a document comprising a subset of words in the the context’s Wikipedia page." which is rather vague. I wonder whether this construct needs to be carefully designed in order to get Table 2b.
In sum, I have a feeling that the inference technique and the embedding technique are both useful, but perhaps they should have been presented separately so that each could have been explored in greater depth.
Thank you for your review and your detailed comments!
“(line 5 of Algorithm 1) uses the very first prediction of local parameters (thus ignoring the optimisation in step 3), this is unclear to me.”
We apologize for the misunderstanding. Our intention was to point out that line 5 updated the global variational parameters holding theta fixed by ascending in the ELBO. We’ve updated the notation to make this a bit clearer. In our early experiments, we also tried variants of line 5 such as updating phi to minimize the KL divergence with between psi_1 and psi_M but we found taking a gradient step to work comparably.
“Please, discuss why you expect the initialisation problem to be worse in the case of sparse data.”
We hypothesize this is because of rare features in sparse data. The gradient signal for learning the global variational parameters comes (via stochastic backpropagation) from taking derivatives of the ELBO. As the vocabulary size grows, each minibatch is less likely to have every word appearing in it and the gradient signal that tells the inference network how to perform inference for documents with rare words gets weaker. This in turn means the inference network takes longer to learn how to perform inference well, which affects the quality of inference early on in learning. By optimizing the variational parameters, the gradients for the generative model are rendered less susceptible to limitations in the inference network.
We attempted to probe into this question empirically with the experiment depicted in Figure 2(c). We took data that was originally sparse and we artificially made it less so by restricting the subset of features to the most frequently occurring ones. We found that our method helps most as the data gets sparser and more-high dimensional.
“Are the authors pretty sure that in Figure 2b models with M=1 have reached a plateau (so that longer training would not allow them to catch up with M=100 curves)? As the authors explain in the caption, x-axis is not comparable on running time, thus the question.”
While this is not an experiment we have tried on the wikipedia data, we have tried this on the RCV2 data where we trained for 400 rather than 200 epochs. We did not find that M=1 caught up to M=100.
On the issue of running time, the training curves presented in Figure 2(a) and (b) suggest that much of the gains in learning from our method are obtained early on in optimization and the number of steps of optimizing the variational parameters may be slowly reduced as a means of reducing the overall training time.
“I think the authors could have walked readers through them.”
We have added in more detail in the main text describing the significance of the two plots and why the Jacobian may be appropriate to study the number of active units in the model.
“I wonder whether this construct needs to be carefully designed in order to get Table 2b.”
This is a good point. We found that the contextual word vectors varied were depending on the length of the document we used for inference and how well the model was trained. We’re currently looking into methods to quantify this.
However, the table exists primarily as a proof of concept that such an operation is indeed possible with this method.
As for the code, the repository will contain finalized the procedure to setup the data for the experiment.
We thank all the reviewers for their careful reading of our work and for their suggestions.
Below we highlight additional experiments carried out based on suggestions by reviewers and clarify what set out to achieve with this work. Following which, we respond to individual reviewers inline.
A] We have added to the paper additional experiments and insights that we highlight here:
* We study how the proposed method for learning performs when varying the sparsity and dimensionality of the data. (Figure 2(c))
* We have added a section that sheds light on how the feature representations are constructed when using gradients with respect to z in linear and non-linear models. (the last subsection of Section 3: Methodology)
* We have studied the effect of varying hyperparameters in the inference network while continuing the optimize the variational parameters (Appendix A)
* We make code for our work available here: https://github.com/rahulk90/inference_introspection
B] In response to [R4] regarding the novelty of our work, we outline the contributions:
a] Improving learning: Underfitting is a problem in DLGMs and to improve learning on sparse non-negative data, we blend inference networks with more traditional methods from SVI. On two (large and small) standard text benchmarks, we study the effect of this additional optimization in the context of five different axes:
(i) Depth of the generative model
(ii) Inputs to the inference network
(iii) Number of steps of additional optimization of the variational parameters
(iv) Dimensionality of the data
(v) Depth and hidden size of the inference network: (*new* added to Appendix A)
Overall, across almost all these axes, we find that optimizing the variational parameters improves learning in large models on high-dimensional data.
b] Building Interpretable Models: We propose and study a novel method of introspection into learned DLGMs. We study two distinct uses for the Jacobian matrix:
(i) Visualizing Latent Space: We show that it may be used as a readily available measure to visualize how much of the latent space is utilized by the generative model.
Unlike the statistic used in Burda et. al (which depends on the learned inference network), the singular values of the Jacobian allow one to directly measure and read off the number of active units in a variational autoencoder. In this work, we use them to understand the effect of optimizing the variational parameters.
(ii) Embeddings: Given any learned model, we show that one may obtain embeddings of features using the Jacobian matrix. We evaluated these embeddings on three diverse kinds of data: text, movie ratings (supplementary material) and EHR data and found the method to yield qualitatively sensible results. Text is naturally sequential and the surrounding words present a natural source of context. In domains such as movie ratings, this may not be the case and defining a context in order to train a method such as Word2Vec may be difficult. The Jacobian represents a simple method for obtaining feature representations from any learned VAE. In domains such as medicine, interpretability plays a crucial role in the adoption of methods into the clinical workflow and this paper attempts to bridge the gap between powerful density estimation and interpretability.
C] Concerns brought up by multiple reviewers:
* Using vectors in a supervised task: This is an excellent suggestion that we are working on. In this work we investigated some of the interesting properties of Jacobian vectors and how they varied as a function of the generative model on diverse domains such as text and medical data.
* Quantitative Results: While our quantitative results are competitive on the medical data, for text data we believe that by augmenting the model with the ability to also take local context into account during learning (as is done in the works of Neelakantan et al and Chen et al brought up by R1), we can hope for further improvements and more comparisons in follow up work.
Indeed, we outperform a comparable method described in Huang et. al that leverages only global information.

Meta-review:
role: PC
data: comment: The paper introduces a number of ideas / heuristics for learning and interpreting deep generative models of text (tf-idf weighting, a combination of using an inference networks with direct optimization of the variational parameters, a method for inducing context-sensitive word embeddings). Generally, the last bit is the most novel, interesting and promising one, however, I agree with the reviewers that empirical evaluation of this technique does not seem sufficient. 
 
 Positive:
 -- the ideas are sensible 
 -- the paper is reasonably well written and clear
 
 Negative
 -- most ideas are not so novel
 -- the word embedding method requires extra analysis / evaluation, comparison to other methods for producing context-sensitive embeddings, etc

Decision: Reject