Title: Multi-modal Variational Encoder-Decoders
Authors: Iulian V. Serban
Alexander G. Ororbia II
Joelle Pineau
Aaron Courville
Venue: ICLR 2017
Keywords: Deep learning
Structured prediction
Natural language processing

Abstract: Recent advances in neural variational inference have facilitated efficient training of powerful directed graphical models with continuous latent variables, such as variational autoencoders. However, these models usually assume simple, uni-modal priors — such as the multivariate Gaussian distribution — yet many real-world data distributions are highly complex and multi-modal. Examples of complex and multi-modal distributions range from topics in newswire text to conversational dialogue responses. When such latent variable models are applied to these domains, the restriction of the simple, uni-modal prior hinders the overall expressivity of the learned model as it cannot possibly capture more complex aspects of the data distribution. To overcome this critical restriction, we propose a flexible, simple prior distribution which can be learned efficiently and potentially capture an exponential number of modes of a target distribution. We develop the multi-modal variational encoder-decoder framework and investigate the effectiveness of the proposed prior in several natural language processing modeling tasks, including document modeling and dialogue modeling.

TL;DR: Learning continuous multimodal latent variables in the variational auto-encoder framework for text processing applications.

Review discussion:
Since it gives such dramatically better results on 20-NG, what is the exact difference between G-NVDM and NVDM? Is it the interpolation/gating between prior and posterior? Or is it the learned prior? Why does learning a variance and mean for the Gaussian prior help, when the latent space is already transformed by an affine transformation by the decoder and encoder networks? It seems that a standard Gaussian prior should have the same expressive capabilities since the posterior should be able to learn to transform itself into the "whitened" prior space.
Hello,
The main difference between the G-NVDM and NVDM is that: 1) the prior mean and variance parameters are learned (by taking the derivative of the variational lower bound with respect to each), and 2) the posterior is calculated as an interpolation (or gating) of both the generated prior and posterior parameters.
Our choice for the above design decisions, for both the document models and dialogue models, is largely experimental as we empirically found that this led to better performance on the data-sets used. We interpret our choice for learning the prior as simply updating the model's general beliefs about the data (especially in the case of document models, where the prior is not conditioned on any context information like the dialogue models) since it might find that a standard Gaussian space is, perhaps, a less-than-reasonable prior. Since calculating the gradients with respect to the prior is rather cheap in a fully-differentiable architecture, we saw no reason not to try updating the prior in an effort to see if the baseline performance could be further improved.
However, the interpolation mechanism led to the greatest improvement in performance.
Thanks,
Alex And Iulian
Why there is no comparison between your models and the baseline momdels (e.g. NVDM, docNADE or Replicated Softmax) in the word query similarity test? The argument "It is clear that the piecewise variables affect what is uncovered by the model with respect to the data, as each model returns different, but relevant results with respect to the query word" does not seem to stand. The latent variable represents the document level semantics which is multi-modal, but the evaluation is carried out by only using the decoder parameter matrix which is fixed. Basically, the words are tied to fixed embeddings. How is the multi-modality of the document semantics related to this?
Hello,
The focus of the paper is to show that the proposed piecewise variables can benefit neural variational encoder-decoder architectures, such as the NVDM and the VHRED.
As such (and to also avoid further clutter/yet more space for our manuscript) we focused on comparing the models with piecewise variables against the improved baseline, the G-NVDM (which encompasses the NVDM -- our target was to beat out the improved competition/baseline, the G-NVDM). The work of Miao et al. (2015) already showed that the NVDM, or rather, neural variational inference in general, was a better approach to document modeling than docNade and the Replicated Softmax model (reinforcing the initial results of Mnih & Gregor (2014)), so we felt that investigation along this line would have been redundant and chose to focus on the effect that using different latent variables had.  We chose to analyze the decoder matrix because in the document models (unlike the VHRED models, where the decoder is not directly connected to the latents), the direct input to the decoder is the generated latent variable state (when conditioned on a given document).
As such, examining the synaptic connections connecting latent variables to word/output units can lend some insight towards the "impact" the latents have on the outputs in general (this is different than what is learned in the weights connecting the input units to the first hidden layer of the inference network, which would effectively be the word-embeddings). More importantly, we went beyond analyzing only the decoder matrix (since this is still, admittedly, a bit too indirect when measuring impact) and provided an approximate posterior analysis, found in the appendix, to confirm if the piecewise variables themselves were picking up different aspects of the data much as the dialogue encoder-decoder models appeared to.
However, we do recognize that yet even further analysis of the piecewise variables learned by our proposed architectures is desirable (on perhaps even lesser-explored document data-sets in order to see what might be uncovered), and as such we will update the manuscript to reflect AnonReviewer3's comment, mentioning that a complementary, focused investigation is the subject of future work.
In addition, we will modify the statement that AnonReviewer3 quoted such that it is less severe in its claim.
Thanks,
Alex And Iulian
UPDATE: I have read the authors' rebuttal and also the other comments in this paper's thread. My thoughts have not changed.
The authors propose using a mixture prior rather than a uni-modal
prior for variational auto-encoders. They argue that the simple
uni-modal prior "hinders the overall expressivity of the learned model
as it cannot possibly capture more complex aspects of the data
distribution."
I find the motivation of the paper suspicious because while the prior
may be uni-modal, the posterior distribution is certainly not.
Furthermore, a uni-modal distribution on the latent variable space can
certainly still lead to the capturing of complex, multi-modal data
distributions. (As the most trivial case, take the latent variable
space to be a uniform distribution; take the likelihood to be a
point mass given by applying the true data distribution's inverse CDF
to the uniform. Such a model can capture any distribution.)
In addition, multi-modality is arguably an overfocused concept in the
literature, where the (latent variable) space is hardly anymore worth
capturing from a mixture of simple distributions when it is often a
complex nonlinear space. It is unclear from the experiments how much
the influence of the prior's multimodality influences the posterior to
capture more complex phenomena, and whether this is any better than
considering a more complex (but still reparameterizable) distribution
on the latent space.
I recommend that this paper be rejected, and encourage the authors to
more extensively study the effect of different priors.
I'd also like to make two additional comments:
While there is no length restriction at ICLR, the 14 page document can
be significantly condensed without loss of describing their innovation
or clarity. I recommend the authors do so.
Finally, I think it's important to note the controversy in this paper.
It was submitted with many significant incomplete details (e.g., no experiments,
many missing citations, a figure placed inside that was pencilled in
by hand, and several missing paragraphs). These details were not
completed until roughly a week(?) later. I recommend the chairs discuss
this in light of what should be allowed next year.
Thank you for your review
> I find the motivation of the paper suspicious...
Indeed, it may appear that many researchers now focus on multi-modality, but we believe it is an important area of research.
In fact, our quantitative results on the document modeling tasks strongly support this claim, where the multi-modal latent variable models significantly outperform the uni-modal Gaussian latent variable models across three different tasks.
In fact, multi-modality is one of the major motivations for incorporating discrete latent variables into VAEs, which is what at least 3 other ICLR submissions focus on:
“Discrete Variational Autoencoders” by Rolfe
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables” by Maddison et al.
“Categorical Reparameterization with Gumbel-Softmax” by Jang et al.
Of course, our approach differs from these, because our goal is to learn continuous latent variables with multi-modal probability density functions.
For a lot of NLP applications, there are plenty of intuitive arguments for why one would like multi-modality in the latent variable space. Suppose we have a generative model over sports articles. Some of the generated articles might be related to football and others to hockey.
But clearly there exists no sport that lies in between football and hockey (at least not yet!), so if the latent variable contains different regions for football and hockey articles, then there must be a region of low probability mass between them. By definition, such a probability density is multi-modal and cannot be represented by a uni-modal Gaussian variable.
In theory, it could be captured in the decoder module which takes as input a uni-modal latent variable sample, but in practice this is going to be very difficult to learn as our experiments have shown.
> the 14 page document can be significantly condensed without loss
We agree, and will shorten the paper for the camera-ready version (see our response to Reviewer #3 above).
> It was submitted with many significant incomplete details
We sincerely apologize for the incomplete submission. We did not plan for this, but found bugs in our experiments and had to re-launch several of them just before the deadline, which is why it took a whole week extra before we had the final results.
-Alex And Iulian
This paper proposes a piecewise constant parameterisation for neural variational models so that it could explore the multi-modality of the latent variables and develop more powerful neural models.
The experiments of neural variational document models and variational hierarchical recurrent encoder-decoder models show that the introduction of the piecewise constant distribution helps achieve better perplexity on modelling documents and seemly better performance on modelling dialogues.
The idea of having a piecewise constant prior for latent variables is interesting, but the paper is not well-written (even 14 pages long) and the design of the experiments fails to demonstrate the most of the claims.
The detailed comments are as follows:
--The author explains the limitations of the VAEs with standard Gaussian prior in the last paragraph of 3.1 and the last paragraph of 5.1.
Hence, a multimodal prior would help the VAEs overcome the issues of optimisation.
However, there is a lack of evidence showing the multimodality of the prior helps break the bottleneck.
--In the last paragraph of 6.1, the author claimed the decoder parameter matrix is directly affected by the latent variables.
But what the connects the decoder is a combination of a piecewise constant and Gaussian latent variables. No matter what is discovered in the experiments, it only shows z=<z_gaussian, z_piecewise> is multimodal.
However, z=<z_gaussian1, z_gaussian2> can be multimodal as well. None of the claims in this paragraph stands.
--In the quantitative evaluation of NVDM, there is an incremental model from z=z_gaussian to z=<z_gaussian, z_piecewise>. As the prior is learned together with the variational posterior, a more flexible prior would alleviate the regularisation imposed by the KL term. Certainly, more parameters are applied as well, so a fair comparison would at least be z=<z_gaussian, z_piecewise> and z=<z_gaussian1, z_gaussian2> which equals to a double sized z_gaussian.
--The results shown in Table 3 are implausible. I cannot believe the author used gradients to evaluate the model.
--Eq. 5 is confusing, adding a multiplication sign might help.
--3.1 can be deleted because people attending ICLR are familiar with VAEs.
Typos:
as well as the well as the generated prior->  as well as the generated prior
Thank you for your detailed review.
> paper is not well-written (even 14 pages long)
We agree that the paper is too long. We aim to shorten it by moving the mathematical derivations and the equations for the NVDM and VHRED models to the appendix. Hopefully, this will also make the ideas more clear.
> the experiments fails to demonstrate the most of the claims …
We disagree with this conclusion. We have carried out experiments on four different tasks (20-Newsgroup, RCV1, CADE, and Twitter dialogues), while comparing to the most competitive baseline model (a similar model with multivariate Gaussian latent variables). On all document modeling tasks, we demonstrate significant performance improvements w.r.t. perplexity. On the document modeling tasks we further demonstrate the utility of the piecewise constant variables through word query similarity and gradient analysis. On the dialogue task, we demonstrate their utility through gradient analysis and qualitatively using examples.
We believe our experimental evaluations are at least as well-designed and conclusive --- if not more conclusive --- compared to several other ICLR submissions. OUR EXPERIMENTS ARE ON COMPLEX, REAL-WORLD TEXT DATASETS, which contain plenty of MULTI-MODAL STRUCTURE. Most of these experiments took several days (sometimes weeks) to execute on machines with TitanX GPUs. Please compare the impact and computational complexity of these experiments to other ICLR submissions such as: “The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables” and “Categorical Reparameterization with Gumbel-Softmax”. These submissions focus their experiments on MNIST and OMNIGLOT, which their reviewers seem to be satisfied with.
> However, z=<z_gaussian1, z_gaussian2> can be multimodal as well...
All Gaussian distributions are uni-modal by definition, so concatenating two samples from multivariate Gaussian distributions is still uni-modal.
Therefore, z=<z_gaussian1, z_gaussian2> cannot be multi-modal.
> a fair comparison would at least be z=<z_gaussian, z_piecewise> and z=<z_gaussian1, z_gaussian2> which equals to a double sized z_gaussian.
We did take this into account during hyper-parameter search. For the G-NVDM model, we experimented with up to 200 latent Gaussians variables, but found that this performed worse due to overfitting.
> The results shown in Table 3 are implausible…
Researchers routinely analyse hidden unit activations in neural network models.
For example, see Miao et al (2015) and Karpathy et al. (2015). In our case, we were interested in seeing how much changing a word would affect the latent variable models, which is why we compute the gradient w.r.t. the word embedding.
Of course, this does not yield quantitative evidence showing how well the model performs on different tasks, but it does illustrate what words the latent variables are sensitive to which, in turn, shows what they have learned to encode.
References
Miao et al, Neural Variational Inference for Text Processing. 2015.
Karpathy et al, Visualizing and Understanding Recurrent Networks, 2015.
- Alex And Iulian
I apologize for that I missed the footnote about the operation of z, and I thought it is an MLP.
However, it doesn't affect my question. What I wanted to emphasize here is that a simple nonlinearity after gaussian latent variables can be multimodal. Do we really need to have a hybrid gaussian piecewise constant distribution as the multimodal prior? If the mutimodality of z is critical, we can still have it by using an unimodal gaussian prior. In this case, "it appears that the piecewise variables affect what is uncovered by the model with respect to the data, as each model returns different, but relevant results with respect to the query word ..." doesn't make sense.
The improvement on perplexity is significant, which is also very exciting. And I do believe the piecewise constant might be promising for VAEs.
However, what I was questioning here is the careless claims made by the author without evidence. I would recommend the author to spend more time on rewriting the paper.
Yes, in theory, the lower levels of the decoder could potentially learn to represent multi-modal latent variables.
However, this is very difficult to learn through SGD optimization.
On the other hand, the piecewise constant latent variables can represent multi-modal random variables directly (e.g. for each dimension, the model has n=3 or n=5 specific parameters controlling one multi-modal distribution). This makes it a lot easier to learn multi-modal aspects of the data, which is why the piecewise constant latent variables achieve substantially better results on our 3 document modeling tasks, and the reason why they qualitatively encode different aspects of the data distribution.
-Alex And Iulian
We agree that some of the language in the experiments can be revised to address your concern (and make some of the claims less vague) and will do so for the final version. (See response to second comment as well as these two are related.)
-Alex And Iulian
The authors introduce some new prior and approximate posterior families for variational autoencoders, which are compatible with the reparameterization trick, as well as being capable of expressing multiple modes. They also introduce a gating mechanism between prior and posterior. They show improvements on bag of words document modeling, and dialogue response generation. The original abstract is overly strong in its assertion that a unimodal latent prior p(z) cannot fit a multimodal marginal int_z p(x|z)p(x)dz with a DNN response model p(x|z) ("it cannot possibly capture more complex aspects of the data distribution", "critical restriction", etc).
While the assertion that a unimodal latent prior is necessary to model multimodal observations is false, there are sensible motivations for the piecewise constant prior and posterior.
For example, if we think of a VAE as a sort of regularized autoencoder where codes are constrained to "fill up" parts of the prior latent space, then there is a sphere-packing argument to be made that filling a Gaussian prior with Gaussian posteriors is a bad use of code space.
Although the authors don't explore this much, a hypercube-based tiling of latent code space is a sensible idea.
As stated, I found the message of the paper to be quite sloppy with respect to the concept of "multi-modality. " There are 3 types of multimodality at play here: multimodality in the observed marginal distribution p(x), which can be captured by any deep latent Gaussian model, multimodality in the prior p(z), which makes sense in some situations (e. g. a model of MNIST digits could have 10 prior modes corresponding to latent codes for each digit class), and multimodality in the posterior z for a given observation x_i, q(z_i|x_i).
The final type of multimodality is harder to argue for, except in so far as it allows the expression of flexibly shaped distributions without highly separated modes. I believe flexible posterior approximations are important to enable fine-grained and efficient tiling of latent space, but I don't think these need to have multiple strong modes. I would be interested to see experiments demonstrating otherwise for real world data.
I think this paper should be more clear about the different types of multi-modality and which parts of their analysis demonstrate which ones. I also found it unsatisfactory that the piecewise variable analysis did not show different components of the multi-modal prior corresponding to different words, but rather just a separation between the Gaussian and the piecewise variables.
As I mention in my earlier questions, I found it surprising that the learned variance and mean for the Gaussian prior helps so dramatically with G-NVDM likelihood when the powerful networks transforming to and from latent space should make it scale-invariant. Explicitly separating out the contributions of a reimplemented base model, prior-posterior interpolation and the learned prior parameters would strengthen these experiments.
Overall, the very strong improvements on the text modeling task over NVDM seem hard to understand, and I would like to see an ablation analysis of all the differences between that model and the proposed one.
The fact that adding more constant components helps for document modeling is interesting, and it would be nice to see more qualitative analysis of what the prior modes represent. I also would be surprised if posterior modes were highly separated, and if they were it would be interesting to explore if they corresponded to e.g. ambiguous word-senses.
The experiments on dialog modeling are mostly negative results, quantitatively. The observation that the the piecewise constant variables encode time-related words and the Gaussian variables encode sentiment is interesting, especially since it occurs in both sets of experiments. This is actually quite interesting, and I would be interested in seeing analysis of why this is the case. As above, I would like to see an analysis of the sorts of words that are encoded in the different prior modes and whether they correspond to e.g. groups of similar holidays or days.
In conclusion, I think the piecewise constant variational family is a good idea, although it is not well-motivated by the paper. The experimental results are very good for document modeling, but without ablation analysis against the baseline it is hard to see why they should be with such a small modification in G-NVDM. The fact that H-NVDM performs better is interesting, though. This paper should better motivate the need for different types of multi-modality, and demonstrate that those sorts of things are actually being captured by the model.
As it is, the paper introduces an interesting variational family and shows that it performs better for some tasks, but the motivation and analysis is not clearly focused. To demonstrate that this is a broadly applicable family, it would also be good to do experiments on a more standard datasets like MNIST. Even without an absolute log-likelihood improvement, if the method yielded interpretable multiple modes this would be a valuable contribution.
> "The original abstract is overly strong in its assertion that a unimodal latent prior p(z) cannot fit a multimodal marginal int_z p(x|z)p(x)dz"
We did not argue for multi-modality of the output space. Naturally, with a powerful decoder, the marginal p(x) and the conditional p(x|z) could potentially be multi-modal.
However, using a Gaussian prior, by definition the latent variable prior p(z) is uni-modal. This could in principle hurt the representations learned. If the latent factors are truly multi-modal, trying to represent them or compress them down to a uni-modal space will act as a strong regularization and make the training process more difficult. That's why our goal is to learn a multi-modal prior p(z).
We will clarify this distinction in the final version of the paper.
> "I found it surprising that the learned variance and mean for the Gaussian prior helps so dramatically with G-NVDM likelihood"
There are other differences between NVDM and G-NVDM beyond learning the prior parameters. One important difference is the gating mechanism which allows the model to interpolate between the posterior and prior in order to calculate the final generated posterior parameters (beyond minor details such as different activation functions, technical modifications, etc. as mentioned in the paper). This is simply something that we found helped improve performance in preliminary experiments (as well as in some experiments in the past).  We felt that a fairer comparison would be to first improve the baseline model (i.e., the NVDM) and then compare our proposed hybrid models against the improved baseline (as opposed to only/exclusively reporting a previously published as is commonly done). This was especially important given that we intended to jointly learn the priors and use the gating mechanism in the proposed models as well.
The mission of the paper was to show that the proposed prior improved the encoder-decoder models in the challenging text problems we chose to explore (or at least uncovered interesting information using the piecewise variables).  Many of the models trained in the paper (especially the dialogue models) are fairly expensive to train, and we further felt that reporting various degradations of the models would clutter the paper and detract from focusing on the piecewise variables themselves.
However, we do agree that an ablation test would be appropriate and will add an appendix in the final version exploring the effects of each modification.
> "... a hypercube-based tiling of latent code space is a sensible idea."
We appreciate this interpretation, and will give it more thought.
> "I also found it unsatisfactory that the piecewise variable analysis did not show different components of the multi-modal prior corresponding to different words"
This is exactly what our analysis using gradients was aimed at.
We will try to visualize the impact of the latent variable components in another way.
> "The experiments on dialog modeling are mostly negative results"
That is not accurate. The standard G-VHRED model is neither better than nor worse than the H-VHRED. Human subjects simply cannot tell them apart.
-Alex And Iulian

Meta-review:
role: PC
data: comment: This paper explores a variational autoencoder variant.
 
 ICLR gives authors some respect that other conferences don't. It is flexible about the length of the paper, and allows revisions to be submitted. The understanding should be that authors should in turn treat reviewers with respect. The paper should still be finished. Reviewers can't be expected to read a churn of large revisions. The final paper should be roughly the right length, unless with very good reason.
 
 This paper was clearly not finished, and now is too long, with issues remaining. I hope that it will be submitted again, but not until it is actually ready.

Decision: Reject