Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper investigates deep generative models with multiple stochastic nodes and gives them meaning by semi-supervision. From a methodological point of view, there is nothing fundamentally novel (it is very similar to the semi-supervised work of Kingma et al; although this work has sometimes more than two latent nodes, it is not a complex extension). There is a fairly classical auxiliary variable trick used to make sure the inference network for y is trained over all data points (by supposing y is in fact is a latent variable with an observation \tilde y; the observation is y if y is observed, or uninformative for unobserved y). Alternatively, one can separate the inference used to learn the generative model (which throws out inference over y if it is observed), from an inference used to 'exercise' the model (approximate the complex p(y|x) in the model by a simpler q(y|x) - effectively inferring the target p(y|x) for the data where only x is collected). Results are strong, although on simple datasets. Overall this is a well written, interesting paper, but lacking in terms of methodological advances.

Minor:
- I feel the title is a bit too general for the content of the paper. I personally don't agree with the strong contrast made between deep generative models and graphical models (deep generative models are graphical models, but they are more typically learned and un-interpretable than classical graphical models; and having multiple stochastic variables is not exclusive to graphical models, see DRAW, Deep Kalman Filter, Recurrent VAE, etc.). The word 'structure' is a bit problematic; here, the paper seems more concerned with disentangling and semanticizing the latent representation of a generative model by supervision. It is debatable whether the models themselves have structure.
review_title: Review

final_score: rating: null
review_content: Could the authors clarify why the described model can help introduce structural constraints on the latent variables in a VAE?  From the paper it seems the most significant thing the model does is to handle labeled and unlabeled data in a single formulation in Eq.4, but it is unclear to me how the claim on being able to specify structural constraints on the latents is supported in the paper.  The three experiments do not seem to have structural constraints on the latents, and the latents are not much more structural than previous work.
review_title: structural constraints on the latents

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a variant of the semi-supervised VAE model which leads to a unified objective for supervised and unsupervised VAE.  This variant gives software implementation of these VAE models more flexibility in specifying which variables are supervised and which are not.

This development introduces a few extra terms compared to the original semi-supervised VAE formulation proposed by Kingma et al., 2014.  From the experiment results it seems that these terms do not do much as the new formulation and the performance difference between the proposed method and Kingma et al. 2014 are not very significant (Figure 5).  Therefore the benefit of the new formulation is likely to be just software engineering flexibility and convenience.

This flexibility and convenience is nice to have, but it is better to demonstrate a few situations where the proposed method can be applied while for other previous methods it is non-trivial to do.

The paper's title and the way it is written make me expect a lot more than what is currently in the paper.  I was expecting to see, for example, structured hidden variable model for the posterior (page 4, top), or really "structured interpretation" of the generative model (title), but I didn't see any of these.  The main contribution of this paper (a variant of the semi-supervised VAE model) is quite far from these.

Aside from these, the plug-in estimation for discrete variables only works when the function h(x,y) is a continuous function of y.  If however, h(x, y) is not continuous in y, for example h takes one form when y=1 and another form when y=2, then the approach of using Expectation[y] to replace y will not work.  Therefore the "plug-in" estimation has its limitations.

review_title: A variant of the semi-supervised VAE.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces a variant of the semi-supervised variational auto-encoder (VAE) framework. The authors present a way of introducing structure (observed variables) inside the recognition network.

I find that the presentation of the inference with auxiliary variables could be avoided, as it actually makes the presentation unnecessarily complicated. Specifically, the expressions with auxiliary variables are helpful for devising a unified implementation, but modeling-wise one can get the same model without these auxiliary variables and recover a minimal extension of VAE where part of the generating space is actually observed. The observed variables mean that the posterior needs to also condition on those, so as to incorporate the information they convey. The way this is done in this paper is actually not very different from Kingma et al. 2014, and I am surprised that the experiments show a large deviation in these two methods' results. Given the similarity of the models, it'd be useful if the authors could give a possible explanation on the superiority of their method compared to Kingma et al. 2014. By the way, I was wondering if the experimental setup is the same as in Kingma et al. 2014 for the results of Fig. 5 (bottom) - the authors mention that they use CNNs for feature extraction but from the paper it's not clear if Kingma et al. do the same. 

On a related note, I was wondering the same for the comparison with Jampani et al. 2015. In particular, is that model also using the same rate of supervision for a fair comparison?

The experiment in section 4.3 is interesting and demonstrates a useful property of the approach.

The discussion of the supervision rate (and the pre-review answer) is helpful in giving some insight about what is a successful training protocol to use in semi-supervised learning.

Overall, the paper is interesting but the title and introduction made me expect something more from it. From the title I expected a method for interpreting general deep generative models, instead the described approach was about a semi-supervised variant of VAE - naturally including labelled examples disentangles the latent space, but this is a general property of any semi-supervised probabilistic model and not unique to the approach described here. Moreover, from the intro I expected to see a more general approximation scheme for the variational posterior (similar to Ranganath et al. 2015  which trully allows very flexible distributions), however this is not the case here.

Given the above, the contributions of this paper are in defining a slight variant of the semi-supervised VAE, and (perhaps more importantly) formulating it in a way that is amendable to easier automation in terms of software. But methodologically there is not much contribution to the current literature. The authors mention that they plan to extend the framework in the probabilistic programming setting. It seems indeed that this would be a very promising and useful extension. 

Minor note: three of Kingma's papers are all cited in the main text as Kingma et al. 2014, causing confusion. I suggest using Kingma et al. 2014a etc.

review_title: null

final_score: rating: null
review_content: I would appreciate if the authors could clarify:

a. Why is q(y|x) missing in equation (3)? It appears correctly in (4), but why not in (3)? I understand that we extend the z-layer with variables y in (2), and in (3) we assume additionally that q(z,y|x) = q(z|y,x) q(y|x). So why is q(y|x) missing? (Or maybe I'm just missing something?)

b. I don't quite get what the auxiliary variable offers in (4), in terms of explanation or formulation. Of course if we insert and marginalize it out, we go back to (2). If we don't marginalize it out but at the same time use the delta, we basically set it to y. So what is the role of this representation?

c. I was wondering if the discussion about "Supervision rate" is particular to the model presented. What are the practical implications from this observation?

d. In the discussion about plug-in estimation for discrete variables, can you please clarify out of these approaches, which is the one *you* decided to use and why?



review_title: Clarifications

Meta-review: The paper is a clearly presented application of deep generative models in the semi-supervised setting. After reviewing the discussion and responses, the reviewers felt that the paper while interesting, is limited in scope, and unfortunately not yet ready for inclusion in this year's proceeding.
Decision: Reject