Venue: ICLR 2018 Workshop

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces the VCAE, a model that takes the form of a variational autoencoder but includes an additional intermediate latent variable.

The overall motivation of the paper is unclear to me; the paper simply states that the standard VAE may not be effective due to a "straightforward encoder-decoder structure". This is a vague statement, so it is not clear what the limitation of the VAE is and how the proposed VCAE addresses it.

It is also not clear why an additional latent variable should lead to better performance compared to a standard VAE with varying NN architectures.

The paper also contains many grammar errors.

The expectation of the first KL term in Eq. 1 is wrong, because it should be taken with respect to q(s|x) and q(s|x).

Finally, I found it surprising that the authors did not cite "Categorical Reparameterization with Gumbel-Softmax" by Eric Jang et al., who proposed the concrete distribution independently of Maddison et al. 
review_title: The motivation is unclear. Poor writing quality

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a surrogate variable to sidestep the learning of the latent variable in variational auto-encoders.
The intuition is simple and clear. However, I feel the experiments are not good enough.

First of all, the authors should give more explanation about the models Concrete-s and Concrete-z.
I'm a bit confused here. What is the difference between these two models? Is Concrete-s the model that has two layer of discrete latent variables?
What is Concrete-z? And what could we learn from it by comparing the models with VCAE?
I think these are all very important points to be explained in this paper.

Secondly, in the comparison of VAE and VCAE, are the number of parameters the same in both models? For fairness, should we compare the VAE with hierarchies to the proposed VCAE?
Also, a deterministic counterpart (adding an extra deterministic layer to the VAE) seems to be necessary in this case for comparison as well.

To sum up, the numbers look very promising, but it is not quite clear where the benefits come from (the surrogate variable or the increase of parameters).
If the authors are able to improve the experiment section including model ablation, and the numbers are still positive, I believe it could be a good paper in the future. 
review_title: needs more work on experiments

final_score: rating: 3: Clear rejection
review_content: The authors propose a variational composite autoencoder, where a surrogate latent variable, s, is placed in between the observed and latent variables, x and z, respectively. This is done to alleviate issues associated with learning latent variables when the model is presented with highly structured data. The model consists of two encoders, onde from x to s and the other from x to z. The objective is the standard variational lower bound with two KL terms, one for each encoder.

I do not think the results presented in Tables 1 and 2 are necessarily comparable because the authors are essentially comparing 1-layer encoders (VAE, Concrete-s, NVIL and REBAR) with a 2-layer model (VCAE). A more direct comparison will be to use 2 layers in all models. In the NVIL paper for instance, one can see that going from 1 to 2 layers all SBNs, results in a gain of about 10 nats, which will make the results of NVIL and VCAE about the same.

The paper can be organized a little bit later, for example in the evaluation section the authors first present the model used for the second experiment (Table 2 and Figure 3), but naturally proceed to present results for the first experiment. Also, Figure 1 can be more general, as the author are not only considering latent and surrogate variables in the simplex.

The paper needs some proofreading.
review_title: VARIATIONAL COMPOSITE AUTOENCODERS

Meta-review: Based on the reviews, this paper has not been accepted for presentation at the ICLR workshop. However, the conversation and updates can continue to appear here on OpenReview.
Decision: Reject