Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method for learning disentangled representations.  The approach is used on both supervised (where the factors to be disentangled are known) and unsupervised settings. The authors demonstrate the efficacy of their approach in both settings on several datasets with both quantitative and qualitative results.

This task is an important one. However, I found that the contribution of this paper is fairly small. The proposed approach seems reasonable but it is mostly a work of engineering and provides little insights into the problem nor the proposed model.

The setup where labeled data (c) also seems a bit unnatural (this also seems to be confirmed by the fact that the authors had to build datasets for the problem). Perhaps the authors could give examples of situations where this would naturally arise. In practice, it seems difficult to obtain these data for all required variables to be disentangled.

The unsupervised results are more interesting but not very much explored (a single set of sampled faces). I was also curious as to why the learned Y's are blurry. This sort of two-stage generation is also potentially interesting, I was wondering if the authors had ideas to generalize this idea. 

I also was not convinced by the experiments which are mostly qualitative. I did not find that this set of experiments provide enough support to the proposed method.


Detailed comments:
- It is a bit unclear to me how the authors propose to obtain independent posteriors over z and c. Is it purely empirical or is there a formal reason that guarantees it?

- Some of the figures your report are compelling but it is a bit unclear to the reader if the results are general (e.g., the examples could have been hand-picked). Are there any quantitative measures you could provide (in addition to Tables 1 and 2 which don't measure the quality of the approach)?

- Comparing to CGAN seems reasonable but given the task at hand, it seems like other methods could have been tried (although I do realize that no one may have done this before for deep generative models).



Other comments:
- In Figure 3, it would be good to label the upper trapezoid.

- Some paragraphs are very long and the manuscript may benefit from segmenting them into multiple paragraphs.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: Summary:
The paper proposes the use of a hierarchical model for a generative modeling task. They propose a framework of introducing an intermediate latent variable to enforce the independence of the control and noise variable. 
The paper report extensive experimental results to validate the proposed hierarchical model. 
The authors also provide the anonymized code to observe the exact implementation in TensorFlow to visualize the latent variable traversals.

Comments:
The paper proposes the use of a hierarchical model for a generative modeling task by introducing an intermediate latent variable to enforce the independence of the control and noise variable. 
The paper report extensive experimental results to validate the proposed hierarchical model. 
This type of framework of crude to fine hierarchical generative model has already been successfully introduced by StackGAN and it's recent variants.
On the unsupervised disentangled feature learning, the framework provides incremental advancement by using beta-VAE in conjunction with GAN to use the best of both the worlds. 
Even though the proposed approach is similar to StackGAN, the experiments and the results mentioned in the paper are noteworthy.

Questions to Authors:
There are 2 main claims of novelty made in the paper.
1. Architectural Biases: 
How is the approach different in comparison to the StackGAN and it's variable which also use multiple levels of crude to fine image generation?
2. Unsupervised control variable discovery:
This part is just the use of existing disentanglement VAEs to extract the control variables. So how does the paper try to make contributions to improve the disentangled features with the proposed method?
Apart from combining these to existing ideas, what can be considered as an added novelty to improve the quality of the disentangled features?

In summary, I find there is no novelty involved apart from combining the already existing SOTA model in disentangled feature learning (beta-VAE) and image generation (StackGAN).

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper proposes a hybrid technique for rendering “control-variate” and class-conditional image in two steps, first by generating an approximate rendering of the image (“Y”) conditional on the control variate and then filling in the details with a conditional GAN dependent on a latent noise variable Z (although I note that the caption of Figure 2 which identifies “Z” as the identity makes this rather confusing).

To ensure that Z is used to explain aspects of the model that are separate from the controlled variation, Z is combined in the refinement model at later steps (since otherwise the posterior over Z and Y conditional on X could induce entanglement between the variables).

In the “supervised” setting where the control variates are observed, Y can be learned as a simple regression problem independent of the other parts of the model, and this two-stage refinement process is demonstrated (using inception scores) to generate convincing samples, including when C consists of up to 10 control variates. In the unsupervised setting, a beta-VAE is used to learn a disentangled representation of X as a proxy for C, but then the data is regenerated using a two step process.

Readability suggestion: the paper starts with a very nice motivating example, but when the setup is provided, i.e., that (x,c) pairs are the input to the learner, the intended content of c is not immediately clear- control variates could assume anything from general context information to privileged information. A similarly informative example would be great!

Clarification regarding lemma 1: it seems that if the true posterior cannot be expressed by q, a gap will necessarily remain, even in the “limit” of perfect learning. Is this correct?

Overall: this paper makes a convincing case that it can be used to generate higher quality images, but not that this improves the quality of the disentangled representations. In fact, the separate training seems to make this unlikely.

review_title: Official Blind Review #3

Meta-review: The paper addresses the problem of learning disentangled representations in supervised and unsupervised settings. 

In general, the problem of representation learning in of course a core problem in ICLR. However, in the set-up described by the authors, R2 commented on the the set-up for supervised being a bit unnatural in as detailed labels need to be given (somewhat confusingly, the labels are called control variates in the paper).

Several reviewers commented on the novelty of the paper being on the low side, with R2 commenting the contribution being fairly small, and R3 noting similarities to stackgan.

There were also some comments on quality, and clarity. On the topic of technical quality, R2 did note that the authors present extensive results, but R3 mentions that the case for the disentanglement improving is not sufficiently supported. In terms of clarity, there was some initial confusing about e.g. the inference procedure, though the authors addressed these issues in the discussion. 
Decision: Reject