Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents evidence that even a tiny bit of supervision over the factors of variation in a dataset presented in the form of semi-supervised training labels or for unsupervised model selection, can result in models that learn disentangled representations. The authors perform a thorough sweep over multiple datasets, different models classes and ways to provide labeled information. Overall, this work is a well executed and rigorous empirical study on the state of disentangled representation learning. I think experimental protocol and models trained models will prove extremely useful for future work and advocate for accepting this paper.

Comments

1) Would it be possible to use the few labeled factors of variation in a meta-learning setup rather than as a regularizer?

2) The paper provides high level conclusions about the impact of having supervised model selection or semi-supervised learning in models in general, but doesn’t offer much discussion into their behavior under specific settings (i.e.) it seems to be hard to pick a winner amongst presented model. Some are better with 100 labeled examples but don’t scale as well as others when an order of magnitude more labeled data is available. It is certainly hard to discuss all the thousands of experimental observations, but the paper can benefit from some more fine-grained analysis.

Minor
Figure 4 is hard to comprehend without a model to index mapping similar to Figure 3
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper considers the challenge of learning disentangled representations---i.e. learning representations of data points x, r(x), that capture the factors of variation in an underlying latent variable z that controls the generating process of x---and studies two approaches for integrated a small number of data points manually labeled with z (or noisier variants thereof): one using these to perform model selection, and another incorporating them into unsupervised representation learning via an additional supervised loss term.  This investigation is motivated by recent results concluding that inductive biases are needed otherwise learning disentangled representations in an unsupervised fashion is impossible.  The paper poses its overall goal as making this injection of inductive biases explicit via a small number (~100 even) of (potentially noisy) labels, and reports on exhaustive experiments on four datasets.

I think that this paper merits acceptance because (a) the motivation of taking a necessity in practice (somehow selecting models / injecting inductive biases) and making it more explicit in the approach is a good one, and because the thorough empirical survey (and simple, but novel, contribution of a new semi-supervised representation learning objective) are likely valuable contributions to this community.

One negative comment overall would be that the results are not that surprising: that is, the fact that using labels either (a) to do model validation or (b) in a semi-supervised fashion would help is not too surprising.  However, I believe in the context of (a) making more explicit a practical (and theoretically) necessary step in the pipeline of learning representations, and (b) contributing a comprehensive empirical study, this is a worthwhile contribution.

Minor notes:
- Fig. 1 isn't the most intuitive- ideally would be better explained for the headlining figure
- 8.57 P100 GPU years is ~= $75k based on a cursory glance at cloud instance pricing at monthly rates... this is a lot to reproduce these experiments...
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: After rebuttal edit:
No clarifications were made, so I keep my score as is.

------------------------------------------------------
Claims: Explicitly requiring a small number of labels allows for successful learning of disentangled features by either using them as a validation set for hyper parameter tuning, or using them as a supervised loss. 

Decision: Reject. This paper needs a substantial rewrite to make clear what specific contributions are from the multitude of experiments run in this study. As is, the two contributions stated in the introduction are both obvious and not particularly significant -- that having some labels of the type of disentanglement desired helps when used as a validation set and as a small number of labels for learning a disentangled representation space. There are no obviously stated conclusions about which types of labels are better than others (4.2). Section 3.2 seems to have some interesting findings that small scale supervision can help significantly and fine-grained labeling is not necessarily needed, but I don't understand why that finding is presented there when Fig. 4 seems to perform a similar experiment on types of labels with no conclusion based on its results. Conclusion sentence of 4.3 is hard to decipher, but I assume is just saying S^2/S beats U/S even when S^2/S is subject to noisy labels. Overall, I find it very difficult to absorb the huge amount of results and find the analysis not well presented.


review_title: Official Blind Review #1

Meta-review: This paper addresses the problem of learning disentangled representations and shows that the introduction of a few labels corresponding to the desired factors of variation can be used to increase the separation of the learned representation. 

There were mixed scores for this work. Two reviewers recommended weak acceptance while one reviewer recommended rejection. All reviewers and authors agreed that the main conclusion that the labeled factors of variation can be used to improved disentanglement is perhaps expected. However, reviewers 2 and 3 argue that this work presents extensive experimental evidence to support this claim which will be of value to the community. The main concerns of R1 center around a lack of clear analysis and synthesis of the large number of experiments. Though there is a page limit we encourage the authors to revise their manuscript with a specific focus on clarity and take-away messages from their results. 

After careful consideration of all reviewer comments and author rebuttals the AC recommends acceptance of this work. The potential contribution of the extensive experimental evidence warrants presentation at ICLR. However, again, we encourage the authors to consider ways to mitigate the concerns of R1 in their final manuscript. 

Decision: Accept (Poster)