Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper addresses the problem of devising a quantitative benchmark to evaluate the capability of algorithms to disentangle factors of variation in the data. 

*Quality* 
The problem addressed is surely relevant in general terms. However, the contributed framework did not account for previously proposed metrics (such as equivariance, invariance and equivalence). Within the experimental results, only two methods are considered: although Info-GAN is a reliable competitor, PCA seems a little too basic to compete against. The choice of using noise-free data only is a limiting constraint (in [Chen et al. 2016], Info-GAN is applied to real-world data). 
Finally, in order to corroborate the quantitative results, authors should have reported some visual experiments in order to assess whether a change in c_j really correspond to a change in the corresponding factor of variation z_i according to the learnt monomial matrix.

*Clarity*
The explanation of the theoretical framework is not clear. In fact, Figure 1 is straight in identifying disentanglement and completeness as a deviation from an ideal bijective mapping. But, then, the authors missed to clarify how the definitions of D_i and C_j translate this requirement into math. 
Also, the criterion of informativeness of Section 2 is split into two sub-criteria in Section 3.3, namely test set NRMSE and Zero-Shot NRMSE: such shift needs to be smoothed and better explained, possibly introducing it in Section 2.

*Originality*
The paper does not allow to judge whether the three proposed criteria are original or not with respect to the previously proposed ones of [Goodfellow et al. 2009, Lenc & Vedaldi 2015, Cohen & Welling 2014, Jayaraman & Grauman 2015]. 

*Significance*
The significance of the proposed evaluation framework is not fully clear. The initial assumption of considering factors of variations related to graphics-generated data undermines the relevance of the work. Actually, authors only consider synthetic (noise-free) data belonging to one class only, thus not including the factors of variations related to noise and/or different classes.

PROS: 
The problem faced by the authors is interesting

CONS:
The criteria of disentanglement, informativeness & completeness are not fully clear as they are presented.
The proposed criteria are not compared with previously proposed ones - equivariance, invariance and equivalence [Goodfellow et al. 2009, Lenc & Vedaldi 2015, Cohen & Welling 2014, Jayaraman & Grauman 2015]. Thus, it is not possible to elicit from the paper to which extent they are novel or how they are related..
The dataset considered is noise-free and considers one class only. Thus, several factors of variation are excluded a priori and this undermines the significance of the analysis.
The experimental evaluation only considers two methods, comparing Info-GAN, a state-of-the-art method, with a very basic PCA.


**FINAL EVALUATION**
The reviewer rates this paper with a weak reject due to the following points.
1) The novel criteria are not compared with existing ones [Goodfellow et al. 2009, Lenc & Vedaldi 2015, Cohen & Welling 2014, Jayaraman & Grauman 2015].
2) There are two flaws in the experimental validation:
	2.1) The number of methods in comparison (InfoGAN and PCA) is limited.
	2.2) A synthetic dataset is only considered.

The reviewer is favorable in rising the rating towards acceptance if points 1 and 2 will be fixed. 

**EVALUATION AFTER AUTHORS' REBUTTAL**
The reviewer has read the responses provided by the authors during the rebuttal period. In particular, with respect to the highlighted points 1 and 2, point 1 has been thoroughly answered and the novelty with respect previous work is now clearly stated in the paper. Despite the same level of clarification has not been reached for what concerns point 2, the proposed framework (although still limited in relevance due to the lack of more realistic settings) can be useful for the community as a benchmark to verify the level of disentanglement than newly proposed deep architectures can achieve. Finally, by also taking into account the positive evaluation provided by the fellow reviewers, the rating of the paper has been risen towards acceptance.   

review_title: The idea is interesting and, despite the limited experimental setup, the proposed framework can ease future investigations 

final_score: rating: 7: Good paper, accept
review_content: ****
I acknowledge the author's comments and improve my score to 7.
****

Summary:
The authors propose an experimental framework and metrics for the quantitative evaluation of disentangling representations.
The basic idea is to use datasets with known factors of variation, z, and measure how well in an information theoretical sense these are recovered by a representation trained on a dataset yielding a latent code c.
The authors propose measures disentanglement, informativeness and completeness to evaluate the latent code c, mostly through learned nonlinear mappings between z and c measuring the statistical relatedness of these variables.
The paper ultimately is light on comprehensive evaluation of popular models on a variety of datasets and as such does not quite yield the insights it could.

Significance:
The proposed methodology is relevant, because disentangling representations are an active field of research and currently are not evaluated in a standardized way.

Clarity:
The paper is lucidly written and very understandable.

Quality:
The authors use formal concepts from information theory to underpin their basic idea of recovering latent factors and have spent a commendable amount of effort on clarifying different aspects on why these three measures are relevant.
A few comments:
1. How do the authors propose to deal with multimodal true latent factors? What if multiple sets of z can generate the same observations and how does the evaluation of disentanglement fairly work if the underlying model cannot be uniquely recovered from the data?
2. Scoring disentanglement against known sources of variation is sensible and studied well here, but how would the authors evaluate or propose to evaluate in datasets with unknown sources of variation?
3. the actual sources of variation are interpretable and explicit measurable quantities here. However, oftentimes a source of variation can be a variable that is hard or impossible to express in a simple vector z (for instance the sentiment of a scene) even when these factors are known. How do the authors propose to move past narrow definitions of factors of variation and handle more complex variables? Arguably, disentangling is a step towards concept learning and concepts might be harder to formalize than the approach taken here where in the experiment the variables are well-behaved and relatively easy to quantify since they relate to image formation physics.
4. For a paper introducing a formal experimental framework and metrics or evaluation I find that the paper is light on experiments and evaluation. I would hope that at the very least a broad range of generative models and some recognition models are used to evaluate here, especially a variational autoencoder, beta-VAE and so on. Furthermore the authors could consider applying their framework to other datasets and offering a benchmark experiment and code for the community to establish this as a means of evaluation to maximize the impact of a paper aimed at reproducibility and good science.

Novelty:
Previous papers like "beta-VAE" (Higgins et al. 2017) and "Bayesian Representation Learning With Oracle Constraints" by Karaletsos et al (ICLR 16) have followed similar experimental protocols inspired by the same underlying idea of recovering known latent factors, but have fallen short of proposing a formal framework like this paper does. It would be good to add a section gathering such attempts at evaluation previously made and trying to unify them under the proposed framework.

review_title: This paper has good formal ideas about evaluating disentangling representations, a solid step forward on understanding learned representations. It would benefit from more thoughts on basic theory and more empirical work to be seen as a relevant benchmark going forward.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors consider the metrics for evaluating disentangled representations. They define three criteria: Disentanglement, Informativeness, and Completeness. They  learning a linear mapping from the latent code to an idealized set of disentangled generative factors, and then define information-theoretic measures based on pseudo-distributions calculated from the relative magnitudes of weights. Experimental evaluation considers a dataset of 200k images of a teapot with varying pose and color.

I think that defining metrics for evaluating the degree of disentanglement in representations is  great problem to look at. Overall, the metrics approached by the authors are reasonable, though the way pseudo-distribution are define in terms of normalized weight magnitudes is seems a little ad hoc to me.  

A second limitation of the work is the reliance on a "true" set of disentangled factors. We generally want to learn learning disentangled representations in an unsupervised or semi-supervised manner, which means that we will in general not have access supervision data for the disentangled factors. Could the authors perhaps comment on how well these metrics would work in the semi-supervised case?

Overall, I would say this is somewhat borderline, but I could be convinced to argue for acceptance based on the other reviews and the author response. 

Minor Commments:

- Tables 1 and 2 would be easier to unpack if the authors were to list the names of the variables (i.e. azimuth instead of z_0) or at least list what each variable is in the caption. 

- It is not entirely clear to me how the proposed metrics, whose definitions all reference magnitudes of weights, generalize to the case of random forests. 
review_title: A reasonable if somewhat ad-hoc approach 

Meta-review: The paper proposes evaluation metrics for quantifying the quality of disentangled representations. There is consensus among reviewers that the paper makes a useful contribution towards this end. Authors have addressed most of reviewers' concerns in their response.
Decision: Accept (Poster)