Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper shows that having local isometry together with non-Gaussianity of latent variables and sufficient number of samples guaranteed to learn a disentangled representation, without any domain-specific knowledge. The authors discussed some potential manifolds for which local isometry is satisfied. The authors also conducted several experiments to evaluate the disentanglement of the obtained representation by their method in contrast to the comparable approaches. The authors demonstrated the latent embedding and the mean correlation coefficient as qualitative and quantitative disentanglement measures, respectively.  


main_review: Originality: 
The paper is addressing a challenging and important problem. Although the local isometry property has been studied before, the authors provide a new theoretical analysis for the disentangled representation which is quite valuable and expands the finding of the previous disentanglement studies by Locatello et al., 2018, 2020. They show that in the presence of enough training samples, satisfying two conditions: isometry and non-Gaussianity on the latent space is enough to obtain disentangled factors. They also discuss the sensitivity of the isometry to the parametrization of each factor.

Quality: 
The paper is technically sound with enough theory and experimental studies. Theorem 1 shows if there exist an isometric mapping from latent space to the data space, using Hessian eigenmaps and ICA, the latent representation is disentangled. Theorem 2 generalizes the result of the first theorem by considering another mapping from latent space to an alternative space, which relaxes the local isometry constraint on the latent factors.

Clarity: 
The paper is well written and well organized.  

Significance: 
Both theoretical and experimental results seem valuable. Experimental results show that both HILLE+ICA and HAE obtain significantly better MCC compared to the prior methods. The results show that the HILLE+ICA and HAE have roughly the same reconstruction, generation, and interpolation performance compared to the vanilla AE.

limitations_and_societal_impact: - As discussed by the authors, there are some practical limitations for using the HILLE+ICA in representation learning. ICA/fastICA suffers from convergence issue and it is not easy to substitute ICA with another method to satisfy non-Gaussianity.  

- According to Theorem 1, latent dimensions are assumed to be uniform, which is a strong assumption in many domains of application. For instance, in the gene expression dataset which is also mentioned in section 3, such uniformity is not present. Can the authors explain how the theoretical results are satisfied for dataset with heterogeneity and rare observations?

- It seems that both HILLE and HAE suffer from poorer reconstruction compared to AE. Is this due to the objective in Eq.1 ?
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper studies the disentanglement of samples generated by GANs. It argues that the local isometry and non-Gaussian property are the keys to disentanglement. Some intuitive experiments based on GAN latent space discovery are conducted to support this insight.
main_review: Pros: 
1. The writing is clear and easy to follow.
2. The review of the traditional manifold learning method is helpful for current GAN research.

Concerns:
1. The two theorems in this paper are a bit tricky. First, a local isometry is a very strong condition, which basically cannot happen in practice. Second, these two theorems do not explain why intuitively disentanglement is connected with isometry, and what is the role of non-Gaussian property? Third, is there any in-depth analysis on the disentangled representations? For example, what is the convergence rate to the disentangled representations with respect to the data volume? Are there any more efficient algorithms? Is isometry or non-Gaussian necessary in disentanglement?
2. I must correct a logical mistake in this paper. That "you can get disentangled representations at local isometry and non-Gaussian condition" does not mean "you cannot get disentangled representations otherwise." So the content of this paper has presented cannot answer the question 'is disentanglement possible.'
3. The experiments are too weak. Why is GANSpace sufficient to support your claim? Any insights or theory?
limitations_and_societal_impact: The paper mentioned the limitations in Sec.6.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a novel method to learn disentangled representations from data. Their approach uses some inductive biases on the transformation between the latent and observed spaces, as well as on the latent distribution. Namely, they assume the transformation satisfies the assumption of the local isometry, and that the latent factors are (non-jointly) non-gaussian.
The disentangled representations are learned by first applying a spectral transform to the data, followed by a linear ICA unmixing. 
Their method is backed by a theoretical analysis, as well as an empirical validation.
main_review: The authors propose a novel method for learning disentangled representations, which is an important topic in unsupervised learning.
The proposed approach is based on a spectral decomposition followed by a linear ICA step, instead of learning a full generative model.
This gives it a considerable edge in run time (albeit this was only explored in 2-d), but takes away the generative capabilities. This is somehow addressed by the Hessian Auto-Encoder proposed in section 4.1, although it's not clear how it compares to state of the art generative models like VAEs and GANs (which have seen variants developed for disentangled representations).

Overall, the paper is well presented and easy to read. The proposed method is novel, and, as far as I know, hasn't been proposed before. 

This being said, I have 2 serious concerns:
1- The dimension of the latent space in the experiments was $d = 2$. The authors didn't report the performance of their approach for a higher latent dimension, which I find alarming. From personal experience, disentangled/identifiable representations are harder to find in higher dimensions.
In addition, their approach has much shorter run time, but the experiments were only conducted for $d = 2$. Does this still hold for a much larger latent dimension, say $d = 500$?

2- Figure 6 would be much more informative if the authors used a box plot of violin plot. More generally, error bars are missing from all figures and numbers, and there is no indication that the MCC figures reported were averaged over multiple runs and not for a specific seed.

As well as some other comments:
3- It would benefit the presentation if the spectral methods were better explained in the paper, and not simply relegated to the appendix, as they are core to the proposed method. The same can be done for fastICA, without going into the detail of the algorithm, but just explaining what ICA does for e.g.

4- Did the authors explore the impact of applying pointwise nonlinearities to the latent space of the datasets and see how it impacts performance? This would give experimental validation to Theorem 2.

5- The authors should specify that the "dimension of the manifold" is equal to that of the latent space.

6- Terrible notations: it seems that the indices are used to both designate the index of an observation in a dataset (e.g. line 25 $\{x_i\}$) and the component of a multivariate r.v. (e.g. line 39 $y_1(z_1, z_2)$). This is very confusing when referring to the distances between points (e.g. line 66 and 205, $d(x_i, x_j)$).

Overall,  I think that the theoretical results of this paper are worthwhile, but the experimental section lacks depth and polish. I will give a score of 5, but will be happy to increase it if the other reviewers or the authors' rebuttal convinces me to do so. 

limitations_and_societal_impact: The limitations of the proposed method, as well as how it relates to prior work, were well discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper follows the recent question of when disentanglement is possible and shows that 'local isometry' is sufficient to ensure that it is possible (not guaranteed) to learn a disentangled representation. This result largely follows from know results in the ICA community alongside the isometry assumption. A proof-of-concepts algorithm that combines HLLE with ICA demonstrates feasibility on synthetic data constructed to satisfy the isometry assumption. While the paper is largely correct, I would argue that it is uninteresting as it (as far as I can tell) only applies to flat manifolds, which is the case we already understand well through linear models.
main_review: The paper considers disentanglement of models where data $x$ is approximated as $x = g(z)$. Here $z$ is a low-dimensional (latent) variable and $g$ is a smooth function. The guiding assumption is that $J^T J = I$, where $J$ is the Jacobian of $g$ and $I$ is the identity matrix. This is assumed true for all $z$. Based on this assumption, the paper shows that classic identifiability results from (Euclidean) ICA holds, such that we can recover the true generative factors (i.e. we can disentangle).

This may sound exciting, but I remain unconvinced. I may have misread the paper, so I am happy to revise my view if the authors can provide more convincing arguments than those put forward in the paper.

Here's my issue:
The matrix $M = J^T J$ ($M$ is a short-hand I here introduce) is the metric tensor of the pull-back metric associated with $g$. If $M = I$ everywhere, then the manifold spanned by $g$ must have zero curvature everywhere, i.e. it must be flat. This leads to two comments from my side:

1. How reasonable is it that actual data manifolds are flat? I would argue that this assumption is so strong that it effectively is uninteresting. Consider data lying on a sphere; even this simple setting is too restrictive for the assumption to work out.

2. In this light, the result that we can disentangle seems rather obvious given that we know that ICA can disentangle linear models. The extension from 'linear' to 'flat' then seem rather incremental. There is still value in the extension, but given that the paper signals that the extension is more significant, I get rather skeptical.

On possible counter-argument to the above is given by the following statement from the paper (lines 97-98):

"
Indeed, assuming that the dimension of z is much smaller than the dimension of x, then any manifold
can be approximated by an isometric mapping. [2].
"

This is, however, all too vague to give this reader much comfort. Clearly any manifold can be approximated by a flat manifold; but can it be well-approximated? Gauss's Theorema Egregium suggest that we perhaps should not be approximating curved objects with flat ones.

On a more positive note, I appreciate that the paper has the relevant proofs as part of the paper rather than hidden in an appendix.  This is great.

I miss a citation to the paper

  Disentangling Disentanglement in Variational Autoencoders
  Emile Mathieu, Tom Rainforth, N. Siddharth, Yee Whye Teh

This gives a partial counter-argument to the Locatello paper, namely that the applied variational inference provides an inductive bias that helps make the models less susceptible to identifiability issues.

# After rebuttal and discussion

I appreciate the rebuttal and follow-up discussions which clarified many things. Based on these I will increase my score to recommend acceptance. **However**, this increase in the score rests on the assumption that the discussions points will be incorporated into the final paper. I think a significant rewrite of several sections of the paper are needed (see our discussions for details) for the paper to be publishable. I trust the authors to implement such changes as it is in their own best interest to do so.

I have increased my score to 6, which I feel is as high as I can go given the state of the submitted manuscript. Had I reviewed a version with the changes implemented, I might have gone higher.

limitations_and_societal_impact: There is no discussion of societal impact, but this can be forgiven considering the theoretical/conceptual nature of the work.

I do, however, think that the limitations are not sufficiently discussed. As far as I can tell, the assumptions imply that only flat manifolds can be considered, which render the work largely uninteresting (see above).

The final paragraph of the paper has a nice discussion of the limitations of the proposed algorithm and its reliance on HLLE. That was great!
review_title: null

Meta-review: This paper examines the hotly debated topic of whether one can learn disentangled representations from unlabelled data.  The key contribution is to show that a combination of non-Gaussianity and local isometry is sufficient to make unsupervised disentanglement theoretically possible.  This is then backed up by a proof-of-concept algorithm and experiments.

After receiving initially quite mixed review scores, this paper has undergone substantial discussions by the reviewers, both publically with the authors and behind closed doors, resulting in two reviewers who initially recommend rejection increasing their scores to back marginal acceptance.  Perhaps the key underlying issue that has been debated is how strong the assumptions required by the theoretical results actually are in practice.  After much back and forth, the general consensus on this seems to be that the assumptions are actually not unreasonable, but need to be much more carefully explained and discussed in the paper.

More generally, the overall sentiment of the reviewers, which I concur with myself, is that the underlying ideas in the paper are interesting and potentially quite significant, but that noticeable work on the writing of the paper itself is needed to improve the clarity and ensure the work can be properly appreciated by its audience.  Though, at the time of writing, Reviewer pb6M still also has concerns about the relationship between the theory and experiments in the paper and which mapping the local isometry assumption applies to, I agree with the authors that this concern seems to be based on a misunderstanding rather than a genuine issue.

Putting this together, I believe that the decision of acceptance comes mostly down to whether the clarity issues are deemed too severe for publication / require an unacceptably large amount of changes for the camera-ready.  I think this is quite a tight call, but recommend giving the author's the benefit of the doubt that the required changes will be adequately incorporated.  The basis for this is that I think that the work does have the potential to stimulate notable interest in an area that is quickly moving, such that the risk of the clarity issues not being solved is outweighed by the potential advantages of ensuring the ideas are published.

If this decision is upheld, I sincerely hope that the authors will make the required efforts to improve the paper for the camera-ready as the changes required are non-trivial and it would be a shame to leave the writing of the paper letting down the underlying technical content.  I would also very strongly suggest the authors change the title of the paper, which is inappropriately general and does a disservice to previous work that has considered the same question.
Decision: Accept (Poster)