Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Synopsis of the paper

This paper presents a new model for assessing to what extent a deep
generative model is disentangled. In contrast to existing methods, this
paper proposes an approach founded on topological considerations: by
assessing the topological dissimilarity of submanifolds of a given model,
which are conditioned on an individual factor. The overarching idea is
that a fully-disentangled model should exhibit high topological
dissimilarity for *different* factors, while exhibiting low topological
dissimilarity for the *same* factors (but with different values for said
factors).

These topological dissimilarity assessments are achieved by employing
persistent homology, a method from topological data analysis that
generalises ordinary simplicial homology (i.e. 'counting
high-dimensional holes') to a multi-scale setting (in which point clouds
are being assessed). Previous work for GAN comparison, viz. the
'geometry score' is extended by a new internal metric, the Wasserstein
distance.

A set of experiments demonstrates the utility of the proposed approach,
validating previous knowledge.

# Summary of the review

This paper is tackling a highly relevant topic (disentanglement
analysis) and provides a novel, fresh perspective that is mathematically
well justified. The paper is chock full of excellent ideas that make me
excited about their potential; I am sure that this will be shine a light
on the potential of topology-driven methods.

However, while the paper is well written for the most part, I cannot
fully endorse it for publication yet because of two reasons:

1. The paper is missing crucial details about its comparison method.
   While I know the existing work to piece everything together, I am
   not sure the method as such is reproducible at the moment.

   I am agreeing with most of the claims in the paper, but I would be
   hard-pressed to implement exactly the same method in the same
   fashion, because not all the details are there.

2. While most of the paper flows very well, in an almost conversational
   style that I very much appreciate, there are also several places at
   which concepts are not explained correctly or with sufficient
   precision. There are also some issues with the terminology that make
   it harder to fully understand what is going on.

If these two points were to be rectified in a revision of the paper,
it would definitely strengthen the message of this paper!

Please see below for detailed comments.

# Detailed comments

- The introduction is really 'on point' and makes an excellent case for
  requiring more foundational approaches

- The way 'topology' is introduced in the introduction is very
  imprecise; I find the discussion of 'high-density' and 'low-density'
  to be problematic here. Algebraic topology, which is where persistent
  homology originated from, has no notion of density; all considerations
  of 'holes' are based on purely algebraic arguments. While it is true
  that you can incorporate density information into persistent homology,
  this discussion is slightly misleading.

  I would point out rather that holes are typically considered to be the
  boundaries of these low-density regions (if the 'density argument' is
  left in there).

- Some of the terms are overloaded and used multiple times: 'latent
  dimension' is used equivalently to 'latent factor', it seems. I would
  strongly suggest to use a unified terminology.

- The term 'homeomorphic dimensions' is vague. I also have to point out
  that the method used in this paper is only *invariant* under
  homeomorphism! More precisely, two topological objects can have the
  same homology (or persistent homology, or barcode, etc.) even though
  they are **not** homeomorphic. This is a crucial detail that is not
  handled correctly in the paper at present; for example, Figure
  5 claims that 'homeomorphic groups' are shown, but the approach cannot
  detect this. If anything, the case should be made for depicting that
  these groups are 'likely' homeomorphic. Alternatively, one could write
  that the proposed approach assesses the *topological similarity* of
  models.

- Often, the term 'topology' is also used as an equivocation.
  Unfortunately, the term is overloaded in mathematics as well: a space
  can *have* a topology (i.e. systems of open sets etc.), but I have the
  impression that the paper often means to say 'shape' or 'topological
  features' (connected components, cycles, etc.) rather than 'topology'.

- The explanation of a manifold could be improved; why not mention the
  definition in terms of 'locally looks like some $\mathbb{R}^d$'? The
  discussion of 'open discs' might leave some readers baffled.

- I disagree with the sentence 'Under the manifold hypothesis, the latent
  space of a deep generative model has an extremely dense underlying
  manifold with few, if any, holes, [...]'. Holes characterise the shape
  of a manifold; the manifold hypothesis only posits the hypothesis that
  data originate from a manifold. It does not say anything about the
  *shape* of the manifold itself, and I do not agree with the notion
  that having no holes is somehow 'better'.

  A generative model that 'learns' that all samples can be arranged
  along a cycle in some latent space can be very appropriate depending
  on the data set.

  (Moreover, Figure 2 in some sense tells the opposite story of this
  sentence.)

- The definition/introduction of (persistent homology) at the end of
  Section 2 could be improved: the Vietoris--Rips complex goes back to
  a 1927 paper by L. Vietoris (https://link.springer.com/article/10.1007/BF01447877);
  an excellent introduction to modern topological concepts is provided
  in the book 'Computational Topology: An Introduction' by Edelsbrunner
  and Harer.

  I also disagree with homology 'approximating the topology'. Homology
  is *one* specific invariant in algebraic topology. In this sense, it
  does not approximate anything but describes a data set in terms of
  its topological features.

- The term 'symplectic vertex' is not correct here, as far as
  I understand. Everything is discussed in terms of simplicial
  complexes; I do not see the connection to symplectic topology here. 

- The explanation of the Vietoris--Rips complex could be improved.
  I feel that the explanation in terms of creating edges based on the
  proximity criterion will leave most readers baffled. I would suggest
  adding a brief explanation/illustration about this, maybe using Figure
  2d.

- What is meant by having a manifold 'assume topology $\tau$'? Does this
  refer to the manifold expressing a certain a set of barcodes?

- In Figure 3, I would rather write '[...] the submanifolds conditions on
  a specific rotation [...]'. Upon first reading the figure,
  I misunderstood its central point, namely that *if* the rotation is
  fixed, so that only the scale varies, the resulting submanifold will
  not have any holes, as the variation in scale provides a simple 1D
  structure.

- I would not say that submanifolds have different 'topologies' than
  their super-manifold; rather they might/will have different homology
  maybe or 'shape'?

- While I fully agree with the thrust of the method, I would need more
  details on p. 4. How is the partitioning of the manifold achieved in
  practice? How are the factors chosen? How are their respective values
  chosen? Is it possible to provide more examples here?

- When discussing conditional submanifold topology, consider mentioning
  how these manifolds are being obtained in practice, maybe by providing
  more examples.

- I do not understand the main thrust of the 'Topological asymmetry'
  section. Is the main argument that manifolds conditioned on the same
  factor should have the same shape and, moreover, that manifolds coming
  from the same factor should be more similar to one another than
  manifolds coming from different factors?

- What does it mean that factors are 'not homeomorphic' in Figure 4? Is
  the ground-truth topology known?

- The statement '...that best separates similar and dissimilar
  topologies...' should again be rephrased and made more precise, for
  example by adding '...topological features, as measured using
  persistent homology...'.

- In the 'Metric' section, I would caution strongly against this term.
  The RLT/Wasserstein calculations might result in a metric in the
  mathematical sense (for sure, the Wasserstein distance constitutes
  a metric on the space of barcodes, as the paper mentions), but I am
  not sure whether the *score* calculated in this section constitutes
  a metric. If anything, this would have to be proven.

- The 'Metric' section is also hard to read because of differences in
  terminology. Why is $\xi$ used to represent an index, whereas other
  Greek letters are used for functions? What is $\delta$? Is it the
  proposed measure of disentanglement?

- After the 'Metric' section, I am missing a concluding section, that
  summarises how the method works in practice. The following questions
  need to be answered:

  1. How are topological features computed (i.e., which point cloud is
     being used and which filtration)?

  2. How is the Vietoris--Rips complex computed in practice? How is its
     scale parameter $\epsilon$ chosen?

  3. What is the dimensionality used to compute topological features,
     i.e. are high-dimensional features being used at all?


- '...to those of the reals.': what does this mean?

- It seems that $\tau$ is not used consistently; is $\rho$ meant in
  Figure 6?

All in all, this paper makes me very excited about the potential of
topology-based methods, and I envision that, with some changes, this
will be a strong publication in the future!

# Style & clarity

The paper is well-written; here are some additional suggestions for
further improving its clarity: 

- 'generated dSprites' --> 'generated from the dSprites data set'?

- 'by composition' --> 'by the composition'

- The term 'group' is unfortunately somewhat overloaded as well; is
  'factor' more appropriate in some places? Or is the paper actually
  talking about manifold groups?

- I would strongly suggest to check whether 'persistent homology' might
  not be ore appropriate in many places instead of 'homology'. The
  latter should be reserved to the setting of *one* simplicial complex,
  but the main argument of the paper is that one has to adopt
  a multi-scale description of the data set. 

# Update after initial revision

I have re-read the revised version multiple times now, and I thank the authors for the amount of work they put into their revision. I am raising my score for the next discussion round. That being said, I want to point out why I cannot fully endorse this paper yet. First, from the point of topological data analysis, the central algorithm is a comparatively small extension of the Geometry Score paper. I agree that this is a superb idea, yet the main contribution for me lies in the application of that technique to disentanglement—and for this to be fully understandable, some more work is needed. For example, putting the central algorithmic details into the appendix will make the adoption of the method that much harder. 

Moreover, while I appreciate the overall story and description of the method, I do not think that readers will understand how this disentanglement is actually _achieved_ by means of the proposed TDA approach. I would therefore prefer to see a more 'technical' or 'algorithmic' description of the contributions in the main paper, in particular since I think that the ideas of conditional submanifold topology require more attention.

My expertise is more the topology and less so the application of disentanglement; nevertheless, this paper strikes me as highly ambitious with a lot of potential, yet somewhat unfinished in its present form. I do believe that it has the potential to be extremely impactful with some additional modifications (concerning conciseness, but also experimental details).

I fully realise that this is not yet the desired outcome for the authors; I shall endeavour to discuss this further with my fellow reviewers to see that we can reach a consensus!
review_title: An exciting demonstration of topology-based assessments of generative models

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

The paper proposes a novel metric for evaluating disentanglement by taking a manifold-topological perspective on the representations learnt. The key insight is that for a disentangled representation, when we fix a certain factor of variation at different values the topology of the conditional sub-manifolds should be similar. Using this insight the paper proposes a metric for disentangling which does not require annotations of the factors of variation and is more general than previous such tests.

Strengths
+ Having an approach that is general and easy to compute across datasets and models makes a lot of sense

Weaknesses

It would be nice to further clarify the intuitions for how disentangling relates to the manifold structures using more examples in the paper for people who are not familiar with the manifold topology literature. For example, in Fig. 3 it would be nice to show qualitatively what an entangled model would do for the rotation scale disentangling situation.

It seems somewhat surprising that \beta-VAE_{B} does so much worse than \beta-VAE on CelebA, were the hyperparameters tuned separately for the CelebA dataset?

review_title: Interesting looking paper

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new metric to quantify disentanglement by leveraging ideas from manifold topology, more precisely the homology of conditional submanifolds. It has a strong theoretical grounding (clearly better than existing metrics) and results appear to be promising.

However it is at times quite hard to follow, and I am not entirely confident that I fully understood how the Wasserstein RLTs were estimated in practice. I’d still recommend publication, given the novelty and importance of a theoretical sound metric for disentanglement.

Comments/questions:
1. The Introduction, Background and Manifold interpretation sections do a good job of introducing the subject, covering the literature and providing enough background about differential geometry, but I think a clear algorithmic demonstration of steps (d) and (e) of Figure 2 is lacking. 
      1. How exactly do you estimate the Wassertein RLT and compute the matrix M?
      2. Writing the exact equations used, e.g. in the Appendix, would be helpful. Overall I found the main text to rely on explanations a bit more than would be necessary, where some math would be clearer (having the code will be most helpful however)
      3. The actual disentanglement metrics $\mu$ and $\mu_{sup}$ might deserve a \begin{equation} for clarity.
      4. You never explicitly say if “high $\mu$ => high disentanglement”, which I think is how one should interpret Table 1 given the main text?
      
2. Providing a bit more explanations about how to interpret the RLT histograms would be valuable. I imagine that if the histograms vary as a function of their conditioning, this indicates entanglement?
      
3. Figure 7 was interesting, but the specific mention of a mismatch in how the classifier method (i.e. Disentanglement (Kim & Mnih) I assume?) ranks $\beta$-TCVA and FactorVAE differently was hard to follow (i.e. these do not appear that close in your $\mu_{sup}$ figure?)
      
4. Do the topological similarity matrices in Figure 5 provide interesting information about which factors are grouped together?
      1. Could you add labels on the axes and discuss what each cluster represents in a few selected models/datasets?
      
5. How does the method relate to what was done in GEOMANCER [1]? They also leverage differential geometry and holonomy (but as a learning signal for representation, not as a metric obviously), so discussing it might be valuable?


[1] https://arxiv.org/abs/2006.12982
review_title: Official Blind Review #2

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper presents a disentanglement metric to measure the intrinsic properties of a generative model with respect to the factor of variation in the dataset. Toward this, the paper first assumes disentangled factors reside in different manifolds. These different manifolds are the sub-manifolds of some manifold M for a given disentangled generative model. The paper considers the fact that in an entangled model the sub-manifolds are not homeomorphic and thus similarity across submanifolds can be measured to evaluate a model’s disentanglement. As such, disentanglement is related to the topological similarity.  For measuring topological similarity, the paper then introduces Wasserstein Relative Living Times. The proposed metric is used to evaluate standard disentanglement methods and datasets demonstrating the importance. 

Pros:
- The paper is well written and the proposed metric is well articulated.
- The manifold interpretation of disentanglement (section 3) is clear and could be considered a stand-alone contribution to the disentanglement community. 
- The experiments covered depth in terms of dataset selection and the number of models considered for the evaluation of the metric. 

Few questions/suggestions:
- Can the authors briefly describe why W. Distance defines a valid metric on barcode space and others don’t, or point out to relevant literature in section 3.1?
- In Li et. al., 2019, MIG-sup was proposed to remedy the weakness of the MIG metric. For the unsupervised portion, since MIG was considered a comparision, can authors also compare their method with MIG-sup? Or at least discuss it along with MIG? 

Li, Z., Murkute, J. V., Gyawali, P. K., & Wang, L. (2019, September). PROGRESSIVE LEARNING AND DISENTANGLEMENT OF HIERARCHICAL REPRESENTATIONS. In International Conference on Learning Representations.

(Update): The score has been updated after the rebuttal phase.
review_title: Disentanglement metric measuring intrinsic properties of a generative model with respect to its factors of variation. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: Introduces unsupervised disentangling metric that measures homeomorphic similarity between submanifolds conditioned on a given factor, and homeomorphic dissimilarity on submanifolds conditioned on different factors. The paper also includes a supervised variant which can directly assess topological similarity of submanifolds with label-spaces. The paper also introduces a novel variation of RLTs that  employs wasserstein distance instead of euclidean distance. 

Strengths:
* Novel application of topological similarity to unsupervised evaluation of disentangling.
* Unsupervised and supervised metrics are both in the same units, and can be directly compared.
* Experimental comparison, using the proposed metric, of many standard disentangling models on challenging datasets.

Weaknesses:
* The comparison of the proposed metric to mutual information gap and the metric from Kim and Mnih 2018 shown in Figure 7 is not convincing. Not only does the ranking change, but the distributions of scores look wildly different between models (e.g., \mu has low variance for VAE in both supervised and unsupervised cases, but both MIG and Kim & Minh show relatively high variance for VAE). It is possible that discrepancy is a good thing, and that the proposed model is capturing disentangling even better than the baseline metrics. However, there is not enough experimental validation to know whether this is the case. The paper would be stronger if it had more experimental validation, including a replication of the reported scores from the MIG (Chen et al., 2018) and (Kim and Minh 2018) paper, and deeper analysis explaining the relative differences in scores for models. Without this component, it is difficult to know whether the model comparison results (Table 1) are meaningful.
* It would be nice to include a discussion of how the proposed metric accounts for “dead” units (e.g., Eastwood & Williams 2018)? These might result in topologically similar submanifolds (since the conditional distributions would be nearly identical) and a worse score for the metric, even though the representation is arguably well-disentangled.

Clarity:
* Typo: Page 6, Line 5: “Figure 9” should be “Figure 4”

Other notes:
* It would be very interesting to see an experiment where the unsupervised metric is used to choose hyper-parameters of a model which is then evaluated in a supervised way, to demonstrate that the unsupervised metric is a valid substitute for a supervised metric.
review_title: Novel application of topological similarity to disentangling metrics, but needs more experimental validation

Meta-review: The paper proposes the use of topological similarity between conditional submanifolds for a given latent dimension as a metric for measuring disentanglement in generative models. To estimate the topological similarity between conditional submanifolds, the authors build upon an earlier work of Relative Living Times (RLT). 

R5 and R4 had concerns on the paper, particularly about the lack of enough novelty in the actual technique (R5) and about the lack of convincing experiments (R4). One of the concerns raised by R4 was around the discrepancies between MIG and FactorVAE. However as noted by other reviewers (R2 and R5), these discrepancies between different popular metrics are well acknowledged in the literature and authors have responded to this point. R2 and R5 also appreciate that avoiding the rotation issue faced by most of these disentanglement metrics is one of the strengths of the proposed metric.

While I tend to agree with R5 that the actual technique is inspired from the earlier work on "Geometry Score", I also think the application to measuring disentanglement in generative models is a novel contribution in itself, especially because current metrics have issues as pointed out by other reviewers -- the paper provides a fresh conditional sub-manifold perspective on disentanglement and a theoretically sound metric for measuring disentanglement. 

Considering this novel perspective and a resulting theoretically sound metric for measuring disentanglement that addresses some of the issues with current metrics, I recommend accepting the paper. 

Decision: Accept (Poster)