Evaluating the Disentanglement of Deep Generative Models through Manifold Topology
Authors: Sharon Zhou
Eric Zelikman
Fred Lu
Andrew Y. Ng
Gunnar E. Carlsson
Stefano Ermon
Venue: ICLR 2021 Poster (2021)
Keywords: generative models
evaluation
disentanglement

Abstract: Learning disentangled representations is regarded as a fundamental task for improving the generalization, robustness, and interpretability of generative models. However, measuring disentanglement has been challenging and inconsistent, often dependent on an ad-hoc external model or specific to a certain dataset. To address this, we present a method for quantifying disentanglement that only uses the generative model, by measuring the topological similarity of conditional submanifolds in the learned representation. This method showcases both unsupervised and supervised variants. To illustrate the effectiveness and applicability of our method, we empirically evaluate several state-of-the-art models across multiple datasets. We find that our method ranks models similarly to existing methods. We make our code publicly available at https://github.com/stanfordmlgroup/disentanglement.

Published as a conference paper at ICLR 2021
EVALUATING THE DISENTANGLEMENT OF DEEP GENERATIVE MODELS WITH MANIFOLD TOPOLOGY
Sharon Zhou, Eric Zelikman, Fred Lu, Andrew Y. Ng, Gunnar Carlsson, Stefano Ermon Computer Science & Math Departments, Stanford University (sharenz, ezel imma, fredlu, ang, ermon 8cs stanford. Barlsson@stanoord.cee
ABSTRACT
Learning disentangled representations is regarded fundamental task for im- proving the generalization, robustness, and interpretability of generative models. However, measuring disentanglement has been challenging and inconsistent, often dependent on an ad-hoc external model or specific to a certain dataset. To address this, we present method for quantifying disentanglement that only uses the gener- ative model, by measuring the topological similarity of conditional submanifolds the learned representation. This method showcases both unsupervised and su- pervised variants. To illustrate the effectiveness and applicability of our method, we empirically evaluate several state-of-this of-the-art models across multiple datasets. We find that our method ranks models similarly to existing methods. We make our code publicly available at http:///wttbb..oooommooooonnonnp
Real
Generated
Generated Interpolations
Shaps
Por5
Figure I: Factors in the dSpritess dataset displaying topological similarity and semantic correspon- dence to respective latent dimensions in disentangled generative model, shown through Wasser- stein RLT
of the persistent homology of submanifolds conditioned on latent dimension and latent interpolations along respective latent dimensions.
INTRODUCTION
Learning disentangled representations important for variety of tasks, including adversarial robustness, generalization to novel tasks, and interpretability (Stutz et al. 2019 Alemi et al. 2017
2013). Recently, deep generative models have shown marked improvement in disentanglemen across an increasing number of datasets and variety of training objectives (Chen et al.) 2016 Lin al. 2020 Higgins al., 2017 Kim and Mnih] 2018 Chen et al. 201 8b) Burgess et al. 2018 8 Karras et al. 2019). Nevertheless, quantifying the extent of this disentanglement has remained challenging and inconsistent. As a result, evaluation has often resorted to qualitative inspection for comparisons between models.
Ridgeway 2016 Bengio al.
Existing evaluation metrics are rigid: while some rely on training additional ad-hoe models that depend on the generative model, such classifier, regressor,
encoder (Eastwood and Williams/ 2018; Kim and Mnih 2018: Higgins et al.) 2017; Chen et al. 2018b Glorot et al. [20 1 Grathwohl and Wilson 2016 Karaletsos al. 2015: Duan et al. 2020), others tuned for a particulan dataset (Karras et al., 2211)). These both pose problems to the evaluation metric's reliability, its
relevance different models and tasks, and consequently, its applicable scope. Specifically, reliance

---

Published as a conference paper at ICLR 2021
on training and tuning external models presents tendency to be sensitive to additional hyperpa- rameters and introduces partiality for models with particular training objectives, c.g. variational methods (Chen et al. 2018b; Kim and Mnih 2018 Higgins et al.] 2017; Burgess et al.] (0118) or adversarial methods with an encoder head on the discriminator (Chen et al. 2016 Lin et al. (020).
In fact, this reliance may provide explanation for the frequent fluctuation in model rankings when new evaluation metrics are introduced (Kim and Mnih 2018): Lin et al.) 2020; Chen et al. 2016).
Meanwhile, dataset-specific preprocessing, such as automatically removing background portions from generated portrait images (Karras et al. 2019), generally limits the scope of the evaluation metric's s
applicability because it depends on the preprocessing procedure and may otherwise be unreliable.
To address this, we introduce an unsupervised disentanglement evaluation metric that can be applied across different model architectures and datasets without training an ad-hoc model for evaluation or introducing -ataset-specific preprocessing step. We achieve this by using topology, the mathematical discipline which differentiates between shapes based on gross features such as holes, loops, etc., alongside density analysis of samples. The combination of these two ideas are the basis for functional persistence, which is one of the areas of application of persistent homology (Cayton 2005 Narayanan and Mitter 2010 Goodfellow et al. (2166). In discussing topology, we walk a fine line between perfect mathematical rigor on the one hand and concreteness for a more general audience on the other. We hope we have found the right level for the machine learning community.
Our method investigates the topology of these low-density regions (holes) by estimating homology, a topological invariant that characterizes the distribution of holes on manifold. We first condition the manifold on each latent dimension and subsequently measure the persistent homology of these condi- tional submanifolds. By comparing persistent homology, we examine the degree to which conditional submanifolds continuously deform into each other. This provides a notion of topological similarity that is higher across submanifolds conditioned on disentangled dimensions than those conditioned on entangled ones. From this, we construct our evaluation metric using the aggregate topological similarity across data submanifolds conditioned on every latent dimension in the generative model.
In this paper, we make several key contributions:
We present an unsupervised metric for evaluating disentanglement that only requires the generative model (decoder) and is dataset- aagasstt-ggosttcc.. order to achieve this, we propose measuring the topology of the learned data manifold with respect to its latent dimensions. Our approach measures the topological dissimilarity measure across latent dimensions, and permits the clustering of submanifolds based on topological similarity.
We also introduce a supervised variant that compares the generated topology to real reference.
For both variants, we develop topological similarity criterion based on Wasserstein distance, which defines a metric on barcode space in persistent homology (Carlsson 2019).
Empirically, we perform an extensive set of experiments to demonstrate the applicability of our method across 10 models and three datasets using both the supervised and unsupervised variants. We find that our results are consistent with several existing methods.
2 BACKGROUND
Our method draws inspiration from the Manifold Hypothesis (Cayton| 2005 2010; Goodfellow et al. (2016), which posits that there exists a -imeennnnioonal manifold Man on which real data lie and Paata (x) is supported, and that generative models g : Z X learn an approximation of that manifold MMmodel As a result, the true data manifold Masta contains high-density regions, separated by large expanses of low-density regions. MMmodet approximates the topology of Maata: and superlevel sets of density within Maata: through the learning process.
Narayanan and Mitter
k-manifold a space X, for example a subset of R for some n, which locally looks like an open set in & (formally, for every point x E X, there is a subset can be reparametrized to an open disc in RR... A coordinate chart for the manifold is an open subset U of & together with a continuous parametrization 9 U + X of a subset of X. An atlas for X is a collection of coordinate charts that cover X. For example, any open hemisphere in a sphere is a coordinate chart, and the collection of all open hemispheres form atlas. We say two manifolds are homeomorphic if there is a continuous map from X to Y that has continuous inverse. Intuitively, two manifolds are homeomorphic if one can be viewed as continuous reparametrization of the other. If we have continuous map f from
2

---

Published as a conference paper at ICLR 2021
Mmodel
(x)
X~ (x)
Simplices, increasing proximity
Wasserstein RLT
(a)
(b)
(c)
(d)
Figure 2: Illustration of obtaining Wasserstein Relative Living Times (W. RLTs) from manifold. (a) learned manifold with holes, on which (b) Pmodal (x) is presumed to be supported (images here generated based on the dSSprites dataset). From Pmodal (x), we obtain (c) samples X. From X, we construct (d) simplicial complexes from increasing the proximity of balls over time, producing distribution of holes of varying dimensionalities, an RLT. In this example. we first have no holes the simplicial complex (homology group Ho), then both 1 1-dimensional hole and no hole (H).H)). and finally I-dimensional hole (H₂). From the W. barycenter of many RLTs, we obtain (e) W. RLT. Fig, (a) is drawn from Hanson (1994) and (c)(d) are drawn from Khrulkov and Oseledets (2018).
manifold X to I,,, and are given two nearby points T and ÿ in R", it is often useful to compare the subsets (z) and f (z)) which are manifolds (where the Jacobian matrix of f is maximal rank). They are frequently homeomorphic, and we will be using topological invariants that can distinguish between non-homeomorphic manifolds.
Among the easiest topological invariants to numerically estimate is homology (Hatcher) 2005). which characterizes the number of k-dimensional holes in a topological space such as manifold. Intuitively, these holes correspond to low-density regions on the manifold. The field of persistent homology offers several methods for estimating the homology of a topological space from data samples (Carlsson 2019). Recent work on Relative Living Times (RLTs) (Khrulkov and Oseledets) 2018) has applied persistent homology to generative model data manifolds, also enabling direct comparison of generated data manifolds to real ones. For low-dimensional data (and images), points correspond to their vector representation (flattened matrices of pixels): for high-dimensional data, particularly images, points correspond to vectorized embeddings from pretrained VGG16 (Simonyan and Zisserman 2015).
To obtain RLTs, we first construct family of simplicial complexes-grape graph-like structures-from data samples, each starting with a set of vertices representing the data points and no edges (Figure These are witness complexes that characterize the topology of a set of points, a common method for statistically estimating topological invariants in persistent homology (Carlsson 2019; Lim et al. 2020). These simplicial complexes approximate the persistent homology of the data manifold by identifying k-dimensional holes present in the simplices varying levels of proximity. Proximity is defined with dissimilarity metric (Euclidean distance) between points. It is used to build simplicial complex in which a collection of points spans simplex if all points have proximity measure less than some threshold, as shown in Figure 2d Varying the threshold gives persistent homology. As proximity increases, simplices are added, creating varying numbers of k-dimensional holes, which gives rise to persistence barcodes (Carlsson) 2019 Zomorodian and Carlsson 2005; Ghrist. 2008).
RLTs are vectorizations of these persistence barcodes, specifically the discrete distributions over the duration of each k-dimensional hole as it appears and disappears, or their lifetime relative to other holes. This is merely one method to (partially) vectorize persistence barcodes efficiently, and we leave it to future work to explore alternate methods (Adcock et al. 2013 Bubenik (015)) To measure the topological similarity between data samples representing two generative model manifolds, Khrulkov and Oseledets (018) then take the Euclidean mean of several RLT's produce discrete probability distribution, called a Mean Relative Living Time; they propose employing the Euclidean distance between Mean Relative Living Times as the measure of topological similarity between two sets of data samples, known as the Geometry Score. Additional background is in Appendix B
3 MANIFOLD INTERPRETATION OF DISENTANGLEMENT
We use prevailing definitions of disentanglement where a disentangled model has a factorized latent space corresponding bijectively factors of variation (Shu et al.) 2019 Higgins al. 2018
Duan
3

---

Published as a conference paper at ICLR 2021
Scale
Rocation
Entangled
Figure 3: Consider a set of images of a spinning heart with different sizes. We can embed these images on any group that combines a rotational and scalar invariance (i.e. group with the SO(2,1) symmetry), visualizing this as a conical shell. The submanifolds conditioned on a given rotation have holes, while those conditioned on scale have ID hole. Notably, the topology of the submanifold when holding scale fixed is different from the topology when holding rotation fixed. Latent dimensions of disentangled model would embed images on each axis. By contrast, an entangled model may embed one dimension on an axis and the other in spiral.
ct al. 2020). We frame our approach primarily the definition in Shu et al. (011)), with more formal connection between their approach and our method in Appendix A
From the manifold perspective, disentanglement is an extrinsic property that is dependent on the generative model's atlas. Consider a disentangled generative model with manifold M that assumes topology We can define another generative model g' with the same underlying manifold M and but it is entangled and has different atlas. In fact, we can define several alternate disentangled and entangled atlases, provided there multiple valid factorizations of the space. As result, need method that can detect whether an atlas is disentangled.
In this paper, we slice M into submanifolds Us M that are conditioned on factor s; value v. These conditional submanifolds may have different homology from their supermanifold M. If we observe samples from one factor, e.g. X,,=v ~ =1 at varying values of v. we find that all samples Xe,=v appear identical, except with respect to that single factor of variation s set to different value of v. For generative model, the correspondence between latent dimensions 2, and factors s; is not known upfront. As result, we perform this procedure by conditioning on each latent dimension zj-
Conditional submanifold topology. For two submanifolds have the same topology, there needs to be continuous and invertible mapping between them. First, assume that there exists an invertible mapping, or encoder e: X + Z, and generative model g : Z - X, where both functions are continuous. Then, for a given and x = g(z), we can recover by the composition z e(g(z)). We can also construct a simple linear mapping ( z - z', which adapts a factor's value, such that z' l(e(g(z))) remains continuously deformable. This holds across factors, where the manifold is topologically symmetric with respect to different factors, i.e. its conditional submanifolds are homeomorphic. As example, consider disentangled generative model (20, 21, 22) that traces tri-axial ellipsoid n²²
27z + 5n + = 1. If we condition the model on varying values of each factor, the resulting submanifolds are ellipses and have the same topology.
Most complex manifolds have submanifolds that have non-hon homeomorphic factors of variation. For example, consider generative model g( 21) that traces cylindrical shell with angle zn, height 21, and for simplicity, no thickness. The submanifolds conditioned on angle 2g form lines (no holes), while the submanifolds conditioned on height 2 form circles (a ID hole). However, the topology remains the same for a given factor. A visualization of this principle on a cone is shown in Figure 3 Taken together, this means that submanifolds within factor (intra-factor) are homeomorphic, while submanifolds between factors fextrr-fctoor) can be either homeomorphic or non- non- homeomophhe....
Topological asymmetry. Because topologically asymmetric submanifolds are non non- ho--omoooooopiic. using single e that continuously deforms across submanifolds no longer holds under disentanglement. To address this, assume that for each factor j, there exists continuous invertible encoder e, X - Z, that exclusively encodes information on j from generated sample. In the cylindrical shell example, this means continuously deforming across submanifolds conditioned on varying values of ag using eo (deforming between lines) and likewise for ž using an el (deforming between circles). Note that this formulation prevents continuous deformations between lines and circles. More generally, we cannot continuously deform across submanifolds conditioned by arbitrary factors and expect the topology to be preserved. This procedure now amounts to performing latent traversals along an axis and observing the topology of the resulting submanifolds. In a disentangled model, the j-conditional
4

---

Published as a conference paper at ICLR 2021
Algorithm I: Procedure for producing W. RLTs on generated images: For generator g with latent prior P and N samples, D= latent dimensions, and ni samples per latent dimension, returns a W. RLT for each dimension. RLT is the RLT procedure from Khrulkov and Oseledets (2018).
for i in 1: do Sample z(() ~ P (note: E .RNX..)
end for
for latent dimension d in 1: D= do
for 1 in 1: mnd do z'+ (coopy(z) Set 2'2 4 k Pd
Compute e₂ 4- (mbeddinggg((a)) (e.g. using VGGI6]
Compute rit[d, k] + (L, 1/128, Lo = 64, n = 100)
end for
Compute WB [d] + (.BBarycenterrrt((d..) end for return WB
submanifolds exhibit the same topology by continuous composition of (z(())), using linear mapping that only adapts factor j across the traversal, i.e. = (g)))()
In an entangled model by contrast, more than one factor-such as both the angle and height in the cylindrical shell example-exhibit variation along a dimension æj. Put another way, the topology on submanifolds conditioned on 2j changes when multiple factors contribute to variation along this dimension. Concretely, following the cylindrical shell example, a dimension that encodes height and, after certain height threshold, also begins to adapt the angle will result in a topology that changes to include a 2D hole. Consequently, submanifolds conditioned on the same latent dimension 2j have the same topology in disentangled model, yet different topology in an entangled one.
Because we cannot assume that the data manifold of a generative model is completely symmetric, we only consider submanifolds to be homeomorphic along the same factor in disentangled model. By contrast, since these submanifolds are not homeomorphic in an entangled model, we can measure the similarity across submanifolds to evaluate model's disentanglement. Using this notion of intra-factor topological similarity, we may sufficiently measure disentanglement in most cases, but it does not shield us from the scenario where a generative model learns single trivial factor along all dimensions, i.e. factorization of one. If we assume that there exists assymmetries in the data manifold, then ensuring that the manifold exhibits topological dissimilarity between certain factors would disarm that case. We operationalize this by identifying homeomorphic clusters of factors, whereby each cluster has a distinct topology to ensure there is not a factorization of one. Within clusters, we measure topological similarity, but between clusters, we calculate topological dissimilarity. Consequently, topological similarity and dissimilarity form the basis of our evaluation metric. A more principled treatment of how manifold topology measures disentanglement is in Appendix [A]
3.1 TOPOLOGICAL SIMILARITY USING WASSERSTEIN RELATIVE LIVING TIMES
To estimate the topological similarity between conditional submanifolds, build on Relative Living Times (Khrulkov and Oseledets] 2018) and introduce Wasserstein (W.) Relative Living Times. Wasserstein distance, unlike Euclidean distance, defines a metric on barcode space (arlssoon 2019); recall that barcodes are the discrete distributions representing the presence and absence of different k-dimensional holes (more formally known as Betti numbers, or k- th homology groups), vectorized to form RLTs CCalssson 2019). This strongly motivates us to consider Wasserstein over Euclidean distance, which we find empirically to improve separation between distinct factors of variation (on real disentangling dataset). detailed in Appendix C.
Figure 4: Wasserstein RLTs from factors in the CelebA dataset, based on the same cluster above, and in distinct clusters below. As one can see, the W. RLTs within a cluster (above) are more similar to each other than to those outside of that cluster (below). Additional examples are in Appendix D.
Thus, in lieu of the Euclidean mean across RLTs,
5

---

Published as a conference paper at ICLR 2021
Spectrally Coclustered Topological Similarity Matrices
B-VAE
P-TVVAE
InfeGAN-CR
FactorVAE
InfoGAN
VAE
B-VAE
ProGAN
StyleGAN
CelebA-HQ
dSprites
WGAN-GP
BEGAN
CelebA
Figure 5: Topological similarity matrices across experimental conditions. Dark clusters along the diagonal indicate homeomorphic clusters. Darker values indicate greater similarity (lower W. distance) to each other. Note that models spectrally cocluster differently across dataset settings.
we equivalently employ the W. barycenter (Aguch and Carlier 2011). For distances between W. barycenters, we employ standard W. distance.
The W. barycenter Pw of the distributions P1...Pv corresponds finding the minimum cost for transporting Fw to each ps, where cost is defined in W-2 distance: fw = arg min, Σ A,W (g.P)). where À 2 and وド= ' 1. This is a weighted Fréchet mean, p = arg ming Σ j=1 Azd(9.P)), where d = W2 In contrast, Euclidean distance, or the l2 norm, is defined using
Because distributions represent discrete unnormalized counts of k-dimensional holes, leverage recent work in unbalanced optimal transport (Chizat et al. 20188] Frogner et al. (0155) that assumes that P; are not normalized probabilities containing varying cumulative mass. The unbalanced W. barycenter modifies the W. distance to penalize the marginal distributions based on the extended KL. divergence (Chizat al.) [2018] Dognin et al.) 2019). Unlike Euclidean, Hellinger, or total variation distance, W. distance defines a valid metric on barcode space in persistent homology (Carlsson 2019).
We provide the procedure for W. RLTs for the generated data manifold in Algorithm and the real data manifold in Algorithm? 2 of Appendix | We also show in Appendix C that the use of both W. RLTS and W. distance result in distance metric on sets of RLT's that best separates similar and dissimilar topological features, measured using persistent homology.
3.2 EVALUATION METRIC
Equipped with procedure for measuring topological similarity, we develop a disentanglement evaluation metric from intra-cluster topological similarity and extra-cluster topological dissimilarity. Beginning with intra-factor topological similarity, topology of (x)dee((|s, = v) varies with respect to a factor s; at different values of u. Specifically, partieular factor 8; value v, while allowing other factors 8 to vary. We then measure the topology of this conditional submanifold. For each factor 8i, we find the topology of conditional submanifolds at varying values of A disentangled model would exhibit topological similarity within the set of submanifolds conditioned on the same s;. We visualize similar and dissimilar W. RLTs on factors of the CelebA dataset Figure
concerned with the degree which the
condition the manifold
For a generative model, the correspondence between latent dimensions zj and factors 8; is not known upfront. As result, we perform this procedure by conditioning on each latent dimension 2y. We then assess pairwise topological similarity across latent dimensions "j.Ad(2),22) 2x), where d is the W. distance between W. RLTTs, This operation constructs j-dimensional similarity matrix M. We use spectral coclustering (Dhillon 2001) on M to cocluster zj into biclusters, which represent different clusters of likely homeomorphic submanifolds conditioned on shared factor. Spectral coclustering uses SVD to identify, in our case, the < j most likely biclusters, the subsets of rows that are most similar to columns in M. The resulting biclusters create a correspondence from latent dimensions 2 to cluster of homeomorphic submanifolds conditioned total variance of intra-cluster variance and extra-clustes variance on the biclusters in to find the value for c. Aggregating biclusters in M, we obtain c-dimensional matrix M' (see examples in
factor h.. We then minimize the

---

Published as a conference paper at ICLR 2021
Using My we compute score j that rewards high intra-cluster similarity Pe: tr(M)) and low extra-cluster similarity Pte = E. M - +t(MM) where a indexes the bicluster rows of M'. This score is based on the normalized cut objective in spectral coclustering to measure the strength of associations within bicluster (Dhillon 2001). As a result, the unsupervised evaluation metric
Figure 5)
J Pc Pro.
Supervised variant. In order to capture the correspondence between the learned and real data topology, we present a supervised variant that uses labels of relevant factors on the real dataset to represent the real data topology. This is motivated in large part by F. Locatello et al. (2019), who importantly show that learning a fully disentangled a model requires supervision; further discussion is in Appendix B While this supervised variant requires labeled data, there classifiers or encoders that might favor one training criterion over another. Persistent homology is computed in the same way for the real data submanifolds the generated data submanifolds, except that we have desired clusters of factors 8; upfront. See Figure [ a comparison between real and generated Wasserstein RLTs of two dSprittes factors. The major difference that the generated data manifold is no longer compared to itself, but to the real data manifold, where topological similarity now computed between the two manifolds: Vi,d(2;: Ej, 8i), where factors in the real topology form a specific factorization, so model that finds an alternate factorization and scores well on the unsupervised evaluation metric may not fare well on the supervised variant.
external ad-hoc
W2. Note that the relevant
We use the same spectral coclustering procedure, though this time on a X matrix. Note that because is not a square matrix, Pr: = Ea=o M' and Pic = -E-0 Lb=g M' - Pc. where c = min (c, i) so we only consider the real factors if >> i. Finally, we normalize the final score by number of factors
Pse p//i,
to penalize methods that do not find any correspondence to some factors. Ultimately, the supervised evaluation metric favors submanifolds, conditioned on clusters of latent dimensions, which are similar to the conditional submanifolds of the ground truth dataset.
Limitations. In Figure 6 we highlight cases where our evaluation metric may face limitations. delineated from scenarios where it would behave as expected. The first limitation is that it is theoretically possible for two factors to be disentangled and, under cases of complete symmetry, still have the same topology. This is more likely in datasets with trivial topologies that are significantly simpler than dSprites. While partial symmetry is handled in the evaluation metric with spectral coclustering of homcomorphic factors, complete symmetry is not. Similar Topology
Different Topology
Because we assume the manifold is not exactly symmetric, do not account for all factors to present symmetry. In order to safeguard against this case, we would need to consider the covariance of topological similarities across pairwise conditional manifolds. This requires selecting fixed points from that hold two dimensions constant, and subsequently verifying that the topologies do not covary,
Disentangled Entangled
Expected
Not applicable under the manifold
Low
interpretation
Unlikely complex data manifolds
Expected High μ
However, this approach comes with high computational cost for benefit only to, for the most part, simple toy datasets. we assign Dirichlet process prior over all pos- sible topologies (Ranganathan) 2008) and treat the number of factors as the number of samples, we find the probabil- ity of having only single set of all homeomorphic factors decreases factorially with the number of dimensions ri.
Figure 6: binations considered in our method.
entanglement com-
An additional limitation of our method is that RLTs do not compute a full topology of the data manifold, but instead efficiently approximate one topological invariant, homology, so that we can comparatively rank generative models on disentanglement. Our overall approach of measuring disentanglement is general enough to incorporate measurements of other topological invariants.
4 EXPERIMENTS
Across an extensive set of experiments, our goal is to show the extent to which our metric is able to evaluate across different generative model architectures, training criteria, and datasets. We
7

---

Published as a conference paper at ICLR 2021
(Ours)
Mutual Information Gap
H. (Ours)
Disentanglement (Kim Mrihh)
VAE
Fartrkit
p-revs
Vi
ponsg
ATCOsE
Vate
Festrtiat
IsOGAN
vac
IstoGNN
IndsiANcC
Unsupervised
Supervised
Figure 7: Comparisons of y MIG (Chen et al. 2018b) and Psur classifier-bssed disentanglement score (Kim and Mnih 2018) on the dSprites dataset.
additionally show that metric performs similarly to existing disentanglement metries, without the same architectural or sppecic needs. We use pretrained model checkpoints where possible or else train with default hyperparameters. For each model and dataset combination we compute ja following Algorithms and 3. To compute HsuP we also use Algorithm to generate the W. RLTs of the real dataset. For the embedding function in Algorithms and 2, we used pretrained VGG 16 (Sioonyan and Zisserman 2015) with the last 3 layers removed to embed image samples into 64 feature dimensions. Additional training details are in Appendix G
Datasets. We present empirical results on three datasets: (1) dSprites (Matthey et al. 2017) is canonical disentanglement dataset whose five generating factors (shape, scale, orientation, x-position, y-position] are complete and independent, i.e. they fully describe all combinations in the dataset; (2) CelebA is a popular dataset for disentanglement and image generation, and is comprised of over 202K human faces, which we align and crop to be 64 x 64 px (Liu et al. (011)). There are also 40 attribute labels for each image; and (3) Celeba-HQQ. a higher resolution subset of CelebA consisting of 30K images Karras et al. 2018).
Generative models. We compare ten canonical generative models, including a standard VAF, 8- VAE (Higgins et al., (017). 3-VAEB (Burgess et al., 2018). FactorVAE (Kim and Mnih 2018)
3-TCVAE (Chen et al., 2018b), InfoGAN (Chen et al. (2166), InfoGAN-CR (Lin et al. 2020
BEGAN (Berthelot et al. 2017). WGAN-GP (Gulrajani al. 2017), ProGAN (Karras et al. 2018
and StyleGAN (Karras et 2019). We evaluate VAE and InfoGAN variants on dSprites and CelebA, WGAN-GP and BEGAN CelebA, and ProGAN and StyleGAN CelebA-HQ. We match models to datasets, on which they have previously demonstrated strong performance and stable training.
Metric parity. We find that (f. [suP) ) rank models similarly several other frequently cited metrics, including: (1) an information-theoric metric MIG that uses an encoder (Chen et al. 2018b), (2) supervised metric from (Kim and Mnih, 2018) that uses a classifier, and (3) dataset-specific
metric PPL (Karras et al.) 2019) that caters to face datasets such as CelebA-HQ We use scores from their respective papers and prior work (Chen et al., 2018b Kim and Mnih 2018 Lin et al.) 2020)] Karras et al. 2019) to show that our method ranks most or all models the same across each metric (ju compared to MIG and PPL., AsuP to the supervised method). The source of deviation from MIG is the ranking of B-V:E; nevertheless, both of our scores exhibit exceptionally high variance across runs, suggesting that 3-VAE has inconsistent disentanglement performance (see Figure 7). The classifier method ranks 3- 3-TCAAE and FactorVAE quite far apart, while ours ranks them similarly. We find that their nearly identical training objectives should rank them more closely and do not find this disparity particularly unexpected. Finally, our method agrees with PPL rankings on CelebA-HQ
As shown in Table I these experiments highlight several key observations:
Performance is not only architecture-dependent but also dataset-dependent. This highlights the importance of having a metric that can cater to comparisons across these facets. Nevertheless, we note that 3-VAEB shows especially strong results both metrics and dataset settings.
As expected, the VAE and InfoGAN variants designed for disentanglement show greater performance on than their GAN counterparts. However, Msse, we find that BEGAN is able to perform inseparably close to 3- B-E, suggesting that the model learns dependent factors consistent with the attributes in CelebA.
8

---

Published as a conference paper at ICLR 2021
Model
Dataset
J4
Paur
Model
Dataset
1
Psup
3- g
dSprites
23.53 1 8.14
3,55 1 4.25
B-VAEB
CelebA
4.73 2.27
0.29 1 0.25
3-TCVAB
dSprites
14.92 3,46
-0.79 + 1.35 1.35
3-TCVAE
CelebA
10.66 1 2.48
0.04 0.36
InfoGAN-CH
dSprites
9.73 4.03
1.85 2.63
InfoGAN-CE
CclebA
0.72
0.07 0.15
FactorVAE
dSprites
8.66 + 1.83
0.35 0.90
FactorVAE
CelebA
8.53
-0.14
InfoGAN
dSprites
7.42 + 1.19
0.16
InfoGAN
CelebA
1.11
0.00
VAE
dSprites
7.05 + 1.25
1.54 + 1.27
VAE
CelchA
6.98
0.00
8-VAE
dSprites
6.53 2.89
1.81 2.90
3-VAE
CelchA
15.10+8.99
0.13+
StyleGAN
CelebA-HQ
1.03 0.24
0.77 0.07
BEGAN
CelchA
0.85 + 0.25
0.22 + 0.10
ProGAN
CelebA-HQ
0.68 + 0.08
0.37 + 0.45
WGAN-GP
CelebA
0.83 + 0.29
0.07 = 0.13
Table 1: Experimental results several generative models and dataset settings for unsupervised
and supervised Msup metrics, across five runs. We find that, consistent with other disentanglement metrics, model architecture that evaluated supercedes all others on every metric and dataset. Higher values indicate greater disentanglement.
With similar training objectives, 3-TCVAE and Factor VAE demonstrate comparable strong performances on μl across both dSprites and CelebA. 3-TCVAE displays slight, yet consistent, improvements FactorVAF: which may point to FactorVAE's underestimation of total correlation (Chen et al.) 20186). Nevertheless, Factor VAE demonstrates higher /sup on dSprites.
StyleGAN demonstrates consistently higher disentanglement, compared to ProGAN, which supports architectural decisions made for StyleGAN (Karras al. 2019).
Different datasets and requirements for evaluating disentanglement may favor an unsupervised variant over a supervised variant, and vice versa. Without known factors or without factors of interest, the unsupervised variant is clearly the more favorable or in some cases the only option.
One benefit of using the unsupervised variant even the factors are known when multiple combinations of factors are valid to disentangle the data. For example, one can imagine disentangling color into RGB [red, green, blue), HSL. (hue, saturation, lightness), or HSV (hue, saturation, value), J. The unsupervised evaluation metric would be satisfied with model that disentangles into any of these. A supervised evaluation metric, for which the known factors are RGB, however, would not. RGB is human (supervised) prior that dictates not only whether but how the factors disentangle. In effect, this leads to differences in rankings across the two variants. Nevertheless, in the case of disentanglement dataset, such dSprites, where the factors were created first and the data was created from those factors, we would expect that the known factors are reasonable ones, if not the best ones according human judgment, to disentangle. That is, while there may be alternatives, this is a good, and quite possibly the most intuitive, human prior.
However, datasets in the wild, which CelebA veers closer to, differ in this respect. We do know the complete set of factors of variation for CelebA's human faces, and the attributes provided in the metadata are subset at best. In this case, if one wishes to measure disentanglement generally, we suggest using unsupervised approaches to assess disentanglement of learned model. If one wishes to measure disentanglement specific factors, e.g. sunglasses and hair color, then the supervised variant would be more appropriate. Nevertheless, different variants suit different needs, and across datasets and variants, we may expect different models to emerge as superior in their ability to disentangle.
CONCLUSION
In this paper, we have introduced a disentanglement evaluation metric that measures intrinsic proper- ties of a generative model with respect to its factors of variation. Our evaluation metric circumvents the typical requirements of existing evaluation metrics, such as requiring an ad-hoc model, a particular dataset, canonical factorization. This opens up the stage for broader comparisons and datasets. Our contributions also consider several cases of disentanglement, where labeled data is available (unsupervised) or where direct comparisons user-speciffed, semantically interpretable factors are desired (supervised). Ultimately, this work advances our ability to leverage the intrinsic properties of generative models observe additional desirable facets and apply these properties important outstanding problems in the field.
models

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
We would like to thank Torbjörn Lundh, Samuel Bengmark, and Rui Shu for their helpful feedback and encouragement the preparation of manuscript.
REFERENCES
A. Adcock, Carlsson, and G. Carlsson. The ring of algebraic functions on persistence bar codes. Homology, Homotopy and Applications, 18, 04 2013.
M. Agueh and G. Carlier. Barycenters in the wasserstein space. SIAM Journal Mathematical Analysis, 43(2)9904 924, 2011.
A. Alemi, I. Fischer, J. Dillon, and K. Murphy. Deep variational information bottleneck. In ICLR, orggabb///12..... 00410
2017. URL httthttps://arrt
Y. Bengio, A. Courville, and P. Vincent. Representation learningg review and new perspectives. IEFE transactions on pattern analysis and machine intelligence, 3598):1798-1828, 2013.
D. Berthelot, T. Schumm, and L. Metz. Began: boundary equilibrium generative adversarial networks. arXiv preprini arXiv: /703. 10717, 2017.
P. Bubenik. Statistical topological data analysis using persistence landscapes. The Journal of Machine Learning Research, 16(1):77-102, 2015.
C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins. and A. Lerchner. Under- standing disentangling in beta-vae. arXiv preprint 1xXv::8804003599 2018
G. Carlsson. Persistent homology and applied homotopy theory. Handbook of Homotopy Theory, 2019.
L. Cayton. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 12(1-17):1 2005.
N. Chen, A. Klushyn, R. Kurle, X. Jiang, J. Bayer, and P. van der Smagt. Metrics for deep generative models. PMLR, 2018a.
T. Q. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pages 2610- 2620, 2018b.
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172-2180, 2016.
L. Chizat, G. Peyré, B. Schmitzer, and F.-X. Vialard. Scaling algorithms for unbalanced optimal transport problems. Mathematics of Computation, 87(314):25332660) 2018.
I. S. Dhillon. Co-clustering documents and words using bipartite spectral graph partitioning. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 269-274, 2001.
Dognin, Melnyk, Y. Mroueh, J. Ross, C. D. Santos, and T. Sereu. Wasserstein barycenter model ensembling. ICLR, 2019,
S. Duan, L. Matthey, A. Saraiva, N. Watters, C. P. Burgess, A. Lerchner, and I. Higgins. Unsupervised model selection for variational disentangled representation learning. ICLR, 2020.
C. Eastwood and C. K. Williams. A framework for the quantitative evaluation of disentangled representations. ICLR, 2018.
Locatello et al. Challenging common assumptions in the unsupervised learning of disentangled representations. ICML, 2019.
10

---

Published as a conference paper at ICLR 2021
C. Frogner, C. Zhang, H. Mobahi, M. Araya, and T. A. Poggio. Learning with wasserstein loss. In Advances in Neural Information Processing Systems, pages
2015.
R. Ghrist. Barcodes: the persistent topology of data. Bulletin of the American Mathematical Society, 45(1):61-75, 2008.
X. Glorot, A. Bordes, and Y. Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. ICMI,, 2011.
I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
W. Grathwohl and A. Wilson. Disentangling space and time in video with hierarchical variational auto-encoders, arXiv preprint arXiv: 16/2.44// 2016.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pages 5767-5777, 2017.
A. J. Hanson. A construction for computer visualization of certain complex curves. Notices of the Amer. Math. Soc, 41(9): 1994.
A. Hatcher. Algebraic topology. Cambridge University Press, 2005,
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR, 2(5):6, 2017.
I. Higgins, D. Amos, D. Pfau, Racaniere, L. Matthey, D. Rezende, and A. Lerchner. Towards definition of disentangled representations. arXiv preprint arXiv: 2018.
D. Horak, S. Yu, and G. Salimi-Khorshidt Topology distance: A topology- based approach for evaluating generative adversarial networks. arXiv preprint arXiv:2002.12054 2020.
Karaletsos, S. Belongie, and Rätsch. Bayesian representation learning with oracle constraints. ICLR, 2015.
T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ICLR, 2018.
Karras, S. Laine, and T. Aila. style-based generator architecture for generative adversarial networks. CVPR, 2019.
V. Khrulkov and I. Oseledets. Geometry score: A method for comparing generative adversarial networks. ICML, 2018
H. Kim and A. Mnih. Disentangling by factorising. PMLR, 2018
Li, J. Murkute, K. Gyawali, and L. Wang. Progressive learning and disentanglement of hierarchical representations. arXiv preprint arXiv:2002. 10549, 2020.
S. Lim, F. Memoli, and O. B. Okutan. Vietoris-rips persistent homology, injective metric spaces, and the filling radius. arXiv preprint arXiv:2001.0755 07558, 2020.
Lin, K. K. Thekumparampil, G. Fanti, and S. Oh. Infogan-cr: Disentangling generative adversarial networks with contrastive regularizers. ICML, 2020.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
M. Vlachos et al. Improving co-cluster quality with application to product recommendations. CIKM, 2014.
L. Matthey, I. Higgins, D. Hassabis, and A. Lerchner. dsprites: Disentanglement testing sprites dataset.
2017.
H. Narayanan and S. Mitter. Sample complexity of testing the manifold hypothesis. Advances in neural information processing systems, pages 1786-1794, 2010.
11

---

Published as a conference paper at ICLR 2021
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2888 2011.
D. Pfau, I. Higgins, A. Botev, and S. Racanière. Disentangling by subspace diffusion. Advances in Neural Information Processing Systems, 33, 2020.
A. Ranganathan. Probabilistii topological maps. PhD thesis, Georgia Institute of Technology, 2008.
K. Ridgeway, A survey of inductive biases for factorial representation-lcaming CoRR abs/16/2.05299, 2016.
B. Rieck, M. Togninalli, C. Bock, M. Moor, M. Horn, T. Gumbsch, and K. Borgwardt. Neural persistence: A complexity measure for deep neural networks using algebraic topology. ICLR 2019, 2018.
H. Shao, A. Kumar, and Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 315-323, 2018.
R. Shu, Y. Chen, A. Kumar, S. Ermon, and B. Poole. Weakly supervised disentanglement with guarantees. ICLR 2020, 2019.
A. Shukla, S. Uppal. S. Bhagat, Anand, and P. Turaga. Geometry of deep generative models for disentangled representations. In Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing, pages 1-8, 2018.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
http: /ttpvv/arvvvvvv//am/ abs/abs/////oooooooae/.aaa/aaaar 1409 1556
D. Stutz, M. Hein, and B. Schiele. Disentangling adversarial robustness and generalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6976- 6987, 2019.
Lin al. On efficient optimal transport: An analysis of greedy and accelerated mirror descent algorithms. ICML, 2019.
A. Zomorodian and G. Carlsson. Computing persistent homology. Discrete & Computational Geometry, 33(2):249-217 2005.
12


Reviews:
[{"review_id": "kX6lU2VI_Ix", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a disentanglement metric to measure the intrinsic properties of a generative model with respect to the factor of variation in the dataset. Toward this, the paper first assumes disentangled factors reside in different manifolds. These different manifolds are the sub-manifolds of some manifold M for a given disentangled generative model. The paper considers the fact that in an entangled model the sub-manifolds are not homeomorphic and thus similarity across submanifolds can be measured to evaluate a model’s disentanglement. As such, disentanglement is related to the topological similarity.  For measuring topological similarity, the paper then introduces Wasserstein Relative Living Times. The proposed metric is used to evaluate standard disentanglement methods and datasets demonstrating the importance. \n\nPros:\n- The paper is well written and the proposed metric is well articulated.\n- The manifold interpretation of disentanglement (section 3) is clear and could be considered a stand-alone contribution to the disentanglement community. \n- The experiments covered depth in terms of dataset selection and the number of models considered for the evaluation of the metric. \n\nFew questions/suggestions:\n- Can the authors briefly describe why W. Distance defines a valid metric on barcode space and others don’t, or point out to relevant literature in section 3.1?\n- In Li et. al., 2019, MIG-sup was proposed to remedy the weakness of the MIG metric. For the unsupervised portion, since MIG was considered a comparision, can authors also compare their method with MIG-sup? Or at least discuss it along with MIG? \n\nLi, Z., Murkute, J. V., Gyawali, P. K., & Wang, L. (2019, September). PROGRESSIVE LEARNING AND DISENTANGLEMENT OF HIERARCHICAL REPRESENTATIONS. In International Conference on Learning Representations.\n\n(Update): The score has been updated after the rebuttal phase.", "extra_scores": {}, "extra_text": {}}, {"review_id": "iK3mpIDampL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: Introduces unsupervised disentangling metric that measures homeomorphic similarity between submanifolds conditioned on a given factor, and homeomorphic dissimilarity on submanifolds conditioned on different factors. The paper also includes a supervised variant which can directly assess topological similarity of submanifolds with label-spaces. The paper also introduces a novel variation of RLTs that  employs wasserstein distance instead of euclidean distance. \n\nStrengths:\n* Novel application of topological similarity to unsupervised evaluation of disentangling.\n* Unsupervised and supervised metrics are both in the same units, and can be directly compared.\n* Experimental comparison, using the proposed metric, of many standard disentangling models on challenging datasets.\n\nWeaknesses:\n* The comparison of the proposed metric to mutual information gap and the metric from Kim and Mnih 2018 shown in Figure 7 is not convincing. Not only does the ranking change, but the distributions of scores look wildly different between models (e.g., \\mu has low variance for VAE in both supervised and unsupervised cases, but both MIG and Kim & Minh show relatively high variance for VAE). It is possible that discrepancy is a good thing, and that the proposed model is capturing disentangling even better than the baseline metrics. However, there is not enough experimental validation to know whether this is the case. The paper would be stronger if it had more experimental validation, including a replication of the reported scores from the MIG (Chen et al., 2018) and (Kim and Minh 2018) paper, and deeper analysis explaining the relative differences in scores for models. Without this component, it is difficult to know whether the model comparison results (Table 1) are meaningful.\n* It would be nice to include a discussion of how the proposed metric accounts for “dead” units (e.g., Eastwood & Williams 2018)? These might result in topologically similar submanifolds (since the conditional distributions would be nearly identical) and a worse score for the metric, even though the representation is arguably well-disentangled.\n\nClarity:\n* Typo: Page 6, Line 5: “Figure 9” should be “Figure 4”\n\nOther notes:\n* It would be very interesting to see an experiment where the unsupervised metric is used to choose hyper-parameters of a model which is then evaluated in a supervised way, to demonstrate that the unsupervised metric is a valid substitute for a supervised metric.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RbsZbVDXYX5", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Synopsis of the paper\n\nThis paper presents a new model for assessing to what extent a deep\ngenerative model is disentangled. In contrast to existing methods, this\npaper proposes an approach founded on topological considerations: by\nassessing the topological dissimilarity of submanifolds of a given model,\nwhich are conditioned on an individual factor. The overarching idea is\nthat a fully-disentangled model should exhibit high topological\ndissimilarity for *different* factors, while exhibiting low topological\ndissimilarity for the *same* factors (but with different values for said\nfactors).\n\nThese topological dissimilarity assessments are achieved by employing\npersistent homology, a method from topological data analysis that\ngeneralises ordinary simplicial homology (i.e. 'counting\nhigh-dimensional holes') to a multi-scale setting (in which point clouds\nare being assessed). Previous work for GAN comparison, viz. the\n'geometry score' is extended by a new internal metric, the Wasserstein\ndistance.\n\nA set of experiments demonstrates the utility of the proposed approach,\nvalidating previous knowledge.\n\n# Summary of the review\n\nThis paper is tackling a highly relevant topic (disentanglement\nanalysis) and provides a novel, fresh perspective that is mathematically\nwell justified. The paper is chock full of excellent ideas that make me\nexcited about their potential; I am sure that this will be shine a light\non the potential of topology-driven methods.\n\nHowever, while the paper is well written for the most part, I cannot\nfully endorse it for publication yet because of two reasons:\n\n1. The paper is missing crucial details about its comparison method.\n   While I know the existing work to piece everything together, I am\n   not sure the method as such is reproducible at the moment.\n\n   I am agreeing with most of the claims in the paper, but I would be\n   hard-pressed to implement exactly the same method in the same\n   fashion, because not all the details are there.\n\n2. While most of the paper flows very well, in an almost conversational\n   style that I very much appreciate, there are also several places at\n   which concepts are not explained correctly or with sufficient\n   precision. There are also some issues with the terminology that make\n   it harder to fully understand what is going on.\n\nIf these two points were to be rectified in a revision of the paper,\nit would definitely strengthen the message of this paper!\n\nPlease see below for detailed comments.\n\n# Detailed comments\n\n- The introduction is really 'on point' and makes an excellent case for\n  requiring more foundational approaches\n\n- The way 'topology' is introduced in the introduction is very\n  imprecise; I find the discussion of 'high-density' and 'low-density'\n  to be problematic here. Algebraic topology, which is where persistent\n  homology originated from, has no notion of density; all considerations\n  of 'holes' are based on purely algebraic arguments. While it is true\n  that you can incorporate density information into persistent homology,\n  this discussion is slightly misleading.\n\n  I would point out rather that holes are typically considered to be the\n  boundaries of these low-density regions (if the 'density argument' is\n  left in there).\n\n- Some of the terms are overloaded and used multiple times: 'latent\n  dimension' is used equivalently to 'latent factor', it seems. I would\n  strongly suggest to use a unified terminology.\n\n- The term 'homeomorphic dimensions' is vague. I also have to point out\n  that the method used in this paper is only *invariant* under\n  homeomorphism! More precisely, two topological objects can have the\n  same homology (or persistent homology, or barcode, etc.) even though\n  they are **not** homeomorphic. This is a crucial detail that is not\n  handled correctly in the paper at present; for example, Figure\n  5 claims that 'homeomorphic groups' are shown, but the approach cannot\n  detect this. If anything, the case should be made for depicting that\n  these groups are 'likely' homeomorphic. Alternatively, one could write\n  that the proposed approach assesses the *topological similarity* of\n  models.\n\n- Often, the term 'topology' is also used as an equivocation.\n  Unfortunately, the term is overloaded in mathematics as well: a space\n  can *have* a topology (i.e. systems of open sets etc.), but I have the\n  impression that the paper often means to say 'shape' or 'topological\n  features' (connected components, cycles, etc.) rather than 'topology'.\n\n- The explanation of a manifold could be improved; why not mention the\n  definition in terms of 'locally looks like some $\\mathbb{R}^d$'? The\n  discussion of 'open discs' might leave some readers baffled.\n\n- I disagree with the sentence 'Under the manifold hypothesis, the latent\n  space of a deep generative model has an extremely dense underlying\n  manifold with few, if any, holes, [...]'. Holes characterise the shape\n  of a manifold; the manifold hypothesis only posits the hypothesis that\n  data originate from a manifold. It does not say anything about the\n  *shape* of the manifold itself, and I do not agree with the notion\n  that having no holes is somehow 'better'.\n\n  A generative model that 'learns' that all samples can be arranged\n  along a cycle in some latent space can be very appropriate depending\n  on the data set.\n\n  (Moreover, Figure 2 in some sense tells the opposite story of this\n  sentence.)\n\n- The definition/introduction of (persistent homology) at the end of\n  Section 2 could be improved: the Vietoris--Rips complex goes back to\n  a 1927 paper by L. Vietoris (https://link.springer.com/article/10.1007/BF01447877);\n  an excellent introduction to modern topological concepts is provided\n  in the book 'Computational Topology: An Introduction' by Edelsbrunner\n  and Harer.\n\n  I also disagree with homology 'approximating the topology'. Homology\n  is *one* specific invariant in algebraic topology. In this sense, it\n  does not approximate anything but describes a data set in terms of\n  its topological features.\n\n- The term 'symplectic vertex' is not correct here, as far as\n  I understand. Everything is discussed in terms of simplicial\n  complexes; I do not see the connection to symplectic topology here. \n\n- The explanation of the Vietoris--Rips complex could be improved.\n  I feel that the explanation in terms of creating edges based on the\n  proximity criterion will leave most readers baffled. I would suggest\n  adding a brief explanation/illustration about this, maybe using Figure\n  2d.\n\n- What is meant by having a manifold 'assume topology $\\tau$'? Does this\n  refer to the manifold expressing a certain a set of barcodes?\n\n- In Figure 3, I would rather write '[...] the submanifolds conditions on\n  a specific rotation [...]'. Upon first reading the figure,\n  I misunderstood its central point, namely that *if* the rotation is\n  fixed, so that only the scale varies, the resulting submanifold will\n  not have any holes, as the variation in scale provides a simple 1D\n  structure.\n\n- I would not say that submanifolds have different 'topologies' than\n  their super-manifold; rather they might/will have different homology\n  maybe or 'shape'?\n\n- While I fully agree with the thrust of the method, I would need more\n  details on p. 4. How is the partitioning of the manifold achieved in\n  practice? How are the factors chosen? How are their respective values\n  chosen? Is it possible to provide more examples here?\n\n- When discussing conditional submanifold topology, consider mentioning\n  how these manifolds are being obtained in practice, maybe by providing\n  more examples.\n\n- I do not understand the main thrust of the 'Topological asymmetry'\n  section. Is the main argument that manifolds conditioned on the same\n  factor should have the same shape and, moreover, that manifolds coming\n  from the same factor should be more similar to one another than\n  manifolds coming from different factors?\n\n- What does it mean that factors are 'not homeomorphic' in Figure 4? Is\n  the ground-truth topology known?\n\n- The statement '...that best separates similar and dissimilar\n  topologies...' should again be rephrased and made more precise, for\n  example by adding '...topological features, as measured using\n  persistent homology...'.\n\n- In the 'Metric' section, I would caution strongly against this term.\n  The RLT/Wasserstein calculations might result in a metric in the\n  mathematical sense (for sure, the Wasserstein distance constitutes\n  a metric on the space of barcodes, as the paper mentions), but I am\n  not sure whether the *score* calculated in this section constitutes\n  a metric. If anything, this would have to be proven.\n\n- The 'Metric' section is also hard to read because of differences in\n  terminology. Why is $\\xi$ used to represent an index, whereas other\n  Greek letters are used for functions? What is $\\delta$? Is it the\n  proposed measure of disentanglement?\n\n- After the 'Metric' section, I am missing a concluding section, that\n  summarises how the method works in practice. The following questions\n  need to be answered:\n\n  1. How are topological features computed (i.e., which point cloud is\n     being used and which filtration)?\n\n  2. How is the Vietoris--Rips complex computed in practice? How is its\n     scale parameter $\\epsilon$ chosen?\n\n  3. What is the dimensionality used to compute topological features,\n     i.e. are high-dimensional features being used at all?\n\n\n- '...to those of the reals.': what does this mean?\n\n- It seems that $\\tau$ is not used consistently; is $\\rho$ meant in\n  Figure 6?\n\nAll in all, this paper makes me very excited about the potential of\ntopology-based methods, and I envision that, with some changes, this\nwill be a strong publication in the future!\n\n# Style & clarity\n\nThe paper is well-written; here are some additional suggestions for\nfurther improving its clarity: \n\n- 'generated dSprites' --> 'generated from the dSprites data set'?\n\n- 'by composition' --> 'by the composition'\n\n- The term 'group' is unfortunately somewhat overloaded as well; is\n  'factor' more appropriate in some places? Or is the paper actually\n  talking about manifold groups?\n\n- I would strongly suggest to check whether 'persistent homology' might\n  not be ore appropriate in many places instead of 'homology'. The\n  latter should be reserved to the setting of *one* simplicial complex,\n  but the main argument of the paper is that one has to adopt\n  a multi-scale description of the data set. \n\n# Update after initial revision\n\nI have re-read the revised version multiple times now, and I thank the authors for the amount of work they put into their revision. I am raising my score for the next discussion round. That being said, I want to point out why I cannot fully endorse this paper yet. First, from the point of topological data analysis, the central algorithm is a comparatively small extension of the Geometry Score paper. I agree that this is a superb idea, yet the main contribution for me lies in the application of that technique to disentanglement—and for this to be fully understandable, some more work is needed. For example, putting the central algorithmic details into the appendix will make the adoption of the method that much harder. \n\nMoreover, while I appreciate the overall story and description of the method, I do not think that readers will understand how this disentanglement is actually _achieved_ by means of the proposed TDA approach. I would therefore prefer to see a more 'technical' or 'algorithmic' description of the contributions in the main paper, in particular since I think that the ideas of conditional submanifold topology require more attention.\n\nMy expertise is more the topology and less so the application of disentanglement; nevertheless, this paper strikes me as highly ambitious with a lot of potential, yet somewhat unfinished in its present form. I do believe that it has the potential to be extremely impactful with some additional modifications (concerning conciseness, but also experimental details).\n\nI fully realise that this is not yet the desired outcome for the authors; I shall endeavour to discuss this further with my fellow reviewers to see that we can reach a consensus!", "extra_scores": {}, "extra_text": {}}, {"review_id": "M-J_zgIWtg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper proposes a novel metric for evaluating disentanglement by taking a manifold-topological perspective on the representations learnt. The key insight is that for a disentangled representation, when we fix a certain factor of variation at different values the topology of the conditional sub-manifolds should be similar. Using this insight the paper proposes a metric for disentangling which does not require annotations of the factors of variation and is more general than previous such tests.\n\nStrengths\n+ Having an approach that is general and easy to compute across datasets and models makes a lot of sense\n\nWeaknesses\n\nIt would be nice to further clarify the intuitions for how disentangling relates to the manifold structures using more examples in the paper for people who are not familiar with the manifold topology literature. For example, in Fig. 3 it would be nice to show qualitatively what an entangled model would do for the rotation scale disentangling situation.\n\nIt seems somewhat surprising that \\beta-VAE_{B} does so much worse than \\beta-VAE on CelebA, were the hyperparameters tuned separately for the CelebA dataset?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "B2TxuNdoqbq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new metric to quantify disentanglement by leveraging ideas from manifold topology, more precisely the homology of conditional submanifolds. It has a strong theoretical grounding (clearly better than existing metrics) and results appear to be promising.\n\nHowever it is at times quite hard to follow, and I am not entirely confident that I fully understood how the Wasserstein RLTs were estimated in practice. I’d still recommend publication, given the novelty and importance of a theoretical sound metric for disentanglement.\n\nComments/questions:\n1. The Introduction, Background and Manifold interpretation sections do a good job of introducing the subject, covering the literature and providing enough background about differential geometry, but I think a clear algorithmic demonstration of steps (d) and (e) of Figure 2 is lacking. \n      1. How exactly do you estimate the Wassertein RLT and compute the matrix M?\n      2. Writing the exact equations used, e.g. in the Appendix, would be helpful. Overall I found the main text to rely on explanations a bit more than would be necessary, where some math would be clearer (having the code will be most helpful however)\n      3. The actual disentanglement metrics $\\mu$ and $\\mu_{sup}$ might deserve a \\begin{equation} for clarity.\n      4. You never explicitly say if “high $\\mu$ => high disentanglement”, which I think is how one should interpret Table 1 given the main text?\n      \n2. Providing a bit more explanations about how to interpret the RLT histograms would be valuable. I imagine that if the histograms vary as a function of their conditioning, this indicates entanglement?\n      \n3. Figure 7 was interesting, but the specific mention of a mismatch in how the classifier method (i.e. Disentanglement (Kim & Mnih) I assume?) ranks $\\beta$-TCVA and FactorVAE differently was hard to follow (i.e. these do not appear that close in your $\\mu_{sup}$ figure?)\n      \n4. Do the topological similarity matrices in Figure 5 provide interesting information about which factors are grouped together?\n      1. Could you add labels on the axes and discuss what each cluster represents in a few selected models/datasets?\n      \n5. How does the method relate to what was done in GEOMANCER [1]? They also leverage differential geometry and holonomy (but as a learning signal for representation, not as a metric obviously), so discussing it might be valuable?\n\n\n[1] https://arxiv.org/abs/2006.12982", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes the use of topological similarity between conditional submanifolds for a given latent dimension as a metric for measuring disentanglement in generative models. To estimate the topological similarity between conditional submanifolds, the authors build upon an earlier work of Relative Living Times (RLT). 

R5 and R4 had concerns on the paper, particularly about the lack of enough novelty in the actual technique (R5) and about the lack of convincing experiments (R4). One of the concerns raised by R4 was around the discrepancies between MIG and FactorVAE. However as noted by other reviewers (R2 and R5), these discrepancies between different popular metrics are well acknowledged in the literature and authors have responded to this point. R2 and R5 also appreciate that avoiding the rotation issue faced by most of these disentanglement metrics is one of the strengths of the proposed metric.

While I tend to agree with R5 that the actual technique is inspired from the earlier work on "Geometry Score", I also think the application to measuring disentanglement in generative models is a novel contribution in itself, especially because current metrics have issues as pointed out by other reviewers -- the paper provides a fresh conditional sub-manifold perspective on disentanglement and a theoretically sound metric for measuring disentanglement. 

Considering this novel perspective and a resulting theoretically sound metric for measuring disentanglement that addresses some of the issues with current metrics, I recommend accepting the paper. 

Decision: Accept (Poster)