When Is Unsupervised Disentanglement Possible?
Authors: Daniella Horan
Eitan Richardson
Yair Weiss
Venue: NeurIPS 2021 Poster (2021)
Keywords: machine learning
manifold learning
representation learning
disentanglement

Abstract: A common assumption in many domains is that high dimensional data are a smooth nonlinear function of a small number of independent factors. When is it possible to recover the factors from unlabeled data? In the context of deep models this problem is called “disentanglement” and was recently shown to be impossible without additional strong assumptions [17, 19]. In this paper, we show that the assumption of local isometry together with non-Gaussianity of the factors, is sufficient to provably recover disentangled representations from data. We leverage recent advances in deep generative models to construct manifolds of highly realistic images for which the ground truth latent representation is known, and test whether modern and classical methods succeed in recovering the latent factors. For many different manifolds, we find that a spectral method that explicitly optimizes local isometry and non-Gaussianity consistently finds the correct latent factors, while baseline deep autoencoders do not. We propose how to encourage deep autoencoders to find encodings that satisfy local isometry and show that this helps them discover disentangled representations. Overall, our results suggest that in some realistic settings, unsupervised disentanglement is provably possible, without any domain-specific assumptions.

When is Unsupervised Disentanglement Possible?
Daniella Horan and Eitan Richardson and Yair Weiss
School of Computer Science and Engineering The Hebrew University of Jerusalem Jerusalem, Israel
{daniella. hhr.eeaaa...rnn...mmmmmmmaaaaaee eitan www.ttaa..aaaaddoonnnonooooo weits//sos........e..ssssses... huji iiiiia. ac il
Abstract
A common assumption in many domains is that high dimensional data are a smooth nonlinear function of small number of independent factors. When is it possible to recover the factors from unlabeled data? In the context of deep models this problem is called "disentanglement" and was recently shown be impossible without additional strong assumptions [17 19] In this paper, we show that the assumption of local isometry together with non- non-Gaussianity of the factors, is sufficient to provably recover disentangled representations from data. We leverage recent advances in deep generative models construct manifolds of highly realistic images for which the ground truth latent representation is known, and test whether modern and classical methods succeed recovering the latent factors. For many different manifolds, we find that spectral method that explicitly optimizes local isometry and Gaussianity consistently finds the correct latent factors, while baseline deep autoencoders do not. We propose how to encourage deep autoencoders to find encodings that satisfy local isometry and show that this helps them discover disentangled representations. Overall, our results suggest that in some realistic settings, unsupervised disentanglement is provably possible, without any Nomain-speciff assumptions.
0.12313
(a) catl
(b) HLLE+ICA
(c) IRMAE
Figure 1: (a) Manifold of highly realistic images with two independent latent factors. Given unlabled images, is it possible recover the true factors? HLLE+ICA (b) succeeds while baseline autoen- coder (c) learns a highly entangled embedding. Similarly, the representation found by HLLE+ICA approximately satisfies local isometry while the encoding of the baseline autoencoder does (bottom). In this paper we show that the assumptions of local isometry and non-Gaussianity are sufficient provably recover disentangled representations.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Introduction
With the recent success of deep learning approaches, there has been resurgence of interest in the problem of recovering the independent factors that generate high dimensional data. As a motivating example, consider the cat1 manifold shown in figure Ta. Each image is of size 256 x 256 generated as smooth, nonlinear function of just two independent latent factors: the fur color and pose of the cat. Is it possible recover these latent factors, given only samples of images?
but is
This problem closely related, yet different from the problem of learning a deep generative model from data. Given training set {2') }. deep generative models find parameters such that x' ge(2") where each latent variable z' is sampled from a base distribution, and the different components of are independent. Tremendous progress in deep generative models has been achieved by methods including Generative Adversarial Networks (GAN) 9 16]. Variational Auto-Encoders (VAE) and their variants [18] and optimization- based approaches
While deep generative models have achieved remarkable success in generating realistic images given random input, the latent variables that they find are often very different from the desired "independent factors" [19]. Returning to our motivating example figure deep generative model may generate perfectly realistic images given random latent vectors where some of the coordinates of z have only a weak relationship to the true latent factors, and no single coordinate encodes a latent factor perfectly, desired.
Figure illustrates this issue. the top of column (c) we show the representation learned by particular autoencoder [5] for the data on the left (see figure for additional baseline autoencoders). The embedding shown as a function of the ground truth factors (21: fur color, 22: cat pose). The autoencoder learned two latent dimensions,
and y/2, and we plot y1 (Z1, 22), (2 22) using the "jet" colormap that goes from blue to red. If each learned dimension encodes only one factor, then these plots should have contours that are parallel to the axes. For this particular autoencoder, however, this is not the case. Even though gives very low reconstruction loss (see supplementary), it can be seen that neither fur color nor pose are directly encoded by either of the Iearned factors. Such representation is called "entangled" [3, and there has been much recent work attempting to quantify the extent to which learned representation is "disentangled" 8 [5]
2
g
2 9-
0
Figure 2: For any generative model, J. we can always apply transformation q to the latent code and add q- the generator obtain an equivalent generative model, ÿ. In the linear case, the code is sufficiently non-Gaussian any linear transformation q will make the distribution more Gaussian and so is possible to differentiate between the true z, and any alternative é, g. In non-linear case, even if we assume non- non-Gaussian prior, there exist many invertible functions & that preserve the distribution. This enables the construction of 9. such that the pair ê, is indistinguishable from the true g.
An intriguing result from the recent literature on disentanglement that the problem impossible without additional assumptions, even with infinite data [19 17). This issue is well-understood in the context of linear functions of latent factors, i.e. Independent Component Analysis [22. In the linear ICA setting, it is known that while the problem is non-identifiable when the latent factors

---

are Gaussian, it can be solved if the latent factors have sufficiently non non- Gaussian distribution. The recent work of [99 [7]. points out that in the nonlinear case, the identifiability problem is much more severe: even if the independent factors are non-Gaussian, there can be many non-linear transformations & = q(()) that preserve the distribution of z. By defining 9(2) = g (()), for g(z) x, we can perfectly generate the data from latent variables that have the desired distribution yet are highly entangled (see figure for a graphical representation of the impossibility result). This suggests that disentanglement is impossible without additional strong assumptions the architecture and the data even with infinite data, and indeed Locatello et al. demonstrated empirically that many deep autoencoder methods fail to learn a disentangled representation.
In this paper we show the power of one very general assumption that can enable the learning of disentangled representations. The assumption of local isometry, i... that an change in the latent representation corresponds to an e change in the observed variables. In the bottom of figure b.c we show scatter plot of the distances ((x", and d(u': .y') for randomly chosen pairs of nearby points. If the embedding satisfies local isometry, these points should lie on straight line. As can be seen, the highly entangled autoencoder embedding does not satisfy local isometry: there is almost no correlation between the magnitude of change in the embedding space Y and the magnitude change in the original input space X.
Specifically, we prove that as the number of datapoints goes to infinity, a combination of two classical methods: Hessian Eigenmaps 771 and fastICA [12], is guaranteed to find a disentangled representation when the manifold parameterization satisfies local isometry and the base distribution is sufficiently non-Gaussian. Figure [ shows the result of our proposed method on the same data that the autoencoder was trained on. The contours in the plots are now axis-aligned, indicating that a perfectly disentangled representation was learned. As shown in the bottom plot, there is strong correlation between the pairwise distances of points in the input space X and their corresponding representations in the embedding space Y.
In the remaining sections, we first derive analytical conditions on manifold parameterizations that guarantee that disentanglement will be possible; and discuss settings in which these conditions hold. We leverage recent advances deep generative models to construct manifolds of highly realistic images for which the ground truth latent representation is known, and test whether different methods succeed in recovering a disentangled representation. For many different manifolds, we find that methods that explicitly optimize local isometry and non-Gaausiannty consistently find the correct latent factors, while baseline deep autoencoders do not. Finally, we discuss ways to encourage deep autoencoders to find encodings that satisfy local isometry, and show that this helps them discover disentangled representations. Overall, our results suggest that in some realistic settings, disentanglement is possible without any domain-specific assumptions.
2 Local Isometry + non-Gaussianity make disentanglement possible
Our main theoretical result shows that a combination of two classical algorithms is guaranteed find disentangled representations under two classical assumptions: local isometry and non- -ausssiiaantt. We first define the assumptions and the algorithms and then state the result.
Local Isometry: Intuitively, a mapping from a latent representation z to an observed vector æ satisfies local isometry if change in corresponds to change in z. Formally, mapping g from is said to satisfy local isometry if for all 2, the Jacobian J(2) ag satisfies:
B2
J())=
As we discuss in section 3 local isometry has been widely used in manifold learning algorithms and any manifold can be approximated to arbitrarily high accuracy using an isometric parameterization.
:on-Gaussianity: Given distribution over p(z), we say that z is sufficiently Gaussinnn,,, if linear ICA algorithm is guaranteed to recover 2 (up to permutation) from a linear mixing Az, given infinite training examples. As mentioned above, this assumption is crucial enable disentanglement in linear generative models.
The two algorithms that we will use HILLE and fastICA 13
Hessian Eigenmaps (HLLE) belongs to family of manifold learning techniques that first estimate a local parameterization around each datapoint and then align the local parameterizations by solving
3

---

Specifically, it is based on the insight that the local isometry assumption means that local PCA will recover a parameterization that is locally isometric to the true parameterization. Given a dataset (x' (x'),= ji=1. 1. an integer K that determines the size of the
an eigenvector problem (e.g. 22 30 28
neighhorhoods, and d the dimensionality of the manifold (or equivalently the dimension of the latent code), the algorithm proceeds in the following steps:
For each point æ", identify its K nearest neighbors using Euclidean distance.
Perform a local principal components analysis (PCA) within each neighborhood and use the principal components in each neighborhood to define a global, . x M, matrix H.
Calculate the d+1 1 eigenvectors of H with smallest eigenvalues. Discard the trivial, constant eigenvector to obtain a d dimensional embedding Y. The embedding of x', which is a vector of length d denoted yy is then computed from the ith index of the eigenvectors.
As shown in 77. given infinite samples from a manifold for which the mapping from satisfies local isometry. the HILEE algorithm is guaranteed to find representation of x that is the same as 2 up to linear transformation: y² = Ae' + b for some matrix A.
The CastICA algorithm [12 searches for projections of the data (u - wf which maximize the deviation between the expectation of a nonlinear function of the projection, G(u), and the same expectation with a Gaussian distribution. It is guaranteed to be consistent "for most reasonable choices of G. and distributions of the sources. In particular, if G(u) u², the condition is fulfilled for any distribution of non-zero zero kurtosis. In that case, it can also be proven that there are no spurious optima" [13.. [.]] In other words, if p(z) has nonzero kurtosis and we observe infinite samples from x Az, then fastICA with G(u) = ut is guaranteed to correctly recover A.
Finally, we refer to HLLE+ICA as an algorithm that first runs HLLE and then runs ICA on the embedding found by HLLE. We are now ready to state our main result.
Definition: A representation y(²) is said to be a disentangled representation of the true latent factors if there exists permutation and pointwise nonlinearity I such that y = (f)))).
Theorem 1:
Let {a"} be samples from a manifold x = g(z). Assume that the support of p(2) is connected. If there exists an alternative representation of the manifold x 92(s) with A2(2). for any matrix A and pointwise nonlinearity f. such that 92 satisfies local isometry and f(z) is sufficiently non-Gaussian; then the number of samples goes infinity, HILE+ICA will provably find a disentangled representation of the true latent factors 2.
Proof: Donoho and Grimes 7 showed that as the number of examples goes to infinity, the H matrix constructed in the HILE algorithm converges to the (continuous) Hessian operator on the manifold whose d +1 1 eigenfunctions with zero eigenvalue are spanned by the latent factors. They also showed that the support of p(=) is connected, then HILLE will recover the isometric embedding of the manifold, up to a linear transformation. In our setting, the results of Donoho and Grimes guarantee that HLLE will recover up to a linear transformation. Since s is assumed to be a lincar mixing of independent factors, the assumption of non- -ou-saassiy means that running fastICA on the output of HLLE will provably recover a disentangled representation.
Figure 3 Fillustrates our theorem for a two dimensional manifold created by translating white squares on black background. The location of each square is sampled from a uniform distribution. The mapping from (the horizontal and vertical position) to (the image of the square) has been shown to satisfy local isometry by [6. Thus, HILEE (left) is guaranteed to find a representaton that is related to the true positions by a linear transformation. However, this is not perfectly disentangled representation just yet. Nevertheless, since the distribution is uniform, not Gaussian, applying ICA on the result of HLLE (middle) indeed recovers perfectly disentangled representation.
Note that the definition of local isometry is sensitive to rescalings: if we parameterize the squares manifold in terms of horizontal and vertical position, then it will satisfy local isometry. However, we can also parameterize the same manifold using log(z), exp(y) and such parameterization does not satisfy local isometry. Nevertheless, Theorem 1 guarantees that we will still recover a disentangled representation, regardless of the rescaling of the true factors. In the supplementary material, we show that another classical manifold learning algorithm, Laplacian EigenMaps (LEM), followed by fastICA, will also recover disentangled representations under certain conditions, but unlike HLLE+ICA, LEM+ICA is very sensitive to rescalings.

---

HLLE
y2
HLLE+ICA
(a) embeddings
(b) shapes-Xx - xy
Figure The results of running HLLE+ICA on 3000 samples from 2D manifold created by translating a white square black background. The results of [6] guarantee that this manifold satisfies local isometry and our theorem guarantees that HLLE followed by ICA will recover a perfectly disentangled representation. As be seen, the algorithm work perfectly even with finite data.
Are the assumptions reasonable?
Our theorem based on two assumptions: non-Gaussiantty and local isometry. While both as- sumptions have been used in machine learning for over 20 years, it is natural to ask whether these assumptions are reasonable in the context of disentanglement.
Regarding non-Gaus non-Gaussianity, this assumption has been successfully used in linear ICA in a wide range of real-world applications including analysis of gene expression data, extracting features from stock market data and removing artifacts from EEG data [14. In particular, linear ICA algorithms can successfully disentangle sources with different distributions and in the presence of outliers.
The history of local isometry in manifold learning goes back to to work of Roweis and Saul 25), who used the assumption that manifolds are locally linear the basis of LLE, and Tenenbaum al. (29) who used the stronger assumption of global isometry as the basis of ISOMAP. Donoho and Grimes 77 showed how interpret [LEE in terms of local isometry and presented HILE more principled alternative to LLE. They also showed that local isometry is a more general assumption than the global one used by ISOMAP, and which covers many interesting articulation manifolds. Taken together, these algorithms have been used in a remarkably wide range of applications.
Manifold learning approaches based local isometry have been shown be successful precisely the type of manifolds that are typically used in the disentanglement literature. Many recent papers variant of it, which consists of simple 2D objects translating and rotating in the image plane. These types of manifolds were shown to satisfy local isometry by Donoho and Grimes (6). An additional example that is very common in the disentanglement literature is images of faces at different poses, illumination and expressions. Donoho and Grimes 6 have shown that a manifold of images of faces with different expressions satisfies local isometry, and such datasets are successfully handled with HLLE.
dsprites (23],
As mentioned above, any manifold can be approximated by one that satisfies local isometry. This follows from the Nash-Kuiper theorem (1956) 27 which shows that any manifold of dimension embedded in n dimensional space (n 2 d + 1), can be approximated to arbitrary accuracy with an isometric parameterization.
Note that this result does not contradict Gauss's Theorema Egregium which states that Gaussian curvature is preserved under isometries and hence, for example, impossible to find a global, exact isometry from a partial sphere to the plane. But this impossiblity is only for global, exact
5

---

isometries. Indeed we have found that running HLLE on data sampled from partial sphere will find an embedding that satisfies local isometry almost perfectly.
As the preceding discussion shows, there always exists an approximate parameterization of a manifold that satisfies local isometry, but for theorem to hold, this parameterization should correspond (up
to linear mixing and rescalings) with the "ground truth" factors that we seek. In the next section we ask whether this holds in realistic image manifolds.
ffhq
cat2
church
kitchen
Figure 4: Realistic 2D image manifolds generated with StyleGAN and GANSpace. Each dataset consists of 3000 samples. The rows correspond to the ž direction and the columns to the 22 direction.
Experiments
We conducted experiments on two simple synthetic manifolds and realistic synthetic manifolds. The two simple manifolds were constructed by squares on black background, where varied the 2D position in the first manifold (shapes-xy) and the diagonal position and the color in the second (shapes-xc). Similar manifolds have been used in many recent papers disentanglement (e.g. [I] 151, and we ensured that the latent representation has a non-Gaussian distribution by sampling the latent variables from independent uniform distributions.
In addition to these simple manifolds, we also leveraged recent advances in deep generative models to create realistic image manifolds where each image is complex, nonlinear function of small number of known latent variables.
Specifically we use GANSpace [10 to find interpretable semantic directions in the W space of StyleGAN2 [6), using PCA. To generate d-dimensional manifold we select d such directions and obtain their corresponding PCA components. We then uniformly sample the coefficients of these components, while fixing the coefficients of the remaining components to a constant (0). For example, for the ffhq manifold, we use the first two PCA components identified by GANSpace, which correspond to head pose and gender, as visible in figure 4 select correspond pose and fur pattern. The following table lists the 8 realistic manifolds that we generated using GANspace and their independent factors.
For the cat2 manifold, the directions
ffhq
cat1
cat:
horse
car
kitchen
bedroom
church
21
gender
pose
pose
pose
viewpoint color
structure
viewpoint
zoom level
22
pose
color
pattern
color
cabinet
illmiination
building material
For each dataset we ran both modern deep autoencoder methods as well as the two classical methods described previously: HLLE+ICA (our proposed method) and LEM+ICA. HLLE has a single free parameter & (the number of neighbors) and we find by choosing the k for which the local isometry score is maximal. The local isometry score is defined as the correlation coefficient between distances in the input space d(x; .²)) and distances in the embedding space d(y y²), when we only consider pairs of nearby points x² _ for which d(x', x²) the pairs of points. Since ((y""r²)) is sensitive to a rescaling of the latent representations, we first compute diagonal rescaling of the representations that minimizes the MSE between the rescaled d(y², y²) and d(xx,, ). For some values of l we found that fastICA did not converge (less than 5% of the runs) and simply ignored the results with that value.
and was set to include the closest 5% among

---

HELIWAA
HAR
LIMSICA
PCA
ARL BELTAIIAA
AFC
DEMAR 12HICA
IRMAE
VAR 125+ICA
VARL
35
0.96499
0.71023
0.50648
Figure 5: The embeddings found for the cat1 manifold shown in figure by various methods. None of the baseline methods manage to learn a disentangled representation.
1.0
0.8
0.6
0.4
0.2
0.0
HAE
HLLE+ICA
PCA
AE
LEM+ICA
AE 1281CAA
IRMAE 1RMAE_128+ICA 128+ICA
IRMAE
VAE, VAE_128+IC 128+IC
VAE
Figure 6: Boxplots for the Mean Correlation Coefficient (MCC) for all methods, over the 8 styleGAN manifolds. The box plots are computed using Matplotli'ss s boxplot command and the box extends from the lower to upper quartile values of the data, with line at the median. The whiskers extend from the box to show the range of the data. Outliers are plotted as separate dots. Our proposed methods succeed in finding disentangled representation for all 8 manifolds, while the baseline autoencoders are no better than PCA.
We also measured the performance of baseline autoencoders the datasets. Since these methods have many free parameters, we do not claim that the results shown here are exhaustive. Our main focus is show that the disentanglement tasks we use here are trivial: simply using an autoencoder with small bottleneck along with standard regularization methods is not sufficient to ensure disentanglement, even if excels alternative tasks (e.g. reconstruction). The autoencoders we use were all taken from the code of recent paper [15] and includes an implementation of "vanilla" autoencoder (AE) with 4 conv+relu layers with stride=2 and one fully-connected layer. The Implicit Rank Bank-Minimizing Autoencoder (IRMAE) includes in addition a number of fully connected linear layers and this has been shown to implicitly regularize the solution towards low rank embedding. We also tested basic implementation of a 3-VAE with the value of 3 that was found [15) to provide the best performance. Results for all auto-encoders are for 100 iterations (as shown in the supplementary material, the reconstruction quality continues to improve with more iterations but the embedding is largely unchanged). Since deep autoencoders sometimes fail to learn good in
representations with very narrow bottlenecks, we trained two versions of all autoencoders: which the bottleneck was of width 2 (the true dimensionality of the manifold), and another with a bottleneck of width 128. In the second case, we ran fastICA to find two dimensional projection of the 128 dimensional latent vector that is as Gaussaansi as possible. We also provide the results of a PCA encoder baseline.
Qualitative results are shown 3 the cat1 manifold. None of the baseline autoencoder methods managed to find disentangled representation, while HLLE+ICA and LEM+ICA did find one. Evidently the VAE suffered from "posterior collapse", i.e. "when the variational distribution closely matches the uninformative prior" [1].

---

As quantitative performance metric we used the Mean Correlation Coefficient (MCC) which is standard in the nonlinear ICA community [77. To compute it, we first solve the problem permuting the components of the embedding y(*) so it best matches the latent factors 2(r), using the Hungarian algorithm. We then measure the Pearson correlation between the permuted embedding and the latent variables. The results are summarized in figure our proposed method consistently found disentangled embedding, while baseline autoencoders did not.
As expected, run-times of the spectral methods were much faster than the autoencoders. With k = 30 neighbors, running HLLE took about 6 seconds per manifold and ICA took less than second, while training the autoencoder took over an hour.
4.1 Hessian Auto-Encoder (HAE)
We saw that our proposed HLLE+ICA algorithm continuously succeeds in finding disentangling representations, while the autoencoders fail to so. However, these autoencoders offer a built- in decoder and also enable out-of-sample extensions. These are two valuable advantages that HLLE+ICA lacks. In this section we propose way to achieve both, by encouraging autoencoders to learn representations which satisfy local isometry.
In order to do so, we use a "student-teacher paradigm. We first run HLLE+ICA and then train the autoencoder to minimize the following loss:
L(O1, (0;,002;,11)
Σ [1I((a:0)) w'11 -
[ID(w':2)
x²11
where 0 are the parameters of the encoder E(x), 02 are the parameters of the decoder D(y) and y is the embedding calculated by HLLE+ICA. This essentially decouples the learning of the encoder and the decoder. We use exactly the same encoder and decoder architectures as the baseline methods. As can be seen in the embedding visualizations in figures 7 this method, which we call the Hessian Autoencoder (HAE), gives near perfect disentanglement on all manifolds, as opposed to the baseline autoencoders which often find highly entangled embeddings. This entanglement can also be in the interpolations in the figures. The baseline autoencoders often jump abruptly across the latent space, while HAE transitions smoothly along the manifold, gradually moving in both dimensions, as desired.
Whereas the vanilla autoencoders only strive for good reconstruction quality, the HAE also aims to learn its HLLE+ICA teacher's s embeddings. As a result of its more complex loss, typically requires more iterations achieve the same reconstruction quality the vanilla AE. However, with more iterations it is indeed able to achieve the same high quality results as its vanilla AE counterpart. The results in figure show reconstructions with 1000 iterations for HAE and 100 iterations for the vanilla AE. A more detailed comparison can be found in the supplementary material.
5 Related Work
Our work was inspired by the recent work of Locatello al. [19] who point that "unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data". They conducted extensive empirical experiments on "more than 12000 models covering most prominent methods and evaluation metrics in reproducible large-scale experimental study seven different data sets" and observed "that while the different methods successfully enforce properties encouraged by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision".
We agree with the fundamental need for inductive biases, but our work suggests that a relatively general inductive bias (local isometry) makes disentanglement possible without supervision.
Khemakhem et al. I7 extend the impossibility results and relate them to classical results on identifiability of nonlinear ICA. They state that "models with any form of unconditional prior p(z) are unidentifiable" and argue for using conditional priors (where is conditioned on a third variable such as time). Our work again acknowledges the non-identifiability issue without inductive bias, but shows that a relatively general inductive bias allows us estimate the correct latent factors even with an unconditional prior.
8

---

HAB
AE
Figure Interpolations in the embedding space of HAE and the vanilla autoencoder, around chosen point (upper left image of each grid). The rows correspond to an change in the first dimension and the columns in the second. For each manifold and model, the corresponding embeddings are visualized to the right of the interpolation. On both manifolds, HAE's two directions perfectly match the true underlying factors, whereas the AE's do not. As can be seen, entangled embeddings result in discontinuous interpolations.
Atzmon et al. 2 suggested using local isometry as a regularizer in deep autoencoders and compared this regularizer other regularizers in the context of visualization of high dimensional data. Our focus, on the other hand, has been to show that local isometry together with non- -on-Gaussianity is sufficient guarantee disentanglement.
Mathieu et al. 222 have argued that the stochastic encoder used by VAE may provide an inductive bias that enables disentanglement. Our experimental results, well those of others (20 13). suggest that in many cases this inductive bias is not sufficiently strong to guarantee disentanglement, and our theoretical results show that local isometry is sufficiently strong.
There has been much work on performance evaluation methods for disentanglement used the simple MCC performance measure, and verified using visual inspection that it captures the success or failure of disentanglement. A second major difference between our work and most existing papers evaluating disentanglement is the use of more realistic, complex dataset that define using StyleGAN. While we have also shown results on simple 2D shapes on constant background, we believe the StyleGAN datasets are much more realistic and capture much wider range of latent factors that affect images.
11
Here we
The fact that LEM is highly sensitive to the relative scaling of different dimensions was previously pointed out by Singer and Coifman 26 and by (24) Both of them suggested methods for improving LEM: either by pre-computing a local covariance at each point and using it do define affinity or by post-processing the L.E output. On our datasets, HI.LE consistently outperformed LEM but we believe improvements to LEM can be of great interest given theoretical results.

---

AE
HAB
IRMAR
AE
HAE
IRMAF
AE
HAE
IRMAE
AE
HAE
IRMAE
Figure 8: Interpolations in the embedding space of the different autoencoders generated by linearly interpolating between the encodings of the left and right image and then decoding. HAE transitions smoothly along the manifold between the two endpoints, while the baseline autoencoders often fail to do so. For each manifold and model, the corresponding embeddings are visualized to the right of the interpolation sequence. As can be seen, entangled embeddings result in discontinuous interpolations. All models were run for 100 iterations (see supplementary material for discussion on the reconstruction quality).
Conclusions and Limitations
The main question addressed in our work is whether disentanglement is possible. Our theoretical and empirical results show that when we make two modest assumptions: (1) local isometry and (2) non- Non-Gaussianity. the answer is "yes"; and combination of two classical methods, HILE and fastICA, can successfully learn disentangled representations less than minute. Our work does not contradict the recent impossiblity theorems on disentanglement which argue that the task is impossible without additional assumptions, but it shows that two very general assumptions suffice with need for Nomain--pecifii assumptions. Moreover, in set of experiments with highly complex realistic image manifolds, the assumptions appear to hold and disentanglement possible with a simple, well- well-understood method.
While our work has shown that disentanglement is possible, we do not mean to suggest that the problem has been solved by running HI.LE followed by fastICA. A major limitation of our theoretical result is that we focus on the infinite data limit and similarly the datasets we constructed correspond to very dense sampling of the manifold (3000 images for 2D manifold). When the sampling sparser, is known that spectral methods can fooled by sampling noise. Furthermore, running spectral methods on large datasets (e.g. millions of images) is challenging and may require special-purpose approximations. We hope that our "possibility results" " will encourage more research that will allow similarly successful disentanglement in broader range of datasets.
10

---

Acknowledgements
Support from the Israeli Science Foundation, Israeli Ministry of Science and Technology and the Gatsby Foundation is gratefully acknowledged.
References
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. 1rX1v:/701.0700 2017.
Wasserstein gan,
arXiv preprint
[2] Matan Atzmon, Amos Gropp, and Yaron Lipman. Isometric autoencoders. obs/2006.092289, 2020.
CoRR,
[3] Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-122 2013.
[4] Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. CoRR, abs/ 7177777777776 2017.
[5] Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representa- tions. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[6] David Donoho and Carrie Grimes. Image manifolds which are isometric to euclidean space. Journal of Mathematical Imaging and Vision, page 5-24, 2005.
[7] David L. Donoho and Carrie Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10):5591- 5596, 2003.
[8] Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018.
[9] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661. 2014.
[10] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable GAN controls. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[11] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick. Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a con- strained variational framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[12] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural Nenworks, 13(4-5):411-44 1-430, 2000.
[13] Aapo Hyvärinen. Fast and robust fxed-poiim algorithms for independent component analysis. IEEE TRANS. NEURAL NETW, (0(3):6266634, 1999.
[14] Aapo Hyvärinen, Juha Karhunen, and Erkki Oja. Independent Component Analysis. Wiley, 2001.
[15] Li Jing, Jure Zbontar, and Yann LeCun. Implicit rank-minimizing autoencoder. Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.

---

[16] Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 8107-8116. IEEE, 2020.
[17] Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]. volume 108 of Proceedings of Machine Learning Research, pages 2207-2217. PMLR, 2020.
[18] Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. Found. Trends Mach. Learn., 12(4):307-392, 2019.
[19] Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. CoRR, abs/1811 11t./811.//0 12359, 2018
[20] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look at the unsupervised learning of disentangled representations and their evaluation. J. Mach. Learn. Res., 21:209:1-209::2 2020.
[21] James Lucas, George Tucker, Roger B. Grosse, and Mohammad Norouzi. Understanding posterior collapse in generative latent variable models. In Deep Generative Models for Highly Structured Data, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019. OpenReview.net, 2019.
[22] Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Teh. Disentangling disentangle- ment in variational autoencoders, 2019.
[23] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- ment testing sprites dataset. http://ziiiilcccmmmmemmmmmttttts 2017.
[24] David Pfau and Christopher P. Burgess. Minimally redundant laplacian eigenmaps. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018
[25] Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 29065550))2333-2336, 2000.
[26] Amit Singer and Ronald R. Coifman. Non-linear independent component analysis with diffusion maps. Applied and Computational Harmonic Analysis, 25(2):226 239, 2008.
[27] Arne Sletsjø. The nash-kuiper theorem, 2015.
[28] Yee W Teh and Sam T Roweis. Automatic alignment of local representations. In Advances in neural information processing systems, pages 865-872 Citeseer, 2003.
[29] JB Tenenbaum, V De Silva, and JC Langford. A global geometric framework for nonlinear dimensionality reduction. Science, pages 2319-2323, 2000.
[30] Tianhao Zhang, Jie Yang, Deli Zhao, and Xinliang Ge. Linear local tangent space alignment and application to face recognition. Neurocomputing, 7077-9):1547-155, 2007.
12


Reviews:
[{"review_id": "xKMSyE-ffxT", "reviewer": "Reviewer_R2z6", "summary": "The paper follows the recent question of when disentanglement is possible and shows that 'local isometry' is sufficient to ensure that it is possible (not guaranteed) to learn a disentangled representation. This result largely follows from know results in the ICA community alongside the isometry assumption. A proof-of-concepts algorithm that combines HLLE with ICA demonstrates feasibility on synthetic data constructed to satisfy the isometry assumption. While the paper is largely correct, I would argue that it is uninteresting as it (as far as I can tell) only applies to flat manifolds, which is the case we already understand well through linear models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper considers disentanglement of models where data $x$ is approximated as $x = g(z)$. Here $z$ is a low-dimensional (latent) variable and $g$ is a smooth function. The guiding assumption is that $J^T J = I$, where $J$ is the Jacobian of $g$ and $I$ is the identity matrix. This is assumed true for all $z$. Based on this assumption, the paper shows that classic identifiability results from (Euclidean) ICA holds, such that we can recover the true generative factors (i.e. we can disentangle).\n\nThis may sound exciting, but I remain unconvinced. I may have misread the paper, so I am happy to revise my view if the authors can provide more convincing arguments than those put forward in the paper.\n\nHere's my issue:\nThe matrix $M = J^T J$ ($M$ is a short-hand I here introduce) is the metric tensor of the pull-back metric associated with $g$. If $M = I$ everywhere, then the manifold spanned by $g$ must have zero curvature everywhere, i.e. it must be flat. This leads to two comments from my side:\n\n1. How reasonable is it that actual data manifolds are flat? I would argue that this assumption is so strong that it effectively is uninteresting. Consider data lying on a sphere; even this simple setting is too restrictive for the assumption to work out.\n\n2. In this light, the result that we can disentangle seems rather obvious given that we know that ICA can disentangle linear models. The extension from 'linear' to 'flat' then seem rather incremental. There is still value in the extension, but given that the paper signals that the extension is more significant, I get rather skeptical.\n\nOn possible counter-argument to the above is given by the following statement from the paper (lines 97-98):\n\n\"\nIndeed, assuming that the dimension of z is much smaller than the dimension of x, then any manifold\ncan be approximated by an isometric mapping. [2].\n\"\n\nThis is, however, all too vague to give this reader much comfort. Clearly any manifold can be approximated by a flat manifold; but can it be well-approximated? Gauss's Theorema Egregium suggest that we perhaps should not be approximating curved objects with flat ones.\n\nOn a more positive note, I appreciate that the paper has the relevant proofs as part of the paper rather than hidden in an appendix.  This is great.\n\nI miss a citation to the paper\n\n  Disentangling Disentanglement in Variational Autoencoders\n  Emile Mathieu, Tom Rainforth, N. Siddharth, Yee Whye Teh\n\nThis gives a partial counter-argument to the Locatello paper, namely that the applied variational inference provides an inductive bias that helps make the models less susceptible to identifiability issues.\n\n# After rebuttal and discussion\n\nI appreciate the rebuttal and follow-up discussions which clarified many things. Based on these I will increase my score to recommend acceptance. **However**, this increase in the score rests on the assumption that the discussions points will be incorporated into the final paper. I think a significant rewrite of several sections of the paper are needed (see our discussions for details) for the paper to be publishable. I trust the authors to implement such changes as it is in their own best interest to do so.\n\nI have increased my score to 6, which I feel is as high as I can go given the state of the submitted manuscript. Had I reviewed a version with the changes implemented, I might have gone higher.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion of societal impact, but this can be forgiven considering the theoretical/conceptual nature of the work.\n\nI do, however, think that the limitations are not sufficiently discussed. As far as I can tell, the assumptions imply that only flat manifolds can be considered, which render the work largely uninteresting (see above).\n\nThe final paragraph of the paper has a nice discussion of the limitations of the proposed algorithm and its reliance on HLLE. That was great!", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "_uVYN1ydwfi", "reviewer": "Reviewer_tdhX", "summary": "The authors propose a novel method to learn disentangled representations from data. Their approach uses some inductive biases on the transformation between the latent and observed spaces, as well as on the latent distribution. Namely, they assume the transformation satisfies the assumption of the local isometry, and that the latent factors are (non-jointly) non-gaussian.\nThe disentangled representations are learned by first applying a spectral transform to the data, followed by a linear ICA unmixing. \nTheir method is backed by a theoretical analysis, as well as an empirical validation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a novel method for learning disentangled representations, which is an important topic in unsupervised learning.\nThe proposed approach is based on a spectral decomposition followed by a linear ICA step, instead of learning a full generative model.\nThis gives it a considerable edge in run time (albeit this was only explored in 2-d), but takes away the generative capabilities. This is somehow addressed by the Hessian Auto-Encoder proposed in section 4.1, although it's not clear how it compares to state of the art generative models like VAEs and GANs (which have seen variants developed for disentangled representations).\n\nOverall, the paper is well presented and easy to read. The proposed method is novel, and, as far as I know, hasn't been proposed before. \n\nThis being said, I have 2 serious concerns:\n1- The dimension of the latent space in the experiments was $d = 2$. The authors didn't report the performance of their approach for a higher latent dimension, which I find alarming. From personal experience, disentangled/identifiable representations are harder to find in higher dimensions.\nIn addition, their approach has much shorter run time, but the experiments were only conducted for $d = 2$. Does this still hold for a much larger latent dimension, say $d = 500$?\n\n2- Figure 6 would be much more informative if the authors used a box plot of violin plot. More generally, error bars are missing from all figures and numbers, and there is no indication that the MCC figures reported were averaged over multiple runs and not for a specific seed.\n\nAs well as some other comments:\n3- It would benefit the presentation if the spectral methods were better explained in the paper, and not simply relegated to the appendix, as they are core to the proposed method. The same can be done for fastICA, without going into the detail of the algorithm, but just explaining what ICA does for e.g.\n\n4- Did the authors explore the impact of applying pointwise nonlinearities to the latent space of the datasets and see how it impacts performance? This would give experimental validation to Theorem 2.\n\n5- The authors should specify that the \"dimension of the manifold\" is equal to that of the latent space.\n\n6- Terrible notations: it seems that the indices are used to both designate the index of an observation in a dataset (e.g. line 25 $\\{x_i\\}$) and the component of a multivariate r.v. (e.g. line 39 $y_1(z_1, z_2)$). This is very confusing when referring to the distances between points (e.g. line 66 and 205, $d(x_i, x_j)$).\n\nOverall,  I think that the theoretical results of this paper are worthwhile, but the experimental section lacks depth and polish. I will give a score of 5, but will be happy to increase it if the other reviewers or the authors' rebuttal convinces me to do so. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the proposed method, as well as how it relates to prior work, were well discussed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7.5 hours"}, "extra_text": {}}, {"review_id": "5P8DqejvBbj", "reviewer": "Reviewer_pb6M", "summary": "This paper studies the disentanglement of samples generated by GANs. It argues that the local isometry and non-Gaussian property are the keys to disentanglement. Some intuitive experiments based on GAN latent space discovery are conducted to support this insight.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1. The writing is clear and easy to follow.\n2. The review of the traditional manifold learning method is helpful for current GAN research.\n\nConcerns:\n1. The two theorems in this paper are a bit tricky. First, a local isometry is a very strong condition, which basically cannot happen in practice. Second, these two theorems do not explain why intuitively disentanglement is connected with isometry, and what is the role of non-Gaussian property? Third, is there any in-depth analysis on the disentangled representations? For example, what is the convergence rate to the disentangled representations with respect to the data volume? Are there any more efficient algorithms? Is isometry or non-Gaussian necessary in disentanglement?\n2. I must correct a logical mistake in this paper. That \"you can get disentangled representations at local isometry and non-Gaussian condition\" does not mean \"you cannot get disentangled representations otherwise.\" So the content of this paper has presented cannot answer the question 'is disentanglement possible.'\n3. The experiments are too weak. Why is GANSpace sufficient to support your claim? Any insights or theory?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper mentioned the limitations in Sec.6.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "4Z9viDPpQA", "reviewer": "Reviewer_nrJU", "summary": "The paper shows that having local isometry together with non-Gaussianity of latent variables and sufficient number of samples guaranteed to learn a disentangled representation, without any domain-specific knowledge. The authors discussed some potential manifolds for which local isometry is satisfied. The authors also conducted several experiments to evaluate the disentanglement of the obtained representation by their method in contrast to the comparable approaches. The authors demonstrated the latent embedding and the mean correlation coefficient as qualitative and quantitative disentanglement measures, respectively.  \n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe paper is addressing a challenging and important problem. Although the local isometry property has been studied before, the authors provide a new theoretical analysis for the disentangled representation which is quite valuable and expands the finding of the previous disentanglement studies by Locatello et al., 2018, 2020. They show that in the presence of enough training samples, satisfying two conditions: isometry and non-Gaussianity on the latent space is enough to obtain disentangled factors. They also discuss the sensitivity of the isometry to the parametrization of each factor.\n\nQuality: \nThe paper is technically sound with enough theory and experimental studies. Theorem 1 shows if there exist an isometric mapping from latent space to the data space, using Hessian eigenmaps and ICA, the latent representation is disentangled. Theorem 2 generalizes the result of the first theorem by considering another mapping from latent space to an alternative space, which relaxes the local isometry constraint on the latent factors.\n\nClarity: \nThe paper is well written and well organized.  \n\nSignificance: \nBoth theoretical and experimental results seem valuable. Experimental results show that both HILLE+ICA and HAE obtain significantly better MCC compared to the prior methods. The results show that the HILLE+ICA and HAE have roughly the same reconstruction, generation, and interpolation performance compared to the vanilla AE.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- As discussed by the authors, there are some practical limitations for using the HILLE+ICA in representation learning. ICA/fastICA suffers from convergence issue and it is not easy to substitute ICA with another method to satisfy non-Gaussianity.  \n\n- According to Theorem 1, latent dimensions are assumed to be uniform, which is a strong assumption in many domains of application. For instance, in the gene expression dataset which is also mentioned in section 3, such uniformity is not present. Can the authors explain how the theoretical results are satisfied for dataset with heterogeneity and rare observations?\n\n- It seems that both HILLE and HAE suffer from poorer reconstruction compared to AE. Is this due to the objective in Eq.1 ?\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines the hotly debated topic of whether one can learn disentangled representations from unlabelled data.  The key contribution is to show that a combination of non-Gaussianity and local isometry is sufficient to make unsupervised disentanglement theoretically possible.  This is then backed up by a proof-of-concept algorithm and experiments.

After receiving initially quite mixed review scores, this paper has undergone substantial discussions by the reviewers, both publically with the authors and behind closed doors, resulting in two reviewers who initially recommend rejection increasing their scores to back marginal acceptance.  Perhaps the key underlying issue that has been debated is how strong the assumptions required by the theoretical results actually are in practice.  After much back and forth, the general consensus on this seems to be that the assumptions are actually not unreasonable, but need to be much more carefully explained and discussed in the paper.

More generally, the overall sentiment of the reviewers, which I concur with myself, is that the underlying ideas in the paper are interesting and potentially quite significant, but that noticeable work on the writing of the paper itself is needed to improve the clarity and ensure the work can be properly appreciated by its audience.  Though, at the time of writing, Reviewer pb6M still also has concerns about the relationship between the theory and experiments in the paper and which mapping the local isometry assumption applies to, I agree with the authors that this concern seems to be based on a misunderstanding rather than a genuine issue.

Putting this together, I believe that the decision of acceptance comes mostly down to whether the clarity issues are deemed too severe for publication / require an unacceptably large amount of changes for the camera-ready.  I think this is quite a tight call, but recommend giving the author's the benefit of the doubt that the required changes will be adequately incorporated.  The basis for this is that I think that the work does have the potential to stimulate notable interest in an area that is quickly moving, such that the risk of the clarity issues not being solved is outweighed by the potential advantages of ensuring the ideas are published.

If this decision is upheld, I sincerely hope that the authors will make the required efforts to improve the paper for the camera-ready as the changes required are non-trivial and it would be a shame to leave the writing of the paper letting down the underlying technical content.  I would also very strongly suggest the authors change the title of the paper, which is inappropriately general and does a disservice to previous work that has considered the same question.
Decision: Accept (Poster)