Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)

Abstract: A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex generative process. Recent breakthrough work by Khemakhem et al. (2019) on nonlinear ICA has answered this question for a broad class of conditional generative processes. We extend this important result in a direction relevant for application to real-world data. First, we generalize the theory to the case of unknown intrinsic problem dimension and prove that in some special (but not very restrictive) cases, informative latent variables will be automatically separated from noise by an estimating model. Furthermore, the recovered informative latent variables will be in one-to-one correspondence with the true latent variables of the generating process, up to a trivial component-wise transformation. Second, we introduce a modification of the RealNVP invertible neural network architecture (Dinh et al. (2016)) which is particularly suitable for this type of problem: the General Incompressible-flow Network (GIN). Experiments on artificial data and EMNIST demonstrate that theoretical predictions are indeed verified in practice. In particular, we provide a detailed set of exactly 22 informative latent variables extracted from EMNIST.

Reviews and author responses:

I have read the comments and the appendix is certainly improved. 

The new added section 4.4.3 admits that selection of $u$ is not clear. I think this section and the subsequent one could also be merged into the conclusions, as a limitation and subject for future work. 

I realize that my final comment was incomplete and should have read 
"Arguably, the key contribution of the current paper is in 3.1. and 3.2, but these are rather hastily written and quite short. Overall, the balance of known results and new contributions should be better balanced and highlighted."

but I feel this is also addressed rafter reading the other replies.

I am now more confident that the paper would be a valuable contribution to ICLR.


Thank you very much for your thorough rebuttal and for addressing many of my issues.

1. The new section is great and very clear. But I would somehow have expected to see it in the Theory section, perhaps at the end?

3. I see thanks. I was more looking for a Beta-VAE baseline which would fail to find the two-dimensional manifold embedded into the ten dimensions. But this feels out of scope for now, so this could be left to further work.

4. Great addition thanks, and Figure 15 in the Appendix is a great demonstration of the effects. I guess it'd be even better if the spectrum itself would be sufficient (i.e. without resorting to sampling from the model to check), but this is promising.

5. Yes I saw Figure 6/7, but I was looking for something more drastic with stronger gaps in the Ground truth supports (and where increasing data points would not be sufficient. I'm aware this is a completely different setting which departs from your support assumptions however...). But now looking at Figure 7 more carefully, the reconstructed space does indeed look rather different from the ground truth, so the effect is already quite strongly there, hence I see your point.

Thank you for your thoughtful comments. We address each one in turn:

1. We have added a section (4.4.3) explaining our justification for the use of the digit labels for EMNIST and point out that other data sets may not have such an obvious candidate for u.

2. We agree that there is no canonical set of variables for EMNIST. For this reason, we allow the reader to see all 22 of the variables our model discovered in Appendix F, figures 10 to 15, and trust that they find them convincing enough to agree that they are at least approximately correct.

3. We do not expect that a VAE would perform worse on the experiments on toy data. The result we wish to emphasise is that the model discovers the two-dimensional manifold embedded in ten dimensions. Our justifications for using invertible networks rather than VAEs are given in the Introduction (final paragraph before summary, starting with “We introduce a variant...).

4. We have amended the main text to make this justification clear. See Section 4.4.2, paragraph starting with “The model encodes information…”

5. We performed exactly such an experiment where a ‘gap’ is present when less data is available but becomes filled as the number of data points is increased. It can be found in Appendix E, figures 6 and 7. 

6. We have modified the captions in Figure 4 as suggested.

7. Thank you for bringing these works to our attention. We agree that they are relevant and have included them in the related work section. 

Thank you for your thoughtful comments.

You are correct that the latent dimension of the model must be chosen at least as large as the dimension of the true generating latent space. This is one of the reasons why we prefer invertible neural networks / normalizing flows over VAEs, as in such models, the dimension of the latent space is necessarily the same as the dimension of the data, in order to preserve bijectivity. Therefore, we never choose the dimension of the model’s latent space, so in this context, an experiment where the latent space is too small is not possible. 

Regarding the magnitude of the standard deviations of noise dimensions: we have added a section (3.3) explaining the role of volume-preservation in our model. Since the EMNIST data set is relatively noise-free, noise does not play a large role in explaining the data, hence we expect it to also play a small role in the derived latent space, quantified by its standard deviation. With more noisy data we might observe non-informative dimensions which have a similar standard deviation to the informative dimensions.

Thank you for your thoughtful comments. 

We agree that the paper as submitted was not sufficiently self-contained. We have addressed this by reproducing the necessary proofs from Khemakhem et al. (2019) in the appendices, alongside our extension of them, so that our article can now be read as a standalone piece.

Regarding selection of the variable u (side information): the major requirement that u must fulfill is that it must condition the distribution of the latent space variables z, as detailed in equation (1). We have added a section (4.4.3) explaining our justification for the use of the digit labels for EMNIST and point out that other data sets may not have such an obvious candidate for u.

We have removed the canonical parameters of a multivariate Gaussian (formerly equations (2) to (4)).

Thank you for bringing to our attention our omission of SVAE. We have added it to the related work section.


This paper builds upon the recent theoretical framework on nonlinear ICA, put forward in recent work Khemakhem et al. (2019) that draw a lot of attention. The latter work provides an extension of the basic nonlinear ICA that is closely related to a VAE with a conditional factorized prior, essentially introducing side information (with assumed extra observables u) to resolve non-identifiability issues. 

This paper proposes an invertible architecture related to RealNVP and NICE, coined as  GIN: the General Incompressible-flow network. 

On the positive side,

A key feature of the proposed methodology is model selection (such as determining the model order) that is in general a hard problem even in linear latent variable models. 
The performance is illustrated on the EMNIST dataset, as well as carefully constructed synthetic experiments. The semantic descriptions of each captured dimension, as detailed in the appendix, is particularly interesting. 
The experimental section is quite extensive. 

On the negative side, 

The paper is largely based on the results of a recent technical report (Khemakhem et al. (2019)) and is not self contained, hence rather hard to digest. Even the proofs in the appendix and the intuition requires the reading of this longer technical report. 

It is also not at all clear where the side information (variables u) is coming from. On EMNIST a natural candidate is the digit labels (and this turns out to be is indeed the case in the experimental section) but it is not very clear what conditions need to be satisfied. What prevents us selecting simply a subset of observations x as u? 

Lacking an explicit motivation, the exercise of writing the canonical parameters of a multivariate Gaussian in (2) - (4) is not very informative. This needs to be better motivated.

The prior structure resembles quite closely the general hierarchical priors SVAE proposed in https://arxiv.org/abs/1603.06277. It would be also informative to discuss the links with this approach.

Arguably, the key contribution of the current paper is in 3.1. and 3.2, but these are rather hastily written and quite short. Overall, the balance of known results and new contributions 


Based on a recent work on identifying the joint distribution over observed and latent variables, the paper presents an extension of it, where the dimension of the latent space is not assumed to be known. Moreover, the authors propose a new neural network architecture based on previous propositions (RealNVP and NICE) to perform this identification.

The paper's presentation is relatively clear, although all the theoretical results are relegated to the appendix. The extension to unknown latent space dimension seems to be quite straightforward, given the recent work that this paper is based on. However, the experimental results performed on EMNIST are quite convincing and the results are interesting. 
Overall, I think this paper would be interesting to the ICLR audience.

It seems currently that there is a need to choose a dimension to be large enough (i.e., larger than the true dimension of the latent space). I would have appreciated some discussion (and some experiments) if this dimension is chosen too small. 
Also, when the dimension is chosen large enough, the non-informative learned latent variables are determined by looking at their standard deviations. Do they always have to be small compared to the informative latent variables?

This paper extends recent work by Khemakhem et al on nonlinear ICA to allow for unknown number of generative factors. This is tackling an important problem in the field of generative modeling, where one would like to extract the generative factors of a dataset that independently control its features (i.e. disentanglement).

The paper is very clearly written, does a great job at motivating the problem and presenting the recent results from Khemakhem et al, before extending them with some simple theorems and demonstrating their application on toy data + EMNIST.
I think that this research direction is extremely promising, and obtaining a theoretical understanding of when/why disentangling could work would be particularly valuable to the field, and I would lean towards acceptance so that this work gets more attention.

It is slightly surprising however that their empirical results seem to indicate that these theoretical conditions do not seem to be necessary, which should be investigated further (and might help illustrate the dichotomy in some claims and results obtained in the disentangling literature recently).

1.	It was unclear to me how one should/would decide what to use for $u$ or what to leave out to be factorized by the method. 
	This may be out of scope for this current work, but one way to answer that would be to leverage datasets with more fully labelled factorised data (e.g. dSprites [1]) and present how one should leverage these with $u$, in order to identify the original generative factors.
2.	Related, using EMNIST was interesting, but given the lack of “accepted” generative factors to be recovered, it is hard to tell if the 22 variables found are “correct” or more similar to using a generative model which would entangled the data more (and hence would falsely introduce extra latent variables).
3.	The toy dataset, with its random RealNVP network to produce the data, was less “mixed” than I expected, looking at Figure 2B. In its projected view, the clusters are still rather easy to identify by eye, which surprised me somehow? Could you comment a bit more on how difficult is the task, or present a VAE baseline that would fail to explain that data?
4.	I did not see how the threshold for selecting 22 latent variables on EMNIST was set? Was the 23rd latent variable significantly less informative? The spectrum on Figure 3A was not precise enough to assess this fairly and the single mention of “measured by the standard deviations of test data transformed to the latent space” was too vague to reverse-engineer the decision.
5.	It was interesting to read about the observations of when this method should fail. It would be interesting if a dataset with explicit “gaps” would be constructed to analyze this case.
6.	Figure 4 might benefit from mentioning “what” each variable controls for directly in their individual caption / on top of them, instead of having to read this through in the full figure caption.
7.	The current model in the end is modeling the latent state using a mixture of gaussians (although these now have a theoretical connection to the true generative factors). How much does this differ to existing generative models using VAEs with a mixture of gaussian prior [2, 3]?


References:
[1] dSprites dataset: https://github.com/deepmind/dsprites-dataset 
[2] https://arxiv.org/abs/1611.02648
[3] https://arxiv.org/abs/1902.03717


Meta-review: This paper builds on the recent theoretical work by Khemakhem et al. (2019) to propose a novel flow-based method for performing non-linear ICA. The paper is well written, includes theoretical justifications for the proposed approach and convincing experimental results. Many of the initial minor concerns raised by the reviewers were addressed during the discussion stage, and all of the reviewers agree that this paper is an important contribution to the field and hence should be accepted. Hence, I am happy to recommend the acceptance of this paper as an oral. 

Decision: accept-spotlight