Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for nonlinear ICA with that restriction that the mixing function is a volume preserving transformation. The authors prove that given this restriction as well as a few other conditions the sources are identifiable. The model set up is very similar to the one used in Sorrenson et al.'s work "Disentanglement by nonlinear ica with general incompressible-flow networks (gin)": A real NVP model is made volume preserving by a slight modification of the flow map and this flow is used to learn the mixing function. The sources are assumed conditional independent given an auxiliary variable $u$, which are the labels of classes in their experiments. The mixing function is learned by maximizing the Gaussian likelihood of the reconstruction sources, which is also conditionally independent given $u$.

The authors demonstrate empirically that some of the conditions of their identifiability proof might not be necessary because the sources can still be reconstructed empirically although the condition is not satisfied. Hence, identifiability might even hold true for an even larger class of problems. They also compare their method to iVAE and show that their framework can identify the true sources of generated data much better than this baseline. Furthermore, they apply the framework to MNIST and can identify a few important source variables explaining most of the variance of the dataset.
main_review:  The framework is mainly reused from Sorrenson et al.'s work. The authors simplify the loss derived by Sorrenson et al.; however, they do not make clear why this is necessary or desirable. Their main contribution are two novel identifiability theorems, which state that under a few more conditions in addition to the volume preserving mixing function the true sources can be identified. Empirically, not all of the conditions seem to be necessary. Even when a non-volume-preserving map is used, the true sources of synthetically generated data can be reconstructed in their ablation study in section 6.2.1. This ablation study raises the question whether the formulation of this framework for nonlinear ICA is to rigid, i.e. model used is too restricted.

The method outperforms iVAE, which is an important baseline in the field.

As a minor command, I want to add that there is a typo in the first sentence of section 6.2: it should be "variables", not "variabls".
summary_of_the_review: I appreciate the two novel identifiability theorems. However, the ablation study  in the experiments section lets the conditions appear to restrictive. Furthermore, it is not clear what distinguishes this work from Sorrenson et al.'s article apart form the identifiability theorems. Therefore, I tend towards rejecting this article but I am open to enter a discussion with the authors and the other reviewers.

Edit:
Given the helpful clarification for the authors to distinguish their work from Sorrenson et al.'s article I will raise my score by one.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: [Sorrenson et al 2020] proposed to do nonlinear ICA via volume preserving mixing functions, when a variable u is observed that makes the latent variables conditionally independent. Via maximum likelihood optimization, the ICA model uses a volume-preserving encoder and then maximises a latent distribution, which is a factorized Gaussian conditional on the observed variable u.

The submitted paper does a theoretical identifiability analysis on that procedure. It assumes that the data is generated by an independent distribution, given observed u, then mixed by a volume-preserving mixing function. Furthermore, it assumes that both the mixer and the encoders are twice differentiable, there exist two observed u variables, u^1 and u^2 such that that the two resulting distributions of encoded latent variables have the same mean and that the set of ratios of the two standard deviations for each variable is distinct. If these assumptions hold, the paper proves that the original sources are identifiable up to a permutation and dimension-wise diffeomorphism.

The authors do three experiments: 2D mixture of Gaussians, triangle images and MNIST. For the first two, the authors quantitatively estimate the source identification and for all datasets, qualitative results on identification are given.
main_review: Strengths:
-	The authors provide theoretical support for the use of the pre-existing GIN model for ICA. They propose novel assumptions and prove this leads to identifiability.
-	The paper is very clearly written.

Weaknesses:
-	The authors claim that a volume-preserving mixing function is a natural restriction and is easily satisfied. I would like to see a stronger argument why this is true, as it seems easy to think of non-volume-preserving mixing functions. Such an argument should include why the triangle dataset and MNIST would be generated by volume-preserving mixing functions. 
-	The experiments find that none of the assumptions in the main theorem are necessary for identifiability to hold. More discussion about what necessary conditions would look like would improve the paper.
-	The experiments are not very strong. Only quantitively analyzes the simple triangle dataset, so does not include a non-trivial dataset with known sources, so that identifiability can be quantified. Also, the experiments show large overlap with the experiments of [Sorrenson et al 2020]. An additional experiment with more complicated images with known sources would improve the paper.
-	It is unclear why the model does not fully succeed in identifying the true sources in the triangle dataset. Is one of the assumptions not satisfied? Are there learning difficulties?

Further comments:
-	It seems that there is a constraint that q(z|u) must be equal to the push-forward of p(s|u) through $g \circ f$, in other words, that Figure 1 can be interpreted as a commuting diagram. However, this is never explicitly stated in the definition of the estimating model. Could the authors please clarify this?
-	I believe the appendix should be separately provides as supplementary material.

Typos:
-	In (18) in the appendix, z_0 should be s_0?
-	Above (23) in the appendix, should positive-defined be positive definite? Or positive semi-definite?

summary_of_the_review: The paper provides an interesting theoretical analysis of an existing method. I weakly recommend acceptance, as the assumptions are not sufficiently motivated, the assumptions are shown to not be necessary and the experimental evaluation is not very strong.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose a framework for nonlinear ICA, in which the mixing function is a volume-preserving transformation, and the conditions for the sources can be relaxed compared to some prior works. The authors prove the identifiability of the proposed framework and implement the framework by volume-preserving Flow-based models. Numerical experiments on both synthetic and real data are performed to corroborate the theoretical results.
main_review: The nonlinear ICA problem is of significant research interest, and it is important to provide an identifiability guarantee for nonlinear ICA. In this work, the authors impose some reasonable assumptions on both the sources and mixing functions and provide two theorems for nonlinear and linear identifiability respectively. In particular, the theoretical results lead to the insight that the main indeterminacy of a nonlinear ICA framework using volume-preserving transformations is the rotation of latent variables. The authors further perform experiments to show that their method outperforms the state-of-the-art nonlinear ICA method iVAE. 

This paper is well-written, and the motivations and contributions are clearly presented. 
summary_of_the_review: Overall, my impression is that this work provides some solid contributions to the interesting nonlinear ICA problem. One issue I hope the authors can address is including the source codes in a code appendix.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose to use volume preserving transformation to solve the disentanglement problem in latent variable models such as ICA. The authors have explained that in literature that ICA can be expressed as a factorial member of the exponential family, however, this arrives at a solution which is not identifiable. The authors have explained that there are two ways of achieving disentanglement, this could be done by either (1) restricting the sources or (2) restricting the function f.


Interestingly that authors have proposed to use a volume-preserving transformation typically found in normalizing flows to be able obtain identifiability. There is a volume preserving transformation from the sources to the mixed signal which represents the generative process. Then there is a volume preserving transformation from the mixed signal to the latent space (which is not the source signal). Then a factorial multivariaate Gaussian is used to identify the conditional independence of each source signal.

In the classical ICA problem setup, the dataset does not contain any labels. While in this setting the input dataset for ICA appears to contain labels (u?).
main_review: I think there are some merits in terms of the novelty of the paper. However, the setting for ICA appears to be different from the classical setup as it requires labels. In the classical ICA setup, there appears to be no labels in the dataset. Understandably you are trying to solve the same problem but you have labels to train the model. 

My questions are:
(1) How do you select $n$ in Equation (8)

(2) How do you obtain \mu (u) and \sigma (u) in equation (8)

(3) So does this approach still work if we don't have any labels for the data?

(4) Since the transformations are volume preserving, does that mean you recover the source signals in the correct scale?
summary_of_the_review: The paper proposes a simple and elegant solution to the disentanglement problem encountered in ICA. So I recommend in accepting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors reformulate ICA in a way that makes it approachable with Flow-based methods. This yields a practical approach that the authors can apply to MNIST and to another dataset of that level of simplicity.
main_review: I will admit that I'm not familiar with the recent developments when it comes to ICA and to Flow-based modeling of densities, but this seems like a good paper. It is difficult for me to assess the importance of the original contribution from the authors.

Q: Does the term "point-wise linear" refer to the kind of model described in https://arxiv.org/pdf/2001.06988.pdf ?
summary_of_the_review: This is a good paper even though I do not have familiarity with the recent developments on that specific topic.
review_title: null

Meta-review: This paper proposes an identifiable nonlinear ICA model based on volume-preserving transformations. The overall approach is very similar to the GIN method published @ ICLR 2020. There is a weak consensus among the reviewers that this paper has some merit, although none pushed for acceptance. After reviewing the paper myself, I agree that the contributions here appear to be incremental, but the results do push this growing field of identifiable latent variable models forward.
Decision: Accept (Poster)