Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a criterion for non-linear ICA as applied to Blind Source Separation inspired by Independence of Causal Mechanisms in the causal discovery literature. The general idea is to constrain the gradients of the output with respect to the input sources to be orthogonal to each other; this is linked to the notion that the mixing mechanism is in some sense independent (or not "too fine-tuned") to the input sources. The authors show that this constraint eliminates a number of known counterexamples to identifiability in non-linear ICA, and test it experimentally to show that helps to characterize true solutions (both as an evaluation metric, and as a regularizer). The authors conclude with a discussion putting their contributions into context.
main_review: Overall, the paper is clearly written and the technical properties and limitations of the method are well-discussed. The experiments are well-designed to support the claims in the paper.

My main concern with the paper is that the discussion of ICM as motivation for IMA is a bit long and imprecise. While I do appreciate the review of ICM ideas, the notion of "algorithmic independence" is still rather vague (this is, IMO, often a problem in a lot of the ICM literature). Although two different measurements of the concept are given (in terms of Kolmogorov complexity and IGCI), a concrete counterexample of what it would mean to have "fine-tuning" between the cause and the conditional would be helpful. In particular, is there an example that could be given (e.g., a modification of the cocktail party example) where this independence would not hold? Perhaps, an operator might have changed some properties of the recording to better capture certain parts of the input sources?

A concrete example of this type would also give a better sense of when this functional restriction may or may not be appropriate in practice. When specifying identifying assumptions, this kind of grounding is generally useful because, by definition, there is no way to test these assumptions in practice.
limitations_and_societal_impact: The authors do a good job of noting exactly what they establish regarding IMA in the paper---that their measure eliminates certain counterexamples---and what they do not---full identifiability under the IMA restriction.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes applying the theory of independent causal mechanisms (ICM), from causal discovery, to nonlinear ICA approaches to the blind source separation problem (BSS).

ICM is based on the idea that variables in a system are algorithmically independent or do not share information with each other. The paper shows, however, that ICM is not sufficient for BSS as spurious solutions exist which are not equivalent to the truth.

To address these difficulties, the paper proposes a new condition for identifiability which places an orthogonality condition on the columns of the Jacobian and provides information theoretic and geometric interpretations. The paper then shows theoretically that a large class of spurious solutions are not admitted under the IMA formulation. 

Experiments on toy examples demonstrate that the model is identifiable under these spurious solutions.
main_review: In general the paper is well written (though a bit difficult to parse for a non-specialist) and the theoretical contributions are rigorous. I did not check proofs. 

I am familiar with the related causal inference literature but less so the ICA literature so it is difficult for me to assess the significance of the class of spurious solutions references in the paper and whether these correspond to real world obstacles or are more of general theoretical interest. If the authors could clarify this, it may influence my final score as it seems this is sort of the crux of whether the paper constitutes a significant advance since IMA is not compared to other ICA approaches in any other way.

I found the experimental results section to be particularly weak as no baselines are included, only the spurious cases are investigated and there are no results on real data. This leaves the reader unclear on several points: (i) whether IMA works similarly in practice to existing ICA approaches (despite the theoretical results), (ii) whether IMA is competitive enough with other ICA approaches on the non-spurious cases to justify the support for the spurious cases, (iii) whether the spurious cases are of practical significance in any real world datasets.

The paper would be significantly improved if other ICA baselines were included as well as experiments on the non spurious cases.

===

I increased my review to 7 based on the discussions and other reviews. See discussion below.
limitations_and_societal_impact: The paper discusses limitations
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a new regularization scheme to improve identifiability in the _nonlinear blind source separation problem_ (nonlinear ICA) inspired by the notion of _independence of mechanism_ coming from causal inference. The idea is to consider only mixing functions which have a Jacobian matrix with orthogonal columns everywhere. They show how this extra assumption allows to exclude two classical types of degeneracies in nonlinear ICA, although without showing identifiability. The theoretical claims are validated by synthetic experiments. 

main_review: Originality:

Recent works on nonlinear ICA have focused on imposing structure on the distribution of the source via auxiliary variables. Instead, this work proposes to put restrictions on the mixing function f. Although different assumptions on f have been explored in the past (cited by the authors), their independence of mechanism assumption (IMA) hasn’t (to the best of my knowledge). 

Quality:

I have not reviewed the proofs, however the various theoretical claims seem reasonable and are clearly presented. I was happy to read about new findings in that interesting direction. However, I have some concerns with the overall framing of the paper.

The inspiration for the authors' approach is the principle of independent causal mechanism (ICM). However, it is not clear to me what is the link between ICM and the orthogonality of the columns of the Jacobian of the mixing function. The authors seem to suggest a connection, but it felt somewhat vague. ICM is about algorithmic independence of mechanisms in a structural causal model (SCM), and here we regularize for orthogonality of the columns of the Jacobian of the mixing function.  

In causality, the term “mechanism” is typically reserved to the conditional distribution of a variable given its direct causal parents (or its associated structural equation). In this work, “mechanism” is used to denote a partial derivative of f w.r.t. some source s_i. This is an unnecessary and confusing clash of terminology in my opinion. Consequently, I do not think independent mechanism analysis is a good name for this contribution.

The authors also seem to suggest a connection between the orthogonality of partial derivatives and their algorithmic independence (lines 192-200), but again, this is a bit vague and unjustified (or maybe I missed an important point).  

The above points can be summarized as follows: I am not convinced that ICM and algorithmic independence is the correct language to describe the authors’ contribution, unless more justification is added to the paper.

Section 3 shows how “regularizing” for algorithmic independence between p(s) and f via a proxy (IGCI) is not sufficient for identifiability. It is always interesting to have negative results, but, following up on the above paragraph, the connection to ICM feels a bit forced.

Instead of showing identifiability for the model class they consider, the authors show how it excludes two types of degeneracies, namely the Darmois construction and the “rotated-Gaussian measuring preserving automorphisms”. This is not sufficient to show the model class is identifiable, although it is a good step in that direction. The authors acknowledge this fact at the very end of the paper (in the discussion section), I believe this should’ve been stressed earlier in the paper as well as near the theorem statements, since I believe it is not a typical thing to do in this literature (usually, there’s a theorem showing identifiability).

In corollary 4.11, it would be nice to have an intuition for why we need non-Gaussianity, can’t we just apply corollary 4.9 directly (since A is a conformal map)? What am I missing?
It would be good to have experiments with non-conformal maps satisfying the IMA assumption (I didn’t see any).

Minor: 
Would be great to have an intuition for why the Darmois construction yields independent sources.

The presentation of information geometric causal inference is very fast and I could not understand its meaning. 

Line 332: Would be good to have a brief description of the MCC and the Amari metrics.

Clarity:

I enjoyed reading the paper overall. 

Significance:

I believe this work has average significance, given how strong the conditions are on the mixing function and the fact that we do not know if it holds in practice (even partially) due to lack of real world experiments. 

That being said, I believe imposing constraints on the mixing function to improve identifiability is an interesting direction with potential significance.

-------
After reading the author's rebuttal, I decided to raise my score from 5 to 6. See my comment above.



limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose to borrow concepts from causality, in particular "independent causal mechanisms" , to propose a novel framework termed "independent mechanism analysis" (IMA), which can provide non-spurious solutions to nonlinear blind source separation. IMA can be seen as a restricted nonlinear ICA model, where the contributions of the latent variables $z_i$ to the nonlinear mixing $\mathbf{f}$ are independent, i.e. the columns of the Jacobian $J_\mathbf{f}$ are orthogonal.

main_review: The proposed IMA framework is novel, and provides an alternative solution to the nonlinear ICA (equivalently nonlinear BSS) problem. It is by now known that nonlinear ICA is not possible when dealing with general nonlinear mixing functions by only assuming independence of the latent components. Recent approaches usually involve using inductive biases in the form of auxiliary information to condition the latent distribution, without constraining the mixing. The authors propose to take a different path, by constraining the mixing instead.
The paper is very well written, and was a joy to read. Section 2 does a very good job of laying the ground for the IMA framework. Readers without knowledge about nonlinear ICA or the ICM framework will find this section very helpful.
The theoretical results are sound, and the assumptions are well justified. 
The new IMA contrast functions is very useful, and can be used in practice to rule out a lot of spurious solutions to the nonlinear ICA problem.

Overall, this work is original, and of great significance. Learning identifiable representations is an important topic in unsupervised learning, and the provided framework is an important step in this direction.
I don't have much to criticize about the paper. I would have liked to see a full identifiability theory, but I believe it will follow in future work, and I am satisfied with the quality and quantity of the current contribution.
limitations_and_societal_impact: The limitations of the paper are well explained and discussed in section 6.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper aims at the identifiability of nonlinear ICA. It proposes a principle as a constraint of mixture functions based on the modified independent causal mechanism principle in the causal discovery literature; investigates the identifiability of nonlinear ICA under such an assumption; then applies it to nonlinear ICA. It also shows that the identifiability results can cover some non-identifiable cases in the existed literature.
main_review: 1. The idea of the work is interesting: it adjusted the ICM principle in the related field of causal discovery for nonlinear ICA.
2. The paper is in general well written and properly introduces the method. The theoretical analysis and examples are appreciated.

My minor concern is how restrictive is the proposed IMA in line 199 which imposes constraints on the determinant of the Jacobin matrix. Although the information-theoretic interpretation and geometric interpretation were trying to further explain IMA, I didn't get help from the given information without further details and explanation.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The authors propose a novel method for non-linear ICA. The idea is to constrain partial derivatives of the mixing function with respect to each source to be orthogonal. The authors connect this idea to the principle of independent mechanisms. The authors show that the constraint allows to deal with known counterexamples to identifiability in non-linear ICA. The method is evaluated on simulated toy examples. Reviewer hbzz praised the originality and significance of the work. Several reviewers (tDZV, ovUQ, 989y) criticized that the connection of IMA to other principles (such as algorithmic independence and information geometric causal inference) is vague or difficult to understand. In the responses, the authors expanded on the connections between the principles and proposed to de-emphasise the connection to algorithmic independence, while emphasizing the connection to information geometric causal inference. Several reviewers (VVZG, ovUQ, hbzz) agree that the paper is well-written. Overall, the authors have addressed the main concerns in the discussion. 
Decision: Accept (Poster)