Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a novel method to identify post-nonlinear mixture models under mild assumptions by removing the nonlinear transformations, after which standard linear mixture model identification methods can be employed to identify the latent components. The problem is formulated as a constraint optimisation problem that can be uniquely solved given infinite data. Additionally a finite sample analysis with probability bounds is derived. An algorithmic implementation of the method is provided and evaluated on synthetic examples for both independent and dependent components to support the theoretical claims.

strengths_and_weaknesses: strengths:
+ very interesting contribution to a well established and frequently encountered problem
+ high quality paper, well written with good explanations, despite the fairly complex/abstract techniques employed
+ significant and surprising identifiability result 
+ potential to lead to relevant novel applications in various fields

weakness:
- experimental evaluation could/should have been stronger (perhaps move more proof details to the supplement)

As far as I can tell results are sound and novel, although I have to admit the subject is somewhat outside of my area of expertise, hence reduced confidence. Nevertheless: clear accept.

minor remarks:
- abstract reads a bit like an introduction, but ok
- 28: typo ’non(n)egativty’
- 60: missing word ‘.. those (that) are ..’
- 149: missing word ‘.. (are) all locally …’
- 287: Fig.7 => Fig.1
- 301: Fig.7 => Fig.2

questions: - eq(2) vs. eq(3): do I understand correctly that the property M > K (‘A is a tall matrix’, l.125) is what makes the system in (3) identifiable whereas (2) is not? Then why is the relation in eq(9) of the form K(K+1) > 2M? Or should both hold?

limitations: No concerns on this aspect.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose an apporach for provably identifying subspaces under post-nonlinear mixtures. The main idea is the formulation of subspace identification problem as nonconvex optimization problem where one learns both a nullspace matrix and an invertible map that removes non-linearities. Theoretical results show that subspace identification is possible under certain assumptions that involve degrees of freedom of latent variables and the  of the nullspace of the linear mixing matrix A.
strengths_and_weaknesses: Strengths:
1) The paper is easy to follow and well-written.
2)  The approach is very interesting and the reported results are novel and quite insightful. It is quite surprising that properties of non-linear function g do not play any role in the conditions for subspace identification and removal of non-linear mapping.
3) The authors provide simulated and a real EEG experiment showing promising results for their method.

Weakness:
1) The optimization problem is non-convex and might be hard to guarantee convergence to optimal f and Q. 
The authors do not provide any insight as to  possible directions regarding the analysis of the optimization problem.
2) The proposed method compares with very standard classifiers e.g. SVM and logistic regression. It would be interesting to compare the performance  more state-of-the-art approaches e.g.  a DNN classifier.
questions: 1) It is not clear why $\mathbf{Q}$ should be dense. Why the orthogonality constraint is not sufficient to avoid trivial solutions?
2) It seems that the non-linear map should be defined as : $g: \mathbb{R}^M \rightarrow \mathbb{R}^M $. Can the results be generalized to mappings $g: \mathbb{R}^M \rightarrow \mathbb{R}^N$ for any N? 
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors prove that the existence of a nontrivial  null space  associated with the underlying mixing system suffices to guarantee identification/removal of the unknown nonlinearity. Consequently, a simple learning criterion is proposed. A sample complexity analysis is offered to characterize the performance of the proposed approach under realistic settings. An optimization algorithm using BCD is proposed and evaluated through numerical experiments. 
strengths_and_weaknesses: [Strengths]

Strength 1: It is proved that the existence of a nontrivial null space associated with the underlying mixing system suffices to guarantee identification/removal of the unknown nonlinearity.

Strength 2: A sample complexity analysis is provided to characterize the performance of the proposed approach under realistic settings. 


[Weaknesses]

Weakness 1: As each dimension is modeled with an individual neural network, the PNL model would incur high computational cost when the data feature dimension M is large.

Weakness 2: There are examples where the A matrix may not have a nontrivial orthogonal complement such that the proposed approach fails. 
questions: 
Question 1:  In Eq. (5c), to avoid degenerated solutions $f_m(\cdot)=c$, the functions $f_m(\cdot)$ is assumed to be invertible. Is this constraint too stringent? 

Question 2:  In Eq. (5d), is the non-sparsity constraint $\|Q\|_0=MD$ too restricted? 

Question 3: In Section 5 "Sample Complexity Analysis", Theorem 2 starts with "Sample Complexity" but does not give an explicit bound on the sample complexity. It is suggested to give an explicit expression of the sample complexity. 
limitations: Yes, the limitations of the proposed approach (i.e., the efficiency, the non-existence of orthogonal component, and the non-convexity of the proposed criterion) have been adequately discussed in the last paragraph.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles a very challenging problem: blind source separation under nonlinear mixing. The authors made a reasonable simplification that the nonlinearity is element-wise, to make the problem more tractable. The authors proposed a null-space identification algorithm to tackle the nonlinearity and the mixing matrix. Theoretical analysis on the identifiability properties were also provided to justify the proposed algorithm, together with convincing simulation results. The paper is very interesting and sheds lights on related independent component analysis problems.
strengths_and_weaknesses: Strengths:
1) The authors proposed a novel null-space identification approach to simultaneously tackle the nonlinearity and the mixing matrix. 
2) An autoencoder reconstruction loss is included to force the transformation invertible. Moreover, end-to-end optimization can be carried out in the block coordinate descent sense. 
3) The theory is elegant and the experimental result is promising.

Weakness:
1) The algorithm could identify the null space of the linear mixing matrix and a nonlinear transform. It is still unclear to me how to guarantee  the recovery of the original signals from the null space, as there is still linear transform ambiguity there. As a comparison, the classic infomax algorithm (Bell and Sejnowski 1995) directly estimates a linear unmixing matrix based on a mutual information maximization criterion.

2) The requirement for a dense null-space matrix Q seems unnecessary for me. Take an extremely simple case, where the mixing matrix A is the first K columns of a M*M identity matrix, and the nonlinearity function g is an identity mapping. In this case, Q should be in the null space of A, which means that the first K rows of Q must be zero, which violates the dense requirement. 

3) It would be nice if the authors can compare this proposed method with other nonlinear ICA / BSS methods, for example, classical methods reviewed in the following paper:
Deville Y, Duarte LT. An overview of blind source separation methods for linear-quadratic and post-nonlinear mixtures. In International Conference on Latent Variable Analysis and Signal Separation 2015 Aug 25 (pp. 155-167). Springer.
questions: 1) In Section 6, the reconstruction network r is required to have the same structure of the transformation network f. How important is this requirement? What if a different architecture is used for r?

2) How is the regularization parameter \lambda determined? Is the result sensitive to the setting of \lambda?


limitations: The element-wise nonlinearity function is assumed throughout the paper. Can the proposed null-space algorithm be extended to arbitrary nonlinear mixing in equation (2)?
review_title: null

Meta-review: metareview: The post-nonlinear (PNL) mixture model has been studied for a while in BSS community.  This paper presents a method to identity nonlinearity under mild conditions so that after removing the nonlinear transformations, a method for source separation  from a linear mixture is employed. All of reviewers agree that the paper is well written and has a solid contribution in BSS. Two important contributions are: (1) it is proved that the existence of a non-trivial null space associated with the underlying mixing systems suffices to guarantee identification/removal of the unknown nonlinearity; (2) a sample complexity analysis is provided to characterize the performance of the proposed approach. A downside is in weak experiments, which has been improved during the author rebuttal period. Therefore, I am pleased to suggest the paper to be accepted.

confidence: Certain
Decision: Accept