Provable Subspace Identification Under Post-Nonlinear Mixtures
Authors: Qi Lyu
Xiao Fu
Venue: NeurIPS 2022 Accept (2022)

Abstract: Unsupervised mixture learning (UML) aims at identifying linearly or nonlinearly mixed latent components in a blind manner. UML is known to be challenging: Even learning linear mixtures requires highly nontrivial analytical tools, e.g., independent component analysis or nonnegative matrix factorization. In this work, the post-nonlinear (PNL) mixture model---where {\it unknown} element-wise nonlinear functions are imposed onto a linear mixture---is revisited. The PNL model is widely employed in different fields ranging from brain signal classification, speech separation, remote sensing, to causal discovery. To identify and remove the unknown nonlinear functions, existing works often assume different properties on the latent components (e.g., statistical independence or probability-simplex structures). This work shows that under a carefully designed UML criterion, the existence of a nontrivial {\it null space} associated with the underlying mixing system suffices to guarantee identification/removal of the unknown nonlinearity. Compared to prior works, our finding largely relaxes the conditions of attaining PNL identifiability, and thus may benefit applications where no strong structural information on the latent components is known. A finite-sample analysis is offered to characterize the performance of the proposed approach under realistic settings. To implement the proposed learning criterion, a block coordinate descent algorithm is proposed. A series of numerical experiments corroborate our theoretical claims.

Provable Subspace Identification Under Post-Nonlinear Mixtures
Qi Lyu
Xiao Fu School of EECS Oregon State University Corvallis, OR 97331 xiao. xiao. wto.ruooorgoontttee.eie
School of EECS
Oregon State University Corvallis, OR 97331 1yugi0oregonstate.ce
Abstract
Unsupervised mixture learning (UML) aims at identifying linearly or nonlinearly mixed latent components blind manner. UML known to challenging: Even learning linear mixtures requires highly nontrivial analytical tools, e.g., in- dependent component analysis or nonnegative matrix factorization. In this work, the post-nonlinear (PNL) mixture model-where unknown clement-wise nonlinear functions are imposed onto linear mixture-is revisited. The PNI. model is widely employed in different fields ranging from brain signal classification, speech separation, remote sensing, to causal discovery. To identify and remove the un- known nonlinear functions, existing works often assume different properties the latent components (e.g., statistical independence or probability-simple: structures). This work shows that under carefully designed UML criterion, the existence of a nontrivial null space associated with the underlying mixing system suffices to guarantee identification/removal of the unknown nonlinearity. Compared prior works, our finding largely relaxes the conditions of attaining PNL identifiability, and thus may benefit applications where strong structural information on the latent components is known. A finite-sample analysis is offered to characterize the performance of the proposed approach under realistic settings. To implement the proposed learning criterion, block coordinate descent algorithm is proposed. A series numerical experiments corroborate our theoretical claims.
Introduction
Unsupervised mixture learning (UML.) has been broadly used in machine learning tasks, e.g., for recovering mixed latent components or extracting informative data representations. Central to understanding UML is the concept of identifiability. The identifiability of UML concerned with whether the latent components can be identified (up to certain to inconsequential ambiguities) with gutthntees-wwthout using any label training data. This is challenging problem due to the unsupervised nature. To address the identifiability challenge, linear mixture learning problems were tackled via exploiting properties of the latent components, e.g., statistical independence (T] nonnegativity M. suasinstationaaiiy source separation (BSS) methods.
5 6 and boundedness (7]. This leads to cohort of blind
However, linear mixture models (LMMs) oftentimes fail to perform well on real-world data, due to the unknown nonlinear distortions arising in data generation and acquisition. In the past two decades, nonlinear mixture models (NMMs) have attracted unprecedentedly more attention (8-14). Nonetheless, the identifiability of the latent components much harder establish under NMMs. was shown in (15) that the NMMs are in general not identifiable, even with strong assumptions, e.g.. statistical independence among the latent components. To establish NMM identifability, line of work exploited observable auxiliary variables or specific priors (i.e., extra information about the data
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Another line of or the latent components on top of statistical independence); see, e.g. [3 14
methods took an alternative route via utilizing the structures of the nonlinear distortions. In particular. the post-nonlinear mixture (PNI.) model [8 9 18 20) is reasonable extension of the classical LMM by assuming unknown nonlinear transformations individually imposed onto the outputs of a
underlying linear mixture. This kind of "structured NMM" is used in wide range of applications, e.g.
hyperspectral imaging [1]2 brain data classification (0 22 24). and causal discovery [12] 26).
In order to identify the generative model/latent components under the PNL model, key step is to
remove the nonlinear distortions in an unsupervised manner. To this end, the existing methods often explicitly rely on some properties of the latent components, e.g., statistical independence (8] [8].
availability of multiview data [20] and the sum-to-one and nonnegativity (or probability-simplles structure of the latent components [99 27 These structures stem from specific applications, and thus are meaningful-yee not always available. A natural question is whether the nonlinear distortions are do not hold?
still removable if the widely used structural assumptions (e.g., those in 18 20 27
In this work, we offer positive answer to the above inquiry- and build new learning algorithm based on our analyses. Our detailed contributions are as follows.
Subspace- Based PNL Model Identification. We propose new nonlinearity identification and removal criterion under the PNL model. We show that, as long as the underlying unknown linear mixing system of the PNL model admits a nontrivial orthogonal complement (or, transpose has a nontrivial null space), the unknown nonlinear transformations are identifiable and removable, under mild conditions. Notably, our approach does not use any specific structural assumptions of the latent components. Our finding presents fairly different result compared to existing nonlinear mixture learning techniques. The latter often heavily rely on the properties of the latent components. Hence, our result may offer new insights and a useful alternative to this long-standing challenge. An immediate consequence is that our method can work with various types of latent components, including those not covered by existing PNL model identification approaches in (8)
20 27
Performance Analysis Under Realistic Settings. The model identification analysis is conducted under the population case where unlimited data is available, with the assumption that a universal function learner can be used to cancel any nonlinear distortion. To advance the understanding under more realistic settings, we also consider the realistic scenario where one only has access to finite samples and non-universal function learners. The analysis a combination or statistical generalization theory and numerical differentiation, which is reminiscent of the recently developed framework
20 28
with nontrivial additional efforts to accommodate our learning criterion.
Optimization Algorithm Design, Based on the theoretical analysis, we formulate the identification problem as constrained optimization criterion. The optimization problem is challenging, as it involves learning unknown nonlinear functions [represented by neural networks (NNs)) under nonconvex constraints. We propose a block coordinate descent (BCD) algorithm that alternates between two subproblems, namely, null space basis learning and nonlinear function learning. This way, the latter disentangled with constraints, and thus off-the-shelf NN training techniques (e.g.. Adam (30) can be readily employed to handle the subproblem. A series of experiments are conducted for validating our theoretical claims.
Notations. I, x, and X represent a scalar, a vector, and a matrix, respectively; ||æ||o. |>> I1. and ||e||2 denote the to function, the & norm, and the Euclidean norm, respectively; @ and denote the Khatri-Rao product and the Hadamard product, respectively; f is used to denote a function f(.) : : R- R; f', f" and f(r) denote the first-order, the second-order, and the nih order derivatives of the function respectively; o denotes the function composition; R(X (X) and N(X) mean the range space and the null space of X, respectively.
2 Background
2.1 UML and Model Identification
In UMI., the classical LMM is defined as follows 31
(1) where ay E R is the 9th observation sample, 8; E RK contains the K latent components, and A E RM is the mixing system. The latent components in I are generally not identifiable,
3 Ase, f 1,2, ,N

---

if A is not known BT As remedy, information about A and/or is often used to establish identifiability. For instance, statistical independence among [st]; is used in independent component analysis (ICA) [1] and the nonnegativity of A E R and S
81,-' sN) E R are used for more conditions that can
in nonnegative matrix factorization (NMF) assist LMM identification.
also see 5 32 34
The LMM in T is often over-simplified for modeling the data generation/aacquiittoon processes in real-world applications- wwhee nonlinear effects are widely observed. In many cases, the NMMs are of more interest. A typical NMM is as follows [35]
Zz ((s). 8= 1,2, 1.2,.....
where g RKK + RM is nonlinear "mixing system". Compared to the LMM, the NMM in can represent much more complex data generation/acquisitton processes. However, it was shown in 335] that the model in ? is not identifiable, even if the latent components satisfy some fairly strong assumptions, e.g., statistical independence. A workaround is to use more information of or the latent components, e-g., the availability of auxiliary variables 13314 16). Another approach is by exploiting structural assumptions of the nonlinear mixing process. This leads the suite of PNL model learning works (see, c.g., 8 27 36)), which will be our focus.
additional conditions
2.2 Post-Nonlinear Mixture Model
The PNL model a natural extension to the LMM, which expressed as follows
18
3t = (As)), = 1,2, ............ .N
(3)
where g = [9 (.)... gM [-) with each 9m (m.()) R - R being an invertible nonlinear function. We assume 8, p(s) and the support of p(s) continuous and open domain S.
The PNL model has a myriad of of applications across different fields, e.g., causal discovery 12 25 26). hyperspectral imaging [2I] and brain data classification 22 24
It considered an appropriate model when the unknown nonlinear distortions happen at the sensor end in data acquisition. also offers valuable generalization to the linear mixture generative models in data analytics, e.g., those
used in NMF and data clustering 37 38..
An important remark is that the challenge of UML under the PNL model boils down to identifying and removing g-). as this will lead to a undersodert LMM as ① Then, any LMM identification approach, e.g., those in f= 1...... N.
34 139, can be applied to identify the latent components for
Existing works tackle the g-identification and removal problem from various angles. In 8 18 the statistical independent among the latent components (i.e., [Se]a for ....., K) is used. Recently, [1]27] proposed to identify the PNL model with latent components satisfying the probability simplex constraint (i.e., 1 1a 1, st Z 0) that often arises in soft clustering problems. The work in (20) shows that g--) can be provably removed if multiple views of data exist. These methods offered insightful and useful approaches remove (-)). but the use of specific structural assumptions on the latent components make their applicability relatively narrow.
We propose to identify and remove the nonlinear distortions via exploiting underlying subspace that often naturally exists under PNL models, instead of resorting to structural assumptions of sy as in prior works. Our method starts with the following observation: If the mixing system A is tall matrix, N (AT) is nontrivial subspace. This implies that if only the linear mixture part is considered (i.e., if g(-) can be removed completely), there exist vectors v such that
v Ase 0, Ae,
(4)
where the nonzero is any vector from ()A"). In the next section, will show how this simple equation in allows for constructing a provable g g-identification and removal criterion.
3 Proposed Approach
In this section, we propose a learning criterion to identify/remove the nonlinear transformations (-). We first consider the population case where infinite data points are available; that is, all æ; p(#) over the continuous support A' are available. Then, we will move forward to consider the finite-sample case in Sec. 3.2 Under the infinite-data assumption, our g-identification formulation as follows:
3

---

find f() [fi()), faI" s.t. Q" f(()) 0, Vxz E X fm (-) : - R is invertible Vm.
(5a)
(5b)
(5c)
l = MDD, Q¹Q=1
(5d)
where Q RMxD D is the dimension of the null space (i.c., (AA)) in which D > 1 and D M. The lg function constraint (5d) means that Q is constrained to be a dense matrix, which will prove important for establishing identifiability; this will become clearer in Sec. 3.1 In the formulation, (.) is nonlinear invertible function that we look for. Ideally, we hope to learn f., = 9m (-) which cancels the nonlinear transformation. Then, Q forms a basis of N (AT). Note that under orthogonality constraint in (5d), the trivial solution, i.e., Q O, is avoided. Besides, the invertibility constraint in iclis used to prevent other degenerate solutions, c.g., f(*{) = c for all & with constant from happening.
3.1 Provable Nonlinearity Removal
We will show that solving S guarantees that the nonlinear transformation g(-) removed from the PNL model. To this end, we first introduce the following condition:
Definition (Locally Free Components) Consider real-valuee random vector v = [ei..... val E Rd, where in resides in continuous and open set V: S IR. Assume that the following holds for all p(uz|w=3) > O, with continuous support Vije=; given any Ø-jr
where w- denotes the vector with all entries but Vj. Then, the components in v are all locally free components.
Note that the definition above means that each fi has the "freedom" to change (at least locally) no matter what value the øーj part takes. As special case, statistically independent random variables clearly satisfy the definition. However, statistical independence much stronger condition. Dependent variables could also be locally free components. For instance, vectors from the probability simplex, i.c.,
v VE AEAK, A = ( 0).
(6)
also satisfy Definition D that is, any Generally, if ..,..., 2a not completely dependent, the locally free condition is not hard to meet.
of the K components in v are locally free components.
Simply speaking, with o; and Vy being free variables, we could observe any possible value of uj given a fixed üg. Using this notion, consider the functional equation q h(Av) that holds for all v. Then, we can always observe
q [A[u.. -y. Da] ) -, with any fixed ü-y. where vy is from the continuous support of p(u,|ū=5) > 0. Taking derivative of w.r.t. by leads the following:
(q o a T)') (Alv),... .y.... Ca] 0,
as the terms 00./80; for any + j. This property will help us show the following main result:
Theorem (Nonlincarity Identification and Removal) Under the model in B assume that the criterion is solved. In addition, assume that A E Mx is drawn from any joint absolutely continuous distribution, that K of the K components of locally free components (ef. Def. and that the learned hi.. = Sm Jm is twice disferentiable for all [M). Suppose that
K(K +1 M.
2
Then, almost surely, any f()) that a solution of 5 satisfies Ti (x)(x) = o 9m( = Cmn + d. Vm E [M], where cm and dm is constant, at the limit of infinite data.

---

Theorem indicates that if any solution f(-) of ( is found, then the composition hm = fm o gm an affine function for all m. Notably, the theorem does not require all the K latent components to be locally free components, as long as the condition <(K+1)/2 2 M is satisfied- means that some components are even allowed to completely dependent. Intuitively, if K is too small then it means that the remaining K K components do not provide useful information, while a larger K implies that there is more diversity among the latent components to assist nonlinearity removal. Note that the premise here is that the null space M(A') must exist, i... M K should hold as rAnk(A)= = min M, K) with probability one for any A that is drawn from a joint continuous distribution. Otherwise, one may fail to learn a nonzero Q. On the other hand, if M is overly large, i... M 1(/+1//2 the model is not identifiable as there is lack of information (i.e., the number of locally free components is not large enough) to pin down all the 9: (-)sss with i = 1,-' M. We show the proof as follows:
Proof: By solving 5 we have the following equation for all sy E S
Q h(s, = O,
(10)
where h fog is the function composition. For simplicity, we drop the subscript "(" as the equation holds for any s E S.
First, show the existence of solution to It is obvious that one can let f = (i.., fm 9m for all m) and make the columns of Q to be an orthogonal basis of ((A ). In addition, such a Q can always be made dense. This is because A follows a joint continuous distribution, which indicates that the probability of (AA being orthogonal to any axis is zero.
Next, we show that such a solution is unique up to only affine ambiguities. By the assumptions, there are K out of the K components of 8 that are locally free. Thus, by taking the second-order (cross) derivatives of Eq. 10 w.r.t. each of the variables, as shown in we have the following:
(a @ aa))
((([As])) : ([As)a))))
(aR² RRl
0, V S,
(11)
(a @ a2)
a R- a
B
where ag is the keth column of A. Without loss of generality, we have assumed that the first K variables of s are locally free components.
Note that if one can show Ri ([As].) 0 for any over any then it indicates that 7i (-) is an
affine function for all m over S. To this end, we show that Q @ B has full column rank, where the
R(R-11 XM
sizes of the matrices involved are Q E RM and B E R
By Lemma 1 of [40). for matrices U E Raxm and V E Rbxm, U ④ V has full column rank if krank(U) + krank( V) 2 m + with krank(U) and krank (V) being the Kruskal rank of U and V. respectively. The Kruskal rank of matrix U is defined as the largest number such that every set of r columns of U is linearly independent. For Q O B, we do not have control over Q since it is optimization variable. But the Kruskal rank ko is at least I since have the constraint |iQl|o = MD. In terms of B, we can show that the matrix B has full Kruskal rank min R R 11/2, almost surely. The detailed explanation can be found in Appendix B
To derive the conditions required for model identification (i.e., rank (QT O B) = M), we consider the following two cases:
a) For the case of krank( B) following conditions are met:
R(K+1) 2
raakk (QT ④ (B) = M with probability one when the
K(K 1)
2 and 1 K(K 11
K(K + 1)
M 2
M +1, which holds if M
2
5

---

b) For the case of krank(B) = M, rank Q" following conditions are met:
1 B M with probability one when the
K(K +1
K(K + 2 M.
M
and
1 M M which holds
2
The two cases imply that if R(++)) 2 M is satisfied, then the nonlinear functions Im (.).ssss can be removed almost surely.
3.2 Finite-Sample Identifiability Analysis
The identifiability analysis in Sec. 3.1 is based on the assumption that (uncountably) infinite samples available. However, always has work with finite samples in practice. In addition, the proof of Theorem D assumed that the [:)(s's are universal function approximators. Nonetheless, in practice, the function learners, e.g., neural networks, may have nontrivial approximation errors for modeling nonlinear functions. In this section, we provide analysis under more realistic conditions.
To begin with, note that when only N i.i.d. samples are available, one could rewrite the finite-sample version of (5) as follows
min Q
Σ t=1
Q
(ax))
(12a)
s.t.
Q MD, I
(12b)
where F; + is an invertible function class used for modeling the inverse of function (. In
the population case (i.e., N -o), (12) and 5 have the same solutions. To characterize the finite sample performance of (12) under f(.) that has limited expressive power, we will use the following definition and assumptions:
Definition 2 (Function Class and Inverse) The function E G is an invertible continuous nonlin- ear function. We define its inverse class G-1 function class that contains all the u's satisfying u g(y) = oz + 3 with + 0, Vg E G.
Assumption (Realization Gap) For any u G-1, there exists f suPaex [f(m)) -oon)-uu(mm u(Im)l & v4, Vm E [M].
F such that
Assumption (Boundedness) The 4th-order derivatives of fm F and gm exist. In addition. ( (-)l and (n) (.)) are bounded for all n E [4] and m E [M]. Any 4th-order derivative of q. h( (As)) with & = [D] bounded by Ca- In addition, A has bounded elements.
Assumption 3 (Neural Network Structure) The function fmn()) is parameterized by one-hidder layer neural network with Rt neurons and 1-Lipschitz nonlinear activation function (()) IR + with ((0)
== = (fllfmmm) = w. ((w)r), |w:112 5 B, i = 1,2].
(13)
where (1y) (...)).....(.nnn w; E RR for 1,2.
Next, we show the Umite-ssmple identifiability result for the case where the above neural network class used. Our analysis can be readily generalized cover more complex neural networks such as deep convolutional neural network (CNN) and ResNet. However, we use this simple neural network as showcase, since our goal is to reveal insights other than covering complex neural structures.
Using the neural networks from Assumption we have the following lemma:
Note that commonly used activation functions, e.g., tanh, rectified linear unit (ReLU) and exponential linear unit (ELU). all satisfy this condition; see 41

---

Lemma 1 Assume that F is the function class defined in Assumption Band the input x: is bounded by [mml S Co for all m. Then, the loss function QQ f has the following Rademacher complexity
R N
R S 2DMB
(14)
The proof can be found in Appendix D Note that Lemma indicates that the complexity of the function class is proportional to the width of the neural network R and inversely proportional to the sample size N. With Lemma
D
we have the following theorem:
Theorem (Finite-Sample Identifiability) Under the generative model L assume that Assump- tions and 3 hold. Suppose that 2; for E [N] are i...d. samples from X according certain
distribution D. Denote any solution of 112 as f [f1..... .fas and hm fm 9m for E [M].
Then, with probability of at least 1 - ô. the following holds:
CaBAC (VR+ (2log(()))))
1/2
CaVMV ( OTBB)
E
h" (As)
O
(15)
min (Q
B)N1/4
where o min' in the linear system
By denotes the smallest singular value of the matrix Q T c B, which is defined
The detailed proof is relegated to Appendix D The proof consists of three steps: 1) treat the loss function as supervised regression problem and estimate the "generalization error" using the empirical error and Rademacher complexity; 2) approximate the linear system in (11 using the generalization error and numerical differentiation; and 3) bound and characterize the second-order derivatives based on the approximated linear system.
Theorem implics that the second-orde derivatives of (-) (-)'s approach zero with sufficiently large sample size N. However, the term is not determined by N and only decreases to when F is a universal function approximator (which could be attained if R is large enough). Therefore, there is always trade-off between the complexity (encoded by key parameters like depth and width) of the neural network and the effectiveness of model identification given fixed sample size N. According to Theorem 2 one hopes to employ expressive enough neural network for canceling 9m)(),, but excessively increasing the expressiveness would hurt the performance under a given N-this is consistent with our experience in many neural network learning problems.
4 Optimization Design
In this section, we will design an optimization scheme to implement the criterion in 12 Specifically, propose to the following formulation:
min Q.f
Σ Q f(x)))1
(16a)
s.t. Q fm() fm(() invertible.
(16b)
Note that the formulation in 16 does not ensure a dense Q. Nonetheless, Q is initialized randomly using any continuous distribution, iterative optimizers would never return Q that contains zeros, the iterative algorithm can be approximated by a continuous transformation. We use the neural network structure defined in Assumption to approximate (.). Note that with sufficiently large R, the function f.).) is a universal approximator 42 43) In addition, to promote invertibility, we introduce another neural network r(.), with the same structure as that of f(-), make sure that I (æx) can be converted back to xy, This idea is from autoencoder follows:
[4 The problem is then recast
N
N
min
Σ Q
f(0)))
+1
Be - (((()))
(17)
QQ=II.fr
Q
N
1=1
2=1

---

where the hyperparameter À 2 0 is used for balancing the energy between the loss and the regularizer term. addition, we define C L1 + AC2.
The problem can be handled by block coordinate descent (BCD) algorithm as shown in Algorithm Note that any stochastic optimizer designed for neural network training (e.g., (30) be used here for the f-) and r(-) updates. We use 0 and O, to denote the parameters of the corresponding neural networks. Besides, we denote L and L; for 1, as estimates of L and L; over random batch B, respectively. For example, L BTT Eies IIQ (xx and L2 is defined in identical way.
The term Vo, L2 denotes the derivative of Eg taken w.r.t. 0,. The other derivative terms are defined follwoing the same manner.
We refer the algorithm post-nonlinear subspace identification. This is because Theorem indi- cates that the method identifies range(S") where S [ &N] for N > K after removing the nonlinear distortions (technically, this needs d,,, 0 for all (ef. Theorem T --which automatically
holds if our norm minimization term the objective becomes zero).
Algorithm 1: Post-Nonlinear Subspace Identification.
Data: æy for & = 1, ,N
Result: f
while stopping criterion not reached do
Q + Ul: M - D M, where U UDVT = SVD with F =f(<x), .M f((x)))
while stopping criterion is not reached do
Draw a random batch
Vo, Vo, AV
Vo, <- Co:
Update f and r using Vo, and Vo, E. with any stochastic optimizer, respectively;
end
end
Numerical Experiments
5.1 Synthetic Data
We generate data following the model in B.. For the neural networks representing fm and rm (-). we use 256 with ReLU activations. We use the Adam optimizer (30] with the initial learning rate being 24 for the network optimization part. For the hyperparameters, we set X = le-4, The nonlinear functions (.)))s are selected to be variants of e" sigmoid((r) and tanh(x) guarantee invertibility. The source code can be found online
Independent Latent Components. In this simulation, we make K = 3 and M 5. Each element of 8e is drawn independently from the uniform distribution U[ with N = 10000, and the mixing matrix A is drawn from the normal distribution. Under this setting, it obvious that Q R5x2
Fig. shows the composition (+) f' 9.)(). One can see that all the five compositions are visually affine, which means that the algorithm has successfully identified and removed (-)). Prior works on post-nonlinear ICA oftentimes involve computation and sample-size demanding implementations. For example, in [8. nonparameteric density estimation step of the learned components needs be involved every iteration in order measure and promote statistical independence. In comparison, the proposed approach is much more straightforward and easier to implement, as it does not rely on statistical independence.
Dependent Latent Components. In this case, we consider the latent components drawn from the probability simplex in K The mixing matrix A is the same as before. Note that the latent components are dependent. Consequently the dimension of the null space is M K + +1 because there are only K = K free variables in sy. For this simulation, we set N = 10, 000, K = 3, and
M = Accordingly, the dimension of A is M K + 1 3 which makes Q 5 matrix.
"atts://github..oo//ly

---

Figure
Learned function compositions with independent latent components.
9
Input
Ingat
Isput
Input
i1pot
Figure 2: Learned function compositions with dependent latent components.
Fig. shows the results. obvious that the nonlinear distortions are successfully removed. Readers might have noticed that this case the condition in ! is actually violated since R(R+))/2 Interestingly, the method still works. This is likely because in this numerical example Q tends to have full Kruskal rank while in our proof we only used that trank(Q") 1. It turns out that the identifiability condition could actually be much improved if Q" has full Kruskal rank. We refer interested readers to further discussions in Appendix A
Validating Theorem 2 In this part, we demonstrate the impact of R (i.e., the width of the employed neural network) on the performance of nonlinearity removal, as stated in Theorem 2 To be specific, measure the subspace distance (see the definition in (20) between the estimated range (S") and the ground truth range(S) (S"). where S [si,.... sN). This value is within [0.1] with 0 being the best, The setting is the same as that of the independent case.
Table I shows the results, which are averaged over random trials. One can see that when (-) is not expressive enough, i.e., when R 8, the performance is far from satisfactory. This is because the term // dominates in the bound (15). As the number of neurons increases, the subspace distance keeps decreasing, which indicates that the latent subspace well identified. However, the function is overly complex (R 1024), the performance starts to deteriorate since in this case the R becomes
dominant on the right hand side of (15). In addition, one can see that larger N leads to better
performance in general, which also validates our claim in (5)
5.2 Real Data
Dataset. We use the human face electroenceepallggam (EEG) datase The EEG signals were collected when subject was shown with pictures of real human faces or scrambled faces. The scrambled face images were generated from the real faces with Fourier transformation by random phase permutations (45)). At every sample, the EEG signal is 330-dimensional vector It, which are measured through 130 sensors (channels) all over the subject's scalp. The task is to use H, for é 1, ... 1,..., N as the training data learn classifier. The classifier aims to determine whether the subject sees a real face image or scrambled ones when a new D; is collected.
Metric. We use number of unsupervised representation learning (URL. method to extract low- dimensional embeddings of æy's. ss Then, these embeddings are used to train classifiers based on support vector machines (SVM) and logistic regression (LR). We have N 27, 692 samples of æz, which are split as training. validation and test sets with 24794, 1449, and 1449 samples, respectively. as indirect measure of the performance.
We use the classification eтror to
We follow the hypothesis in 455 There, the model is H = g(As; with PNL structure, where se are the brain generated "clean" electronic signals, and A and g(-) represent the propagation, mixing, and distortions on the sensor end, respectively (23). Hence, we use our method to remove g(:) and then reduce the dimension of As; to K" via PCA. Our belief is that if the PNI. learning method can remove g()), ) then the extracted undistorted signals will benefit training linear classifiers such SVM
'htps://wwww.r www. wwl.ac..acoooo..o. htt..i/o.........//////t/tt///na www.uu///mmmttt/mmmcccssss

---

Table 1: Subspace distance with independent components.
R 8
R 16
=
R =
R 128
R 256
R 1024
N 10000
0.47.10.08
0.22110055
0.08.10.02 0.07+0002
0.051:0022 0.05++0.01
0.04-0001
0.0310.01
0.07
N 20000
0.44+0.06
0.20
0.03=0.01
0.02+0.01
0.06=0.001
Table 2: Average classification error= (best error attained in the 5 trials) on the EEG data. (K') is the feature dimension for training the classifier.
Raw
PCA (30) Autoencoder (30) Proposed (30) nICA (16 (30) PNL-ICA (30)
SVM 0.46 (0.43) 0.43 (0.41) 0,43-00:3 (0,40) 0,40:10.03 (0.35) 0.44:00.22 (0.43) 0.461.0.05 (0.38)
LR 0.47=0..4 (0.42) 0.45=0.04 (0.41) 0.46:10.33 (0.41) 0.39:10.03 (0.35) 0.43++0.03 (0.39) 0.44=0.03 (0.39)
and LR. We include PCA (applied onto the data), autoencoder, and nonlinear ICA (nICA) based approaches [6 as baselines.
Results. Table 2 shows the classification error. The results are averaged over 5 random trials. For all methods, the reduced dimension K' (for our method, we have D {10,30, 50, 70} using the validation set. We use fully connected networks that have two hidden layers and 256 neurons on each layer. The same network structure used for the autoencoder baseline. The activation functions are all ReLU, We use Adam 30) optimizer for neural network updates, where the initial learning rate 1e_1
K') is selected over
One can see that our method has the lowest classification error, compared to other baselines. Note that brain signal classification problem quite challenging. and thus using the raw data only outputs an accuracy that is slightly better than random guesses. Using our method, the average classification errors are reduced by 6% and 7% from the those using the raw data for SVM and LR, respectively, which is substantial. This also shows the usefulness of the PNL model for EEG data. The PCA is a linear method that does not capture the nonlinearity of the data, and thus the result under PCA is less promising. The autoencoder can be understood not ensure provable g()) removal. The nICA variants also exhibit higher error rates relative to the proposed method. This may be because the ICA frameworks hinge structural assumptions on the latent components, which may not always be available practice.
nonlinearity removal method, but it does
Conclusion
In this work, we revisited the PNL model from model identification perspective. We proposed a new framework to identify and remove the unknown nonlinear distortions in the data generating/acquisition process. Compared to existing works, the proposed approach does not require the latent components to satisfy specific structural assumptions, e.g., statistical independence. Consequently, our new result offers PNL model identification tool to a much wider spectrum of applications. In addition, we provided analysis of the cases where only finite samples and non-universal function learners are available. We designed a constrained optimization scheme to implement proposed learning criterion. Our theoretical claims were supported by experiment results.
There are a number of limitations. First, the PNL model would incur high computational cost when the data feature dimension M is large, since each dimension induces an individual neural network. Second, there are meaningful applications where the A matrix may have nontrivial orthogonal complement (e.g., in PNL-based causal direction estimation (2)). yet this current framework does not guarantee nonlinearity removal in those cases. Third, the PNL problem's 's learning criterion is a nonconvex optimization problem. The proposed algorithm does not guarantee to solve the formulated learning criterion, which creates gap between the identifiable results (by assuming optimization optimality) and the attainable results in practice.
Acknowledgement This work was supported by the National Science Foundation (NSF) CAREER Award under Project SBCCS-214488
10

---

References
[1] Comon, "Independent component analysis, new concept?" Signal Processing, vol. 36,
no. 3, pp. 287 314, 1994.
[2] X. Fu, K. Huang, and N. D. Sidiropoulos, "On identifiability of nonnegative matrix factorization," " IEEE Signal Process. Lett., vol. 25, no. 3, pp.
2018.
[3] X. Fu. K. Huang, N. D. Sidiropoulos, and W.-K. Ma. "Nonnegative matrix factorization for signal and data analyties: Identifiability, Algorithms, and Applications," IEEE Signal Process. Mag., vol. 36, no. 2. pp. 59-80, March 2019.
[4] N. Gillis, Nonnegative matrix factorization. SIAM, 2020.
[5] L.. D. Lathauwer and J. Castaing. "Blind identification of underdetermined mixtures by simul- taneous matrix diagonalization," IEEE Trans. Signal Process., vol. 56, no. 3, pp. 1096 - 1005, Mar. 2008,
[6] X. Fu, W.-K. Ma, K. Huang, and N. D. Sidiropoulos, "Blind separation of quasi-stationary sources: Exploiting convex geometry in covariance domain," IEEE Trans. Signal Process., vol. 63, no. 9. pp. 2306-2320, May 2015.
[7] S. Cruces, "Bounded component analysis of linear mixtures: A criterion of minimum convex perimeter," IEEE Trans. Signal Process., vol. 58, no. 4, pp. 2141-2154, April 2010.
[8] A. Taleb and C. Jutten, "Source separation in post-nonlinear mixtures," IEEE Trans. Signal Process., vol. 47, no. 10, pp. 2807-2820, 1999.
[9] A. Ziehe, M. Kawanabe, S. Harmeling, and K.-R. Müller, "Blind separation of post-nonlinear mixtures using linearizing transformations and temporal decorrelation," Journal of Machine Learning Research, vol. 4, no. Dec, pp. 1319-1338, 2003.
[10] A. Hyvarinen and H. Morioka, "Nonlinear ICA of temporally dependent stationary sources," in International Conference on Artificial Intelligence and Statistics, vol. 54, 20-22 Apr 2017, pp. 460-469.
[11] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel, "Contrastive learning inverts the data generating process," in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 18-24 Jul 2021, pp. 12 1299-12290..
[12] K. Zhang and A. Hyvärinen, "On the identifiability of the post-nonlinear causal model," in 25th Conference on Uncertainty in Artificial Intelligence (UAI 2009). AUAI Press, 2009, pp. 647-655.
[13] A. Hyvarinen and H. Morioka, "Unsupervised feature extraction by time-contrastive learning and nonlinear ICA," in Advances in Neural Information Processing Systems, vol. 29, 2016.
[14] A. Hyvarinen, H. Sasaki, and R. Turner, "Nonlinear ICA using auxiliary variables and general- ized contrastive learning." in International Conference on Artificial Intelligence and Statistics, 2019, pp. 859-868.
[15] A. Hyvarinen, "Fast and robust fixed-point algorithms for independent component analysis." JEEE Trans. Neural Netw.,, vol. 10, no. 3, pp. 226-634,1 1999.
[16] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen, "Variational autoencoders and nonlinear ICA: A unifying framework," " in International Conference Artificial Intelligence and Statistics. PMLR, 2020, pp. 2207-2217
[17] L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf, "The incomplete Rosetta stone problem: Identifiability results for multi-vies nonlinear ICA," in Proceedings of UAI 2020, 2020, pp. 217-227.
[18] S. Achard and C. Jutten, "Identifiability of post-nonlineal mixtures," " IEEE Signal Process. Lett., vol. 12, no. 5, pp. 423-426. 2005.
[19] Q Lyu and X. Fu, aidentifabblityygurraaa via autoencoder." IEEE Trans. Signal Process., vol. 69, pp. 4921-4936 2021.
simplex-structured post-nonlinear mixture learning
[20]
"Nonlinear multiview analysis: Identifiability and neural network-assisted implementa- tion." IEEE Trans. Signal Process., vol. 68, pp. 2697-2712, 2020.

---

[21] Y. Altmann, A. Halimi, N. Dobigeon, and J.-Y. Tourneret, "A post nonlinear mixing model for hyperspectral images unmixing." in 2011 IEEE International Geoscience and Remote Sensing Symposium, 201 1, pp. 1882-1885.
[22] F. Oveisi, "EEG signal classification using nonlinear Independent Component Analysis." " in Proc. IEEE ICASSP 2009, April 2009, pp. 361-364.
[23] A. Ziehe, K. Muller, G. Nolte, B. Mackert, and G. Curio, "Artifact reduction in magne- toneurography based on time-delayed second-order correlations," IEEE Trans. Biomedical Eng.. vol. 47, no. 1, pp. 75-87, Jan 2000.
[24] F. Oveisi, S. Oveisi, A. Eframian, and I. Patras, NNnnlnnear Independent Component Analysis for EEG-Based Brain Computer Interface Systems." Independent Component Analysis for Audio and Biosignal Applications, Edited by Ganesh R. Naik, p. 165, 2012.
[25] K. Zhang and A. Hyvärinen, "Distinguishing causes from effects using nonlinear acyclic causal models," in Causality: Objectives and Assessment. PMLR, 2010, pp- 157-164
[26] K. Uemura, T. Takagi, K. Takayuki, H. Yoshida, and S. Shimizu, "A multivariate causal discovery based on post-nonlinear model," in First Conference on Causal Learning and Reasoning, 2022. [Online]. Available: https//opeeneeviw..e///oonnm
[27] B. Yang, X. Fu, N. D. Sidiropoulos, and K. Huang, "Learning nonlinear mixtures: Identifiability and algorithm," IEEE Trans. Signal Process., vol. 68. pp. 2857-2869, 2020.
[28] Q. Lyu, X. Fu, W. Wang, and S. Lu, "Understanding latent correlation- based multiview learning and self-supervision: An identifiability perspective." in International Conference on Learning Representations, 2022. [Online]. Available: http://pppnneiiee..eetfromrmmm// wwt.://oooooooiiiiiSSSSSSSSSS
[29] Q Lyu and X. Fu, "Finite-sample analysis of deep CCA-based unsupervised post-nonlinear multimodal learning," " IEEE Transactions on Neural Networks and Learning Systems, pp. 1-7, 2022.
[30] D. P. Kingma and J. Ba. "Adam: A method for stochastic optimization." in International Conference on Learning Representations, 2015.
[31] P. Comon and C. Jutten, Handbook of Blind Source Separation. Elsevier, 2010.
[32] M. Zibulevsky and B. A. Pearlmutter, "Blind source separation by sparse decomposition in signal dictionary." " Neural computation, vol. 13, no. pp. 863-882, 2001.
[33] X. Fu, K. Huang, B. Yang, W.-K. Ma, and N. D. Sidiropoulos, "Robust volume minimization- based matrix factorization for remote sensing and document clustering," IEEE Trans. Signal Process., vol. 64, no. 23, pp. 6254-6268, Dec 2016.
[34] N. Gillis and S. Vavasis, "Fast and robust recursive algorithms for separable nonnegative matrix factorization." " IEEE Trans. Pattern Anal. Mach. Intell. vol. 36, no. 4, pp. 698-714, April 2014.
[35] A. Hyvärinen and P. Pajunen, "Nonlinear Independent Component Analysis: Existence and uniqueness results," Neural Networks, vol. 12, no. 3, pp. 429-439, 1999.
[36] E. Oja, "The nonlinear PCA learning rule in Independent Component Analysis," Neurocomput- ing, vol. 17, no. 1, pp. 25-45, 1997.
[37] D. Lee and H. Seung. "Leerrning the parts of objects by non-negative matrix factorization," Nature, vol. 401, no. 6755, pp. 788-791, 1999.
[38] B. Yang, X. Fu, and N. D. Sidiropoulos, "Learning from hidden traits: Joint factor analysis and latent clustering," JEEE Trans. Signal Process., vol. 65, no. 1, pp. 256-269, 2017.
[39] X. Fu, W.-K. Ma, T.-H. Chan, and J. M. Bioucas-Dias, "Self-dictionary sparse regression for hyperspectral unmixing: Greedy pursuit and pure pixel search are related," IEEE J. Sel. Topics Signal Process.. vol. 9, no. 6, pp. 1128-1144 2015.
[40] N. D. Sidiropoulos, R. Bro, and G. B. Giannakis, "Parallel factor analysis in sensor array processing," IEEE transactions on Signal Processing, vol. 48, no. 8, pp. 2377-2388, 2000. [41] L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus, "Regularization of neural networks using dropconnect," in International Conference on Machine Learning. PMLR, 2013, pp- 1058-1066.
[42] G. Cybenko, "Approximation by superpositions of sigmoidal function," Mathematcs of Control, Signals and Systems, vol. 2, no. 4, pp. 303-314, 1989.
12

---

[43] K. Hornik, "Approximation capabilities of multilayer feedforward networks," Neural networks, vol. 4, no. 2. pp. 251-257, 1991.
[44] G. E. Hinton and R. S. Zemel, "Autoencoders, minimum description length and Helmholtz free energy." in Advances in Neural Information Processing Systems, 1994, pp. pp. 3-10.
[45] Ashburner, G. Barnes, C. Chen, J. Daunizeau, G. Flandin, K. Friston, D. Gitelman, Kiebel,
J. Kilner, V. Litvak el al., "Spm8 manual," Functional Imaging Laboratory, Institute of Neurol- ogy, 2012.
[46] G. Bacciagaluppi. M. J. Donald, and P. E. Vermaas, "Continuity and discontinuity of definite properties in the modal interpretation," Helvetica Physica Acta, vol. 68, no. 7, 679, 1995.
[47] T. W. Anderson, An introduction to multivariate statistical analysis. Wiley New York, 1962.
[48] T. Jiang, N. D. Sidiropoulos, and M. Ten Berge, "Almost-sure identifiability of multidi- mensional harmonic retrieval," IEEE Trans. Signal Process. vol. 49, no. 9, pp. 1849-1859, 2001.
[49] Liang, "CS229T/STAT231: Statistical Learning Theory," 2016.
[50] P. L. Bartlett and S. Mendelson, "Rademacher and Gaussian complexities: Risk bounds and structural results," " Journal of Machine Learning Research, vol. 3, no. Nov, pp. 463-482, 2002.
[51] S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: From theory to algo- rithms. Cambridge university press, 2014.
[52] K. Mørken, "Numerical algorithms and digital representation, 2013," Department of Mathemat- ics, Centre of Mathematics for Applications, University of Oslo, 2018
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "oUN2ih_UTI", "reviewer": "Reviewer_zio7", "summary": "The authors propose an apporach for provably identifying subspaces under post-nonlinear mixtures. The main idea is the formulation of subspace identification problem as nonconvex optimization problem where one learns both a nullspace matrix and an invertible map that removes non-linearities. Theoretical results show that subspace identification is possible under certain assumptions that involve degrees of freedom of latent variables and the  of the nullspace of the linear mixing matrix A.", "questions": "1) It is not clear why $\\mathbf{Q}$ should be dense. Why the orthogonality constraint is not sufficient to avoid trivial solutions?\n2) It seems that the non-linear map should be defined as : $g: \\mathbb{R}^M \\rightarrow \\mathbb{R}^M $. Can the results be generalized to mappings $g: \\mathbb{R}^M \\rightarrow \\mathbb{R}^N$ for any N? ", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper is easy to follow and well-written.\n2)  The approach is very interesting and the reported results are novel and quite insightful. It is quite surprising that properties of non-linear function g do not play any role in the conditions for subspace identification and removal of non-linear mapping.\n3) The authors provide simulated and a real EEG experiment showing promising results for their method.\n\nWeakness:\n1) The optimization problem is non-convex and might be hard to guarantee convergence to optimal f and Q. \nThe authors do not provide any insight as to  possible directions regarding the analysis of the optimization problem.\n2) The proposed method compares with very standard classifiers e.g. SVM and logistic regression. It would be interesting to compare the performance  more state-of-the-art approaches e.g.  a DNN classifier.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lAMBeKv2Khl", "reviewer": "Reviewer_TZbZ", "summary": "This paper tackles a very challenging problem: blind source separation under nonlinear mixing. The authors made a reasonable simplification that the nonlinearity is element-wise, to make the problem more tractable. The authors proposed a null-space identification algorithm to tackle the nonlinearity and the mixing matrix. Theoretical analysis on the identifiability properties were also provided to justify the proposed algorithm, together with convincing simulation results. The paper is very interesting and sheds lights on related independent component analysis problems.", "questions": "1) In Section 6, the reconstruction network r is required to have the same structure of the transformation network f. How important is this requirement? What if a different architecture is used for r?\n\n2) How is the regularization parameter \\lambda determined? Is the result sensitive to the setting of \\lambda?\n\n", "limitations": "The element-wise nonlinearity function is assumed throughout the paper. Can the proposed null-space algorithm be extended to arbitrary nonlinear mixing in equation (2)?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The authors proposed a novel null-space identification approach to simultaneously tackle the nonlinearity and the mixing matrix. \n2) An autoencoder reconstruction loss is included to force the transformation invertible. Moreover, end-to-end optimization can be carried out in the block coordinate descent sense. \n3) The theory is elegant and the experimental result is promising.\n\nWeakness:\n1) The algorithm could identify the null space of the linear mixing matrix and a nonlinear transform. It is still unclear to me how to guarantee  the recovery of the original signals from the null space, as there is still linear transform ambiguity there. As a comparison, the classic infomax algorithm (Bell and Sejnowski 1995) directly estimates a linear unmixing matrix based on a mutual information maximization criterion.\n\n2) The requirement for a dense null-space matrix Q seems unnecessary for me. Take an extremely simple case, where the mixing matrix A is the first K columns of a M*M identity matrix, and the nonlinearity function g is an identity mapping. In this case, Q should be in the null space of A, which means that the first K rows of Q must be zero, which violates the dense requirement. \n\n3) It would be nice if the authors can compare this proposed method with other nonlinear ICA / BSS methods, for example, classical methods reviewed in the following paper:\nDeville Y, Duarte LT. An overview of blind source separation methods for linear-quadratic and post-nonlinear mixtures. In International Conference on Latent Variable Analysis and Signal Separation 2015 Aug 25 (pp. 155-167). Springer.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fP1HfVcBYSD", "reviewer": "Reviewer_3LT3", "summary": "The paper introduces a novel method to identify post-nonlinear mixture models under mild assumptions by removing the nonlinear transformations, after which standard linear mixture model identification methods can be employed to identify the latent components. The problem is formulated as a constraint optimisation problem that can be uniquely solved given infinite data. Additionally a finite sample analysis with probability bounds is derived. An algorithmic implementation of the method is provided and evaluated on synthetic examples for both independent and dependent components to support the theoretical claims.\n", "questions": "- eq(2) vs. eq(3): do I understand correctly that the property M > K (‘A is a tall matrix’, l.125) is what makes the system in (3) identifiable whereas (2) is not? Then why is the relation in eq(9) of the form K(K+1) > 2M? Or should both hold?\n", "limitations": "No concerns on this aspect.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n+ very interesting contribution to a well established and frequently encountered problem\n+ high quality paper, well written with good explanations, despite the fairly complex/abstract techniques employed\n+ significant and surprising identifiability result \n+ potential to lead to relevant novel applications in various fields\n\nweakness:\n- experimental evaluation could/should have been stronger (perhaps move more proof details to the supplement)\n\nAs far as I can tell results are sound and novel, although I have to admit the subject is somewhat outside of my area of expertise, hence reduced confidence. Nevertheless: clear accept.\n\nminor remarks:\n- abstract reads a bit like an introduction, but ok\n- 28: typo ’non(n)egativty’\n- 60: missing word ‘.. those (that) are ..’\n- 149: missing word ‘.. (are) all locally …’\n- 287: Fig.7 => Fig.1\n- 301: Fig.7 => Fig.2\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G3tWHtlSyN_", "reviewer": "Reviewer_Gx16", "summary": "The authors prove that the existence of a nontrivial  null space  associated with the underlying mixing system suffices to guarantee identification/removal of the unknown nonlinearity. Consequently, a simple learning criterion is proposed. A sample complexity analysis is offered to characterize the performance of the proposed approach under realistic settings. An optimization algorithm using BCD is proposed and evaluated through numerical experiments. ", "questions": "\nQuestion 1:  In Eq. (5c), to avoid degenerated solutions $f_m(\\cdot)=c$, the functions $f_m(\\cdot)$ is assumed to be invertible. Is this constraint too stringent? \n\nQuestion 2:  In Eq. (5d), is the non-sparsity constraint $\\|Q\\|_0=MD$ too restricted? \n\nQuestion 3: In Section 5 \"Sample Complexity Analysis\", Theorem 2 starts with \"Sample Complexity\" but does not give an explicit bound on the sample complexity. It is suggested to give an explicit expression of the sample complexity. ", "limitations": "Yes, the limitations of the proposed approach (i.e., the efficiency, the non-existence of orthogonal component, and the non-convexity of the proposed criterion) have been adequately discussed in the last paragraph. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\nStrength 1: It is proved that the existence of a nontrivial null space associated with the underlying mixing system suffices to guarantee identification/removal of the unknown nonlinearity.\n\nStrength 2: A sample complexity analysis is provided to characterize the performance of the proposed approach under realistic settings. \n\n\n[Weaknesses]\n\nWeakness 1: As each dimension is modeled with an individual neural network, the PNL model would incur high computational cost when the data feature dimension M is large.\n\nWeakness 2: There are examples where the A matrix may not have a nontrivial orthogonal complement such that the proposed approach fails. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept