Nonparametric Evaluation of Noisy ICA Solutions
Authors: Syamantak Kumar
Derek Bean
Peter Bickel
Purnamrita Sarkar
Venue: NeurIPS 2024 poster (2024)
Keywords: ICA
Independent Component Analysis
Kurtosis
Characteristic Function
Cumulant Generating Function
Blind Signal Separation
Convergence
Primary area: other
TL;DR: We propose a non-parametric score to evaluate algorithms for noisy ICA and consequently propose a meta-algorithm which utilizes the score to choose the best algorithm for any given data distribution.

Abstract: Independent Component Analysis (ICA) was introduced in the 1980's as a model for Blind Source Separation (BSS), which refers to the process of recovering the sources underlying a mixture of signals, with little knowledge about the source signals or the mixing process. While there are many sophisticated algorithms for estimation, different methods have different shortcomings. In this paper, we develop a nonparametric score to adaptively pick the right algorithm for ICA with arbitrary Gaussian noise. The novelty of this score stems from the fact that it just assumes a finite second moment of the data and uses the characteristic function to evaluate the quality of the estimated mixing matrix without any knowledge of the parameters of the noise distribution. In addition, we propose some new contrast functions and algorithms that enjoy the same fast computability as existing algorithms like FASTICA and JADE but work in domains where the former may fail. While these also may have weaknesses, our proposed diagnostic, as shown by our simulations, can remedy them. Finally, we propose a theoretical framework to analyze the local and global convergence properties of our algorithms.

Nonparametric Evaluation of Noisy ICA Solutions
Syamantak Kumar!
Purnamrita Sarkar?
Peter Bickel
Derek Bean"
Department of Computer Science, UT Austin
2 Depaatmment of Statistics and Data Sciences, UT Austin
3 DDepartment of Statisties, University of California, Berkeley Department of Statistics, University of Wisconsin, Madison syamant auteeeeas.............aa....aaaa edu, purna. sarkareaustin, utexas edu, bickel@stat berkeley.edur derekbestat wisc. edu
Abstract
Independent Component Analysis (ICA) was introduced in the 1980's model for Blind Source Separation (BSS), which refers to the process of recovering the sources underlying a mixture of signals, with little knowledge about the source signals or the mixing process. While there are many sophisticated algorithms for estimation, different methods have different shortcomings. In this paper, we develop nonparametric score to adaptively pick the right algorithm for ICA with arbitrary Gaussian noise, The novelty of this score stems from fact that just assumes finite second moment of the data and uses the characteristic function to evaluate the quality of the estimated mixing matrix without any knowledge propose some new
of the parameters of the noise distribution. In addition, contrast functions and algorithms that enjoy the same fast computability as existing algorithms like FASTICA and JADE but work in domains where the former may fail. While these also may have weaknesses, our proposed diagnostic, as shown by simulations, remedy them. Finally, propose theoretical framework analyze the local and global convergence properties of our algorithms.
Introduction
Independent Component Analysis (ICA) was introduced in the 1980's as model for Blind Source Separation (BSS) 13 12], which refers the process of recovering the sources underlying a mixture of signals, with little knowledge about the source signals or the mixing process. It has become
powerful alternative to the Gaussian model, leading to PCA in factor analysis. good account of the history, many results, and the role of dimensionality in noiseless ICA can be found in [28 5]. In one of its first forms, ICA based on observing n independent samples from a k-dimensional population:
Bz +
(I)
where B E Rd (d 2 k) is an unknown mixing matrix, E R is a vector of statistically independent mean zero "sources" or "factors" and g ( Σ) Gaussian noise vector with covariance matrix E, independent of z. We denote diag (a) == cov (z) and S = cov (x: = diag (a) B" + The goal of the problem is to estimate
dimensional mean zero
ICA was initially developed in the context of g O. now called noiseless ICA. In that form, it spawned an enormous literature [7
42 and at least two well- 29 and JADE 9 which are widely used. The general model with
g
35 37 23
established algorithms FASTICA
nonzero g and unknown noise covariance matrix Σ. known as noisy ICA, has developed more slowly,
with its own literature 4 55 49
30
31 41.
In the following sections, we will show that various ICA and noisy ICA methods have distinct shortcomings. Some struggle with heavy-tailed distributions and outliers, while others require approximations of entropy-basee objectives, which have their own challenges (see eg. (2)).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Although methods for noiseless cases may sometimes work in noisy settings 49 they are not always reliable (e.g.. see Figure in 49 and Theorem 2 in [T]). Despite the plethora of algorithms for noisy and noiseless ICA, the literature has largely been missing a diagnostic to decide which algorithm to pick for a given dataset. This our primary goal.
The paper is organized as follows. Section [oontains background and notation. Section 3.1 contains the independence score, its properties, and a Meta algorithm [ that utilizes this score. Section 3.3 introduces new contrast functions based on the natural logarithm of the characteristic function and moment generating function of z u. Section presents the global and local convergence results for noisy ICA. Section 5 demonstrates the empinical performance of our new contrast functions as well as the Meta algorithm applied to a variety of methods for inferring B.
2 Background and overview
This section contains a general overview of the main concepts of ICA.
se[n] for (x(() IRd. Scalar random variables and matrices are represented using upper case letters, respectively, In denotes the k- dimensional identity matrix. Entries of vector (matrix M) are represented as V/ (Myy). M(:, i) (i)) represents the 2th column (row) of matrix M. Mt represents the inverse of M. |I.]| represents the Euclidean l2 norm for vectors and operator norm for matrices. ||-1|e similarly represents the Frobenius norm for matrices.
Notation: We denote vectors using bold font as v E IR4. The dataset is represented as {x(()) ]
(É [.]) denotes the I1 represents the
expectation (empirical average); x 11 y denotes statistical independence,
subgaussian Orlicz norm (see Def. 2.5.6 in [46). X, = Or (an) implies that Xu is stochastically as
bounded i.e, Vc > 0, there exists a finite M such that sup,, P
(1
Xm M) S c (See 45
Identifiability: One key issue in Eq 1 is the identifiability of A =- B- which holds up to permutation and scaling of the coordinates of 2 if, at most, one component of z is Gaussian (see [14] (3). If d & and Σ are unknown, it is possible, as we shall see later, to identify the canonical versions of B and Σ. For d > h and unknown Σ, it is possible (see [T5]) to reduce to the d = k k. For classical factor models, when is Gaussian, identifiability
case. In this work, we assume can only be identified up to rotation and scale, leading to non-interpretability. This suggests fitting strategies focusing on recovering B through nongaussianity as well as independence. In the noisy model (Eq[), [ additional difficulty is that recovery of the source signals becomes impossible even is known exactly. This is because, the ICA models x = ((z + r) + g and x = Bz + (Br + g) are indistinguishable (see 48) pages 2-3), This is resolved in 48 via maximizing the Signal-to- Interference-pluss posecce--puNNssseeee ratio (SINR). In classical work on ICA, a large class of algorithms (see (37) choose to optimize measure of non- -on-aaussiantty referred to contrast functions.
Methods for estimating B typically optimize an empirical contrast function. Formally, define contrast function f(u P) by f B& P - Th where Bg is the unit ball in R and P is essentially the class of mean zero distributions of g. f(u) P) = fu](z) = Eg~rig("" (u" z)] ] for any random variable x ~ P. The most popular example of contrast function is the kurtosis, which is defined as f(u P) Eg-r (u x) E (uT=)²²² 3.
Contrast functions:
Rgd More concretely, for suitable choices
The aim is to maximize an empirical estimate, f(u) P). Notable contrast functions are the negentropy. mutual information (used by INFOMAX 77, the tanh function, etc. (See (37) for further details).
There are two broad categories of the existing fitting strategies. (I) Find A such that the components of Ar are both as nongaussian and as independent as possible. See, for example, the multivariate cumulant based method JADE [9] and characteristic function- based methods 11). The latter we shall call the PFICA method, (II) Find successively the jth row of A, denoted by :(j.)) E Rd, j = 1,..., k such that A(j, :)" x are independent and each as nongaussian as possible, that is, estimate A(1,:), project, and then apply the method again on the residuals successively. The chief representative of these methods is FastICA [7 based on univariate kurtosis.
Fitting strategies:
In the noiseless case one can first prewhiten the dataset using the empirical variance-covariane matrix, E, of z(), using Đ- 1-1/2 Therefore, WLOG, B can be assumed to be orthogonal. Searching over orthogonal matrices not only simplifies algorithm design for fitting
From noiseless to noisy ICA:
If B is not invertible, this can be replaced by the Moore-Penrose inverse.

---

strategy (I) but also makes strategy (II) meaningful. It also greatly simplifies the analysis of kurtosis- based contrast functions (see (18). For noisy ICA. prewhitening requires knowledge of the noise
covariance, and therefore many elegant methods 4 48 49
avoid this step.
Individual shortcomings of different contrast functions and fitting strategies: To our knowledge, no single ICA algorithm or contrast function works universally across all source distributions. Adaptively determining which algorithm is useful in given setting would be an important tool in practitioner's toolbox. Key challenges include:
Cumulants: Even though contrast functions based on even cumulants have no spurious local optima (Theorem 49
they can vanish for some non-Gaussian distributions. Our experiments (Section (5)
show that kurtosis-based contrast functions can perform poorly even when there are few independent components with zero kurtosis. They also suffer under heavy-tailed source distributions [ 26].
PFICA: PFICA can outperform cumulant- based methods some situations (Section 5 However, computationally much slower, and poorly performs for Bernoulli(p) sources with small p.
FastICA: 22 show that FastICA's use of Negentropy approximations for computational efficiency may result in contrast function where optimal directions do not correspond to directions of low entropy. tanh is another popular smooth contrast function used by FastICA. However, in [52), the authors show that this tends to return spurious solutions for some distributions (52).
Contrast functions are usually nonconvex and may have spurious local optima. However, they may still, under suitable conditions, converge to maximal directions. We introduce such a contrast function which vanishes iff is Gaussian in Section 3.3 and does not require higher moments.
Our contributions:
1) Using Theorem we obtain computationally efficient scoring method for evaluating the B estimated by different algorithms. Our score can also extended to evaluate each demixing direction in sequential settings, which we do not pursue here.
2) We propose new contrast functions that work under scenarios where cumulant-based methods fail. We propose fast sequential algorithm as in [49 to optimize these and further provide theoretical guarantees for convergence.
3 Main contributions
In this section, we present the "Independence score" and state the key result that justifies it. We conclude with two new contrast functions.
3.1 Independence score
While there are many tests for independence [22 34
66, we choose the Characteristic function-
based one because it is easy to compute and has been widely used in normality testing [19 20 25 and ICA 11 21
Let us start with noiseless ICA to build intuition. Say we have estimated the inverse of the mixing matrix, i.c. B-1 using a matrix F. Ideally, i B then Fr = 2, where is a vector of independent random variables. Kac's theorem
3 tells us that E |exp(it" 2)] =
I E (x))( if and only if 2; are independent. In 21 novel characteristic function- based objective (CHFICA), is further analyzed and studied by (PFICA). Here one minimizes E (it" Fz)] I E (Ez)) over F. We refer to this objective as the uncorrected independence score. We propose to adapt the CHFICA objective using estimable parameters, S, to the noisy ICA setting. We will minimize:
tf (iag(FSF"
t FSSF
Att, = E [ix Fœ) exp
(exp(it, (exp(it((Fx))) (Fx);)) exp
2
JoiNI
PRODUCT
where E[xx="] and can be estimated using the sample covariance matrix. Hence, we do not require knowledge of any model parameters. We refer to this score the (corrected) independence score. The second terms in JOINT and PRODUCT (Eq corrects the original score by canceling out the additional terms resulting from the Gaussian noise using the covariance matrix of the

---

data (estimated via the sample covariance). See [2]] for related score requiring knowledge of the unknown Gaussian covariance matrix Σ. We are now ready to present our first theoretical result for consistency of (t,, F)P).
Theorem 1. If E R is invertible and the joint and marginal characteristic functions of all independent components, { are twice-differene difffefiffren,...ee, then vt E R A(t,F)P = 0 iff
DB- where D is permulation of an arbitrary diagonal matrix.
Unfortunately, F is not uniquely defined if Σ is unknown as we have noted in Section ! The proof of Theorem D is provided in the Appendix Section A When using this score in practice, S is replaced with the sample covariance matrix of the data, and the expectations are replaced by the empirical estimates of the characteristic function. The convergence of this empirical score, At. FIP). to the population score, (t,, F(P), is given by the following theorem.
Theorem 2. Let F : {F E R
||
1},
subgaussian(o) and Cş =
max(1, k log (n) Tr(S)). Then, we have
(1
1 121|5|| o (|S])
((L sup (Lt, Bee Op FEF
n
This bound shows that uniformly over F, the empirical average (o A(t, FIP,, is close to the population score. This guarantees that as long as the difference between the population scores of the two candidate algorithms is not too small, the meta-algorithm can pick up the better score.
Remark 1 (Subgaussianity assumption). The subgaussianity assumption in Theorem the concentration of the sample covariance matrix since it is used in the score (see Theorem A.3, line 825). It can be relaxed if the concentration in the operator norm is ensured. Appendix Section 1. contains experiments with more super-Gaussian Gaussian source signals.
ensures
The proof of this result deferred to the Appendix section A.1.1 It is important to note that Att F PIP)
is not easy to optimize. As we show in Section 3.2 objective functions that are computationally
more amenable to optimize for ICA, e.g., cumulants, satisfy some properties (see Assumption [ The independence score does not satisfy the first property (Assumption (a)). In the noiseless case, whitening reduces the problem to B being orthogonal. This facilitates efficient gradient descent in the Stiefel manifold of orthogonal matrices [ل]] However, in the noisy setting. the prewhitening matrix contains the unknown noise covariance Σ, making optimization hard. Furthermore, as noted
in Remark in practice, it is better to use Ee-,((o.1.) (tt, FIP) M 21 25 instead of using fixed
vector, t, evaluate (t,, F' PP) Any iterative optimization method requires repeatedly estimating this expectation at each gradient step. Therefore, instead of using this score directly as the optimization objective, we use it choose between different contrast functions after extracting the demixing matrix with each. This process is referred to as the Meta algorithm (A (Algorithm D.
25 11 Condistion of Indepandence score with Areari dbtance
Algorithm 1 Meta-algorithm for choosing best candidate algorithm.
Input: Algorithm list C. Data X Ra*k for j in range[[. I. size (C) do
Inteperderre
Bj & (X) (Extract mixing matrix
Bj using 5th candidate algorithm)
8, + B
| t, B 1P
]
end for
Ameri
0.5 0.0
1.2
Figure 1: Correlation of independence score (with std. dev.) with Amari error between B' = eB + (1 - <) 1 and B. averaged over 10 random runs.
i. < arg minjee(sre((C)) c)) [ōy]
return Bi
Remark (Averaging over t). Algorithm| 7 averages the independence score over t N (0. It). While Eq defines the independence score for one direction t, in practice, however, there may be some directions such that a non- gaussian signal has a small score in that direction. Hence, is desirable average over t, following the convention in [1] 21 25y.

---

To gain an intuition of the independence score, we conduct an experiment where we use the dataset mentioned in Section (Figure 2b) and compute the independence score for B' cB c) I (0.5, 1). As we increase e, B' approaches B, and hence the Amari error dæ' (see Eq decreases. Figure shows the independence score versus Amari error, indicating that the independence score accurately predicts solution quality even without knowing the true B.
3.2 Desired properties of contrast functions
The following properties are often useful for designing provable optimization algorithms for ICA. Assumption 1 (Properties of contrast functions). Let J be contrast function defined for mean zero random variable X. Then,
(a) f() + Y) = f(u[X) + f( Y (b) = (0)
- 0. (0(0 =0 ((u(G) 0, G (O,
G~NN
(d)WLOG, f(u]X) = X) (symmetry)
f(l for non-gaussian X and independent non-gaussian G, which means that the additive independent Gaussian noise does not change f. Property (c) ensures that [f(u G)| is minimal for Gaussian. Property (d) holds without loss of generality because one can always symmetrize the distribution.
Properties (a) and (c) ensure that G X)=
3.3
New contrast functions
In Section
we provided a discussion of the individual shortcomings of different contrast functions for existing contrast functions. Before we introduce new contrast functions in this section, we revisit the algorithmic issues posed by the added Gaussian noise with unknown E in Eq [
2
Prewhitening the data is challenging for noisy ICA because E[zx"" includes the unknown Gaussian covariance matrix Σ. 4 show that it is enough to quasi orthogonalize the data, i.e., multiply by matrix, which makes the data have diagonal covariance matrix (not necessarily the identity). Subsequent work [50) [49] uses cumulants and the Hessian of f f') to construct a matrix C == BDBT where D a diagonal matrix, and then use this matrix to achieve quasi-orthogonalizatin D may not be positive semidefinite (e.g. when components of z have kurtosis of different signs). To remedy this, in 49]. the authors propose computing the C matrix using the Hessian of f(u) at some fixed unit vector and then perform an elegant power method in the Euclidenn space:
In general,
u Vf(C'uu-lP)// (ttp//IV/CCCC.CC (.
Aeeudo-Fucliden space is generalization of the Euclidean space used by 48 Here the produce between vectors v is given as u Av,
Algorithm 2 Pseudo-Euclidean Power Iteration for ICA (Algorithm 2 in 48 B is the recovered matrix for the ICA model in Eq
is a running estimate of Bt.
Input: C E R V
Á + 0, B for j in range[1, 11, k] do
Draw uniformly random from Sk-1 while Convergence (up to sign) do AAu
u + V (Ctu) // (Ctu)ll2
end while
B, - u, Ag + (ttBy/ [(1B/((00B)) B,]
end for return B,
NNoe that : I. Lln/2/ti, i= 1... [n/2] has symmetric distribution, and also remains noisy version of linear combination of source signals with the same mixing matrix.
5

---

In this section, we present two new contrast functions based on the logarithm of the characteristic function (CHF) and the cumulant generating function (CGF). These do not depend on particular cumulant, and the first only requires finite second moment, which makes it suitable for heavy- tailed source signals. We will use [(u)] P) or [(((z) where æ ~ P interchangeably to represent the population contrast function. In constructing both of the following contrast functions, we use the fact that, like the cumulants, the CGF and CHF-based contrast functions satisfy Assumption [ To satisfy Assumption (c), subtract out the part resulting from the Gaussian noise, which leads to the additional terms involving Su.
Recall that S denotes the covariance matrix of z. We maximize the absolute value of following:
CHF-based contrast function:
f(u = log Eexp(iix x) + log E Bexp(-ite x) + u Su log(EE [[s(Ecos("' cos x²²² log(EEsin( x))² u² Su
(4)
The intuition is that this is exactly zero for zero mean Gaussian data and maximizing this leads to extracting non-gaussian signal from the data. also satisfies Assumption a), b) and c).
CGF-based contrast function: The cumulant generating function also has similar properties (see 533 for the noiseless case). In this case, maximize the absolute value of following:
1 =u² Sru
f(u P)) = log lE exp((" x)
(5)
2
Like CHF, this vanishes iff x is mean zero Gaussian, and satisfies Assumption 1 a), b) and c). However, it cannot be expected to behave well in the heavy-tailed case. In the Appendix (Section A.
Theorem A.I), we show that the Hessian of both functions obtained from Eq 4 Eq 5 evaluated some u is, in fact, of the form BDB where D is some diagonal matrix.
4 Global and local Convergence: loss landscape of noisy ICA
Here we present sufficient conditions for global and local convergence of broad class of contrast functions. This covers the cumulant-basse contrast functions and our CHF and CGF-based proposals.
4.1
Global convergence
The classical argument for global optima kurtosis for noiseless ICA in [8] assumes WLOG that B is orthogonal. This reduces the optimization over into one for v Bu. Since B is orthogonal, ||||= = 1 translates into ||||| = 1. It may seem that this idea should extend to the noisy case due to property 1 a) and e) by optimizing over Bu over potentially non-orthogonal mixing matrices B.
However, for not-orthogonal B, the norm constraint on v is no longer vTe 1, rendering the original argument invalid. In what follows, we extend the original argument to the noisy ICA setting by introducing pseudo-euclidean constraints. Our framework includes large family of contrast functions, including cumulants.
To our knowledge, this the first work to characterize the loss functions in the ayeudo-euclided space. In contrast, (50) provides global convergence of the cumulant- methods by convergence argument of the power method itself.
Consider the contrast function f {u P) == Ez-r [g (xTu)) u)] and recall the definition of the quasi orthogonalization matrix C = BDB" where D is diagonal matrix. For simplicity, WLOG let us assume that D is invertible. We now aim find the optimization objective that leads to the pseudo- BecudoEEuliddaan update in Eq For of (C-'u(P) u((P)
Ex~P [g (u C-
--)z]], consider the following:
{C- u
Σ
E
[g (B -u), 24/ [.))
Σ Egg (ox2,) TE[k]
Σ f ((as/Dale sE[k]
(6)
where we define a := B norm constraint u² CC = 1. The key point is that for suitably defined f. one can construct matrix C' = "DB" from the data even when B is unknown. So our new objective to optimize
lu and é; = 2 D We now examine f(C-luu subject to the "pseudo"
(C u/ P) s.t. u TC--==1

---

Using the Lagrange multiplier method, optimizing the above simply gives the power method update (Eq u x V u). Furthermore, optimizing to the following transformed objective:
S (os
s.t. Co./Dar =1
max a
Now we provide our theorem about maximizing f (C)) Analogous results hold for minimiza- tion. We will denote the corresponding derivatives evaluated at la as (4(()) and f (lol))))
Theorem 3. Let C be matrix of the form BDB", where d; == Da = f" (uil=)) zs) for some random u,- IR + R. Assume that for every non- aon-Gaussian independent component, Assumption 7 holds and the third derivative, (u](x) does not change the sign in the half-lines [0, 00), (co, 0f. Then f (C-lu) x) with the constraint of (u, u)c == has local maxima at B c,, E S4- All solutions satisfying e B-'u= & O, Vi E [1, k] are minima, and all solutions with i e. B uu + 0} < h are saddle points. These are the only fixed points.
Let S+ (i ; de > 0}. Consider a conirast fanction f foncttissanction
Note that the above result immediately implies that for Č= the same optimization in Theorem will have maxima at all u such that B e,, E. S_.
Corollary 3.1. 2 cumulant-base contrast functions for k > 1, have maxima and minima at the directions of the columns of B. provided (Zi)se(s) have a corresponding non-zero cumulant.
Proof. This proof (see Appendix Section A.1 is immediate since cumulants satisfy (a), (c). For (c) and (d), note that ful Z) is simply n K2y ZZ) where K2y(2 (Z) is the 2jth cumulant of Z.
Theorem 3 caa be easily extended to the case where C not invertible. The condition on the third derivative, f" (ulX), may seem strong, and it is not hard to construct examples of random variables Z where this fails for suitable contrast functions (see (6) ). However, for such settings, hard to
separate Z from Gaussian. See the Appendix A.4.I for more details.
4.2 Local convergence
In this section, establish local convergence result for the power iteration-based method described 04
in Eq Define Vt E IR, [d], the functions qi (t) := f
21). Then, without loss of Dis
generality, we have the following result for the first corner:
Theorem 4. Let a: B-1u; and a" = e1. Let the contrast function f(([), satisfy assumptions
(a), (b), (c), and (d). Let B = |- 1-122 1 12]] and define C1, C22 C3 > 0 such that Vi E [d].
supzEB
[x((a)] lo((z)) (x)
< 1. Define e = || [Belll
BlI max
Camogl
12 (at)
19: (a:))
(af)
Begl
Let |a00 - 221 R, R S max
1) Then, we have VI 2 1, 22 5e
a a²1
S
||: a"1l
Therefore, we can establish linear convergence without the third derivative constraint required for global convergence (see Theorem 33. by assuming some mild regularity conditions on the contrast functional and sufficiently close mitiialization. The proof is included in Appendix Section A.1.2
Remark 3. Let a B- denote the demixed direction, Theorem 4 shows that if the initial un is such that an is close to the ground truth a (which is WLOG taken to be e1), then there is geometric convergence to a". Ig1 (a]) and lg (at) essentially quantify how non non-gaussian the corresponding independent component is since they are zero for a mean zero Gaussian. When these are large ox2 quantities, is small, and the initialization radius |1:00 -x" 2|2 can be relatively larger.
5 Experiments
In this section, provide experiments to compare the exxed-point algorithms based on the character- istic function (CHF), the cumulant generating function (CGF) (Eqs and 5) with the turtosis-based algorithm (PEGI-Ka [49). We also compare against noiseless ICA algorithms FastICA, JADE,
and PFICA. These six algorithms are included as candidates for the Meta algorithm (see Algorithm
MMTLAB implementations (under the GNU General Public License) can be found at FastICA and JADE
The code for PFICA was provided request by the authors.
7

---

D. We also present the Uncorrected Meta algorithm (Unc. Meta) to denote the Meta algorithm
with the uncorrected independence score proposed in CHFICA (2. Table Figure provide
experiments on simulated data, and Tigure@ppon provides an application for image demixing [6]. We
provide additional experiments on denoising MNIST images the Appendix Section A.2
Experimental setup: Similar to [49], the mixing matrix B is constructed as B = UAVT, where U, V k dimensional random orthonormal matrices, and a diagonal matrix with Aa [1,3]. The covariance matrix Σ of the noise g follows the Wishart distribution and is chosen to be & &RA!. where & is the number of sources, and
random Gaussian matrix. Higher values of the noise power can make the noisy ICA problem harder. Keeping B fixed, we report the median of 100 random runs of data generated from a given distribution (different for different experiments). The ouagiiortooonnllizationn matrix for CHF and CGF is initialized as B using the mixing matrix, B, estimated via PFICA. The performance of CHF and CGF is based on a single random initialization of the vector in the power method (see Algorithm 2 Pro M2 2022 CPU with 8 GB RAM.
Our experiments were performed on a Macbook
Due to the inherent ambiguity in signal recovery discussed in Section 2 we measure error using the accuracy of estimating B. We report the widely used Amari error (2 24 II [0 for our results. For estimated demixing matrix B, define W B-1 B, after normalizing the rows of B-1 and B=1 Then we measure d dB, as
Error Metries:
Wel maxy Wel
W max; W 1g
B
(8)
Variance reduction using Meta: We first show that the independence score can also be used to pick the best solution from many random initializations. Figure[(a)) (a), the top panel has a histogram of 40 runs of CHE, each with one random initialization. The bottom panel shows the histogram of 40 experiments, where, for each experiment, the best out of 30 random initializations are picked using the independence score. The top and bottom panels have (mean, standard deviation) (0.51, 0.51 and (0.39,0.34) respectively. This shows a reduction in variance and overall better performance.
Effect of varying kurtosis: For our second experiment (see Table T. we use k 5 independent components, sample size 10 and noise power p = 0.2 from a Bernoullip) distribution, where we vary p from 0.001 to 0.5 1 I2. The last parameter makes kurtosis zero. Different algorithms
Scaled K4
994
194
95
15
5
2
0.8
0.13
Algorithm
Meta
0.007
0.010
0.011
0.010
0.011
0.011
0.0128
0.01981
0.023
CHF
1.524
0.336
0.011
0.010
0.011
0.011
0.0129
0.0213
0.029
CGF
0.007
0.011
0.011
0.016
0.029
0.044
0.05779
0.06521
0.071
PEGI
0.007
0.010
0.011
0.010
0.012
0.017
0.02795
0. 0.13097
1.802
PFICA
1.525
0.885
0.540
0.024
0.023
0.023
0.0212
0.0224
0.024
JADE
0.021
0.022
0.021
0.022
0.022
0.023
0.029
0.089
1.909
FastICA
0.024
0.027
0.026
0.026
0.026
0.027
0.02874
0.0703
Unc. Meta
1.52
0.049
0.041
0.0408
0.0413
0.0414
0.0413
0.0416
0.0419
Table 1: Median Amari enror with varying p in Bernoulli()) data. The scaled kurtosis is given as K4 : (1 - 6p (1 - p))/(p(1 p)). Observe that the Meta algorithm (shaded in red) performs par or better than the best candidate algorithms. FastICA did not converge for zero- kurtosss data.
perform differently (best candidate algorithm is highlighted in bold font) for different p. In particular, characteristic -unetion-based methods like PFICA and CHF perform poorly for small values of p. We attribute this to the fact that the characteristic function is close to one for small p. bartedis-based algorithms like PEGI, JADE, and FASTICA perform poorly for kurtosis close to zero. Furthermore, the Uncorrected Meta algorithm performs worse than the Meta algorithm since it shadows PFICA.
Effect of varying noise power: For our next two experiments, we use Uniform (- v3, v3), 3 are from Exponential(5) and 3 are from (Bernoulli (1 V and hence have zero kurtosis. In these experiments, vary the sample size (Figure 2b n fixing p 0.2, and
9, out of which 3 are from
8

---

the noise power (Figure 2ap p fixing 10", respectively. Note that with most mixture distributions, it is easily possible have low or zero kurtosis. We include such signals in our data to highlight some limitations of PEGI-K4 and show that [ choose adaptively to obtain better results. Figure shows that large noise power leads to worse performance for all methods. PEGI, JADE,
Nete
(a) Variation of performance with noise power (b) Variation of performance with sample size
with
Hatlogram of Amari Scores with best out multiple runa
62
(c) Histograms of Amari error with n = 10 and noise power p = 0.2
Figure 2: Amari error in the log-scale on y axis and varying noise powers (for 10º) and varying sample sizes (for = 0.2) on 2 axis for figures 2a and (2 respectively. For figure 2c the top panel contains histogram of 40 runs with one random initialization. The bottom panel contains a histogram of 40 runs, each of which is the best independence score out of 30 random initializations.
and FastICA perform poorly consistently, because of some zero kurtosis components. CGF suffers because of heavy-tailed components. However, CHF and PFICA perform well consistently. The Meta algorithm mostly picks the best algorithm except for few points, where the difference between the two leading algorithms is small. The Uncorrected Meta algorithm (which uses the independence score without the additional correction term introduced for noise) performs identically PFICA.
Effect of varying sample size: Figure 2b shows that the noiseless ICA methods have good per- formance at smaller sample sizes. However, they suffer a bias in performance compared to their noiseless counterparts as the sample size increases. The Meta algorithm can consistently pick the best option amongst the candidates, irrespective of the distribution, leading significant improvements in performance. What is interesting is that up to a sample size of 10a, PFICA dominates other algorithms and Meta performs like PFICA. However, after that CHF dominates, and one can see that Meta starts to have a similar trend as CHF. We also see that the Uncorrected Meta algorithm (which uses the independence score without the additional correction term introduced for noise) has near identical error as PFICA and has a bias for large
Conclusion
ICA is a classical problem that aims to extract independent non-Gaussian components. The vast literature both noiseless and noisy ICA introduces many different inference methods based

---

(a) Source Images
(b) Mixed Images
(c) Demixed Images using CHF-based contrast function (A P) 5.26 x 10 10
(d) Demixed Images using kurtosis-based contrast function (A(t, FIP) = 2.48 10
We demix images using ICA by flattening and linearly mixing them with 4 matrix B (i.id entries ~ (00 1)) and Wishart noise (p = 0.001). The CHF-daaedd method (c) recovers the original sources well, upto sign. The Kurtosis-based method (d) fails to recover the second source. This is consistent with its higher independence score. The Meta algorithm selects CHF from candidates CHF, CGF, Kurtosis, FastCAA, and JADE. Appendix Section A.2 provides results for other contrast functions and their independence scores.
Figure 3:
variety of contrast functions for separating the non-Gaussian signal from the Gaussian noise. Each has its own of shortcomings. We aim identify the best method for a given dataset in data-driven fashion. In this paper, we propose a nonparametric score, which is used to evaluate the quality of the solution of any inference method for the noisy ICA model. Using this score, design Meta algorithm, which chooses among a set of candidate solutions of the demixing matrix. We also provide new contrast functions and a computationally efficient optimization framework. While they also have shortcomings, we show that our diagnostic can remedy them. We provide uniform convergence properties of our score and theoretical results for local and global convergence of our methods. Simulated and real-world experiments show that our Meta-algorithn matches the accuracy of the best candidate across various distributional settings.
A Acknowledgments and Disclosure of Funding
The authors thank Aiyou Chen for many important discussions that helped shape this paper and for sharing his code of PFICA. The authors also thank Joe Neeman for sharing his valuable insights and the anonymous reviewers for their helpful suggestions in improving exposition of the paper. SK and PS were partially supported by NSF grants 22 2016969, 2019844, and DMS 2109155.
10

---

References
[1] Shun-Ichi Amari and J.-F. Cardoso. Blind source separation--eeippaaantrii statistical approach. IEEE Transactions on Signal Processing, 45(11):2622-2700, 1997.
[2] Shun-ichi Amari, Andrzej Cichocki, and Howard Yang. A new learning algorithm for blind signal separation. Advances neural information processing systems, 8, 1995.
[3] Joseph Anderson, Navin Goyal, Anupama Nandi, and Luis Rademacher. Heavy-tailed analogues of the covariance matrix for ica. In Satinder P. Singh and Shaul Markovitch, editors, AAAI, pages 1712-1718. AAAI Press, 2017.
[4] Sanjeev Arora, Rong Ge, Ankur Moitra, and Sushant Sachdeva. "provable ica with unknown gaussian noise, with implications for gaussian mixtures and autoencoders". In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, NIPS, pages 2384 2392, 2012.
[5] Arnab Auddy and Ming Yuan. Large dimensional independent component analysis: Statistical optimality and computational tractability, 2023.
[6] F.R. Bach and Jordan. Kernel independent component analysis. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)., volume 4, pages IV-876, 2003.
[7] Anthony Bell and Terrence J Sejnowski. An information-maaxxmizziioo approach blind separation and blind deconvolution. Neural computation, (66):1129-113) 1995.
[8] Jean-Frangois Cardoso. High-order contrasts for independent component analysis. Neural computation, 11(1):157-192, 1999.
[9] J.F. Cardoso and A. Souloumiac. Blind beamforming for non- con-gaussian signals. IEE Proceedings F (Radar and Signal Processing), -003682-770(8, December 1993.
[10] Aiyou Chen and Peter J. Bickel. Efficient independent component analysis. The Annals of Statistics, 34(6):2825- 2855, 2006.
[11) Aiyou Chen and P.J. Bickel. Consistent independent component analysis and prewhitening. IEEE Transactions on Signal Processing, 53(10):3625-3632. 2005.
[12] Pierre Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-
31 314, 1994.
[13] Pierre Comon and Christian Jutten. Handbook of Blind Source Separation: Independent component analysis and applications. Academic press, 2010.
[14] George Darmois. Analyse générale des liaisons stochastiques: etude particulière de l'analyse factorielle linéaire. Revue de I' I'Institut international de statistique, pages 2-8, 1953.
[15] M. Davies. Identifiability issues in noisy ica. IEEE Signal Processing Letters, 1 1(5))470-473, 2004.
[16] Laurent de Vito. Ica for demixing images.
https://gtthub.cmm/llv/// 2024. Accessed: 2024-05-20.
[17] N. Delfosse and P. Loubaton. Adaptive separation of independent sources: a deflation approach. In Proceedings ICASSP '94, IEEF International Conference on Acoustics, Speech and Signal Processing, volume iv, pages IV/41-IV/44 vol.4, 1994.
[18] Nathalie Delfosse and Philippe Loubaton. Adaptive blind separation of convolutive mixtures. In ICASSP, pages 1940-29433 IEEE Computer Society, 1996.
[19] Bruno Ebner and Norbert Henze. Bahadur efficiencies of the epps-pulley test for normality. arXiv preprint arXiv:2 13962, 2021.
[20] Bruno Ebner and Norbert Henze. On the eigenvalues associated with the limit null distribution of the epps-pulley test of normality, Statistical Papers, 64(3):739-752, 2023.
11

---

[21] Jan Eriksson and Visa Koivunen. cinitterristicfuctiionn based independent component analysis. Signal Processing, (3(10):2195-2200) 2003.
[22] Arthur Gretton, Kenji Fukumizu, Choon Hui Teo, Le Song, Bernhard Schölkopf, and Alexan- der J. Smola. A kernel statistical test of independence. In Proceedings of the 20th International Conference on Neural Information Processing Systems, NIPS 07, page 585-592, Red Hook, NY, USA, 2007. Curran Associates Inc.
[23] Trevor Hastie and Rob Tibshirani. Independent components analysis through product density estimation. Advances in neural information processing systems, 15, 2002.
[24] Trevor Hastie and Rob Tibshirani. Independent components analysis through product den- sity estimation. In Proceedings of the 15th International Conference on Neural Information Cambridge, MA, USA, 2002. MIT Press.
Processing Systems, NIPS 02, page
[25] Norbert Henze and Thorsten Wagner. A new approach to the bhep tests for multivariate normality. Journal of Multivariate Analysis, 62(1):1-23, 1997.
[26] A. Hyvarinen. One-unit contrast functions for independent component analysis: statistical analysis. In Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop, pages 388-397, 1997.
[27] Aapo Hyvarinen. Fast and robust fixed- point algorithms for independent component analysis. IEEE transactions Neural Networks, 10(3):6266334, 1999.
[28] Aapo Hyvärinen, J. Karhunen, and Erkki Oja. Independent component analysis. John Wiley & Sons, 2001.
[29] Aapo Hyvärinen and Erkki Oja. A fast fixed-point algorithm for independent component analysis. Neural Computation, 9(7):1483-1492 1997.
[30] Pauliina Ilmonen and Davy Paindaveine. Semiparametrically efficient inference based on signed ranks in symmetric independent component models. Annals of Statistics, 39:2448-2476, 2011.
[31] Pauliina Ilmonen and Davy Paindaveine. Semiparametrically efficient inference based on signed ranks in symmetric independent component models. 2011.
[32] Elena Issoglio, Paul Smith, and Jochen Voss. On the estimation of entropy in the fastica algorithm. Journal of Multivariate Analysis, 181:104689, 2021.
[33] M. Kac. Sur les fonctions indépendantes (i) (propriétés générales). Studia Mathematica, 6:46-58, 1936.
[34] Sergey Kirshner and Barnabás Póczos. Ica and isa using schweizer-wolte measure of dependence. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, page 464 471, New York, NY, USA, 2008. Association for Computing Machinery.
[35] Te- Won Lee, Mark Girolami, and Terrence Sejnowski. Independent component analysis using extended infomax algorithm for mixed subgaussian and supergaussian sources. Neural computation, 1 1122):477444, 1999.
[36] Peter McCullagh. Does the moment- geneeettnggnnrningggggg function characterize distribution? The American Statistician, 48(3):208-208, 1994.
[37] Erkki Oja and A Hyvarinen. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411-40 2000.
[38] Erkki Oja, Aapo Hyvärinen, and Patrik Hoyer. Image feature extraction and denoising by sparse coding. Pattern Analysis & Applications, 2:104-110, 1999.
[39] Dinh Tuan Pham. Blind separation of instantaneous mixture of sources via an independent component analysis. IEEE Transactions Signal Processing, (4(11):2768-277)) 1996.
12

---

[40] Dinh Tuan Pham and P. Garat. Blind separation of mixture of independent sources through quasi- maximumm likelihood approach. IEEE Transactions on Signal Processing. 45(7):1712- 1725, 1997.
[41] Karl Rohe and Muzhe Zeng. Vintage factor analysis with varimax performs statistical inference. arXiv preprint arXiv:2004.05387, 2020.
[42] Hao Shen, Stefanie Jegelka, and Arthur Gretton. Fast kernel-based independent component analysis. IEEE Transactions Signal Processing, 57(9):3498 -5188, 2009.
[43] Viktor Skitovitch. On property of the normal distribution. DAN SSSR, 89:217-219, 1953,
[44] Fabian J. Theis. Multidimensional independent component analysis using characteristic func- tions. In 2005 13th European Signal Processing Conference, pages 1-4, 2005.
[45] Aad W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[46] Roman Vershynin. High-Dimensional Probability. Cambridge University Press, Cambridge, UK, 2018.
[47] Roman Vershynin. High-ddimensional probabilit:: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[48] James Voss, Mikhail Belkin, and Luis Rademacher. Optimal recovery in noisy ICA. CoRR. abs/1502.04148 2015.
[49] James R Voss, Mikhail Belkin, and Luis Rademacher. A pseudo-euclidean iteration for optimal recovery noisy ica. In Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[50] James R Voss, Luis Rademacher, and Mikhail Belkin. Fast algorithms for gaussian noise invari- ant independent component analysis. C.J. Burges, L.. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[51] Martin J Wainwright. figh-diimessonal statistics: A non-asymptotic viewpoint, volume 48.
Cambridge university press, 2019.
[52] Tianwen Wei. study of the fixed points and spurious solutions of the deflation-based fastica algorithm. Neural Comput. Appl., 28(1):13-244 jan 2017.
[53] Arie Yeredor. Blind source separation via the second characteristic function. Signal Processing, 80(5):897-900) 2000.
13

---

A
Appendix
The Appendix is organized follows-
Section
A.1
proves Theorems
1 and
Section
A.2
provides additional experiments for noisy ICA
Section A. 3
provides the algorithm to compute independence scores via sequential proce-
dure
explores the third derivative constraint in Theorem and provides examples where holds
Section A.4
contains surface plots of the loss landscape for noisy ICA using the CHF-based contrast functions
Section A.3
A.1 Proofs
Proof of Theorem We will give an intuitive argument for the easy direction for k = 2. We will show that if F' = DB-1 for some permutation of diagonal matrix D, then after projecting using that matrix, the data is of the form + g' for some Gaussian vector g'. Let k: 2. Let the entries of this vector 21 + 9 and 22 + 922 where 2,, i {1,2] are mean zero independent non-Gaussian random variables and g/. i E {1. 2} are mean zero possibly dependent Gaussian variables. The Gaussian variables are independent of the non-Gaussian random variables. Let 4, E {1,2) 2) be arbitrary but fixed real numbers. Let t (2) and let Eg denote the covariance matrix of the Gaussian g'.
Assume for simplicity var(z;) = 1 for i E {1,2}. Denote by A == cov(Fx) = FSFT I Egg The JOINT part of the score is given by:
(- [T + diag( (A))t
JOINT = E 21 E [itez]] exp
2
The PRODUCT part of the score is given by
t (diag( Egg') A)t
PRODUCT EE [it] 21
[262 exp
2
is not hard see that JOINT equals PRODUCT. The same argument generalizes 2. We next provide proof for the harder direction here. Suppose that A (t = 0, i.e,
[exp Fz)] exp ( (-
t diag (FSF
t FFFF
II (exp(it, (Fz),)) exp
2
(A.9)
We then prove that F must be of the form DB for D being permutation of diagonal matrix. Then, taking the logarithm, we have
In (E lexp(it Fx)))
In (B (ex))
2 _t (diag (FSFT) FSFT){
t (diag (FEF") FEF*)
Under the ICA model we have, Bz g. Therefore, from Eq A.9 using the definition of the characteristic function of a Gaussian random variable and noting that cov (g) E, we have
In (E [exp(it" F=)]) In (E [exp(it" FB2)) In (E [exp(it [exp(it"Fg))) In (E [ixp( FBz]]) 1 FEF
2
and similarly,
In (E (xx)(( Σ In (E (FB2)), o diag (FIF*) t
14

---

Therefore, from Eq A.9
gF (t) = In E [epp(( (it
PeFFPr FBz)])
Σ In LE (ity (FBz),))
must be a quadratic function of is important note that the second term is an additive function w.r.t f1, f2, fg.. For simplicity, consider the case of l = 2 and assume that the joint and marginal
characteristic functions of all signals twice differentiable. Consider 8 (gr (t)) then
0618t2
2 8 (gF (t)) const.
(A.10)
0ti
To simplify the notation, let (1) = [e fj = Vo. == FB. The above then equivalent to
2
Σ Mix (Mitt + M2xtz) = const.
(A.11)
Note that M invertible since F, are invertible. Therefore, let t, then
Σ M2x const.
which implies that either M2 = 0 or f" (sx) = const, for k = 2. For any k E {1, 2). since 2j non-Gaussian, its logarithmic characteristic function, i.e. f& cannot be a quadratic function, so
Ma
which implies that each column of M has zero. Since M is invertible, thus M is either diagonal matrix or permutation a diagonal matrix. Now to consider 2, we just need apply the above k = 2 argument to pairwise entries of (4,.... (x] with other entries fixed. Hence proved.
D
Proof of Theorem
We start with considering the following constrained optimization problem
sup / (CC''ul) u iu=1
By an application of lagrange multipliers, for the optima Uopt. we have
vC -C Uopt VC uopt PP) C
Ukpt
with multiplier RR. This leads to fixed-point iteration very similar the one [49]. which the motivation for considering such optimization framework. We now introduce some notations to simplify the problem.
Let Ba, 2 := di and df == var(z,) at where a; := (.)( Then,
(C- uu-)) aD
sup f aD=
sup
c==l
We will use f(u P) or f(ulx) where P interchangeably. By Assumption I (a), that (D-lalz) = E / d, )x) For simplicity of notation, we will use ha(os/d;) = (as/d.l=s).
since the functional form can be different different random variables z;. Therefore, we seek to characterize the fixed points of the following optimization problem-
sup
h (ay//dd
(A.12)
s.t
1
d
1=1
d, + 0 ViE [k)
15

---

where a, d E R$. We have essentially moved from optimizing in the (...)c- space to the (-..)--
space. We find stationary points of the Lagrangian
h
L(a, X)
14
C4
Σ
(A.13)
d
di
1=1
i=1
The components of the gradient of [(a, A) w.r.t given
C(a,
_
1
h
Bj
da,
dj
dj
dj
At the fixed point (a, A) we have,
Vj E [k]
h2
Xaj = 0
dj
By Assumption (b), for a; 0, the above is automatically zero. However, for {j: : 0}, have,
A
(A.14)
0g
So. for all {j : ay # 0}. we must have the same value and sign of n, hi (2)) By assumption in the theorem statement, 1i"" does not change sign on the half-lines x > 0 and x < Along with Assumption (b), this implies that
A or [0, oo), sgn(h:(z)) sgn(M((x)) = sgn(h sgn(n:"" (x)" = =
(A.15)
V E (-c0, 0J. sgn(h;(x)) = -gg(((((zr) = sgm(h" (z)) = sgn (z)) = K2 (A.16)
where K1, {-1, 1 are constants. Furthermore, we note that Assumption D (d) ensures that (2) is symmetric function. Therefore,
R, sg)((((z))) sgn(h (z)) K. { 1, 1}.
Then, since d; = h (u;).
sgn (d) = sgn (h+( Vx e R
(A.17)
Now, using A. 14
aj
sgn(A) A) = sgn
sgn(a;)
di
)
aj
C3j
sgn
x
sgn
sgn(a,), using Eq 4.15 A.16
dg
sgn (dz)
sgn
hj
(Kj
d
1, using Eq
A.17
(A.18)
Keeping this in mind, we now compute the Hessian, E B of the lagrangian, L(ar, A) at the fixed point, (a, A). Recall that, we have h²(0) and (0) 0. Thus, for {i a; = 0}.
H -/44
(A. 19)
This implies that sgn (d: H.) sgn (A) -1 for (i or 0}. using Eq A.18
For (i: a4 # 0}. we have
Hig = do,0o, c(a,2)
8²
= 1(i 11==j) =j
R
di
dz
dg
1/
1(;= j)
for + 0. using A.14
d3
1
di )
1( = 5)
1/
for a; *
d,a.
d,
di
16

---

We consider the pseudo inner product space nn- for optimizing a. Furthermore, since we are in this pseudo-inner product space, we have
(v. Hu) D-1 = -D Hu
So we will consider the positive definite-ness of the matrix H == D- to characterize the fixed points. Recall that for a differentiable convex function f. have V æ. y dom(f) S R
f(y) 2 f(x) + Vf(x)* (y x)
Therefore, for {i a; * 0}. we have
ai (3 d;
O3
sgn (a) = sgn (d:) sgn (daa;) sgn
h1g
d:
d;
sgm (a:) sgn
h"""
ai
using
convexity/concavity of h' (.).)
di
sgn (a:) x sgn
M/
using A.15 and A.16
d;
sgn (a:) sgn (A) x sgn (a:)
using A.14
sgn (A)
using Eq A.17
(A.20)
Let S = (i: 0, # 0}. We then have the following cases
1. Assume that only one o; is nonzero, then Vo orthogonal this direction, (v, Ho) D- p-1 by Eq A.19 Thus this gives local maxima.
2. Assume more than 0g nonzero, but |S|< k. Then we have for i & S, H3j < 0 using
A.19 For i S, Ha 0 from Eq A.20 Hence these are saddle points.
3. Assume we have = [A:]. i.e. Vi, o4 o O. In this case, H > 0. So, we have a local minima.
This completes our proof.
Theorem A.1. Consider the data generated from the noisy ICA model (Eq Z). If f(u is defined as in Eq 4 or 5 we have V f (u) P)) = BD,.B", for some diagonal matrix Dw, which can be different for the differerent contrast functions.
Remark 4. The above theorem is useful because it shows that the Hessian of the contrast functions based on the CHF (eg oor CGF 5) is of the form B BDB where D is some diagonal matrix. This matrix can be precomputed at some vector and used as the mairix C in the power iteration update (see eq 3 in Algorithm of [49].
Proof of Theorem A.7 First, we show that the CHF-based contrast function (see Eq 4, the Hessian, of the correct form.
CHF-based contrast function
[(u|P) log E eexp(iu" Bz) + log Eexp(-iu" Bz) + u" B diagla)
where diag(a) is the covariance matrix of z. For simplicity of notation, define d(z;u) = (iup Bz)].
Texp(iuu Bz)a]
E Texp( inu" 2)2²] 0(2; -u)
Vf(u = =iB
+ 2B diag(a) BT
((z;()
p(u;))
17

---

Define
E [exp(iu' Bz)z2²
H (u 2)
=))uu;
(22; u)
Taking second derivative, we have:
[u)() B (--=B((H((( (- Hz; -u) + diag(a)) B²
All we have to show this point that Huu; z) diagonal. We will evaluate the k; l entry. Let Bg denote the 1t column of B. Let Vi = u² B jEh,E 2j- Using independence of the components of z, we have, for 1: + E.
E [exp(iu) (iup Bx2g iu"'eez Bez + Kxee
E |Texp( iu BB2x) E (ixp(iu) BBze)
((2;u)
E [exp(inu (iu Bxzx) (exp(iuu Bear)]
(A.21)
Now we evaluate:
[exp(iu' Bz)zk] ( u)
E [exp(ii'' iuu" BBZR) [exp( itu" BB2k)]
e (u(u 2)
(A.22)
Using Eqs A.21 and A.22
that indeed Hu: 2) is diagonal.
Now, we prove the statement about CGF-based contrast function. CGF-based contrast function
We have,
exp
(u' Bz) Bz]
V (u)P)-
B diag (a) BTu.
E (exp (u B2)]
|exp (u" Bz) Bzz"
E eep (u Bz) Bz E [exp (u Bz) xT [²
V2
(u)P)
B diag (a) B
E lexp (uT Bz)]
E eepp (u² B2)]²
E exp (u Bz) 2z E (exp (u'" Bz)
E eex (u B2) 2] E [exp (u Bz) 277
diag (a)
BT
[exp (u Bz)]²
exp (u" Bz)
Covariance of P(=).
E (exp (u. Bz)]
exp appuu"n B2)
The new probability density IP (z) { are independent, and the new tilted density also factorizes over the 2, s, therefore, the
is an exponential tilt of the original pdf, and since
Elexpp B2)
covariance under this tilted density is also diagonal.
A.1.1 Uniform convergence
In this section, we provide the proof for Theorem First, we state some preliminary results about the uniform convergence of smooth function classes which will be useful for our proofs.
A.1.1.1 Preliminaries
Let D : suPo,ØET (0.0) denote the diameter of set T, and let Nxx (6; T) denote the 8-covering number the px metric. Then, we have the following standard result:
Proposition A.2. (Proposition 5.17 from (57)) Let (Xe,0 E T} be a zero-mean subgaussian process
with respect to the metric px. Then for any å E [0, DJ such that N (6;
2 10, we have
sup (Xo X²) 0.0eT
< 2E
sup PevTTxppxss
(Xo Xz)
+ 44 D2 / log (Nx (6:T))
18

---

Remark 5. For zero-mean subgaussian processes, Proposition A.2 implies, E (supoer X6l 1 E Se (Xo -X;)
from (57)) For any b-uniformly bounded class of functions C, any positive integer n 2 1, any scalar O, and a set of i..d datapoints X (+) [ijjn] we have
Proposition A.3. (Theorem 4.10 from
sup SEC
1
/ (x(1)
- -E(
2 2 Ex,
sup SEC
Σ a (x((e)
+
==1
with probability at least 1 exp
n 26
Here (Eilietn) are i.i.d rademacher random variables.
A.1.1.2 Proof of theorem 2
For dataset {x(") [ie[m] xx(() E IR,, consider the following definitions:
(t, F(P) Ex [exp(it" Fx)] o(t,
1
Σ
exp (it" litt Fxli))
(A.23)
n j=1
1
(t, FI))
II En [exp(it
((t,
exp(it;(Fxx1))
(A.24)
j=l
Let Att, F) be the empirical version where the expectation is replaced by sample averages. Now define:
A P)) lo(t, P) exp (-t" diag(FSF"))) (t) (t. FIP) exp (-t
Att, (()) = ((tt,FP) exp (-t" diag(F ((,,F(P) exp FSE"))))
Theorem A... Les F = {F Rkk |F 1}. Assume that x subgaussian()). We have:
BLEN A(t Op (1 ( |
log2(
sup ((E,N FEF
where Cs := max(1a k1 log (n) Tr(S)).
Proof. Define & (t |||s 2 log n. We will the fact that P(E9) 2/n (Example 2.12, 51). Also note that by construction, At,, FIP) < 1. Observe that:
sup EIENOO.L F(P,F) E,E FEF
sup PEF BLEN(O,
ELEN Att F Att,F(P)
sup FEF
Att, F) - A(t,F())
< BLEN(U.J))
sup
Att, Att, F P3|
+ EIEN(O
sup FEF A(t, ((t, -(, 180 P(E))
PEF
< BIEN(O.L))
sup PEF
At, P) Att F P)
3
+ 2/n
h2
& mmxx(k o STI) logg((((gn (nC*)
=pp
+ 2/nn, using Theorem A.5
n
The second inequality follows from Jensen's inequality, the fourth follows from R[(sp(()) S sup(EE()).
19

---

Theorem A.5. Ler ;= = E R | 1}. Assume that æ ~ subgaussian((), We have:
[t, P) A(; Op [It|ly m|ss o log((CC))
sup FEF
7
where C == ma]
Proof.
- ((t,FlP) ott, FIP) exp (-t" diag(FSFF)))
+ fttt exp (-t"
- cxp (-t" diag( diag(FFFF))t)
+ (t, FIP)
(tt, exp (-t"
+ (t, FIP)
exp (-t' - - (-FFSF*+)
Finally, for some S' AS (1 A)S.
lexp(-
t' aop(-t"'o
FSFFFtt) - exp( t FSF t) = | Os
s 5))
< expl - --- FS t F(S F
$ |1111115- $11= 11:11p00pppp o² ||||| v / k
where the last result follows from Theorem 4.7.1 in 47 Now, for the second term, observe that:
((t.FIP) (exp ( (-t" diag(FSF")) - exp (-t" diag(FSF")) 1t) I exp (-t"
exp (-t" diag(F($ -)F=)(t) -
(A.25)
We next have that, with probability least 1 1/n.
|iiag (F(S (((S-S)(T\[ || s (|| S o² |||||
log n
Thus with probability least 1/n, using the inequality |1 1 2/xl Vo E [- leads to:
Eq A.25
p(t,, FIP) cxp (-t" diag(F$F*)))) exp (-t" diag(FFF")))) 1t)) <20|111/ 0z (|SS
1: log a
Note that the matrices diag F(S F) and FSFT are positive semi-definite and is unit-norm. Observe that, using Lemmas A.6 and A.7
we have:
sup FEF
At, P) At. FI))
(T((S) log (nC;)
log n
122 | ||og ||tIly
<0p
|IeIly
+ Op ||t||o2 ||S|||
+ Op
S||SS mmx (,, 04 |||) log(nCr)
Op
||0|||
20

---

Lemma A.6. Define F == {F E R ||F||s 1}. Let (2(1) sem be i.idd samples from
subgaussian( 6) distribution. We have:
=Op (IeIN)
k: (T()S) log (nCr)
explit' Px(i)) E [exp(it' Fx)]
sup FET
where C max(1, | [t|1)Tr(SS).
Proof. Let ot, (1) = explit" Fx()). Next, we note that || I imply |F I ||:]|- Therefore,
sup FEF 7 Σ exp(it" Fx(1)) Exr [exp(it" Fx)]
exp(in' x0)))()))) Fz [exp(<u> [))]
sup MERSS|
M |
Let fx (u) = E, =E(uux We will argue for unit vectors u, and later scale them by ||E||_ Define B₁ = [u E IR* ||ul| S 1]. <(u:x) = exp(inu x) and let
d(u, u)) = Σ (E(U; - E(u'; (1))
Then have,
|| exp(iu" x)1l2 = |||||| ||n]l=sinnu"" x) + cos(u s ||>||
Then,
(()))(0)) - ((t;)(())
s min
( E(u; x (1))) 2111,2) ,2)
min [1120) u'll .l.)
: Let S = E1 (() (z(1)² T /n and nn = n Tr
We next have,
D = max d(u, 2²)² s min 4n, max Σ 1l 10 1111
< 4 min [n, Tr
Next, we bound the covering number N8;; BR, du). Note that N(;; Bx. du) I N(8/ VT: Bx, (1-112) since,
d d u²)² Σ (E(u;z()) =('':z())) (9))
(12( z2'12
Tr (x(() (z(1)) 7) 1/ee u'l1²
Using Cauchy-Schwerd Schwarz, we have:
fx(()) fx(2x)) Σ xl E(u; (1))ss Vnd(u, u')2
21

---

Therefore, have (δ; Bkg du) 1
Tm
Therefore, using Proposition A.2
Ez.c
sup fx( fx(2')l
- 2
sup (fx(u) - fx(((()) + 4E D Dylog log (o;; Bk du)
FEF
a(u,au))
(3
$ 2
inf 620
8yn + 8V 8/ min fn, Tn Tn)
2k
{
0 Ex v /k: miiinn, Tn) log (max(Tn, ni))
setting 6 Vmin(1, Ta/n)
vlnO (Vmin(1, Tr(S)) log (n max(1, Tr))))) Cauchy-Schward inequality
Now we recall that || S | since vectors appear as t (T((() this simply leads scaling and ||||| by ||t]] Dividing both sides by we have
k V
Eg,e
fup (t) - fx(("))
(Vmin(1) min | [t]))T((S) log (n max(1, (T(r(r)))))
FEF
Thus, using Proposition
A.3 and using the definition of Cr, we have,
V h o (vitt log (nC+)) = =11e1l ( log (nC:)
sup PEF
(t, FIP) - ptt, P)]
Lemma A.7. Let F== = {F E kxk || 1}. We have:
k2 S
sup FEF
(t, FIP) - F()F F SOp
||e|||
n
Proof. Let BB == {u R ||u||| S 1 Now define bh F:z)
F z)). Also define
(F) E, er/l (.)( Thus, using the same argument as in Lemma A.6 for t |
and (1) + 11,0
Sl|S|llog
k
sup b (t, F(P) (t, Op FEF
|tyly
Finally, we see that:
||| |log
k
sup PEF
(t,
$ Op
k |||S|llog
(v
(1eel1 |>l|S||| llgg
=Op
Σ 1611
=118110r
n
The above true because, for |aæl, |bx| S 1, = 1,...,k.
k- 1 a
II a
[ bi 1=1
ΣI be(0)+1 6)+1)
II
as+1
bs41l
x=j+2
j=0
A.1.2 Local convergence
In this section, we provide the proof of Theorem 4 Recall the ICA model from Eq
t = Bz + g diag (a) == cov (z), Σ = cov (g) = B diag (a) B" + Σ
22

---

We provide the proof for the CGF-based contrast function. The proof for the CHF-based contrast function follows similarly. From the Proof of Theorem A.I we have,
Vf (u)P) exp (u' Bz) Bz] B diag (a) B²u,
(exp (u" Bz)]
(u(P) I |exp (u" Bz) Baz"
E eeepp (u" Bz) Bz E2 [exp (u" Bz) x² B²
B diag (a) B
E lexp (u² Bz)]
IE exxp (u² Bz)]²
E exp (u B2) 22 [exp (u" Bz)]
E eexp (u B2) 2) B (exp (u Bz) 2²
diag (a)
BT
lexp (u" Bz)]²
(z) exp (u" Bz)
Covariance of
E (exp (u Bz]]
IP (2) exp (u" Bz)
The new probability density
is exponential tilt of the original pdf, and since
E [exp (u Bz]]
{=:1²_ are independent, and the new tilted density also factorizes over the 2,'s, therefore, the covariance under this tilted density is also diagonal.
Let := BD. We denote the jth column of B as B5, Define functions
(u) In (E Texp (u" (u" B.zz)]) Var (zi) BB,
(u) := Vr; (u)
Then fu (u)P) E -- (u). v (u(P) Eí Di (u). For fixed point iteration, Ux+1 vf(c-',,1) IV UT]P] Consider the function (C(C-tu(P) P). Let e, denote the 7th axis-aligned unit basis vector of R Since B is full-rank, can denote B B-'u. We have,
Vf u(p)=
Σ gb (C-'u)
cxp
T (C-1)² B
Bi
Var (z)) B.B..
E
cxp
uT (C-1)²
B,s2)
E
u
(B")
Do e;2;
B
Var (zi) Bee DJ B- ''
1=1
E
exp
TT
(BT)
DT ei2s
E
exp
(D)) Zi
Var (21)
B,
(Do)
E
Di exp
(Do) 2i
For & E IR, define the function ga (t) : IR +
R as
E
exp
(Do). 23
2/
qi (t)
Var (2)) (Do)ss
E
exp
(Dole
21
Note that di (0) = E [
= 0.
For t
12, .t) E Rd. let (t) =
[g (4)) 22 (12) ... qa (L,))² Rd Then,
Vf (C- = Bq (a)
23

---

Therefore, if ftu+1 = Ba++1. then
Bq (a+)
Bat+1
Bq (a4)
q (a:)
0:+1
|| Bq (ax)ll
)
Vi [d], (au+1);
[qq (a:)
e1
Bq (a")
At the fixed point, a"
and
Bei. Therefore,
[eil
[8q (ar)
9i (()z)))
vi [2, dl, (a++1);
ai 3 (ar+1);
(A.26)
BBq (a:)) 1 Be
91 ((ar);)
for = 1, (a+t1) -a
(A.27)
[B (a:)
Note smoothness assumptions on g: (.) mentioned in Theorem
Vi E [d].
1. SuPxe[-10 -|-- (t) S
2. -uPpe[-11--1la.ll) // (f) 02
3. suPbc[- 1 4/ (1) |s 63
Since Vk: ||u 1, therefore, Ak, ||a||/ || We seek to prove that the sequence (a:) converges a².
A.1.2.1 Taylor expansions
2i (1s) | (y)
Consider the function ga (v) =
for y E Rdd We start by computing the gradient,
||
Vygi (y)
Lemma A.8. Let gi (y) == 9: (y:)
then we have
[ (v) |I'
1
0q: (vi)
(2:)
O| Bq (v)
0q: (1s). Oy.
for
[Bq (1)
Oy:
[Bq (y) 112
Oq (y:)
Vysi (w)),
df (1₂)
Qe (1r) e Bd Bq (v)
for ti
|| Bg (v) ||
| (z) 1|
Proof. The derivative w.r.t yr is given as
dg: (y)
Oq. (21)
qi (vL)
i B|l (3)
Oy.
|| BBq (v) ||
Oy:
| (2) 1|2
Oy.
Oq: (11)
ai (11)
a|lBq (w)
0g: (y1)
Bqq (3)
Oy:
[[qq (z)) 1|
0q: (xt.)
Oy.
Note that
V Il Ayll2
AT Ay
A
Therefore,
0g. (v)
Oq: (ys)
qi (y6)
1
Øg: (1s)
e.
BT Bq (v)
Oys
| (z) of (v.)
Oy:
|Bq (v) 11 B (v))
Oy.
(v.) eef BT Bq (v)
(A.28)
[ (v)
[Bq (y) ||?
24

---

For j * i, the derivative w.r. yj is given as
dg. (v)
(w:)
1| Bq (y)
Ogg (Vj)
Oyj
| (v) || Ouj (yj)
θyj
4/ (2,)
qi (w)) e B" Bq (v)
=
(x)
| (v) ||a
(A.29)
Next, we bound d (1) and de (1). Lemma A.9. Under the smoothness assumptions on q:(-) mentioned in Theorem
we have for
t
[-|| B 1-11 21,, | [|2].
1. 1i (1) 21 (af)) S 02 -2 ail, loi (1) of (af) S c3 - ajjl
2. vi, 96 (t) وا cat2 2 14 (t) S calt|
Proof. First consider qi (t) and of (t) for + 1. Using Taylor expansion around t = 0, we have for some c (0,t) and Vi of
f J (c)
9: (t) = qi (0) + 14/
and
d. (t) = of (0) tq." (0)
Now, we know that qi (0)
df (0) 0. Then using Assumption 1 we have, for
B 12, || '12),
cat and
19. (t) i
(A.30)
10 (1) S cs|t]
(A.31)
Similarly, using Taylor expansion around aj T l2 for Q1 (.) we have, for some d' E (0, t)
@ (1) = @1 (af) (t - az) 0 (c') and
di (f) कi (af)
aj) 4 (c')
Therefore, using Assumption
we have, for E [- B 2| [
[2]
In (t) 1 (ai) 02 |t 05 and
(A.32)
ld (t) &i (af) cs
(A.33)
A.1.2.2 Using convergence radius
In this section, we use the Taylor expansion results for (-) and of (-) in the previous section to analyze the following functions for y E TR...
1. (y) Byll
e. 3T Bq
2. v (y:
[[ (v) 2
Under the constraints specified in the theorem statement have,
y a" 2| R R < max (02/ C3, 1}.
6: || BBr max
{
C2
1
cR <
(A.34)
Bey |l
[at)) laf (at)l
10
We start with (z).
25

---

Lemma A.10. A E R satisfying A.34
let 8 (g) := q (v) 91 (ai) 01. Then, we have
(1 - elly II) l0 (ai)] Be || S [q (v) S (1 + + o l (af) Be
2. [ (v) || 62 ly a²ll
Proof. Consider the vector o (y) := (y) ạ1 (at) ei Then,
C2 | aql for using Eq A.32 La 2 (v) for l # 1, using Eq A.30
1(0 (0)))) -
(A.35)
Note that
8 (yy 1S c2 1 a²l
(A.36)
Now consider (y). Using the mean-value theorem on the Euclidean norm have,
(BTBy (v)) T 5 (
|| [q (y) || = |m (ai) Beill || + (B) ||
(A.37)
where y (y) = pa (y) + (1 (1) 91 (ai) e1, / E (0. 1). Then,
1 B [I
(B" By (v)) T (z)
1
| By (v) (z)
B (v)
1
|| (1) (v)
BB (y)
BII ||8 (y)ll
S c2 ||]| | a²I|, using Eq A.36
S |B 11 sy-cc a=
S 1/1 (af) Be, |y ||, using Eqg A.34
(A.38)
Therefore using A.37
(1 - 1 a² lmn (af) [ei || S |Bq (v) ||S (1 + by a II) lan (af) [Bee
(A.39)
Finally, we consider the function (9: e: B" Bq (y)
|g (v)
Lemma A.11. Vy E _d satisfying (A.34). we have
1. For i = 1, - 5cll a كا Qn (a:T) (u(1) < 1 + 5c fly a²)
Be,
2. For i + 1, 1qı (aï) (y: i) (1 5e| y a" aIll)
Bel
Proof. Define 0 = elly a || for convenience of notation. We have,
B TT Bq (y) [[q (v) 1|2
e. T Be
e, B B8 (y) BT
in (aii
| (y) 12 using definition of 8 (v)
(A.40)
[8 (y) 1|2
Therefore, if i = 1, then using Lemma A.10
(1+8)* + @i ea Td B8 (y)
Ti (af ef BTT Bq (y)
+ 4i (af) ef BTT Bō (y)
<
[Bq (v) 1
Bqq (v) 1
(1 0)-
BQq (v) ||
(A.41)
Using the following inequalities-
1
2 1100, 0 2 0, and,
5 1 + 50,0
(1 + 0)2
(1 0)3
26

---

and observing that using Eg A.34
and Lemma A.I0
41 (at) ej B" BÒ y
Bei
191 (af))
[B || (v)ll
[Bq (y) 1
(1 0)* 91 (ai)) 2 Be, ||
02 [B|
|1 a²1l
(1 0)²
191 (aii)
[Be
$ (1 0)²
lyy a'Il
Therefore we have from Eq A.41
50 S (aî) (y; - 50
(A.42)
where we used 9 clly a²Il For the case of + 1, using Lemma A.10
we have
e. B" Bg"Bqle (y)
191 (af) lo (y: 19 (ai)
[ (y) ||
[eill
191 (ai)
Bq (y)
Be,
(1 0)²
Bei
Beill
$ (1 + 50)
(A.43)
|Be1]
We now operate under the assumption that Eq A.34 holds for = a and inductively show that holds for y 0+++1 well. Recall the function Si (v)
Qi (3%)
By applying the mean-value
Bg 4
theorem for (-) for the points a and a²,, we have from Eq A.26 and A.27 (a++1);; az!
1g (ax)) 9i (a*))
Vga (8,)² (a:: - aa)
for Bo
A,) Naa" X (0,1)
VV (B,) |||a: a'll
(A.44)
Note the induction hypothesis assumes that Eq A.34 is true for : an Since Vi, X E (0. 1), therefore Aq.A.34 holds for B, as well. Squaring and adding Eq A.44 for i [d] and taking square-root, we have
|a++1 a²ll S lla: - aa'l
V (B:) 11?
1=1
Ogs (v) Dy, B.
< |la0 a²ll,
(A.45)
i=1
Let us consider the expression Giz = dgi (v) زyh
Vi,j [k]. For the purpose of the subsequent
9
analysis, we define A = e |lar: a² ||. We divide the analysis into the following cases
Case 1:
Using Lemma A.8
di ((81)))
[ q1 (( BT Bq (B1) | (B,) 112
GGul= =
|Bw [1
27

---

From Lemma A.9
be ((B)))) 11l (af)) c3 ((8,), -ail
[1+ c3 ((8 ajl
1al (af)
d (aj)
||a a"ll
s lal (af))
[1+ C3
lo' (af))
s 1al (of)) (1 + 8)
and,
④n ((81),) v 11 (at) +02 1(81), - -ail
(B ail
gi (ai)
1 1 02
1/1 (aj)
||, a²'
< lm (af)
1 02
1/2 (a1)
I 91 (a])
[[1
From Lemma A.10
||Ba 11 2 (1 - 0) | (ai)l Beil ||
From Lemma A.11 and 0 S 1
41 ((B2),, et B² Bw (B1) || BBw (8,) 1|
60
I 60
Therefore,
1+ 0
lo (a:)) 0
GGul s 6
1 1-0 0
Be, lll:: (ai)
7.5c l91 (at))
1 10
|a. a=Il
since 0 <
Be |l (ai)l
Case 2: 0:1=1,j-
Using Lemma A.8 A.8
9/ (B.1)) 91 ((B)))) BT Bq (B1)
[Gi5
BB (B,)
|Bq (B,) 1|
From Lemma A.9
df (181),) s ca (B)), - -a3 s ca (at); - -azl
since a₃ = 0
From Lemma A.10
(Bi) 1| (1 - 0) |m [ Beill
From Lemma A.9
qq ((B,),) s 1gi (af) + c2 (22((1,, - aji
1 1 02 (B1), añl
Imn (ai)
1/1 (ai)
I Ign (af)l (1 0)
From Lemma A.1I
e BTT Bq (8,) Bg (B,) 1|2
(1 + 50)
Be,
<
Im (ai)
Be
28

---

Therefore,
11+0) |G15] S C3 1 1- 0 (1+ + 50)
|| |Be,
|e
(ai); -aj s 203
(at)) 03
la2 (at) |I| Beil?
l (af) Bei 1|
Case 3 =1
Using Lemma A.8
((B,), di ((B,), eî BTT Bq (B.) [[qq (B,) 11?
Gn
|| (B,) ||
From Lemma A.9
lsi ((B)))) < lai (af)l + c3 1(8,), - ail
1 | 03 ((B)) ajl
= lu' (ai)
l (ai)
$ lgi (af) (1 (1+0)
From Lemma A.10
[Bq (8,) (1 1 8) lgı (a Beil
From Lemma A.9
03 ((B)); - aai²² s 2
C3 ((at); -a
las ((B,),)1
2
From Lemma A.11
ee B" Bq (B,)
+ 50
Bq (B,)
1q1 (ai)
Therefore,
Gall
S c3
(1+ (1 50)
lg' (afi)
((a)); (aa))-a;)) -a;)²
2
1-0
q11 (ai)l²
[Be
la' (at))
$ c3 la (at)1² |[Belll
((at); a;)²
loi (ai))
S c
I(at); a;
1/1 (at))² |Beil
lo' (at)
< 02
I(az); oz7l
191 (at)) |B
Case 4: 1,
Using Lemma A.8
df ((B.).
9: ((B.), eT B" Bq (B.) |Ba (B,) 1|
|Gul
[Bq (
From Lemma A.9
((..)) S c3 (8)), - aa5l < c3 (a:t;; - aa;l
From Lemma A.10
[B (B.) 12 (1 0) loi (af)) Bei
From Lemma A.9
C3 ((B,); a;)² < c3 ((ar), - at)² - 23R -alllaaa"" aa -
l9s ((B)))) <
2
2
29

---

From Lemma A.I1
e/ B" BTBBBBQQ6 (6,)
Be,
(1 + 50)
[ (B,)
lqi (af) Be
Then,
ge ((B:),) e B² Bqq (B,)
< (1 +50) | - a²Il
|Be
C2
[[q (8,) 11
Bei
(1) (af))
| Beil
S (1 + 50) ||a a²Il
C2
(g (af)
< (1 + 50) |a: a²)
- 20
Therefore,
C3 (a.) ag
203
Ga S (1 0) la (a])
(a:); - agl
Beil
4t (af)l Beill
Case $:1 # 1
Using Lemma A.8
[q ((B,)) II of (BB)),)
qi ((B+),) e, BT Bq ((B))
Giyl
[[ ((B,)) 1|
From Lemma A.9
Jas (1B)),) C3
(B aj
C3 (a:); 0j
From Lemma A.10
|B (8,) (1 -) 00) |g) (af) Bei Beil
From Lemma A.9
(B,), - -0t)² - C3 ((a:); -,²² - call
lq: ((B)),)
(at); - az
2
2
From Lemma A.1I
e B" Bq (B.)
+1+ 50)
Be,
BBq (B:) 112
4 (ai)
|B |
Therefore,
|Giy <
2RR
1 50
Be,
(a:);
(a(((a)) -a,l
[(ar);
R5 [ee
(ar), a;) (()() a?l
1-
11
(a+))²
Beg 1
1/1 (af) |Bee 1|
A.1.2.3 Putting everything together
Putting all the cases together in Eq A.45 have
a || 10.-00
< C llar. a² where
7.5c lg1 (af)l
2cs
B
C2 lo' (af)l
+ -(
2c3
+
+
|e blqi (ai)
11
(ar)
||
|
!/ (af)1= Bee
[q) (af) [|Be
C:=
(1a
22 C2R2 |B]]
+
lel (ai)) Be
Then we have, C <
6 B 1 max
For linear convergence, we require CR 1,
0u
01 (of)
which is ensured by the condition mentioned in Theorem
30

---

A.2
Additional experiments for noisy ICA
A.2.1
Experiments with super-gaussian source signals
Table 2 shows the Amari error in the presence of super-Gaussian signals. The signals
Uni- form distribution V3-v3, v3), 2) Bernoulli 2/2 + v12 Laplace(O, 0.05) (mean 0 and standard deviation 0.05), 4) Exponential($), and 5) and 6) Student's I-distribution with and degrees of freedom, respectively. The Meta algorithm closely follows the best candidate algorithm even in the presence of many super- signals.
Table 2: Variation of Amari error with Sample Size for Heavy-Tailed distributions, averaged over 100 random runs. Noise power = 0.001, number sources k = 6.
200
500
1000
5000
10000
Algorithm
Meta CHF CGF PEGI
0.44376
0.25215
0.20222
0.11435
0.0838
1.10103
0.70823
0.52529
0.21344
0.09194
1.84266
1.51216
1.3702
0.84351
0,58753
2.27873
1.86561
1.71474
1.33709
1,23322
PFICA
0.39237
0.25468
0.21222
0.12878
0.12347
JADE
0.70174
0.39246
0.28199
0.12652
0.09686
FastICA
0.66441
0.3869
0.28419
0.13215
0.09946
A.2.2 Image demixing experiments
In this section, provide additional experiments for image-demixing using ICA. We mix images using flattening and linearly mixing them with 4 x 4 matrix B (i.id entries ~ (O 1)) and Wishart noise (p = 0.001). Demixing is performed using the opNRnot demixing matrix (see Section 2 and the results are shown in Figure A.J along with their corresponding independence scores. The CHF-bbsed method recovers the original sources well, upto sign. The Kurtosis and CGF-based method fails to recover the second source. This is consistent with their higher independence score. The Meta algorithm selects CHF from candidates CHF, CGF, Kurtosis, FastICA, and JADE.
A.2.3 Image denoising experiments
In this experiment, Noisy ICA algorithms and show that the Meta algorithm can pick the best-denoised image based on the independence score proposed in our work.
the ICA-based denoising technique proposed in 38 compare candidate
We use the noisy MNIST dataset and further add entrywise Gaussian noise with variance proportional to sin? (ci + (c1 50) to the (i.j)th pixel. Training images flattened to create 784- dimensional vector, and PCA is performed to reduce dimensionality to 25, on which subsequently ICA is performed. The original and denoised images along with their independence Figure A.2 We note that Cased denoising provides qualitatively better results, while CGF-based denoising provides the worst results. This is consistent with their corresponding independence scores.
shown in
A.3 Algorithm for sequential calculation of independence scores
In this section, we provide an algorithm for the computation of the independence score proposed in the manuscript, but in sequential manner. The detailed algorithm is provided as Algorithm 3 We assume that we have access to a matrix of the form C' T
The power method (see Eq 3)
essentially extracts one column of (up to scaling) at every step. 49 provides elegant way to use the pseudo-Euclidean space to successively project the data onto columns orthogonal to the ones extracted. This enables one to extract each column of the mixing matrix. For completeness, present the steps of this projection. After extracting the tth
31

---

(a) Source Images
(b) Mixed Images
(c) Demixed Images using Cas-dased contrast function (A P) 5.26 10
10 3).
(d) Demixed Images using Kurtosis based contrast function (A(t, F(P) 2.48 10
(e) Demixed Images using CGF-based contrast function (A(t, FIP) = 4 4 10
(f) Demixed Images using FastICA (A FIP) 7711100
32
(g) Demixed Images using JADE (A(L, F(P) = 6.8 x 10 10-3).
Figure A.I: Image-Demine Demixiin using ICA

---

Original Images
(b) CHF-based denoising (A (t. P)P) = 444 10 10
(c) Kurtosis- based denoising (A P) = 1.7 x 10-4)
(d) CGF-based denoising (A (t. P) = 9.2 x 10-")
(e) FastICA-base denoising (A P)) = 5.9 1
(I) JADE-based denoising (t,t, F) = 4.6 x 10-6) Figure A.2: Image Denoising using ICA
column ue (1), the algorithm maintains two matrices. The first, denoted by U, estimates the mixing matrix B one column at time (up to scaling). The second, denoted by V estimates B-1 one row time. It is possible to extend the independence score in Eq to the sequential setting as follows. Let (f) be the jth vector extracted by the power method (Eq until convergence. After extracting columns, we project the data using min(e + 1, k) projection matrices. These would be mutually independent we indeed extracted different columns of B. Let C = BDB" for some diagonal matrix D. For convenience notation, denote B, it. Set-
Ci
(A.46)
uli) Ctuli)
When () B,/[BBII,, 005) ) Die, datapoint. Thus
|B [B,ll Let æ denote an arbitrary
oa Die,
V( Byz + ((.))V(i:))
(A.47)
Thus the projection all other columns j > l is given by:
UV Σ
B
B5z +
Σ
Byz
j=1
j=
33

---

All data
0.122
0.11
0.044
0.02
0.55
Dot product true column
Figure A.3: independence score with errorbars from 50 random runs
where g - Fg. where some k k matrix. So have min(é, k) vectors of the form B; + 9,- and when f < i an additional vector which contains all independent random variables 2j, > & along with mean-zero Gaussian vector. Then we can project each vector to a scalar using unit random vectors and check for independence using the Independence Score defined in When = k. then all we need are the j = 1,..., . & projections on the directions identified via ICA in Eq A.47
As an example, we conduct an experiment (Figure A.3 where we mixing matrix B using the same generating mechanism in Section Now create vectors which interpolate between (B:, 1) and a fixed arbitrary vector orthogonal to B(:, 1). As the interpolation changes, we plot the independence score for direction q. The dataset is the 9-dimensional dataset, which has independent components from many different distributions (see Section 5
This plot clearly shows that there is clear negative correlation between the score and the dot product of a vector with the column (:: 1). To be concrete, when q has small angle with ((:, 1). the independence score is small, and the error bars are also very small. However, as the dot product decreases, the score grows and the error bars become larger.
Algorithm Independence Score after extracting columns. and are running estimates of B and B upto é columns and rows respectively.
Input
E E Rank U E Rx E Rixk Number of random projections M kn min(E 1,
for j in range[1, do
Y, 7x X (U.))))()))) end for
if l < then
Yeu1 X (I-VVot) VTUT)
end
for j in ra .] do random unit vector in R range[1, Ko] do for W(:.)) + Y,L
end for
Let E I.o represent row of W $ 5 cov(W)
+- Σng Sij
B E. Si
s(j) |É ex, iay) exp(-B) - -1 exxp(())))xp((-)) exp(-7)
end for Return mean(s), (tdee())
34

---

(a)
(h)
(c)
(d)
Figure A.4: Plots of third derivative of CGF-based contrast function for different datasets Bernoulli(p 0.5) A.4a Uniform(24 (-v3, v3) A.4bb Poisson(A
1) A.4c and Exponential(). A.4d
Note that the
sign stays the each half-line
We refer this function as A (X, V,V, which takes input, the data X, the number of columns é and matrices U, V which are running estimates of B and B upto & columns and rows respectively.
A.4 More details for third derivative condition in theorem 3
Theorem requires the condition "The third derivative of hu(u)) does not change the sign in the half line [0, oc) for the non- non-Gaus Gaussin random variable considered in the ICA problem.". In this section, we provide sufficient conditions with respect contrast functions and datasets where this holds. Further, we also provide an interesting example to demonstrate that our Assumption (d) might not be too far from being necessary.
CGF-based contrast function. Consider the cumulant generating function of a random variable X, i.e. the logarithm of the moment generating function. Now consider the contrast function
(t) -CGF(())va(XXee'
We first note that it satisfies Assumption [a)(d). Next, we observe that it is enough for distribution to have all cumulants of the same sign to satisfy Assumption ((d) for the CGF. For example, the Poisson distribution has all positive cumulants. dedepicsss the third derivative of the contrast function for a Bernoulli(1/2), Uniform, Poisson, and Exponential.
Logarithm of the symmetrized characteristic function. Figure A.5 depicts the third derivative of this contrast function for the logistic distribution where Assumption 1(d) holds. Although the Assumption does not hold for large class of distributions here, we believe that the notion of having same sign can relaxed up to some bounded parameter values instead the entire half line, so that the global convergence results of Theorem still hold. This belief is further reinforced by Figure A.6 which shows that the loss landscape of functions where Assumption (d) doesn' 't hold still smooth.
35

---

Figure A.5: Plots of the third derivative of the CHF-based contrast function for Logistic(O,, dataset.
A.4.1 Towards necessary condition
Consider the probability distribution function (see (36]) f(z) ke (a (cs(C)))) for constants b. c > O. For (x) to be valid pdf, we require that f()) 2 0, and I f(x) dx 1. Therefore, we have,
ke 4 (a + boos((Tx)) dx
(A.48)
Noting standard integration results, have that,
f e 2 dx = V2x and
e cos(CTx) dr 22Te 22²
(A.49)
Therefore, VIEla+be"
Some algebraic manipulations yield the following:
MGF (a + be
e 2² (a+ be
22
cos(cTt))
Therefore, In =- In(a + be Sg2
+ 1 + In(a + - cos(ctt)).
We now compute the mean, μ4, and variance o².
The mean, jr can be given as
p d( since f(x) is an even function.
The variance, a can therefore be given as
pcc 0² o²= (a=(aa² e 2-2 do +
e --2 COS(CTT)) dr
Now, note that Therefore,
1 c- 2 x2 dr 2 and 1 c- 2 cos(CTx) dr
2 2+2
2 2n12)
be x 2
2 2m - (2((1=)7))) 0² = k (av27 + be
be
The symmetrized CGF can therefore be written sym CGF(() = In(MGF II(MMGF
-22nn be 2d2 + 2 2 2In(a be 220
cos(cct))
Therefore, g(1) sym CGF(() (1
112 can be written as
atde
=ي2 cos(cat)) + bc 2 2g2
g(t) = lnna + be --22)) + 2ln(a +
+ be 02.2 42
36

---

Finally, (t) can be written as
bc n 22
g' d
2ln be 2-2 cos(cct)) +
a a be -pp 1
dt
2bere + sin(ct!) c2 a be cos(etl)
2bc
2r2
c2,3
a + be
gt) can be written
- 2 2g² cos(cct)) + be 2
2bc x - 22² o2.2 a + be
be cos(et())2
o2
and (t) can be evaluated as:
(a2 222-282 2b² -p== abe 2 cos(ctt)
g² 2 2 sin(ct()
(a + be di2 cos(cat))
Lets set a 2,b -1, 4. Then, we have that the pdf, [(x) ke - cos(4x))
ke 7 (1 (2r)) The corresponding functions are:
tf (2-e- - cos((1))
Elexp(eX)] (2 - c)
16e
s(1) = sym CGF(1) var (ar(X))2 = 22n(22 E- 21n(2 - =co((4)))
6
8e 8 sin(4()
32c
)((() 2 e cos( 4t) 2 - c-
200s(44)) (4t) c
32c
g"(() 32c
(2 e cos( [1))²
2 e
(4 - 2e + 2c cos(4t))
(I) 128e
sin (4t)
(2 - c cos((4t))
Closeness Gaussian MGF: can be seen there that g'r changes sign in the half-line. However, the key is to note that the MGF of this distribution is very close to that of Gaussian and can be made arbitrarily small by varying the parameters a and This renders the CGsed contrast function ineffectual. This leads us to believe that Assumption ((d) is not far from being necessary to ensure global optimality of the corresponding objective function.
A.5 Surface plots of the CHF-based contrast functions
Figure deplets the loss landscape of the Characteristic function (CHF) based contrast function described in Section 3.3 of the manuscript. We plot the value of the contrast function evaluated at
B- 7z
for x, y E [-1, 1, 1). As shown in the figure. We have rotated the data
y
that the columns of B align with the X and Y axes. The global maxima occur at u aligned with the columns of B.
37

---

1.0
(h)
Tatte
(c)
(d)
Figure A.6:
Surface plots for zero- eero-kurtosis (e and A.6d and Uniform (A.6a A.6b) data with n 10000
points, noise-power = 0.1 and number of source signals, k 2.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contributions are the Meta algorithm for non-parametric selection of the best noisy ICA algorithm for given distribution, along with 2 new contrast functions of independent interest.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should elearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the Experiments (Section 5 there are some noise/sample-size regimes where other algorithms may perform slightly better than our proposed contrast functions, but show that even then our diagnostic can pick out the best algorithm.
Guidelines:
38

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Every theorem statement mentions the assumptions and the location of the proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 5 provides extensive details of the experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
39

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We make the code for experiments available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines thttps://nips.cm /c/
pub Sublic//e gwwd//s.s/eee/eeg Sildde//Cooie Cwt///udeeeeeeeiieeeeeeeeeooaale wswssssoooolo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (https)
/ //nips.coom cc/ public/ ides /odeee//ooeuubbr ss i onPol imPolicy) icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
40

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Section provides extensive details of the experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Figure and 2 show error bars and error distribution respectively. Fig- ure (c)(C) show the mean over 100 runs, similar to
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experiments were performed single Macbook Pro M2 2022 CPU with 8 GB RAM.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
41

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurrr:t:eeee httpppppppppcclccc/ccccccaiaaaa.
Answer:
[Yes]
Justification:
We abide by the NeurIPS Code of Ethics in our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any models with a risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
42

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: MATLAB implementations (under the GNU General Public License) be found at CastICA and JADE The code for PFICA was provided on request by the authors.
Guidelines:
The answer NA means that the paper does
existing
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package ppppesswithseeeeo snneerswittee swithcode com/ dat aseas has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
should be provided. For popular datasets,
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets to require documentation licensing.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We only use publicly available datasets in Section
43

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have experiments with crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
44


Reviews:
[{"review_id": "o3ClDFmrnS", "reviewer": "Reviewer_Dz1S", "summary": "This work proposes a modification of the original CHFICA characteristic function to the noisy ICA case without requiring knowledge of the noise distribution parameters.\nThe modified independence score is then used to select the best out of multiple ICA methods based on the score assigned to their solutions, in a per dataset basis. In practice, an average score is evaluated over random directions t, which renders the score computationally inefficient for direct Minimization.\nThe empirical estimate of the new independence score is shown to converge to the population score if the data covariance is concentrated.\nEmpirical evidence is shown that the new empirical score correlates well with the well-established Amari error.\n\nThis work also introduces two computationally efficient contrast functions to be Maximized (CHF-based and CGF-based) for BSS, both of which do not require higher moments and, thus, work well on data with near-zero kurtosis, like Bernoulli(p) with low p.\nTheoretical analyses of local and global convergence are also included for a large class of smooth contrast functions that meet certain desirable properties (Assumption 1). Properties (a) and (c) are critical as they yield contrast functions that are not influenced by additive independent Gaussian noise.\nThe CHF contrast only requires a finite second moment and is suitable for heavy-tailed sources.\nThe CGF contrast is not appropriate for heavy-tailed sources.\nGlobal convergence: The Hessian of both contrasts is shown to have the form C = BDB^T. The maxima of the Hessian-adjusted contrasts is shown to occur at B^-1 u = e_i for positive elements of D. Fast sequential power method optimization in a pseudo-Euclidean space defined by C^-1 was shown by [48]. Before optimization, C need only be computed once for some random vector u and just reused during the power iteration updates. A requirement of no sign change of the third derivative of the contrast over the half-lines is required to sufficiently distinguish between Gaussian and near-0 kurtosis source distributions.\nLocal convergence: Linear convergence is attainable without the third derivative requirement. Geometric convergence is attained at some low epsilon based on how non-gaussian the source is.\nContrary to prior work on cumulant-based methods, convergence of contrasts meeting Assumption 1 was established based on the characterization of the contrasts in the pseudo-Euclidean space and not on the convergence of the power method.\n\nExperimental results support Meta's ability to select the optimal solution over various algorithms, showing variance reduction in the estimates, uniform performance across high and near-0 kurtosis.\nEvaluation of the CHF and CGF contrasts show that CHF outperforms other methods at higher noise, but only at sample size n > 20000.\nThe Meta algorithm picks up the slack of CHF by selecting the competing PFICA solution when n < 20000.", "questions": "Questions were included along with the weaknesses.", "limitations": "Yes, the authors addressed the limitations of the work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Originality :\n- The work introduces a somewhat novel combination of known ideas in a reasonably novel formulation that adapts previous work to the case of additive Gaussian noise.\n- The work differs from and extends previous contributions, dealing with near-zero kurtosis sources and additive Gaussian noise.\n- The manuscript cites related work on contrast functions for BSS and noiseless ICA methods, adequately indicating the sources of inspiration.\n\nQuality:\n - The work is technically sound, with thorough proofs.\n - The theoretical claims are well supported by the experiments.\n - Strengths and weaknesses are discussed well.\n\nClarity:\n - The work is written well overall, focusing on the key points and contributions.\n\nSignificance :\n - The results are important as they establish a consistent framework to select the optimal solution over a range of BSS methods for noisy ICA based on the proposed modified independence score.", "weaknesses": "Quality:\n - The code is not friendly to readers as there is no 1-to-1 correspondence with the notation in the paper. Suggestion: improve documentation and tidy up the codebase (especially comments and function arguments) for readability.\n\nClarity:\n - Despite the well-written manuscript and thorough methodology and proofs, the paper contains many typos (both in text and proofs) and inconsistencies in notation (especially between Appendix and main manuscript) that limit the clarity. Besides fixing these typos, explicitly include some of the \"obvious\" steps currently omitted in the derivation; that would greatly improve readability. Also add some high-level intuitions as noted below.\n - Clarify the origin of new contrast functions and how they came about. It is unclear what motivated or led to these otherwise \"semi-arbitrary\" functions.\n - Define O_P(.)\n - It appears that the new independence score does not meet some of the properties in Assumption 1. Please, clarify if that is indeed the case and, if so, which properties specifically.\n - Appendix A.3: Clarify that Algo 2 is NOT specific to contrasts based on power iteration only, but any sequential method. It is suggested to make one separate sub-section covering how the projection is done and another for the Algo description itself. Also clarify what the \"Total\" (non-sequential) Algo would look like, for comparison. \n - Lines 119-122: t.T should be bold face.\n - Eq. 2: missing last t in the PRODUCT term.\n - Section 3.2: Change \"Properties of Contrast Functions 1.\" to \"Assumption 1. (Properties of Contrast Functions)\"\n - Line 175: remove \"them\"\n - Line 182: \"note\" --> \"not\"\n - Footnote 2 (page 5): add \"and remains a linear combination of z\"\n - Theorem 4, line 254: Assumption 1(d) is NOT about the third derivative.\n - Line 271: Define the value of k in this experiment. I see 11 mentioned in passing in line 786...\n - Line 277: Fix contradicting statements: \"CHF and CGF are initialized using the B estimated via PFICA\" and \"CHF and CGF is based on a single random initialization\". \n - Line 283: \"also used\" --> \"also be used\"\n - Table 1 caption: Clarify that since median is being reported, Meta is better than the best method “on average”, but on any individual experiment it is identical to the best method.\n - Line 299: refs to 2b and 2c are in flipped order.\n - Line 315: In the noise power experiment, it is stated that \"the difference between the two leading algorithms is small.\" Therefore, the same acknowledgement should be indicated in line 315 after \"CHF dominates, and one can see that Meta starts following CHF\"\n - Line 456: Assumption 1(d) is NOT about the 3rd derivative constraint...\n - Line 464: Missing ' in g.\n - Line 472: For consistent notation, it should be Delta(t,F | P) = 0, not Delta_F(t) = 0 \n - Lines 474-477: there is an extra transpose symbol at the last t (several times).\n - Line 510: (e) --> (d)\n - Notation and conditions in lines 524-529 is not consistent with Theorem 3 in the main manuscript.\n - Line 777: w_i is not defined, g_i should be ~g_i.\n - Lines 784-789: the r defined here has no relation to the r in Algo 2. Please pick another letter.\n - Algo 2:\n\t - Clarify why it is necessary to evaluate Y_j per component and why it would not be equivalent to just compute W = X V^T U^T r\n\t - for j in range[1, M] : M is not defined, but I think it would be l + 1? Also, the letter j is used for many different things inside the same loop. Change to \" for p in range[1, l + 1] \", and s(j) --> s(p).\n\t - \" W(:,j) ← Y_j r \" --> \" W(:,a) ← Y_a r \"\n\t - I suspect r is equivalent to t in Eq. 2, if so change it to t.\n - Appendix A.4: Verify that all occurrences of Assumption 1 (d) are correct (i.e., referring to symmetry not the third derivative)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind words regarding our work's originality, quality, clarity, and significance. We appreciate your detailed comments and suggestions, which we address below. All typographical and grammatical errors will be corrected in the revised manuscript and are not addressed individually here. \n\n**[Re: Code]** We will reorganize the codebase to enhance readability for the revised submission\n\n**[Re: Clarity]** In the revised version of the manuscript, we will fix typographical and notation errors, and add further details in the derivations to enhance readability and clarity\n\n**[Re: $O_P$]** $O_P$ refers to the Big-Oh notation in probability. We will include a formal definition in the revised manuscript\n\n**[Re: Occurrences of Assumption 1 (d)]** We will double-check and ensure that occurrences of Assumption 1(d) refer to the symmetry of source distributions, not the third-derivative condition\n\n**[Re: Properties of Independence Score]** The new independence score does not satisfy property (a) of Assumption 1, so we do not consider it as an optimization objective but rather treat it as a diagnostic score for the Meta algorithm\n\n**[Re: Appendix A.3]** We will reorganize the section to improve the clarity of Algorithm 2 and add further details about the total algorithm that would look like which uses this to choose the best candidate. Thank you for pointing out the wider applicability of the algorithm apart from power-method-based iterative approaches.\n\n**[Re: Y_j per component]** Thank you for pointing it out. It is indeed true that the evaluation of Y_j can be done as suggested.\n\n**[Re: Choice of k for experiments]** The value of k depends on the particular experiment and is specified in lines 289 and 297.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7SjrY1Ck30", "reviewer": "Reviewer_N8km", "summary": "The presented paper focuses on the problem of noisy ICA which remains a significant challenge in classical machine learning.\nThe authors introduces a nonparametric independent score extending the work in [21] to evaluate the estimation of the demixing matrices without requiring any prior knowledge of underlying noise distribution parameters. \nFurther. the authors propose some new contrast functions and provides a very interesting discussion on the convergence for the presented contrast functions of noisy ICA which can be also applied for cumulant based contrast functions.", "questions": "In equation 2 in the exponent of the product term \\mathbf{t} seems to be missing?", "limitations": "The methods focus primarily on Gaussian noise, which might limit their applicability in scenarios where noise distributions do not conform to this assumption.\nThe global convergence analysis heavily depends on the assumption that the noise is Gaussian. This dependency raises questions about the generalizability of the convergence results to other types of noise.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The introduction of a nonparametric independence score is innovative.\nThe paper provides an extensive theoretical analysis, including the development of new contrast functions and a detailed discussion on convergence properties.", "weaknesses": "Two main weaknesses stood out to me in the paper:\n1.  The proposed study heavily rely on Gaussian noise assumptions. The performance of suggested score might not hold in cases with different noise characteristics. Similarly, the proposed contrast functions could have limited scope of applicability in real life scenarios..\n2. The meta algorithm's effectiveness is contingent on the pool of candidate algorithms it selects from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind words regarding our independence score, contrast functions, and our theoretical analysis. We address your questions and suggestions below.\n\n**[Re: Gaussian noise assumption]** The classical noisy ICA problem adds Gaussian noise to a mixture of non-gaussian independent components. This is an illustrious problem with applications ranging from signal processing and image analysis to biomedical data analysis. Nearly all the literature we found focuses on Gaussian noise. While it will be interesting to analyze the role of non-gaussian noise, that is outside the scope of this paper. \n\n**[Re: effectiveness of Meta]** You are right. We hope that given the multitude of ICA methods we can include in the candidate pool, one can use this property as a strength of our algorithm. \n\n**[Re: $\\mathbf{t}$ missing]**: Yes, that is a typographical error. We will fix it in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3cFuM5S0V", "reviewer": "Reviewer_c27m", "summary": "This paper proposes a nonparametric score to adaptively pick the best noisy ICA algorithm from a set of candidates. This “independence score” is based on the characteristic function-based objective (CHF) introduced by Eriksson&Koivunen in 2003. \n\nIn practice, this independence score evaluates the inverse mixing matrix obtained by an ICA algorithm without requiring accessing to the true sources. \n\nIn addition, the paper proposes some new contrast functions and algorithms and present simulation results showing the effectiveness of the proposed independence score.", "questions": "Regarding assumption of Theorem 2 (sub-Gaussianity). I understand that assumption is needed to guaranty the concentration of the norm operator, which is a rather strong assumption. Do you think is possible to relax that assumption and keep the method working? Have you make any experiments with super-Gaussian sources, for example?", "limitations": "Yes, the experimental results section discuss some limitations of the proposed algorithm in highly noisy scenarios.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tSolid theoretical approach that justifies the proposed independence score and the new derived objective functions\n\n•\tGood review of relevant previous studies on ICA and noisy ICA.\n\n•\tExperimental results include synthetically generated sources aswell as real data (images, MNIST) synthetically mixed.", "weaknesses": "•\tThe proposed approach focused only in the recovery of the proper unmixing matrix (B^{-1}) while in noisy ICA, the final goal is to recover the clean sources. There is no discussion about how to recover the sources after the proper unmixing matrix is obtained.\n\n•\tThe experimental part focused on synthetically mixed signals only. The paper would be greatly benefited from the inclusion of some real-world source separation problem.\n\n•\tTheorem 2 assumes Subgaussian mixtures, which ensures the concentration of the sample covariance matrix. This assumtion is rather strong and it would be good to analyze what happen if it is not met.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind words regarding our theoretical approach, literature review, and experiments. We address your comments, suggestions, and questions below:\n\n**[Re: Recovery of source signals]** We address this concern in lines 60-65 of our paper. Under the noisy ICA model (Eq 1), both $\\mathbf{x} = B(\\mathbf{z}+\\mathbf{g’}) + \\mathbf{g}$ and $\\mathbf{x} = B\\mathbf{z} + (B\\mathbf{g’} + \\mathbf{g})$ for Gaussian $\\mathbf{g}’$ are indistinguishable (see Voss et al (2015) pg 3). This makes recovery of the source signals impossible in the noisy ICA setting, even if the mixing matrix is known exactly because part of the noise could always be interpreted as part of the signal while preserving the model definition.  Voss et al. (2015) propose learning the source signals by maximizing the Signal-to-Interference-plus-Noise ratio (SINR). Typically, in all related papers, algorithms are compared using the accuracy of estimating the $B$ matrix. But, with the estimated $B$ matrix one can obtain an SINR optimal estimation of the signals as in the previous citation, which we will clarify. \n\n**[Re: Experiments with real-world source separation data]** Like many previous methodological papers, we focused on synthetically mixed real data. Having access to the ground truth signals and mixing matrix allows us to better understand the quality of the solutions of the algorithms. For example, many of the available sound-based (see e.g. [a]) datasets mix different sound sources synthetically.\n\n**[Re: Subgaussianity assumption in Theorem 2]:** Yes, as long as the sample covariance matrix concentrates around the population covariance in the operator norm, our proof holds. For example, [b] shows that if $\\mathbb{E}[|X^T u|^q]\\leq L^q$ is bounded for all unit vectors $u \\in \\mathbb{R}^{d}$ (Eq 1.6), then the covariance matrix concentrates at a rate $(\\frac{d}{n})^{\\frac{1}{2}-\\frac{1}{q}}$. Under such distributions, which are more general than sub-gaussians, Theorem 2 will hold with a different error rate. We will clarify this. Our experiments include exponential source signals, which do not follow the subgaussian assumption. We have added new experimental results (see pdf) with other super-gaussian sources (Laplace, Student’s t with 3, 5 degrees of freedom, and exponential) to answer your question. The Meta algorithm closely follows the best candidate algorithm even in the presence of many super-gaussian signals, which reflects that the sub-gaussianity assumption in Theorem 2 is not critical. \n\nReferences:\n\n[a] Massachusetts Institute of Technology. \"Independent Component Analysis (ICA) Benchmark.\" MIT Media Lab. Accessed August 6, 2024.\n\n[b] Vershynin, Roman. \"How close is the sample covariance matrix to the actual covariance matrix?.\" Journal of Theoretical Probability 25, no. 3 (2012): 655-686.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fWP3ArYePo", "reviewer": "Reviewer_uqTT", "summary": "The authors consider the noisy ICA problem. They first propose a numerical objective function that can be used as a guide to assess the quality of an existing ICA algorithm. This function is not suitable for optimization and therefore, the authors propose to use it in a meta-algorithm, where the purpose is to select the best ICA solutions out of several, where each solution is possibly produced by a separate algorithm. Then, they propose new contrast functions that are specifically suited for the noisy ICA problem. Finally, they study critical points of contrast functions so as to validate their use in noisy ICA problems. They provide numerical experiments to back up their claims.", "questions": "- line 123, \"We propose to adapt the CHFICA objective using estimable parameters to the noisy ICA setting\" : I don't understand this sentence. What are the \"estimable parameters\" here? Is it $S$?\n- eqn 2 : What's the motivation for the additional terms (the second factors in both \"JOINT\" and \"PRODUCT\")?\n- eqn 2 : In the last term, is a $t$ missing?\n- Thm 2 : How does knowing this bound help? Please discuss how this theorem is useful in the text.\n- line 175, \"...in this section them, ...\" : typo?\n- line 214 : What is a pseudo-Euclidean space?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper consider the noisy ICA problem, which is a challenging problem and is arguably more realistic than the noise-free ICA problem. The paper also demonstrates via numerical experiments that the proposed meta algorithm and the contrast functions can uncover the components in an image mixing problem.", "weaknesses": "The paper touches on too many things and is not very readable. For instance, I don't see the connection between the meta algorithm and the new contrast functions. Similarly, I would have welcomed more detail on the development of the meta algorithm, and the choice of randomization in its computation. \nOccasionally, something is introduced out of nowhere and it's hard to understand the motivation. For instance, I don't see the development for the new contrast functions. \nOverall, I'd have preferred a paper that coherently and clearly explores a single idea. Currently, it reads like a collection of not clearly-developed ideas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the challenging nature and wider applicability of the noisy ICA problem and for your kind words regarding our experiments with the Meta algorithm. All typographical and grammatical errors will be corrected in the revised manuscript and are not addressed individually here. \n\n**[Re: Connection between the meta-algorithm and the new contrast functions]** - Most methods for noisy ICA center around designing an appropriate contrast function. Lines 87-103 in the paper, under “Individual shortcomings of different contrast functions and fitting strategies,” explain the different issues with different types of existing contrast functions. As explained in lines 104-110, this leads to two challenges: a) how do we adaptively pick an appropriate contrast function for the dataset at hand, and b) can we design better contrast functions? Our paper gives a full pipeline for a new adaptive algorithm for choosing from a set of contrast functions, which includes established ones and the new ones we introduce.\n\n**[Re: Development of the meta-algorithm and the choice of randomization in its computation]**\nLines 115-124 build intuition for the score used in the Meta algorithm using the noiseless case. For the noisy case, one needs to account for the characteristic function of the Gaussian noise, whose covariance matrix is unknown. The second terms in JOINT and PRODUCT essentially cancel out the additional terms using \\textit{estimable} parameters, i.e. the covariance matrix S of the data (estimated via the sample covariance).\n\nLines 148-151 discuss why we choose not to use the independence score as an optimization objective but instead use it to choose the best algorithm among candidates. This leads to the meta-algorithm in Algorithm 1. \n\nRemark 2 explains the choice of randomization in the Meta algorithm. Essentially, all related papers that use the characteristic functions in ICA or for Gaussianity testing (see the citations in Remark 2, line 155) average over $t$ sampled from a spherical Gaussian. \n\n**[Re: Development of contrast functions]** Section 3.2 (lines 163-168) provides the mathematical properties useful in a contrast function for designing provable optimization algorithms for ICA. Lines 169-172 explain what each of these properties implies. Section 3.3 then develops the new contrast functions that satisfy these important properties. \n\n**[Re: Estimable parameters]** Yes, S is the estimable parameter here. We will clarify it in the revised manuscript.\n\n**[Re: Second-factors in JOINT and PRODUCT]** See the response on the development of the meta-algorithm above.\n\n**[Re: Bound in Theorem 2]:** This bound shows that uniformly over $F$, the empirical average $\\mathbb{E}_{\\mathbf{t}\\in \\mathcal{N}(0,I_k)}\\Delta(\\mathbf{t},F|\\hat{P})$ is close to the population score. This guarantees that as long as the difference between the population scores of the two candidate algorithms is not too small, the meta-algorithm can pick up the better score. We will clarify this in the manuscript.\n\n**[Re: Pseudo-Euclidean Space]** A pseudo-Euclidean space is a generalization of Euclidean space used by Voss et. al. where the produce between vectors $u, v \\in \\mathbb{R}^{d}$ is given as $u^T A v$ where $A$ does not need to be positive definite. See also [a] for details.\t\n\nReferences: \n\n[a] Wikipedia contributors. \"Pseudo-Euclidean Space.\" Wikipedia, The Free Encyclopedia. Accessed August 6, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to first thank all the reviewers for their valuable suggestions and insightful feedback. We believe we have addressed nearly all of their main technical questions. In what follows, we will address some important points each reviewer has raised. We will correct all the typographical issues pointed out and will not address them here.

### **Re: Clarity (Reviewer Dz1S)**

 Thank you for noting that our manuscript, proofs, and methodology are well-written and thorough. In the revised version of the manuscript, we will fix typographical and notational errors and add further details in the derivations to enhance readability and clarity.

### **Re: Gaussian noise assumption (Reviewer N8km)**

 The classical noisy ICA problem adds Gaussian noise to a mixture of non-gaussian independent components. This illustrious problem has applications ranging from signal processing and image analysis to biomedical data analysis. Nearly all the literature we found focuses on Gaussian noise. While it will be interesting to analyze the role of non-gaussian noise, that is outside the scope of this paper.

### **Re: Recovery of source signals (Reviewer c27m)**

 We address this concern in lines 60-65 of our paper. Under the noisy ICA model (Eq 1), both $\mathbf{x} = B(\mathbf{z}+\mathbf{g’}) + \mathbf{g}$ and $\mathbf{x} = B\mathbf{z} + (B\mathbf{g’} + \mathbf{g})$ for Gaussian $\mathbf{g}’$ are indistinguishable (see Voss et al (2015) pg 3). This makes recovery of the source signals impossible in the noisy ICA setting, even if the mixing matrix is known exactly because part of the noise could always be interpreted as part of the signal while preserving the model definition.  Voss et al. (2015) propose learning the source signals by maximizing the Signal-to-Interference-plus-Noise ratio (SINR). Typically, in all related papers, algorithms are compared using the accuracy of estimating the $B$ matrix. But, with the estimated $B$ matrix one can obtain an SINR optimal estimation of the signals as in the previous citation, which we will clarify. 


### **Re: Subgaussianity assumption in Theorem 2] (Reviewer c27m)**

Yes, as long as the sample covariance matrix concentrates around the population covariance in operator norm, our proof holds. For example, [a] shows that if $\mathbb{E}[|X^T u|^q]\leq L^q$ is bounded for all unit vectors $u \in \mathbb{R}^{d}$ (Eq 1.6), then the covariance matrix concentrates at a rate $(\frac{d}{n})^{\frac{1}{2}-\frac{1}{q}}$. Under such distributions, which are more general than sub-gaussians, Theorem 2 will hold with a different error rate. We will clarify this. Our experiments include exponential source signals, which do not follow the subgaussian assumption. To answer your question, we have now added new experimental results (see pdf) with other super-gaussian sources (Laplace, Student’s t with 3, 5 degrees of freedom, and exponential). The Meta algorithm closely follows the best candidate algorithm even in the presence of many super-gaussian signals, which reflects that the sub-gaussianity assumption in Theorem 2 is not critical. 

### **Re: Connection between the meta-algorithm and the new contrast functions] (Reviewer uqTT)**

Most methods for noisy ICA center around designing an appropriate contrast function. Lines 87-103 in the paper, under “Individual shortcomings of different contrast functions and fitting strategies,” explain the different issues with different types of existing contrast functions. As explained in lines 104-110, this leads to two challenges: a) how do we adaptively pick an appropriate contrast function for the dataset at hand, and b) can we design better contrast functions? Our paper gives a full pipeline for a new adaptive algorithm for choosing from a set of contrast functions, which includes established ones and the new ones we introduce.

### **Re: Development of the meta-algorithm and the choice of randomization in its computation] (Reviewer uqTT)**

 Lines 115-124 build intuition for the score used in the Meta algorithm using the noiseless case. For the noisy case, one needs to account for the characteristic function of the Gaussian noise, whose covariance matrix is unknown. The second terms in JOINT and PRODUCT essentially cancel out the additional terms using \textit{estimable} parameters, i.e. the covariance matrix $S$ of the data (estimated using the sample covariance).

Lines 148-151 discuss why we do not use the independence score as an optimization objective and instead use it to choose the best algorithm amongst candidates. This leads to the meta-algorithm in Algorithm 1. 

Remark 2 explains the choice of randomization in the Meta algorithm. Essentially, all related papers that use the characteristic functions in ICA or for Gaussianity testing (see the citations in Remark 2, line 155) average over $t$ sampled from a spherical Gaussian. 

### **Re: Origin and development of contrast functions (Reviewer Dz1S)**

  Section 3.2 (lines 163-168) provides the mathematical properties that are useful in a contrast function for designing provable optimization algorithms for ICA. Lines 169-172 explain what each of these properties imply. Section 3.3 then develops the new contrast functions that satisfy these important properties.

References: 

[a] Vershynin, Roman. "How close is the sample covariance matrix to the actual covariance matrix?." Journal of Theoretical Probability 25, no. 3 (2012): 655-686.

Meta-review: This paper presents a modification of the noiseless characteristic function-based ICA to the noisy ICA without requiring the knowledge of the Gaussian noise distribution parameters. Two computationally efficient contrast functions are introduced in the paper. Reviewers claimed that the origin of these new contrast functions was not clear, but during the reviewer-author discussion period it has been clarified up to a certain point. One of serious concerns is that the paper seems to a collection of not clearly-developed ideas, rather than exploring coherently and clearly a single idea. I  believe this is partially true. During the AC-reviewer discussion period, there was a mix of support and concern. One reviewer with negative comments agreed that he/she would not mind accepting the paper even though he/she wanted to keep his/her rating as a minority. Consider all the comments raised by reviewers, I think that paper has a contribution in the direction of noisy ICA. I would like to ask the authors to revise the paper carefully, improving the clarity in the final version.
Decision: Accept (poster)