Title: Nonparametric Evaluation of Noisy ICA Solutions
Venue: NeurIPS 2024
Accepted: yes
Keywords: ICA

Independent Component Analysis

Kurtosis

Characteristic Function

Cumulant Generating Function

Blind Signal Separation

Convergence
Abstract: Independent Component Analysis (ICA) was introduced in the 1980's as a model for Blind Source Separation (BSS), which refers to the process of recovering the sources underlying a mixture of signals, with little knowledge about the source signals or the mixing process. While there are many sophisticated algorithms for estimation, different methods have different shortcomings. In this paper, we develop a nonparametric score to adaptively pick the right algorithm for ICA with arbitrary Gaussian noise. The novelty of this score stems from the fact that it just assumes a finite second moment of the data and uses the characteristic function to evaluate the quality of the estimated mixing matrix without any knowledge of the parameters of the noise distribution. In addition, we propose some new contrast functions and algorithms that enjoy the same fast computability as existing algorithms like FASTICA and JADE but work in domains where the former may fail. While these also may have weaknesses, our proposed diagnostic, as shown by our simulations, can remedy them. Finally, we propose a theoretical framework to analyze the local and global convergence properties of our algorithms.

Nonparametric Evaluation of Noisy ICA Solutions

Abstract

Independent Component Analysis (ICA) was introduced in the 1980's as a model for Blind Source Separation (BSS), which refers to the process of recovering the sources underlying a mixture of signals, with little knowledge about the source signals or the mixing process. While there are many sophisticated algorithms for estimation, different methods have different shortcomings. In this paper, we develop a nonparametric score to adaptively pick the right algorithm for ICA with arbitrary Gaussian noise. The novelty of this score stems from the fact that it just assumes a finite second moment of the data and uses the characteristic function to evaluate the quality of the estimated mixing matrix without any knowledge of the parameters of the noise distribution. In addition, we propose some new contrast functions and algorithms that enjoy the same fast computability as existing algorithms like FASTICA and JADE but work in domains where the former may fail. While these also may have weaknesses, our proposed diagnostic, as shown by our simulations, can remedy them. Finally, we propose a theoretical framework to analyze the local and global convergence properties of our algorithms.

1 Introduction

Independent Component Analysis (ICA) was introduced in the 1980's as a model for Blind Source Separation (BSS) [13, 12], which refers to the process of recovering the sources underlying a mixture of signals, with little knowledge about the source signals or the mixing process. It has become a powerful alternative to the Gaussian model, leading to PCA in factor analysis. A good account of the history, many results, and the role of dimensionality in noiseless ICA can be found in [28, 5]. In one of its first forms, ICA is based on observing n independent samples from a k -dimensional population:

where B ∈ R d × k ( d ≥ k ) is an unknown mixing matrix, z ∈ R k is a vector of statistically independent mean zero 'sources' or 'factors' and g ∼ N (0 , Σ) is a d dimensional mean zero Gaussian noise vector with covariance matrix Σ , independent of z . We denote diag ( a ) := cov ( z ) and S := cov ( x ) = B diag ( a ) B T +Σ . The goal of the problem is to estimate B .

ICA was initially developed in the context of g = 0 , now called noiseless ICA. In that form, it spawned an enormous literature [17, 39, 40, 1, 8, 27, 35, 37, 23, 11, 6, 42] and at least two wellestablished algorithms FASTICA [29] and JADE [9] which are widely used. The general model with nonzero g and unknown noise covariance matrix Σ , known as noisy ICA, has developed more slowly, with its own literature [4, 50, 49, 30, 31, 41].

In the following sections, we will show that various ICA and noisy ICA methods have distinct shortcomings. Some struggle with heavy-tailed distributions and outliers, while others require approximations of entropy-based objectives, which have their own challenges (see eg. [32]).

Although methods for noiseless cases may sometimes work in noisy settings [49], they are not always reliable (e.g., see Figure 1 in [49] and Theorem 2 in [11]). Despite the plethora of algorithms for noisy and noiseless ICA, the literature has largely been missing a diagnostic to decide which algorithm to pick for a given dataset. This is our primary goal.

The paper is organized as follows. Section 2 contains background and notation. Section 3.1 contains the independence score, its properties, and a Meta algorithm 1 that utilizes this score. Section 3.3 introduces new contrast functions based on the natural logarithm of the characteristic function and moment generating function of x T u . Section 4 presents the global and local convergence results for noisy ICA. Section 5 demonstrates the empirical performance of our new contrast functions as well as the Meta algorithm applied to a variety of methods for inferring B .

2 Background and overview

This section contains a general overview of the main concepts of ICA.

Notation: We denote vectors using bold font as v ∈ R d . The dataset is represented as { x ( i ) } i ∈ [ n ] for x ( i ) ∈ R d . Scalar random variables and matrices are represented using upper case letters, respectively. I k denotes the k - dimensional identity matrix. Entries of vector v (matrix M ) are represented as v i ( M ij ). M (: , i ) ( M ( i, :) ) represents the i th column (row) of matrix M . M † represents the Moore-Penrose inverse of M . ∥ . ∥ represents the Euclidean l 2 norm for vectors and operator norm for matrices. ∥ . ∥ F similarly represents the Frobenius norm for matrices. E [ . ] ( ̂ E [ . ] ) denotes the expectation (empirical average); x ⊥ ⊥ y denotes statistical independence. ∥ . ∥ ψ 2 represents the subgaussian Orlicz norm (see Def. 2.5.6 in [46]). X n = O P ( a n ) implies that X n a n is stochastically bounded i.e, ∀ ϵ > 0 , there exists a finite M such that sup n P (∣ ∣ ∣ X n a n ∣ ∣ ∣ > M ) ≤ ϵ (See [45]).

Identifiability: One key issue in Eq 1 is the identifiability of A := B -1 1 , which holds up to permutation and scaling of the coordinates of z if, at most, one component of z is Gaussian (see [14, 43]). If d = k and Σ are unknown, it is possible, as we shall see later, to identify the canonical versions of B and Σ . For d > k and unknown Σ , it is possible (see [15]) to reduce to the d = k case. In this work, we assume d = k . For classical factor models, when z is Gaussian, identifiability can only be identified up to rotation and scale, leading to non-interpretability. This suggests fitting strategies focusing on recovering B -1 through nongaussianity as well as independence. In the noisy model (Eq 1), an additional difficulty is that recovery of the source signals becomes impossible even if B is known exactly. This is because, the ICA models x = B ( z + r ) + g and x = B z +( B r + g ) are indistinguishable (see [48] pages 2-3). This is resolved in [48] via maximizing the Signal-toInterference-plus-Noise ratio (SINR). In classical work on ICA, a large class of algorithms (see [37]) choose to optimize a measure of non-gaussianity, referred to as contrast functions.

Contrast functions: Methods for estimating B typically optimize an empirical contrast function. Formally, we define a contrast function f ( u | P ) by f : B k ×P → R where B k is the unit ball in R k and P is essentially the class of mean zero distributions on R d . More concretely, for suitable choices of g , f ( u | P ) ≡ f ( u | x ) = E x ∼ P [ g ( u T x )] for any random variable x ∼ P . The most popular example of a contrast function is the kurtosis, which is defined as f ( u | P ) = E x ∼ P ( u T x ) 4 / E [ ( u T x ) 2 ] 2 -3 . The aim is to maximize an empirical estimate, ˆ f ( u | P ) . Notable contrast functions are the negentropy, mutual information (used by INFOMAX [7]), the tanh function, etc. (See [37] for further details).

Fitting strategies: There are two broad categories of the existing fitting strategies. (I) Find A such that the components of Ax are both as nongaussian and as independent as possible. See, for example, the multivariate cumulant-based method JADE [9] and characteristic function-based methods [21, 11]. The latter we shall call the PFICA method. (II) Find successively the j th row of A , denoted by A ( j, :) ∈ R d , j = 1 , . . . , k such that A ( j, :) T x are independent and each as nongaussian as possible, that is, estimate A (1 , :) , project, and then apply the method again on the residuals successively. The chief representative of these methods is FastICA [27] based on univariate kurtosis.

From noiseless to noisy ICA: In the noiseless case one can first prewhiten the dataset using the empirical variance-covariance matrix, ˆ Σ , of x ( i ) , using ˆ Σ -1 / 2 . Therefore, WLOG, B can be assumed to be orthogonal. Searching over orthogonal matrices not only simplifies algorithm design for fitting

strategy (I) but also makes strategy (II) meaningful. It also greatly simplifies the analysis of kurtosisbased contrast functions (see [18]). For noisy ICA, prewhitening requires knowledge of the noise covariance, and therefore many elegant methods [4, 48, 49] avoid this step.

Individual shortcomings of different contrast functions and fitting strategies: To our knowledge, no single ICA algorithm or contrast function works universally across all source distributions. Adaptively determining which algorithm is useful in a given setting would be an important tool in a practitioner's toolbox. Key challenges include:

Cumulants: Even though contrast functions based on even cumulants have no spurious local optima (Theorem 3, [49]), they can vanish for some non-Gaussian distributions. Our experiments (Section 5) show that kurtosis-based contrast functions can perform poorly even when there are a few independent components with zero kurtosis. They also suffer under heavy-tailed source distributions [3, 11, 26].

PFICA: PFICA can outperform cumulant-based methods in some situations (Section 5). However, it is computationally much slower, and poorly performs for Bernoulli( p ) sources with small p .

FastICA: [32] show that FastICA's use of Negentropy approximations for computational efficiency may result in a contrast function where optimal directions do not correspond to directions of low entropy. tanh is another popular smooth contrast function used by FastICA. However, in [52], the authors show that this tends to return spurious solutions for some distributions [52].

Contrast functions are usually nonconvex and may have spurious local optima. However, they may still, under suitable conditions, converge to maximal directions. We introduce such a contrast function which vanishes iff x is Gaussian in Section 3.3 and does not require higher moments.

Our contributions:

1) Using Theorem 1 we obtain a computationally efficient scoring method for evaluating the B estimated by different algorithms. Our score can also be extended to evaluate each demixing direction in sequential settings, which we do not pursue here.

2) We propose new contrast functions that work under scenarios where cumulant-based methods fail. We propose a fast sequential algorithm as in [49] to optimize these and further provide theoretical guarantees for convergence.

3 Main contributions

In this section, we present the 'Independence score' and state the key result that justifies it. We conclude with two new contrast functions.

3.1 Independence score

While there are many tests for independence [22, 34, 42, 6], we choose the Characteristic functionbased one because it is easy to compute and has been widely used in normality testing [19, 20, 25] and ICA [11, 21, 44]. Let us start with noiseless ICA to build intuition. Say we have estimated the inverse of the mixing matrix, i.e. B -1 using a matrix F . Ideally, if F = B -1 , then Fx = z , where z is a vector of independent random variables. Kac's theorem [33] tells us that E [ exp( it T z ) ] = ∏ k j =1 E [exp( it j z j )] if and only if z i are independent. In [21], a novel characteristic functionbased objective (CHFICA), is further analyzed and studied by [11] (PFICA). Here one minimizes | E [ exp( it T Fx ) ] -∏ k j =1 E [exp( it j ( Fx ) j )] | over F . We refer to this objective as the uncorrected independence score. We propose to adapt the CHFICA objective using estimable parameters , S , to the noisy ICA setting. We will minimize:

where S = E [ xx T ] and can be estimated using the sample covariance matrix. Hence, we do not require knowledge of any model parameters. We refer to this score as the ( corrected ) independence score. The second terms in JOINT and PRODUCT (Eq 2) corrects the original score by canceling out the additional terms resulting from the Gaussian noise using the covariance matrix S of the

data (estimated via the sample covariance). See [21] for a related score requiring knowledge of the unknown Gaussian covariance matrix Σ . We are now ready to present our first theoretical result for consistency of ∆( t, F | P ) .

Theorem 1. If F ∈ R k × k is invertible and the joint and marginal characteristic functions of all independent components, { z i } i ∈ [ k ] , are twice-differentiable, then ∀ t ∈ R k , ∆( t, F | P ) = 0 iff F = DB -1 where D is a permutation of an arbitrary diagonal matrix.

Unfortunately, F is not uniquely defined if Σ is unknown as we have noted in Section 1. The proof of Theorem 1 is provided in the Appendix Section A.1. When using this score in practice, S is replaced with the sample covariance matrix of the data, and the expectations are replaced by the empirical estimates of the characteristic function. The convergence of this empirical score, ∆( t , F | ˆ P ) , to the population score, ∆( t , F | P ) , is given by the following theorem.

Theorem 2. Let F := { F ∈ R k × k : ∥ F ∥≤ 1 } , x ∼ subgaussian ( σ ) and C k := max(1 , k log ( n ) Tr( S )) . Then, we have

This bound shows that uniformly over F , the empirical average E t ∈ N (0 ,I k ) ∆( t, F | ˆ P ) , is close to the population score. This guarantees that as long as the difference between the population scores of the two candidate algorithms is not too small, the meta-algorithm can pick up the better score.

Remark 1 (Subgaussianity assumption) . The subgaussianity assumption in Theorem 2 simply ensures the concentration of the sample covariance matrix since it is used in the score (see Theorem A.3, line 825). It can be relaxed if the concentration in the operator norm is ensured. Appendix Section A.2.1 contains experiments with more super-Gaussian source signals.

The proof of this result is deferred to the Appendix section A.1.1. It is important to note that ∆( t, F | ˆ P ) is not easy to optimize. As we show in Section 3.2, objective functions that are computationally more amenable to optimize for ICA, e.g., cumulants, satisfy some properties (see Assumption 1). The independence score does not satisfy the first property (Assumption 1 (a)). In the noiseless case, whitening reduces the problem to B being orthogonal. This facilitates efficient gradient descent in the Stiefel manifold of orthogonal matrices [11]. However, in the noisy setting, the prewhitening matrix contains the unknown noise covariance Σ , making optimization hard. Furthermore, as noted in Remark 2, in practice, it is better to use E t ∼ N (0 ,I k ) ∆( t, F | ˆ P ) [11, 21, 25] instead of using a fixed vector, t , to evaluate ∆( t, F | ˆ P ) . Any iterative optimization method requires repeatedly estimating this expectation at each gradient step. Therefore, instead of using this score directly as the optimization objective, we use it to choose between different contrast functions after extracting the demixing matrix with each. This process is referred to as the Meta algorithm (Algorithm 1).

Algorithm 1 Meta-algorithm for choosing best candidate algorithm.

2.5×10~3

Correlation ofIndependencescorewithAmari distance

2

IndependenceScore

1.5

0.5

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Amari distance from true mixingmatrix

To gain an intuition of the independence score, we conduct an experiment where we use the dataset mentioned in Section 5 (Figure 2b) and compute the independence score for B ′ = ϵB +(1 -ϵ ) I, ϵ ∈ (0 . 5 , 1) . As we increase ϵ , B ′ approaches B , and hence the Amari error d B ′ ,B (see Eq 8) decreases. Figure 1 shows the independence score versus Amari error, indicating that the independence score accurately predicts solution quality even without knowing the true B .

3.2 Desired properties of contrast functions

The following properties are often useful for designing provable optimization algorithms for ICA.

Assumption 1 (Properties of contrast functions) . Let f be a contrast function defined for a mean zero random variable X . Then,

Properties (a) and (c) ensure that f ( u | G + X ) = f ( u | X ) for non-gaussian X and independent non-gaussian G , which means that the additive independent Gaussian noise does not change f . Property (c) ensures that | f ( u | G ) | is minimal for a Gaussian. Property (d) holds without loss of generality because one can always symmetrize the distribution. 2

3.3 New contrast functions

In Section 2, we provided a discussion of the individual shortcomings of different contrast functions for existing contrast functions. Before we introduce new contrast functions in this section, we revisit the algorithmic issues posed by the added Gaussian noise with unknown Σ in Eq 1.

Prewhitening the data is challenging for noisy ICA because E [ xx T ] includes the unknown Gaussian covariance matrix Σ . [4] show that it is enough to quasi orthogonalize the data, i.e., multiply it by a matrix, which makes the data have a diagonal covariance matrix (not necessarily the identity). Subsequent work [50, 49] uses cumulants and the Hessian of f ( u ) to construct a matrix C := BDB T where D is a diagonal matrix, and then use this matrix to achieve quasi-orthogonalization. In general, D may not be positive semidefinite (e.g. when components of z have kurtosis of different signs). To remedy this, in [49], the authors propose computing the C matrix using the Hessian of f ( u ) at some fixed unit vector and then perform an elegant power method in the pseudo-Euclidean space:

A pseudo-Euclidean space is a generalization of the Euclidean space used by [48]. Here the produce between vectors u,v is given as u ⊤ Av ,

Algorithm 2 Pseudo-Euclidean Power Iteration for ICA (Algorithm 2 in [48]) ˜ B is the recovered matrix for the ICA model in Eq 1. ˜ A is a running estimate of ˜ B † .

In this section, we present two new contrast functions based on the logarithm of the characteristic function (CHF) and the cumulant generating function (CGF). These do not depend on a particular cumulant, and the first only requires a finite second moment, which makes it suitable for heavytailed source signals. We will use f ( u | P ) or f ( u | x ) where x ∼ P interchangeably to represent the population contrast function. In constructing both of the following contrast functions, we use the fact that, like the cumulants, the CGF and CHF-based contrast functions satisfy Assumption 1 (a). To satisfy Assumption 1 (c), we subtract out the part resulting from the Gaussian noise, which leads to the additional terms involving u ⊤ Su .

CHF-based contrast function: Recall that S denotes the covariance matrix of x . We maximize the absolute value of following:

The intuition is that this is exactly zero for zero mean Gaussian data and maximizing this leads to extracting non-gaussian signal from the data. It also satisfies Assumption 1 a), b) and c).

CGF-based contrast function: The cumulant generating function also has similar properties (see [53] for the noiseless case). In this case, we maximize the absolute value of following:

Like CHF, this vanishes iff x is mean zero Gaussian, and satisfies Assumption 1 a), b) and c). However, it cannot be expected to behave well in the heavy-tailed case. In the Appendix (Section A.1, Theorem A.1), we show that the Hessian of both functions obtained from Eq 4 and Eq 5 evaluated at some u is, in fact, of the form BDB T where D is some diagonal matrix.

4 Global and local Convergence: loss landscape of noisy ICA

Here we present sufficient conditions for global and local convergence of a broad class of contrast functions. This covers the cumulant-based contrast functions and our CHF and CGF-based proposals.

4.1 Global convergence

The classical argument for global optima of kurtosis for noiseless ICA in [18] assumes WLOG that B is orthogonal. This reduces the optimization over u into one for v = Bu . Since B is orthogonal, ∥ u ∥ = 1 translates into ∥ v ∥ = 1 . It may seem that this idea should extend to the noisy case due to property 1 a) and c) by optimizing over v = Bu over potentially non-orthogonal mixing matrices B .

However, for non-orthogonal B , the norm constraint on v is no longer v T v = 1 , rendering the original argument invalid . In what follows, we extend the original argument to the noisy ICA setting by introducing pseudo-euclidean constraints. Our framework includes a large family of contrast functions, including cumulants.

To our knowledge, this is the first work to characterize the loss functions in the pseudo-euclidean space. In contrast, [50] provides global convergence of the cumulant-based methods by a convergence argument of the power method itself.

Consider the contrast function f ( u | P ) := E x ∼ P [ g ( x T u )] and recall the definition of the quasi orthogonalization matrix C = BDB T , where D is a diagonal matrix. For simplicity, WLOG let us assume that D is invertible. We now aim to find the optimization objective that leads to the pseudo-Euclidean update in Eq 3. For f ( C -1 u | P ) = E x ∼ P [ g ( u T C -1 x )] , consider the following:

where we define α := B -1 u and ˜ z i = z i /D ii . We now examine f ( C -1 u ) subject to the 'pseudo' norm constraint u T C -1 u = 1 . The key point is that for suitably defined f , one can construct a matrix C = BDB T from the data even when B is unknown. So our new objective is to optimize

Using the Lagrange multiplier method, optimizing the above simply gives the power method update (Eq 3) u ∝ ∇ f ( C -1 u ) . Furthermore, optimizing Eq 6 leads to the following transformed objective:

Now we provide our theorem about maximizing f ( C -1 u | P ) . Analogous results hold for minimization. We will denote the corresponding derivatives evaluated at t 0 as f ′ ( t 0 | z i ) and f ′′ ( t 0 | z i ) .

̸

Theorem 3. Let C be a matrix of the form BDB T , where d i := D ii = f ′′ ( u i | z i ) for some random u i . Let S + = { i : d i > 0 } . Consider a contrast function f : R → R . Assume that for every non-Gaussian independent component, Assumption 1 holds and the third derivative, f ′′′ ( u | X ) , does not change the sign in the half-lines [0 , ∞ ) , ( ∞ , 0] . Then f ( C -1 u ∣ ∣ x ) with the constraint of ⟨ u,u ⟩ C -1 = 1 has local maxima at B -1 u = e i , i ∈ S + . All solutions satisfying e T i B -1 u = 0 , ∀ i ∈ [1 , k ] are minima, and all solutions with | { i : e T i B -1 u = 0 } | < k are saddle points. These are the only fixed points.

̸

Note that the above result immediately implies that for ˜ C = -C , the same optimization in Theorem 3 will have maxima at all u such that B -1 u = e i , i ∈ S -.

Corollary 3.1. 2 k th cumulant-based contrast functions for k > 1 , have maxima and minima at the directions of the columns of B , provided { z i } i ∈ [ k ] have a corresponding non-zero cumulant.

Proof. This proof (see Appendix Section A.1) is immediate since cumulants satisfy (a), (c). For (c) and (d), note that f ( u | Z ) is simply u 2 j κ 2 j ( Z ) , where κ 2 j ( Z ) is the 2 j th cumulant of Z .

Theorem 3 can be easily extended to the case where C is not invertible. The condition on the third derivative, f ′′′ ( u | X ) , may seem strong, and it is not hard to construct examples of random variables Z where this fails for suitable contrast functions (see [36]). However, for such settings, it is hard to separate Z from a Gaussian. See the Appendix A.4.1 for more details.

4.2 Local convergence

In this section, we establish a local convergence result for the power iteration-based method described in Eq 3. Define ∀ t ∈ R , ∀ i ∈ [ d ] , the functions q i ( t ) := f ′ ( α i D ii ∣ ∣ ∣ z i ) . Then, without loss of generality, we have the following result for the first corner:

-1 ∗

Therefore, we can establish linear convergence without the third derivative constraint required for global convergence (see Theorem 3), by assuming some mild regularity conditions on the contrast functional and a sufficiently close initialization. The proof is included in Appendix Section A.1.2.

Remark 3. Let α = B -1 u denote the demixed direction, u . Theorem 4 shows that if the initial u 0 is such that α 0 is close to the ground truth α ∗ (which is WLOG taken to be e 1 ), then there is geometric convergence to α ∗ . | q 1 ( α ∗ 1 ) | and | q ′ 1 ( α ∗ 1 ) | essentially quantify how non-gaussian the corresponding independent component is since they are zero for a mean zero Gaussian. When these are large quantities, ϵ is small, and the initialization radius ∥ α 0 -α ∗ ∥ 2 can be relatively larger.

5 Experiments

In this section, we provide experiments to compare the fixed-point algorithms based on the characteristic function (CHF), the cumulant generating function (CGF) (Eqs 4 and 5) with the kurtosis-based algorithm (PEGIκ 4 [49]). We also compare against noiseless ICA 3 algorithms - FastICA, JADE, and PFICA. These six algorithms are included as candidates for the Meta algorithm (see Algorithm

1). We also present the Uncorrected Meta algorithm ( Unc. Meta ) to denote the Meta algorithm with the uncorrected independence score proposed in CHFICA [21]. Table 1 and Figure 2 provide experiments on simulated data, and Figure 3 provides an application for image demixing [16]. We provide additional experiments on denoising MNIST images in the Appendix Section A.2.

Experimental setup: Similar to [49], the mixing matrix B is constructed as B = U Λ V T , where U, V are k dimensional random orthonormal matrices, and Λ is a diagonal matrix with Λ ii ∈ [1 , 3] . The covariance matrix Σ of the noise g follows the Wishart distribution and is chosen to be ρ k RR T , where k is the number of sources, and R is a random Gaussian matrix. Higher values of the noise power ρ can make the noisy ICA problem harder. Keeping B fixed, we report the median of 100 random runs of data generated from a given distribution (different for different experiments). The quasi-orthogonalization matrix for CHF and CGF is initialized as ˆ B ˆ B T using the mixing matrix, ˆ B , estimated via PFICA. The performance of CHF and CGF is based on a single random initialization of the vector in the power method (see Algorithm 2). Our experiments were performed on a Macbook Pro M2 2022 CPU with 8 GB RAM.

Error Metrics: Due to the inherent ambiguity in signal recovery discussed in Section 2, we measure error using the accuracy of estimating B . We report the widely used Amari error [2, 24, 11, 10] for our results. For estimated demixing matrix ̂ B , define W = ̂ B -1 B , after normalizing the rows of ̂ B -1 and B -1 . Then we measure d ̂ B,B as -

Variance reduction using Meta: We first show that the independence score can also be used to pick the best solution from many random initializations. In Figure 2 (a), the top panel has a histogram of 40 runs of CHF, each with one random initialization. The bottom panel shows the histogram of 40 experiments, where, for each experiment, the best out of 30 random initializations are picked using the independence score. The top and bottom panels have (mean, standard deviation) (0.51, 0.51) and (0.39,0.34) respectively. This shows a reduction in variance and overall better performance.

Effect of varying kurtosis: For our second experiment (see Table 1), we use k = 5 independent components, sample size n = 10 5 and noise power ρ = 0 . 2 from a Bernoulli( p ) distribution, where we vary p from 0.001 to 0 . 5 -1 / √ 12 . The last parameter makes kurtosis zero. Different algorithms

perform differently (best candidate algorithm is highlighted in bold font) for different p . In particular, characteristic function-based methods like PFICA and CHF perform poorly for small values of p . We attribute this to the fact that the characteristic function is close to one for small p . Kurtosis-based algorithms like PEGI, JADE, and FASTICA perform poorly for kurtosis close to zero. Furthermore, the Uncorrected Meta algorithm performs worse than the Meta algorithm since it shadows PFICA.

Effect of varying noise power: For our next two experiments, we use k = 9 , out of which 3 are from Uniform ( - √ 3 , √ 3 ) , 3 are from Exponential (5) and 3 are from ( Bernoulli ( 1 2 -√ 1 12 )) and hence have zero kurtosis. In these experiments, we vary the sample size (Figure 2b), n fixing ρ = 0 . 2 , and

the noise power (Figure 2a) , ρ fixing n = 10 5 , respectively. Note that with most mixture distributions, it is easily possible to have low or zero kurtosis. We include such signals in our data to highlight some limitations of PEGIκ 4 and show that Algorithm 1 can choose adaptively to obtain better results. Figure 2a shows that large noise power leads to worse performance for all methods. PEGI, JADE,

- Uncorrected Meta

·..CHF

Uncorrected Meta

Meta

CGF

Kurtosis

......CHF

OCGF

Meta

--JADE

Kurtosis

PFICA

--JADE

PFICA

FastICA

FastICA

10

10

10"

中

5

10

0

0.2

0.4

0.6

8'0

1.2

1.4

1.6

1.8

HistogramofAmariScoreswithbestoutof multipleruns

15

10

0

0.2

0.4

0.6

80

1.2

1.4

1.6

1.8

2

and FastICA perform poorly consistently, because of some zero kurtosis components. CGF suffers because of heavy-tailed components. However, CHF and PFICA perform well consistently. The Meta algorithm mostly picks the best algorithm except for a few points, where the difference between the two leading algorithms is small. The Uncorrected Meta algorithm (which uses the independence score without the additional correction term introduced for noise) performs identically to PFICA.

Effect of varying sample size: Figure 2b shows that the noiseless ICA methods have good performance at smaller sample sizes. However, they suffer a bias in performance compared to their noiseless counterparts as the sample size increases. The Meta algorithm can consistently pick the best option amongst the candidates, irrespective of the distribution, leading to significant improvements in performance. What is interesting is that up to a sample size of 10 4 , PFICA dominates other algorithms and Meta performs like PFICA. However, after that CHF dominates, and one can see that Meta starts to have a similar trend as CHF. We also see that the Uncorrected Meta algorithm (which uses the independence score without the additional correction term introduced for noise) has a near identical error as PFICA and has a bias for large n .

6 Conclusion

ICA is a classical problem that aims to extract independent non-Gaussian components. The vast literature on both noiseless and noisy ICA introduces many different inference methods based on a

(a) Source Images

(b) Mixed Images

variety of contrast functions for separating the non-Gaussian signal from the Gaussian noise. Each has its own set of shortcomings. We aim to identify the best method for a given dataset in a data-driven fashion. In this paper, we propose a nonparametric score, which is used to evaluate the quality of the solution of any inference method for the noisy ICA model. Using this score, we design a Meta algorithm, which chooses among a set of candidate solutions of the demixing matrix. We also provide new contrast functions and a computationally efficient optimization framework. While they also have shortcomings, we show that our diagnostic can remedy them. We provide uniform convergence properties of our score and theoretical results for local and global convergence of our methods. Simulated and real-world experiments show that our Meta-algorithm matches the accuracy of the best candidate across various distributional settings.

Acknowledgments and Disclosure of Funding

The authors thank Aiyou Chen for many important discussions that helped shape this paper and for sharing his code of PFICA. The authors also thank Joe Neeman for sharing his valuable insights and the anonymous reviewers for their helpful suggestions in improving the exposition of the paper. SK and PS were partially supported by NSF grants 2217069, 2019844, and DMS 2109155.

References

Shun-Ichi Amari and J.-F. Cardoso. Blind source separation-semiparametric statistical approach. IEEE Transactions on Signal Processing , 45(11):2692-2700, 1997.

Shun-ichi Amari, Andrzej Cichocki, and Howard Yang. A new learning algorithm for blind signal separation. Advances in neural information processing systems , 8, 1995.

Joseph Anderson, Navin Goyal, Anupama Nandi, and Luis Rademacher. Heavy-tailed analogues of the covariance matrix for ica. In Satinder P. Singh and Shaul Markovitch, editors, AAAI , pages 1712-1718. AAAI Press, 2017.

Sanjeev Arora, Rong Ge, Ankur Moitra, and Sushant Sachdeva. "provable ica with unknown gaussian noise, with implications for gaussian mixtures and autoencoders". In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, NIPS , pages 2384-2392, 2012.

Arnab Auddy and Ming Yuan. Large dimensional independent component analysis: Statistical optimality and computational tractability, 2023.

F.R. Bach and M.I. Jordan. Kernel independent component analysis. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03). , volume 4, pages IV-876, 2003.

Anthony J Bell and Terrence J Sejnowski. An information-maximization approach to blind separation and blind deconvolution. Neural computation , 7(6):1129-1159, 1995.

Jean-François Cardoso. High-order contrasts for independent component analysis. Neural computation , 11(1):157-192, 1999.

J.F. Cardoso and A. Souloumiac. Blind beamforming for non-gaussian signals. IEE Proceedings F (Radar and Signal Processing) , 140:362-370(8), December 1993.

Aiyou Chen and Peter J. Bickel. Efficient independent component analysis. The Annals of Statistics , 34(6):2825 - 2855, 2006.

Aiyou Chen and P.J. Bickel. Consistent independent component analysis and prewhitening. IEEE Transactions on Signal Processing , 53(10):3625-3632, 2005.

Pierre Comon. Independent component analysis, a new concept? Signal processing , 36(3):287314, 1994.

Pierre Comon and Christian Jutten. Handbook of Blind Source Separation: Independent component analysis and applications . Academic press, 2010.

George Darmois. Analyse générale des liaisons stochastiques: etude particulière de l'analyse factorielle linéaire. Revue de l'Institut international de statistique , pages 2-8, 1953.

M. Davies. Identifiability issues in noisy ica. IEEE Signal Processing Letters , 11(5):470-473, 2004.

Laurent de Vito. Ica for demixing images. https://github.com/ldv1/ICA_for_ demixing_images , 2024. Accessed: 2024-05-20.

N. Delfosse and P. Loubaton. Adaptive separation of independent sources: a deflation approach. In Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing , volume iv, pages IV/41-IV/44 vol.4, 1994.

Nathalie Delfosse and Philippe Loubaton. Adaptive blind separation of convolutive mixtures. In ICASSP , pages 2940-2943. IEEE Computer Society, 1996.

Bruno Ebner and Norbert Henze. Bahadur efficiencies of the epps-pulley test for normality. arXiv preprint arXiv:2106.13962 , 2021.

Bruno Ebner and Norbert Henze. On the eigenvalues associated with the limit null distribution of the epps-pulley test of normality. Statistical Papers , 64(3):739-752, 2023.

Jan Eriksson and Visa Koivunen. Characteristic-function-based independent component analysis. Signal Processing , 83(10):2195-2208, 2003.

Arthur Gretton, Kenji Fukumizu, Choon Hui Teo, Le Song, Bernhard Schölkopf, and Alexander J. Smola. A kernel statistical test of independence. In Proceedings of the 20th International Conference on Neural Information Processing Systems , NIPS'07, page 585-592, Red Hook, NY, USA, 2007. Curran Associates Inc.

Trevor Hastie and Rob Tibshirani. Independent components analysis through product density estimation. Advances in neural information processing systems , 15, 2002.

Trevor Hastie and Rob Tibshirani. Independent components analysis through product density estimation. In Proceedings of the 15th International Conference on Neural Information Processing Systems , NIPS'02, page 665-672, Cambridge, MA, USA, 2002. MIT Press.

Norbert Henze and Thorsten Wagner. A new approach to the bhep tests for multivariate normality. Journal of Multivariate Analysis , 62(1):1-23, 1997.

A. Hyvarinen. One-unit contrast functions for independent component analysis: a statistical analysis. In Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop , pages 388-397, 1997.

Aapo Hyvarinen. Fast and robust fixed-point algorithms for independent component analysis. IEEE transactions on Neural Networks , 10(3):626-634, 1999.

Aapo Hyvärinen, J. Karhunen, and Erkki Oja. Independent component analysis . John Wiley & Sons, 2001.

Aapo Hyvärinen and Erkki Oja. A fast fixed-point algorithm for independent component analysis. Neural Computation , 9(7):1483-1492, 1997.

Pauliina Ilmonen and Davy Paindaveine. Semiparametrically efficient inference based on signed ranks in symmetric independent component models. Annals of Statistics , 39:2448-2476, 2011.

Pauliina Ilmonen and Davy Paindaveine. Semiparametrically efficient inference based on signed ranks in symmetric independent component models. 2011.

Elena Issoglio, Paul Smith, and Jochen Voss. On the estimation of entropy in the fastica algorithm. Journal of Multivariate Analysis , 181:104689, 2021.

M. Kac. Sur les fonctions indépendantes (i) (propriétés générales). Studia Mathematica , 6:46-58, 1936.

Sergey Kirshner and Barnabás Póczos. Ica and isa using schweizer-wolff measure of dependence. In Proceedings of the 25th International Conference on Machine Learning , ICML '08, page 464-471, New York, NY, USA, 2008. Association for Computing Machinery.

Te-Won Lee, Mark Girolami, and Terrence J Sejnowski. Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources. Neural computation , 11(2):417-441, 1999.

Peter McCullagh. Does the moment-generating function characterize a distribution? The American Statistician , 48(3):208-208, 1994.

Erkki Oja and A Hyvarinen. Independent component analysis: algorithms and applications. Neural networks , 13(4-5):411-430, 2000.

Erkki Oja, Aapo Hyvärinen, and Patrik Hoyer. Image feature extraction and denoising by sparse coding. Pattern Analysis & Applications , 2:104-110, 1999.

Dinh Tuan Pham. Blind separation of instantaneous mixture of sources via an independent component analysis. IEEE Transactions on Signal Processing , 44(11):2768-2779, 1996.

Dinh Tuan Pham and P. Garat. Blind separation of mixture of independent sources through a quasi-maximum likelihood approach. IEEE Transactions on Signal Processing , 45(7):17121725, 1997.

Karl Rohe and Muzhe Zeng. Vintage factor analysis with varimax performs statistical inference. arXiv preprint arXiv:2004.05387 , 2020.

Hao Shen, Stefanie Jegelka, and Arthur Gretton. Fast kernel-based independent component analysis. IEEE Transactions on Signal Processing , 57(9):3498-3511, 2009.

Viktor P Skitovitch. On a property of the normal distribution. DAN SSSR , 89:217-219, 1953.

Fabian J. Theis. Multidimensional independent component analysis using characteristic functions. In 2005 13th European Signal Processing Conference , pages 1-4, 2005.

Aad W. Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

Roman Vershynin. High-Dimensional Probability . Cambridge University Press, Cambridge, UK, 2018.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

James R. Voss, Mikhail Belkin, and Luis Rademacher. Optimal recovery in noisy ICA. CoRR , abs/1502.04148, 2015.

James R Voss, Mikhail Belkin, and Luis Rademacher. A pseudo-euclidean iteration for optimal recovery in noisy ica. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

James R Voss, Luis Rademacher, and Mikhail Belkin. Fast algorithms for gaussian noise invariant independent component analysis. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Tianwen Wei. A study of the fixed points and spurious solutions of the deflation-based fastica algorithm. Neural Comput. Appl. , 28(1):13-24, jan 2017.

Arie Yeredor. Blind source separation via the second characteristic function. Signal Processing , 80(5):897-902, 2000.

A Appendix

The Appendix is organized as follows -

Section A.1 proves Theorems 1, 2, A.1, 3 and 4

Section A.2 provides additional experiments for noisy ICA

Section A.3 provides the algorithm to compute independence scores via a sequential procedure

Section A.4 explores the third derivative constraint in Theorem 3 and provides examples where it holds

Section A.5 contains surface plots of the loss landscape for noisy ICA using the CHF-based contrast functions

A.1 Proofs

Proof of Theorem 1. We will give an intuitive argument for the easy direction for k = 2 . We will show that if F = DB -1 for some permutation of a diagonal matrix D , then after projecting using that matrix, the data is of the form z + g ′ for some Gaussian vector g ′ . Let k = 2 . Let the entries of this vector be z 1 + g ′ 1 and z 2 + g ′ 2 , where z i , i ∈ { 1 , 2 } are mean zero independent non-Gaussian random variables and g ′ i , i ∈ { 1 , 2 } are mean zero possibly dependent Gaussian variables. The Gaussian variables are independent of the non-Gaussian random variables. Let t i , i ∈ { 1 , 2 } be arbitrary but fixed real numbers. Let t = ( t 1 , t 2 ) and let Σ g ′ denote the covariance matrix of the Gaussian g ′ . Assume for simplicity var ( z i ) = 1 for i ∈ { 1 , 2 } . Denote by Λ := cov ( Fx ) = FSF T = I +Σ g ′ . The JOINT part of the score is given by:

The PRODUCT part of the score is given by

It is not hard to see that JOINT equals PRODUCT. The same argument generalizes to k > 2 . We next provide proof for the harder direction here. Suppose that ∆ F ( t | P ) = 0 , i.e,

We then prove that F must be of the form DB -1 for D being a permutation of a diagonal matrix. Then, taking the logarithm, we have

Under the ICA model we have, x = Bz + g . Therefore, from Eq A.9, using the definition of the characteristic function of a Gaussian random variable and noting that cov ( g ) = Σ , we have

and similarly,

Therefore, from Eq A.9,

must be a quadratic function of t . It is important to note that the second term is an additive function w.r.t t 1 , t 2 , · · · t k . For simplicity, consider the case of k = 2 and assume that the joint and marginal characteristic functions of all signals are twice differentiable. Consider ∂ 2 ( g F ( t )) ∂t 1 ∂t 2 , then

To simplify the notation, let ψ j ( t ) := E [ e itz j ] , f j ≡ log ψ j , M := FB . The above is then equivalent to

Note that M is invertible since F, B are invertible. Therefore, let s ≡ M T t , then

which implies that either M 1 k M 2 k = 0 or f ′′ k ( s k ) ≡ const , for k = 1 , 2 . For any k ∈ { 1 , 2 } , since z k is non-Gaussian, its logarithmic characteristic function, i.e. f k cannot be a quadratic function, so

which implies that each column of M has a zero. Since M is invertible, thus M is either a diagonal matrix or a permutation of a diagonal matrix. Now to consider k > 2 , we just need to apply the above k = 2 argument to pairwise entries of { t 1 , · · · , t k } with other entries fixed. Hence proved.

Proof of Theorem 3. We start with considering the following constrained optimization problem -

By an application of lagrange multipliers, for the optima u opt, we have

with multiplier ν ∈ R . This leads to a fixed-point iteration very similar to the one in [49], which is the motivation for considering such an optimization framework. We now introduce some notations to simplify the problem.

Let u = Bα , z ′ i := z i d i and a ′ i := var ( z ′ i ) = a i d 2 i , where a i := var ( z i ) . Then,

We will use f ( u | P ) or f ( u | x ) where x ∼ P interchangeably. By Assumption 1-(a), we see that f ( D -1 α | z ) = ∑ k i =1 f ( α i d i | z i ) . For simplicity of notation, we will use h i ( α i /d i ) = f ( α i /d i | z i ) , since the functional form can be different for different random variables z i . Therefore, we seek to characterize the fixed points of the following optimization problem -

̸

where α, d ∈ R k . We have essentially moved from optimizing in the ⟨ ., . ⟩ C -1 space to the ⟨ ., . ⟩ D -1 space. We find stationary points of the Lagrangian

The components of the gradient of L ( α, λ ) w.r.t α are given as

At the fixed point ( α, λ ) we have,

Now, using A.14,

Keeping this in mind, we now compute the Hessian, H ∈ R k × k , of the lagrangian, L ( α, λ ) at the fixed point, ( α, λ ) . Recall that, we have h ′ i (0) = 0 and h ′′ i (0) = 0 . Thus, for { i : α i = 0 } ,

This implies that sgn ( d i H ii ) = sgn ( λ ) = -1 for { i : α i = 0 } , using Eq A.18.

̸

For { i : α i = 0 } , we have

̸

By Assumption 1-(b), for α i = 0 , the above is automatically zero. However, for { j : α j = 0 } , we have,

̸

So, for all { j : α j = 0 } , we must have the same value and sign of 1 α j h ′ j ( α j d j ) . By assumption in the theorem statement, h ′′′ i ( x ) does not change sign on the half-lines x > 0 and x < 0 . Along with Assumption 1-(b), this implies that

where κ 1 , κ 2 ∈ {-1 , 1 } are constants. Furthermore, we note that Assumption 1-(d) ensures that h i ( x ) is a symmetric function. Therefore, ∀ x ∈ R , sgn ( h i ( x )) = sgn ( h ′′ i ( x )) = κ, κ ∈ {-1 , 1 } . Then, since d i = h ′′ i ( u i ) ,

̸

̸

We consider the pseudo inner product space ⟨ ., . ⟩ D -1 for optimizing α . Furthermore, since we are in this pseudo-inner product space, we have

So we will consider the positive definite-ness of the matrix ˜ H := D -1 H to characterize the fixed points. Recall that for a differentiable convex function f , we have ∀ x, y ∈ dom ( f ) ⊆ R n

Therefore, for { i : α i = 0 } , we have

̸

̸

Let S := { i : α i = 0 } . We then have the following cases -

Assume that only one α i is nonzero, then ∀ v orthogonal to this direction, ⟨ v, Hv ⟩ D -1 < 0 by Eq A.19. Thus this gives a local maxima.

Assume more than one α i are nonzero, but | S | < k . Then we have for i ̸∈ S , ˜ H jj < 0 using A.19. For i ∈ S , ˜ H ii > 0 from Eq A.20. Hence these are saddle points.

̸

Assume we have S = [ k ] , i.e. ∀ i , α i = 0 . In this case, ˜ H ≻ 0 . So, we have a local minima.

This completes our proof.

Theorem A.1. Consider the data generated from the noisy ICA model (Eq 1). If f ( u | P ) is defined as in Eq 4 or Eq 5, we have ∇ 2 f ( u | P ) = BD u B T , for some diagonal matrix D u , which can be different for the differerent contrast functions.

Remark 4. The above theorem is useful because it shows that the Hessian of the contrast functions based on the CHF (eq 4 or CGF (eq 5) is of the form BDB T , where D is some diagonal matrix. This matrix can be precomputed at some vector u and used as the matrix C in the power iteration update (see eq 3) in Algorithm 2 of [49].

Proof of Theorem A.1. First, we show that the CHF-based contrast function (see Eq 4), the Hessian, is of the correct form.

CHF-based contrast function

where diag( a ) is the covariance matrix of z . For simplicity of notation, define ϕ ( z ; u ) := E [ exp( iu T Bz ) ] .

Define

Taking a second derivative, we have:

̸

All we have to show at this point is that H ( u ; z ) is diagonal. We will evaluate the k, ℓ entry. Let B i denote the i th column of B . Let Y \ k,ℓ := u T B ∑ j = k,ℓ z j . Using independence of the components of z , we have, for k = ℓ ,

̸

Now we evaluate:

Using Eqs A.21 and A.22, we see that indeed H ( u ; z ) is diagonal.

Now, we prove the statement about the CGF-based contrast function.

CGF-based contrast function

We have,

The new probability density P ( z ) . exp ( u T B z ) E [exp( u T B z )] is an exponential tilt of the original pdf, and since { z i } d i =1 are independent, and the new tilted density also factorizes over the z i 's, therefore, the covariance under this tilted density is also diagonal.

A.1.1 Uniform convergence

In this section, we provide the proof for Theorem 2. First, we state some preliminary results about the uniform convergence of smooth function classes which will be useful for our proofs.

A.1.1.1 Preliminaries

Let D := sup θ, ˜ θ ∈ T ρ X ( θ, ˜ θ ) denote the diameter of set T , and let N X ( δ ; T ) denote the δ -covering number of T in the ρ X metric. Then, we have the following standard result:

Proposition A.2. (Proposition 5.17 from [51]) Let { X θ , θ ∈ T } be a zero-mean subgaussian process with respect to the metric ρ X . Then for any δ ∈ [0 , D ] such that N X ( δ ; T ) ≥ 10 , we have

Remark 5. For zero-mean subgaussian processes, Proposition A.2 implies, E [sup θ ∈ T X θ ] ≤ E [ sup θ, ˜ θ ∈ T ( X θ -X ˜ θ ) ]

Proposition A.3. (Theorem 4.10 from [51]) For any b -uniformly bounded class of functions C , any positive integer n ≥ 1 , any scalar δ ≥ 0 , and a set of i.i.d datapoints { X ( i ) } i ∈ [ n ] we have

with probability at least 1 -exp ( -nδ 2 2 b 2 ) . Here { ϵ i } i ∈ [ n ] are i.i.d rademacher random variables.

A.1.1.2 Proof of theorem 2

For dataset { x ( i ) } i ∈ [ n ] , x ( i ) ∈ R k , consider the following definitions:

Let ∆( t, F | ˆ P ) be the empirical version where the expectation is replaced by sample averages. Now define:

Theorem A.4. Let F := { F ∈ R k × k : ∥ F ∥≤ 1 } . Assume that x ∼ subgaussian ( σ ) . We have:

where C k := max(1 , k log ( n ) Tr( S )) .

Proof. Define E = { t : ∥ t ∥≤ √ 2 k log n } . We will use the fact that P ( E c ) ≤ 2 /n (Example 2.12, [51]). Also note that by construction, ∆( t, F | P ) ≤ 1 . Observe that:

The second inequality follows from Jensen's inequality, the fourth follows from E [sup( . )] ≤ sup( E [ . ]) .

Theorem A.5. Let F := { F ∈ R k × k : ∥ F ∥≤ 1 } . Assume that x ∼ subgaussian ( σ ) . We have:

where C t := max(1 , ∥ t ∥ 2 Tr( S )) .

Proof.

Finally, for some S ′ = λS +(1 -λ ) ˆ S ,

where the last result follows from Theorem 4.7.1 in [47]. Now, for the second term, observe that:

We next have that, with probability at least 1 -1 /n ,

Thus with probability at least 1 -1 /n , using the inequality | 1 -e x |≤ 2 | x | , ∀ x ∈ [ -1 , 1] , Eq A.25 leads to:

Note that the matrices diag ( F ( S -̂ S ) F T ) and FSF T are positive semi-definite and t is unit-norm. Observe that, using Lemmas A.6 and A.7, we have:

Lemma A.6. Define F := { F ∈ R k × k : ∥ F ∥≤ 1 } . Let { x ( i ) } i ∈ [ n ] be i.i.d samples from a subgaussian ( σ ) distribution. We have:

where C t := max(1 , ∥ t ∥ 2 Tr( S )) .

Proof. Let ϕ ( t, F ; x ( i ) ) = exp( it T Fx ( i ) ) . Next, we note that ∥ F ∥ 2 ≤ 1 imply ∥ ∥ F T t ∥ ∥ ≤ ∥ t ∥ . Therefore,

Let f X ( u ) = ∑ i ϵ i ξ ( u ; x ( i ) ) . We will argue for unit vectors u , and later scale them by ∥ t ∥ . Define B k := { u ∈ R k , ∥ u ∥ ≤ 1 } , ξ ( u ; x ) := exp( iu T x ) and let

Then we have,

Then,

Let ̂ S := ∑ i x ( i ) ( x ( i ) ) T /n and τ n := n Tr ( ̂ S ) . We next have,

Next, we bound the covering number N ( δ ; B k , d u ) . Note that N ( δ ; B k , d u ) ≤ N ( δ/ √ τ n ; B k , ∥ . ∥ 2 ) since,

Using Cauchy-Schwarz, we have:

Therefore, we have N ( δ ; B k , d u ) ≤ ( 3 √ τ n δ ) k . Therefore, using Proposition A.2,

Now we recall that ∥ F T t ∥≤ ∥ t ∥ , since all vectors t appear as t T F T x ( i ) , this simply leads to scaling τ and ∥ S ∥ by ∥ t ∥ 2 . Dividing both sides by n , we have

Thus, using Proposition A.3 and using the definition of C t , we have,

Lemma A.7. Let F = { F ∈ R k × k : ∥ F ∥≤ 1 } . We have:

Proof. Let B k := { u ∈ R k , ∥ u ∥ ≤ 1 } Now define ψ j ( t, F ; x ) = E [exp( it j F T j x )] . Also define f ( j ) X ( F ) = 1 n ∑ i ϵ i ψ j ( t, F ; x ( i ) ) . Thus, using the same argument as in Lemma A.6 for t ←∥ t ∥ e j and x ( i ) ← t j x ( i ) ,

Finally, we see that:

The above is true because, for | a i | , | b i |≤ 1 , i = 1 , . . . , k ,

A.1.2 Local convergence

In this section, we provide the proof of Theorem 4. Recall the ICA model from Eq 1,

We provide the proof for the CGF-based contrast function. The proof for the CHF-based contrast function follows similarly. From the Proof of Theorem A.1 we have,

The new probability density P ( z ) exp ( u T Bz ) E [exp ( u T Bz )] is an exponential tilt of the original pdf, and since { z i } d i =1 are independent, and the new tilted density also factorizes over the z i 's, therefore, the covariance under this tilted density is also diagonal.

Let C := BD 0 B T . We denote the i th column of B as B .i . Define functions

Then f ( u | P ) = ∑ d i =1 r i ( u ) , ∇ f ( u | P ) = ∑ d i =1 g i ( u ) . For fixed point iteration, u k +1 = ∇ f ( C -1 u k | P ) ∥∇ f ( C -1 u k | P ) ∥ . Consider the function ∇ f ( C -1 u | P ) . Let e i denote the i th axis-aligned unit basis vector of R n . Since B is full-rank, we can denote α = B -1 u . We have,

For t ∈ R , define the function q i ( t ) : R → R as -

Note that q i (0) = E [ z i ] = 0 . For t = ( t 1 , t 2 , · · · t d ) ∈ R d , let q ( t ) := [ q 1 ( t 1 ) , q 2 ( t 2 ) · · · q d ( t d )] T ∈ R d . Then,

Therefore, if u t +1 = Bα t +1 , then

At the fixed point, α ∗ = e 1 ∥ Be 1 ∥ and Bq ( α ∗ ) ∥ Bq ( α ∗ ) ∥ 2 = Be 1 . Therefore,

Note the smoothness assumptions on q i ( . ) mentioned in Theorem 4, ∀ i ∈ [ d ] ,

Since ∀ k, ∥ u k ∥ = 1 , therefore, ∀ k, ∥ α t ∥≤ ∥ B -1 ∥ 2 . We seek to prove that the sequence { α t } n k =1 converges to α ∗ .

A.1.2.1 Taylor expansions

Consider the function g i ( y ) := q i ( y i ) ∥ B q ( y ) ∥ for y ∈ R d . We start by computing the gradient, ∇ y g i ( y ) .

Lemma A.8. Let g i ( y ) := q i ( y i ) ∥ B q ( y ) ∥ , then we have

Proof. The derivative w.r.t y i is given as -

Note that

Therefore,

̸

̸

For j = i , the derivative w.r.t y j is given as -

Next, we bound q i ( t ) and q ′ i ( t ) .

Lemma A.9. Under the smoothness assumptions on q i ( . ) mentioned in Theorem 4, we have for t ∈ [ -∥ B -1 ∥ 2 , ∥ B -1 ∥ 2 ] ,

̸

Proof. First consider q i ( t ) and q ′ i ( t ) for i = 1 . Using Taylor expansion around t = 0 , we have for some c ∈ (0 , t ) and ∀ i = 1 ,

Now, we know that q i (0) = q ′ i (0) = 0 . Then using Assumption 1, we have, for t ∈ [ -∥ B -1 ∥ 2 , ∥ B -1 ∥ 2 ] ,

Similarly, using Taylor expansion around α ∗ 1 = 1 ∥ B e 1 ∥ 2 for q 1 ( . ) we have, for some c ′ ∈ (0 , t )

Therefore, using Assumption 4, we have, for t ∈ [ -∥ B -1 ∥ 2 , ∥ B -1 ∥ 2 ]

A.1.2.2 Using convergence radius

In this section, we use the Taylor expansion results for q i ( . ) and q ′ i ( . ) in the previous section to analyze the following functions for y ∈ R d ,

Under the constraints specified in the theorem statement we have,

We start with w ( y ) .

̸

Lemma A.10. ∀ y ∈ R d , satisfying (A.34) , let δ ( y ) := q ( y ) -q 1 ( α ∗ 1 ) e 1 . Then, we have

Proof. Consider the vector δ ( y ) := q ( y ) -q 1 ( α ∗ 1 ) e 1 . Then,

̸

Note that

Now consider w ( y ) . Using the mean-value theorem on the Euclidean norm we have,

where γ ( y ) = µ q ( y ) + (1 -µ ) q 1 ( α ∗ 1 ) e 1 , µ ∈ (0 , 1) . Then,

Therefore using A.37,

Finally, we consider the function v ( y ; i ) := e T i B T B q ( y ) ∥ B q ( y ) ∥ 2 .

Lemma A.11. ∀ y ∈ R d satisfying (A.34) , we have

̸

Proof. Define θ := ϵ ∥ y -α ∗ ∥ for convenience of notation. We have,

Therefore, if i = 1 , then using Lemma A.10,

Using the following inequalities -

and observing that using Eq A.34, and Lemma A.10,

Therefore we have from Eq A.41,

̸

where we used θ := ϵ ∥ y -α ∗ ∥ . For the case of i = 1 , using Lemma A.10 we have

We now operate under the assumption that Eq A.34 holds for y = α t and inductively show that it holds for y = α t +1 as well. Recall the function g i ( y ) := q i ( y i ) ∥ B q ( y ) ∥ . By applying the mean-value theorem for g i ( . ) for the points α t and α ∗ , we have from Eq A.26 and A.27 -

Note the induction hypothesis assumes that Eq A.34 is true for x = α t . Since ∀ i, λ i ∈ (0 , 1) , therefore Eq A.34 holds for all β i as well. Squaring and adding Eq A.44 for i ∈ [ d ] and taking a square-root, we have

Let us consider the expression G ij := ∂g i ( y ) ∂y j ∣ ∣ ∣ ∣ ∣ β i , ∀ i, j ∈ [ k ] . For the purpose of the subsequent ∗

analysis, we define θ := ϵ ∥ α t -α ∥ . We divide the analysis into the following cases -

Case 1 : i = 1 , j = 1

From Lemma A.9, and,

Therefore,

̸

Case 2 : i = 1 , j = 1

From Lemma A.9,

From Lemma A.9,

Therefore,

̸

From Lemma A.9,

From Lemma A.9,

Therefore,

̸

Case 4 : i = 1 , j = 1 , i = j

̸

From Lemma A.9,

From Lemma A.9,

Then,

Therefore,

̸

Case 5 : i = 1 , j = 1 , i = j

̸

̸

From Lemma A.9,

From Lemma A.9,

Therefore,

A.1.2.3 Putting everything together

Putting all the cases together in Eq A.45, we have ∥ α t +1 -α ∗ ∥ ∥ α t -α ∗ ∥ ≤ C ∥ α t -α ∗ ∥ , where

Then we have, C ≤ 5 ∥ B ∥ F ∥ B e 1 ∥ 2 max { c 3 | q 1 ( α ∗ 1 ) | , c 2 2 | q 1 ( α ∗ 1 ) | 2 } . For linear convergence, we require CR < 1 , which is ensured by the condition mentioned in Theorem 4.

A.2 Additional experiments for noisy ICA

A.2.1 Experiments with super-gaussian source signals

Table 2 shows the Amari error in the presence of super-Gaussian signals. The signals are 1) a Uniform distribution U ( - √ 3 , √ 3) , 2) Bernoulli ( 1 2 + 1 √ 12 ) , 3) Laplace(0, 0.05) (mean 0 and standard deviation 0.05), 4) Exponential(5), and 5) and 6) a Student's t -distribution with 3 and 5 degrees of freedom, respectively. The Meta algorithm closely follows the best candidate algorithm even in the presence of many super-Gaussian signals.

A.2.2 Image demixing experiments

In this section, we provide additional experiments for image-demixing using ICA. We mix images using flattening and linearly mixing them with a 4 × 4 matrix B (i.i.d entries ∼ N (0 , 1) ) and Wishart noise ( ρ = 0 . 001 ). Demixing is performed using the SINR-optimal demixing matrix (see Section 2) and the results are shown in Figure A.1 along with their corresponding independence scores. The CHF-based method recovers the original sources well, upto sign. The Kurtosis and CGF-based method fails to recover the second source. This is consistent with their higher independence score. The Meta algorithm selects CHF from candidates CHF, CGF, Kurtosis, FastICA, and JADE.

A.2.3 Image denoising experiments

In this experiment, we use the ICA-based denoising technique proposed in [38] to compare candidate Noisy ICA algorithms and show that the Meta algorithm can pick the best-denoised image based on the independence score proposed in our work.

We use the noisy MNIST dataset and further add entrywise Gaussian noise with variance proportional to sin 2 ( c 1 π ( i + j ))( c 1 = 50) to the ( i, j ) th pixel. Training images are flattened to create a 784dimensional vector, and PCA is performed to reduce dimensionality to 25, on which subsequently ICA is performed. The original and denoised images along with their independence score are shown in Figure A.2. We note that CHF-based denoising provides qualitatively better results, while CGF-based denoising provides the worst results. This is consistent with their corresponding independence scores.

A.3 Algorithm for sequential calculation of independence scores

In this section, we provide an algorithm for the computation of the independence score proposed in the manuscript, but in a sequential manner. The detailed algorithm is provided as Algorithm 3. We assume that we have access to a matrix of the form C = BDB T .

The power method (see Eq 3) essentially extracts one column of B (up to scaling) at every step. [49] provides an elegant way to use the pseudo-Euclidean space to successively project the data onto columns orthogonal to the ones extracted. This enables one to extract each column of the mixing matrix. For completeness, we present the steps of this projection. After extracting the i th

(a) Source Images

(b) Mixed Images

(a) Original Images

(b) CHF-based denoising (

∆(

t, F

|

P

) = 4

.

4

×

10

-

6

)

(c) Kurtosis-based denoising (

∆(

t, F

|

P

) = 1

.

7

×

10

-

4

)

(d) CGF-based denoising (

∆(

t, F

|

P

) = 9

.

2

×

10

-

6

)

(e) FastICA-based denoising (

∆(

t, F

|

P

) = 5

.

9

×

10

-

6

)

When u ( j ) = B j / ∥ B j ∥ , v ( j ) = ( B T ) -1 D † e j e T j D † e j ∥ B j ∥ = ( B T ) -1 e j ∥ B j ∥ . Let x denote an arbitrary datapoint. Thus

Thus the projection on all other columns j > ℓ is given by:

Alldata

0.018

0.016

0.014

Independencescore

0.012

0.01

0.008

0.006

0.004

0.002

0

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Dotproductwithtruecolumn

where ˜ g = Fg , where F is some k × k matrix. So we have min( ℓ, k ) vectors of the form z i B i +˜ g i , and when ℓ < k an additional vector which contains all independent random variables z j , j > ℓ along with a mean-zero Gaussian vector. Then we can project each vector to a scalar using unit random vectors and check for independence using the Independence Score defined in 2. When ℓ = k , then all we need are the j = 1 , . . . , k projections on the k directions identified via ICA in Eq A.47.

As an example, we conduct an experiment (Figure A.3), where we fix a mixing matrix B using the same generating mechanism in Section 5. Now we create vectors q which interpolate between B (: , 1) and a fixed arbitrary vector orthogonal to B (: , 1) . As the interpolation changes, we plot the independence score for direction q . The dataset is the 9-dimensional dataset, which has independent components from many different distributions (see Section 5).

This plot clearly shows that there is a clear negative correlation between the score and the dot product of a vector with the column B (: , 1) . To be concrete, when q has a small angle with B (: , 1) , the independence score is small, and the error bars are also very small. However, as the dot product decreases, the score grows and the error bars become larger.

Algorithm 3 Independence Score after extracting ℓ columns. U and V are running estimates of B and B -1 upto ℓ columns and rows respectively.

(a)

(b)

(c)

(d)

We refer to this function as ∆( X,U,V,ℓ ) which takes as input, the data X , the number of columns ℓ and matrices U , V which are running estimates of B and B -1 upto ℓ columns and rows respectively.

A.4 More details for third derivative condition in theorem 3

Theorem 3 requires the condition 'The third derivative of h X ( u ) does not change the sign in the half line [0 , ∞ ) for the non-Gaussian random variable considered in the ICA problem.". In this section, we provide sufficient conditions with respect to contrast functions and datasets where this holds. Further, we also provide an interesting example to demonstrate that our Assumption 1-(d) might not be too far from being necessary.

CGF-based contrast function. Consider the cumulant generating function of a random variable X , i.e. the logarithm of the moment generating function. Now consider the contrast function

We first note that it satisfies Assumption 1(a)-(d). Next, we observe that it is enough for a distribution to have all cumulants of the same sign to satisfy Assumption 1(d) for the CGF. For example, the Poisson distribution has all positive cumulants. Figure A.4 depicts the third derivative of the contrast function for a Bernoulli(1/2), Uniform, Poisson, and Exponential.

Logarithm of the symmetrized characteristic function. Figure A.5 depicts the third derivative of this contrast function for the logistic distribution where Assumption 1(d) holds. Although the Assumption does not hold for a large class of distributions here, we believe that the notion of having the same sign can be relaxed up to some bounded parameter values instead of the entire half line, so that the global convergence results of Theorem 3 still hold. This belief is further reinforced by Figure A.6 which shows that the loss landscape of functions where Assumption 1(d) doesn't hold is still smooth.

A.4.1 Towards a necessary condition

Consider the probability distribution function (see [36]) f ( x ) = ke -x 2 2 ( a + b cos( cπx )) for constants a, b, c > 0 . For f ( x ) to be a valid pdf, we require that f ( x ) ≥ 0 , and ∫ ∞ -∞ f ( x ) dx = 1 . Therefore, we have,

Noting standard integration results, we have that,

Therefore, k = 1 √ 2 π ( a + be -c 2 π 2 2 ) . Some algebraic manipulations yield the following:

Therefore, ln( MGF f ( x ) ( t )) = -ln( a + be -c 2 π 2 2 ) + t 2 2 +ln( a + be -c 2 π 2 2 cos( cπt )) .

We now compute the mean, µ , and variance σ 2 .

The mean, µ can be given as :

The variance, σ 2 can therefore be given as :

Now, we note that ∫ ∞ -∞ e -x 2 2 x 2 dx = √ 2 π and ∫ ∞ -∞ e -x 2 2 x 2 cos( cπx ) dx = e -c 2 π 2 2 √ 2 π (1 -c 2 π 2 ) Therefore,

The symmetrized CGF can therefore be written as

Therefore, g ( t ) = sym CGF ( t ) -(1 -bc 2 π 2 e -c 2 π 2 2 a + be -c 2 π 2 2 ) t 2 can be written as :

Finally, g ′ ( t ) can be written as :

g ′′ ( t ) can be written as :

and g ′′′ ( t ) can be evaluated as:

Lets set a = 2 , b = -1 , c = 4 π . Then, we have that the pdf, f ( x ) = ke -x 2 2 (2 -cos(4 x )) = ke -x 2 2 (1 + 2 sin 2 (2 x )) . The corresponding functions are :

Closeness to a Gaussian MGF: It can be seen there that g ′′′ ( t ) changes sign in the half-line. However, the key is to note that the MGF of this distribution is very close to that of a Gaussian and can be made arbitrarily small by varying the parameters a and b . This renders the CGF-based contrast function ineffectual. This leads us to believe that Assumption 1(d) is not far from being necessary to ensure global optimality of the corresponding objective function.

A.5 Surface plots of the CHF-based contrast functions

Figure A.6 depicts the loss landscape of the Characteristic function (CHF) based contrast function described in Section 3.3 of the manuscript. We plot the value of the contrast function evaluated at B -1 u , u = 1 √ x 2 + y 2 ( x y ) for x, y ∈ [ -1 , 1] . As shown in the figure. We have rotated the data so that the columns of B align with the X and Y axes. The global maxima occur at u aligned with the columns of B .

(a)

(b)

(c)

(d)

Surface plot for Uniform data

0.8

0.065

0.055

0.6

0.055

0.06

0.055

0.05

0.4

0.05

900

0.045

0.04

0.045

02

0.045

0.035

0.04

0.2

0.03

-0.4

0.025

0.035

-0.6

0.5

0.03

0.8

0.8

0.6

-0.4

-0.2

0.2

44

0.8

Surface plot for Zero-Kurtosis data

V

90'0

0.06

0.045

0.6

0.045

0.05

00

0.04

0.04

00

900'0

0.02

0.03

0.2

0.01

S20'0

5200

-0.4

L015

0.015

S'0-

0.01

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main contributions are the Meta algorithm for non-parametric selection of the best noisy ICA algorithm for a given distribution, along with 2 new contrast functions of independent interest.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the Experiments (Section 5, there are some noise/sample-size regimes where other algorithms may perform slightly better than our proposed contrast functions, but we show that even then our diagnostic can pick out the best algorithm.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Every theorem statement mentions the assumptions and the location of the proof in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 5 provides extensive details of the experimental setup.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We make the code for our experiments available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 5 provides extensive details of the experimental setup.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Figure 1 and 2(a) show error bars and error distribution respectively. Figure 2(b),(c) show the mean over 100 runs, similar to [49]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our experiments were performed on a single Macbook Pro M2 2022 CPU with 8 GB RAM.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We abide by the NeurIPS Code of Ethics in our work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any models with a risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: MATLAB implementations (under the GNU General Public License) can be found at - FastICA and JADE. The code for PFICA was provided on request by the authors.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets to require documentation or licensing. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We only use publicly available datasets in Section 5.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not have experiments with crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the noisy ICA problem. They first propose a numerical objective function that can be used as a guide to assess the quality of an existing ICA algorithm. This function is not suitable for optimization and therefore, the authors propose to use it in a meta-algorithm, where the purpose is to select the best ICA solutions out of several, where each solution is possibly produced by a separate algorithm. Then, they propose new contrast functions that are specifically suited for the noisy ICA problem. Finally, they study critical points of contrast functions so as to validate their use in noisy ICA problems. They provide numerical experiments to back up their claims.\n\nSTRENGTHS:\nThe paper consider the noisy ICA problem, which is a challenging problem and is arguably more realistic than the noise-free ICA problem. The paper also demonstrates via numerical experiments that the proposed meta algorithm and the contrast functions can uncover the components in an image mixing problem.\n\nWEAKNESSES:\nThe paper touches on too many things and is not very readable. For instance, I don't see the connection between the meta algorithm and the new contrast functions. Similarly, I would have welcomed more detail on the development of the meta algorithm, and the choice of randomization in its computation. \nOccasionally, something is introduced out of nowhere and it's hard to understand the motivation. For instance, I don't see the development for the new contrast functions. \nOverall, I'd have preferred a paper that coherently and clearly explores a single idea. Currently, it reads like a collection of not clearly-developed ideas.\n\nQUESTIONS:\n- line 123, \"We propose to adapt the CHFICA objective using estimable parameters to the noisy ICA setting\" : I don't understand this sentence. What are the \"estimable parameters\" here? Is it $S$?\n- eqn 2 : What's the motivation for the additional terms (the second factors in both \"JOINT\" and \"PRODUCT\")?\n- eqn 2 : In the last term, is a $t$ missing?\n- Thm 2 : How does knowing this bound help? Please discuss how this theorem is useful in the text.\n- line 175, \"...in this section them, ...\" : typo?\n- line 214 : What is a pseudo-Euclidean space?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a nonparametric score to adaptively pick the best noisy ICA algorithm from a set of candidates. This “independence score” is based on the characteristic function-based objective (CHF) introduced by Eriksson&Koivunen in 2003. \n\nIn practice, this independence score evaluates the inverse mixing matrix obtained by an ICA algorithm without requiring accessing to the true sources. \n\nIn addition, the paper proposes some new contrast functions and algorithms and present simulation results showing the effectiveness of the proposed independence score.\n\nSTRENGTHS:\n•\tSolid theoretical approach that justifies the proposed independence score and the new derived objective functions\n\n•\tGood review of relevant previous studies on ICA and noisy ICA.\n\n•\tExperimental results include synthetically generated sources aswell as real data (images, MNIST) synthetically mixed.\n\nWEAKNESSES:\n•\tThe proposed approach focused only in the recovery of the proper unmixing matrix (B^{-1}) while in noisy ICA, the final goal is to recover the clean sources. There is no discussion about how to recover the sources after the proper unmixing matrix is obtained.\n\n•\tThe experimental part focused on synthetically mixed signals only. The paper would be greatly benefited from the inclusion of some real-world source separation problem.\n\n•\tTheorem 2 assumes Subgaussian mixtures, which ensures the concentration of the sample covariance matrix. This assumtion is rather strong and it would be good to analyze what happen if it is not met.\n\nQUESTIONS:\nRegarding assumption of Theorem 2 (sub-Gaussianity). I understand that assumption is needed to guaranty the concentration of the norm operator, which is a rather strong assumption. Do you think is possible to relax that assumption and keep the method working? Have you make any experiments with super-Gaussian sources, for example?\n\nLIMITATIONS:\nYes, the experimental results section discuss some limitations of the proposed algorithm in highly noisy scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
The presented paper focuses on the problem of noisy ICA which remains a significant challenge in classical machine learning.
The authors introduces a nonparametric independent score extending the work in [21] to evaluate the estimation of the demixing matrices without requiring any prior knowledge of underlying noise distribution parameters. 
Further. the authors propose some new contrast functions and provides a very interesting discussion on the convergence for the presented contrast functions of noisy ICA which can be also applied for cumulant based contrast functions.

STRENGTHS:
The introduction of a nonparametric independence score is innovative.
The paper provides an extensive theoretical analysis, including the development of new contrast functions and a detailed discussion on convergence properties.

WEAKNESSES:
Two main weaknesses stood out to me in the paper:
1.  The proposed study heavily rely on Gaussian noise assumptions. The performance of suggested score might not hold in cases with different noise characteristics. Similarly, the proposed contrast functions could have limited scope of applicability in real life scenarios..
2. The meta algorithm's effectiveness is contingent on the pool of candidate algorithms it selects from.

QUESTIONS:
In equation 2 in the exponent of the product term \mathbf{t} seems to be missing?

LIMITATIONS:
The methods focus primarily on Gaussian noise, which might limit their applicability in scenarios where noise distributions do not conform to this assumption.
The global convergence analysis heavily depends on the assumption that the noise is Gaussian. This dependency raises questions about the generalizability of the convergence results to other types of noise.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a modification of the original CHFICA characteristic function to the noisy ICA case without requiring knowledge of the noise distribution parameters.\nThe modified independence score is then used to select the best out of multiple ICA methods based on the score assigned to their solutions, in a per dataset basis. In practice, an average score is evaluated over random directions t, which renders the score computationally inefficient for direct Minimization.\nThe empirical estimate of the new independence score is shown to converge to the population score if the data covariance is concentrated.\nEmpirical evidence is shown that the new empirical score correlates well with the well-established Amari error.\n\nThis work also introduces two computationally efficient contrast functions to be Maximized (CHF-based and CGF-based) for BSS, both of which do not require higher moments and, thus, work well on data with near-zero kurtosis, like Bernoulli(p) with low p.\nTheoretical analyses of local and global convergence are also included for a large class of smooth contrast functions that meet certain desirable properties (Assumption 1). Properties (a) and (c) are critical as they yield contrast functions that are not influenced by additive independent Gaussian noise.\nThe CHF contrast only requires a finite second moment and is suitable for heavy-tailed sources.\nThe CGF contrast is not appropriate for heavy-tailed sources.\nGlobal convergence: The Hessian of both contrasts is shown to have the form C = BDB^T. The maxima of the Hessian-adjusted contrasts is shown to occur at B^-1 u = e_i for positive elements of D. Fast sequential power method optimization in a pseudo-Euclidean space defined by C^-1 was shown by [48]. Before optimization, C need only be computed once for some random vector u and just reused during the power iteration updates. A requirement of no sign change of the third derivative of the contrast over the half-lines is required to sufficiently distinguish between Gaussian and near-0 kurtosis source distributions.\nLocal convergence: Linear convergence is attainable without the third derivative requirement. Geometric convergence is attained at some low epsilon based on how non-gaussian the source is.\nContrary to prior work on cumulant-based methods, convergence of contrasts meeting Assumption 1 was established based on the characterization of the contrasts in the pseudo-Euclidean space and not on the convergence of the power method.\n\nExperimental results support Meta's ability to select the optimal solution over various algorithms, showing variance reduction in the estimates, uniform performance across high and near-0 kurtosis.\nEvaluation of the CHF and CGF contrasts show that CHF outperforms other methods at higher noise, but only at sample size n > 20000.\nThe Meta algorithm picks up the slack of CHF by selecting the competing PFICA solution when n < 20000.\n\nSTRENGTHS:\nOriginality :\n- The work introduces a somewhat novel combination of known ideas in a reasonably novel formulation that adapts previous work to the case of additive Gaussian noise.\n- The work differs from and extends previous contributions, dealing with near-zero kurtosis sources and additive Gaussian noise.\n- The manuscript cites related work on contrast functions for BSS and noiseless ICA methods, adequately indicating the sources of inspiration.\n\nQuality:\n - The work is technically sound, with thorough proofs.\n - The theoretical claims are well supported by the experiments.\n - Strengths and weaknesses are discussed well.\n\nClarity:\n - The work is written well overall, focusing on the key points and contributions.\n\nSignificance :\n - The results are important as they establish a consistent framework to select the optimal solution over a range of BSS methods for noisy ICA based on the proposed modified independence score.\n\nWEAKNESSES:\nQuality:\n - The code is not friendly to readers as there is no 1-to-1 correspondence with the notation in the paper. Suggestion: improve documentation and tidy up the codebase (especially comments and function arguments) for readability.\n\nClarity:\n - Despite the well-written manuscript and thorough methodology and proofs, the paper contains many typos (both in text and proofs) and inconsistencies in notation (especially between Appendix and main manuscript) that limit the clarity. Besides fixing these typos, explicitly include some of the \"obvious\" steps currently omitted in the derivation; that would greatly improve readability. Also add some high-level intuitions as noted below.\n - Clarify the origin of new contrast functions and how they came about. It is unclear what motivated or led to these otherwise \"semi-arbitrary\" functions.\n - Define O_P(.)\n - It appears that the new independence score does not meet some of the properties in Assumption 1. Please, clarify if that is indeed the case and, if so, which properties specifically.\n - Appendix A.3: Clarify that Algo 2 is NOT specific to contrasts based on power iteration only, but any sequential method. It is suggested to make one separate sub-section covering how the projection is done and another for the Algo description itself. Also clarify what the \"Total\" (non-sequential) Algo would look like, for comparison. \n - Lines 119-122: t.T should be bold face.\n - Eq. 2: missing last t in the PRODUCT term.\n - Section 3.2: Change \"Properties of Contrast Functions 1.\" to \"Assumption 1. (Properties of Contrast Functions)\"\n - Line 175: remove \"them\"\n - Line 182: \"note\" --> \"not\"\n - Footnote 2 (page 5): add \"and remains a linear combination of z\"\n - Theorem 4, line 254: Assumption 1(d) is NOT about the third derivative.\n - Line 271: Define the value of k in this experiment. I see 11 mentioned in passing in line 786...\n - Line 277: Fix contradicting statements: \"CHF and CGF are initialized using the B estimated via PFICA\" and \"CHF and CGF is based on a single random initialization\". \n - Line 283: \"also used\" --> \"also be used\"\n - Table 1 caption: Clarify that since median is being reported, Meta is better than the best method “on average”, but on any individual experiment it is identical to the best method.\n - Line 299: refs to 2b and 2c are in flipped order.\n - Line 315: In the noise power experiment, it is stated that \"the difference between the two leading algorithms is small.\" Therefore, the same acknowledgement should be indicated in line 315 after \"CHF dominates, and one can see that Meta starts following CHF\"\n - Line 456: Assumption 1(d) is NOT about the 3rd derivative constraint...\n - Line 464: Missing ' in g.\n - Line 472: For consistent notation, it should be Delta(t,F | P) = 0, not Delta_F(t) = 0 \n - Lines 474-477: there is an extra transpose symbol at the last t (several times).\n - Line 510: (e) --> (d)\n - Notation and conditions in lines 524-529 is not consistent with Theorem 3 in the main manuscript.\n - Line 777: w_i is not defined, g_i should be ~g_i.\n - Lines 784-789: the r defined here has no relation to the r in Algo 2. Please pick another letter.\n - Algo 2:\n\t - Clarify why it is necessary to evaluate Y_j per component and why it would not be equivalent to just compute W = X V^T U^T r\n\t - for j in range[1, M] : M is not defined, but I think it would be l + 1? Also, the letter j is used for many different things inside the same loop. Change to \" for p in range[1, l + 1] \", and s(j) --> s(p).\n\t - \" W(:,j) ← Y_j r \" --> \" W(:,a) ← Y_a r \"\n\t - I suspect r is equivalent to t in Eq. 2, if so change it to t.\n - Appendix A.4: Verify that all occurrences of Assumption 1 (d) are correct (i.e., referring to symmetry not the third derivative)\n\nQUESTIONS:\nQuestions were included along with the weaknesses.\n\nLIMITATIONS:\nYes, the authors addressed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}