Shared Independent Component Analysis for Multi-Subject Neuroimaging
Authors: Hugo Richard
Pierre Ablin
Bertrand Thirion
Alexandre Gramfort
Aapo Hyvarinen
Venue: NeurIPS 2021 Poster (2021)
Keywords: neuroimaging
fMRI
MEG
shared response modeling
component analysis
independent component analysis
multi-view learning

Abstract: We consider shared response modeling, a multi-view learning problem where one wants to identify common components from multiple datasets or views. We introduce Shared Independent Component Analysis (ShICA) that models each
view as a linear transform of shared independent components contaminated by additive Gaussian noise. We show that this model is identifiable if the components are either non-Gaussian or have enough diversity in noise variances. We then show that in some cases multi-set canonical correlation analysis can recover the correct unmixing matrices, but that even a small amount of sampling noise makes Multiset CCA fail. To solve this problem, we propose to use joint diagonalization after Multiset CCA, leading to a new approach called ShICA-J. We show via simulations that ShICA-J leads to improved results while being very fast to fit. While ShICA-J is based on second-order statistics, we further propose to leverage non-Gaussianity of the components using a maximum-likelihood method, ShICA-ML, that is both more accurate and more costly. Further, ShICA comes with a principled method for shared components estimation. Finally, we provide empirical evidence on fMRI and MEG datasets that ShICA yields more accurate estimation of the components
than alternatives.

Shared Independent Component Analysis for Multi-Subject Neuroimaging
Hugo Richard
Pierre Ablin DMA CNRS and ENS Paris, France
Bertrand Thirion
Alexandre Gramfort Inria Université Paris-
Inria
Inria
Université Paris-Saclay
Université Paris-Saclay
Palaiseau, France
Palaiseau, France
Palaiseau, France
Aapo Hyvärinen
Department of Computer Science
University of Helsinki Helsinki, Finland
Abstract
We consider shared response modeling, multi-view learning problem where one wants to identify common components from multiple datasets or views. We introduce Shared Independent Component Analysis (ShICA) that models each view as a linear transform of shared independent components contaminated by additive Gaussian noise. We show that this model is identifiable if the components are either non-Gane an-Gaussian or have enough diversity in noise variances. We then show cases multi-set canonical correlation analysis
that in unmixing matrices, but that even a small amount of sampling noise makes Multiset propose to joint diagonalization after
recover the correct
CCA fail. To solve this problem, Multiset CCA, leading to a new approach called ShICA-J. We show via simulations that ShICA-J leads to improved results while being very fast fit. While ShICA-J is based on second-order statistics, we further propose to leverage non-Gaussianity of the components using maximun-likelhhodd method, ShICA-ML, that is both more accurate and more costly. Further, ShICA comes with principled method for shared components estimation. Finally, we provide empirical evidence on fMRI and MEG datasets that ShICA yields more accurate estimation of the components than alternatives.
Introduction
In many data science problems, data are available through different views. Generally, the views represent different measurement modalities such as audio and video, or the same text that may be available in different languages. Our main interest here is neuroimaging where recordings from multiple subjects. In particular, it is of interest to find common patterns or responses that are shared between subjects when they receive the same stimulation or perform the same cognitive task 16 51]
made
popular line of work to perform such shared response modeling is group Independent Component Analysis (ICA) methods. The fastest methods [14 are among the most popular, yet they are not grounded on principled probabilistic models for the multiview setting. More principled approaches exist [5]] 27 but they do not model subject-specific deviations from the shared response. However, such deviations expected in most neuroimaging settings, the magnitude of the response may differ from subject to subject 46), as may any noise due to heartheats, respiratory artefacts or head
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

movements [39]. Furthermore, most GroupICA methods are typically unable to separate components whose density close to a Gaussian.
Independent vector analysis (IVA) [6 5 is powerful framework where components are independent within views but each component of given view depend the corresponding component in other views. However, current implementations such as IVA-L [36], IVA-G [5], IVA-L-SOS ([]] IVA-GGD
IVA with Kotz distribution [6] estimate only the view-specific components, and do not model or extract a shared response which is the main focus in this work.
On the other hand, the shared response model [16] is popular approach to perform shared response modeling, yet it imposes orthogonality constrains that are restrictive and not biologically plausible.
In this work we introduce Shared ICA (ShICA), where each view is modeled as a linear transform of shared independent components contaminated by additive Gaussian noise. ShICA allows the principled extraction of the shared components (or responses) in addition to view-specific components. Since is based on a statistically sound noise model, it enables optimal inference (minimum mean square error, MMSE) of the shared responses.
Let us note that ShICA is longer the method of choice when the concept of common response is either not useful or not applicable. Nevertheless, we believe that the ability to extract common response an important feature in most contexts because highlights stereotypical brain response to stimulus. Moreover, finding commonality between subjects reduces often unwanted inter-subject variability.
The paper is organized as follows. We first analyse the theoretical properties of the ShICA model, before providing inference algorithms. We exhibit necessary and sufficient conditions for the ShICA model to be identifiable (previous work only shows local identifiability (7). in the presence of Gaussian or non-Gaussian components. We then use Multiset CCA to fit the model when all the components are assumed to be Gaussian. We exhibit necessary and sufficient conditions for Multiset CCA to be able recover the unmixing matrices (previous work only gives sufficient conditions (38). In addition, we provide instances of the problem where Multiset CCA cannot recover the mixing matrices while the model is identifiable. We next point out practical problem even small sampling noise can lead to large error in the estimation of unmixing matrices when Multiset CCA used. To address this issue and recover the correct unmixing matrices, we propose to apply joint diagonalization to the result of Multiset CCA yielding new method called ShICA-J. We further introduce ShICA-MI, a maximum likelihood estimator of ShICA that models non-Gaue non-Gaussian components using Gaussian mixture model. While ShICA-ML yields more accurate components, ShICA-J is significantly faster and offers great initialization to ShICA-ML. Experiments on fMRI and MEG data demonstrate that the method outperforms existing GroupICA and IVA methods.
2 Shared ICA (ShICA): an identifiable multi-view model
Notation We write vectors in bold letter v and scalars in lower case a. Upper case letters M are used denote matrices. We denote M the absolute value of the determinant of M. x ~ NN E) means that x I² follows multivariate normal distribution of mean μ E IR and covariance E R The j.j entry of a diagonal matrix Σ, is denoted Eij. the entry of y: is denoted Yij- Lastly, 8 is the Kronecker delta.
Model Definition In the following, X1 XI,...,X,, Xm E RR denote the m observed random vectors obtained from the m different views. We posit the following generative model, called Shared ICA (ShICA): f
x, As
(1)
where TRP contains the latent variables called shared components, A1,..., Am Rpxp the invertible mixing matrices, and n, E TR are individual noises. The individual noises model both the deviations of a view from the mean -i.e. individual differences and measurement noise. Importantly, we explicitly model both the shared components and the individual differences in probabilistic framework enable an optimal inference of the parameters and the responses.
We assume that the shared components are statistically independent, and that the individual noises Gaussian and independent from the shared components: p(s) = IT p(sy) and n, (0, E,). where the matrices E; are assumed diagonal and positive. Without loss of generality, components

---

are assumed to have unit variance E(ss" = Ig. We further assume that there are at least 3 views: m 2-
In contrast to almost all existing works, we assume that some components (possibly all of them) may be Gaussian, and denote G the set of Gaussian components: 8j (0,,1) for G. The other components are non-Gaussian: for & 9, Sy is non-Gaussian.
Identifiability The parameters of the model θ (A1, m. E1y...; Em)) We are inter- ested in the identifiability of this model: given observations Xi...., Xm generated with parameters O, are there some other O' that may generate the same observations? Let us consider the following assumption that requires that the individual noises for Gaussian components are sufficiently diverse:
Assumption (Noise diversity in Gaussian components). For all j:j' E G.j + j', the sequences (Et))== and )=1.))= different where Eig is the j.j entry of E4
It is readily seen that there is one trivial set of indeterminacies in the problem: if P E RPxP is sign and permutation matrix (i.e. a matrix which has one coefficient on each row and column, and 0's elsewhere) the parameters [AIP.... Am P. ET P..... P Em P) also generate X1,. Xm- The following theorem shows that under the above assumption, these are the only indeterminacies of the problem.
Theorem 1 (Identifiabilityy. We make Assumption We let θ' = -AA,.... A' Ey..... )))) another set of parameters, and assume that they also generate x1...., ,,m- Then, there exists sign and permutation matrix P such that for all i. A/ A,P, and Σ = P.PP
The proof is in Appendix A. Identifiability
Gaussian case is a consequence of the identifiability results in [59 and in the general case, local identifiability results can be derived from the work of 77] However local identifiability only shows that for given set of parameters there exists neighborhood in which other set of parameters can generate the same observations [52 In contrast, the proof of
Theorem shows global identifiability.
Theorem shows that the task of recovering the parameters from the observations is well-posed problem, under the sufficient condition of Assumption [ We also note that Assumption i necessary for identifiability. For instance, if and j' are two Gaussian components such that Dij Eny for all i, then global rotation of the components j.j' yields the same covariance matrices. The current work assumes 2 3, in appendix B we give an identifiability result for 2, under stronger conditions.
Estimation of components with noise diversity via joint-diaggnallzation
We now consider the computational problem of efficient parameter inference. This section considers components with noise diversity, while the next section deals with non-Gaussian components.
3.1 Parameter estimation with Multiset CCA
If assume that the components are all Gaussian, the covariance of the observations given by
Cjj = [ A. p + 6G Σ, A are sufficient statistics and methods using only second order information, like Multiset CCA, are candidates to estimate the parameters of the model. Consider the matrix R containing m x blocks of size p x such that the block i, is given by Cyz. Consider the matrix D identical C excepts that the non-diagonal blocks are filled with zeros:
Cim
C
C
D
:
(2)
Cml
Crem
Omm
Generalized CCA consists of the following generalized eigenvalue problem:
Cu XDu,
A > 0. E Rm
(3)
Consider the matrix = [u' ....,... u²] E mpp formed by concatenating the p leading eigenvectors of the previous problem ranked in decreasing eigenvalue order. Then, consider U to be formed of m

---

blocks of size stacked vertically and define (W*)T to be the i-th block. These m matrices are the output of Multiset CCA. We also denote A - ... - Ap the p leading eigenvalues of the problem.
An application of the results of (38 shows that Multiset CCA recovers the mixing matrices of ShICA under some assumptions.
Proposition 1 (Sufficient condition for solving ShICA via Multiset CCA (8]). Let Tijk (1 $. Assume that (Fajk)) is non-increasing. Assume that the maximum eigenvalue Vy of matrix R( of general element (Taske) is such that Vv = Axp- Assume that x1...Ap are distinct. Then, there exists scale matrices I such that W. = T;A,' for all i.
= + =,
This proposition gives a sufficient condition for solving ShICA with Multiset CCA. It needs particular structure for the noise covariances as well as specific ordering for the eigenvalues. The next theorem shows that only need Åι ...Ap A to be distinct for Multiset CCA to solve ShICA:
Assumption 2 (Unique eigenvalues). Ap.A.pp are distinct.
Theorem 2. We only make Assumption [ Then, there exists permutation matrix P and scale matrices T3 such that W; = AT, for all
The proof is in Appendix A.2 This theorem means that solving the generalized eigenvalue problem 3 allows to recover the mixing matrices up to a scaling and permutation: this form of generalized CCA recovers the parameters of the statistical model. Note that Assumption 2 is also necessary condition. Indeed, if two eigenvalues are identical, the eigenvalue problem is not uniquely determined.
We have two different Assumptions, and 2 the first of which guarantees theoretical identifiability as per Theorem D and the second guarantees consistent estimation by Multiset CCA as per Theorem Next we will discuss their connections, and show some limitations of the Multiset CCA approach. To begin with, we have the following result about the eigenvalues of the problem
(3) and the Σij.
Proposition 2. For j S p, let XÚ the largest solution of Ex (114 1. Then, X Ai...., are the p largest eigenvalues of problem
It is easy to see that we then have Åy..... Xp greater than 1, while the remaining cigenvalues are lower than 1. From this proposition, two things appear clearly. First, Assumption 2 implies Assumption [ Indeed, if the Xy 's are distinct, then the sequences (Eij): must also be different from the previous proposition. This is expected as from Theorem 2 Assumption [ implies identifiability, which in turn implies Assumption
Prop.Zales allows derive cases where Assumption I holds but not Assumption 2 The following Proposition gives a simple case where the model is identifiable but cannot be solved using Multiset CCA:
Proposition 3.
Assume that for two integers j.J', Jr, the sequence (Eig): ) is permutation of (Eay) ) Lee. that there exists a permutation of (1....,P). n, such that for all i, Eij- = Em(O)). Then, >3 = Ayr.
In this setting, Assumption [oolds so ShICA is identifiable, while A does does not hold, so Multiset CCA cannot recover the unmixing matrices.
3.2 Sampling noise and improved estimation with joint diagonalization
The consistency theory for Multiset CCA developed above is conducted under the assumption that the covariances Cij are the true covariances of the model, and not approximations obtained from observed samples. practice, however, a serious limitation of Multiset CCA is that even a slight error of estimation on the covariances, due to "sampling noise", can yield large error in the estimation of the unmixing matrices, as will be shown next.
We begin with an empirical illustration. We take m = 3, p 2, and Σ; such that A 2+ and A2 2 for 8 > O. In this way, we can control the eigen-gap of the problem, E. We take W; the outputs of Multiset CCA applied to the true covariances Ciz. Then, we generate a perturbation A = 8- S, where is a random positive symmetric pm pm matrix of norm 1, and 8 > 0 controls the scale of the pertur- bation. We take Ai the p x p block of A in position (i. j). and W; the output of Multiset CCA applied to the covariances Ci + Aij. We finally compute the sum of the Amari distance between the W; and We the Amari distance measures how close the two matrices are, up to scale and permutation (4)

---

displays the median Amari distance over 100 random repetitions, as the perturbation scale increases. The different curves correspond to different values of the eigen- gap s. We see clearly that the robustness of Multiset CCA critically depends on the eigen-gap, and when it is small, even a small perturbation of the input (due, for instance, to sampling noise) leads to large estimation errors.
10-1
Amari distance
10
10
10-7
10
10
10-10
10
This problem is very general and well studied [53]: the mapping from matrices to (generalized) eigenvectors is highly non-smooth. However, the gist of our method is that the span of the leading p eigenvectors is smooth, long as there is large enough gap between Ap and Ap+1. For our specific problem we have the following bounds, derived from Prop.2 2
10 10-4
10
100
Perturbation scale
Figure 1: Amari distance between true mixing matrices and estimates of Multi- set CCA when covariances are perturbed. Different solid curves correspond dif- ferent eigen- gaps. The black dotted line shows the chance level. When the gap is small, small perturbation can lead to complete mixing. doint-diagonalization (colored dotted lines) fixes the problem.
Proposition 4. We let maax maxij Zij and amin minsy Eng- Then, Ap 2 1+ 140max'
m-1 while Apt1 1
1+0
As consequence, we have Ap
Ap+1
140min I+omax the gap between these cigenvalues increases with m, and decreases with the noise power.
In this setting, when the magnitude of the perturbation is smaller than Ap - Ap+1. 153 indicates
that Span(W Wm [ Span(W,,... Wm. [n]T), where [W,,... W., E R is the
vertical concatenation of the W,'s. s. turn, this shows that there exists a matrix Q E Roxp such that
Wi QW, for all
We propose to joint-iiagonalizztion to recover the matrix Q. Given the WP', con- sider the of symmetric matrices K; W.C.W.WT where Cú is the contaminated covari- ance of x,. Following Eq. 44. we have QK,Q" QK,QT = PT,A, T.PT. Since Č₁ is close to Cn = Ai + E.)AT the matrix almost diagonal. In other words, the matrix Q is an approximate diago- nalizer of the K; ss, and we approximate Q by diagonalaaaninnoionnn of the K, 's In Fig | we see that this procedure mitigates the problems of Calltiset-CCAA, and gets uniformly better performance regardless of the eigen- gap. In practice, we use fast joint-diagonalization algorithm [ to minimize a joint-diagonalization criterion for positive symmetric matrices
W,C,W,T. and using Theorem 2 have
CTA,,
48 The estimated unmixing
matrices U, QW, correspond to the true unmixing matrices only up some scaling which may be different from subject to subject: the information that the components are of unit variance is lost. As a consequence, naive averaging of the recovered components may lead inconsistant estimation. We now describe a procedure to recover the correct scale of the individual components across subjects.
Scale estimation We form the matrices Tej U.C.,UT. In order estimate the scalings, we solve min Eijs [ (Tt)) Il where the &; are diagonal matrices. This function is readily min- imized with respect to of the +; by the formula
Algorithm 1 ShICA-J
Input: Covariances Cif = E[x,x]]
(W:): + Mu(tise)CCA((Cuuaa)
4, Eies 4 diag((.,))
Eins (Ydiag(Y,))* (derivations in Appendix 20 We then iterate the previous formula over until con- vergence. The final estimates of the unmixing trices are given by (6,U,)?11 The full procedure, called ShICA-J, is summarized in Algorithm
Q - (W..Diigg(WWW,,.....
Tú < QW.C. WT WTQQ
(#e); + Scaling(( (Tann)((TT))))
Return: Unmixing matrices ($:QW)),
3.3 Estimation of noise covariances
practice, it is important to estimate noise covari- ances E; in order to take advantage of the fact that
5

---

some views are noisier than others. As it is well known in classical factor analysis, modelling noise variances allows the model virtually discard variables, or subjects, that are particularly noisy.
Using the ShICA model with Gaussian components, we derive an estimate for the noise covariances directly from maximum likelihood. We use especially fast because noise updates are in closed-form. Following derivations given in appendix D.I the sufficient statistics in the E-step are given by
oppectaioonmmaxiiizziion (EM) algorithm, which is
m
-1
m
m
E[s|x]
Σ
E y:)
V
1)-1
(5)
Incorporating the M-step we get the following updates that only depend on the covariance matrices:
Σ, < diag(C; (Cts - 2V/ss|x شØ= b C3+ + V[s دリ= Σx= (1 C
V(s/x] + V[s|x)
4 ShICA-ML: Maximum likelihood for non- Gau-Gaussian components
ShICA-J only uses second order statistics. However, the ShICA model (I) allows for non- non-Gaussian components. We now propose an algorithm for fitting the ShICA model that combines covariance information with non-Gaussianity in the estimation to optimally separate both Gaussian and non- Gaussian components. We estimate the parameters by maximum likelihood. Since most non-Gaussian components in real data are super-Gassiian have the super-Gaussian density p(8,) (N(s;; 0, () 0,,;;
21 13
we assume that the non-Gaussian components
We propose to maximize the log- log-likelihood using generalized EM 41 22 Derivations are available
in Appendix EE] Like in the previous section, the E-step is in closed-form yielding the following sufficient statistics:
Eee(t Oa og and V[sg[x]
Ece
Oa
E att
E Ece Oa
E[sy|x] xX
Eae(0.5,1.5) Oa
where 0, NW;;; 0, Es + a). Dj
E.
and ΣS (, 2g 1 with y; Wxx, Noise Σ,
updates are in closed- form and given by: Σ, + (ong((y-Elss>> (y, E + V|s However, no closed-fore form is available for the updates of unmixing matrices. We therefore perform quasi- quasi-Newton updates given by W; + (I- )(? W. W,, where م E IR is chosen by backtracking line-
search, W.W,, abb..d Suddbe dad ( is approximation of the Hessian of the negative complete Ena likelihood and gW: = - + (E,)
(y:) the gradient.
We alternate between computing the statistics E[s]x], V/s|x] (E-step) and updates of parameters E; ...... m (M-step). Let us highlight that our EM algorithm and particular the E-step
and W; for 11...M resembles the one used in [40]. However because they assume noise on the sensors and not on the components, their formula for E[s[x] involves a sum with 2 terms whereas we have only terms. The resulting method is called ShICA-ML.
Minimum mean square error estimates in ShICA In ShICA-J well as in ShICA-ML. we have a closed-form for the expected components given the data TE[s) [x], shown in equation (5) and (6) respectively. This provides minimum mean square estimates of the shared components, and is an important benefit of explicitly modelling shared components in a probabilistic framework.
Related Work
ShICA combines theory and methods coming from different branches of "component analysis". can be viewed as a GroupICA method, as an extension of Multiset CCA, as an Independent Vector Analysis method or, crucially, as an extension of the shared response model. the setting studied here, ShICA improves upon all existing methods.

---

Gr GroupICA methods extract independent components from multiple datasets. In its original form 14. views are concatenated and then a PCA is applied yielding reduced data on which ICA is applied. One can also reduce the data using Multiset CCA instead of PCA, giving method called CanICA [58. Other works [24 32 apply ICA separately on the datasets and attempt to match the decompositions afterwards. Although these works provide very fast methods, they do not rely on a well defined model like ShICA. Other GroupICA methods impose some structure on the mixing matrices such as the tensorial method of [10 or the group tensor model in (27] (which assumes identical mixing matrices up to a scaling) or (54] (which assumes identical mixing matrices but different components). In ShICA the mixing matrices are only constrained to be invertible. Lastly, maximum- maximum-ikellhood based methods exist such as Multi Viewiii [51] (MVICA) or the full model of [27]. These methods are weaker than ShICA as they the same noise covariance across views and lack a principled method for shared response inference.
Multiset CCA In its basic formulation, CCA identifies shared space between two datasets. The extension to more than two datasets is ambiguous, and many different generalized CCA methods have been proposed. [33] introduces 6 objective functions that reduce to CCA when = 2 and [2] considered 4 different possible constrains leading to 24 different formulations of Multiset CCA. The formulation used in ShICA-J is refered to in 42 SUMCORR with constraint 4 which is one of the fastest as it reduces to solving a generalized eigenvalue problem. The fact that CCA solves a well defined probabilistic model has first been studied in [9] where it is shown that CCA is identical to multiple battery factor analysis [12 (restricted to 2 views). This latter formulation differs from our model in that the noise is added on the sensors and not the components which makes the model unidentifiable. Identifiable variants and generalizations can be obtained by imposing sparsity on the mixing matrices such as in [8 34 [2] or non-negativity (20) The work in [38] exhibits set of sufficient (but not necessary) conditions under which a well defined model can be learnt by the formulation of Multiset CCA used in ShICA-J. The set of conditions we exhibit in this work are necessary and sufficient. We further emphasize that basic Multiset CCA provides a poor estimator as explained in Section 3.2
Independent vector analysis Independent vector analysis 36 (IVA) models the data as a linear mixture of independent components x; A₁S; where each component Sty of given view can depend on the corresponding component in other views ((st)) ) are not independent). Practical implementations of this very general idea assume distribution for p((s:))(= In IVA-L (36),
(((s)))=1) x exp(- --y E [st))² (so the variance of each component in each view is assumed to be the same), in IVA-G 5 or in [60]. (((s;) y ~ NO, R.s)) and 23 proposed a normal averre-Gamma density. Let us also mention IVA-L-SOS [I. IVA-GGD
and IVA with Kotz distribution 6 that assume a non-Gaussian density general enough so that they can use both second and higher order statistics to extract view-specific components. The model of ShICA can be seen as instance of IVA which specifically enables extraction of shared components from the subject specific components, unlike previous versions of IVA. In fact, ShICA comes with minimum mean square error estimates for the shared components that is often the quantity of interest. The IVA theory provides global identifiability conditions in the Gaussian case (IVA-G) [59] and local identifiability conditions in the general case 7 from which local identifiability conditions of ShICA could be derived. However, in this work, we provide global identifiability conditions for ShICA. Lastly, IVA can be performed using joint diagonalization of cross covariances [37 [9] although multiple matrices have to be learnt and cross cross-covariances are not necessarily symmetric positive definite which makes the algorithm slower and less principled.
Shared response model ShICA extracts shared components from multiple datasets, which is also the goal of the shared response model (SRM) [66. The robust SRM (57] also allows to capture subject specific noise. However these models impose orthogonality constraints on the mixing matrices while ShICA does not. Deep variants of SRM exist such as [17] but while they release the orthogonality constrain, they are not very easy to train or interpret and have many hyper-parameters to tune. ShICA leverages ICA theory to provide much more powerful model of shared responses.
The main limitation of this work is that the model cannot reduce the dimension inside each view: : there are as many estimated sources sensors. This might be problematic when the number of sensors is very high. In line with other methods, view-specific dimension reduction has to be done by some external method, typically view-specific PCA. Using specialized methods
Limitations
7

---

ShICA-ML
ShICA-J
Multiset CCA
CanICA
MVICA
Amari distance
10a
10-5
(a)
(c)
10a
10a
10
10a
10
10*
10a
10a
10
Number of samples
Figure 2: Separation performance: Algorithms are fit on data following model (a) Gaussian
components with noise diversity (b) Non-Gaussian components without noise diversity (c) Half of the components are Gaussian with noise diversity, the other half is non-Gaussian without noise diversity.
for the estimation of covariances should also be of interest for ShICA-J, where it only relies on sample covariances. Finally, ShICA-ML uses simple model of Super-Gausian distribution, while modelling the non-gaussianities in more detail in ShICA-MI should improve the performance.
Experiments
Experiments used Nilearn 3 and MNE [26 for fMRI and MEG data processing respectively, as well
as the scientific Python ecosystem: Matplotlib 31, Scikit-leam 455. Numpy (29 and Scipy [61]. We
use the Picard algorithm for non-Gaussian ICA [2] and mvlearn for multi-view ICA 47). The above
libraries use open-source licenses. fMRI experiments used the following datasets: sherlock [[5)
forrest (28 raiders [49] and gallant [4) The data we use do not contain offensive content or
identifiable information and consent was obtained before data collection. Computations were run on
a large server using up 100 GB of RAM and 20 CPUs in parallel.
Separation performance In the following synthetic experiments, data are generated according to model II with p = 4 components and m = 5 views and mixing matrices are generated by sampling coefficients from standardized Gaussian. Gaussian components are generated from standardized
Gaussian and their noise has standard deviation Σ (obtained by sampling from uniform density between and 1) while non non-Gaussian components are generated from a Laplace distribution and their noise standard deviations components are non-Gause Gaussian or half of the components are Gaussian and half are non- non-Gausiian. We vary the number of samples between 102 and 10 and display in Fig 2 the mean Amari distance across subjects between the true unmixing matrices and estimates of algorithms as function of n. The experiment is repeated 100 times using different seeds. We report the median result and error bars represent the first and last deciles.
equal. We study 3 cases where either all components are Gaussian, all
When all components Gaussian (Fig. (a)), CanICA cannot separate the components all. In contrast ShICA-J, ShICA-MI, Multiset CCA and MVICA are able to separate them, but Multiset CCA needs many shows that correcting for the rotation due to sampling noise improves the results. Looking at error bars, we also that the performance of Multiset CCA varies quite lot with the random seeds: this shows that depending on the sampling noise, the rotation can be very different from identity. MVICA sample than Multiset CCA to reach low amari distance but still outperforms
samples than ShICA-J or ShICA-ML to reach low amari distance, which
needs even CanICA.
When none of the components are Gaussian (Fig. (b)), only CanICA, ShICA-ML and MVICA able to separate the components, as other methods do not make use of non- -ou-Gaussiantty. Finally, in the hybrid case (Fig. 2 (c)), ShICA-ML is able to separate the components as it can make use of both non-Gaussianity and noise diversity. MVICA is a lot less reliable than ShICA-ML, is uniformly worse and error bars are very large showing that for some seeds it gives poor results. CanICA, ShICA-J and MultisetCCA cannot separate the components at all, Additional experiments illustrating the separation powers of algorithms
available in Appendix H.I
As we can see, MVICA can separate Gaussian components to some extent and therefore does not completely fail when Gaussian and non-Gaussian components present. However MVICA is lot

---

SHICA-ML
Multisct OCA
MVICA
10
ShiCA-J
CanICA
8 moem
Pitting time (in s)
10a
10"2
(In
109
10-1
SMCA
SMFiA-J
Moltinn CGA
MVICA
CarCA
10-4
10-3
10-2
10-3
10P
Amari distance
(a)
Figure 3: Left: Computation time. Algorithms fit on data generated from model with Gaper-Gaussian density. For different values of the number of samples, we plot the Amari distance and the fitting time. Thick lines link median values across seeds. Right: Robustness war. intra-subject variability in MEG. (top) 22 distance between shared components corresponding to the same stimuli in different trials. (bottom) Fitting time.
less reliable than ShICA-ML: MVICA is uniformly worse than ShICA-ML and the error bars are very large showing that for some seeds it gives poor results.
Computation time We generate components using slightly super Gaussian density: 8, d(x) with d(xx) = _r/=:0.2 and x (0, 1). We vary the number of samples between 102 and 10 We compute the mean Amari distance across subjects and record the computation time. The experiment repeated 40 times. We plot the Amari distance as function of the computation time in Fig Ga] Each point corresponds to the Amari distance/computation time for a given number of samples and a given seed. We then consider for given number of samples, the median Amari distance and computation time across seeds and plot them the form thick line. From Fig [a we see that ShICA-J the method of choice when speed is a concern while ShICA-ML yields the best performance in terms of Amari distance at the cost of an increased computation time. The thick lines for ShICA-J and Multiset CCA are quasi-flat indicating that the number of samples does not have strong impact on the fitting time as these methods only work with covariances. On the other hand CanICA or MVICA computation time is more sensitive the number of samples.
In the following experiments we consider the Cam-CAN dataset [56). We use the magnetometer data from the MEG m 100 subjects chosen randomly among 496. In appendix F we give more information about Cam-CAN dataset. Each subject is repeatedly presented three audio-visual stimuli. For each stimulus, we divide the trials into two sets and within each set, the MEG signal is averaged across trials to isolate the evoked response. This procedure yields chunks of individual data per stimulus). We study the similarity between shared components corresponding to repetitions of the same stimulus. This gives measure of robustness of each ICA algorithm with respect to intra-subject variability. Data are first reduced using a subject-speciff PCA with 10 components. The initial dimensionality of the data before PCA 102 as we only use the 102 magnetometers. Algorithms are run times with different seeds on the 6 chunks of data, and shared components to repetitions of the same stimulus they should yield similar components. For each component and for each stimulus, we therefore measure the & distance between the two repetitions of the stimulus. This yields 300 distances per algorithm that are plotted on Fig 3b
Robustness w.r.t intra-subject variability in MEG
extracted. When two chunks of data correspond
The components recovered by ShICA-ML have much lower variability than other approaches. The performance of ShICA-J competitive with MVICA while being much faster fit. Multiset CCA yields satisfying results compared with ShICA-J. However we see that the number of components that do not match at all across trials is greater Multiset CCA. Additional experiments on MEG data are available in Appendix H.3

---

Reconstructing the BOLD signal of missing subjects We reproduce the experimental pipeline of [5T] bench- mark GroupICA methods using their ability to recon- struet fMRI data of a left-out subject. The preprocessing involves a dimension reduction step performed using the shared response model [16]. Detailed preprocessing We call unmix-
ShiCA-J
CanICA
Forrest
gallarnt
R s00m
0.20
6.126
6.300
0.15
6.076
pipeline is described in Appendix F ing operator the product of the dimension reduction operator and unmixing matrix and mixing operator its pseudoinverse. There is one unmixing operator and one mixing operator per view. The unmixing operators are learned using all subjects and 80% of the runs. Then they applied the remaining 20% of the runs using 80% the subjects yielding unmixed data from which shared components
Faiders
sberlock
5c0re
0.15
Number componestts
sherlock
forrest
extracted. The unmixed data combined by averaging (for SRM and other baselines) or using the MMSE estimate for ShICA-J and ShICA- ML. We then apply the mixing operator of the remaining 20% subjects on the shared components reconstruct their data. Reconstruction accuracy is measured via the coefficient of determination, a.k.a. R² score, that yields for each voxel the relative discrepancy between the true time course and the predicted one. For each compared algorithm, the experiment is run 25 times with different seeds to obtain error bars. We report the mean R² score across voxels in region of interest (see Appendix [+ for details) and display the results in Fig 4 The error bars represent a 95% confidence interval. The chance level is given by the R² score of an algorithm that samples the coefficients its unmixing matrices and dimension reduction operators from standardized Gaussian. The median chance level is below 10" on all datasets. ShICA-ML yields the best R² score all datasets and for any number of components. ShICA-J yields competitive results with respect to MVICA while being much faster to fit. A popular benchmark especially in the SRM community is the time-segment matching experiment [16] we include such experiments in Appendix [.2)] In appendix G we give the performance of ShICA-ML, ShICA-J and MVICA in form of a table.
107
Fis time 6)
10P
z01-
Faiders
gallant
107-
102
50
Namber of componenta
Figure 4: Reconstructing the BOLD sig- nal of missing subjects. (top) Mean Rt score between reconstructed data and true data. (bottom) Fitting time.
7 Conclusion, Future work and Societal impact
We introduced the ShICA model as a principled unifying solution the problems of shared response modelling and GroupICA. ShICA is able to use both the diversity of Gaussian variances and non- Gaussianity for optimal estimation. We presented two algorithms to fit the model: ShICA-J, a fast algorithm that uses noise diversity, and ShICA-ML, a maximum likelihood approach that can use non-Gan Con--aassiantty on top of noise diversity. ShICA algorithms come with principled procedures for shared components estimation, as well as adaptation and estimation of noise levels in each view (subject) and component. On simulated data, ShICA clearly outperforms all competing methods in terms of the trade-off between statistical accuracy and computation time. On brain imaging data, ShICA gives more stable decompositions for comparable computation times, and more accurately predicts the data of one subject from the data of other subjects, making it good candidate to perform transfer learning. Our code is available at hhtpps:///// / giuh........ com/hugort caaggorichar//Sh....
'Regarding the ethical aspects of this work, we think this work presents exactly the same issues as any brain imaging analysis method related to ICA.
10


Reviews:
[{"review_id": "yFGU1vSDGAR", "reviewer": "Reviewer_v4RY", "summary": "The authors propose the shared independent component analysis which can account for shared components $\\mathbf{s}$ and individual differences through the noise signal $\\mathbf{n}$. Notably the noise can vary across subjects and views in magnitude whereas the shared components can both be Gaussian or non-Gaussian (through an invoked super-gaussian two component fixed gaussian mixture). The model is optimized through an associated EM procedure in which mean and variance of the shared sources are estimated in the E step and the associated mixing matrices and noise variances estimated in the M step. As no closed form solution for the M step exists when considering super-Gaussian noise sources an associated Newton optimization procedure is here invoked. The procedure is demonstrated on synthetic and real datasets to provide superior recovery when compared to existing component models including canICA, multiview ICA and multiset CCA whereas mathematical properties in regards to model identifiability are carefully derived.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript is well written and considers a well studied problem with an extensive review of the existing approaches well positioning the current approach within the relevant literature on independent vector analysis, group ICA, multiview ICA and multiset canonical correlation analysis.\n\n$\\textbf{Originality}$\nThe approach is rather straightforward however the mathematical derivations of properties appear non-trivial. In particular, the result of identifiability even facing Gaussian sources through properties of the individual noise covariance matrices $\\Sigma_i$ is useful and differentiates this approach in general from conventional methods. Furthermore, the experimental evaluation is solid and the approach found to be meritable providing enhanced recovery. All in all the mathematical rigour and the convincing experimental section can warrant publication of this manuscript.\n\n$\\textbf{Quality}$\nThe manuscript is well written and sound. The experimentation is convincing and includes an extensive synthetic analysis as well as an analysis of real magneto-encephalography (MEG) data. The results are promising and the proposed approach appears both efficient and accurate in terms of component recovery.\n\n$\\textbf{Clarity}$\nThe manuscript is well written and clear in its presentation. \n\nIn proof of theorem 2 note:that $s + n^i$ should be $s + n_i$\n\nIt is unclear to me how the approach relates and differ to the approach of \nKlami, Arto, Seppo Virtanen, and Samuel Kaski. \"Bayesian Canonical Correlation Analysis.\" Journal of Machine Learning Research 14.4 (2013).\nIn particular, this approach also accounts for view specific covariances and seems very related to the current approach when not including also individual factors but only shared factors and not imposing the mixture of two Gaussian prior. The authors thus need to relate their approach to this work.\n\n$\\textbf{Significance}$\nThe manuscript falls within a large literature on decomposition approaches accounting for individual variability while preserving shared component extraction. The work is carefully positioned within this literature and forms a promising although rather straightforward framework. However, the merits in terms of model identifiability are interesting and admits Gaussian component extraction providing efficiency and favorable results as demonstrated in the experimental validation which warrents publication.\n\n\nResponse rebuttal:\nI have read all reviews and rebuttals. I am satisfied with the authors’ response to my concerns and still find the manuscript marginally above the acceptance threshold. I find the experimental section sufficiently convincing and although the approach is rather straightforward it has favorable merits whereas non-trivial derivations of properties are provided which I find can warrant publication. I am thus inclined to keep my marginal accept score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact is sufficiently discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "The analysis is based on existing datasets published in [45].", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "lEfzT7JeQUH", "reviewer": "Reviewer_vUQJ", "summary": "This paper proposed a Shared Independent Component Analysis (ShICA) that models each dataset as a linear transform of shared independent components contaminated by additive Gaussian noise.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The author proposed a Shared ICA for multi-subject data which can extract both Gaussian and non-Gaussian components. Some major concerns:\na.\tIn simulation, the author only simulated 4 components and 5 subjects’ data to do the comparisons, which is too small to test the performance of the proposed method. At least the subject number N>100, and component number around 10.  \nb.\tThe well-known GICA and IVA are not compared with the proposed method.\nc.\tThere is no independence constraint in mCCA, so it is obviously that mCCA can’t deal with noise as stated in abstract. It is unfair to compare mCCA with other ICA related method with respect to noise. \nd.\tApart from simulation, no real fMRI data decomposition was compared. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In term of novelty and performance improvement, the current paper is not enough. The overall quality is poor, and it does not meet the publication requirement.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "YMIPM7vd7T", "reviewer": "Reviewer_6yar", "summary": "The paper examines a multi-view learning problem in which one wants to identify common shared components from multiple datasets. The proposed method, Shared Independent Component Analysis (ShICA) has identifiability if components express a sufficient level of non-Gaussianity as well as enough diversity in noise variation. The key to ShICA that makes it different from the common multiset CCA is that it incorporates joint diagonalization across the views. The paper additionally introduces a maximum likelihood version of the proposed method, ShICA-ML, which is more accurate but less computationally efficient.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper does an excellent job of clearly identifying the current methods used in independent component extraction from multiple views (in section 5, Related work). Although the reviewer sees why the related methods are added right before the results section (to provide benchmarks), he was not certain as to the main contributions of the model until reading this section. It would be useful to add a clearer description in the introduction as to the main limitations of the current methods and how ShICA addresses these. A short paragraph explaining this would do. The reviewer also appreciates the candid explanation of limitations for the current ShICA method.\n\nThe presentation of the results are clear, which show that ShICA has a clear advantage in regards to separation performance, computation time, robustness with respect to intra-subject variability in MEG, reconstructing the BOLD signal of missing subjects.\n\nThe paper is very excellent and clearly presents the methodology and results in a way that convince the reader the advantages of the method. The main suggestion from the reviewer is to make some of the advantages of ShICA more clear in the paper (add a paragraph in the introduction to explain the main contributions of ShICA and maybe even ShICA-ML).\n\nHowever, after reading some other reviewer comments, I have some concerns about the methodology and originality of the paper. Therefore, I must give a lower score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "30 minutes"}, "extra_text": {}}, {"review_id": "LNwBh9PGHV", "reviewer": "Reviewer_Mkd2", "summary": "This paper develops Shared Independent Component Analysis (ShICA) that learns a linear transform of shared independent components from a multi-view dataset. The ShICA is identifiable when the components are non-Gaussian or if the noise variances show “enough diversity”. A number of empirical studies have demonstrated that the proposed method outperforms the classic CCA on fMRI and MEG datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The following are the major concerns and minor comments:\n\n1)  In this paper, the empirical studies are reported using Amari distance and R2 score. It makes the produced results unclear. Indeed, we cannot compare the performance of the proposed method with the previous studies (e.g., Chen et al. 2015) that evaluated the related methods with the same datasets. Moreover, I cannot find clear information about the experiment scheme, including how cross-validation is applied (across subjects or instances), how the neural responses are learned after mapping to the shared space (in the same order, or in shuffled form), etc. I would like to compare the performance of the proposed method with other approaches using the regular metrics like classification accuracy and in the form of different schemes proposed in other related studies, including Chen et al. 2015, etc.\n\n2) In this paper, performances of the various techniques are reported by manually selecting the hyperparameters --- e.g., number of components in Figure 5. The optimum values for each hyperparameter should be learned using internal cross-validation or a grid search. In the current format, we cannot have any inference from the empirical studies. Here are two problems: imagine the number of components. There is no guarantee that different techniques will produce the same performance with the same number of components. You must evaluate each technique with its best performance, obviously using the optimum hyperparameters. In addition, it seems that the performances of most techniques are increasing when the number of components is increasing. You have to show us when this tendency is not consistent anymore. Then we can compare the best of other techniques (e.g., SRM) with the best of yours.\n\n3) In this paper, handling non-Gaussian noise is described as a novelty of the proposed method in comparison with CCA-based approaches such as SRM. However, I did not find an explanation to see why it is important? In fMRI datasets, each voxel shows the BOLD signal as a proxy to trace the oxygen usage of around 10000 neurons. We can assume that the distribution of one voxel (the BOLD signal) is related to the aggregation of all distributions corresponding to the oxygen usage of the associated neurons. Although we obviously do not have each of these distributions, the central limit theorem clearly explains that the Gaussian distribution may be the best assumption on the voxel (aggregated) distribution. To keep the procedure conjugate during the factor analysis, assuming the Gaussian noise in the other techniques makes more sense --- for instance, it can even help us to control SNR.\n\n4) There are several related works that should be compared with the proposed method. For instance, SRM has been published in 2015 and there are several new extensions of SRM that can produce better performances in comparison with SRM, including MN-SRM (AISTAT 2018) and RSRM (ICASSP 2018).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Empirical studies and scientific evidence are my primary concerns. The proposed method was not compared with the related state-of-the-art techniques. Furthermore, the empirical studies did not follow the same scheme as other studies for evaluating learned models in a trustable manner. For instance, hyperparameters (such as the number of components) are not tuned to produce the best performance for each technique. As another example, they did not report the prevalent evaluation metrics such as the classification accuracy that is used in other studies to compare the performances. It is hard to compare the produced results with other studies that are even using the same datasets.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "40"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers recommend rejection and two reviewers recommend acceptance. After reading the reviews, the rebuttal, the internal discussion among reviewers and after my own reading of the paper, I believe this work provides a genuine contribution to ICA for multiple views and therefore I recommend Accept. The method provides identifiable components (under certain restrictions) which makes it robust, and the empirical section shows this advantage of the method and its usefulness in downstream tasks, e.g. reconstructions of left-out subjects in fMRI data. Two reviewers that recommend rejection have mainly asked for comparisons against other baselines. However, I believe that the authors have included suitable baselines when testing their model. I recommend the authors follow the recommendations of the reviewers regarding the presentation and fixing any typos.
Decision: Accept (Poster)