Shared Independent Component Analysis for Multi-Subject Neuroimaging

Abstract: We consider shared response modeling, a multi-view learning problem where one wants to identify common components from multiple datasets or views. We introduce Shared Independent Component Analysis (ShICA) that models eachview as a linear transform of shared independent components contaminated by additive Gaussian noise. We show that this model is identifiable if the components are either non-Gaussian or have enough diversity in noise variances. We then show that in some cases multi-set canonical correlation analysis can recover the correct unmixing matrices, but that even a small amount of sampling noise makes Multiset CCA fail. To solve this problem, we propose to use joint diagonalization after Multiset CCA, leading to a new approach called ShICA-J. We show via simulations that ShICA-J leads to improved results while being very fast to fit. While ShICA-J is based on second-order statistics, we further propose to leverage non-Gaussianity of the components using a maximum-likelihood method, ShICA-ML, that is both more accurate and more costly. Further, ShICA comes with a principled method for shared components estimation. Finally, we provide empirical evidence on fMRI and MEG datasets that ShICA yields more accurate estimation of the componentsthan alternatives.


Reviews and author responses:

The authors propose the shared independent component analysis which can account for shared components $\mathbf{s}$ and individual differences through the noise signal $\mathbf{n}$. Notably the noise can vary across subjects and views in magnitude whereas the shared components can both be Gaussian or non-Gaussian (through an invoked super-gaussian two component fixed gaussian mixture). The model is optimized through an associated EM procedure in which mean and variance of the shared sources are estimated in the E step and the associated mixing matrices and noise variances estimated in the M step. As no closed form solution for the M step exists when considering super-Gaussian noise sources an associated Newton optimization procedure is here invoked. The procedure is demonstrated on synthetic and real datasets to provide superior recovery when compared to existing component models including canICA, multiview ICA and multiset CCA whereas mathematical properties in regards to model identifiability are carefully derived.  The manuscript is well written and considers a well studied problem with an extensive review of the existing approaches well positioning the current approach within the relevant literature on independent vector analysis, group ICA, multiview ICA and multiset canonical correlation analysis.

$\textbf{Originality}$
The approach is rather straightforward however the mathematical derivations of properties appear non-trivial. In particular, the result of identifiability even facing Gaussian sources through properties of the individual noise covariance matrices $\Sigma_i$ is useful and differentiates this approach in general from conventional methods. Furthermore, the experimental evaluation is solid and the approach found to be meritable providing enhanced recovery. All in all the mathematical rigour and the convincing experimental section can warrant publication of this manuscript.

$\textbf{Quality}$
The manuscript is well written and sound. The experimentation is convincing and includes an extensive synthetic analysis as well as an analysis of real magneto-encephalography (MEG) data. The results are promising and the proposed approach appears both efficient and accurate in terms of component recovery.

$\textbf{Clarity}$
The manuscript is well written and clear in its presentation. 

In proof of theorem 2 note:that $s + n^i$ should be $s + n_i$

It is unclear to me how the approach relates and differ to the approach of 
Klami, Arto, Seppo Virtanen, and Samuel Kaski. "Bayesian Canonical Correlation Analysis." Journal of Machine Learning Research 14.4 (2013).
In particular, this approach also accounts for view specific covariances and seems very related to the current approach when not including also individual factors but only shared factors and not imposing the mixture of two Gaussian prior. The authors thus need to relate their approach to this work.

$\textbf{Significance}$
The manuscript falls within a large literature on decomposition approaches accounting for individual variability while preserving shared component extraction. The work is carefully positioned within this literature and forms a promising although rather straightforward framework. However, the merits in terms of model identifiability are interesting and admits Gaussian component extraction providing efficiency and favorable results as demonstrated in the experimental validation which warrents publication.


Response rebuttal:
I have read all reviews and rebuttals. I am satisfied with the authors’ response to my concerns and still find the manuscript marginally above the acceptance threshold. I find the experimental section sufficiently convincing and although the approach is rather straightforward it has favorable merits whereas non-trivial derivations of properties are provided which I find can warrant publication. I am thus inclined to keep my marginal accept score.
 Limitations and societal impact is sufficiently discussed.

The paper examines a multi-view learning problem in which one wants to identify common shared components from multiple datasets. The proposed method, Shared Independent Component Analysis (ShICA) has identifiability if components express a sufficient level of non-Gaussianity as well as enough diversity in noise variation. The key to ShICA that makes it different from the common multiset CCA is that it incorporates joint diagonalization across the views. The paper additionally introduces a maximum likelihood version of the proposed method, ShICA-ML, which is more accurate but less computationally efficient.  The paper does an excellent job of clearly identifying the current methods used in independent component extraction from multiple views (in section 5, Related work). Although the reviewer sees why the related methods are added right before the results section (to provide benchmarks), he was not certain as to the main contributions of the model until reading this section. It would be useful to add a clearer description in the introduction as to the main limitations of the current methods and how ShICA addresses these. A short paragraph explaining this would do. The reviewer also appreciates the candid explanation of limitations for the current ShICA method.

The presentation of the results are clear, which show that ShICA has a clear advantage in regards to separation performance, computation time, robustness with respect to intra-subject variability in MEG, reconstructing the BOLD signal of missing subjects.

The paper is very excellent and clearly presents the methodology and results in a way that convince the reader the advantages of the method. The main suggestion from the reviewer is to make some of the advantages of ShICA more clear in the paper (add a paragraph in the introduction to explain the main contributions of ShICA and maybe even ShICA-ML).

However, after reading some other reviewer comments, I have some concerns about the methodology and originality of the paper. Therefore, I must give a lower score. Yes.

 Dear reviewer,

We thank you for taking time to review our article and for your insightful comments. 

1. (notation issue in proof of Theorem 2) Indeed, we will correct this

2. (comparison to Bayesian CCA) Unless we missed something, the Bayesian CCA uses a Gaussian prior for the shared components (which is why it is a CCA method). This changes many things because it cannot use non-Gaussianity to separate sources. In this regard it is similar to ShICA-J. However we believe that ShICA-J is much faster to fit (though in order to be so fast, it does not use maximum likelihood estimators). Furthermore, our method is based on an exact likelihood computation, while the Bayesian CCA needs a variational approximation. Lastly, a big difference in Bayesian CCA is that the noise is added outside as opposed to our approach where the noise is added inside. As you already noticed, this is what makes our approach so efficient.

About our source density, it is not so problematic to use in the model a density that is not the correct one. In one-view ICA any super Gaussian density can be used to recover super Gaussian sources from a linear mixture (Cardoso, J. F. (1998). Blind signal separation: statistical principles. Proceedings of the IEEE, 86(10), 2009-2025.). While things are more complex in ShICA, we still see experimentally that our density can recover the sources even when the exact density is unknown. Note that in the synthetic experiments Figure 2, 3 and 6 we never used the density of the model but always a different one.

3. (straightforward framework) We find it a bit surprising to see as a drawback the fact that the model is “straightforward”. We tend to think that “simple” models are easier to interpret and understand and often a lot faster to fit so we would rather see this as an advantage of our approach. As you noted, the proofs of the properties we derive are not straightforward at all. We believe this is also what is beautiful: a seemingly simple model may be very powerful !

 Dear reviewer,

We thank you for taking time to review our article and for your insightful comments. 

1. In appendix F, some information about the experimental design are available but others are missing. Descriptions of the study cohort and experimental design are too long to be included in the rebuttal. We refer the reviewer to [12] for Sherlock and [23] for Forrest. The cohort description for Raiders and Gallant is available in [40]. The protocol used for Raiders is the same as the one used in (Haxby, Neuron 2011) and the protocol used for Gallant is the same as the one used in (Nishimoto, Current Biology 2011). We will include the corresponding references in appendix F.
From the paragraph corresponding to the reconstruction experiment: “the unmixing operators are learned using all subjects and 80% of the runs. Then they are applied on the remaining 20% of the runs using 80% of the subjects yielding unmixed data from which shared components are extracted.”. The unmixed data are combined by averaging (for SRM and other baselines) or using the MMSE estimate for ShICA-J and ShICA-ML. We will add this sentence to the above paragraph.
The experiment you describe (using accuracy instead of R2 score) is in fact available in appendix Figure 8. It is actually not the same datasets as in Chen at al. 2015: we use full brain data while they use ROIs. Also note that the Raiders data is different: it involves different subjects and was acquired in a different neuroimaging center. However, we used the brainiak code for SRM (https://github.com/brainiak/brainiak/blob/master/brainiak/funcalign/srm.py). The experimental code will be made public upon acceptance. (Note that the code for ShICA is already available in the supplementary material along with some examples to show that it works).

2. We believe it is actually more informative to give the performance of the methods on a range of hyper-parameters than to select a particular one. While we agree that it would be great to have the results with even more components, SRM tends to be very slow when the number of components is on the order of 100-200 (since we are using full brain data). The fact that ShICA is better for any number of components we tried, convinced us that ShICA is indeed a good technique.

3. We believe you may have misunderstood this part. We do not assume that the noise is non-Gaussian, just the sources, as is common practice in ICA.  The fact that brain sources are mostly super Gaussian is acknowledged in the work of (Delorme, A., Palmer, J., Onton, J., Oostenveld, R., & Makeig, S. (2012). Independent EEG sources are dipolar. PloS one, 7(2), e30135.) and  (Calhoun, V. D., & Adali, T. (2006). Unmixing fMRI with independent component analysis. IEEE Engineering in Medicine and Biology Magazine, 25(2), 79-90.). 
Note that ShICA can also separate Gaussian sources (see results on synthetic experiments Figure 2 and 6). 

4. Judging by Figure 4 in the MN-SRM paper, there is no improvement compared to classical SRM on the classification task. While RSRM is slightly better, it has one additional hyper-parameter to tune, which makes it less practical to use than SRM. For these reasons, we decided to only benchmark SRM. While we believe it would be interesting to benchmark against other baselines, we believe the benchmarks we performed are extensive and convincing enough.


 Dear reviewer,

We thank you for taking time to review our article and for your insightful comments. 

a. While we agree that N > 100 and k=10 would be a more realistic setting, we believe our message is completely independent of this. We chose these numbers so that the computation time of all methods benchmarked in Figure 2 and 3 is small enough. Note that the number of time samples is very large which allows for statistically significant estimates.
When performing the experiments on real data, these methods were used in large scale settings. The experiments on the MEG CamCAN dataset used either 496 subjects (with 10 components that are displayed in appendix section G.4) or 100 subjects and 10 components (to produce Figure 4).

b. As highlighted in [41] GICA usually performs worse than CanICA. Note that in some implementations (such as the one in GIFT), the PCA in GICA uses whitening and therefore is equivalent to CanICA.
IVA is not compared because it does not provide a shared response.

c. We believe you may have misunderstood this part. The interesting thing about mCCA is that it can provably be used to fit ShICA (under some assumptions including noise diversity). Therefore, it **can** deal with the *modelling* noise in the ShICA model. Where it fails is when there is *sampling* noise, i.e. inaccuracies in the estimation of the covariances due to a limited number of samples. As demonstrated in Figure 2, ShICA-J and mCCA do not use non-Gaussianity to separate sources but noise diversity.   In contrast, classical ICA methods such as CanICA can use non Gaussianity but not noise diversity. From experiments on real and synthetic data we see that both capabilities are useful to find the sources. ShICA-ML can exploit both.

d. We strongly believe visual inspection of decompositions is not a good way to compare ICA based methods. It is not an objective measure and one can always find a good explanation for how a set of components look. In contrast we rely on objective measures based on real data experiments: robustness, time segment matching accuracy or R2 reconstruction scores (see figures 4, 5, 8, 9, 10) .
We validate our methods on multiple modalities.  It is clearly not true that we only used simulated data. See the experiments Figures 4, 5, 8, 9, 10.
Note that a decomposition on MEG data is available in section G.4 of the appendix.


 Dear reviewer,

We thank you for taking time to review our article and for your insightful comments.
We will follow your suggestion and describe in more details the advantages of ShICA in the introduction.

This paper proposed a Shared Independent Component Analysis (ShICA) that models each dataset as a linear transform of shared independent components contaminated by additive Gaussian noise.  The author proposed a Shared ICA for multi-subject data which can extract both Gaussian and non-Gaussian components. Some major concerns:
a.	In simulation, the author only simulated 4 components and 5 subjects’ data to do the comparisons, which is too small to test the performance of the proposed method. At least the subject number N>100, and component number around 10.  
b.	The well-known GICA and IVA are not compared with the proposed method.
c.	There is no independence constraint in mCCA, so it is obviously that mCCA can’t deal with noise as stated in abstract. It is unfair to compare mCCA with other ICA related method with respect to noise. 
d.	Apart from simulation, no real fMRI data decomposition was compared. 
 In term of novelty and performance improvement, the current paper is not enough. The overall quality is poor, and it does not meet the publication requirement.

This paper develops Shared Independent Component Analysis (ShICA) that learns a linear transform of shared independent components from a multi-view dataset. The ShICA is identifiable when the components are non-Gaussian or if the noise variances show “enough diversity”. A number of empirical studies have demonstrated that the proposed method outperforms the classic CCA on fMRI and MEG datasets.  The following are the major concerns and minor comments:

1)  In this paper, the empirical studies are reported using Amari distance and R2 score. It makes the produced results unclear. Indeed, we cannot compare the performance of the proposed method with the previous studies (e.g., Chen et al. 2015) that evaluated the related methods with the same datasets. Moreover, I cannot find clear information about the experiment scheme, including how cross-validation is applied (across subjects or instances), how the neural responses are learned after mapping to the shared space (in the same order, or in shuffled form), etc. I would like to compare the performance of the proposed method with other approaches using the regular metrics like classification accuracy and in the form of different schemes proposed in other related studies, including Chen et al. 2015, etc.

2) In this paper, performances of the various techniques are reported by manually selecting the hyperparameters --- e.g., number of components in Figure 5. The optimum values for each hyperparameter should be learned using internal cross-validation or a grid search. In the current format, we cannot have any inference from the empirical studies. Here are two problems: imagine the number of components. There is no guarantee that different techniques will produce the same performance with the same number of components. You must evaluate each technique with its best performance, obviously using the optimum hyperparameters. In addition, it seems that the performances of most techniques are increasing when the number of components is increasing. You have to show us when this tendency is not consistent anymore. Then we can compare the best of other techniques (e.g., SRM) with the best of yours.

3) In this paper, handling non-Gaussian noise is described as a novelty of the proposed method in comparison with CCA-based approaches such as SRM. However, I did not find an explanation to see why it is important? In fMRI datasets, each voxel shows the BOLD signal as a proxy to trace the oxygen usage of around 10000 neurons. We can assume that the distribution of one voxel (the BOLD signal) is related to the aggregation of all distributions corresponding to the oxygen usage of the associated neurons. Although we obviously do not have each of these distributions, the central limit theorem clearly explains that the Gaussian distribution may be the best assumption on the voxel (aggregated) distribution. To keep the procedure conjugate during the factor analysis, assuming the Gaussian noise in the other techniques makes more sense --- for instance, it can even help us to control SNR.

4) There are several related works that should be compared with the proposed method. For instance, SRM has been published in 2015 and there are several new extensions of SRM that can produce better performances in comparison with SRM, including MN-SRM (AISTAT 2018) and RSRM (ICASSP 2018).
 Empirical studies and scientific evidence are my primary concerns. The proposed method was not compared with the related state-of-the-art techniques. Furthermore, the empirical studies did not follow the same scheme as other studies for evaluating learned models in a trustable manner. For instance, hyperparameters (such as the number of components) are not tuned to produce the best performance for each technique. As another example, they did not report the prevalent evaluation metrics such as the classification accuracy that is used in other studies to compare the performances. It is hard to compare the produced results with other studies that are even using the same datasets.

Meta-review: Two reviewers recommend rejection and two reviewers recommend acceptance. After reading the reviews, the rebuttal, the internal discussion among reviewers and after my own reading of the paper, I believe this work provides a genuine contribution to ICA for multiple views and therefore I recommend Accept. The method provides identifiable components (under certain restrictions) which makes it robust, and the empirical section shows this advantage of the method and its usefulness in downstream tasks, e.g. reconstructions of left-out subjects in fMRI data. Two reviewers that recommend rejection have mainly asked for comparisons against other baselines. However, I believe that the authors have included suitable baselines when testing their model. I recommend the authors follow the recommendations of the reviewers regarding the presentation and fixing any typos.

Decision: accept