Multi-View Independent Component Analysis with Shared and Individual Sources
Authors: Teodora Pandeva
Patrick Forré
Venue: Submitted to ICLR 2023 (2023)
Keywords: multiview independent component analysis
independent component analysis
blind source separation
multiview representation learning

Abstract: Independent component analysis (ICA) is a blind source separation method for linear disentanglement of independent latent sources from observed data. We investigate the special setting of noisy linear ICA where the observations are split among different views, each receiving a mixture of shared and individual sources. We prove that the corresponding linear structure is identifiable, and the shared sources can be recovered, provided that sufficiently many diverse views and data points are available. To computationally estimate the sources, we optimize a constrained form of the joint log-likelihood of the observed data among all views. We show empirically that our objective recovers the sources in high dimensional settings, also in the case when the measurements are corrupted by noise. Finally, we apply the proposed model in a challenging real-life application, where the estimated shared sources from two large transcriptome datasets (observed data) provided by two different labs (two different views) lead to a more plausible representation of the underlying graph structure than existing baselines.

Under review as a conference paper at ICLR 2023
MULTI- INDEPENDENT COMPONENT ANALYSIS WITH SHARED AND INDIVIDUAL SOURCES
Anonymous authors Paper under double-blind review
ABSTRACT
Independent component analysis (ICA) is a blind source separation method for linear disentanglement of independent latent sources from observed data. We in- vestigate the special setting of noisy linear ICA where the observations are split among different views, each receiving mixture of shared and individual sources. We prove that the corresponding linear structure is identifiable, and the shared sources can be recovered, provided that sufficiently many diverse views and data points are available. To computationally estimate the sources, we optimize con- strained form of the joint log-likelihood of the observed data among all views. We show empirically that our objective recovers the sources in high dimensional settings, also in the case when the measurements are corrupted by noise. Finally, we apply the proposed model in challenging real-life application, where the estimated shared sources from two large transcriptome datasets (observed data) provided by two different labs (two different views) lead to more plausible rep- resentation of the underlying graph structure than existing baselines.
INTRODUCTION
We consider linear multi-view blind source separation (BSS) problem in the context of independent component analysis (ICA) where the different views share latent sources but also have view-specific ones. The modeling strategy presented in this work is inspired by applications from the biomedi- cal domain where linear BSS problems have often been encountered due to the nature of the data
(Vigário et al. 1997 McKeown Sejnowski, 1998: Sompairac et al. 2019).
Linear multi-view BSS solutions, such as Group ICA (Calhoun et al., 2001 independent vector analysis (TVA) and its corresponding variations (Lee et a
Anderson et al.) 2011 [014 Eng- berg et al. 2016 Vía et al. 2011), have been widely used for analyzing fMRI and EEG data. Typical applications include multi- subject studies for unraveling group-level brain activity patterns in the
data (Salman et al. 2019 Huster et al.) 2015 Congedo et al. 2010 Durieux & Wilderjans/ 2019
Congedo et al. 2010)) The main assumption in all those models is that each view (c.g., subject data) linear mixture of brain activity patterns shared across views (e.g., the group-level brain activity pattern). However, there is a growing tendency in neuroscience to shift the focus from group-level inference extracting setividual-specifii signals (Dubois & Adolphs 2016). For instance, one can be interested in investigating the individual (individual-speciin brain functions) and shared (behav- ioral phenotypes) patterns in individuals' brain activity in natural stimuli experiment (Seghier & Price, 2018:) Bartolomeo et al., 2017: Dubois & Adolphs 2016). Unfortunately, the aforementioned -ieei-view methods cannot be directly applied in this case, unless they are part two step pro- cedure (e.g. (Long et al.) 2020)): applying ICA/IVA on the different views followed by statistical analysis to separate the individual from the shared sources.
Moreover, this particular multi-view task is not only relevant for neuroscience but also for computa- tional biology, where ICA has been a standard tool for analyzing omics data (Sompairac et al. 2019 Avila Cobos et al.) 2018 Fraunhoffer al. 2022). For example, in cancer research, one assumes that the observed measurements of tissue/tumor samples are a linear mixture of cell-type specific expressions (latent sources) (Avila Cobos et al. 2018). Now, consider the task where want to aggregate heterogeneous experimental datasets (c.g., from different labs or modalities) to improve cancer prediction. By utilizing the prior knowledge that the datasets have shared and experiment-

---

Under review as a conference paper at ICLR 2023
specific information, transform this data integration task into linear multi-view BSS the one discussed before.
Model Summary. To address this and similar appli- cations, we formalize the stated BSS model as linear noisy generative model for multi-view data regime, assuming that the mixing matrix and number of in- dividual sources are view-specific. By requiring that the sources are non-Gaussian and mutually indepen- dent, and the linear mixing matrices are with full col- umn rank we provide identifiability guarantees for the mixing matrices and latent sources. We adopt a maxi- mum likelihood approach for the joint log- ligelikll of the observed views, which we use to estimate the mix- ing matrices. Furthermore, we suggest novel strategy for data integration of transcriptome datasets. We show empirically that method works well compared to the baseline methods when the estimated components are used for a graph inference task.
80
Sd
Ed
3d
d = 1... D
Figure I: A graphical representation of
Contributions. Our contribution can be summarized follows:
1. We provide theoretical guarantees for the identifiability of the recovered linear structure and the sources and noise distributions.
2. We propose optimization procedure based on MLE for estimating the mixing matrices.
2 RELATED WORK
The existing body of work on linear multi-view BSS, inspired by the ICA literature, considers mostly shared response model applications (i.e., individual sources), some of them adopting maximum likelihood approach (Guo & Pagnoni] [2008) Richard et al. 2020: (021) to model the noisy views of the proposed models. Many of these approaches, such Group ICA (Calhoun et 2001), shared response ICA (SR-ICA) (Zhang et al. 2016). and Multi ViewICA, incorporate a dimensionality re- duction step for every view (CCA (Varoquaux et al., 2009 Richard al., 2021) I) or PCA) extract the mutual signal between the multiple objects before applying an ICA procedure the reduced data. However, there are no guarantees that the pre-processing procedure will entirely remove the influence of the object-specific sources on the transformed data. Moreover, independent vector anal- ysis (IVA) relaxes the assumption about the shared sources across views by allowing them to be dependent (Lee et al. 2008, Anderson et al.) 2011 2014 Engberg et al. 2016 Vía et al. 201 However, as the previously discussed models, they fix the number of latent sources to be equal across views. To the best of our knowledge, two existing ICA-based methods exist for extracting shared and individual sources from data. Maneshi et al. (2016) proposes a heuristic way of using FastICA for the given task without discussing the identifiability of the results. The other exploits temporal correlations (Lukic et al. 2002) rather than the -au-Gaussanntty of the sources and thus is applicable the context we are considering.
A common tool for analyzing multi-view data is canonical correlation analysis (CCA), initially pro- posed by Hotelling (1936). It finds datasets' projections that maximize the correlation between the projected variables. Gaussian-Cer CCA (Bach & Jordan 2005), its kernelized version (Bach & Jor- dan (002) and deep learning (Andrew et al.) 2013) formulations of the classical CCA problem aim to recover shared latent sources of variations from the multiple views. There are extensions of CCA that model the observed variables as linear combination of group-specific and dataset-specific latent variables. (Klami et al. 2013) estimated with Bayesian inference methods or exponential families with MCMC inference (Virtanen 2010). However, most of them assume that the latent sources are Gaussian non-linearly related the observed data (Wang et al. 2016) and thus lack identifiability results.
Existing non-linear multiview versions such (Tian al. 2020 Federici et al. 2020) cannot cover both shared and individual signals across multiple measurements, and do not assure the iden- tifiability of the proposed generative models. There are identifiable deep non-linear versions of
2

---

Under review as a conference paper at ICLR 2023
ICA which can be employed for this task. However, they make stronger assumptions achieve identifiability which are hard to be satisfied in real-life applications (e.g. (Hyvärinen et al. 2019
3 PROBLEM FORMALIZATION
Consider the following D-view multivariate linear BSS model
Zd Aa(ša + ca) Adosu Ad1sa Aded, d E (l....,D),
where we assume that for d = I....,D
za R is a random vector with M[ra]] = 0,
latent random sources with and so IR and Sa R being the shared and individual sources and E[ša] = and Var [Sa] = Ika
(s0 8 s
Aa Raa is mixing matrix with full column rank, Adu and Aa the columns corresponding to the shared and individual sources,
~ (0, o 20 is Gaussian noise, or measurement error, on the sources (similar to (Richard al. 2020 2021)).
Additionally, assume that the variables 801..... 80c: 811,.... 81(k1 (e)--e)).... 8p1 8 D -c)- Ell..... Eik, ,..,, EDkp are mutually independent. Thus, we require that the noise vari- ables do not influence the latent signal and vice versa. In this work, we aim estimate the mixing matrices Ag from given observations of zz, I...., D. Note that for D = 1 the model becomes a standard linear BSS model with non nau-Gaasiian latent sources = $1 C1 (see (Comon 1994 Hyvärinen & Oja 2000: Bell & Sejnowski, 1995)).
IDENTIFIABILITY RESULTS
unsupervised machine learning methods, the reliability of the algorithm cannot be directly verified outside of simulations due to the non-existence of labels. For this reason, theoretical guarantees are necessary to trust that the algorithm estimates the quantities of interest. For a BSS problem solution, such as ICA, we want the sources and mixing matrices to be (up to certain equivalence relations) unambiguously determined (or identifiable) by the data, at least in the large sample limit.
Identifiability results for noiseless single-view ICA proved by (Comon 1994). It turns out that if at most one the latent sources is normal and the mixing matrix is invertible, then both the mixing be recovered almost surely up to permutation, sign and scaling. However,
matrix and sources this result does not hold in the general additive noise setting. [avies (2004) shows that if the mixing matrix has full column rank, then the structure identifiable, but not the latent sources. In the following we extend Comon (1994); Davies (2004); Kagan et al. 1973) results for the noisy setting inspired from our model (see l). Compared to previous work, we provide identifiability guarantees for the sources and noise distributions.
Theorem 4.1. Let Tl,.... FI for D 2 2 be random vectors with the following two representations:
(1)
(2)
1
+
(1) = xd A
+ +( g)
de (1....,,D)
with the following properties for 1,2
1. 1 RP is (non-random) matrix with full column rank, i.e. rank(A = k.(0)
(1)
R
(02 (m)) is a -variate normal random variable, and e (0) ~ ~NO0, 0d
(0
2.
3. 200
(5) ,84
(T) IT with s0 E ( and s TTRK" (1) - is random vector such that:

---

Under review as a conference paper at ICLR 2023
= (a) the components of 84 are mutually independent and each of them a.s. non- constant random variable,
(6)
(b) d is non-normal with O mean and unit variance.
(0) 4. Ed is independent from s6 ( and (0) cd 1 and Ed 1
(i)
(1)
Then, e(1) cl2) =: and for all d L,..., D we get that kg) k
ka, and there exist
sign matrix D and permutation matrix Pa Rhaaka such that:
A
A PaTd.
farthermrre the sources and noise distributions are identifiable, i.e.
$
(2) (1)
11)
d od
Theorem 4.1 proved in Appendix A assures the identifiability of the mixing matrices and sources and noise distributions up sign and permutation for multi-view (D 2 2) noisy ICA model. This is a more general case than since here the noise distribution can be view-specific and the mixing matrices can be non-square. We also provide identifiability results for the more general single-view setting (see Theorem A.2 In the single-view case compared to the multiview one, we get similar but weaker results, i.e. the source and noise distributions identifiable only if the latent sources do not have normal components. This means that for any of the latent sources variables
w with Lww then v and w are non-normal. Recent works by Richard et (2020) 2021); Anderson et al. (2014) provide identifiability results for shared response modeling by imposing different assumptions to the one in Theorem 4.1 For example, under additional assumptions about the noise covariance matrices, the requirement about the non-Gaussianity of the sources in the ShICA model can be relaxed (Richard) et al. (021). Similar results about the shared sources are provided in Theorem [A.4 by imposing additional requirements about the sources variance that are not covered by our model assumptions.
in both equivalent representations in Theorem A.2 if we have 5(6)
5 OPTIMIZATION
Here, derive the joint log- likeliiol of the observed views which mixing matrices. Let 2d Waza &d + ed, and 2d (1) == S0 cao E R and 2(2 := sa + cd1 Rka-s, i.e. 2d
for estimating the
(=d (1)T (2)T Furthermore, let P P2500 be the probability distribution of (2) and
Wal = det Wa). Then the data log-likelihood I for N observed samples per view is given by
N
D
C(W,,... WD)
log (8))
log
(=
N
log Wal
z
1=1 d=1
d=1
D
D
trace Z
trace ((T) +
202 d=1
D
where 2(11) | for d I,..., D is the data matrix that stores N observations of z and
(1) I and f30) exp 56 -. Ei-12 20
exp
D D|[so
su
ps, (5o)
20
We further simplify the loss function by assuming that the data matrices X1 E R ,g., e R are whitened. That consists of centering and linearly transforming the random variables' realizations Da such that the resulting variable d. = Ka has unit variance, E[ = I where Ka is the whitening matrix. Thus, from the last equation we get that I,, = B| = (1 + o²) 2)) KaAdAJ A KT It follows that the matrix (1 11+0z 02) KgAa is orthogonal, which we estimate by the matrix W., After training set Å K- which differs from the one by (1 (
4

---

Under review as a conference paper at ICLR 2023
Due to the orthogonal constraints the objective function becomes
L(W1,... WD) α log fo (5)
N
log Pz (2d (2))
1
Σ
trace ZZ12Z(11)) T(
2D0²
d=1
d=1
(3)
DI| 80 (1 + 22) ) 1|
where here (50) = exp
pSo (50) This results from the fact that
20
and W = 1. All proof' can be found Appendix B the proof of equation 2 extends the one in (Richard et al. 2020). Note that in our optimization procedure, both (50) and Pzz!"), we approximate by the negative of nonlinear function g()). e.g-
after whitening we have trace( 2 d
((s) log cosh(s) for super-Gaussial
g(s) - for sub-Gaussian sources. Moreover, since we do not estimate o² we treat it as a Lagrange multiplier via the relation À
1+:²2 o²
For the parameter estimation of the orthogonal unmixing matrices, we use the transformation framework proposed (Lezcano- 2019 Lezcano-Casado & Martunez-Rubie 2019 The framework developed in (Lezcano-Casado) 2019) allows us to transform manifold optimization problems to unconstrained (Euclidean) optimization problems. To accomplish the transforma- tion, the scheme uses trivializations, Φ V - M. which are smooth, surjective mappings be- tween Euclidean spaces (e.g. V) to the manifold (denoted by .M) (Lezcano-Casadee (019). Thus, the optimization problem defined in 3 maxwj...WW.eM L(W,,... Wp) becomes equivalent to ((yD)) for .M being the Stiefel manifold and á a trivialization as the one described above. Subsequently, we can apply L-BFGS or (stochastic) gradient descent com- pute approximate minimizers to our parameter estimation problem. Note that an alternative opti- mization approach Riemannian optimization (Sato 2021 Boumal et al. 2014). Moreover, there are other L-BFGS based methods, such as Ablin et al. (2018) that develop optimization solver for ICA under orthogonal constraints which relies on the Lie exponential map parametrization.
nax
EXPERIMENTS
Preprocessing. Before running any of the ICA-based methods (our or the baselines) we whiten each single view by performing PCA to speed up computation.
We used the python library pytorch (Paszke et al.) 2017) to implement our method. We model each view with separate unmixing matrix. To impose orthogonality constraints on the unmixing matrices, we made of the geotorch library, which
Model Implementation.
Hept Plasid Miaieeee
Amari Distance
the extension of pytorch (Lezcano-Casatee 2019). The stochastic gradient based method applied for training is L-BFGS.
Initialization. We estimate the mixing matrix up to scale (due to the whitening) and permutation (see Sections 4 and 5). To force the algorithm to output the shared sources in the same order across all views we initialize the unmixing ma- trices by means of CCA. This follows from the fact that the CCA weights are orthogonal matrices due whitening, and the transformed views" components are paired and ordered across views such that the first pair correspond to the largest singular value from the singular value decomposition of the cross correlation matrix.
Compute time
Figure 2: Comparison of the comput- ing time (x-axis) and Amari distance (y-axis). The data is generated accord- ing to a shared response model with 5 views, 5 sources and 1000 samples and noise with the variance The addition "O" to model's name refers to learning with orthogonal con- straints.
Training. For all conducted experiments fixed the param- eter À from equation to 1. In the simulated data experiments conducted each experiment 50 times and based that provided error bars all figures where applicable. For the computational specifics of the real-life data experiment please refer to Section 6.2
Baselines Implementation. We compare our method to the standard single-view ICA method In- fomax (pi card library (Ablin et al. 2018)). To adopt it to the multi-view setting, Infomax
5

---

Under review as a conference paper at ICLR 2023
10r
10
disancee
Annai distance
1.00
ShICA
SNCA
SHICAMI
SHGGAA
method
GroupICA
GroupICA
MullivteoCCA
20
40
60
80
100
10
15
20
25
30
45
50
Shared Sources
Sources
(a) Noiseless views according [o
(b) Noisy views shared response model
Figure 3: We compare the performance of our method, ShICA, Infomax, GroupICA, MuliVViewCAA and ShICA-MI. Amari distance (the lower the better) for different number of shared sources (x-axis). The error bars correspond 95% confidence intervals based on independent datasets come from two different views with total number sources 100 and sample size 1000. We vary number of shared sources from 10 to 100. Figure (bb The data is generated according to a model, where no
of the experiment. Figure The
individual sources are present and the noise per view is uniformly sampled from the interval [L. 2]. The number of views is 10 and the sample size 1000. We vary the number of sources from 10 50 (y-axis).
on each view separately and then we apply the Hungarian algorithm (Kuhn Yaw 1955) to match components from different views based on their cross-correlation.
For the shared response model settings, we compare it to related methods such as Multi ViewICA Richard et al. (2020), ShiCA, ShICA-ML Richard et al. (2021). and GroupICA proposed by Richard et al. (020). The latter involves a two-step pre-po processscnnng procedure, first whitening the data in the single views and then dimensionality reduction on the joint views. The code for the latter ca (Richard et al. (Richard et al,
models based on https////ggtttbb..oo/o/oo/ooeeee
2020 and for ShICA-ML
com/ httphhhoooohhhaaaahicoe
2021
For the data integration experiment we use method based partial least squares estimation, closely related to CCA, that extracts between-views correlated components and view-specific ones. This method is provided by the OmicsPLS package Bouhaddani et al. (2018) and especially developed for data integration task for omics data. We refer to this method as PLS. We also make use of IVA-L-SOS (Bhinge al. 2019) as a representative of the independent vector analysis methods which aligns well with this task. This method assumes linear noiseless model x. Agsa, d E {I..... D} with Laplacian independent sources 5d E I per view where the following between view dependence holds: (sii, SD) ~ Laplace(O, 2) with sæ being the i-th component of sa. The implementation of this method is provided by the independent-vect -aaaerrrn-eettt---allllll watpeettrraaaaalllaaaaaaaa ysss package provided by (Lehmann et al. 2022).
6.1 SYNTHETIC EXPERIMENTS
Data Simulation. We simulated the data using the Laplace distribution exp(-> and the mixing matrices are sampled with normally distributed entries with mean 1 and 0.1 standard deviation. The realizations of the observed views are obtained according to the proposed model. In the different scenarios described below we vary the noise distribution.
Evaluation. The quality of the mixing matrix estimation is measured with the Amari distance (Amaii et al., (995),, which cancels the estimated matrix differs from the ground truth one up to scale and permutation. More experiments than the one stated below are provided in Appendix D.2
Noiseless views. Figure 3al we consider a noiseless view setting, where we fixed the dimension to be 100 and we vary the number of shared sources from 10 to 100 in two view setting, We can see that as soon as the ratio of shared sources to individual sources gets around 1:1 we can recover almost all shared and individual sources compared to the baseline methods which cannot perform well in higher dimensions. The reason the performance drop of our method is insufficient number of samples for the learning task. See Appendix D.2 for additional experiments in the noisy case.

---

Under review as a conference paper at ICLR 2023
Table 1: Shared Sources Recovery Performance
Views
0.1
0 0.5
0
0.975 + 0.002
0.940 + 0.003
0.812 0.003
0.281 + 0.01
0.981 + 0.001
0.967 0.001
0.922 + 0.003
0.5 + 0.04
10
0.980 + 0.002
0.972 0.002
0.95 + 0.003
0.741
Robustness to model misspecification in a shared response model application. Here we apply our method to a shared response setting, i.e. no individual sources are available. For this experiment the views have view-specific variance uniformly sampled from [1, 2]. Our method shows similar performance to the competitor model MultiViewIC. in this special (see Figure 3b)
Computing time. Figure 2 the computing times of response models trained on the same datasets. The experiment carried out on local machine using CPU cores in parallel. It seems that MultiViewICA has similar performance as ShICA-ML but much faster. Our method is faster than ShICA-ML and shows similar mean perfromance the Infomax method.
Mean cross correlation (MCC) of shared sources. We explore how well our method recovers shared sources. We estimate them by taking their average across views and compute the mean cross correlation (MCC) between the estimates and the ground truth. That involves matching estimated with the ground truth components by using the Hungarian algorithm and then computing the mean over all correlations between the aligned pairs. In this experiment, we fixed the total number of sources be 60 and the shared to be 30. We investigated four cases corresponding noise standard deviation = 0.1, 1, 2. and reported the mean MMC and its standard deviation As expected by increasing the number of views we get better estimates of the shared sources, i.e. the MMC score increases.
different
from 50 runs in Table I
6.2 DATA FUSION OF TRANSCRIPTOME DATA
Background. Transcriptome datasets are relevant for the field of genomics. After preprocessing they have the form of random data matrices, where each row correspond to gene and each column refers to an experiment. Based on these datasets, scientists try to infer gene-gene interactions in the genome.
Co-regulation Inference. Since the transcriptome datasets are in the high- wwweeniion-lwwssmplle....e regime (number of genes>number of samples), usually graphical lasso (Friedman al. (007)) well-suited for inferring graphical structure from the observed data. More precisely in this application, we want to estimate undirected graph with nodes referring to the genes and with edges connecting genes with a common regulator.
Legend
Selected
reguion genes
Genes not part of the reguion
True positive edges from selected regulon
True positive edges from other regulons
Undirected graph
Possibly false positive edges
Data Integration Task. To boost the graphi- cal lasso performance, we would like to com- bine as many experiments as possible. Since the datasets usually come from different labs there is non-biological noise present. There- fore, just pooling the two datasets together for performing downstream tasks can lead to sub- optimal results. The goal of the data integration task is to "denoise" the datasets, such that the trans- formed data can be used as samples for the graph inference task. In this kind of application usually dimensionality reduction step is required. This done by a dimensionality reduction step fol- lowed by a feature extracting algorithm, such as IVA-L-SOS, or combined dimensionality reduction method and a feature extractor such PLS. We follow the latter approach for both, method and
Figure 4: Induced subgraph with vertices from selected regulon (in the main text denoted by R) (not dotted nodes) and its neighbors (dotted nodes). The green edges helong to the ground truth and the red one possibly false pos- itives.

---

Under review as a conference paper at ICLR 2023
10000
0.6
0.5
8000
True Positives
0.4
6000
1 0.3
4000
IVA---SOS
0.2
IVA-L-SOS
Our Method
Method
2000
0.1
Inforax
Informxx
0
0.0
500
1000
1500
2000 2500
2000
4000
6000
8000
10000 12000
False Positives
Edges
(a) True Positive False Positive pairs
(b) Fl score
After the proposed hyperparameter tuning, we compare the top ten EBIC glasso models with our model, PLS, Infomax, and IVA-L-SOS. We order the edges from the selected networks according to their strength. In Figure 5a count the truc positives (y-axis) and possibly false positive edges in the first 100, 200... edges (x-axis). It seems that our method and PLS outperform the other two methods, and for our method, the true positive/false positive rate increases faster than for PL.S. In Figure we compute the average regulon Fl score (y-axis) from the selected edges (x-axis). Our method seems to perform better than PLS but worse than IVA-L-SOS and Infomax. That might result from the proposed Fl score prioritizing sparse graphs.
Figure 5:
Infomax, and directly optimize for unmixing matrices Wa of dimension ca x ka with ca S ka for d = 1,...,D. D. That means that by minimizing the loss function with respect to possibly non-square Wi, Wi,-.. Wn we directly extract Ga features, which are then used for the graph inference task.
Data Assumptions. We do a one-to-one translation to our proposed model by assuming that each experiment is noisy linear combination of independent gene pathways and that some pathways are active for both datasets (shared sources).
Datasets. In this example, consider the bacterium B. subtilis, for which very rich collection of the discovered gene-gene interactions are publicly available, which we use our ground truth model. For this data integration task we use two vast publicly available datasets (Arrieta-rre Ortiz al.) 2015 Nicolas et al. 2012). Each of the datasets contain gene expression levels of about 4000 genes measured across more than 250 experimental outcomes. For detailed description of the datasets we refer to Appendix
Experiment. As in most real-life applications, the number of latent sources per dataset is unknown. We treat it as hyperparameter for each model, i.e., we perform grid search on (50, 60,
200) for the total number of sources for both datasets. Note that for IVA-L- SOS the number of sources in both datasets should coincide. The number of shared sources for our method and PLS varies between 10, 20, 30, and 40. We fit 30 graphical lasso models for different penalization parameters on the estimated components. We select the top 10 models by employing a statistical goodness-of-fit measure, called EBIC, for each combination of hyperparameters (see Appendix [opendix[[forr for more details). Then, the hyperparameter setting selected, yielding the best true positive/false positive ratio curves (as the ones shown in Figure 5ay. The resulting hyperparameter settings are IVA-L-SOS (130 latent sources), Our Method (50 for dataset 1, 60 for dataset 2, 40 shared sources), PLS (180 for dataset 1, 80 for dataset 2, 10 shared sources), and Infomax (200 for dataset 50 for dataset 2).
True Positives vs False Positives. The below-deseribed evaluation is used for our hyperparameter selection. The output graph from the graphical lasso for each pre-processing method compared to the ground truth one. The evaluation strategy is as follows. For each estimated graph, we order the edges according to their strength. Then we count the true positive and false positive edges in the first 100, 200,... ... edges. Then for each method separately, we select the hyperparameter combination for which the graphical lasso has the best true positive/false positive ratio curves. The results are depicted in Figure [5] where the best models for each method are compared. We can conclude that our model boosts the graphical lasso's performance compared to the others. We also run the graphical lasso the pooled data without any pre-processing. Surprisingly, the method
8

---

Under review as a conference paper at ICLR 2023
10 Regulons =0
10 Reguiors
0.6
1.0
0.5
0.8
0.6
26.3
0.2
MA--0S
Methed
0.4
6.3
0.7
0.0
3000
10000
MA.
PLS
Infomax
Our Method
Regulons
50.5 Seguloms
100
1.0
0.4
0.3
0.5
0.2
MNL-DD
Method
0.4
10000
MA--500
PLS
Infomax
Our Method
Figure 6: Fl score curves per model (left) and best model regulon performance (right) two groups regulons depending on their size (between 10 and 50, and between 50 and 400). In the first group all methods show comperable results. In the second case (regulon size > 50) it seems that our method shows hetter Fl score for the large regulons.
outputs an empty graph, i.c., the goodness-of-fi measure we use evaluates the empty graph as the best model describing the data.
F1 score. We are also interested if the output graph resembles the known regulon structure. A regulon is set of genes that controlled unit by the same regulator. The species of interest has 225 known regulons, which form cliques in our ground truth graph. We compute the precision and recall per regulon from the graphical lasso output in the following way. First, select subgraph induced by the regulon genes denoted by R and their neighbors as depicted in Figure 4 The precision the ratio of the true positive edges and the total number of edges in the selected subgraph. The recall is the ratio of the number of nodes in the biggest connected component with vertices in and the cardinality of R. Note that the recall score does not incorporate any information about the connectivity of the selected subgraph. In Figure 55 we order the edges according to their strength and from the first 100, 200, edges we the Fl score per regulon from the corresponding precision and recall. It seems that IVA-L-SOS and Infomax outperform the other methods. The reason is that the output graphs are sparser in both cases and recover smaller regulons better (most of the regulons have size < 10). Due the overlapping structure of regulons, i.c. larger regulons encompass smaller ones, we want to compare the aggregated F1 scores for the larger regulons. From Figure 6 we than the other methods. In medium size regulon group Infomax, IVA-L-SOS and our method show comparable results.
that method recover very large regulons with higher F1 scores
DISCUSSION
We proposed a novel noisy linear ICA approach that utilizes the prior knowledge that the different views share information to infer both shared and view-specific sources. Compared to other models from related fields, our model does not assume that all views have the same dimensionality. We provided theoretical guarantees for the identifiability of the model's linear structure and the latent sources in distribution. We adopted a maximum likelihood strategy for estimating the unmixing ma- trices by maximizing the joint log- likelikeliooo of the observed views. Our empirical results showed that our model performs well high-dimensional data regimes that often resemble real-life applica- tions. We also suggested novel strategy for combining transcriptome data and empirically showed that our model improves the performance of a graphical inference model chosen for the particular task. In future work, would like to address some possible extensions to model. such the case when the noise has an arbitrary covariance matrix. Moreover, to state the identifiability re- sults of model, assumed that the individual sources from the different views are mutually independent. That might not the case in real-life applications like the one considered above.

---

Under review as a conference paper at ICLR 2023
8 ETHICS STATEMENT
Our model can be used in many scientific domains such as neuroscience, genomics, physics, medicine, causal discovery, etc., where the proposed model assumptions hold. Therefore, its so- cietal impact is tied to the ethical concerns of the domains it is applied to.
REPRODUCIBILITY STATEMENT
The code and transcriptome data and ground truth model be found under https:// anonymous 4open www.sscieeeeeeeeeee / shi ndf - C4-C797/ We provided detailed proofs of the theorem and other theoretical results stated in the main paper. Furthermore, all synthetic experi- ments are run multiple times with different random seeds and the reported results are in terms of mean and 95% confidence interval. Additional information regarding the data integration exper- iment, such as pseudo code of algorithm and explanation of the main steps are provided in the appendix.
REFERENCES
Pierre Ablin, Fran-François Cardoso, and Alexandre Gramfort. Faster ica under orthogonal con- straint. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4464-4468. IEEF, 2018.
Shun-ichi Amari, Andrzej Cichocki, and Howard Yang. A new learning algorithm for blind signal separation. Advances in neural information processing systems, 8, 1995.
Matthew Anderson, Tuelay Adali, and Xi-Lin Li. Joint blind source separation with multivariate gaussian model: Algorithms and performance analysis. IEEE Transactions on Signal Processing, 60(4):1672-1683, 2011.
Matthew Anderson, Geng-Shen Fu, Ronald Phlypo, and Tülay Adahı Independent vector analysis: Identification conditions and performance bounds. IEEE Transactions Signal Processing, 62 (17))4399-4410 2014.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp. 1247-1255, PMLR, 2013.
Mario L Arrieta-Ortiz, Christoph Hafemeister, Ashley Rose Bate, Timothy Chu, Alex Greenfield, Bentley Shuster, Samantha N Barry, Matthew Gallitto, Brian Liu, Thadeous Kacmarezyk, et al. An experimentally supported model of the bacillus subtilis global transcriptional regulatory net- work. Molecular systems biology, 11(11):839, 2015.
Francisco Avila Cobos, Jo Vandesompele, Pieter Mestdagh, and Katleen De Preter. Computational deconvolution of transcriptomics data from mixed cell populations. Bioinformatics, 1966 1979, 2018.
Francis R Bach and Michael Jordan. Kernel independent component analysis. Journal of machine learning research, 3(Jul): 1-44)) 2002.
Francis Bach and Michael Jordan. A probabilistic interpretation of canonical correlation analysis. 2005.
Paolo Bartolomeo, Tal Seidel Malkinson, and Stefania De Vito. Botallo's error, the quandaries of the universality assumption. Cortex, 86: 176-185, 2017.
Anthony J Bell and Terrence J Sejnowski. An information-maxiiizztion approach blind separa- tion and blind deconvolution. Neural computation, 7(6):1129-1155 1995.
Suchita Bhinge, Rami Mowakeaa, Vince D Calhoun, and Tülay Adah. Extraction of time-varying spatiotemporal networks using parameter-tuned constrained iva. IEEE transactions on medical imaging, 387)):115511755 2019.
10

---

Under review as a conference paper at ICLR 2023
Said el Bouhaddani, Hae- Han Uh, Geurt Jongbloed, Caroline Hayward, Lucija Klarić, Szymon M Kictbasa, and Jeanine Houwing-Duisterma Duistermaat. Integrating omics datasets with the omicspls pack- age. BMC bioinformatics, 19(1):1-9, 2018.
Nicolas Boumal, Bamdev Mishra, P-A Absil, and Rodolphe Sepulchre. Manopt, matlab toolbox for optimization on manifolds. The Journal of Machine Learning Research, 15(1):1455-1459 2014.
Vince D Calhoun, Tulay Adali, Godfrey D Pearlson, and James J Pekar. A method for making group inferences from functional mri data using independent component analysis. Human brain mapping, 140151140-55,, 2001.
Jiahua Chen and 7ehua Chen. Extended bayesian information criteria for model selection with large model spaces. Biometrika, 95(3):759-77) 2008.
Pierre Comon. Independent component componeen analysis, new concept? Signal processing, 36(3):287-31) 1994.
Marco Congedo, Roy E John, Dirk De Ridder, and Leslie Prichep. Group independent component analysis of resting state eeg in large normative samples. International Journal of Psychophysiol- ogy. 78(2):89-99)) 99, 2010.
Mike Davies. Identifiability issues in noisy ica. IEEE Signal processing letters, 11(5):470-473, 2004.
Julien Dubois and Ralph Adolphs. Building science of individual differences from fmri. Trends in cogmilive sciences, 20(6):425-443, 2016.
Jeffrey Durieux and Tom F Wilderjans. Partitioning subjects based on high-dimensional fmri data: comparison of several clustering methods and studying the influence of ica data reduction in big data. Behaviormetrika, 46(2):271-3111 2019.
Astrid ME Engberg, Kasper Andersen, Morten Mørup, and Kristoffer H Madsen. Independent vector analysis for capturing common components in fmri group analysis. In 2016 international workshop on pattern recognition in neuroimaging (prni), pp. 1-4. IEEE, 2016.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. ICLR, 2020.
Rina Foygel and Mathias Drton. Extended bayesian information criteria for gaussian graphical models. arXiv preprint 101.::0011 2010.
Nicolas Fraunhoffer, Analía Meilerman Abuelafia, Martin Bigonnet, Odile Gayet, Julie Roques, Remy Nicolle, Gwen Lomberk, Raul Urrutia, Nelson Dusetti, and Juan Iovanna. Multi-omics data integration and modeling unravels new mechanisms for panereatic cancer and improves prognos- tic prediction. NPJ precision oncology, 6(1):1-16, 2022.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatisties, 9(3):33--44 441,, 12 2007. ISSN 1465-4644. doi: 10. biostatistics/kxx0445 htR.stt:: ///oii doi http://ttoooogo/./////o 1093/0 http://ossttttssicss//o/oeeeee
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In
Uncertainty in Artificial Intelligence, pp. 217-227. PMLR, 2020.
Ying Guo and Giuseppe Pagnoni. A unified framework for group independent component analysis for multi-subject fmri data. Neurolmage, 1278:1077 2008.
Harold Hotelling. Relations between two sets of variates. In Breakthroughs in statistics, pp. 162- 190. Springer, 1936.
Maxim
Multivariate transformation for- mula correct? Mathematics Stack Exchange, 2019. URL https::/math..
sstackecchngee. com/ http://////////33/// q/3325459 h.t.tt/ssmaatt...acccccaa.gooom/ (ver- sion: 2019-08-17).

---

Under review as a conference paper at ICLR 2023
Rene J Huster, Sergey M Plis, and Vince D Calhoun. Group-level component analyses of eeg: validation and evaluation. Frontiers in neuroscience, 9:254, 2015.
Aapo Hyvärinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources.
In Artificial Intelligence and Statistics, pp. 460-469. PMLR, 2017.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 334-5))411-4300 2000.
Aapo Hyvärinen. Hiroaki Sasaki, and Richard Tumnr. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 359-868. PMLR, 2019.
Abram Meerovich Kagan, Yurii Vladimirovich Linnik, Calyampudi Radhakrishna Rao, et al. Char- acterization problems in mathematical statistics. Wiley-Interscience, 1973.
Arto Klami, Seppo Virtanen, and Samuel Kaski. Bayesian canonical correlation analysis. Journal of Machine Learning Research, 14(4). 2013.
H. W. Kuhn and Bryn Yaw. The hungarian method for the assignment problem. Naval Res. Logist. Quart, pp. 83-97, 1955.
Jong-Hwan Lee, Te- Won Lee, Ferenc A Jolesz, and Seung-Schik Yoo. Independent vector analysis (iva): multivariate approach for fimri group study. Neuroimage, 40(1):86-109, 2008.
Isabell Lehmann, Evrim Acar, Tanuj Hasija, M.A.B.S. Akhonda, Vince D. Calhoun, Peter J. Schreier, and Tulay Adali. Multi-task fmri data fusion using iva and parafac2. 2022.
Mario Lezcano-Casado. Trivializations for gradient-based optimization on manifolds. In Advances in Neural Information Processing Systems, NeurIPS, pp. 9154-9164, 2019.
Mario Sezcano-Casado and David Martunez-Rubo.. Cheap orthogonal constraints in neural net- works: A simple parametrization of the orthogonal and unitary group. In International Confer- ence on Machine Learning, pp. 3794-3803, PMLR, 2019.
Qunfang Long, Suchita Bhinge, Vince D) Calhoun, and Tülay Adali. Independent vector analysis for common subspace analysis: Application to multi-subject fmri data yields meaningful subgroups of schizophrenia. Neurolmage, 216: 16872, 2020.
Ana S Lukic, Miles N Wernick, Lars Kai Hansen, and Stephen C Strother. An ica algorithm for analyzing multiple data sets. In Proceedings. International Conference on Image Processing. volume 2. pp. II-II. IEEE, 2002.
Paul Lévy. Propriétés asymptotiques des sommes de variables aléatoires indépendantes ou en- chaînées. The Annals of Probability, 1(1):9-18, 1935. ISSN 00911798. URL hattp://www
Mona Maneshi, Shahabeddin Vahdat, Jean Gotman, and Christophe Grova Validation of shared and specific independent component analysis (ssica) for between-group comparisons in fmri. Fron- fiers in neuroscience, 10:417, 2016.
Martin McKeown and Terrence J Sejnowski. Independent component analysis of fmni data: exam- ining the assumptions. Human brain mapping, 6(5-6)336-337 1998.
Ricardo Pio Monti, Kun Zhang, and Aapo Hyvärinen. Causal discovery with general non- non-linear relationships using non-lincar ica. In Uncertainty in artificial intelligence, pp. 186-195. PMLR, 2020.
Pierre Nicolas, Ulrike Mäder, Etienne Dervyn, Tatiana Rochat, Aurélie Leduc Nathalie Pigeon- neau, Elena Bidnenko, Elodie Marchadier, Mark Hoebeke, Stéphane Aymerich, et al. Condition- dependent transcriptome reveals high-level regulatory architecture in bacillus subtilis. Science, 13536072):1133-1106 2012.
12

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Hugo Richard, Luigi Gresele, Aapo Hyvarinen, Bertrand Thirion, Alexandre Gramfort, and Pierre Ablin. Modeling shared responses in neuroimaging studies through multiview ica. Advances in Neural Information Processing Systems, 33:19149-19162 2020..
Hugo Richard, Pierre Ablin, Bertrand Thirion, Alexandre Gramfort, and Aapo Hyvarinen. Shared independent component analysis for multi-subject neuroimaging. Advances in Neural Information Processing Systems, 34:29962-29971. 2021.
Mustafa S Salman, Yuhui Du, Dongdong Lin, Zening Fu, Alex Fedorov, Eswar Damaraju, Jing Sui, Jiayu Chen, Andrew R Mayer, Stefan Posse, et al. Group ica for identifying biomarkers in schizophrenia: adaptive networks via spatially constrained ica show more sensitivity to group differences than spatio-temporal regression. Neurolmage: Clinical, 22:101747, 2019.
Hiroyuki Sato. Riemannian Optimization and Its Applications. Springer, 2021.
Mohamed Seghier and Cathy Price. Interpreting and utillsing intersubject variability brain function. Trends in cognitive sciences, 22(6):517-530, 2018
Nicolas Sompairac, Petr V Nazarov, Urszula Czerwinska, Laura Cantini, Anne Biton, Askhat Molkenov, Zhaxybay Zhumadilov, Emmanuel Barillot, Francois Radvanyi, Alexander Gorban, al. Independent component analysis for unraveling the complexity of cancer omics datasets. International Journal of molecular sciences, 20(18):4414, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020.
Gaël Varoquaux, Sepideh Sadaghiani, Jean Baptiste Poline, and Bertrand Thirion. Canica: Model- based extraction of reproducible group-level ica patterns from finri time series. arXiv preprint arXiv:0911.4650, 2009.
Javier Vía, Matthew Anderson, Xi-Lin Li, and Tülay Adalı. A maximum likelihood approach for independent vector analysis of gaussian data sets. In 2011 IEEE International Workshop on Machine Learning for Signal Processing, pp. 1-6. IEEE, 2011.
Ricardo Vigário, Veikko Jousmäki, Matti Hämäläinen, Riitta Hari, and Erkki Oja. Independent com- ponent analysis for identification artifacts in magneooencephalographi recordings. Advances in neural information processing systems, 10, 1997.
Seppo Virtanen. Bayesian exponential family projections. PhD thesis, Aalto University, 2010.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Liveseu. Deep variational canonical correla- tion analysis. arXiv preprint arXiv:1610.03444. 2016.
Hejia Zhang, Po-Hsuan Chen, Janice Chen, Xia Zhu, Javier S Turek, Theodore L Willke, Uri Has- son, and Peter Ramadge. searchlight factor model approach for locating shared information in multi-subject fmri analysis. arXiv preprint arXiv: (609009933.. 2016.
13


Reviews:
[{"review_id": "dueYfh5ODj", "reviewer": "Reviewer_g5xN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors extend ICA for multi-view settings, building on similar latent variable models that have been used for modelling correlations between multiple views. This paper extends the formulation for supporting non-gaussian latent distributions to enable identification of individual sources, following the standard ICA principles. The method is demonstrated to outperform competing methods in specific conditions, mostly when there are tens of views.", "summary_of_the_review": "Technically okay paper that suffers from presentation issues (lack of transparent motivation, algorithmic details on very high level) and incremental contribution. I do not see potential for high impact because the need for the problem remains vague. Empirical experiments are reasonable, but not strong enough to overcome the other problems.", "strength_and_weaknesses": "Strengths:\n- The theoretical guarantee for identifiability for multi-view learning setups is a valuable contribution\n\nWeaknesses:\n- The motivation for multi-view BSS remains poor. The concert example is clearly artificial and the other examples (fMRI etc) are not explained well enough to understand why they correspond to multi-view BSS tasks. The linearity assumption is also poorly motivated, by merely stating that it holds in these applications. It is far from obvious for the reader why fMRI or biological datasets would exhibit prominently linear mixing.\n- Core method is poorly explained and seems to be fairly incremental", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a---T-BWCg", "reviewer": "Reviewer_Kgmf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a new technique for multi-view blind source separation. The formulation involves a linear summation of a linear approximation of the blind sources from different views (Eq.(1)). The authors then derive a theorem to ensure the identity so that the source signal can be uniquely recovered from the set of mixed signals.\n", "summary_of_the_review": "Overall, the paper has a well-evaluated experiment section. The writing of the paper is easy to follow as it is written in dot-point form. But this is non-standard for papers published at ICLR.\n\nI am very familiar with the field of multi-view learning and blind source separation, so I was able to follow the paper. However, I felt that readers who are less familiar with this area will be unable to follow the paper as the introduction (Section 1), Problem Formulation (Section 2), and the theorem in Section 3 are not well written.\n\nI have recommended rejecting the paper because of the combination of (i) clarity of the writing in both motivating the problem and formulating the problem, and (ii) the novelty proposed is incremental which can pretty much be summarized by Eq.(1)", "strength_and_weaknesses": "Strengths:\n(1) The paper considers the theoretical aspects of multi-view blind source separation\n(2) The experiments have properly evaluated against its baselines and other state-of-the-art approaches\n\nWeaknesses:\n(3) The approach is very simplistic and lacks any novelty. The entire contribution can pretty much be summarized as Eq.(1) (followed with theoretical guarantees and optimization) which in my opinion does not meet the bar for publication at ICLR.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "GhyJPOuDi0", "reviewer": "Reviewer_8FCr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of ICA with multiple views, where the goal is to estimate independent sources shared by a group of subject. The originality of this work lies in the addition of individual sources: the data of each subject is a mixture of individual and shared sources, allowing for a more flexible model. The authors demonstrate the identifiability of the proposed model, and propose an algorithm to recover sources and mixing matrices. The authors finally validate their approach on synthetic and genomics problems.", "summary_of_the_review": "This is a solid paper: the authors study a new model that is relevant to the community, establish some theoretical results about it, and then provide a fast practical algorithm to maximize its likelihood. Finally, the empirical study is convincing. For me, this paper is worthy of acceptance.", "strength_and_weaknesses": "# Strengths\n- The authors consider a realistic and important problem: indeed, in practice the common assumption of group ICA  that all sources are shared is problematic, and assuming a mix of individual and shared sources is an elegant solution to this problem\n- While expected, the identifiability result provides a solid theoretical basis for the method\n- The proposed algorithm is sound, as it maximizes a likelihood under orthogonal constraint, as is classical in single subject ICA.\n- The experiments are convincing, and clearly show that the proposed algorithm is able to recover the sources, and brings something new on the table in practice.\n- The paper is very well written, it was a pleasure to read.\n\n# Weaknesses\n- The practical optimization method could probably be improved. Trivialization methods were originally developed for deep learning. Here using these methods destroys some of the nice structure of the problem. For instance we lose the Hessian structure that is used in [Ablin et al 18] and [Richard et al. 20] to get fast algorithms. I think that this is why the proposed method is slower than MultiviewICA (fig.3), while using a similar optimization technique as in [Richard et al. 20] should give a similar runtime.\n- The  authors do not release code for the peer review phase, which greatly hinders the reproducibility of this work", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6tK0m1rCawQ", "reviewer": "Reviewer_vDv7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of multiview linear independent component analysis. The identifiability is analyzed and a constrained joint log-likelihood-based formulation is provided. Experiment results are obtained on real data for a source separation task.", "summary_of_the_review": "Overall, the reviewer finds the paper studies an important problem, but the experiments need to be improved, and novelty and challenges should be clarified", "strength_and_weaknesses": "Strength: ICA is a fundamental and important problem, the identifiability of which is also of great importance.\n\nWeakness: \n\n1. the submission focus on the linear case, which limits the applicability of the proposed approach, but I can understand that \n2. justification for assumptions are not sufficient\n3. experimental results are not sufficient", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject