L0-Sparse Canonical Correlation Analysis
Authors: Ofir Lindenbaum
Moshe Salhov
Amir Averbuch
Yuval Kluger
Venue: ICLR 2022 Poster (2022)

Abstract: Canonical Correlation Analysis (CCA) models are powerful for studying the associations between two sets of variables. The canonically correlated representations, termed \textit{canonical variates} are widely used in unsupervised learning to analyze unlabeled multi-modal registered datasets. Despite their success, CCA models may break (or overfit) if the number of variables in either of the modalities exceeds the number of samples. Moreover, often a significant fraction of the variables measures modality-specific information, and thus removing them is beneficial for identifying the \textit{canonically correlated variates}. Here, we propose $\ell_0$-CCA, a method for learning correlated representations based on sparse subsets of variables from two observed modalities.
Sparsity is obtained by multiplying the input variables by stochastic gates, whose parameters are learned together with the CCA weights via an $\ell_0$-regularized correlation loss. 
We further propose $\ell_0$-Deep CCA for solving the problem of non-linear sparse CCA by modeling the correlated representations using deep nets. We demonstrate the efficacy of the method using several synthetic and real examples. Most notably, by gating nuisance input variables, our approach improves the extracted representations compared to other linear, non-linear and sparse CCA-based models.

Published as a conference paper at ICLR 2022
SPARSE CANONICAL CORRELATION ANALYSIS
Ofir Lindenbaum
Moshe Salhov Amir Averbuch School of Computer Science Tel Aviv University Tel Aviv, Israel
Yuval Kluger
Faculty of Engineering Bar Ila University Ramat Gan, Israel of Fringin@gaaii com comm
School of Medicine
Yale University New Haven, CT. USA
ABSTRACT
Canonical Correlation Analysis (CCA) models are powerful for studying the as- sociations between two sets of variables. The canonically correlated represen- tations, termed canonical variates are widely used in unsupervised learning to analyze unlabeled multi-modal registered datasets. Despite their success, CCA models may break (or overfit) if the number of variables in either of the modal- ities exceeds the number of samples. Moreover, often a significant fraction of the variables measures modality-specific information, and thus removing them is beneficial for identifying the canonically correlated variates. Here, we propose f-CCAA a method for learning correlated representations based on sparse subsets of variables from two observed modalities. Sparsity is obtained by multiplying the input variables by stochastic gates, whose parameters are learned together with the CCA weights via an Lo fo-regularieed correlation loss. We further propose fo-eeep CCA for solving the problem of non-linear sparse CCA by modeling the corre- lated representations using deep nets. We demonstrate the efficacy of the method using several synthetic and real examples. Most notably, by gating nuisance input variables, our approach improves the extracted representations compared to other models.
linear, non-linear and sparse
INTRODUCTION
Canonical Correlation Analysis (CCA) (Hoteling 1936 Thompson 2005). is a classic statistical method for finding the maximally correlated linear transformations of two modalities (or views). Us- ing modalities X E RDD*xN xN and Y e CDD* xN which are centered and have N samples with D and DV features, respectively, CCA seeks canonical vectors a E RD and b E RD" such that a X, and u = Y, will maximize the sample correlations between the canonical variates, i.e.
a X
p(a xa
max a, by0
||a T
X || bT 2
To identify non-linear relations between input variables, several extensions of CCA have been pro- posed. Kernel methods such Kernel CCA (Bach & Jordan| 2002), Konmperaettric CCA (Michaeli et al. 2016) or Multi-view Diffusion maps (Lindenbaum et al. a 2020 Salhov et al., 2020) learn the non non-linear relations in reproducing Hilbert spaces. These methods have several shortcomings: they are limited to a designed kernel, and they have poor interpolation and extrapolation capabilities.
Deep CCA (Andrew et al., 2013) overcomes these limitations by learning two non-linear transfor- mations parametrized using neural networks. The model has several extension, see for example (Wang et 2016 Gundersen et al. 2019 Karami & Schuurmans 2021).
CCA models have been widely used in biology (Pimentel et al., 2018). neuroscience (Al-Shargie ct al. 2017), medicine (Zhang et al., al. 2017), and engineering (Chen et al. 2017), for unsupervised or semi-supervised learning. By extracting meaningful dimensionality reduced representations, CCA improves downstream tasks such as clustering, classification, or manifold learning. One key limita- tion of CCA is that it requires more samples than features, i.e., N D" D",
Dd, However, if we have
M.S. is co affiliated with Playtika Tsrael

---

Published as a conference paper at ICLR 2022
Observed View
Observed View 2
Gates
Neural Subnets
Figure 1: Illustration of fg-DCCA. Data from two observed views propagate through stochastic gates (defined in Eq. 4) The gates output is fed into two neural sub-nets that have shared loss (see Eq. 5) We compute this shared loss based on the neural sub-nets outputs (with dimension d = 3 in this example). Our shared loss combines total correlation term with a differentiable regularization term which induces sparsity in the input variables (by sparsifying the gates).
more variables than samples, the estimation based on the closed-form solution of the CCA problem (in Eq. D breaks (Suo et al., 2017) (because the covariance matrix is low rank). Moreover, in high dimensional data, often some of the variables do not measure the phenomenon that common to both modalities (therefore are not correlated) and thus should be omitted from the transformations. For these reasons, there has been a growing interest in studying sparse CCA models.
Sparse CCA (SCCA) models seek linear transformations which are based on a sparse subset of vari- ables from the input modalities X and Y. Sparsifying the feature space improves interpretability, prevents overfitting, and removes the degeneracies inherent to N < D DV (once the number of selected variables is smaller than N the covariance matrix becomes full rank). To encourage sparsity of the canonical vectors and 6, several authors (Wiesel et al., 2008 Cai et al. 2019) propose an to regularized variant of Eq. [ However, these schemes are greedy and therefore may lead to sub- optimal solutions. As demonstrated by (Waaijenborg al. 2008 Parkhomenko et al. 2009 Witten et al. 2009 Hardoon & Shawe- Taylor
2011 et al.) 2017) replacing the En norm by & is differ- entiable and leads sparse solution to Eq. Several authors extended these ideas by considering group sparsity (Chen & Liu 2012 Klami et al. 2013 Solari et al. 2019). However, these methods are limited linear transformations and may lead to shrinkage of the canonical vectors due to the & regularizer. There has been limited work on extending these models to sparse non-linear CCA. Specifically, there are two kernel- extensions: two-stage kernel CCA (TSKCCA) by (Yoshida et al. 2017) and SCCA based on -chhidt Independence Criterion (SCCA-HSIC) by (Uurtio jtt al. 2018). However, these models suffer from the same limitations as KCCA and do not scale to high dimensional regime.
We present lg-CA, a simple yet effective method for learning correlated representation based on sparse subsets of the input variables by minimizing an to regularized loss. Our to regularization relics on a recently proposed Gaussian- based continuous relaxation of Bernoulli random variables, termed gates (aaaada et al. 2020). The gates are applied to the input features to sparsify the canoni- cal vectors. The model and gates parameters are trained jointly via gradient descent to maximize the correlation between the representations of X and Y while simultaneously selecting only the subsets of most correlated input features. Our contributions are four-folds: (i) By modeling the transforma- tions using two neural networks, we provide a natural solution to the problem of sparse non-linear correlation analysis. (ii) We then propose a novel initialization scheme that helps the model identify small subsets of correlated features in the challenging regime of D > N. (ii) We apply the pro- posed method to synthetic data and demonstrate that our approach improves the estimation of the canonical vectors compared with existing baselines. (iv) We demonstrate the efficacy of the pro- posed scheme to several real work applications; specifically, we demonstrate that it leads to more reliable and interpretable representations than other linear and non-linear data fusion schemes.
2 SPARSE CCA
The problem in Eq. has closed-form solution based the cigendecomposition of C.yC C.C-CCC.
Cgr yr and C- C C where Cz, Cy are within view sample covariance ma- trices and Ozy, Cyz are cross-view sample covariance matrices. However, if N is smaller than the number of input variables (Dz or D the sample covariance may be rank deficient, and the closed-form solution becomes meaningless. To overcome this limitation, we consider the problem of sparse CCA. Sparse CCA reduces the number of active variables and therefore the covariance ma- trix becomes full rank. Sparsity also improves model interpretability but may reduce the maximal (training) correlation value obtained in Eq. (Shalev-Shwarte et al. 2010).
2

---

Published as a conference paper at ICLR 2022
Sparse CCA deals with identifying maximally correlated representations based linear combina- tions of sparse subsets of the input variables in X and Y. The problem can be formulated as
min b
p(a" Fo'xxbby YY + |aaloo + >'Ilb|lo
where x and X are regularization parameters which control the sparsify the input variables. |allo and |Bllo are smaller than N, remove the degeneracy inherent to Eq. and identify meaningful correlated representations based on a sparse subset of input variables.
3 PROBABILISTIC REFORMULATION OF SPARSE CCA
The sparse CCA problem formulated in Eq. becomes intractable for large D or D moreover, due to the discrete nature of the lo norm, the problem is not amenable to gradient-based optimization schemes. Fortunately, as demonstrated sparse regression, probabilistic models such as the spike- and-slab (George & McCulloch] 1993; Kuo & Mallick 1998] Zhou et al. 2009) Polson & Sun 2019) provide a compelling alternative. More recently, differentiable probabilistic models such as [oouizos et al.) 2017 Yamada et al. 2020) Jana et al. 2021) were proposed for sparse supervised learning. Here, we adopt these ideas by rewriting the canonical vectors as a = 0" @ s" and 3 0 @ 8 where @ denotes the Hadamard product (element wise multiplication), and 0" RD 0" E RRO" The vectors 8* E 1) 8 E {0. 1) are Bernoulli random vectors with parameters π" = (-f :-.., Th= and π (mi, Th These Bernoulli variables, sparsify the coefficients of the canonical vectors. Now, the problem in Eq. 2 can be reformulated as an expectation minimization, which is parameterized by π" and π² Specifically, based on the following theorem, we can reformulate the problem in Eq.
gates and
Theorem 3.1 The solution of the sparse CCA problem in Eq. 2 is equivalent the solution of the following probabilistic problem
min " tT,,
E
x(a" X + 1 \*
OU
where t" @ 8 and f 0" ( 8 and the expectation is taken with respect to the random Bernoulli variables s" and (which are parameterized by π and T
Note that the expected values of the Eg norms can be expressed using the Bernoulli parameters as E||s* lo = Σ nF, and R.|ss*|o = Σ -.- The proof relies on the fact that the optimal solution to Eq. 2 is valid solution to Eq. and vice versa. The proof is presented in the Appendix, Section H
and follows the same construction as the proof of Theorem in (Yin et al. 2020).
3.1 CONTINUOUS RELAXATION FOR SPARSE CCA
Due to the discrete nature of s" and s" differentiating the leading term in Eq. B is not straightfor- ward. Although solutions such as REINFORCE (Williams) 1992) or the ttrighh-througe estimator (Bengio et al. 2013) enable differentiating through discrete random variables, they still suffer from high variance. Furthermore, they require many Monte Carlo samples for effective training (Tucker al. 2017). Recently, several authors (Maddison al. 2016 Jang al., [0066] Louizos et al.j 2017 have demonstrated that using continuous reparametrization of discrete random variables can reduce the variance of the gradient estimates. Here, following the reparametrization presented in
Yamada et al. 2020, Lindenbaum et al. 2021). we use Gaussian- relaxation for the Bernoulli random variables.
Each relaxed Bernoulli variables is defined by drawing from a centered Gaussian E; N N(0,a3) then shifting by it and truncating its values using the following hard Sigmoid function
z; max(O, min(1, μ4 +c)).
Using these relaxed Bernoulli variables, and 3 = 0² c zz We incorporate these vectors into the objective defined in Eq. 3 in which the to regularization terms can be expressed as
define the gated canonical vectors a 0 o z
z 10 = P(z 2 0) =
1=
D= Σ
erf
(- /20/ (-veo)
(- Mr (-
2 2
1=1
3

---

Published as a conference paper at ICLR 2022
where erf() is the Gaussian error function, and is defined similarly for B||2"|10-
To learn to model parameters 0", @ and gate parameters je, μ we first draw realizations for the gates, then update the parameters by applying gradient descent to minimize
E
X x, Y) + 1*1|z"11o + > llo
Post training, we remove the stochasicity from the gates and use all variables such that z = max(0, min(1, (z)) (defined similarly for z²).
3.2 GATE INITIALIZATION
If the gates are initialized with μ4; = 0.5, they will approximate "fair" Bernoulli variables. This is a reasonable choice if no prior knowledge about the solution is available; however, we can utilize the closed-form solution of the CCA problem to derive more suitable parameter initialization for the gate. Specifically, given the empirical covariance matrix Cay
XYT we denote the thresholded (N (N-1J+
covariance matrix by C ay: with values defined as follows
(CCoy)ss if I 8 (Czy)is = 10,
otherwise,
where d is the selected threshold value based on the desired sparsity for X and Y, specifically, we assume that percent of the values should be zeroed, then 5 is set to be the r-th percentile of ICay)]/ Then, we compute the leading singular vectors u and of Czy- We further threshold the absolute values of these vectors using the same percentile used for Cay- The initial values of the parameters of the gates are then defined by μ² = ū + 0.5, and μ² 5 0.5, where ū and ō are the thresholded versions of the absolute value of the singular vectors. Using this procedure, we increase the initial probability for all gates in the support of i and T based on the singular vectors of Ozy-
3.3 Eo DEED CCA
To extend our model to non-linear function estimation, we can formulate the problem of sparse non-linear CCA by modeling the transformations using deep nets as (Andrew et al.] 2013 Wang et al. 2015b). We introduce two random Bernoulli relaxed vectors into the input layers of two neural networks trained in tandem to maximize the total correlation. Denoting the random gating vectors z and nd for view X and Y. respectively, our fo-Deep CCA (Go-DCCA) loss is defined by
(()((()) + A3 2 llo + X3
(5)
where f(X) f(2 @ x E R and g ( = g("" @ Y| E RRx are modeled deep networks with model parameters 0 (0 )v), and gate parameters μ [==,,1²) The functions ff and g embed the data into a d-dimensional space (this formulation could also be used for linear f and g). The functional (f{(), gX\)) measures the total correlation between the two dimensional outputs of the deep nets, this the sum over correlation values computed between pairs of coordinates. Exact details on the computation of this term appear in the following subsection. The regularization parameters X=, X control the sparsity of the input variables. The vectors 2t and n are Bernoulli relaxed vectors, with elements defined based on Eq. 4
Figure. T highlights the proposed architecture. We first pass both observed modalities through the gates. Then, we feed these into view-specific neural sub-nets. Finally, minimize the shared loss term in 5lby optimizing the parameters of the gates and the neural sub-nets.
3.4 ALGORITHM DETAILS
Denoting the centered representations for Y L L E Rd (computed using the coupled neural sub-nets), respectively, the empirical covariance matrix between these representations be expressed Č.,, Nー r (T))T Using a similar notations, we express the regularized empiri-
cal covariance matrices of X and Y as Č. N- V ( I I and Č2 = N 4 (VVV) +7I,
4

---

Published as a conference paper at ICLR 2022
where the matrix 7I (y >> 0) is added to stabilize the invertibility of C2 and Cy. The total correla- tion in Eq. (i.e. (()()), g.g(()))))))) (Y)) can be expressed using the trace of Č 8-112/ yy C
To learn the parameters of the gates and of the representations 0 we apply full batch gradient decent to the loss in Eq. [S] Specifically, we use Monte Carlo sampling estimate the left part of Eq. []] This is repeated for several steps (epochs), using one Monte Carlo sample between each gradient step as suggested by (Louizos et al. 2017) and (Yamada et al. 20000, and worked well in our experiments. After training, we remove the stochastic part of the gates and use only the variables 1 E (1,.., and E [I,..., Dy} such that z > 0 and 2% > 0. Finally, we apply linear CCA to the low dimensional extracted embedding to remove this arbitrary scaling and use the normalized generalized eigenvectors (of the closed-fort CCA solution) as our canonical vectors. Empirically, observe that our method also works well with stochastic gradient descent, as long as the batch size not too small. Alternatively, for small batches we can use a variant of the total correlation loss presented in (Wang al., (2015c) or (Chang et al. 2018). In the Appendix section [ we present a pseudo-code of Lo-DCCA evaluate the complexity of the method, and extend it for multi-modal setting (more than two views).
4 RELATED WORK
The problem of of lo sparse CCA was studied in (Wiesel et al. 2008 Cai et al. 2019). However, both rely greedy heuristic which iteratively adds non zero elements the support of and b. Alternatively, as proposed by (Suo et al. 2017). an (n f1-regularized problem can be described as
a, b = argmin CCCCCo'a (oov(aXx000 X,," + 71 ///a/11 + T2 b|11)
subject to
|
X S 1,
1 Y 2 <
where Ti and are regularization parameters. A number of variants have been proposed to this problem (Waaijenborg et al. 2008 Parkhomenko et al. 2009) Witten et al. |009 Hardoon Taylor 2011), but they all suffer from parameter shrinkage and therefore lead to a solution which is less consistent with the correct canonical vectors (see Table
& Shawe-
5 EXPERIMENTAL RESULTS
We validate the effectiveness of the proposed approach on a wide range of tasks. First, using syn- thetic data, we demonstrate that C-CCA correctly identifies the canonical vectors in a challenging regime of N < D DV Next, using coupled video dataset, we demonstrate that l-CCA can identify the common information from high dimensional data, and embed it into correlated, low- dimensional representations. Then, data to demonstrate that 4-CCCA finds meaningful representations of the data even in a noisy regime. Finally, genetic measurements. In all experiments validation sets are used for tuning the hyperparameters of all basclines by maximizing the total correlation on the validation set. We refer the reader the Appendix for a complete description of the baselines, training procedure, and parameters choice for all methods.
noisy images from MNIST and multi-channel seismic
DC-C to improve cancer sub-type classification using high dimensional
5.1 SYNTHETIC EXAMPLE
To generate samples from X, Y E DxN, we follow the procedure described in (uoo al. E.y
2017)
considering data sampling from the following distribution X
0
Σn
N
Σ yz E,
where Σny = po E. ( We study three cases for the covariance matrices Σ Σn Ey
Model I. Identity: Σ- DD Model II. Toeplitz: Σi, Pu ji-sl i 1,..., D.
F.,, where Σ = r- Fr,j = 1== + 0.5Li + 0.41:=5- V.,
Model III. Sparse inverse: Σt,j
For all three cases, the vectors Φ,, n RR,, are sparse with nonzero elements and po 0.9. The indices of the active elements are chosen randomly with values equal to 1/v5. In this setting, the
5

---

Published as a conference paper at ICLR 2022
100
0.5
90
0.55
0.3
0.50
50
40
01
0.70
30
20
-01
10
0.60
10P
102
100
200 300
400
500
600
700
800
Index
Figure 2: Left: Regularization path of C-CCA on data generated from the linear model (described in Section 5.1p. Values on the left y-axis (green) represent the sum of active gates (by expectation). Values on the right y-axis (blue) represent the empirical correlation between the estimated repre- sentations, i.e. o" X Y' where e and f are the estimated canonical vectors. Dashed lines indicate the correct number of active coefficients (10) and true correlation (0.9). Note that for small values of = X= X², the model selects many variables and attains higher correlation value; in this case, fu-CCA suffers from overfitting. Right: True canonical vector Φ along with the estimated vectors using fu-CCA (Ø) and CCA (â). Due to the small sample size, CCA overfits and fails to identify the correct canonical vectors.
canonical vectors a and maximizing the correlation objective in Eq. | are & and n7. respectively (see Proposition 1 in (Suo et al. 2017)).
Model I- Identity covariance
(N, D", DD)
(400,800,800)
(500,600,600)
(700,1200, 1200)
PMA (Witten et al. 2009)
(1.170,1.170)
(0.850,0.850)
(1.090,1.090)
IP-SCCA (Mai & Zhang 2019
(1.658,1.647)
(1.051,1.051)
(1.544,1.542)
SCCA-I (Hardoon & Stawe-Taylon 2011
(1.602,1.140)
(1.143.0.282)
(1.160,0.18))
SCCA-II (Gao et al. 2017)
(0.060,0.066)
(0.053,0.057)
(0.045,0.043)
mod-SCCA (Suo al. 2017
(0.056,0.062)
(0.05,0.056)
(0.045,0.043)
l--CA
(0.003,0.009)
(0.002,0.002)
(0.001,0.002)
Model II- Toeplitz covariance
PMA (Witten et al. 2009)
(1.038,10067)
(1.115.0.943)
(1.098,0.890)
IP-SCCA (Mai & Zhang 2019
(NA,NA)
(NA.NA)
(NA,NA)
SCCA-I Hardoon & Shawe- Taylor 2011
(1.219,1.188 (1.351,1.299) (1.382,1.357)
SCCA-II [ao et al. 2017)
(0.213,0.296)
(0.145,0.
(0. (0.000)
mod-SCCA (Suo et al. 2017
(0.173,0.218)
(0. (0.136.0.098)
(0. 10..0.00)86)
l--CA
(0.101,0.079)
(0.098,0.072)
(0.026,0.039)
Model III- Sparse Inverse covariance
PMA (Witten et al. 2009)
(0.930,1.050)
(0.670,0.450)
(0. (0.760,0.580)
IP-SCCA (Mai & Zhang 2019
(0.654.0.653)
(0.092,0.091)
(0.282,0.285)
SCCA-I Hardoon & Shawe- Taylor 2011
(1.375,0.966)
(1.041 50.502)
(0.985,0.364)
SCCA-II [Gao et al. 2017)
(0.129,0.190
00.069.0.061
(0.051,0.041
mod-SCCA (Suo et al. 2017
(0.092,0. 00.092,0.1499
(0.068.0.8) 0...). (0.026,0.036)
(0.050,0.08
l--CAA
(0.10)
(0.009,0.005)
Table 1: Evaluating the estimation quality of the canonical vectors Φb and n7. Each pair indicates (Eo) en)) which are the estimation errors of & and respectively. We compare the proposed t-- CCA to other sparse CCA schemes considering three types of covariance matrices for generating X and Y. and different dimensions (N, D D*). The description of all three covariances appears in Section 5.1 We highlight the smallest error obtained across all methods using boldface.
Using Model I, we first generate N = 400 samples, with D = 800, and estimate the canonical vectors based on CCA and lo-CA. In Fig. 2 we present a regularization path of the proposed scheme. Specifically, we apply CoCCA to the data described above using various values of = x= > We present the to of active gates (by expectation) along with the empirical correlation
between the extracted representations, defined by = p( X, YY) As evident from the left panel, a wide range of À values Ieads to the correct number of active coefficients (10) and the correct correlation value (po 0.9). Next, in the right panel we present the values of &, the fo-CCA estimate (using À 30) of the canonical vector b. and the CCA spectral based estimate of the canonical vector à. Due to the low number of samples, the solution by CCA is wrong and not sparse, while the CCCC solution correctly identifies the support of Φb

---

Published as a conference paper at ICLR 2022
[ø""2] Next, evaluate the estimation error of Φ using <Φb = 2(1 1 Φ1), and en is defined similarly. In Table I we present the estimation errors of & and p (averaged over 100 simulations) for Models I, II and III (identity, Toeplitz and sparse inverse covariance matrices). As baselines, we compare the performance significantly outperforms all baselines in its ability to learn the correct canonical vectors. Here, restrict our evaluation to five baselines since code was not available We have implemented (Wiesel et al. 2008) but observed that it does not converge to the correct canonical vectors (the estimation errors Eg and Ep were 2). In the Appendix, section F we provide runtime evaluation of the method for different values of N and D.
leading sparse CCA models. As evident from these experiments, fo-CCA
other sparse CCA models.
5.2 MULTI VIEW OF SPINNING PUPPETS
As an illustrative example, we use a dataset collected by (Lederman & Talmon 2018). The authors have generated two videos capturing rotations of 3 desk puppets. One camera captures two puppets, while the other captures another two, where one puppet is shared across cameras. A snapshot from both cameras appears in Fig. 3 All puppets different frequencies. In both videos, there is a shared underlying parameter, namely the rotation of the common bulldog. We use 400 images from each camera, where each image has 240 320
placed spinning devices that rotate the dolls
76800 pixels (using grayscaled version of the colored image); therefore, N < D. and direct application of CCA would fail. We apply the proposed scheme using X X= 50, a linear activation and embedding dimension d = 2. fo-CCA converges to embedding with total correlation of 1. 1.99 using 372 and 403 pixels from views X and Y. The active gates are presented in the right panels of Fig. 3 In this example, the gates correctly attenuate the modality-specific information and preserve subsets pixels capturing common information, allowing for embedding gated videos into a shared correlated representation.
Figure 3: Left: two samples from the spinning puppets videos. The videos capture the rotation of desk puppets. Arrows indicate the spinning direction of each puppet. We use fo-CCA to identify sparse subset of pixels that are correlated from both videos. Right: the values of the gates for each video z and z² After training, the values of the gates are binary (i.e., {0, 1}), with 372 and 403 active gates for the left and right videos, respectively. to CCACA correctly select correlated subsets of pixels that highlisht the common Dupnet (Bulldog) in this examnle.
Figure 4: The correlated 4--CAA representations (visualized using MVDM (Lindenbaum et al., 2020)) of the Yoda+Bulldog video (left) and Bulldog+Bunny (right). We superimpose each em- bedding with 6 images corresponding to 6 points in the embedding spaces. The transformations are based on the information described by pixels whose gates are active (presented in Fig. B). The resulting embeddings are correlated with each other, with total correlation of p = 1.99. The struc- ture captured by the embeddings correctly represent the angular rotation of the Bulldog, which the common latent parameter in this experiment.

---

Published as a conference paper at ICLR 2022
8
9
Figure 5: Images from the coupled noisy MNIST dataset. the bottom right both panels, we presents the active gates (white values within green frame). There are 277 and 258 active gates for view I and II, respectively.
In Fig. we present the coupled two-dimensional embedding of both videos. Both embeddings are correlated with each other and reveal the correct angular orientation of the Bulldog (which is the common latent parameter in this experiment). This indicates the ability of lo-CCA to identify correlated latent variables in the regime of N < D*, DV.
5.3 NOISY MNIST
We use two noisy variants of MNIST (LeCun et al. 2010) as our coupled views. The first view is created by adding noise drawn uniformly from [0, 1 all pixels. The second view is created by placing a random patch from a natural image in the background of the handwritten digits. Random
samples from these modalities are presented in Fig. 5 Each view consists of 62, 000 samples, of which we use 40, 000 for training, 12, 000 for testing and 10, 000 are used as a validation set.
We train DCCAA to embed the data into a correlated 10 dimensional space while selecting subsets of input pixels. Our model selects 277, and 258 pixels from both modalities respectively (see bottom right corner of Fig. 5). Next, we evaluate the quality of learned embedding by applying k-means to the stacked embedding of both views. We run k-means (with k = 10) using 20 random initializations and record the run with the smallest sum of square distances from centroids. Given the cluster assignment, i-means clustering accuracy (KM) and mutual information (MI) are measured using the true labels. Additionally, we train a Linear-S S SVM (SVM) model on our train and validation datasets. SVM classification accuracy is measured DCCD leads to higher classification and clustering results compared with several linear and non- linear modality fusion models appear in Table 2 In the Appendix, we provide the implementation details and present experimental results demonstrating the accuracy vs. sparsity trade-off of 40- DCCA.
the remaining test set. The embedding provided by
5.4 SEISMIC EVENT CLASSIFICATION
Next, we evaluate the method using dataset of seismic events studied by (Lindenbaum et al. 2018). Here, we focus on 537 explosions which are categorized into quarries. Each event is recorded using two directional channels facing east (E) and north (N); these comprise the coupled views for the correlation analysis. Following the analysis by (Lindenbaum et al.) (2018), the input features are sonogram representations of the seismic signal. Sonograms bins equally tempered on a logarithmic scale. Each sonogram z E 1R17 with 89 time bins and 13 frequency bins. An example of sonograms from both channels appears in the top of Fig. 6
time-frequency representations with
We create the noisy seismic data by adding sonograms computed based on vehicle noise from Examples of noisy sonograms appear in the middle row of Fig. 6 We hold 20% of the data as validation set, and train DDCCC to embed the data in 3 dimensions. In Table we present the MI, k-means and SVM accuracies computed based on DCCA embedding. Furthermore, we compare the performance with several other baselines. Here, the proposed scheme improves performance in all metrics while identifying a subset of 17 and 16 features from channel E and N, respectively. The active gates are presented in the bottom row of Fig. 6 Our results indicate that even in the presence of strong noise, 6 -CCCA correctly activates the gates in frequency bins that coincide with the energy stamps of the primary and secondary waves (P and S in the top left of Fig, 6
8

---

Published as a conference paper at ICLR 2022
Frequency
Frequency
Time
Time
Frequency Frequency
Frequency Frequency
Time
Time
Time
Clean sample sonograms of explosion based on the E and N channels (left and right, respectively). Arrows highlight the Primary (P) and Secondary (S) waves caused by the ex- plosion. Middle: noisy sonograms generated by adding sonograms of vehicle recordings. Bottom: the active gates for both channels. Note that the gates are active at time-frequency bins which corre-
Figure 6: Top:
spond to the P and S waves (see top left panel).
MINIST
Lindenhau
NIIN
METABRIC
SVM[
KM
Daia PCA
0.130
86.5
0.001
35.1
41.3
0.36
36.5
63.8
Chirrooof
med SCCA NWW
0.342
23.9
63.1
0.610
71.7
86.9
0.12
21.0
26.0
SCCA-IISIC
Curto
2019
NA
NA
NA
0.003
38.7
49.5
NA
NA
NA
KCCA (Bach
A177
0.443
5112
85.3
0.006
38.4
92.5
0.35
30.8
61.3
multiview-i
NOCA Michieir Wilte
1,030
47.5
77.2
0.700
86.8
91.4
0.72
DCCA Kiinlow di 2i
2013
1.970
93.2
93.2
0.830
94.9
94.6
0.79
45.2
72.1
DCCAF
Waing amai 20136 1.940
91.8
94.0
0.92
97.0
97.0
0.68
42.9
69.0
87.1
88.4
0.85
0.82
20.7
(noe:
Table 2: Evaluation of correlated embedding extracted from the Noisy MNIST, Seismic, and METABRIC (cancer type) datasets. The representation extracted by f-DCCC leads to higher clus- tering accuracy (KM), and classification accuracy (SVM) compared with several baselines. MI indicates the Mutual Information between cluster assignments and sample labels. We use NA to denote simulations which did not converge.
5.5
CANCER SUB-TYPE CLASSIFICATION
Accurate classification of cancer sub-type is vital for extending the life span of patients by person- alized treatments (Zhu et al.) 2020). This task is challenging since the number of measured genes is typically much larger than the number of observations. Here, we multi-modal observations from the METABRIC data (Curtis et al. 2012) and attempt to find correlated representations to improve cancer sub-type classification. The data consists of 1, 112 breast cancer patients which are annotated by 10 subtypes based on InClust (Dawson et al. 2013). We observe two modalities, namely the RNA gene expression data, and Copy Number Alteration (CNA) data. The dimensions of these modalities are 15, 709 and 47, 127, respectively. We compute the fo-DCCA 10 dimensional embedding (and all baseline embeddings) and demonstrate using k-means and SVM that the representation identified using DCCA can lead to more accurate cancer sub-type classification (see 2).
CONCLUSION
This paper presents method for learning sparse non-linear transformations that maximize the canonical correlations between two modalities. Our approach is realized by gating the input lay- of two neural networks, which are trained optimize their output's total correlations. Input variables are gated using a regularization term which encourages sparsity. We further propose novel scheme initialize the gates based on thresholded cross-covariance matrix. Our method can learn informative correlated representations even when the number of variables far exceeds the number of samples. Finally, we demonstrate that the proposed scheme outperforms existing algo- rithms for linear and non-linear canonical correlation analysis. We describe our future directions in the Appendix section
7 ACKNOWLEDGMENT
The work of YK was supported by the National Institutes of Health R01GM131642, UMIPA05141, U54AG076043, P50CA121974, and U01DA053628.

---

Published as a conference paper at ICLR 2022
REFERENCES
Fares Al-Shargie, Tong Boon Tang, and Masashi Kiguchi. Assessment of mental stress effects on prefrontal cortical activities using canonical correlation analysis: an fnirs-ceg study. Biomedical optics express, 8(5):2533 2599, 2017.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International Conference on Machine Learning, pp. 1247-1255, 2013.
Francis R Bach and Michael Jordan. Kernl independent component analysis. Journal of machine learning research, 3(Jul):1-45 2002.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on paltern analysis and machine intelligence, 3588):1798-18288 2013.
Adrian Benton, Huda Khayrallah, Biman Gujral, Dee Ann Reisinger, Sheng Zhang, and Raman Arora. Deep generalized canonical correlation analysis. In Proceedings of the 4th Workshop Representation Leurning for NLP (RepLANLP-2019), pp. 1-6, 2019.
Sarah M Berharrtt, Pallave Dasari, David Walsh, Amanda R Townsend, Timothy J Price, and Wendy V Ingman. Hormonal modulation of breast cancer gene expression: implications for intrinsic subtyping in premenopausal women. Frontiers in oncology, 2016.
Jia Cai, Wei Dan, and Xiaowei Zhang. 10-based sparse canonical correlation analysis with applica- tion to cross-language document retrieval. Neurocomputing, 329:32 02: 2019.
D. Carroll. Generalization of canonical correlation analysis to three or more sets of variables. Proceedings of the 76th annual convention of the American Psychological Association, 3:227- 228, 1968.
Xiaobin Chang, Tao Xiang, and Timothy M Hospedales. Scalable and effective deep eca via soft decorrelation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- fion, pp. 1488-1497, 2018.
Kamalika Chaudhuri, Sham M Kakade, Karen Livescu, and Karthik Sridharan. Multi-view cluster- ing via canonical correlation analysis. In Proceedings of the 26th annual international conference on machine learning, pp. 129-136. ACM, 2009.
Xi Chen and Han Liu. An efficient optimization algorithm for structured sparse cca, with applica- tions to equi mapping. Statistics in Biosciences, 4(1):3-26 26, 2012.
Zhiwen Chen, Steven X Ding, Tao Peng, Chunhua Yang, and Weihua Gui, Fault detection for non- gaussian processes using generalized canonical correlation analysis and randomized algorithms. IEEE Transactions on Industrial Electronics, 6552)1155-11567, 2017.
Christina Curtis, Sohrab P Shah, Suet-Feung Chin, Gulisa Turashvili, Oscar M Rueda, Mark J Dun- ning. Doug Speed, Andy Lynch, Shamith Samarajiwa, Yinyin Yuan, al. The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature, 486(7403): 346, 2012.
Sarah-Jane Dawson, Oscar M Rueda, Samuel Aparicio, and Carlos Caldas. A new genome-driven integrated classification of breast cancer and its implications. The EMBO journal, 32(5):617-628, 2013.
Chao Gao, Zongming Ma, Harrison H Zhou, et al. Sparse cca: Adaptive estimation and computa- tional barriers. Annals of Statistics, 45(5):2074-1001, 2017.
Edward I George and Robert E McCulloch. Variable selection via gibbs sampling. Journal of the American Statistical Association, 88(423):881-8 -889 1993.
Gregory Gundersen, Bianca Dumitrascu, Jordan T Ash, and Barbara E Engelhardt. End-to-end training of deep probabilistic cca on paired biomedical observations. In Uncertainty in artificial intelligence, 2019.
10

---

Published as a conference paper at ICLR 2022
David R Hardoon and John Shawe-Taylor. Sparse canonical correlation analysis. Machine Learning, 83(3):331-355 2011.
Paul Horst. Generalized canonical correlations and their applications to experimental data. Journal of Clinical Psychology, 17(4):331-34, 1961.
Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3/4):321-37) 1936.
Soham Jana, Henry Li, Yutaro Yamada, and Ofir Lindenbaum. Support recovery with stochastic gates: Theory and application for linear models. arXiv preprint arXiv:2110.15960, 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztioon with gumbel-softmax, arXiv preprint arXiv: 16l.:6611.. 1.01 144, 2016.
Mahdi Karami and Dale Schuurmans. Deep probabilistic canonical correlation analysis. In Pro-
ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9055-8063, 2021.
J. R. Kettering. Canonical analysis of several sets of variables. Biometrika, 58(3) 451, 12 1971. ISSN 04004,344, doi: 10. 1...099/ioomet/58...555.. URI bi biomet/58..3 433
Arto Klami, Seppo Virtanen, and Samuel Kaski. Bayesian canonical correlation analysis. Journal of Machine Learning Research, 14(4). 2013.
I Lynn Kuo and Bani Mallick. Variable selection for regression models. Sankhyā: The Indian Journal of Statistics, Series B, pp. 65-81, 1998.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online). Available: http:://ann.lecau http://www..ee....o////////m/aaa wwtp./////////ssrtt..mmm 2010.
Roy R Lederman and Ronen Talmon. Learning the geometry of common latent variables using internatin-diffusiion Applied and Computational Harmonic Analysis, 44(3)5509-536, 2018.
Ofir Lindenbaum, Yuri Bregman, Neta Rabin, and Amir Averbuch. Multiview kemnels for low- dimensional modeling of seismic events. IEEE Transactions on Geoscience and Remote Sensing, 36(6):3300-3100, 2018.
Ofir Lindenbaum, Arie Yeredor, Moshe Salhov, and Amir Averbuch. Multi-view diffusion maps. Information Fusion, 55:127-149 2020.
Ofir Lindenbaum, Uri Shaham, Erez Peterfreund, Jonathan Svirsky, Nicolas Casey, and Yuval Kluger. Differentiable unsupervised feature selection based on a gated laplacian. Advances in Neural Information Processing Systems, 34, 2021.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. arXiv preprint arXiv: 2017.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv/6611.00712 2016.
Qing Mai and Xin Zhang. An iterative penalized least squares approach to sparse canonical corre- lation analysis. Biometrics, 75(3):734-744 2019.
Roderick P McDonald. A unified treatment of the weighting problem. Psychometrika, 33(3):351- 381, 1968.
James John McKeon. Canonical analysis: Some relations between canonical correlation, fuctor analysis. discriminant function analysis, and scaling theory. Number 13. Psychometric Society, 1966.
Tomer Michaeli, Weiran Wang, and Karen Livescu. Nonparametric canonical correlation analysis. In International Conference on Machine Learning, pp. 1967-1976, 2016.

---

Published as a conference paper at ICLR 2022
Elena Parkhomenko, David Tritchler, and Joseph Beyene. Sparse canonical correlation analysis with upplication to genomic data integration. Statistical applications in genetics and molecular biology, 8(1), 2009.
Karl Pearson. Principal components analysis. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 6(2):559, 1901.
Harold Pimentel, Zhiyue Hu, and Haiyan Huang. Biclustering by sparse canonical correlation anal- ysis. Quantitative Biology, 6(1 (11):56-67, 67, 2018.
Nicholas G Polson and Lei Sun. Bayesian O-regularized least squares. Applied Stochastic Models in Business and Industry, 35(3)711-773, 2019.
Hugo Richard, Luigi Gresele, Aapo Hyvärinen, Bertrand Thirion, Alexandre Gramfort, and Pierre Ablin. Modeling shared responses in neuroimaging studies through multiview ica, arXiv preprint arXiv:2006. 06635, 2020.
Moshe Salhov, Ofir Lindenbaum, Yariv Aizenbud, Avi Silberschatz, Yoel Shkolnisky, and Amir Averbuch. Multi-view kernel consensus for data analysis. Applied and Computational Harmonic Analysis,
2020.
Shai Shalev-Shwartz, Nathan Srebro, and Tong Zhang. Trading accuracy for sparsity in optimization problems with sparsity constraints. SIAM Journal on Optimization, 20(6)) 2010.
Omid S Solari, James B Brown, and Peter J Bickel. Sparse canonical correlation analysis via concave minimization. arXiv preprint (Tt.:19999079477 2019.
Xiaotong Suo, Victor Minden, Bradley Nelson, Robert Tibshirani, and Michael Saunders. Sparse canonical correlation analysis. arXiv preprint (t0.::7705.00666 2017.
Arthur Tenenhaus and Michel Tenenhaus. Regularized generalized canonical correlation analysis. Psychometrika, 76(2):277, 2011.
Arthur Tenenhaus, Cathy Philippe, Vincent Guillemot, Kim-Anh Le Cao, Jacques Grill, and Vincent Frouin. Variable selection for generalized canonical correlation analysis. Biostatistics, 156::5669 583, 02 2014. ISSN 1465-4644. doi: 10.1099/bbssaaiisiic/k000000.111 Utttttts::///0 org/ 10. I0.1003/b.estaaiisiies///ss
Arthur Tenenhaus, Cathy Philippe, and Vincent Frouin. Kernel generalized canonical correlation analysis. Computational Statistics and Data Analysis, 90:114-131, 2015. ISSN 0167-9473. doi: http://0ooo.op/lll...l0lccsss...
URL httpt: //www sciencedirect. com/
Bruce Thompson. Canonical correlation analysis. Encyclopedia of statistics in behavioral science, 2005.
George Tucker, Andriy Mnih, Chris Maddison, Dieterich Lawson, and Jascha Sohl-Dickstein. Re- bar: Low-variance, unbiased gradient estimates for discrete latent variable models. urXiv preprint 17X 2017.
Viivi Uurtio, Sahely Bhadra, and Juho Rousu. Sparse non-linear cca through hilbert-schmidt inde- pendence criterion. In 2018 IEEE International Conference on Data Mining (ICDM), pp. 1278- 1283. IEEE, 2018.
Viivi Uurtio, Sahely Bhadra, and Juho Rousu. Large-scale sparse kernel canonical correlation anal- ysis. In International Conference on Machine Learning, pp. 1383-6391. PMLR, 2019.
John P Van de Geer. Linear relations amongk sets of variables. Psychometrika, 49(1):79-9) 94, 1984.
Sandra Waaijenborg, Philip C Verselewel de Witt Hamer, and Aeilko H Zwinderman. Quantifying the association between gene expressions and dna-markers by penalized canonical correlation analysis. Statistical applications in genetics and molecular biology. 7(1), 2008
12

---

Published as a conference paper at ICLR 2022
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi view representation learning. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pp. 1083-1092. JMLR.org. 2015a.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International conference on machine learning, pp. 1083-1092. PMLR, 2015b.
Weiran Wang, Raman Arora, Karen Livescu, and Nathan Srebro. Stochastic optimization for deep cca via nonlinear orthogonal iterations. In 2015 53rd Annual Allerton Conference on Communi- cation, Control, and Computing (Allerton), pp. 688-695. IEEE, 2015c.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correla- tion analysis. arXiv preprint arXiv:/610.03445 2016.
Ami Wiesel, Mark Kliger, and Alfred Hero III. greedy approach to sparse canonical correlation analysis. arXiv preprint arXiv:0801. 2008.
Ronald J Williams. Simple statistical gradient-oollowing algorithms for connectionist reinforcement learning. Machine leamning, 8(3-4)2229-255 1992.
Danicla M Witten, Robert Tibshirani. and Trevor Hastie. A penalized matrix decomposition, with applications sparse principal components and canonical correlation analysis. Biostatistics, 10 (3):515-534, 2009.
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates, In Proceedings of Machine Learning and Systems 2020, pp. 8952-8963. 2020.
Mingzhang Yin, Nhat Ho, Bowei Yan, Xiaoning Qian, and Mingyuan Zhou. Probabilistic best subset selection via gradient-based optimization. arXiv e-prints, pp. arXiv-2006, 2020.
Kosuke Yoshida, Junichiro Yoshimoto, and Kenji Doya. Sparse kernel canonical correlation analysis for discovery of nonlinear interactions in digh-siinnsio data. BMC bioinformatics, 18(1):1- 11, 2017.
Yu Zhang, Guoxu Zhou, Jing Jin, Yangsong Zhang, Xingyu Wang, and Andrzej Cichocki. Sparse bayesian multiway canonical correlation analysis for eeg pattern recognition. Neurocomputing, 225: 2017.
Mingyuan Zhou, Haojun Chen, John W Paisley, Lu Ren, Guillermo Sapiro, and Lawrence Carin. Non-parametric bayesian dictionary learning for sparse image representations. NIPS, volume 9, pp. 2295 2303. Citeseer, 2009.
Wan Zhu, Longxiang Xie, Jianye Han, and Xiangqian Guo. The application of deep learning in cancer prognosis prediction. Cancers, 12(3):603, 2020.
A
APPENDIX
B ADDITIONAL EXPERIMENTAL DETAILS
In the following sections we provide additional experimental details required for reproduction of the experiments provided in the main text. All the experiments were conducted using Intel(R) Xeon(R) CPU E5-2620 v3 @2.4Ghz x2 (12 cores total).
B.1 SELECTING THE EMBEDDING DIMENSION
Choosing the embedding dimension might require application-speeific considerations. For example, for clustering purposes, if the number of clusters is known (K), we might select d = K to have sufficient number of dimensions to separate the clusters. However, for other applications, we can validation set to tune the embedding dimension d; precisely, by increasing d until the total correlation in the validation set stops increasing. Using this procedure we can estimate the most significant number of correlated emhedding coordinates.
13

---

Published as a conference paper at ICLR 2022
B.2 SYNTHETIC EXAMPLE
For the linear model we use learning rate of 0.005 with 10, 000 epochs. The values of X and A are both set to 30. These values were obtained using cross validation procedure. The standard deviation of the injected noise set to 0.5 by (Ymaada et al. 2020). They have selected this value as it maximized the gradient of the regularization term at initialization. Empirically, we ob- serve that for e -CCCA smaller values of translated to improved convergence. Specifically, we used o = 0.25, which worked well in our experiments.
For each data model, we the method 100 times with different realizations of X and Y. In Table
1 we compare the average results of 2 -C-CCA to PMA (Witten et al. 22009). IP-SCCA (Mai & Zhang, 2019), SCCA-I (Hardoon & Shawe-Taylor 201T),CCAA (Gao al. 2017), and mod-SCCA (Suo 2017). Results of SCCA-II, PMA, and mod-SCCA were simulated by (Suo et al. 2017).
ct al.
To demonstrate the tuning procedure of X and Xy, we generate data based on Model-I (described in Section [.1] and evaluate the total correlation on training and validation samples. The results As demonstrated in this plot, the correlation values on unseen samples can indicate the optimal values for the regularization parameters. Namely, the model leads to lower correlation values when selecting nuisance variables or removing some of the informative variables.
presented in Fig.
100
90
Zz Zg lo + Z b
80
70
0.90
60
0.88
40
30
20
0.86
10
000000000
0.84
10a
10a
10
10
Figure 7: Regularization path of CCAA on data generated from the linear model I (described in Section 5.1). Values on the left y-axis (green) represent the sum of active gates (by expectation). Values the right y-axis (blue) represent the empirical correlation between the estimated repre- sentations, i.e. و o Yi, where & and r are the estimated canonical vectors. Dashed lines indicate the correct number of active coefficients (10) and true correlation p (0.9). In this example, present the correlation values training samples (blue circles), and on unseen samples (blue aster- isks). Red vertical lines indicate the range X that maximize the correlation on the validation set. Within this range of values the model selects the "correct" set of coefficients.
B.3 NOISY MNIST
In this subsection we provide additional details regarding the noisy MNIST experiment. In Fig. we present the performance as function of the number of active gates (pixels) controlled by X
g
A². The Mutual Information (MI) score, k-means, and linear Support Vector Machine (LSVM) accuracies were computed based on -CCCA embedding with learning rate of 0.01. Furthermore, the number of epochs (~ 4000) was tuned by early stopping using validation set of size 10000.
To learn 10 dimensional correlated embedding, we use the same architecture as suggested by (Wang et al. 2015a) consisting of three hidden layers with 1000 neurons each. The number of dimensions in the embedding was selected based on the number of classes in MNIST. This architecture is used for -CCCA, DCCA (Andrew et al. 2013) and DCCAE (Wang et al. 2015b). Note that for DCCDCC using small values of the regulaniattion parameters X and X", increases the number of selected features and degrades the performance. This is duo to the fact that more features are selected
14

---

Published as a conference paper at ICLR 2022
more noise is introduced into the extracted representation (of size 10). It is interesting to note that the k-means was more robust to the introduced noise than the LSVM.
The regularization parameters X and A balance between the correlation loss and the amount of sparsification performed by the gates. These hyper parameters are tuned using the validation set by
maximizing the total correlation value. We compare g DC-CA to CCA (Chaudhuri et al. 2009
mod-SCCA (Suo et al. 2017). SCCA-HSIC (Uurtio et al. 2018 KCCA (Bach & Jordan 2002
NCCA (Michaeli al. 2016) multiview-ICA (Richard et al. (020), DCCA (Andrew al. 2013
5
and DCCAE (Wang et al. 2015bj. For all methods we use an embedding with dimension 10, and
evaluate performance with k-maans using 20 random initilizations, and using LSVM by performing
training on the training samples and testing on the remaining samples (split defined in the main text).
The hyperparameters of all methods are tuned to maximize the total correlation on a validation set. The solution obtained by mod-SCCA is based on ~ 320 variables. In this experiment we tried to train SCCA-HSIC (Uurtio et al., 2018) 0 for over two days, but did not converge. Furthermore, we believe that the poor performance of the kernel methods stems from the high level of noise in the input images. We note that grad-KCCA (Uurtio et al. 2019) did not converge in this experiment, therefore we did not report its performance on MNIST.
2.100
SVH
0.98
M-Mesns
2.075
2.050
0.90
2.025
Accuracy
Score
0.92
1.950
1.925
0.90
1,900
200
300
400
500
800
700
200
300
400
500
600
700
# of selected features
# of selected features
Figure 8: k-means and SVM classification accuracy (left) and mutual information score (right) vs. the number of selected features.
B.4 SEISMIC EVENT CLASSIHICATION
Using the seismic data, we compare the performance of CD-DCCA with a linear and non-linear activation. In this example. we use learning rate of 0.01 with 2000 epochs. The number of neurons for the five hidden layer are: 300, 200, 100,,0 50, and 40 respectively, with a tanh activation after each layer. The number of dimensions in the embedding (d = 3) was selected based on the number of classes in the data. Parameters are optimized manually to maximize the correlation on validation set. In Fig. 9 we present SVM accuracy for different levels of sparsity. The presented number of features is the average over both modalities, and SVM performance is evaluated using 5-folds cross validation. We compare e -CDCAA to CCA (Chaudhuri et al. 2009). mod-SCCA (Suo ct al. 2017). SCCA-HSIC (Uurtio et al.] 2018)) KCCA (Bach & Jordan| 2002), NCCA (Michaeli al. 2016), multiview-ICA (Richard al. 2020), DCCA (Andrew al.) 2013), and DCCAE
[Wagg et al. 20155). For all methods we use an embedding with dimension d = 3, and evaluate performance with k-means using 20 random initilizations, and using linear SVM by performing 5-folds cross validation. For the kernel methods we evaluated performance by constructing a kernel using 5, 10, ..., 50, nearest neighbors and selected the value which maximized performance
2 https//sw.........oo.oooc.o.occ. htt.://mmmpsiiioooo......... hCm.html
hhtpp://gistggiituu http://shhg.h..oooo////////////a
http:///oonooooccccccccoo////l/d v3.zip
http:///ittt...commaaaaaaa/ll/lp tensorflow
15

---

Published as a conference paper at ICLR 2022
Linear
0.98
Tanh
0.96
Accuracy
0.94
0.92
0.90
0.88
0.86
200
40t
600
800
1000
# of selected features
Figure 9: Classification accuracy on the noisy seismic data. Performance is evaluated using linear SVM in the 3 dimensional embedding. Comparing performance of -D-CCCA for different levels of sparsity, and using linear and nonlinear activation (tanh).
in terms of total correlation. In this example, the optimal solution obtained mod-SCCA Suo et al. (207)) is based on 200 input variables, while the solution obtained SCCA-HSIC Uurtio et al. 2018 is based ~ 77 variables. This means that method uses much fewer input variables and leads to substantially higher clustering and classification accuracies (sce Table 3).
C Low DIMENSIONAL SYNTHETIC EXAMPLE
Following an interesting suggestion raised during the review process, we provide here an evaluation of the method for the regime of D & N. We use a noisy version of the synthetic data model pre- sented in Section 5.1
Specifically, we generate samples from X, Y E DXNN using the following
x
Σ Σ, ),,") where Exg Po ( E With Σ IID-
Ex Eny
distribution
Y
The we use additive noise to create the two modalities, namely X X + Y Y Where 6x and Ey are drawn from ((0, N). We use is beneficial in the regime where there are more samples than variables. In table we present the average errors for different noise levels (values of x)). As evident from this table, the proposed approach can provide more accurate estimate of the sparse canonical vectors in the regime of D < N.
20 and N 4000 to evaluate if our method
D 20, N 4000)
2
CCA
(0.0098,0.008))
(0.034.0.036)
(0.256,0.258)
SCCA-II (Gao et al. 2017 CCA
(0.0013.0.000) (0.0131,0.0097)
(0.6003,0.5881)
(0.0005,0.0004) (0.0065,000044) (0.1075,0.0633)
Table 3: Evaluating the estimation quality the canonical vectors & and n. Each pair indicates (c en), which the estimation emoor of and 7 respectively. We compare the proposed t-- CCA standard CCA and to SCCA-II (Gao et al. 2017). Different columns indicate different additive noise levels.
D CANCER SUB-TYPE CLASSIFICATION
In the main text, we demonstrated that the embedding extracted with fDCDC leads to more accu- rate cancer sub-type classification. Here, we provide additional details on the experiment described in Section 5.5 In this example, neurons for the hidden layers are: 500, 300, 100, with tanh activation after each layer. The number of dimensions in the embedding (d 10) selected based on the number of classes in
learning rate of 0.5 with 2000 epochs. The number of
16

---

Published as a conference paper at ICLR 2022
Algorithm 1 Co-DCA
Input:
Coupled data, (X, } regularization parameters X² A", number of epochs T. Initialize the gate parameters: ju 0.5 for .....,, D and u 0.5 for I,..., D for t = to T do
Sample stochastic gate (STG) vectors z", z² defined based on equation 2² @ Y.
Apply the STG the data X z X and Y
Compute the loss L E p ( ). g
X" ||z llo X llo] where the calculation of
the total correlation م is based on Section 3.4 Where f(x) = f(z o x100) E RRd and g(8) gzz @ 0 E RdxN
Update 0t 0 V 0 L, π π LL π." = TT - VV.LL
8 0 Vo
end for Optional: apply linear CCA on f(X), and g\)) to remove arbitrary scalings.
the data. Parameters are optimized manually to maximize the correlation on a validation set. We
compare -CCCCCA CCA (Chaudhuri al. 2009), mod-SCCA (Suo et al. 2017). SCCA-HSIC (Uurtio et al. 2018). KCCA (Bach & Jordan 2002), NCCA (Michaeli et al. (0166), multiview-ICA
(Richard et al. 2020), DCCA (Andrew et al. 2013 and DCCAE Wang al. 2015b). We the
same SVM and k-means scheme as described in Section B.4 The solution obtained by mod-SCCA
based on 940 variables.
To provide additional intuition on why sparsity is useful in biomedical upplications, we further analyze the results presented in Section [3.5] (the METABRIC dataset). the experiment provided in this section, our method selects 169 genes from the RNA data and extracts an embedding that leads to improved identification (clustering and classification) of cancer subtypes. To understand the variables selected by model are interpretable, we analyze genes selected by the model. We have observed that out the 169 selected genes, six are part of the known risk factors for breast cancer, namely the well-known PAM50 list. The PAM50 contains 50 genes known to be critical variables for classifying breast cancer subtypes (Brnhhard et al. (2016). Our model has identified six genes (in unsupervised fashion) out of the 42 possible PAM50 genes that were initially available in the METABRIC data. To demonstrate that this is a statistically significant finding, we compute the probability of selecting 169 genes out of total of 15019 genes while having overlap of 6 genes with another list of size 42. This probability is 5,39E-6 (the p-value), which indicates that this overlap unlikely to have happened by chance. To conclude, our method was able to identify known markers for breast cancer subtypes classification without using any labels. Notably, one of these genes 'ERBB2', which corresponds to the HER2/neu protein. 'ERBB2' is known to have a large copy number in patients classified to the HER2 cancer subtype. Thus the method found strong association between the RNA and CNA of this gene.
E PSEUDOCODE
In Algorithm
we provide pseudocode description of the proposed approach.
F RUN TIME ANALYSIS
Our method requires computing covariance matrices for kg and ky variables with N samples each iteration. This scales like O(N22 + NE.). To demonstrate the computational efficiency of our method, we run Cc-CAA for different values of and D and evaluate the empirical computational complexity of the method. In Fig. [0 we present the average runtime over 100 runs, the data is generates following Model from Section 5.1
G GENERALIZED 4 -ASSE SPARSE CCA
The proposed 4 CCA algorithm was designed and demonstrated on coupled datasets. However, in many applications, one may have access to multiple (> 2) datasets. There are many real-life
17

---

Published as a conference paper at ICLR 2022
Average run time [sec]
D
D 900
12
9
200
Number of samples N
Figure 10:
Run time evaluation of the proposed approach. We measure the average runtime (over 100 runs) for different values of N and D D
cases where more than two modalities available such as hyper-spectral imaging, medical imag- ing and more. The problem of CCA was generalized decades ago to include multiple data sets by (Horst, 1961 McKeon 1966; McDonald 1968) and (Carroll 1968) to name some. These ap- proaches were later summarized in (Kettering 1971). Since then, many extensions have been pro- posed such as (Van de Geer 1984).. Some of the modern extensions of multi-view CCA comprise its regularised (Tenenhaus & Tenenhaus, 2011), kerneiised (Tenenhaus et al. 2015) sparse (Tenen- haus al.) 2014) and deep (Benton et al. 2019) variants. In this section, we extend the proposed Lo-DCCA to multi-view datasets.
Let number of views and X the corresponding k-th view 1 S S K. The Deep Generalized CCA that was proposed in (Benton et al. 2019) aims to solve
min
E GG-
stt GGT
GER LER
where G is the desired shared representation, and Us is a linear transformation from the k-th deep network output, f(x*), to the shared representation space.
While the proposed method in Section 3.3 describes solution for coupled views, this solution can be naturally extended to include multiple views by introducing a shared common space into the optimization problem in Eq Denoting the random gating vectors z for view X The generalized to GC-DCCA loss is defined by
K G U f(X Aell=*loo s.t. GC =I,
min
(7)
GeRaxd UVeRaxa &
k=1
where 5(X²t) = fz x E d and A is the regularization parameter that control the sparsity k-th selected gate vector z The optimization problem in Eq 7 can be solved using standard optimizers such as gradient descent. The initialization of G and U plus the analysis of above suggested generalized gated canonical correlation are left for further research.
H PROOF OF THEOREM
We first note that the equivalence of the solutions means that if we denote the solution to Eq.
by (ā, b). then there is corresponding set of values π² n. 0", 0", such that (ā, (a. b). where a = Theorem 1 in (Yin et al. 2020 To prove the Theorem, the deterministic sparse CCA problem (in Eq. 2 is a valid solution to the probabilistic sparse CCA problem (in Eq. 3) and vice versa.
x and B
O s", The proof follows the same construction as the proof of
will show that the optimal solution to
18

---

Published as a conference paper at ICLR 2022
First, denote by b) the optimal solution of the problem defined in Eq. 2 then by setting à ā @ s and 3 b @ 8", where म." if à; * and .दे otherwise, we get feasible solution to Eq. which leads to the same objective value.
Now, want show that the optimal solution to Eq. 3 is also feasible solution Eq. Denoting the optimal solution to Eq. 3 by & and B. with á
@ 8 and / = 0" c 8". if p(s TT) (T") point mass densities (i.e., π7" π {0,1)9), then the solution is effectively deterministic and â b= would be a valid solution to problem Eq, 2 If p(s" TT) and p(s πT) are not point mass densities, we will now show by contradiction that all the points in the support of p(s" π²)) (or ))) would lead to the same objective values. Assume without
loss of generality that {*ī ......... { E supplp(s") [+")], if not all values in the support lead to the same objective value, this means that there exist sf 8m such that L(sf) < [.(* where
L(s*) p((@"
S + X5 8t 10 19115"
$" which is the objective from Eq. 3 with optimal values of 0" ," Now if we set म" sf would obtain Eg=~p(s= きh) L(s") K Egr (1 L(s") which contradicts the assumption that #
is optimal. This means that all points in supp[p(s") #²)) lead to the same objective value, therefore,
they are also feasible solutions to Eq. 2 Now, showing that the solution to Eq. feasible
solution to Eq. and vice versa completes our proof.
I LIMITATIONS AND FUTURE DIRECTIONS
The proposed method provides an effective solution to the problems of sparse linear and nonlinear CCA. One advantage of the suggested method compared with existing sparse CCA solutions, is that it can embed data into d dimensional space and learn the sparsity pattern simultaneously. The proposed CC-C problem albeit not being convex leads to an empirically stable solution across different settings. Nonetheless, the method has some limitations, specifically, tuning À requires cross validation procedure, which could be costly in high dimensional regime. Another caveat the existing approach is that it lacks guarantees when trained small batches. In the future, aim extend the method to enable compatibility with small batch training. Furthermore, we are currently working on evaluating how the estimation error scales with respect the dimension, sparsity, for different structures of the covariance matrix. Furthermore, we are working on bounding the estima- tion error of our method. We observe that using a Gaussian based data model (similar the one used in section 5.1 D. it is possible to bound the estimation error (in probability) by D(D/N+ck/NN where is the number of variables, is the number of measurements,
the true sparsity of the conical vectors, and is an independent constant. Although this bound provides some guarantee, it is ubiquitous in the context of the regime studied our manuscript (i.c., N & D). Therefore, we are currently working improving this bound. Empirically, observe that the method works well in regimes that existing & and Eg based approaches fail.
19


Reviews:
[{"review_id": "u8V6Se-OGHx", "reviewer": "Reviewer_N6Rd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Idea of sparse non-linear CCA for N << D case is novel and interesting and is valuable for such real-world cases. Loss function that minimizes correlation and increases sparsity provides a unique way to do feature selection for multi-modal cases.\n\nAlgorithm presented that does l_0 feature selection using Gaussian relaxed Bernoulli variables is interesting and novel application of for sparse-DCCA problem.\n\nEmpirical results on synthetic data show lowest error in learned correlations from sparse-DCCA compared to other SOTA techniques. Results on multi-view, MNIST digits, Seismic event and METABRIC datasets show that technique is able to pick relevant input features in challenging noisy and high dimensional (N<<D) cases with few examples.\n\nA strength of the presented method is that it is able to sub-select relevant features from high dimensional inputs so it is expected to perform well compared to techniques that are not specifically designed for that. It would be helpful to see performance of sparse-DCCA for cases of N>>D compared to other techniques for comparison.", "summary_of_the_paper": "Paper presents a novel sparse nonlinear-CCA method which works especially for cases where number of sample are less than max input feature dimension (D) N << D. It presents a novel algorithm that uses Gaussian relaxation of Bernoulli gating variables that sub-select sparse set of features. Empirical results on synthetic and real world data show efficacy of the method.", "summary_of_the_review": "Novelty of the paper is extension of idea of nonlinear-CCA method to sparse case where where number of sample are less than max input feature dimensions (D) N << D. Results presented are convincing and support technical details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TP3j3r6ls_-", "reviewer": "Reviewer_R5ad", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is technically sound and interesting, in that the authors demonstrate how sparse CCA with l_0 norm can be effectively solved, and how the sparsity constraint can be integrated into the deep CCA model. My concerns lie mainly in the experimental section.\n\nIn the supplementary material, the authors mention that the parameters are optimized to maximize the correlation for the cancer classification task. Why don’t choose parameters based on accuracy instead of correlation? Does high correlation guarantee better accuracy or vice versa? Are the parameters of all compared methods chosen in the same manner?\n\n2. How are lambda_x and lambda_y selected since they directly control the sparsity of variables?\n\n3. The authors claim that the proposed method is efficient in C.2 (Run Time Analysis), it seems that the model scales when k increases. Besides, training a DCCA model for classification seems to be time-consuming. Therefore, it is also important to compare the time complexity and running time (or training time) of compared methods.\n\n4.  In Table 2, why don’t the authors include the results of the proposed sparse CCA method?\n\n5. The datasets included are mainly for vector-based applications. Recently, there are several tensor-based CCA methods, it is interesting to also report results on some high-dimensional datasets, for example, Gait and JAFFE.", "summary_of_the_paper": "The paper proposes a sparse canonical correlation analysis method based on l_0 norm. A continuous relaxation scheme is adopted for solving sparse CCA. The proposed model is then extended to nonlinear function estimation and combined with deep neural networks. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed methods.", "summary_of_the_review": "The paper is technically sound, but the authors should carefully revise their experimental section to address my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B1GusF06rqJ", "reviewer": "Reviewer_LbVS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Comments:\n\nOriginality & Quality: The paper proposes a stochastic gating based approach for l0-CCA which uses ideas from a couple of recent papers on gaussian relaxation of discrete/bernoulli random variables. The resulting algorithm is straightforward and it is also possible to \"backprop into it\", so it is also amenable to non-linear \"deep\" extensions. The results on various synthetic and real-world datasets show the superior performance of the proposed method.\n\nWhile I like the proposed approach, there are a couple of questions/concerns that I have:\n\n1). The paper mentions that standard l0-CCA relies on greedy optimization and that the l1-CCA leads to parameter shrinkage. Both these are valid concerns. Though, it is unclear what recovery guarantees does the stochastic gating approach has? Optimal subset selection is a NP-Hard problem, so clearly the proposed approach is also performing some approximation. Can it be theoretically compared to the greedy l0-solution?\n\n2). The paper shows predictive accuracies of the different CCA approaches on real-world datasets. However, it doesn't show the number of features selected by the different methods and how they compare to each other. If we only cared about predictive accuracy then why select features as shown by (Shalev-Shwartz et al. 2010)? And, also sparsity leads to interpretability so it is unclear how different approaches compare on that front. \n\n\n\nClarity: The paper is well written and puts itself nicely in context of previous work. The overall presentation of the paper is good.\n\nSignificance: The paper addresses an important problem of sparse CCA using a l0 penalty. The proposed approach uses stochastic gating to enforce sparsity which does not rely on a greedy/heuristic solution as previous l0-sparse CCA approaches. ", "summary_of_the_paper": "The paper proposes a new method for l0-CCA using stochastic gating which allows for an efficient algorithm and also permits a deep-version of the l0-CCA. Results are shown on synthetic and real-world datasets that highlight the superior performance of the proposed approach. \n\nMain Contributions: \n\n1). The paper proposes a new approach for l0-CCA that uses a continuous relaxation of a bernoulli random variable via stochastic gating. \n\n2). The paper extends the proposed method to l0-deep-CCA which allows non-linear interactions between the two CCA views.\n\n3). Results are shown on synthetic and real-world datasets which show the superior performance of the proposed l0-CCA methods.", "summary_of_the_review": "An interesting paper that solves an \"old\" problem of sparse-CCA using some recent advances in stochastic gating based sparsity solution. Results are comprehensive, but the focus is less on feature selection and more on predictive power. If predictive power is the primary concern, why care about sparsity?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AOdt88evV-f", "reviewer": "Reviewer_snhT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is very critical to select relevant features from high dimensional multimodal data and such method is very demanding in lots of real cases. \n\nOne key strength of the presented method is that it is able to sub-select relevant features from high dimensional inputs and it showed better performance compared to others for predictive purpose. \n\nThe paper is well written, and the overall presentation of the paper is good.\n\nQuestions:\n\n1. How to select the most relevant features in the multimodal learning field is crucial for model interpretation and downstream analysis and it will have wide applications in biology or human genetics. For those fields, researchers are more interested to select which gene, protein, or biomarkers are most important for disease development. Even though the paper showed it’s outperformed predictive performance, however, it missed the detailed model interpretation part, feature selection. I would be interested to see one biology dataset application for model interpretability on this method. \n\n2. In figure 2, the wide range of lambda showed around 40 coefficients. Please comment on how could the algorithm avoid the wrong number of coefficients?\n\n3. How does the proposed method compare to other methods regarding accuracy vs sparsity?\n\n", "summary_of_the_paper": " I have seen this manuscript before. The main contribution of this paper is to combine two existing works, Gaussian-based continuous relaxation of Bernoulli random variables, gates (Yamada et al., 2020), and sparse CCA (Suo et al., 2017).", "summary_of_the_review": "I like the paper overall. However, I think the model interpretation is more important than the prediction for multimodal learning/CCA, specifically, the key strength of this proposed paper is feature selection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Canonical correlation analysis is a method for studying associations between two sets of variables. However these methods lose their effectiveness when the number of variables is larger than the number of samples. This paper proposes a method, based on stochastic gating, for solving a $\ell_0$-CCA problem where the goal is to learn correlated representations based on sparse subsets of variables. Essentially, this paper combines ideas from Yamada et al. and Suo et al. who introduced Gaussian-based relaxations of Bernoulli random variables, and sparse CCA respectively. They also extend their methods to work with nonlinear functions by integrating deep neural networks into the $\ell_0$-CCA model. They gave experimental results on various synthetic and real examples, including to feature selection on biological data. The author response addressed a number of the reviewers' concerns, including by providing additional experiments and analyzing the genes selected by their model on the METABRIC dataset. Overall this is a solid contribution both from a theoretical and experimental standpoint.
Decision: Accept (Poster)