Deep Gated Canonical Correlation Analysis
Authors: Ofir Lindenbaum
Moshe Salhov
Amir Averbuch
Yuval Kluger
Venue:  (2021)

Abstract: Canonical Correlation Analysis (CCA) models can extract informative correlated representations from multimodal unlabelled data. Despite their success, CCA models may break if the number of variables exceeds the number of samples. We propose Deep Gated-CCA, a method for learning correlated representations based on a sparse subset of variables from two observed modalities. The proposed procedure learns two non-linear transformations and simultaneously gates the input variables to identify a subset of most correlated variables. The non-linear transformations are learned by training two neural networks to maximize a shared correlation loss defined based on their outputs. Gating is obtained by adding an approximate $\ell_0$ regularization term applied to the input variables. This approximation relies on a recently proposed continuous Gaussian based relaxation for Bernoulli variables which act as gates. We demonstrate the efficacy of the method using several synthetic and real examples. Most notably, the method outperforms other linear and non-linear CCA models.

Under review as a conference paper at ICLR 2021
DEEP GATED CANONICAL CORRELATION ANALYSIS
Anonymous authors Paper under double-blind review
ABSTRACT
Canonical Correlation Analysis (CCA) models can extract informative correlated representations from multimodal unlabelled data. Despite their success, CCA models may break if the number of variables exceeds the number of samples. We propose Deep Gated-CCA, method for learning correlated representations based on a sparse subset of variables from two observed modalities. The proposed procedure learns two non-linear transformations and simultaneously gates the input variables to identify a subset of most correlated variables. The non- linear transformations are learned by training two neural networks to maximize a shared correlation loss defined based on their outputs. Gating is obtained by adding an approximate t regularization term applied to the input variables. This approximation relies recently proposed continuous Gaussian based relaxation for Bernoulli variables which act as gates. We demonstrate the efficacy of the method using several synthetic and real examples. Most notably, the method outperforms other linear and non-linear CCA models.
INTRODUCTION
Canonical Correlation Analysis (CCA) (Hoteling 1936 Thompson 2003), is a classic statistical method for finding the maximally correlated linear transtormations of two modalities (or views). Using modalities X E R and Y RP x which are centered and have N samples with D2 and D, features respectively. CCA seeks canonical vectors a; RRxx, and b; RDy, such that u, a? X, and v; = b, 1,.., N, maximize the sample correlations between u, and uú where u, (v:) form an orthonormal basis for i = 1,....d, d, i.e.
ai, b
argmax (u. u.. (
Corr(u;, mz).
While CCA enjoys closed-form solution using generalized eigen pair problem, it is restricted to the linear transformations A [a...... az] and = [b_.... byl.
In order identify non-linear relations between input variables, several extensions of CCA have been proposed. Kernel methods such as Kernel CCA (Bach & Jordan| 2002). Non-paramatric CCA (iichalli et al. 2016) Multi-view Diffusion maps (Lindenbaum al. (020) learn the non-linear relations in reproducing Hilbert spaces. These methods have several shortcomings: they are limited designed kernel, they require (NN2) computations for training, and they have poor interpolation and extrapolation capabilities. To overcome these limitations, Andrew et al. (2013) have proposed Deep CCA, to learn parametric non-linear transformations of the input modalities X and Y. The functions are learned by training two neural networks to maximize the total correlation between their outputs.
Linear and non-linear canonical correlation models have been widely used in the setting of unsupervised or semi-supervised learning. When is set dimension satisfying < Dr. Dy. these models find dimensional reduced representations that may be useful for clustering, classification or manifold learning in many applications. For example, in biology (Pimentel et al.) 2018 neuroscience (Al-Shargie et al. 2017), medicine (Zhang al. 2017). and engineering (Chen) et al. 2017). One key limitations of these models that they typically require more samples than features, i.c. N Dz, Dy. However, if we have more variable than samples, the estimation based on the closed form solution of the CCA problem (in Eq. IT) breaks (Suo et al. 2017). Moreover, in high dimensional data, often some of the variables not informative and thus should be omitted

---

Under review as a conference paper at ICLR 2021
from the transformations. For these reasons, there has been growing interest in studying sparse CCA models.
Sparse CCA (SCCA) (Waaijenborg al. 2008 Hardoon & Shawe-Taylor 2011 Suo et al. 2017) uses an & penalty to encourage sparsity of the canonical vectors a, and b;- This can not only remove the degeneracy inherit to N Dz, Dy, but improve interpetability and performance. One caveat of this approach its high computational complexity, which can reduced by replacing the orthonormality constraints on u, and v; with orthonormality constraints on a, and b;. This procedure known as simplified-SCCA (Parkhomenko et al., 2009) Witten et al. 2009), which enjoys a closed form solution. There has been limited work on extending these models to sparse non- linear CCA. Specifically, there are two kernel based extensions, two-stage kernel CCA (TSKCCA) by Yoshida et al. (2017) and SCCA based on Hilbert-Schmin Independence Criterion (SCCA- HSIC) by Uurtio et al. (2018). However, these models suffer from the same limitations as KCCA and are not scalable to a high dimensional regime.
This paper presents a sparse CCA model that can be optimized using standard deep learning methodologies. The method combines the differentiable loss presented in DCCA (Andrew et al, 2013) with an approximate to regularization term designed to sparsify the input variables of both X and Y. Our regularization relies on recently proposed Gaussian based continuous relaxation of Bernoulli random variables, termed gates (Yamada et al., 2020 The gates are applied to the input features to sparsify X and Y. The gates parameters are trained jointly via stochastic gradient decent to maximize the correlation between the representations of X and Y, while simultaneously selecting only the subsets of the most correlated input features. We apply the proposed method to synthetic data, and demonstrate that our method can improve the estimation of the canonical vectors compared with SCCA models. Then, we use the method to identify informative variable in multi- channel noisy seismic data and show its advantage over other CCA models.
1.1 BACKGROUND
1.2 DEEP CCA
Andrew et al. (2013), present a deep neural network that learns correlated representations. They proposed Deep Canonical Correlation Analysis (DCCA) which extracts two nonlinear transformations of X and Y with maximal correlation. DCCA trains two neural networks with a joint loss aiming to maximize the total correlation of the network's outputs. The parameters of the networks are learned by applying stochastic gradient decent to the following objective:
x 0%
arggmax Corr Cor(((X;xx):g(X;;e
0x, Oy
where 0x and 0y are the trainable parameters, and f(X),g(Y) IRd are the desired correlated representations.
1.3 SPARSE CCA
Several authors have proposed solutions for the problem of recovering sparse canonical vectors. The key advantages of sparse vectors that they enable identifying correlated representations even in the regime of N < Dz. Dy and they allow unsupervised feature selection. Following the formulation by Suo et al. 2017), SCCA could be described using the following regularized objective
a, b = argmin
- Cov (ar(a" T X, X, + +1//a/11 + + b111),
subject to
| X
11,
y 1
where Ti and T2 are regularization parameters for controlling the sparsity of the canonical vectors a and b. Note that the relaxed inequality constrain a T X and b Y makes the problem bi-convex, however, if ||aX|] X|2 < or |b6 X|| < 1, then the covariance in the objective is no longer equal to the correlation.
2

---

Under review as a conference paper at ICLR 2021
1.4 STOCHASTIC GATES
In the last few years, several methods have been proposed for incorporating discrete random variables into gradient based optimization methods. Towards this goal, continuous relaxations of discrete random variables such as (Maddison et al. 2016 Jang et al. 2017) have been proposed.
Such relaxations have been used several applications, for example, model compression (Louizos et al., 2017), feature selection or for defining discrete activations (Jang et al. 2016). We focus on Gaussian-bas relaxation of Bernoulli variables, termed Stochastic Gates (STG) (Yamada al. 2020). which were originally proposed for supervised feature selection. We denote the STG random vector by z E [0, 1]P, where each entry is defined as
z max(0, min eli))),
where ur(i] a trainable parameter for entry the injected noise <[v] is drawn from (00 a²) and is fixed throughout training. This approximation can be viewed as a clipped, mean-shifted, Gaussian we illustrate generation of the transformed random variable z[i] for
random variable. In Fig. [ p|i]] 0.5 which represents a "fair" relaxed Bernoulli variable.
== 1
Ta(E)
f2(z)
Z 0
0.0
0.2
0.4
Z
From left to right, pdf of the Gaussian injected noise the hard Sigmoid function (defined in Eq. g and the pdf of the relaxed Bernoulii variable for 0.5 corresponding "fair" Bernoulli variable. The trainable parameter zu can shift the mass towards or Here, refer to one element of the random vector and omit the index
Figure 1:
2 DEEP GATED CCA
2.1 MODEL
appealing to try to combine ideas from Sparse CCA into the rich differentiable model Deep CCA. However, straight forward & regularization of the input layer of neural network does not work in practice because it makes the learning procedure unstable. This was observed in the supervised setting by Li al. (2016); Feng & Simon (2017). This instability occurs because the objective is not differentiable everywhere. To overcome this limitation, we use the STG random variables (see Eq. 3) by multiplying them with the features of X and Y. Then, by penalizing for active gates using a regularization term EE|zz|o. we can induce sparsity in the input variables.
We formulate the problem of sparse nonlinear CCA by regularizing deep neural network with correlation term. We introduce two random STG vectors into the input layers of two neural networks which are trained in landem to maximize the total correlation. Denoting the random gating vectors zu and z, for view X and Y respectively, the Deep Gated CCA (DG-CCA) loss is defined by
L(O. Ez, ( Corr( 0x), (z 0r)) AmllZe + X,, [zy]lo],
(4)
0y), ). μ = (ux, )y) are the model parameters, and Az. Ay are regularization parameters that control the sparsity of the input variables. The vectors 2, and Zy random STG vectors, with elements defined based on Eq.
where = (0x,0y):
Fig. 2 higghlight the proposed architecture. Each observed modality is first passed through the gates. Then, the outputs of the gates are used as inputs to a view-specific neural sub-net. Finally, the shared loss term in Eq. is minimized by optimizing the parameters of the gates and neural sub-nets.
3

---

Under review as a conference paper at ICLR 2021
Observed View
Observed View
Gates
Neural Subnets
Loss
Figure 2: The proposed architecture. Data from two views is propagated through stochastic gates. The gates output is fed into neural sub-nets that have shared loss. The shared loss is computed both neural sub-nets output representations (with dimension d this example). shared loss combines total correlation term and a differentiable regularization term which induces sparsity in the input variables.
2.2 ALGORITHM DETAILS
We now detail the procedure used in DG-CCA for minimizing the loss L(O, μ1) (in Eq. regularization is based on parametric expectation and therefore, can be expressed by
4 The
D. Σ P(z
Dr Σ
M (-M=lil) V20
Ez 1z/1
2 erf
where erf() is the Gaussian error function, and is defined similarly for Ez, 1/zyllll Denoting the centered output representations of Y,Y by x By E RdxN respectively, the empirical covariance matrix between these representations can be expressed as Σxy N-T 4 M. Using similar notations, we express the regularized empirical covariance matrices of X and Y as Σx - x V + +I and Σy N- My T + II, where the matrix yI (7 0) is added to stabilize the invertability of Σx and Σv. The total correlation in Eq. 4 can be expressed using the trace of Ex 112
E Ex ExyE, Ey 1/2
To learn the parameters of the gates a and of the representations 0 we apply stochastic gradient decent to L(0, 14). Specifically, we used Monte Carlo sampling to estimate the left part of Eq. This is repeated for each batch, using one Monte Carlo sample per batch as suggested by Louizos et al. (2017) and Yamada et al. (2020), and worked well in experiments. After training remove the stochastic part of the gates, and use only variables iz E {1,..., Dx} and i, E {I...., Dy) such that 7zz [ir] 0 and Z [4n] O.
RESULTS
In the following section we detail the evaluation of the proposed approach using synthetic and real datasets. We start with two linear examples, demonstrating the performance of DG-CCA when N & Dx, Dy. Then, we use noisy images from MNIST and seismic data measured using two channels demonstrate that DG-CCA finds meaningful representations of data even in noisy regime. For a full description of the training procedure as well as the baseline methods, we refer the reader the Appendix.
3.1 SYNTHETIC EXAMPLE
We start by deseribing simple linear model also experimented by Suo al. (007).. Consider
x
Σx
Σxy Σy
data generated from the following distribution
N
). where Σx
Σyx
Σy I s00, and Exy -Ex(on=) Ey. Here Φ 500
sparse with 5 nonzero elements

---

Under review as a conference paper at ICLR 2021
Linear model
Method
P
E
Em
PMA
0.71
1.17
1.17
SCCA
0.90
0.060
0.066
mod-SCCA
0.90
0.056
0.062
DG-CCA
0.90
0.027
0.025
Table Evaluating the estimation quality of the canonical vectors &p and vy.
and p 0.9. The indices of the active elements are chosen randomly with values equal to 1/ 1/v5. In this setting, based on Proposition in (Suo al.] 2017), the canonical vectors a and which maximize the objective in Eq. [ are & and respectively.
Using this model we generate 400 samples and estimate the canonical vectors based on CCA and DG-CCA. In Fig. 3 we present a regularization path of the proposed scheme. Specifically, we apply DG-CCA to the data described above using various values of = A = Ag. We present the to of active gates (by expectation) along with the empirical correlation between the extracted representations Exy, which is also wide range of À values such that DG-CCA converges to true number of coefficients (10) and correct correlation value (0.9). Next, of the canonical vector c. and the CCA based estimate of the canonical vector å. The solution by CCA is wrong and not sparse, while the DG-CCA solution correctly identifies the support of Φ. Finally, we evaluate the estimation error of Φ using E 2(1 1Ø² Φ)), and En is defined similarly. In Table we present the estimated correlation along with the estimation errors of sb and (averaged over 100 simulations). As baselines we present the results simulated by [Suo et al. (2017) (mod-SCCA), comparing the performance PMA (Witten al. 2009) and SCCA (Chen et al.
@XYT;"" r As evident from the left panel, there is
present the values of Φ, the DG-CCA estimate (using À 30)
2013
100
0.6
Ez, 2z lo + Ez, 2
0.90
80
0.85
0.4
0.80
50
0.75(
0.2
000000000000
0.70
30
0.65
0.0
0.60
0
-0.2
Index
Figure 3: Left: Regularization path of DG-CCA on data generated from the linear model. The left g-axis (green) represents the sum of active gates (by expectation) after training. The right y-axis represents the empirical correlation between the estimated representations, iec. estimated canonical vectors. Dashed lines indicate the correct number active coefficients (green) and true correlation p (blue). Note that for small values of À = A X, the model select more variables than needed and attains higher correlation value, this is similar over-fitting phenomenon that CCA suffers from. Right: True canonical vector & along with the estimated vectors using DG-CCA (Φ) and CCA (â).
3 Y where & and f are the
3.2 MULTI VIEW SPINNING PUPPETS
As illustrative example we use dataset collected by Lederman & Talmon (2018) for multiview learning. The authors have generated two videos capturing rotations of 3 desk puppets. One camera captures two puppets, while the other captures another two, where cameras. snapshot from both cameras appears in the top row of Fig. 4 All puppets are placed spinning device that rotates the dolls at different frequencies. In both video there is shared
puppet is shared across
5

---

Under review as a conference paper at ICLR 2021
parameter, namely the rotation of the common buildog. Even thought the Buildog is captured from a slightly different angle, we attempt to use CCA to identify a linear transformation that projects the two Bulldogs in to common embedding. We use subset of the spinning puppets dataset, with 400 images from each camera. Each image has 240 x 320 = 76800 pixels (using a gray scaled version of the colored image), therefore, there more feature than samples and direct application of CCA would fail. We apply the proposed scheme using X = Ag = 50, linear activation and embedding dimension d 2. DG-CCA converges to embedding with correlation of 1.99 using 372 and 403 pixels from views X and Y. The active gates are presented in the bottom row of Fig. 4
In Fig. 5 we present the coupled two dimensional embedding of both videos. Both embeddings are highly correlated with the angular orientation of the Bulldog. Note that adjacent images in the embedding are not necessarily adjacent in the original ambient space, this is because the Bunny and the Yoda puppets are gated and do not affect the embedding.
Figure 4: Top: two samples from the spinning puppets videos. Arrows indicate the spinning direction of cach puppet. Bottom: the converged active gates for each video. There are 372 and 403 active gates for the left and right videos respectively.
3.3 NOISY MNIST
MNIST (LeCun et al. 2010 which consists of 28 28 grayscale digit images, with 60K/10K images for training/testing, is well known and widely investigated dataset. We use a challenging variant of MNIST with coupled noisy views. The first view is created by adding noise drawn uniformly from [0, to all pixels. The second view is created by placing a random patch from natural image in the background of the handwritten digits. Both views are based on different digit instances drawn from the same digit class. Random samples from both views are presented in Fig. 6 Both views consists of 62, 000 samples, which we use 40, 000 for training 12, 000 for testing and 10, 000 are used validation set. Here the validation set is used for early stopping of the training procedure and optimizing = A Ay-
Multi-view processing of the two noisy views can generate informative representation of the noisy MNIST data. In the following we will focus on performing unsupervised embedding of each noisy MNIST view into correlated 10 dimensional space.
By reducing the correlation cost, the DG-CCA leamss which per-view pixels are relevant and informative in correlation maximization sense. In the bottom right corner of Fig 6 we present the location of the active gates. DG-CCA selects features (pixels) within an oval-like shape in the center of each view thus capturing the digit information and reducing the influence of the noise.

---

Under review as a conference paper at ICLR 2021
Ses gren
Figure 5: The two DG-CCA embedding of the Yoda-Bulldog video (left) and Bulldog+ Bunny (right). We overlay each embedding with images corresponding points the embedding spaces. The resulting embeddings are correlated with the angular rotation Bulldog, which is the common rotating puppet this experiment.
7
8
9
Figure 6: Images from noisy MNIST (left) and corresponding images from background MNIST (right). In the bottom right of both figures, presents the active gales (white values within green frame). There are 277 and 258 active gates for view and II respectively.
To measure the class separation in the learned embedding, we apply k-meaas clustering to the stacked embedding of the two views. We run k-means (with A: = 10) using 20 random initializations and record the run with the smallest sum of square distances from centroids. Given the cluster assignment, k-means clustering accuracy (KM ACC) and mutual information (MI) are measured using the true labels. Additionally, we train a Linear-SVM (LSVM) model on our train and validation datasets. LSVM classification accuracy (LSVM ACC) is measured the remaining test set. The performance of DG-CCA compared with several baselines appears in Table g In the appendix provide all implementation details and provide experiment demonstrating the performance for various values X = X₂ Agy
3.4 SEISMIC EVENT CLASSIFICATION
Next, we evaluate the method using a dataset studied by Lindenbaum al. (2018). The data consists of 1609 seismic events. Here, we focus 537 explosions which are categorized into 3 quarries. The events occurred between the years 2004 2015, in the southern region of Israel and Jordan. Each event is recorded using two directional channels facing east (E) and north (N), these comprise the coupled views for the correlation analysis. Following the analysis by Lindenbaum et al. (2018), the input features are Sonogram representations of the seismic signal. Sonograms time frequency

---

Under review as a conference paper at ICLR 2021
Noisy MNIST
Seismie
Method
MI
KM ACC (%)
LSVM (%)
MI
KM ACC (%)
LSVM (%)
Raw Data
0.130
16.6
86.6
0.001
35.5
77.7
CCA
1.290
66,4
75.8
0,003
38.1
40.4
SCCA
0.342
23.9
63.1
0.610
71.7
86.9
SCCA-HSIC
NA
NA
NA
0.003
38.7
49.5
KCCA
0,943
50.2
85.3
0,006
38.4
92.5
NCCA
1.030
47.5
77.2
0.700
86.8
91.4
DCCA
1.970
93.2
93.2
0.830
94.9
94.6
DG-CCA
2.05
95.4
95.5
0.97
98.1
97.2
Table 2: Performance evaluation on the Noisy MNIST and seismic datasets.
representations with bins equally tempered on logarithmic scale. Each Sonogram & E 12157 with 89 time bins and 13 frequency bins. An example of Sonograms from both channels appears in the top row Fig. 7
We create the noisy seismic data by adding sonograms computed based on vehicle noise from I Examples noisy sonograms appear in the middle row of Fig We omit 20% of the data as a validation set. Then train DG-CCA embed the data in dimensions using several values for
- Xy. Then, we use the model that attains maximal correlation on the validation set. In T present the MI, k-means and SVM accuracies computed based DG-CCA embedding. Furthermore, we compare the performance with several other baselines. Here, the proposed scheme improves performance in all 3 metrics while identifying subset of 71 and 68 features from channel E and N respectively. The active gates are presented in the bottom row of Fig. 7
Frequency Freguency Frequency
Frequency
Time
Time
Frequency Frequency
20
40
Time
Time
Time
Figure 7: Top: Clean sample Sonograms an explosion based on the and N channels (left right respectively). Arrows highlight the Primary (P) and Secondary (S) waves caused by the explosion. Middle: Noisy sonograms generated by adding sonograms of vehicle recordings. Bottom: the active gates for both channels. Note that the gates active time frequency bins which correspond to the P and S waves (scc top left figure).
CONCLUSION
In this paper we present a method for learning sparse non-linear transformations which maximize the canonical correlations between modalities. Our method is realized by gating the input layers of two neural networks which are trained to maximize their output's 's total correlations. Input variables are gated using regularization term which encourages sparsity. This allows us learn informative representations even when the number of variables far exceeds the number of samples. Finally, we demonstrate that the method outperforms existing methods for linear non-linen linerr canonical correlation analysis.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Fares Al-Shargie, Tong Boon Tang, and Masashi Kiguchi. Assessment of mental stress effects on prefrontal cortical activities using canonical correlation analysis: an fnirs-ceg study. Biomedical optics express, 8(5):2583-5588, 2017.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International Conference on Machine Learning, pp. 1247-1255, 2013.
Francis R Bach and Michael Jordan. Kernel independent component analysis. Journal of machine learning research, 3(Jul):1444) 2002.
Kamalika Chaudhuri, Sham M Kakade, Karen Livescu, and Karthik Sridharan. Multi-view
clustering via canonical correlation analysis. In Proceedings of the 26th anmual international conference on machine learning, pp. 129-136. ACM, 2009.
Mengjie Chen, Chao Gao, Zhao Ren, and Harrison H Zhou. Sparse cca via precision adjusted iterative thresholding. arXiv preprint 1rXiv:1311.61 2013.
Zhiwen Chen, Steven X Ding, Tao Peng. Chunhua Yang, and Weihua Gui. Fault detection for non- gaussian processes using generalized canonical correlation analysis and randomized algorithms. IEEE Tramsactions Industrial Electronics, 65(2):1559-155 2017.
Jean Feng and Noah Simon. Sparse-inpe input neural networks for digendimensional nonparametric regression and classification. arXiv preprint arXiv.1711.07592, 2017.
David R Hardoon and John Shawe-Taylor. Sparse canonical correlation analysis. Machine Learning, 83(3):331-33,, 2011.
Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3/4):32--37,, 1936.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint arXiv:161 1.0/ 144, 2016
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. 2017.
URL httpt:ttt///xxtttt org/ abs/16ll..11........... 01144
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. 2010.
Available: http://yamm.ecm..coo//edx//////
Roy R Lederman and Ronen Talmon. Learning the geometry of common latent variables using alternating-difftsion.. Applied and Computational Harmonic Analysis, 44(3):509-536, 2018.
Yifeng Li, Chih- Yu Chen, and Wyeth W Wasserman. Deep feature selection: theory and application to identify enhancers and promoters. Journal of Computational Biology, 23(5):32-333 -366 2016.
Ofir Lindenbaum, Yuri Bregman, Neta Rabin, and Amir Averbuch. Multiview kernels for low- dimensional modeling of seismic events. IEEE Transactions Geoscience and Remote Sensing, 56(6):3300-333 -310, 2018
Ofir Lindenbaum, Arie Yeredor, Moshe Salhov, and Amir Averbuch. Multi-view diffusion maps. Information Fusion, 55:127-149, 2020.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint arXiv:1712.01312 2017.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint /ttp:////11.0//2// 2016.
Tomer Michaeli, Weiran Wang, and Karen Livescu. Nonparametric canonical correlation analysis.
In International Conference on Machine Learning, pp. 1967-1976. 2016.
Elena Parkhomenko, David Tritchler, and Joseph Beyene. Sparse canonical correlation analysis with application to genomic data integration. Statistical applications in genetics and molecular biology, 8(1), 2009.

---

Under review as a conference paper at ICLR 2021
Harold Pimentel, Zhiyue Hu, and Haiyan Huang. analysis. Quantitative Biology, 6(1):56-63 2018.
Biclustering by sparse canonical correlation
Xiaotong Suo, Victor Minden, Bradley Nelson, Robert Tibshirani, and Michael Saunders. Sparse canonical correlation analysis. arXiv preprint 1ttt::1705.00666 2017.
Bruce Thompson. Canonical correlation analysis. Encyclopedia of statistics in behavioral science, 2005.
Viivi Uurtio, Sahely Bhadra, and Juho Rousu. Sparse non-linear сca through hilbert-schmidt independence criterion. In 2018 IEEE International Conference on Data Mining (ICDM), pp- 1273-1283. IEEE, 2018.
Sandra Waaijenborg, Philip C Verselewel de Witt Hamer, and Aeilko H Zwinderman. Quantifying the association between gene expressions and dna-markers by penalized canonical correlation analysis. Statistical applications in genetics and molecular biology, 7(1), 2008
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi- view representation learning, In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, pp. 1083-1092. JMI.Rorgg 2015.
Daniela M Witten, Robert Tibshirani, and Trevor Hastie. A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics, 10 (3):515-534, 2009.
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In Proceedings of Machine Learning and Systems 2020, pp. 8952-8963. 2020.
Kosuke Yoshida, Junichiro Yoshimoto, and Kenji Doya. Sparse kernel canonical correlation analysis for discovery of nonlinear interactions in high-dimensional data. BMC bioinformatics, 18(1):1- 11, 2017.
Yu Zhang, Guoxu Zhou, Jing Jin, Yangsong Zhang, Xingyu Wang, and Andrzej Cichocki. Sparse bayesian multiway canonical correlation analysis for eeg pattern recognition. Neurocomputing, 225:103-110, 2017.
A APPENDIX
A.1 GATES INITIALIZATION
The Gaussian based stochastic gates suggested by Yamada et al. (2020) are based on trainable parameters μ and constant parameter These control the mean and standard deviation of the injected noise respectively. Yamada et al. (2020) have initialize all values of şe to 0.5, in this case, the gates approximate "fair" Bernoulli parameters. This is reasonable choice, if no prior knowledge about the solution is available, however, we can utilize the closed form solution of the CCA problem to derive smarter initialization procedure for the parameters of the gates. Specifically, given the
empirical covariance matrix Cxy XYT (N-1)
we denote the thresholded covariance matrix by SxY,
with values defined as follows
(SxY) (Cxy))ss if (CCy)) > (0,
otherwise.
Where 5 a threshold value selected based on the estimated sparsity of and Y. Specifically, assume that percent of the values should be zeroed, then d is set to be the r-th percentile of [Cxx)] Then we compute the leading singular vectors u and a of Sxy. We further threshold the absolute values of these vectors (using the same percentile used for Sxy. The initial values of the parameters of the gates are then defined by μx = ū + 0.5, and by = + 0.5, where i and are the thresholded versions of the absolute value of the singular vectors.
The standard deviation of the injected noise a was set to 0.5 by (Yamada et al., 2020). They have selected this value it maximized the gradient of the regularization term at initialization.
10

---

Under review as a conference paper at ICLR 2021
Empirically, we have observed that for DG-CCA smaller values of translate to improved convergence. Specifically, we have used o = 0.25 which worked well in our experiments. Studying the effect of is an open question that aim to pursue in future study.
A.2 ADDITIONAL EXPERIMENTAL DETAILS
In the following sections we provide additional experimental details required for reproduction of the experiments provided in the main text.
A.2.1 SYNTHETIC EXAMPLE
For the linear model
learning rate of 0.005 with 10, 000 epochs. The values of X and Xy are both set to 30. These values were obtained using a cross validation procedure. We run the method 100 times with different realizations of X and Y. Importantly following Suo et al. (2017) we present the average errors for the estimation of the canonical vectors, however the median values are one order of magnitude better, specifically Exp 0.0017 and En 0.0020.
A.2.2 NOISY MNIST
In this subsection we provide additional details regarding the noisy MNIST experiment. In Fig we present the performance as function of the number of active gates (pixels) controlled by X Ag- The MI score, k-means and SVM accuracy were computed based on DG-CCA embedding with learning rate of 0.01. Furthermore, the number of epochs (~ 4000) was tuned by early stopping using random validation of size 10000. To learn 10 dimensional correlated embedding, we use the same architecture as suggested by (Wang et al., 2013) consisting of three hidden layers with 1000 neurons each. The number of dimensions in the embedding classes in MNIST. This architecture is used for both DCCA and DG-CCA. Note that for DG-CCA using small values of the regularization parameters X and Ay. increases the number of selected features and the degrades performance. This is duo to the fact that as more features are selected more noise is introduced into the extracted representation (of size 10).It is interesting note that the k-means was more robust to the introduced noise than the LSVM.
selected based on the number of
The regularization parameters A and Xy balances between the correlation loss and the amount of sparsification performed by the gates. These hyper parameters are tuned using the validation set in by maximizing the total correlation value. We compare DG-CCA to CCA (Chaudhuri et 2009 KCCA (Bach & Jordan 2002 NCCA (Michaeli et al. 2016 and DCCA (Andrew et al. 2013) For all methods we use an embedding with dimension 10, and evaluate performance with m-mass using 20 random initilizations, and using LSVM by performing training on the training samples and testing on the remaining samples (split defined in the main text). In this experiment we tried to train
SCCA-HSIC (Uurtio et al.) 20118) for over two days, but it did not converge. Furthermore, we believe that the poor performance of the kernel methods are degraded due to the high level of noise in the input images.
A.2.3 SEISMIC EVENT CLASSIFICATION
Using the seismic data, we compare the performance of DG-CCA with linear and non-linear activation. In this exaple, we use learning rate of 0.01 with 2000 epochs. The number of neurons for each hidden layer are: 300, 200, 100, 50, 40, with Tanh activation. he number of dimensions in the embedding (d = 3) was selected based on the number classes in the data. Parameters are optimized manually to maximize the correlation
validation set. In Fig. g we present SVM accuracy for different levels of sparsity. The presented number of features is the average over both modalities, and SVM performance is evaluated using 5-folds cross validation. We compare DG- CCA to CCA (Chaudhuri et al.) 2009), SCCA (Suo et al. 2017). SCCA-HSIC (Uurtio et al. 2018), KCCA (Bach & Jordan 2002). NCCA (Michaeli et al. 2016) and DCCA (Andrew at 2013) 3). For
http:///aiiaiillacc/////y/////ss decprooosssoooooiiiiiiiooooo Catpmmposstion.CC..hii hCm.

---

Under review as a conference paper at ICLR 2021
2.100
SVM
2,075
20000
0.96
Accuracy
Score
1.973
1.950
1.925
0.90
1,900
200
300
400
500
900
700
selected features
#
of selected features
Figure 8: k-means and SVM classification accuracy (left) and mutual information score (right) vs. the number of selected features.
all methods we use an embedding with dimension 3, and evaluate performance with k-menss using 20 random initilizations, and using linear SVM by performing 5-folds cross validation. For the kernel methods we evaluated performance by constructing kernel using k = 5, 10, 50, nearest neighbors and selected the value which maximized performance in terms of total correlation.
Linear
0.90
Tanh
0.96
Accuracy
0.94
0.92
0.90
0.88
0.86
200
400
600
800
1000
# of selected features
Figure 9: Classification accuracy the noisy seismic data. Performance is evaluated using linear SVM in the dimensional embedding. Comparing performance DG-CCA for different levels sparsity. and using linear and nonlinear activation (Tanh).
12


Reviews:
[{"review_id": "rWE5HoiDBew", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines an approximate $L_0$ regularization on the canonical vectors with CCA to encourage the CCA for getting sparse vectors. In addition, the CCA is computed on embeddings from a neural network, which make it possible to capture non-linear correlations.\n\nOverall, the paper is well written and easy to follow. The paper seems to be a combination of deep CCA (Andrew et al. 2013) and Louizos et al. 2017. In particular, the $L_0$ regularization approximation is very similar to that proposed in Louizos et al. 2017. It would be great if the authors could be more clear on illustrating the differences (if any). Therefore, the novelty of this paper is unclear. \n\nThe experiments could be improved. Since most of the experiments were carried out on relatively small datasets with reasonable sized model, it would be great to have multiple runs that illustrate the stability/variance of the method. In addition, the major benefit of using neural networks as embedding function is the ability to capture non-linear relationships. It would be great to add a synthetic example to illustrate this benefit. The authors mentioned the use of early stopping and hyper-parameter selection, however, it is not clear based what criteria those actions were carried out. My guess is that it is based on the objective in eq. 4 on the validation set. It would be great if the authors could make this clear, because from the synthetic experiments, $\\lambda$ plays a quite important role for the final performance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YEyAeZBsEtu", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Paper summary:\n\nThis paper proposes a DL method for learning sparse non-linear transformations that maximize correlations between two views. In particular, each view is passed through a separate network. Stochastic Gating is applied to the input layer of each network. The two networks are jointly trained by maximising the correlation between their outputs. Sparsity is obtained by imposing L0 regularization terms on the Stochastic Gating variables.\n\n2. Strong points of the paper:\n\nStochastic Gating gives way to an objective function that can be optimized through Stochastic Gradient Descent.\n\nThe method can detect correlation between two views even when data size is less than the number of dimensions, as demonstrated by the experimental results.\n\n3. Weak points of the paper:\n\nThe proposed method is very similar to DCCA paper of Andrew et al. The only difference is Andrew et al. use L2 regularization, while the authors use L0 regularization.\n\nSimilarly to DCCA, the method suffers from the two issues.\n\nFirst, the method learns non-linear transformations that however are hard to interpret. Non-linear CCA can be achieved by learning linear transformations through non-linear correlation measure, such as HSIC. HSIC-CCA [1] can also learn sparse representations. Given the rising importance of explainable AI, non-linear transformations seem to be a drawback.\n\nSecond, the method relies on Stochastic Gradient Descent. However, the loss function is not decomposable into batches. This makes batch training somewhat random.\n\n4. Conclusion:\n\nFor the above reasons, I find the contributions of the paper to be marginal.\n\n[1] Billy Chang, Uwe Krüger, Rafal Kustra, Junping Zhang: Canonical Correlation Analysis based on Hilbert-Schmidt Independence Criterion and Centered Kernel Target Alignment. ICML (2) 2013: 316-324", "extra_scores": {}, "extra_text": {}}, {"review_id": "KFwh2klSf6J", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose a new non-linear CCA variant that learns mappings that are sparse with respect to the input variables, using approximate $l_0$ regularisation, to improve performance for applications with large number of features but few samples.\n\nReasons for score: I am leaning towards rejection due to the straightforward nature of the work. The method combines two existing techniques in fairly obvious way and despite good empirical comparisons has also issues in evaluation since more recent comparison methods are missing.\n\nDetailed feedback: The related work and importance of the application are well covered, and the technical solution is sound. The conceptual novelty is, however, fairly limited; several sparse CCA variants have been proposed in the past and switching to proper sparsity ($l_0$ vs more common $l_1$) is a natural thing to do. Furthermore, in recent years technical solutions building on stronger sparsity priors have been proposed for closely related models (e.g. Boyveyron et al. \"Bayesian variable selection of globally sparse probabilistic PCA\" (2018) discusses this in detail for PCA, and many algorithmic details for PCA generalise easily for CCA by interpreting CCA as group-sparse PCA).\n\nThe specific technical solution presented here appears to be new, but builds directly on existing and relatively obvious choices: The loss matches Andrew et al. (2013) and the $l_0$ approximation is from Yamada et al. (2020). Even though the specific formulation in the latter is recent, the underlying auxiliary variable construct has been used for similar purposes before. The simplicity of the technical approach is highlighted by the fact that the whole model description takes only slightly more than one page of the paper. In summary, the paper does not make fundamental conceptual or technical contributions. It certainly has potential for being a useful practical tool for the task, but the required scientific insight is limited.\n\nThe empirical demonstrations are nice and illustrative, but carried out on somewhat simplified benchmark data. They do show that the method works well in comparison against reasonably chosen competing methods, but do not clearly indicate qualitative change in CCA applications. The advantage over ordinary DCCA, published already 7 years ago, is not particularly striking in Table 2, and in recent years quite a few deep CCA variants have been proposed but are not compared against (or cited). Consequently, we cannot really evaluate whether this advances the field in practice; there is potential, but as it is the empirical comparisons do not seem sufficient to overcome the lack of technical and conceptual contribution.", "extra_scores": {}, "extra_text": {}}, {"review_id": "InP7y7fCC8W", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new deep CCA method to learn non-linear relationships between two modalities. It trains two neural networks each for a modality to maximize the total correlations of their output representations. Gating is applied to input variables by associating each with a latent Bernoulli variables which is then relaxed with the clipped Gaussian random variable. Experiments on one synthetic and two real datasets demonstrate the superiority of the proposed method.\n\nBelow are specific comments.\n\n1. In the last sentence of Section 2.2, it is unclear to me how the stochastic part of the gates is removed to determine whether $z_x[i_x]$ is equal to or larger than zero. Is $z_x[i_x]$ determined based on the estimated $\\mu_x[i_x]$: $z_x[i_x] > 0$ if $\\mu_x[i_x] > 0$ and $z_x[i_x] = 0$ otherwise?\n\n2. Minor comments (notation inconsistencies/abuse, typos, etc.):\n\nThe sentence \"For example, in biology ... and engineering (Chen et al., 2017)\" is not complete (sentence fragment). Please rephrase it or join it to the preceding sentence.\nIs \"the degeneracy inherit to $N < D_x,D_y$\" supposed to be \"the degeneracy inherent to $N < D_x,D_y$\"?\nThe word \"interpetability\" is misspelled.\nIn the middle subfigure of Figure 1, it is clearer if the label $\\epsilon$ and tick values $\\{-0.5,0,0.5\\}$ are added along the horizontal axis.\n\"straight forward\" should be spelled as \"straightforward\" (no space).\n\nEq. (4): It seems $\\boldsymbol{z}_x^T \\boldsymbol{X}, \\boldsymbol{z}_y^T \\boldsymbol{Y}$ should be written as $\\mathop{{\\rm diag}}\\left(\\boldsymbol{z}_x\\right) \\boldsymbol{X}, \\mathop{{\\rm diag}}\\left(\\boldsymbol{z}_y\\right) \\boldsymbol{Y}$. Note that the $\\boldsymbol{X}, \\boldsymbol{Y}$ here represent the observed data matrices of dimensions $D_x \\times N, D_y \\times N$ [rather than random vectors based on which the data are observed].\n\nIn Section 2.2, third line, the expression of the regularization:\n- $\\mathbb{P}(\\boldsymbol{z}_x[i] \\geq 0)$ should be $\\mathbb{P}(\\boldsymbol{z}_x[i] > 0)$ or $\\mathbb{P}(0 < \\boldsymbol{z}_x[i] \\leq 1)$;\n- For consistency, it should write $\\|\\boldsymbol{z}\\|_0$ as $\\|\\boldsymbol{z}_x\\|_0$ and the index $i$ as $i_x$.\n\n\"a similar notations\" should be \"similar notations\".\n\nIn Section 2.2, first paragraph, last sentence \"The total correlation in Eq. 4 can be expressed using the trace of ...\"\n- \"total correlation\" should be \"total squared correlation\".\n\nIn Section 3.1, second paragraph, $\\hat{\\rho}=\\bm{\\hat{\\phi}}\\boldsymbol{X}\\boldsymbol{Y}^T\\bm{\\hat{\\eta}}^T$ should be $\\hat{\\rho}=\\bm{\\hat{\\phi}}^T\\boldsymbol{X}\\boldsymbol{Y}^T\\bm{\\hat{\\eta}}$.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to sparse CCA with deep neural nets, performing simultaneous feature selection with stochastic gating and canonical correlation maximization.  The reviewers think that there is merit in defining an objective function that optimizes the goals jointly throughout the networks. However, the paper has not clearly presented the novelty in methodology. In particular, the reviewers agree that the paper needs to clearly distinguish itself from the two building blocks (Andrew et al. 2013 and Louizos et al. 2017), and demonstrate the significance of combining the two techniques theoretically and/or experimentally. Also, there is a large literature in sparsifying classical method. Sufficient discussions and comparisons with prior work can better position the current work in the literature.
Decision: Reject