Deep Gated Canonical Correlation Analysis

1 INTRODUCTION . Canonical Correlation Analysis ( CCA ) ( Hotelling , 1936 ; Thompson , 2005 ) , is a classic statistical method for finding the maximally correlated linear transformations of two modalities ( or views ) . Using modalities X ∈ RDx×N and Y ∈ RDY ×N , which are centered and have N samples with Dx and Dy features respectively . CCA seeks canonical vectors ai ∈ RDX , and bi ∈ RDY , such that , ui = aTi X , and vi = b T i Y , i = 1 , ... , N , maximize the sample correlations between ui and vi , where ui ( vi ) form an orthonormal basis for i = 1 , ... , d , i.e . ai , bi = argmax 〈ui , uj〉=δi , j , 〈vi , vj〉=δi , j , i , j=1 , ... , d Corr ( ui , vi ) . ( 1 ) While CCA enjoys a closed-form solution using a generalized eigen pair problem , it is restricted to the linear transformationsA = [ a1 , ... , ad ] andB = [ b1 , ... , bd ] . In order to identify non-linear relations between input variables , several extensions of CCA have been proposed . Kernel methods such as Kernel CCA ( Bach & Jordan , 2002 ) , Non-paramatric CCA ( Michaeli et al. , 2016 ) or Multi-view Diffusion maps ( Lindenbaum et al. , 2020 ) learn the non-linear relations in reproducing Hilbert spaces . These methods have several shortcomings : they are limited to a designed kernel , they requireO ( N2 ) computations for training , and they have poor interpolation and extrapolation capabilities . To overcome these limitations , Andrew et al . ( 2013 ) have proposed Deep CCA , to learn parametric non-linear transformations of the input modalities X and Y . The functions are learned by training two neural networks to maximize the total correlation between their outputs . Linear and non-linear canonical correlation models have been widely used in the setting of unsupervised or semi-supervised learning . When d is set to a dimension satisfying d < Dx , Dy , these models find dimensional reduced representations that may be useful for clustering , classification or manifold learning in many applications . For example , in biology ( Pimentel et al. , 2018 ) , neuroscience ( Al-Shargie et al. , 2017 ) , medicine ( Zhang et al. , 2017 ) , and engineering ( Chen et al. , 2017 ) . One key limitations of these models is that they typically require more samples than features , i.e . N > Dx , Dy . However , if we have more variable than samples , the estimation based on the closed form solution of the CCA problem ( in Eq . 1 ) breaks ( Suo et al. , 2017 ) . Moreover , in high dimensional data , often some of the variables are not informative and thus should be omitted from the transformations . For these reasons , there has been a growing interest in studying sparse CCA models . Sparse CCA ( SCCA ) ( Waaijenborg et al. , 2008 ; Hardoon & Shawe-Taylor , 2011 ; Suo et al. , 2017 ) uses an ` 1 penalty to encourage sparsity of the canonical vectors ai and bi . This can not only remove the degeneracy inherit to N < Dx , Dy , but can improve interpetability and performance . One caveat of this approach is its high computational complexity , which can be reduced by replacing the orthonormality constraints on ui and vi with orthonormality constraints on ai and bi . This procedure is known as simplified-SCCA ( Parkhomenko et al. , 2009 ; Witten et al. , 2009 ) , which enjoys a closed form solution . There has been limited work on extending these models to sparse nonlinear CCA . Specifically , there are two kernel based extensions , two-stage kernel CCA ( TSKCCA ) by Yoshida et al . ( 2017 ) and SCCA based on Hilbert-Schmidt Independence Criterion ( SCCAHSIC ) by Uurtio et al . ( 2018 ) . However , these models suffer from the same limitations as KCCA and are not scalable to a high dimensional regime . This paper presents a sparse CCA model that can be optimized using standard deep learning methodologies . The method combines the differentiable loss presented in DCCA ( Andrew et al. , 2013 ) with an approximate ` 0 regularization term designed to sparsify the input variables of both X and Y . Our regularization relies on a recently proposed Gaussian based continuous relaxation of Bernoulli random variables , termed gates ( Yamada et al. , 2020 ) . The gates are applied to the input features to sparsify X and Y . The gates parameters are trained jointly via stochastic gradient decent to maximize the correlation between the representations of X and Y , while simultaneously selecting only the subsets of the most correlated input features . We apply the proposed method to synthetic data , and demonstrate that our method can improve the estimation of the canonical vectors compared with SCCA models . Then , we use the method to identify informative variable in multichannel noisy seismic data and show its advantage over other CCA models . 1.1 BACKGROUND . 1.2 DEEP CCA . Andrew et al . ( 2013 ) , present a deep neural network that learns correlated representations . They proposed Deep Canonical Correlation Analysis ( DCCA ) which extracts two nonlinear transformations of X and Y with maximal correlation . DCCA trains two neural networks with a joint loss aiming to maximize the total correlation of the network ’ s outputs . The parameters of the networks are learned by applying stochastic gradient decent to the following objective : θ∗X , θ ∗ Y = argmax θX , θY Corr ( f ( X ; θX ) , g ( Y ; θY ) ) , ( 2 ) where θX and θY are the trainable parameters , and f ( X ) , g ( Y ) ∈ Rd are the desired correlated representations . 1.3 SPARSE CCA . Several authors have proposed solutions for the problem of recovering sparse canonical vectors . The key advantages of sparse vectors are that they enable identifying correlated representations even in the regime of N < Dx , Dy and they allow unsupervised feature selection . Following the formulation by Suo et al . ( 2017 ) , SCCA could be described using the following regularized objective a , b = argmin [ − Cov ( aTX , bTY ) + τ1‖a‖1 + τ2‖b‖1 ] , subject to ‖aTX‖2 ≤ 1 , ‖bTY ‖2 ≤ 1 , where τ1 and τ2 are regularization parameters for controlling the sparsity of the canonical vectors a and b . Note that the relaxed inequality constrain on aTX and bTY makes the problem bi-convex , however , if ‖aTX‖2 < 1 or ‖bTX‖2 < 1 , then the covariance in the objective is no longer equal to the correlation . 1.4 STOCHASTIC GATES . In the last few years , several methods have been proposed for incorporating discrete random variables into gradient based optimization methods . Towards this goal , continuous relaxations of discrete random variables such as ( Maddison et al. , 2016 ; Jang et al. , 2017 ) have been proposed . Such relaxations have been used in several applications , for example , model compression ( Louizos et al. , 2017 ) , feature selection or for defining discrete activations ( Jang et al. , 2016 ) . We focus on a Gaussian-based relaxation of Bernoulli variables , termed Stochastic Gates ( STG ) ( Yamada et al. , 2020 ) , which were originally proposed for supervised feature selection . We denote the STG random vector by z ∈ [ 0 , 1 ] D , where each entry is defined as z [ i ] = max ( 0 , min ( 1 , µ [ i ] + [ i ] ) ) , ( 3 ) where µ [ i ] is a trainable parameter for entry i , the injected noise [ i ] is drawn fromN ( 0 , σ2 ) and σ is fixed throughout training . This approximation can be viewed as a clipped , mean-shifted , Gaussian random variable . In Fig . 1 we illustrate generation of the transformed random variable z [ i ] for µ [ i ] = 0.5 which represents a “ fair ” relaxed Bernoulli variable . 2 DEEP GATED CCA . 2.1 MODEL . It is appealing to try to combine ideas from Sparse CCA into the rich differentiable model of Deep CCA . However , a straight forward ` 1 regularization of the input layer of a neural network does not work in practice because it makes the learning procedure unstable . This was observed in the supervised setting by Li et al . ( 2016 ) ; Feng & Simon ( 2017 ) . This instability occurs because the objective is not differentiable everywhere . To overcome this limitation , we use the STG random variables ( see Eq . 3 ) by multiplying them with the features of X and Y . Then , by penalizing for active gates using a regularization term E‖z‖0 , we can induce sparsity in the input variables . We formulate the problem of sparse nonlinear CCA by regularizing a deep neural network with a correlation term . We introduce two random STG vectors into the input layers of two neural networks which are trained in tandem to maximize the total correlation . Denoting the random gating vectors zx and zy for viewX and Y respectively , the Deep Gated CCA ( DG-CCA ) loss is defined by L ( θ , µ ) = Ezx , zy [ − Corr ( f ( zTxX ; θX ) , g ( zTy Y ; θY ) ) + λx‖zx‖0 + λy‖zy‖0 ] , ( 4 ) where θ = ( θX , θY ) , µ = ( µX , µY ) are the model parameters , and λx , λy are regularization parameters that control the sparsity of the input variables . The vectors zx and zy are random STG vectors , with elements defined based on Eq . 3 . Fig . 2 highlights the proposed architecture . Each observed modality is first passed through the gates . Then , the outputs of the gates are used as inputs to a view-specific neural sub-net . Finally , the shared loss term in Eq . 4 is minimized by optimizing the parameters of the gates and neural sub-nets . 2.2 ALGORITHM DETAILS . We now detail the procedure used in DG-CCA for minimizing the loss L ( θ , µ ) ( in Eq . 4 ) . The regularization is based on a parametric expectation and therefore , can be expressed by Ezx‖z‖0 = Dx∑ i=1 P ( zx [ i ] ≥ 0 ) = Dx∑ i=1 ( 1 2 − 1 2 erf ( −µx [ i ] √ 2σ ) ) , where erf ( ) is the Gaussian error function , and is defined similarly for Ezy‖zy‖0 . Denoting the centered output representations of X , Y by ΨX , ΨY ∈ Rd×N respectively , the empirical covariance matrix between these representations can be expressed as Σ̂XY = 1N−1ΨXΨ T Y . Using a similar notations , we express the regularized empirical covariance matrices of X and Y as Σ̂X = 1 N−1ΨXΨ T X + γI and Σ̂Y = 1 N−1ΨY Ψ T Y + γI , where the matrix γI ( γ > 0 ) is added to stabilize the invertability of Σ̂X and Σ̂Y . The total correlation in Eq . 4 can be expressed using the trace of Σ̂ −1/2 Y Σ̂Y XΣ̂ −1 X Σ̂XY Σ̂ −1/2 Y . To learn the parameters of the gates µ and of the representations θ we apply stochastic gradient decent to L ( θ , µ ) . Specifically , we used Monte Carlo sampling to estimate the left part of Eq . 4 . This is repeated for each batch , using one Monte Carlo sample per batch as suggested by Louizos et al . ( 2017 ) and Yamada et al . ( 2020 ) , and worked well in our experiments . After training we remove the stochastic part of the gates , and use only variables ix ∈ { 1 , ... , DX } and iy ∈ { 1 , ... , Dy } such that zx [ ix ] > 0 and zy [ iy ] > 0 .

Summary: This paper presents a new deep CCA method to learn non-linear relationships between two modalities. It trains two neural networks each for a modality to maximize the total correlations of their output representations. Gating is applied to input variables by associating each with a latent Bernoulli variables which is then relaxed with the clipped Gaussian random variable. Experiments on one synthetic and two real datasets demonstrate the superiority of the proposed method.