Deep Gated Canonical Correlation Analysis

1 INTRODUCTION . Canonical Correlation Analysis ( CCA ) ( Hotelling , 1936 ; Thompson , 2005 ) , is a classic statistical method for finding the maximally correlated linear transformations of two modalities ( or views ) . Using modalities X ∈ RDx×N and Y ∈ RDY ×N , which are centered and have N samples with Dx and Dy features respectively . CCA seeks canonical vectors ai ∈ RDX , and bi ∈ RDY , such that , ui = aTi X , and vi = b T i Y , i = 1 , ... , N , maximize the sample correlations between ui and vi , where ui ( vi ) form an orthonormal basis for i = 1 , ... , d , i.e . ai , bi = argmax 〈ui , uj〉=δi , j , 〈vi , vj〉=δi , j , i , j=1 , ... , d Corr ( ui , vi ) . ( 1 ) While CCA enjoys a closed-form solution using a generalized eigen pair problem , it is restricted to the linear transformationsA = [ a1 , ... , ad ] andB = [ b1 , ... , bd ] . In order to identify non-linear relations between input variables , several extensions of CCA have been proposed . Kernel methods such as Kernel CCA ( Bach & Jordan , 2002 ) , Non-paramatric CCA ( Michaeli et al. , 2016 ) or Multi-view Diffusion maps ( Lindenbaum et al. , 2020 ) learn the non-linear relations in reproducing Hilbert spaces . These methods have several shortcomings : they are limited to a designed kernel , they requireO ( N2 ) computations for training , and they have poor interpolation and extrapolation capabilities . To overcome these limitations , Andrew et al . ( 2013 ) have proposed Deep CCA , to learn parametric non-linear transformations of the input modalities X and Y . The functions are learned by training two neural networks to maximize the total correlation between their outputs . Linear and non-linear canonical correlation models have been widely used in the setting of unsupervised or semi-supervised learning . When d is set to a dimension satisfying d < Dx , Dy , these models find dimensional reduced representations that may be useful for clustering , classification or manifold learning in many applications . For example , in biology ( Pimentel et al. , 2018 ) , neuroscience ( Al-Shargie et al. , 2017 ) , medicine ( Zhang et al. , 2017 ) , and engineering ( Chen et al. , 2017 ) . One key limitations of these models is that they typically require more samples than features , i.e . N > Dx , Dy . However , if we have more variable than samples , the estimation based on the closed form solution of the CCA problem ( in Eq . 1 ) breaks ( Suo et al. , 2017 ) . Moreover , in high dimensional data , often some of the variables are not informative and thus should be omitted from the transformations . For these reasons , there has been a growing interest in studying sparse CCA models . Sparse CCA ( SCCA ) ( Waaijenborg et al. , 2008 ; Hardoon & Shawe-Taylor , 2011 ; Suo et al. , 2017 ) uses an ` 1 penalty to encourage sparsity of the canonical vectors ai and bi . This can not only remove the degeneracy inherit to N < Dx , Dy , but can improve interpetability and performance . One caveat of this approach is its high computational complexity , which can be reduced by replacing the orthonormality constraints on ui and vi with orthonormality constraints on ai and bi . This procedure is known as simplified-SCCA ( Parkhomenko et al. , 2009 ; Witten et al. , 2009 ) , which enjoys a closed form solution . There has been limited work on extending these models to sparse nonlinear CCA . Specifically , there are two kernel based extensions , two-stage kernel CCA ( TSKCCA ) by Yoshida et al . ( 2017 ) and SCCA based on Hilbert-Schmidt Independence Criterion ( SCCAHSIC ) by Uurtio et al . ( 2018 ) . However , these models suffer from the same limitations as KCCA and are not scalable to a high dimensional regime . This paper presents a sparse CCA model that can be optimized using standard deep learning methodologies . The method combines the differentiable loss presented in DCCA ( Andrew et al. , 2013 ) with an approximate ` 0 regularization term designed to sparsify the input variables of both X and Y . Our regularization relies on a recently proposed Gaussian based continuous relaxation of Bernoulli random variables , termed gates ( Yamada et al. , 2020 ) . The gates are applied to the input features to sparsify X and Y . The gates parameters are trained jointly via stochastic gradient decent to maximize the correlation between the representations of X and Y , while simultaneously selecting only the subsets of the most correlated input features . We apply the proposed method to synthetic data , and demonstrate that our method can improve the estimation of the canonical vectors compared with SCCA models . Then , we use the method to identify informative variable in multichannel noisy seismic data and show its advantage over other CCA models . 1.1 BACKGROUND . 1.2 DEEP CCA . Andrew et al . ( 2013 ) , present a deep neural network that learns correlated representations . They proposed Deep Canonical Correlation Analysis ( DCCA ) which extracts two nonlinear transformations of X and Y with maximal correlation . DCCA trains two neural networks with a joint loss aiming to maximize the total correlation of the network ’ s outputs . The parameters of the networks are learned by applying stochastic gradient decent to the following objective : θ∗X , θ ∗ Y = argmax θX , θY Corr ( f ( X ; θX ) , g ( Y ; θY ) ) , ( 2 ) where θX and θY are the trainable parameters , and f ( X ) , g ( Y ) ∈ Rd are the desired correlated representations . 1.3 SPARSE CCA . Several authors have proposed solutions for the problem of recovering sparse canonical vectors . The key advantages of sparse vectors are that they enable identifying correlated representations even in the regime of N < Dx , Dy and they allow unsupervised feature selection . Following the formulation by Suo et al . ( 2017 ) , SCCA could be described using the following regularized objective a , b = argmin [ − Cov ( aTX , bTY ) + τ1‖a‖1 + τ2‖b‖1 ] , subject to ‖aTX‖2 ≤ 1 , ‖bTY ‖2 ≤ 1 , where τ1 and τ2 are regularization parameters for controlling the sparsity of the canonical vectors a and b . Note that the relaxed inequality constrain on aTX and bTY makes the problem bi-convex , however , if ‖aTX‖2 < 1 or ‖bTX‖2 < 1 , then the covariance in the objective is no longer equal to the correlation . 1.4 STOCHASTIC GATES . In the last few years , several methods have been proposed for incorporating discrete random variables into gradient based optimization methods . Towards this goal , continuous relaxations of discrete random variables such as ( Maddison et al. , 2016 ; Jang et al. , 2017 ) have been proposed . Such relaxations have been used in several applications , for example , model compression ( Louizos et al. , 2017 ) , feature selection or for defining discrete activations ( Jang et al. , 2016 ) . We focus on a Gaussian-based relaxation of Bernoulli variables , termed Stochastic Gates ( STG ) ( Yamada et al. , 2020 ) , which were originally proposed for supervised feature selection . We denote the STG random vector by z ∈ [ 0 , 1 ] D , where each entry is defined as z [ i ] = max ( 0 , min ( 1 , µ [ i ] + [ i ] ) ) , ( 3 ) where µ [ i ] is a trainable parameter for entry i , the injected noise [ i ] is drawn fromN ( 0 , σ2 ) and σ is fixed throughout training . This approximation can be viewed as a clipped , mean-shifted , Gaussian random variable . In Fig . 1 we illustrate generation of the transformed random variable z [ i ] for µ [ i ] = 0.5 which represents a “ fair ” relaxed Bernoulli variable . 2 DEEP GATED CCA . 2.1 MODEL . It is appealing to try to combine ideas from Sparse CCA into the rich differentiable model of Deep CCA . However , a straight forward ` 1 regularization of the input layer of a neural network does not work in practice because it makes the learning procedure unstable . This was observed in the supervised setting by Li et al . ( 2016 ) ; Feng & Simon ( 2017 ) . This instability occurs because the objective is not differentiable everywhere . To overcome this limitation , we use the STG random variables ( see Eq . 3 ) by multiplying them with the features of X and Y . Then , by penalizing for active gates using a regularization term E‖z‖0 , we can induce sparsity in the input variables . We formulate the problem of sparse nonlinear CCA by regularizing a deep neural network with a correlation term . We introduce two random STG vectors into the input layers of two neural networks which are trained in tandem to maximize the total correlation . Denoting the random gating vectors zx and zy for viewX and Y respectively , the Deep Gated CCA ( DG-CCA ) loss is defined by L ( θ , µ ) = Ezx , zy [ − Corr ( f ( zTxX ; θX ) , g ( zTy Y ; θY ) ) + λx‖zx‖0 + λy‖zy‖0 ] , ( 4 ) where θ = ( θX , θY ) , µ = ( µX , µY ) are the model parameters , and λx , λy are regularization parameters that control the sparsity of the input variables . The vectors zx and zy are random STG vectors , with elements defined based on Eq . 3 . Fig . 2 highlights the proposed architecture . Each observed modality is first passed through the gates . Then , the outputs of the gates are used as inputs to a view-specific neural sub-net . Finally , the shared loss term in Eq . 4 is minimized by optimizing the parameters of the gates and neural sub-nets . 2.2 ALGORITHM DETAILS . We now detail the procedure used in DG-CCA for minimizing the loss L ( θ , µ ) ( in Eq . 4 ) . The regularization is based on a parametric expectation and therefore , can be expressed by Ezx‖z‖0 = Dx∑ i=1 P ( zx [ i ] ≥ 0 ) = Dx∑ i=1 ( 1 2 − 1 2 erf ( −µx [ i ] √ 2σ ) ) , where erf ( ) is the Gaussian error function , and is defined similarly for Ezy‖zy‖0 . Denoting the centered output representations of X , Y by ΨX , ΨY ∈ Rd×N respectively , the empirical covariance matrix between these representations can be expressed as Σ̂XY = 1N−1ΨXΨ T Y . Using a similar notations , we express the regularized empirical covariance matrices of X and Y as Σ̂X = 1 N−1ΨXΨ T X + γI and Σ̂Y = 1 N−1ΨY Ψ T Y + γI , where the matrix γI ( γ > 0 ) is added to stabilize the invertability of Σ̂X and Σ̂Y . The total correlation in Eq . 4 can be expressed using the trace of Σ̂ −1/2 Y Σ̂Y XΣ̂ −1 X Σ̂XY Σ̂ −1/2 Y . To learn the parameters of the gates µ and of the representations θ we apply stochastic gradient decent to L ( θ , µ ) . Specifically , we used Monte Carlo sampling to estimate the left part of Eq . 4 . This is repeated for each batch , using one Monte Carlo sample per batch as suggested by Louizos et al . ( 2017 ) and Yamada et al . ( 2020 ) , and worked well in our experiments . After training we remove the stochastic part of the gates , and use only variables ix ∈ { 1 , ... , DX } and iy ∈ { 1 , ... , Dy } such that zx [ ix ] > 0 and zy [ iy ] > 0 .

Summary: This paper proposes a DL method for learning sparse non-linear transformations that maximize correlations between two views. In particular, each view is passed through a separate network. Stochastic Gating is applied to the input layer of each network. The two networks are jointly trained by maximising the correlation between their outputs. Sparsity is obtained by imposing L0 regularization terms on the Stochastic Gating variables.