L0-Sparse Canonical Correlation Analysis

1 INTRODUCTION . Canonical Correlation Analysis ( CCA ) ( Hotelling , 1936 ; Thompson , 2005 ) , is a classic statistical method for finding the maximally correlated linear transformations of two modalities ( or views ) . Using modalitiesX ∈ RDx×N andY ∈ RDy×N , which are centered and haveN samples withDx and Dy features , respectively , CCA seeks canonical vectors a ∈ RDx , and b ∈ RDy , such that , u = aTX , and v = bTY , will maximize the sample correlations between the canonical variates , i.e . max a , b6=0 ρ ( aTX , bTY ) = aTXY T b ‖aTX‖2‖bTY ‖2 . ( 1 ) To identify non-linear relations between input variables , several extensions of CCA have been proposed . Kernel methods such as Kernel CCA ( Bach & Jordan , 2002 ) , Non-parametric CCA ( Michaeli et al. , 2016 ) or Multi-view Diffusion maps ( Lindenbaum et al. , 2020 ; Salhov et al. , 2020 ) learn the non-linear relations in reproducing Hilbert spaces . These methods have several shortcomings : they are limited to a designed kernel , and they have poor interpolation and extrapolation capabilities . Deep CCA ( Andrew et al. , 2013 ) overcomes these limitations by learning two non-linear transformations parametrized using neural networks . The model has several extension , see for example ( Wang et al. , 2016 ; Gundersen et al. , 2019 ; Karami & Schuurmans , 2021 ) . CCA models have been widely used in biology ( Pimentel et al. , 2018 ) , neuroscience ( Al-Shargie et al. , 2017 ) , medicine ( Zhang et al. , 2017 ) , and engineering ( Chen et al. , 2017 ) , for unsupervised or semi-supervised learning . By extracting meaningful dimensionality reduced representations , CCA improves downstream tasks such as clustering , classification , or manifold learning . One key limitation of CCA is that it requires more samples than features , i.e. , N > Dx , Dy . However , if we have more variables than samples , the estimation based on the closed-form solution of the CCA problem ( in Eq . 1 ) breaks ( Suo et al. , 2017 ) ( because the covariance matrix is low rank ) . Moreover , in high dimensional data , often some of the variables do not measure the phenomenon that is common to both modalities ( therefore are not correlated ) and thus should be omitted from the transformations . For these reasons , there has been a growing interest in studying sparse CCA models . Sparse CCA ( SCCA ) models seek linear transformations which are based on a sparse subset of variables from the input modalities X and Y . Sparsifying the feature space improves interpretability , prevents overfitting , and removes the degeneracies inherent to N < Dx , Dy ( once the number of selected variables is smaller thanN the covariance matrix becomes full rank ) . To encourage sparsity of the canonical vectors a and b , several authors ( Wiesel et al. , 2008 ; Cai et al. , 2019 ) propose an ` 0 regularized variant of Eq . 1 . However , these schemes are greedy and therefore may lead to suboptimal solutions . As demonstrated by ( Waaijenborg et al. , 2008 ; Parkhomenko et al. , 2009 ; Witten et al. , 2009 ; Hardoon & Shawe-Taylor , 2011 ; Suo et al. , 2017 ) replacing the ` 0 norm by ` 1 is differentiable and leads to a sparse solution to Eq . 1 . Several authors extended these ideas by considering group sparsity ( Chen & Liu , 2012 ; Klami et al. , 2013 ; Solari et al. , 2019 ) . However , these methods are limited to linear transformations and may lead to shrinkage of the canonical vectors due to the ` 1 regularizer . There has been limited work on extending these models to sparse non-linear CCA . Specifically , there are two kernel-based extensions : two-stage kernel CCA ( TSKCCA ) by ( Yoshida et al. , 2017 ) and SCCA based on Hilbert-Schmidt Independence Criterion ( SCCA-HSIC ) by ( Uurtio et al. , 2018 ) . However , these models suffer from the same limitations as KCCA and do not scale to a high dimensional regime . We present ` 0-CCA , a simple yet effective method for learning correlated representation based on sparse subsets of the input variables by minimizing an ` 0 regularized loss . Our ` 0 regularization relies on a recently proposed Gaussian-based continuous relaxation of Bernoulli random variables , termed gates ( Yamada et al. , 2020 ) . The gates are applied to the input features to sparsify the canonical vectors . The model and gates parameters are trained jointly via gradient descent to maximize the correlation between the representations ofX and Y while simultaneously selecting only the subsets of most correlated input features . Our contributions are four-folds : ( i ) By modeling the transformations using two neural networks , we provide a natural solution to the problem of sparse non-linear correlation analysis . ( ii ) We then propose a novel initialization scheme that helps the model identify small subsets of correlated features in the challenging regime of D > N . ( iii ) We apply the proposed method to synthetic data and demonstrate that our approach improves the estimation of the canonical vectors compared with existing baselines . ( iv ) We demonstrate the efficacy of the proposed scheme to several real work applications ; specifically , we demonstrate that it leads to more reliable and interpretable representations than other linear and non-linear data fusion schemes . 2 SPARSE CCA . The problem in Eq . 1 has a closed-form solution based on the eigendecomposition of C−1x CxyC −1 y Cyx and C −1 y CyxC −1 x Cxy , where Cx , Cy are within view sample covariance matrices and Cxy , Cyx are cross-view sample covariance matrices . However , if N is smaller than the number of input variables ( Dx or Dy ) , the sample covariance may be rank deficient , and the closed-form solution becomes meaningless . To overcome this limitation , we consider the problem of sparse CCA . Sparse CCA reduces the number of active variables and therefore the covariance matrix becomes full rank . Sparsity also improves model interpretability but may reduce the maximal ( training ) correlation value obtained in Eq . 1 ( Shalev-Shwartz et al. , 2010 ) . Sparse CCA deals with identifying maximally correlated representations based on linear combinations of sparse subsets of the input variables inX and Y . The problem can be formulated as min a , b − ρ ( aTX , bTY ) + λx‖a‖0 + λy‖b‖0 , ( 2 ) where λx and λy are regularization parameters which control the sparsify the input variables . If ‖a‖0 and ‖b‖0 are smaller than N , we can remove the degeneracy inherent to Eq . 1 , and identify meaningful correlated representations based on a sparse subset of input variables . 3 PROBABILISTIC REFORMULATION OF SPARSE CCA . The sparse CCA problem formulated in Eq . 2 becomes intractable for large Dx or Dy , moreover , due to the discrete nature of the ` 0 norm , the problem is not amenable to gradient-based optimization schemes . Fortunately , as demonstrated in sparse regression , probabilistic models such as the spikeand-slab ( George & McCulloch , 1993 ; Kuo & Mallick , 1998 ; Zhou et al. , 2009 ; Polson & Sun , 2019 ) provide a compelling alternative . More recently , differentiable probabilistic models such as ( Louizos et al. , 2017 ; Yamada et al. , 2020 ) were proposed for sparse supervised learning . Here , we adopt these ideas by rewriting the canonical vectors as α = θx sx and β = θy sy , where denotes the Hadamard product ( element wise multiplication ) , and θx ∈ RDx , θy ∈ RDy . The vectors sx ∈ { 0 , 1 } Dx , sy ∈ { 0 , 1 } Dy are Bernoulli random vectors with parameters πx = ( πx1 , ... , πxDx ) and πy = ( πy1 , ... , π y Dy ) . These Bernoulli variables , act as gates and sparsify the coefficients of the canonical vectors . Now , the problem in Eq . 2 can be reformulated as an expectation minimization , which is parameterized by πx and πy . Specifically , based on the following theorem , we can reformulate the problem in Eq . 2 . Theorem 3.1 The solution of the sparse CCA problem in Eq . 2 is equivalent to the solution of the following probabilistic problem min πx , πy , θx , θy E [ − ρ ( αTX , βTY ) + λx‖sx‖0 + λy‖sy‖0 ] , ( 3 ) where α = θx sx and β = θy sy and the expectation is taken with respect to the random Bernoulli variables sx and sy ( which are parameterized by πx and πy ) . Note that the expected values of the ` 0 norms can be expressed using the Bernoulli parameters as E‖sx‖0 = ∑ πxi , and E‖sy‖0 = ∑ πyi . The proof relies on the fact that the optimal solution to Eq . 2 is a valid solution to Eq . 3 , and vice versa . The proof is presented in the Appendix , Section H , and follows the same construction as the proof of Theorem 1 in ( Yin et al. , 2020 ) . 3.1 CONTINUOUS RELAXATION FOR SPARSE CCA . Due to the discrete nature of sx and sy , differentiating the leading term in Eq . 3 is not straightforward . Although solutions such as REINFORCE ( Williams , 1992 ) or the straight-through estimator ( Bengio et al. , 2013 ) enable differentiating through discrete random variables , they still suffer from high variance . Furthermore , they require many Monte Carlo samples for effective training ( Tucker et al. , 2017 ) . Recently , several authors ( Maddison et al. , 2016 ; Jang et al. , 2016 ; Louizos et al. , 2017 ) have demonstrated that using a continuous reparametrization of discrete random variables can reduce the variance of the gradient estimates . Here , following the reparametrization presented in ( Yamada et al. , 2020 ) , we use Gaussian-based relaxation for the Bernoulli random variables . Each relaxed Bernoulli variables zi is defined by drawing from a centered Gaussian i ∼ N ( 0 , σ2 ) , then shifting it by µi and truncating its values using the following hard Sigmoid function zi = max ( 0 , min ( 1 , µi + i ) ) . ( 4 ) Using these relaxed Bernoulli variables , we can define the gated canonical vectors as α = θx zx and β = θy zy . We incorporate these vectors into the objective defined in Eq . 3 , in which the ` 0 regularization terms can be expressed as E‖zx‖0 = Dx∑ i=1 P ( zxi ≥ 0 ) = Dx∑ i=1 ( 1 2 − 1 2 erf ( − µ x i√ 2σ ) ) , where erf ( ) is the Gaussian error function , and is defined similarly for E‖zy‖0 . To learn to model parameters θx , θy and gate parameters µx , µy we first draw realizations for the gates , then we update the parameters by applying gradient descent to minimize E [ − ρ ( αTX , βTY ) + λx‖zx‖0 + λy‖zy‖0 ] . Post training , we remove the stochasicity from the gates and use all variables such that zxi = max ( 0 , min ( 1 , µxi ) ) > 0 ( defined similarly for z y ) .

Summary: The paper proposes a sparse canonical correlation analysis method based on l_0 norm. A continuous relaxation scheme is adopted for solving sparse CCA. The proposed model is then extended to nonlinear function estimation and combined with deep neural networks. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed methods.