L0-Sparse Canonical Correlation Analysis

1 INTRODUCTION . Canonical Correlation Analysis ( CCA ) ( Hotelling , 1936 ; Thompson , 2005 ) , is a classic statistical method for finding the maximally correlated linear transformations of two modalities ( or views ) . Using modalitiesX ∈ RDx×N andY ∈ RDy×N , which are centered and haveN samples withDx and Dy features , respectively , CCA seeks canonical vectors a ∈ RDx , and b ∈ RDy , such that , u = aTX , and v = bTY , will maximize the sample correlations between the canonical variates , i.e . max a , b6=0 ρ ( aTX , bTY ) = aTXY T b ‖aTX‖2‖bTY ‖2 . ( 1 ) To identify non-linear relations between input variables , several extensions of CCA have been proposed . Kernel methods such as Kernel CCA ( Bach & Jordan , 2002 ) , Non-parametric CCA ( Michaeli et al. , 2016 ) or Multi-view Diffusion maps ( Lindenbaum et al. , 2020 ; Salhov et al. , 2020 ) learn the non-linear relations in reproducing Hilbert spaces . These methods have several shortcomings : they are limited to a designed kernel , and they have poor interpolation and extrapolation capabilities . Deep CCA ( Andrew et al. , 2013 ) overcomes these limitations by learning two non-linear transformations parametrized using neural networks . The model has several extension , see for example ( Wang et al. , 2016 ; Gundersen et al. , 2019 ; Karami & Schuurmans , 2021 ) . CCA models have been widely used in biology ( Pimentel et al. , 2018 ) , neuroscience ( Al-Shargie et al. , 2017 ) , medicine ( Zhang et al. , 2017 ) , and engineering ( Chen et al. , 2017 ) , for unsupervised or semi-supervised learning . By extracting meaningful dimensionality reduced representations , CCA improves downstream tasks such as clustering , classification , or manifold learning . One key limitation of CCA is that it requires more samples than features , i.e. , N > Dx , Dy . However , if we have more variables than samples , the estimation based on the closed-form solution of the CCA problem ( in Eq . 1 ) breaks ( Suo et al. , 2017 ) ( because the covariance matrix is low rank ) . Moreover , in high dimensional data , often some of the variables do not measure the phenomenon that is common to both modalities ( therefore are not correlated ) and thus should be omitted from the transformations . For these reasons , there has been a growing interest in studying sparse CCA models . Sparse CCA ( SCCA ) models seek linear transformations which are based on a sparse subset of variables from the input modalities X and Y . Sparsifying the feature space improves interpretability , prevents overfitting , and removes the degeneracies inherent to N < Dx , Dy ( once the number of selected variables is smaller thanN the covariance matrix becomes full rank ) . To encourage sparsity of the canonical vectors a and b , several authors ( Wiesel et al. , 2008 ; Cai et al. , 2019 ) propose an ` 0 regularized variant of Eq . 1 . However , these schemes are greedy and therefore may lead to suboptimal solutions . As demonstrated by ( Waaijenborg et al. , 2008 ; Parkhomenko et al. , 2009 ; Witten et al. , 2009 ; Hardoon & Shawe-Taylor , 2011 ; Suo et al. , 2017 ) replacing the ` 0 norm by ` 1 is differentiable and leads to a sparse solution to Eq . 1 . Several authors extended these ideas by considering group sparsity ( Chen & Liu , 2012 ; Klami et al. , 2013 ; Solari et al. , 2019 ) . However , these methods are limited to linear transformations and may lead to shrinkage of the canonical vectors due to the ` 1 regularizer . There has been limited work on extending these models to sparse non-linear CCA . Specifically , there are two kernel-based extensions : two-stage kernel CCA ( TSKCCA ) by ( Yoshida et al. , 2017 ) and SCCA based on Hilbert-Schmidt Independence Criterion ( SCCA-HSIC ) by ( Uurtio et al. , 2018 ) . However , these models suffer from the same limitations as KCCA and do not scale to a high dimensional regime . We present ` 0-CCA , a simple yet effective method for learning correlated representation based on sparse subsets of the input variables by minimizing an ` 0 regularized loss . Our ` 0 regularization relies on a recently proposed Gaussian-based continuous relaxation of Bernoulli random variables , termed gates ( Yamada et al. , 2020 ) . The gates are applied to the input features to sparsify the canonical vectors . The model and gates parameters are trained jointly via gradient descent to maximize the correlation between the representations ofX and Y while simultaneously selecting only the subsets of most correlated input features . Our contributions are four-folds : ( i ) By modeling the transformations using two neural networks , we provide a natural solution to the problem of sparse non-linear correlation analysis . ( ii ) We then propose a novel initialization scheme that helps the model identify small subsets of correlated features in the challenging regime of D > N . ( iii ) We apply the proposed method to synthetic data and demonstrate that our approach improves the estimation of the canonical vectors compared with existing baselines . ( iv ) We demonstrate the efficacy of the proposed scheme to several real work applications ; specifically , we demonstrate that it leads to more reliable and interpretable representations than other linear and non-linear data fusion schemes . 2 SPARSE CCA . The problem in Eq . 1 has a closed-form solution based on the eigendecomposition of C−1x CxyC −1 y Cyx and C −1 y CyxC −1 x Cxy , where Cx , Cy are within view sample covariance matrices and Cxy , Cyx are cross-view sample covariance matrices . However , if N is smaller than the number of input variables ( Dx or Dy ) , the sample covariance may be rank deficient , and the closed-form solution becomes meaningless . To overcome this limitation , we consider the problem of sparse CCA . Sparse CCA reduces the number of active variables and therefore the covariance matrix becomes full rank . Sparsity also improves model interpretability but may reduce the maximal ( training ) correlation value obtained in Eq . 1 ( Shalev-Shwartz et al. , 2010 ) . Sparse CCA deals with identifying maximally correlated representations based on linear combinations of sparse subsets of the input variables inX and Y . The problem can be formulated as min a , b − ρ ( aTX , bTY ) + λx‖a‖0 + λy‖b‖0 , ( 2 ) where λx and λy are regularization parameters which control the sparsify the input variables . If ‖a‖0 and ‖b‖0 are smaller than N , we can remove the degeneracy inherent to Eq . 1 , and identify meaningful correlated representations based on a sparse subset of input variables . 3 PROBABILISTIC REFORMULATION OF SPARSE CCA . The sparse CCA problem formulated in Eq . 2 becomes intractable for large Dx or Dy , moreover , due to the discrete nature of the ` 0 norm , the problem is not amenable to gradient-based optimization schemes . Fortunately , as demonstrated in sparse regression , probabilistic models such as the spikeand-slab ( George & McCulloch , 1993 ; Kuo & Mallick , 1998 ; Zhou et al. , 2009 ; Polson & Sun , 2019 ) provide a compelling alternative . More recently , differentiable probabilistic models such as ( Louizos et al. , 2017 ; Yamada et al. , 2020 ) were proposed for sparse supervised learning . Here , we adopt these ideas by rewriting the canonical vectors as α = θx sx and β = θy sy , where denotes the Hadamard product ( element wise multiplication ) , and θx ∈ RDx , θy ∈ RDy . The vectors sx ∈ { 0 , 1 } Dx , sy ∈ { 0 , 1 } Dy are Bernoulli random vectors with parameters πx = ( πx1 , ... , πxDx ) and πy = ( πy1 , ... , π y Dy ) . These Bernoulli variables , act as gates and sparsify the coefficients of the canonical vectors . Now , the problem in Eq . 2 can be reformulated as an expectation minimization , which is parameterized by πx and πy . Specifically , based on the following theorem , we can reformulate the problem in Eq . 2 . Theorem 3.1 The solution of the sparse CCA problem in Eq . 2 is equivalent to the solution of the following probabilistic problem min πx , πy , θx , θy E [ − ρ ( αTX , βTY ) + λx‖sx‖0 + λy‖sy‖0 ] , ( 3 ) where α = θx sx and β = θy sy and the expectation is taken with respect to the random Bernoulli variables sx and sy ( which are parameterized by πx and πy ) . Note that the expected values of the ` 0 norms can be expressed using the Bernoulli parameters as E‖sx‖0 = ∑ πxi , and E‖sy‖0 = ∑ πyi . The proof relies on the fact that the optimal solution to Eq . 2 is a valid solution to Eq . 3 , and vice versa . The proof is presented in the Appendix , Section H , and follows the same construction as the proof of Theorem 1 in ( Yin et al. , 2020 ) . 3.1 CONTINUOUS RELAXATION FOR SPARSE CCA . Due to the discrete nature of sx and sy , differentiating the leading term in Eq . 3 is not straightforward . Although solutions such as REINFORCE ( Williams , 1992 ) or the straight-through estimator ( Bengio et al. , 2013 ) enable differentiating through discrete random variables , they still suffer from high variance . Furthermore , they require many Monte Carlo samples for effective training ( Tucker et al. , 2017 ) . Recently , several authors ( Maddison et al. , 2016 ; Jang et al. , 2016 ; Louizos et al. , 2017 ) have demonstrated that using a continuous reparametrization of discrete random variables can reduce the variance of the gradient estimates . Here , following the reparametrization presented in ( Yamada et al. , 2020 ) , we use Gaussian-based relaxation for the Bernoulli random variables . Each relaxed Bernoulli variables zi is defined by drawing from a centered Gaussian i ∼ N ( 0 , σ2 ) , then shifting it by µi and truncating its values using the following hard Sigmoid function zi = max ( 0 , min ( 1 , µi + i ) ) . ( 4 ) Using these relaxed Bernoulli variables , we can define the gated canonical vectors as α = θx zx and β = θy zy . We incorporate these vectors into the objective defined in Eq . 3 , in which the ` 0 regularization terms can be expressed as E‖zx‖0 = Dx∑ i=1 P ( zxi ≥ 0 ) = Dx∑ i=1 ( 1 2 − 1 2 erf ( − µ x i√ 2σ ) ) , where erf ( ) is the Gaussian error function , and is defined similarly for E‖zy‖0 . To learn to model parameters θx , θy and gate parameters µx , µy we first draw realizations for the gates , then we update the parameters by applying gradient descent to minimize E [ − ρ ( αTX , βTY ) + λx‖zx‖0 + λy‖zy‖0 ] . Post training , we remove the stochasicity from the gates and use all variables such that zxi = max ( 0 , min ( 1 , µxi ) ) > 0 ( defined similarly for z y ) .

Summary: The paper proposes a new method for l0-CCA using stochastic gating which allows for an efficient algorithm and also permits a deep-version of the l0-CCA. Results are shown on synthetic and real-world datasets that highlight the superior performance of the proposed approach.   Main Contributions:   1). The paper proposes a new approach for l0-CCA that uses a continuous relaxation of a bernoulli random variable via stochastic gating.   2). The paper extends the proposed method to l0-deep-CCA which allows non-linear interactions between the two CCA views.  3). Results are shown on synthetic and real-world datasets which show the superior performance of the proposed l0-CCA methods.