Canonical Correlation Analysis with Implicit Distributions

1 INTRODUCTION . Canonical correlation analysis ( CCA ) ( Hotelling , 1936 ) is a powerful technique which has been widely adopted in a range of multi-view learning problems ( Haghighi et al. , 2008 ; Kim et al. , 2007 ) . Generally , this technique aims at projecting two set of multivariate data into a subspace so that the transformed data are maximally linear correlated . Since classical CCA captures correlations under linear projections , it is represented as linear CCA . Due to the conjugacy of the prior and the likelihood , probabilistic CCA ( PCCA ) ( Bach & Jordan , 2005 ) presents analytic solution for the posterior , which provides probabilistic interpretation for linear CCA . Specifically , let x ∈ Rdx and y ∈ Rdy be the random vectors in each view respectively , z ∈ Rdz denotes the latent variable , then the posterior is given as p ( z|x , y ) = p ( x , y|z ) p ( z ) p ( x , y ) . ( 1 ) To interpret linear CCA , PCCA adopts the following two assumptions : ( 1 ) The latent codes z follows Gaussian distribution ; ( 2 ) The data in each view are transformed through linear projection . As the prior and the likelihood are conjugate , the posterior p ( z|x , y ) also follows Gaussian distribution , which makes linear correlation an ideal criteria for the multi-view learning problem . In addition , as the two views are both transformed through linear mapping , p ( x , y|z ) can be modeled with the joint covariance matrix , with which the conditional independent assumption is satisfied . p ( x , y|z ) = p ( x|z ) p ( y|z ) . ( 2 ) Consequently , as maximum likelihood estimation on the joint covariance matrix leads to an analytic solution for the posterior , PCCA addresses the probabilistic interpretation for linear CCA , which greatly deepens the understanding of CCA based models . However , due to the linear projection , linear CCA can only capture the linear correlation among the views , while the nonlinear dependency that commonly exists in practical multi-view learning problems can not be properly captured . Recently , a stream of nonlinear CCA variants , such as Kernel CCA ( KCCA ) ( Lai & Fyfe , 2000 ) and Deep CCA ( DCCA ) ( Andrew et al. , 2013 ) , were proposed to capture the nonlinear dependency , but the probabilistic interpretation provided in PCCA fails to generalize to these models for mainly two reasons : 1 ) As nonlinear dependency is to be captured , the Gaussian assumption made on the prior distribution is restricted . Consequently , the linear correlation is no longer an ideal criteria , as the high-order dependency in the latent space can not be captured . 2 ) As the mappings to the latent space are complex in these nonlinear variants , conjugacy between the prior and the likelihood is violated . Consequently , the posterior distributions are intractable in these cases , making it restricted to be modelled with Gaussian assumption . Furthermore , the intractability also makes it hard to validate the conditional independent assumption in equation 2 for practical tasks . To overcome the bottlenecks caused by the assumptions on the prior and the posterior distributions , in this paper , we provide a novel perspective for CCA based on implicit distributions . Specifically , we present minimum Conditional Mutual Information ( CMI ) as a new criteria for multi-view learning problem , which overcomes the limitations of PCCA . To further eliminate the explicit distribution requirement in the direct estimation of CMI , we derive an objective whose minimization implicitly leads to the proposed criteria . Based on this objective , we present an implicit probabilistic formulation for CCA , named Implicit CCA ( ICCA ) . Proposed ICCA is presented to encompass most of the existing probabilistic CCA variants . It also provides a flexible framework to design new CCA extensions that can handle implicit distributions . As an instantiation , we design adversarial CCA ( ACCA ) under the ICCA framework . Specifically , ACCA adopts holistic information for the encoding of the data , and adopts adversarial learning scheme to achieve a consistent constraint for the incorporated encodings . This elegant formulation of ACCA enables it to handle CCA problems with implicit distributions . It also leads to the consistent encoding of the involved variables , which benefits ACCA ’ s ability in capturing nonlinear dependency . Extensive experiments verify that ACCA outperforms the baselines in capturing nonlinear dependency between the two views and consequently achieves superior performance in the cross-view generation task . 2 RELATED WORK . Recently , various nonlinear CCA variants were proposed to improve the performance of CCA . However , assumptions on the prior and posterior distributions in these models restrict their ability in capturing the nonlinear dependency in real-world multi-view learning problems . As shown in Table 1 , considering the adopted extension strategy , most of the nonlinear CCA variants can be grouped into two categories . For the first category , the nonlinear extension is conducted by capturing linear correlations for nonlinear transformed data . The representative works are Kernel CCA ( KCCA ) ( Lai & Fyfe , 2000 ) and Deep CCA ( DCCA ) ( Andrew et al. , 2013 ) . Although the nonlinear mappings are complex , the common space is still assumed to be Gaussian distributed , as they still adopt the linear correlation as the criteria . As the conjugacy of the prior and the likelihood is violated , this distribution assumption restricted . For the other category , the extension is conducted by capturing high-order dependency for linear transformed data . Most of the existing works adopt mutual information or its variants as the nonlinear dependency measurement , corresponding representative models are Canonical Information Analysis ( CIA ) ( Vestergaard & Nielsen , 2015 ) and least-squares canonical dependency analysis ( LSCDA ) ( Karasuyama & Sugiyama , 2012 ) , respectively . However , explicit distributions are still required to estimate the adopted criteria in these methods . The estimation would be extremely complicated for the high dimensional problem . This greatly restricts its ability in handle high-dimensional multi-view learning problem for large datasets . In ( Wang et al. , 2016 ) , two generative nonlinear CCA variants , VCCA and Bi-VCCA were proposed , which both can handle implicit likelihood distributions with variational inference . However , the connection between its objective and that of CCA is not obvious . Furthermore , the prior p ( z ) and the approximate posteriors q ( z|x , y ) are still required to be explicit distribution so that the KL-divergence can be computed . However , as the data are usually complex in real-world applications , assumptions on these two distributions restricts the power of these CCA models ( Mescheder et al. , 2017 ) . For example , if the data in the two views are from two separated clusters , the complex data structure in each view makes it restrictive to make certain distribution assumptions for the latent space . Drawbacks caused by the restricted distribution assumptions in the existing methods makes is reasonable and promising to propose nonlinear CCA models which can handle implicit distributions . 3 IMPLICIT CCA - AN IMPLICIT PROBABILISTIC FORMULATION OF CCA . In this section , we propose an implicit probabilistic formulation for CCA , named Implicit CCA ( ICCA ) . First , we present PCCA and discuss its limitations as preliminary . Next , we present minimum CMI as a new criteria to overcome the limitations of PCCA . Then , we present a general probabilistic formulation of CCA ( ICCA ) based on the proposed criteria , which provides a flexible framework to design CCA extensions with implicit distributions . Connections between ICCA and existing probabilistic CCA variants are then presented . 3.1 PRELIMINARY- FORMULATION OF LINEAR CCA AND PCCA . Let X = { x ( i ) } Ni=1 and Y = { y ( i ) } Ni=1 be the inputs with pairwise correspondence in multi-view scenario . Classic linear CCA aims to find linear projections for the two views , ( W ′ xX , W ′ yY ) , such that the linear correlation between the projections are mutually maximized . ( W∗x , W ∗ y ) = arg max Wx , Wy corr ( W ′ xX , W ′ yY ) = arg max Wx , Wy W ′ xΣxyWy√ W′xΣxxWxW ′ yΣyyWy ( 3 ) Where Σxx and Σyy denote the covariance of X and Y respectively , and Σxy denotes the crosscovariance of X and Y . Based on the graphical diagram depicted in Figure 1 . ( a ) , PCCA provides probabilistic interpretation for linear CCA by proving that maximum likelihood estimation for the parameters in equation 4 leads to the canonical correlation directions of linear CCA . z ∼ N ( 0 , Id ) , x|z ∼ N ( Wxz + µx , Φx ) , y|z ∼ N ( Wyz + µy , Φy ) , ( 4 ) where Wx , Wy , Φx , Φy , µx and µy are the model parameters ; d denotes the dimension of the projected space . As discussed in the introduction , since nonlinear dependency is required to be captured in nonlinear CCA models , PCCA can not be generalized to these models for two reasons : • The linear correlation criteria adopted in PCCA can not capture the high-order dependency between the variables . • The conditional conditional independent assumption adopted in PCCA can not be generalized to nonlinear CCA variants . Consequently , there are two points to consider in probabilistic interpretation of nonlinear CCA variants : ( 1 ) a new criteria which can measure nonlinear dependency is required ; ( 2 ) how to avoid the conditional independent assumption to benefit the generalization ability of the proposed model . 3.2 MINIMUM CMI - A NEW CRITERIA FOR MULTI-VIEW LEARNING PROBLEM . In this paper , we present minimum Conditional Mutual Information ( CMI ) as a new criteria for the CCA-based models , which overcomes the aforementioned limitations of PCCA simultaneously . Given three random variables X , Y and Z , CMI , defined in equation 5 , measures the expected value of the mutual information between X and Y given Z ( Rahimzamani & Kannan , 2017 ) . I ( X ; Y|Z ) = ∫∫∫ p ( z ) p ( x , y|z ) log p ( x , y|z ) p ( x|z ) p ( y|z ) dzdxdy = H ( X|Z ) −H ( X|Y , Z ) , ( 5 ) where the entropy H ( X|Z ) quantifies the uncertainty of X when Z is known , and H ( X|Y , Z ) quantifies the remaining uncertainty of X when Y , Z are both known . where the entropy H ( X|∗ ) quantifies the uncertainty of X when ∗ is known . Obviously , CMI overcomes the limitations of PCCA from both the two aspects . First , mutual information ( MI ) is a general correlation metric which can measure the nonlinear dependency between two random variables ( Cover & Thomas , 2012 ) , it is a suitable statistic for interpreting nonlinear CCA variants . Second , measuring CMI of the multi-view inputs given the common latent space avoids the conditional independent assumption of PCCA . Furthermore , for multi-view learning problem , as the mutual dependency of X and Y are maximumly captured in the common latent space Z , further introducing Y as the condition has not much influence on the uncertainty of X . Thus the difference between H ( X|Z ) and H ( X|Y , Z ) in equation 5 , is to be minimized . Consequently , minimum CMI can be adopted as a new criteria for CCA .

Summary: This paper proposes to improve deep variational canonical correlation analysis (VCCA, Bi-VCCA) by 1) applying adversarial autoencoders (Makhzani et al. ICLR 2016) to model the encoding from multiple data views (X, Y, XY) to the latent representation (Z); and 2) introducing q(z|x,y) to explicitly encode the joint distribution of two views X,Y. The proposed approach, called adversarial canonical correlation analysis (ACCA), is essentially the application of adversarial autoencoder to multiple data views. Experiments on benchmark datasets, including the MNIST left right halved dataset, MNIST noisy dataset, and Wisconsin X-ray microbeam database, show the proposed ACCA result in higher dependence (measured by the normalized HSIC) between two data views compared to Bi-VCCA.