Deep Multi-View Learning via Task-Optimal CCA

1 INTRODUCTION . Parallel modalities of data are increasingly common in a variety of applications , including images and text , audio and video , parallel texts of different languages , and a variety of medical imaging and omics modalities for each patient . Each view provides essential information for classification and , when used together , can form a more accurate model . This is especially important for difficult discriminative tasks such as those with a small training set size . Canonical Correlation Analysis ( CCA ) is the most common method for computing a shared representation from two views of data by computing a space in which they are maximally correlated ( Hotelling , 1936 ; Bie et al. , 2005 ) . In this paper we will demonstrate that , through optimizing for both discriminative features and correlation between views , we can improve classification accuracy for three real world scenarios . CCA is an unsupervised method but has been applied to many discriminative tasks ( Kan et al. , 2015 ; Sargin et al. , 2007 ; Arora & Livescu , 2012 ) . While some of the correlated CCA features are useful for discriminative tasks , many represent properties that are of no use for classification and obscure correlated information that is beneficial . This problem is magnified with recent non-linear extensions of CCA that use deep learning to make significant strides in improving correlation ( Andrew et al. , 2013 ; Wang et al. , 2015a ; 2016 ; Chang et al. , 2018 ) but often at the expense of discriminative capability ( cf . §5.1 ) . Therefore , we present Task-Optimal CCA ( TOCCA ) , a new deep learning technique to project the data from two views to a shared space that is also discriminative ( Fig . 1 ) . Implementing a task-optimal variant of CCA required a fundamental change in formulation . We show that the CCA objective can equivalently be expressed as an ` 2 distance minimization in the shared space plus an orthogonality constraint . Orthogonality constraints help regularize neural networks ( NNs ) ( Huang et al. , 2018 ) ; we present three techniques to accomplish this . While our method is derived from CCA , by manipulating the orthogonality constraints , we obtain deep CCA approaches that compute a shared latent space that is also discriminative . Our family of solutions for supervised CCA required a crucial and non-trivial change in formulation . We demonstrate the effectiveness and versatility of our model for three different tasks : 1 ) crossview classification on a variation of MNIST ( LeCun , 1998 ) , 2 ) regularization when two views are available for training but only one at test time on a cancer imaging and genomic data set with only 1,000 samples , and 3 ) semi-supervised representation learning to improve speech recognition . All experiments showed a significant improvement in accuracy over previous state-of-the-art . In addition , our approach is more robust in the small sample size regime than alternative methods . Overall , our experiments on real data show the effectiveness of our method in learning a shared space that is more discriminative than previous methods for a variety of practical problems . 2 RELATED WORK . CCA was initially used for unsupervised data analysis to gain insights into components shared by two sources ( Andrew et al. , 2013 ; Wang et al. , 2015a ; 2016 ) . CCA has also been used to compute a shared latent space for cross-view classification ( Kan et al. , 2015 ; Wang et al. , 2015a ; Chandar et al. , 2016 ; Chang et al. , 2018 ) , for representation learning on multiple views that are then joined for prediction ( Sargin et al. , 2007 ; Dorfer et al. , 2016b ) , and for classification from a single view when a second view is available during training ( Arora & Livescu , 2012 ) . Recent non-linear extensions of CCA implemented via NNs make significant improvements in correlation ( Andrew et al. , 2013 ; Wang et al. , 2015a ; 2016 ; Chang et al. , 2018 ) but with little focus on discriminative capability . Most prior work that boosts the discriminative capability of CCA is linear only ( Lee et al. , 2015 ; Singanamalli et al. , 2014 ; Duan et al. , 2016 ) . More recent work using NNs still remains limited in that it optimizes discriminative capability for an intermediate representation rather than the final CCA projection ( Dorfer et al. , 2016b ) , or optimizes the CCA objective only during pre-training , not while training the task objective ( Dorfer et al. , 2018 ) . We advocate to jointly optimize CCA and a discriminative objective by computing the CCA projection within a network layer while applying a task-driven operation such as classification . Experimental results show that our method significantly improves upon previous work ( Dorfer et al. , 2016b ; 2018 ) due to its focus on both the shared latent space and a task-driven objective . The latter is particularly important on small training set sizes . While alternative approaches to multi-view learning via CCA exist , they typically focus on a reconstruction objective . That is , they transform the input into a shared space such that the input could be reconstructed – either individually or reconstructing one view from the other . Variations of coupled dictionary learning ( Shekhar et al. , 2014 ; Xu et al. , 2015 ; Cha et al. , 2015 ; Bahrampour et al. , 2015 ) and autoencoders ( Wang et al. , 2015a ; Bhatt et al. , 2017 ) have been used in this context . CCA-based objectives , such as the model used in this work , instead learn a transformation to a shared space without the need for reconstructing the input . This task may be easier and sufficient in producing a representation for multi-view classification ( Wang et al. , 2015a ) . 3 BACKGROUND . We first introduce CCA and present our task-driven approach in §4 . Linear and non-linear CCA are unsupervised and find the shared signal between a pair of data sources , by maximizing the sum correlation between corresponding projections . Let X1 ∈ Rd1×n and X2 ∈ Rd2×n be meancentered input data from two different views with n samples and d1 , d2 features , respectively . CCA . The objective is to maximize the correlation between a1 = w > 1 X1 and a2 = w > 2 X2 , where w1 and w2 are projection vectors ( Hotelling , 1936 ) . The first canonical directions are found via arg max w1 , w2 corr ( w > 1 X1 , w > 2 X2 ) and subsequent projections are found by maximizing the same correlation but in orthogonal directions . Combining the projection vectors into matrices W1 = [ w ( 1 ) 1 , . . . , w ( k ) 1 ] and W2 = [ w ( 1 ) 2 , . . . , w ( k ) 2 ] ( k ≤ min ( d1 , d2 ) ) , CCA can be reformulated as a trace maximization under orthonormality constraints on the projections , i.e. , arg max W1 , W2 tr ( W > 1 Σ12W2 ) s.t . W > 1 Σ1W1 = W > 2 Σ2W2 = I ( 1 ) for covariance matrices Σ1 = X1XT1 , Σ2 = X2X T 2 , and cross-covariance matrix Σ12 = X1X T 2 . Let T = Σ−1/21 Σ12Σ −1/2 2 and its singular value decomposition ( SVD ) be T = U1diag ( σ ) U > 2 with singular values σ = [ σ1 , . . . , σmin ( d1 , d2 ) ] in descending order . W1 and W2 are computed from the top k singular vectors of T as W1 = Σ −1/2 1 U ( 1 : k ) 1 and W2 = Σ −1/2 2 U ( 1 : k ) 2 where U ( 1 : k ) denotes the k first columns of matrix U . The sum correlation in the projection space is equivalent to k∑ i=1 corr ( ( w ( i ) 1 ) > X1 , ( w ( i ) 2 ) > X2 ) = k∑ i=1 σ2i , ( 2 ) i.e. , the sum of the top k singular values . A regularized variation of CCA ( RCCA ) ensures that the covariance matrices are positive definite by computing the covariance matrices as Σ̂1 = 1 n−1X1X > 1 + rI and Σ̂2 = 1 n−1X2X > 2 + rI , for regularization parameter r > 0 and identity matrix I ( Bilenko & Gallant , 2016 ) . DCCA . Deep CCA adds non-linear projections to CCA by non-linearly mapping the input via a multilayer perceptron ( MLP ) . In particular , inputs X1 and X2 are mapped via non-linear functions f1 and f2 , parameterized by θ1 and θ2 , resulting in activations A1 = f1 ( X1 ; θ1 ) and A2 = f2 ( X2 ; θ2 ) ( assumed to be mean centered ) ( Andrew et al. , 2013 ) . When implemented by a NN , A1 and A2 are the output activations of the final layer with do features . Fig . 2 ( a ) shows the network structure . DCCA optimizes the same objective as CCA ( equation 1 ) but using activations A1 and A2 . Regularized covariance matrices are computed accordingly and the solution for W1 and W2 can be computed using SVD just as with linear CCA . When k = do ( i.e. , the number of CCA components is equal to the number of features in A1 and A2 ) , optimizing the sum correlation in the projection space ( equation 2 ) is equivalent to optimizing the following matrix trace norm objective ( TNO ) LTNO ( A1 , A2 ) = ‖T‖tr = tr ( T > T ) 1/2 , where T = Σ−1/21 Σ12Σ −1/2 2 as in CCA ( Andrew et al. , 2013 ) . DCCA optimizes this objective directly , without a need to compute the CCA projection within the network . The TNO is optimized first , followed by a linear CCA operation before downstream tasks like classification are performed . This formulation does not allow for combining directly with a supervised term . SoftCCA . While DCCA enforces orthogonality constraints on projections W > 1 A1 and W > 2 A2 , SoftCCA relaxes them using regularization ( Chang et al. , 2018 ) . Final projection matrices W1 and W2 are integrated into f1 and f2 as the top network layer . The trace objective for DCCA in equation 1 can be rewritten as minimizing the ` 2 distance between the projections when each feature in A1 and A2 is normalized to a unit variance ( Li et al. , 2003 ) , leading to1 L ` 2 dist ( A1 , A2 ) = ‖A1− A2‖2F . Regularization in SoftCCA penalizes the off-diagonal elements of the covariance matrix 1We use this ` 2 distance objective in our formulation . Σ , using a running average computed over batches as Σ̂ and a loss of LDecorr ( A ) = ∑do i6=i |Σ̂i , j | . Overall , the SoftCCA loss takes the form L ` 2 dist ( A1 , A2 ) + λ ( LDecorr ( A1 ) + LDecorr ( A2 ) ) . Supervised CCA methods . CCA , DCCA , and SoftCCA are all unsupervised methods to learn a projection to a shared space in which the data is maximally correlated . Although these methods have shown utility for discriminative tasks , a CCA decomposition may not be optimal for classification because features that are correlated may not be discriminative . Our experiments will show that maximizing the correlation objective too much can degrade performance on discriminative tasks . CCA has previously been extended to supervised settings in three ways : 1 ) with methods that are linear only ( Singanamalli et al. , 2014 ; Lee et al. , 2015 ; Kan et al. , 2015 ; Duan et al. , 2016 ) , 2 ) by maximizing the total correlation between each view and the training labels in addition to each pair of views ( Lee et al. , 2015 ; Singanamalli et al. , 2014 ) , and 3 ) with Linear Discriminant Analysis ( LDA ) -style approaches to encourage class separation ( Kan et al. , 2015 ; Dorfer et al. , 2016b ; Elmadany et al. , 2016 ) .2 LDA approaches to supervision are generative rather than discriminative . Importantly , we will show in §5.3 that encouraging class separation with an LDA-style objective performs significantly inferior to a softmax . Further , Dorfer et al . ( 2016b ) did not apply LDA to the shared space itself but to the NN layer below it , and Elmadany et al . ( 2016 ) did not validate the shared space created , only its use in multi-view classification using both views for training and test . Dorfer et . al ’ s CCA Layer ( CCAL ) is the closest to our method . It optimizes a task loss operating on a CCA projection ; however , the CCA objective itself is only optimized during pre-training , not in an end-to-end manner ( Dorfer et al. , 2018 ) . Further , their goal is retrieval with a pairwise rank loss , not classification . Instead of computing the CCA projection explicitly within the network , we optimize the non-linear mapping into the shared space together with the task objective , requiring a fundamental change in formulation . We optimize for the shared space with the ` 2 distance between activations ( similar to SoftCCA ) and propose three different ways to apply the orthogonality constraints of CCA .

Summary: This paper addresses the problem of jointly performing CCA with task labeling. The problem is timely and important as it is challenging to perform CCA jointly with the task classification (see below) and hence previous work typically perform this in a pipeline - that is, first projecting the data using a pre-trained CCA and then training a task classifier using the projected representation. As the authors note, this may be problematic as CCA may delete important information that is relevant for the classification, if training is not done jointly.