Deep Multi-View Learning via Task-Optimal CCA
Authors: Heather D. Couture
Roland Kwitt
J.S. Marron
Melissa Troester
Charles M. Perou
Marc Niethammer
Venue:  (2020)
Keywords: multi-view
components analysis
CCA
representation learning
deep learning

Abstract: Canonical Correlation Analysis (CCA) is widely used for multimodal data analysis and, more recently, for discriminative tasks such as multi-view learning; however, it makes no use of class labels.  Recent CCA methods have started to address this weakness but are limited in that they do not simultaneously optimize the CCA projection for discrimination and the CCA projection itself, or they are linear only. We address these deficiencies by simultaneously optimizing a CCA-based and a task objective in an end-to-end manner. Together, these two objectives learn a non-linear CCA projection to a shared latent space that is highly correlated and discriminative. Our method shows a significant improvement over previous state-of-the-art (including deep supervised approaches) for cross-view classification (8.5% increase), regularization with a second view during training when only one view is available at test time (2.2-3.2%), and semi-supervised learning (15%) on real data.

Under review as a conference paper at ICLR 2020
DEEP MULTI- VIEW LEARNING VIA TASK-OPTIMAL CCA
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-view learning seeks to form better models by making use of multiple feature sets representing the same samples. Exploiting feature correlations during train- ing can enable better models. The traditional method for computing correlations between feature sets is Canonical Correlation Analysis (CCA), which finds linear projections that maximize correlation between feature vectors, essentially com- puting a shared embedding for two views of data. More recently, CCA has been used for multi-view discriminative tasks; however, CCA makes no use of class labels. Recent CCA methods have started address this weakness but are limited in that they do not simultaneously optimize the CCA projection for discrimination and the CCA projection itself, or they are linear only. We address these defi- ciencies by simultancously optimizing CCA-based and a task objective in an end-to-end manner. Together, these two objectives learn a non-linear CCA pro- jection to a shared latent space that is highly correlated and discriminative. Our method shows a significant improvement over previous state-of-the-art (including deep supervised approaches) for cross-view classification (8.5% increase), regu- larization with a second view during training when only one view is available at test time (2.2-3.2%), and supervise learning (15%) real data.
INTRODUCTION
Parallel modalities of data are increasingly common in variety of applications, including images and text, audio and video, parallel texts of different languages, and a variety of medical imaging and omics modalities for each patient. Each view provides essential information for classification and, when used together, can form more accurate model. This is especially important for difficult discriminative tasks such as those with small training set size. Canonical Correlation Analysis (CCA) is the most common method for computing a shared representation from two views of data by computing space in which they are maximally correlated (Hotelling 1936 Bie et al.) 2005). In this paper we will demonstrate that, through optimizing for both discriminative features and correlation between views, we can improve classification accuracy for three real world scenarios.
CCA is an unsupervised method but has been applied to many discriminative tasks (Kan et al. 2015 Sargin et al., 2007; Arora & Livescu 2012). While some of the correlated CCA features useful for discriminative tasks, many represent properties that are of no use for classification and obscure correlated information that is beneficial. This problem is magnified with recent non-linear extensions of CCA that use deep learning to make significant strides in improving correlation (Andrew et al., 2013 Wang et al.] 2015a 2016; Chang et al., 2018) but often at the expense of discriminative capability (cf. 1..1). Therefore, we present Optimal CCA (TOCCA), new deep learning technique to project the data from two views to a shared space that is also discriminative (Fig. [T.
Implementing task-optimal variant of CCA required fundamental change in formulation. We show that the CCA objective can equivalently be expressed as an f2 distance minimization in the shared space plus orthogonality constraint. Orthogonality constraints help regularize neural networks (NNs) (Huang et al. 2018);; we present three techniques to accomplish this. While our method is derived from CCA, by manipulating the orthogonality constraints, we obtain deep CCA approaches that compute a shared latent space that is also discriminative.
Our family of solutions for supervised CCA required crucial and non-trivial change in formulation. We demonstrate the effectiveness and versatility of our model for three different tasks: 1) cross- view classification on variation of MNIST (LeCun 1998), 2) regularization when two views

---

Under review as a conference paper at ICLR 2020
Goal: shared embedding that is also discriminative
Solution: multi-task NN
Applications:
Cross-view daassffication
Discriminative
Train: y y [(A.)
X, A,
X A,
Test: y ((A.)
Correlated
Multi-view regularization
during training
Train: y A,) Test: y y f(A.)
A, 10... O0
00
O0 A
NN
NN
Multi-view prediction
Train: AIA,A.) Test (IA,A.)
Class
Class 2
Class
00.. 00
00.. X
Figure 1: Our goal with Task-Optimal CCA is to compute shared space that is also discriminative. We do this by using NNs to compute an embedding for each view while simultaneously optimizing for correlation in the embedded space and task-optimal objective. This setup is beneficial for three scenarios: 1) training a classifier with the embedding from one view and testing with the embedding of the other view (55.1). 2) when two views are available for training but only one at test time (5..2) and 3) when both views used for both training and testing ($5.3). The embeddings for views X1 and X2 are represented by Ai and A2, respectively. classifier f(A) then predicts the class each sample. order to compare with unsupervised variants of CCA, the classifier f may be computed subsequent to the CCA emhedding.
available for training but only one at test time on cancer imaging and genomic data set with only 1,000 samples, and 3) semi-supervised representation learning to improve speech recognition. All experiments showed a significant improvement in accuracy over previous state-of-the-an arrt In addition, our approach is more robust in the small sample size regime than alternative methods. Overall, our experiments on real data show the effectiveness of our method in learning a shared space that is more discriminative than previous methods for variety of practical problems.
2 RELATED WORK
CCA was initially used for unsupervised data analysis gain insights into components shared by two sources (Andrew et al.
2013) Wang et al., 2015a 2016). CCA has also been used to compute a shared latent space for cross-view classification (Kan et al., 2015 Wang et al., 2015a Chandar al. 2016/Chang et al. 2018). for representation learning on multiple views that are then joined for prediction (Sargin et al. 2007 Dorfer et al.] 2016b), and for classification from a single view when second view is available during training (Arora & Livescu 2012). Recent non-linear extensions of CCA implemented via NNs make significant improvements in correlation (Andrew et al., 2013
Wang et al.
2015a 2016 Chang al. (2018) but with little focus on discriminative capability.
Most prior work that boosts the discriminative capability of CCA is linear only (Lee et al. 2015 Singanamalli et al., 2014 Duan al., (016). More recent work using NNs still remains limited in that it optimizes discriminative capability for an intermediate representation rather than the final CCA projection (Dorfer et al. 2016b). or optimizes the CCA objective only during pre-training, not while training the task objective (Dorfer et al. 2018). We advocate to jointly optimize CCA and discriminative objective by computing the CCA projection within network layer while applying task-driven operation such as classification. Experimental results show that our method significantly improves upon previous work (Dorfer et al., 2016b 2018) due to its focus on both the shared latent space and task-driven objective. The latter is particularly important small training set sizes.
While alternative approaches to multi-view learning via CCA exist, they typically focus on a recon- struction objective. That is, they transform the input into shared space such that the input could be reconstructed either individually or reconstructing one view from the other. Variations of coupled dictionary learning (Shekhar et al. [2014] Xu et al. 2015 Cha al.) 2015 Bahrampour et al. 2015) and autoencoders (Wang et al. 2215aa] Bhatt et al. (017) have been used in this context. SCA--asse objectives, such the model used in this work, instead leam transformation to a shared space without the need for reconstructing the input. This task may be easier and sufficient in producing a representation for multi-view classification (Wang et al. 2015a).
2

---

Under review as a conference paper at ICLR 2020
3 BACKGROUND
We first introduce CCA and present our task-driven approach in + Linear and non-linear CCA are unsupervised and find the shared signal between pair of data sources, by maximizing the sum correlation between corresponding projections. Let X E R2 and X2 E Rddax be mean- centered input data from two different views with n samples and d1, d2 features, respectively.
CCA. The objective is to maximize the correlation between а₃ = w X1 and 22 w X2, where Wi and W2 are projection vectors (Hotelling 1936). The first canonical directions are found via
arg max coIr (w] X1, W. X²)
W1.W2
and subsequent projections are found by maximizing the same correlation but in orthogonal di- rections. Combining the projection vectors into matrices W1 = [w] (2) "i......"" wi (k) and W2 =
[w2 1 (k S min(d1, d2)). CCA can be reformulated as trace maximization under or- thonormality constraints on the projections, i.e.,
arg max ( W) W.
s.t. W, E, W = W E, =I
(1)
for covariance matrices Σ = X,X, Σ X2XY, and cross-covariance matrix Σ = X Σ3 1/2 and its singular value decomposition (SVD) be T U, ) with singular values a = [o1..... din)dd in descending order. WN and W₂ are computed from
Let T 1/3 E
the top & singular vectors of T as W1 = Σ {(: and W2 Σ u where U( denotes the & first columns of matrix U. The sum correlation in the projection space is equivalent to
k
COTr (orr((!!') (w! T
( X1, (w2
(1)
i.e, the sum of the top i singular values. A regularized variation of CCA (RCCA) ensures that the covariance matrices are positive definite by computing the covariance matrices as Σ X + and Eg X +rI, for regularization parameter and identity matrix I Bilenno & Gallant 2016).
DCCA, Deep CCA adds non-linear projections to CCA by non-linearly mapping the input via a mul- tilayer perceptron (MLP). In particular, inputs X, and X₂ are mapped via non-linear functions fi and f2. parameterized by 0₁ and 02. resulting in activations A₁ = (X( O1) and A₂ = fe(X;; 02) (assumed to be mean centered) (Andrew et al. 2013). When implemented by NN, A and A are the output activations of the final layer with d, features. Fig. ((a) shows the network structure. DCCA optimizes the same objective CCA (equation I but using activations A₁ and A2. Reg- ularized covariance matrices are computed accordingly and the solution for W1 and W₂ can be computed using SVD just as with linear CCA. When do (i.e., the number of CCA components is equal to the number of features in A and A2), optimizing the sum correlation in the projection space (equation? 2 is equivalent to optimizing the following matrix trace norm objective (TNO)
CIoO (A1, A2) T (T T) 1/2
in CCA (Andrew et al. 2013). DCCA optimizes this objective directly, without a need to compute the CCA projection within the network. The TNO is optimized first, followed by a linear CCA operation before downstream tasks like classification performed. This formulation does not allow for combining directly with supervised term.
where T =
SoftCCA. While DCCA enforces orthogonality constraints on projections WT Ai and W. Ag. SoftCCA relaxes them using regularization (Chang et al.) 2018). Final projection matrices Wi and W₂ are integrated into J1 and /2 the top network layer. The trace objective for DCCA in equation cann be rewritten as minimizing the & distance between the projections when each feature in A, and A2 is normalized to a unit variance (Li al. 2003), leading to Ci, (A. Azll? Regularization in SoftCCA penalizes the off dia-onaa elements of the covariance matrix
||A
We use this & distance objective formulation.
3

---

Under review as a conference paper at ICLR 2020
(a) DCCA
(c) TOCCA-W (Ours)
(d) TOOCA-SD (Ours)
TNO
Task Icee
Task loss
Impross dscriminability
....00
O0O
O0O Y
ftas
MIP
O0
00
Loss
B
6 Loss
00
Decorr
Loss
Decorr
Whitening
Whitoning
Decorr
Decorr
00
00.+00
O
OO...00
00
MLP
MIP
00
C 00.000
00.+.00
O0.+000
00...00
O0 000
Xi
X
(b) SoftCCA
Max. sum correlation (equiv. to ty loss)
2 Such that projections are orthogonal
Figure 2: Deep CCA architectures: (a) DCCA maximizes the sum correlation in projection space by optimizing
equivalent loss, the trace objective (TNO) (Andrew al. 2013); (b) SofCCCA relaxes the orthogonality constraints by regularizing with soft decorrelation (Decorr) and optimizes the & distance the projection space (equivalent to sum correlation with activations normalized to unit variance) (Chang et al. (018). Our TOCCA methods add a task loss and apply CCA orthogonality constraints hy regularizing in two ways: (c) TOCCA-W uses whitening and (d) TOCCA-SI SD uses Decorr. The third method that we propose, TOCCA-NI,, simply removes the Decorr components of TOCCA-SI..
E, using running average computed over batches E and loss of Leccor (A) Eips Eigh. Overall, the SoftCCA loss takes the form
Lia dist [A1, A2) + X( + De (Az))
Supervised CCA methods. CCA, DCCA, and SoftCCA are all unsupervised methods to learn projection to a shared space in which the data is maximally correlated. Although these methods have shown utility for discriminative tasks, a CCA decomposition may not optimal for classification because features that correlated may not be discriminative. Our experiments will show that maximizing the correlation objective too much can degrade performance on discriminative tasks.
CCA has previously been extended to supervised settings in three ways: 1) with methods that linear only (Singanamalli et al. 2014; Lee et al. 2015 Kan et al.) 2015 Duan et al. 2016), 2) by maximizing the total correlation between each view and the training labels in addition each pair of views (Lee et al.] 2015 Singanamalli et al.] 2014), and 3) with Linear Discriminant Analysis (LDA))-tyle approaches to encourage class separation (Kan et al.) 2015 Dorfer al., 2016b El- madany et al. 2016 2 LDA approaches to supervision are generative rather than discriminative. Importantly, we will show in (5.3 that encouraging class separation with an LDA-style objective performs significantly inferior to a softmax. Further, Dorfer et al.] (2016b) did not apply LDA to the shared space itself but to the NN layer below and Elmadany et al. (2016) did not validate the shared space created, only its use in multi-view classification using both views for training and test.
Dorfer et. al's CCA Layer (CCAL) the closest our method. optimizes task loss operating on CCA projection; however, the CCA objective itself is only optimized during pre-training, not an end-to-end manner (Dorfer et al. 2018). Further, their goal is retrieval with a pairwise rank loss, not classification. Instead of computing the CCA projection explicitly within the network, optimize the non-linear mapping into the shared space together with the task objective, requiring fundamental change in formulation. We optimize for the shared space with the & distance be- tween activations (similar to SoftCCA) and propose three different ways to apply the orthogonality constraints of CCA.
4 TASK-OPTIMG CCA (TOCCA)
To compute shared latent space that is also discriminative, we reformulate DCCA to add task- driven to the optimization objective. The CCA component finds features that correlated
*Gatto & Dos Santos 2017) use similar technique with LDA but apply it as convolutional filter single view; it is not multi-viww method.
4

---

Under review as a conference paper at ICLR 2020
between views, while the task component ensures that they also discriminative. This model can be used for representation learning on multiple views before joining representations for prediction (Sargin et al., 2007; Dorfer et al., 201 and for classification when two views are available for training but only one at test time (Arora & Liveseu 2012). In { we demonstrate both use cases on real data. Our methods and related NN models from the literature are summarized in Tab. A2 Fig.
shows schematic diagrams,
Challenges and solutions. While DCCA optimizes the sum correlation with an equivalent loss function (TNO), the CCA projection itself is computed only after optimization. Hence, the projec- tions cannot be used to optimize another task simultaneously. The main challenge in developing a task-optimal form of deep CCA that discriminates based on the CCA projection is in computing this projection within the network - a necessary step to enable simultaneous training of both objec- tives. We tackle this by focusing on the two components of DCCA: maximizing the sum correlation between activations A and A2 and enforcing orthonormality constraints within A 1 and Ag. We achieve both by transforming the CCA objective and present three methods that progressively relax the orthogonality constraints.
We further improve upon DCCA by enabling mini-batch computations for improved flexibility and test performance. DCCA was developed for large batches because correlation is not separable across batches. While large batch implementations of stochastic gradient optimization can increase com- putational efficiency via parallelism, small batch training provides more up-to-date gradient caleu- lations, allowing a wider range of learning rates and improving test accuracy (Masters & Luschil 2018). We reformulate the correlation objective as the & distance (following SoftCCA), enabling separability across batches. We ensure normalization to one via batch normalization without the scale and shift parameters (loffe & Szegedy 2015). [Wan et al. (2016) also developed a stochastic mini-batch solution to DCCA but handled the orthonormality constraints in a different way (dis- cussed below).
Task-driven objective. First, we apply non-lineat functions fi and fz with parameters O (via MLPs) to each view X1 and X2, i.e., A fi (Xx;; 01) and A2 = f2((2x, (X2; 02). Second, task- specific function (A; (A; 6ms.. operates on the outputs A₁ and Ag. particular, fi and fz are optimized so that the E2 distance between A₁ and A₂ is minimized; therefore, fusk can be trained to operate on both inputs Ai and A2. We combine CCA and task-driven objectives as a weighted sum with hyperparameter for tuning. This model is flexible, in that the task-driven goal can be used for classification (Krizhevsky et al., (2012; Dorfer et al.] (2066a), regression (Katzman et al., 2016), clustering (Caron et al. (018), or any other task. Other prior attempts integrate classifier into deep CCA only used LDA (Kan et al. 2015 Dorfer et al., 2016b; Elmadany et al. 2016). See Tab. A2 for overview.
Orthogonality constraints. The remaining complications for mini-batch optimization are the or- thogonality constraints, for which we propose three solutions, each handling the orthogonality con- straints of CCA in different way: whitening, soft decorrelation, and no decorrelation.
1) Whitening (TOCCA-W). CCA applies orthogonality constraints to A₁ and A2- We accomplish this with a linear whitening transformation that transforms the activations such that their covari- ance becomes the identity matrix, i.c., features are uncorrelated. Decorrelated Batch Normalization (DBN) has previously been used to regularize deep models by decorrelating features (Huang et al.) 2018) and inspired our solution. In particular, we apply a transformation orthonormal, i.e., BB =I I.
UA to make B
We use a Zero-phase Component Analysis (ZCA) whitening transform composed of three steps: rotate the data to decorrelate it, rescale each axis, and rotate back to the original space. Each trans- formation learned from the data. Any matrix UeRdd.xd Xdo satisfying U --1 whitens the data, where Σ denotes the covariance matrix of A. As U is only defined up to rotation, it is not unique. PCA whitening follows the first two steps and uses the cigendecomposition of Σ: UPCA = A T for = diag(A)..... .., Ad,) and = -Vv...... Va.). where (A;, v:) are the eigenvalue, eigenvector pairs of Σ. As PCA whitening suffers from stochastic axis swapping, neu- rons are not stable between batches (Huang et al.) 2018). ZCA whitening uses the transformation UZCA- -1/2// in which PCA whitening is first applied, followed by a rotation back to the original space. Adding the rotation V brings the whitened data B as close as possible to the original data A (Kessy et al. 2015).
5

---

Under review as a conference paper at ICLR 2020
Computation of UZCA is clearly depend on E. While Huang al. (2018) used running average of UZCA over batches, we apply this stochastic approximation to for each view using the update (A)) = (k) (1-o) for batch l where Σ the covariance matrix for the current batch and a E (0, 1) the momentum. We then compute the ZCA transformation from (k) to do whitening as B = fzcA (A) ZCA
AcAA. A. At test time, U(A) from the last training batch is used. Algorithm A1
describes ZCA whitening in greater detail. In summary, TOCCA W integrates both the correlation and task-driven objectives, with decorrelation performed by whitening, into
Lusk ( (BI),, + Lunck (s (B2); Y) + À Lt₂ dist (B1, 11,Ba) B2)
where B, and B2 are whitened outputs of A and 2,, respectively, and Y is the class labels. This is a novel approach to integrating the orthogonality constraints of CCA into a NN as it is the first to use ZCA whitening in this manner. WWng et al. (2016)'s stochastic mini-batch solution to DCCA used nonlinear orthogonal iterations and does not state what type of whitening operation was used.
2) Soft decorrelation (TOCCA-SD), While fully independent components may be beneficial in regularizing NNs on some data sets, a softer decorrelation may be more suitable on others. this second formulation relax the orthogonality constraints using regularization, following the Decor loss of SoftCCA (Chang et al. 2018). The loss function for this formulation is
A2) (fukk (A1), Y) t (A2). Y) +) Ce, dist A1, A2) (CDeorr (A1) + (A2))
While this solution is based on SoftCCA, our experiments ponent is essential when using the model for classification.
(55) will demonstrate that the task com-
3) No decorrelation (TOCCA-ND). When CCA is used in an unsupervised manner, some form of orthogonality constraint or decorrelation is necessary ensure that and f2 do not simply produce multiple copies of the same feature. While this result could maximize the sum correlation, is not helpful in capturing useful projections. In the task-driven setting, the discriminative term ensures that the features in fi and /2 are not replicates of the same information. TOCCA-ND therefore removes the decorrelation term entirely, forming the simpler objective
Liskk (usk (A1), Y) + Crsk (auk (A2), Y) + ALi, dist (A1, A2)
These three models allow testing whether whitening or decorrelation benefit a task-driven model.
Computational complexity. Due to the eigendecomposition, TOCCA -W has a complexity of (d)) compared to (de) for TOCCA-SD. with respect to output dimension do- However, d, is typically small (< 100) and this extra computation is only performed once per batch. The difference in runtime less than 6,5% for a batch size of 100 9.4% for batch size of 30 (Tab. A4..
Summary. All three variants are motivated by adding task-driven component to deep CCA. TOCCA-ND is the most relaxed and directly attempts to obtain identical latent representations. Ex- periments will show that whitening (TOCCA-W) and soft decorrelation (TOCCA-SD) provide a ben- eficial regularization. Further, since the & distance that we optimize was shown to equivalent to the sum correlation (ef. } SoftCCA paragraph), all three TOCCA models maintain the goals of CCA, just with different relaxations of the orthogonality constraints. Our method is the first to si- multancously optimize for CCA and discriminative task with end-to-ent training. See Tab, A2 for an overview.
EXPERIMENTS
We validated our methods on three different data sets: MNIST handwritten digits, the Carolina Breast Cancer Study (CBCS) using imaging and genomic features, and speech data from the Wis- consin X-ray Microbeam Database (XRMB). Our experiments show the utility of our methods for 1) cross-view classification, 2) regularization with second view during training when only view is available at test time, and 3) representation learning on multiple views that are joined for prediction.
Implementation Each layer of our network consists of fully connected layer, followed by ReLU activation and batch normalization (Toffe & Szegedy 2015). Our implementations of DCCA, SoftCCA, and Joint DCCA/DeepL.A (Dorfer et al. 2016b) also use ReLU activation and batch
Cooee submitted with this paper and will also he available publicly on GitHub after the review period.

---

Under review as a conference paper at ICLR 2020
normalization. We modified CaaL-Crank (Dorfer et al. 2018) to use softmax function and cross- entropy loss for classification, instead of a pairwise ranking loss for retrieval, referring to this modi- fication as CCAL-fee- We used the Nadam optimizer and tuned hyperparameters via random search; settings and ranges are specified in Tab. A3 The same hyperparameter tuning procedure was used for our methods and those we compare with. We used Keras with the Theano backend and an Nvidia GeForce GTX 1080 Ti.
validation set
The following experiments compare our methods with two linear methods (CCA and RCCA), two unsupervised deep methods (DCCA and SoftCCA) and two supervised deep methods (Joint DCCA/DeepLDA and CC Many other variants exist ( but the ones we selected are the current state-of-the-art in each of these classes. We did not run a direct comparison with Wang jet al.] (2015a) as Chang et al.] (2018) already showed that SofCCCA superior. We chose Joint DCCA/DeepL.DA to represent supervised LDA-style CCA methods rather than comparing with all methods in this group (Kan et al. 2015 Elmadany al. 2016
5.1 CROSS- VIEW CLASSIFICATION ON MNIST DIGITS
We formed a multi-view data set from the MNIST handwritten digit data set (LeCun 1998). Fol- lowing Andrew al. (2013), we split each 28 28 image in half horizontally, creating left and right views that are each 14 x 28 pixels. All images were flattened into a vector with 392 features. The full data set consists of 60k training images and 10k test images. We used random set of up to 50k for training and the remaining training images for validation. We used the full 10k image test set.
In order to validate both the discriminativeness of the embedding and the success in finding a shared space, we studied performance on view classification. We evaluated cross-view classification aceuracy by first computing the projection for each view, then we trained a linear SVM on one view's projection, and finally we used the other view's projection at test time. While the task-driven methods presented in this work learn a classifier within the model, this test setup enables a fair comparison with the unsupervised CCA variants and validates the discriminativity of the features learned. It is also the standard method in the literature to test CCA methods for classification. Notably, using the built-in softmax classifier (not shown) performed similarly the SVM, as much of the power of our methods comes from the representation learning part. We do not compare with simple supervised NN because this setup does not learn the shared space necessary for cross-view classification. We report results averaged over five randomly selected training/validation sets; the test set always remained the same.
Correlation vs. classification accuracy We first demonstrate the importance of adding a task- driven component to DCCA by showing that maximizing the sum correlation between views is not sufficient. Fig. B (lefi) shows the sum correlation vs. cress-view classification accuracy across many different hyperparameter settings for DCCA (Andrew et al,, 2013). SofCCCA (Chang et al.) 2018), and TOCCA. We used 50 components for each; thus, the maximum sum correlation was 50. The sum correlation was measured after applying linear CCA to ensure that components were indepen- dent. With DCCA a larger correlation tended to produce a larger classification accuracy, but there was still a large variance in classification accuracy amongst hyperparameter settings that produced similar sum correlation. For example, with the two farthest right points in the plot (colored red), their classification accuracy differs by 10%, and they are not even the points with the best classi- fication accuracy (colored purple). The pattern is different for SoftCCA. There was an increase in classification accuracy as sum correlation increased but only up to a point. For higher sum corre- lations, the classification accuracy varied even more from 20% to 80%. Further experiments (not shown) have indicated that when the sole objective is correlation, some of the projection directions are simply not discriminative, particularly when there are a large number of classes. Hence, optimiz- ing for sum correlation alone does not guarantee discriminative model. TOCCA-W and TOCCA-SD show much greater classification accuracy across a wide range of correlations and, overall, the best accuracy when correlation is greatest.
Effeet of batch size. Fig. (right) plots the batch size vs. classification accuracy for a training set size of 10, 000. We tested batch sizes from 10 to 10,000; a batch size of 10 or 30 was best for all
'While Elmadany et al. 2016) ran experiments on MNIST, they used the embeddings from both views for training and test; hence, their results are not directly comparable to our cross-view classification results. When we did test multi-view classification MNIST, achieved 98.5% vs. their reported 97.2%.
7

---

Under review as a conference paper at ICLR 2020
CCA
SofCCCA Joint DCCA/DeepLDA
TOCCA-W TOCCA-SD
Conto with highest conelations Contig hightst accuracy
RCCA
DCCA
CCAL-Le
TOCCA-ND
10
0.85
08
0.8
Accuracy
0.6
Accuracy
9.6
Accuracy
0.91
0.4
024
0.2
0.80
DCCA
SoftCCA
00
0.
10
20
30
50
0.99
10
80
100
1,000
10,000
Sum correlation
Sum correlation
Batch size
1.0
10
0.87
Accuracy
Accuracy
Accuracy
0.6
0.0
0.
0.
0.4
TOCCA-W
TOCCA-SD
0%
10
20
30
40
50
0.0
10
20
30
40
50
300
1,000
10,000
50,00
Sum correlation
Sum correlation
Training size
Figure 3: Left: Sum correlation vs. cross-view classification accuracy (on MNIST) across different hyper- parameter settings on a training set size 10,000 for DCCA (Andrew et al. 2013). SofICCA (Chang et al.] 2018). TOCCA-W, and CD. For unsupervised methods (DCCA and SofiCCA), large correlations do not necessarily imply good accuracy. Right: The effect of batch size on classification accuracy for each TOCCA method on MNIST (training size of 10,000), and the effect of training set size on classification accuracy for each method. Our TOCCA variants uutf-erformed all others across training set sizes.
CCA
RCCA
DCCA
SofTCCA
CCAL-Los
TOCCA-i
variation of MNIST. Each method was used to compute projections for the two views (left and right sides the images) using 10,000 training examples. The plots show a visualization of the projection for the left view with each digit colored differently. TOCCA-SD and TOCCA-ND (not shown) produced similar results to TOCCA-W.
Figure 4: t-SNE plots for CCA methods
three variations of TOCCA. This is in line with previous work that found the best performance with batch size between 2 and 32 (Masters & Luschi 2018). We used a batch size of 32 in the remaining experiments MNIST.
Effect of training set size. We manipulated the training set size in order to study the robustness of methods. In particular, Fig. (right) shows the cross-view classification accuracy for training set sizes from n = 300 to 50,000. While we expected that performance would decrease for smaller training set sizes, some methods were more susceptible this degradation than others. The clas- sification accuracy with CCA dropped significantly for n 300 and 1,000, due to overfitting and instability issues related to the covariance and cross-covariance matrices. SoftCCA shows similar behavior (prior work (Chang et al. 2018) on this method did not test such small training set sizes).
Across all training set sizes, our TOCCA variations consistently exhibited good performance, e.g., increasing classification accuracy from 78.3% to 86.7% for = 1,000 and from 86. 1% to 94.6% for n 50,000 with TOCCA-SD. Increases accuracy over TOCCA-ND were small, indicating that the different decorrelation schemes have only a small effect on this data set; the task-driven component is the main reason for the success of our method. In particular, the classification accuracy with 1,000 did better than the unsupervised DCCA method on 10,000. Further, TOCCA with 72 = 300 did better than linear methods on n = 50,000, clearly showing the benefits of the proposed formulation. We also examined the CCA projections qualitatively via 2D t-SNE embedding (Van Der Maaten & Hinton 2008). Fig. shows the CCA projection of the left view for each method. As expected, the task-driven variant produced more clearly separated classes.
8

---

Under review as a conference paper at ICLR 2020
Table I: Classification accuracy for different methods of predicting Basal genomic subtype from images grade from gene expression. Linear SVM and DNN were trained on single view, while all other methods were trained with both views. By regularizing with the second view during training, all TOCCA variants improved classification accuracy. The standard error is in parentheses.
Method
Training data
Test data
Task
Accuracy
Method
Training data
Test data
Task
Accuracy
Linear SVM
Image only
Image
Basal 0.777 (0.003)
Linear SVM
GE only
GB
Grade 0,832 (0.012)
NN
Image only
Image
Basal 0.808 (0.006)
NN
GE only
Grade (0.012)
CCAL-C.
Image+GE
Image
Basul 0.807 (0,008)
CCAL-C
GE+ GE+image
GB
Grade 0,804 (0.022)
"OCCA-X
Image+GFF
Image
Basal 0.830 (0.006)
TOCCA-W
GF+image
GR
Grade 0.862 (0.013)
DOCCA-BD
Image+GE
Image
Basal 0.818 (0.006)
TOCCA-8D
GE+image
GE
Gride 0.856 (0.011)
-D
Image+GE
Image
Basal 0.816 (0.004)
TOCCA-ND ND
GE+image
GB
Grade 0.856 (0.011)
5.2 REGULARIZATION FOR CANCER CLASSIFICATION
In this experiment, we address the following question: Given two views available for training but only one at test time, does the additional view help to regularize the model?
We study this question using 1.003 patient samples with image and genomic data from CBC$ (Troester et al.] 2018). Images consisted of four cores per patient from a tissue microarray that was stained with hematoxylin and eosin. Image features were extracted using VGG16 backbone (Simonyan & Zisserman 2015). pre-trained on ImageNet, by taking the mean of the 512D output of the fourth set of cony. layers across the tissue region and further averaging across all core images for the same patient. For gene expression (GE), we used the set of 50 genes in the PAM50 array (Parker et al. 2009). The data set was randomly split into half for training and one quarter for validationntesting: we report the mean over eight cross- validation runs. Classification tasks included predicting 1) Basal vs. non- non-Basal genomic subtype using images, which is typically done from GE, and 2) predicting grade vs. 3 from GE, typically done from images. This is not multi-task classification setup; it a means for one view to stabilize the representation of the other. The first task is also a valuable clinical use case. Genomic analysis is expensive and not routinely performed, while histologic imaging is standard practice by pathologists for detecting cancer and assessing its aggressiveness. In working with our clinical collaborators, our goal has been to predict tumor subtypes from images- something that is too complex for pathologists. We hope that this will one day make tumor subtypes accessible more patients and improve treatment decisions. This experiment demonstrates that the second view of data can help regularize during training even if it not available for test patients.
We tested different classifier training methods when only one view was available at test time: a) a linear SVM trained on one view, b) a deep NN trained on one view using the same architecture as the lower layers of TOCCA, c) CCAL-Loe trained on both views, d) TOCCA trained on both views. Tab. lists the classification accuracy for each method and task. When predicting genomic subtype Basal from images, all our methods showed an improvement in classification accuracy; the best result was with TOCCA-W, which produced 2.2% improvement. For predicting grade from GE, all our methods again improved the accuracy by up to 3.2% with TOCCA- W. These results show that having additional information during training can boost performance at test time. Notably, this experiment used a static set of pre-trained VGG16 image features in order to assess the utility of the method. The network itself could be fine-tuned end-to-end with our TOCCA model, providing an easy opportunity for data augmentation and likely further improvements in classification accuracy.
5.3 SEMI-SUPERVISED LEARNING FOR SPEECI RECOGNITION
Our final experiments use speech data from XRMB, consisting of simultancously recorded acoustic and articulatory measurements. Prior work has shown that CCA-based algorithms can improve phonetic recognition (Wang et al.) 2015b a 2016 Dorfer et al. 2016b). The 45 speakers were split into 35 for training. 2 for validation, and 8 for testing - a total of 1, 1,429,236 samples for training, 85,297 for validation, and 111.3 14 for testing The acoustic features are 1 112D and the articulatory ones are 273D. We removed the per-speaker mean & variance for both views. Samples are annotated with one of 38 phonetic labels.
http://ttic.uchh.cag.

---

Under review as a conference paper at ICLR 2020
Our task this data set was representation learn- ing for multi-view prediction - that is, using both views of data to learn shared discrimina- tive representation. We trained each model us- ing both views and their labels. To test each
Table XRMB classification results.
Method
Task
Accuracy
Baseline
0.591
CCA
0,589
CCA model, we followed prior work and concate- nated the original input features from both views with the projections from both views. Due to the large training set size, we used Linear Discrimi- nant Analysis (LDA) classifier for efficiency. The same construction was used test time. This setup was used to assess whether a task-optimal DCCA model can improve discriminative power. We tested TOCCA with task-driven loss of LDA (Dorfer et al. 2016a) or softmax to demonstrate the flexibility of our model.
RCCA
0.588
DCCA
0,620
SofTCCA
0,635
Joint DCCA/DeepLDA
LDA
0,633
CCAL-4
Softmax
0.642
TOCCA-W
LDA
0,710
TOCCA-SD
LDA
0,677
TOCCA-ND
LDA
0.677
TOCCA-W
Softmax
0.795
VOCCA-8D
Softmax
0.785
TOCCA-ND
Softmax
0.785
We compared the discriminabilityy of a variety of methods to learn shared latent representation. Tab. 5 lists the classification re- sults with a baseline that used only the original input features for LDA. Although deep methods, i... DCCA and SofiCCA, improved upon the linear methods, all TOCCA variations significantly outper- formed previous state-of-the-at techniques. Using softmax consis- tently beat LDA by a large margin. TOCCA-SD and TOCCA-ND
Table 6: Semi-superviser classi- fication results on XRMB using TOCCA-1 W.
Labeled data
Accuracy
100%
0.795
30%
0.762
produced equivalent results as a weight of on the decorrelation term performed best. However, TOCCA- W showed the best result with an improvement of 15% over the best alternative method.
10%
0.745
3%
0.684
1%
0.637
TOCCA can also be used in semi-supervised manner when labels are available for only some samples. Tab, lists the results for TOCCA-W V in this setting. With 0% labeled data, the result would be similar DCCA. Notably, a large improvement over the unsupervised results in Tab. 5 is seen even with labels for only 10% of the training samples.
DISCUSSION
We proposed method to find shared latent space that is also discriminative by adding task- driven component to deep CCA while enabling end-to-end training. This required fundamental change in formulation because Deep CCA does not compute the embeddings directly it optimizes an equivalent objective; therefore, we could not simply add an additional term. Instead, we found an alternative formulation by replacing the CCA projection with lg distance minimization and orthogo- nality constraints on the activations, and we implemented this in three different ways. TOCCA-W W or TOCCA-SD performed the best, dependent the data set both of which include some means of decorrelation to provide a regularizing effect to the model and thereby outperforming TOCCA-ND.
TOCCA showed large improvements over state-of-the-art in cross-view classification accuracy on MNIST and significantly increased robustness to small training set size. On CBCS, TOCCA pro- vided regularizing effect when both views were available for training but only one at time. TOCCA also produced a large increase over state-of-te saate-of-the--ee of-the-art for multi-view representation learning on a much larger data set, XRMB. On this data set we also demonstrated semi-supervised approach to get large increase in classification accuracy with only small proportion of the labels. Using similar technique, our method could be applied when some samples are missing a second view.
Classification tasks using softmax operation or LDA were explored in this work; however, the formulation presented can also be used with other tasks such as regression or clustering. Another possible avenue for future work entails extracting components shared by both views as well as individual components. This approach has been developed for dictionary learning (Lock et al. al, 2013) Ray et al. 2014; Feng et al. 2018) but could be extended to deep CCA-based methods. Finally, have yet to apply data augmentation to the proposed framework; this could provide a significant benefit for small training sets.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep Canonical Correlation Analy- sis. Proc. ICML, 2013.
Raman Arora and Karen Livescu. Kernel eea for multi-view learning of acoustic features using artic- ulatory measurements, In Symposium Machine Learning in Speech and Language Processing, 2012.
Soheil Bahrampour, Nasser M. Nasrabadi, Asok Ray, and W. Kenneth Jenkins. Multimodal Task- Driven Dictionary Learning for Image Classification. arXiv preprint: 1502.01094, 2015.
Gaurav Bhatt, Piyush Jha, and Balasubramanian Raman. Common Representation Learning Using Step-based Correlation Multi-Modal CNN. arXiv preprint: 1711.00003, 2017.
Tijl De Bie, Nello Cristianini, and Roman Rosipal Eigenproblems in pattern recognition. In Hand- book of Geometric Computing, pp. 129 167. Springer Berlin Heidelberg, 2005.
Natalia Y. Bilenko and Jack Gallant. Pyrcea: regularized kernel canonical correlation analysis in Python and its applications to neuroimaging. Frontiers in Neuroinformatics, 10, nov 2016.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu-
pervised learning of visual features. In Proc. ECCV, 2018.
Miriam Cha, Youngjune Gwon, and H. T. Kung. Multimodal sparse representation learning and applications. arXiv preprint: 1511.0624 1.0638,, 2015.
Sarath Chandar, Mitesh M. Khapra, Hugo Larochelle, and Balaraman Ravindran. Correlational Neural Networks. Neural Computation, 28(2):257-285, feb 2016.
Xiaobin Chang, Tao Xiang, and Timothy M. Hospedales. Scalable and Effective Deep CCA via Soft Decorrelation. In Proc. CVPR, 2018.
Matthias Dorfer, Rainer Kelz, and Gerhard Widmer. Deep linear discriminant analysis. In Proc. ICLR, 2016a.
Matthias Dorfer, Gerhard Widmer, and Gerhard Widmerajku At. Towards Deep and Discriminative Canonical Correlation Analysis. In Proc. ICML Workshop on Multi-view Representaiton Learn ing, 2016b.
Matthias Dorfer, Jan Schlüter, Andreu Vall, Filip Korzeniowski, and Gerhard Widmer. End-to-end cross-modality retrieval with CCA projections and pairwise ranking loss. International Journal of Multimedia Information Retrieval, 1(2):117-128, jun 2018
Kanghong Duan, Hongxin Zhang, and Jim Jing Yan Wang. Joint learning of cross cross-modal classifier and factor analysis for multimedia data classification. Neural Computing and Applications, 27 (2):459-468, feb 2016.
Nour El Din Elmadany, Yifeng He, and Ling Guan. Multiview learning via deep diseriminative canonical correlation analysis. In Proc. ICASSP, 2016.
Qing Feng, Meilei Jiang. Jan Hannig, and JS Marron. Angle-based joint and individual variation explained. Journal of Multivariate Analysis, 166:241-265, 2018.
Bernardo B. Gatto and Eulanda M. Dos Santos. Discriminative canonical correlation analysis net- work for image classification. In Proc. ICIP, 2017.
Harold Hotelling. Relations between two sets variates. Biometrika, 28(3/4):31-3377, dec 1936.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated Batch Normalization. In Proc. CVPR, 2018.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proc. ICMI, 2015.

---

Under review as a conference paper at ICLR 2020
Meina Kan, Shiguang Shan, Haihong Zhang, Shihong Lao, and Xilin Chen. Multi-view Discrimi- nant Analysis. IEEE PAMI, 2015.
Jared Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deep Survival: A Deep Cox Proportional Hazards Network. arxiv preprint: 1606.00931, 2016.
Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. arXiv preprint: 1512.00809, 2015.
Alex Krizhevsky, lly Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp- 1106-1114, 2012.
Yann LeCun. The mnist database of handwritten digits. http://yaam.eeca..oo/eeddad// wwttsecb////emi/tt////////m///// 1998.
George Lee, Asha Singanamalli, Haibo Wang, Michael D Feldman, Stephen R Master, Natalie N C Shih, Elaine Spangler, Timothy Rebbeck, John E Tomaszewski, and Anant Madabhushi. Super- vised multi-view canonical correlation analysis (sMVCCA): integrating histologic and proteomic features for predicting recurrent prostate cancer. IEEE Transactions on Medical Imaging, 34(1): 284-97, jan 2015.
Dongge Li, Nevenka Dimitrova, Mingkun Li, and Ishwar K. Sethi. Multimedia content processing through cross-modal association. In Proc. ACM International Conference on Multimediu, 2003,
Eric Lock, Katherine A Hoadley, J S Marron, and Andrew B Nobel. Joint and Individual Vari- ation Explained (JIVE) for Integrated Analysis of Multiple Data Types. The Annals of Applied Statistics, 7(1):523-555 542,, mar 2013.
Dominic Masters and Carlo Lusehi. Revisiting Small Batch Training for Deep Neural Networks. arxiv preprint: 1804.07612, 2018.
Joel S Parker, Michael Mullins, Maggie CU Cheang, Samuel Leung, David Voduc, Tammi Vickery, Sherri Davies, Christiane Fauron, Xiaping He, et al. Supervised risk predictor of breast cancer based on intrinsic subtypes. Journal of Clinical Oncology, 27(8): 1160)::110-11100 1607
Priyadip Ray, Lingling Zheng. Joseph Lucas, and Lawrence Carin. Bayesian joint analysis of het- crogencous genomics data. Bioinformatics, 30(10):1370-6 may 2014.
Mehmet Emre Sargin, Yücel Yemez, Engin Erzin, and A Murat Tekalp. Audiovisual synchronization and fusion using canonical correlation analysis. IEEE Transactions on Multimedia, 9(7):1396- 1403, 2007.
Sumit Shekhar, Vishal M Patel, Nasser M Nasrabadi, and Rama Chellappa. Joint sparse representa- tion for robust multimodal biometrics recognition. IEEE PAMI, 36(1):113-26, jan 2014.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. ICLR, 2015.
Asha Singanamalli, Haibo Wang, George Lee, Natalie Shih, Mark Rosen, Stephen Master, John Tomaszewski, Michael Feldman, and Anant Madabhushi. Supervised multi-view canonical cor- relation analysis: fused multimodal prediction of disease diagnosis and prognosis. In Proc. SPIE Medical Imaging, 2014.
MA Troester. Xuezheng Sun, Emma H. Allott, Joseph Geradts, Stephanie M Cohen, Chui Kit Tse, Erin Kirk, Leigh B Thorne, Michelle Matthews, Yan Li. Zhiyuan Hu. Whitney R. Robinson, Katherine A. Hoadley, Olufunmilayo I. Olopade, Katherine E. Reeder-Hayes, H. Shelton Earp, Andrew F. Olshan, LA Carey, and Charles M. Perou. Racial differences in PAM50 subtypes in the Carolina Breast Cancer Study. Journal of the National Cancer Institute, 2018
L Van Der Maaten and G Hinton. Visualizing high-dimensional data using t-sne. journal of machine learning research. Journal of Machine Learning Research, 9:26, 2008
Weiran Wang, Raman Arora, Karen Liveseu, and Jeff Bilmes. On deep multi view representation learning. In Proc. ICMI., 201
12

---

Under review as a conference paper at ICLR 2020
Weiran Wang, Raman Arora, Karen Livescu, and Jeff A. Bilmes. Unsupervised learning of acoustic features via deep canonical correlation analysis. In Proc. ICASSP, 2015b.
Weiran Wang, Raman Arora, Karen Livescu, and Nathan Srebro. Stochastic optimization for deep CCA via nonlinear orthogonal iterations. In Proc. Allerton Conference Communication, Con- trol, and Computing, 2016.
Xing Xu, Atsushi Shimada, Rin-ichiro Taniguchi, and Li He. Coupled dictionary learning and feature mapping for cross-modal retrieval. Proc, International Conference on Multimedia and Expo, 2015.
13

---

Under review as a conference paper at ICLR 2020
A APPENDIX
This appendix includes additional details on our TOCCA algorithm and experiments, including 1) comparison of our formulation with other related CCA approaches, 2) pseudocode for the ZCA whitening algorithm used by TOCCA-W, 3) details on hyperparameter selection, and 4) training runtime experiments.
A.1 COMPARISON OF TOCCA WITH RELATED ALGORITIIMS
Our TOCCA methods finds a shared latent space that is also discriminative by changing the CCA for- mulation in order add task-driven component. Tab. A2 compares three TOCCA. formulations with other related methods (discussed in 33. CCA is the baseline linear method with goal of max- imizing the correlation between a of orthogonal linear projections on two views of data. DCCA and SoftCCA are unsupervised deep methods. DCCA optimizes an equivalent objective to CCA but uses non-lineat projections implemented with a NN; however, the projections are not computed in the network, only after optimization is complete. SoftCCA changes the correlation objective to, equivalently, minimize & distance between projections and relaxes the orthogonality constraints by using regularization. CCAL-frank does compute the CCA projections in the network but does not optimize the final NN for correlation; it instead focuses on a pairwise ranking loss for use in use retrieval. Our family of TOCCA methods were detailed in 44) In this supervised formulation, the same l distance as SoftCCA and simultaneously optimize a task-driven objective. We handle the orthogonality constraints in three different ways: with whitening (TOCCA-W), with regulariza- tion (TOCCA-SD) as was used in SoftCCA, and with no explicit decorrelation (TOCCA-ND).
Table A2: A comparison our proposed task-optimal deep CCA methods with other related ones from the literature: DCCA (Andrew et al. 2013). SoftCCA (Chang et al. 2018). CCAL-Lunk (Dorfer et al., 2018) CCAL-Crak uses pairwise ranking loss with cosine similarity to identify matching and non-matching samples for image retrieval
classification. A, and A₂ are centered outputs from two feed-forward networks. 2 = =A"A A'AA is computed from single (large) batch (used in DCCA): E is computed as running mean over batches (for all other methods). funk (A; Ousk) speciii function with parameters Oask. e.g., softmax operation for classification.
Method
Objective
CCA
-n( EisWa) |i Ei 1n
WTY, = WW
DCCA
where Th = TT (TNO, equivalent to CCA objective)
CCA(W" A,, Az) computed after optinization complete
SofCCCA
CCAL-C.a (B, Bs)
where B, CCA(A) Cuvs pairwise ranking loss where Bi = Ui B2 Ug s.l. B B, = B B, B,=
TOCCA-M (B, Bu, +)+ Ce BB)
rOCCA-SD TaskAA Y++ A Co auld (As. de (oanelld)) + Lacemlds)
Whitening
TOOCA-ND
Task(A,, +++ Li, Ab)
Loss functions
fg dist
(A, A2) 1|A, Adll
Decorr
Erase (A) Ens, [Eagl
where Et running mean across butches of 3 Al where Crust can be crussentropy any other task-driven loss
Task
A.. Av. (d Y)+ (dal)
A.2 ALGORITHM FOR WHITENING
TOCCA-W W uses whitening to achieve orthogonality (see for details). The goal is to transform the activations such that their covariance becomes the identity matrix. We use ZCA whitening which first applies PCA whitening to decorrelate the data and rescale each axis, followed by rotation back to the original space. The final rotation reduces the stochastic axis swapping problems of PCA whitening (Huang et al. 2018). Pseudocode for ZCA whitening is shown in Algorithm AT
14

---

Under review as a conference paper at ICLR 2020
Algorithm A1 Whitening layer for orthogonality.
Input: activations AcRduxn
Hyperparameters: batch size momentum
Parameters of layer: mean μ4, covariance y if training then
A - ap + (1 - a) 1x1 (Update mean]
= A /4 (Mean center )ata)
a+ (1 () m- A A/ (Update covariance)
Σ Σ + e/ eAdd for numerical stability)
A, eig(E) (Compute eigendecomposition)
U + VA- 1// {Compute transformation matrix)
else
À <- A μ (Mean center data)
end
B t UA (Apply ZCA whitening transform)
return B
A.3 IMPLEMENTATION DETAILS: HYPERPARAMETERS
random search over hyperparameters was used to train our methods and those that we compare with. The hyperparameter settings and ranges for each data set are provided in Tab. A3 Random search these intervals was performed 100 times for MNIST and CBCS. Fewer tries were done for XRMB because of the much greater runtime on this large data set. A larger batch size was used for XRMB to improve runtime. The hyperparameter ranges were initially set as an educated guess and, in some cases, were widened for particular data set (for all methods) after observing results.
Table A3: Hyperparameter settings and scarch ranges for the experiments each data set.
Hyperparameter
MNIST
CBCS
XRMB
Hidden layers
[0,4]
4 1,000
Hidden layer size
500
200
Output layer size
50
50
112
Loss function weight À
[109,10-] 10-11
[10*, 10-5)
[10
Momentum a
0.99
0.99
0.99
Weight decay
[10-3, 10-61 0
[10-2 10-9). 0
1103,10011
Soft decorrelation regularizer
[10210-6 10-6
[10 10
[10" 10-6
Batch size
32
100
50,000
Learning rate Epochs
[10 2, 10 10-4
[10-1.10 10
[100, 10
200
400
100
A.4 RUNTIME EXPERIMENTS
The computational complexity of TOCCA-B W is greater than that of TOCCA-SI SD due to the eigende- composition operation (see the end of 44); however, this extra computation is only carried once per batch. A runtime comparison of the two methods on all three data sets is provided in Tab. A4 The difference in runtime was less than 6.5% for batch size of 100 9.4% for batch size of 30.
Table A4: Training runtime for each data set.
Data set
Batch size
Epochs
TOCCA-W
TOCCA-SD
MNIST
100
200
488 s
418 s
MNIST
30
200
1071
1036 s
CBCS
100
400
103 s
104 s
XRMB
50.000
100
3056
3446 s
15


Reviews:
[{"review_id": "rkeDzTZ6KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a interesting paper on an important topic, but it was a main weakness: it assumes that the reader is deeply familiar with the CCA. In order to make the paper more accessible to a general audience, the authors should:\n1) have at least one sentence in the abstract that explains in layman terms why is CCA important and how it works (\"multi-view learning\" does not suffice); given that you have the term \"multi-view learning\" in the title, you should explain what it is and how it can benefit from CCA\n2)  re-organize the current intro, which reads more like related work, into a more traditional format\n     - one intuitive paragraph on what is multi-view learning (MVL), what is CCA, how does CCA help MVL \n     - one intuitive paragraph on an illustrative example on how MVL & CCA help solving a problem\n     - one intuitive paragraph on how the proposed approach works\n     - one paragraph summarizing the main findings/results \n3) ideally, add a section with an illustrative running example, which would have a huge impact on the paper's readability (far more than, say, than the current Appendix) ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SkgeAag0tH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of jointly performing CCA with task labeling. The problem is timely and important as it is challenging to perform CCA jointly with the task classification (see below) and hence previous work typically perform this in a pipeline - that is, first projecting the data using a pre-trained CCA and then training a task classifier using the projected representation. As the authors note, this may be problematic as CCA may delete important information that is relevant for the classification, if training is not done jointly. \n\nAs the authors note, the main challenge in developing a task-optimal form of deep CCA that discriminates based on the CCA projection is in computing this projection within the network. To deal with this, the authors propose approximations for two steps: maximizing the sum correlation between activations A1 and A2 and enforcing orthonormality constraints within A1 and A2. Particularly, the authors present three methods that progressively relax the orthogonality constraints. Also, because correlation is not separable across batches, SGD is not possible for deep CCA training and the authors deal with this too (although this was dealt with in previous work on deep CCA as well).\n\nThis is an empirical paper in the sense that no guarantees are provided for the proposed techniques. The experiments are thorough, convincing and the span a range of applications. The results demonstrate the value of the proposed approximations, and I hence recommend a weak accept of the paper. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJl61Mt2FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality:\n\nCCA is a generative model that learns a shared subspace based on  two (or multi) views of the data. Being generative, it might not have strong discriminative power for some downstream classification tasks. Previous approaches to infuse discriminative power into the shared subspace estimated by CCA are linear. So, this paper proposes to learn 1) non-linear 2) discriminative subspaces for CCA. The paper accomplishes this by simply adding a task specific term to the optimization objective of DeepCCA (Andrew et. al. 2013), which involves just adding a task-specific MLP on top and minimizing the associated loss-function. \n\n\n1). The novelty of the proposed approach is limited. It just adds an extra term (extra neural network layer) with a corresponding weighting hyperparameter to the objective function of a previous method (DeepCCA) without much motivation.\n\n\n2). The experimental setup and results are sound but some of the tasks seem contrived to show the improved performance of TOCCA methods. For instance, in the cross-view MNIST classification the authors use only projection from one view at training time and use the other view at test-time. What's the motivation for this setup? Why not split the data into train and test set by splitting observations, then train on both the views at train time and test on the held-out observations at test-time? I hope I am not missing something. \n\n3). Similarly, for the \"Regularization for Cancer Classification\" task, it's assumed that only one view is available at test time. Why is that? What are the real-world examples of such setups?  \n\n\nQuality:\n\nThe paper is technically sound, though it is a trivial extension of a previous method. The experimental setup is somewhat contrived to show the superiority of the proposed method. \n\n\nClarity:\n\nThe paper is well organized and is well written in general. The supplementary material contains more results and code will be available after the review period. \n\n\nSignificance:\n\nThe paper solves an important problem by infusing discriminative power into generative subspaces learned by CCA but the results are not that important in my eyes. Since the empirical setup is a little contrived it is hard to even know whether a simple two-step approach that first estimates CCA subspace and then uses those projections in a SVM or MLP would perform comparable or better if given a fair-chance to compete.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main contribution of this paper is the training of a supervised model jointly with deep CCA for improving the representations learned in a setting where the training data is multi-view.  The claimed technical contribution is modifications to deep CCA to enable it to play nicely with the minibatch gradient-based training used for the supervised loss.  Pros:  This is an important problem with many applications.  Cons:  The novelty is minimal.  Some previous work has done joint training of supervised models with CCA, and some has addressed training deep CCA in a stochastic setting.  The reviewers (and I) are unconvinced that the differences from previous work are sufficient, and the paper does not carefully compare with the previous work.  The contribution to the tasks may be quite significant, however, so the paper may fit in well in an application-oriented conference/journal.
Decision: Reject