Title: Deep Generalized Canonical Correlation Analysis
Authors: Adrian Benton
Huda Khayrallah
Biman Gujral
Drew Reisinger
Sheng Zhang
Raman Arora
Venue: ICLR 2017
Keywords: Unsupervised Learning
Deep learning
Multi-modal learning

Abstract: We present Deep Generalized Canonical Correlation Analysis (DGCCA) – a method for learning nonlinear transformations of arbitrarily many views of data, such that the resulting transformations are maximally informative of each other. While methods for nonlinear two-view representation learning (Deep CCA, (Andrew et al., 2013)) and linear many-view representation learning (Generalized CCA (Horst, 1961)) exist, DGCCA is the first CCA-style multiview representation learning technique that combines the flexibility of nonlinear (deep) representation learning with the statistical power of incorporating information from many independent sources, or views. We present the DGCCA formulation as well as an efficient stochastic optimization algorithm for solving it. We learn DGCCA representations on two distinct datasets for three downstream tasks: phonetic transcription from acoustic and articulatory measurements, and recommending hashtags and friends on a dataset of Twitter users. We find that DGCCA representations soundly beat existing methods at phonetic transcription and hashtag recommendation, and in general perform no worse than standard linear many-view techniques.

TL;DR: A multiview representation learning technique that can learn nonlinear mappings from arbitrarily many views to a shared semantic space -- Deep Generalized Canonical Correlation Analysis.

Review discussion:
The proposed method is simple and elegant; it builds upon the huge success of gradient based optimization for deep non-linear function approximators and combines it with established (linear) many-view CCA methods. A major contribution of this paper is the derivation of the gradients with respect to the non-linear encoding networks which project the different views into a common space. The derivation seems correct. In general this approach seems very interesting and I could imagine that it might be applicable to many other similarly structured problems.
The paper is well written;
but it could be enhanced with an explicit description of the complete algorithm which also highlights how the joint embeddings G and U are updated.
I don’t have prior experience with CCA-style many-view techniques and it is therefore hard for me to judge the practical/empirical progress presented here.
But the experiments seem reasonable convincing;
although generally only performed on small and medium sized datasets.
Detailed comments:
The colours or the sign of the x-axis in figure 3b seem to be flipped compared to figure 4.
It would be nice to additionally see a continuous (rainbow-coloured) version for Figures 2, 3 and 4 to better identify neighbouring datapoints;
but more importantly: I’d like to see how the average reconstruction error between the individual network outputs and the learned representation develop during training.  Is the mismatch between different views on a validation/test-set a useful metric for cross validation? In general, it seems the method is sensitive to regularization and hyperparameter selection  (because it has many more parameters compared to GCCA and different regularization parameters have been chosen for different views) and I wonder if there is a clear metric to optimize these.
- The detailed algorithm and updates for U, G are given in Algorithm 1 in Appendix B on page 13. We will highlight the key updates in the main text as well.
- Figures 3(b) and Figure 4(a) show different things. Figure 3(b) shows the shared representation G whereas Figure 4(a) shows the projection of View 1 (in Figure 2(a)) projected onto its corresponding subspace, in other words it shows U’X_1. They do look similar in that the two components are well separated.
- We can easily add view-specific reconstruction errors and rainbow plots. Note that from Figure 4 it appears that in terms of reconstruction error, View 3 is worse than View 2 which is worse than View 1.
- We found that reconstruction error was a reasonable proxy in choosing regularization parameters, but not the best model for a downstream task.
For example, if the L2 regularization penalty was too low, the reconstruction error would be orders of magnitude larger than a correctly regularized model, and downstream task performance would be very poor.
However, we did not purely rely on reconstruction error to select models, since networks with narrower output layers will necessarily have lower reconstruction error.
Ultimately we selected models that perform best on the downstream task.  We can include a figure containing reconstruction error vs. downstream task performance to illustrate this.
This paper proposes a deep extension of generalized CCA. The main contribution of the paper is deriving the gradient update for the GCCA objective.
I disagree with the claim that “this is the first Multiview representation learning technique that combines the flexibility of nonlinear representation learning with the statistical power of incorporating information from many independent resources or views”.  [R1] proposes a Multiview representation learning method which is both non-linear and capable of handling more than 2 views. This is very much relevant to what authors are proposing. The objective function proposed in [R1] maximizes the correlation between views and minimizes the self and cross reconstruction errors. This is intuitively similar to nonlinear version of PCA+CCA for multiple views. Comparing these 2 methods is crucial to prove the usefulness of DGCCA and the paper is incomplete without this comparison. Authors should also change their strong claim.
Related work section is minimal. There are significant advances in 2-view non-linear representation learning which are worth mentioning.
References:
[R1] Janarthanan Rajendran, Mitesh M. Khapra, Sarath Chandar, Balaraman Ravindran: Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning. HLT-NAACL 2016: 171-181
- Thanks for the additional reference, we will discuss it in the revision as it is related work.
However, the “Bridge Correlational Neural Networks” paper considers a setting where instead of several parallel views, there is a bridge view that provides a parallel view between other views.
Therefore the problem reduces to two-view CCA and their nonlinear/deep extensions. This is clearly very different from the setting we consider here as DGCCA has no such notion of a bridge view. We emphasize again that extending correlative learning to many views, say p views, using (two-view) CCA would require solving p-choose-2 CCA-like problems and various heuristics to combine them. We will add a discussion to that effect; future work/extensions will compare with bridge Corr-nets.
- We discuss and compare with the most relevant related work in two view learning here, especially CCA and Deep CCA. For extensive literature, we refer the readers to our prior work including the DCCA and DCCAE papers available online. We emphasize again that this is a non-trivial extension of DCCA and various modifications/extensions of DCCA can be considered for the proposed method here as well, but that is not the focus here.
The authors propose a method that extends the non-linear two-view representation learning methods, and the linear multiview techniques, and combines information from multiple sources into a new non-linear representation learning techniques.
In general, the method is well described and seems to lead to benefits in different experiments of phonetic transcription of hashtag recommendation.
Even if the method is mostly a extension of classical tools (the scheme learns a (deep) network for each view essentially), the combination of the different sources of information seems to be effective for the studied datasets.
It would be interesting to add or discuss the following issues:
- what is the complexity of the proposed method, esp. the representation learning part?
- would there by any alternative solution to combine the different networks/views? That could make the proposed solution more novel.
- the experimental settings, especially in the synthetic experiments, should be more detailed. If possible, the datasets should be made available to encourage reproducibility.
- the related work is far from complete unfortunately, especially from the perspective of the numerous multiview/multi-modal/multi-layer algorithms that have been proposed in the literature, in different applications domaines like image retrieval or classification, or bibliographic data for example (authors like A. Kumar, X. Dong, Ping-Yu Chen, M. Bronstein, and many others have proposed works in that direction in the last 5 years). No need to compare to all these works obviously, but a more complete description of the related could help appreciating better the true benefits of DGCCA.
- The code and other resources are already posted online (see the footnote on page 2).  The synthetic data are included in the linked repository.
- Yes, the key idea in extending correlative learning to many views, say p views, is to compute correlation between every pair of views, store them in a pxp symmetric matrix and maximize some norm of that matrix. The norm that we consider in this paper is the spectral norm, as it leads to tractable solutions.  Naïve approaches solve p-choose-2 CCA-like problems and come up with heuristics to combine them. We will add a discussion to this effect.
- We greatly appreciate pointers to additional references; there has been immense activity in this area and it has been hard to catch up with the related work. We have done extensive review of related work in machine learning conferences as well as speech and language processing, but we do realize that there have been several remarkable contributions in computer vision and information retrieval. We will rectify this in the revision.
- It is hard to characterize the computational complexity of the proposed methodology as is the case with any deep learning technique (it being a highly non-convex optimization problem). We can, and will, indeed discuss the computational cost per iteration and memory requirements of the proposed method (which we had included in a previous draft.)
Thank you for your helpful comments.  We just uploaded a revised draft, incorporating the reviewers' suggestions, and hopefully addressing many of their concerns.  Below are the things to note:
- The linear GCCA solution for G and U is included in Appendix A, along with a full gradient derivation: "... the rows of G are the top r (orthonormal) eigenvectors of M, and $U_j = C_{jj}^{−1} Y_j G^T$" (reviewer 2)
- In the last paragraph of the Optimization subsection (page 4), we include big-Oh notation for the gradient update time complexity.  We leverage the GCCA solution presented in [R1] to scale DGCCA to large datasets. (reviewer 2)
- We qualify the pronouncement of being "the first nonlinear multiview learning technique" with the adjective "CCA-style".
Although our work focuses on extending CCA-based multiview methods, we recognize that others have attempted to learn embeddings by merging information from multiple views. (reviewer 5)
- In Section 5, "Other Multiview Work", we include a discussion of a non-CCA-based techniques for nonlinear representation learning from multiple views, and how they differ from DGCCA.  As reviewers 2 and 5 mention, the multiview learning literature is vast, and we are not able to address all models proposed that make learned representations from more than one view.
However, we do hope that this section will clarify how our proposed model differs from other representation learning techniques exploiting multiple views. (reviewers 2 and 5)
- Appendix C includes a short discussion of how the DGCCA objective reconstruction error relates to downstream task performance for Twitter hashtag recommendation.
In short, we found that high reconstruction error is a strong signal for poor downstream performance, but there is significant variation in downstream performance between embeddings learned by models with low reconstruction error. (reviewer 4)
We also trained Bridge Correlational Neural Network embeddings for Twitter hashtag and friend recommendation in a series of preliminary experiments.  We swept over hidden layer width in the same range as the DGCCA experiments, $\lambda \in \{0.0, 0.1, 1.0, 10.0\}$ (the strength of the correlation term in the DGCCA objective), and used either the Twitter user ego text view or their friend network view as the pivot view, since these were the solely most effective views for hashtag and friend recommendtion.  Other learning parameters were left at the defaults and networks were trained for 50 epochs.
However, the performance of these embeddings was much worse than the CCA-style models we compare to (R@1000=0.06 for hashtag recommendation.)  We grant that downstream performance would be improved by tuning learning parameters, but these preliminary experiments underscore the fact that this class of models is not a panacea, and may not be appropriate for these recommendation tasks.  We explicitly note this in the text, since these models assume that all views should be correlated with a pivot view representation.  Entraining all embeddings to a single view is, thus, probably not as effective at hashtag recommendation as learning a CCA-style joint representation for all views. (reviewer 5)
Please let us know if any of these revisions are confusing, or if you have any additional suggestions.
[R1] Pushpendre Rastogi, Benamin Van Durme, and Raman Arora. Multiview LSA: Representation Learning via Generalized CCA. Proceedings of NAACL. 2015.

Meta-review:
role: PC
data: comment: This is largely a clearly written paper that proposes a nonlinear generalization of a generalized CCA approach for multi-view learning. In terms of technical novelty, the generalization follows rather straightforwardly. Reviewers have expressed the need to clarify relationship and provide comparisons to existing proposals for combining deep learning with CCA. As such the paper has been evaluated to be borderline. The proposed method appears to yield significant gains on a speech dataset, though comparisons on other datasets appear to be less conclusive. Some basic baselines as missing, e.g., concatenating views and running a deep model, or using much older nonlinear extensions of CCA such as kernel CCA (e.g. accelerated via random features, and combined with deep representations underneath).

Decision: Reject