Title: Deep Variational Canonical Correlation Analysis
Authors: Weiran Wang
Xinchen Yan
Honglak Lee
Karen Livescu
Venue: ICLR 2017

Abstract: We present deep variational canonical correlation analysis (VCCA), a deep multi-view learning model that extends the latent variable model interpretation of linear CCA~\citep{BachJordan05a} to nonlinear observation models parameterized by deep neural networks (DNNs). Computing the marginal data likelihood, as well as inference of the latent variables, are intractable under this model. We derive a variational lower bound of the data likelihood by parameterizing the posterior density of the latent variables with another DNN, and approximate the lower bound via Monte Carlo sampling. Interestingly, the resulting model resembles that of multi-view autoencoders~\citep{Ngiam_11b}, with the key distinction of an additional sampling procedure at the bottleneck layer. We also propose a variant of VCCA called VCCA-private which can, in addition to the ``common variables'' underlying both views, extract the ``private variables'' within each view. We demonstrate that VCCA-private is able to disentangle the shared and private information for multi-view data without hard supervision.

TL;DR: A deep generative model for multi-view representation learning

Review discussion:
While I can see the advantage of only conditioning on x (instead of x, y) in the encoder for certain applications, doesn’t this reduce the quality of the lower bound? It would be interesting to know by how much. If q(z | x) is needed at test time, one could also train it separately, e.g. by dropping y from q(z | x, y) only after training and fixing p(x, y, z).
Thanks for the comment.
We chose to parameterize q(z|x) as this feature mapping is what we use during test, so it is very natural to derive the lower bound with this posterior and train it "end-to-end" (to match the situation we have for testing).
It is possible that also parameterizing q(z|x,y) can help improve the performance. A similar approach was taken in the following paper for structured output prediction
Sohn et al. Learning Structured Output Representation using Deep Conditional Generative Models. NIPS 2015.
where the authors derive two variational lower bounds and optimize the convex combination of them.
We will investigate this approach in future work.
Hi,
I have a few questions regarding this paper:
Q1: In the same instructive way in which you compare VCCA to MVAE, would it be possible to compare VCCA to DCCAE? Are the two easy to relate?
Q2: Related to Q1. How come DCCAE outperforms VCCA in most of your experiments?
Thanks for the comments.
-- It might be possible to derive the relationship between VCCA and DCCAE (or a similar model) rigorously, but we have not done it so far. One could think of deriving two variational bounds of the data likelihood by parameterizing q(z|x) and q(z|y), optimizing the combination of the two lower bounds while encouraging the two approximate posteriors to agree. This model is similar to DCCAE.
-- The improved model VCCA-private does better than DCCAE in 2 out of 3 experiments.
We have so far improved the VCCA/VCCA-private performance on MIR-FLICKR significantly, matching previous state-of-the-art result we were aware of for the same setting. This result will be updated in the paper soon.
Dear reviewers,
We just uploaded a slightly modified version. This version
-- contains the updated results on Flickr and the proposed methods significantly outperform others
-- reorganized the figures and tables to make the paper more compact
Thanks!
Summary:
This paper describes the use of variational autoencoders for multi-view representation learning as an alternative to canonical correlation analysis (CCA), deep CCA (DCCA), and multi-view autoencoders (MVAE). Two variants of variational autoencoders (which the authors call VCCA and VCCA-private) are investigated. The method’s performances are compared on a synthetic MNIST dataset, the XRMB speech-articulation dataset, and the MIR-Flickr dataset.
Review:
Variational autoencoders are widely used and their performance for multi-view representation learning should be of interest to the ICLR community. The paper is well written and clear. The experiments are thorough. It is interesting that the performance of MVAE and VCCA is quite different given the similarity of their objective functions. I further find the analyses of the effects of dropout and private variables useful.
As the authors point out, “VCCA does not optimize the same criterion, nor produce the same solution, as any linear or nonlinear CCA”. It would have been interesting to discuss the differences of a linear variant of VCCA and linear CCA, and to compare it quantitatively.
While it might not make sense to use variational inference in the linear case, it would nevertheless help to understand the differences better.
The derivations in Equation 3 and Equation 13 seem unnecessarily detailed given that VCCA and VCCA-p are special cases of VAE, only with certain architectural constraints. Perhaps move to the Appendix?
In Section 3 the authors claim that “if we are able to generate realistic samples from the learned distribution, we can infer that we have discovered the underlying structure of the data”. This is not correct, a model which hasn’t learned a thing can have perfectly realistic samples (see Theis et al., 2016). Please remove or revise the sentence.
Minor:
In the equation between Equation 8 and 9, using notation N(x; g(z, theta), I) as in Equation 6 would make it clearer.
Thanks for your comments.
We will consider moving some derivations into the appendix, and rephrase the sentence you mentioned.
UPDATE: I have read the replies on this thread. My opinion has not changed.
The authors propose deep VCCA, a deep version of the probabilistic CCA model by using likelihoods parameterized by nonlinear functions (neural nets). Variational inference is applied with an inference network and reparameterization gradients. An additional variant, termed VCCA-private, is also introduced, which includes local latent variables for each data point (view). A connection to the multi view auto encoder is also shown.
Since the development of black box variational inference and variational auto-encoders, the methodology in model-specific papers like this one are arguably not very interesting. The model is a straightforward extension of probabilistic CCA with neural net parameterized likelihoods. Inference is mechanically the same as any black box approach using the reparameterization gradient and inference networks. The approach also uses a mean-field approximation, which is quite old given the many recent developments in more expressive approximations (see, e.g., Rezende and Mohamed (2015); Tran et al. (2016)).
The connection to multi-view auto encoders is at first insightful, but no more than the difference between MAP and variational inference. This is a well-known insight: in the abstract, the authors argue that the key distinction is the additional sampling, but ultimately what matters is the KL regularizer. Even with noisy samples, the variances of a normal variational approximation would collapse to zero and thus become a point mass approximation, equivalent to optimizing a point estimate from the MVAE objective. (I suspect the authors know this to some degree due to their remarks in the paper, but it is unclear.)
That said, I think the paper has strong merits in application. The experiments are strong, comparing to alternative multi-view approaches under a number of interesting data sets.
While the use of "private variables" is simple, they demonstrate how it can successfully disentangle the per-view latent representation from the shared view. It would have been preferable to compare to methods using probabilistic inference, such as full Bayes for the linear CCA.
There are also a number of approximations taken to almost be standard in the paper which may not be necessary, such as the use of a mean-field family or the use of an inference network. To separate out how much the approximate inference is influencing the fit of the model, I strongly recommend using MCMC and non-amortized variational inference on at least one experiment.
+ Rezende, D. J., & Mohamed, S. (2015). Variational Inference with Normalizing Flows. Presented at the International Conference on Machine Learning.
+ Tran, D., Ranganath, R., & Blei, D. M. (2016). The Variational Gaussian Process. Presented at the International Conference on Learning Representations.
-- The main contributions of our paper are:
* We propose a deep multi-view learning model that extends the latent variable model interpretation of linear CCA.
* We also propose VCCA-private as an extension, which is able to disentangle the shared and private information for multi-view data without strong supervision.
* Our model provides tight connections to previous work, including the probabilistic interpretation of CCA and multimodal autoencoders.
* The proposed model is as good as the SOA model in MNIST/XRMB experiment and outperforms SOA model in Flicker experiment.
Besides the performance, the VCCA can be trained in an end-to-end style and is relatively easy to optimize.
Previously, the most successful multi-view deep learning methods are deterministic (CCA, contrastive loss) or undirected graphical models based on RBMs (Srivastava and Salakhutdinov JMLR 2014, Sohn et al., at NIPS 2014). The deterministic methods are not designed for generation. The RBM based models require relatively complicated pipelines for training (layer-wise pre-training, mean-field approximations and MCMCs) and tuning a large number of hyperparameters.
-- Thank you for pointing out related work on inference methods; our contribution and more advanced variational inference/learning techniques (e.g. ,Rezende and Mohamed (2015); Tran et al. (2016)) are complementary. We will be happy to investigate this in our future work.
-- You are correct about "what matters is the KL regularizer". In early stages of our experiments, we actually tuned a trade-off parameter for the KL regularizer; deviating from 1 (as the variational lower bound dictates) did not improved the performance. We will rephrase the superficial resemblance of "sampling procedure".
This paper considers the case where multiple views of data are learned through a probabilistic deep neural network formulation. This makes the model non-linear (unlike e.g. CCA) but makes inference difficult.
Therefore, the VAE framework is invoked for inference.
In [Ref 1] the authors show that maximum likelihood estimation based on their linear latent model leads to the canonical correlation directions.
But in the non-linear case with DNNs it's not clear (at least with the present analysis) what the solution is wrt to the canonical directions. There's no such analysis in the paper, hence I find it a stretch to refer to this model as a CCA type of model.
In contrast, e.g. DCCA / DCCAE are taking the canonical correlation between features into account inside the objective and provide interpretations.
[Ref 1] F. R. Bach and M. I. Jordan. A probabilistic interpretation of canonical correlation analysis. Technical Report 688, 2005.
There is also a significant body of very related work on non-linear multi-view models which is not discussed in this paper.
For example, there's been probabilistic non-linear multi-view models [Ref 2, 3], also extended to the Bayesian case with common/private spaces [Ref 4] and the variational / deep learning case [Ref 5].
[Ref 2] Ek et al. Gaussian process latent variable models for human pose estimation. MLMI, 2007.
[Ref 3] Shon et al. Learning shared latent structure for image synthesis and robotic imitation. NIPS, 2006.
[Ref 4] Damianou et al. Manifold relevance determination. ICML, 2012.
[Ref 5] Damianou and Lawrence. Deep Gaussian processes. AISTATS, 2013.
I can see the utility of this model as bringing together two elements: multi-view modeling and VAEs. This seems like an obvious idea but to the best of my knowledge it hasn't been done before and is actually a potentially very useful model.
However, the question is, what is the proper way of extending VAE to multiple views? The paper didn't convince me that VAE can work well with multiple views using the shown straightforward construction.
Specifically, VCCA doesn't seem to promote the state of the art in terms of results (it actually is overall below the SOA), while the VCCA-private seems a quite ill-posed model: the dimensionalities d have to be manually tuned with exhaustive search;
further, the actual model does not provide a consinstent way of encouraging the private and common variables to avoid learning redundant information. Relying only on dropout for this seems a quite ad-hoc solution (in fact, from Fig. 4 (ver2) it seems that the dropout rate is quite crucial). Perhaps good performance might be achieved with a lot of tuning (which might be why the FLICKR results got better in ver2 without changing the model), but it seems quite difficult to optimize for the above reasons. From a purely experimental point of view, VCCA-private doesn't seem to promote the SOA either.
Of course one wouldn't expect any new published paper to beat all previous baselines, but it seems that extension of VAE to multiple views is a very interesting idea which deserves some more investigation of how to do it efficiently.
Another issue is the approximate posterior being parameterized only from one of the views. This makes the model less useful as a generic multi-view model, since it will misbehave in tasks other than classification.
But if classification is the main objective, then one should compare to a proper classification model, e.g. a feedforward neural network.
The plots of Fig. 8 are very nice.
Overall, the paper convinced me that there is merit in attaching multiple views to VAE.
However, it didn't convince me a) that the proposed way to achieve this is practical b) that there is a connection to CCA (other than being a method for multiple views). The bottom line is that, although the paper is interesting, it needs a little more work.
-- Thanks for the references [2-5].  [3] and [4] are quite relevant in that they use the same graphical model and we will cite and discuss them in the next version soon.
-- We improved the result on Flickr simply because we trained the same model longer after the deadline.
-- There was not too much tuning involved to make VCCA work: standard deviation (a scalar) of Gaussian observation models, dropout rate (0.2 is selected for all experiments), and feature dimensionality. For optimization, we have always used Adam with 0.0001 learning rate for 200--300 epochs. We will make our code available and provide complete recipes for replicating our experiments.
-- Dropout was indeed important to remove redundancy in learning useful features in deep variational auto-encoders (Sohn et al., at NIPS 2015).
In addition, dropout is a very standard technique for auto-encoder style training to overcome overfitting to the training data. Please note that we set the dropout rate to 0.2 for all the experiments without tuning the rate too much.
In summary, we believe the drop-out is a simple and efficient method in deep variational auto-encoder training that also applies to our case.
-- We derive approximate posterior from one of the views mainly due to our multi-view representation learning (other possible derivations are given in last paragraph of page 3): we have large amount of unlabeled multi-view data from which we learn the feature transformation, and the feature transformation is applied to unseen single-view data in downstream tasks. The downstream tasks have little labeled single-view data and thus one could not train a powerful classifier well (e.g., this is the case for flickr, only 25000 images out of 1 million has topic labels). This is a common setting, see Ngiam et al ICML 2012, Srivastava and Salakhutdinov JMLR 2014, Sohn et al NIPS 2014, and
Zheng et al. A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data. TPAMI, 2015.
For the XRMB speech corpus, we had trained powerful discriminative DNN features in this setting, see Section 5 of
Wang et al.  Unsupervised Learning of Acoustic Features via Deep Canonical Correlation Analysis. ICASSP 2015.
and the DNN features' performance is weaker than that of deep CCA and thus weaker than that of VCCA-private.
We conducted experiments in the unsupervised feature learning setting in order to make direct comparison with previous work, but it is straightforward to extend our method to semi-supervised setting by combining the objectives of representation learning and downstream task learning.
-- Please note that the objectives of feature learning and downstream task learning shall be correlated but not identical, so the eventual tuning criteria is the downstream validation set performance. This justifies why we also tune the feature dimensionality. As a concrete example, too high a feature dimensionality will be problematic for the training of recognizers (Gaussian mixture model-HMMs)  on XRMB.
-- Our method (as well as several other papers we discuss in related work) is inspired by the probabilistic interpretation of linear CCA, so we believe that the connection is interesting and nontrivial. We hope that our work will stimulate more research on developing a unified framework that explains many previous approaches.
Dear reviewers,
We have modified our response to your review, acknowledging the related work pointed out by several reviewers, and clarifying our main contributions.
Thanks!

Meta-review:
role: PC
data: comment: The application of VAEs to multiview settings, snd the general problem of learning representations over multiple modalities is of increasing importance. After discussion and reviewing the rebuttals, the reviewers felt that the paper is not yet ready. The application is strong, but that more can be done in terms of praticality, generalisation using other posteriors, and the comparisons made. For this reason, the paper is unfortunately not yet ready for inclusion in this year's proceedings.

Decision: Reject