Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: While I can see the advantage of only conditioning on x (instead of x, y) in the encoder for certain applications, doesn’t this reduce the quality of the lower bound? It would be interesting to know by how much. If q(z | x) is needed at test time, one could also train it separately, e.g. by dropping y from q(z | x, y) only after training and fixing p(x, y, z).
review_title: Lower bound and using q(z | x) instead of q(z | x, y)

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper considers the case where multiple views of data are learned through a probabilistic deep neural network formulation. This makes the model non-linear (unlike e.g. CCA) but makes inference difficult. Therefore, the VAE framework is invoked for inference.

In [Ref 1] the authors show that maximum likelihood estimation based on their linear latent model leads to the canonical correlation directions. But in the non-linear case with DNNs it's not clear (at least with the present analysis) what the solution is wrt to the canonical directions. There's no such analysis in the paper, hence I find it a stretch to refer to this model as a CCA type of model. In contrast, e.g. DCCA / DCCAE are taking the canonical correlation between features into account inside the objective and provide interpretations.

[Ref 1] F. R. Bach and M. I. Jordan. A probabilistic interpretation of canonical correlation analysis. Technical Report 688, 2005.

There is also a significant body of very related work on non-linear multi-view models which is not discussed in this paper. For example, there's been probabilistic non-linear multi-view models [Ref 2, 3], also extended to the Bayesian case with common/private spaces [Ref 4] and the variational / deep learning case [Ref 5].

[Ref 2] Ek et al. Gaussian process latent variable models for human pose estimation. MLMI, 2007.
[Ref 3] Shon et al. Learning shared latent structure for image synthesis and robotic imitation. NIPS, 2006.
[Ref 4] Damianou et al. Manifold relevance determination. ICML, 2012.
[Ref 5] Damianou and Lawrence. Deep Gaussian processes. AISTATS, 2013.

I can see the utility of this model as bringing together two elements: multi-view modeling and VAEs. This seems like an obvious idea but to the best of my knowledge it hasn't been done before and is actually a potentially very useful model.

However, the question is, what is the proper way of extending VAE to multiple views? The paper didn't convince me that VAE can work well with multiple views using the shown straightforward construction. Specifically, VCCA doesn't seem to promote the state of the art in terms of results (it actually is overall below the SOA), while the VCCA-private seems a quite ill-posed model: the dimensionalities d have to be manually tuned with exhaustive search; further, the actual model does not provide a consinstent way of encouraging the private and common variables to avoid learning redundant information. Relying only on dropout for this seems a quite ad-hoc solution (in fact, from Fig. 4 (ver2) it seems that the dropout rate is quite crucial). Perhaps good performance might be achieved with a lot of tuning (which might be why the FLICKR results got better in ver2 without changing the model), but it seems quite difficult to optimize for the above reasons. From a purely experimental point of view, VCCA-private doesn't seem to promote the SOA either. Of course one wouldn't expect any new published paper to beat all previous baselines, but it seems that extension of VAE to multiple views is a very interesting idea which deserves some more investigation of how to do it efficiently.

Another issue is the approximate posterior being parameterized only from one of the views. This makes the model less useful as a generic multi-view model, since it will misbehave in tasks other than classification. But if classification is the main objective, then one should compare to a proper classification model, e.g. a feedforward neural network.

The plots of Fig. 8 are very nice. Overall, the paper convinced me that there is merit in attaching multiple views to VAE. However, it didn't convince me a) that the proposed way to achieve this is practical b) that there is a connection to CCA (other than being a method for multiple views). The bottom line is that, although the paper is interesting, it needs a little more work.

review_title: null

final_score: rating: null
review_content: Hi,
I have a few questions regarding this paper:
Q1: In the same instructive way in which you compare VCCA to MVAE, would it be possible to compare VCCA to DCCAE? Are the two easy to relate? 

Q2: Related to Q1. How come DCCAE outperforms VCCA in most of your experiments? 
review_title: VCCA and DCCAE

final_score: rating: 7: Good paper, accept
review_content: 7

Summary:
This paper describes the use of variational autoencoders for multi-view representation learning as an alternative to canonical correlation analysis (CCA), deep CCA (DCCA), and multi-view autoencoders (MVAE). Two variants of variational autoencoders (which the authors call VCCA and VCCA-private) are investigated. The method’s performances are compared on a synthetic MNIST dataset, the XRMB speech-articulation dataset, and the MIR-Flickr dataset.

Review:
Variational autoencoders are widely used and their performance for multi-view representation learning should be of interest to the ICLR community. The paper is well written and clear. The experiments are thorough. It is interesting that the performance of MVAE and VCCA is quite different given the similarity of their objective functions. I further find the analyses of the effects of dropout and private variables useful.

As the authors point out, “VCCA does not optimize the same criterion, nor produce the same solution, as any linear or nonlinear CCA”. It would have been interesting to discuss the differences of a linear variant of VCCA and linear CCA, and to compare it quantitatively. While it might not make sense to use variational inference in the linear case, it would nevertheless help to understand the differences better.

The derivations in Equation 3 and Equation 13 seem unnecessarily detailed given that VCCA and VCCA-p are special cases of VAE, only with certain architectural constraints. Perhaps move to the Appendix?

In Section 3 the authors claim that “if we are able to generate realistic samples from the learned distribution, we can infer that we have discovered the underlying structure of the data”. This is not correct, a model which hasn’t learned a thing can have perfectly realistic samples (see Theis et al., 2016). Please remove or revise the sentence.

Minor:
In the equation between Equation 8 and 9, using notation N(x; g(z, theta), I) as in Equation 6 would make it clearer.
review_title: Linear VCCA

final_score: rating: 5: Marginally below acceptance threshold
review_content: UPDATE: I have read the replies on this thread. My opinion has not changed.

The authors propose deep VCCA, a deep version of the probabilistic CCA model by using likelihoods parameterized by nonlinear functions (neural nets). Variational inference is applied with an inference network and reparameterization gradients. An additional variant, termed VCCA-private, is also introduced, which includes local latent variables for each data point (view). A connection to the multi view auto encoder is also shown.

Since the development of black box variational inference and variational auto-encoders, the methodology in model-specific papers like this one are arguably not very interesting. The model is a straightforward extension of probabilistic CCA with neural net parameterized likelihoods. Inference is mechanically the same as any black box approach using the reparameterization gradient and inference networks. The approach also uses a mean-field approximation, which is quite old given the many recent developments in more expressive approximations (see, e.g., Rezende and Mohamed (2015); Tran et al. (2016)).

The connection to multi-view auto encoders is at first insightful, but no more than the difference between MAP and variational inference. This is a well-known insight: in the abstract, the authors argue that the key distinction is the additional sampling, but ultimately what matters is the KL regularizer. Even with noisy samples, the variances of a normal variational approximation would collapse to zero and thus become a point mass approximation, equivalent to optimizing a point estimate from the MVAE objective. (I suspect the authors know this to some degree due to their remarks in the paper, but it is unclear.)

That said, I think the paper has strong merits in application. The experiments are strong, comparing to alternative multi-view approaches under a number of interesting data sets. While the use of "private variables" is simple, they demonstrate how it can successfully disentangle the per-view latent representation from the shared view. It would have been preferable to compare to methods using probabilistic inference, such as full Bayes for the linear CCA. 

There are also a number of approximations taken to almost be standard in the paper which may not be necessary, such as the use of a mean-field family or the use of an inference network. To separate out how much the approximate inference is influencing the fit of the model, I strongly recommend using MCMC and non-amortized variational inference on at least one experiment. 

+ Rezende, D. J., & Mohamed, S. (2015). Variational Inference with Normalizing Flows. Presented at the International Conference on Machine Learning.
+ Tran, D., Ranganath, R., & Blei, D. M. (2016). The Variational Gaussian Process. Presented at the International Conference on Learning Representations.
review_title: Review: Deep Variational Canonical Correlation Analysis

Meta-review: The application of VAEs to multiview settings, snd the general problem of learning representations over multiple modalities is of increasing importance. After discussion and reviewing the rebuttals, the reviewers felt that the paper is not yet ready. The application is strong, but that more can be done in terms of praticality, generalisation using other posteriors, and the comparisons made. For this reason, the paper is unfortunately not yet ready for inclusion in this year's proceedings.
Decision: Reject