Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a deep extension of generalized CCA. The main contribution of the paper is deriving the gradient update for the GCCA objective.

I disagree with the claim that “this is the first Multiview representation learning technique that combines the flexibility of nonlinear representation learning with the statistical power of incorporating information from many independent resources or views”.  [R1] proposes a Multiview representation learning method which is both non-linear and capable of handling more than 2 views. This is very much relevant to what authors are proposing. The objective function proposed in [R1] maximizes the correlation between views and minimizes the self and cross reconstruction errors. This is intuitively similar to nonlinear version of PCA+CCA for multiple views. Comparing these 2 methods is crucial to prove the usefulness of DGCCA and the paper is incomplete without this comparison. Authors should also change their strong claim.

Related work section is minimal. There are significant advances in 2-view non-linear representation learning which are worth mentioning. 

References:
[R1] Janarthanan Rajendran, Mitesh M. Khapra, Sarath Chandar, Balaraman Ravindran: Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning. HLT-NAACL 2016: 171-181


review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a method that extends the non-linear two-view representation learning methods, and the linear multiview techniques, and combines information from multiple sources into a new non-linear representation learning techniques. 

In general, the method is well described and seems to lead to benefits in different experiments of phonetic transcription of hashtag recommendation. Even if the method is mostly a extension of classical tools (the scheme learns a (deep) network for each view essentially), the combination of the different sources of information seems to be effective for the studied datasets. 

It would be interesting to add or discuss the following issues:

- what is the complexity of the proposed method, esp. the representation learning part?
- would there by any alternative solution to combine the different networks/views? That could make the proposed solution more novel.
- the experimental settings, especially in the synthetic experiments, should be more detailed. If possible, the datasets should be made available to encourage reproducibility. 
- the related work is far from complete unfortunately, especially from the perspective of the numerous multiview/multi-modal/multi-layer algorithms that have been proposed in the literature, in different applications domaines like image retrieval or classification, or bibliographic data for example (authors like A. Kumar, X. Dong, Ping-Yu Chen, M. Bronstein, and many others have proposed works in that direction in the last 5 years). No need to compare to all these works obviously, but a more complete description of the related could help appreciating better the true benefits of DGCCA.

review_title: Deep Generalised Canonical Correlation Analysis

final_score: rating: 7: Good paper, accept
review_content: The proposed method is simple and elegant; it builds upon the huge success of gradient based optimization for deep non-linear function approximators and combines it with established (linear) many-view CCA methods. A major contribution of this paper is the derivation of the gradients with respect to the non-linear encoding networks which project the different views into a common space. The derivation seems correct. In general this approach seems very interesting and I could imagine that it might be applicable to many other similarly structured problems.
The paper is well written; but it could be enhanced with an explicit description of the complete algorithm which also highlights how the joint embeddings G and U are updated. 
 
I don’t have prior experience with CCA-style many-view techniques and it is therefore hard for me to judge the practical/empirical progress presented here. But the experiments seem reasonable convincing; although generally only performed on small and medium sized datasets.
  
Detailed comments: 

The colours or the sign of the x-axis in figure 3b seem to be flipped compared to figure 4.
  
It would be nice to additionally see a continuous (rainbow-coloured) version for Figures 2, 3 and 4 to better identify neighbouring datapoints; but more importantly: I’d like to see how the average reconstruction error between the individual network outputs and the learned representation develop during training.  Is the mismatch between different views on a validation/test-set a useful metric for cross validation? In general, it seems the method is sensitive to regularization and hyperparameter selection  (because it has many more parameters compared to GCCA and different regularization parameters have been chosen for different views) and I wonder if there is a clear metric to optimize these.

review_title: Good paper

Meta-review: This is largely a clearly written paper that proposes a nonlinear generalization of a generalized CCA approach for multi-view learning. In terms of technical novelty, the generalization follows rather straightforwardly. Reviewers have expressed the need to clarify relationship and provide comparisons to existing proposals for combining deep learning with CCA. As such the paper has been evaluated to be borderline. The proposed method appears to yield significant gains on a speech dataset, though comparisons on other datasets appear to be less conclusive. Some basic baselines as missing, e.g., concatenating views and running a deep model, or using much older nonlinear extensions of CCA such as kernel CCA (e.g. accelerated via random features, and combined with deep representations underneath).
Decision: Reject