Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper addresses the problem of jointly performing CCA with task labeling. The problem is timely and important as it is challenging to perform CCA jointly with the task classification (see below) and hence previous work typically perform this in a pipeline - that is, first projecting the data using a pre-trained CCA and then training a task classifier using the projected representation. As the authors note, this may be problematic as CCA may delete important information that is relevant for the classification, if training is not done jointly. 

As the authors note, the main challenge in developing a task-optimal form of deep CCA that discriminates based on the CCA projection is in computing this projection within the network. To deal with this, the authors propose approximations for two steps: maximizing the sum correlation between activations A1 and A2 and enforcing orthonormality constraints within A1 and A2. Particularly, the authors present three methods that progressively relax the orthogonality constraints. Also, because correlation is not separable across batches, SGD is not possible for deep CCA training and the authors deal with this too (although this was dealt with in previous work on deep CCA as well).

This is an empirical paper in the sense that no guarantees are provided for the proposed techniques. The experiments are thorough, convincing and the span a range of applications. The results demonstrate the value of the proposed approximations, and I hence recommend a weak accept of the paper. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This is a interesting paper on an important topic, but it was a main weakness: it assumes that the reader is deeply familiar with the CCA. In order to make the paper more accessible to a general audience, the authors should:
1) have at least one sentence in the abstract that explains in layman terms why is CCA important and how it works ("multi-view learning" does not suffice); given that you have the term "multi-view learning" in the title, you should explain what it is and how it can benefit from CCA
2)  re-organize the current intro, which reads more like related work, into a more traditional format
     - one intuitive paragraph on what is multi-view learning (MVL), what is CCA, how does CCA help MVL 
     - one intuitive paragraph on an illustrative example on how MVL & CCA help solving a problem
     - one intuitive paragraph on how the proposed approach works
     - one paragraph summarizing the main findings/results 
3) ideally, add a section with an illustrative running example, which would have a huge impact on the paper's readability (far more than, say, than the current Appendix) 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Originality:

CCA is a generative model that learns a shared subspace based on  two (or multi) views of the data. Being generative, it might not have strong discriminative power for some downstream classification tasks. Previous approaches to infuse discriminative power into the shared subspace estimated by CCA are linear. So, this paper proposes to learn 1) non-linear 2) discriminative subspaces for CCA. The paper accomplishes this by simply adding a task specific term to the optimization objective of DeepCCA (Andrew et. al. 2013), which involves just adding a task-specific MLP on top and minimizing the associated loss-function. 


1). The novelty of the proposed approach is limited. It just adds an extra term (extra neural network layer) with a corresponding weighting hyperparameter to the objective function of a previous method (DeepCCA) without much motivation.


2). The experimental setup and results are sound but some of the tasks seem contrived to show the improved performance of TOCCA methods. For instance, in the cross-view MNIST classification the authors use only projection from one view at training time and use the other view at test-time. What's the motivation for this setup? Why not split the data into train and test set by splitting observations, then train on both the views at train time and test on the held-out observations at test-time? I hope I am not missing something. 

3). Similarly, for the "Regularization for Cancer Classification" task, it's assumed that only one view is available at test time. Why is that? What are the real-world examples of such setups?  


Quality:

The paper is technically sound, though it is a trivial extension of a previous method. The experimental setup is somewhat contrived to show the superiority of the proposed method. 


Clarity:

The paper is well organized and is well written in general. The supplementary material contains more results and code will be available after the review period. 


Significance:

The paper solves an important problem by infusing discriminative power into generative subspaces learned by CCA but the results are not that important in my eyes. Since the empirical setup is a little contrived it is hard to even know whether a simple two-step approach that first estimates CCA subspace and then uses those projections in a SVM or MLP would perform comparable or better if given a fair-chance to compete.

review_title: Official Blind Review #2

Meta-review: The main contribution of this paper is the training of a supervised model jointly with deep CCA for improving the representations learned in a setting where the training data is multi-view.  The claimed technical contribution is modifications to deep CCA to enable it to play nicely with the minibatch gradient-based training used for the supervised loss.  Pros:  This is an important problem with many applications.  Cons:  The novelty is minimal.  Some previous work has done joint training of supervised models with CCA, and some has addressed training deep CCA in a stochastic setting.  The reviewers (and I) are unconvinced that the differences from previous work are sufficient, and the paper does not carefully compare with the previous work.  The contribution to the tasks may be quite significant, however, so the paper may fit in well in an application-oriented conference/journal.
Decision: Reject