Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Paper presents a novel sparse nonlinear-CCA method which works especially for cases where number of sample are less than max input feature dimension (D) N << D. It presents a novel algorithm that uses Gaussian relaxation of Bernoulli gating variables that sub-select sparse set of features. Empirical results on synthetic and real world data show efficacy of the method.
main_review: Idea of sparse non-linear CCA for N << D case is novel and interesting and is valuable for such real-world cases. Loss function that minimizes correlation and increases sparsity provides a unique way to do feature selection for multi-modal cases.

Algorithm presented that does l_0 feature selection using Gaussian relaxed Bernoulli variables is interesting and novel application of for sparse-DCCA problem.

Empirical results on synthetic data show lowest error in learned correlations from sparse-DCCA compared to other SOTA techniques. Results on multi-view, MNIST digits, Seismic event and METABRIC datasets show that technique is able to pick relevant input features in challenging noisy and high dimensional (N<<D) cases with few examples.

A strength of the presented method is that it is able to sub-select relevant features from high dimensional inputs so it is expected to perform well compared to techniques that are not specifically designed for that. It would be helpful to see performance of sparse-DCCA for cases of N>>D compared to other techniques for comparison.
summary_of_the_review: Novelty of the paper is extension of idea of nonlinear-CCA method to sparse case where where number of sample are less than max input feature dimensions (D) N << D. Results presented are convincing and support technical details.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a sparse canonical correlation analysis method based on l_0 norm. A continuous relaxation scheme is adopted for solving sparse CCA. The proposed model is then extended to nonlinear function estimation and combined with deep neural networks. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed methods.
main_review: This paper is technically sound and interesting, in that the authors demonstrate how sparse CCA with l_0 norm can be effectively solved, and how the sparsity constraint can be integrated into the deep CCA model. My concerns lie mainly in the experimental section.

In the supplementary material, the authors mention that the parameters are optimized to maximize the correlation for the cancer classification task. Why don’t choose parameters based on accuracy instead of correlation? Does high correlation guarantee better accuracy or vice versa? Are the parameters of all compared methods chosen in the same manner?

2. How are lambda_x and lambda_y selected since they directly control the sparsity of variables?

3. The authors claim that the proposed method is efficient in C.2 (Run Time Analysis), it seems that the model scales when k increases. Besides, training a DCCA model for classification seems to be time-consuming. Therefore, it is also important to compare the time complexity and running time (or training time) of compared methods.

4.  In Table 2, why don’t the authors include the results of the proposed sparse CCA method?

5. The datasets included are mainly for vector-based applications. Recently, there are several tensor-based CCA methods, it is interesting to also report results on some high-dimensional datasets, for example, Gait and JAFFE.
summary_of_the_review: The paper is technically sound, but the authors should carefully revise their experimental section to address my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  I have seen this manuscript before. The main contribution of this paper is to combine two existing works, Gaussian-based continuous relaxation of Bernoulli random variables, gates (Yamada et al., 2020), and sparse CCA (Suo et al., 2017).
main_review: It is very critical to select relevant features from high dimensional multimodal data and such method is very demanding in lots of real cases. 

One key strength of the presented method is that it is able to sub-select relevant features from high dimensional inputs and it showed better performance compared to others for predictive purpose. 

The paper is well written, and the overall presentation of the paper is good.

Questions:

1. How to select the most relevant features in the multimodal learning field is crucial for model interpretation and downstream analysis and it will have wide applications in biology or human genetics. For those fields, researchers are more interested to select which gene, protein, or biomarkers are most important for disease development. Even though the paper showed it’s outperformed predictive performance, however, it missed the detailed model interpretation part, feature selection. I would be interested to see one biology dataset application for model interpretability on this method. 

2. In figure 2, the wide range of lambda showed around 40 coefficients. Please comment on how could the algorithm avoid the wrong number of coefficients?

3. How does the proposed method compare to other methods regarding accuracy vs sparsity?


summary_of_the_review: I like the paper overall. However, I think the model interpretation is more important than the prediction for multimodal learning/CCA, specifically, the key strength of this proposed paper is feature selection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new method for l0-CCA using stochastic gating which allows for an efficient algorithm and also permits a deep-version of the l0-CCA. Results are shown on synthetic and real-world datasets that highlight the superior performance of the proposed approach. 

Main Contributions: 

1). The paper proposes a new approach for l0-CCA that uses a continuous relaxation of a bernoulli random variable via stochastic gating. 

2). The paper extends the proposed method to l0-deep-CCA which allows non-linear interactions between the two CCA views.

3). Results are shown on synthetic and real-world datasets which show the superior performance of the proposed l0-CCA methods.
main_review: Comments:

Originality & Quality: The paper proposes a stochastic gating based approach for l0-CCA which uses ideas from a couple of recent papers on gaussian relaxation of discrete/bernoulli random variables. The resulting algorithm is straightforward and it is also possible to "backprop into it", so it is also amenable to non-linear "deep" extensions. The results on various synthetic and real-world datasets show the superior performance of the proposed method.

While I like the proposed approach, there are a couple of questions/concerns that I have:

1). The paper mentions that standard l0-CCA relies on greedy optimization and that the l1-CCA leads to parameter shrinkage. Both these are valid concerns. Though, it is unclear what recovery guarantees does the stochastic gating approach has? Optimal subset selection is a NP-Hard problem, so clearly the proposed approach is also performing some approximation. Can it be theoretically compared to the greedy l0-solution?

2). The paper shows predictive accuracies of the different CCA approaches on real-world datasets. However, it doesn't show the number of features selected by the different methods and how they compare to each other. If we only cared about predictive accuracy then why select features as shown by (Shalev-Shwartz et al. 2010)? And, also sparsity leads to interpretability so it is unclear how different approaches compare on that front. 



Clarity: The paper is well written and puts itself nicely in context of previous work. The overall presentation of the paper is good.

Significance: The paper addresses an important problem of sparse CCA using a l0 penalty. The proposed approach uses stochastic gating to enforce sparsity which does not rely on a greedy/heuristic solution as previous l0-sparse CCA approaches. 
summary_of_the_review: An interesting paper that solves an "old" problem of sparse-CCA using some recent advances in stochastic gating based sparsity solution. Results are comprehensive, but the focus is less on feature selection and more on predictive power. If predictive power is the primary concern, why care about sparsity?
review_title: null

Meta-review: Canonical correlation analysis is a method for studying associations between two sets of variables. However these methods lose their effectiveness when the number of variables is larger than the number of samples. This paper proposes a method, based on stochastic gating, for solving a $\ell_0$-CCA problem where the goal is to learn correlated representations based on sparse subsets of variables. Essentially, this paper combines ideas from Yamada et al. and Suo et al. who introduced Gaussian-based relaxations of Bernoulli random variables, and sparse CCA respectively. They also extend their methods to work with nonlinear functions by integrating deep neural networks into the $\ell_0$-CCA model. They gave experimental results on various synthetic and real examples, including to feature selection on biological data. The author response addressed a number of the reviewers' concerns, including by providing additional experiments and analyzing the genes selected by their model on the METABRIC dataset. Overall this is a solid contribution both from a theoretical and experimental standpoint.
Decision: Accept (Poster)