Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposes an approach to find a map between two feature spaces to maximize correlation between them and to use the resulting map for inference. A theoretical exposition is given and some empirical results are provided showing that the approach speeds up convergence on supervised MNIST and can be used for image completion (again on MNIST).

The paper should be rejected for the following reasons. First, the approach looks very similar to deep CCA, but the connection is never mentioned. This connection needs to be clarified. The objective function needs to be clearly stated and related to the loss function in eq. (7). In particular, I would suggest to give a clear definition of the problem before delving into the theory in section 2. In its current version, it is difficult to assess how the parts of section 2 relate to the overall objective. The paper severely lacks in relation to relevant related work. Half(!) of the 14 referenced papers are by the author himself. This can be verified since the double blind review process is compromised as the paper links to code in the author’s public github account. Finally, the empirical results are quite incomplete. It is not clear how the results compare to generative methods like VAEs, which are referenced as a motivation for this work in this work.

The improvement in convergence from RFA for supervised learning is interesting and this aspect deserves more analysis. It would be useful to look at the total amount of computation required to reach a given loss. I also wonder how this differs from simply mapping the output of the first network to a low-rank space via PCA. Is the dual-view really necessary in this case since the information content in the label space must be very limited, beyond simple class balance statistics?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper considers how to learn correlations between two spaces, e.g., input/output, in order to generate data in one space conditioned on values from the other. This is performed by modeling features with neural networks and optimizing an objective function that maximizes a measure of correlation between the features versus learning a generative model such as a CVAE. Some illustrative examples using MNIST are provided.

My decision is to reject. I think there is value in the approach, but it is hard to see clearly at the moment given that the exposition is difficult to follow and the experiments aren't very compelling. If these issues could be addressed (concrete suggestions below), and some of the follow-on work in the last section could be performed, I think there could be a pretty interesting contribution here.

***

Decision-related suggestions/questions:

* Include more datasets in the experimental section. The second sentence of the introduction lists possibilities such as time series and multi-modalities that I would have been very interesting.

* The first claim that there is no tunable variable in the objective function is a little hard to parse. Clearly, the rank of the low-rank approximation must be set, and the features of the two spaces need to be learned. Some clarification here would be helpful. 

* There are a number of unfortunate typos/grammar issues/presentation choices that really impact clarity. Some examples:
	* In the third paragraph under theory, "...linear spaces spanned by the probability distributions..." should probably be "...linear spaces spanned by all probability distributions..." (?)
	* The following sentence is a run-on. 
	* In (8), occurrences of g_*(x_n) should be replaced with g_*(y_n).
	* The replacement of the (low) rank symbol, k_0, with the sample size symbol, n, in the second paragraph of section 4.1.
	* Introducing a "Bayesian estimator for an l^2 distance" w/o explanation. What does this mean?

* How should the low-rank parameter k_0 be selected generally given that the singular value distribution may not always be useful in selecting it?

* Can anything be said quantitatively or qualitatively about the sample complexity required to estimate the matrices of (8) well enough to estimate the features?

* Is there an interpretation for why both spaces require the same feature dimension, k_0?

***

Comments not related to decision:

* It is generally good to avoid sweeping statements such as the first sentence of the introduction. Perhaps consider replacing with a simple statement on the intended goal of the paper: "...produce a useful model of correlations... for the task of data generation..."

* Consider placing a concrete, motivating example prior to the theory section as it is hard to digest (from an ML perspective) without a clear context. The analytical example with the Gaussian from the supplementary material is one option.

* The last statement of the paragraph under (3) needs a reference.

* It seems strange to have the supervised learning experiment of 4.1 as the first experimental result of the paper since it is an unintended and unexplained side-effect of the approach. Also, the claim of "faster convergence" should be demonstrated in wall-clock time.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes how the correlation between two different types of data can be extracted from learned representations. The proposed metric can also be used as an alternative to cross entropy loss. The paper provides analytical calculations as well as real data sets simulations/experiments. However there are significant draw-becks:

1) Similarity and Metric Learning is a booming area in machine learning with several different directions focusing on different problems. The paper fails to locates itself in the literature, how it compares itself into other techniques (both analytically and experimentally). 

2) The proposed technique seems to be very similar to SVD of learned representations. Connection to quantum field theory is well established but more simpler comparisons to SVD and other spectral techniques are not provided in metric learning.

3) Novelty is not clear. There are interesting experiments in disentangled feature feature extraction and data generation. However, they are mostly proof of concept and lack of baselines. It is not clear what problem this technique solves better compared to other existing solutions. 

Paper is mostly written clearly. I do suggest putting Appendix A1 to the main paper though.
review_title: Official Blind Review #3

Meta-review: This manuscript proposes an approach for estimating cross-correlations between model outputs, related to deep CCA. Authors note that the procedure improves results when applied to supervised learning problems.

The reviewers have pointed out the close connection to previous work on deep CCA, and the author(s) have agreed. The reviewers agree that the paper has promise if properly expanded both theoretically and empirically.
Decision: Reject