Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: A method is proposed for learning correlated representations of two data views that purports to be superior to existing approaches in terms of correlation captured, utility of representations for classification, and training efficiency (since stochastic optimization can be used).
strength_and_weaknesses: The paper employs some interesting techniques and could be of value, but a few points were not clear to me.

1) The proposed technique is claimed to have the advantage of not needing to "balance the two modalities". What is meant by that? How do existing methods for learning correlated representations suffer from "unbalanced modalities"? Using some modality-specific transformation certainly sounds reasonable, independent of concerns about not being "balanced".
2) The derivation of Theorem 1 and (especially) corollary are suspect. How can you maximize the sum of the correlations of the components without any orthogonality/uncorrelatedness constraint? Is there anything then stopping the components from all being equal, specifically all representing the first canonical correlation? That is what the uncorrelatedness constraint is there to prevent.
3) Eq. (5) is not clear to me. How does this represent the Wasserstein distance? Can you explain how this adversarial learning achieves what you claim?
4) In many places, "constraints" are discussed (e.g., "we constrain the latent representations for each of the two modalities to follow the same distribution") but as far as I can tell these are not really constraints; they take the form of penalties in the joint objective.
clarity,_quality,_novelty_and_reproducibility: In light of all of my questions above, I have to say clarity is quite low. Quality is uncertain. I am not familiar enough with recent work to be very confident about novelty. I do know that using separate encoders for each modality, including autoencoders, has been tried. I also know there have been other methods proposed for training DCCA-style models without having to use full-batch optimization as in the original paper. If indeed the approach is novel, it should probably be compared with some of these existing techniques in terms of training time. Since the approach combines several things (autoencoding initial layers, penalizing Wasserstein distance, using dot product objective that can be decomposed over examples) it would also be good to have ablation experiments to determine which of these components are necessary for the claimed performance.
summary_of_the_review: There may be some value to the proposed method, but the motivation is suspect and the paper is less than clear in several places.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the canonical correlation analysis under an adversarial learning framework. They propose an effective adCCA that can learn better representations. The experiments show the proposed method is competitive compared to many recent deep CCA. 
strength_and_weaknesses: The proposed method seems interesting. The experiments are sufficient to show the effectiveness of the proposed adCCA. I have some concerns listed as follows:

1. The preliminary results show that latent features $z_x$ and $z_y$ have the same latent distribution, with a mean of $mu$ and a variance of $sigma2$. And section 2.2 presents this shared latent distribution is achieved by using batch normalization. This is confusing information. As illustrated in Figure 1, adCCA employs two distinct neural networks, $g(x)$ and $g(y)$, to extract the corresponding latent features. However, it's difficult to understand how to use the BN techniques to constrain the two modality features to follow the same distribution. 
2. On page 5, the authors said that ''This permits the use of the modality specific structures and removes the requirement of modality balance.'' Although this sentence is easy to follow, this statements are not supported by experiments or conclusions from previous works. I think it's better to clarify these statements (maybe from relevant literatures). Or this statements seem to be the authors' hypothesis.
3. The descriptions of the neural networks through the whole paper are missing, which is very important for Reproducibility. 
4. According to the Figure 5 and Table 3, we find that higher correlation may not lead to better class prediction. What is the cause of this phenomenon? Is there a trade-off problem between correlation and accuracy? Moreover, we also find that most of the joint representation learning methods achieve worse results. Is this a problem inherent in this type of approach? Or is this problem due to the asymmetric information between different modalities? I think the authors should give more analysis in the experiments.
5.In Algorithm 1, the initialization of $z_x$ and $z_y$ is link to a pre-trained model. So, how much does this initialization process affect the result? For example, do we remove it? Because in the next part, we also update the parameter of the AE network. Moreover, after we achieve two pre-trained AEs, shall we fix the parameter $\Phi$ during the second iterative optimization?

clarity,_quality,_novelty_and_reproducibility: The proposed method is interesting, but the current submission has several issues that need to be addressed. Some details are missing. I write my main concerns and comments in the box of [Strength And Weaknesses].

summary_of_the_review: This paper proposed a deep CCA model under adversarial learning framework. The proposed method is novel and the experiments are sufficient.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work derived a new deep adversary CCA method, which is called adCCA, from DCCA framework. The authors frame CCA problem under the assumption that the representations of different modalities obey an identical distribution in a common latent subspace. Compared to other DNN-based CCA methods, the authors claim 1) their approach is able to learn highly correlated representations across modalities, without deforming original structure of each modal. 2) they simplified analytical solution of CCA problem with their identical distribution assumption. 3) extensive experiments valid their method and demonstrate the robustness of their model. 
strength_and_weaknesses: Strengths:
- They conducted extensive experiments to demonstrate the effectiveness of their method

Main concerns:
- The paper is hard to follow due to the incoherence and imprecision in their writing. For instance, the second paragraph in introduction section "it is temping ...". It is unclear to me what does it refer to in this sentence. the second paragraph on page 2, "This avoids having to balance the two modalities." I don't quite get what messages the authors try to convey via this sentence. 
- All theorems were built on a faithless assumption. They assume "latent representations from modality x and y follow the same distribution $\pi$ with the mean $\mu$ and variance $\sigma^2$" which is weird to me. In practice, the distribution discrepancy is commonly occurred among data that come from different sources/views/modalities. At least, at the initial stage I don't think the representation distribution obey their assumption. However, they derived their theorems from their fundamental assumption which is untrustworthy. 
- The authors emphasized that their method is able to learn maximally correlated representations without comprising structure in each modalities. I was wondering how do the authors prove the structures among modalities were not comprised. How should I understand the correlation of representations is maximized while the structure are not deformed? I would very much appreciate the authors if they can help me understand this.
- I don't think it is fair to claim resolving adCCA with mini-batch optimization as one major contribution of this work since  DCCA (Wang et al.) already confirmed stochastic mini-batch optimization works for DCCA based methods.
- I would expect the authors compare their method with [1,2] from the methodology perspective. 
[1]. Benjamin Dutton, Adversarial Canonical Correlation Analysis, 2020
[2] Fan et al., Deep Adversarial Canonical Correlation Analysis, 2020
clarity,_quality,_novelty_and_reproducibility: Clarity: The current version of paper is not clearly presented. The paper can be further polished to make it easier to follow. 
Quality: The technique parts, for instance theorems, need to be improved, otherwise I don't think the paper is ready to publish.
Novelty: I think the novelty is very limited.
Reproducibility: No enough datapoint as I don't try their code yet.



summary_of_the_review: The current version of paper is not ready to publish since there are some concerns need to be addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work derived a deep adversary CCA method, with the intuition/assumption that representations of different modalities obey an identical distribution in a common latent subspace. 

Strengths:
The authors had a reasonably good survey of literature and some experimental study. 

Weakness:

Most reviewers complained about the writing being not precise and some claims not supported. As mentioned by reviewers, having a mini-batch training procedure does not make a significant contribution as it was achieved by DCCA already. In my opinion, the key difference by this paper is to simultaneously minimize distribution-wise difference in latent space representations, while maximizing sample-wise correlation. It is not very clear whether theses two terms are contradictory or complimentary to each other, and this point should probably be investigated thoroughly. I also have concerns on the two-step approach where a first auto-encoding step is used to extract features for correlation based learning: unless the auto-encoding step is loss-less,  correlated dimensions that are not contributing significantly to reconstruction are already lost in that step.
justification_for_why_not_higher_score: The reviewers found common issues for the paper, and I have additional concerns (discussed in the meta review).
Decision: Reject