Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ### Scope & Problem setting
The paper studies the problem of representation learning from multi-view data from an identifiability perspective. In particular, the focus is on latent correlation maximisation approaches as commonly used in (nonlinear) canonical correlation analysis (CCA) and artificial multi-view self-supervised learning (AM-SSL). The paper postulates a generative model by which views $(x^1,x^2)$ are generated by invertible functions $f_i$ applied to a mixture of independent shared $z$ and private $c^i$ components, $x^i=f^i(z,c^i)$, for $i=1,2$. 


### Theory
 It is shown that latent correlation maximisation (with a suitable objective and constraints, as well as with invertible encoders) identifies, or separates, both (i) the shared component $z$ and (ii) the private components $c^1,c^2$ up to arbitrary invertible functions. Further, (iii) a finite sample analysis is provided.


### Algorithm & Experiments
The paper proposes a learning objective that combines latent correlation maximisation with a regulariser and a reconstruction objective to encourage independence among $(z,c^1,c^2)$ and invertibility of the encoders. Several experiments on synthetic data and image benchmarks are used to validate the theory and compare the proposed method against DCCA, BarlowTwins, and BYOL.
main_review: ### Strengths
- The paper presents sound (I only checked the proof of Thm. 1 in detail and could not find any major flaws.) theory proving identifiability of both shared and private components for the assumed multi-view setting.
- A finite sample analysis and result is provided which is rare in this area and thus appreciated.
- The paper is overall well-written and well structured. 
- Based on the presented theory, a new learning method is developed and implemented.
- The experimental evaluation is relatively diverse (multiple datasets and baselines)


### Weaknesses
- The theoretical analysis relies on two relatively strong and restrictive assumptions. 
   - The first is **invertibility of the learnt encoders** which, as stated by the authors, ensures that all information is preserved and corresponds to assuming the reconstruction task is solved. In their proposed algorithm, this is implemented in the form of a reconstruction objective within an auto-encoding framework. However, the (DCCA and) AM-SSL methods cited in the paper do not use such reconstruction objectives, partly owing to the complex nature of the data these methods are often applied to (e.g., natural images where perfect reconstruction is not feasible). Instead, these methods avoid trivial representations, e.g., via redundancy reduction (BarlowTwins) or using moving averages/stop gradients (BYOL/SimSiam). Unfortunately, **this undermines one of the paper's main claims that its theory explains the success of DCCA and AM-SSL methods**. In order to justify this claim, further theoretical analysis would be needed that explains why the aforementioned methods still work *without explicitly enforcing invertibility*. There is existing work that actually addresses learning with non-invertible encoders, see, e.g., [S1, S2], (von Kügelgen et al., 2021); cf. contrastive learning theory literature [S3, S4]. 
   - The second is **mutual independence between shared and private components**. This assumption seems central to the proofs of Thms. 1  and 2, but is only very briefly discussed in 3.2. This assumption may be less restrictive than invertibility (see also Questions below), but it deserves a more extensive discussion and justification. Moreover, since specifying a latent distribution is part of specifying a generative model, I believe this should be moved earlier to the beginning of 3.1 and inform the comparison with other generative models of multi-view data. 
- For *some* of the theoretical results, it is unclear to what extent they are novel or can already be found in similar form in the literature. In particular, there seem to be large parallels between Thm. 1 and the cited work of (von Kügelgen et al., 2021) which also identifies the shared latent component without the above assumptions and uses the same notion of identifiability up to invertible function. However, this is not discussed in the theory or related work sections.

### Questions/comments for the authors
- For the assumed generative model, starting from a setting where $(z,c^1,c^2)$ are *dependent*, can this equivalently be transformed into a setting with independent latents (and possible different $f^1,f^2$), and, if so, how? In other words, does the independence assumption come without loss of generality when arbitrary invertible $f^1,f^2$ are allowed?
- It seems that $D,D_1,D_2$ are implicitly assumed to be known. If so, this should be highlighted as an assumption. Also, did you perform any ablation on this? 
- The finite sample analysis at the end of 3.3 is hard to parse for non-experts. It would be nice if you could provide some additional intuition on Assumption 2 and Thm. 3.
- I did not understand exactly how the clustering experiment in 6.1 directly evaluates Thm. 1, i.e., the existence of an invertible function mapping between ground truth and inferred latents. Could you please clarify? In particular, how are the ground truth shared components used in the evaluation? It may be nice to complement the existing presentation with performance at predicting the ground truth latents, or is this the reported clustering accuracy in Fig. 1? Could you also show t-SNE for $\hat{z}^2$?
- What does your analysis have to say/how does it relate to contrastive AM-SSL approaches such as SimCLR? Can this also be phrased as  latent correlation maximisation?

### Other minor comments, suggestions, and typos
- p.2, 2nd para: The claim that "theoretical understanding has been lacking" seems too strong and should be toned down and complemented with references, acknowledging at least some of the many works from previous years that seek to theoretically understand CCA & AM-SSL.
- p.2, (i): the key assumptions (e.g., invertibility) and type of identifiability (up to invertible function) should be stated here for transparency 
- p.3, end of 2.1: presumably the criterion should hold for all x?
- p.7, 1st para: is "HSIC [...] maximizes the correlation" a typo and should read "minimizes"?
- p.7, Reformulation para: invertibility cannot be "enforced" by using an autoencoder (at least in practice), this should be toned down, e.g., to encourage or promote as used in the next sentence
- p.14, 3rd last para: there seems to be a typo in that the partial derivative of $h_S$ w.r.t. $c^1$ (instead of $z$) should be zero; otherwise this contradicts the first part of the sentence.
- p.14, 2nd para: $h_P$ has not yet been defined

### References
[S1] Zimmermann, Roland S., et al. "Contrastive Learning Inverts the Data Generating Process." (2021).

[S2] Tian, Yuandong, Xinlei Chen, and Surya Ganguli. "Understanding self-supervised learning dynamics without contrastive pairs." (2021).

[S3] Arora, Sanjeev, et al. "A theoretical analysis of contrastive unsupervised representation learning." (2019).

[S4] Tosh, Christopher, Akshay Krishnamurthy, and Daniel Hsu. "Contrastive learning, multi-view redundancy, and linear models." (2021).
summary_of_the_review: Overall, I enjoyed reading this paper. However, I have two main concerns (see Weaknesses above for details): (i) the invertibility assumption used to obtain the theoretical results is at odds with the fact that latent correlation-based multi-view learning typically does not use invertible encoders. This is not made sufficiently clear, and the claims that the presented analysis explains or helps understand the success of current methods is not fully justified and should be toned down. (Such an explanation would also have to explain why these methods work without invertible encoders.) (ii) part of the theory appears weaker than results in (von Kügelgen et al., 2021) which are not compared to in sufficient detail; in particular, I encourage the authors to include a more complete comparison.

I believe the paper also provides valuable novel contributions, in particular, the finite sample analysis (Thm. 3), the new learning method (Sec. 4), and the thorough experimental evaluation (Sec. 6); I actually see these as the main contributions (Thms. 1 and 2 seem to follow quite directly from invertibility and independence) and think they could feature more prominently. Together with addressing (i) and (ii) above, this would make the paper a solid and well-rounded submission, and I remain open to increasing my score depending on the authors' response.

### Post-rebuttal updates: 
The authors have provided detailed responses to my comments and have made several modifications to the manuscript that satisfactorily address my two main concerns (see the discussion below for details). I have decided to increase my score as a result.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper offers a theoretical analysis of neural canonical correlation analysis (CCA) type methods. This reveals conditions under which these methods may work. The paper then proposes regularizers that approximately realize these conditions, which result in a new CCA-type algorithm. Empirical results indicate that the approximate algorithm behaves as dictated by the theory.
main_review: Multi-view data (e.g. getting both visual and audio data of the same situation) is often analysed with CCA-style algorithms. Contemporary nonlinear CCA models rely on neural encoders to find suitable representations. This paper develops a theoretical analysis of a particular CCA model. This model differs in subtleties from existing models, but reflect the same intuitions as current state-of-the-art. The paper provides theoretical evidence that suitable representations can be recovered from multi-view data. While I am not an expert in this area, this does seem like valuable insights.

To me, the largest caveat of the paper are the subtleties which is where the proposed model differs from others. These allow for theoretical insights, but it is unclear to which extend the assumptions are reasonable. For example, 

* In Eq. 6b it is stated that $f^{(q)}$ should be invertible. This seem like a rather strict assumption in practice. In particular does it not imply that $f^{(q)}: \mathbb{R}^D \rightarrow \mathbb{R}^D$? Is it reasonable that $f^{(q)}$ is dimensionality preserving?

* Eq. 6d seems to be a rather important assumption about independence, but I find it rather hard to determine if that's a suitable assumption. I get that it comes with mathematical convenience, but is it reasonable?

In general, I would have liked the paper to have a remark after both Assumption  1 & 2 about the reasonability of these assumptions.

Algorithmically, the assumed constraints are converted in "soft constraints" (regularization) and an appropriate optimization is proposed. While the empirical analysis is limited to fairly simple settings, the evidence does point towards these regularizers working well. It would have been nice with an empirical study of the sensitivity towards the scaling of these regularizers (e.g. $\lambda$).

### Minor comments ###

* In Sec. 3.1, the paragraph "Generative Models of Multiview Analysis" feels more like a comment on previous work and it broke my flow while reading. Perhaps this paragraph could be moved elsewhere?

* In Theorem 2, it is unclear what is meant by "a certain invertible function".

* The citation associated with t-SNE in Sec. 6.1 appears to be about SNE. This should be corrected.

* The parenthesis remark in the conclusion ("which is later realized...") should perhaps indicate that the realization is approximate.

### Update after rebuttal ###
I am happy with the given replies and retain my positive score.
summary_of_the_review: The paper is generally easy to read and tackles an interesting problem with a novel solution (as far as I can tell). The issues I have pointed to can fairly easily be fixed. Hence the positive score (I would have given a 7 if that was an option).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a general multi-view learning approach and provides a theoretical analysis for the proposed method. Specifically, each view is considered as a common/shared and private/view-specific components. Then, the multi-view learn problem is converted to a identification and disentanglement problems. The theoretical analysis provides the bounds of the proposed model. Experimental results demonstrate the effectiveness for downstream tasks.
main_review: This paper proposed a novel approach for multi-view learning. The pros and cons are listed below:

Pros:

The general motivation of the proposed model is reasonable and logical.

Consider there are a few multi-view learning works provide the theoretical analysis, while this work gives a comprehensive analysis of the model. It is potential to extend to wider multi-view methods for deeply understanding multi-view scenario.

The experimental results demonstrate the effectiveness of the proposed modules.

Cons:

The experimental results mainly based on relatively small and simple datasets. The potential extension to a more large and sophisticated datasets and sample formats should be discussed.

The computational cost of the proposed model should be discussed. The potential solution for large-scale applications should be discussed.

summary_of_the_review: This paper proposed a novel approach for multi-view learning problem. A theoretical analysis of the proposed model is provided. In experiments, most of the datasets are small scale and simple data format. More discussions and analysis about its potential real-world applications could be discussed.
review_title: null

Meta-review: All reviewers agreed that this is a strong paper, that the methodological contributions are both relevant and significant, and that the experimental validation is convincing. I fully share this viewpoint!
Decision: Accept (Spotlight)