Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work extends the contrastive learning based nonlinear mixture identification framework in (von Kugelgen et al. 2021) and considers a case where the two modalities have two different generative functions. The major contribution is the models’ identifiability analysis. The setting and the analysis can be considered as extensions of those in (von Kugelgen et al. 2021). A multimodal nonlinear mixture model is adopted, and a contrastive learning-based criterion is analyzed. The major claimed contribution lies in analyzing the case where two views have different generative nonlinear functions, but in (von Kugelgen et al 2021) assumed that the two views share the same generative nonlinear system.
strength_and_weaknesses: Strength

- The considered problem is interesting, and meaningful. Compared to the work of (von Kugelgen et al. 2021), the two different generative functions for two views, respectively, can be applied to a wider range of applications, e.g., text v.s. image type multiview data.

- The paper is well-written.

Weakness

- Some technical assumptions are not easy to follow. Although the paper is in general pleasant to read, the part around Assumptions 1-2 are a bit hard to digest. Some more explanations and relating to physical meaning may help.

- There work only established block identifiability of the shared components, but did not consider identifying the private components.

- The work's identifiability result was established by assuming that infinite data samples are available, which is not realistic.

**A major concern is that discussion regarding a very relevant existing work is missing**. The submission claims that a major contribution is that the work established block-identifiability of a multiview nonlinear mixture model when different modalities have different generating functions. However, the reviewer hopes to draw attention of the authors to the following paper:

(Lyu et al 2022) Lyu, Qi, Xiao Fu, Weiran Wang, and Songtao Lu. "Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective." In ICLR 2022

This recent work (Lyu et al 2022) has already established a similar identifiability result under a similar multiview model with different view-generating mechanisms. The work was not discussed in this submission, but it is very relevant: 

Some similarities are listed as follows:

- Using this work’s notation, the generative model with x_1 = f_1(c,m_1) and x_2=f_2(c,m_2) was considered in (Lyu et al 2022), and the identifiability of c and m_1 and m_2 were established using a nonlinear multimodal learning criterion. This is quite similar to the generative model considered in this submission, where the latents can be split to shared and view-specific components.

- The identifiability in both works were established via latent shard component matching with invertibility regularization. The ideas share the same spirit. 

- The criterion in (Lyu et al 2022) was also related to a multimodal contrastive learning criterion in Appendix I.2. This makes the learning criteria of the two works even more similar to each other.

I hope to mention that some additional contributions in (Lyu et al 2022):

- Unlike this paper that only considers infinite-sample cases, (Lyu et al 2022) also has finite-sample identifiability analysis.  

- Unlike this submission that only considers block-identifying of the shared components, (Lyu et al 2022) also proved that the private components can be block-identified under some conditions.

The reviewer also acknowledges that there are some differences:

- The model considered in this submission is slightly different, where the private components are split into s and m_i for i=1,2. But the model in (Lyu et al 2022) does not have the s part.
- The assumptions used to establish identifiability are different in the two works.
- The proof techniques are not the same.
- Due to the introduction of s in this submission, the framework in this submission is perhaps more suitable for causal learning.

Given the similarity in terms of the generative models and the learning criteria in both works, the fact that (Lyu et al 2022) is completely missing from the bibliography of this submission is unfortunate. Proper discussions regarding the very relevant prior work (Lyu et al 2022) and existing block-identifiability results may make the contributions of this submission clearer.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is in general written in a clear and easy-to-follow manner. However, the technical assumptions may benefit from more discussion and explanation.

Quality: The paper has a good quality in terms of technical contents and presentation.

Novelty:  One of the major concern of the reviewer is that the paper has missed a very relevant existing work (Lyu et al 2022) that has established block-identifiability of a similar model. But (Lyu et al 2022) was not discussed or mentioned in this paper. This makes the contributions of this work relative to (Lyu et al 2022) a bit unclear. 

Reproducibility: no major issue identified.
summary_of_the_review: This is a well-written and well-organized paper. The major concern lies in its novelty relative to an existing work. This should be properly acknowledged and discussed, which would have made the contributions of this work clearer.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies contrastive learning (e.g., in representation learning with multiview data or multimodal image/caption pairs). This paper presents new identifiability results for multimodal contrastive learning by extending recent work in identifiability results for multiview contrastive learning. Their main contribution is to extend the analysis of multiview non-linear ICA with a single shared latent variable to a multimodal case where it is possible to have a disentangled set of latent variables across modality-shared and modality-specific parts, which all jointly generate the observed multimodal data. Their main result shows that it is still possible to recover shared factors in this new setup, specifically that one can block-identify latent factors shared between modalities. They empirically verify these results on synthetic data and multimodal image/text paired data.
strength_and_weaknesses: Strengths:
1. The paper is well motivated and described. The ideas are clear and experiments are on several large multimodal datasets.
2. The paper is largely clear with clear figures and exposition. It was a joy reading this paper.
3. The paper makes important contributions to the understanding of contrastive, multiview and multimodal learning, with important insights for future work.

Weaknesses:
1. The synthetic and real data experiments should vary the contribution across modality-general style and content and modality-specific to better determine how one can identify the important modality-specific factors across different settings. For example does identifiability get harder with more modality-general information? or more modality-specific information?
2. More real-world multimodal datasets can be experimented with, see https://arxiv.org/abs/2107.07502, especially datasets with a broad range of ratios in modality-general vs modality-specific information.
3. The theory, while seems to be right and justified, is rather high-level, and does not delve into the details of learning and optimization (e.g., guarantees on how to learn such disentangled latent variables, or to encode different modalities). A lot of the proof follows by extending prior work in multiview ICA combined with the newly made assumptions in the paper, which is okay, but seems rather high-level. What if the assumptions are not made? What are some directions to further improve this line of theoretical study?
clarity,_quality,_novelty_and_reproducibility: The paper is largely clear with clear figures and exposition. It was a joy reading this paper. The paper makes important contributions to the understanding of contrastive, multiview and multimodal learning, with important insights for future work backed up by theory and experiments.
summary_of_the_review: The paper makes important contributions to the understanding of contrastive, multiview and multimodal learning, with important insights for future work backed up by theory and experiments, but there can be deeper empirical experiments and concluded insights to be more impactful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a contrastive learning method combined with a generative process where the latent space involves the disentangled latent variables "content", "style" and "domain-specific" elements. The authors provide theoretical analysis using AlignUniform loss, a variant of the InfoNCE loss and demonstrate that contrastive learning in a multi-modality setting could block-identify the common latent attributes across domains like contents. Empirically, the authors conduct simulations with InfoNCE loss on both synthetic and Causal3DIdent data for numerical justifications. 
strength_and_weaknesses: **Strength**

The paper is well-written and organized, making it easy to follow. The proposed generative model is well presented and justified in a mathematical way. Combining the generative model with the contrastive framework is fairly novel in the ICA context. The generative process is reasonable and can be naturally combined with the contrastive framework. The theoretical analysis is well-defined with the given hypothesis. From the empirical perspective, the authors have shown their main experiments reasonably, and the results support the claims of the paper. The details included should be sufficient for reproducibility. 

**Weakness**

Although the math analysis is clearly elaborated, the training of the proposed method is not clarified and I need to figure it out by reading their code. Also, after reading through the code, I found some details are omitted in the paper. For example, the generative function f and the contrastive encoder g need to be separately trained (please correct me if I understand it in the wrong way). Maybe adding a paragraph or an algorithm box will better clarify this part.

The practical training loss and the loss used for analysis are not the same. Although these two loss functions have a close correlation, there exists an approximation gap in terms of the number of samples used in the calculation. 

The generative process also introduces several variables/hyper-parameters that need to handle. For example, the generative functions $f_1$ and $f_2$ need to be invertible; the dimensionality and covariances of the latent variables need to be carefully chosen. Some additional ablation studies might be needed to address these concerns. 

The latent variables, even the whole framework, should be affected by the performance of the generative functions. The authors should provide either theoretical or empirical studies to instruct the impacts. For example, how does the contrastive encoder perform in terms of the reconstruction error of the generative models? 

For practical use, since the framework has some assumptions which can hardly be satisfied in real-world cases, it is hard to evaluate the contributions from this perspective. The experiments are conducted at the toy level, and it is unclear what would happen in bigger and more complex data. Some relaxation or approximation are in need in the future.

Some math notations are not fully explained. Please see the clarity part.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is overall clear to the readers. I list some points that are not precise below:

The notation in 5.1 needs clarification. I may miss out on those definitions, but I did not find the definition of $\boldsymbol{a}$ and $B$, which are related to the mean of style distribution. 

The presentation of Fig. 3 is a little bit confusing to me. For example, in (a),  it seems the authors mix the results of categorial and numerical attributes in the same plot. Are there any specific reasons to do so? On the one hand, it weakens the legibility of the figure; on the other hand, these metrics are not comparable in the same plot.



**Novelty**

The proposed method has novelty in the context of multi-modal ICA, though there are some existing works in combining self-supervised models with generative models in both single- and multi-modal scenarios. 

**Quality** 

The proposed method is technically sound. The theoretical and empirical analyses are comprehensive. Some ablation studies may be needed.

**Reproducibility**

The authors include details for the algorithm and model parameterization, which should be useful for reproducing their results. A demo code is also provided.
summary_of_the_review: The topic of this paper is interesting in the study of contrastive learning in a multi-modal case. The proposed method is well supported with both theoretical and empirical evidence. Some important perspectives regarding the proposed method are either not clarified or comprehensively studied at this moment.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addressed the problem of identifiability for multimodal representation learning and showed that contrastive learning can block-identify latent factors shared between heterogenous modalities. The authors first prove this theorem based on some assumptions. Afterwards, some experimental results are offered to corroborate their theoretical results.
strength_and_weaknesses: This paper proves that contrastive learning can block-identify latent factors shared between heterogenous modalities from both theoretical and experimental aspects. The authors describe their settings and experiments in a specific way, which is understandable.
However, I still have some concerns about this paper. 
1) Theorem 1 assumes that the number of content variables is known or that it can be estimated. It seems satisfied in the settings of Multimodal3DIdent. But what about more complex natural datasets like COCO? Or some modalities beyond image and text? It needs more in-depth discussions, although the authors have mentioned in the limitation part. 
2) In the experimental part, the authors use Convnet for text encoding, which is the same as the convolutional network used for image encoder. Why not use more general and classical sequential neural network, like LSTM or GRU? They are more widely used in multimodal tasks.

clarity,_quality,_novelty_and_reproducibility: This paper is in good shape and the writing is understandable. The authors clearly express their main idea through this article.
This paper extends the previous work on the proof of identifiability to the multimodal setting. To some extent, it can help the research of multimodal representation learning.

summary_of_the_review: In general, this article is well organized and reasonable, and the conclusion is helpful. But I still have some concerns, see Strength And Weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a novel identifiability analysis of a certain class of multimodal contrastive learning algorithms. The result is centered around a model with a latent variable partitioned into content (shared across modalities), style and and modality-specific information. The main theoretical result guarantees a form of block identifiability of the shared latent content. This work distinguishes itself by allowing for certain types of dependencies between the latent factors and by allowing for two distinct decoders, corresponding to the two modalities. The theory is illustrated in both a controlled synthetic dataset and a more realistic one presenting image/text pairs.   
strength_and_weaknesses: Strengths:
- This paper is very well written and fun to read (even on a Saturday morning).
- The theoretical contribution is significant.
- The technical assumptions are very clearly presented, a quality which is sometimes lacking in this line of work.
- The topic is timely, given how popular multimodal contrastive learning is.
- Presents a realistic experiments with image/text data.

Weaknesses:
- I would have appreciated more experiments where theoretical assumptions are violated, for example, what happens if Assumption 2 doesn't hold?
- The theory applies only to continuous data (as is made clear by the assumption that both encoders f_1 and f_2 are diffeomorphisms), but the main applications of multimodal contrastive learning is with image/text data.
clarity,_quality,_novelty_and_reproducibility: Clarity:
- As I said, this paper is very clearly written. 
- Assumption 2: I was a little bit confused at first with how (8) relates to p_(\tilde{s} | s). It might be useful to write explicitly that p(\tilde{s} | s) = sum_A p(A)p(\tilde{s} | s, A).
- Definition 1: “...contains all and only information about c, i.e., if there exists an invertible function…”. What is said before the “i.e.” is different from what is said after it: even if \hat{c} = h(c) for some invertible function h, \hat{c} might still contain information about s. Statistical independence and disentanglement are not the same. This is actually discussed in 5.1. A similar problematic formulation is employed in “Implication and scope” of Section 7.
- Theorem 1: I was slightly confused when reading that p_{z} must be a continuous density, since at first I thought z contains \tilde{s}, and if that were the case, that would imply that p(A) puts all of its mass on A = {1,..., n_s}. But z does not contain \tilde{s}. Maybe that’s worth emphasizing. 

Novelty/originality:
- I have never seen this result before. 
- Even if this work clearly builds on previous identifiability results, it seems to extend them in a non-trivial way. 

Improvement suggestions:
- I could not find the full joint over p(z) in the main text. It might be worth writing it down somewhere, to make explicit the fact that s can depend on c.
- Looking at the generative model for the latent variable z (figure 1) I find myself wondering which pieces of the model are absolutely crucial for the identifiability result to follow through and which are just there for sake of generality. For example, does the result follow through when either m_1 and m_2 are empty (i.e. just absent from the model)? Is dependency between s and c necessary? Is the dependency between s and \tilde{s} necessary? What if s and \tilde{s} are empty?

Minor:
- Typo in Assumption 2 (ii). “... is smooth w.r.t. Both \tilde{s}_A and s_A” ?
- The authors might want to consider citing “Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA” by Lachapelle et al (2022) in “Limitation and outlook” when mentioning interventions on the generative process and actions in reinforcement learning.
summary_of_the_review: I very much enjoyed reading this manuscript. It is well written, easy to follow and its topic is very timely. The identifiability results are novel and very clearly presented. I found the experiments with the realistic image/text data to be very convincing, although I was expecting more experiments where the assumptions of the theory are violated. The main drawback is that the theory applies only to continuous data and that one of the main application of multimodal contrastive learning is image/text data, but this point is acknowledge by the authors. Still, this remains an important contribution and I am very happy to recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper provides identifiability results for multi-view SSL. The problem setup involves shared variables, style variables, and private variables, and is a combination/generalization of

von Kugelgen et al. 2021: same modality (one generation function), latent space consists of content + style, dependencies between content and style are allowed
Lyu et al. 2022: two modalities (two generation function), private variables are independent from shared ones.
The data generation process combines formulations from both.

The present paper shows the extracted latent representations matched by the two modalities are the shared variables, and thus cannot contain style and cannot contain private variables.

Strength:
The paper studies a setup that generalizes those of previous work, and still achieve identifiability.

Weakness:
While the paper solves additional challenge, the proof techniques mostly follow that of previous work (specifically  von Kugelgen et al. 2021).
justification_for_why_not_higher_score: The proof of main theorem largely reuses the proof techniques of von Kugelgen et al. 2021. von Kugelgen et al. 2021 showed the "can not contain style" result without private variables, and allows dependency between content and style. The current paper additionally argued the "can not contain private variables" part, with an argument different from Lyu et al. 2022.

The new development here is, in my opinion, somewhat limited/incremental.


justification_for_why_not_lower_score: Reviewers are generally happy with the results and the presentations.
Decision: Accept: poster