Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This manuscript demonstrates a modality gap phenomenon for multi-modal contrastive models like CLIP. Specifically, the authors analyze why the gap exists and its importance; (a) the modality gap is born at random initialization and the contrastive learning objective encourages the gap, and (b) changing the modality gap can affect zero-shot and fairness performances on downstream tasks. Furthermore, the authors provide a theoretical analysis of the modality gap phenomenon. 
strengths_and_weaknesses: Strengths
- The writing is clear and easy to understand
- This manuscripts study interesting intrinsic phenomenon of contrastive-based multi-modal models (e.g., CLIP)
- Theoretical analysis supports the existence of the modality gap at randomly initialized weights

Weakness
Overall, the backups are not enough to support the claims and the improvements in the experiments are marginal
- The cone effect phenomenon in pretrained models; for example, the authors claim the avg 0.56 cosine similarity score (in Figure (a)) is high enough to show the embedding space is a narrow cone. But, compared to the random initialization (in Figure (b)), the score is already significantly reduced from 0.99 to 0.56. Is there any other baseline (such as standard contrastive learning on ImageNet) to show the values are meaningful?
- Inconsistent claims; the authors claim closing the gap increases the contrastive loss. But, Figure (d) contradicts the claim - closing the gap reduces the contrastive loss. In my understanding, the contrastive loss forces to close the gap between given pair of data (e.g., image and text pair), where the temperature parameter (i.e., sharpening parameter) in contrastive loss can control the magnitude of the loss. For example, Figure 8 in Supp. shows that high temperatures can remove the gap. So, after training, the gap could be negligible in cases.
- Experimental results are not supportive; improvements in Tables 1 and 2 seem to be marginal and simply obtained from the best results from the search space of hyperparameter $\lambda$ in Sec 4.2.
- The authors do not provide a proper guide for the modality gap; which one brings the benefits for multi-modal learning - closing or increasing the gap?
questions: I hope the authors could resolve my concerns in the weakness part above.
limitations: The authors do not provide any limitation or potential negative social impact of their work
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates the geometric misalignment of multi-modal data representations in CLIP representation space. The authors investigate the existence of the gap under different random initialization, real or random input data and during the optimization of the model with contrastive learning objectives. They also provide theoretical analysis of the misalignment.

strengths_and_weaknesses: Strengths:
The paper is well written and provides extensive analysis (both theoretical and empirical) of the geometric misalignment of representations under different modelling assumptions. Proofs and code are provided. The influence of the geometric misalignment is also investigated when using representations for real-world downstream tasks.

Weaknesses:
As far as I am aware, the observation about the geometric misalignment of multi-modal data representations is not novel but is typically referred to as heterogeneity gap (discussed for example, in https://ieeexplore.ieee.org/document/8715409). 
While the experiments are interesting and insightful, I am doubtful about the significance of the results. In particular, I am not convinced by the results on the downstream tasks which are in my view the important ones. The authors investigate only one value of the modified gap and the results are not very significant (especially for Table 1). 

questions: I kindly ask the authors to elaborate on the following questions:
- In Sec 2.1, you investigate the cone effect on 3 pretrained models. I was wondering if this could be specific to the MSCOCO dataset and/or the selection of the 5000 embeddings? 
- You never specify the dimension of the representations. How does this affect the gap? It is a known fact that higher dimensions lead to the diminishing effect of the distances.
- In general, I think it would be helpful to quantify the extent of misalignment. This could be done with geometric methods for evaluation of representations such as Delaunay Component Analysis (Poklukar et al, ICLR 20211),Improved Precision and Recall (Kynkäänniemi et al, NeurIPS 2019) or similar. 
- The experiments suggest that the gap is present either when network parameters follow a Gaussian distribution (related to initialization) or the data (when using random noise). I assume the input data is also normalized before given to the network, and that the network uses batch or layer normalizations. Have you perhaps investigated whether the gap arises in more “non-Gaussian” setups?
Is the gap specific for the cosine similarity?

limitations: The experiments in the paper are specific to the CLIP model, which limits the scope of the work and makes it more difficult to judge the significance of the contribution. If no other models are added into the evaluation, I believe the introduction and abstract should reflect that, i.e., I do not think it is fair to claim that the work investigates the gap for general multi modal contrastive representation learning models if this is in fact not the case.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes ‘modality gap’ in multimodal representation learning. The authors observe that embeddings of different modalities under a CLIP-style contrastive training objective will fall in different regions of the embedding space. They further provide explanations for the modality gap and a theoretical analysis.

strengths_and_weaknesses: \+ The paper presents an interesting phenomenon and unique insight in multimodal representation learning

\+ The conclusion is evaluated with many experiments and backed with theoretical analysis.

\+ The paper is well written and clearly organized. 

&nbsp;


\- What’s unclear to me is that: modality gap seems to stem from the inductive bias of deep neural networks. The key is that using two encoders (with different initialization) to process input data will produce a gap in the representation space, and a contrastive learning objective preserves the gap. I don’t quite see the influence of data modality in the problem statement. As stated in L108-110, the gap exists even if the two encoders operate on the exact same data (i.e., no multi modality). If so, why is the gap termed as ‘modality gap’? What differentiates the gap in a multi-modal learning setting from a uni-modal setting? 

If we consider a uni-modal setting, take Mean Teacher [1] in semi-supervised learning as an example: given an image, the consistency loss aims to minimize the predictions of a teacher and student model. The teacher model is the exponential moving average of the student model weights (i.e., the two models have different weights and operate on same data modality). Does the gap still exist here if we use a contrastive learning objective? 

Conversely, if we use a single model for multiple modalities (e.g., Omnivore [2]), where different modalities are tokenized and fed into the same transformer. Will the modality gap still exist here?

[1] Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.
[2] Omnivore: A Single Model for Many Visual Modalities
questions: + Figure 3, the objective of contrastive learning is cosine similarity while Figure 3 plots Euclidean distance. Could the authors provide plots for the cosine similarity or comment on reasons for using Euclidean distance? Is is possible that the embedding shift is related to difference between cosine distance and Euclidean distance?

+ Is the proposed modality gap specific to multi-modal learning? Does the gap exist for unimodal contrastive learning?

+ Section 4.3 (simulating mismatched data) seems an interesting investigation. What’s the multi-modal data being used here? Also, I wonder if the authors have some insight on how the modality gap correlates with multimodal data misalignment (e.g., an image is paired with a wrong caption due to data collection error). 

+ The improvement in Table 1 seems quite small. Is the reported number given by exhaustively iterating over all possible shifting? Are the results averaged over multiple runs / random seeds?

+ Line309: shwon -> shown
limitations: 
As the authors mention in the paper, an important future direction is to investigate how the modality gap influence downstream task performance.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper investigates the modality gap issue in existing multi-modal models. Specifically, this work empirically shows the gap phenomenon across different neural networks and different modalities, and makes a conclusion that the gap issue is a combination of model initialization and contrastive learning optimization. This paper further theoretically proves how inductive bias in deep neural networks creates narrow representation cones in the embedding space. Another contribution of this work is to empirically investigate the impact of the modality gap distance in downstream tasks.

strengths_and_weaknesses: ---

Originality: to the best of my knowledge, this is the first work that investigates the modality gap of multi-modal models. It is interesting to see that neural networks with random initialization create narrow cones in the embedding space. This paper also theoretically proves how it happens.

---

Quality: 

Strengths: the empirical study and theoretical proof of the cone effect in deep neural networks are solid and well supported. Specifically, this work first empirically demonstrates that the cone effect widely exists in deep neural networks, then theoretically prove that each network layer narrows the representation cone.

Weakness: (1) as shown in Table 1, the impact of modality gap on downstream tasks is marginal. (2) All experiments are performed on small-scale datasets. For example, the pretraining is performed on MSCOCO, and downstream tasks are performed on CIFAR10, CIFAR100, and SVHN. What’s the zero-shot performance on ImageNet? If the CLIP model pretrained on large-scale datasets, does the modality gap still have an impact on the downstream tasks? 

---

Clarity: this paper is well written and well organized. It is easy to follow. Since this is an empirical work and built upon existing models, there is no reproduction issue.

---

Significance: the investigation of the modality gap in multi-modal models is interesting and would benefit the community.

---

questions: 1. What’s the zero-shot performance on ImageNet?
2. If the CLIP model is pre-trained on large-scale datasets, does the modality gap still have an impact on the downstream tasks? 

limitations: Not applicable
review_title: null

Meta-review: metareview: This paper investigates the gap between representations when training with a contrastive objective, through the characterisation of the gap in various settings, and building a theoretical analysis of this gap.

The reviewers mostly agree that the paper tackles an interesting problem through investigation and characterisation of the inductive biases provided by CLIP-style models, and the experiments appear to cover a good number of cases.

The primary issues with the work however appear to be with some of the framing---it comes across as an investigation into something a bit more generic than the title suggests, and the claims to novelty, while reasonable, are also a bit too strong given the existence of the heterogeneity gap. The authors argue that finding that multi-modal data project to separate subspaces is somewhat reasonable, but I still don't think that supports as strong a claim as given.

On balance, though it appears as if the paper has more merits than issues, and most of the issues raised could be addressed with a bit of work. I would strongly urge the authors to actually make the edits for framing, clarity, and incorporating the additional experiments from the rebuttal into the manuscript, as requested by the reviewers.
confidence: Less certain
Decision: Accept