On Erroneous Agreements of CLIP Image Embeddings

Authors: Siting Li, Pang Wei Koh, Simon Shaolei Du
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multimodal Learning, CLIP, LLaVA, cosine similarity, erroneous agreement

Abstract:
Recent research suggests that the failure of Vision-Language Models (VLMs) in visual reasoning could be attributed to the CLIP image encoder ambiguously encoding distinct images into embeddings with high cosine similarity, namely *erroneous agreements*. In this paper, we show that they are not the sole issue, as multimodal large language models (MLLMs) may extract distinct information even from image embeddings with high cosine similarities. On Subset A of the What'sUp benchmark, where the Left/Right image pairs are embedded by CLIP with average cosine similarity greater than 0.99, CLIP's performance is near random guess. In contrast, LLaVA-1.5-7B, which uses the same image encoder as CLIP, achieves nearly 100\% accuracy. This discrepancy is also observed between LLaVA-1.5-7B and CLIP-like models on similar benchmarks. To investigate this performance gap, we conduct controlled experiments to test the effect of varying evaluation methods, training data, and language processing choices. We find that the CLIP image embeddings contain more extractable information than previously suggested, but it is likely obscured by the inadequate vision-language alignment of the CLIP's paradigm. Motivated by this observation, we reconsider the LLaVA-1.5 model on the MMVP benchmark, for which prior work showed that it could not distinguish image pairs with high cosine similarity. We observe a performance gain brought about by an alternative decoding algorithm, which attends more to visual input. Further, we show that the accuracy significantly increases if the model can take both images as input to emphasize their nuanced differences. Both findings indicate that LLaVA-1.5 did not utilize extracted visual information sufficiently. In conclusion, our findings suggest that while improving image encoders could benefit VLMs, there is room to enhance the models with a fixed image encoder through better strategies for extracting and utilizing visual information.

Decision: reject
Reviewer scores: 3, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730143769289

thread_type:
official_review

<thread_object>
creator: reviewer_7vbu

<content>
summary:
Previous works have argued that the poor performance of VLMs on simple visual reasoning tasks is due to their dependence on CLIP encoder. They show that CLIP can encode two visually different images with high cosine similarity (called erroneous agreement) and argue that many VLMs fail due because they use CLIP as their vision encoder.

In this paper the authors show that with better extraction and utilization methods, clip encoder can still be used for downstream tasks of visual reasoning. They show experiments with LLaVA-1.5 and show that it performs good on benchmarks despite using CLIP as its vision encoder.

strengths:
1. Important analysis shown in section 4 (Investigating the performance gap)- this section answers the questions related to training data, language model and evaluation method. This analysis is important to make the claim that visual information extraction is the key factor in determining the performance gap on downstream tasks. And these other factors (eval method, language encoder, training data) are not contributing much to the improved performance.

2. Detailed benchmarking of the models on different datasets and good ablation studies.

3. They show, using a different decoding method, that even with a fixed pre-trained image encoder if we try to 'force' VLMs to attend to visual features while decoding (and not just relying on language priors), we can perform good on downstream visual reasoning tasks.  Although they used a previously proposed decoding strategy M3ID  (Favero et al., 2024).

weaknesses:
1. The authors show that the visual feature extraction technique in LLaVA (a two layer MLP) is an important step in distinguishing between two erroneous images. But they do not provide an convincing argument on why is it an important step. An analysis on "why just adding a 2-layer MLP on top of pre-trained CLIP makes it so much better?" would have been an amazing addition to the paper. 

2. On Spearman's rank correlation (also asked in the questions): Since CLIP is trained using loss based on cosine similarity, I think using Spearman's rank correlation to show that two embeddings are "fully opposed" is not correct. For example, consider the example given on LN 232-233. Although the ranks of the dims are reversed giving  ρ = −1, their absolute values are pretty close. And if we assume (in an ideal world) them to be separable features, for example the embeddings could be of dog images and the features are 'ear-length' , 'fur color', 'nose-shape', both the embeddings will still show two very similar looking dogs (and not 'fully opposite') even though the embedding might have ρ = −1.

questions:
Would a high negative Spearman's rank correlation show that the embeddings are quite different? 

LN 232-236 says: "While SC (fv(v1), fv(v2)) > 0.989, Spearman’s rank correlation coefficient can tell their sharp difference: ρ = −1, showing that they are fully opposed in this sense. Therefore, the difference in visual inputs might still be extracted through other means when erroneous agreements occur"

How does ρ = −1 show that the embeddings are 'fully opposed'? If the authors could show this or cite a paper that shows this, that would be great.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730321808691

thread_type:
official_review

<thread_object>
creator: reviewer_dmvn

<content>
summary:
This paper examines the performance of LLaVA-1.5-7B on visual reasoning tasks, specifically WhatsUp and MMVP, and concludes that its suboptimal performance is not due to CLIP's visual features. While CLIP visual features effectively capture semantic similarities, they occasionally misinterpret spatial differences in object placement (e.g., "mug on the left" vs. "mug on the right"), which results in high cosine similarity (over 0.95) despite subtle image differences—referred to as "erroneous agreements." The authors show that CLIP’s visual features are accurate; instead, they attribute the performance issues to LLaVA not making effective use of these features. They further demonstrate that poor alignment between visual and textual inputs, not the visual features themselves, explains the bad performance in CLIP models for these tasks and datasets. Unlike CLIP, LLaVA does not exhibit this alignment problem, and this is shown quantitatively. Finally, the authors try better decoding strategies in Llava like M3ID such that the decoding better makes use of the visual features. They also show that  multiple image inputs works better to highlight the difference in images. They also explore performance gaps related to evaluation methods, training data, and the text encoder.

strengths:
- This paper delivers a valuable message to the community by advocating for enhancing Multimodal LLMs and keeping the image encoder fixed. Previous research suggested that the image encoder introduced issues by producing "erroneous agreements" (similar embeddings for semantically similar but visually distinct images). However, this paper counters that claim, attributing the problem instead to the the model not utilizing these visual features effectively. 

- Interesting observation of better decoding algorithms and methods for evaluating specific tasks.

weaknesses:
- There is an incoherent story. The abstract initially suggests that LLaVA performs well on reasoning tasks and achieves high accuracy, yet later the paper claims LLaVA performs poorly on MMVP, contradicting the initial statement. They also mention that LLava is able to extract the correct information from the visual features, and that it does not face issues (L186, and demo image). Only later is it clarified that LLaVA performs well on WhatsUp but not on MMVP. In general, I feel there is an unclear and confusing story. 

- WhatUp, MMVP, COCO-spatial and GQA-spatial are not really well-known datasets and publicly-agreed on to measure reasoning. I actually came to know them after reading this paper. Measuring reasoning on MMLMs are usually not done on these datasets. These datasets are not enough to reflect model reasoning and to come up with general conclusions about LLava or MMLMs in general. The authors don’t show ablation and analysis results using their ablation strategies, on important reasoning tasks such as VQA, GQA, OK-VQA, VCR and others (specifically, those that LLava reports on). I feel the scope, task and datasets are not enough to reach the standard required for ICLR.

questions:
In general the second weakness is the biggest to me. I would like to hear what the authors say on this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535525771

thread_type:
official_review

<thread_object>
creator: reviewer_2b8n

<content>
summary:
The paper challenges the prevailing belief that Vision-Language Models' (VLMs) failures in visual reasoning are primarily due to CLIP image encoder's "erroneous agreements" (where distinct images have high cosine similarity). Using LLaVA-1.5-7B as an example, they demonstrate that MLLMs can successfully extract distinct information from similar image embeddings, achieving high accuracy on tasks where CLIP performs poorly. This suggests that the limitation lies not in the image embeddings themselves, but in how effectively models extract and utilize the encoded information.

strengths:
Provides compelling empirical evidence through controlled experiments across multiple benchmarks.
Challenges and refines an important assumption in the field about VLM limitations.
Demonstrates that existing architectures might be more capable than previously thought, just requiring better utilization strategies.

weaknesses:
The paper's scope might be too focused on LLaVA-1.5 as the primary example, potentially limiting the generalizability of findings
While the paper shows that information can be extracted from similar embeddings, it doesn't fully tackle why LLaVA-1.5 is able to do this.

questions:
How do these findings generalize to other MLLMs beyond LLaVA-1.5?
What specific mechanisms allow MLLMs to extract distinct information from seemingly similar embeddings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730575492656

thread_type:
official_review

<thread_object>
creator: reviewer_fplb

<content>
summary:
This paper provides a comprehensive study to analyze the answers supplied by the VLMs. Specifically, It compares the performances of CLIP and LlaVa-1.5-7B in the What’s Up and MMVP benchmarks. These benchmarks ask questions about a pair of images that contain the same objects and background but in different positions. This paper shows that the LlaVa-1.5-7B can perform better than CLIP in these benchmarks even when LlaVa uses CLIP as a visual encoder, and the average cosine similarity of the CLIP embedding of the image pair is greater than 0.95. Moreover, it provides ablation studies to explain this behavior.

strengths:
This paper provides some interesting insights to show that the metric commonly used to measure the embedding similarity (Cosine Similarity) does not depict all aspects of vector pairs. Therefore, it suggested a complementary metric, Spearman’s rank correlation coefficient. However, table 1 only provides the average Cosine Similarity.

weaknesses:
The paper is challenging to follow, primarily due to the absence of a clear statement of its main contributions in the Introduction. Its content closely parallels the CVPR24 paper, "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs," raising concerns about the originality of this work. The CVPR24 paper highlights that Visual Language Models (VLMs), often relying on CLIP as the visual encoder, struggle with recognizing fine-grained details, such as object locations. It introduces the MMVP benchmark to evaluate these limitations comprehensively. I encourage the authors to clarify how their contributions provide novel insights beyond this existing research.

questions:
I recommend including Spearman's rank correlation coefficient in Table 1 to enhance the analysis. Additionally, a more comprehensive study would be valuable. For example, could the authors provide Spearman's rank correlation coefficient and cosine similarity for the questions with the highest- and lowest-accurate answers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732310017438

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Reviews, Responses, and Changes to the Paper

comment:
We thank the reviewers for their comments and suggestions regarding our submission. We appreciated positive comments on our contributions, e.g., the paper "challenges and refines an important assumption in the field about VLM limitations" from Reviewer `2b8n`, and it "delivers a valuable message to the community ... counters that (previous) claim" from Reviewer `DMVn`. We also found that reviewers gave positive feedbacks on our ablation studies with analysis (Reviewer `7vBu`) and the interesting observation in the discussion (Reviewer `7vBu` and Reviewer `DMVn`). 

We also thank the reviewers for their constructive criticisms. Below, we summarize the major issues and our responses to them:

### **Concerns on the storyline being unclear and confusing from Reviewer `DMVn` and Reviewer `fpLb`.**

Our storyline is as follows: Previous research attributes LLaVA-1.5's low MMVP performance to deficiencies in the CLIP vision encoder (namely erroneous agreements). We show that LLaVA-1.5 (which uses the CLIP vision encoder) excels on image pairs with erroneous agreements on What'sUp benchmark, whereas CLIP model (using both vision encoder and text encoder) has a poor performance. Through ablation, we pinpoint the key differences between LLaVA-1.5 and CLIP that drives this gap and revisit MMVP benchmark to uncover the true cause of LLaVA-1.5's shortcomings.
    
From the feedback of Reviewer `DMVn`, we found that the What'sUp and MMVP benchmark and model performance on them might be unfamiliar to the readers and potentially causes confusion. So we added more description in the introduction to indicate that LLaVA-1.5 perform well on What'sUp but not on more challenging MMVP, guiding the readers to better understand the storyline. 
    
### **Confusion about the function and description of the Spearman's rank correlation from Reviewer `7vBu` and Reviewer `fpLb`.** 

*First, Spearman's rank correlation only serves as a toy example and does not appear in any other contexts in our submission*. It motivates our exploration and comparison using LLaVA-1.5 on extracting information from highly similar embeddings. However, we did not propose to put it into use in experiments and it is not part of our core contribution. 
    
Second, for Reviewer `7vBu`'s question about how high negative Spearman's rank correlation show that the embeddings are "fully opposed," we did not intend to show that the vectors are fully opposed in the sense of cosine similarity. Instead, they contain opposed order information. We clarify this by replacing "they are opposed in this sense" with "their order information is fully opposed" in the new version, which better conveys our intended meaning.

## Summary of changes to the draft

We have made several changes to the draft based on the reviewers' suggestions (all changes are highlighted in blue in the updated PDF). Specifically:

- In response to Reviewer `2b8n`'s concern about the generalizability of our findings beyond LLaVA-1.5, we included evaluations on What'sUp for two other MLLMs (LLaMA-3-V-8B and Phi-3-V-3.8B) in Appendix B.5, demonstrating that our findings in Section 3 generalize to these models as well.
- For concerns on the storyline raised by Reviewer `DMVn`, we refined the Figure 1 and its caption, and added descriptions in introduction (L52-L54, L67) and Section 3 (L193-L195).
- For Reviewer `DMVn`'s concerns about the scope and dataset choice, we added more captions to Figure 2 and included our reasoning in the related work section (L104–L107).
- To clarify the meaning of "visual information extraction" and the toy example as suggested by Reviewer `7vBu`, we rephrased descriptions in the introduction (L74–L76) and Section 3 (L243–L244).
- We combined the original Table 5 and Table 6 to save space and improve readability.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates whether CLIP's image embeddings are truly the bottleneck for vision-language models' (VLMs) performance on visual reasoning tasks. Using LLaVA-1.5 as the primary example, the authors challenge the previous assumption that erroneous agreements in CLIP embeddings (high cosine similarity between visually distinct images) are responsible for VLMs' poor visual reasoning performance.

### Strengths:

1. Novel perspective challenging existing assumptions:
> "delivers a valuable message to the community ... counters that claim, attributing the problem instead to the model not utilizing these visual features effectively" - Reviewer DMVn

2. Systematic experimental methodology: 
> "Demonstrates that existing architectures might be more capable than previously thought, just requiring better utilization strategies" - Reviewer 2b8n

3. Technically sound ablation studies:
> "Important analysis shown in section 4 (Investigating the performance gap)... visual information extraction is the key factor in determining the performance gap" - Reviewer 7vBu

## Weaknesses:

1. Limited scope and generalizability:
> "WhatsUp, MMVP, COCO-spatial and GQA-spatial are not really well-known datasets... These datasets are not enough to reflect model reasoning and to come up with general conclusions" - Reviewer DMVn

2. Methodological concerns:
> "The novel experiments are not comparable with OpenAI-CLIP-ViT-L/14-336 because, as the author outlined, they leverage different visual encoders" - Reviewer fpLb

3. Insufficient evidence for claims:
> "I find the claim that the authors make not well-supported by experimental evidence. The benchmarks the authors used are not enough to support this claim" - Reviewer DMVn

### Justification

Despite interesting insights and thorough revisions, fundamental concerns about methodology, generalizability, and evidence remain. All reviewers rated the paper below the acceptance threshold (three at "marginally below" and one "reject"), indicating the contribution does not meet the conference bar. Key issues include:

1. Limited benchmark selection that doesn't fully support the claims
2. Methodological issues in model comparisons
3. Lack of evidence that findings generalize beyond LLaVA


While the paper raises important questions about VLM limitations and CLIP embeddings, the methodological concerns and limited scope make it unsuitable for acceptance at this time. The authors' revisions, while thorough, did not fully address these core issues as evidenced by maintained negative ratings from all reviewers.