Cross the Gap:  Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

Authors: Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, Andrew D. Bagdanov
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: intra-modal misalignment, CLIP, modality gap, modality inversion, textual inversion, visual inversion, image retrieval, text retrieval, image classification

Abstract:
Pre-trained multi-modal Vision-Language Models like CLIP are widely used off-the-shelf for a variety of applications. In this paper, we show that the common practice of individually exploiting the text or image encoders of these powerful multi-modal models is highly suboptimal for intra-modal tasks like image-to-image retrieval. We argue that this is inherently due to the CLIP-style inter-modal contrastive loss that does not enforce any intra-modal constraints, leading to what we call intra-modal misalignment. To demonstrate this, we leverage two optimization-based modality inversion techniques that map representations from their input modality to the complementary one without any need for auxiliary data or additional trained adapters. We empirically show that, in the intra-modal tasks of image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance with respect to intra-modal baselines on more than fifteen datasets. Additionally, we demonstrate that approaching a native inter-modal task (e.g. zero-shot image classification) intra-modally decreases performance, further validating our findings. Finally, we show that incorporating an intra-modal term in the pre-training objective or narrowing the modality gap between the text and image feature embedding spaces helps reduce the intra-modal misalignment. The code is publicly available at: https://github.com/miccunifi/Cross-the-Gap.

Decision: accept (poster)
Reviewer scores: 3, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730272818414

thread_type:
official_review

<thread_object>
creator: reviewer_bnbn

<content>
summary:
This paper demonstrates that CLIP models, trained exclusively with inter-modal contrastive loss, perform suboptimally on intra-modal tasks like image-to-image and text-to-text retrieval. The authors attribute this limitation to a modality gap and the absence of intra-modal supervision during pre-training. To address this, they employ modality inversion techniques, transforming inputs across modalities to bridge this gap and enhance intra-modal performance. Through extensive experiments, they show that CLIP’s intra-modal performance improves by leveraging the structured nature of inter-modal cosine similarities, which are more consistent than intra-modal ones.

strengths:
Well written, easy to follow.
Extensive experiments and analysis.

weaknesses:
Lack. of alternative baselines, consider evaluating alternative baselines, such as using an image-captioning model to generate captions for query images, followed by image-to-image retrieval based on these captions. This could provide a comparative perspective on the effectiveness of the proposed modality inversion techniques.

Minor: Are the ticks and crosses correct in Table 2. Right. ?

questions:
1. Does combining the native image representation with its inverted (cross-modal) image representation provide any performance benefits?

2. For text-to-text retrieval, have you tried the method on a purely textual task with less than 77 tokens context? Existing text retrieval datasets could be summarized to 77 tokens using an LLM. My assumption is that this approach would only be effective when the text can be represented visually, which might explain its success with text derived from image-captioning datasets.

3. Have you considered alternative baselines for image-to-image retrieval, such as using a captioning model?

4. Do you have any intuitions on why the image-image retrieval score for EuroSat dataset goes down with modality inversion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730587209855

thread_type:
official_review

<thread_object>
creator: reviewer_ikx3

<content>
summary:
The authors point out in this paper that the common practice of using intra-modal similarities from pre-trained CLIP encoders is inherently suboptimal for intra-modal tasks. To address this, they adapt optimization-based modality inversion techniques (OTI and OVI) and leverage them to transform native modality inputs into intermodal representations for similarity measures. Experiments across three settings—image-to-image retrieval, text-to-text retrieval, and zero-shot image classification—on 15 datasets and with 5 different pre-trained models validate the authors’ hypothesis, demonstrating the effectiveness of optimization-based modality inversion.

strengths:
- The idea is good and interesting. Using inter-modal CLIP representations for intra-modal similarity measures makes sense. If this has not been done before, it is a novel idea.

- The paper is well-motivated, with comprehensive experiments, promising results, and clear writing.

- The authors validate their hypothesis and demonstrate their method's effectiveness through extensive experiments involving over 15 datasets, 3 different types of CLIP models, and 5 different pre-trained models.

weaknesses:
- The method is quite simple. OVI is similar to OTI. There is nothing new about the inversion. But applying the inversion in this particular context can be appreciated. The authors might want to tune down the claim on the methodology contribution.

- It is a surprise to see the proposed method also helps with image classification tasks. Why? Image classification is supposed to be an inter-modal task. In the open-vocabulary setting, it is often done by comparing CLIP image features with a set of CLIP text features on the class vocabulary, which is already inter-modal. 

- Missing baseline: For the image-to-image retrieval task, the authors convert the query image to a corresponding textual feature using OTI, then calculate semantic similarity between this feature and the other image features (obtained from the CLIP image encoder) to get the final results. An important baseline is missing, i.e., converting all the images into OTI features and then measuring the similarity among OTI features. This can demonstrate your proposed inter-modal CLIP representations are better for intra-modal tasks.

questions:
- Please answer those issues pointed out in the weakness.
- In Table 1, the use of OTI features outperforms native image features on all datasets except for the EuroSAT dataset. What distinguishes the EuroSAT dataset from the others? Why does it show a performance decrease with OTI features? 	
- The authors use the template sentence “a photo of” concatenated with the pseudo-word tokens in Section 4.1. Have you tried other templates?
- It would be better to provide more training details (e.g., training time, memory usage, and training data).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647234945

thread_type:
official_review

<thread_object>
creator: reviewer_u71q

<content>
summary:
The paper points out the problem of  intra-modal misalignment in CLIP, which limits its effectiveness for intra-modal tasks such as image-to-image retrieval. To address this issue, the paper proposes to transform intra-modal tasks to inter-modal ones via Optimization-based Textual Inversion (OTI) and Optimization-based Visual Inversion (OVI), which map representations to the complementary modality. The experiments demonstrate its effectiveness on fifteen datasets.

strengths:
1. The paper tackles an intriguing problem, since CLIP encoders are widely used in various tasks and serves as components in other models like LVLM. Addressing the intra-modal misalignment issue could enhance CLIP's performance in image and text understanding.
2. The presentation quality is high. The paper is well-written, easy to follow, and features clear and informative figures.

weaknesses:
1. The analysis of the intra-modal misalignment problem could benefit from more quantitative insights. In Figure 1, the authors illustrate that intra-modal similarity scores can be higher for the same class than for different ones. Providing statistics to demonstrate the significance and frequency of this issue would strengthen the argument.
2. The experiments lack comparative analysis. While the related works section mentions existing works that explore intra-modal misalignment, it would be beneficial to compare these methods in the experimental study to validate the paper's contributions.

questions:
It is interesting to know if it is possible to have a more systematic approach for selecting the number of tokens and optimization steps used in OTI and OVI, as discussed in Figure 2.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704341613

thread_type:
official_review

<thread_object>
creator: reviewer_pj7m

<content>
summary:
This paper investigates the suboptimal performance of using pre-trained multi-modal Vision Language Models (VLMs) like CLIP for intra-modal tasks. It highlights the modality gap, a disparity between text and image feature spaces due to contrastive pre-training, which leads to poor performance in tasks like image-to-image retrieval when using individual text or image encoders. The authors propose a modality inversion technique to transform native modality inputs into inter-modal representations, leveraging CLIP's inter-modal alignment. Extensive experiments demonstrate significant performance improvements over intra-modal baselines in image retrieval, text retrieval, and zero-shot image classification tasks.

strengths:
- The paper introduces a novel modality inversion method that significantly outperforms standard intra-modal approaches by exploiting inter-modal alignments of CLIP models, providing a new direction for enhancing VLMs' utility in single-modal tasks.

-  With experiments spanning multiple datasets and tasks, the paper offers robust evidence of the proposed method's efficacy, showcasing its broad applicability in the field of multi-modal learning.

weaknesses:
- The contribution is incremental, as the OTI has already been introduced by existing methods.  
- In zero-shot classification, current methods utilize both image and text encoders concurrently. Therefore, only utilizing the image encoder compromises fairness in the experimental comparisons for this task.  
- Several relevant comparative methods are omitted. In particular, the proposed method achieves similar effects to prompt learning and adapter learning, which should be discussed and included in further comparisons.

questions:
Please see the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732477109626

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response and Revised Manuscript Upload

comment:
We thank all reviewers for their insightful feedback aimed at improving the quality of our work. The reviewers agree that the paper is *well-written* (**U71q**, **ikx3**, **BNBn**) and *well-motivated* (**ikx3**), addresses an *intriguing problem* (**U71q**), and provides a *comprehensive and extensive* (**pj7m**, **ikx3**, **BNBn**) *empirical evaluation* spanning multiple datasets and tasks.

We have uploaded a revised version of the manuscript improving the clarity and including the changes suggested by the reviewers. We highlight in blue all the changes for clearer visualization. Please note that the paper lines we mentioned in the responses to the reviewers refer to the original paper and not the revised one. Below we provide a summary of the main additions to the revised manuscript. 

### **Additional baselines (**pj7m**, **U71q**, **BNBn**)**
We expanded the "*Additional Experiments*" section (**Appendix F**) to include an adapter-based approach for image-to-image and text-to-text retrieval and a captioning-based method for image-to-image retrieval. Employing modality inversion techniques, as proposed in our work, achieves the best performance even after including these new baselines.

### **Quantitative insights on intra-modal misalignment (**pj7m**)**
We have added a new section to the main paper (**Sec. 2** of the revised manuscript) to provide quantitative insights on the impact of the intra-modal misalignment issue. We thank Reviewer **pj7m** for the suggestion, as we believe that this addition provides additional strength to our claims.

### **Additional Experiments (**BNBn**, **ikx3**)**
We included all the additional experiments suggested by the reviewers. Specifically, we added the following paragraphs to the "*Additional Experiments*" section (**Appendix F**):
- *Intra-OTI Similarity Comparisons* (**ikx3**): we applied OTI to all the images and measured the similarity among OTI-inverted features to perform image-to-image retrieval;
- *Impact of the OTI Template Sentence* (**ikx3**): we studied the impact on the performance of OTI template sentence on the image-to-image retrieval task;
- *Text-to-text Retrieval on Purely Textual Datasets* (**BNBn**): we evaluated the performance of OVI on purely textual datasets;
- *Combining Native and Inverted Features* (**BNBn**): we assessed wheter combining native and inverted features improves performance on image-to-image retrieval.

We again thank the reviewers for their thoughtful and insightful feedback. We are happy to address any further questions or provide additional clarifications.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733170884733

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Global Response

comment:
As the discussion period comes to a close, we would like to thank all reviewers again for their thoughtful and constructive feedback which has helped us improve the quality of our manuscript. We are still perplexed by Reviewer pj7m reducing their rating from 5 to 3, without further engaging in the discussion, despite our efforts to address all weaknesses mentioned in their original review. We remain open to any additional feedback that could help us improve our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper points out the problem of intra-modal misalignment in CLIP, which limits its effectiveness for intra-modal tasks such as image-to-image retrieval. To address this issue, the paper proposes to transform intra-modal tasks to inter-modal ones via Optimization-based Textual Inversion (OTI) and Optimization-based Visual Inversion (OVI), which map representations to the complementary modality. The experiments demonstrate its effectiveness on fifteen datasets. Major concerns of reviewers include lack of quantitative analysis of inter-modal misalignment, lack of baseline, and experimental reasonableness. The author solved the above problems during the rebuttal. So the final vote is acceptance.