Mitigate the Gap: Improving Cross-Modal Alignment in CLIP

Authors: Sedigheh Eslami, Gerard de Melo
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: CLIP, modality gap, cross-modal alignment, multi-modal representation learning

Abstract:
Contrastive Language--Image Pre-training (CLIP) has manifested remarkable improvements in zero-shot classification and cross-modal vision-language tasks. Yet, from a geometrical point of view, the CLIP embedding space has been found to have a pronounced modality gap. This gap renders the embedding space overly sparse and disconnected, with different modalities being densely distributed in distinct subregions of the hypersphere. In this work, we propose AlignCLIP, in order to improve the alignment between text and image embeddings, and thereby reduce the modality gap. AlignCLIP increases the cross-modal alignment, and yields gains across several zero-shot and fine-tuning downstream evaluations by sharing the learnable parameters between the modality encoders and a semantically-regularized separation objective function on the uni-modal embeddings. The source code and model checkpoints for reproducing our experiments are available at https://github.com/sarahESL/AlignCLIP.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730191288024

thread_type:
official_review

<thread_object>
creator: reviewer_zbcg

<content>
summary:
The paper proposed a novel method named AlignCLIP to address a recognized problem -- modality gap, for the CLIP network. The improved components include (i) a shared transformer and (ii) an intra-modality separation module. The proposed method surpassed the existing methods in most cases.

strengths:
1. Technical solid paper with clear stated motivation and well organized content.
2. The proposed Intra-Modality Separation module is novel and is benefit to the community.
3. The proposed method surpassed the existing methods in most cases.

weaknesses:
1. The idea of shared transformer is less novel. From my understanding, since the shared transformer for CLIP has been proposed by You et. al., then extending it to the projection layer is just an incremental improvements. The paper should discuss why the extension to the projection layer is necessary comparing to the existing methods. For example, in terms of "extend the sharing to the extent possible" in line #168m, what should we do if using CLIP-ResNet instead of CLIP-ViT.
2. Since the paper claims the (i) SharedCLIP (ii) IMSep module. It is not clear that which module contributes more on the overall performance. Since the SharedCLIP is strongly dependent on transformer structure, while IMSep module is more related to the general multi-modality contrastive learning task, therefore, the IMSep moduel has better expansibility for various model structures. Therefore, an ablation study on this is necessary, i.e. (original_CLIP+)SharedCLIP vs original_CLIP+IMSep, if I missed this experiment, please point it out during the rebuttal.
3. It's necessary to add some discussion for Table.5, regarding the COCO T->I. It seems the SharedCLIP works better in the T->I scenario, but what reason cause the degradation while adding IMSep? Some discussion and analysis regarding why this degradation occurs would be benifit for researchers to understand the limitation of the proposed model, for example, is it due the complexity of COCO dataset itself, or due to the IMSep while performing T->I. How is the performance while applying original_CLIP+IMSep only on T->I?

[You et. al] Learning visual representation from modality-shared contrastive language-image pre-training. In ECCV, pp. 69–87, 2022.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730237668662

thread_type:
official_review

<thread_object>
creator: reviewer_j1qt

<content>
summary:
The paper introduces AlignCLIP, a novel training strategy that aims to improve the cross-modal alignment of CLIP-like models by addressing the modality gap between image and text in the shared embedding space. The authors propose to share the learnable parameters between the vision and language encoders to align the two modalities more closely, and to add an Intra-Modality Separation objective function that pushes semantically dissimilar image embeddings apart. Authors train from scratch CLIP, a SharedCLIP (with shared learnable parameters between the vision and language encoders), and AlignCLIP on CC12M. Extensive experiments demonstrate that AlignCLIP improves zero-shot classification, robustness to distribution shifts, and multi-modal retrieval tasks compared to the original CLIP.

strengths:
- AlignCLIP demonstrates significant improvements to the original CLIP, with measurable gains in downstream tasks such as zero-shot classification and robustness to distribution shifts.
- The IMSep objective adds a novel approach to handling intra-modality representations, ensuring that semantically dissimilar image embeddings are spread apart without affecting semantically similar pairs.
- The paper provides both quantitative and qualitative analyses (I really appreciate Fig. 5), providing clear insights into improvements in retrieval tasks and the reduction of the modality gap.

weaknesses:
- AlignCLIP’s performance on fine-tuned retrieval tasks shows marginal improvements over SharedCLIP, with SharedCLIP sometimes even outperforming AlignCLIP (see Tab. 5). 
- Ablation studies on the key components, while present, could be expanded. In particular, it would be interesting to explore how much the "Pre-trained Sentence Encode" improves the performance and the benefits of using the ImSep loss component. I am afraid that the smaller improvements in such cases could arise just from using an additional pre-trained encoder (i.e. SBERT all-mpnet-base-v2) at training time.

**Additional Consideration**. I noticed several significant errors within the method section Sec. 3.1 and the method figure (Fig. 2) that impact its reliability. In particular, the figure shows:
- *Conv2d Applied to Image Patches*: I believe this is an error stemming from a misunderstanding of the common CLIP practice of using a Conv2d with a stride equal to the image patch size (that is equivalent to applying a standard linear layer to the flattened patches). So depicting a Conv2d after the patches, as the authors proposed, seems incorrect.
- *Max Pooling Applied to Text Embeddings*: Similarly, I suspect that applying max pooling to the text token patches is another mistake. The authors stated: "Following the original CLIP, we use max-pooling for text embeddings." (line 165). However, I believe this results from a misunderstanding of OpenAI's CLIP implementation, which uses an argmax operation to select the EOS token.
These errors impact the clarity and accuracy of the figure and may mislead readers regarding the model architecture.

questions:
- Q1. How would AlignCLIP perform with different backbone architectures besides ViT-B-16? Is the performance consistent with other vision transformers?
- Q2. Could you explain why AlignCLIP’s retrieval performance only marginally improves over SharedCLIP in certain settings? Are there any specific modifications you plan to address this?
- Q3. Conducting an ablation study of CLIP using the ImSep loss would be insightful. If such experiments haven't been performed yet, evaluating whether this approach might outperform or underperform compared to AlignCLIP in zero-shot classification and cross-modal retrieval could offer meaningful comparisons.
- Q4. While mitigating the modality gap is a promising approach to enhancing cross-modal alignment, it might be worth discussing its effectiveness given that the standard definition of the "modality gap" [1]—the difference between the centroids of the two modalities—can be zero even if individual image-text pairs aren't perfectly aligned. Clarifying why this approach remains effective under these conditions could strengthen the understanding of its impact.

[1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (https://arxiv.org/abs/2203.02053)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730281008711

thread_type:
official_review

<thread_object>
creator: reviewer_xflj

<content>
summary:
This paper proposes AlignCLIP, a solution to the text-image alignment issue in CLIP. Specifically, AlignCLIP (1) shares encoder parameters and (2) enhances cross-modal alignment without disrupting embedding distances by introducing IMSep. IMSep particularly adjusts the unimodal embeddings, placing semantically similar image embeddings close but dissimilar image embeddings apart. Experimental results demonstrate substantial improvements in zero-shot classification, robustness to distribution shifts, and multi-modal retrieval, highlighting AlignCLIP's superior alignment and performance over existing CLIP models.

strengths:
- AlignCLIP effectively addresses the text-image alignment issue in CLIP, offering meaningful performance gains over existing methods.  

- I like the idea of IMSep, adjusting unimodal embeddings for enhancing cross-modal alignment. 

- The method has been verified under various tasks-- especially achieving gains both in zero-shot classification and retrieval seem meaningful.

weaknesses:
- I am not very confident in the evaluation settings. Are all competing models trained on the same dataset? If not, how is performance comparability ensured?

- The ideas are interesting; however, this still falls within contrastive learning. Could you strengthen the value of your method by connecting it to existing work? While I believe your approach is new, a clearer link to prior work would enhance its contribution.

questions:
Please see the above weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730390164251

thread_type:
official_review

<thread_object>
creator: reviewer_6cpp

<content>
summary:
This paper introduces AlignCLIP, a variant of the CLIP model aimed at reducing the modality gap between image and text embeddings in cross-modal learning. The authors propose two main modifications: SharedCLIP, which shares the learnable parameters of vision and language encoders, and Intra-Modality Separation (IMSep), an additional objective that regularizes the distances within each modality based on semantic dissimilarity. Through extensive experiments, the authors demonstrate improvements in cross-modal alignment and zero-shot performance, with AlignCLIP consistently outperforming baseline CLIP on a range of benchmarks.

strengths:
- The authors address a meaningful challenge in multimodal learning, i.e., the modality gap, which affects alignment quality in embedding spaces.
- The paper evaluates performance on various benchmarks (e.g., zero-shot classification and retrieval tasks) and robustness against distributional shifts, making the findings broad in scope.
- Visualizations, like the DOSNES projection and alignment score plots, effectively show the improvements in modality alignment and embedding spread across models.

weaknesses:
- Both SharedCLIP and IMSep primarily extend existing contrastive learning techniques. Sharing parameters between encoders is a known approach, and the IMSep loss largely repurposes InfoNCE without substantial modifications. The paper’s novelty, therefore, is limited.
- Empirical comparisons with relevant baselines are lacking, as AlignCLIP is only compared with the original CLIP (and SharedCLIP, also introduced in this paper), omitting several cited "naive" approaches for modality gap reduction.
- For example, the paper repeatedly claims that naive alignment approaches "likely" negatively impact downstream zero-shot performance and backs up this with a theoretical argument, but no qualitative or quantitative results are provided to show that this is indeed the case.
- Table 7 also compares AlignCLIP with two other CLIP variations, but from what I understand, these variations were not designed to mitigate the modality gap.
- In L198-200, the text claims the denominator gets strongly minimized, but the next sentence in L200-202 says it is not strongly minimized. This seems to be a mistake.
- L269 says "In AlignCLIP, we set $\alpha=1$ and $\beta=0.5$", $\beta$ is never introduced in the paper. Moreover, in appendix A.2, it says $\alpha=0.5$ was used, which contradicts this line.

questions:
- The paper states that naive shifts harm downstream performance due to distorted relative distances. Could the authors provide empirical evidence or experiments to support this claim? e.g., Liang et al. (2022)
- The choice of hyperparameters, such as the weight of the IMSep objective in total loss($\alpha$), appears arbitrary. Did the authors conduct sensitivity analyses on these hyperparameters, and if so, could they report their findings?
- While alignment scores improve, does this necessarily translate to enhanced downstream performance? Could the authors discuss specific cases where improved alignment directly correlates with task gains?
- How significant are the alignment improvements in practical, real-world applications? For instance, does better alignment consistently yield qualitatively superior results in retrieval scenarios?
- Do you expect your method to scale easily to larger backbones (e.g., ViT-L or ViT-H) using a dataset like LAION-400M instead of CC12M?
- How is the pre-trained sentence encoder (SBERT) used to calculate semantic distance? Is it robust across various domains and vocabularies in the dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730449365477

thread_type:
official_review

<thread_object>
creator: reviewer_mh82

<content>
summary:
This paper proposes AlignCLIP to reduce the modality gap in CLIP embedding space. The method leverages a shared transformer to align the image and text embeddings. An additional loss function is also introduced to separate semantically distinct unimodal embeddings. Experimental results show that the proposed method can reduce the modality gap and improve the performance on downstream tasks.

strengths:
1. Investigating the multimodal embedding space of vision-language models is an interesting topic.
2. The proposed framework seems easy to implement while still being very effective.

weaknesses:
1. The technical contribution appears limited, as prior work has also leveraged the shared transformer framework [1, 2].

2. The stated goal of this work is to reduce the modality gap within the CLIP embedding space. So basically, the approach should rely on the pre-trained CLIP embedding space. However, the proposed method utilizes a single-encoder framework, which is completely different from CLIP’s two-tower architecture. This raises questions about whether the obtained embedding space is relevant to the CLIP embedding space, which could benefit from further clarification.

3. To my knowledge, the effect of the modality gap on downstream task performance remains an open question. In some cases, a larger modality gap can actually improve performance on certain datasets [3]. This paper could be strengthened by including a more in-depth discussion or insights.




[1] You et al. Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training. \
[2] Girdhar et al. OMNIVORE: A Single Model for Many Visual Modalities. \
[3] Liang et al. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning.

questions:
please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730586214970

thread_type:
official_review

<thread_object>
creator: reviewer_wimx

<content>
summary:
The paper proposes a method to mitigate the modality gap in the representational space of CLIP. The authors introduce two modifications to CLIP (termed AlignCLIP) to achieve this: (1) Sharing the weights of the vision and text transformers in CLIP (termed ‘SharedCLIP’), and (2) using a new Intra-modality separation loss, which encourages separation between images in CLIP space while respecting some of the semantics of the similarities between the images.

The paper’s experiments study the effects of the modifications on a range of downstream tasks, and demonstrate that the AlignCLIP reduces the modality gap and improves zero-shot image classification, linear probe accuracy, robustness to distribution shifts, and multi-modal retrieval accuracy.

strengths:
**Well motivated paper:** Understanding and mitigating the modality gap is an important direction in CLIP and its variants, which are ubiquitously used in many applications

**Clearly written:** Clear flow of ideas, motivations well explained

**Comprehensive ablation studies:** The authors suggested two directions (SharedCLIP + IMSep loss) for attempting to reduce the modality gap, and the effects of both SharedCLIP and SharedCLIP + IMSep loss were tested. Further, the impact of enforcing intra-modality separation on images, texts and their combination was tested individually. Finally, the effect of the rescaling mechanism to control separation of similar images in the batch was also tested individually.

weaknesses:
**Potentially limited applicability:** The approach of AlignCLIP is only applicable where transformer architecture can be used for both modality encoders. This may limit the applicability to other multimodal models with a modality gap using non-transformer architectures: The authors could measure the performance of the proposed IMSep loss without the SharedCLIP architecture, as the IMSep loss could be applied to any non-transformer architecture. (See Question 2)

**Unfair comparison to baselines:** I appreciate the authors’ efforts to have a comparison with popular baselines in Table 7. However, as far as I understand, the numbers for CLIP, CyCLIP, and DFSep in Table 7 were taken from the DFSep paper [a] . However, important hyperparameters that are used to train CLIP, CyCLIP, and DFSep in [a] are different to what the authors used to train SharedCLIP and AlignCLIP in their paper. As a result, these comparisons may not be fair. For instance, authors in [a] trained their models for 64 epochs using batch size of 128, and used 1024 dimensional features. Whereas authors of this paper trained for 30 epochs using 512 batch size, and 768 dimensional features. Further, CLIP models in [a] used ResNet-50 as the image encoder, whereas the authors of this paper used ViT-B-16 backend.

- I would recommend that the authors either re-implement CyCLIP and DFSep using the same hyperparameters as in their experiments, or repeat their experiments using the same hyperparameters as used in CyCLIP and DFSep in [a].
- In either case, these differences should be clearly stated in the paper.

[a] Q. Jiang et al., “Understanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE, Jun. 2023, pp. 7661–7671. doi: 10.1109/CVPR52729.2023.00740.

questions:
1. **Why did the authors use “photo of {caption}” prompt in multi-modal retrieval, instead of just the caption itself?**
Were there any intuitions the authors had about this approach? 

2. **Did the authors check performance using just the IMSep loss, without using the architecture of SharedCLIP?**
This would allow the approach to be used regardless of encoder architecture. 

3. **Did the authors track how the learnable temperature parameter changes during training in AlignCLIP?** Low temperature is known to be an important factor in the emergence of the modality gap.  Therefore, it would be interesting to note the trend and/or the final value of learned temperature in AlignCLIP.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646555872

thread_type:
official_review

<thread_object>
creator: reviewer_3rs9

<content>
summary:
This paper proposes the modality gap problems existing in CLIP and shows the multi-modalities distribusion characteristics. To handle this problem, They proposed a novel method named AlignCLIP. The proposed AlignCLIP mitigates the multi-modal gaps with sharing the parameters between all modalities. To align the text-image and image-image well, they also propose the intra-modality separation objective function. Then, they have provided extensive experiment evidance to prove the effectiveness of  AlignCLIP.

strengths:
This paper shows the multi-modalities distribusion characteristics and proposes two novel idea to mitigate such modal gap. This is a clear and well-writen paper, easy to follow. The experiments for ablation is sufficient. This idea of AlignCLIP is reasonable.

weaknesses:
1. Less evidance to prove the modality gap existing.
2. It seems this paper is not the first to propose modality gap problems.
3. I doubt that Sharing the parameter space between the vision and language encoders may cause each modality like text or image features not learning well.
4. The object function should consider more about the text modality.

questions:
1. The authors have provided DOSNES projection of the CLIP-encoded image–text pairs from CC3M with ViT-B-32, I wonder how it will be with ViT-L-14 and other image-text datasets?
2.  are the existing CLIP-based methods performing the same phenomen like original CLIP ? 
3. The authors have mentioned that there are several works also studying the modality gaps, what is the difference between your findings and these works?
4. Sharing the parameter space between the vision and language encoders may cause each modality like text or image features not learning well, can you provide the parameter comparison between CLIP and AlignCLIP?
5. The proposed object function may cause the learning direction preferring the image based tasks, can you provide more evidance to prove it ?
6. The authors should provide more comarison experiments between existing more recent SOTA methods and AlignCLIP in each task settings.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732564768486

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Response

comment:
We wish to thank all the reviewers for their constructive feedback and invaluable effort in reviewing our paper. We are pleased to see that the reviewers found our paper excellently well-presented [`3Rs9`, `WiMx`, `mh82`, `6cpP`, `XfLj`, `ZbcG`] and encouraged to see that reviewers [`WiMx`, `6cpP`, `XfLj`] found the motivation of studying and mitigating the modality gap important. The reviewers also recognized the technical novelty and effectiveness of our proposed Intra-Modality Separation objective [`3Rs9`, `XfLj`, `J1QT`, `ZbcG`, `mh82`] and found it beneficial for the community. Furthermore, reviewers viewed our experiments sufficient [`3Rs9`], comprehensive [`WiMx`], broad in scope [`6cpP`, `XfLj`] and appreciated the visualizations as well as the qualitative examples in our manuscript [`6cpP`, `J1QT`], providing clear insights into improvements .

We addressed the concerns raised by the reviewers in the revision and individual responses. Major changes are as follows: 

1. **Addition of the ablation study on IMSep loss without sharing parameters**: In response to the concerns about the limitation of AlignCLIP to transformer model and necessity of sharing parameters raised by reviewers \[`WiMx, J1QT, ZbcG`\] and the possibility of using IMSep in two-tower architectures \[`mh82`\], we conducted an ablation study on IMSep loss without sharing parameters using the CC12M dataset and the same hyperparameters reported in the paper. We evaluated the performance of this setting for zero-shot image classification, image classification with linear probing,  image classification with distribution shifts,  zero-shot multi-modal retrieval and  fine-tuned multi-modal retrieval and observed that the **addition of the IMSep loss to the clip loss, without sharing parameters, is indeed effective in reducing the modality gap and enhancing all downstream tasks**. These results suggest that **IMSep loss is also effective in dual-stream two-tower architectures without sharing architecture parameters**.

2. **Enhancement of the SOTA comparison:** Regarding the enhancement of our SOTA comparisons with a more fair experimental setting \[`WiMx`\] and comparing to naive shifting approaches \[`6cpP`, `3Rs9`\], we have fully incorporated reviewers’ suggestions and provided the updated comparisons. Our new comparisons show that the **AlignCLIP model achieves even stronger results** **in comparison to SOTA models**.

3. **Including experiments using additional transformer backends** other than ViT-B-16: Reviewers \[`3Rs9`, `6cpP`, `J1QT`\] have asked whether our method is effective when choosing **other transformer backends**. To study this case, we have **extended** our experiments and repeated them using the **ViT-S-16** backend and provide the results. We observe **the same trend of improvements** when using the ViT-S-16 backend, showing that our method is effective for at least for **two transformer architectures**.
</content>
</thread_object>

</thread>


Metareview:
metareview:
CLIP has been studied for a long time, but how to effectively handle the pronounced modality gap is still a challenging problem in different downstream tasks. This paper researches this problem from the geometrical point of view, and the authors first visualize this phenomenon using the pre-trained ViT. They propose to reduce the modality gap by sharing the parameter space and highlighting the intra-modality separation capability. For the shared parameters, they build two shared encoders, including a Transformer Encoder and a Projector Layer. For the intra-modal discriminative learning, they formulate an InfoNCE-inspired contrastive loss, which considers visual- and text-level feature similarities combined with Visual-Text feature pairs. Experimental results on classification and retrieval tasks to validate their claims on alignment. 

All 7 reviewers pointed out the advantages of the proposed method and they offered a positive rating score, including 6 marginally above the acceptance and 2 acceptance ratings. Generally, the overall novelty of this work is the proposed discriminative criterion based on the cross-modal (diagonal elements of the similarity matrix) information alignment and intra-modal (off-diagonal entries of the similarity matrix) discriminative learning. Hence, the overall idea of this work is good but lower than the excellent bar of 'spotlight', and my final recommendation is accepted with 'poster'.