Mitigate the Gap: Improving Cross-Modal Alignment in CLIP
Authors: Sedigheh Eslami
Gerard de Melo
Venue: ICLR 2025 Poster (2025)
Keywords: CLIP
modality gap
cross-modal alignment
multi-modal representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This work introduces a novel approach for reducing the modality gap in the CLIP embedding space while improving the downstream performance.

Abstract: Contrastive Language--Image Pre-training (CLIP) has manifested remarkable improvements in zero-shot classification and cross-modal vision-language tasks. Yet, from a geometrical point of view, the CLIP embedding space has been found to have a pronounced modality gap. This gap renders the embedding space overly sparse and disconnected, with different modalities being densely distributed in distinct subregions of the hypersphere. In this work, we propose AlignCLIP, in order to improve the alignment between text and image embeddings, and thereby reduce the modality gap. AlignCLIP increases the cross-modal alignment, and yields gains across several zero-shot and fine-tuning downstream evaluations by sharing the learnable parameters between the modality encoders and a semantically-regularized separation objective function on the uni-modal embeddings. The source code and model checkpoints for reproducing our experiments are available at https://github.com/sarahESL/AlignCLIP.

Published as a conference paper at ICLR 2025
MITIGATE THE GAP: IMPROVING CROSS-MODAL
ALIGNMENT IN CLIP
Sedigheh Eslami
Gerard de Melo
Hasso Plattner Institute/ Dastitute/Universitty Uiversity of Potsdam Potsdam, Germany
Hasso Plattner Institute/Universti of Potsdam Potsdam, Germany
sedi gheh. es ami @hpi .
gera rd. deme o@hpi.chhoooooooo
ABSTRACT
Contrastive Language-Image Pre-training (CLIP) has manifested remarkable im- provements in zero-shot classification and cross-modal vision-language tasks. Yet, from a geometrical point of view, the CLIP embedding space has been found have a pronounced modality gap. This gap renders the embedding space overly sparse and disconnected, with different modalities being densely distributed in distinct subregions of the hypersphere. In this work, we propose AlignCLIP, in alignment between text and image embeddings, and thereby reduce the modality gap. AlignCLIP increases the cross-modal alignment, and yields gains across several zero-shot and fine-tuning downstream evaluations by sharing the learnable parameters between the modality encoders and a semantically- regularized separation objective function on the uni-modi embeddings. The source code and model checkpoints for reproducing experiments available pttt:a///githuub..om//aaaaat
order to improve
INTRODUCTION
One the most prominent vision-language models is OpenAl's Contrastive Language-Image Pre-training (CLIP) model (Radford et al. (021). CLIP is dual-stream vision-languse coder trained for learning a joint representation space, in which image and text modalities
be jointly embedded. It has demonstrated exceptional zero-shot capabilities for image classification, multi-modal retrieval as well as robustness to natural distribution shifts, and is
widely used in diverse domains (Eslami et al., 2023; Bleidt al. 2024
Lin et al. 2022)
Despite the outstanding performance of CLIP, recent work has
shed light on a pronounced modality gap the CLIP embed-
ding space (Liang et al.) 2022 Tyshchuk et al. 2023 Schrodi al., (024), leading
large distances between image and text embeddings. We illustrate this phenomenon in Figure [ which shows the DOSNES (Lu et al., 2019) projection of CLIP-encoded image text pairs from CC3M (Sharma al. 2018) using the pre-trained ViT-B-32 backend. As can be seen. each modality densely populates separate small subregion of CLIP's representation embedding space.
Image
Teat
Previous work (Liang al.] 2022] Schrodi et al. 2024 Tyshchuk et al.) 2023) has studied the effects of closing the gap by translating the embeddings of of the modalities to those of the other modality, using the average distance of the image-tex. text pairs, e.g., shifting the image embedding i for each image i by A towards the language embedding t,
in which A N 2 4; + E. V; and N is the total number of image text pairs. However, such naive translations likely harm the relative distances of the unpaired images and texts, and therefore distort the meaningful structure of the embedding space. We show this by simple 2D example in
text
Figure 1: Modality gap in CLIP

---

Published as a conference paper at ICLR 2025
Section A.1 In contrast, work proposes to reduce the gap by first sharing the parameter space between the vision and language encoders and then imposing an intra- modaaity separation objective. The goal of sharing parameters between the encoders is to increase the inductive bias towards shared cross-modal encodings, and thereby, reducing the cross-modal gap. The intra-modality separation more directly aims at encouraging the embeddings within the visual modality be reasonably expanded and pushed towards the language modality. Hence, each modality is pushed towards the embeddings of the opposite modality and thereby, the modality gap can get reduced. Our extensive experimental results support that the two aforementioned refinements show substantial improvement of the tross-modal alignment while improving the performance across wide variety of downstream tasks.
To summarize, our contributions are as follows:
1. In Section A.I, we show that the previous naive approaches for closing the gap leads to damaged relative distances of the unpaired samples, which hurts the performance in downstream tasks.
2. Then, we propose sharing the learnable parameters of the vision and language encoders and study its effects modality gap reduction and furthermore, downstream performance.
3. In addition, we propose AlignCLIP. semantically-reguaritn
intra-modality separation objective function that enforces larger distances for the image embeddings of semantically dissimilar images, and at the same time, smaller distances on those of similar images.
4. We demonstrate that AlignCIIP achieves significant alignment of the modalities, and also substantially improves zero-shot classification, robustness to distribution shifts, and classification with linear probing.
5. We further investigate the impacts of AlignCLIP for the zero-shot and fine-tuned multi- modal retrieval tasks and provide qualitative analysis of the image rankings as well as cosine similarities of the embeddings.
2 BACKGROUND, NOTATIONS AND CONCEPTS
Given a set of N image-text pairs, we consider the CLIP image encoder fy to obtain the 12- normalized vector Ex E R' as d-dimensional embedding vector for image E; i.c., fv(u;) = E. and the CLIP text encoder fr to obtain the 12 no-morale text embedding é E Rdd for the text sample t; i.e., (t, E. We denote batch of encoded image- pairs by E, Raxd for images and E, E 1xd for texts, where refers to the batch size.
Background on CLIP. The contrastive objective in CLIP is the average of the vision to language and the language to vision Info-NCE contrastive loss functions formulated as:
expl(Et . + /r] أز= exp : /
expl -())) -
Lo
log
log
N
N
expl( : 2 /T]
(1)
respectively, where r a learnable temperature parameter. The overall CLIP loss is then:
1 Ceip o [ + L.-o).
(2)
In practice, symmetric cross-entropy loss is employed using the vision and language logits. The label y E IR, which the index of the image-text pair in the batch, represents the correspondence of the paired samples. For batch of image-text pairs, Y E I& denotes the set of labels. The visual and textual logits, Dv E b and De R6x6 are then calculated as:
i. = exp(T)
A =
(3)
respectively. Ultimately, the overall CLIP loss is calculated using the cross-entropy loss, H, as:
Lelp = 2
HH Y) H(),,
(4)
2

---

Published as a conference paper at ICLR 2025
Posttional
Embeeddnng
[805] Taken Posing
Embadding
yallow blonket
Transfomer Eneodar
Nomolustion Layer
Projection Layer
Propend loomable (CS) Taken
Rutlored Patias
Lines Prieeien
Pathh Dropout
[cS]] Takon Polling Pooling
Prsitional fibeeddigg
Figure Overview of sharing the transformer and projection layer in SharedCLIP.
Cross-Modal Alignment Score. In contrastive representation learning, the goal is to learn similar representations for positive pairs and distant representations for irrelevant negative samples. In cross-modal vision-language learning, often the paired image-texts form the positive pair distribution. Alignment entails mapping positive pairs to close embedding vectors such that perfect alignment is achieved when f(²) (xx2 for given encoding function f and a randomly drawn positive pair of samples 2,, (Wang & Isola 2020). In the multi-modaa scenario, we are particularly interested studying the alignment property for the CLIP embedding space, as represents the modality gap (Jiang et al. 2023), and therefore, we define perfect alignment as fT((;) fv(v:).
We adopt the alignment measurement proposed by Goel al. (2022) and define it as the average cosine similarity between the paired image and text embeddings in the CLIP embedding space:
alignment
alignment [-1,
Higher scores demonstrate better alignment, and, therefore, decrease in the modality gap.
ALIGNCLIP
We attain cross-modal alignment in AlignCLIP by two main refinements: sharing the parameter space between the vision and language encoders, as well as learning to spread out and push apart the uni-modal embeddings. The following subsections describe each of these refinements in detail.
3.1 SHARING THE LEARNABLE PARAMETER SPACE IN CLIP
AlignCLIP employs a transformer-base encoder architecture (Vaswani et al. 2017). where the transformer is shared between the two vision and language modality encoders. We suspect that one of the main reasons that the modality gap exists in the original CLIP's embedding space the fact that with dual- encoder architecture, each modality has a separate disentangled set of parameters for optimization, i.e., fr has the parameters 9 and fv has 6', which are initialized and optimized disjointly. As result, even with the same encoder architecture for each modality, the final learned parameters and 0' can diverge quite radically, leading to distinct functions that map each modality to completely different subregions of the embedding space. In other words, training two transformers separately has no explicit inductive bias towards a shared region of the encoding space.
Therefore, seek to align the outputs of the vision and language encoding functions by increasing the overlap of and 8' to the extent possible. We achieve this by sharing the layers of the vision and share the transformer encoder well the projection
language encoding towers. Specifically, layer between the vision and language modalities. Sharing the layers between the modality encoders increases the inductive bias towards shared region of the embedding space, and therefore, reduces

---

Published as a conference paper at ICLR 2025
the modality gap. We utilize standard transformer encoder architecture (Dosovitskiy et al. 2020). An overview of our refined overall model architecture is given in Figure 2 Yellow components are designated for the language modality, green components for the visual modality, and the blue parts shared between the two modalities. For encoding texts, the yellow and blue parts of the model actively contribute in the emhedding calculations. Similarly, when encoding images, the green components as well as the blue ones are invoked for the computation. Following the original CLIP, we use the [EOS] token pooling for text embeddings and the [CLS] token embedding for the image embedding. We refer to this architecture as SharedCLIP throughout the rest of the paper. Sharing transformers in CLIP has been previously investigated from the perspective of downstream performance (You et al.) 2022). In this work, we extend the sharing to the extent possible, i.e., we include the projection layer as well, and study the impacts with respect to the cross-modal alignment property.
3.2 INTRA-MODALITY SEPARATION
As shown in Figure D each modality resides in distinct dense subregion of the CLIP embedding space. We hypothesize that this phenomenon is a direct result of the cross-modal contrastive objective in CLIP, which merely optimizes the relative distances of image embeddings and text embeddings. The cross-modal contrastive loss alone is not sufficient for imposing meaningful distances within the uni-modal embeddings, i..., pairwise distances of text embeddings or pairwise distances of image embeddings. Therefore, we define an objective function that enforces reasonable distances within each modality by separating the uni-modal embeddings that are semantically dissimilar. In other words, we impose semantically--reguarizzd Intra-Modality Separation (IMSep) in addition to the CLIP's objective function.
IMSep is achieved by an InfoNCE contrastive loss, in which image text pairs are the positive samples and all pairwise image-image combinations are considered as negative samples, respectively:
N log
expl(Et / /T] exp x /
C =
(6)
N
The distinguishing role of the IMSep loss, in comparison to the CLIP loss, is the enforcement of minimizing the pairwise image image-image cosine similarities, i.e., the denominator in Eq. 6 As result, this objective spreads apart different image embeddings. However, one should notice that while pushing the image embeddings away from each other, some samples might indeed bear some semantic similarity, and therefore ought not to be substantially separated. To account for this, we regularize the intra-modee modaltyyylyty separation with respect to the pairwise semantic distances of samples. Denoting the semantic distance of the image sample from j by D.j. the regularized loss becomes:
exp - E [| eDp E Et)) |T]
C
log
In Eq. 7 the distance Dij controls the minimization of (e) a by reducing these in-modality dot product values the samples are semantically close (similar), and therefore, the denominator does not get strongly minimized, since the values already small. Conversely, the denominator obtains larger values when Diy is large, when images are semantically dissimilar, and therefore, the denominator gets strongly minimized.
In practice, IMSep aims at optimizing Eq. Jusing the cross-entropy loss, Given a batch of encoded image-text pairs, IMSep creates the ÿvsep logits and then minimizes the cross-entropy loss over Y and @veep- To this end, first the pairwise cosine similarities of the images are calculated by:
E, where V E bx6
(8)
For the semantic regularization, and therefore, calculating Diy, we utilize the pairing texts of the images as the semantic supervision signal, and invoke a pre-trained sentence encoder for encoding each text as EØ RR We denote the corresponding batch of semantically encoded texts Eg E 6xx and proceed to calculate the distance matrix D E R from the similarity matrix S E R
S EgE E, ?
D 1-
4

---

Published as a conference paper at ICLR 2025
Inlbrae
R----8
M
PAAA
D
9 veep
1.3.
Figure 3: Schematic summary of the Intra-Modality Separation approach in AlignCLIP.
We rely on in order to re-scale V by:
VD V D, where V/D E ggxb
(10)
and @ denotes the glement-wise product. The re-scaling mechanism encourages reducing the dot product values in for semantically similar images, while increasing for dissimilar images.
Afterwards, for calculating the logits of positive samples, calculate M = E,E, and mask the M. We then obtain Diveop logits by:
non-diagonal values by diag(M)
Dveep exp(r) [diag(MM) + Vpl, Dveep Rbxb In Figure 3 approach for obtaining IMSep schematically summarized. Ultimately, define the Intra-Modalin Separation loss as:
(11)
Limse (iveep: Y),
(12)
and adopt the core of the CLIP loss to represent the cross- modal separation:
LeReep H Y) (
(13)
The final loss function optimized in AlignCLIP is:
C LCRsep + aL
(14)
Note that is sufficient define the Intra-Modality Separation function for only one of the modalities, image in our case, since the cross-modal objective defined by LeReep already behaves supervision for the other modality, text our case, and enforces Intra-Modality Separation the opposite modality as well. To better support this statement, we extensively experiment on the effects of adding a similar Intra- Modalttyy Separation on text embeddings in Section A.3 and show that it sufficient to impose the Intra-Modality Separation the image embeddings.
4 EXPERIMENTS
4.1 TRAINING DATASET AND SETUP
We used the Conceptual Caption 12M (CC12M) dataset (Changpinyo et al. 2021 for pre-training the models. In order to ensure a fair comparison of CLIP, we configured our setup for SharedCLIP and AlignCLIP to similar to that the original CLIP with the ViT-B- 16 backend. The pre-trained semantic encoder utilized in AlignCLIP for re-scaling image-image cosine similarities is the SBERT all-mpnet-basee baaneet-baeev22 model. Moreover, in terms of ablations, we train a version AlignCLIP without sharing parameters, i.e., IMSep, and compare its performance. We trained all models from scratch using the OpenCLIP implementation (Cherti et al. 2023 Ilharco et al. 202T). AlignCLIP, we set 0.5. Section A.2 provides more details about the training setup.

---

Published as a conference paper at ICLR 2025
MODEL
CC3M
MSCOCO
IMAGENET-IK
CIFAR-100
CIFAR-10
CLIP
0.42
0.47
0.41
0.38
0.40
SHAREDCLIP
0.59
0.62
0.57
0,54
0.54
IMSEP
0.61
0.64
0.59
0.58
0.6
ALIGNCLIP
0.64
0.67
0.63
0.62
0.64
Table 1: Comparison of the alignment score.
Positive Samples
25000
Samples
Auq-0.62 4.g-0.67
CUP
SharddtL AlinnCPP
10000
5000
0.0
0.2
0.4
0.6
0.8
Cosine Similariiie
Figure Cumulative distribution pairwise cosine similarities of positive samples in MSCOCO.
4.2 CROSS-MODAL ALIGNMENT
We start by reporting and comparing the alignment scores when using CLIP, SharedCLIP, and AlignCLIP models on the validation sets from CC3M, MSCOCO as well as the ImageNet-1K, CIFAR-100, and CIFAR-10 test datasets. Table[[smmmariees the corresponding alignment scores. We observe that the original CL.IP model has relatively low alignment scores, varying within [0.38, 0.47), across all five datasets. In contrast, sharing the parameter space in SharedCLIPP results in noticeable improvements of up to 0.17 in the alignment scores. We also observe that IMSep noticeably improves the Blignment.Furthennn 0.62 to 0.67. In other words, AlignCLIP reduces the average angle between the paired image-text embedding from about 70" to 47", which can be interpreted as reducing the modality gap.
using AlignCLIP yields even better alignment scores, ranging from
We also plot the cumulative distribution of cosine similarities of the positive samples from MSCOCO when encoded using the original CLIP, SharedCLIP, and AlignCLIP in Figure g We find that using SharedCIIIP noticeably shifts the distribution of the cosine similarity of positive samples towards higher similarity values. A higher similarity of positive samples, i.e., image-text pairs, means achieving greater cross-modal alignment, and thus lower modality gap. Furthermore, AlignCLIP shifts the distribution even more to the right, resulting in better reduction of the modality gap.
For more comprehensive comparison of the distribution of each modality and studying the modality gap, we visualize the DOSNES projection of the encoded image-texts from the MSCOCO dataset in Figure [5] As can be seen, the uni-modal embeddings of the original CLIP are densely located on opposite sides of the hypersphere. In contrast, the embeddings get spread out, hence reducing the modality gap, when using the SharedCLIP model. This observation confirms our hypothesis that sharing layers in SharedCLIP adds greater inductive bias towards a shared embedding space, in comparison training separate transformers CLIP, and therefore, reduces the gap. Finally, the embedding space of AlignCLIP achieves the best spread of the uni-modal embeddings and substantially decreases the gap between image and text embeddings, showing the effectiveness of pushing the image embeddings apart from each other by the IMSep loss. In summary, Table [[] Figures I and]3] confirm that AlignCLIP achieves an embedding space with the least sparsity, least modality gap, and the best cross-modal alignment.

---

Published as a conference paper at ICLR 2025
Image
(a) Original CLIP
(b) SharedCLIP
(c) AlignCLIP
Figure 5: DOSNES visualization the multi-modal embeddings using CC3M
MODEL
IMAGENET-IK
CIFAR-100
CIFAR-10
FLOWERS-103
STANFORD CARS
ToP1
Top5
Top1
Top5
ToPl
ToP5
Topl
ToP5
ToP1
ToP5
CLIP
31.4
58.7
28.1
55.9
61.6
95.6
19.9
40.6
11.7
36.4
SHAREDCLIP
32.1
59.7
26.4
54.7
56.9
95.2
18.2
38.8
10.7
35.4
IMSEP
32
59
31
60.1
61.6
96.2
20.7
40.5
9.3
32.7
ALIGNCLIP
32.8
60.6
36.5
66.3
97.8
18.8
40.3
38.1
Table 2: Accuracy scores for zero-shot image classification.
4.3 CLASSIFICATION
CLIP's pre-training objective resulted in outstanding image classification performance when tested in either zero-shot linear probing settings (Radford al. 202T. Therefore, we further assess the performance of SharedCLIP and AlignCLIP in classification tasks.
Zero-Shot Image Classification. We conduct zero-shot classification experiments on ImageNet- IK (Russakovsky et al. 2015), CIFAR-100, CIFAR-10 (Krizhevsky et al., 2009). Flowers- 102 Nilssack Zisserman 2008), and Stanford Cars Kraasse et al. 2013). with the combination of text prompts used by CLIP (Radford et al.) 2021), c.g., "a photo of the [label)". The experimental results summarized in Table 2 show that SharedCL.IP improves the classification performance on the ImageNet-1K K dataset and reduces the accuracy on the rest of the datasets. In contrast, AlignCLIP yields the best scores across all five datasets. It achieves 1.4% and 2% improvement of Top-1 and Top-5 accuracy, respectively, on ImageNet-IK when compared to the original CLIP. On CIFAR- 10, AlignCLIP achieves up to 8% and 2% improvements in Top-1 and Top-5 accuracy scores. Similarly, using AlignCLIP for CIFAR-100 results in about 8% and 11% enhancement in Top-1 and Top-5 accuracy scores. The trend of improvement is also observed on the Flowers- 102 and Stanford Cars datasets. Our experiments thus evince that via sharing parameters and the additional IMSep separation, AlignCLIP enhances the downstream zero-shot image classification task.
Linear Probing. We further test the performance of SharedCLIP and AlignCLIP when performing linear probing for image classification and report the Top-1 accuracy results in Table 3 For all datasets, we train the linear classifier layer with a batch size of 128, for 30 epochs, with AdamW optimization, and a cosine scheduler with starting learning rate of 5 x 10 Table shows that SharedCLIP generally outperforms CLIP in this setting and furthermore, AlignCLIP achieves the best scores, improving the image classification task with linear probing by up to 5% in comparison to CLIP. Overall, our experiments show that AlignCIIIP and IMSep achieve the best classification performance in both zero-shot and linear probing settings.
4.4 ROBUSTNESS TO NATURAL DISTRIBUTION SHIFT
In the zero-shot image classification task, CLIP has additionally shown impressive robustness to natural distribution shifts and promising generalizability out-of-distribution images. Therefore,
7

---

Published as a conference paper at ICLR 2025
MODEL
IMAGENET-IK
CIFAR-100
CIFAR-10
FLOWERS-102
STANFORD CARS
CLIP
50.0
62.6
85,0
71.5
42.9
SHAREDCLIP
$1.2
85.0
IMSEP
52.2
65.9
86.6
79.5
46.3
ALIGNCLIP
51.5
67.4
87.2
76.8
45.6
Table 3: Accuracy scores for image classification with linear probing.
MODEL
IMAGENETV2
IMAGENET-R
IMAGENET-A
INAGENETSKETCH
Top1
Top5
Top1
Tor5
Top1
Toe5
Topl
Top5
CLIP
27.1
53.3
39.8
65.8
6.5
25.4
19.4
41.8
SHAREDCLIP
27.4
53.5
40.1
67.3
6.7
25.5
20.6
43.2
IMSEP
27.5
53.3
40.9
67.5
6.7
24.6
20.2
45.1
ALIGNCLIP
29.1
54.4
41.2
67.3
7.0
24.6
20.7
43.2
Table 4: Accuracy scores for zero-shot classification and natural distribution shift.
expand our evaluations and investigate to what extent SharedCIII and AlignCLIP are affected by natural distribution shifts. We use the ImageNetV2, ImageNet-R, ImageNet-A, and ImageNetSketch datasets for these evaluations and report the corresponding results in Table 4 in terms Top-I and Top-5 accuracy. An initial observation is that SharedCLIP generally improves the classification accuracy in comparison to the CLIP model. Additionally, when comparing to the CLIP model, AlignCLIP achieves about 2% improvement in Top-1 and Top-5 accuracy scores, leading to the conclusion that sharing parameters and applying IMSep in AlignCLIP improve the robustness to natural distribution shifts in comparison to the original CL.IP model.
4.5 MULTI-MODAL RETRIEVAL
In addition classification, we evaluate SharedCLIP and AlignCLIP in the applications zero-shot and fine-tuned image-to-text and text-to-image retrieval using the MSCOCO (Lin et al., 2014) and Flickr30K (Plummer et al. 2015) datasets. In all settings, the text prompt photo of the [caption]] )" is used. When fine-tuning, the batch size was set to 128 and the AdamW optimizer with learning rate 5 x 10 and a weight decay of 0.2 was used. For the fine-tuning experiments, each model was fine-tuned for 8 and 20 epochs on MSCOCO and Flickr, respectively.
The results
zero-shot and fine-tuned evaluations are reported in Table Our experiments show that both SharedCLIP and AlignCLIP improve the retrieval results measured by ( 5, 10) on both datasets when compared to the original CLIP model. In addition, AlignCLIP achieves the best overall results in comparison SharedCLII. When testing text-to-image retrieval in the zero-shot setting on the MSCOCO dataset, SharedCLII outperforms AlignCLIP at R 10 by less than 1%. After fine-tuning, the recall difference reduced to 0.5%. Similarly, for the image-to-texl retrieval on the Flickr dataset, SharedCLIP outperforms AlignCLIP by about 1% at R (1, 5) and fine-tuning makes the performances marginally different.
For a more comprehensive understanding of the differences in results in the retrieval task, Figure 6A provides an example text query for which SharedCLIP achieves higher R@5 image retrieval in comparison to ALignCLIP. We provide the images at rank 1-3 and observe that all models retrieve images that are semantically relevant for the query text, but might not necessarily In this example, such characteristics of the dataset can particularly affect the evaluation results of AlignCLIP, due to the semantic regularization IMSep, as it will make the semantically similar images closer to each other and can retrieve semantically close samples that are not labeled as the ground truth. We suspect that this phenomenon is more noticeable in the retrieval scenario in comparison to image classification datasets, as in the classification tasks, classes are more semantically disjoint. In Figure[8] we further provide the ground truth texts for each image. Moreover, Figure provides more of such, and Figure 10 illustrates text rankings given sample image queries.
the ground truth.
8

---

Published as a conference paper at ICLR 2025
MSCOCO
FLICKR30K
MODEL
1 T
T T-1
T
R R@0
R
R
R@]
R@5
R
R
R
R 10
R@1
R@5
R 10
CL.IP
31.4
57.0
68.6
20.5
44.1
55.9
53.2
80.5
88.6
39.9
69.0
78.5
SHIAREDCLIP
33.6
59.6
70.8
21.8
45.4
57.3
58.3
83.6
89.8
42.6
70.0
79.1
IMSEP
33.7
60.8
71,5
21.5
45.1
56.9
56.8
82.8
89.2
42
69.9
79.4
ALIGNCLIP
34.0
59.7
70.9
21.9
45.0
56.8
57.2
82.3
89.5
41.8
70.2
79.1
CLIP
39.6
67.5
78.3
26.7
53.4
65.7
64.8
87.8
93.9
47.4
75.9
84.1
E
SHAREDDLIP
40.7
69.2
79.7
27.9
55.0
66.7
66.5
89.1
94.1
48.9
76.4
84.3
IMSEP
41
69.8
79.7
27.8
54.5
66.8
64.3
87.5
93.6
48.5
76.5
84.8
ALIGNCLIP
41.7
69.3
80.1
27.9
54.5
66.2
66
89.1
94.5
49.3
76.7
84.4
Table 5: Zero-shot (ZS) and fine-tuned (FT) cross-modal retrieval summarized with @@ (1, 5, 10).
Blue color represents the best performance for ZS. Numbers in bold represent the best scores.
CLP
SharedCLIP
AlignCLIP
The sun shining down on
CUP
SharedCUP
AlignCLIP
7:
1;
T.
a surfer
641
on the
839
0.2
861
0.4
0.8
071
beach
4:
15
41
4
h
Rank
1:
000
1
033
867
071
039
1
food
noodlees and beof
A
B
Figure 6: Qualitative analysis of the image rankings and cosine similarities on MSCOCO.
In contrary, in O8 we provide an example where AlignCLII retrieves the most accurate caption or image. We use CLIP, SharedCLIP, and AlignCLIP to encode the images and texts, and report the the right side. As can be seen, when querying the first image (on the left), the
cosine similarities similarity of the first image and the second text (on the right) using the CLIP embeddings is higher in comparison to the ground truth caption, suggesting that the second text will, incorrectly, get selected as the predicted caption when using CLIP. This shortcoming still appears when using SharedCLIIP for encoding the images and texts. However, when using AlignCLIIP, the cosine similarity of the first image and the first text is higher in comparison to the second text, meaning that when querying the first image, the ground truth caption, which is semantically more correct in comparison to the second text, successfully gets selected. This suggests that the semantic regularization of the Intra-Modality Separation AlignCLIP, which is calculated using the semantics of the text samples, potentially contributes to capturing more details in the captions, and, hence, improving the retrieval performance.
4.6 ABLATION STUDY
This section provides an ablation study on the effectiveness of the re-scaling mechanism proposed in Eq. TO To this end, we compare the performance of AlignCLII with and without the re-scaling mechanism on ImageNet-1K: CIFAR-100, CIFAR-10 for zero-shot classification, ImageNetV2 for robustness to distribution shift, and the MSCOCO and Flickr30K datasets for the retrieval task. For the retrieval tasks, R@ 10 is reported. From the summarized results in Table 6 we can substantially conclude that the re-scaling mechanism is effective in controlling the separation of similar image samples in the batch, and therefore, increasing the performance of the downstream tasks.
4.7 COMPARISON WITH STATE-OF-THE-ARE MODELS
Table 7 provides comparison of SharedCLIP and AlignCLIP with the state-of-the -he CLIP-based models. One of the prominent previous studies that also reports the alignment score is the CyCLIP model (Goel al. 2022), which trained using the CC3M dataset. For fair comparison with CLIP

---

Published as a conference paper at ICLR 2025
MODEL
IMAGENET-IK
CIFAR-100
CIFAR-10
IMAGENETV2
MSCOCO
FLICKR
Top1
ToP1
TOP1
TOP1
1-T
T-
1
T
+
ALIGNCLIP-WO RESCALING
32.8
34.7
64.2
27.9
71.4
56.4
89.5
78
ALIGNCTIP
32.8
36.5
69.4
29.1
70.9
56.8
89.5
79.1
Table 6: Ablation study on the re-scaling mechanism in AlignCLIP.
MODEL
IMAGENET-IK
CIFAR-100
CIFAR-10
IMAGENET-A
ALIGNMENT
Top1
Top5
Top1
Top5
Top1
Top5
ToP1
ToP5
IMAGENET-IK
CLIP (RADFORD RT Al..
2021
16.5
34.1
23.7
49.7
51.3
92.7
4
15.8
0.36
NAIVE SHIFTING (LANN
2022
16.6
34.6
24.5
52.5
51.2
91.2
43.3
68.8
0.16 (.
CyCLIP (GOEL ET Al..
2022
15
32
21.8
48
44.7
85.4
3.3
13.6
0.32
SHAREDCLIP
17.4
34.9
52.1
3.7
14.5
0.45
ALIGNCLIP
17.6
34.9
26.3
54
57.9
93.7
4.2
16.1
0.63
Table 7: Comparison with the state-of-the-art CLIP-based models.
and CyCLIP, we re-train these models with the same hyperparameters as SharedCLIP and AlignCLIP. Furthermore, we train versions of SharedCLIP and AlignCLIP with the CC3M dataset and compare the results in T We also provide a comparison with the case of naive shifting proposed in Liang et al. (2022). Our results show a promising improvement of alignment scores while improving the downstream performance in comparison to the previous models.
5 RELATED WORK
Modifications of CLIP. Prior work has sought improve several different aspects of CLIP, e.g..
introducing mulli-positinn contrastive loss in Lee et al. (2022). an additional image-based contrastive loss in Mu al. (2022), optimizing the geometrie consistency of the image and text embeddings in Goel et al. (2022), proposing
conerrattttive training regimen based on image-tex textge-eetteee pairs in Zhou al. 2023). employing pairwise sigmoid loss for scaling up the batch size in Zhai et al. (2023), ) and domain adaptation by refining the fine-tuning in Hu et al. (2024). In contrast, AlignCI.IP studies the cross-modal alignment in CL.IP by investigating effective refinements of CLIP.
Modality Gap in CLIP. The modality gap in CLIP was first studied by Liang et al. (2022). They showed that the modality gap is caused by a combination of the model initialization and the contrastive loss optimization, and attempted to close the gap by shifting the embeddings. In similar study, Tyshchuk et al. (202)) measured the alignment of image and text embeddings CLIP with a focus on isotropic properties. More recently. Schrodi et al. (2024) showed that one of the key factors contributing to both the modality gap and the bias in CLIP is the information imbalance between the two Condalities.uurhermmee
Jiang al. (2023) showed that exact modality alignment is suboptimal for downstream tasks, as the prediction error has a lower bound that is dependent on the modality information gap. Our AlignCLIP work shows that naive shifting of the embeddings distorts the relative distances of the unpaired samples, which can harm the downstream performance. We show that under the same information gap, i.e., using the same training datasets, parameter sharing between the modality encoders as well as senrtialllyregularizeede separation of the uni-moo modal embeddings reduce the gap while improving the downstream tasks.
6 CONCLUSION
This work presents AlignCLIP, novel framework for enhancing the cross-modal alignment in the CLIP embedding space. AlignCLIP reduces the gap by increasing the inductive bias towards shared embeddings as well semantically-regularizzz modifications of the latent space. We show that AlignCLIP substantially improves the cross-modal alignment and further achieves the best results for image classification, robustness distribution shifts and improves the multi-modal retrieval performance in comparison to CLIP. Our qualitative analysis on the retrieval task demonstrates the changes of the ranking of the neighborhoods for semantically similar images.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We acknowledge the financial support from the German Federal Ministry for Education and Research (BMBF) within the project KI-Servicezentrrum Berlin Brandenburg (011S22092). We also thank Ali Ghaffaari, Maryam Hosseini, and Mina Rezaei for insightful discussions.
REPRODUCIBILITY STATEMENT
We have provided the details of our training and evaluations setups in Section A.2 Clear explanations of the shortcoming of the previous naive attempts at reducing the gap has been provided in Section [A. Furthermore, the source code for training our models and model checkpoints for reproducing our are available at htttps://github... wttph//...oooaaaaatttt/eees/se/e
REFERENCES
Tibor Bleidt, Sedigheh Eslami, and Gerard de Melo. ArtQuest: Countering hidden language biases in ArtVQA. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2024), pp. 7326-7335, January 2024.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3558-3568, June
2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2822 June 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Sedigheh Eslami, Christoph Meinel, and Gerard de Melo. PubMedCLIP: How much does CLIP bene- fit visual question answering in the medical domain? In Andreas Vlachos and Isabelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EACI. 2023, pp. 1181-1193, Dubrovnik, Croatia, May 2023. Association for Computational I Linguistics. doi: 10. 1861886/1/2222.. findings-eacl.88. URL https: Sitttss//aclanthat //aaa/aaanhloooyy./. org/ 2023. findings eacl. 88
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp- 6704-6719. Curran Associates, Inc., neurips www.ris..c/pppp.......eeeppeppee fillss/pape//2/2///
2022. URL httpp:://proeeeee 12040
Paperraper--- Conference pdf
Xuefeng Hu, Ke Zhang, Lu Xia, Albert Chen, Jiajia Luo, Yuyin Sun, Ken Wang, Nan Qiao, Xiao Zeng, Min Sun, Cheng-Hao Kuo, and Ram Nevatia. Reclip: Refine contrastive language image pre- training with source free domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2994-3003, January 2024.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL (tRtps/ts:: //do://oo...m www.ooorgooe. 528811111/81 If you use this software, please cite it as below.
zenodo. 5143773
Qian Jiang. Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Dinh Tran, Xu, Belinda Zeng. and Trishul Chilimbi. Understanding and constructing latent modality structures in multi-modal representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7661-7671, June 2023.

---

Published as a conference paper at ICLR 2025
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for iin-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, June 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uniclip: Unified framework for contrastive language-image pre- training- In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 1008-1019. Curran Asso- ciates, Inc., 2022. URL
wwtpspprccpppppppppppepeeepeee.e files/paper) 0027:144222233334455abaaa2227122
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learn- ing. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 17612-17625 Curran Asso- ciates, Inc.,
URL wttp:tteee///ppooeee...........s
Tsung-YY Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ziyi Lin, Shijie Geng, Renrui Zhang, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, Peng Gao, and Hongsheng Li. Frozen CLIP models efficient video learners. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Pro- ceedings of ECCV 2022, volume 13695 of Lecture Notes in Computer Science, pp. 388 404. Springer, 2022. doi: 10.1000/978-3-0330 3-18-----3--111 19833-5
URL /ttp:tts:///ol.oorgg/ 1007/
377-3-031-19633-33
Yao Lu, Jukka Corander, and Zhirong Yang. Doubly stochastic neighbor embedding on spheres. Pattern Recognition Letters, 128: 100-100 2019. ISSN 0167-8655, doi: http:///6i.org//0..00/// 1pptrec.2019.08.000 026. URL htttt:/t////.. htwp::///wwww..// wwtp::/wwweeeeeeireeec..... http://sr..ccooccccceeeeetttttat
p11/80167665118305000
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In European conference on computer vision, pp. 529-544. Springer, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision Graphics & Image Processing, pp. 722-729, 2008. doi: 10. 101100/VGIIP.220.....
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-plee to-raae correspondences for richer image-to-sentence models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. $748-8763, PMLR, 18-24 Jul 2021. URL mlr. www.sprsseeeeeee.e / vv2v2111331 39/
ht https://proceedia
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 2015. doi: 10.1007/s 1263-015-081642
12

---

Published as a conference paper at ICLR 2025
Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive rision-languee representation learning. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL https: http///ppe/////eeeeeee/ee i ew net/ forum? (0=
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.1166...0 06653v1//188 1238. URL ht..t:ttp.............. //acl antho logy
Drg/P18-1238
Kirill Tyshchuk, Polina Karpikova, Andrew Spiridonov, Anastasiaa Prutianova, Anton Razzhigaev, and Alexander Panchenko. On isotropy of multimodal embeddings. Information, 14(7), 2023. ISSN 2078-2489. doi: 10.3390/in0014000399. URL httttstss:: /wwww//wwwww.//////.////.... wwwp..dppi...m ctt.:pcccom///oo wttp:/cccm/m7772///11 7/392
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL (bdocsdccdddeeeoococcocoocccaaa Papeeaaperppppe File/3ffee24354 /eee9/eeeleeee
files//ppe::/2117
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pp. 9929-9939. PMLR, 13-18 Jul 2020. URL
bttps://proceecings www.smlr..prrrs /vl1 http:///v1//wwaggg200 2k html
Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Learning visual representation from modality-shared contrastive language-image pre-training. In European Conference on Computer Vision, pp. 69-87, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 11975-11986, October 2023.
Jinghao Zhou, Li Dong, Zhe Gan, Lijuan Wang, and Furu Wei. Non-contrastive learning meets language-iman pre-training. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11028-1 11038, June 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX / SUPPLEMENTAL MATERIAL
A.1 DESTRUCTIVE EFFECTS OF NAIVE EMBEDDINGS TRANSLATION FOR CLOSING THE GAP
In this section, provide a 2D example for showing how the translation proposed in Liang et al. (002) (and later adopted in Schrodi et al. (2024) is not guaranteed to maintain the relative distances of the unpaired image-tea text samples, and therefore, can distort the meaningful structure of the embedding space. Figure ( provides 2 image-text pairs, i.e., (Ti, 11) and (72, 12). As derived in Eq. [5] the distance of the first image is closest the first text, and the distance of the second image is closest to the second text. However, in Eq. I6] after shifting by A, as proposed by Liing et al. (2022), the distances of the second text to either of the images become equal and fail maintain the relative distances after the shift. As result, in downstream tasks, e.g., image retrieval given T2, both J and /2 will be retrieved, which incorrect.
d1.1 1
11ll = |1- 11- V v5
d1.2 = 17 - 12ll = 2.5, = v6.25 V6.5
d2,1 [T2 -4 -11-1.5,
v2.25 V4.5
(15)
d2.2 = [22 - /2|l |[]2,0]] v = V4
d,, < d1,2, d2. d2.1
A = (4 1.5] - [2,5 2.5 [-4, 1.5]
1 = - 1/ = 12 + A 2 -1
d, 1|7 /[ll = || 0.5]| /4 V = V4.25
422 = ||7 - -/ll = [| 2|| = 22.5 +4 V6.25
(16)
d2 IT2 -1 ||2.5, 0|| V625
22 = T2 121 = I| 1.5]|| = /4 2.25 = V6.25
4, < <1,2 22,1
A.2 TRAINING DATASET AND SETUP
For pre-training, we used the CC12M dataset (Changpinyo et al. 2021). We adopted a transformer encoder consisting of 12 layers and 12 heads in CLIP, SharedCLIP, and AlignCLIP. The same input pre-processing and augmentations employed in CLIP have been used for SharedCLIP as well as AlignCLIP, including random cropping of images to the size 224 224. The image patch size for encoding visual data is set to 16 x 16. When encoding texts, the maximum sequence length is set to 77 tokens and the vocabulary size for the embedding layer is set to 49,408. The output embedding dimensionality for both the vision and language modalities is set to 768. We chose our setup to be quite similar that of the original CLIP with the ViT-B-16 backend, in order ensure fair comparison of CLIP and our proposed model. The SBERT all-mpnet-base-e model has been utilized in AlignCLIP for re-scaling mmage-image cosine similarities in Eq. [0]
In order to fairly compare the effectiveness of each model, we trained all of the models from scratch using the CC12M dataset and the OpenCLII implementation (Cherti et al. 2023 Ilharco al. 2021). For all models, we used Adam W optimization with a starting learning rate of 1 x 10-4 cosine scheduler, 10,000 warmup steps, and a weight decay of 0.1. The initial temperature value for all models were set to 0.07. Each model was trained using an NVIDIA H100 GPU with batch size 512 for epochs. We used the checkpoint from the last epoch in our evaluations of downstream experiments. In AlignCLIP, we set 0.5.
A.3 EFFECTS OF TEXT EMBEDDING SEPARATION
we proposed IMSep to enforce Intra-Modality Separation among the image embed- dings. An alternative scenario could be to enforce the separation amongst the text embeddings, or even, to enforce the separation inside the image embeddings as well as the text embeddings. To ex- amine these alternatives, trained versions of AlignCI.IP without image-image separation, instead
In Section 3.2
14

---

Published as a conference paper at ICLR 2025
introducing text-text separation, which we refer to AlignCLIP-TI. We also used the re-scaling mechanism described in Eq. [0] Furthermore, we trained a version including both image-image embeddings separation and text-text embeddings separation, i.e., HighCLLI--II-TI We followed the same experimental setup as described in Section A.2 and compare the results for zero-shot image classification well linear probing in Tables 8 and respectively. As be seen, imposing the Intra-Modality Separation merely on the image embeddings yields the best results.
MODEL
IMAGENRT-IK
CIFAR-100
CIFAR-10
FLOWERS-
STANFORD CARS
Tor1
Top5
Topl
Tor5
Top1
Top5
Tor1
Tor5
Top1
ToP5
ALIGNCLIP (TT)
31.1
58.6
31.5
60.9
64.8
95.5
18.7
39.5
10.0
36.4
ALIGNCLIP (II)
32.8
60.6
36.5
66.4
69.3
97.8
18.8
40.3
11.8
38.1
ALIGNCLIP (II-TT)
32.4
60.0
31.2
61.8
66.4
96.9
18.7
39.9
11.8
37.1
Table 8: Accuracy scores for zero-shot image classification.
MODEL
IMAGENET-IK
CIFAR-100
CIFAR-10
FLOWERS-102
STANFORD CARS
ALIGNCLIP (TT)
$1.2
64.5
86.3
74.3
41.6
ALIGNCLIP (II)
51.5
67.4
87.2
76.8
45.6
ALIGNCLIP (II-TT)
50.3
63.4
85.8
72.2
42.2
Table 9: Accuracy scores for image classification with linear probing-
A.4 QUALITATIVE EXAMPLES
In this section, further qualitative examples are provided. Figure Tigure&prrviin provides the ground truth for the Top-3 images retrieved in Figure 6 Figure 9 provides
Proviees further examples of the rankings when querying texts. Moreover, Figure 10 illuttrate the text rankings when an image is queried.
A.5 TREND OF THE TEMPERATURE VALUE
In this section, provide analysis of the learned temperature value in each of the CLIP, Shared- CLIP, and AlignCLIP models. We traced the value of and provide the plots in Figure
MODEL
CC3M
MSCOCO
IMAGRNET-IK
CIFAR-100
CIFAR-10
CLIP
0.4
0.44
0.38
0.58
0.36
SHAREDCLIP
0.58
0.61
0.57
0.56
0.57
ALIGNCLIP
0.67
0.69
0.66
0.64
0.65
Table 10: Comparison of the alignment score with ViT-S-16.
A.6 EXPERIMENTAL RESULTS USING THE VIT-S-16 TRANSFORMER
In this section, we provide the results of using AlignCLIP with the ViT-S- 16 backend in Tables 10
15

---

Published as a conference paper at ICLR 2025
MODEL
IMAGENET-IK
CIFAR-I00
CIFAR-I0
FLOWERS- 102
STANFORD CARS
ToP1
ToP5
ToP1
ToP5
ToP1
TOP5
ToP1
ToP5
ToP1
ToP5
CLIP
30.5
57.7
25.3
551.
57.5
94.9
16.8
37.6
SHAREDCLIP
29.8
56.5
27.6
56.7
61.3
95.8
17.7
37
8.8
30.9
ALIGNCLIP
30.8
57.8
27.8
95.9
20.6
38.8
32.9
Table 11: Accuracy scores for zero-shot image classification with ViT-S-16.
MODEL
IMAGENETV2
IMAGENET-R
IMAGENET-A
IMAGENETSKETCH
ToP1
ToP5
ToP1
ToP5
ToP1
ToP5
ToP1
ToP5
CLIP
26.1
52
36.3
63
5.7
22.4
39.6
SHAREDCLIP 25.1
50.3
36.5
62.8
5.4
22.2
18
39.6
ALIGNCLIP
26.4
52.4
38.1
64.5
6.3
23.4
18.6
39.6
Table 12: Accuracy scores for zero-shot classification and natural distribution shift with ViT-S-16.
I1
[1, 2]
T
[-1, 1]
I
I [1.5, 0.5]
T2
[-3, 0.5]
[-0.5, 0.5]
[-2.5, -1]
I
Figure 7: example illustrating that naive embedding shifts can distort the relative distance of the unpaired images and texts.
MSCOCO
FLICKR30K
MODEL
T
T
R@S
R
R@1
R@
R@1
R@S
10
R@1
R@S
R@ 10
CLIP
30.4
56.9
67.4
19.9
42
53.7
52.1
77.9
85.8
38.2
66
76.1
SHAREDCLIP
31.3
57
68.1
20.3
42.2
54.1
52.1
80.9
87.4
38.6
66.4
76.1
ALIGNCLIP
31.8
57.1
68.1
19.9
42.2
53.9
52.7
80.6
87.2
38.7
66.6
76.4
CLIP
38.2
66
76.3
25
50.5
62.8
61
84.8
91.1
45.2
72.1
82
E
SHAREDCLIP
38.5
66
76.8
25.2
51.1
63.4
61.1
85.1
91.7
45.4
73.4
82,2
ALIGNCLIP
39
66.8
76.4
25.4
50.8
63
61.1
85.4
91.6
45.5
72.8
82
Table 13: Zero-shot (ZS) and fine-tuned (FT) cross-modal retrieval summarized with R@ (1, 5, 10). Blue color represents the best performance in the 7S setting. Numbers in bold represent the best scores with ViT-S-16.
16

---

Published as a conference paper at ICLR 2025
country skier the slopes sunny day nan is surfing on a wave in the ocean
The sun shining down on surfer on the beach
holding sur suoab Malking beach next to the
wonan wet suit carrying a surfboard into the ocean
Figure 8: Ground truth texts the Top-3 retrieved images Figure
CLIP
SharedCLIP
AlignCLIP
CLIP
SharedCLIP
AlignCLIP
small
girls
Rank
hydrant has sone
CLO
wearing
blue
writing
jackets pose for
side
Rank 2
photo
CLC
Bank 3
Sone
small pepgeroni pizza next to a fork
oranges that are
Rank
Rank-
sitting
on sore
wood
Rank 2
Rank 2
Rank= 3
Rank 3
Figure 9: Qualitative analysis of the image rankings.
17

---

Published as a conference paper at ICLR 2025
Rankl A photo of Three expty park benches sitting next to orange and pink walls
Rank2 A photo The bench sitting on the side of the road is multi-colored.
Rank3 A photo of An orange on the middle of park banch
SharedCLJP
Rankl A photo of bench with coupte of purple benches
X
Rank2 photo of The wooden park benches painted dark purple.
Rank3 A photo of Three enpty benches face each other beneath canору.
Rank1 A photo of Three enpty park benches sitting next to orange and pink walls
AlignCLIP
Rank2
steel benches
valk way.
Rankl photo bench couple of purpte benches
Rank1
rusted Rank2 A photo of A road sign sttanding near telephone pole.
CLIP
xx*
STOP
Rank3 photo of stop sign that is the side of pole.
SharedCLIP
Rank1 4 photo of stop sign in front of road in small toun in the hills wrecked trucks lot
Rank2 photo Road is Listered with at Rank3 A photo of stop sign and street signs near mountains.
Lone and white
Rank1 photo of
sign leens to right at snall intersection hills.
of read Rank1 photo of stop sign and street signs near mountains.
photo stop sign
b
AlignCLIP
Figure 10: Qualitative analysis of the text rankings given image queries.
(a) Original CLIP
(b) SharedCLIP
(c) AlignCLIP
Figure 11: Trend of learnable temperature value: The Y-Axis shows value and the X-Axis shows the step number.
18

---

Published as a conference paper at ICLR 2025
B LIMITATIONS
For the zero-shot classification experiments, we have tested the models using the ImageNet- k,, CIFAR-IO, CIFAR-I00, Stanford Cars, and Flower-100 102 datasets. Further classification analysis using benchmark datasets such as Hateful Memes has thus far not been conducted in this work. Moreover, this study is limited to the English language and should be followed up by further analyses on multilingual and cross- cross-lingual representation learning. Additionally, the re-scaling mechanism in the intra-modality separation loss dependant on the choice of pre-trained sentence encoder and has not been extensively benchmarked in this work. Pre-trained image encoders could well be explored for providing the semantic supervision for the re-scaling mechanism.
C IMPACT STATEMENT
This paper presents research seeking to advance the field of machine learning at a fundamental level with regard to cross-modal representation learning. We acknowledge that this line of work has broad range of potential societal implications. For instance, vision-language models may exhibit harmful biases and stereotypes, particularly when trained data crawled from the Web. Due their incorporation in prominent generative AI models, vision-language models may also contribute toward the model's s ability to produce images portraying trademarked characters notable figures. These sorts of concerns need to be carefully considered before incorporating such models into real-world applications.
19


Reviews:
[{"review_id": "wSS45v1aLN", "reviewer": "Reviewer_3Rs9", "summary": "This paper proposes the modality gap problems existing in CLIP and shows the multi-modalities distribusion characteristics. To handle this problem, They proposed a novel method named AlignCLIP. The proposed AlignCLIP mitigates the multi-modal gaps with sharing the parameters between all modalities. To align the text-image and image-image well, they also propose the intra-modality separation objective function. Then, they have provided extensive experiment evidance to prove the effectiveness of  AlignCLIP.", "questions": "1. The authors have provided DOSNES projection of the CLIP-encoded image–text pairs from CC3M with ViT-B-32, I wonder how it will be with ViT-L-14 and other image-text datasets?\n2.  are the existing CLIP-based methods performing the same phenomen like original CLIP ? \n3. The authors have mentioned that there are several works also studying the modality gaps, what is the difference between your findings and these works?\n4. Sharing the parameter space between the vision and language encoders may cause each modality like text or image features not learning well, can you provide the parameter comparison between CLIP and AlignCLIP?\n5. The proposed object function may cause the learning direction preferring the image based tasks, can you provide more evidance to prove it ?\n6. The authors should provide more comarison experiments between existing more recent SOTA methods and AlignCLIP in each task settings.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "This paper shows the multi-modalities distribusion characteristics and proposes two novel idea to mitigate such modal gap. This is a clear and well-writen paper, easy to follow. The experiments for ablation is sufficient. This idea of AlignCLIP is reasonable.", "weaknesses": "1. Less evidance to prove the modality gap existing.\n2. It seems this paper is not the first to propose modality gap problems.\n3. I doubt that Sharing the parameter space between the vision and language encoders may cause each modality like text or image features not learning well.\n4. The object function should consider more about the text modality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7RRZo17qjQ", "reviewer": "Reviewer_WiMx", "summary": "The paper proposes a method to mitigate the modality gap in the representational space of CLIP. The authors introduce two modifications to CLIP (termed AlignCLIP) to achieve this: (1) Sharing the weights of the vision and text transformers in CLIP (termed ‘SharedCLIP’), and (2) using a new Intra-modality separation loss, which encourages separation between images in CLIP space while respecting some of the semantics of the similarities between the images.\n\nThe paper’s experiments study the effects of the modifications on a range of downstream tasks, and demonstrate that the AlignCLIP reduces the modality gap and improves zero-shot image classification, linear probe accuracy, robustness to distribution shifts, and multi-modal retrieval accuracy.", "questions": "1. **Why did the authors use “photo of {caption}” prompt in multi-modal retrieval, instead of just the caption itself?**\nWere there any intuitions the authors had about this approach? \n\n2. **Did the authors check performance using just the IMSep loss, without using the architecture of SharedCLIP?**\nThis would allow the approach to be used regardless of encoder architecture. \n\n3. **Did the authors track how the learnable temperature parameter changes during training in AlignCLIP?** Low temperature is known to be an important factor in the emergence of the modality gap.  Therefore, it would be interesting to note the trend and/or the final value of learned temperature in AlignCLIP.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Well motivated paper:** Understanding and mitigating the modality gap is an important direction in CLIP and its variants, which are ubiquitously used in many applications\n\n**Clearly written:** Clear flow of ideas, motivations well explained\n\n**Comprehensive ablation studies:** The authors suggested two directions (SharedCLIP + IMSep loss) for attempting to reduce the modality gap, and the effects of both SharedCLIP and SharedCLIP + IMSep loss were tested. Further, the impact of enforcing intra-modality separation on images, texts and their combination was tested individually. Finally, the effect of the rescaling mechanism to control separation of similar images in the batch was also tested individually.", "weaknesses": "**Potentially limited applicability:** The approach of AlignCLIP is only applicable where transformer architecture can be used for both modality encoders. This may limit the applicability to other multimodal models with a modality gap using non-transformer architectures: The authors could measure the performance of the proposed IMSep loss without the SharedCLIP architecture, as the IMSep loss could be applied to any non-transformer architecture. (See Question 2)\n\n**Unfair comparison to baselines:** I appreciate the authors’ efforts to have a comparison with popular baselines in Table 7. However, as far as I understand, the numbers for CLIP, CyCLIP, and DFSep in Table 7 were taken from the DFSep paper [a] . However, important hyperparameters that are used to train CLIP, CyCLIP, and DFSep in [a] are different to what the authors used to train SharedCLIP and AlignCLIP in their paper. As a result, these comparisons may not be fair. For instance, authors in [a] trained their models for 64 epochs using batch size of 128, and used 1024 dimensional features. Whereas authors of this paper trained for 30 epochs using 512 batch size, and 768 dimensional features. Further, CLIP models in [a] used ResNet-50 as the image encoder, whereas the authors of this paper used ViT-B-16 backend.\n\n- I would recommend that the authors either re-implement CyCLIP and DFSep using the same hyperparameters as in their experiments, or repeat their experiments using the same hyperparameters as used in CyCLIP and DFSep in [a].\n- In either case, these differences should be clearly stated in the paper.\n\n[a] Q. Jiang et al., “Understanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE, Jun. 2023, pp. 7661–7671. doi: 10.1109/CVPR52729.2023.00740.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lv2MmEBaHL", "reviewer": "Reviewer_mh82", "summary": "This paper proposes AlignCLIP to reduce the modality gap in CLIP embedding space. The method leverages a shared transformer to align the image and text embeddings. An additional loss function is also introduced to separate semantically distinct unimodal embeddings. Experimental results show that the proposed method can reduce the modality gap and improve the performance on downstream tasks.", "questions": "please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Investigating the multimodal embedding space of vision-language models is an interesting topic.\n2. The proposed framework seems easy to implement while still being very effective.", "weaknesses": "1. The technical contribution appears limited, as prior work has also leveraged the shared transformer framework [1, 2].\n\n2. The stated goal of this work is to reduce the modality gap within the CLIP embedding space. So basically, the approach should rely on the pre-trained CLIP embedding space. However, the proposed method utilizes a single-encoder framework, which is completely different from CLIP’s two-tower architecture. This raises questions about whether the obtained embedding space is relevant to the CLIP embedding space, which could benefit from further clarification.\n\n3. To my knowledge, the effect of the modality gap on downstream task performance remains an open question. In some cases, a larger modality gap can actually improve performance on certain datasets [3]. This paper could be strengthened by including a more in-depth discussion or insights.\n\n\n\n\n[1] You et al. Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training. \\\n[2] Girdhar et al. OMNIVORE: A Single Model for Many Visual Modalities. \\\n[3] Liang et al. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qk97NBZzX8", "reviewer": "Reviewer_6cpP", "summary": "This paper introduces AlignCLIP, a variant of the CLIP model aimed at reducing the modality gap between image and text embeddings in cross-modal learning. The authors propose two main modifications: SharedCLIP, which shares the learnable parameters of vision and language encoders, and Intra-Modality Separation (IMSep), an additional objective that regularizes the distances within each modality based on semantic dissimilarity. Through extensive experiments, the authors demonstrate improvements in cross-modal alignment and zero-shot performance, with AlignCLIP consistently outperforming baseline CLIP on a range of benchmarks.", "questions": "- The paper states that naive shifts harm downstream performance due to distorted relative distances. Could the authors provide empirical evidence or experiments to support this claim? e.g., Liang et al. (2022)\n- The choice of hyperparameters, such as the weight of the IMSep objective in total loss($\\alpha$), appears arbitrary. Did the authors conduct sensitivity analyses on these hyperparameters, and if so, could they report their findings?\n- While alignment scores improve, does this necessarily translate to enhanced downstream performance? Could the authors discuss specific cases where improved alignment directly correlates with task gains?\n- How significant are the alignment improvements in practical, real-world applications? For instance, does better alignment consistently yield qualitatively superior results in retrieval scenarios?\n- Do you expect your method to scale easily to larger backbones (e.g., ViT-L or ViT-H) using a dataset like LAION-400M instead of CC12M?\n- How is the pre-trained sentence encoder (SBERT) used to calculate semantic distance? Is it robust across various domains and vocabularies in the dataset?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors address a meaningful challenge in multimodal learning, i.e., the modality gap, which affects alignment quality in embedding spaces.\n- The paper evaluates performance on various benchmarks (e.g., zero-shot classification and retrieval tasks) and robustness against distributional shifts, making the findings broad in scope.\n- Visualizations, like the DOSNES projection and alignment score plots, effectively show the improvements in modality alignment and embedding spread across models.", "weaknesses": "- Both SharedCLIP and IMSep primarily extend existing contrastive learning techniques. Sharing parameters between encoders is a known approach, and the IMSep loss largely repurposes InfoNCE without substantial modifications. The paper’s novelty, therefore, is limited.\n- Empirical comparisons with relevant baselines are lacking, as AlignCLIP is only compared with the original CLIP (and SharedCLIP, also introduced in this paper), omitting several cited \"naive\" approaches for modality gap reduction.\n- For example, the paper repeatedly claims that naive alignment approaches \"likely\" negatively impact downstream zero-shot performance and backs up this with a theoretical argument, but no qualitative or quantitative results are provided to show that this is indeed the case.\n- Table 7 also compares AlignCLIP with two other CLIP variations, but from what I understand, these variations were not designed to mitigate the modality gap.\n- In L198-200, the text claims the denominator gets strongly minimized, but the next sentence in L200-202 says it is not strongly minimized. This seems to be a mistake.\n- L269 says \"In AlignCLIP, we set $\\alpha=1$ and $\\beta=0.5$\", $\\beta$ is never introduced in the paper. Moreover, in appendix A.2, it says $\\alpha=0.5$ was used, which contradicts this line.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mS8FNDwEK8", "reviewer": "Reviewer_XfLj", "summary": "This paper proposes AlignCLIP, a solution to the text-image alignment issue in CLIP. Specifically, AlignCLIP (1) shares encoder parameters and (2) enhances cross-modal alignment without disrupting embedding distances by introducing IMSep. IMSep particularly adjusts the unimodal embeddings, placing semantically similar image embeddings close but dissimilar image embeddings apart. Experimental results demonstrate substantial improvements in zero-shot classification, robustness to distribution shifts, and multi-modal retrieval, highlighting AlignCLIP's superior alignment and performance over existing CLIP models.", "questions": "Please see the above weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- AlignCLIP effectively addresses the text-image alignment issue in CLIP, offering meaningful performance gains over existing methods.  \n\n- I like the idea of IMSep, adjusting unimodal embeddings for enhancing cross-modal alignment. \n\n- The method has been verified under various tasks-- especially achieving gains both in zero-shot classification and retrieval seem meaningful.", "weaknesses": "- I am not very confident in the evaluation settings. Are all competing models trained on the same dataset? If not, how is performance comparability ensured?\n\n- The ideas are interesting; however, this still falls within contrastive learning. Could you strengthen the value of your method by connecting it to existing work? While I believe your approach is new, a clearer link to prior work would enhance its contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BhIBi5muYI", "reviewer": "Reviewer_J1QT", "summary": "The paper introduces AlignCLIP, a novel training strategy that aims to improve the cross-modal alignment of CLIP-like models by addressing the modality gap between image and text in the shared embedding space. The authors propose to share the learnable parameters between the vision and language encoders to align the two modalities more closely, and to add an Intra-Modality Separation objective function that pushes semantically dissimilar image embeddings apart. Authors train from scratch CLIP, a SharedCLIP (with shared learnable parameters between the vision and language encoders), and AlignCLIP on CC12M. Extensive experiments demonstrate that AlignCLIP improves zero-shot classification, robustness to distribution shifts, and multi-modal retrieval tasks compared to the original CLIP.", "questions": "- Q1. How would AlignCLIP perform with different backbone architectures besides ViT-B-16? Is the performance consistent with other vision transformers?\n- Q2. Could you explain why AlignCLIP’s retrieval performance only marginally improves over SharedCLIP in certain settings? Are there any specific modifications you plan to address this?\n- Q3. Conducting an ablation study of CLIP using the ImSep loss would be insightful. If such experiments haven't been performed yet, evaluating whether this approach might outperform or underperform compared to AlignCLIP in zero-shot classification and cross-modal retrieval could offer meaningful comparisons.\n- Q4. While mitigating the modality gap is a promising approach to enhancing cross-modal alignment, it might be worth discussing its effectiveness given that the standard definition of the \"modality gap\" [1]—the difference between the centroids of the two modalities—can be zero even if individual image-text pairs aren't perfectly aligned. Clarifying why this approach remains effective under these conditions could strengthen the understanding of its impact.\n\n[1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (https://arxiv.org/abs/2203.02053)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- AlignCLIP demonstrates significant improvements to the original CLIP, with measurable gains in downstream tasks such as zero-shot classification and robustness to distribution shifts.\n- The IMSep objective adds a novel approach to handling intra-modality representations, ensuring that semantically dissimilar image embeddings are spread apart without affecting semantically similar pairs.\n- The paper provides both quantitative and qualitative analyses (I really appreciate Fig. 5), providing clear insights into improvements in retrieval tasks and the reduction of the modality gap.", "weaknesses": "- AlignCLIP’s performance on fine-tuned retrieval tasks shows marginal improvements over SharedCLIP, with SharedCLIP sometimes even outperforming AlignCLIP (see Tab. 5). \n- Ablation studies on the key components, while present, could be expanded. In particular, it would be interesting to explore how much the \"Pre-trained Sentence Encode\" improves the performance and the benefits of using the ImSep loss component. I am afraid that the smaller improvements in such cases could arise just from using an additional pre-trained encoder (i.e. SBERT all-mpnet-base-v2) at training time.\n\n**Additional Consideration**. I noticed several significant errors within the method section Sec. 3.1 and the method figure (Fig. 2) that impact its reliability. In particular, the figure shows:\n- *Conv2d Applied to Image Patches*: I believe this is an error stemming from a misunderstanding of the common CLIP practice of using a Conv2d with a stride equal to the image patch size (that is equivalent to applying a standard linear layer to the flattened patches). So depicting a Conv2d after the patches, as the authors proposed, seems incorrect.\n- *Max Pooling Applied to Text Embeddings*: Similarly, I suspect that applying max pooling to the text token patches is another mistake. The authors stated: \"Following the original CLIP, we use max-pooling for text embeddings.\" (line 165). However, I believe this results from a misunderstanding of OpenAI's CLIP implementation, which uses an argmax operation to select the EOS token.\nThese errors impact the clarity and accuracy of the figure and may mislead readers regarding the model architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EDzen94bku", "reviewer": "Reviewer_ZbcG", "summary": "The paper proposed a novel method named AlignCLIP to address a recognized problem -- modality gap, for the CLIP network. The improved components include (i) a shared transformer and (ii) an intra-modality separation module. The proposed method surpassed the existing methods in most cases.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Technical solid paper with clear stated motivation and well organized content.\n2. The proposed Intra-Modality Separation module is novel and is benefit to the community.\n3. The proposed method surpassed the existing methods in most cases.", "weaknesses": "1. The idea of shared transformer is less novel. From my understanding, since the shared transformer for CLIP has been proposed by You et. al., then extending it to the projection layer is just an incremental improvements. The paper should discuss why the extension to the projection layer is necessary comparing to the existing methods. For example, in terms of \"extend the sharing to the extent possible\" in line #168m, what should we do if using CLIP-ResNet instead of CLIP-ViT.\n2. Since the paper claims the (i) SharedCLIP (ii) IMSep module. It is not clear that which module contributes more on the overall performance. Since the SharedCLIP is strongly dependent on transformer structure, while IMSep module is more related to the general multi-modality contrastive learning task, therefore, the IMSep moduel has better expansibility for various model structures. Therefore, an ablation study on this is necessary, i.e. (original_CLIP+)SharedCLIP vs original_CLIP+IMSep, if I missed this experiment, please point it out during the rebuttal.\n3. It's necessary to add some discussion for Table.5, regarding the COCO T->I. It seems the SharedCLIP works better in the T->I scenario, but what reason cause the degradation while adding IMSep? Some discussion and analysis regarding why this degradation occurs would be benifit for researchers to understand the limitation of the proposed model, for example, is it due the complexity of COCO dataset itself, or due to the IMSep while performing T->I. How is the performance while applying original_CLIP+IMSep only on T->I?\n\n[You et. al] Learning visual representation from modality-shared contrastive language-image pre-training. In ECCV, pp. 69–87, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)