Cross the Gap:  Exposing the Intra-modal Misalignment in CLIP via Modality Inversion
Authors: Marco Mistretta
Alberto Baldrati
Lorenzo Agnolucci
Marco Bertini
Andrew D. Bagdanov
Venue: ICLR 2025 Poster (2025)
Keywords: intra-modal misalignment
CLIP
modality gap
modality inversion
textual inversion
visual inversion
image retrieval
text retrieval
image classification
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We demonstrate that relying on intra-modal similarities computed with off-the-shelf VLMs is suboptimal due to intra-modal misalignment. We show that approaching intra-modal tasks inter-modally via modality version significantly improves performance.

Abstract: Pre-trained multi-modal Vision-Language Models like CLIP are widely used off-the-shelf for a variety of applications. In this paper, we show that the common practice of individually exploiting the text or image encoders of these powerful multi-modal models is highly suboptimal for intra-modal tasks like image-to-image retrieval. We argue that this is inherently due to the CLIP-style inter-modal contrastive loss that does not enforce any intra-modal constraints, leading to what we call intra-modal misalignment. To demonstrate this, we leverage two optimization-based modality inversion techniques that map representations from their input modality to the complementary one without any need for auxiliary data or additional trained adapters. We empirically show that, in the intra-modal tasks of image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance with respect to intra-modal baselines on more than fifteen datasets. Additionally, we demonstrate that approaching a native inter-modal task (e.g. zero-shot image classification) intra-modally decreases performance, further validating our findings. Finally, we show that incorporating an intra-modal term in the pre-training objective or narrowing the modality gap between the text and image feature embedding spaces helps reduce the intra-modal misalignment. The code is publicly available at: https://github.com/miccunifi/Cross-the-Gap.

Published as a conference paper at ICLR 2025
CROSS THE GAP: EXPOSING THE INTRA-MODAL MISALIGNMENT IN CLIP VIA MODALITY INVERSION
Marco Mistretta' Alberto Baldrati! Lorenzo Agnolucci'
Marco Bertini', Andrew D. Bagdanov'
UUniversity of Florence, Media Integration and Communication Center (MICC), Italy
University of Pisa, Italy
{name. surname) @ @unifi. fitt
ABSTRACT
Pre-trained multi-modal Vision-Language Models like CLIP are widely used off- the-shelf for a variety of applications. In this paper, we show that the common practice of individually exploiting the text or image encoders of these power- ful multi-modal models is highly suboptimal for intra-modal tasks like image- to-image retrieval. We argue that this is inherently due the CLIP-style inter- modal contrastive loss that does not enforce any mntra-odall constraints, lead- ing to what we call intra-moda misalignment. To demonstrate this, we lever- age two optimization bassed modality inversion techniques that map represen- tations from their input modality to the complementary one without any need for auxiliary data additional trained adapters. We empirically show that, in the intra-modal tasks image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance with respect to intra- modal baselines on more than fifteen datasets. Additionally, we demonstrate that approaching native model task (e.g. zero-shot image classification) intra-modally decreases performance, further validating our findings. Finally, we show that incorporating an intra-modal term in the pre-training objective or nar- rowing the modality gap between the text and image feature embedding spaces helps reduce the intra-modal misalignment. The code is publicly available at: https://gitthbs www.tttobbccommtccmmmmoot ccuni Fi/Cross Gap
INTRODUCTION
In recent years the availability of massive, pre-trained Vision-Langueen Models (VLMs) has en- abled a wide variety of applications ranging from zero-shot image segmentation (Zhou et al. 2022a Lüddecke & Ecker 2022) to visual question answering (Song et al. 2022, Parelli et al. 2023) These models are typically composed of independent image and text encoders which are simultaneously trained on massive corpora of image-text pairs to align the text and image embeddings of associ- ated inputs. For example, the Contrastive Language-Imas a corpus of 400M image-text pairs to map inputs from both modalities into a shared embedding space (Radford et al.) (021). CLIP is trained with an inter-modal contrastive loss that aims to max- imize the similarity of corresponding image-text samples while minimizing the similarity with all the other examples within batch.
Pre-training (CLIP) model is trained on
Despite CLIP's shared embedding space, visual and textual features lie in distinct regions. This phenomenon, known the modality gap (Liang et al.) (022). originates from model initializa- tion, and the inter-modal contrastive loss preserves and worsens it during training. Moreover, we note that CLIP's contrastive training strategy focuses on inter-modal (i.e. image-text) similarities between paired samples and disregards intra-modal (i.e. image-image and text-text) similarities. Consequently, the intra-image and intra-text similarities between CLIP representations might not faithfully correspond to those of the actual images or texts, as depicted in the left section of Fig. []] We refer to this issue as intra-modal misalignment. A simple experiment aimed at quantifying this problem is presented in Appendix B
Thhee authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
Intra-modal misalignment
From intra-modal to inter-modal
but
is not dog!
Quary
marifckd
Intromodal Approach
Inter-mrodll
Image Encoder
Image Encoder
Modality Gap
Encouter
Encoder
Yoa
Tot
Not used!
Moddilty
Inseesson
Tdog
hnnl
mraniiid)
"s cute cat"
Gallery
amiling'
Similanty
Sinuladly
The inter-modal contrastive loss used in pretraining enforces paired images and texts be at a given distance (i.e. Ndoy and reat) but does not encourage intra-modal alignment. Consequently, intra-modal similarity scores might not correspond to those of actual images and texts (i.e. di < d2). Right: We show that the common practice of individually exploiting only one encoder is suboptimal and approaching intra- tasks (e.g. image-tor sageett--mmage retrieval) inter-modally via modality inversion improves performance.
Figure 1: Motivation and overview. Left:
Aspects of this misalignment have been accounted for in the limited scope of zero- and few-shot image classification (Udandarao et al., 2023; Yi et al. 2024). However, many recent works over- look this phenomenon and still employ CLIP representations for intra-modal comparison, which leads to suboptimal similarity measurements. Examples range from KNN-based image classifica- tion (Geirhos al., (024) text-to-image generation (Gal et al. 2022 Ruiz et al.
2023) and video
synthesis (Esser et al. 2023 Zhang et al., 2024). For instance, Esser et (2023) measure the tem-
poral consistency of generated videos via intra-modal CLIP similarity between consecutive frames.
In this paper we argue that relying on intra-modal similarities computed using pre-trained CLIP encoders is inherently suboptimal. To support this conduct an extensive study of the behavior of intra-modas similarities on the intra- modal tasks of image-to-image and text-to-text retrieval. We perform this analysis by transforming atra-modal tasks into inter-modal ones to leverage CLIP's inter-modal alignment.
Specifically, we map features from their native modality (i.e. the same as the input) into their comple- mentary one. We refer to this process as modality inversion. To perform modality inversion we adapt Optimization-base Textual Inversion (OTI) (Baldrati et al., (023) and introduce Optimization-bel based Visual Inversion (OVI). OTI and OVI are iterative modality inversion strategies that map image fea- tures into text features and vice versa while keeping the encoders frozen. These techniques operate at the Lingle-feature level, i.e. they do not require external data nor the training of mapping network.
Our experiments show that tackling intra-modal tasks midr-ly via modality inversion as il- lustrated in the right side of Fig. D outperforms intra-modal baselines more than fifteen datasets. To additionally support our claim that this performance improvement stems from inter-modal align- ment and not the modality inversion process itself, transform inter-modal tasks into intra-modal ones. Specifically, we show that applying modality inversion to the inherently inter-modal zero-shot image classification task yields worse performance than the inter-modal baseline.
Moreover, we investigate whether the inclusion of an intra-modal loss during image-text contrastive pre-training reduces intra-modal misalignment. For this analysis we use SLIP (Mu al. (022), which adds just such an intra-modal loss to improve the alignment within the image embedding space. Results confirm that adding intra-modal loss terms during the pre-training of VLMs signifi- cantly mitigates intra-mout modal misalignment. Finally, we study the relation between the modality gap phenomenon and the intra- modal misalignment. In particular, similar to Liang et al.) (2022) we fine- tune CLIP to reduce the modality gap and observe a decrease in the performance of approaching intra-modal tasks inter-modally. This indicates that a narrower modality gap diminishes the impact of intra-modal misalignment.
2

---

Published as a conference paper at ICLR 2025
The main contributions of this work are:
we conduct thorough and comprehensive study of CLIP's intran intra-modal misalignment, and our analyses show that the common practice of relying on intra-modal similarities com- puted through pre-trained CL.IP encoders is inherently suboptimal.
we propose transform intra-modal tasks to inter-modal ones via modality inversion to exploit CLIP's inter-modal aligmment. To this end we introduce OVI, a single-feature level modality inversion strategy that maps textual features into the image embedding space;
we conduct extensive experiments that show that approaching intra-modal tasks inter- modally significantly outperforms intra-modal baselines more than fifteen datasets; and
we demonstrate that adding intra-modal loss terms during VLM pre-training or reducing the modality gap mitigates the impact of intra-modal misalignment.
2 RELATED WORK
Contrastively trained Vision-Language Models. VLMs have become increasingly popular for their ability to learn aligned representations across visual and textual modalities (Radford et al.
2021 Jia et al. 2021: Zhai et al.] 2022 2023 Mu et al., 2022 Li et al.) 2021). This alignment enables VL.Ms to be used in a broad variety downstream tasks, including image-text retrieval and zero-shot image classification, by projecting images and text into shared feature space.
The most prominent example is CLIP (Radford et al., 2021), which maximizes the similarity be- tween paired images and text captions while minimizing the similarity with the other samples in the batch. SigLIP (Zhai et al. 2023), on the other hand, employs a sigmoid- -aanddd-asedd contrastive loss that considers only the single image-text pairs while neglecting the other samples in the same batch.
More recently, several approaches have extended the CLIP-style style contrastive loss by incorporating intra-modal similarities into the training objectives (Mu et al., 2022 Li et al.. 202T). For instance, SLIP (Mu et al. (202)) integrates a self-supervised component that maximizes the similarity between different augmentations of the same image, with a strategy akin SimCLR (Chen al. 2020).
The modality gap in multi-modal models. Liang et al. (2022) demonstrated a consistent phe- nomenon affecting VLMs known as the modality gap. This refers to the separation between fea- ture embeddings of different modalities (e.g. text and images) within their shared representation space (Liang al.] (022). The modality gap arises due both model initialization and the con- trastive learning objective used during training. At initialization, independent encoders for each modality produce embeddings that are restricted to distinct regions (or cones) within the representa- tion space. During training, the contrastive learning process preserves and worsens this separation. Several works have studied the causes and implications of the modality gap in CLIP (Shi et al. [023] Schrodi al. 2024; Zhang et al. 2025). [Schrodi et al. (024) analyzed the embedding space and demonstrated that a minimal number of embedding dimensions often as few as two are sufficient to perfectly separate the image and text modalities.
Intra-modal misalignment. Some studies have investigated the problem of misaligned intra- modal embedding distances within the context of zero- and few-shot image classification (Udandarao al., 2023 Yi et af. [0044). To address this, Udandarao et al. (2023) propose mitigating the issue by com- puting similarities in the image-text space, rather than working exclusively with image embeddings, thereby leveraging the inter-modal nature of the feature representations. Similarly, CODER (Yi al. 2024) introduces an enhanced image representation technique based measuring distances between images and their neighboring texts within CLIP's embedding space.
Our contribution with respect to the state-of-the-art. While these prior works have addressed various aspects of intra-modal and inter-modal relationships within VLMs, their scope remains lim- ited, often focusing on specific tasks, datasets, or narrow perspectives on the modality gap and its effects. None of these studies comprehensively investigate the fundamental nature of the intra-modal versus inter-modal similarities across diverse tasks and datasets, nor do they fully explore the poten- tial performance improvements achievable by leveraging inter-modal comparisons for intra-modal problems. The motivation behind our work is to shed light on the phenomenon of intra-modal mis- alignment, and its relationship to the modality gap, and to demonstrate the importance of either ensuring mndra alignment during pre-training or comparing solely representations that belong to different modalities.
3

---

Published as a conference paper at ICLR 2025
3 CLIP PRELIMINARIES
CLIP is a vision-language model trained align images and textual captions in shared embedding space (Radford et al. 2.21). It consists of an image encoder fa and a text encoder Jø- Given an image 7, the image encoder extracts its feature representation (T( E TRd, where d is the size of the shared embedding space. Likewise, for a given textual caption Y., first a word embedding layer Eg maps each tokenized word to the token embedding space V. Then, the text encoder Iø generates the textual feature representation (E, E R.
When using& a Vision Transformer (ViT) (Dosovitskiy et al. 2020) as the visual encoder fe, the encoding process begins by splitting the image into U fixed-size non-overlapping patches. Each patch is then transformed into a corresponding patch embedding {w1, W2...., Wy through linear projection by the patch embedding layer Ew, where each we resides in the patch embedding space W. A learnable class (CLS) token e is concatenated with the patch embeddings, resulting in the input to the vision transformer being / = {c, W1, W2,... wy ). Finally, the CLS token of the final transformer layer is projected into the shared embedding space via a linear projection to obtain the final representation fo(lc, E.. = fo) E R For brevity, when unnecessary will omit both the patch embedding layer Ew and the token embedding layer E,, and use the simplified notations fo( instead of [o( Enw (T)) and (Y) instead of go (Y)).
Given a batch of image-caption pairs B = [(In: Yo) CLIP aims to maximize the cosine sim- ilarity for the N correct pairs while minimizing it for the N22 - other pairs. This is achieved by optimizing symmetric, multi-class N N-pair contrastive loss (Sohn (016)). Let of fo( and TiT = 96 (E (Y.)) denote the image and text embeddings, respectively. The CLIP loss is:
N Σ n=1
Em exp (((C())))////)) )/T) + log exp(c(up,
exp(o(V7, (f)/T)
CALP
log
(I)
N
Σ exp(c(U77 (AT.)))))))) w//)/T))
where .) denotes the cosine similarity, and r is a temperature parameter. As shown by Liang eet al. (2022), Eq. (I) leads to measurable separation between embeddings of the different modali- ties, creating what is known as the modality gap. This gap is significantly affected by the temperature T, with a larger gap occurring as the temperature decreases.
Note that CLIP's training loss focuses exclusively on inter-modal similarities between paired sam- ples while neglecting intra-modal similarities. For example, consider an image feature anchor 4/y and two distinct text features 4hz and v expressing the same concept. The loss enforces both viJ and v/Z be cosine distance from th,, where the cosine distance is defined as d(Eb wB) = 1 c( ). This is equivalent to d(w/, ((11,151)
meaning the text embeddings lie on hypersphere of radius centered at t... The absence intra-modal constraints leaves the alignment between vi+ and 2/ž remains uncalibrated; thus, we have 0 S d(w), v/7) S. 2r. This indicates that, while both text features are equidistant from the image feature, their intra-modal similarity is not constrained in any way, leading to intra-modal misalignment. We argue that such misalignment must either mitigated via additional intra-modal losses during pre-training or must be compensated by tackling intra-modal tasks mnter-
d(wr, (i)))
4 FROM INTRA-MODAL TO INTER-MODAL VIA MODALITY INVERSION
Due to the modality gap, images and text features lie in distinct regions in CLIP's shared embedding space. Previous work introduced modality inversion techniques to map features from the native
modality to the complementary one (Ramesh et al. 2022, Patel et al. 2024 Li et al., 2023). For instance, Ramesh et al. (2022) trains a diffusion model to generate CLIP's image features from text captions for text-to-image generation.
Our goal is to demonstrate that tackling intra-modal tasks in inter-modal way outperforms intra- modal baselines. To this end, we propose to employ a modality inversion strategy to derive represen- tations that exploit both native and complementary modality encoders. However, existing modality inversion techniques rely on external data or the training of mapping network, making the inversion process dependent external factors (Ramesh et al. 2022
Patel et al. 2024
Li et al. 2023).
4

---

Published as a conference paper at ICLR 2025
To minimize the impact of external biases in our analysis, we choose to rely on two modality inversion strategies that operate at a single-feature level, iee that map each individual feature to its complementary modality without the need for any external resources. Specifically, Optimization-base Textual Inversion (OTT) (Baldrati et al. 2023 Agnolucci et al. 2024) and we introduce Optimization-based Visual Inversion (OVI) map image the text embedding space and vice versa while keeping the encoders frozen. Both are iterative and optimization- bassed ap- proaches. The core concept behind OTI and OVI is to learn vectors of trainable parameters that are passed through the encoder of the complementary modality to yield features aligned with the representations of the native modality encoder. By optimizing these input vectors while keeping the encoder weights fixed, we ensure that the output features retain the pre-training alignment. In the following we define OTI and OVI for CLIP, but they can be applied to any VLM that maps images and texts into a shared embedding space. To make our analysis more comprehensive, in Appendix we present two additional experiments in which modality inversion is performed using a pre-trained captioner an adapter.
adapt
4.1 OPTIMIZATION BASED TEXTUAL INVERSION (OTI)
Starting from an image 1, OTI involves iteratively optimizing a set of R pseudo-tokens v² {u', 15, ..,,....,0 " with of E for i E fI..... R), for a given number of optimization steps S. We refer to o" as pseudo-tokens since they belong to the token embedding space V but are not associated with any existing words. Algorithm i Appendix (apendix[A]shoveee shows the pseudo-code of OTI.
The pseudo-tokens v" are randomly initialized and concatenated with the template sentence "a photo of to form y.. ("aa photo of"), v*] input into the CLIP text encoder 9o to obtain 1/r go v..)). Then we extract the features of the image with the CLIP image encoder fa, resulting in / (T). Since we aim obtain textual feature representation d that captures the informative content of I, we minimize the gap between image and text features via a cosine loss:
Loss 1 - c(vl, (T).
Note that while we adapt OTI from Baldrati et al. (2023) our goal is significantly different. Their work focuses on deriving a single pseudo-token that captures the informative content of the image I and can interact with existing words to form meaningful sentences (c.g., "a photo of v² that is running ... "). thus they use OTI for combining inputs from both modalities. In contrast, we use OTI purely as a mapping technique from visual to textual features. We do not focus on the pseudo- tokens themselves but aim obtain a final representation that effectively captures the content of the image I. Additionally, the original OTI technique employs regularization loss that exploits an auxiliary vocabulary to constrain the pseudo-token to reside in CLIPP's token embedding space. However we are not interested in using the learned v* in different contexts and more importantly, we aim to avoid influencing the inversion process with external data. For this reason we do not use regularization loss.
4.2 OPTIMIZATION-BAM BASED VISUAL INVERSION (OVI)
We propose the OVI approach to map text features from the CLIP text embedding space to the visual embedding space. Note that since OVI learns vectors of trainable parameters in the patch embedding space W, it can be applied only to ViT-based image encoders.
Given sentence Y, we first extract its text features श = (6 OVI then optimizes a set of P randomly initialized pseudo-patcher w² [w[ ... w where each w E W. This optimization performed for a fixed number of optimization steps Similarly to the terminology introduced in Sec. 4.1 refer to w" as pscudo-patcher since they belong to the patch embedding space W but are not associated with any existing image. Algorithm 2 in Appendix A illlsstrates the psdudo-code of the OVI method.
Since the ViT employs learned positional embeddings, the number of input patches U to the image encoder is fixed. Consequently, when P < U directly using w" input is impossible. In such cases, we apply nearest searebot interpolation to repeat the ascudo-patches and match the required number of U patches.
5

---

Published as a conference paper at ICLR 2025
Specifically, given the pre-trained CLS token e, the input to the ViT is given by: I = {c, wi, wi....,, wi W22 W2. Wy. W w wph
(3)
H1 times
H2 times
Hp times
where HI..... Hp represent the number of times each pseudo-patch is repeated, and H1 +... Hp U. The specific values are given by the nearest-neighbor interpolation. Finally, the input ww- is passed through CLIP's image encoder to obtain the features ক so(Tw-). To obtain visual feature representation bhy that captures the informative content of Y. we minimize the gap between the image and text features using the same cosine-based loss in Eq.
4.3 CROSSING THE MODALITY GAP WITH OTI AND OVI
The goal of OTI and OVI is to map features from the native modality into corresponding features in the complementary modality. We observe that in cases where the loss Coos approaches zero, the com- plementary features converge to the native ones, thus drifting into the native modality embedding manifold. This undermines the goal of leveraging CLIP's image-text alignment.
For OTI, in our experiments the loss never approaches zero within reasonable number of opti- mization steps when considering a single pseudo-token (i.e. R = 1). We argue that this stems from the strong inductive biases of the frozen encoders and the modality gap, making challenging for a single pseudo-token to bridge the distance between image and text representations. Never- theless, the OTI-inverted features retain the informative content of the corresponding image. As a result, the potential drift related to Leos does not pose significant issue, and inter-modal alignment is preserved. In all experiments we use R = 1 unless stated otherwise.
Also for OVI we observe that the loss only approaches zero when the number of pseudo patches P is relatively large. Unlike OTI, we find that for some experiments a single pseudo-patch (i.e. P = 1) is insufficient for embedding the informative content of the corresponding text. We believe that this discrepancy stems from the inherent differences between images and texts. Specifically, in textual inputs single word (or pseudo-token) instance, the sentences "a photo of a building" and "a photo of dog" convey completely different meanings, despite differing by only one word. In contrast, a single (pseudo-)patch has less influence on the overall semantic content of an image. Therefore, while single pseudo-token is enough for an effective modality inversion with OTI, more pseudo-patches may be necessary when applying OVI. Consequently, in our experiments, we employ a number of pseudo- pseudo-patches P ranging from 4, based on the considered model (see Appendix [ for more details). For such values, pseudo-patches effectively embed the informative content of the input text. Moreover, the inter- modal alignment is maintained and the drift does not constitute a significant problem.
significantly alter the meaning of sentence. For
EXPERIMENTAL RESULTS
Here report broad range of experiments supporting claims. We first evaluate two intra- modal tasks: image-to tongeett-image and text-to-text retrivvvll We show that transforming intra-modal tasks into inter-modal ones via OTT and OVT consistently improves performance by better aligning with the original CLIP training objective. To confirm that this outcome does not stem from the modal- ity inversion process itself, we evaluate a natively inter-modal task zero-shot image classification and show that making it intra-modal hinders the performance. Finally, we analyze the behavior of modality inversion techniques, and we study how adding intra-modal loss terms during VLM pre- vre-traniing or narrowing the modality gap affects intra-modal misalignment. In the following, we denote as inter-modal approaches those involving inter-modal similarity comparisons, i.e. similar- ity comparisons between features of two different modalities (such as image-text, OTI-image, and OVI-text). Conversely, intra-modal approaches refer methods that employ intra-modal similarity comparisons (such image-image, text-text, OTI-OTI, and OVI-OVI.
To ensure comprehensive analysis, we experiment using multiple CLIP models with different backbones and pre-training datasets. We also consider SigLIP to prove that our observations are not specific to the CLIP loss but generalize to other inter-modal contrastive losses. Specifically, we OpenAl CLIP with VTT-B/32 and ViT-L/14 backbones, OpenCLIP (OPEN) pre-trained on the DataComp dataset (Gadre et al. (024) with the same backbones, and SigLIP-B/16 Implementation details and description of all datasets used are given in Appendices A and

---

Published as a conference paper at ICLR 2025
Table Performance (mAP) evaluation on the image-to-image retrieval task. Blue rows indicate the usage of OT Oi'--eeeee features, while white rows refer to the intra-modal baseline. v and X denote inter-modal and intra-modas approaches, respectively.
720xford
RPaais
Flowers
EuroSAT
Food/01
SUN3V7
Callch
LCTIIO
ImsgaNor
Average
Inter modal
SOP
Cas
Pas
DTD
Backbone
CLIP
B/32
X
24.6
22.9
34.4
42.6
70.3
67.9
24.6
30.5
62.0
14.5
14.4
28.1
47.9
47.2
34.7
32.3
34.3
77.1
47.1
48.6
21.4
39.2
43.0
40.8
57.5
76.9
43.3
47.3
84.0
25.8
34.1
59.0
53.0
39.1
83.2
60.0
33.1
52.0
1/14
47.1
41.2
62.4
77.1
50.5
56.0
86.0
27.1
37.7
56.3
55.9
43.5
87.3
62.8
38.2
55.3
OPEN
B/32
34.3
32.1
43.0
44.5
50.8
54.4
75.8
74.7
46.7
50.5
50.5
44.1
77.0
78.0
20.1
19.6
40.9
36.9
56.4
54.5
39.6
42,9
36.2
37.8
83.3
82.5
45,7
48.2
27.3
24.7
47.3
49.5
L/14
56.4
58.9
50.7
51,9
73.2
87.7
65.4
72.6
67.3
92.7
32.5
34.3
44.3
63.11
63.8
61.1
63.2
45.8
89.7
62.6
64.7
42.6
60.4
63.6
Sig
L.P
B/16
X
39.4
49.9
50.6
73.9
65.7
56.5
87.0
37,9
39.9
$2.4
56.3
42.8
87.3
56.7
35.9
55.5
41.8
53.0
79.1
71.8
89.7
37.6
52.9
59.0
88.9
54,9
58.3
5.1 IMAGE-TO-IMANE RETRIEVAL
Pre-trained CLIP image encoders are often used extract features for image-to-image similarity comparisons, For this reason, we study the image-to-image retrieval task.
Experiment design. The objective is retrieve images from gallery that are visually similar to a given query image. We consider a total of 15 datasets commonly employed for image-to-image retrieval and image classification. We consider two strategies. In the first, which we call intra- modal, we directly compare the features of the query image with those of the gallery. In the second, we transform the intra-modal mmage-to-image retrieval task into an inter-modal one by applying OTI to the query image. Then, we use the resulting OTT-inverted features to query the gallery.
Results. We report the results in Tab. I Approaching the task inter-modally using the OTI-inverted features outperforms the intra-modal baseline, achieving an average absolute improvement ranging from 2% to 3%. Specifically, we observe that the performance gain obtained across a large va- riety of datasets with different class granularity and diverse domains, spanning from birds (CUB) to monuments (R.ariss). Moreover, we notice that the intra-modal misalignment phenomenon is independent of the pre-training dataset (CLIP vs. OpenCLIP) and pre-training contrastive loss (CLiP/OpenCLIP vs. SigLIP) since the performance improvement is consistent across all the con- sidered VLMs. Finally, we note that OTI-inverted features cannot contain more informative content than the native ones used by the intra-modal baseline since they are obtained simply by map- ping them to the complementary modality single-feature level and without using any external resources. Thus, the observed improvement is solely attributable to inter-modal alignment rather than to a more enriched representation.
5.2 TEXT-TO-TEXT RETRIEVAL
Although text features from pre-trained CLIP models are not commonly used for text-to-text tasks, we believe that is important to show that our findings also apply text-to-text comparisons.
Experiment design. Using the CLIP text encoder for text-only tasks presents several challenges. Specifically, the CLIP text encoder is trained short, descriptive texts. As a result, using it for tasks such as sentiment analysis or text classification, which involve longer texts and abstract con- cepts, results in a mismatch with the pre- pre-training data. Moreover. VLMs have limited input token capacity (e.g. 77 tokens for CLIP), which makes them unsuitable for longer texts. To avoid these problems, we formulate an intra-modal text-to-text retrieval task using image captioning datasets. Specifically, we select datasets in which each sample consists of an image and multiple associated captions (e.g. Flickr3KK (Plummer et al. 2015). These captions are comparable to those used in VLM training and are short enough to avoid token limit issues. We ignore the images and use the first caption associated with each image as the query text. The goal is to retrieve the other captions related to the same image from gallery of all captions in the dataset. In Appendix G we also report experimental results on purely textual text-to-text retrieval datasets. To address the issue of texts exceeding the input token limit of VLMs,
an L.M to summarize the texts.

---

Published as a conference paper at ICLR 2025
Performance (mAP) evaluation on the text-to-text retrieval task. Purple rows indicate the usage of OVI-inverted features, while white rows refer to the motall baseline. Right: Per- formance (accuracy) evaluation on the zero-shot image classification task. Blue rows indicate the usage of OT1-inverted features, while white rows refer to the inter-modal baseline. v and X denote inter-modal and intra-modal approaches, respectively.
Table Left:
ImapeNet
COCO
Averogg
UCP101
Ancrage
Inter
Inter
Backhons
CUIP
BFR2
35.1
37.7
60.4
87.5
67.0
19.1
4366
45.2
80.5
62.0
91.2
62.0
62.1
61,9
1/14
X
35.7
38.2
76.8
93.6
79.3
32,5
5300
58.1
91.0
67.6
96.9
74.2
73,5
72.2
29.4
39.5
41.3
OPEN
B/32
32.0
43.6
42.6
46.3
58.4
90.3
73.5
34.4
53U1
56.5
830
69,4
1/14
47.0
95.01
82.5
47.6
75.0
B/15
60.1
29.6
43.4
40.8
44.4
85.8
43.9
62.11
42.3
89.3
69.6
75.7
As in the image-to-imagee retrieval experiments detailed in Sec. 5.1 we consider two strategies. In the former, we use the query text features to retrieve from the gallery. In the latter, we approach the task inter-modally by applying OVI to each query to obtain the complementary features.
Results. Table 2 (left) shows the results. Analogously to image-to-image retrieval, using the OVI- inverted features outperforms the intra-modal baseline. Specifically, the absolute performance gains range from 1% to 5% across all datasets, backbones, and VLMs. This outcome proves that intra- modal similarity comparisons lead to suboptimal performance independently from the considered modality. Hence, the intra-modal misalignment phenomenon involves both the image and text em- bedding spaces.
5.3 ZERO-SHOT IMAGE CLASSIFICATION
We evaluate the performance of modality inversion on inter-modal tasks, such as zero-shot image classification and image-text retrieval. We expect that transforming inter-modal tasks intra-modal ones hinders performance due to intra-modal misalignment. Here we consider zero-shot image classification, while we report experiments on image-text retrieval in the supplementary material.
Experiment design. CLIP-like models perform zero-shot image classification by predicting the output class based the similarity between the input image and set of textual prompts in the form of "a photo of a /CLASSJ", where CLASS represents each class name, such as "cat" or "dog"" Following Zhou et al. (2022b), we take into account 11 datasets (see the supplementary material for more details). We consider three strategies. The first is the inter-modal baseline, which compares the features of the input image and the set of prompts. the second, we apply OTI the input image. In the third, OVI is applied to each textual prompt.
Results. In Tab. 2 (right) we report the performance of the first two strategies described above. Results for the third strategy are given in Appendix G As expected, using modality inversion con- sistently leads to performance degradation across different VL.Ms and backbones. Note that the datasets used in zero-shot image classification Sec. 5.T This allows us to reuse the same OTI-inverted features for both tasks. Interestingly, the results are opposite: performance improves in image-to-image retrieval but decreases in zero-shot image classification. This contrast arises because, in the former, we transform an intra-modal task into an mnter-modal one, while in the latter, we do the reverse. This experiment demonstrates that modality inversion does not inherently improve performance, as the same OTI-inverted features can either enhance hinder results depending the nature of the task.
also employed for image-to-image retrieval in
5.4 ANALYZING MODALITY INVERSION
In this section we study how and why transforming native modality features into complementary ones via modality inversion leads performance improvement intra modal tasks. For brevity, we consider only OTL. but we find that the same considerations apply to OVL. We consider the Cars dataset (Krause al. 2013) and the CLIP ViT-B/32 model.
8

---

Published as a conference paper at ICLR 2025
0.0
totan ORt
OTtimeee tolens, ctss 17;
(TT) ontiienn (4 rilens ctss 1900)
30
sumple Doratty
Took mage Image trage-inapu
0.4
28
28-
0.2
Intw madil perforrance
0.0
250
400
600
8:0
1020
200
400
1000
5.0
0.2
0.4
0.5
0.5
202
Number
Number of atimization
(a)
(b)
(c)
Figure 2: (a, b) Loss values and retrieval performance over OTI optimization steps for different numbers of pseudo-tokens R. (c) Distribution of pairwise amage-image text-image, and OTI-image cosine similarities. We consider the OTl-inverted features using four pseudo-tokens (R = 4) at two distinct optimization steps: the performance peak (step 17) and the final step (step 1000).
In Figs. aaa) and 2bb) we investigate how the values of the loss Less and the image-to-image retrieval performance vary based on the number of optimization steps and ppeudottokens R. First, we notice that with a single pseudo-token (i.e. R 1) the loss does not reach zero within reasonable number of optimization steps. Conversely, as R increases e..e. the number of trainable parameters grows) the loss decreases more rapidly and approaches zero. As discussed in Sec. 4.3 as the loss decreases the OTI-inverted features shift away from the text manifold towards the image manifold, approaching the original native image features. This phenomenon is reflected in the image retrieval performance shown in Fig. 2bb). since for enough optimization steps and pseudo-tokens the performance ap- proaches those obtained by the native image features. Moreover, we observe that, regardless of the value R, the best performance corresponds to a relatively low number of optimization steps.
We argue that, in proximity to the performance peak observed during the optimization process, the OTI-inverted features capture the informative content of the corresponding image while retaining the inter-modal alignment. To support this claim, we compute cosine similarities for image-image, text-image, and OTl-image feature pairs. Specifically, we analyze the OTI-inverted features using four pseudo-tokens (R = 4) at two distinct optimization steps: the performance peak (step 17) and the final step (step 1000), where performance approaches the intra-modal baseline. In Fig. 2(c) we plot the distribution of these pairwise similarities. We observe that, at the performance peak, the similarity distribution of OTI-image matches the text-image similarity distribution, while at the final step aligns with the image-image similarity distribution, confirming the drift of the OTI-inverted features toward the image manifold. This suggests that OT1-inverted features perform best when aligned with image features in the same way as text features, confirming our hypothesis that the performance improvement obtained by OTI stems from leveraging CLIP's inter-modal alignment.
Finally, we notice that R = is not the optimal choice to achieve the best performance when using OTL Still, we use R = 1 in the experiments the associated OTI-inverted features are less prone to drift towards the native image features, thus being more robust to the number of optimization steps. Moreover, the main objective of this work is not achieve the best results the downstream tasks but rather show that using VLMS intra-modally is suboptimal.
5.5 THE ROLE OF IOTRA-MODAL CONSTRAINTS
We investigate whether incorporating an intra-modal loss term during image-text contrastive pre- training effectively mitigates the issue of intra-modal misalignment. To this end, we consider SLIP Muu et al., (022). which adds a self-supervised intra-modal loss based on SimCLR (Chen 2020) to the standard CLIP inter-modal contrastive loss LCUP (see Appendix C for more details). Such intra-modal loss encourages the model to produce similar representations for two augmentations of the same image, aiming to improve the intra-modal alignment within the image embedding space.
To verify this, we perform an image-to-image retrieval experiment following the evaluation protocol from Sec. 5.1 We report the results in Tab. 3 Notably, the OTI-inverted features achieve comparable performance to the native image ones. This contrasts with results from VLMs trained solely with an inter-modal contrastive loss (see Tab. [ in which OTI led to substantial performance boost. This experiment proves that SLIP's s intra intra-modal loss effectively reduces intra-modal misalignment and suggests the importance of including such loss when pre-training VLMs.

---

Published as a conference paper at ICLR 2025
Table 3: Performance (mAP) evaluation on the image-to-image retrieval task using SLIP model. Blue rows indicate the usage of OTl-inverted features, while white rows refer to the intra-modal baseline. v and X denote modal and intra-modal approaches, respectively.
ROxfood
RParis
Flowers
Aissmii
EuroSAr
FoodlO1
SUN397
Caleeh
UCFI0
Imagete
Averge
Inter modal
S0P
Cars
Pass
DTD
Backbone
B/16
16.6
36.2
66.2
9.1
29.8
65.5
14.5
SLIP
16.2
48.8
36.4
79.3
5.0
19.3
65.1
9.0
30,5
50.6
20.0
26.4
67.6
40.6
14.8
35.3
19.5
46.4
36.3
75.3
69.2
9.7
28.8
56.5
24.25
71.0
17.4
36.7
19.6
45.8
38.0
75.
5.5
23.3
70.2
9.8
29.7
53.7
25.2
27.8
72.3
41.4
18.2
37.1
5.6
THE ROLE OF THE MODALITY GAP
Table 4: Impact of the modality gap on the per- formance (mAP) for the image-to-image retrieval task on image retrieval datasets. Blue rows in- dicate the usage of OTI-inverted features, while white rows refer the intra-modal baseline. / and X denote inter-modal and intra-modal ap- proaches, respectively.
During CLIP pre-training. the temperature pa- rameter in Eq. ① critically affects the modal- ity gap: higher temperatures considerably re- duce or close (Liang et al. 2022). To exam- ine the impact of the modality gap on the intra- modal misalignment, we fine-tune a CLIP VIT- B/32 model on the COCO dataset (Lin et al.) 2014) using a temperature т 1.0, which closes the modality gap. As reference, we repeat the experiment with r = 0.01, i.e. the value employed during CLIP pre-training. See Tab. A2 for more details on the magnitudes of the modality gap for the different models.
R.Oxford
R.riis
Average
Inter modal
CUB
SOP
Caus
Temperature
15.9
23.7
29.3
46.6
19.3
27.0
pap)
17.4
X
24.0
35.0
43.1
68.6
25.7
39.3
24.1
35.2
44.0
70.2
27.6
We reproduce our image-to-iman retrieval experiments using these fine-tuned models and report results in Tab | In the absence the modality gap (r 1) tackling intra-modal tasks inter-modally does not improve performance. This shows that closing the modality gap reduces the intra-modal misalignment. The results of the reference model stem from the fine-tuning strategy. As also observed by Liang et al. (2022), we note that using higher temperature values during training leads to an overall performance decrease downstream tasks, despite reducing the modality gap. For this reason, we argue that in practice simply increasing the temperature value in Eq. does not represent viable strategy to address intra- modal misalignment.
0.01) prove that this outcome does
6 CONCLUSIONS
In this work show that relying on intra-modal similarities computed with off-the-shelf VLMs suboptimal for intra-modal tasks like image-to-image and text-to-text retrieval. This stems from the inter-modal contrastive loss employed for pre-training these models that leads to a modality gap and intra-modal misalignment. We propose to transform intra-modd modal tasks to inter-modal ones via two hingleffeature level modality inversion techniques. We demonstrate that this strategy improves performance as exploits the inter-modal alignment of VLMs. Finally, we show that employing an intra-modal loss component during VLM pre-training or reducing the modality gap alleviates the impact of mndal misalignment.
Limitations. Our analyses demonstrate the significance of intra-modal misalignment when exploit- ing pre-trained CLIP models, but fall short of offering practical alternatives. The modality inversion techniques we propose are computationally expensive. They are based on iterative optimization of learnalle input parameters (150 optimization steps for OTI and 1000 for OVI in our experiments). This limits their practical applicability and future work should concentrate on efficient methods to mitigate the intra-modal misaligument.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by funding from the Italian national project Collaborative Explainable neuro-symbolic Al Decision Support Assistant, CA14DSA, CUP B13C23005640006
REPRODUCIBILITY STATEMENT
We have taken steps in this work to ensure the reproducibility of our results. All models and datasets used in our experiments are publicly available and we release the complete source code. In the main paper and appendices material model architectures, training and evaluation protocols, and hyperparameters. All random seeds are fixed ensuring that others can replicate our results with the provided code. We believe that the measures we have taken to ensure reproducibility will facilitate straightforward replication and verification of our findings, as well allow the community to build upon our results in the future.
provide complete details of all experimental setups, including
REFERENCES
Lorenzo Agnolucci, Alberto Baldrati, Marco Bertini, and Alberto Del Bimbo. iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval. arXiv preprint arXiv:2405.02951,
2024. 5
Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8948-8957, 2019.
Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-Shot Composed Image Retrieval with Textual Inversion. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 15338- 15347, 2023.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 6-12, 2014, Proceedings, Part VI 13, pp. 461- Springer, 2014. 18
discriminative compo-
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning,
pp. 1597-1607. PMLR, 2020. 3 9 16
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3606-3613, 2014. 18
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009. I7
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint 24Xiv:2407.2 1783, 2024. 20
Jeremy Elson, John R Douceur, Jon Howell, and Jared Saul. Asirra: captcha that exploits interest- aligned manual image categorization. CCS, 7:366-374, 2007. 15
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germani- dis. Structure and suntent-guided video synthesis with diffusion models. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 7346-7356, 2023. 2

---

Published as a conference paper at ICLR 2025
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pp. 178-178 IEEE, 2004. 17
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets, Advances Neural Information Prucessing Systems, 36, 2024. 6
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprini arXiv:2208. 01618, 2022. 2
Robert Geirhos, Priyank Jaini, Austin Stone, Sourabh Medapati, Xi Yi, George Toderici, Abhijit Ogale, and Jonathon Shlens. Towards flexible perception with visual memory. arXiv preprint arXiv:2408. 17,72 2024. 2
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019. 17
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916.
PMLR, 2021. 3
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1128-113, 2015, 18 19
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013. 18
Ken Lang. Newsweeder: Learning to filter netnews, In Machine learning proceedings 1995, pp. 331-339. Elsevier, 1995. 19
Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding CLIP latents for zero-shot captioning via text-only training. arXiv preprint arXiv:2303.03032, 2023.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre- training paradigm. arXiv preprint arXiv:2110. 05,0008, 2021. 3
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the Gap: Understanding the Modality Gap in Multi-Modal Contrastive Representation Learning. Advances in Neural Information Processing Systems, 15:17612-1722 2022.
17
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ESsion-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014. 10
19 21
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning, Advances
in neural information processing systems, 36, 2024. 20
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
Bkg6Ri C
15
Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7086-
7096, 2022. [
12

---

Published as a conference paper at ICLR 2025
Andrew Maas, Raymond E Daly. Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011. 19
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint orXiv:1306.51550 2013. 18
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets
language-image pre-training. In European conference on computer vision, pp. 529-544. Springer, 2022.
3 9 16
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp- 722-729. IEEE, 2008. 18
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 4004-4012, 2016. 18
Maria Parelli, Alexandros Delitzas, Nikolas Hars, Georgios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Clip-guided vision-language pre-training for question answer- ing in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5607-5612, 2023. !
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012. 18
Maitreya Patel, Changhoon Kim, Sheng Cheng, Chitta Baral, and Yezhou Yang. Eelipse: A
resource-efficient text-to-image prior for image generations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9069-9078, 2024.
20
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region- cesion-to-phree to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
18 19
Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondiej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In Proceedings of the IEEE conference
computer vision and pattern recognition, pp. 7770-5 2018. 18 21
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International conference on machine learning,
pp. 8748-8763. PMLR, 2021.
1 18
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022. 4
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 22500-
22510, 2023.
Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Representation Learning. arXiv preprint
2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2225278-252944 2022. 21
13

---

Published as a conference paper at ICLR 2025
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp-
2556-2565, 2018. 20
Peiyang Shi, Michael C Welle, Márten Björkman, and Danica Kragie. Towards Understanding the Modality Gap in CLIP. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
Kihyuk Sohn.
Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, and Furu Wei. Clip models are few-shot learners:
Empirical studies vqa and visual entailment. arXiv preprint arXiv:2203.07/99 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 1XI1:1212.0042. 2012. 18
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. SuS-X: Training-Free Name-Only Trans- fer of Vision-Languuage Models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2725-22222222 2736, 2023. 3
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-201 dataset. 2011.
18
Chao Yi, Lu Ren, De-Chuan Zhan, and Han-Jia Ye. Leveraging Cross-Modal Neighbor Representa- tion for Improved CLIP Classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27402-274444 27 2024.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learn- ing Research. 21
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18123-18133, 2022. 3
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1975--1986, 2023. 16
Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang. Kuan-Chich Wang, James Zou, and Serena Yeung. Diagnosing and Rectifying Vision Models using Language. In International Conference on Learning Representations (ICLR), 2023.
Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. Avid: Any-length video inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7162-7172, 2024. 2
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference Computer Vision, pp. 696-712. Springer, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2344 2022b.
17
1
14

---

Published as a conference paper at ICLR 2025
Algurithm Optinization based Textual Inversion (OT))
Algurithm Optimization bused Visual Inversion (OVI)
Input: image number pseudo-tokens number of opti
Input: Text Y. number of pseudo- patches P. number optimiza-
mization steps 5
tion steps S
Initialize n² [ri 2 .....: 1/R)
Initialize uv" = [MI]. ........ .,
3: Extract image fearures:
fo(
3: Extract text features: v/r 9o (X))
4: for = to do
4: for 1 to do
5: Form Y "* [E, ("a photo of"),, 1²1
Form input f using Ey.
6: Extract text features: Vor g +*
6: Extract image features: 17 = so(lu*)
7: Compute loss: foes
cos (192, 1/T)
Compute loss: Lon.
cos (Vz,, vT)
Update v² minimize Le.e
8: Update w" minimize Cao
9: end for
9: end
10: Oulput: OTI inverted features VT (Y
10: Output: OVI-inverted features thy fo(tw=)
Algorithms 1 and 2. Left: OTI maps an image into the textual embedding space by optimizing pseudo-tokens. Right: OVI maps text into the visual embedding space by optimizing pseudo- patches. Both approaches iteratively minimize the cosine distance between the feature representa- tions of the native and complementary modality.
APPENDIX A IMPLEMENTATION DETAILS
OTI and OVI. We report the pseudo-code of Optimization-busee Textual Inversion (OTI) and Optimization-basel Visual Inversion (OVT) in Algorithm [ and Algorithm 2 respectively. Un- less stated otherwise, we use the same hyperparameters for OTI and OVI. We employ the Adam
Loshchilov & Hutter (2019) optimizer with learning rate equal to 0.02, 3 = 0.9, 83 = 0.999, and weight decay 0.01. We perform 150 optimization steps for OTI and 1000 steps for OVI. For OTI, we consistently use single pseudo-token (R = 1). In contrast, for OVI, we employ a number of petuddo-pacchss P ranging from 1 to 4, depending on the considered model (see Sec. D for more details). On average, when using the CLIP ViT/B-32 model, OTI takes approximately 0.2 seconds per image, while OVI takes around 0.5 seconds per prompt on single A 100 GPU (40GBs) with a batch size of 2048. The memory usage scales linearly with the batch size. Specifically, when using the CLIP ViT-B/32 model, OTI requires approximately 1,878 MiB plus 18.6 MiB per sample in the batch. For example, with a batch size of 128, the memory consumption is about 4,260 MiB. For OVI, the memory usage is approximately 2,218 MiB plus 16.2 MiB per sample, resulting in about 4,290 MiB with the same batch size. We use mixed precision to save memory and increase computational efficiency. In downstream tasks all the features are normalized to have a unit L2-norm.
CLIP fine-tuning, To investigate the role of the modality gap on the intra-modal misalignment, we perform fine-tuning of the CLIP model using different loss temperatures (see Sec. (36)). In particular, we fine-tune the CLIP ViT B/32 model on the COCO training set for 30k steps, using batch size of 512 and a learning rate of 1e-6. As an optimizer we employ AdamW with 3 = 0.9, 82 = 0.98 and a weight decay of 0.2. To mitigate possible overfitting issues, we train only the final projection layers. We train two different models, in the first we set the loss temperature parameter 1.0 (first two rows of Tab, 4 while in the second use 0.01 (last two rows of Tab.
APPENDIX B MORE INSIGHTS ON INTRA-MODAL MISALIGNMENT
To provide quantitative insights into the intra-modal misalign- ment issue we conduct a simple experiment using the CLIP ViT-B/32 model and the "Dogs vs Cats" dataset (Elson et al. 2007]. This dataset consists of 25K images evenly distributed between two classes: dog and cat. Our goal to demonstrate that, despite inter-modal alignment, the intra-modal similarity scores are misaligned, i.e. they might not reflect those of actual images and texts, as illustrated in the left section of Fig. T
0.3
0.6
DJ 0.8
0.9
1.0
Smibdit
Figure Al: Distribution of pair- wise dog-dog and dog-cat image similarities. Overlap highlights the intra-modal misalignment issue.
We start by filtering out images with incorrect inter-modal alignment to class-specific prompts. Specifically, we remove dog images that exhibit higher similarity to the prompt "a photo of cat" than to the prompt "a photo of dog". Then
15

---

Published as a conference paper at ICLR 2025
the dog-related prompt to query the gallery of all images and filter out the minimal number of images that are incorrectly ranked for this query. We repeat the same procedure for cat images. This filtering ensures perfect mnter-modal alignment and text-image retrieval scores.
On the resulting filtered dataset, we perform image-to- toaageto-image retrieval using dog images as queries and the whole set of images the gallery. If inter-modal alignment guarantees mntra-modal align- ment, all dog images should rank higher than cat images for any dog query, resulting in perfect retrieval. However, our results contradict this assumption. Specifically, we observe mean Average Precision (mAP) of 83. % and an average R-Precision of 73.2%, where R-Precision represents the precision at rank R, with being the total number of relevant items for a given query. These find- ings indicate that on average at least 26.8% of dog images are ranked below cat images for a given dog query. Figure AT qualitatively illustrates this issue, revealing significant overlap between the distributions of pairwise dog-dog and dog-cat image similarities. We observe similar results when employing cat images as queries. Given the evidence of intra-modal misalignment in such toy dataset, we believe that the issue is likely to be even more pronounced in more complex datasets with more classes.
APPENDIX C ADDITIONAL VLMS
In this section, we provide a more detailed explanation the SigLIP and SLIP models, highlighting their key differences from CLIP.
SigLIP. In SigLIP ZZhai et al. 2023). given a batch of image-caption pairs B = {(4, Y N training maximizes the cosine similarity for the N correct pairs and minimizes it for the Nº N incorrect pairs. Unlike the softmax-base employs sigmoid-based loss that avoids global normalization factors. Each image-text pair is processed independently, transforming the learning task into a binary classification problem across all pair combinations. The matching pair (4,, Y,) receives a positive label, while all other pairs (I,, Yjyz) receive negative labels. SigLIP consists of an image encoder So and text encoder Ja- We denote the image and text emheddings as vs = fo():) and ta/p = go (V:), respectively. The loss employed by SigLIP is:
1 1
contrastive loss from Eq. ① used in CLIP, SigLIP
SigLL Σ log
(1
where -)-,-) denotes the cosine similarity, T is a learnable temperature parameter, b is a learnable bias, and zij is the label for given image and text input (zıg 1 if = j and žÚj -1 otherwise). Similar to CLIP, SigLIP's loss does not include explicit intra-modal constraints; the loss focuses solely on inter-modal alignment between image and text embeddings, without directly enforcing indal alignment.
SLIP. SLIP Mu et al., 2022) is a VL.M trained with both language supervision and image self- supervision. Its loss function combines two components: the moder-modal loss used in CLIP (Eq. I.) and an modal self superviee term. For the self-supervised component, SLIP uses an adapta- tion of SimCLR (Chen et al.) (020)). This loss, denoted as LSiMCLR, maximizes features similarities between two different views (i.e. augmentation) of the same image:
exp (clup()) (7)))) )/)
2N
SSiCCLR = 2N
log
(5)
Σ Ik= exp (c(V), w/)/7)²
2N
1=1
where vb/ indicates the image embedding of a sample j. p(1) and (()) are two augmented views of the image i, c(.,-) denotes the cosine similarity, and T is temperature parameter.
The final loss used SLIP is a combination of CLIP and self-supervised losses:
LSLP = LCUP + LSICLLR-
(6)
By incorporating the intra-modal self-supervised loss, SLIP encourages better intra-modal align- ment within the image embedding space. We confirm this empirically in Tab. 3
16

---

Published as a conference paper at ICLR 2025
APPENDIX D SELECTING THE NUMBER OF PSEUDO- FOR OVI
In Sec. 4.3 we observe that for certain experi- ments single pseudo-patch (i.e. P 1) is not enough to encapsulate the informative content of the associated text. To determine the optimal number of pseudo-patcher for each VLM, we conducted a text-to-text retrieval experiment on the Flickr30K (Plummer et al. 2015) validation set, varying the number of from to 16.
Table A1: Ablation on the number of OVI pseudo- patches for text-to-text retrieval on the Flickr30K validation set. The highest mAP score in each row highlighted in bold, with the correspond- ing value of P representing the number of pseudo- patches used in the experiments.
Number of Pscudo-Patches P
Table Al presents the results of this ablation. We observe that the ideal number pseudo- patches changes depending
VLM
Backbone
Intra-modal
2
8
16
R/32
51.4
54.5
52.9
51.8
51.6
51.6
the model. In particular, larger VLMs with a greater num- ber of input patches U tend to require more pseudo-patches. We hypothesize that this is be- cause, as the number of patches increases, the influence of single pseudo-patch decreases, necessitating a larger number capture suffi- cient information.
CLIIP
L/14
52.6
51.7
55.2
56.0
55.3
54.1
B/32
57.3
59.6
57.9
57.5
57.4
57.4
OPEN
62.5
61.2
SigLIP
B/16
56.3
45.2
58.0
60.1
59.9
59.4
B/16
45.8
46.4
46.4
46.1
45.9
45.9
SLIP
L/16
49.8
48.9
50.0
49.9
49.8
APPENDIX E DIFFERENT VLM, DIFFERENT MODALITY GAP
In Sec. [.6 show that in the absence of the modality gap, tackling intra-modal tasks inter- aiter-modally does not improve performance. This demonstrates that closing the modality gap helps reduce intra- modal misalignment.
Following Liang et al. 2022) we define the modality gap as the difference between the two centroids of the image and text modality embeddings:
N
gap
34
Vi
where 2; and ys are the L.2 2-2ommaaldeed image and text embeddings, and N is the number of pairs.
To facilitate clearer comparison across differ- ent VLMs, in Table[A2] report the magnitude of the modality gaps evaluated on the COCO validation split. We observe that integrating intra-modal constraint (e.g. SLIP) or using higher temperature in the contrastive loss (e.g. fine-tuned model with temperature helps reduce or even eliminate the modality gap. By analyzing Tabs. to 4 confirm our hypothesis that exists a positive correlation be- tween the magnitude of the modality gap and the improvement in approaching intra-modal tasks mnter-modally using OTI (or OVI).
Table A2:
| for different VLMs on COCO.
VLM
Backbone
Less
||
B/32
0.82
CLIP
Car
L/14
0.82
B/32
0.82
OPEN
Cur
1)
L/14
0.30
SigLIP
B/16
Lugur
1.05
B16
0.57
Ceur LsunCi
L/16
0.49
Fine-tuned
Lour( 1)
0.007
B/32
CaulT 0.01)
APPENDIX F DATASET DETAILS
Our experimental evaluation is performed on 18 datasets. Here we report all the evaluated splits and details of the datasets used in our experiments.
Zero-shot Image Classification. Following Zhou et al. (2022b). we validate our zero-shot im- age classification experiments 11 publicly available datasets with diverse characteristics: Im- ageNet (Deng et al.] 2009) for large-scale object classification; Caltech101 (Fei-Fei et al. (004) for general object classification; FuroSAT (Helber al. 2019) for satellite image recognition;
17

---

Published as a conference paper at ICLR 2025
Food101 (Bossard et al.. 2014), FGVCAireral (Maji et al. 2013), OxfordPets (Prkhi al. 2012) Flowers (Nilsback& & Zisserman 2008). and StanfordCars (Krause et al. 2013) for fine- -ne-grained classification; UCFI01 (Soomro et al. 2012) for action recognition; and the Describable Textures Dataset (DTD) (Cimpoi et al., 2014) for texture classification.. Following Zhou et al. (2022b), we discard the "BACKGROUND Google" and "Faces easy" classes from Caltech101, For UCF101 video dataset- - we follow Radford et al. (2021) and use the middle frame of each video clip as the input image. In all classification experiments, we report the accuracy results on the test set.
image-to-Image Retrieval. For image-to-image retrieval experiments, we use the 11 datasets also employed for zero-shot image classification and four widely used datasets commonly used for metric learning and image retrieval: CUB-200-200
(CUB) (Wah et al. (011). Stanford Online Products (SOP) (Oh Song et al. 2016), ROxford (Radenovie et al. 2018). and RPrris (Radenović et al.) 2018), for a total of 15 datasets. In the 11 datasets used for zero-shot image classification, we use the test set as the query set and the training set as the gallery. For CUB, the entire dataset is used as both the query and gallery sets. In SOP, both the query and gallery sets are taken from the test set.
In all experiments involving ROxford and RParis, we follow the standard benchmark and include the RIM distractor set, containing 1 million images, negative samples for all the queries. For brevity in the paper we report only the metric calculated on the Easy setting Radenović et al. (2018). For image-to-imasd
retrieval evaluation, use the standard mean Average Precision (mAP) metric. Importantly, by using the same 11 datasets employed for zero-shot classification, we can evaluate the performance of the same OTI-inverted features both inter-modal zero-shot classification and intra-modal image retrieval tasks
Text-to-Text Retrieval. We performed our text-to-text retrieval experiments using three image- caption datasets: COCO (Lin et al., 2014), Flickr30K (Plummer et al. 2015), and nocaps (Agrawal al. 2019). We selected these datasets for two reasons: they contain short, descriptive text similar to the ones used to train VLMs, and they provide multiple captions for each image. In our evaluation, we use the first caption of each image as the query and aim to retrieve the other captions associated with the same image from gallery of all captions in the dataset. On average, COCO and Flickr30K images have captions each, while nocaps images have 10. We use the Karpathy split (Karpathy) & Fei-Fei 2015) for both COCO and Flickr30K and report results using captions from the test split. For nocaps, we report results on the validation split. Although these datasets contain images associated with captions, we ignore the images in this setting. We use mAP as the evaluation metric similar to the image retrieval experiments.
APPENDIX G ADDITIONAL EXPERIMENTS
Here we report additional experiments to support our claims about the importance of approaching tasks inter-modally when using constrastively trained VLMs.
Zero-shot Image Classification with OVL Due to space limitation, in Sec. 5.3 we provide a brief overview of how CLIP-like models can perform zero-shot image classification. Here, we offer more detailed explanation for clarity. Given an image I and a set of textual prompts y = (Y; i = 1,..., C), where C. is the number of classes, each text prompt Y, is formatted as: "a photo of J", with CLASS; representing a specific class name, such as "cat" or "dog". . Let the image features be denoted vi [T) and the text features for each prompt by vor gV.)). The probability of the image belonging to each class is then given by:
exp(e(1t, w1)/T)
ply = +11) = ES exp(c(U),)))) (1)/7)
(8)
where o(.,)) denotes cosine similarity and T is temperature parameter.
In Sec. 5.3 we transform zero-shot image classification from being natively inter-modal to intra- modal by applying OTI to the input image I (see the right section of Tab. (2) As expected, this consistently leads to performance degradation for different VLMs and backbones, demonstrating that modality inversion does not inherently improve performance. Similarly, we perform an exper- iment where approach the zero-shot classification task intra-modally by applying OVI to each textual prompt, with results reported in Tab. A3, Consistent with our previous findings, we observe that approaching classification intra-modally hinders the performance.
18

---

Published as a conference paper at ICLR 2025
Table A3: Performance (accuracy) evaluation on the zero-shot image classification task. Purple rows indicate the usage of OVI-inverted features, while white rows refer to the inter-modal baseline. V and X denote inter-modal and intra-modal approaches, respectively.
Flowers
Aireraft
EuroAA
Food101
SUN397
Caltech
UCF101
ImageNet
Average
Inter modal
Caus
Pets
DTD
Backbone
60.4
87.5
67.0
19.1
43.6
45.2
80.5
62.0
91.2
62.0
62.1
61.9
B/32
CLIP
37.4
59.9
35.0
9.2
26.2
18.9
65.1
44.1
83.9
51.2
42.9
43.1
76.8
93.6
79.3
32.5
53.0
58.1
91.0
67.6
94.9
74.2
73.5
72.2
L/14
46.9
71.1
65.1
23.3
41.4
23.8
73.6
46.2
41.6
63.5
54.8
50.1
88.4
90.3
73.5
24,4
53.9
56.5
83.0
67.0
96.2
61.6
68.6
69.4
B/32
OPEN
81.4
82.1
62.4
17.9
45.8
36.6
56.9
93.6
55.1
59.6
60.7
93.7
95.0
82.5
47.6
62.7
68.0
92.3
74.2
97.6
75.0
78.9
78.9
1/14
X
78.6
85.3
71.1
35.9
48.6
47.9
86.2
50.7
92.9
62.4
67.3
66.1
Sig
LIP
90.7
94.1
85,8
43.9
62.0
42.3
89.2
69.6
97.4
74.9
75.7
75.1
B/16
X
67.2
68.9
32.6
23.5
40.5
14.2
59.6
27.8
35.1
21.0
22.1
37.5
Table A4: Performance evaluation on the image-to-text and on the text-to-image retrieval task. Blue rows and Purple rows indicate the usage of OTI- and OVI-inverted features, respectively. White rows refer to the inter-modal baselines. v and X denote inter-modal and intra-modal approaches, respectively.
Imago-to-Text
Text-toImmage
Flick330k
COCO
Flickr30k
COCO
Inter modal
Backbone
R@1
R@s
R@ 10
R@)
R@5
R 10
R@I
R@5
R@ 10
R@1
R@5
R@ 10
78.8
94.9
98.2
50.1
75.0
83.5
58.8
83.5
90.0
30.5
56.0
66.9
B/32
X
64.5
74.6
25.6
85.2
97.4
99.2
56.3
79.3
86.6
64.9
87.3
92.0
36.5
61.0
71.1
X
75.8
92.9
95.9
49.0
72.8
81.2
60.7
84.8
90.3
33.2
55.1
67.7
OPEN
B/32
x
79.2
72.8
90.3
93.8
96.2
94.1
$3.5
49.2
77.7
73.4
86.0
82.0
57.4
61.1
81.5
84.9
90.9
88.4
37.1
33.1
58.0
62.4
72.7
68.4
<
89.1
98.6
99.7
63.3
84,2
90.4
73.4
91.8
95.5
45.7
70.1
79.2
1./1
X
86.0
97.7
98.9
60.8
81.5
88.3
67.4
88.1
93.0
39.0
63.4
73.2
LIP
B/16
89.0
98.0
99.2
65.7
85.4
91.2
74.6
92.3
95.6
47.8
72.4
81.0
X
81.8
95.5
97.3
57.0
79.0
86.2
57.9
82.6
88.7
33.7
58.2
68.9
Text Retrieval with OTI and OVI. To provide additional experimental evidence that trans- forming inter-modal tasks in intra-modal ones hinders performance, we conduct an experiment on image-text retrieval using the COCO (Lin et al. 2014) and Flickr30K (Plummer et al. 2015) datasets. We use the Karpathy splits Karpathy & Fei-Fei (2015) for both datasets and report results on the test split. Following standard the evaluation benchmark, we report Recall@ K scores with K = 1, and 10. Specifically, image-to-text retrieval, we apply OTI to the query image, while in text-to-image retrieval, apply OVI to the query text. We then compare the results with the inter-modal baseline, which directly compares image and text features. Results in Tab, [A4 confirm findings from the zero-shot image classification task: transforming an inter-modal task into an intra-modal one leads to performance degradation due to intra-modal misalignment.
Text-to-text Retrieval Purely Textual Datasets. In Sec. 5.2 conduct text-to-text retrieval experiment using image captioning datasets to avoid a mismatch with VLMs pre-training data. In this section, evaluate the performance of OVI on purely textual datasets using the CLIP ViT B/32 model. Specifically, we select seven datasets from the NanoBEIR benchmark spanning diverse domains such as scientific documents (SciDOCS) and climate-related texts (ClimateFEVER) We discard Question-Answert
(QA) datasets and those with queries whose average length exceeds the context length of CLIP's text encoder (77 tokens). Additionally, we include the IMDB Reviews (Maas et al., 2011) and the 20 Newsgroups (Lang) 1995) datasets.
19

---

Published as a conference paper at ICLR 2025
Table A5: Performance (mAP) evaluation on the text-to-text retrieval task using purely textual datasets. Purple rows indicate the usage of OVI-inverted features. v and X denote inter-modal and intra-modal approaches, respectively.
NFCorpus
IMDB
20News.
Climate
DBPedia
PEVER
SeiDocs
SciFact
Average
Inter
NQ
Method
modal
Baseline
52.2
19.2
11.2
30.3
58.4
8.9
23.3
13.5
26.3
27.0
OVI
52.3
33.1
15.3
39.1
70.5
12.2
33.6
16.8
33.2
34.0
Table A6: Performance (mAP) comparison between the proposed modality inversion techniques and the adapter-based approach the image-to-image (left) and text-to-text (right) retrieval tasks. Blue rows and Purple rows indicate the usage of OTI- and OVI-inverted features, respectively. and X denote inter-modal and intra- intra-modal approaches, respectively.
ROxford
Flickr30k
72Paris
Average
COCO
Average
Inter modal
CUB
SOP
Cars
Inter modal
Method
Method
Baseline Adapter
22.9
34.4
42.6
67.9
24.6
38.5
Baseline Adapter
51.7
26.2
35.1
37.7
23.7
35.0
44.3
69.5
25.5
39.6
51.9
28.3
37.8
39.3
011
24.6
35.1
43.0
70.3
28.0
40.2
OVI
54.8
28.3
38.8
40.6
All selected datasets comprise texts that cannot be easily represented visually. Examples include "Learning Actionable Representations with Coaa-Condi Policies" (SciDocs), "Atheism, phi- losophy, and the absence of belief in deities" (20 Newsgroup), and "The carbon footprint on wind energy is significant" (ClimateFEVER) Since gallery texts often exceed CLIP's context length, we employ the Llama-3.2. 1-21 B.- to fit within the token limit.
Large Language Model (Dubey et al. 2024 to summarize them
We report the results Tab. A5 OVI achieves significant performance improvement over the intra-modal baseline. This outcome demonstrates that OVI is effective even when considering texts that can not be easily represented visually.
From Intra-modal to Inter-modal via Adapters. To broaden comparative analysis we con- duct an additional experiment where we train two single-layer linear adapters: one maps image features to text features (aligned with the goal of OTI), and the other maps text features to image features (aligned with the goal of OVT). For training, we leverage the LLaVA-CC3M dataset (Liu al. 2024), which comprises 595K image-text pairs. This dataset is derived by filtering the CC3M dataset (Sharma et al.) (218)) to achieve a more balanced distribution of concept coverage. We train each adapter using a cosine loss that minimizes the distance between the adapter output and the corresponding complementary features. Additionally, following Patel et al. (2024), we also employ a CLIP-based contrastive loss component.
Table A6 presents the results for image-to-image and text-to-text retrieval tasks using the CLIP ViT-B/32 model. The adapter-based approach improves performance over the intra-modal baseline for both tasks. These findings support our claim that approaching the task inter-modally enhances performance thanks CLIP's inherent inter-modal aligmment. Interestingly, we observe that OTI and OVI outperform the adapter-based approach in most scenarios. This result emphasizes the effectiveness of OTI and OVI, as they do not require a training dataset but rather map individual features directly to the complementary modality without relying external resources.
From Intra-modal Inter-modal via Captioning, We compare the performance of OTI on image- to-image retrieval with captioning- based approach. Specifically, given a query image, first generate the caption using a pre-trained captioning model, then use CLIP's text encoder to extract the text features to perform retrieval.
20

---

Published as a conference paper at ICLR 2025
Table A7: Performance (mAP) comparison between the proposed OTI technique and the captioning- based approach on the image-to-image retrieval task. Blue rows indicate the usage of OTI-inverted features. v and denote inter-modal and intra-modal approaches, respectively.
ROxford
RParis
Average
Inter modal
CUB
SOP
Cars
Method
Baseline
22.9
34,4
42.6
67.9
24.6
38.5
DeCap
4.4
2.0
0.1
1.2
2.5
2.0
CoCa (COCO)
3.5
0.8
0.0
0.7
1.8
1.4
CoCa (LAION)
17.6
3.9
8,4
28.2
23.6
16.3
OTI
24.6
35.1
43.0
70.3
28.0
40.2
We experiment with three pre-trained captioning models: DeCap (Li et al. 2023). which directly generates captions from CLIP image features; CoCa (LAION)) Yu al.), trained the Laion2B Schuhmann et al. 2022) dataset; and CoCa (COCOF pre-trained on Laion2B and fine-tuned on COCO (Lin al. 2014).
Table A7 shows the results using the CLIP VIT- B/32 model. Regardless the captioning model, the captioning-based approaches achieve unsatisfactory performance, even falling short of the intra-moda base- line despite leveraging CLIP's inter-modal alignment. This outcome stems from the fact that the generated captions are not discriminative enough to perform im- age retrieval. This is particularly evident in fine-grat grained domains such the buildings of the ROxford and RParis datasets (Radenović et al., 2018). Figure A2 shows example of generated captions for randomly chosen image from the ROxford dataset. We observe that all the captioning models generate generic and sufficiently discriminative captions. CoCa (LAION)
DeCap:
large building clock tower on the front.
CoCa (COCO): old building
towers has a clock on
CoCa (LAION):
souls college, axford, united kingdom.
Figure A2: Captions generated by pre- trained captioning models for image from the ROxford dataset.
produces more precise description than the other models, reflecting its higher performance.
Intra-OTh OTI Similarity Comparisons. To further support our claim that the performance improve- retrieval stems from CLIP's inter-modal alignment and not from the modal- ity inversion itself, we perform an experiment where we apply OTI to both query and gallery images. Since OTI maps image features into text features, this intra- (TT- strategy involves intra-modal simi- larity comparisons within the text embedding space.
ment in
Table A8 shows results on image retrieval datasets using the CLIP VIT- B/32 model. We observe that employ- ing inter-modal similarity comparisons by applying OTI only to the query images achieves better performance than using intra-modal similarities with the intra-OTT approach. This confirms that modality in- version techniques do not inherently im- prove performance. Instead, their effec- tiveness lies in leveraging CLIP's inter- modal alignment by transforming intra-modal tasks into inter-modal ones.
Table A8: Performance (mAP) evaluation on the image-to-ing to-image retrieval task. v and X denote inter- modal and intra-modal approaches, respectively.
ROxford
RPPris
Average
Inter modal
SOP
Cus
Method
Baseline Intra-OTI
22.9
34.4
42.6
67.9
24.6
38.5
21.3
31.9
42.3
68.2
24.9
37.7
(ours)
43.0
Impact of the OTI Template Sentence. As detailed in Sec. 4.1 for OTI we concatenate the tem- plate sentence "a photo of" with the pseudo-token s² to craft the prompt "a photo of u²" To study the impact of the template sentence the performance, we test the following prompts: 1) "an image of "m 2) "we see 5" in this photo"; and 3) "!""" (the empty prompt).
https://hssginggaccc.cc/aaaaaaco ViT-- 32-oonn
http:://wwwgnnnaacccccco///////a finetuned
21

---

Published as a conference paper at ICLR 2025
Table A9: Impact the performance (mAP) of the OTI template sentence used in image-to to-get retrieval. Each prompt is given by the combination a template sentence with the pseudo-token e".
ROxford
Average
R.Prrs
CUB
S0P
Cars
OTI Prompt
"v" (empty prompt)
24.0
34.6
43.7
69.6
28.2
40.0
"We see v" in this photo"
24.5
34.7
43.0
69.7
28.3
40.0
image of et
24.0
34.8
43.1
70.7
28.3
40.2
"A photo of n. (ours)
24.6
35.1
43.0
70.3
28.0
40.2
Table A9 reports the image-to-image retrieval results using the CLIP ViT-B/32 model. We observe that all the considered prompts achieve comparable performance. These results demonstrate the robustness of the OTI technique to the template sentence.
Combining Native and OTI-Inverted Features. We conduct experiment image-to- mnage-to-image retrieval to assess whether combining native image features with the corresponding OTI-inverted features improves the performance. Let र [o( be the native image features and 1/T 9 "" be the OTI-inverted features. We query the gallery using a weighted combination of native and OTI- inverted representations:
VI = a ur + (1 a) 1/1,
(9)
where a E [0, 1] is a weighting factor that controls the contribution of each component.
Table repprrsss the results image- to-image retrieval datasets for vary- ing values of using the CLIP ViT- B/32 model. Interestingly, for large enough, combining native and inverted features obtains better results than re- lying solely on either of them. No- tably, regardless of the value, we ob- serve that employing OTI-inverted fea- tures always improves the performance over the intra-modal baseline. leave further investigation of the combination of native and OTI-inverted features to future work.
Table A10: Performance (mAP) evaluation of the combi- nation between native and OTI-inverted features for vary- ing weighting factors a for image-to tongg-tt--mage retrieval.
ROxford
RRPriss
Average
CUB
SOP
Cars
Method
Baseline (a -0=00 0)
22.9
34.4
42.6
67.9
24.6
38.5
OTI (a = 0.25)
24.0
35.6
44.9
70.1
25.9
40.1
OTI (a 0.50)
24.6
36.1
46.7
71.0
27.0
41.1
OTI (a 0.75)
24.8
35.9
46.3
71.1
27.7
41.2
OTI (a 1) (ours)
24.6
35.1
43.0
70.3
28.0
40.2
22


Reviews:
[{"review_id": "wt4ng4gsOf", "reviewer": "Reviewer_pj7m", "summary": "This paper investigates the suboptimal performance of using pre-trained multi-modal Vision Language Models (VLMs) like CLIP for intra-modal tasks. It highlights the modality gap, a disparity between text and image feature spaces due to contrastive pre-training, which leads to poor performance in tasks like image-to-image retrieval when using individual text or image encoders. The authors propose a modality inversion technique to transform native modality inputs into inter-modal representations, leveraging CLIP's inter-modal alignment. Extensive experiments demonstrate significant performance improvements over intra-modal baselines in image retrieval, text retrieval, and zero-shot image classification tasks.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a novel modality inversion method that significantly outperforms standard intra-modal approaches by exploiting inter-modal alignments of CLIP models, providing a new direction for enhancing VLMs' utility in single-modal tasks.\n\n-  With experiments spanning multiple datasets and tasks, the paper offers robust evidence of the proposed method's efficacy, showcasing its broad applicability in the field of multi-modal learning.", "weaknesses": "- The contribution is incremental, as the OTI has already been introduced by existing methods.  \n- In zero-shot classification, current methods utilize both image and text encoders concurrently. Therefore, only utilizing the image encoder compromises fairness in the experimental comparisons for this task.  \n- Several relevant comparative methods are omitted. In particular, the proposed method achieves similar effects to prompt learning and adapter learning, which should be discussed and included in further comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0xKNozD8Fv", "reviewer": "Reviewer_U71q", "summary": "The paper points out the problem of  intra-modal misalignment in CLIP, which limits its effectiveness for intra-modal tasks such as image-to-image retrieval. To address this issue, the paper proposes to transform intra-modal tasks to inter-modal ones via Optimization-based Textual Inversion (OTI) and Optimization-based Visual Inversion (OVI), which map representations to the complementary modality. The experiments demonstrate its effectiveness on fifteen datasets.", "questions": "It is interesting to know if it is possible to have a more systematic approach for selecting the number of tokens and optimization steps used in OTI and OVI, as discussed in Figure 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles an intriguing problem, since CLIP encoders are widely used in various tasks and serves as components in other models like LVLM. Addressing the intra-modal misalignment issue could enhance CLIP's performance in image and text understanding.\n2. The presentation quality is high. The paper is well-written, easy to follow, and features clear and informative figures.", "weaknesses": "1. The analysis of the intra-modal misalignment problem could benefit from more quantitative insights. In Figure 1, the authors illustrate that intra-modal similarity scores can be higher for the same class than for different ones. Providing statistics to demonstrate the significance and frequency of this issue would strengthen the argument.\n2. The experiments lack comparative analysis. While the related works section mentions existing works that explore intra-modal misalignment, it would be beneficial to compare these methods in the experimental study to validate the paper's contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kiKUUTJb7b", "reviewer": "Reviewer_ikx3", "summary": "The authors point out in this paper that the common practice of using intra-modal similarities from pre-trained CLIP encoders is inherently suboptimal for intra-modal tasks. To address this, they adapt optimization-based modality inversion techniques (OTI and OVI) and leverage them to transform native modality inputs into intermodal representations for similarity measures. Experiments across three settings—image-to-image retrieval, text-to-text retrieval, and zero-shot image classification—on 15 datasets and with 5 different pre-trained models validate the authors’ hypothesis, demonstrating the effectiveness of optimization-based modality inversion.", "questions": "- Please answer those issues pointed out in the weakness.\n- In Table 1, the use of OTI features outperforms native image features on all datasets except for the EuroSAT dataset. What distinguishes the EuroSAT dataset from the others? Why does it show a performance decrease with OTI features? \t\n- The authors use the template sentence “a photo of” concatenated with the pseudo-word tokens in Section 4.1. Have you tried other templates?\n- It would be better to provide more training details (e.g., training time, memory usage, and training data).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is good and interesting. Using inter-modal CLIP representations for intra-modal similarity measures makes sense. If this has not been done before, it is a novel idea.\n\n- The paper is well-motivated, with comprehensive experiments, promising results, and clear writing.\n\n- The authors validate their hypothesis and demonstrate their method's effectiveness through extensive experiments involving over 15 datasets, 3 different types of CLIP models, and 5 different pre-trained models.", "weaknesses": "- The method is quite simple. OVI is similar to OTI. There is nothing new about the inversion. But applying the inversion in this particular context can be appreciated. The authors might want to tune down the claim on the methodology contribution.\n\n- It is a surprise to see the proposed method also helps with image classification tasks. Why? Image classification is supposed to be an inter-modal task. In the open-vocabulary setting, it is often done by comparing CLIP image features with a set of CLIP text features on the class vocabulary, which is already inter-modal. \n\n- Missing baseline: For the image-to-image retrieval task, the authors convert the query image to a corresponding textual feature using OTI, then calculate semantic similarity between this feature and the other image features (obtained from the CLIP image encoder) to get the final results. An important baseline is missing, i.e., converting all the images into OTI features and then measuring the similarity among OTI features. This can demonstrate your proposed inter-modal CLIP representations are better for intra-modal tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aJfVbuItwa", "reviewer": "Reviewer_BNBn", "summary": "This paper demonstrates that CLIP models, trained exclusively with inter-modal contrastive loss, perform suboptimally on intra-modal tasks like image-to-image and text-to-text retrieval. The authors attribute this limitation to a modality gap and the absence of intra-modal supervision during pre-training. To address this, they employ modality inversion techniques, transforming inputs across modalities to bridge this gap and enhance intra-modal performance. Through extensive experiments, they show that CLIP’s intra-modal performance improves by leveraging the structured nature of inter-modal cosine similarities, which are more consistent than intra-modal ones.", "questions": "1. Does combining the native image representation with its inverted (cross-modal) image representation provide any performance benefits?\n\n2. For text-to-text retrieval, have you tried the method on a purely textual task with less than 77 tokens context? Existing text retrieval datasets could be summarized to 77 tokens using an LLM. My assumption is that this approach would only be effective when the text can be represented visually, which might explain its success with text derived from image-captioning datasets.\n\n3. Have you considered alternative baselines for image-to-image retrieval, such as using a captioning model?\n\n4. Do you have any intuitions on why the image-image retrieval score for EuroSat dataset goes down with modality inversion?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Well written, easy to follow.\nExtensive experiments and analysis.", "weaknesses": "Lack. of alternative baselines, consider evaluating alternative baselines, such as using an image-captioning model to generate captions for query images, followed by image-to-image retrieval based on these captions. This could provide a comparative perspective on the effectiveness of the proposed modality inversion techniques.\n\nMinor: Are the ticks and crosses correct in Table 2. Right. ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)