Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions
Authors: Jihoon Kwon
Kyle Min
Jy-yong Sohn
Venue: NeurIPS 2025 poster ()
Keywords: Compositional Reasoning
Vision-Language Models
Contrastive Learning
CLIP
Primary area: deep_learning

Abstract: Despite recent advances, vision-language models trained with standard contrastive objectives still struggle with compositional reasoning -- the ability to understand structured relationships between visual and linguistic elements.
This shortcoming is largely due to the tendency of the text encoder to focus on individual words rather than their relations, a limitation reinforced by contrastive training that primarily aligns words with visual objects.
In this paper, we introduce REconstruction and Alignment of text Descriptions (READ), a fine-tuning method designed to enhance compositional reasoning by adding two auxiliary objectives to the contrastive learning: (1) a token-level reconstruction objective, where a frozen pre-trained decoder reconstructs paraphrased captions based on the embedding of the original caption; and (2) a sentence-level alignment objective, which explicitly aligns paraphrased sentences in the embedding space.
We show that READ-CLIP, a model derived by applying the READ method to the pre-trained CLIP model, achieves the state-of-the-art performance across five major compositional reasoning benchmarks, outperforming the strongest conventional fine-tuning baseline by up to 4.1%.
Furthermore, applying READ to existing CLIP variants (including NegCLIP and FSC-CLIP) also improves performance on these benchmarks.
Quantitative and qualitative analyses reveal that our proposed objectives -- reconstruction and alignment -- offer complementary benefits: the former encourages the encoder to capture relationships between words within a caption, while the latter ensures consistent representations for paraphrases expressed with different wording.

Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions
Jihoon Kwon Seoul National University kog0712esnu, ac.kr kr
Kyle Min" Oracle kyle www.mmiinnce...mmmmmmmm com
Jy-yong Sohn Yonsei University jysohn1108@yonsei.. ac. kr
Abstract
Despite recent advances, vision-language models trained with standard contrastive objectives still struggle with compositional reasoning the ability to understand structured relationships between visual and linguistic elements. This shortcoming is largely due to the tendency of the text encoder to focus on individual words rather than their relations, limitation reinforced by contrastive training that primarily aligns words with visual objects. In this paper, we introduce REconstruction and Aligument of text Descriptions (READ), fine-tuning method designed enhance compositional reasoning by adding two auxiliary objectives to the contrastive learning: (I) token-level reconstruction objective, where frozen pre-trained decoder reconstructs alternative captions based on the embedding of the original caption: and (2) sentence- level aligument objective, which explicitly aligns para- phrased sentences in the embedding space. We show that READ-CLIP a model derived by applying the READ method the pre- preitreined CLIP model, achieves the state-of-the thrt performance across five major compositional reasoning bench- marks, outperforming the strongest conventional fine-tuning baseline by up to 4. Furthermore, applying the READ to existing CLIIP variants (including NegCI.IP and FSC-CLIP) also improves performance on these benchmarks. Quantitative and qualitative analyses reveal that proposed objectives reconstruction and alignment offer complementary benefits: the former encourages the encoder capture relationships between words within caption, while the latter ensures consistent representations for paraphrases expressed with different wording.
Introduction
Recent advances in Vision-Language Models (VLMs) have significantly enhanced the ability to align images with text descriptions 88 677. A key driver of this progress is contrastive pre- vee-training, such as CLIP [45), which learns to embed images and texts into shared multi-modal space, in way that the distance in the embedding space represents the semantic similarity of image-text pairs. VLMs trained with this standard contrastive objective have been widely applied to diverse downstream tasks, including poccbbccayy object detection [ 722 semantic segmentation [5] 30 34), cross-modal
retrieval [12 36 71, and multi-modal generation
53)
Despite their remarkable progress, current VLMs still face challenges with compositional reasoning the ability understand structured relationships between visual and linguistic elements (37 56 64
Numerous studies have shown that VL.Ms commonly fail on even simple compositional tasks that humans find straightforward a horse eating grass, VLMs often assign higher similarity score to the incorrect caption "the grass is cating the horse" than to the correct caption "the horse is eating the grass", highlighting the limitation of the VLMs in capturing syntactic and relational structures [64] These failures underscore the need for further research on compositional reasoning achieve reliable and robust vision-languaa language
19 24 28
42
58 66. 69). For instance, when given an image of
understanding in real-world applications [4] 29 33 39 56 57 60
Work partially done while Intel Labs. 'Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Paraphrased cuption #ll: long table with... long aole has
CLIP
CREPE
WhatsUp
(1) Token-level Reconstruction Loss
43.9
NegCLIP
41.5
-1.3
CLPCLIP
FSC CLIP
Decoder
READ-CLIP
(Ours)
Sentence-Revel Alignment Loss
Imuge-Texx Text Contrastive Loss
76.2
87.0
VALSE 760
SugarCrepe
CLIP Text Encodde
Image Encoden
66.2
Paraphrazed caption 82: "A table is compleme -nted by wooden
Original caption:
"A long sable with...
CreprCrepo++ (ITT)
SugarCrepot+ (TOT)
Figure Overview of the training objectives used in READ-CLIP (left), VLM that applies REconstruction and Aligument tett Descriptions (READ) method to the pretrained CLIP model 45 READ our proposed fine-tuning method that enhances compositional reasoning in VLMs by augmenting contrastive learning with two auxiliary objectives. The auxiliary objectives consist of two components: token-level reconstruction and sentence-level alignment. The performance of READ-CLIP (right) on compositional reasoning benchmarks demonstrates that it consistently outperforms conventional state-of -fe-----tthhttt--- methods across diverse aspects of compositional reasoning.
Prior works have identified the text encoder primary bottleneck for compositional reasoning in VLMs trained with the contrastive objective. Specifically, the text encoder often fails to capture the relationship between words in sentence [10 23 65 This limitation is largely attributed to the contrastive objective, which trains the text encoder to align caption with its corresponding image, without encouraging the encoder to capture the relationships between words [0 64] As result, the text encoder focuses on words that refer to objects depicted in the image, as they mainly contribute to the image-text alignment, thereby limiting its ability to learn compositional reasoning (7).
Conventional approaches to overcome this limitation can be categorized into two parts. The first approach is to modify the contrastive objective by introducing hard negatives [48 -semantically different examples that are nonetheless difficult for the model to distinguish from the positives. Here, the model is trained to bring each sample closer to its positives and push it farther from hard negatives, thus improving the compositional reasoning capability [41] 50 54 64). However, these approaches that rely solely on hard negatives often encourage the model to focus patterns specific to those negatives, rather than developing genuine compositional reasoning B 14] 19].
The second approach is add auxiliary objectives the standard contrastive objective. However, these efforts either supervise both image and text encoders jointly [51] 770 or focus solely on the image encoder 22] 18 Although the text encoder serves primary bottleneck for compositional reasoning, limited attention has been paid to adopting auxiliary objectives for the text encoder, aimed
improving the compositional reasoning capability.
In order to enhance the compositional reasoning capability of VLMs more effectively, we tackle the underexplored challenge of improving the text encoder via targeted training objective. Specifically, our contributions are as follows:
In Sec. 3] we propose REconstruction and Alignment of text Descriptions (READ), fine-tuning method designed to enhance the text encoder by adding two auxiliary objectives to the standard contrastive objective: (I) token-level reconstruction and (2) sentence-level alignment, as in Fig. First, the token-level reconstruction objective trains the text encoder to produce embeddings from the original caption that enable frozen decoder to reconstruct each token of an alternative caption. Second, the sentence-leve alignment objective explicitly aligns paraphrased captions in the embedding space to reflect their shared semantics, even when they are expressed differently.
In Sec. [ we provide experiments demonstrating the READ method is effective across wide range of compositional reasoning benchmarks. Specifically, we introduce READ-CLIP. a VLM derived by applying the READ method to the pre-trained CLIP model (45), which achieves the state-of-the-art performance on five compositional reasoning benchmarks. READ-CLIP outperforms the famous baseline NeIC 64 by an average of 4.5% across benchmarks, and outperforms the strongest baseline FSC-CLIP (38 by up to 4.1%. Furthermore, applying

---

the READ method to existing CLIP variants (including NegCLIP and FSC-CLIP) consistently improves performance across these benchmarks, with gains of up to 2.4%.
Our analysis in Sec. 5 demonstrates that the two objectives in the READ method reconstruction and alignment provide complementary benefits for compositional reasoning. The former encourages the encoder to capture relationships between words within a caption, while the latter ensures consistent representations for paraphrases even expressed with different wording. We also find that reconstructing an alternative caption, rather than the original caption, reduces overfitting to exact wording and improves the ability of VLMs learn relational understanding.
2 Related Work
Compositional Reasoning in Contrastive VLMs. VLMs trained with the contrastive objective often struggle with compositional reasoning [36, 64]. as the text encoder tends to overlook relationships between words due to the training objective that prioritizes image-text alignment based on object
mentions 7 10 23 64 [5]. To address this limitation, common approach is to introduce hard
negatives by modifying the contrastive objective. These approaches typically generate hard negative
captions via rule-based perturbation [6 64], language models (7 688. scene graphs (20) [8] [54, or construct hard negative pairs by altering both text and image [1] These methods have been shown to be effective; for example, NegCLIP improves over CLIP by 23.4% on ARO 64] and
CE-CLIP (68] achieves a 7.2% gain on VALSE [40] [68] Among these, DAC [6] highlights that training with well-aligned captions improves compositional reasoning, while TSLVC [7] finds that using paraphrased captions in analogy loss improves image classification performance.
Beyond contrastive learning, recent work has proposed adding auxiliary objectives to improve the compositional reasoning. Some methods supervise both image and text encoders, such as SF- CLIP [11, which uses masked distillation from pre-trained models, and IL-CLIP [70], which employs codebook alignment and iterative re-initialization. Other approaches target only the image encoder: SDS-CLIP 2 uses distillation from diffusion models [4)], and CLIP-SGVI [18] introduces scene-graph loss. Although the text encoder has been identified as the primary bottleneck [10] 23
65]. approaches that specifically introduce auxiliary objectives for the text encoder for improved compositional reasoning capability of VLMs remain searce.
For the purpose of language understanding, various recent works have focused on training a text encoder-decoder architecture in way that the sentence put into the encoder is reconstructed at the output of the decoder. It is reported that such reconstruction objective is beneficial for improving the performance of encoders on various For instance, MASS [5 reconstructs masked fragments of the original sentence, while RetroMAE [62 reconstructs the original sentence from pooled embedding. These approaches have shown that the auxiliary reconstruction objective can encourage the text encoder to capture both syntactic and semantic relationships among the words in the sentence (55] 62). However, reconstructing the caption under the encoder-decoder structure has not been explored as a training objective for VLMs. Despite the use of auxiliary objectives in VLMs 31 633, these approaches do not aim to reconstruct the input caption in the text modality. We introduce a text reconstruction loss during mne-tuning, aiming to enhance the compositional reasoning ability of VLMs.
Reconstruction Objectives for Training Encoders.
language understanding benchmarks 26 32 59
3 Method
In this section, we formally define our proposed REconstruction and Alignment of text Descriptions (READ) method for improving the compositional reasoning performance of VLMs. The READ method is fine-tuning method using three types of losses: conventional contrastive loss reviewed in Sec. 3JJ and two auxiliary losses proposed in Sec. 3.2 and Sec. 3.3 namely, the token-level reconstruction loss and the sentence- alignment loss. The final form of the fine-tuning loss in the READ method given in Sec. 3.4
3.1 Contrastive Loss
We consider a batch of B image-text pairs, denoted as T.) where I, and T, represent the i-th image and its associated caption. The image and text encoders are denoted by Jr and JT. which produce embeddings u, = (T.) and v; = [T(T)), respectively. For convenience, we define the index set [B] : (1,2....,1) B}.
3

---

Token-level Reconstruction Loss
(2) Sentence-level Alignment Loss
Paraphrased caption y Mi; "a long table with plant
b ..... "1
%
Text Decoder x²())
Linear Projectur W
Text Fncoder f(())
1/2
Encoder f1(0
"
Original caption 7
caption
Paraphrased 77: table is complemented by unoden chairs.
"A long restaurant table with rounded back chairs."
"A long restaurant table with rattan rounded back chairs."
Figure 2: Illustration of our proposed auxiliary objectives of REconstruction and Alignment of text Description (READ) method. Given pairs of captions an original and its paraphrase that share common meaning, the token-level reconstruction (1) trains the text encoder produce embeddings from the original caption such that frozen pre-trained decoder can reconstruct each token of the paraphrased caption. This reconstruction encourages the encoder to capture relationships between words within caption, which are critical for reconstructing its paraphrase. In contrast, the sentence- level alignment (2) aligns the pair of captions the embedding space. This alignment encourages the encoder to capture underlying semantie relationships across the paraphrased captions.
Suppose we are given a batch image-text pairs {4, Ti) For each i, E [B]. the similarity between the i-th image I, with embedding u, and the j-th text Tÿ with embedding by is defined as (I;,T)) == exp (cos (u,, 0z)] /T) where T is learnable temperature parameter. The standard
contrastive losses used in CLIP 45 is represented
L = 2
(CI-T + LT-1),
(1)
where each component is defined as
L
Σ log B
1
644, Ti)
CT
log 1 (T;, 1) B
Σt 61,, T2)*
E
(T, 1
which are dubbed as image-to-text loss and the text-to-image loss, respectively, Note that for the standard loss in Eq. each image I, has one positive caption T; and B negative captions {T} {Tihjzi-
The READ method uses variant of the standard contrastive loss in Eq. to improve the compositional reasoning capability, motivated by the following two observations. First, prior works [0] 23 65
have identified the text encoder as a primary bottleneck for compositional reasoning in VLMs trained with the contrastive objective. Second, recent works [41
54 64 showed that compositional reasoning of CLIP model is improved by introducing additional hard negative captions semantically different captions that are nonetheless difficult for the model to distinguish from the positives - into the training loss. Motivated by these observations, the contrastive loss our proposed READ method uses hard negatives in the text domain, details of which are given as below.
For each sample index i, let (T Jm=1 be the set of M hard negative captions associated with the positive caption T; Incorporating these hard negatives into the denominator, the image-to-text loss in Eq. 2is is modified as
1
Σ log
1(I, Tτ)
B
Io,, T,) Em 044, T
Inserting this modified image-to-text loss Eq. I the contrastive loss in the READ method
CContrastive
2 (Ci-r+ + CT-+)).
3.2 Token-Level Reconstruction Loss
Here we define our proposed token-level reconstruction loss and discuss how this loss promotes compositional reasoning. Given the i-th image- text pair (4, T,), consider set of K captions

---

{y; =k=1 describing the same image I, which serve as target sequences for the reconstruction. We refer to this set as the alternative captions of 4, As shown in the left-hand side of Fig. 2 the token- level reconstruction loss measures how well each token in the alternative caption y is reconstructed at the output of the text decoder, once the original caption T; given as the input of the text encoder.
To be specific, let U; = (T;) be the text embedding of the original caption T; for the i-th sample. In order reconstruct texts from the embedding vy, we employ a pre-trained frozen decoder и, While the text encoder fr is initialized from pre-trained text encoder. one can use any off-the-shelf pre-trained text decoder for æ. Thus, the encoder and the decoder may have different embedding dimensions. To address this potential difference, we introduce a learnable projector W that maps the encoder output to the decoder input. Specifically, given the text embedding vs we apply a linear projection to obtain h2 = WT,, which is used as the input to the decoder π. Then, for each & E [K], the decoder predicts the k-th alternative caption yh , ViLL composed of tokens, conditioned on the projected embedding h;. Thus, the token-level reconstruction loss is defined
B K
Token Reconstruction
log (y!*)
(k)
h.).
(5)
B K
where the log-likelihood value is
log ग
(1)
14:)
Σ 1=1
log क
(k) Vist
Milcer Ms)
(6)
Now we discuss how the proposed loss is beneficial for improving the compositional reasoning performance. Since the decoder is frozen and conditioned solely on the text embedding U, the proposed reconstruction loss trains the text encoder fr to embed a caption T, such that a frozen pre-trained decoder reconstruct the tokens in the alternative caption. The encoder fr is thus encouraged to capture the relationships between words within a caption, as these are necessary for reconstructing its alternatives, which promotes the compositional reasoning.
3.3
Sentence-Level Alignment Loss
Effective compositional reasoning requires not only understanding the relationships between words within a sentence, but also recognizing semantic similarity even when sentences convey the same meaning using different expressions. To this end, as shown the right- side of Fig. 2 we additionally employ a sentence-level alignment loss to explicitly align text embeddings of paraphrased captions that describe the same image. For each image-text pair (J,, T.). we generate a paraphrase TV of T; through augmentation, forming a text pair (T; T/). Here, the pair (T; T() is treated as positive, while paraphrases (T,, TT) from other samples the batch serve as negatives. The sentence-level alignment loss is then defined as
B log
(Ti, TD 1コ= (T,, TT)
Leentence Alignment
B
where á is the similarity metric defined in Sec. 3.1 with slight abuse of notation This alignment loss encourages the encoder to embed paraphrased captions (T;, Ti) close together in the embedding space, thus letting the encoder capture the semantic relationships between sentences that express the same meaning using different wording and phrasing.
3.4 Fine- Tane-TTningg Loss of The READ Method
The fine-tuning loss used for the READ method combines the above components:
BREAD = Lcentrattive + a Lyoken Reconstruction + 3 Sentenne Aligument:
(8)
where and 3 are hyperparameters controlling the relative contribution of the auxiliary losses. Together, these losses operate in complementary way by capturing relational structure at different levels: the token-level reconstruction loss captures relationships between words within a sentence, while the sentence-level alignment loss captures semantic similarity across paraphrased sentences.
The definition of é in Sec. 3.1 compares image and a sentence, while here compare sentences
5

---

Table I: Compositional reasoning performance (%) of the pre-trained CLIP model (ViT--/32, top row) and its fine-tuned variants (rows 2-7) across five major benchmarks. All models are fine-tuned on 100K samples from the MS-COCO dataset [35]. Among various fine-tuning methods, READ- CLIP achieves the highest average accuracy of 64.1%.
WhatsUp
VALSE
CREPE
SugarCrepe
SugarCrepe++
Avg.
Models
ITT
TOT
CLIP 45 (VIT-B/32)
41.0
67.4
23.9
73.2
60.0
46.7
52.0
Fine-tumed:
MS-COCO. 100K Samples
Triplet-CLIP
41.6
64.2
15.0
82.7
61.7
57.4
53.8
GNM-CLIP [50]
41.6
70.7
17.4
77.9
60.2
60.0
54.6
CE-CLIP
68
40.7
76.0
34.8
86.0
55.7
57.0
58.4
NegCLIP
64
42.4
73.7
30.5
83.6
65.0
62.5
59.6
FSC-CLIP
38
39,8
74.4
42.5
85.2
67.9
64.4
62,4
READ-CLIP (Ours)
43.9
76.2
41.5
87.0
69.8
66.2
64.1
4 Experiments
In this section, we empirically evaluate the effectiveness of our proposed READ method. To be specific, we fine-tune the pre-trained CLIP model using the READ method, where the fine- tuned model dubbed as READ-CLIP. We compare READ-CLIP with various baselines on major compositionnl reasoning benchmarks. We begin in Sec. 4.1 by describing the experimental setup and present the experimental results in Sec. 4.2 Codes are available this GitHub repository
4.1 Experimental Setup
Training: We use the MS-COCO dataset 335 for all experiments. We follow training practices
established in prior work compositional reasoning 38
54 64 68), using 100K subsample with
the Karpathy split [25). 5 training epochs, batch size of 256, and the ViT-B/32 architecture. As defined in Eq. 8 our training loss consists of three components: the standard contrastive loss, the token-level reconstruction loss, and the sentence-level alignment loss. We provide implementation details for each component, including hyperparameters, and other specifics, in Appendix A.
We compare HEAD-CLIP against recent sfate-of-the-art finettuing methods designed to improve compositional reasoning in VLMs. To evaluate against method relying solely hard negatives, we include NegCLIP (64, which uses rule-based negatives. To compare with methods that construct synthetic image-text negative pairs, include GNM-CLIP (50) and Triplet-CLIP [41).
Baselines:
We also consider methods that improve the effectiveness of hard negative captions by incorporating multiple contrastive objectives, including CE-CLIP
68 and FSC-CLIP 38
Evaluation: We evaluate READ-CLIP and the baselines on five benchmarks (24).
CREPE (37), VALSE [40], SugarCrepe [19], and SugarCrepe++ [[]]-each designed to assess different aspect of compositional reasoning. All benchmarks are evaluated using accuracy, which measures whether positive pairs ranked above all negatives. For each benchmark containing multiple subtasks, we report the accuracy averaged over the subtasks, following prior work [38 68
Details of each benchmark provided in Appendix A.2
4.2 Results
READ-CLIP outperforms baselines various compositional benchmarks. Table reports compositional reasoning performance of READ-CLIP and baselines across five benchmarks. READ- CLIP achieves the highest average accuracy of 64.1%, outperforming the pre-trained CLIP by 12.1% 12.. NegCLIP-a strong and widely cited baseline-by by 4.5%, and the second-best model FSC-CLIP by 1.7%. Notably, READ-CLIP ranks first on four benchmarks and second on the remaining one, demonstrating consistently strong performance. This consistent outperformance highlights the advantage of enhancing compositional reasoning in the text encoder through the READ method.
Reconstruction and alignment losses provide complementary benefits. We conduct an ablation study analyze the benefits of the auxiliary losses introduced in the READ method: token-level reconstruction and sentence-level alignment, summarized Table Compared to the contrastive- only baseline row 1, adding token-level reconstruction in improves average accuracy from

---

Table Ablation study analyzing how the reconstruction and alignment objectives in our proposed READ method contribute to its performance of READ-CLIP, both individually and jointly. The reconstruction loss improves accuracy on WhatsUp, CREPE, VALSE, and SugarCrepe. The inclusion of the alignment loss improves SugarCrepe++. The combination of both losses results in the highest overall accuracy, indicating their combined contributions to compositional reasoning.
UpatsUp VAL.SE CREPE SugarCrepe SugarCrepet+ Avg.
Reconstruction
Alignment Loss
ITT
TOT
Loss
(1)
40.5
74.7
38.4
86.4
69.3
66.0
62.2
(2)
43.6
76.6
41.6
86.9
69.7
64.8
63.9
(3)
V
43,0
75.5
40.6
86.8
70.2
67.0
63.8
(4)
V
43.9
76.2
41.5
87.0
69.8
66.2
64.1
Table 3: Analysis of the impact of key hyperparameter selection on the average accuracy (%) of our proposed READ method: weights for the token reconstruction loss (a) and sentence alignment loss (8), the number of target sequences (K) used in the token reconstruction loss, and the size of the T5 [46) decoder model employed for computing reconstruction loss. Gray cells indicate the hyperparameter configuration that yields the highest average accuracy.
a
B (Sentence Align. Loss)
K (Num. of Targets)
Decoder Model (T5 46
(Token Reconst. Loss)
Value
Avg. Acc.
Value
Avg. Acc.
Value
Avg. Acc.
Size
Avg. Acc.
63.8
63.9
63.8
63.8
0.1
64.1
0.1
63.4
64.1
Small
64.0
0.2
64.0
0.2
64.0
63.5
Base
64.1
63.4
0.5
64.1
63.9
Large
64.1
1.0
62.7
1.0
64.0
64.0
XL
63.4
2.0
61.8
2.0
63.8
5
63.7
XXL
63.9
62.2% to 63.9%, notably enhancing WhatsUp by 3.1%, VALSE by 1.9%, and CREPE by 3.2%, while
adding sentence-level alignment row substantially enhances SugarCrepe++ ITT to 70.2% and TOT to 67. 67.0%. Combining both losses in row 4 achieves the highest average accuracy of 64.1% confirming that each objective offers complementary benefit.
READ-CLIP is robust to hyperparameter selection. To assess the robustness of READ, we analyze its sensitivity four major hyperparameters used for training READ-CLIP: the weight and / for the auxiliary objectives in the READ method, the number K of target sequences, and the size of the T5 decoder [46] used for reconstruction. Table 3 summarizes the average accuracy across the five benchmarks for each configuration. The performance of READ-CLIP remains stable over wide range of hyperparameter values. In particular, varying the loss weights a and / results in only modest performance differences. Also, it achieves strong performance even with single target sequence (K 1) and T5-Large decoder, demonstrating robust gains with minimal computational overhead.
READ provides consistent gains when applied to diverse fine-tuning methods. The experiments so far applied READ to modified CL.PP objective (Eq. 4 that incorporates hard negative captions only. To establish broader applicability, it is crucial to verify whether the READ method consistently provides gains when applied on top of diverse einettunig methods. Therefore, we evaluate READ alongside three baselines: (I) naive CLIP with standard contrastive loss (Eq. I (2) NegCLIP 41). and (3) FSC-CLIP [8), without additional hyperparameter tuning. Table 4 presents the results of applying the READ to these three fine-tume tuning baselines, illustrating its impact across diverse fine- tuning methods. Across all three settings, augmenting the baseline with READ leads to consistent performance improvements on the majority of benchmarks. When applied to naive CLIP, READ improves the average accuracy from 57.0% to 58.2%, with consistent gains across all five benchmarks. For Ne and FSC-CLIP, the average accuracy increases from 61.5% to 62.5% and from 62.8% to 63.6%, respectively. In both cases, READ leads to clear improvements on WhatsUp, VALSE, CREPE, and SugarCrepe, while maintaining comparable performance on SugarCrepe++.
7

---

Table Performance comparison of baseline finettuning methods CLIP [45]. NegCLIP 64 and FSC-CLIP [38] and their READ -uAD-auumented counterparts, to assess the effectiveness of READ method. READ consistently improves performance on WhatsUp, VALSE, CREPE, and SugarCrepe, while preserving comparable accuracy on SugarCrepe*+.
WhatsUp VALSE CREPE SugarCrepe
SugarChepe++
Avg.
Methods
TOT
Fine-tuned MS-COCO, 100K Samples, 5 epoch
CLIP
41.3
70.4
15.5
81.8
66.2
66.5
57.0
READ
43.3 (+:m)
71.3 (-09)
17.3
83.0 (1..2)
68.2 (-2m)
66.2 (4.)
58.2
NegCLIP
41.3
75.4
34.4
84.5
68.0
65.6
61.5
+ READ
43.7 (+24)
76.5 (-1D)
36.7 (+2.))
85.2 (40.7)
68.1 1 (-0.1)
64.9 (40.)
62.5 (+1.))
FSC-CLIP
41.3
73.9
42.7
85.8
68.1
65.1
62.8
READ
43.2 (1.9))
74.4 (.0.5)
45.1 (22
86.6 (+0.5)
67.1 (4.0)
64.8 (4.3)
63.6 (+0.))
Puslne
whl Revouet
wis
w/
X Negative
Recomss Laty
Brewuu Lows
Brewset
Leirs
Lone
WhatsUp
A lapop lef
amechar
A2
41
A can in finer of
hoadplonnz
23
81
liting arncheir
43
X
headphones
Zehind
02
laptop under
cas haselpores
A lapenn the ritratt srmchair
.44
.14
x can 2a oi
headphotes
24
24
la frewe of pursor diployynng
SugarCrepe
skatebaant
A cat sitting in frost montur luat disphaiing stmmiaatd pisture of anodher oot.
This das riling x behind a tai oa skatchurii
Figure 3: Representative examples illustrating how applying the reconstruction loss affects caption rankings based on image-caption cosine similarity on WhatsUp (24) and SugarCrepe [9]. Positive (V) and negative captions (X) are shown with their rankings based image-caption cosine similarity.
Analysis
Our results in Sec. 4 show that the reconstruction and alignment losses the READ method consistently improve compositional reasoning, with each offering complementary benefits. To better understand the role of our proposed loss, we present a qualitative analysis Sec.p.Iam and Sec. (.2 We then quantitatively analyze the key components of each loss. In Sec. 5.3 we examine the benefit of reconstructing an alternative caption instead of the original one within the token-level reconstruction loss. Subsequently, Sec. 5.4 investiigs how the quality and diversity of LL L.MM-generated paraphrases used the sentence-level alignment loss affect compositional reasoning.
5.1 Toven-Level Reconstruction Enhances Encoding of Compositional Relationships
Fig. illustrates the effect observed in Table 2 (rows vs. 2), showing that incorporating the recon- struction loss improves compositional reasoning across benchmarks. Specifically, observe that the reconstruction loss helps lower the ranking of negative captions that differ from positive ones. These negative captions typically involve subtle structural edits-such as swapping. replacing, or inserting single words or short phrases-that preserve most of the original wording while altering the underly- ing meaning. This improved discrimination between correct captions and their negative counterparts suggests that the reconstruction loss enables the encoder to recognize semantic differences between those captions by capturing the relationships between words.
5.2 Sentence-Level Alignment Promotes Semantic Consistency
Fig. 4 iillustrate the effect observed Table vs. 3), showing how the alignment loss enhances compositional reasoning by affecting the ranking of two positive captions in SugarCrepe++ [1]]

---

Nepatiee
black pour monding aets white goats
A persse svey ainng woodrr
REPE ACC
SWAP
bick poat postioond oest to whtte goors
22
A pense wenning grey alirt and hat miiingg on a veooien bench.
#1
A2
Two black posts standing rest to shile
J3
A person pirkk shin and X viling wookles
bench.
Figure 4: Representative examples from each category (SWAP and REPLACE) of SugarCrepe+* dataset, showing how applying the alignment loss improves the ranking of positive captions. In Sugarcrepe++, each image is paired with two positive captions (V) that worded differently. In ITT (image-to-text) evaluation, a prediction considered accurate both positive captions are ranked higher than all negatives based image-caption cosine similarity.
Average
Original Paraghruse Reconst.
Averag Similarity
Origiral Record. Pos Pio
Parppecno Recons Pos Pos
Ascuracy
Pos 1ogg-Nog
66.5
67.1
66.5
66.0)
66.2
1.0
2-Nig
54.2
47.0
0.95
41.7
39.1
38.6
0.85
8
16
20
spoch
0.8
12
16
20
Figure 5: Comparison of reconstructing a paraphrased caption versus the original one, measured by the performance of trained encoder SugarCrepe*+ [11] TOT (text-to-text) benchmark, for various training epochs. In the TOT evaluation, average accuracy (left) is computed by checking whether the cosine similarity between the positive caption pair is higher than that of any positive- negative pairs. Average similarity (right) measures cosine similarity between caption pairs. Here, Pos1 and Pos2 positive pairs, while others negative pairs.
where each image paired with two paraphrases (denoted as Pos1 and Pos2) that convey the same meaning but differ in expression. We observe that applying the aligmment loss leads to improved ranking for some positive captions that were previously ranked below negatives. This effect arises because the alignment loss encourages the encoder to embed paraphrased captions closer together in the embedding space, despite differences in wording. Consequently, both captions become embedded closer the corresponding image than to negative caption, which in turn strengthens the ability in vision-language compositional reasoning.
5.3 Reconstructing Alternative Captions Mitigates Overfitting and Enhances Generalization
Recall that the reconstruction loss in Sec. 3.2 is designed to reconstruct an alternative caption rather than the original one. here, we explore the effect of using such alternative caption on the compositional reasoning performance of the trained encoder. To be specific, we conduct experiments on the SugarCrepe++ TOT (text-to-text) benchmark, comparing two variants of READ-CLIP: one that reconstructs the original caption and the other that reconstructs alternative caption.
The left plot in Fig. 5 shows that reconstructing alternative caption leads significantly more stable accuracy across training epochs, whereas using the original caption results gradual performance decline. To better understand this effect, in the right plot of Fig. 5 we compare the similarity of positive/negative caption pairs in the trained embedding space. We compare three pairs, where Pos1- Pos2 indicates the positive pair, and other pairs are negative. One can confirm that reconstructing the original caption causes steady decline in similarity between Pos1 and Pos2 over time, which undesired. This phenomenon can be interpreted as follows: when trained exactly reconstruct the original caption, the encoder increasingly overfits to exact wording and phrasing rather than capturing the underlying relationships between words within a caption. contrast, reconstructing an alternative caption better preserves semantic similarity between positive captions and maintains greater discrimination from negatives. These findings indicate that reconstructing an alternative caption mitigates overfitting to exact wording and enhances generalization in relational understanding.

---

Table Compositional reasoning performance under varying quality LLM-ggenrrted paraphrases in Eq. ( We randomly replaced 10% or 20% of -LM-ggnerated paraphrases with unrelated captions from the dataset to simulate lower-quality paraphrases during training.
WhatsUp VALSE CREPE SugarCrepe SugarCrepe++ Avg.
Model
ITT
TOT
READ-CLIP
43.9
76.2
41.5
87.0
69.8
66.2
64.1
READ-CLIP (10% Noise)
43.6
76.0
39.0
86.9
67.1
64.7
62.9
READ-CLIP (20% Noise)
43.4
76.0
38.6
86.8
65.1
62.9
62.1
Table 6: Compositional reasoning performance under varying diversity of LLM-generated paraphrases in Eq. 7 We generated multiple paraphrases per caption and randomly sampled one during each training step examine whether increased diversity improves robustness.
WhatsUp VALSE CREPE SugarCrepe SugarCrepe++ Avg.
Model
ITT
TOT
READ-CLIP (num,
43.9
76.2
41.5
87.0
69.8
66.2
64.1
READ-CLIP (nump 3)
43.4
76.4
41.1
86.5
70.0
66.4
64.0
READ-CLIP (num, 5)
43.6
76.0
41.3
86.5
70.8
66.6
64.1
5.4 Sentence-Level Alignment is Robust to Quality and Diversity of Paraphrases
Recall that the sentence-level alignment loss in Sec. 3.3 uses LLM-geeerated paraphrases encourage semantic consistency between captions with different wordings. In here, we investigate how the quality and diversity of these paraphrases affect compositional reasoning performance.
First, we assess whether the performance of compositional reasoning of our proposed READ method is sensitive to lower-quality paraphrases. To this end, we intentionally inject noise by randomly replacing --geerrrete paraphrases with unrelated captions from the dataset two levels: 10% and 20%. As shown in T5 performance drops by only 1.2-2.0% on average when 10-20% of paraphrases are replaced with noise, indicating that the sentence-level alignment loss reasonably robust moderate degradation in paraphrase quality.
Next, we examine whether increasing diversity of LLM-generated paraphrases improves the perfor- mance of compositional reasoning. We vary the number of LLM- L-M-enerrtted paraphrases per caption (nump E f1. 3, 53) and randomly sample one at each training step. All other components in Eq. remain unchanged, thereby the only difference is the diversity of available paraphrases. Table that while increasing num, slightly improves performance on SugarCrepe++ [1 where recognizing paraphrased captions as semantically equivalent is critical - the average performance across all bench- marks remains nearly unchanged. This suggests that a single number of L.IM- Lfnngertteee paraphrase is already sufficient for effective alignment, and additional diversity provides only marginal benefits.
6 Conclusion
We introduced READ, a fine-tuning method that enhances compositional reasoning in contrastively trained VLMs by integrating token-level reconstruction and sentence-level alignment objectives. READ explicitly captures compositional relationships, enabling READ-CLIP to outperform other fine-tuning baselines across diverse benchmarks. We hope this work provides practical approach for compositionality-avaa fine-tuning of VLMs, and encourages further exploration of auxiliary objectives to strengthen the compositional reasoning ability of text encoders.
Limitation. While our method designed to leverage multiple captions per image, can still be applied to the dataset with only single caption per image by generating additional paraphrases using LLMs, although this introduces additional complexity. our reconstruction loss, without exploring the impact of alternative generative architectures [6 44 We did not assess the effect of fine-tuning the decoder as well, which may influence the compositional reasoning capability of our proposed method.
addition, we only used T5 (46) decoder in
10

---

Acknowledgements
This work partially supported by the National Research Foundation of Korea (NRF) grant funded by the Ministry of Science and ICT (MSIT) of the Korean government (RS-2024-00345351, RS- 2024-00408003), and Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by MSIT (RS-2023-00259954,, RS-20255-02283048).
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736. 2022.
[2] Samyadeep Basu, Shell Xu Hu, Maziar Sanjabi, Daniela Massiceti, and Soheil Feizi Distilling knowledge from text-to- sonttto--mme generative models improves visio- cinilling reasoning in clip. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 6105-6115 13, 2024.
[3] Paola Cascante- Bascante-Boonila, Khaled Shchada, James Scale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gul Varol, Aude Oliva, Vicente Ordonez. Rogerio Feris, et al. Going beyond nouns with vision & language models using synthetic data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 20155-20165, 2023.
[4] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision- language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14455-14465, 2024.
[5] Mchdi Cherti and Romain Beaumont. Clip benchmark, May 2025.
[6] Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante- Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and aligned captions (dac) promote compositional reasoning in vl models. Advances Neural Information Processing Systems, 36:76137-761555 2023.
[7] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2657-2668, 2023.
[8] Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. arXiv preprint arXiv:2202. 10936, 2022.
[9] Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang. Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. Aha: A lssio-laaguaag-modde for detecting and reasoning over failures in robotic manipulation. arXiv preprint crXiv:2410.00371, 2024.
[10] Sri Harsha Dumpala, David Arps, Sageev Oore, Laura Kallmeyer, and Hassan Sajjad. Seeing syntax: Uncovering syntactic learning limitations in vision-language models, arXiv preprint arXiv:2412.08111, 2024.
[11] Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, and Hassan Sajjad. Sugarcrepe++ dataset: Fision-languaee model sensitivity semantic and lexical alterations. arXiv preprint arXiv:2406.11177 2024.
[12] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097 2021.

---

[13] Robert Geirhos, Jern-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning deep neural networks. Nature Machine Intelligence, 2(1 2:11):6611111111 ):665-673, 2020.
[14] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 181 12231, 2018.
[15] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmen- tation with image-level labels. In European Conference on Computer Vision, pages 540-557. Springer, 2022.
[16] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
[17] Xiuye Gu, Tsung-Yo Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2 13921, 2021.
[18] Roci Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson. Incorporating structured representations into pretrained vision & language models using scene graphs. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14077- 14098. Singapore, December 2023. Association for Computational Linguistics.
[19] Cheng-Yre Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugar- crepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36, 2024.
[20] Yufeng Huang. Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, et al. Structure-clip: Towards scene graph knowledge enhance multi- modal structured representations. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 2417-2425, 2024.
[21] Drew A Hudson and Christopher D Manning. Gqa: A dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709 2019.
[22] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXivv2410.21227. 2024.
[23] Amita Kamath, Jack Hessel, and Kai-Wei Chang. Text encoders bottleneck compositionality in contrastive languaee models. arXiv preprint arXiv:2305. 14897, 2023.
[24] Amita Kamath, Jack Hessel, and Kei-Wei Chang. What's "up" with vision-language models? investigating their struggle with spatial reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 9161-9175, Singapore, December 2023. Association for Computational Linguistics.
[25] Andrej Karpathy and Li Fei-Fei, Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 3128-3137, 2015.
[26] Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Advances in neural information processing systems, 28, 2015.
[27] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
12

---

[28] Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image retrieval from contextual descriptions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 3426-3440, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[29] Evelina Leivada, Elliot Murphy, and Gary Marcus. Dall- e 2 fails to reliably capture common syntactic processes. Social Sciences & Humanilies Open, 8(1):100648, 2023.
[30] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation. arXiv preprint arXiv:2201.03554, 2022.
[31] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-languse languge understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[32] Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1-39, 2024.
[33] Linjie Li, Zhe Gan, and Jingjing Liu. A closer look at the robustness of vision-and daiion-and-anguugee pre-trained models. arXiv preprint arXiv:2012.08673 2020.
[34] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061-7070, 2023.
[35] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[36] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputingg 508:293-304 2022.
[37] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910-10921, 2023.
[38] Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, and Junmo Kim. Preserving multi-modal capabilities of pre-trained vims for improving lingusiin compositionality. arXiv preprint 2rXiv:2410.052/11 2024.
[39] Maya Okawa, Ekdeep S Lubana, Robert Dick, and Hidenori Tanaka. Compositional abilities emerge multiplicatively: Exploring diffusion models on synthetic task. Advances in Neural Information Processing Systems, 36, 2024.
[40] Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Jacer Calixto, and Albert Gatt. VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8253-8280, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[41] Maitreya Patel, Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, and Yezhou Yang. Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives. Advances in neural information processing systems, 2024.
[42] Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize diagnose and optimize: Towards fine-grained vision-language understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13279-13288, 2024.
13

---

[43] Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. Grounded situation recognition. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 314-332. Springer, 2020.
[44]
Alec Radford.
Improving language understanding by generative pre-training, 2018.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR, 2021.
[46] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[47] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint 0rX1::2204.06125, 1(2):3, 2022.
[48] Joshua Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint 0rXiv:2010.04592, 2020.
[49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[50] Ugur Sahin, Hang Li, Qadeer Khan, Daniel Cremers, and Volker Tresp. Enhancing multi- modal compositional reasoning of visual language models with generative negative mining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5563-5573, 2024.
[51] Sepehr Sameni, Kushal Kafle, Hao Tan, and Simon Jenni. Building vision-language models on solid foundations with masked distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14216-14226, 2024.
[52] Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. Foil it! find one mismatch between image and language caption. arXiv preprint
2017.
[53] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video, Text-to-video generation without text-video data. arXiv preprint arXiv:2209. /4792, 2022.
[54] Harman Singh, Pengchuan Zhang, Qifan Wang, Mengjiao Wang, Wenhan Xiong, Jingfei Du, and Yu Chen. Coarse-to-fine contrastive learning in image-text-graph space for improved vision-language compositionality. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 869-893, Singapore, December 2023. Association for Computational Linguistics.
[55] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu, Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02455 2019.
[56] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238-5248, 2022.
[57] Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is picture worth thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems, 37:75392-75421, 2025.
[58] Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 1 11998-
1 12008, 2023.
14

---

[59] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung. Iz Beltagy. Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, pages 22964 22984. PMLR, 2022.
[60] Yi Ru Wang. Jiafei Duan, Dieter Fox, and Siddhartha Srinivasa. Newton: Are large language models capable of physical reasoning? arXiv preprint arXiv:2310.07018. 2023.
[61] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[62] Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. Retromae: Pre-training retrieval-oriented language models via masked auto-encoder. arXiv preprint arXiv:2205. 2022.
[63] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 27X1v:2205.01917, 2022.
[64] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou, When and why vision-language models behave like bags-of-words and what to do about it? In International Conference on Learning Representations, 2023.
[65] Arman Zarei, Keivan Rezaei, Samyadeep Basu, Mehrdad Saberi, Mazda Moayeri, Priyatham Kattakinda, and Soheil Feizi. Mitigating compositional issues in text-to-image generative models via enhanced text embeddings.
[66] Yunan Zeng, Yan Huang, Jinjin Zhang. Zequn Jie, Zhenhua Chai, and Liang Wang. Investigating compositional challenges in vision-language models for visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14141-1415/
2024.
[67] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[68] Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross- modal hard negatives to enhance visio-linguistic compositional understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13774-13784, 2024.
[69] Tiancheng Zhao, Tianqi Zhang. Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. An explainable toolbox for evaluating pre-trained vision-language models. In Wanxiang Che and Ekaterina Shutova, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 30-37, Abu Dhabi, UAE December 2022. Association for Computational Linguistics.
[70] Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Iterated learning improves compositionality in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13785-13795, 2024.
[71] Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-iman pretraining. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16793-16803, 2022.
[72] Xingyi Zhou, Rohit Girdhar, Armand Joulín, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision, pages 350-368. Springer, 2022.
15

---

[73] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995-5004, 2016.
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes].
Justification: The claims made in the abstract and introduction accurately reflect the paper's contributions. The motivation (limitations of compositional reasoning in contrastive VLMs), method (READ with two auxiliary objectives for the text encoder), and results (consistent improvements across benchmarks and compatibility with other methods) are all clearly stated and supported by empirical evidence throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
The paper includes dedicated Limitations section that discusses the scope generalization with respect to training data (e.g., dependence on high-quality, multi- caption datasets), architectural choices in the reconstruction objective (e.g., reliance decoder), and the absence of exploration into alternative decoder types or objectives. These limitations are clearly acknowledged and contextualized with respect to potential future directions.
Justification:
T5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e-g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover
17

---

limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
Justification: This work focuses on empirical evaluation and does not contain theoretical assumptions or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification: We provide all necessary details reproduce our main experimental results, including architecture, training protocol, hyperparameters. data splits, and implementation specifics Appendix A..I All models are trained using publicly available datasets and baselines, and paraphrased captions are generated via an open API with specified parameters.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions provide some reasonable avenue for reprodueibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
18

---

(c) If the contribution is new model (e.g,, , large language model), then there should either be a way access this model for reproducing the results a way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case closed-source models, it may that access the model limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: We provide an anonymous open-access GitHub repository containing all necessary materials to reproduce the main experimental results, including code, model checkpoints, and data. The repository includes detailed instructions on environment setup, data, and exact commands to run each experiment. Scripts are provided to reproduce the result of READ-CLIP.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please the NeurlPS code and data submission guidelines (https://niis.cc) for more details.
public/guides
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc.. wwt.cc/pppppppu...eeeeee wetgggudd/oeeuuuuieeeeeeeeeeeeee
wwwsssssooopcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: All training and evaluation details-including dataset splits, optimizer, learning rate, warmup steps, batch size, weight decay, decoder choice, loss weights, and paraphrase generation method-an are specified in the main text (Sec. 4.1) and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
19

---

7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No].
Justification: None of the existing work, including NegCLIP, CE-CLIP, and FSC-CLIP, report statistical significance. To ensure fair comparison, we follow this convention and report single-run results standard benchmarks. For training, we fix all random seeds to 2025, Evaluation is deterministic, and thus variability due randomness does not arise.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eιror rates).
If error bars are reported in tables or plots, The authors should explain text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes].
Justification: We report that all experiments can be conducted on single NVIDIA A100 40GB GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes].
Justification: Our research does not involve human subjects, personal data, high-risk assets. All datasets and models used are publicly available, properly licensed, and fully credited. We have carefully reviewed the NeurlPS Code of Ethics and confirm that our work
20

---

adheres to all its principles regarding safety, faimess,, environmental impact, data licensing. and reproducibility.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special considera- tion due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: Our paper proposes fine-tuning framework (READ) improve compositional reasoning in vision-language models. This may lead to more accurate and interpretable
VLMs. We do not foresee direct risks of misuse, as model does generate open-ended content or interface with end users.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, scraped datasets)?
Answer: [NA].
Justification: The paper does not release any high-risk assets. All datasets used are publicly available and widely adopted compositional reasoning benchmarks with no known safety concerns. Our released model is fine-tuned version of CLIP (ViT-B/32) on caption- based reasoning tasks, and is intended for academic use only. It does not have generative capabilities and poses minimal risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should released with necessary safeguards to allow for controlled use of the model, for example by requiring
21

---

that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
SugarCrepe, WhatsUp, VALSE, and SugarCrepe+++ all of which are publicly released under MIT licenses (with the exception of CREPE-Produuctiityy for which the license is unspecified). We cite the original papers introducing these benchmarks and describe their licenses and image sources in Appendix A.2 (see Table 7 All datasets were used accordance with their published terms of use. No scraped proprietary assets were used.
Justification: We use five existing
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the pack- age should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
The paper does not introduce any new assets. All datasets used are publicly available benchmarks, our method is a fine-tuning approach applied to existing models. No new datasets or models are released.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/modae as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
22

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: This work does not involve any crowdsourcing or experiments with human participants. All data and evaluations were conducted using publicly available resources without human subject interaction.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material fine, but if the main contri- bution of the paper involves human subjects, then as much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA].
This study does involve research with human subjects. All experiments were conducted on publicly available datasets without any interaction with individuals or collection of personal data, and therefore IRB approval was required.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA].
Justification: LLMs were used solely for writing assistance (e.g.. wording, formatting), and did not contribute to the design, implementation, or evaluation the proposed methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https://neuripp wttps///.ccccooooeecceeeeeeeeeee what should should not be described.
h 20////2255//// for
23

---

<task>
Rewrite the following image description:
</task>
"IIpptt*:
(caption)
"outppt""
Figure 6: A prompt for generating synthetic paraphrased captions.
Benchmark
License
Image Source
CREPE- CREPE-Produc Productivity 37
Unspecified
Visual Genome
27
SugarCrepe [9
MIT
COCO 35
SugarCrepe++
MIT
COCO 35
WpatsUp 24
MIT
Custom-collected, COCO R5
GQA
21
VALSE 40
MIT
Visual7W 73, COCO
35
SWiG 43
FOIL
52
Table 7: Benchmarks used in the evaluation, along with license and image source.
A
Appendix
A.1
Implementation Details
Batch Sampling for Training. As described in Sec. 4.1, all our experiments are conducted using the COCO (35) dataset. The COCO dataset is image-caption dataset consisting of images, each associated with a set of captions. During training, we first sample a batch of B images from this
dataset. For each i {1, 2, ................................ B), let T, (T+")) with I (where I, denotes the i-th image). From this set, we randomly select one caption TІ E Ta to
(1) J denote the set of N captions associated
form positive image-tex, pair (L, T.). In this way, obtain batch of B image-text pairs, denoted as (T))
Details of Each Loss Component. Recall that our proposed REconstruction and Alignment of text Descriptions (READ) method comprises three components in the final loss (Eq. x the standard contrastive loss, the token-level reconstruction loss, and the sentence-level alignment loss. For the contrastive loss (Eq. 4 we incorporate M = 3 hard negative captions per image, generated via rule-based perturbations as proposed in NegCLIP (64). For the reconstruction loss (Eq. 5 we use a frozen decoder extracted from the pre-trained encoder-decoder language model, T5 Large [46].
To obtain { } we randomly sample K = 1 element from the caption set T; associated with each image-text pair (4, T.) and use it as an alternative caption. For the alignment loss (Eq. 7 we generate paraphrased caption TV for each T; via augmentation using large language models. Specifically, prior to training, we generate one paraphrased caption for every caption in each image's original caption set using the gpt-4o- 1opt-4o-mini-2200 1wlo-mmiii2222--1 2024-----222----1111 model 22 This is done by applying a simple prompt as shown in Fig 6 with a temperature of 1.0 and all other parameters set to their obtain synthetic caption set for each image. Given a batch of sampled image-text pairs (44, B | during training, we randomly sample one caption from the union of the original and synthetic caption sets associated to Jt, and use it as TV. Finally, the weighting factors in Eq. g are set to a = 0.1 and 8 = 0.5.
default values [22]. From this augmentation,
Training Details. We fine-tune all models using the Huggingface transf ormers 61 library The Adam W optimizer is used with learning rate of 1.0 x 10 cosine annealing schedule, 50 warmup steps, and weight decay of 0.I. Training is performed with bf mixed precision for computational efficiency. All experiments are conducted using a single A000 40GB GPU.
A.2 Evaluation Details
Description Benchmarks. WhatsUp 24 evaluates spatial reasoning by testing whether models can interpret relative object positions. CREPE 37 measures compositional reasoning at varying

---

Table 8: Detailed results on CREPE [37]
Model
Atom
Negate
Swap
Total
CLIP
(Pre-trained)
18.9
35.3
17.3
23.9
TriPlet-CLIP
18.5
15.3
GNM-CLIP
21.7
17.3
CE-CLIP
40.4
42.7
34.8
NegCLIP 64
32.1
42.8
30.5
FSC-CLIP 38
40.5
45.5
42.5
READ-CLIP 37.1
26.2
61.2
41.5
Table 9: Detailed results on WHATSUP [24].
Model
COCO-Spatial
GQA-Spatial
Whats-up
Total
CLIP 45
(Pre-trained)
44.5
47.8
30.7
41.0
Triplei-CLIP
41
49.2
47.1
28.5
41.6
GNM-CLIP 50
44.8
47.4
32.6
41.6
CE-CLIP
43.7
47.8
30.7
40.7
NegCLIP
64
45.1
47.7
34.4
42.4
FSC-CLIP
38
47.7
41.9
29.6
39.8
READ-CLIP
51.6
48.1
31.8
43,9
complexity levels using logical operations such as conjunction, negation, and attribute swapping. VALSE [40] assesses fine-gra ameeggrined linguistic understanding, including object existence, quantity, action semantics, and coreference resolution. SugarCrepe [19] focuses on relational reasoning through hard negative captions crafted with natural linguistic variation. SugarCrepet [T] extends SugarCrepe by adding paraphrased positive caption and introduces two tasks: (1) image-to-text (ITT), which tests whether all paraphrased positives for given image are ranked above all negatives, and (2) text-to-text (TOT), which evaluates semantic consistency by checking whether each positive paraphrase pair is ranked above all negative pairs in the absence of visual context. Since our study aims to improve the compositional reasoning capability of VLMs such as CLIP, we primarily adopt the ITT metric as a major focus for evaluation, while including TOT as a supplementary measure.
Licensing of the Benchmarks. We conduct our evaluation on five publicly available compositional reasoning benchmarks. Table summarizss their license information and image sources. All datasets used for training and evaluation are either MIT-licensed or publicly released for research use.
A.3
Supplementary Experimental Results
This section provides extended experimental results that complement the main paper. We include both (1) detailed, category- results for each benchmark and (2) additional evaluations on zero-shot image classification datasets further assess the generalization ability of READ-CLIP.
A.3.1 Detailed Version of Experimental Results in the Main Paper
To complement the results presented in Table I we report category-wise results on all evaluation benchmarks, including CREPE, VALSE, WhatsUp, SugarCrepe, and SugarCrepe++ (ITT and TOT). Table g 1 present detailed breakdowns for individual benchmarks. These results provide finer- grained analysis of compositional reasoning performance supplementing the aggregated scores shown in the main paper. Overall, the detailed results confirm that READ-CLIP consistently improves over baselines across categories. In addition, we provide Fig. and Fig. g which present extended qualitative examples that respectively complement Fig. 3 and Fig. in the main paper.
25

---

Table 10: Detailed results on VALSE 40
Model
Actions
Coreference
Counting
Existence
Noun Phrases
Plurals
Relations
Total
CLIP 45
(Pre-trained)
74.3
61.7
69.3
90.4
57.9
66.0
67.4
TripletCCIIP
72.6
54.8
54.0
59.4
91.4
61.3
60.9
64.2
GNM-CLIP
72.1
76.0
CE-CLIP 68
85.0
59.9
67.6
78.2
94.4
78.8
74.0
76.0
NegCLIP 66
84.1
60.2
65.7
75.3
93.4
70.3
69.6
73.7
FSC-CLIP 38
82.9
59.4
66.3
77.6
93.5
72.7
75.3
74.4
READ-CLIP
95.8
75.0
76.2
Table 11: Detailed results on SugarCrepe 19
Att., Obj and Rel. transformation: Attribute, Object, and Relation, respectively.
denote the targets of
Model
Add Att.
Add Obj.
Replace Att.
Replace Obj.
Replace Rel.
Swap Att.
Swap Obj.
Total
CLIP
45
(Pre-trained)
69.5
77.0
80.3
90.7
69.4
64.1
61.2
73.2
Triplet CL.IP
41
85.5
87.5
86.7
94.5
83.2
73.1
68.6
82.7
GNM-CLIP
50
79.9
88.4
84.9
93.2
67.8
70.0
61.2
77.9
CE-CLIP
68
91.9
92.3
90.2
94.4
81.4
76.7
75.1
86.0
NegCLIIP
64
85.3
74.6
ISC-CLIP 38
86.7
90.2
89.2
94.3
80.4
77.8
77.6
85.2
READ-CLIP
87.7
90.3
91.0
94.9
80.6
82.7
81.6
87.0
Table 12: Detailed results on image-to-text subset SugarCrepe*+ 11.
Att., Obj., and Rel.
denote the targets of transformation: Attribute, Object, and Relation, respectively.
Model
Replace Att.
Replace Obj.
Replace Rel.
Swap Att.
Swap Obj.
Total
CI.IP
(Pre-trained)
65.7
87.0
56.5
45.0
45.8
60.0
Triplet-CLIP
71.7
87.0
62.3
48.5
39.2
61.7
GNM-CLIP
89.5
41.2
60.2
CF-CTIIP
62.4
81.9
$3.5
40.5
55.7
NegCLIP 64
69.7
89.8
52.6
58.1
54.7
65.0
FSC-CLIP 38
73.5
90.4
60.1
60.4
55.1
67.9
READ-CLIP
72.2
90.1
57.5
66.2
62.9
69.8
Table 13: Detailed results on text-to-text subset of SugarCrepe++ [1] Att., Obj., and Rel. denote the targets of transformation: Attribute, Object, and Relation, respectively.
Model
Replace Alt.
Replace Obj.
Replace Rel.
Swap Alt.
Swap Obj.
Total
CLIP
451
(Pre-trained)
59.3
83.7
38.6
32.7
19.2
46.7
TriP
41
74.1
92.3
52.3
43.2
24.9
57.4
IN
50
76.9
95.9
$1.9
48.9
26.1
60.0
CE-CLIP
68
74.2
89.6
52.0
42.8
26.5
57.0
NegCLIIP
64
76.4
94.6
51.7
56.6
33.1
62.5
FSC-CLIP
38
96.3
29.0
64.4
READ-CLIP
77.3
97.6
58.0
56.8
66.2
A.3.2 Additional Experimental Results
To further verify the generalization capability of READ-CLIP, we conducted additional evaluation beyond the main benchmarks. We assessed the model's zero-shot image classification performance across 23 widely used benchmarks [5]. We compared READ-CLIP with the original CLIP (45
pre-trained model and six representative compositional reasoning fine-tuning methods.
The results show that all fine-tuned models, including READ-CLIP, achieve lower average performance than the original CLIP across the 23 datasets. This finding aligns with trends reported in previous study (38] where improvements in compositional understanding often come the cost of general zero-shot capability. Such trade-offss highlight the inherent difficulty of maintaining broad generalization while adapting models specifically for compositional reasoning.
In Table 14
26

---

Postive Negstio
wile
who
Reconst.
Reconnt
Reconat. Lass
Reconst.
Lows
Lass
Lon
lapiop
ofa
WhansUpp
Aoulphowwzz
can bichind
A srmohain
43
headphones
A2
X A lapiop uddea annchair
41
#2
X
can the left of heudphones
81
43
A laprop fight armehoie
right
headphones
/ tush can ground
Rower in sundos box:
83
there lieer Inss can listds
bench between
X
lata in ganden Bosas:
CREPE
ground there liner. frnsh can artoss
bench between Bussst
Sower garden banch
greund thome a lince. stlle.
bas betwsen bonch
bench, sundes
31
there is liner.
bas betoeeen box
X
46
A6
a flare.
Dosypt pow
X
trash location liner.
01
85
82
A reporter intervious poticenoan
There exocily
VALSE
A policemun inservieus
There ure exuntly
82
sitting of musitor that displaying picnure of cnnttberr cat.
ridlna
SugarCrope
Auteebosel belind day.
sitting momico that disploying anauimmcee picture of another
dag ridlng
mur
skatchoand.
Figure Extended representative examples for Fig. 3 including additional examples from
CREPE (37 and VALSE [40], as well as WhatsUp [4] and SugarCrepe [19]. These extended
examples additionally include broader range of benchmarks where applying the reconstruction loss proved effective.
Table 14: Additional results of zero-shot image classification performance across various datasets.
Here, CLIP 45
the pre-trained model, while the other models are fine-tuned versions of CLIP 455
ww.000 www.oiieel ioliiia
20136
Medel
CLUP CX
19.8
50.5
$4.8
63A
43.3
62.3
97.1
76.5
NectLiil b4
$2.6
35.9
18.9
53.0
15.0
43.0
4011
1537
15.8
63.0
76A
30.3
45.5
603
40/8
51.2
27.2
20.7
540
58.
92.
96.7
79.6
56.6
CNNS-CLIP
18.5
$1.8
63-4
4:00
$4.3
77.
BACALLM
72.7
19.4
90.4
63.9
19.9
92.1
58.5
54.
942
510
35.2
42.2
30.
97.1
72.9
13.7
39.9
30.0
83.7
14.8
40.3
31.1
475
90
53.
723
249
45.5
524
351
46.8
187
45.3
301
39.8
51.7
96.1
63.8
3000
READ-CLUP
82.1
916
51.5
319
-1
19
916
27

---

Prssrive X Negative
wio
Aligumont
Algemen
Allgnment
Lors
Less
Loss
Loss
nunds child
parzon in skins
blork goor standing
piece of
itunda by child huidling
person is grnm shin
R3
81
black goor possioond nest tew while
43
02
piece of cake
plate.
stards by persaa holding
shild in green
Two biack poats standing ucsa to white gown.
83
SWAL
piece cake a plate.
sundwicchs beside bow!
Trays of pontries and
83
An open laptop and speakers of denk
M
81
benel of soup placed
Speokers and open
benide mays of sauturicbe
toptog crr poriiioned on of desk.
and pastrics.
sandwiches boside traga scup.
Speaisers top an.ureen lepton
Two zelvus grazing wille
kid wvaring vellow is
another horsc stanaling and
81
81
holding pizza ie bes.
81
81
staring whilee another two
mundinn
unffi holding a pizza in bes.
zohnns are grazing.
sebeas while
REPLACE ACCE
horse standing staiig
wearing green a gizza
person leaning up aguinal
peros govy but sining on woodlan bench.
minal rail while hotding
minbon colored ambreiia
parzon puritioned
wwaring gany shin aitting a voodon bencil.
while bolding
ralsbow colored unbrolla.
person leaning up
X pink slint and
A
X
while
ninbow
Figure 8: Extended representative examples for Fig. 4 including additional examples each category (SWAP and REPLACE) of SugarCrepe++. These extended examples further illustrate the effectiveness of applying the alignment loss across diverse cases.
28


Reviews:
[{"review_id": "fx4pNCsYyD", "reviewer": "Reviewer_UjDy", "summary": "In this paper, the authors propose a novel fine-tuning framework for CLIP-like VLM, including reconstruction and alignment of text descriptions to enhance the text encoder. Extensive experiments have demonstrated that the proposed algorithm can significantly enhance compositional reasoning capabilities.", "questions": "Besides the above concerns raised by the reviewer, please answer the following questions:\n\n1. Please provide additional details regarding the paraphrased captions. How are the captions generated, and which tool is used for generating them?\n2. If the granularity of the captions is further enhanced, would the performance improve accordingly?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is well-written, with clear logic and comprehensive experiments. The proposed algorithm is reasonable and has been validated as effective.\n\nWeekness:\n1. Although the technical details differ, the proposed algorithm shares a similar motivation with two existing CLIP-based papers, FG-CLIP[1] and FineCLIP[2]. Both of these aim to enhance the fine-grained representation capability of the CLIP model by leveraging fine-grained image-level and region-level multimodal contrastive learning. Please compare the proposed algorithm with these works in terms of algorithmic details and performance (if possible).\n2. The training dataset used in this study is COCO, which contains many images featuring similar objects and collected from similar scenes. How can it be ensured that fake negative pairs do not adversely affect the model's training? \n\n[1] FG-CLIP: Fine-Grained Visual and Textual Alignment \n[2] FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer 1 (UjDy)\n\nWe thank the Reviewer UjDy for the detailed review and constructive suggestions. We appreciate your acknowledgement that the paper is well-written, with clear logic and comprehensive experiments. Please find our answers to your comments and questions as follows.\n\n\n**[R1-1]**: `Although the technical details differ, the proposed algorithm shares a similar motivation with two existing CLIP-based papers, FG-CLIP and FineCLIP. Both of these aim to enhance the fine-grained representation capability of the CLIP model by leveraging fine-grained image-level and region-level multimodal contrastive learning. Please compare the proposed algorithm with these works in terms of algorithmic details and performance (if possible)`\n\n\nWe appreciate the reviewer’s thoughtful feedback. While **READ shares the high-level motivation** of improving CLIP's fine-grained compositional understanding, our approach is **distinct in scope, algorithmic design, and empirical setting**. Below, we clarify these differences in depth and provide **quantitative comparisons** to support our claims.\n\n\n#### **1. Overall Distinction**\n\n- **FG-CLIP** and **FineCLIP** enhance **visual representations** by explicitly aligning image regions and parts with fine-grained textual cues using large-scale region-level annotation or dynamic distillation strategies.\n- In contrast, **READ introduces lightweight, text-centric auxiliary losses** to structurally regularize the **text encoder**:  \n  - *(i)* a **token-level reconstruction loss**, and  \n  - *(ii)* a **sentence-level alignment loss** using LLM-generated paraphrases.\n\nThe main differences between FG-CLIP [Xie'25], FineCLIP [Jing'24], and READ-CLIP are summarized in the table below:\n\n`[Table R1]` Conceptual Differences of READ-CLIP and baselines\n| Method      | Targeted Modality | Key Mechanisms                                     | Training Data Scale                                 |\n|-------------|-------------------|----------------------------------------------------|---------------------------------------------|\n| **FG-CLIP** | Image encoder     | Region-level contrastive alignment, hard negatives | 1.6B global + 12M images / 40M boxes         |\n| **FineCLIP**| Image encoder     | Multi-grained contrastive + self-distillation      | MS-COCO (100K) + 970K region-labels              |\n| **READ** *(Ours)* | Text encoder      | Token reconstruction + sentence alignment           | MS-COCO (100K)              |\n\n\n#### **2. Performance Comparison on Compositional Reasoning Benchmarks**\n\n`[Table R2]` Performance Comparison between READ-CLIP and baselines\n\n| Model                   | CREPE | SUGARCREPE | VALSE | WHATSUP |\n|-------------------------|-------|------------|-------|---------|\n| FG-CLIP (ViT-B/16)      | 15.7  | 80.4       | 69.6  | 40.7    |\n| FineCLIP (ViT-B/16)     | 14.2  | 80.7       | 67.8  | 40.8    |\n| **READ-CLIP (ViT-B/32)**| **41.5** | **87.0**   | **76.2** | **43.9** |\n\n`Table R2` shows that, even with the **coarse granularity (i.e., large patch size and lower spatial resolution) of ViT-B/32 relative to ViT-B/16**  and **modest training data scale** (as shown in `Table R1`), **READ-CLIP consistently outperforms FG-CLIP and FineCLIP across all compositional reasoning benchmarks**, highlighting the effectiveness of its text-encoder-centric approach for enhancing compositional reasoning capability in CLIP.\n\nWe hope this clarifies the detailed difference between READ, FG-CLIP and FineCLIP, and provides a clear message that READ is a simple and effective alternative.\n\n***References:***\n\n[Xie'25] Xie, C., Wang, B., Kong, F., Li, J., Liang, D., Zhang, G., Leng, D., & Yin, Y. (2025). FG-CLIP: Fine-grained visual and textual alignment. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025).\n\n[Jing'24] Jing, D., He, X., Luo, Y., Fei, N., Yang, G., Wei, W., Zhao, H., & Lu, Z. (2024). FineCLIP: Self-distilled region-based CLIP for better fine-grained understanding. In Advances in Neural Information Processing Systems, 37 (NeurIPS 2024). \n\n---\n\n**[R1-2]**: `The training dataset used in this study is COCO, which contains many images featuring similar objects and collected from similar scenes. How can it be ensured that fake negative pairs do not adversely affect the model's training?`\n\n\nThank you for raising this important concern. We fully agree that the **semantic false negative problem**—i.e., negative pairs that are semantically similar to positives—is a key challenge in contrastive learning.\n\nNonetheless, **MS-COCO remains the standard for CLIP-style contrastive fine-tuning**, and is widely used not only in recent compositional representation learning studies such as NegCLIP [Yuksekgonul'23] and FSC-CLIP [Oh'24], but also in recent multi-modal representation learning such as M^3-Mix [Oh'23] and CLIP-Refine [Yamaguchi'23]. These prior works have shown that, **with COCO datasets, contrastive objectives leads to meaningful performance gains**, indicating that the practical effect of false negatives is limited in this setting. Thus, we believe that our experimental setup is both **standardized and empirically sound**.\n\nWe appreciate the reviewer’s thoughtful question and hope this clarifies our position.\n\n***References:***\n\n[Yuksekgonul'23] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., & Zou, J. (2023). When and why vision-language models behave like bags-of-words, and what to do about it? Proceedings of the 11th International Conference on Learning Representations (ICLR 2023). \n\n[Oh'24] Oh, Y., Cho, J. W., Kim, D.-J., Kweon, I. S., & Kim, J. (2024). Preserving multi-modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).\n\n[Oh'23] Oh, C., So, J., Byun, H., Lim, Y., Shin, M., Jeon, J.-J., & Song, K. (2023). Geodesic multi-modal mixup for robust fine-tuning. Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023).\n\n[Yamaguchi'23] Yamaguchi, S., Feng, D., Kanai, S., Adachi, K., & Chijiwa, D. (2025). Post-pre-training for modality alignment in vision-language foundation models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025).\n\n---\n\n\n**[R1-3]**: `Please provide additional details regarding the paraphrased captions. How are the captions generated, and which tool is used for generating them?`\n\nThank you for your question regarding the generation process of paraphrased captions used in our method.\n\nAs described in **Appendix A.1** of the supplementary material, the **sentence-level alignment loss** in READ requires semantically equivalent yet lexically distinct captions for each image–caption pair. To obtain these, we generate paraphrases using a structured LLM-based procedure:\n\n- **Model**: `gpt-4o-mini-2024-07-18` via the OpenAI API \n- **Prompt format**: Shown in *Appendix Figure 6* \n- **Generation parameters**: `temperature = 1.0` (all other settings default) \n- **Timing**: Paraphrases are generated *offline*, prior to training \n- **Quantity**: One paraphrase per original caption\n\nThese synthetic captions are used **only** for supervising the sentence-level alignment loss. \nFor the **token-level reconstruction loss**, we instead leverage the *multiple original captions* provided by MS-COCO, without any LLM-based augmentation. These details are in the Appendix of the submitted manuscript. \n\nWe will include these details in the main body of the revised manuscript.\n\n\n---\n\n**[R1-4]**: `If the granularity of the captions is further enhanced, would the performance improve accordingly?`\n\n\nThank you for your insightful question. We agree that the **granularity of captions can affect model performance**. Indeed, prior work on compositional reasoning—such as FSC-CLIP [Oh'24], DAC [Doveh'23-1], and TSLVC [Doveh'23-2]—has demonstrated that replacing generic captions from datasets like CC3M with **fine-grained, VLM-generated descriptions** can lead to consistent improvements.\n\nIn our case, READ is designed to promote understanding relationship between words through both reconstruction and alignment objectives. As such, we believe that **incorporating more fine-grained captions would further enrich the training signal and potentially enhance model performance**. \n\nWe appreciate this valuable suggestion and will incorporate a discussion on the potential impact of caption granularity on READ's performance in the revised manuscript.\n\n***References:***\n\n[Oh'24] Oh, Y., Cho, J. W., Kim, D.-J., Kweon, I. S., & Kim, J. (2024). Preserving multi-modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).\n\n[Doveh'23-1] Doveh, S., Arbelle, A., Harary, S., Herzig, R., Kim, D., Cascante-Bonilla, P., Alfassy, A., Panda, R., Giryes, R., Feris, R., Ullman, S., & Karlinsky, L. (2023). Dense and aligned captions (DAC) promote compositional reasoning in VL models. In Advances in Neural Information Processing Systems, 37 (NeurIPS 2023 Spotlight).\n\n[Doveh'23-2] Doveh, S., Arbelle, A., Harary, S., Panda, R., Herzig, R., Schwartz, E., Kim, D., Giryes, R., Feris, R., Ullman, S., & Karlinsky, L. (2023). Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023)."}, {"review_id": "TsaFDb4VD8", "reviewer": "Reviewer_a9tz", "summary": "The paper proposes READ to enhance CLIP-like models (ITM) compositional reasoning by adding two auxiliary losses to the usual image-text contrastive objective via finetuning: 1) token-level reconstruction, where a frozen T5 decoder is forced to reconstruct a paraphrase of the input caption from its text-encoder embedding, encouraging the encoder to encode intra-sentence relations; 2) sentence-level alignment; where contrastive alignment of paraphrase pairs is enforced, pushing their embeddings together while keeping other captions apart. Authors show results on standard benchmarks.", "questions": "- How sensitive is READ to paraphrase quality (noisy paraphrases)?\n\n- Does READ hurt zero-shot generalization on standard CLIP tasks (e.g., ImageNet, etc)?\n\n- Why not use the same loss objectives with another text backbone?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- This paper is overall easy to read, the method is described really well. It is also well motivated, and figures help understand the whole pipeline. Formulations seem correct, extensive results in standard benchmarks, and good analysis.\n\n- This paper focuses on CLIP-like models, given that VLMs cover both auto-regressive and ITM, it should be good to make the distinction explicitly.\n\n- This paper explores only T5 text encoder. While this is ok, relevant prior work is not further discussed. As an example, VQAScore for Evaluating Text-to-Visual Models [1] explicitly fine-tunes a T5 text encoder and explains the rationale behind this decision. Is this something the authors of READ also leverage, thus the positive results?\n\n- Evaluation only limited to relatively old benchmarks: it is highly encouraged to report results on more recent/more challenging/realistic benchmarks, e.g.: SugarCrepe++, Winoground, ConMe [2].\n\n[1] Lin, Zhiqiu, et al. \"Evaluating text-to-visual generation with image-to-text generation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[2] Huang, Irene, et al. \"Conme: Rethinking evaluation of compositional reasoning for modern vlms.\" Advances in Neural Information Processing Systems 37 (2024): 22927-22946.", "was_revised": true, "final_justification": "The authors have addressed my questions, and I'm increasing my score accordingly.", "rebuttal": "## To Reviewer 2 (a9tz)\n\nWe thank the Reviewer a9tz for the detailed review and constructive suggestions. We appreciate your acknowledgement that the paper is well motivated, easy to read, and analyzed well. Please find our answers to your comments and questions as follows.\n\n\n\n**[R2-1]**: `This paper focuses on CLIP-like models, given that VLMs cover both auto-regressive and ITM, it should be good to make the distinction explicitly.`\n\n\nThank you for the helpful observation. As the reviewer correctly points out, vision-language models (VLMs) can be broadly categorized into three classes:\n\n1. **Image–Text Contrastive (ITC) Models** – e.g., CLIP, which learn via contrastive objectives over image–text pairs using dual encoders.\n2. **Auto-Regressive Generation Models** – which generate textual descriptions or answers conditioned on images.\n3. **Image–Text Matching (ITM)-Based Models** – which treat alignment as a classification task over paired inputs.\n\nOur work focuses specifically on the **first category (ITC)**, targeting CLIP-style dual-encoder architectures. \n\nWe acknowledge that this distinction was **not made sufficiently clear** in the submitted manuscript. We will clarify this distinction in the revised manuscript.\n\n---\n\n**[R2-2]**: `This paper explores only T5 text encoder. While this is ok, relevant prior work is not further discussed. As an example, VQAScore for Evaluating Text-to-Visual Models explicitly fine-tunes a T5 text encoder and explains the rationale behind this decision. Is this something the authors of READ also leverage, thus the positive results?`\n\nThank you for raising this important point.\n\nFirst, we would like to clarify the difference in the role of T5 between VQAScore and READ-CLIP. VQAScore explicitly **fine-tunes** the T5 text decoder, whereas READ-CLIP fine-tunes the dual encoders while making use of a **frozen** T5 text decoder. In our method, the T5 text decoder remains completely frozen throughout training and is only used to provide sentence-level reconstruction supervision for the text encoder.\n\nOur rationale for using T5 is closely tied to the core idea of our work. One of the key motivations of READ is to improve the training of the encoder-only CLIP architecture by introducing a **token-level reconstruction loss**. This, however, requires a high-quality pre-trained text decoder. T5, as a widely adopted encoder-decoder model with a strong pre-trained decoder and cross-attention capabilities, is a natural fit for this purpose. Importantly, it allows us to achieve the reconstruction objective without modifying the CLIP backbone, aligning with our design goal of enhancing alignment in a minimally invasive way.\n\nWe will clarify this design choice in the revised manuscript to avoid any potential confusion.\n\nAs per the reviewer's suggestion, we will further include relevant prior work.\n\n---\n\n**[R2-3]**: `Evaluation only limited to relatively old benchmarks: it is highly encouraged to report results on more recent/more challenging/realistic benchmarks, e.g.: SugarCrepe++, Winoground, ConMe.`\n\n\nThank you for this suggestion. \n\n#### 1. On SugarCrepe++\n\nWe would like to note that **SugarCrepe++ is already included in our evaluation**, as shown in **Tables 1–4 and Figures 4–5** of the main paper. This benchmark is central to our compositional evaluation suite.\n\n**READ-CLIP outperforms other baselines on SugarCrepe++**, as shown in Table 1 of the submitted manuscript. \n\n\n#### 2. On Winoground and ConMe\n\nWinoground and ConMe are important benchmarks. However, Winoground **go beyond compositionality alone** and often requires advanced capabilities, as stated in a related work:\n\n> *“..., we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images”* \n> — *Diwan et al. 2022 [Diwan'22]*\n\nAdditionally, ConMe is designed to evaluate **large-scale generative VLMs** (e.g., Flamingo, PaLI, GPT-4V) and **is not used as a standard benchmark for studies on contrastive CLIP fine-tuning**.\n\nRecall that our focus was to assess **compositional reasoning** of **vision-language models trained with contrastive learning**. While testing on Winoground/ConMe is an interesting direction for future exploration, we believe their exclusion does not detract from the core objectives of this study.\n\nWe hope this clarification addresses the concern, and we thank the reviewer again for the constructive feedback.\n\n***References***:\n\n[Diwan'22] Diwan, A., Berry, L., Choi, E., Harwath, D., & Mahowald, K. (2022, December). Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 2236–2250). Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.\n\n---\n\n**[R2-4]**: `How sensitive is READ to paraphrase quality (noisy paraphrases)?`\n\n\nThank you for this insightful question.\n\nWe interpret the reviewer’s concern as asking how robust our method is when some of the paraphrases used for sentence-level alignment are not true paraphrases which serve as semantic noise.\n\nTo resolve the reviewer's question, we conducted an additional experiment where we injected noise into the paraphrase data by randomly replacing a fraction of paraphrase captions with **unrelated captions from the dataset**. Specifically, we ran experiments with **10%** and **20%** of the paraphrase pairs being replaced with mismatched captions.\n\nThe results are shown below:\n\n| Model                    | CREPE | SUGARCREPE | VALSE | WHATSUP | SUGARCREPE++ (ITT) | SUGARCREPE++ (TOT) | AVG  |\n|--------------------------|-------|------------|--------|---------|---------------------|---------------------|------|\n| READ-CLIP                | 41.5  | 87.0       | 76.2   | 43.9    | 69.8                | 66.2                | 64.1 |\n| READ-CLIP (10% Noise)    | 39.0  | 86.9       | 76.0   | 43.6    | 67.1                | 64.7                | 62.9 |\n| READ-CLIP (20% Noise)    | 38.6  | 86.8       | 76.0   | 43.4    | 65.1                | 62.9                | 62.1 |\n\nAs expected, increasing noise leads to performance degradation. However, the drop is relatively modest—**only 1.2 to 2.0 points in average score**—indicating that **READ is reasonably robust even in the presence of moderate paraphrase noise**.\n\nWe will include this analysis and discussion in the revised manuscript.\n\n---\n\n**[R2-5]**: `Does READ hurt zero-shot generalization on standard CLIP tasks (e.g., ImageNet, etc)?`\n\n\nThank you for raising this important question. To address the reviewer's question, we ran additional experiments on two standard CLIP tasks: cross-modal retrieval and image classifications tasks.\n\nFrom the results shown in the below tables, one can confirm that READ-CLIP maintains or even improves the performance of CLIP on **cross-modal retrieval benchmarks**, while some performance drop is observed on *image classification tasks* when READ is applied to CLIP.\n\n##### 1. Zero-shot Cross-modal Retrieval (Flickr30K)\n\n| Model                  | Direction | R@1   | R@5   |\n|------------------------|-----------|--------|--------|\n| CLIP (Pre-trained)     | i2t       | 79.3   | 95.0   |\n|                        | t2i       | 58.8   | 83.4   |\n| **READ-CLIP (Fine-tuned)** | i2t   | **79.8** | **95.1** |\n|                        | t2i       | **68.9** | **90.6** |\n\nThe above result shows that applying READ improves **Recall@1 (text-to-image)** by **+10.1%**, demonstrating that READ effectively enhances text representations in line with its design goals. Note that READ-CLIP is fine-tuned only on COCO dataset, \nwhich implies that the improvement on Flickr30K is a strong indicator of READ’s cross-dataset generalization ability.\n\n##### 2. Zero-shot Image Classification\n\n| Model             | MNIST | CIFAR10 |\n|-------------------|--------|---------|\n| CLIP (Pre-trained)| 48.25  | 89.83   |\n| Triplet-CLIP      | 34.31  | 89.11   |\n| GNM-CLIP          | 54.33  | 88.45   |\n| CE-CLIP           | 40.56  | 85.90   |\n| FSC-CLIP          | 50.38  | 89.08   |\n| **READ-CLIP**     | 47.95  | 87.14   |\n\nThe above result shows that applying READ incurs performance drop on image classification tasks. We note that this slight degradation on classification tasks is a common trade-off observed in prior work on CLIP-based compositional fine-tuning.\n\nWe will report these results in the revised manuscript.\n\n---\n\n**[R2-6]**: `Why not use the same loss objectives with another text backbone?`\n\n\nThank you for this thoughtful question. We believe your question refers to our choice of using CLIP encoders (backbone), and we would like to share our thoughts in more detail below.\n\nAlthough we applied READ to vanilla CLIP backbone in the submitted manuscript, our proposed objectives are also applicable to **any vision-language model (VLM) that uses the dual-encoder architecture**, as the reviewer pointed out. These VLMs include a variety of off-the-shelf CLIP variants with different text encoders or backbone modifications. \n\nWe will revise the manuscript to explicitly state that **READ can be directly applied to other pre-trained dual-encoder models** in a plug-and-play fashion during fine-tuning."}, {"review_id": "1zLyTBlqp4", "reviewer": "Reviewer_Zdd4", "summary": "This paper proposes a fine-tuning approach for vision-language models (VLMs) such as CLIP to enhance their compositional reasoning capabilities. The method introduces two auxiliary loss terms to the standard contrastive loss: (1) a token-level reconstruction loss that encourages the model to reconstruct paraphrased versions of the input sentence using a frozen pre-trained decoder, and (2) a sentence-level alignment objective. This fine-tuning strategy yields improved performance across multiple compositional reasoning benchmarks. Furthermore, integrating this approach with other compositional reasoning techniques leads to additional performance gains.", "questions": "1. The paper presents aggregate performance across five benchmarks; however, it would be helpful to understand how the proposed approach performs on specific subsets of these benchmarks. Are there particular subsets where the method shows significant improvement, or conversely, any subsets where performance degrades?\n2.According to Appendix A.1, the paraphrases used for the token-level reconstruction loss are selected from the COCO dataset. However, COCO captions are often diverse and may not be true paraphrases, with many describing different semantic content. How did the authors address this issue during training, particularly when the paired captions are not semantically equivalent?\n3. When comparing results in Tables 1 and 4, it is unclear why applying READ to the fine-tuned CLIP model (Table 4: average performance of CLIP+READ = 58.2) underperforms compared to applying READ to the pre-trained CLIP model (Table 1: CLIP+READ = 64.1). Could the authors clarify the cause of this performance drop? Is there any explanation for why the fine-tuning process may negatively affect READ's effectiveness?\n\nI am willing to improve my score if the above issues are clarified.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and easy to follow\n- The idea of reconstructing paraphrased versions of input sentences is innovative -- particularly, the use of an off-the-shelf decoder to reconstruct paraphrases encourages the network to capture the syntactic structure and word order of the input text.\n- The introduction of a sentence-level alignment loss is another noteworthy contribution. This objective encourages the model to learn semantic representations by bringing semantically equivalent sentences—despite lexical variation—closer in the embedding space.\n\nWeaknesses:\n- **Limited evaluation results:** While Tables 1, 2 and 3 present results across 5 different benchmarks, the evaluation on the subsets of the benchmarks is missing. Reporting performance on benchmark subsets would offer deeper insights into which aspects of compositionality and semantics the proposed method improves.\n- **Clarification on the Use of \"Paraphrase\":** The term paraphrase in the token reconstruction loss appears to be misleading. According to Section A.1 of the appendix, these paraphrases are selected from the COCO dataset, which consists of diverse captions that may not always be semantically equivalent. This distinction is not clearly addressed in the paper and may explain why the method performs best for K=1 in Table 3.\n- **Reliability of LLM-Generated Paraphrases:** For the sentence-level alignment loss, the paraphrases are generated by the LLM GPT-4o-mini. However, the paper does not discuss the reliability or semantic fidelity of these generated paraphrases, nor does it mention whether any post-processing or filtering was applied. This information is crucial for assessing the robustness of the alignment loss.\n- **Performance on other downstream tasks:** It is necessary to show that the proposed approach retains the performance of CLIP on other downstream tasks, while improving the performance on compositionality benchmarks. For example, Table 6 in the NegCLIP paper (https://openreview.net/pdf?id=KRLUvxh8uaX) provides such an analysis, showing performance retention across standard vision and language benchmarks. Including a similar evaluation would help establish the robustness and general applicability of the fine-tuning strategy.", "was_revised": true, "final_justification": "Most of my concerns are resolved by the authors rebuttal. But I would still like to know how training with sentences which are not semantically similar lead to improved performance.", "rebuttal": "## To Reviewer 3 (Zdd4)\n\n\nWe thank the Reviewer Zdd4 for the detailed review and constructive suggestions. We appreciate your acknowledgement that our idea is innovative, the contribution is noteworthy, and the paper is easy to follow. Please find our answers to your comments and questions as follows.\n\n\n**[R3-1]** `The evaluation on the subsets of the benchmarks is missing.`\n\nBelow we provide a comprehensive **per-subset performance**. Due to the space limitation, we compare with CLIP and FSC-CLIP (the latest baseline).\n\n**READ-CLIP consistently outperforms CLIP across most subsets**, not just in average scores. Notably:\n\n- On **SugarCrepe and SugarCrepe++**, READ-CLIP shows **up to 20 percentage point improvements** in **replace** and **swap** tasks.\n- On **VALSE**, READ-CLIP has strong gains in **existence** (69.3 → 80.8), **plurals** (57.9 → 73.8), and **relations** (66.0 → 75.0), all of which require understanding fine-grained semantic variation.\n- On **WHATSUP**, READ-CLIP yields improvements across all subcategories.\n- On **CREPE**, significant performance improvement is observed on **Atom** (18.9 → 37.8) and **Swap** (17.3 → 61.2), while performance drop is observed on **Negate**. \n\n\nWe will include the complete table as well as the above discussion in the revised manuscript.\n\n### SugarCrepe\n* att: attribute, obj: object, rel: relation\n\n| Model|Add (att)|Add (obj)|Replace (att)|Replace (obj)|Replace (rel)|Swap (att)|Swap (obj)|Total|\n|-|-|-|-|-|-|-|-|-|\n| CLIP (Pre-trained) | 69.5    | 77.0    | 80.3         | 90.7         | 69.4         | 64.1      | 61.2      | 73.2   |\n| READ-CLIP         | **87.7**    | **90.3**    | **91.0**         | **94.9**         | **80.6**         | **82.7**      | **81.6**      | **87.0**   |\n| FSC-CLIP          | 86.7    | 90.2    | 89.2         | 94.3         | 80.4         | 77.8      | 77.6      | 85.2   |\n\n\n### SugarCrepe++ (ITT)\n\n|Model|Replace (att) |Replace (obj)|Replace (rel)|Swap (att)|Swap (obj)|Total|\n|-|-|-|-|-|-|-|\n| CLIP (Pre-trained) | 65.7 | 87.0 | 56.5 | 45.0 | 45.8 | 60.0 |\n| READ-CLIP | **72.2** | 90.1 | 57.5 | **66.2**     | **62.9**     | **69.8**  |\n| FSC-CLIP | 73.5 | **90.4** | **60.1** | 60.4  | 55.1 | 67.9 |\n\n\n### VALSE\n\n| Model | Actions | Coreference | Counting | Existence | Noun Phrases | Plurals | Relations | Total |\n|-|-|-|-|-|-|-|-|-|\n| CLIP (Pre-trained) | 74.3 | 54.4 | 61.7 | 69.3 | 90.4 | 57.9 | 66.0 | 67.4 |\n| READ-CLIP | **86.3** | 55.7 | **69.0** | **80.8** | **95.8** | **73.8** | 75.0 | **76.2** |\n| FSC-CLIP | 82.9 | **59.4** | 66.3 | 77.6      | 93.5 | 72.7 | **75.3** | 74.4 |\n\n\n### WHATSUP\n\n| Model | COCO-Spatial | GQA-Spatial | Whats-up | Total |\n|-|-|-|-|-|\n| CLIP (Pre-trained) | 44.5 | 47.8 | 30.7    | 41.0  |\n| READ-CLIP         | **51.6** | **48.1** | **31.8**    | **43.9**  |\n| FSC-CLIP          | 47.7 | 41.9 | 29.6    | 39.8  |\n\n\n### CREPE\n\n| Model | Atom | Negate | Swap  | Average |\n|-|-|-|-|--|\n| CLIP (Pre-trained) | 18.9 | 35.3   | 17.3  | 23.9  |\n| READ-CLIP | 37.1 | 26.2   | **61.2**  | 41.5  |\n| FSC-CLIP | **40.5** | **41.3**   | 45.5  | **42.5** |\n\n---\n\n**[R3-2]** `The term \"paraphrase\" in the token reconstruction loss is misleading.`\n\nAs the reviewer correctly observes, captions in the COCO dataset describe different aspects of the same image and are not necessarily semantically equivalent. We used such alternative captions for the reconstruction loss. \n\nIn the submitted manuscript, we referred to these alternative captions as “paraphrases,” but **this term is misleading** as pointed out by the reviewer. More precisely, our method samples **caption variants**—captions associated with the same image that describe complementary or overlapping content. \n\nIn the revised manuscript, **we will revise the terminology** from “paraphrase” to “caption variant” or “alternative caption” to better reflect their nature and avoid confusion.\n\nWe also agree with your interpretation that the performance peak at $K = 1$ in Table 3 may stem from this potential semantic difference among captions. We will explicitly include this explanation in the revised manuscript to clarify the notation and interpretation of our experimental result in Table 3.\n\n\n---\n\n**[R3-3]** `LLM-Generated Paraphrases are Reliable?`\n\nThank you for raising this important question.\n\nFirst, we did not apply any post-processing or filtering to the LLM-generated paraphrases used during training. This decision was motivated by recent studies that demonstrate the effectiveness of using LLM-generated captions in contrastive learning pipelines, even without filtering [Fan'23, Doveh’23-1, Doveh’23-2].\n\n\nAs per the reviewer's comment, **we ran additional experiments validating the reliability of the LLM-generated captions**. To be specific, we randomly sampled **1,000 paraphrase pairs** (original vs. LLM-generated) from our training set and assessed their semantic consistency using **two stronger LLMs**:  \n- **GPT-4.1-2025-04-14**\n- **Gemini-2.5-Flash**\n\nEach model is independently prompted to determine whether the two captions are **semantically equivalent** using a strict boolean format, where the detailed prompt is shown below:\n\n#### Prompt (used in both models):\n```text\n<task>\nGiven two image captions, decide whether they are semantically equivalent (i.e., paraphrases).\n\nReturn the result **only** as a strict JSON object with one boolean field named \"paraphrase\".\n\nExample formats:\n{\"paraphrase\": True}\n{\"paraphrase\": False}\n\nNo additional text, explanations, or metadata.\n</task>\n\n**Input**:\nCaption 1: {original_caption}\nCaption 2: {synthetic_caption}\n```\n\n#### Result:\n\n| Evaluator         | % Judged as Paraphrase | Std Dev   |\n|------------------|------------------------|-----------|\n| **GPT-4.1**       | 99.7%                  | 0.0547    |\n| **Gemini-2.5-Flash** | 98.1%                  | 0.1366    |\n\nThese results indicate that **more than 98%** of our paraphrases were judged to preserve the original meaning, providing strong empirical support for the semantic reliability of our LLM-generated captions.\n\nTo further demonstrate the reliability of LLM-generated captions, we conducted additional experiments analyzing how the choice of different LLMs (GPT-4o-mini, GPT-4.1, Gemini-2.5-flash) for caption generation impacts the compositional reasoning performance of READ-CLIP, averaged out over 5 benchmarks used in the manuscript. As shown in the result below, the performance of READ-CLIP remains consistently stable regardless of which LLM is used to generate captions.\n\n| Model                       | AVG  |\n|-----------------------------|------|\n| READ-CLIP (GPT-4o-mini)     | 64.1 |\n| READ-CLIP (GPT-4.1)         | 64.3 |\n| READ-CLIP (Gemini-2.5-flash)| 64.2 |\n\nWe will include this discussion in the revised manuscript.\n\n***References:***\n\n[Fan'23] Fan et al. Improving CLIP training with language rewrites. NeurIPS 2023.\n\n[Doveh'23-1] Doveh et al. Dense and aligned captions (DAC) promote compositional reasoning in VL models. In NeurIPS 2023.\n\n[Doveh'23-2] Doveh et al. Teaching structured vision & language concepts to vision & language models. In CVPR 2023.\n\n---\n\n**[R3-4]** `Performance on other downstream tasks?`\n\nAs per the reviewer's questions, we ran additional experiments on two downstream tasks -- cross-modal retrieval and image classification.\n\nDue to the space limitation, please check the detailed result in our response to `[R2-2]` -- one can confirm that READ-CLIP maintains or even improves the performance of CLIP on **cross-modal retrieval benchmarks**, while some performance drop is observed on *image classification tasks* when READ is applied to CLIP. We note that this slight degradation on classification tasks is a common trade-off observed in prior work on CLIP-based compositional fine-tuning [Oh'24].\n\nWe will report these results in the revised manuscript.\n\n[Oh'24] Oh et al. Preserving multi-modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality. In EMNLP 2024.\n\n---\n\n**[R3-5]** `Why applying READ to the fine-tuned CLIP model (Table 4: average performance of CLIP+READ = 58.2) underperforms compared to applying READ to the pre-trained CLIP model (Table 1: CLIP+READ = 64.1)?`\n\nLet us clarify the implication of the results in Tables 1 and 4. Let us focus on three methods which are fine-tuned versions (using MS-COCO) of the same pre-trained CLIP model, where different fine-tuning loss is used for each method: \n\n- **Method 0** (Table 1, last row): Uses our full READ loss as in Equation (8), where the contrastive component $\\mathcal{L}'_{I \\rightarrow T}$ **includes** hard negatives.\n\n- **Method 1** (Table 4, Row 1): Standard CLIP baseline, fine-tuned with only the vanilla contrastive loss (average of losses in Equation (2)), **without** using hard negatives.\n\n- **Method 2** (Table 4, Row 2): Uses the full READ loss from Equation (8), **but** replaces \\(\\mathcal{L}_{I \\rightarrow T}^{**\\prime**}\\) \nwith the standard contrastive loss \n\\(\\mathcal{L}_{I \\rightarrow T}\\) in Equation (2), **without** using hard negatives.\n\nAs such, the READ-CLIP in Table 1 and the READ-CLIP in Table 4 refer to **two different variants** of our method, where the former includes hard negatives in the loss, while the latter does not. **The performance difference of two methods** (64.1 for method 0 and 58.2 for method 2) **implies the importance of including hard negatives in the fine-tuning loss**. \n\nWe acknowledge that this distinction may not have been sufficiently clear in the initial submission and could understandably cause confusion. To address this, we will update the revised manuscript to make these differences clear."}, {"review_id": "uQXLCJXwyN", "reviewer": "Reviewer_gRGm", "summary": "This paper introduces READ, a fine-tuning method aimed at enhancing compositional reasoning in VLMs such as CLIP. The authors propose an approach that adds two auxiliary losses to the standard Contrastive Loss:\n- A Token-level Reconstruction Loss, which helps the text encoder better capture intra-sentence relationships.\n- A Sentence-level Alignment Loss, designed to improve the encoder's ability to recognize semantic similarity between sentences that use different expressions to convey the same meaning.\n\nThe authors apply their method to CLIP and its variants, evaluate it on five compositional reasoning benchmarks, and report consistent improvements over existing similar techniques. They also provide ablation studies to assess the contribution of each component of the proposed loss.", "questions": "- Clarification on the paraphrasing process: The paper mentions paraphrase generation through augmentation, but the details remain unclear. Did you use a specific LLM to generate paraphrases? If so, which one, and under what settings? Or are you using a VLM to do this, conditioning on both text and image? If a VLM was used for paraphrasing, this could raise a concern about circularity, i.e., relying on VLM, that you criticized for poor compositional reasoning, to improve compositional reasoning of other VLMs. Clarifying this aspect would strengthen the methodological transparency of the paper.\n- Highlighting the originality of the contribution: While the reconstruction and alignment losses are motivated and well implemented, their novelty may not be immediately evident to readers who are less familiar with the literature. I recommend emphasizing more explicitly how your approach differs from prior work using reconstruction losses, specifically, by explaining that although similar techniques exist for general-purpose encoder training, they have not been applied to the problem of improving compositional reasoning in contrastive VLMs. Making this distinction more visible could better communicate the originality of your work.\n- More qualitative examples: The qualitative examples presented in the main paper are insightful. However, I suggest including additional qualitative examples, especially failure cases and edge scenarios, maybe in the appendix. This would help readers better understand the behavior of the model under different conditions and strengthen the empirical analysis.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "*Strenghts*:\n- The problem of compositional reasoning is well explained in the introductory sections of the paper, providing useful context and motivation.\n- The method achieves strong results, showing a clear overall improvement over other CLIP-based models addressing similar compositional reasoning challenges. In the only benchmark where READ does not rank first, it still ranks second.\n- The experimental section is comprehensive and well-structured, covering key aspects of the proposed method. The ablation studies in Table 2 and Table 3 are particularly valuable, and the inclusion of the experiment in Table 4â€”evaluating READ when applied to different fine-tuning baselinesâ€”is especially noteworthy. Overall, Section 4 demonstrates a solid and thorough evaluation of the method.\n\n*Weaknesses*:\n- This is more of a minor issue, but some parts of the method section, particularly the mathematical formulation, are difficult to follow. The notation is heavy, with many variables introduced at once, which can make the equations hard to parse on a first read. A clearer presentation of the method, possibly with simplified or modular notation, would improve readability.\n- Another potential issue is that, on a first read, the method may appear to require the generation of multiple captions per image, which could be perceived as a limitation. It would be helpful to clarify early in the paper, either in the Introduction or Related Work, that datasets like MS-COCO already exist and provide multiple captions per image, making the method more accessible. Additionally, when MS-COCO is mentioned in Section 4.1, a brief description of its structure would help readers who may not be familiar with the dataset.", "was_revised": true, "final_justification": "The rebuttal effectively addressed my concerns, and the clarifications provided reinforce the validity and relevance of the proposed contributions. I thus maintain my accept rating.", "rebuttal": "## To Reviewer 4 (gRGm)\n\nWe thank the Reviewer gRGm for the detailed review and constructive suggestions. We appreciate your acknowledgement that our method achieves strong result, our ablation studies are valuable, and our evaluation is solid and thorough. Please find our answers to your comments and questions as follows.\n\n\n**[R4-1]** `This is more of a minor issue, but some parts of the method section, particularly the mathematical formulation, are difficult to follow. The notation is heavy, with many variables introduced at once, which can make the equations hard to parse on a first read. A clearer presentation of the method, possibly with simplified or modular notation, would improve readability.`\n\nThank you for highlighting this point. To address this concern, we plan to improve the clarity in the revised manuscript by:\n- **Modularizing the notation** around core conceptual units (e.g., alignment vs. reconstruction),\n- **Adding intuitive summaries** between mathematical blocks to guide the reader through key steps.\n\nWe will ensure that these revisions preserve mathematical precision while significantly improving readability. Thank you again for this constructive suggestion.\n\n---\n\n**[R4-2]** `Another potential issue is that, on a first read, the method may appear to require the generation of multiple captions per image, which could be perceived as a limitation. It would be helpful to clarify early in the paper, either in the Introduction or Related Work, that datasets like MS-COCO already exist and provide multiple captions per image, making the method more accessible. Additionally, when MS-COCO is mentioned in Section 4.1, a brief description of its structure would help readers who may not be familiar with the dataset.`\n\n\nWe thank the reviewer for the thoughtful comment. \n\nIn the revised manuscript, we will clarify early in the paper (Introduction or Related Work) that datasets such as MS-COCO **already** provide multiple human-written captions per image. This will help readers understand that our method does not require any additional caption generation. Furthermore, when introducing MS-COCO in Section 4.1, we will include a brief description of its structure to better support readers who may be less familiar with the dataset.\n\n\n---\n\n**[R4-3]** `Clarification on the paraphrasing process: The paper mentions paraphrase generation through augmentation, but the details remain unclear. Did you use a specific LLM to generate paraphrases? If so, which one, and under what settings? Or are you using a VLM to do this, conditioning on both text and image? If a VLM was used for paraphrasing, this could raise a concern about circularity, i.e., relying on VLM, that you criticized for poor compositional reasoning, to improve compositional reasoning of other VLMs. Clarifying this aspect would strengthen the methodological transparency of the paper.`\n\n\n\nThank you for raising this important point.\n\nIn our experiments, paraphrasing was performed using a **text-only LLM**, not a vision-language model. As mentioned in *supplementary material*, to generate paraphrased captions for our sentence-level alignment loss, we use the `gpt-4o-mini-2024-07-18`, and details are as follows:\n\n- Each original caption was provided **as text-only input**.\n- No image or visual features were used during generation.\n- We used a simple prompt (shown in Appendix Fig. 6), with **temperature = 1.0** and all other decoding parameters left at default settings.\n- For every caption in the COCO dataset, we generated **one paraphrased variant** offline before training.\n\nWe will include these details in the revised manuscript. \n\n\n---\n\n\n**[R4-4]** `Highlighting the originality of the contribution: While the reconstruction and alignment losses are motivated and well implemented, their novelty may not be immediately evident to readers who are less familiar with the literature. I recommend emphasizing more explicitly how your approach differs from prior work using reconstruction losses, specifically, by explaining that although similar techniques exist for general-purpose encoder training, they have not been applied to the problem of improving compositional reasoning in contrastive VLMs. Making this distinction more visible could better communicate the originality of your work.`\n\nThank you for this helpful suggestion. As you rightly pointed out, the core novelty of our work lies in introducing token-level reconstruction and sentence-level alignment objectives into the contrastive training of vision-language models (VLMs)—**not for general-purpose encoding, but explicitly to improve compositional reasoning**.\n\nWhile text reconstruction and text alignment objectives have appeared in prior work (e.g., for general-purpose language model training), to the best of our knowledge, **our work is the first to adapt these textual supervision for improving compositional generalization in VLMs trained with contrastive learning like CLIP**.\n\nWe acknowledge that this distinction may not be immediately clear to readers unfamiliar with the relevant literature. Accordingly, we will revise the manuscript to emphasize this point more explicitly in the Introduction and Related Work sections.\n\n\n---\n\n**[R4-5]** `More qualitative examples: The qualitative examples presented in the main paper are insightful. However, I suggest including additional qualitative examples, especially failure cases and edge scenarios, maybe in the appendix. This would help readers better understand the behavior of the model under different conditions and strengthen the empirical analysis.`\n\nThank you for this thoughtful suggestion. As per the reviewer's comment, we will augment the appendix in the revised manuscript with additional qualitative examples, including:\n\n- **Cases where both READ-CLIP and other baselines fail**, highlighting shared limitations in compositional reasoning.\n- **Cases where READ-CLIP fails but other baselines succeed**, which can reveal specific weaknesses unique to our method."}]

Author rebuttal:

Meta-review: The paper proposes a fine-tuning method for CLIP-like vision-language models that enhances compositional reasoning by incorporating two auxiliary objectives (a token-level reconstruction, which encourages the text encoder to capture intra-sentence structure by reconstructing a paraphrased caption, and a sentence-level alignment, which pulls paraphrase pairs together in the embedding space while pushing apart unrelated captions). Across the reviews, reviewers were positive about the paper being clearly written and well-motivated, and thought the proposed objectives were novel, technically sound and effective. They also noted the strong empirical performance across a variety of benchmarks.

Initially there were some concerns raised including limited evaluation on more recent benchmarks, lack of generalization analysis across diverse backbones, and concerns about the reliability of paraphrase generation. However, these were largely addressed in the rebuttal and discussion, with the authors providing extended experiments (e.g., on Winoground and zero-shot tasks), clarifying distinctions in related work, justifying design choices (e.g., T5 usage), and committing to camera-ready updates. For the camera-ready version, the authors should be to incorporate these important elements from the rebuttal/discussion (especially clarifications on paraphrase quality and terminology, updated evaluation results on Winoground and zero-shot tasks, and improved exposition of related work distinctions).

The AC recommends acceptance.
Decision: Accept (poster)