Natural Language Inference Improves Compositionality in Vision-Language Models
Authors: Paola Cascante-Bonilla
Yu Hou
Yang Trista Cao
Hal Daumé III
Rachel Rudinger
Venue: ICLR 2025 Poster (2025)
Keywords: text to image evaluation
image to text generation
natural language inference
sentence decomposition
large language models
visual question answering
question generation
benchmark
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Compositional reasoning in Vision-Language Models (VLMs) remains challenging as these models often struggle to relate objects, attributes, and spatial relationships. Recent methods aim to address these limitations by relying on the semantics of the textual description, using Large Language Models (LLMs) to break them down into subsets of questions and answers. However, these methods primarily operate on the surface level, failing to incorporate deeper lexical understanding while introducing incorrect assumptions generated by the LLM. In response to these issues, we present Caption Expansion with Contradictions and Entailments (CECE), a principled approach that leverages Natural Language Inference (NLI) to generate entailments and contradictions from a given premise. CECE produces lexically diverse sentences while maintaining their core meaning. Through extensive experiments, we show that CECE enhances interpretability and reduces overreliance on biased or superficial features. By balancing CECE along the original premise, we achieve significant improvements over previous methods without requiring additional fine-tuning, producing state-of-the-art results on benchmarks that score agreement with human judgments for image-text alignment, and achieving an increase in performance on Winoground of $+19.2\%$ (group score) and $+12.9\%$ on EqBen (group score) over the best prior work (finetuned with targeted data).

Published as a conference paper at ICLR 2025
NATURAL LANGUAGE INFERENCE IMPROVES COMPOSITIONALITY IN VISION-LANGUAGE MODELS
Paola Rascante-Bonille 1.2 Yu Hou!
Yang Trista Cao Hal Daumé III Rachel Rudinger University of Maryland, College Park Stony Brook University SUniversity of Texas at Austin
ABSTRACT
Compositional reasoning in Vision-Language Models (VLMs) remains challeng- ing as these models often struggle to relate objects, attributes, and spatial relation- ships. Recent methods aim address these limitations by relying on the semantics of the textual description, using Large Language Models (LLMs) to break them down into subsets of questions and answers. However, these methods primarily operate the surface level, failing to incorporate deeper lexical understanding while introducing incorrect assumptions generated by the LLM. In response to these issues, we present Caption Expansion with Contradictions and Entailments (CECE), a principled approach that leverages Natural Language Inference (NLI) to generate entailments and contradictions from a given premise. CECE produces lexically diverse sentences while maintaining their core meaning. Through ex- tensive experiments, we show that CЕCE enhances interpretability and reduces overreliance on biased or superficial features. By balancing CECE along the orig- inal premise, we achieve significant improvements over previous methods with- out requiring additional fine-tuning, producing wate-of-theeard results on bench- marks that score agreement with human judgments for image-text alignment, and achieving an increase in performance Winoground of +19.20 2% (group score) and + +2.9% on EqBen (group score) over the best prior work (finetuned with targeted data). Project page: https::/cece-vlnn wwt.wwww.b..mmmm.mm..... github.ih..o.o
INTRODUCTION
Trained with internet-scale data, Large-scale Vision-Language Models (VLMs) often struggle to re- late objects, attributes, understand spatial relationships, and grasp subtle changes in meaning due
to small variations in images or word order (Thrush et al. 2022 Diwan et al. 2022 Wang et al.
20236 Bitton- et al.) 2023] Yuksekgonul a 2023 Tong al. [0224] Saxon et al. 2024
Fu et al. 2024). With a seeming inability to handle semantically modular scenarios, their opaque
nature makes difficult to understand their decision-making
processes (Dziri et al. 2023 Kamath
ct al. 2023 2024). Furthermore, internal biases play major role in affecting the model s perfor-
mance across various tasks (Zhou et al.) 2022 Tiong et al.) 2024; Howard et al. 2024 Fraser &
Kiritchenko 2024 Raj al., 2022).. Recent works have explored ways mitigate these issues by
breaking down a problem into smaller tasks. Typically, a Large Language Model (LLM) is prompted
to create small programs (i.e.. Visual Programming (VP) (Gupta & Kembhavi] 2023 Hu al. 2023 Surís et al., 2023] Subramanian et al.) 2023 Cho et al. 2023b; Koo et al.) [024 Hu et al. [0024b] or deconstruct the textual description in set of validation questions with their corresponding ex- pected answers (i.e., Sentence Decomposition via Semantics (SDS) (Cho et al. 2023a Wu et al., 2023 Yarom al., 2023 Mitra al., 2024 Zhang et al., 2024 Wan al.] 2024). While these methods provide interpretability, they also tend to degrade the VLM performance when evaluating challenging benchmarks that introduce pairs of images and captions that require extensive real-world
knowledge and reasoning (Lin al. 2024).
Consider Figure [ the first block shows two images with bananas. If we use a VLM to compute the likelihood of answering "yes" given each image and the text "there is a banana split", ", the VLM incorrectly assigns a higher probability to the second image. When decomposing the sentence through SDS, the LLM will output sentences like; "there is banana" " and "the banana is split" (output examples taken from Cho et al. (2023))) Although the SDS decomposition seems correct, it preserves the same lexical surface of the text and is unable to incorporate additional information

---

Published as a conference paper at ICLR 2025
Original Caption/maage Pair
Sentence Decomposition via Semantics (SDS)
Caption Expansion via CECE
Test (T)
Pisss /T
PiYes T,
TTs
PiYesT,
PoYes IT.
there banana
90.4%
93.1%
62.1%
43.0%
plcces
the banana split
45.0%
79,9%
leatana's pazl spen
61.9%
Test (T)
PoYes (T
Po I))
(hene is spiit bunana
60.6%
<
64.6%
SDS Score
<
86.2%
Final Score
62.0%
44.9%
Original Caption/tmage Paie
Sentence Decomposition via Semantics (SDS)
Caption Expansion via CECE
PiYs P
PCYes T,
(Tx
PL IT,
P/
something meiag
45.3%
perwon mrtion
94.1%
97.18
person
1.3%
foum place seuther
object is being novedd
58.79
PoYes I'T,
T,
recing it over
24.3%
Finsll
1.7%
CRCR Final Score
75.5%
Figure 1: Examples from Winoground dataset. The first column shows the output of LLa Va-Va 1.6 when computing the likelihood of answering "yes" given the image and text. The second column shows the sentence decomposition proposed in prior work (SDS), which follows the original cap- tion semantics. The third column shows our proposed Caption Expansion with Contradictions and Entailments (CECE). In all cases, the model is only allowed to evaluate one image and text at time.
that could be leveraged by the VLM to make correct prediction - both images contain bananas, but the VLM might have learned stronger correlation of banana split being dessert. Likewise, if we look at the second block in Figure [ and follow the same process, the SDS output will not only preserve the same lexical surface, but will make wrong assumptions and introduce some of the biases present in the LLM (e.g., racing might have strong correlations with cars).
To address these limitations, we introduce Caption Expansion with Contradictions and Entail- ments (CECE), principled approach that leverages Natural Language Inference (NLI) to improve the compositional capabilities of VLMs. With CECE, we instruct an LLM to use NLI which is used to determine the relationship between two sentences, premise and a hypothesis (Bowman et al. 2015) and generate entailments (hypotheses that logically follow from the premise) and contradictions (hypotheses that logically incompatible with the premise). In this way, the LLM is instructed to produce lexically diverse sentences while preserving the underlying meaning of the original captions. It is important to note that while the outputs generated by SDS be considered subset of entailments, CECE expands the scope by generating a wider variety of captions that cap- ture nuanced relationships beyond what SDS would typically output, including both entailments and contradictions. This allows the L.M to leverage common sense and break down the captions using world knowledge, while also mitigating the hallucinations and biases introduced by the L.M.M
Revisiting Figure CECE generates sentences that are lexically different from the original caption but preserve meaning (e.g.. "split" entails dividing things "in one or more pieces"). By expand- ing the captions with both entailments and contradictions, the LLM incorporates world knowledge and common sense while mitigating hallucinations and biases. For example, "racing over" is parsed into "a person in motion" " and "an object being moved from one place another". which entails a different perceptual inference the caption, and reduces the likelihood of stereotypical associations. As whole, CECE goes beyond the lexical boundaries of SDS, introducing richer contextual information for improved reasoning.
To leverage both entailments and contradictions generated via CECE, we evaluate the likelihood of the VLM answering "yes" given the image and each entailment, and the likelihood of the VLM answering "no" given the image and each contradiction. These scores are then aggregated using weighting value to balance the contributions of both entailments and contradictions. With CECE, we incorporate both positive and negative reasoning cues: entailments provide semantic inclusion (focusing on subset relations), while contradictions provide semantic exclusion (focusing on subset complements). In addition, we aggregate the VLM likelihood of answering "yes" given the im- age and original caption. We found that including the original caption provides additional context for balancing out the VLM outputs. In a way, the original caption serves as a direct reference that helps the model minimize the risk of semantic drift, where caption expansions may diverge from the intended meaning. Our results show that incorporating CECE along with the original caption fur- ther improves the amage-to-tery and text-to- text-to-image alignment, providing lexical diversity, improved semantic reasoning, and a more interpretable output, without fine-tuning the models, which may compromise their zero-shot capabilities.
2

---

Published as a conference paper at ICLR 2025
Our contributions are summarized as follows: a) We propose Caption Expansion with Contradictions and Entailments (CECE), a principled approach that leverages entailments and contradictions to preserve the semantic meaning while providing lexical diversity of text descriptions. b) We show that CECE significantly outperforms prior decomposition methods, obtaining 47.5% on Winoground (group score) and 47.9% Eqßen (group score) without finetuning. c) We conduct extensive experiments on benchmarks that score agreement with human judgments of alignment for image- text alignment. d) We provide thorough ablation studies and analyses to evaluate the performance of our method under various conditions, and introduce simple ensembling approach that effectively boosts the accuracy when associating each image-text pair.
RELATED WORK
Single-caption Scoring frameworks. Commonly used for evaluating the alignment between text
and images in VLMs (Dai et al. 2023 Liu et al. 2023b [024 Bai et al. (023), these approaches
include similarity scores derived from multimodal encoders (Radford et al. [202I), as well as text
similarity metrics based on image captioning models (Li al.] 2023). However, summarizing the relationship between text and images using single embeddings often fails to capture the semantic granularity needed for fine-gre fine-grained image-text alignment (Zhao al. 2024). Moreover, these met- rics are often uncalibrated and may obscure important nuances; for example, a particular CLIPScore value might indicate good match for pixel art but be considered poor for realistic images, which might affect image-text alignment compared to the human judgments of alignment in text-to-image evaluation metries. More recently, [inn et al.] (2024) introduced VQAScore, which leverages a VLM to computes the likelihood of given image-caption pair, by re-writing the caption as a binary ques- tion ("yes) [no",, yielding significant improvements. Our approach builds on this while aiming to provide a more detailed and semantically diverse evaluation. By generating entailments and con- tradictions, we introduce a mechanism to understand the model's nuanced response to positive and negative cues, thereby addressing some limitations of prior single-embedding scoring methods.
Sentence Decomposition via Semantics (SDS) frameworks. Given the limitations of single- caption scoring frameworks, recent works have explored more sophisticated evaluation methods based on sentence decomposition and semantic analysis (Khan et al., 2023 Hu et al.) 2023] Cho et al. 20236). These approaches aim provide a more comprehensive and fine-grained evaluation of text-to-image and image-to-text alignment. Typically, an L.M is instructed to generate subsets of validation questions and expected answers that VLM can evaluate (Wan et al. (0024). Similarly, Sanders et al. (2024) uses entailments to improve video question answering. While these methods provide finer semantic analysis and interpretability to the evaluation process, SDS methods typically produce outputs that are direct entailments of the original caption. Furthermore, Yarom et al. (2023) introduces VNLI, an approach that finetunes a model that receives an image and set of entai ments and contradictions, where the contradictions are defined as identified question-answer pairs with the lowest VQA score. We instead directly instruct the LLM to output entailments and contradictions with the input caption premise, providing strong prior for caption expansion and exclusion, without finetuning the models.
Chain-of-Thougee Prompting frameworks. Recent work has shown promising results when in- corporating Chain-of- Chain-of-Thought (CoTT) prompting (Wei et al. 2022) to enhance compositional rea- soning in challenging vision-language scenarios, using large- VLMs (Dubey al.] 2024 Achiam eet al. 2023: Hu et al. 2024a). Notably, Zhang et al. (2024) introduces CoT for multiple image- to-text matching, through contrastive approach for comparative reasoning. Similarly, two-step prompting strategy is introduced to generate descriptions of the given image, which is then used
by the model to answer specific questions (Wu et al. 2023 Ossowski al., 2024). These works prompt the VLM with multiple images and instruct them to choose the correct one. [Mira et al. (2024) propose to generate scene graphs as an intermediate reasoning step, and instruct the VL.M to pick from two given captions given the image and scene graph. This also reformulates the problem as a multiple-image-t--eee one image and text at a time- we argue that this setup closely aligns with real-world scenarios, where compositional understanding must be robust without the benefit of direct comparisons be- tween multiple images captions. By evaluating each image-caption pair independently. ensure that the model's 's reasoning is not influenced by relative comparisons, thus providing a more realistic assessment of its compositional capabilities.
matching task. In contrast, our approach is only allowed to evaluate
3

---

Published as a conference paper at ICLR 2025
3 METHODOLOGY
To effectively address the limitations of Vision-Language Models (VLMs) in handling complex compositional visual-textual relationships, we introduce CECE: Caption Expansion with Contradic- tions and Entailments. Our approach leverages Natural Language Inference (NLJ) to systematically generate entailments and contradictions for each image-caption pair, capturing the deeper mean- ing of the text and more interpretable metric for image-to-text and text-to-image evaluation and alignment. We describe contradictions via CECE (Section 3.1). We then explain the likelihood computation for each caption expansion (Section 3.2p. Finally, describe the score-balancin mechanism the contributions of entailments, contradictions, and the original captions into a unified evaluation
proposed method, beginning with the generation of entailments and
to integrate
framework (Section 3.3
Captien:
E. Beranz' pod leopen
VOA
preces
Finul
0.64
Barera is it places
fIIN
Sueralape spon
Caption Expansion
Baszn iincaspton
fn
Then gpil hontss
Bunan is Nee trans preces
Basasa one Heor
T-Thew oupii
hesess
fyear
Busaniss pool
Isopen
Figure 2: Complete pipeline of our proposed approach. CECE provides diverse semantic inclusion and exclusion given a caption premise. VLM is then used to compute the likelihood of the captions generated via entailments and contradictions. The scores finally balanced along with the original image-text results to avoid semantic drift and enable better alignment.
Given an evaluation dataset that consists of image-caption pairs, we define: X = (T, as an image-caption pair, where T represents the textual caption and / is the image to evaluate.
3.1 CAPTION EXPANSION
To enrich the semantic representation via caption expansion, we instruct an LLM, denoted as fE- to generate entailments and contradictions (i.e. expansions) based on the original caption T (see Figure - This process outputs two subsets: the entailment set E, consisting of hypotheses that logically follow from T, and the contradiction set C, consisting of hypotheses that are logically incompatible with as follows: (E, C* = fT(T). Here, E represents the subset of generated entailments: E {EL, ......, En ). and C' represents the subset of generated contradictions: C'
Each element in E and C aims to provide a diverse but grounded derivation of the original caption through semantic inclusion and exclusion.
(c), ......, Cn)
3.2 LIKELIHOOD COMPUTATION
Following Lin et al. (2024). we use a Vision-Language Model (VLM), denoted as fVLM to evaluate image-text pairs. To standardize the input for the VLM, we use function q(-) that converts a given text t into a question format, which enables the model to assess the visual-textual alignment a consistent manner. Specifically, g(-) generates yes/no question that captures the essence of the original statement. If we take the example in Figure 2, the caption t "THERE IS A BANANA SPLIT" is formatted q(t) "Does THERE A BANANA SPLIT can be observed in the image? Answer yes or no".
We proceed to evaluate all captions generated in the previous step follows:
Prompt details are in Appendix A.1
4

---

Published as a conference paper at ICLR 2025
(1) For entailments, compute the likelihood of answering "yes" given the image-text pair:
(Li 7) = P("yes" (,e()))
(1)
(2) For contradictions, we compute the likelihood of answering "no" given the image-text pair:
JVLM (c, I) = P("no" I,q(c,))
(2)
(3) For the original caption, we compute the likelihood of answering "yes" as follows:
f (T. P("yes" [[" I,
(3)
This step allows us to assess the model's agreement with both positive cues (entailments) and neg- afive cues (contradictions), probabilities are normalized such that P( P("yes") P("no") = 1. By evaluating these three compo- nents, we better assess the VLM's ability align visual content with textual information, capturing nuanced relationships across different semantic variations.
well as its consistency with the original caption. In all cases, the
3.3 BALANCING SCORES
To integrate the information from entailments, contradictions, and the original caption, we employ two-step balancing process using hyperparameters a and 0z,, to obtain more comprehensive evaluation of the model's ' performance while avoiding semantic drift. By carefully balancing the contributions from entailments, contradictions, and the original caption, this approach ensures that the final assessment remains grounded in the intended meaning of the original text, minimizing the risk of unintended shifts in interpretation that could arise from generated variations (refer to the nuanced scores to the right in Figure 2
First, we define the aggregation function S(.), which computes the average score across all elements in given set. For example, to score all entailment outputs given the set E' containing M elements, the aggregation function is given by:
S(E, I)
pent (e
(4)
M
and score all contradiction outputs in similar way:
S(C,1) = M Σ (osM(
1
(5)
Then, we compute a weighted sum the entailment, contradiction, and the original caption score:
S(X) 02 - (an S(E, I) (1 (1-o1) S(C, [)] 02) S(T,
where S(T, 7) represents the VLM score for the original image-caption pair.
This two-step mechanism allows for a flexible adjustment of the importance assigned to entailments, contradictions, and the original caption in the final assessment, balancing their contributions. By effectively incorporating both positive and negative reasoning cues along with the original context, we aim to achieve a nuanced evaluation that better aligns with human judgments, avoids semantic drifts, and reduces overreliance on biased or superficial features.
EXPERIMENT SETTINGS
Baselines. We compare CECE with a wide range of baseline methods divided into three categories.
1) End-to-end models (i.e., Single-caption Scoring frameworks) including CLIPScore (Radford)
ctt al. 2021), BLIP2 (Li al. 2023), VQAScore (Lin et al. 2024). VIEScore (Ku et al. 2023)
and Gvvll ŌZhag et al. 2023); 2) Visual Programming (VP) frameworks (VisProg Gupta
& Kembhavi 2023), ViperGPT Suris et al. 2023). VPEval (Cho et al. 2023b)); 3) Sentence De-
composition via Semantic (SDS) frameworks (VQ2) Yarom et al. (2023), , DSG (Cho al. 2023b
CCoT (Mira et al., 2024)). Note that VP and SDS frameworks use an LLM for program instruc-
tion sentence decomposition, including ChatGPT (OpenAl 2023) GPT4 (Achiam et al.
2023
5

---

Published as a conference paper at ICLR 2025
FlanT5 (Chung et al. 2024), and several VLMs. including ViLT Kim al. 2021 OWL-ViV (Min-
derer et al. 2022). ) CLIP [Radford et al. 2021 GLIP (Li et al. 2022). GroundDINO (Liu et al.
2023c), LLa VA-15 (Liu et al. 2023a), ALaVA-1.6
(Liu al. 2024). PaLI-17B (Chen et al. 2022
and the finetuned model introduced by
Lin et al.
2024) CLIP-Flan" 13-1111
Implementation. We Llama3.1 70B (Dubey al.
2024) our LLM for caption expansion
through NLI. We further evaluate CECE on different VLMs (BLIPv2 (Li et al.) (023), Instruct- BLIP (Dai et al., 2023), LA- (Liu et al., 2023a). LLaVA-1.6 (Liu al. 2024g) and incorpo- rate a soft-assembling method that balances the results scores of different models, by balancing the scores from entailments and contradictions (VLM scores from LLaVA-115), and the original caption
(VLM scores from LLaVA-1.6). We use = 0.5 and a2 = 0.6 in all experiments. Additional
details are included Section 6
Tasks and Benchmarks. We evaluate the compositional capabilities of CECE in three differ- ent tasks. 1) Image-text matching evaluation through binary retrieval tasks, which require deter- mining the best caption from a pair of candidates for given image, well as determining the best image from a pair candidates for given caption. We report results on two benchmarks (Winoground (Trrush et al. 2022). EqBen (Wang et al.] 2023b)) and the performance is evaluated using three metrics: (i) a text score, which assesses the model's ability to identify the correct cap- tion for given image; (ii) an image score, which measures the model's accuracy in selecting the appropriate image based on a provided caption; and (iii) group score, which evaluates the success- ful matching of both pairs. Score agreement with human judgments of alignment for image-text alignment, using images generated from complex text prompts. We report results on five text-to- image evaluation benchmarks DDrawBench (Saharia al. 2022). EditBench (Wang al. 2023a), COCO-T2I (Lin et al. 2014), TIFA 160 (Hu et al.) 2023). Pick-a-Pic (Kirstain et al. (023)) ). 3D aligmment, which assesses the human judgments of alignment for text-to-3D asset generation. We report results on the StanfordT23D (Wu et al., 2024) benchmark with the human ratings collected by Lin et al. (2024).
Table 1: Performance on challenging compositional benchmarks that require multi-hop reasoning. Tools indicate the vision and language models used for inference. LLM indicates the large language model used for generating the visual programming output or sentence decompositions. DSG is the only method that uses a model fine-tuned for this task. Llama3. 1 indicates the 8B parameter model.
Method
Tooks fook-fVLM
LLM-fE
Winoground
EqBen
Text
Image
Group
Text
Image
Group
Random Chance
25.0
25.0
16.7
25.0
25.0
16.7
Human Evaluation
89.5
88.5
85.5
End-to-end models
CI CLore (Radford
2021
CLIP-1,-14
27.8
11.5
35.0
35.0
25.0
BLIP2m
(L al 2025
BLIPP2
42.8
22.0
18.3
48.6
43.6
35.0
VQAScore
Lil et
2024
InstretBLIIP
44.5
42.8
28.5
49.3
58.6
38.6
VQAScore
1.th
207.4
1LvVA-1.5
41.3
45.0
VQAScure Lib el m
20144
LLaVA-166
46.8
45.8
31.3
46.4
54.3
32.9
VIEScore eu
GPt4-Vision
39.3
40.0
GPT4V-Evar Zilang 21
2023
Gisi-Vision
44,5
49.0
36.3
42,9
40.0
35.0
Visuul Progromining
VisProg (Gupta & Kembhavi
2023
V.L.T WTT
ChaiGPT
3.5
3.5
3.5
7.9
7.9
7.9
ViperGPP (Sms
CLIP, BLIP, GLIP
ChatGP
7.8
7.8
4.3
4.3
4.3
VPEval Cho acal. 20230
BLIPº2, GroundDINO
ChaGGT
12.8
11.0
34.3
25.7
21.4
Sentence Decomposition via Semontics (SDS)
DSG
Cho et
2023g
LLaVA-1.5
Llama3.1 1
5.7
9.5
3.7
10.0
14.3
6.4
Che
2025%
Llama3.1
4.5
10.2
10.7
VQ2
Yarom dF
A724
LLAVA-T5
FlanTS
14.0
27.3
10.0
22.9
40.7
20.0
207258
26.4
DSG
21
LLavA-16
Llama3.1
45.8
45.8
31.3
47.1
44.3
32.1
DSG
CLIP-FlanTS-IIR
38.3
45.7
CC3T
Milra m
2114
LLavA-1.5
GPT4
39.8
37.3
22,3
VQ2
farom cla.
PaLI-17B
FlanTs
Caption Exponsion with Contrudictions and Enteilments (CECE)
CECE (Ours)
BLIPv2
Llama3.1
29.8
39.3
30.0
43.6
21.4
CECE (Ours)
InstrctBLIIP
Llama3.
37.5
46.3
28.8
41.4
57.1
34.3
CECE (Ours)
LLaVA-1.
Llama3.1
47.7
49.7
48.6
CECU (Ours)
LLaVA-1.6
Llama3,1 1
48.0
57.5
38.7
50.7
64,3
40.0
CECE (Ours)"
LLaVA- LLaVA-1.6
Llama3.1
50.0
53.5
39.0
53.6
57.1
40.7
CECE (Ours)
L.5VA
L.ama3.1 I
551.3
55.3
41.0
58.6
57.9
41.4
CECE (Ours)
LLaVA-1.6
Llama3.1 1
$2.0
61.3
42.8
58.6
64.3
47.1
CECU (Ours)*
LLaVA-155 LLaVA-1.6
Llama3.1
55.0
61.3
47.5
57.9
65.0
47.9

---

Published as a conference paper at ICLR 2025
5 RESULTS
Image-text matching evaluation through binary retrieval tasks. We conduct experiments Winoground and EqBen. Results are shown in Table [ The entailments and contradictions gen- erated by CECE uation for a wide range of visual-language model architectures. We demonstrate that our method outperforms prior works (including single-caption scoring methods (i.e. end-to-end models), visual programming, and sentence decomposition approaches that also leverage LLMs). For a fair compar- ison, took the best SDS method (DSG) and their end-to-end framework using Llama3. I and LLaVA-1.6. We also run DSG with the finetuned method introduced by Lin et al. (2024). Note that while CECE outperforms all other methods under similar conditions (one-LLM, one- (LM), the best results are obtained through our score- balancin approach CECE* which leverages both LLaVA-1.5 (scores from entailments and contradictions) and LLaVA-1.66 (scores from the original caption).
be applied wide variety of VLMs, this allows for comprehensive eval-
Table 2: Performance on benchmarks that score agreement with human judgments of alignment for image-text aligmment. Tools indicate the vision and language models used for inference. LLM indicates the large language model to generate the sentence decompositions. Note that none of the models have been specifically fine-tuned for this task.
Method
Tool--vva
LLM-fi
DrawBench
EditBench
COCO-T21
TIFA160
Pick-a-Pic
End tond-tt-enddd models
CLIP-L-14
BLIP2mM CL P0Z3
BLIPv2
60.5
68.0
70.7
57.5
80.0
VQAScure
2014
InstrciBLIP
82.6
75.7
83.0
70.1
83.0
VQAScore Li al 2014
LLaVA-1.5
82.2
70.6
79.4
66.4
76.0
VIE.Score ei ml 2023
Vision
63.9
78,0
GPT4V-Eval (Zhing
2023
GPT4-Vision
Sentence Decomposition vio Semantics (SDS)
VQ2 Yarom
2023
LLaVA-115
FlanTS
52.8
52.8
47.7
48.7
73.0
DSG Cho et al.
LLaVA-1.5
ChaGPT
78.8
69.0
76.2
54.3
70.0
VQ2 Yamo el 2023
Pal.1-17B
FlanTS
82.6
73.6
83.4
ata 20243
Lluma2
Caption Expansion with Contradictions and Entailments (CECE)
ChCR (Qur)
InstracBBI IP
1lama3.1
85.4
76.7
81.4
69.3
84.0
CECE (Ouns)
LLaVA-155
Llama3.1
87.3
75.6
81.3
68.9
86.0
Cbсв (Ours)
LLVVA-16
Llama3.1
86.3
75.9
83.8
70.4
83.0
(Ouns)
76.4
Score agreement with human judgments of alignment for image-text alignment. We show results on five text-to-image evaluation benchmarks in Table These results measure the correlation of each method score with human judgments of alignments for an image generated based on a textual prompt. Human ratings are given on a 1-to-5-Likert scale or by assigning binary match- or-not label. We show AUROC for DrawBench, EditBench, and COCO-T2I, pairwise accuracy for TIFA160, and binary accuracy for Pick-a-Pick CECE consistently outperforms all prior scoring approaches, indicating that caption expansion via NLI better aligns with human judgments when evaluating text-to-image generation methods.
Table 3: Performance on benchmarks that correlate 3D alignment with human agreement.
Method
Pairwise Ace
Pearson
Kendall
End-to-end models
CL.IPScore
61.0
48.1
32.6
BI. VP2ooee
56.6
34.3
23.4
InstructBLIP
68.0
59.5
47.5
LLaVA-1.5
64.9
55.8
40.8
Finetuned on human feedback
ReageReward 66.3
57.1
43.6
PickScore
60.1
41.3
30.3
HPSv2
55,9
31.5
21.9
Caption Expansion Contradictions and Entailments (CECE)
w/ InstructBLIP
68.5
64.0
48.4
w/ LLAVAL5
65.3
57.4
41.8

---

Published as a conference paper at ICLR 2025
3D alignment with human agreement. We show results in 3 We report the pairwise accuracy along with the Pearson and Kendall coefficients, which assume a linear correspondence between human ratings and metric scores. We follow the setting proposed by Lin et al. (2024) and show that CECR consistently outperforms the base model (I.LVA--1.5.
6 ANALYSIS
We conduct an in-depth analysis of CECE through multiple perspectives, including detailed break- downs Winoground and Eqben benchmarks, lexical diversity, semantic drift, and the impact of incorporating entailments and contradictions, Moreover, we demonstrate the robustness of CECE across different model architectures and present comprehensive ablation study to understand the importance of each component in our approach.
Detailed results on Winoground. We show fine- suee-graind results on tags provided by Winoground in T Each sample is grouped per skill category and can include multiple skills. We compare our method against the results from the base end-to-end models, since they outperform prior work based on SDS. Notably, CECE not only outperforms objects and relations from the linguistic side, but it also outperforms in cases where the images need to be interpreted non-literally due to idiomatic uses of language in caption. It also outperforms the base models when a symbolic description must be understood to make a correct prediction (e.g., typically in non-natural images, such as drawings or illustrations).
Table 4: Detailed analysis on Winoground. Results are grouped by linguistic (2) and visual (v) tags. We report results using DSG / CECE with LLaVA-1.6, and DSG* / CECE with LLaVA-1.5 (for entailments and contradictions) and LLaVA- (for the given caption).
Object:
Rebation:
Symboliey
Pragmaticsy
Imagee
Group
Text
Group
Text
Imsae
Group
Group
Group
Human
50.7%
88.65
59.27
86.70
76.92
57.69
96.43
92.86
92.86
41.15
InstrucDDLPP
42.5
49.7
27.7
34.3
33.9
20.2
65.4
38.5
34.5
31.7
21.9
25.0
29.2
8.3
LL.VA-1.5
43.8
33.9
46.3
38.5
26.0
36.6
LL-VA-116
48.2
43.8
40.88
27.9
65.4
46.2
38.5
1..V6-1.5-1.
54.3
43.9
42.9
30.9
53.8
DSG
$1.0
CEcR (Ours)
51.5
42.5
53.8
Ceris(Cumm))
On the complexity of image-text matching. A key challenge of Winoground that the captions are also ambiguous; for example, in the sentence "ir hatched before it was eaten". " "ir" could refer either to the egg or the animal inside the egg. Previously identified by Diwan et al. (2022). we show results the taxonomy of Winoground schemes in Table 5 We compare LLa VA-1.6 VQAScore outputs and DSG scores against CECE. For a fair comparison, we report the results of DSG and CECE using Llama3.1 and LL. Notably, DSG outperforms the base model and our method when evaluating samples tagged as NonComposit ional. These sample pairs are actually not semantically compositional of another since they do not contain semantic entities. On the other hand, our CECE shows stronger results for all other tags, where higher score is expected for the correct image-text pairs when the image is difficult parse (e.g., objects are small or blurry), the wording of the caption makes it difficult to parse (e.g., "yellow duck shoes on"), or common-sense reasoning to match the correct image-text pair is required (e.g., "together hammering something" vs. "hammering something together"). "). It is important to note that in image-tex matching datasets, the captions are syntactically similar, with the key difference of contextual or semantic alterations by swapping objects, relations, or both. Similarly, sentences like "another organism was harmed by a plant, and that plant broke the organism into pieces" " introduce ambiguity regarding the subject, since "another organism" " could be interpreted as either an animal or possibly another plant. Results show that CECE is particularly performant also where world knowledge is required.
Lexical diversity and semantic drift. We conduct comprehensive experiments to measure the lexical diversity between different semantic decomposition methods (SDS) and CECE. We compute the Jaccard Similarity (Real& & Vargas) 1996) between the Winoground caption and the LLM outputs for each technique. Results show a higher similarity between captions generated via DSG (with a score of 0.53) in comparison with captions generated via CECE (with score of 0.32).
8

---

Published as a conference paper at ICLR 2025
Table 5:
Breakdown analysis with Winoground categories from Diwan al. (0222) For fair com-
parison, we report numbers for DSG and CECE under similar conditions (i.e., same LLM and VLM).
LLAVA-1.6
DSG
CECE
Text
Image
Group
Text
Image
Group
Text
Image
Group
Non Comp.
60.0
50.0
40.0
66.7
53.3
46.7
66.7
53.3
43.3
Ambig. Correct
30.4
28.3
17.4
34.8
34.8
21.3
41.3
43.5
26.1
Visually Difficult
47.4
34.2
23.7
26.3
28.9
15.8
39.5
50.0
28.9
Unusual Text
40.0
32.0
26.0
32.0
38.0
22.0
40.0
50.0
34.0
Unusual Image
50.0
41.1
33.9
44.6
33.9
19.6
50.0
58.9
33,9
Complex Reasoning
32.1
32.1
19.2
28.2
28.2
16.7
29.5
46.2
20.5
Human Validation Caption Expansion. We validate the quality of the entailment and con- tradiction captions generated by CECR as described in subsection 3.1 We randomly selected 90 samples and manually annotated whether the generated captions could be entailed from the original caption. Using a Likert scale ranging from (definitely not likely) to 5 (definitely likely), the entail- ment captions received an average score of 4.7, while the contradiction captions received an average score of 1.7. This indicates that the entailment and contradictions are both accurately generated.
Semantic drift and balancing scores. While the lexical diversity CECE provides benefits the image-text and text-image matching, we also observed a level of divergence from the semanties of the original caption. We refer to this as "semantic drift". phenomenon present in LLMs that describes the degradation of text generation quality. [Spataru et al. (2024) defines this as degree of separation generation quality. We mitigate this issue by incorporating the balancing score ap- proach described in subsection 3.3 We show in Figure 3 how different values balance out the contribution between entailments and contradictions (ax)) and the given caption (a2), with a visible trend of highest performance in the middle for each scoring metric.
0.55
0.40
0.50
Text Score
Groupp Score
0.45
0,-0.0
0,-0.0
0,50,4
0.50
0,-0.4
0.40
0.-0.5
0.50.5
0.45
0.00.6
0.-0.7
01-07
01=1.0
0.25
0.0
0.2
0.4
0.6
0.8
1.0
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.55
0.60
0.50
0.55
0.40
Score
000.0
0.-0.4
a.40.4
0,=0.0
Group
0.35
0.-0.4
0=0000
0.40
0.0.5
0.45
m-0.5
0.6.66
0.-0.6
0.-0.6
0.35
-07
0.40
0.25
m=0077
0.0
0.11.
0.-10
0.30
0.0
0.2
0.6
0.8
0.2
0.4
0.6
0.8
1.0
Figure 3: Balancing scores from entailments, contradictions and the original caption. We show how the values affect the matching performance in Winoground. The first row shows results using LLLVA-116 only. The second row shows results using our soft-ensemble approach that balances the scores from LLaVA-1.5 (for entailments and contradictions) and LLaVA-1.6 (for the original caption). Best viewed in color.
Importance of each component and soft-ensembling, We ablate each component our proposed method in three ways. We show in Teble Winoground and Eqben when using (i) only en- tailments, (ii) entailments and contradictions, (ii) entailments, contradictions and the given caption. Our results show that entailments alone outperform prior methods, while all components progres-

---

Published as a conference paper at ICLR 2025
Table Importance of each component. While entailments alone significantly improve the compo- sitional scoring performance, progressively adding each proposed component yields the best match- ing score. Results with Llama3. and LLaVA-1.6.
Wineground
EqBen
Entail.
Contrad.
Caption
Text
Image
Group
Text
Image
Group
49.3
47.3
36.0
45.0
57.9
33.6
31.7
38.0
24.2
20.7
27.1
13.6
46.8
57.5
39.0
47.1
60.0
38.6
52.0
61.3
42.8
58.6
64.3
47.1
Table 7: Mixture of VLMs. We show results of balancing the scores of entailments and contradic- tions (or) and the given caption (a12) with different models.
Winoground
EqBen
InstructBLIP LLaVA-1.5
LLaVA-1.6
Text
Image
Group
Text
Image
Group
49.3
55.5
41.0
48.6
58.6
40.7
51.5
57.3
42.5
51.4
57.9
40.0
55.0
61.3
47.5
57.9
65.0
47.9
sively boost the final matching performance. We further explored different combinations to mitigate the "semantic drift" problem by balancing the matching scores from different models. We show in Table how combining different models boosts the final compositional evaluation.
CONCLUSION
In this work, introduce Caption Expansion with Contradictions and Entailments (CECE), prin- cipled approach designed to enhance compositional reasoning vision-language models. CECE leverages Natural Language Inference to generate diverse entailments and contradictions, aimed to expand the semantic understanding of textual descriptions. We conduct extensive evaluations across multiple compositional benchmarks, including Winoground and EqBen, and demonstrate that CECE significantly outperforms prior methods without requiring additional fine-tuning, achieving notable results in alignment with human judgments for text-to-image evaluation. Through comprehensive analysis, we show that CRCE enhances interpretability and provides balanced semantic repre- sentation, which is crucial for nuanced image-text matching and reasoning. Our results indicate that combining both entailments and contradictions allows vision-language models to consider both inclusive and exclusive semantic cues, leading to interpretable and less biased compositional reason- ing. We encourage future work interpretable multimodal frameworks that can leverage structured semantic expansions across diverse domains and tasks.
Broader Impact. CECE effectively improves the interpretability and robustness of vision-language models, and contributes to fairer Al systems that align more closely with human reasoning, reducing its reliance superficial correlations. CECE also has the potential to improve a wide range of applications, such as assistive technologies for people with visual impairments, educational tools, and creative content generation. However, like other works that leverage LLMs, CECE also poses risks related to the generation of misleading content or misuse in malicious contexts. The enhanced contextual interpretation and semantic descriptions could be used to make fabricated altered visual content more convincing, amplifying the risks associated with misinformation. We encourage the responsible use of frameworks like CFCF, with an emphasis on transparency, ethical guidelines, and mechanisms for monitoring and mitigating potential misuse.
A Acknowledgments. We thank the anonymous reviewers as well the members of the University of Maryland CLIP lab for their thoughtful and thorough feedback. This work was supported by the NSF CAREER Award No. 2339746 (Rudinger). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 088744 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arxi2:2388. /2966, 2023.
Nitzan Bitton-Guctta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! a mark of synthetic and compositional images. In Proceedings of the IEEE/CVF International 2023.
bench-
Conference on Computer Vision, pp.
Samuel R. Bowman, Gabor Angeli, Christopher Pouts, and Christopher D. Manning, A large anno- tated corpus for learning natural language inference. In Lluís Mårquez, Chris Callison-Burch, and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pp. 632 642, Lisbon. Portugal, September 2015. Association for Computational Lin- guistics. doi: 18.18653/ 1/235//11D1--00555 URL hwwp:htt:////ttt //ac::::/cc antho anthoology..rggD155-1-
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.00 06794, 2022.
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine- grained evaluation for text-image generation. arXiv preprint arXiv:2310.1821 2023a.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-to-image generation and evaluation. In NeurlPS, 2023b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cnstructionffietuuaa lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general purpose vinion-language models with instruction tuning, 2023. URL (ttth:ts: http://axivv..rr. aba / 2305. 06500
Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald. Why is winoground hard? investigating failures in visuolinguistic compositionality. Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pp. 2236-2250, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: pettps//aaclon http:s/////claaathoooo...oo org 2022 http:/22222mmmmmmmaam... www.p-mmin.....m....mmmmmmmmm 143
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of trans- formers on compositionality. In Thirty-seventh Conference on Neural Information Processing wwwpeeewwwwweeeeeeeeeeeeeeeeeeee www.ne///forrmmm kckkrc333
Systems, 2023. URL
Kathleen Fraser and Svetlana Kiritchenko. Examining gender and racial bias large vision- language models using novel dataset of parallel images. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 690-669 0030, St. Julian's s, Malta, March 2024. Association for Computational Linguistics. URL https: / a wecennnhhoooyy wwt.oyy.orgg2000..
leacl 1loll1oog..41
11

---

Published as a conference paper at ICLR 2025
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint mmXiv:444..12390 2024.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19953-14962 2023.
Phillip Howard, Avinash Madasu, Tiep Le, Gustavo Lujan Moreno, Anahita Bhiwandiwalla, and Vasudev Lal. Socialcounterfactualss Probing and mitigating intersectional social biases in vision- language models with counterfactual examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11975-11988 2024.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2023.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as visual chain thought for multimodal language models. arXiv preprint arXiv:2406.0944 09400, 2024a.
Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ran- jay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision- languae models. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp.
2024b.
vestigating their struggle with spatial reasoning. arXiv preprint arXiv:2310. 2023. Amita Kamath, Jack Hessel, and Kai- Wei Chang- What's" up" with vasion-language models? in-
Amita Kamath, Cheng- Yu Hsieh, Kai- Wei Chang, and Ranjay Krishna. The hard positive truth about lisiun-language compositionality. In ECCV, 2024.
Zaid Khan, Vijay Kumar BG, Samuel Schulter, Manmohan Chandraker, and Yun Fu. Exploring question decomposition for zero-shot vqa. arXiv preprint arXiv:2310.17050, 2023.
Wonjae Kim, Bokyung Son, and Iddo Kim. Vilt: Vision-and-language transformer without convo- lution region supervision. In International conference on machine learning, pp. 5583-5594. PMLR, 2021.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
Jaywon Koo, Ziyan Yang, Paola Cascante- Bonnilla, Baishakhi Ray, and Vicente Ordonez. Proptest: Automatic property testing for improved visual programming. In The 2024 Conference Em- pirical Methods in Natural Language Processing, 2024. URL https: http::///peeeeeee...eoe
(Eorum?id-Irai IrrummiiiIIKk
Max Ku, Dongfu liang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312. 14867, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, -eng-Neng Hwang, et al. Grounded language- -mage pre- trairrin In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10965-10975. 2022.
12

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In Computer ESconnEECVV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation, 2024. URL
www//ba.aa...ooooo. 2ttp:////244.0299999.9. 01291
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https:/7
1lava Llava-vl.git wtw.gitnu...o//lc.......oo...
01-4300100000 30- 1lava -lava-nnxxt/
Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023c.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, et al. Sim- ple open-vocabulary object detection. In European Conference on Computer Vision, pp. 728-755. Springer, 2022.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig, Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14420-14431, 2024.
OpenAl. ChatGPT.
wwtpppaa......aaaa....aa...aaaa. 2023. Large language model.
Timothy Ossowski, Ming Jiang, and Junjie Hu. Prompting large vision-language models for com- positional reasoning, 2024. URL BttL:tttps://axxv.coom http://www.rr/a/////1....1
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763, PMLR, 2021.
Chahat Raj, Anjishnu Mukherjee, Aylin Caliskan, Antonios Anastasopoulos, and Ziwei Zhu. Biasdora: Exploring hidden biased associations in vision-language models. arXiv preprint arXiv:2407.02066, 2024.
Raimundo Real and Juan M Vargas. The probabilistic basis of jaccard's index of similarity. System- atic biology, 45(3)3380-385, 1996.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494, 2022.
Kate Sanders, Nathaniel Weir, and Benjamin Van Durme. Ty-trees: Multimodal entailment entiimment trees for neuro-symbolic video reasoning, 2024. haattssr//axiiv.orrr/a///2222
peattps://axxiv.oott
Michael Saxon, Fatima Jahara. Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. Who evaluates the evaluations? objectively scoring text-to-image prompt coherence met- rics with (iscorescooee (1s2), 2024.
04251]
13

---

Published as a conference paper at ICLR 2025
Ava Spataru, Eric Hambro. Elena Voita, and Nicola Cancedda. Know when to stop: A study of semantic drift in text generation. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3656-3671, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/
3024.naacl-long.222 202. URL
http:////wwaaaaoooooooooooo///// -twp:///o//222.aaa111a21111 long-long.... 202
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang. Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. Modular visual question answering via code generation. In Proceedings of the 61 Annual Meeting of the Association for Compula- tional Linguistics, Toronto, Canada, 2023. Association for Computational Linguisties.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11888-11898, 2023.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 9238-5248. 2022.
Anthony Tiong, Junqi Zhao, Boyang Li, Junnan Li, Steven Hoi, and Caiming Xiong. What are we measuring when we evaluate large vision-language models? an analysis of latent factors and biases. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Con- ference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers). pp. 3427-3454, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 001.865//11/222444 naaae-4naa----o.1150 188. URL https: httb::://aclattholl.e www. 2024 maa.lll....aaa-l-........caaccl. nwacl- -ongell-ooo..oo 188
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024.
David Wan, Jaemin Cho, Elias Stengel-Esking and Mohit Bansal. Contrastive region guidance: Im- proving grounding vision-language models without training. arXiv preprint arXiv22403.02325, 2024.
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset. Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In Proceedings of the IEEE/VV con- ference on computer vision and pattern recognition, pp. 18359-18369, 2023a.
Tan Wang, Kevin Lin, Linjie Li, Chung -hhnn Lin, Zhengyuan Yang. Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11998- 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v (ision) is human-aligned evaluator for text-to-3d generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22227- 22238, 2024.
Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oguz, James C. Gee, and Yixin Nie. The role of thain-of-thought in complex vision-language reasoning task, 2023. URL https://arxii:
org/ ats//bb//33111 09193
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roce Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 1601-1619. Curran Associates, Inc., 2023. URL https: httpppss/ppooocecinn...errro.ocp
httpppppppppieeseppppeeeepppp 2023/file 6e8e 9c 99 335.ee88888c35387766666611011910
14

---

Published as a conference paper at ICLR 2025
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-languea models behave like bags-of- and what to do about it? The Eleventh International Conference Learning Representations, 2023.
Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs, 2024. URL Sttps: /ttpss///axxvvaa aww.i.iiiivv./... org/ abs 200..4411...000 02582
Xinlu Zhang. Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v (ision) generalist evaluator for vision- language tasks. arXiv preprint arXiv:2311.01361, 2023.
Shihao Zhao, Shaozhe Hao, Bojia Zi, Huaizhe Xu, and Kwan- Yee K. Wong. Bridging different language models and generative vision models for text-to-image generation. ECCV, 2024.
Kankan Zhou, Eason Lai, and Jing Jiang. VLStereoSet: study of stereotypical bias in pre-trained vision-language models. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (eds.), Proceedings of the 2nd Conference of the Asia-Pacijp Chapter of the Association for Computa- tional Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 527-538, Online only, November 2022, Association for Computa- tional Linguistics. URL https:ttp...m : ac httpa/caaaathooooooooooooo wwwphhhlww.......///e2o 2022. aacl -main. 40
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROMPT DETAILS
the caption expansion step, we instruct the LLM to generate both entailments and contradictions in single output rather than in separate steps, this helps the LLM maintain balanced semantic context, leading to more coherent and complementary outputs. Since the LLM often tends to output simple negative statements for contradictions (e.g., using words like "no" or "not"), explicitly instruct the model to avoid such terms. This ensures that the generated contradictions are more diverse and meaningful, rather than being straightforward negations. addition, following prior work (Mitra et al. 2024), we instruct the LLM to output the generated entailments and contradictions JSON, as follows an easy format parse the outputs. Prompt template is in Figure
Prompt Template
Given the sentence:
[Caption] Think step by step. What could be entailed?
First, provide two concise descriptions that entail the sentence. Include attributes mentioned in the sentence (color, size, position, amounts). If there is verb, rephrase it entail the sentence. Some possible verb entailments include:
looking -> (participant) person's eyes over something (do not include the recipient)
kissing -> (participant) person's s 1ips touching someone (do net include the recipient)
talking -> (participant) person's mouth open (do not include the recipient)
hugging -> (participant) person's arms reaching the other person (recipient)
person hitting (participant) person in motion object hitting >> (participant) object damaged
The descriptions need to be specific and implicitly entailed in the assumptions.
sentence. Include world knowledge and For example:
Sentence: A yellow unicorn talks to a tall person
Nouns: plant, unicorn, ball, porson, sky Entailed descriptions: [Yellow unicorn's mouth is open, Yellew unicorn is gesturing]
Then, provide the opposite sentence to each sentence, do not include negations.
For example:
Sentence: Person talks unicorn
Nouns: unicorn, ball, person, sky
Opposite descriptions: [Tall person's mouth is open, Tall person is gesturing]
Finally, output the entailed descriptions in a json format. {
"Entailed doscriptions": [Yellow unicorn's a mouth is open, Yellow unicorn is gesturing] "Opposite descriptions": [Tall person's mouth open, Tall person is gesturing]
Figure 4: Prompt template used in Caption Expansion. (Caption) is replaced for each sample.
16

---

Published as a conference paper at ICLR 2025
A.2 ERROR ANALYSIS
We further examine cases where CECE fails comparing with using original caption only (LLaVA- 1.6) and Sentence Decomposition via Semantics (SDS). We show detailed entailments and contra- dictions in CECE column, where CECE Final Score is from Equation 6 with = 0.5 and Overall Final Score is from Equation 6 where 02 0.6. Note that both a values are kept consistent throughout all our experiments. Figure S shows examples where LLaVA-1.8 correctly predicts the score relationship between images given the text but both SDS and CECE fail. Figure exam- ples where LLa and SDS correctly predict the score relationship between images given the text but CECE fails. Figure 7 shows examples where LLaVA-1.6 1.6, SDS and CECE fail. Finally, show examples where CECE semantically drift from the original caption.
Figures g
Original Caption/lmage Pair
Sentence Decempacitinn via Semanties (SUS)
Caption Expandion CRCK
Entallments
Text (T)
Pitss TI,A)
PYsi By
Tiret
P. A)
PC T.
thone lca percos
25
asraorndss hande are helding speasing
90.6%
peruen the rchide
26.05
73.8%
semoone in un
Text (T)
PrYes (T,A)
P/Na IT,
dere vehicle
97.3%
94.4%
De yeoging device
coning out
18.09
74,4%
water lowands the
veticce speasing
70.4%
<
74.2%
sperying
(T)
Contradictons
PINe (T
PNNe IT.
pround
down
57.2%
35.6%
$5.5%
dene wital
92.2%
$2.9%
dovice
80.85
95.15 19
dere
79.39
78.0%
CECE Final Seare
55.4%
67.8%
SDS Final Score
31.7%
63.8%
Original Captioe/lmong Paic
Sentence Decomposition via Semantics (SDS)
Cuption Exponsion via CECE
Entallments
Text (T)
PTY
Pieei T, B)
Text
P/lai 'T,
Pryes XX
ln tal's leegth than Es bell's leegth
the tohoet body
44.0%
45.9%
in lengds
Text (T))
Pr IT,
Piles T, Ri]
Seore
Mi 1.
24.9%
its isll shoner body
46.%
Contrudictions
Tist:T)
PNo
PNN Ib
ik leegth greaue bedy's length
It limgth (han ia Jal's legili
$7.4%
Sk 59.5%
CRCE Fltal Score
62.8%
63.2%
Ovenail
56.0%
Original CaptinnTnaaee
Decomposiion Semontics (SDS)
Expansion CECE
Tex
P/la T,A)
Pr (NNiTE)))
Entalinents
there ao people
95 36
95.7%
poople siting or suadz
855
sprish
bodies
$9.15
65.45
people are Htadding hesl ano siting
PoYes (TAA)
PoYes TT,
conding
75.56
73.8
Cantradictinns
nare people' bodie scilel prodico
60.45
77.8%
60.4%
SDS Final Stare
$3.1%
83.9%
poople' bodies spright
12.9%
34.7%
uree poople' hodier.r soiled
16.1%
23.5%
Case 1: Correct with VA-166 Wrong with SDS and CECE
Overall Final Score
53,8%
53.0%
Figure 5: Qualitative error analysis: cases where both SDS and CECE fail: a) In the first case, SDS correctly decomposes the caption and focuses on the vehicle, which the one spraying water. On the other hand, through CECE, the LLM incorrectly focuses on person spraying water. However, both cases fail since the VLM is unable identify the vehicle spraying water. b) In the second case, SDS fails to decompose the caption into smaller parts, and repeats the given text. Although CECE produces correct entailments and contradictions, the VLM fails to match the correct image-text pair. c) Similarly, for the third case, the VLM seems to fail to match the correct image, which seems too difficult to parse due to out-of-focus and illumination issues.
17

---

Published as a conference paper at ICLR 2025
Original Caption/Image Pair
Sentence Decompusitio via Semantics (STS)
Caption Expaansonn via
Enfallments
IT.
Teat (T)
P/Yes IT,
PiYes IT,
dey mway
ley molimn
95.9%
98.78
them
70.5%
58.2%
motion
97.4%
97.7%
Text
PMlls A)
PLYes I'I,
they away
SDS Final Score
30.3%
50.7%
Text (T)
P/No IT,
P(No IT,
while pursued
they sre stending still it standing still
94.1%
78.5%
86.5%
71.8%
CBCE Final Seore
94,2% %
85.8%
Overall Final Score
72.5%
72.3%
Original Caption/lmage Pair
Sentence Decompoditonn Semantlcs (SDS)
Caption Expunsion via CBCE
Text (T)
PLYes TT
PVVes T,
Entailmentt
Test (T)
PyYes IT.A)
P/Yes IT.
diagram is
present
arrow the diagram
pointin from left
81.6%
85.0%
Text (T))
PoYes I.A)
Poles I,
esprosed
diegrum
showing
diegram
70.7%
708%
tee movement the diagna lroms left to right
52.1%
$7.3%
loft right
$1.78
Contradictions
left to right
Test (T)
PoNo
PONe T, 19
the diagamm
from
30.9%
26.4%
soguenee
CFCE Score
31.0%
Overall Finad Seare
60.4%
58.6%
Original Caption/Iuaage Pair
Sentence Decomposition via Semanties (SDS)
Caption Expansion via CECK
Entailments
Text (T)
P&Ys IT,A)
PLYes IT, B)
Text (T)
PoYes IT,
Prlies IT,
there is boat house
73.0%
30,0%
structure neor body of watee
93.0%
90.3%
bsat is soonal dockel
32.9%
$6.4%
PoYes
POYes
Contradictions
house
Text
P.NoTT.A)
PiNo
lanallackell
96.5%
CRCR Firal Score
64.0%
72.9%
Case 2: Correct with LLa and SDS; Wrong with CECE
Overall Final Score
60.7%
61.8%
Figure 6: Qualitative error analysis: cases where only CECE fails: a) In the first case, SDS correctly decomposes the caption, although they and it are not concretely stated (e.g., could refer to animals or objects), breaking the sentence into two separate statements is sufficient for the VLM match the correct image. On the other hand, the contradictions generated via CFCE introduce statements that are correct, but lead to incorrect conclusions given nature of the data (i.e., still images). b) Similarly, the contradictions generated for the second case introduce incorrect negative statements that weigh over the entailments and lead to the incorrect matching. c) In the third case, SDS is unable decompose the sentence, and the given description contains the name of the referring object (i.e., boat house). Although both entailments and contradictions generated via CECE provide more detailed fine- fine-grained set of descriptions, the VLM is unable to identify the correct pair.
18

---

Published as a conference paper at ICLR 2025
Original Caption/Emage Puir
Sentence Decompesition via Semanties (SDS)
Caption Expansion via CECE
Fatailments
Test (T)
PLYa (T,))
P/ IT,
Test
individull standing
ground
person is upeidlt
PCYex IT,
IT)
SDS Final Score
37.0%
67.1%
Coutradictions
standing on feet
24.7%
32.1%
Text (T)
PNNo /T. A)
PNNo /T,
penum's fest am off the
ground
74.1%
79.8%
person is hocizonnal
81.2%
$4.2%
CFCF Pinal
Final Score
35.4%
55.4%
Original Enmag
Decomposition Semanties
Caption Expunsiio CECE
Text (T)
POYs T,
PIYes |T, E)
Entailments
Text (T)
PrTes IT, A)
PrYis T,
there are sweet things
93.2%
$8.0%
has
the number swest
66.1%
61.5%
47.9%
amangrd
Text (T)
PCYe IT, A)
PO'is IT, B3
there plute
39.2%
$3.8%
eu even naater
things
the plate has un odd number of sides
39.2%
49.2%
Contradictions
plate an
number of sides
the sweet things are the plate
76.0%
71.5%
Text (T)
PNNo A)
PrNo 8)
plate has odd mimber suret things
57.7%
42.8%
SDS Final Score
69.8%
47.4%
nunber sides
56.1%
61.5%
CECE Final Soare
$2.4%
50.9%
Case 3: Wrong with LLaVA-116, SDS, and CECE
Overall Final Score
$4.0%
49.3%
Figure 7: Qualitative error analysis: cases where LLaVA-1.6, SDS and CECE all fail: a) In the first case, while CECE correctly generates entailments and contradictions, the VLM unable to match the correct image-text pair. In this case. both images cutout and it may be difficult for the model to identify the people the scene. Note that SDS is unable to break the sentence. b) As opposed the previous one, in the second case, SDS is able correctly decompose the given caption, but the VLM is unable to score the matching image-text pair. CECE on the other hand, fails to generate incorrect output. However, the failure margin from CECE is lower than that produced by SDS.
meaningful entailments and contradictions, leading
19

---

Published as a conference paper at ICLR 2025
Caption: a person with long braids is exercising in front of person without braids
Caption: person without braids is exercising in front of person with long braids
Entailments:
Entailments:
E,; person with long braids is moving their body.
E, person's body is moving.
Eg: person without braids is standing in front of the exercising person.
E person is front of another person.
Contradictions:
Contradictions:
C person without braids is moving their body.
C, person with braids is exercising in front of person without braids.
Czi person with long braids is standing front of the exercising person.
Figure 8: Qualitative error analysis: cases where CECE semantically drift from the original caption. In this case, the LLM incorrectly outputs "a person without braids is standing in front of the exer- cising person" " as an entailment. Similarly, the first two entailments for the second caption do not contain any particular attribute to refer to the right person.
Caption: halloween, child dressed insect the right of child dressed mammal
Caption: halloween, child dressed mammal to the right of child dressed insect
Entailments:
Entailments:
E, child dressed as an insect is positioned to the
E, a child dressed as mammal is positioned to the right of child dressed
right child dressed mammal.
insect.
E,: two children are standing next to each other.
E,: there are at least two children, one dressed as a mammal and the other as insect.
Contradictions:
C, child dressed mammal to the right of child dressed an insect.
Contradictions:
C, a child dressed as insect to right child dressed as mammal.
C child dressed as a mammal and a child dressed insect are standing next each other with the mammal
C,: child dressed as insect positioned to the
right.
left a child dressed as mammal.
Figure 9: Qualitative error analysis: cases where CECE semantically drift from the original caption. In this case, the LLM provides the incorrect assumption that only two children are standing next to each other, and is also unable to break the caption into smaller semantic components.
20

---

Published as a conference paper at ICLR 2025
someone the ground spraying water lowards vehicle
Caption:
Caption: someone vehicle spraying water towards ground
Entailments:
Entailments:
E,: someone's hand is moving with hose spray nozzle.
E, someone's hands are holding spraying device.
E,' is coming
spraying device.
E, water is coming out of a hose or spray nozzle towards vehicle.
Contradictions:
C, someone's 's hands are empty.
Contradictions:
Cy: the spraying device dry.
C, someone's hand moving away from hose or spray nozzle.
C,: water coming of hose spray nozzle away from vehicle.
analysis: cases where CECE semantically drift from the original caption. Winoground sample with tag Both,. These are cases where both objects and relations are swapped single swap that changes parts of speech. In this case, while the decomposed captions entail and contradict the original caption, they fail to describe the precise action depicted in the image. This happens when the original captions swap multiple objects, attributes, and actions.
Figure 10: Qualitative
21

---

Published as a conference paper at ICLR 2025
Caption:
there three people and two windows
Caption:
there two people and three windows
Entailments:
Entailments:
E,: three people are present.
E,: two people are present.
E,: two windows present.
E," three windows are present.
Eyi three people are slanding or sitting in room or
Ey: two people are slanding or sitting a room or
area
space
E,: two windows part building structure
E, three windows are part building structure
Contradictions:
C, na people are present.
Contradictions:
Cgi windows present.
C,i there person.
C there is one window.
Figure 11: Qualitative error analysis: cases where CECE semantically drift from the original caption. Winoground sample with tag Symbolicy. These samples contain captions for non-natural images, such as drawings or illustrations. In this case, the entailments and contradictions are mostly correct, but the LLM introduces incorrect assumptions about the location of the people present in the image.
22


Reviews:
[{"review_id": "eRfBblHE11", "reviewer": "Reviewer_Xyex", "summary": "The paper presents Caption Expansion with Contradictions and Entailments (CECE), a new approach to improve compositionality in the vision-language models by transforming the caption into a natural language inference task. The main idea is to paraphrase the caption into contradictions and entailments with a large language model and then combine them with a vision-language model for final prediction. The results on two important benchmarks (image-to-text and text-to-image) show that CECE outperforms existing baselines without any fine-tuning.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, and the motivation and method are clearly explained. The related work is thorough, which makes it easy to understand the existing body of work on the topic.\n\nThe method focuses on manipulating the captions and shows improved performance without any fine-tuning, which is impressive and useful. \n\nThe analysis shows that all the components — caption, entailment, and contradiction — are important for the highest performance on the Winoground and EqBen datasets.", "weaknesses": "**Inference cost.**\nThe main weakness of the method is that it requires a very powerful LLM, such as LLama 3.1 70B, for caption expansion. This adds to the inference and compute cost for generating lexically diverse captions. It would be helpful to know if a smaller model can improve performance similarly. It would be ideal to have an experiment showing the performance of the VLM on a range of language models. \n\n\n**More datasets.**\nThe paper experiments on two datasets, namely, WinoGround and EqBen. However, Winoground is a small dataset with only 1600 examples. It would be helpful to know if the method works on larger datasets such as SugarCrepe [1] or larger synthetic datasets such as the concept binding dataset [2]. \n \n\n[1] SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality. NeurIPS: Datasets and Benchmarks Track, 2023. \n\n[2] Does clip bind concepts? probing compositionality in large image models. EACL Findings, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AwMldWjanZ", "reviewer": "Reviewer_1DuW", "summary": "This paper proposes Caption Expansion with Contradictions and Entailments (CECE) to enhance compositional reasoning capabilities in vision-language models (VLMs) by leveraging a large language model (LLM) to generate entailments and contradictions through natural language inference (NLI) prompts. The core idea is to prompt the LLM to expand each image caption by producing semantically related statements, both for entailments and contradictions. The intuition is that CECE provides the VLM with lexically diverse cues that encourage deeper visual-textual alignment for complex compositional reasoning. The authors conducted extensive evaluations on two common compositionality benchmarks: Winoground and EqBen, CECE achieves competitive results.", "questions": "Minor experimental detail: \n\nDid the authors use any system prompt or chat template for the open-source LLMs during the experiments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and relatively easy to follow. The core ideas are intuitive. I think The proposed method, CECE, addresses a critical limitation in vision-language models, as vanilla pretrained VLMs have traditionally struggled with compositional reasoning tasks. CECE provides a novel solution that enriches the semantic understanding of VLMs and I see it beneficial to the broader community. The experimental results are competitive with clear improvements over existing methods.", "weaknesses": "A crucial limitation of this method is its reliance on two balancing hyperparameters. While the authors provide some analysis, the need for the balancing hyperparmeters could hurt CECE’s generalizability to new tasks and datasets. I think documenting example cases of semantic drift in the appendix would be valuable to the community and it will provide insight into how auxiliary prompting can affect semantic meaning of original captions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WqVsNtuz21", "reviewer": "Reviewer_ZcB9", "summary": "The authors propose a new text-image alignment metric, CECE, that considers the original caption alongside a set of LLM-generated entailed and contradictory statements as opposed to past work breaking down captions into smaller sub-statements. This is motivated by a desire to increase lexical diversity and improve performance on captions involving reasoning or world knowledge. The authors validate their approach on Winoground and EqBen and also measure alignment with human judgements on five benchmarks for text-to-image generation and one benchmark for text-to-3D generation, outperforming prior work on all but StanfordT23D. This is coupled by finer-grained analyses of performance, lexical diversity and correctness on Winoground.", "questions": "1) In most of the Winoground splits on Table 4, CECE performs better on selecting images rather than text unlike the VQAScore baselines. Any ideas why this may be the case?\n\n2) Is there any reason why CECE with InstructBLIP was not included for Tables 2 and 3 (especially considering that VQAScore with InstructBLIP achieves superior performance compared to Llava-1.5)?\n\n3) Is the value of M for the number of entailments and contradictions set in stone or variable based on the input caption?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The method achieves state-of-the art performance on Winoground and EqBen, outperforming the second best evaluated model by large margins of 11.2% and 9.3% respectively. Its alignment with human judgements is similarly strong, albeit with a lesser gap over prior work.\n2) This improvement is achieved via a relatively simplistic inference-time strategy requiring no finetuning.\n3) It was unexpected for me that \"expanding\" a caption via entailed and contradictory statements would lead to better performance than breaking down statements into smaller components as in prior work. This could inspire further inference-time strategies either refining the current approach or combining it with elements of past work.", "weaknesses": "1) One unaddressed limitation of the approach compared to the VQAScore baseline is the inference-time cost of additionally using Llama3.1 70B for caption expansion. This is an integral component of the approach and I would have expected ablations of the LLM (for instance by using smaller variants of Llama3.1) in the same way that there were ablations of the VLM.\n\n2) I think the paper would be improved by further discussions of when CECE underperforms against the baselines. On Table 4, for instance, CECE without ensembling underperforms against the VQAScore baseline for the Text score for the Object_L and Both_L categories (and by a roughly 25% margin for the latter). Similarly, for the Pragmatic_V split, CECE can only decisively outperform the VQAScore baseline after ensembling and is either competitive with or poorer than the VQAScore baseline otherwise. An analysis of why would lead to a better understanding of the strengths and weaknesses of the approach and would help future work improve the method further. \n\n3) On a related note, I think the comparisons for the ensembling strategy would be fairer if a similar ensembling strategy were applied for VQAScore and DSG, particularly when considering the boost ensembling affords on Table 4. For instance, would VQAScore still perform worse if some weighted average of the Llava-1.5 and 1.6 probabilities were used instead?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dHdHwdJkZo", "reviewer": "Reviewer_G1JW", "summary": "This work proposes to utilize entailments and contradictions generated by LLMs conditioned on captions on visio-linguistic compositionality benchmarks. Specifically, the authors prompt LLMs to generate entailment and contradiction statements in a one-shot manner, then calculate a weighted average of the conditional likelihood of a 'yes/no' response from a VL model, conditioned on image-statement pairs. The statements include the generated entailment/contradiction statement and the original caption. And the weights are hyperparameters. The proposed method shows effectiveness on both EqBen and Winoground benchmarks.", "questions": "Refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clear and easy-to-follow. \n- The proposed method shows significant gains compared with other baselines on two visio-linguistic compositionality benchmarks.", "weaknesses": "- Although the proposed methods demonstrate empirical effectiveness, the motivation for including contradiction statements lacks clarity. According to the prompt in Appendix A.1, the LLM is instructed to generate contradiction statements as sentences opposite to the entailment statements. This setup implies that each pair of entailment and contradiction statements may be semantically equivalent. Given this, it is unclear where the benefit lies in incorporating both entailments and contradictions.\n- In Table 6, it would be helpful to include the performance results for using contradictions alone.\n- In the qualitative error analysis in Fig. 5, the entailments and contradictions generated by the proposed method generally represent only a subset of the caption’s full semantics, rather than comprehensively capturing its meaning. This limitation may introduce semantic bias. An ablation study on the impact of the number of entailment/contradiction statements, along with an analysis of the semantic diversity among these statements, would enhance the understanding of their effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)