Natural Language Inference Improves Compositionality in Vision-Language Models

Authors: Paola Cascante-Bonilla, Yu Hou, Yang Trista Cao, Hal Daumé III, Rachel Rudinger
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text to image evaluation, image to text generation, natural language inference, sentence decomposition, large language models, visual question answering, question generation, benchmark

Abstract:
Compositional reasoning in Vision-Language Models (VLMs) remains challenging as these models often struggle to relate objects, attributes, and spatial relationships. Recent methods aim to address these limitations by relying on the semantics of the textual description, using Large Language Models (LLMs) to break them down into subsets of questions and answers. However, these methods primarily operate on the surface level, failing to incorporate deeper lexical understanding while introducing incorrect assumptions generated by the LLM. In response to these issues, we present Caption Expansion with Contradictions and Entailments (CECE), a principled approach that leverages Natural Language Inference (NLI) to generate entailments and contradictions from a given premise. CECE produces lexically diverse sentences while maintaining their core meaning. Through extensive experiments, we show that CECE enhances interpretability and reduces overreliance on biased or superficial features. By balancing CECE along the original premise, we achieve significant improvements over previous methods without requiring additional fine-tuning, producing state-of-the-art results on benchmarks that score agreement with human judgments for image-text alignment, and achieving an increase in performance on Winoground of $+19.2\%$ (group score) and $+12.9\%$ on EqBen (group score) over the best prior work (finetuned with targeted data).

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730606787233

thread_type:
official_review

<thread_object>
creator: reviewer_g1jw

<content>
summary:
This work proposes to utilize entailments and contradictions generated by LLMs conditioned on captions on visio-linguistic compositionality benchmarks. Specifically, the authors prompt LLMs to generate entailment and contradiction statements in a one-shot manner, then calculate a weighted average of the conditional likelihood of a 'yes/no' response from a VL model, conditioned on image-statement pairs. The statements include the generated entailment/contradiction statement and the original caption. And the weights are hyperparameters. The proposed method shows effectiveness on both EqBen and Winoground benchmarks.

strengths:
- The paper is clear and easy-to-follow. 
- The proposed method shows significant gains compared with other baselines on two visio-linguistic compositionality benchmarks.

weaknesses:
- Although the proposed methods demonstrate empirical effectiveness, the motivation for including contradiction statements lacks clarity. According to the prompt in Appendix A.1, the LLM is instructed to generate contradiction statements as sentences opposite to the entailment statements. This setup implies that each pair of entailment and contradiction statements may be semantically equivalent. Given this, it is unclear where the benefit lies in incorporating both entailments and contradictions.
- In Table 6, it would be helpful to include the performance results for using contradictions alone.
- In the qualitative error analysis in Fig. 5, the entailments and contradictions generated by the proposed method generally represent only a subset of the caption’s full semantics, rather than comprehensively capturing its meaning. This limitation may introduce semantic bias. An ablation study on the impact of the number of entailment/contradiction statements, along with an analysis of the semantic diversity among these statements, would enhance the understanding of their effects.

questions:
Refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675693471

thread_type:
official_review

<thread_object>
creator: reviewer_zcb9

<content>
summary:
The authors propose a new text-image alignment metric, CECE, that considers the original caption alongside a set of LLM-generated entailed and contradictory statements as opposed to past work breaking down captions into smaller sub-statements. This is motivated by a desire to increase lexical diversity and improve performance on captions involving reasoning or world knowledge. The authors validate their approach on Winoground and EqBen and also measure alignment with human judgements on five benchmarks for text-to-image generation and one benchmark for text-to-3D generation, outperforming prior work on all but StanfordT23D. This is coupled by finer-grained analyses of performance, lexical diversity and correctness on Winoground.

strengths:
1) The method achieves state-of-the art performance on Winoground and EqBen, outperforming the second best evaluated model by large margins of 11.2% and 9.3% respectively. Its alignment with human judgements is similarly strong, albeit with a lesser gap over prior work.
2) This improvement is achieved via a relatively simplistic inference-time strategy requiring no finetuning.
3) It was unexpected for me that "expanding" a caption via entailed and contradictory statements would lead to better performance than breaking down statements into smaller components as in prior work. This could inspire further inference-time strategies either refining the current approach or combining it with elements of past work.

weaknesses:
1) One unaddressed limitation of the approach compared to the VQAScore baseline is the inference-time cost of additionally using Llama3.1 70B for caption expansion. This is an integral component of the approach and I would have expected ablations of the LLM (for instance by using smaller variants of Llama3.1) in the same way that there were ablations of the VLM.

2) I think the paper would be improved by further discussions of when CECE underperforms against the baselines. On Table 4, for instance, CECE without ensembling underperforms against the VQAScore baseline for the Text score for the Object_L and Both_L categories (and by a roughly 25% margin for the latter). Similarly, for the Pragmatic_V split, CECE can only decisively outperform the VQAScore baseline after ensembling and is either competitive with or poorer than the VQAScore baseline otherwise. An analysis of why would lead to a better understanding of the strengths and weaknesses of the approach and would help future work improve the method further. 

3) On a related note, I think the comparisons for the ensembling strategy would be fairer if a similar ensembling strategy were applied for VQAScore and DSG, particularly when considering the boost ensembling affords on Table 4. For instance, would VQAScore still perform worse if some weighted average of the Llava-1.5 and 1.6 probabilities were used instead?

questions:
1) In most of the Winoground splits on Table 4, CECE performs better on selecting images rather than text unlike the VQAScore baselines. Any ideas why this may be the case?

2) Is there any reason why CECE with InstructBLIP was not included for Tables 2 and 3 (especially considering that VQAScore with InstructBLIP achieves superior performance compared to Llava-1.5)?

3) Is the value of M for the number of entailments and contradictions set in stone or variable based on the input caption?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685451266

thread_type:
official_review

<thread_object>
creator: reviewer_1duw

<content>
summary:
This paper proposes Caption Expansion with Contradictions and Entailments (CECE) to enhance compositional reasoning capabilities in vision-language models (VLMs) by leveraging a large language model (LLM) to generate entailments and contradictions through natural language inference (NLI) prompts. The core idea is to prompt the LLM to expand each image caption by producing semantically related statements, both for entailments and contradictions. The intuition is that CECE provides the VLM with lexically diverse cues that encourage deeper visual-textual alignment for complex compositional reasoning. The authors conducted extensive evaluations on two common compositionality benchmarks: Winoground and EqBen, CECE achieves competitive results.

strengths:
This paper is well-written and relatively easy to follow. The core ideas are intuitive. I think The proposed method, CECE, addresses a critical limitation in vision-language models, as vanilla pretrained VLMs have traditionally struggled with compositional reasoning tasks. CECE provides a novel solution that enriches the semantic understanding of VLMs and I see it beneficial to the broader community. The experimental results are competitive with clear improvements over existing methods.

weaknesses:
A crucial limitation of this method is its reliance on two balancing hyperparameters. While the authors provide some analysis, the need for the balancing hyperparmeters could hurt CECE’s generalizability to new tasks and datasets. I think documenting example cases of semantic drift in the appendix would be valuable to the community and it will provide insight into how auxiliary prompting can affect semantic meaning of original captions.

questions:
Minor experimental detail: 

Did the authors use any system prompt or chat template for the open-source LLMs during the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730818453153

thread_type:
official_review

<thread_object>
creator: reviewer_xyex

<content>
summary:
The paper presents Caption Expansion with Contradictions and Entailments (CECE), a new approach to improve compositionality in the vision-language models by transforming the caption into a natural language inference task. The main idea is to paraphrase the caption into contradictions and entailments with a large language model and then combine them with a vision-language model for final prediction. The results on two important benchmarks (image-to-text and text-to-image) show that CECE outperforms existing baselines without any fine-tuning.

strengths:
The paper is well-written, and the motivation and method are clearly explained. The related work is thorough, which makes it easy to understand the existing body of work on the topic.

The method focuses on manipulating the captions and shows improved performance without any fine-tuning, which is impressive and useful. 

The analysis shows that all the components — caption, entailment, and contradiction — are important for the highest performance on the Winoground and EqBen datasets.

weaknesses:
**Inference cost.**
The main weakness of the method is that it requires a very powerful LLM, such as LLama 3.1 70B, for caption expansion. This adds to the inference and compute cost for generating lexically diverse captions. It would be helpful to know if a smaller model can improve performance similarly. It would be ideal to have an experiment showing the performance of the VLM on a range of language models. 


**More datasets.**
The paper experiments on two datasets, namely, WinoGround and EqBen. However, Winoground is a small dataset with only 1600 examples. It would be helpful to know if the method works on larger datasets such as SugarCrepe [1] or larger synthetic datasets such as the concept binding dataset [2]. 
 

[1] SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality. NeurIPS: Datasets and Benchmarks Track, 2023. 

[2] Does clip bind concepts? probing compositionality in large image models. EACL Findings, 2024.

questions:
See the weaknesses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
CECE improves compositional reasoning in vision-language models (VLMs) by using a large language model (LLM) to generate entailments and contradictions of captions. These expanded captions provide more diverse cues for better visual-textual alignment, leading to improved performance on compositional tasks like Winoground and EqBen, often without fine-tuning.

The paper is well-written, and the motivation and method are clearly explained. The core ideas are intuitive and the experiment results demonstrate the merit of the work.