NL-Eye: Abductive NLI For Images

Authors: Mor Ventura, Michael Toker, Nitay Calderon, Zorik Gekhman, Yonatan Bitton, Roi Reichart
Primary area: datasets and benchmarks
Keywords: Benchmark, Multimodality, Abductive Reasoning, NLI, VLM

Abstract:
Will a Visual Language Model (VLM)-based bot warn us about slipping if it detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet their ability to infer outcomes and causes remains underexplored. To address this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual abductive reasoning skills. NL-Eye adapts the abductive Natural Language Inference (NLI) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. The data curation process involved two steps—writing textual descriptions and generating images using text-to-image models, both requiring substantial human involvement to ensure high-quality and challenging scenes. Our experiments show that VLMs struggle significantly on NL-Eye, often performing at random baseline levels, while humans excel in both plausibility prediction and explanation quality. This demonstrates a deficiency in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a crucial step toward developing VLMs capable of robust multimodal reasoning for real-world applications, including accident-prevention bots and generated video verification.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730631824842

thread_type:
official_review

<thread_object>
creator: reviewer_972w

<content>
summary:
This paper introduces NL-EYE, a benchmark designed to test the abductive reasoning capabilities of Visual Language Models (VLMs) through image-based tasks. The benchmark includes 350 carefully curated triplet examples spanning diverse reasoning categories where models must choose the more plausible hypothesis from a set and provide an explanation. Experiments reveal that while humans excel in this task, current VLMs show notable deficiencies in their reasoning capabilities. The authors conclude that VLMs face significant challenges in visual interpretation, which impacts their ability to reason effectively about images.

strengths:
1. The benchmark is well-designed with diverse reasoning categories.
2. Experiments on the benchmark reveal interesting findings.
3. The analysis is thorough and highlights notable insights into VLM limitations.

weaknesses:
1. While I agree that the benchmark is carefully curated, the filtering condition can be inconsistent and subjective because it is done manually. 
2. This paper focuses primarily on evaluating VLMs' deficiencies but lacks discussion on strategies or methods to improve these models' abductive reasoning capabilities.
3. The paper lacks experiments with additional open-source models. While the current model selection is valid, given the paper's findings about failures in visual interpretation and hypothesis location dependency, testing VLMs with different visual encoders or those trained on multi-image datasets would further support the analysis.

questions:
## Question
1. Have the authors conducted experiments with VLMs that trained on datasets including multiple images such as LLaVA-Onevision or VILA, and with VLMs that use other visual encoders like Cambrian-1?

## Typo
* L260 Validation,and Categorization -> Validation and Categorization

---
### References
* Lin, Ji, et al. Vila: On pre-training for visual language models. CVPR 2024
* Li, Bo, et al. Llava-onevision: Easy visual task transfer. https://llava-vl.github.io/blog/2024-08-05-llava-onevision/
* Tong, Shengbang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Neurips 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645689386

thread_type:
official_review

<thread_object>
creator: reviewer_gcx5

<content>
summary:
This work proposes the NL-Eye benchmark to evaluate the abductive reasoning ability of visual-language models from pure visual perception in multi-image situations, inspired by Natural Language Inference (NLI). The benchmark consists of testing examples for temporal reasoning along with six reasoning categories and images are obtained from the text-to-image generation model. The authors argue that current visual-language models show significantly inferior performance compared to humans on abductive reasoning in multi-image situations and claim that this is due to the lack of purely visual perception abilities compressed from the visual perception modules.

strengths:
This work is distinct from existing multi-image benchmarks in that the objects of perception required to perform reasoning are provided solely through visual perception. NLI-inspired benchmarks that require visual reasoning over multi-images already exist, such as [1], but they are limited in terms of evaluating purely on visual perception, as they require reasoning over a given natural language premise. However, NL-Eye has the unique feature that requires reasoning on pure visual perception, since these premises are provided as images.

[1] A Corpus for Reasoning About Natural Language Grounded in Photographs (Suhr et al., 2019)

weaknesses:
There is a lack of consideration in the experiments as to whether a proper evaluation of the current visual-language model can be made in a multi-image setting. As the authors argue, current benchmarks for testing abductive reasoning are single-image focused, but it should not be overlooked that research on the visual-language model itself is also focused on this. As a result, the authors provide “concatenated” images, which may not be a fair assessment for most visual-language models that currently operate at fixed, squared-sized resolutions. To demonstrate the need for the proposed benchmark, it is required to observe if the same phenomenon is found in visual-language models that can handle flexible resolutions and aspect ratios like [1].

[1] LLaVA-UHD: an LMM Perceiving any Aspect Ratio and High-Resolution Images (Guo et al., 2024)

questions:
It would be nice to be able to determine if the problem this benchmark shows is out-of-domain on the language model side or a limitation of the visual encoder itself. If we split the data in the benchmark for training and test purposes and fine-tuned models improved on the remaining test splits, then we can assume that the main problem was the task was out-of-distribution rather than a lack of performance on visual perception, since most current visual-language models trained with a frozen visual encoder. Have you done any further experiments to see if this limitation on visual reasoning can be improved with some training or not?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679415684

thread_type:
official_review

<thread_object>
creator: reviewer_fepb

<content>
summary:
This paper presents NL-EYE, a benchmark to evaluate VLMs' visual abductive reasoning skills across six reasoning types: physical, functional, logical, emotional, cultural, and social. It includes 350 triplet examples (1,050 images) with temporal annotations indicating event sequence and duration. The study examines model robustness, considering hypothesis order sensitivity and different input formats (individual versus composite images). NL-EYE also assesses models' ability to score single hypotheses, addressing real-world scenarios where multiple alternatives may not be available.

strengths:
1. This paper demonstrates a way to measure VLMs abductive reasoning ability on six diverse reasoning categories using multiple images.
2. The experiments shown in the paper comprehensively evaluate the reasoning ability of VLMs by checking image ordering, exploring different input formats to justify the reasoning gap of the existing VLMs.
3. The analysis section is interesting. The breakdown of performance across reasoning categories and the underlying insights will be useful for the community.

weaknesses:
1. The prompt selection is under-explored. 
2. More detailed in the Questions section

questions:
Q1. It is unclear from the paper how the authors selected the concepts for each individual reasoning category. For example, in the Cultural Reasoning category, which cultures were represented in the generated image. As image generation models are also not good for cultural content generation and the VLMs being better on cultural NLI raise interest in which cultures were highlighted mostly in the data to assess the comprehensiveness of the test set.

Q2. The current prompt for VLM is asking the plausible answer first and then asking for explanation. It would be interesting to reverse this process (i.e., explain each image step-by-step and then conclude the plausible answer) and see how the VLMs react.

Q3. In Tables 2 and 3, LLaVA 1.6 performs better at predicting the plausible image using GPT-4 when converting image to text (Table 3) than when directly inputting images (Table 2). Could this difference be due to LLaVA’s limitations as a predictor, or is the prompt structure (e.g., asking for image descriptions first before selecting a plausible answer) affecting performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693752260

thread_type:
official_review

<thread_object>
creator: reviewer_ldjn

<content>
summary:
This paper introduces a new benchmark NL-EYE that is designed to assess VLMs’ visual abductive reasoning skills. NL-EYE adapts the abductive Natural Language Inference (NLI) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL-EYE consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. Experiments show that VLMs struggle significantly on NL-EYE, often performing at random baseline levels, while humans excel in both plausibility prediction and explanation quality.

strengths:
Previous Visual entailment tasks were mainly in text format. This paper for the first time proposes the task in image formats, and collected a human-curated benchmark. The experiments show that current VLMs cannot do well on the NL-EYE.
Also, one experiment result saying that VLM prediction depends on hypothesis location is interesting.

weaknesses:
1. It is unclear whether the used prompt can best unleash VLMs' performance. For example, from Table 5, it seems no example has been provided, and that may lead to lower VLM performance. 
2. Why do human only achieve 83-85% accuracy if human collected the dataset and this dataset do not require expert knowledge? (Line 426-427) It is a bit confusing to understand.
3. In Table 3, why not try GPT-4o as the Image-to-Text model? Also, why not try Claude models as predictor?
4. The images are generated instead of from real world, and could potentially affect the output. The test size is 350 which might be small.

questions:
See Weakness 1-3. 
Also just out of curiosity, why can't the setting in Table 3 solve this problem? E.g. How did GPT-4o fail the entailment upon the Figure 8 machine-generated captions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717897095

thread_type:
official_review

<thread_object>
creator: reviewer_frsv

<content>
summary:
This paper proposes a benchmark for measuring visual abductive reasoning capability and explains the process of constructing this benchmark. It demonstrates that current multimodal language models lack visual abductive reasoning capability and introduces a novel aspect of verifying image-to-image entailment that has not been previously addressed.

strengths:
- The paper is well-written and easy to read.
- The process of data collection and verification is systematic and meticulous.
- It intriguingly points out the shortcomings of existing visual language models (VLMs) in visual abductive reasoning, with experimental results to substantiate this claim.
- The paper proposes various experimental setups by combining or separating images, changing the order of images, which helps ensure fair testing.
- The benchmark effectively reveals multiple shortcomings of different VLMs, not only evaluating abductive reasoning but also highlighting issues with image location sensitivity and poor visual interpretation.
- Unlike traditional natural language inference (NLI) benchmarks, this approach offers a comprehensive evaluation of multiple aspects.

weaknesses:
- The evaluation criteria are unclear and not well-defined. The use of automatic evaluation for explanations seems inadequate, and manual evaluation, while more accurate, is too costly and varies depending on the person.
- The definition of visual abductive reasoning capability remains unclear; it appears to evaluate abilities including visual interpretation, interpretation of multiple images, and natural language inference, covering a broad range of concepts that are not distinctly defined.

questions:
- For the evaluation with this benchmark, it would be beneficial to have better metrics. Are there methods to quantify image order sensitivity? Could metrics be developed to measure visual understanding and linguistic abstract reasoning capabilities using various forms of input (Text-only, Image-to-Text, Image and Text, etc.)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732224745564

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their valuable feedback. We are pleased that the reviewers acknowledged the novelty of our dataset, particularly in integrating abductive reasoning across multiple image scenes and the quality of our systematic data collection (fRsv). We are also glad that reviewers found our approach clear and original (972W, GcX5, LDjN) and appreciated our comprehensive evaluation of VLMs (fepb, 972W).


In response to the reviewers constructive suggestions, we have incorporated considerable changes into the updated manuscript, (attached in the updated PDF):


- **Added open-source VLMs** to expand our comparative analysis (L427-L428; Table 12).  
- **Explored new prompting strategies**, including reverse process, Chain-of-Thought, and role-specific prompts, to investigate their impact on reasoning performance (L281; Tables 6, 14).  
- **Conducted ablation analysis on real images** (online and photographed) to assess the effect of visual realism on model performance (L1241-L1255, Fig.9).  
- **Introduced comparisons** of image-to-text performance using Claude as a judge (L486-487; Table 13).  
- **Clarified definitions** (L140-141; Appendix A.1)  and **expanded the discussion** (Appendix B) on future directions for improving abductive reasoning in VLMs.  


We deeply appreciate this recognition and have provided detailed responses to each reviewer’s comments below.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732519615372

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers,

As the discussion period comes to an end, we kindly ask you to consider our responses and the additional experiments we conducted. 
We would be happy to discuss our responses further and hope they address any misunderstandings or concerns you may have raised.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces NL-EYE, a benchmark designed to test visual abductive reasoning by requiring models to determine which of two images (hypotheses) better fits a given premise image. Unlike typical visual NLI tasks that rely on textual premises, NL-EYE uses purely visual input and spans diverse reasoning categories. Reviewers praised the careful data collection, the clear problem formulation, and the exploration of model weaknesses, noting that current VLMs struggle with abductive reasoning and even show sensitivity to image order. Although some raised concerns about evaluation metrics, prompt selection, and the small dataset size, the novelty and thoroughness of NL-EYE stand out. The paper opens a new avenue for assessing complex visual reasoning capabilities beyond conventional benchmarks. Given the overall positive assessment of the benchmark's conceptual clarity and its potential to spur new research directions, I recommend acceptance.