NL-Eye: Abductive NLI For Images

## Abstract

Abstract Will a Visual Language Model (VLM)-based bot warn us about slipping if it detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet their ability to infer outcomes and causes remains underexplored. To address this, we introduce NL-Eye , a benchmark designed to assess VLMs’ visual abductive reasoning skills. NL-Eye adapts the abductive Natural Language Inference (NLI) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. The data curation process involved two steps—writing textual descriptions and generating images using text-to-image models, both requiring substantial human involvement to ensure high-quality and challenging scenes. Our experiments show that VLMs struggle significantly on NL-Eye , often performing at random baseline levels, while humans excel in both plausibility prediction and explanation quality. This demonstrates a deficiency in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a crucial step toward developing VLMs capable of robust multimodal reasoning for real-world applications, including accident-prevention bots and generated video verification. 1 1 1 Data and code are available on the project page: https://venturamor.github.io/NLEye/ .

## 1 Introduction

Figure: Figure 1: NL-Eye evaluates the abductive reasoning capabilities of VLMs. The main setup involves a premise image and two hypothesis images, where the model is tasked with inferring which hypothesis is more plausible, and to provide an explanation for its choice.
Refer to caption: x1.png

*Abductive reasoning* refers to the ability to infer and predict plausible outcomes or causes given a context scene Peirce et al. (); Fann (); Douven ().
This reasoning skill is crucial for Visual Language Models (VLMs), as
they are likely to become increasingly integrated into our daily lives (Yildirim et al., ; Anwar et al., ; Chiang et al., ; Shah et al., ).
These models will be required to accurately monitor and interpret daily life scenes and correctly infer plausibility to prevent accidents and provide timely advice. For instance, would a bot warn us from slipping on a wet floor when there is no warning sign? or would it infer a missing pacifier as a cause of a crying baby?

Although this capability is critical,
previous work has mainly evaluated VLMs in a *single scene* setting — such as visual entailment or detecting improbable events like a fire in a closed jar — or in *sequential scenes*, such as next-frame prediction Xie et al. (); Fu et al. (); Hessel et al. (); Fu et al. (); Ganz et al. (); Yarom et al. (); Kadiyala et al. ().
Consequently, it remains unclear to what extent existing VLMs are capable of abductive reasoning.

To address this, we introduce NL-Eye, a benchmark designed to evaluate visual abductive reasoning capabilities of VLMs across multiple images. NL-Eye is inspired by the textual abductive NLI task Bhagavatula et al. () and applies it to the visual domain. In NL-Eye, a VLM is presented with a premise image and one or two hypothesis images. It then needs to infer how likely (plausible) a hypothesis image is to result from or lead to the premise image. The plausibility evaluation can be either done individually or in comparison to an alternative hypothesis. For instance, in Figure 1, the VLM needs to infer that, given the broken leg in the context image, it is more likely that the man slipped on the wet floor which lacked a warning sign (i.e., selecting hypothesis image 1).

Beyond *plausiblity prediction*, NL-Eye facilitates the evaluation of the models’ capability to provide faithful explanations. This allows us to explore whether they are correct for the right reasons rather than relying on shallow heuristics McCoy et al. (). For example, a valid explanation for the broken leg scene would suggest that the presence of a warning sign would have made the man more alert, thereby potentially preventing the accident. In contrast, a shallow explanation might suggest that the man was simply resting on a cozy rainy day.

Each NL-Eye example features a premise image alongside two hypothesis images, annotated with a gold label indicating the index of the more plausible hypothesis. The examples also include a gold explanation detailing why the chosen hypothesis is more plausible than the alternative.
Each example is categorized into one of six *reasoning categories* – physical, logical, emotional, functional, cultural, and social – and includes temporal annotations that specify whether the hypotheses occur *before*, *after*, or *simultaneously*
with the premise, and whether the time duration between the premise and hypothesis scenes is *short* or *long*. This rich annotation aids in diagnosing current VLMs and highlights their strengths and weaknesses. Figure 2 presents a detailed example.

To create NL-Eye, we collected a large pool of high-quality textual scenes created by experienced human annotators.
The resulting scenes were then provided to professional designers who utilized Midjourney
and DALL-E (Ramesh et al., ) to synthesize the corresponding images.
The designers are also tasked with categorizing each example and creating the explanation that is used as the gold label.
The image generation process was iterative, requiring multiple attempts to ensure consistency between the textual descriptions and the visual scenes, as well as visual coherence among the images within the same triplet. This process resulted in a total of 1,050 generated images, yielding 350 image triplets. Overall, NL-Eye is characterized by carefully curated examples, offering high quality both in terms of the scenarios and the consistency and quality of the images.

The first analysis is *human evaluation* where annotators select the more plausible hypothesis and explain their choice. Our results indicate that humans successfully identify the more plausible hypothesis in $85\%$ of the cases. Furthermore, in our assessment of the quality of the human-generated explanations, we find that in $94\%$ of the cases where the correct hypothesis was selected, the humans also provided a valid explanation. This demonstrates that humans perform reasonably well on the NL-Eye tasks.

Next, we design a comprehensive study to evaluate the abductive reasoning abilities of modern VLMs.
We take multiple measures to ensure the robustness of our evaluation, including addressing sensitivity to the order in which hypotheses are presented and exploring various input strategies, such as feeding the model three separate images or presenting it with a single combined-image that composites all three.
Since real-world scenarios may not always provide two alternatives, we also evaluate the model’s ability to assign a plausibility score to a single hypothesis, in addition to comparing two candidates.
We have also developed a framework that utilizes a text-based baseline that processes textual descriptions of visual scenes.
Specifically, we compare the results with gold descriptions and with the captions of the images as generated by the VLMs.
Lastly, evaluating model-generated explanations is challenging, as comparing generated text to a single reference (gold) explanation can be limiting and may not capture the variety and validity of possible correct answers. To address this, we adopt the evaluation proposed by Bitton-Guetta et al. (): human annotators are presented with an image triplet where the correct hypothesis is already labeled and select valid explanations from a provided set.

Our results show that while humans perform well on NL-Eye, VLMs struggle, with most models failing to surpass a random baseline in the *plausibility prediction* task. Even when identifying the plausible hypothesis, VLMs fail to provide accurate explanations in over 50% of cases, revealing a major weakness in their abductive reasoning.
Furthermore, our text-based experiments indicate that these models often succeed in textual reasoning even when they fail to reason over images. Interestingly, when we prompt the VLMs to generate image captions, the resulting captions prove ineffective for solving the task. Consequently, we hypothesize that the VLMs reasoning is hindered by inaccurate visual interpretations.
We also find that these models are sensitive to the order in which the hypotheses are presented and to the input format (three separate images vs a single combined-image). This sensitivity is concerning, as it raises the possibility that the models may not genuinely understand the underlying concepts, potentially relying on superficial cues to make decisions.

To summarize, we introduce NL-Eye  a carefully curated benchmark designed to test the abductive reasoning abilities of VLMs across various categories and temporal relations. We then conduct a comprehensive study evaluating modern VLMs on NL-Eye and find notable deficiencies in their abductive reasoning capabilities. We believe NL-Eye represents a crucial step toward enhancing the reasoning abilities of VLMs, moving them closer to truly understanding complex, real-world scenarios and providing more reliable and interpretable outputs.

## 2 The NL-Eye Benchmark

### 2.1 Tasks

Our objective is to explore and benchmark the abductive reasoning capabilities of modern VLMs. Unlike much of the previous work in NLP, our focus is on reasoning solely based on visual inputs: premise and hypothesis images. The premise image illustrates the context – factual observations about the world and a starting point from which conclusions are drawn. The hypothesis image illustrates a candidate conclusion – a possible event that could occur before, after, or simultaneously with the scenario presented by the premise image. In the context of our study, we refer to the definition of abductive reasoning Nie et al. (); Douven () as a form of logical reasoning that seeks the most plausible hypothesis (conclusion) given a premise (a set of observations).

To perform visual abducting reasoning, the VLM should identify objects and their relationships within each image, understand the relationships between the images, and integrate this information to reason about the plausibility of the hypotheses. We introduce two novel tasks to evaluate those capabilities: Plausibility Prediction and Plausibility Explanation. In the prediction task, the VLM is provided with the premise and hypothesis images. Its goal is to predict the plausibility of the hypothesis images or to determine which one is more plausible.
We argue that VLMs should be capable of not only predicting plausibility but also providing a sensible explanation of their reasoning process. Therefore, in the explanation task, the model is also required to generate a free-form textual explanation justifying why the chosen hypothesis is plausible or at least more plausible than the other.

Figure: Figure 2: Fully annotated examples from NL-Eye. Each example includes the three images, the textual descriptions (prompts) used to generate them, the gold label, an explanation for why the gold is more plausible, and indications of the reasoning category and temporal direction and duration.
Refer to caption: x2.png

### 2.2 Benchmark Structure and Categorization

In this subsection, we describe the structure of each example in our benchmark and discuss the taxonomy we proposed for categorizing the examples. In Figure 2, we present the structure of two examples, which contains: (1) the premise image; (2) two hypothesis images; (3) the label, which indicates the more plausible hypothesis and is given by the benchmark designers; (4) the textual descriptions of the three images that were used for generating the images; (5) the gold explanation, which clarifies why the correct hypothesis is more plausible, and is written by the benchmark designers; (6) reference explanations, which were written and validated by crowd-workers; (7) categorization of the example, which indicates the involved reasoning category, temporal direction and duration.

In §3, we describe the data creation process and specifically elaborate on components (1)-(5). The crowd-worker annotations (component 6) are detailed in §4.2. We next outline our proposed categorization, which serves two purposes: first, to ensure our benchmark is diverse, balanced, and covers a wide range of domains and reasoning types; and second, to aid in diagnosing areas where VLMs fall short.

#### Reasoning Categories

We identify six different categories: Physical, Logical, Emotional, Functional, Social, and Cultural, ranging from physical reasoning (e.g., predicting the color of a rotten banana) to cultural reasoning (e.g., determining if a habit like wearing house shoes implies another cultural trait, such as owning Matryoshka dolls).
Figure 3 presents an example from each category, with formal definitions in Appendix C.1.

#### Temporal Categories

The temporal categories are based on direction and duration. Temporal direction refers to the logical relationship between the premise and hypothesis, indicating whether the event depicted in the premise image causes the hypothesis event (forward), is caused by it (backward), or if the events occur simultaneously (parallel). Examples that do not occur at the same time and are not categorized as parallel can also be classified by temporal duration, which is determined by the time gap between the events depicted in the premise and hypothesis images. These include short-term – when the events occur close in time, possibly in the same physical environment, or with no significant sequence of events separating them, and long-term – when the events take place in noticeably different periods of time.
For instance, in the example on the left in Figure 2, the grated carrots suggest a short-term forward progression within the same environment as the whole carrot in the premise.

Figure: Figure 3: Real examples from each reasoning category in NL-Eye. The more plausible hypotheses are framed in green. The gold explanations are included below each sample.
Refer to caption: x3.png

## 3 Data Curation

Figure: Figure 4: NL-Eye data curation workflow scheme. The process includes three steps: (1) writing textual descriptions, (2) generating images, and (3) generating explanation and categorization. Yellow denotes steps that require human involvement while turquoise denotes model-based generations.
Refer to caption: x4.png

Joining recent efforts in evaluating VLMs with an emphasis on the quality of test sets over their sheer size (Thrush et al., ; Bitton-Guetta et al., ; Bitton et al., ; Bitton-Guetta et al., ; Padlewski et al., ), we carefully curated 350 test set examples.
The creation process of NL-Eye required human involvement at every key step (see Figure 4), enabling the creation of diverse, high-quality examples tailored to the evaluation’s specific goal.

NL-Eye is a multi-image benchmark consisting of daily life scenes. A “scene” refers to a specific setting where objects, people, and actions are arranged in a particular context, which can be represented either textually or visually. The benchmark includes both representations, and the following key steps in its creation process: (1) textual description writing, (2) image generation, and (3) explanation and categorization.

#### Textual Descriptions

Scenes were manually crafted by a group of 20 annotators who were tasked with creating triplets consisting of a premise scene, and two hypothesis scenes, while one is more plausible than the other (see the first step, “scenes writing”, in Figure 4). Each annotator had the flexibility to develop hypotheses across diverse reasoning categories, time directions, time durations, and domains. Annotators’ creativity and experiences generated unique, everyday scenes that are often undocumented or scattered, making it hard to gather automatically similar data.
We manually filtered scenes from the suggested pool based on several key criteria: (1) premise necessity, ensuring the scene is essential for determining the more plausible hypothesis; (2) visual relevance, guaranteeing the scenes can be effectively communicated visually; and (3) uniqueness, verifying that we do not replicate similar existing examples or logical patterns (see examples in Appendix Table C.2). We also applied preferences for receiving a range of challenges, from easy to difficult, as well as diverse time shifts, including varying directions and durations. After applying these filters, we retained 75% of 450 suggested ideas.

#### Image Generation

The images in NL-Eye were manually curated by two of the authors (noted as “the designers”), who have experience with text-to-image models. This careful generation process ensures high-quality images and verifies consistency, alignment between text and images, and overall clarity. The images were generated based on the textual descriptions using mainly Midjourney and DALL-E 3 (Ramesh et al., ). During the prompt augmentation phase, The designers had the option to utilize assistance from Gemini and GPT-4 (Achiam et al., ) to transform the human-generated concise descriptions from step 1, into more detailed prompts with specific visual elements, enhancing visual consistency (see Appendix §C.2). For example, the step 1 textual description, the boy is crying, turns to the augmented prompt, the curly redhead boy with the striped green t-shirt is crying. Once the revised prompts were verified to ensure they don’t interfere with the essential content (e.g., a change such as the teenager is crying or an omission such as the curly redhead boy is wearing striped t-shirt) and manual edits were made if necessary, the prompts were ready for image generation.

Typically, the process begins with generating the premise image. Image generation is an iterative process, involving repeated cycles of manual editing and image-to-image alignment until high quality and consistency are achieved. The image generation phase produces photorealistic images that are visually consistent, meaning that objects, people, and environments appearing in one image of the triplet are the same as in the others. The last guideline arises from the crucial need to exclude style from reasoning considerations in the future evaluation of VLMs on the task. Technically, visual consistency is achieved not only through prompt augmentations but also via inpainting (i.e., editing a specific region of an image using a textual prompt), image-conditioned prompting (generating an image while conditioning on another image), and using the same seed (initial noise distribution number) for all triplet images. See an example of image generation, step 2, in Figure 12 in Appendix §C.2.

#### Explanation

For each example, the designers wrote gold-standard explanations (^2^22The benchmark contains the gold explanations and additional explanations written and validated by annotators, see §4.3.) The gold explanation represents the original reasoning behind the scenes at the time of their curation. The gold explanation clarifies why the correct hypothesis is a more plausible outcome or cause of the premise. It often follows a pattern like “Usually, X tends to Y” or “Because X made Y to…”. Naturally, the explanation written at this stage is not the only possible explanation of the reasoning. Humans can suggest multiple plausible explanations and stories to justify connections between observations, also even for less likely scenarios. For example, examining the example of the Social reasoning category in Figure 3 (bottom-right row), the premise image depicts a wallet with a family photo, the hypothesis images depict a man’s hand with a wedding ring (plausible) and a man’s hand without a wedding ring (less plausible), and the human explanation is a man with a family photo in his wallet is socially and statistically more likely to be married rather than single. However, the man might be married but not wearing a ring, have a family without being married, or the scenes might tell a story of loss and remembrance.

#### Validation and Categorization

The validation process consists of two key checks: (1) image-text alignment and (2) plausibility validation. First, we ensured that the images were correctly aligned with their corresponding texts. Second, we qualitatively assessed each example’s plausibility, evaluating how difficult it is to understand the intended meaning and make any necessary adjustments. This process includes manual verification by the designers, supported by a human baseline (§4.3) with a high human success rate on the plausibility prediction task (85%; see §5) and strong inter-annotator-agreement (67.6%; unanimous votes) confirms the clarity.In addition, the designer classified the examples into relevant categories, as outlined in the previous section.

#### Dataset Statistics

NL-Eye is categorized by reasoning categories, domains and temporal information. The Logical, Social, Physical, Cultural, Functional, and Emotional reasoning categories comprise 28%, 24.6%, 17.8%, 14.3%, 7% and 8%, respectively, of the benchmark examples.
78% of the examples are in the time duration of short-term, divided mainly (68%) with the forward direction. The long-term examples are 22% while 27% of them are also associated with the backward direction. Refer to the histogram of reasoning categories and real NL-Eye examples in Appendix Figures 13 and 17, respectively.

## 4 Experimental Setup

**Table 1: Models and baselines by their input strategy and reasoning approach.**
$$ $\displaystyle\text{Consistency Acc.}(I_{P},I_{H1},I_{H2},H_{gold})=\begin{ cases}1,&\text{if }\quad f(I_{P},I_{H1},I_{H2})=f(I_{P},I_{H2},I_{H1})=H_{gold }\\ 0,&\text{otherwise}\end{cases}$ $$

### 4.1 Tasks and Setups

Recall that the NL-Eye benchmark includes two tasks: Plausibility Prediction and Plausibility Explanation. Both tasks require reasoning about the relationship between the context image, the premise, which is denoted as $I_{P}$, and a candidate image, a hypothesis, denoted by $I_{H}$. In addition to the images, the model $f$ receives a textual query (a prompt) that contains instructions describing the task it should perform. We introduce two setups for solving the tasks: the Triplet setup and the Pairs setup.

#### The Triplet Setup

which is illustrated in the left box of Figure 5 the model receives the query along with three images: the premise ($I_{P}$) and two candidate hypotheses ($I_{H1}$ and $I_{H2}$). In the prediction task, the model’s goal is to determine which of the two hypotheses is more plausible given $I_{P}$, i.e., which is more likely to occur, assuming $I_{P}$ is a true observation about the world. For the explanation task, the model is also required to generate a textual explanation justifying why the chosen hypothesis is more plausible than the other.

#### The Pairs Setup

which is illustrated in the right box of Figure 5, the model $f$ receives the query and two images: the premise $I_{P}$ and the hypothesis $I_{H}$. The prediction task now is to provide a plausibility score that indicates how plausible $I_{H}$ is, given that $I_{P}$ is a factual observation. In our experiments, this score is provided on a 1-10 Likert scale. However, this is not mandatory – the plausibility score can be adapted to the needs of the model developer. For instance, the score could be expressed as a probability or another appropriate metric. In the explanation task, the model is asked to explain why $I_{H}$ can be plausible given $I_{P}$.

#### Reasoning Approaches & Input Strategies

To thoroughly assess the abductive reasoning capabilities of current models, we use two reasoning approaches: Vision-based – where the model is tasked with performing the entire task end-to-end based solely on the visual input, and Text-based – where the final plausibility reasoning is based on textual input.
In the vision-based approach, we experiment with two input strategies for feeding input to the model: (v.1) combined-image – where we concatenate the images (with the premise on the left) to form a single combined image, and (v.2) separate-images – where we feed in one prompt the images to the model separately, starting with the premise. As not all models support both strategies, Table 4 specifies which strategy was used for each model in our study.
In the text-based approach, we utilize two input strategies as well: (t.1) image-to-text – where we ask the model to describe the two images in natural language, and then, using those descriptions, the same model or another performs the plausibility prediction, and (t.2) text-only – we discard the visual inputs altogether and use only the textual descriptions generated when the images were created (see §3).
The vision-based and text-based approaches allow us to understand the model’s weaker capabilities better.

### 4.2 Evaluation

#### Predictions in the Triplet Setup

At first, we evaluated models based on accuracy. However, we found that all models are sensitive to the positioning (in the all-in-one strategy) or the order (in the sequential strategy) of the hypothesis images, that is, whether $I_{H1}$ is placed or fed before or after $I_{H2}$. For example, models may perform differently when given Triplet A versus Triplet B from Figure 5. To address this sensitivity, we provide predictions for both orders of the hypotheses and then aggregate the two predictions. A prediction is considered correct if and only if the model selects the correct hypothesis in both orders. This approach reduces the likelihood of a correct prediction by chance and ensures the model demonstrates consistency. The performance score in the triplet setup is the described consistency accuracy (proportion of correct and consistent predictions; see §A.1).

#### Predictions in the Pairs Setup

In the pairs setup, we aim to evaluate the plausibility score predicted by the model. However, as previously mentioned, we do not want to constrain the model (or developers) to produce a specific score or adhere to a specific scoring function. Therefore, we do not support direct evaluation of the score, i.e., we do not provide a gold standard score against which the predicted score is compared. This raises the question: how do we plan to evaluate models in the pairs setup?
The only assumption we require from the scoring function is order-faithfulness (Gat et al., ): if for a given premise $I_{P}$, the evaluated model $m$ scores one hypothesis $I_{H1}$ higher than another hypothesis $I_{H2}$, then $I_{H1}$ should genuinely be more plausible than $I_{H2}$. Accordingly, for every premise image, we take the two hypotheses and consider the scores of $f$ as correct if and only if the hypothesis scored higher is the gold plausible hypothesis. The performance score in the pairs setup is the described order-faithfulness accuracy (proportion of correctly ordered scores; see §A.1).

#### Human Evaluation of Explanations

Evaluating free-text explanations is a challenging task due to the various ways explanations can be paraphrased and the reasoning involved in determining their validity. To address this, we followed the efficient human evaluation protocol proposed by Bitton-Guetta et al. () and recruited crowd workers from Amazon Mechanical Turk (AMT).
For each triplet of images, the workers were presented with the correct hypothesis and several explanations either written by humans (from the human baseline described in the next subsection) or generated by VLMs. We included only explanations of the correct hypothesis. Then, the workers were tasked to select all the explanations that are logic and justify why the correct hypothesis is more plausible (see Appendix C.3 for additional details).
We consider an explanation as correct if at least one worker selected it. The human evaluation score we present is the proportion of correct explanations.

#### Automatic Evaluation of Explanations

Through automatic evaluation, we aim to demonstrate a more scalable and cheaper approach to assessing the validity of model explanations. Like other recent works, we follow the common practice of employing an LLM as a judge (Zheng et al., ; Chen et al., ). Notice, that current models perform poorly on our visual abductive reasoning tasks, thus, expecting them to succeed in evaluating the validity of explanations generated by other models is pretentious. Instead, we simplify the task by conducting a reference-based evaluation, asking the LLM to determine whether the generated explanation aligns with a gold reference explanation – a task that relies solely on textual reasoning.
The reference explanations include the gold explanations (see §3), augmented with human-written explanations approved during the human evaluation stage. To perform the automatic evaluation, we instruct an LLM (GPT-4o) to determine if the generated explanation aligns with one of the reference explanations (more details are provided in §A.1). The automatic evaluation score is the proportion of generated explanations that the judge LLM predicted as aligning with the references.

### 4.3 Models and Baselines

Table 4 outlines the models used in our study, detailing their configuration, reasoning approach, and input strategy (specific versions in Appendix Table A.1). Below, we provide more details on the models and baselines.

#### VLMs

We employ state-of-the-art closed source VLMs, including, Gemini-1.5-pro (Google, ), GPT-4-vision and GPT-4o (Achiam et al., ), and Claude-Sonnet-3.5 and Claude-Opus-3 (Anthropic, ).
In addition, we employ open-source VLMs, including LLaVa 1.6 (Liu et al., ) and Fuyu (Bavishi et al., ).

#### NLI models

Recall that in the text-only reasoning approach, we provide the model with the gold text descriptions (used to generate the images) and ask it to predict which hypothesis description is more plausible. The predictor models we use in this approach include all the closed-source VLMs mentioned above, as well as fine-tuned NLI models such as DeBERTa-v3 (He et al., )
and BART-L (Lewis et al., ). When the predictor is an LLM, we ask it to determine which hypothesis description is more plausible given the textual premise description. For fine-tuned NLI models, we compute two ‘entailment’ scores between the premise and each hypothesis, and the final prediction is made based on the hypothesis with the higher score.

#### Random baselines

We present two simple baselines. The first is the random baseline, which randomly selects a hypothesis in the triplet setup or assigns a random score in the pairs setup. However, it is inconsistent due to its sensitivity to hypothesis order. To improve consistency, we introduce the dumb pixel baseline, which selects a hypothesis or assigns a score based on a predefined rule using the upper-leftmost pixel. For example, the hypothesis with the brighter pixel is deemed more plausible, or the score is calculated from the pixel’s value.

#### Humans

Currently, there are indications that models can perform inference tasks at a level comparable to, or even exceeding, that of humans. Accordingly, we want to investigate whether these VLMs can match human performance on our tasks that appear straightforward for humans and expect them to succeed. To this end, we recruited 15 crowd-workers on the AMT platform. Pre-qualifications for workers were high approval rates and English-speaking countries. Additional details and guidelines are provided in §C.3.

**Table 2: Main results: Scores for vision-based experiments. Automatic evaluation scores are not presented for Humans since their explanations serve as references. Regardless of the input strategy, VLMs are greatly outperformed by humans and mostly perform on par or even below the baselines.**
$$ $\displaystyle\text{Consistency Acc.}(I_{P},I_{H1},I_{H2},H_{gold})=\begin{ cases}1,&\text{if }\quad f(I_{P},I_{H1},I_{H2})=f(I_{P},I_{H2},I_{H1})=H_{gold }\\ 0,&\text{otherwise}\end{cases}$ $$

## 5 Results

### 5.1 VLMs Fail to Perform Abductive Reasoning With Images

Table 4.3 presents the performance of humans, VLMs, and baselines on both tasks, prediction and explanation, for different setups and input strategies. For detailed results, refer to Appendix §B.1 and §B.2, where we compare human performance to VLM performance and analyzing their alignment.

#### VLMs Fail Where Humans Excel

The results reveal a large performance gap between humans and VLMs on both tasks. Except GPT-4o, which achieves 60% accuracy in the triplet setup for combined-image inputs, all VLMs perform worse than the dumb pixel baseline. The situation is even more concerning for current open-source VLMs, such as LLaVA 1.6 and Fuyu, which score below random baselines. In contrast, human participants achieve 83-85% accuracy in the prediction task and 95% in the explanation task. Notably, the participants are crowd-workers who are not experts or highly skilled. This suggests that the task is neither unsolvable nor particularly difficult. Rather, current VLMs lack the visual abductive reasoning capabilities necessary to solve it effectively. Importantly, the finding that tasks easily handled by humans pose significant challenges for VLMs underscores the relevance of our benchmark and highlights areas where the research community can focus its efforts. In addition, we found that VLMs are better in comparative or relative judgment setups (triplet, selecting which hypothesis is more plausible than the other) than in absolute judgment (pairs, predicting a plausibility score for a single hypothesis). This is unsurprising, as it is a known and well-studied phenomenon of humans (Pollitt, ; Verhavert et al., ) which was also observed in LLM-as-a-judge tasks (Kim et al., ). To the best of our knowledge, we are the first to show it for visual reasoning tasks.

#### Even When Correct, VLM Explanations Are Unhelpful

To assess the quality of the explanations, we conduct human and automatic evaluations.
Since nearly half of the predictions are incorrect, we focused only on explanations for correct predictions, ensuring we can determine whether an explanation is genuinely poor rather than simply a result of the model failing to predict the correct answer. Note that in §6 we provide a qualitative analysis of explanations of wrong predictions to better understand why they fail. Table B.1 in the Appendix reports the number of evaluated explanations of each model (a total of more than 3,800), and the results are reported on the two rightmost columns of Table 4.3 (see human votes distribution in Appendix Table B.1). As can be seen, humans almost always produce correct explanations, as 95% of the explanations were selected by the annotators. On the other hand, at best, only half of the explanations are selected. This demonstrates that even when the VLMs predict correctly, the explanation is unhelpful. In addition, we used valid human explanations as references for the automatic evaluation, which produces scores similar to those of the human evaluation in most cases. The automatic evaluation suggests that VLMs produce explanations that describe different reasoning than humans.

**Table 3: Text-Based: Performance for prediction in the triplet setup. Predictor models perform well and similarly to (vision-based reasoning of) humans when using the gold description. However, VLM describers generate useless captions which do not help solve the task.**
$$ $\displaystyle\text{Consistency Acc.}(I_{P},I_{H1},I_{H2},H_{gold})=\begin{ cases}1,&\text{if }\quad f(I_{P},I_{H1},I_{H2})=f(I_{P},I_{H2},I_{H1})=H_{gold }\\ 0,&\text{otherwise}\end{cases}$ $$

### 5.2 Reasons For Failing to Reason

This subsection presents experiments and analyses to explore why VLMs fail at visual abductive reasoning.

#### VLMs Can Perform Textual Reasoning – The Failure is in Visual Interpretation

Table 5.1 presents the results of text-based experiments aimed at decoupling the textual reasoning capabilities from the visual ones. In the text-only approach, the models are provided with gold-standard descriptions of the images. As shown in the table, the performance of all models, including smaller fine-tuned NLI models, is significantly higher than their performance in vision-based experiments. Strong VLMs, such as GPTs and Claudes, achieve around 80% accuracy, nearing human performance. This indicates that VLMs are capable of textual reasoning.
This finding suggests that the reasoning challenge does not mainly lie in the VLM’s textual components but in the visual ones.
In open-source VLM architectures (Liu et al., ), inference is not performed directly over images. Instead, the models encode images to latent visual representations, which are then passed to the language model component. We hypothesize that poor visual inference results from these visual representations being inaccurate for the reasoning task.

In contrast, when VLMs were tasked with generating descriptions for each image, and these descriptions were used as input to GPT-4o, instead of the gold ones, the performance dropped significantly, aligning with the results from the vision-based experiments. We hypothesize that this indicates a recognition gap, where the generated descriptions either lack sufficient detail to capture the necessary information or are overly detailed (see examples in Appendix Figure 8), making it challenging to reason effectively.

Finally, consider the performance gap in Table 4.3 between separate and combined image input strategies. Except for GPT-4o, which consistently predicts the first hypothesis as more plausible (see Table B.1 in the Appendix), the other three models perform better when using separate image inputs, showing an average improvement of over 10% (48.6% vs. 37.3%). This suggests that when an image is complex and contains many details, as in the case of a combined image, VLMs struggle to encode the necessary details and represent each image successfully.

#### VLMs Predictions Depend on Hypothesis Location

Another factor contributing to the low performance of VLMs in the triplet setup is their lack of consistency. As shown in Table B.1 in the Appendix, all VLMs are highly sensitive to hypothesis order, with performance variations ranging from 5-80%. For example, Gemini-1.5-pro’s performance in the combined-image strategy drops from 82% when the second hypothesis is correct to 46% when the first hypothesis is correct. Similarly, GPT-4o, which performs best in the combined-image strategy (60%), fails in the separate-images approach, scoring 97% when the first hypothesis is correct but only 16% for the second. This suggests GPT-4o almost always predicts the first hypothesis as correct. In contrast, VLMs show much greater consistency in text-only inputs, with performance variation limited to 7%. This indicates that VLMs rely on weak visual encoding, capturing superficial patterns like hypothesis order rather than meaningful image content.

#### VLMs are Better in Correlational and Knowledge-based Reasoning Compared to Causal Reasoning

Relying on the categorization within our benchmark, we present in Figure 6 the vision-based results in the triplet setup for six reasoning categories (left plot) and five temporal categories (right plot). VLMs exhibit a clear dichotomy in their reasoning abilities, excelling in some areas while falling short in others. Interestingly, the patterns are consistent across models, yet they diverge from the performance patterns observed in humans. For example, VLMs perform best in Social and Cultural reasoning, where specific knowledge is key to correctly solving those examples, see Ventura et al. () for extended discussion about VLMs and cultural knowledge.
In contrast, humans perform worst in the Cultural category.
Another interesting observation is that VLM performance on parallel reasoning examples is higher than on forward and backward reasoning tasks. Notably, parallel reasoning may require only understanding correlations between the premise and the hypothesis, whereas forward and backward reasoning require causal reasoning – identifying causes (in backward) or effects (in forward). This suggests that VLMs may be more adept at identifying correlations rather than causal understanding.
Finally, the weakest category for VLMs is Emotional, which aligns with the literature (Lissak et al., ).

## 6 VLM Failure Analysis

This section presents a qualitative analysis of VLMs’ explanations for wrong predictions. We qualitatively analyze 120 explanations – 40 from each VLM: Gemini 1.5, GPT-4 and Claude 3.5.

Figure: Figure 7: Failure factors of model explanation for incorrect plausibility prediction.
Refer to caption: x7.png

We identify five main factors contributing to the models’ failures: (1) style & consistency: When irrelevant visual details influence the decision; (2) time: When the explanation relies on incorrect time direction or duration; (3) ignoring key details: Overlooking important information; (4) missing knowledge: Misinterpreting key details despite recognizing them; (5) failed comparison: Justifying a less plausible hypothesis with logical reasoning. Table B.2 in the Appendix presents illustrative examples of these factors.

As Figure 7 shows, all models struggle with understanding temporal progression. Notably, Claude often relies on style considerations, with 30% of its errors resulting from this factor, indicating an overemphasis on irrelevant visual details. Both Gemini (32%) and GPT-4 (25%) frequently miss key details, suggesting recognition gaps. GPT-4 has the highest rate of failed comparisons, often making the incorrect decision at the final plausibility stage. To further understand these failures, researchers can try to interpret models’ internal thought processes (Toker et al., ).

## 7 Related Work

Recent advances in multimodal learning have enabled models to integrate textual and visual data across diverse tasks (Voulodimos et al., ; Guo et al., ; Qin et al., ). Powerful visual encoders like CLIP (Radford et al., ; Cherti et al., ) and SigLip (Zhai et al., ), coupled with the progress in LLMs (Chowdhery et al., ), have given rise to sophisticated VLMs such as BLIP2 (Li et al., ), GPT-4 (Achiam et al., ), and Gemini (Google, ). These VLMs are pushing the boundaries of multimodal capabilities, tackling tasks like visual question answering (VQA) (Antol et al., ) and visual entailment (VE) (Xie et al., ). Our work builds on and extends research in these areas, with an emphasis on commonsense reasoning.

From Textual to Visual Entailment  A cornerstone of our work is the expansion of Natural Language Inference (NLI), traditionally a text-based task (MacCartney, ; Dagan et al., ; Gekhman et al., ), into the visual domain. While previous research has explored NLI in the context of image-text alignment (e.g., SNLI-VE (Xie et al., ), TIFA (Hu et al., ), WYSIWYR (Yarom et al., ), Mismatch-Quest (Gordon et al., )), and even video-text entailment (Xu et al., ; Bansal et al., ), we introduce a novel framework for image-to-image entailment.
This framework goes beyond simply selecting plausible alternatives by requiring models to explain their choices, thus offering a deeper evaluation of their abductive reasoning.
Furthermore, we introduce a “pairs” setup that requires scoring the plausibility of image pairs, aligning our task more closely with the original formulation of textual entailment.

Synthetic Data for Multi-Image Reasoning  Our work uniquely employs synthetic images generated by models like DALL·E (Ramesh et al., ), allowing greater control over visual complexity and diversity compared to natural image datasets such as Winoground (Thrush et al., ), Sherlock (Hessel et al., ), and VCOPA (Yeo et al., ). By emphasizing multi-image reasoning, we address limitations in existing datasets that focus primarily on single-image tasks, like WHOOPS!(Bitton-Guetta et al., ) and Visual Riddles(Bitton-Guetta et al., ). Our approach complements research on synthetic image understanding (Gokhale et al., ; Wu et al., ; Stöckl, ) and is better suited for commonsense reasoning in real-world contexts, enhancing datasets like SEED-Bench (Li et al., ) and MMToM-QA (Jin et al., ), which tackle different aspects of multi-image reasoning. Unlike ScienceQA (Lu et al., ) and NTSEBench (Pandya et al., ), which focus on diagrams and scientific domains, our dataset employs photorealistic scenes from everyday life, making it more appropriate for evaluating commonsense reasoning. This integration of synthetic data, multi-image reasoning, and image-to-image entailment establishes a new benchmark for assessing VLMs’ reasoning capabilities.

## 8 Conclusion

We introduced NL-Eye, a benchmark designed to assess the visual abductive reasoning capabilities of VLMs across multiple images. This skill is essential for real-world applications, such as accident-prevention bots. We paid special attention to detail in order to ensure that NL-Eye consists of high-quality and challenging examples, which required extensive human involvement at every stage of its curation. Our carefully designed study highlights critical challenges faced by modern VLMs in delivering satisfying plausibility predictions. We demonstrate that although humans perform well on these tasks, VLMs struggle significantly. This indicates a significant limitation of current models’ ability to integrate visual interpretation with logical reasoning. Furthermore, models not only struggle to make correct predictions but also often fail to consistently provide helpful explanations. In future work, we would like to address these gaps, building on our insights to develop new VLM architectures with higher reasoning skills, mirroring human cognitive processes in complex environments.

## Ethics Statement

The NL-Eye benchmark includes AI-generated images, with the potential presence of unpleasant or insensitive content. While we strive to minimize harmful biases, the inclusion of reasoning based on common sense knowledge and cultural perspectives may introduce further bias, particularly related to social norms. Additionally, the labels in this benchmark are based on consensus from human annotators, whose judgments may be influenced by their own cultural backgrounds, which could amplify bias. We also recognize the challenges related to text-to-image (TTI) copyrights, where the ownership of AI-generated content remains unclear. Researchers are encouraged to carefully consider these ethical and legal concerns when utilizing the benchmark.

## Reproducibility

To ensure the reproducibility of our results and promote further research, we will publicly release the NL-Eye benchmark, along with the code. Detailed technical instructions, as well as documentation on how to use and adapt the benchmark, will be provided in a publicly accessible repository. Additional technical details, including model versions and specific configurations used in the experiments, are available in the Appendix (§A.1). By sharing these resources, we aim to foster transparency and support the research community in advancing the evaluation of VLMs.

## Appendix A Appendix

### A.1 Reproducibility and Resources

**Table 4: API and version of closed-source models used for inference on NL-Eye tasks.**
$$ $\displaystyle\text{Consistency Acc.}(I_{P},I_{H1},I_{H2},H_{gold})=\begin{ cases}1,&\text{if }\quad f(I_{P},I_{H1},I_{H2})=f(I_{P},I_{H2},I_{H1})=H_{gold }\\ 0,&\text{otherwise}\end{cases}$ $$

#### Prompt of Auto Evaluation of Explanation.

We combine classes 0 and 1 as a false (invalid) explanation, and 3 as a positive (valid) explanation:

Context: <premise textual description>,
Plausible Hypothesis: <hypthesis 1 textual description>,
Less Plausible Hypothesis: <hypothesis 2 textual description>.
Return 0 (Not logical at all), 1 (Logical but different), or 2 (Logical and same as one of the gold)
if the candidate’s explanation presents the same logical common sense as appears in one of the gold explanations for justifying the plausible hypothesis.
Candidate explanation: <candidate explanation>.
Gold explanations:
Explanation 1: <gold explanation>…

#### Accuracy Metrics: Mathematical Formulation

Here we present the mathematical formulation of the accuracy metrics, based on the notations in Section §4).

Formally, the consistency-accuracy (triplet accuracy) is:

$$ $\displaystyle\text{Consistency Acc.}(I_{P},I_{H1},I_{H2},H_{gold})=\begin{ cases}1,&\text{if }\quad f(I_{P},I_{H1},I_{H2})=f(I_{P},I_{H2},I_{H1})=H_{gold }\\ 0,&\text{otherwise}\end{cases}$ $$

Formally, the order-faithful accuracy (pairs accuracy) is:

$$ $\displaystyle\text{order-faithful Acc.}(I_{P},I_{H1},I_{H2},H_{gold})=\begin{ cases}1,&\text{if }\quad f(I_{P},I_{H1})>f(I_{P},I_{H2})\quad\text{and}\quad H 1=H_{gold}\\ 1,&\text{or }\quad f(I_{P},I_{H1})<f(I_{P},I_{H2})\quad\text{and}\quad H2=H_{ gold}\\ 0,&\text{otherwise}\end{cases}$ $$

## Appendix B Complementary Results

### B.1 Results of Plausibility Prediction and Explanation

**Table 6: Plausibility prediction results of the triplet setup. Order refers to the position of the correct hypothesis image in the input, whether it was presented first (order 1) or second (order 2).**
|  |
| --- |
| (a) MTurk Human Explanation Evaluation Instructions. |
|  |

### B.2 Failure Analysis

#### Are Human and Model Difficulties Aligned?

In the plausibility prediction task using the triplet setup of the separate-images input strategy, humans and VLMs agree on 55% of the predictions (full comparison in Appendix Table B.1).
When humans are incorrect, the model’s success rate falls to 30%, below random chance. While the model outperforms humans in only 5% of cases, its explanations in these instances are rarely accurate. Furthermore, only 21% of the models’ errors overlap with human errors, indicating that humans and models tend to make different types of mistakes. Additional examples are shown in Figures 10 and 11.

**Table 11: Failure factors with examples, as illustrated by the following scene - a premise of a man in a hospital bed with a broken leg and two hypotheses: a wet floor with (less plausible) and without (plausible) a warning sign (Figure 1).**
|  |
| --- |
| (a) MTurk Human Explanation Evaluation Instructions. |
|  |

## Appendix C NL-Eye Dataset Creation - Complementary Information

### C.1 Reasoning Categories Definitions

#### Physical Reasoning.

Involves understanding the physical world and how objects interact within it. The scenes include changes in temperature, phase, shadow’s location, color, shape, etc. This reasoning is inspired by the spatial-temporal reasoning definition (Deza et al., ).

#### Functional Reasoning.

Requires an understanding of objects’ functionalities and tools’ common usage. This type of reasoning involves not just recognizing objects and tools, but also comprehending their intended purposes and how they interact within various contexts. For instance, a hammer is not merely identified by its shape but also by its function of driving nails into surfaces. Functional reasoning allows a model to infer the appropriate use of an object within a given scenario, such as using a knife for cutting or a broom for sweeping.

#### Social Reasoning.

Understanding social norms, relationships, and interactions. Social reasoning allows for the comprehension of social norms and etiquette, such as knowing how to greet someone depends on the context. This includes recognizing familial roles, friendships, professional relationships, and the varying degrees of formality and familiarity in interactions.

#### Emotional Reasoning.

Understanding and interpreting emotions and emotional responses. It refers to the ability to identify a wide range of emotions, including happiness, sadness, anger, fear, surprise, and disgust, and to understand the context in which these emotions arise.

#### Cultural Reasoning.

Involves acknowledging cultural traits and traditions while correctly associating them with their respective cultures. It includes the ability to recognize and interpret cultural symbols, rituals, languages, and behaviors accurately. For instance, it includes understanding that certain gestures may have different meanings in different cultures or that specific holidays and celebrations are unique to particular cultural or religious groups.

#### Logical Reasoning.

Requires an understanding of general processes and broad commonsense. It enables the analysis of situations, draw inferences, and make decisions based on logical principles and widely accepted knowledge. Logical reasoning involves the ability to follow a sequence of steps to solve problems, recognize patterns, and identify relationships between different pieces of information.

**Table 12: NL-Eye textual descriptions examples. One example for each reasoning category. Every example consists of a premise phrase, a plausible hypothesis phrase, and an implausible hypothesis phrase.**
|  |
| --- |
| (a) MTurk Human Explanation Evaluation Instructions. |
|  |

### C.2 Image Generation Step

**Table 13: Examples of suggested textual descriptions (scenes) filtered by specific criteria.**
|  |
| --- |
| (a) MTurk Human Explanation Evaluation Instructions. |
|  |

#### Text-to-Image prompt.

The Text-To-Image prompt (in Midjourney) is consisted of 3 parts, while the last one is optional:

- •
Text description. The textual scene caption, basic or improved.
- •
Photorealistic style. Adding textual styling of photorealistic images by mentioning it is captured with Nikon D850.
- •
Visual consistency. Making an image consistent with another image by setting the same seed number, and referring to the reference image with the flag cref and its conditioning strength with the flag cw ranging from 0-100.

All these parts are aggregated into the following template:

< prompt caption >, captured with a Nikon D850 and a 24-70mm lens at f/2.8 –seed <> –cref <> –cw 80

#### Prompt augmentation.

Augmenting a text description by prompting Gemini or GPT-4 with the following prompt:

Describe visually a specific looks of < interacting component1 >, < interacting component2 > and < environment >.
keep it short and concise, and avoid NSFW words.
and integrate these details into every reference of them in the following captions smoothly and consistently.
do not change the content of the captions besides the visual description integrations.
return in a JSON format:
1) <first image caption>
2) <plausible second image caption>
3) <implausible second image caption>

Note: integrate the environment only if it fits the context of the caption.
For example:
interacting component1: little child,
interacting component2: vaccine,
environment: nurse room,
first image caption: a child gets a vaccine.,
plausible second image caption: a child cries after getting a vaccine.,
implausible second image caption: a child smiles after getting a vaccine.
Response:
improved first image caption: a short curly-haired child wearing a green t-shirt receives
a vaccine with a silver syringe in a nurse’s room filled with toys.
improved plausible second image caption: the short curly-haired child in a green t-shirt
cries after receiving a vaccine in the toy-filled nurse room.
improved implausible second image caption: the short curly-haired child in a green t-shirt smiles after receiving a vaccine in the nurse’s room.

Figure: Figure 12: Zoom into the image generation step in NL-Eye curation, as seen in Figure 4. Yellow color notes a hand-curated stage, while turquoise notes a model-generated stage. All stages require human involvement for fixing, editing and validating.
Refer to caption: x12.png

Figure: Figure 13: Dataset Analysis. A histogram of the NL-Eye examples. The benchmark is also annotated with diverse domains (left): administration, business & work, daily life & hospitality, education, healthcare, religion, science & technology, sports & arts and transportation, and representation of time duration and direction (right) in every reasoning category. Parallel in noted by ”parallel-short”.
Refer to caption: x13.png

### C.3 AMT: Human Performance & Evaluation

Figure: Figure 14: Guidelines for the crowd-workers. Guidelines for the human baseline on the plausibility prediction and plausibility explanation tasks (left), and for human evaluation of explanations (right).
Refer to caption: x14.png

#### Human Performance - Plausibility Prediction and Explanation.

Crowd-workers were instructed to complete the plausibility prediction and explanation tasks based on the following guidelines, in Figure 14 and the questions in Figure 15.

#### Human Evaluation of Explanations.

The explanations are presented in a multiple-choice question format (see Figure 16), where the crowd workers are instructed to select explanations that demonstrate logical reasoning and clearly justify why the correct hypothesis is more plausible than the other candidate (see Figure 14). We conduct the human evaluation of explanations across all the input formats in the vision-based reasoning approach (Triplet setup), focusing on the explanations associated with the correct plausibility predictions, resulting in a total of 3.8k explanations.

Figure: Figure 15: AMT questionnaire (human baseline) screen of the plausibility prediction and plausibility explanation tasks.
Refer to caption: x15.png

Figure: Figure 16: Human evaluation of explanation screen, including (a) instructions provided to participants, and (b) a screenshot of the AMT questionnaire.
Refer to caption: x16.png

## Appendix D NL-Eye Triplet Examples

Figure: Figure 17: NL-Eye examples from each reasoning category (3 triplets per category). Each example consists of 3 images: a premise (left column), a plausible hypothesis (green frame), and a less plausible hypothesis (red frame). While the gold explanations are included in the benchmark, we invite the reader to attempt to create valid explanations on their own.
Refer to caption: extracted/5899175/figures/images/appendix_examples-1.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - For the evaluation with this benchmark, it would be beneficial to have better metrics. Are there methods to quantify image order sensitivity? Could metrics be developed to measure visual understanding and linguistic abstract reasoning capabilities using various forms of input (Text-only, Image-to-Text, Image and Text, etc.)?
soundness: 3
strengths: - The paper is well-written and easy to read.
- The process of data collection and verification is systematic and meticulous.
- It intriguingly points out the shortcomings of existing visual language models (VLMs) in visual abductive reasoning, with experimental results to substantiate this claim.
- The paper proposes various experimental setups by combining or separating images, changing the order of images, which helps ensure fair testing.
- The benchmark effectively reveals multiple shortcomings of different VLMs, not only evaluating abductive reasoning but also highlighting issues with image location sensitivity and poor visual interpretation.
- Unlike traditional natural language inference (NLI) benchmarks, this approach offers a comprehensive evaluation of multiple aspects.
summary: This paper proposes a benchmark for measuring visual abductive reasoning capability and explains the process of constructing this benchmark. It demonstrates that current multimodal language models lack visual abductive reasoning capability and introduces a novel aspect of verifying image-to-image entailment that has not been previously addressed.
weaknesses: - The evaluation criteria are unclear and not well-defined. The use of automatic evaluation for explanations seems inadequate, and manual evaluation, while more accurate, is too costly and varies depending on the person.
- The definition of visual abductive reasoning capability remains unclear; it appears to evaluate abilities including visual interpretation, interpretation of multiple images, and natural language inference, covering a broad range of concepts that are not distinctly defined.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weakness 1-3. 
Also just out of curiosity, why can't the setting in Table 3 solve this problem? E.g. How did GPT-4o fail the entailment upon the Figure 8 machine-generated captions?
soundness: 3
strengths: Previous Visual entailment tasks were mainly in text format. This paper for the first time proposes the task in image formats, and collected a human-curated benchmark. The experiments show that current VLMs cannot do well on the NL-EYE.
Also, one experiment result saying that VLM prediction depends on hypothesis location is interesting.
summary: This paper introduces a new benchmark NL-EYE that is designed to assess VLMs’ visual abductive reasoning skills. NL-EYE adapts the abductive Natural Language Inference (NLI) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL-EYE consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. Experiments show that VLMs struggle significantly on NL-EYE, often performing at random baseline levels, while humans excel in both plausibility prediction and explanation quality.
weaknesses: 1. It is unclear whether the used prompt can best unleash VLMs' performance. For example, from Table 5, it seems no example has been provided, and that may lead to lower VLM performance. 
2. Why do human only achieve 83-85% accuracy if human collected the dataset and this dataset do not require expert knowledge? (Line 426-427) It is a bit confusing to understand.
3. In Table 3, why not try GPT-4o as the Image-to-Text model? Also, why not try Claude models as predictor?
4. The images are generated instead of from real world, and could potentially affect the output. The test size is 350 which might be small.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1. It is unclear from the paper how the authors selected the concepts for each individual reasoning category. For example, in the Cultural Reasoning category, which cultures were represented in the generated image. As image generation models are also not good for cultural content generation and the VLMs being better on cultural NLI raise interest in which cultures were highlighted mostly in the data to assess the comprehensiveness of the test set.

Q2. The current prompt for VLM is asking the plausible answer first and then asking for explanation. It would be interesting to reverse this process (i.e., explain each image step-by-step and then conclude the plausible answer) and see how the VLMs react.

Q3. In Tables 2 and 3, LLaVA 1.6 performs better at predicting the plausible image using GPT-4 when converting image to text (Table 3) than when directly inputting images (Table 2). Could this difference be due to LLaVA’s limitations as a predictor, or is the prompt structure (e.g., asking for image descriptions first before selecting a plausible answer) affecting performance?
soundness: 3
strengths: 1. This paper demonstrates a way to measure VLMs abductive reasoning ability on six diverse reasoning categories using multiple images.
2. The experiments shown in the paper comprehensively evaluate the reasoning ability of VLMs by checking image ordering, exploring different input formats to justify the reasoning gap of the existing VLMs.
3. The analysis section is interesting. The breakdown of performance across reasoning categories and the underlying insights will be useful for the community.
summary: This paper presents NL-EYE, a benchmark to evaluate VLMs' visual abductive reasoning skills across six reasoning types: physical, functional, logical, emotional, cultural, and social. It includes 350 triplet examples (1,050 images) with temporal annotations indicating event sequence and duration. The study examines model robustness, considering hypothesis order sensitivity and different input formats (individual versus composite images). NL-EYE also assesses models' ability to score single hypotheses, addressing real-world scenarios where multiple alternatives may not be available.
weaknesses: 1. The prompt selection is under-explored. 
2. More detailed in the Questions section

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: It would be nice to be able to determine if the problem this benchmark shows is out-of-domain on the language model side or a limitation of the visual encoder itself. If we split the data in the benchmark for training and test purposes and fine-tuned models improved on the remaining test splits, then we can assume that the main problem was the task was out-of-distribution rather than a lack of performance on visual perception, since most current visual-language models trained with a frozen visual encoder. Have you done any further experiments to see if this limitation on visual reasoning can be improved with some training or not?
soundness: 2
strengths: This work is distinct from existing multi-image benchmarks in that the objects of perception required to perform reasoning are provided solely through visual perception. NLI-inspired benchmarks that require visual reasoning over multi-images already exist, such as [1], but they are limited in terms of evaluating purely on visual perception, as they require reasoning over a given natural language premise. However, NL-Eye has the unique feature that requires reasoning on pure visual perception, since these premises are provided as images.

[1] A Corpus for Reasoning About Natural Language Grounded in Photographs (Suhr et al., 2019)
summary: This work proposes the NL-Eye benchmark to evaluate the abductive reasoning ability of visual-language models from pure visual perception in multi-image situations, inspired by Natural Language Inference (NLI). The benchmark consists of testing examples for temporal reasoning along with six reasoning categories and images are obtained from the text-to-image generation model. The authors argue that current visual-language models show significantly inferior performance compared to humans on abductive reasoning in multi-image situations and claim that this is due to the lack of purely visual perception abilities compressed from the visual perception modules.
weaknesses: There is a lack of consideration in the experiments as to whether a proper evaluation of the current visual-language model can be made in a multi-image setting. As the authors argue, current benchmarks for testing abductive reasoning are single-image focused, but it should not be overlooked that research on the visual-language model itself is also focused on this. As a result, the authors provide “concatenated” images, which may not be a fair assessment for most visual-language models that currently operate at fixed, squared-sized resolutions. To demonstrate the need for the proposed benchmark, it is required to observe if the same phenomenon is found in visual-language models that can handle flexible resolutions and aspect ratios like [1].

[1] LLaVA-UHD: an LMM Perceiving any Aspect Ratio and High-Resolution Images (Guo et al., 2024)

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: ## Question
1. Have the authors conducted experiments with VLMs that trained on datasets including multiple images such as LLaVA-Onevision or VILA, and with VLMs that use other visual encoders like Cambrian-1?

## Typo
* L260 Validation,and Categorization -> Validation and Categorization

---
### References
* Lin, Ji, et al. Vila: On pre-training for visual language models. CVPR 2024
* Li, Bo, et al. Llava-onevision: Easy visual task transfer. https://llava-vl.github.io/blog/2024-08-05-llava-onevision/
* Tong, Shengbang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Neurips 2024
soundness: 4
strengths: 1. The benchmark is well-designed with diverse reasoning categories.
2. Experiments on the benchmark reveal interesting findings.
3. The analysis is thorough and highlights notable insights into VLM limitations.
summary: This paper introduces NL-EYE, a benchmark designed to test the abductive reasoning capabilities of Visual Language Models (VLMs) through image-based tasks. The benchmark includes 350 carefully curated triplet examples spanning diverse reasoning categories where models must choose the more plausible hypothesis from a set and provide an explanation. Experiments reveal that while humans excel in this task, current VLMs show notable deficiencies in their reasoning capabilities. The authors conclude that VLMs face significant challenges in visual interpretation, which impacts their ability to reason effectively about images.
weaknesses: 1. While I agree that the benchmark is carefully curated, the filtering condition can be inconsistent and subjective because it is done manually. 
2. This paper focuses primarily on evaluating VLMs' deficiencies but lacks discussion on strategies or methods to improve these models' abductive reasoning capabilities.
3. The paper lacks experiments with additional open-source models. While the current model selection is valid, given the paper's findings about failures in visual interpretation and hypothesis location dependency, testing VLMs with different visual encoders or those trained on multi-image datasets would further support the analysis.

Meta-review: This paper introduces NL-EYE, a benchmark designed to test visual abductive reasoning by requiring models to determine which of two images (hypotheses) better fits a given premise image. Unlike typical visual NLI tasks that rely on textual premises, NL-EYE uses purely visual input and spans diverse reasoning categories. Reviewers praised the careful data collection, the clear problem formulation, and the exploration of model weaknesses, noting that current VLMs struggle with abductive reasoning and even show sensitivity to image order. Although some raised concerns about evaluation metrics, prompt selection, and the small dataset size, the novelty and thoroughness of NL-EYE stand out. The paper opens a new avenue for assessing complex visual reasoning capabilities beyond conventional benchmarks. Given the overall positive assessment of the benchmark's conceptual clarity and its potential to spur new research directions, I recommend acceptance.

Decision: Accept (Poster)