NL-Eye: Abductive NLI For Images
Authors: Mor Ventura
Michael Toker
Nitay Calderon
Zorik Gekhman
Yonatan Bitton
Roi Reichart
Venue: ICLR 2025 Poster (2025)
Keywords: Benchmark
Multimodality
Abductive Reasoning
NLI
VLM
Primary area: datasets and benchmarks

Abstract: Will a Visual Language Model (VLM)-based bot warn us about slipping if it detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet their ability to infer outcomes and causes remains underexplored. To address this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual abductive reasoning skills. NL-Eye adapts the abductive Natural Language Inference (NLI) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. The data curation process involved two steps—writing textual descriptions and generating images using text-to-image models, both requiring substantial human involvement to ensure high-quality and challenging scenes. Our experiments show that VLMs struggle significantly on NL-Eye, often performing at random baseline levels, while humans excel in both plausibility prediction and explanation quality. This demonstrates a deficiency in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a crucial step toward developing VLMs capable of robust multimodal reasoning for real-world applications, including accident-prevention bots and generated video verification.

Published as a conference paper at ICLR 2025
NL-EYE: ABDUCTIVE NLI FOR IMAGES
Mor Ventural Michael Toker'
Nitay Calderon'
Zorik Gekhman 1.2
Yonatan Bitton
Roi Reichart
TTechnion
Google Research
ABSTRACT
Will Visual Language Model (VLM)-based bot warn about slipping if it de- tects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet their ability to infer outcomes and causes remains underexplored. To address this, we introduce NL-EYE, a benchmark designed to assess VLMs' visual abduc- tive reasoning skills. NL-EYE adapts the abductive Natural Language Inference (NLJ) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL- EYE consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. The data curation process involved two steps -eritinritnnggg textual descrip- tions and generating images using eex--to-image models, both requiring substan- tial human involvement to ensure high-quality and challenging scenes. Our ex- periments show that VLMs struggle significantly on NL-EYE, often performing at random bascline levels, while humans excel in both plausibility prediction and explanation quality. This demonstrates a deficiency in the abductive reasoning capabilities of modern VLMs. NL-EYE represents crucial step toward devel- oping VLMs capable robust multimodal reasoning for real-world applications, including accident-preevnntoo bots and generated video verification
INTRODUCTION
Premise
Plousibe Hypothesis
Implousible Hypothesis
Abductive reasoning refers to the ability to infer and predict plausible outcomes causes given a context scene Peirce et al. (1934); Fann 2012); Douven (2021). This reasoning skill is crucial for Visual Lan- guage Models (VLMs), as they are likely to become increasingly integrated into our daily lives (Yildirim et al. 2024 Anwar et al.) 2024: Chiang et al. 2024: Shah jet al. 2023). These models will be re- quired to accurately monitor and interpret daily life scenes and correctly infer plau- sibility to prevent accidents and provide timely advice. For instance, would a bot warn us from slipping on a wet floor when there is no warning sign? or would it infer missing pacifier as cause of a crying baby?
Sliping is more lkely without a floor warning sign
NL-EYE evaluates the abductive reason- ing capabilities of VLMs. The main setup involves premise image and two hypothesis images, where the model is tasked with inferring which hypothesis is more plausible, and provide explanation for its choice.
Figure 1:
Although this capability critical, previous work has mainly evaluated VLMs in single scene setting such as visual entailment or detecting improbable events like fire in a closed jar or sequential scenes, such as next-frame prediction Xie et al. (2019); : Fu
(2022): ; Hessel et al. (202);; ; Fu et al.) (2024); Ganz et al. (2024): Yarom et al. (2024): Kadiyala et al. (0224). Consequently, remains unclear what extent existing VLMs are capable of abductive reasoning.
To address this, we introduce NL-EYE, a benchmark designed to evaluate visual abductive reason- ing capabilities of VLMs across multiple images. NL-EYE inspired by the textual abductive NLI
"Correspondence to: mor, peterrraearppss.ccotooo.oo.
'Our data and code available at ahttps://ventrraoor.t 1o/t.yye/////e

---

Published as a conference paper at ICLR 2025
task Bhagavvtula al. (2019) and applies it to the visual domain. In NL-EYE, VLM is presented with a premise image and one or two hypothesis images. then needs to infer how likely (plausible) hypothesis image is to result from lead to the premise image. The plausibility evaluation can be either done indiviuallyy or in comparison to an alternative hypothesis. For instance, in Figure [ the VLM needs to infer that, given the broken leg in the context image, it is more likely that the man slipped on the wet floor which lacked a warning sign (i.e., selecting hypothesis image 1).
Beyond plausiblity prediction, NL-EYE facilitates the evaluation of the models' capability to pro- vide faithful explanations. This allows us to explore whether they are correct for the right reasons rather than relying on shallow heuristics McCoy et al.] (2019). For example, a valid explanation for the broken leg scene would suggest that the presence of a warning sign would have made the man more alert, thereby potentially preventing the accident. In contrast, a shallow explanation might suggest that the man was simply resting on cozy rainy day.
Each NL-EYE example features a premise image alongside two hypothesis images, annotated with gold label indicating the index of the more plausible hypothesis. The examples also include gold explanation detailing why the chosen hypothesis more plausible than the alternative. Each ex- ample is categorized into one of six reasoning categories physical, logical, emotional, functional, cultural, and social - and includes temporal annotations that specify whether the hypotheses occur before, after, or simultaneously with the premise, and whether the time duration between the premise and hypothesis scenes is short or long. This rich annotation aids in diagnosing current VLMs and highlights their strengths and weaknesses. Figure presents detailed example.
To create NL-EYE, we collected a large pool of high-quality textual scenes created by experienced human annotators. The resulting scenes were then provided professional designers who utilized Midjoumny and DALL-E (Ramesh et al. 2021) to synthesize the corresponding images. The design- ers are also tasked with categorizing each example and creating the explanation that is used as the gold label. The image generation process was iterative, requiring multiple attempts to ensure consis- tency between the textual descriptions and the visual scenes, as well as visual coherence among the images within the same triplet. This process resulted in a total of 1,050 generated images, yielding 350 image triplets. Overall, NL-EYE is characterized by carefully curated examples, offering high quality both in terms of the scenarios and the consistency and quality of the images.
The first analysis is human evaluation where annotators select the more plausible hypothesis and explain their choice. Our results indicate that humans successfully identify the more plausible hy- pothesis in 85% of the cases. Furthermore, in our assessment of the quality of the human- -enaan-geeeaated explanations, we find that in 94% of the cases where the correct hypothesis was selected, the humans also provided a valid explanation. This demonstrates that humans perform reasonably well on the NL-EYE tasks.
Next, we design a comprehensive study to evaluate the abductive reasoning abilities of modern VLMs. We take multiple measures ensure the robustness of our evaluation, including addressing sensitivity to the order in which hypotheses are presented and exploring various input strategies, such as feeding the model three separate images or presenting it with a single combined-image that composites all three. Since real-world scenarios may not always provide two alternatives, we also evaluate the model's ability assign plausibility score single hypothesis, in addition to comparing two candidates. We have also developed framework that utilizes text-based base- line that processes textual descriptions of visual scenes. Specifically, we compare the results with gold descriptions and with the captions of the images as generated by the VLMs. Lastly, evaluat- ing model-generated explanations is challenging, as comparing generated text to a single reference (gold) explanation can be limiting and may not capture the variety and validity of possible correct answers. To address this, we adopt the evaluation proposed by Bitton- Guetta et al. (2023): human annotators are presented with image triplet where the correct hypothesis is already labeled and select valid explanations from a provided set.
Our results show that while humans perform well NL-EYE, VLMs struggle, with most models failing to surpass a random baseline in the plausibility prediction task. Even when identifying the plausible hypothesis, VLMs fail to provide accurate explanations in over 50% of cases, revealing a major weakness in their abductive reasoning. Furthermore, our text-based experiments indicate that these models often succeed in textual reasoming even when they fail to reason over images. Interestingly, when we prompt the VLMs to generate image captions, the resulting captions prove
2

---

Published as a conference paper at ICLR 2025
Deta Colleciin
Prompts
Premse CABBOTS REXT TO
Plousibe Hypothess BoWL wItHr a LOT of C98TC0 CARDOTS
Implausble Hypothesis
Premise 4AN VITONES MONK seses FUES GROVt THE avca
Plausible Hypolihesis KECOO tHe socs LRIN ECOA
Imolousible Hypoliheais AOD with PicKer THe LNMN B00N
A BOOLL WITH CABBOT CURSS
SOATED
Designers
Text- Text-1o- Nort-t-Inngee Imoge
Feetures Explonation
WHY IS THIS SCENE MORE PLAUSIBLE?
Amnottors
gnater produces shredded vegetables
liely grils nimeling
Bynng above the river in the move.
Categary
Temporal
Temparal
Recsoning Category
Temporal Duration
Tempondl Devectio
Funatronal
Logicel
Short
Forward
Figure 2: Fully annotated examples from NL-EYE. Each example includes the three images, the textual descriptions (prompts) used to generate them, the gold label, an explanation for why the gold more plausible, and indications of the reasoning category and temporal direction and duration.
ineffective for solving the task. Consequently, we hypothesize that the VLMs reasoning is hindered by inaccurate visual interpretations. We also find that these models are sensitive the order in which the hypotheses are presented and to the input format (three separate images vs a single combined- image). This sensitivity is concerning, as it raises the possibility that the models may not genuinely understand the underlying concepts, potentially relying on superficial cues to make decisions.
To summarize, we introduce NL-EYE carefully curated benchmark designed to test the abductive reasoning abilities of VLMs across various categories and temporal relations. We then conduct comprehensive study evaluating modern VLMs on NL-EYE and find notable deficiencies in their abductive reasoning capabilities. We believe NL-EYE represents crucial step toward enhancing reasoning abilities of VLMs, moving them closer to truly understanding complex, real-world scenarios and providing more reliable and interpretable outputs.
2
THE NL-EYE BENCHMARK
2.1 TASKS
Our objective is to explore and benchmark the abductive reasoning capabilities of modern VLMs. Unlike much of the previous work in NLP, our focus is reasoning solely based visual inputs: premise and hypothesis images. The premise image illustrates the context factual observations about the world and starting point from which conclusions illustrates a candidate conclusion a possible event that could occur before, after, or simultaneously with the scenario presented by the premise image. In the context of our study, we refer to the definition of abductive reasoning Nie et al. (2020); Douven (2021) as form of logical reasoning that seeks the most plausible hypothesis (concluston) given a premise (a set of observations).
drawn. The hypothesis image
To perform visual abducting reasoning, the VLM should identify objects and their relationships within each image, understand the relationships between the images, and integrate this informa- tion to reason about the plausibility of the hypotheses (see $B. for detailed definitions of the sub- capabilities involved performing visual abductive reasoning). We introduce two novel tasks to evaluate those capabilities: Plausibility Prediction and Plausibility Explanation. In the prediction task, the VLM is provided with the premise and hypothesis images. Its goal is to predict the plausi- bility of the hypothesis images to determine which one is more plausible. We argue that VLMs should be capable of not only predicting plausibility but also providing a sensible explanation of their reasoning process. Therefore, in the explanation task, the model is also required to generate a form textual explanation justifying why the chosen hypothesis is plausible or at least more plausible than the other.
3

---

Published as a conference paper at ICLR 2025
Premse
Plausbbe Hypolfhoss
Implauiill Hypathwos
Premise
Piuuabie Hypathesa
Implo.able Hypothesi.
PHYSICAL
FUUCTIONN
Because banano rot often month
Becouse the roling faathees the dough
LOCICAL
CUL AULTURA
Homes childrnn hypically have Fridge stocked with Friendy Food
Wearing house shoes c common Russion hroft, siilar hoving Mahrycshka dols
EMMOTIONAL
SOCIAL
Treats poulive feedback that makes happy
Becouse family photo more licly inicate married
Figure 3: Real examples from each reasoning category in NL-EYE. The more plausible hypotheses are framed in green. The gold explanations are included below each sample.
2.2 BENCHMARK STRUCTURE AND CATEGORIZATION
In this subsection, we describe the structure of each example in our benchmark and discuss the tax- onomy proposed for categorizing the examples. In Figure 2 we present the structure of two examples, which contains: (1) the premise image; (2) two hypothesis images; (3) the label, which indicates the more plausible hypothesis and is given by the benchmark designers; (4) the textual descriptions the three images that were used for generating the images; (5) the gold explana- tion. which clarifies why the correct hypothesis is more plausible, and is written by the benchmark designers; (6) reference explanations, which were written and validated by crowd-workers: (7) cat- egorization of the example, which indicates the involved reasoning category, temporal direction and duration.
In A describe the data creation process and specifically elaborate on components (1)-(5). The crowd-worker annotations (component 6) are detailed in $3. We next outline our proposed catego- rization, which serves two purposes: first, to ensure our benchmark is diverse, balanced, and covers a wide range of domains and reasoning types; and second, to aid in diagnosing areas where VLMs fall short.
Reasoning Categories We identify six different categories: Physical, Logical, Emotional, Func- tional, Social, and Cultural, ranging from physical reasoning (e.g., predicting the color of a rotten banana) to cultural reasoning (e.g., determining if habit like wearing house shoes implies another cultural trait, such as owning Matryoshka dolls). Figure presents an example from each category, with formal definitions in Appendix B.2
Temporal Categories The temporal categories are based on direction and duration. Temporal di- rection refers the logical relationship between the premise and hypothesis, indicating whether the event depicted in the premise image causes the hypothesis event (forward), is caused by it (back- ward), or if the events oceur simultaneously (parallel). Examples that do not occur the same time and are not categorized as parallel can also be classified by temporal duration, which is determined by the time gap between the events depicted the premise and hypothesis images. These include short-term when the events occur close in time, possibly in the same physical environment, with no significant sequence of events separating them, and long-term when the events take place in noticeably different periods of time. For instance, in the example the left in Figure 2 the grated carrots suggest a short-term forward progression within the same environment as the whole carrot in the premise.

---

Published as a conference paper at ICLR 2025
Triple Sadap
Para Safup
Table 1: Models and baselines by their in- put strategy and reasoning approach.
Approach ++
Vision-Rased Multiple Images
Text-Rused
Corlect
Strategy
Model
One
Image to Text
Only
Text
Gemini-1.5 1.5-PPo
GPT- Vision
GPT-4o
Claude-3.5
More Phovodic Image
Pouble Image Ronks
Rack
Claude-3-Ope
Expenator: Mussum camta
tysically have sigel cont
archoedlgical orighs
Pyrorids ersheceiegiod
doployed manuru
Findags
Ronk
DLIP2-FlanTS.XXL
InstructBLIP
Llava-1,6
Nore Plcable Image /[ Order Accuraay
BART-L-MNLI
Conostency Ascunasy
DeBERTa.<3nni
In the triplet setup (left), the input of the VLM is a triplet of premise and two hypotheses images, and its task is to predict and explain which hypothesis is more plausible. We provide the triplet two times with different orders of the hypotheses (e.g., see A and B), and only if it is consistent and predicts the correct hypothesis for both we consider it an accurate prediction. In the pairs setup (right), the input is a premise and hypothesis, and the VLM should output a plausibility score. For the same premise and two hypotheses, the predictions of the VLM are considered order-faithful and accurate if the correct hypothesis is scored higher than the wrong one.
Figure 4:
EXPERIMENTAL SETUP
3.1 TASKS AND SETUPS
Recall that the NL-EYE benchmark includes two tasks: Plausibility Prediction and Plausibility Explanation. Both tasks require reasoning about the relationship between the context image, the premise, which is denoted Ip, and candidate image, a hypothesis, denoted by Iμ- In addition to the images, the model f receives a textual query (a prompt) that contains instructions describing the task should perform (see Appendix Tables 16 and We introduce two setups for solving the tasks: the Tripler setup and the Pairs setup.
The Triplet Setup which is illustrated in the left box of Figure the model receives the query along with three images: the premise (Ip) and two candidate hypotheses (T and (H2). In the prediction task, the model's goal is to determine which of the two hypotheses is more plausible given Tr. i.e, which is more likely to occur, assuming Tp is a true observation about the world. For the explanation task, the model is also required generate a textual explanation justifying why the chosen hypothesis is more plausible than the other.
The Pairs Setup which is illustrated in the right box of Figure 4 the model f receives the query and two images: the premise Lp and the hypothesis I The prediction task now is to provide a plausibility score that indicates how plausible IN is, given that /p is factual observation. In our experiments, this score is provided on 1-10 Likert scale. However, this not mandatory the plausibility score be adapted to the needs of the model developer. For instance, the score could be expressed as a probability or another appropriate metric. In the explanation task, the model is asked to explain why H can be plausible given Tp.
Reasoning Approaches & Input Strategies To thoroughly assess the abductive reasoning capa- bilities of current models, we use two reasoning approaches: Vision-based where the model is tasked with performing the entire task end-to-end based solely on the visual input, and Text-based where the final plausibility reasoning is based on textual input. In the rision-based approach, we - where
experiment with two input strategies for feeding input to the model: (v.1) combined-imecn we concatenate the images (with the premise on the left) to form single combined image, and (v.2) pyparate-imagee
where we feed in one prompt the images to the model separately, starting with the premise. As not all models support both strategies, Table specifies which strategy was used for each model in our study. In the text-basee approach, we utilize two input strategies as well: (t.1) image-to-lext where we ask the model to describe the two images in natural language, and then,
5

---

Published as a conference paper at ICLR 2025
using those descriptions, the same model another performs the plausibility prediction, and (t.2) text-only we discard the visual inputs altogether and use only the textual descriptions generated when the images were created (see $A). The rision-based and text-based approaches allow us to understand the model's weaker capabilities better.
3.2 EVALUATION
Predictions in the Triplet Setup At first, we evaluated models based on accuracy. However, we found that all models are sensitive to the positioning (in the all-in-one strategy) or the order (in the sequential strategy) of the hypothesis images, that is, whether HH is placed or fed before or after IH2- For example, models may perform differently when given Triplet versus Triplet B from Figure E then aggregate the two predictions. A prediction considered correct if and only if the model selects the correct hypothesis in both orders. This approach reduces the likelihood of correct prediction by chance and ensures the model demonstrates consistency. The performance score in the triplet setup is the described consistency accuracy (proportion of correct and consistent predictions; see {B.3)
To address this sensitivity, we provide predictions for both orders of the hypotheses and
Predictions in the Pairs Setup In the pairs setup, aim to evaluate the plausibility score pre- dicted by the model. However, as previously mentioned, we do not want to constrain the model (or developers) produce a specific score or adhere support direct evaluation of the score, i.c., we do not provide gold standard score against which the predicted score compared. This raises the question: how do we plan evaluate models in the pairs setup? The only assumption we require from the scoring function is order-faithfule al.] 2024): if for a given premise Ip, the evaluated model m scores one hypothesis Iиι higher than another hypothesis I2,, then IFI should genuinely be more plausible than H2. Accordingly, for every premise image, we take the two hypotheses and consider the scores of J as correct if the hypothesis scored higher is the gold plausible hypothesis. The performance score in the pairs setup is the described arder-faithfulnes accuracy (proportion of correctly ordered scores; see BB.))
specific scoring function. Therefore, we do not
(Gat
Human Evaluation of Explanations Evaluating free-text explanations is a challenging task due the various ways explanations can be paraphrased and the reasoning involved in determining their validity. To address this, we followed the efficient human evaluation protocol proposed by Bitton- Guetta et al. (2023) and recruited crowd workers from Amazon Mechanical Turk (AMT). For each triplet of images, the workers were presented with the correct hypothesis and several explanations either written by humans (from the human baseline described in the next subsection) or generated by VLMSs. We included only explanations of the correct hypothesis. Then, the workers were tasked to select all the explanations that are logic and justify why the correct hypothesis more plausible (see Appendix for additional details). We consider an explanation correct if at least one worker selected it. The human evaluation score we present is the proportion of correct explanations.
Automatic Evaluation of Explanations Through automatic evaluation, we aim to demonstrate more scalable and cheaper approach assessing the validity of model explanations. Like other recent works, we follow the common practice of employing an LLM as a judge (Zheng et al. 2023] Chen al. 2024). Notice, that current models perform poorly on our visual abductive reasoning tasks, thus. expecting them to succeed in evaluating the validity of explanations generated by other models is pretentious. Instead, we simplify the task by conducting a reference-based evaluation, asking the LLM to determine whether the generated explanation aligns with gold reference expla- nation - a task that relies solely on textual reasoning. The reference explanations include the gold explanations (see AA), augmented with evaluation stage. To perform the automatic evaluation, we instruct an LLM (GPT-4o) to determine if the generated explanation aligns with one of the reference explanations (more details are provided in I.. The automatic evaluation score is the proportion of generated explanations that the judge L.M predicted as aligning with the references.
written explanations approved during the human

---

Published as a conference paper at ICLR 2025
Table 2: Main results: Scores for vision-based experiments. Automatic evaluation scores are not presented for Humans since their explanations serve as references. Regardless of the input strategy, VLMs are greatly outperformed by humans and mostly perform on par or even below the baselines.
Input Strategy
Prediction
Explanation
Model
Triplet
Pairs
Human
Auto
Humans
85%
83%
95%
Gemini-1.55PP
51%
38%
34%
GPT-4-Vision
46%
40%
39%
37%
Separate
GPT-4o
16%
50%
23%
23%
Images
Claude-Sonnet333
49%
38%
50%
26%
Gemini-1.5-Pro
43%
39%
40%
33%
GPT-4-Vision
41%
34%
37%
27%
GPT-4o
60%
45%
44%
40%
3oudee-Sonnet-35
28%
33%
42%
21%
Combined
Claudc-Opus-3
15%
19%
26%
6%
Image
LLaVA 1.6
14%
42%
15%
4%
Fuyu
4%
44%
10%
2%
Random
25%
45%
Dumb Pixel
50%
50%
3.3 MODELS AND BASELINES
Table outlines the models used in study, detailing their configuration, reasoning approach, and input strategy (specific versions in Appendix Table 15). Below, we provide more details on the models and baselines.
VLMs We employ state-of-the-art closed source VLMs, including, Gemini-1.5-pro (Google 2024), GPT-4-vision and GPT-4o (Achiam et al. 2023), and Claude-Sonnet-3.55 and Claude-Opus- (Anhrropic 2024). In addition, we employ open-source VLMs, including LLaVV 1.6 (Liu et al. 2024 and Fuyu (Bavishi et al. 2023
NLI models Recall that in the text-only reasoning approach, we provide the model with the gold text descriptions (used generate the images) and ask it to predict which hypothesis description more plausible. The predictor models we use in this approach include all the closed-source VLMMs
mentioned above, as well as fine-tuned NLI models such as DeBERTa-v3 (He et al., (023) and BART-L (Lewis et al. 2019). When the predictor is LLM, we ask it to determine which hy- pothesis description is more plausible given the textual premise description. For fine-tuned NLI models, compute two 'entailment' scores between the premise and each hypothesis, and the final prediction is made based on the hypothesis with the higher score.
Random baselines We present two simple baselines. The first is the random baseline, which ran- domly selects hypothesis in the triplet setup assigns random it is inconsistent due to its sensitivity to hypothesis order. To improve consistency, we introduce the dumb pixel baseline, which selects hypothesis or assigns a score based on a predefined rule using the upper-leftmost pixel. For example, the hypothesis with the brighter pixel is deemed more plausible, or the score calculated from the pixel's 's value.
in the pairs setup. However,
Humans Currently, there are indications that models can perform inference tasks at a level com- parable to, even exceeding, that of humans. Accordingly, we want investigate whether these VLMs can match human performance on our tasks that appear straightforward for humans and expect them succeed. To this end, recruited 15 workess the AMT platform. Pre- qualifications for workers were high approval rates and English-speaking countries. Additional details and guidelines are provided in

---

Published as a conference paper at ICLR 2025
4 RESULTS
4.1 VLMS FAIL TO PERFORM ABDUCTIVE REASONING WITH IMAGES
Table 2 presents the performance of humans, VLMs, and baselines on both tasks, prediction and explanation, for different setups and input strategies. For detailed results, refer to Appendix $ and C222 where we provide extended experiments, as well as $ where we compare human per- formance VLM performance and analyze their alignment.
VLMs Fail Where Humans Excel The results reveal a large performance gap between humans and VLMs on both tasks. Except GPT-4o, which achieves 60% accuracy in the triplet setup for combined-image inputs, all VLMs perform worse than the dumb pixel baseline. The situation even more concerning for current open-source VLMs, such as LLaVA 1.6 and Fuyu, which score below random baselines (see additional open-source VLMS in Appendix Table In contrast, hu- man participants achieve 83-85% accuracy in the prediction task and 95% in the explanation task. Notably, the participants are trokerworkers who are not experts or highly skilled. This suggests that the task is neither unsolvable nor partieularly difficult. Rather, current VLMs lack the visual abductive reasoning capabilities necessary to solve it effectively. Importantly, the finding that tasks easily handled by humans pose significant challenges for VLMs underscores the relevance of our benchmark and highlights areas where the research community can focus its efforts. Moreover, we found that VLMS are better in comparative or relative judgment setups (triplet, selecting which hy- pothesis is more plausible than the other) than in absolute judgment (pairs, predicting plausibility score for single hypothesis). This is unsurprising, as of humans (Pollit 2012 Verhavert et al.) 2019) which was also observed in LLM-as LLL----a-judge tasks (Kim al. 2024). To the best of our knowledge, we are the first to show for visual reasoning tasks.
known and well-studied phenomenon
Even When Correct, VLM Explanations Are Unhelpful To assess the quality of the explana- tions, we conduct human and automatic evaluations. Since nearly half of the predictions are incor- rect, we focused only on explanations for correct predictions, ensuring we can determine whether an explanation is genuinely poor rather than simply a result of the model failing to predict the correct answer. Note that in $D we provide qualitative analysis of explanations of wrong predictions to better understand why they fail. Table g in the Appendix reports the number of evaluated explana- tions of each model (a total of more than 3,800), and the results are reported on the two rightmost columns of Table (see human votes distribution in Appendix Table 7). As can be seen, humans almost always produce correct explanations, as 95% of the explanations were selected by the an- notators. On the other hand, at best, only half of the explanations are selected. This demonstrates that even when the VLMs predict correctly, the explanation is unhelpful. In addition, we used valid human explanations as references for the automatic evaluation, which produces scores similar to those the human evaluation in most cases. The automatic evaluation suggests that VLMs produce explanations that describe different reasoning than humuns.
4.2 REASONS FOR FAILING TO REASON
This subsection presents experiments analyzing why VLMs fail visual abductive reasoning.
VLMs Can Perform Textual Reasoning The Failure is in Visual Interpretation Table presents the results of text-based experiments aimed at decoupling the textual reasoning capabili- ties from the visual ones. In the text-only approach. the models are provided with gold-standard descriptions of the images. As shown in the table. the performance of all models, including smaller fine-tuned NLI models, is significantly higher than their performance vision- based experiments. Strong VLMMs such as GPTs and Claudes, achieve around 80% accuracy, nearing human perfor- mance. This indicates that VLMs are capable of textual reasoning. This finding suggests that the reasoning challenge does not mainly lie in the VLM's textual components but in the visual ones. In open-source VLM architectures (Liu et al., 2024), inference is not performed directly over im- ages. Instead, the models encode images to latent visual representations, which are then passed to the language model component. We hypothesize that poor visual inference results from these visual representations being inaccurate for the reasoning task.
8

---

Published as a conference paper at ICLR 2025
Performance for predic- tion in the triplet setup. Predictor models per- form well and similarly to (vision-based reason- ing of) humans when using the gold description. However, VLM describers generate useless cap- tions which do not help solve the task.
Table 3: Text-Based:
Reseaning Colegaries
Temporal Calngeries
40
-10
9.8
05
-04
Reasoning Approach
Describer
Predictor
Prediction Triplet
-10
Gemini-1.5-PP
66%
Emntional
Rexcoona
Photoal
Logioal
Sodial
Calbral
Baskwadd
Fineord
Paraiell
Shart
Long
GPT-4o
80%
Text-Only
Gold
GPT-4
Chanle-Sumnc-3355
79%
78%
Gembn 1.6
Caude-3.Gecmm
3PT4
Chande Opus-3
81%
BART 1. mnh
68%
DeBERTs ali v3
65%
Figure 5: Vision-based performance with sep- arate images for different reasoning categories (left) and temporal categories (right). VLMs struggle with the Emotional and Functional cate-
Gemini-I J.5-Pmm
29%
GPT-4 vision
52%
Claude 3.5
39%
Image-to- Text
Claude LLaVA 1.5
GPT-4o
33%
29%
BLIP 2
40%
gories but perform better on Social and Cultural
Instruct BLIP
35%
ones and parallel reasoning.
In contrast, when VLMs generated descriptions for each image, and these descriptions were used as input to GPT-4o, instead of the gold ones, the performance dropped significantly, aligning with the results from the vision-basee experiments (see Appendix Table [00 Claude 3.5 the predic- tor). We hypothesize that this indicates a recognition gap, where the generated descriptions either lack sufficient detail capture the necessary information are overly detailed (see examples in Appendix Figure 7. making it challenging to reason effectively.
Finally, consider the performance gap in Table 2 between separate and combined image input strate- gies. Except for GPT-4o, which consistently predicts the first hypothesis as more plausible (see Table in the Appendix), the other three models perform better when using separate image inputs, showing an average improvement of over 10% (48.6% vs. 37.3%). This suggests that when an image is complex and contains many details, as in the case of combined image, VLMs struggle to encode the necessary details and represent each image successfully.
VLMs Predictions Depend on Hypothesis Location Another factor contributing to the low per- formance of VLMS the triplet setup is their lack of consistency. As shown in Table 5 in the Appendix, all VLMS are highly sensitive to hypothesis order, with performance variations ranging from 5-80%. For example, Gemini-1.5-pro's performance in the combined-image strategy drops from 82% when the second hypothesis is correct to 46% when the first hypothesis is correct. Sim- ilarly, GPT-40, which performs best in the combined-image strategy (60%), fails in the separate- images approach, scoring 97% when the first hypothesis is correct but only 16% for the second. This suggests GPT-4o almost always predicts the first hypothesis as correct. In contrast, VLMs show much greater consistency in text-only inputs, with performance variation limited 7%. This indicates that VLMs rely on weak visual encoding, capturing superficial patterns like hypothesis order rather than meaningful image content.
VLMs are Better in Correlational and Knowledge-basee Reasoning Compared to Causal Rea- soning Relying on the categorization within our benchmark, we present in Figure 5 the vision- based results in the triplet setup for six reasoning categories (left plot) and five temporal categories (right plot). VLMs exhibit clear dichotomy in their reasoning abilities, excelling in some areas while falling short in others. Interestingly, the patterns are consistent across models, yet they diverge from the performance patterns observed in humans. For example, VLMs perform best in Social and Cultural reasoning, where specific knowledge is key to correctly solving those examples, see Ven- tura et al. (2023) for extended discussion about VLMs and cultural knowledge. In contrast, humans perform worst in the Cultural category. Another interesting observation is that VLM performance on parallel reasoning examples is higher than forward and backward reasoning tasks. Notably, parallel reasoning may require only understanding correlations between the premise and the hy- pothesis, whereas forward and backward reasoning require causal reasoning wwwnuinngiiinnttyyine identifying causes (in

---

Published as a conference paper at ICLR 2025
backward) effects (in forward). This suggests that VLMs may be more adept at identifying relations rather than causal understanding. Finally, the weakest category for VLMs is Emotional, which aligns with the literature (Lissak et al. 2024).
5 RELATED WORK
Recent advances in multimodal learning have enabled models to integrate textual and visual data across diverse tasks (Voulodimos et al. 2018 Guo et al., 2022 Qin et al. 2024). Powerful visual encoders like CLIP (Radford et 2021 Cherti et al. (023) and SigLip (Zhii et al. 2023), coupled
with the progress in LLMs (Chowdhery al.] 2023), have given rise sophisticated VLMS such as
BLIP2 (Li et al., 2023b). GPT-4 (Achiam ct al. (202)) and Gemini (Google (024). These VLMs are pushing the boundaries of multimodal capabilities. tackling tasks like visual question answering
(VQA) (Antol et al., 2015) and visual entailment (VE) (Xie et al. 2019). Our work builds on and extends research in these areas, with an emphasis on commonsense reasoning.
From Textual to Visual Entailment A cornerstone of our work is the expansion of Natural Language Inference (NLI), traditionally text-based task (MacCartney 2009 Dagan et al. 2010 Gekhman et al., 2023). into the visual domain. While previous research has explored NL in the context of image-text alignment (e.g., SNLI-VE (Xie al.] 2019). TIFA (Hu et al., 2023), WYSI- 2024). Mismatch-Ques, (Gordon et al. 2023)), ) and even video-text entail- ment (uu et al. 2021 Bansal al., 2024). we introduce novel framework for image-to-image en- tailment. This framework goes beyond simply selecting plausible alternatives by requiring models to explain their choices, thus offering deeper evaluation of their abductive reasoning. Furthermore, we introduce a "pairs" setup that requires scoring the plausibility of image pairs, aligning our task more closely with the original formulation of textual entailment.
WYR Yarrom et al.
Synthetic Data for Multi- Image Reasoning Our work uniquely employs synthetic images gen- erated by models like DALL-E (Ramesh al., 2022), allowing greater control over visual com- plexity and diversity compared to natural image datasets such as Winoground (Thrush et al. 2022), Sherlock (Hessel al. 2022). and VCOPA Yeo al. 2018). By emphasizing multi-image rea- soning, we address limitations in existing datasets that focus primarily on single-image tasks, like WHOOPSt(Bitton--o Guetta et al. 2023) and Visual
(Bitton-Cuetti et al. 2024). Our approach complements research on synthetic image understanding (Gokhale et al.) [022]] Wu et al., [023) Stöckl 2023) and is better suited for commonsense reasoning in real-world contexts, enhancing datasets like SEED-Bench (Li et al. (2033)) and MMToM-QA (Jin et al. (024), which tackle differ- ent aspects multi-image reasoning. Unlike ScienceQA {Lu et al. 2022) and NTSEBench (Pandya dataset employs photorealistic scenes from everyday life, making it more appropriate for evaluating commonsense reasoning. This integration of synthetic data, multi-image reasoning, and image-to-image entailment establishes new benchmark for assessing VLMs* reasoning capabilities.
eet al. 2024), which focus diagrams and scientific domains,
CONCLUSION
We introduced NL-EYE, a benchmark designed to assess the visual abductive reasoning capabilities of VLMs across multiple images. This skill is essential for real-world applications, such as accident- prevention bots. We paid special attention to detail in order to ensure that NL-EYE consists of high- quality and challenging examples, which required extensive human involvement at every stage of its curation. Our carefully designed study highlights critical challenges faced by modern VLMs in delivering satisfying plausibility predictions. We demonstrate that although humans perform well on these tasks, VLMs struggle significantly. This indicates a significant limitation of current mod- els' ability to integrate visual interpretation with logical reasoning. Furthermore, models not only struggle to make correct predictions but also often fail to consistently provide helpful explanations. In future work, we would like address these gaps, building our insights to develop new VLM architectures with higher reasoning skills, mirroring human cognitive processes in complex environ- ments, as elaborated in +
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
The NL-EYE benchmark includes Al-generated images, with the potential presence of unpleasant or insensitive content. While we strive to minimize harmful biases, the inclusion of reasoning based on common sense knowledge and cultural perspectives may introduce further bias, particularly related to social norms. Additionally, the labels this benchmark are based on consensus from human annotators, whose judgments may be influenced by their own cultural backgrounds, which could amplify bias. We also recognize the challenges related to text-to-image (TTT) copyrights, where the ownership of Al-generated content remains unclear. Researchers encouraged carefully consider these ethical and legal concerns when utilizing the benchmark.
REPRODUCIBILITY
To ensure the reproducibility of our results and promote further research, we will publicly release the NL-EYE benchmark, along with the code. Detailed technical instructions, as well as documen- tation on how to use and adapt the benchmark, will be provided in a publicly accessible repository. Additional technical details, including model versions and specific configurations used in the exper- iments, are available in the Appendix ($I). By sharing these resources, we aim to foster transparency and support the research community in advancing the evaluation of VLMs.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv: 08774, 2023.
Anthropic. Claude ai models, 2024. URL
wome Accessed: [Date]]
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425 2223,2433,222 2015.
Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, and Yan Chang. Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation. arXiv preprint MrXiv:2409.13668 2024.
Hritik Bansal, Yonatan Bitton, Idan Sxpektor, Kai-Wei Chang, and Aditya Grover. Videocon: Robust video-language alignment via contrast captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13927-13937, 2024.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023. URL httthttps://www adept.
ht/blog/fuyu-- 8b.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. Abductive commonsense rea- soning. arXiv preprint arXiv: 000../008...533 2019.
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gard- ner, Rohan Taori, and Ludwig Schmidt. Visit-bench: dynamic benchmark for evaluating instruction-following ssiionaadd-lnnguag models. Advances in Neural Information Processing Systems, 36:26898-266666 26922, 2023.
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! a vision-and-langusal bench- mark of synthetic and compositional images. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2616-2627, 2023.

---

Published as a conference paper at ICLR 2025
Nitzan Pitton-Guette Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton, Idan Szpektor, Amir Globerson, and Yuval Elovici. Visual riddles: a commonsense and world knowledge challenge for large vision and language models. arXiv preprint arXiv:2407.19477 2024.
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-aa-a-juuge: Assessing multimodal Ilm- as-a-judge with vision-language benchmark. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
net/ forum?idddbbFa. HAD79
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2829. IEEE, 2023.
-ien Lewis Chiang, Zhuo Xu, Zipeng Fu, Mithun George Jacob, Tingnan Zhang, Tsang- Wei Edward Lee, Wenhao Yu, Connor Schenck, David Rendleman, Dhruv Shah, al. Mobility vla: Multimodal instruction navigation with long-context vims and topological graphs. arXiv preprint ar arXiv:2407.07775. 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240); 1-113, 2023.
Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. Recognizing textual entailment: Rational, evaluation and approaches- approaches-erratume erratum. Natural Language Engineering, ((1)):05-1105, 2010.
Elena Deza, Michel Marie Deza, Michel Marie Deza, and Elena Deza. Encyclopedia of distances. Springer, 2009.
Igor Douven. Abduction. Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2021 edition, 2021.
Kuang Tih Fann. Peirce's theory of abduction. Springer Science Business Media, 2012.
Xingyu Fu, Ben Zhou, Ishaan Preetam Chandratreya, Carl Vondrick, and Dan Roth. There's time and place for reasoning beyond the image. ArXiv, 024/2203.007555 2022. URL https:
7/ apaii... www.smetticccoolaateoeooo org Serrg/Corpuss ID: 247218503
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404. 12390, 2024.
Roy Ganz, Yair Kittenplon, Aviad Aberdam, Elad Ben Avraham, Oren Nuriel, Shai Mazor, and Ron Litman. Question aware vision transformer for multimodal reasoning. arXiv preprint arX:v:2402.05472, 2024.
Yair Ori Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. Faithful explanations of black-box NLP models using llm-generated counterfactuals, In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttp:://opereeiiceee/ net forum?id
UMf UMfcdRIstC
Zorik Gekhman, Jonathan Herzig, Rose Aharoni, Chen Elkind, and Idan Szpektor. Trueteacher: Learning factual consistency evaluation with large language models. arXiv preprint arXiv:2305. 1171, 2023.
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in eoxttto-image generation. arXiv preprint w:Xiv:22/2.10011 2022.
12

---

Published as a conference paper at ICLR 2025
Google. Gemini pro vision, 2024. URL
Multimodal model.
Brian Gordon, Yonatan Bitton, Yonatan Shafir, Roopal Garg, Xi Chen, Dani Lischinski, Daniel Cohen-Or, and Idan Szpektor. Mismatch quest: Visual and textual feedback for image-text mis- alignment. arXiv preprint arXiv:23122005766, 2023.
Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang. Ralph R Martin, Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey. Computational visual media, 8(3):331-368 2022.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with pradient-disentanglee embedding sharing. The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httphttts:/:ppeeeeeet
net/ forum? i=medidd=s----- XhIxHAA
Jack Hessel, Jena D Hwang, Jac Sung Park. Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. The abduction of sherlock holmes: A dataset for visual abductive reasoning. European Conference on Computer Vision, pp. 558-575. Springer, 2022.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 20417, 2023.
Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B Tenenbaum, and Tianmin Shu. Mmtom-qu: Multimodal theory of mind question answering. arXiv preprint arXiv:2401.08743. 2024.
Likith Anoop Kadiyala, Omer Mermer, Dinesh Jackson Samuel, Yusuf Sermet, and Ibrahim Demir. The implementation of multimodal large language models for hydrological applications: A com- parative study of gpt-4 vision, gemini, llava, and multimodal-gpt. Hydrology, 1 11(9))148, 2024.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.0000
2024.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to--sequence pre- training for natural language generation, translation, and comprehension. CoRR, 2019. URL htttttttp///xxiieeee
13341
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint 11X1v22307.66155, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 2rX1v:2301.12597 2023b.
Shir Lissak, Nitay Calderon, Geva Shenkman, Yaakov Ophir, Eyal Fruchter, Anat Brunstein Klomek, and Roi Reichart. The colorful future of 1lms: Evaluating and improving Ilms as emotional supporters for queer youth. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 2040-2079, Association for URL https:
Computational Linguistics, 2024. doi: -ong. 113
//doi.org// 1ttp:/10..1663////200000 maw.///222.a...........m...co
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Bill MacCartney. Natural language inference. Stanford University, 2009.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Amnual Meeting of the Association for Computational Linguistics, pp. 3428-3448, Florence, Italy, July 2019, Association for Computational Linguisties. doi: 10. 18653/v1/P19-1336 URL https: weclannhhlooo wtwphholl....ooool.eee 1334
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language understanding. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics. pp. 4885 4901, Online, July 2020. Association for Compu- tational Linguistics. doi: 10. (0.18653/v1/2000 hctp://222.2lll2222//1 acl- acl-0aan.-mmmm-eeem 441. URL https::/aclaatholle. org/2020. acl-main. 441
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, et al. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint arXiv:2405.0222 02287, 2024.
Pranshu Pandya, Agney Talwarr, Vatsal Gupta, Tushar Kataria, Vivek Gupta, and Dan Roth. Ntsebench: Cognitive reasoning benchmark for vision language models. arXiv preprint arXiv:2407.10380, 2024.
Charles Sanders Peirce, Charles Hartshorne, and Paul Weiss. Pragmatism and pragmaticism. (No Tūle), 1934.
Alastair Pollitt. Comparative judgement for assessment. International Journal of Technology and Design Education, 22(2):157-170, 2012.
Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12811 12819, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot pout-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
Dhruv Shah, Blažej Osiński, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre- trained models of language, vision, and action. In Conference on robot learning, pp. 492-504. PMLR, 2023.
Andreas Stöckl. Evaluating a synthetic image dataset generated with stable diffusion. Interna- tional Congress Information and Communication Technology, pp. 805-818. Springer, 2023.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- fion, pp. 3238-5248 2022.
Michael Toker, Hadas Orgad, Mor Ventura, Dana Arad, and Yonatan Belinkov. Diffusion lens: Interpreting text encoders in text-to-image pipelines. arXiv preprint 24Xiv:2403.05844 2024.
14

---

Published as a conference paper at ICLR 2025
Mor Ventura, Eyal Ben-David Anna Korhonen, and Roi Reichart. Navigating cultural chasms: Ex- ploring and unlocking the cultural pov of text-to-image models. arXiv preprint arXiv:2310.01929, 2023.
San Verhavert, Renske Bouwer, Vincent Donche, and Sven De Maeyer. A meta-analysis on the reliability of comparative judgement. Assessment in Education; Principles, policy & practice, 26 (5):541-56,, 2019.
Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018()):706834,, 2018.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2096-2105, 2023.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- grained image understanding. arXiv preprint arXiv:1901.06700 2019.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109. 14084, 2021.
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. Advances in Neural Information Processing Systems, 36, 2024.
Jinyoung Yeo, Gyeongbok Lee, Gengyu Wang, Seungtaek Choi, Hyunsouk Cho, Reinald Kim Am- playo, and Seung-won Hwang. Visual choice of plausible alternatives: An evaluation of image- based commonsense causal reasoning. In Proceedings of the Fleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018
Nur Yildirim, Hannah Richardson, Maria Teodora Wetscherek, Junaid Bajwa, Joseph Jacob, Mark Ames Pinnock, Stephen Harris, Daniel Coelho De Castro, Shruthi Bannur, Stephanie Hy-- land, et al. Multimodal healthcare ai: identifying and designing clinically relevant vision-language applications for radiology. In Proceedings of the CHI Conference on Human Factors in Comput- ing Systems, pp. 1-22, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 11975-11986, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: / nips cctpsppr.cpppppppeeeeeee httplpaprrrrll//pppaapepee fitpss/appee///233// haaa222ffaaaa1233322222222222333
Ber: Berchmarks..ht
15

---

Published as a conference paper at ICLR 2025
A DATA CURATION
Joining recent efforts in evaluating VLMs with an emphasis on the quality of test sets over their
SCSEES WDITICC
Hand curched
Scone Wirting
Filerra
STOP
sheer size
(Thrush et al. 2022
Sitton-Guette
Model gesennrel
iet al. 2024
Bitton et al., 2023
Bitton Guetta Padlewski et al., 2024). we care-
IMACE LEUEDATION
Prenne Image Sanenatory
Hypothess
al.,
2023
fully curated 350 test set examples. The cre-
ation process of NL-EYE required human in- volvement at every key step (see Figure 6), en-
Phenot Auprertationl
LIM
Firg Algreeni
Ganenalion
STEP3
Image Editing
for Consistenay and Hignnent
abling the creation of diverse, high-quality ex- amples tailored to the evaluation's 's specific goal.
CEPLEEATION 6EO CATECON2ATION
Pausbiily Vallstion
Gold Exslancher
Fecture
STEP
Wirting
NL-EYE is a multi multi-image benchmark consist- ing of daily life scenes. A "scene" refers to specific setting where objects, people, and ac- tions are arranged in a particular context, which can be represented either textually or visually. The benchmark includes both representations, and the following key steps in its creation pro- cess: (1) textual description writing, (2) image generation, and (3) explanation and categoriza- tion.
Figure 6: NL-EYE data curation workflow scheme. The process includes three steps: (1) writing textual descriptions, (2) generating im- ages, and (3) generating explanation and catego- rization. Yellow denotes steps that require human involvement while turquoise denotes model-based generations.
Textual Descriptions Scenes were manually crafted by group of 20 annotators who were tasked with creating triplets consisting of a premise scene, and two hypothesis scenes, while one is more plausible than the other (see the first step, "scenes writing", in Figure 6). Each annotator had the flexibility to develop hypotheses across diverse reasoning categories, time directions, time durations, and domains. Annotators' creativity and experiences generated unique, everyday scenes that are of- ten undocumented or scattered, making it hard to gather automatically similar data. We manually filtered scenes from the suggested pool based several key criteria: (1) premise necessity, ensuring the scene is essential for determining the more plausible hypothesis; (2) visual relevance, guarantee- ing the scenes can be effectively communicated visually; and (3) uniqueness, verifying that we do not replicate similar existing examples or logical patterns (see examples in Appendix Table[14) We also applied preferences for receiving a range of challenges, from easy to difficult, as well as diverse time shifts, including varying directions and durations. After applying these filters, we retained 75% of 450 suggested ideas.
The images in NL-EYE were manually curated by two of the authors (noted as "the designers"), who have experience with text-to-image models. This careful generation pro- cess ensures high-quality images and verifies consistency, alignment between text and images, and overall clarity. The images were generated based on the textual descriptions using mainly Midjour- ney and DALL-E 3 (Ramesh al., 2021). During the prompt augmentation phase, The designers had the option to utilize assistance from Gemini and GPT-4 (Achiam et al. 2023) to transform the human-generated concise descriptions from step 1, into more detailed prompts with specific visual elements, enhancing visual consistency (see Appendix H For example, the step I textual descrip- tion, the bay is crying, turns the augmented prompt, the curly redhead boy with the striped green t-shirt is crying. Once the revised prompts were verified to ensure they don't 't interfere with the es- sential content (e.g., change such as the teenager is crying or an omission such as the curly redhead boy is wearing striped (-shirt) and manual edits were made if necessary, the prompts were ready for image generation.
Image Generation
Typically, the process begins with generating the premise image. Image generation is an iterative process, involving repeated cycles of manual editing and image-to-image alignment until high qual- ity and consistency are achieved. The image generation phase produces photorealistic images that are visually consistent, meaning that objects, people, and environments appearing in one image of the triplet are the same in the others. The last guideline arises from the crucial need to exclude style from reasoning considerations in the future evaluation of VLMs on the task. Technically, visual consistency is achieved not only through prompt augmentations but also via inpainting (i.e., edit-
16

---

Published as a conference paper at ICLR 2025
ing specific region of an image using textual prompt), image-conditioned prompting (generating an image while conditioning on another image), and using the same seed (initial noise distribution number) for all triplet images. See example of image generation, step 2, in Figure 17 in Appendix S
Explanation For each example, the designers wrote gold-standard explanations The gold ex- planation represents the original reasoning behind the scenes at the time of their curation. The gold explanation clarifies why the correct hypothesis is more plausible outcome or cause of the premise. It often follows a pattern like "Usually, X tends to Y" or "Because X made Y to...". Naturally, the explanation written this stage not the only possible explanation of the reasoning. Humans can suggest multiple plausible explanations and stories to justify connections between observations, also even for less likely scenarios. For example, examining the example the Social reasoning category in Figure 3 (bottom-right row), the premise image depicts wallet with a family photo, the hypothe- sis images depict a man's 's hand with a wedding ring (plausible) and a man's 's hand without a wedding ring (less plausible), and the human explanation is and statistically more likely to be married rather than single. However, the man might be married but wearing ring, have family without being married, the scenes might tell a story of loss and remembrance.
with a family photo in his wallet is socially
The validation process consists of two key checks: (1) image-text aligmment and (2) plausibility validation. First, we ensured that the images were correctly aligned with their corresponding texts. Second, we qualitatively assessed each example's plausibility, eval- uating how difficult it is to understand the intended meaning and make any necessary adjustments. This process includes manual verification by the designers, supported by a human baseline ($3.3) with a high human success rate on the plausibility prediction task (85%; see $4) and strong inter- annotator mnnotator-agreemenn (67.6%; unanimous votes) confirms the clarity.In addition, the designer classi- fied the examples into relevant categories, outlined in the previous section.
Validation and Categorization
Dataset Statistics NL-EYE is categorized by reasoning categories, domains and temporal infor- mation. The Logical, Social, Physical, Cultural, Functional, and Emotional reasoning categories comprise 28%, 24.6%, 17.8%, 14.3%, 7% and 8%, respectively, of the benchmark examples. 78% of the examples are in the time duration of short-term, divided mainly (68%) with the forward di- rection. The long-term examples are 22% while 27% of them are also associated with the backward direction. Refer the histogram of reasoning categories and real NL-EYE examples in Appendix Figures 18 and [6] respectively.
B EXTENDED DEFINITIONS
B.1 SUB-CAPABILITUE OF VISUAL ABDUCTIVE REASONING IN NL-EYE
We propose that visual abductive reasoning can organized into three levels of sub-capabilities, ranging from basic to advanced skills. Below, we outline and define these levels:
Basic Capabilities
Visual Recognition: Comprehending visual elements and their spatial relationships within
a scene.
Object Detection:
Identifying and recognizing objects in the visual input.
Intermediate Capabilities
Handling Multiple Images or Details: Analyzing information across multiple images or managing intricate details within a single image.
Object Tracking: Identifying an object in one image the same object in another, even across changes in perspective or context.
The benchmark contains the gold explanations and additional explanations written and validated by anno-
tators, see
$3.3
17

---

Published as a conference paper at ICLR 2025
Advanced Capabilities
Temporal Understanding: The ability to interpret the sequence, direction, and duration of events in visual inputs. This includes identifying the flow of actions or changes, under- standing their temporal order, and estimating how long they persist.
Recognizing the importance of sequence and distinguishing causal relationships from spurious correlations.
Order Sensitivity and Correlation Handling:
Common Sense Reasoning: Applying general world knowledge infer logical conclu- sions about the visual input.
Plausibility Assessment and Decision-Making: Evaluating likelihood of a scenario based on visual and contextual clues, and selecting the most plausible explanation or out- come from multiple options.
This hierarchical organization clarifies the definition of abductive reasoning in the context of NL- EYE.
B.2 REASONING CATEGORIES DEFINITIONS
Table 4: NL-EYE textual descriptions examples. One example for each reasoning category. Every example consists premise phrase, a plausible hypothesis phrase, and an implausible hypothesis phrase.
Category
Premise
Plausible Hypothesis
Implausible Hypothesis
Physical
child sits on the floor, holding wrapped present in the shape rectangu- box.
A child sits on the floor, holding an unwrapped rect- angular present.
A child sits on the floor. holding an unwrapped ball- shaped present.
Logical
Clothesline with large shirts and small children's 's shirts.
refrigerator full of home- made food, yogurts, and children's 's food.
empty refrigerator with only a few bottles of beer and ketchup.
Emotional
baby stroller with paci- fier lying on the floor next to it.
crying baby sits stroller.
happy baby sits a stroller.
Functional
large thin circle of dough kitchen surface.
A lump of dough and rolling pin on kitchen face.
A lump of dough and pasta maker kitchen surface.
Cultural
digital clock shows 16:00 pm and an image Queen Elizabeth
British old ladies sit and drink hot cups.
British old ladies play con- tract bridge game.
wall.
Social person with kiss mark on the cheek sitting a hol- iday table with family.
Grandmother arrived as guest.
Grandfather arrived as guest.
Physical Reasoning, Involves understanding the physical world and how objects interact within The scenes include changes in temperature, phase, shadow's 's location, color, shape, etc. This reasoning is inspired by the spatial-temporal reasoning definition (Deza et al. 2009).
Functional Reasoning. Requires an understanding of objects' functionalities and tools' common usage. This type of reasoning involves not just recognizing objects and tools, but also comprehend- ing their intended purposes and how they interact within various contexts. For instance, a hammer not merely identified by its shape but also by its function of driving nails into surfaces. Functional reasoning allows a model to infer the appropriate use of an object within a given scenario, such as using knife for cutting broom for sweeping.
18

---

Published as a conference paper at ICLR 2025
Social Reasoning. Understanding social norms, relationships. and interactions. Social reasoning allows for the comprehension of social norms and etiquette, such as knowing how to greet someone depends the context. This includes recognizing familial roles, friendships, professional relation- ships, and the varying degrees of formality and familiarity in interactions.
Emotional Reasoning. Understanding and interpreting emotions and emotional responses. refers to the ability to identify a wide range of emotions, including happiness, sadness, anger, fear, surprise, and disgust, and to understand the context in which these emotions arise.
Cultural Reasoning, Involves acknowledging cultural traits and traditions while correctly associ- ating them with their respective cultures. includes the ability to recognize and interpret cultural symbols, rituals, languages, and behaviors accurately. For instance, it includes understanding that certain gestures may have different meanings different cultures or that specific holidays and cele- brations unique particular cultural religious groups.
Logical Reasoning. Requires an understanding of general processes and broad commonsense. It enables the analysis of situations, draw inferences, and make decisions based on logical principles and widely accepted knowledge. Logical reasoning involves the ability to follow sequence of steps to solve problems, recognize patterns, and identify relationships between different pieces of information.
B.3 ACCURACY METRICS: MATHEMATICALL FORMULATION
Here we present the mathematical formulation of the accuracy metrics, based on the notations in Section $3)
Formally, the consistency-acccurrey (triplet accuracy) is:
consistency Acc.(Ip, IHI, Tuz, H Hgota)
[1, if f(Ip, (e2) f(Ip, IH2, (x1) Hgold
o,
otherwise
Formally, the order-faithful accuracy (pairs accuracy) is:
order-faithfün Acc. IH,, 122 Hi Hgold)
1, if f(Ir,Im) f(Ir,
& H1 H gold
1. or f(Ip,Im) fmI, & H2 Hgold
0, otherwise
19

---

Published as a conference paper at ICLR 2025
C COMPLEMENTARY RESULTS
C.1 RESULTS OF PLAUSIBILITY PREDICTION AND EXPLANATION
Table 5: Plausibility prediction results of the triplet setup. Order refers the position of the correct
hypothesis image in the input, whether it was presented first (order 1) or second (order 2).
Baseline
Model
Consistency Acc.
Order Acc.
Order Acc.
Apparate-Imager
Gemini- Gemini-1.5-pr 1.5-pro
50.57% %
56.0%
75.
GPT-4-vision
45.71%
19%
66.0%
GPT-40 Claude-3.5-sonnet
16.29%
97.43%
16.29%
49.28%
65.9%
59.31 %
Combined Combined-Image
Claude-3-opus
15.14%
30.57%
72.86%
Claude-3.5-somnee
28.29%
76.86%
30.29%
Llava-mistral-7b
14.86%
53.14%
26.0%
Gemini-1.5 1.5-pro
42.57%
46.29%
81.43%
GPT-4-vision
41.14%
54.57%
67.14%
GPT-4o
60.0% %
76.0%
69.14%
Fuyu-8b
4.58%
42.98%
13.75%
Text-Only
GPT-4
78.0%
86.86%
82.86%
GPT-4o
80.0%
83.14%
88.57%
Gemini-1.5
65.8%
72.99%
79.02%
Claude-opus-3
80.57 %
85.43%
87. 77.4%%
Claude-sommet-335
79.43%
82.29%
89.43%
Bart L mnli
68.0%
68.0%
68.0%
DeBeRTa-v3
65%
65%
65%
Table 6: Pairs-setup performance with additional rank information regarding rank differences and absolute values.
37
7.39
GPP--4-rsion
6.97
Genini 1.5gmm
42
1.71
43
3.16
0.65
7.13
6.47
Chande-Smmer--5
16
1.53
41
7.49
0.60
7.07
7.56
GPT:4
45
2.27
0.50
7.37
7.25
Iizage
GPT-4vraion
5.5mm
11,VA
Pryu-80
3.35
2.04
4.06
7.60
Table Human votes of candidate explanations. The percentage of votes reflects annotators' agree- ment with the candidate explanations provided by the models. O-votes notes no-selection, 3-votes notes selected unanimously.
Input Strategy
Model Name
0 Votes (%)
1 Vote (%
Votes (%)
Votes (%
Humans
5%
9%
25 %
60%
GPT-4o
77%
8%
5%
9%
GPT-4-vision
61%
8%
6%
25%
Separate-Imager
Gemini-1.5-pro
62%
9%
7%
23%
Claude-sonnet-3.5
50%
12%
12%
25%
GPT-4o
56%
6%
8%
31%
GPT-4-vision
63%
8%
11%
18%
Gemini-1.5-pp
60%
7%
12%
21%
Combined-Iman Image
Claude-sonnet-3.5
58%
8%
10%
24%
Claude-opus-3
74%
12%
5%
9%
LLaVA 1.6
85%
9%
5%
1%
Fuyu
90%
8%
2%
1%
20

---

Published as a conference paper at ICLR 2025
Table 8: Number of evaluated explanations. The explanations are associated with correct plausi- bility prediction. Human explanations include the correct explanations of 3 crowd-workers, The explanations are evaluated by human and automatically.
Input Strategy
Model
Num Candidate Explanations
Humans
840
Gemini p.5-pro
GPT-4-vision
278
Separate- Images
GPT-4o
340
Claude-somnet-35
265
Claude-Opus-3
309
Claude-sonnct-33.
276
LLaVA 1.6
224
Combbined-Image
Gemini-1.5-pro
298
GPT-4-vision
281
GPT-4o
298
Fuya-8h
Total Model explanations
3,868
COLD ITEXTUAL DESCRIPTIONS!
IMAGE-TO-TEAN ICLAUDE 3.5)
IMAGE
of can and bride hold handsin the oldee looling happy.
The imaye slowes e caugle in what oppears be church or chasel softing. They wulling togethor looking of escoofter with beigtt unies. suggesting joy possibly wedding
The wonan is wouring on dlgurt the shouker wiite dress with long skeeess whic coiciooodlie be wedding gown. Her ling blnde hainis sty el loose wowes
The dressed o
jocket. whiee and has small Ford beutornne
lopel tysiod groam. has way
beard
bodkground shows archnd colong. hanging lighta
lodes len staned glans
wndow, all inleative of church inferic Thare are hirts of other seeale and decand ane the box Aground. soggesfing event foving place.
hoppiness and celeleation, coptured intimate manent between couple they through verue.
The overall onvesphore of the
Two shomes lone bakengnn ls deplay hrder cop mottch
The maye shows lwo amort pphnns indilatl by adea duploying doiing spp riler focna. Balh show "II's Match!" mensages mndaling that two users have not ched with each other the daling platforms. callel "Tndar" with arofin piciune anilng man with dark han night phone shows cap called "Tidd" with a profie oclure of laughing weman with long blanci Bath cap tnler facus faot une miar layou's with heart ioans cohtnn buffora Hikdy far deng messogee
lift dasloys
bereals the "It's one being by handh, suggettin o person hibii The overoll fane image is worm, with soft bicig lightin that emphonzes the bright sonwens of the unortonanes Thes seruo oppware be Bustnaling the moment when two people match en dating capturing the restement and ootantal of sicing work!"
profittal. and
TLAMOOL
IL MAN
shones bebengng mmmnn one a woman? deplay the fncer upp who norrecldh
"The image snarlphenes by sholoying what appurs daling app interface. soneers show profle pelores and On the iift phane. we see a man with cank har and o beard, looking down or ha phone. On the right prane. there's waman with beo. light celoned hai. die lcsling her phone wrlc touching fuce Belo profic
Match' premmantly exployed
on onbuthalooee there are leely the appl's fan llng on pansing on palwerid mates The shanes openar is be mocern smariphones with reichad scnoar, nexerbiing recent iPhone medial. The badky has o worn, aightly sloving effect. erealing moddy almosphere The everall compostion suggests commentory modern dafing occur hightlincchhheeee
buftons, anel
and a gray
-iit.
patertil discenecttion rissed connections that
Figure 7: Image-to-text descriptions example. Detailed descriptions by Claude 3.5 and the gold textual descriptions. In bold style are the key necessary details for succeeding in the plausibility prediction task.
21

---

Published as a conference paper at ICLR 2025
C.2 FURTHER EXPERIMENTS
To complement our primary experiments, we conducted additional analyses to further examine the reasoning capabilities of Vision-Language Models (VLMs) under various conditions. These ex- periments focus on three key aspects: (1) evaluating the performance of additional open-source VLMs that support multi-image imputs and flexible resolutions (Table (2) assessing image-to-text pipelines using Claude 3.5 as predictor, in contrast to GPT-4o in the main experiments (Table IO), and (3) analyzing the sensitivity of VLMs to different prompt formulations (Table Our findings highlight persistent limitations in current VLMs, with most models struggling to perform well on NL-EYE tasks.
Table 9: Additional Open-source VLMs:
Scores for vision-based experiments, including two ad- ditional VLMs that are multi-image and support flexible resolutions and aspect ratios: MiniCPM, which employs SigLIP-4000 visual encoder paired with the MiniCPM-2.44 language backbone (MiniCPM v2.6), and VLaVA-OneVisie which integrates a SigLIP vision encoder with Qwen2 While performance mains below random, using a combined image strategy with these models demonstrates improve- ment, showcasing their enhanced ability to encode and leverage visual information effectively.
language backbone (LLaVA-One Visionnnneii-OOOO (isiion-Qwen27b))
separate images re-
Input Strategy
Model
Triplet Acc.
Humans
85%
Separate
MiniCPM-V:22 V-2.6
12%
Images
LLaVA Owweiision-wweee22-66
18%
MiniCPM- V-2.6
Combined
LLaVA-OneVision
MeeViionn-Qeen-11
23%
Image
LLaVA-1.6
14%
Random
25%
Dumb Pixel
50%
Table 10: Tmage-to-Text: Performance drops observed also with Claude 3.5 predictor, evaluated using Triplet Accuracy based on a separate image input strategy. GPT-4o demonstrates lower performance in the plausibility performance task, it relies on image descriptions. Both models exhibit a significant drop in performance overall.
Describer
Predictor: Claude 3.5
Gemini- 1.5-Pro
50%
GPT-4 vision LLaVA 1.6
44%
36%
BLIP 2
42%
Instruct BLIP
36%
Table 11: Prompts Comparison with GPT-4 Vision: Analyzing different prompts provides valu- able insights; however, a pell-performing VLM (or any model) should be able to understand and follow instructions. Excessive sensitivity to input prompts may indicate difficulty with the task. This table includes results from experiments with 3 additional prompts. The CoT approach shows 3% improvement, while overall, the prompts demonstrate comparable performance.
Prompt Variant
Prompt Template
Triplet Acc. (Separated)
Regular (Table
46%
Reverse Task
First explain, then predict
43%
CoT
Let's 's think step by step
49%
Role
You are causality expert
44%
22

---

Published as a conference paper at ICLR 2025
D FAILURE ANALYSIS
This section examines key failure points and differences in VLM reasoning patterns, with focus on quantitative comparisons to human performance and qualitative failure analysis.
Are Human and Model Difficulties Aligned? In the plausibility prediction task using the triplet setup of the separate-images input strategy, humans and VLMs agree on 55% of the predictions (full comparison Table 12). When humans are incorrect, the model's success rate falls to 30%, below random chance. While the model outperforms humans in only 5% of cases, its explanations in these instances are rarely accurate. Furthermore, only 21% of the models' errors overlap with human errors, indicating that humans and models tend make different types of mistakes. Additional examples are shown in Figures and 12
Plausibility prediction analysis: Model vs. Human Comparison. /notes correct prediction while x notes incorrect one.
Table 12:
Model
Model v /umman v
Model Human
Model xHuman x
Model xHuman v
Claudc-sonnct-355
44%
5%
10%
40%
Gemini-1 5-prro
46%
598
11
38%
GPT-4-vision
41%
4%
11%
42%
Models Avg.
44%
5%
11%
40%
VLM Failure Analysis This section presents qualitative analysis of VLMs' explanations for wrong predictions. We qualitatively analyze 120 explanations 40 from each VLM: Gemini 1.5, GPT-4 and Claude 3.5.
We identify five main factors contributing to the models' failures: (1) style & consistency: When irrelevant visual details influence the decision; (2) time: When the explanation relies incorrect time direction or duration; (3) ignoring key details: Overlooking important information; (4) missing knowledge: Misinterpreting key details despite recognizing them; (5) failed comparison: Justifying a less plausible hypothesis with logical reasoning. Table 13 presents illustrative examples of these factors.
As Figure X shows, all models struggle with understanding temporal progression. Notably, Claude often relies on style considerations, with 30% of its errors resulting from this factor, indicating overemphasis on irrelevant visual details. Both Gemini (32%) and GPT-4 (25%) frequently miss key details, suggesting recogni- tion gaps. GPT-4 has the highest rate of failed comparisons, often making the incorrect deci- sion at the final plausibility stage. To further understand these failures, researchers can try to interpret models' internal thought processes (Toee et 2024).
Mixsnng Ignoned Key Faled Time Consistency Knowkegge Detais Comporiion
3%
8%
19%
16%
32%
25%
24%
19%
14%
Style &
30%
14%
19%
27%
27%
22%
Genisi 1.5
GPF-4
Does Image Naturalness Influence Model Figure 8: Failure factors of model explanation for Performance? Synthesizing the benchmark incorrect plausibility prediction.
provides ficxibility to simulate diverse everyday scenes while maintaining consistency and qual- ity. Extracting triplet scenes from real-world sources, such as videos, poses challenges in terms of efficiency and feasibility, particularly when ensuring consistency between the premise and false hypothesis, which may not exist in the same video.
To evaluate the influence of image style, generated versus realistic, on model performance, we con- ducted ablation study using subset of 20 triplets (60 images) that parallel to the generated
'The sample DDs of the real images subset: [8, 15, 24, 30, 35, 47, 51, 63, 84, 86, 106, 118, 187, 192, 200, 256, 268, 312, 342]
23

---

Published as a conference paper at ICLR 2025
Table 13: Failure factors with examples, as illustrated by the following scene a premise of a man in hospital bed with broken leg and two hypotheses: a wet floor with (less plausible) and without (plausible) a warning sign (Figure
Failure Factor
Example
Style & Consistency
Selecting the image because the window in the background matches the onc in the premise image.
Time
Assuming the hypothesis occurs after the premise and concluding the warning sign placed after an accident.
Ignored Key Details
Ignoring the cast on the man's leg and assuming he's resting rather than injured.
Missing Knowledge
Identifying the sign but not realizing it's a warning sign.
Failed Comparison
Selecting the hypothesis with warning sign, reasoning person was distracted and didn't see it, even though slipping could occur in both cases.
ones, consisting solely of natural images sourced online or photographed by students, as demon- strated in Figure [ We selected examples that were straightforward to produce, focusing on those that did not require consistency across individuals highly nuanced setups, due to the complexity of finding or producing such cases. This resulted in simpler examples that generally yielded higher performance outcomes.
(triplet, separate images) on real images compared to 68% on generated original ones. This suggests that the performance gap is not significantly influenced by the type of images used, but it opens the door for further investigation into how visual styles may may impact the robustness visual abductive reasoning performance in VLMs.
Our analysis reveals that GPT-4 Vision achieves 58% consistency-accuue
IPHOTOREAUSTIC
GEUERATED (PHOTOREAUISICC
NU-EVE EXAMPLES
OULIME EXTRACTIOU
PHOTOGBAPHED
ABLATIOU:
REALISTIC
Figure 9: Examples of the real images subset.
24

---

Published as a conference paper at ICLR 2025
Style & Canustency
01
Masing Knowledge
Deleed
Goude
Choule 3.5.
The second hycord hyporhaau mape mmoee alouab becouse showa the some olden mon en the Beths
hypothesc slouab shows strong cormaciinn
feolure the sare anwient. grarted free und fir serre ginger out, mage
centest
the paroon, Indlsating they part the sama photo or roment elefting Theso raae. showing girl dees not seen be direcffy related to the centex?
difererce
chilk whkc hoporoornees ihees on okder men, living possage free. This could reprovent the refuring maaningful place from thir chichood..
Gemen "nage
upon
The sxcrdd hyset mage shows earry sperts hag sports goggle
setting whichs common place far
refurnng from tep. image couels welnng Rowers. aegesligg pauage Ince
tmage bay
Ighored Key Detads
and groplo mupest haat the person alher preporiin for on has finahed zwer ming sassonn puraan typicoly rokitel to the of spets srogins for owir ving Therefore, the first image plovelle
Cortect
gyms.
afle
is denfial anea
dractly
dog cariesl mage a paccefully sleaping Isage 2 heppy and hedttyy with caner. tis vore loely that the dagin the contoct iaagee fullllng fho
the
Figure 10: VLM failure analysis: Explanations examples. Based on five main factors: (A) Style & consistency, (B) Time, (C) Ignored key details, (D) Missing knowledge and (E) Failed comparison.
25

---

Published as a conference paper at ICLR 2025
Contect
Gemini 1.5:
Image is more plousible because visually matches the fish in the context image and could be toy vansion of it. Moreover, it's unikall that cat would be playing with dead fish (mage 2).'
B3
Esplonchin
Cloude 3.5:
"Hypohheess is more plouséle becouse it shows crumpled paper. The crumpled poper suggests falled attempt folding organni which would naturaly occur before successfully creating the intricate flower shope seen the context iogge..
Figure 11: VLM failure analysis: When the model's plausibility prediction is correct the explana- tion can be either valid (B) or not (A).
A3
Motel Explanctenn
Humon Predctinn
Conteet
Genini 5,
Human
Ross typsully don't suits liveyy socur a setting live dlepidated bathreom Wiiss unModdy sameone would choose to wash their hands the bathroom from the context sigitly plouabs forced use ronndown facilty"
homes, making image
posallly mass The man is wearing a suif and idn't bekeve he is reloted the talet. The non number 2 bathrobe and used the tole!,
a complete
mage, image
being
exploded."
Midel Exdenotion X
Huma Predicion X
GPT-4:
Humon
plousble because shows shanding "Tee Diret hypothess natorcyckst shooped distance. which could legelll precede the contedt mage where the nstorcycket clser and the focus. suagestrg motorcyclist approscited ance first iage sacend hypothesis inoge wide uonler shows offerent cow
It plaustbe for cow be nood country than city
Figure 12: VLM failure analysis: When humans plausibility prediction is incorrect, and model's explanation is correct- the explanation can be either valid (B) or not (A).
26

---

Published as a conference paper at ICLR 2025
E HUMAN PERFORMANCE & EVALUATION
Plausibility Predictionn Explanation
Explanation Evaluation
Canted
Cankod
Answer
Answer
Anawerr:
The pacilier floor next the baby strolle, from the baby's mouth so not calm anymore..
More Plausible Choice: Hypothesis
Exdlrstionn pacilier fell the floor the baby stroller, from the baby's moulh so he not calm anymore.
X
The floral wallpaper matches the stroiler image D....
Guidelline
Guideline
To correctly answer many such questions, you would have to detect small details and tell yourself the resscnable "story" behind the scenes.
1. To correctly select the valld explanations, they have to tall "missing that connects betveen context and image.
2. explanation the wallpaper strolleer style does not explain the baby reaction.
Figure 13: Guidelines for the crowd- workers. Guidelines for the human baseline the plausibility prediction and plausibility explanation tasks (left), and for human evaluation explanations (right).
Human Performance Plausibility Prediction and Explanation. AMT Crowd-worken were in- structed to complete the plausibility prediction and explanation tasks based on the following guide- lines, in Figure 13 and the questions in Figure 14
Human Evaluation of Explanations. The explanations are presented in multiple-choice ques- tion format (see Figure 15). where crowd workers are instructed to select explanations that demonstrate logical reasoning and clearly justify why the correct hypothesis is more plausible than the other candidate (see Figure 13). We conduct the human evaluation explanations across all the input formats in the vision-based reasoning approach (Triplet setup), focusing the explanations associated with the correct plausibility predictions, resulting in a total of 3.8k explanations.
27

---

Published as a conference paper at ICLR 2025
astuctors
Shancats
Context
Part 1: Considering the Context and Two Candidate Images
Nrtw
Given the context image, which 1 2 is more plausible?
01 O2
Provide short explanation how the image you selected connected context image why plnusible happen
How certain are you in your choice?
Somewhtel Unceriain
Part 2: Considering the Context and One Candidate Image at Time
Focus on context and Image (ignore image 2)
How likely this from to
1-1 licely
2
3
4
Very likely
Focus on the context and image 2 (ignore image 1)
How likely is image from O Not Ikaly
OS Very likey
Figure 14: AMT questionnaire (human baseline) screen of the plausibility prediction and plausibility explanation tasks.
28

---

Published as a conference paper at ICLR 2025
Instructions
Summary
Detailed Instructions
Examples
Instructions for task
You will be shown an image triplet and multiple candidete explanations related to the images. Your task to select all the oxplanations that logically connect the context imago the correct image, which framed groen.
Clarification The correct image is marked with green framework, and incorrect image is marked with a red framework.
Clarification
explianations may specific image numbers (1
only explanation
Itself and ignore any specific numbering.
Summary
Detailed Instructions
Examples
examples
examples
Example good selection: Choesing explanations cleany logical reasoning making reasonable image (green-T image. For example, context image sit, umbrellaa:
Examples selections: Choosing explanations rely stylsticc reasons,
example, image, introduuee
Image
image
image
holding umbreila:
sky
Tusually clouds impty coming storm, which the man prepares for holding an umbretta so he won't get wer" shows plausible connection based on common sense and real-world expertence.
"The sky the context Image more simitar to the sky the other image" This explenation locuses on visual similarity without logical reasoning.
"The man looks like he's about to run because people storms tend to runt This explanation assumes behavior that isntt shown or implied by the image, acding ireeevant information.
"The man saw it's about rain so he decided take an umbreita" shows different logical time direction for the scenes progress.
Des lueks trown cade fius, alges with eturso flamss and invigtaar sepores
fhalights
Iogicaly below enteree oftert fighing
depicted
image
the treighhr move in with fre houe to outa fhe like image one fivitgg furs do not down and becaa thare outhoaak
Hepttere more plossile boceuse shows raatakc progressen
one stonano conttdt
housi enguliddin
energen pence seppriie The pou Boring the trelihhsss and the mie tuly of the ane coma slent with the conteed image In conbact Hipotheis 2 stovs fielgga
Hypothool
forlghtr poar knwaing burning house,
what sciin be it taking Therstore the image contuct more scenaro
Ematoonl Hleppone Fieegriess
potentaa
rrggrosion Whit Image thows mow ite ovenn recensatty tooo the contedt image iigicaly contoct image thoun the andl underney poesoioo bocasio depidl evenl anoicca ropporse Eudigter alget with prosiits shage tho fert progroocse after contoct.
trulghter approaching
Figure 15: Human evaluation of explanation screen, including (a) instructions provided to partici- pants, and (b) screenshot of the AMT questionnaire.
29

---

Published as a conference paper at ICLR 2025
F DISCUSSION AND FUTURE DIRECTIONS FOR ENHANCING VISUAL ABDUCTIVE REASONING
To improve VLMs' abductive reasoning, future efforts could focus refining both model architec- tures and training tasks. Given our findings that models can reason from textual descriptions but struggle with images, one approach might to modify the image encoding align with the textual query, thereby enhancing representations of specific areas within the image. Another potential solu- tion train a model to generate image descriptions that are optimized for downstream reasoning tasks.
Building on this, techniques like Chain-of Sicin-oo-Thoooe Thought (CoT) can be extended to the visual domain, gen- erating images part of the reasoning-thinking process. Additionally, VLMs tend to assign undue importance to style image order rather than focusing on semantics. To address this, constructing large-scale training set that emphasizes semantic content over order and style could help models prioritize meaning over superficial patterns, resulting in more robust reasoning. These strategies collectively offer path toward enhancing models' visual abductive reasoning capabilities.
G
NL-EYE EXAMPLES
SOCAL
CMOTIOAAL
CULTUBAL
FUUCTIOUAL
LOCKAAL
PHYSICAL
Typetheas
Inglusble Hppe
Premie
Doabe Hocheas
Sebbable tpehhs
Figure 16: NL-EYE examples from each reasoning category (3 triplets per category). Each example consists of images: a premise (left column), a plausible hypothesis (green frame), and a less plausible hypothesis (red frame). While the gold explanations are included in the benchmark, invite the reader to attempt to create valid explanations on their own.
30

---

Published as a conference paper at ICLR 2025
H NL-EYE DATASET: CURATION DETAILS
Table 14: Examples of suggested textual descriptions (scenes) filtered by specific criteria.
Filter Cri-
Premise
Hypothesis
Hypothesis
Reason
terion
Premise ne- cessity
teacher en- ters school
An apple on the teacher's 's desk
An orange on the teacher's desk
We need to see the teacher to understand it's school setup
Visual rele- vance
Man says hello
Man
enters
Man exits home
It's unclear the man is saying hi or bye
home
Uniqueness
Man with bro- ken leg
Hole in road under construc- tion
Hole in road with warning sign com- pleted work
Repetition of existing ideas (Figure
Text-to-Image Prompt. the last one is optional:
The Text-To-Image prompt (in Midjourney) is consisted of 3 parts, while
Text description. The textual scene caption, basic or improved.
Photorcalistic style. Adding textual styling of photorealistic images by mentioning it is captured with Nikon D850.
Visual consistency. Making image consistent with another image by setting the same seed number, and referring to the reference image with the flag cref and its conditioning strength with the flag ow ranging from 0-100.
All these parts are aggregated into the following template:
< prompt cuption >, captured with Nikon D850 and a 24-70mm lens at ff.8 -seed <> -cref <>
-Cw 80
Prompt Augmentation. following prompt:
Augmenting a text description by prompting Gemini or GPT-4 with the
Describe visually specific looks of < interacting componentl > < interacting component2 >> and environment >. keep short and concise, and avoid NSFW words. and integrate these details into every reference of them in the following captions smoothly and consistently. do not change the content of the captions besides the visual description integrations. return in JSON format: 1) <first image caption>> 2) <plausible second image caption>> 3) <implausible second image caption>>
Note: integrate the environment only it fits the context of the caption.
For example:
interacting componentl: little child, interacting component2: vaccine, environment: nurse room, first image caption: a child gets vaccine,, plausible second image caption: child cries after getting a vaccine., implausible second image caption: child smiles after getting vaccine. Response:
improved first image caption: short curly-haired child wearing a green t-shirt receives a vaccine with silver syringe in nurse's room filled with toys. improved plausible second image caption: the short curly-haired child a green t-shirt cries after receiving a vaccine in the toy toy-filled nurse room. improved implausible second image caption: the short curly-haired child in green t-shirt smiles after receiving a vaccine in the nurse's room.
31

---

Published as a conference paper at ICLR 2025
DAASES CEUGEETINE STOP: 200M
TEP GUAENTT
S
Figure 17: Zoom into the image generation step in NL-EYE curation, as seen in Figure 6 Yellow
color notes hand-curated stage, while turquoise notes model-generated stage. All stages require human involvement for fixing, editing and validating.
Doeann
Dop Dueeher
Foueblyy
Galtural
Enurhonal
Fanchonal
Loguca
Physical
Social
Galtunal
Emchanal
Fanchund
Lopicl
Physicl
Social
Figure 18: Dataset Analysis. A histogram of the NL-EYE examples. The benchmark is also an- notated with diverse domains (left): administration, business work, daily life & hospitality, ed- ucation, healthcare, religion, science & technology, sports & arts and transportation, and repre- sentation of time duration and direction (right) in every reasoning category. Parallel noted by "parallel-short".
32

---

Published as a conference paper at ICLR 2025
1 REPRODUCIBILITY AND RESOURCES
Table 15: API and version of closed-source models used for inference on NL-EYE tasks.
API
Model Version
Used as
Gemini
gemini-1.5-pe
VLM, LLM
GPT
gp1-40-2024-08-08
VLM, LLM
GPT
gp1-4-110-visiion-pe preview
VLM. LLM
Claude
claude-3-opus-22240223
VLM. LLM
Claude
claude-35--omme-22224600
VLM. LLM
Table 16: Textual prompts for task descriptions in different input strategies and setups.
Input Strategy
Setup
Prompt Template
Triplet
Given a context image and hypothesis images (3 total images), which image of the following two (I and 2) is more plausible? The context image happen before or after the hypothesis images. Mention which one more plausible or 2, and explain.
Separate- Images
Pairs
Given a pair of images a context image and hypothesis image mrakk how plausible the hypothesis image is in relation to the context. The context image can occur before after the hypothesis image. Rank the plausibility with score between and 10, where: 1: plausible at all, Slightly plausible, 5: Moderately plausible, Very plausible, 10: Almost necessarily plausible. Explain why.
Triplet
Given context image (left image) and two hypothesis images (middle and right), which hypothesis image (1 or 2) is more plausible? Mention which one is more plausible
Combined-Imanl Image
or 2, and explain. The context image can happen before after the hypothesis images.
Pairs
The first (left) image is the context image. Given a pair of images... Separate- Separate-Images Pairs)
Text-Only
Triplet
Given a context, hypothesisl, and hypothesis2, which hypothesis more plausible? The context can occur before or after the hypotheses.
Image-to- Text
Triplet
Describe the content of the image in detail.
Prompt of Auto Evaluation of Explanation. We combine classes 0 and as false (invalid) explanation, and 3 as a positive (valid) explanation:
Context:
<premise textual description>,
Plausible Hypothesis: <hypthesis I textual description>.
Less Plausible Hypothesis: <hypothesis 2 textual description Return 0 (Not logical at all), 1 (Logical but different), or 2 (Logical and same as one of the gold) if the candidate's explanation presents the same logical common sense as appears one of the gold explanations for justifying the plausible hypothesis. Candidate explanation: <candidate explanation>. Gold explanations:
Explanation 1: <gold explanation>...
33


Reviews:
[{"review_id": "GXpIhae1zP", "reviewer": "Reviewer_fRsv", "summary": "This paper proposes a benchmark for measuring visual abductive reasoning capability and explains the process of constructing this benchmark. It demonstrates that current multimodal language models lack visual abductive reasoning capability and introduces a novel aspect of verifying image-to-image entailment that has not been previously addressed.", "questions": "- For the evaluation with this benchmark, it would be beneficial to have better metrics. Are there methods to quantify image order sensitivity? Could metrics be developed to measure visual understanding and linguistic abstract reasoning capabilities using various forms of input (Text-only, Image-to-Text, Image and Text, etc.)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to read.\n- The process of data collection and verification is systematic and meticulous.\n- It intriguingly points out the shortcomings of existing visual language models (VLMs) in visual abductive reasoning, with experimental results to substantiate this claim.\n- The paper proposes various experimental setups by combining or separating images, changing the order of images, which helps ensure fair testing.\n- The benchmark effectively reveals multiple shortcomings of different VLMs, not only evaluating abductive reasoning but also highlighting issues with image location sensitivity and poor visual interpretation.\n- Unlike traditional natural language inference (NLI) benchmarks, this approach offers a comprehensive evaluation of multiple aspects.", "weaknesses": "- The evaluation criteria are unclear and not well-defined. The use of automatic evaluation for explanations seems inadequate, and manual evaluation, while more accurate, is too costly and varies depending on the person.\n- The definition of visual abductive reasoning capability remains unclear; it appears to evaluate abilities including visual interpretation, interpretation of multiple images, and natural language inference, covering a broad range of concepts that are not distinctly defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KtNxE8bfSp", "reviewer": "Reviewer_LDJN", "summary": "This paper introduces a new benchmark NL-EYE that is designed to assess VLMs’ visual abductive reasoning skills. NL-EYE adapts the abductive Natural Language Inference (NLI) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL-EYE consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. Experiments show that VLMs struggle significantly on NL-EYE, often performing at random baseline levels, while humans excel in both plausibility prediction and explanation quality.", "questions": "See Weakness 1-3. \nAlso just out of curiosity, why can't the setting in Table 3 solve this problem? E.g. How did GPT-4o fail the entailment upon the Figure 8 machine-generated captions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Previous Visual entailment tasks were mainly in text format. This paper for the first time proposes the task in image formats, and collected a human-curated benchmark. The experiments show that current VLMs cannot do well on the NL-EYE.\nAlso, one experiment result saying that VLM prediction depends on hypothesis location is interesting.", "weaknesses": "1. It is unclear whether the used prompt can best unleash VLMs' performance. For example, from Table 5, it seems no example has been provided, and that may lead to lower VLM performance. \n2. Why do human only achieve 83-85% accuracy if human collected the dataset and this dataset do not require expert knowledge? (Line 426-427) It is a bit confusing to understand.\n3. In Table 3, why not try GPT-4o as the Image-to-Text model? Also, why not try Claude models as predictor?\n4. The images are generated instead of from real world, and could potentially affect the output. The test size is 350 which might be small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k132LdYq0G", "reviewer": "Reviewer_fepb", "summary": "This paper presents NL-EYE, a benchmark to evaluate VLMs' visual abductive reasoning skills across six reasoning types: physical, functional, logical, emotional, cultural, and social. It includes 350 triplet examples (1,050 images) with temporal annotations indicating event sequence and duration. The study examines model robustness, considering hypothesis order sensitivity and different input formats (individual versus composite images). NL-EYE also assesses models' ability to score single hypotheses, addressing real-world scenarios where multiple alternatives may not be available.", "questions": "Q1. It is unclear from the paper how the authors selected the concepts for each individual reasoning category. For example, in the Cultural Reasoning category, which cultures were represented in the generated image. As image generation models are also not good for cultural content generation and the VLMs being better on cultural NLI raise interest in which cultures were highlighted mostly in the data to assess the comprehensiveness of the test set.\n\nQ2. The current prompt for VLM is asking the plausible answer first and then asking for explanation. It would be interesting to reverse this process (i.e., explain each image step-by-step and then conclude the plausible answer) and see how the VLMs react.\n\nQ3. In Tables 2 and 3, LLaVA 1.6 performs better at predicting the plausible image using GPT-4 when converting image to text (Table 3) than when directly inputting images (Table 2). Could this difference be due to LLaVA’s limitations as a predictor, or is the prompt structure (e.g., asking for image descriptions first before selecting a plausible answer) affecting performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper demonstrates a way to measure VLMs abductive reasoning ability on six diverse reasoning categories using multiple images.\n2. The experiments shown in the paper comprehensively evaluate the reasoning ability of VLMs by checking image ordering, exploring different input formats to justify the reasoning gap of the existing VLMs.\n3. The analysis section is interesting. The breakdown of performance across reasoning categories and the underlying insights will be useful for the community.", "weaknesses": "1. The prompt selection is under-explored. \n2. More detailed in the Questions section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PpRvgE4CYA", "reviewer": "Reviewer_GcX5", "summary": "This work proposes the NL-Eye benchmark to evaluate the abductive reasoning ability of visual-language models from pure visual perception in multi-image situations, inspired by Natural Language Inference (NLI). The benchmark consists of testing examples for temporal reasoning along with six reasoning categories and images are obtained from the text-to-image generation model. The authors argue that current visual-language models show significantly inferior performance compared to humans on abductive reasoning in multi-image situations and claim that this is due to the lack of purely visual perception abilities compressed from the visual perception modules.", "questions": "It would be nice to be able to determine if the problem this benchmark shows is out-of-domain on the language model side or a limitation of the visual encoder itself. If we split the data in the benchmark for training and test purposes and fine-tuned models improved on the remaining test splits, then we can assume that the main problem was the task was out-of-distribution rather than a lack of performance on visual perception, since most current visual-language models trained with a frozen visual encoder. Have you done any further experiments to see if this limitation on visual reasoning can be improved with some training or not?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This work is distinct from existing multi-image benchmarks in that the objects of perception required to perform reasoning are provided solely through visual perception. NLI-inspired benchmarks that require visual reasoning over multi-images already exist, such as [1], but they are limited in terms of evaluating purely on visual perception, as they require reasoning over a given natural language premise. However, NL-Eye has the unique feature that requires reasoning on pure visual perception, since these premises are provided as images.\n\n[1] A Corpus for Reasoning About Natural Language Grounded in Photographs (Suhr et al., 2019)", "weaknesses": "There is a lack of consideration in the experiments as to whether a proper evaluation of the current visual-language model can be made in a multi-image setting. As the authors argue, current benchmarks for testing abductive reasoning are single-image focused, but it should not be overlooked that research on the visual-language model itself is also focused on this. As a result, the authors provide “concatenated” images, which may not be a fair assessment for most visual-language models that currently operate at fixed, squared-sized resolutions. To demonstrate the need for the proposed benchmark, it is required to observe if the same phenomenon is found in visual-language models that can handle flexible resolutions and aspect ratios like [1].\n\n[1] LLaVA-UHD: an LMM Perceiving any Aspect Ratio and High-Resolution Images (Guo et al., 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4eWu99RNrh", "reviewer": "Reviewer_972W", "summary": "This paper introduces NL-EYE, a benchmark designed to test the abductive reasoning capabilities of Visual Language Models (VLMs) through image-based tasks. The benchmark includes 350 carefully curated triplet examples spanning diverse reasoning categories where models must choose the more plausible hypothesis from a set and provide an explanation. Experiments reveal that while humans excel in this task, current VLMs show notable deficiencies in their reasoning capabilities. The authors conclude that VLMs face significant challenges in visual interpretation, which impacts their ability to reason effectively about images.", "questions": "## Question\n1. Have the authors conducted experiments with VLMs that trained on datasets including multiple images such as LLaVA-Onevision or VILA, and with VLMs that use other visual encoders like Cambrian-1?\n\n## Typo\n* L260 Validation,and Categorization -> Validation and Categorization\n\n---\n### References\n* Lin, Ji, et al. Vila: On pre-training for visual language models. CVPR 2024\n* Li, Bo, et al. Llava-onevision: Easy visual task transfer. https://llava-vl.github.io/blog/2024-08-05-llava-onevision/\n* Tong, Shengbang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Neurips 2024", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark is well-designed with diverse reasoning categories.\n2. Experiments on the benchmark reveal interesting findings.\n3. The analysis is thorough and highlights notable insights into VLM limitations.", "weaknesses": "1. While I agree that the benchmark is carefully curated, the filtering condition can be inconsistent and subjective because it is done manually. \n2. This paper focuses primarily on evaluating VLMs' deficiencies but lacks discussion on strategies or methods to improve these models' abductive reasoning capabilities.\n3. The paper lacks experiments with additional open-source models. While the current model selection is valid, given the paper's findings about failures in visual interpretation and hypothesis location dependency, testing VLMs with different visual encoders or those trained on multi-image datasets would further support the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)