Title: VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision

vlms

visual understanding

visual reasoning

reasoning
Abstract: Vision Language Models (VLMs) excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple perceptual tests. We present an evaluation that tests vision-language models’ capacity for non-local visual reasoning— reasoning that requires chaining evidence collected from multiple, possibly distant, regions of an image.  We isolate three distinct forms of non‑local vision: comparative perception, which demands holding two images in working memory and comparing them; saccadic search, which requires making discrete, evidence‑driven jumps to locate successive targets; and smooth visual search, which involves searching smoothly along a continuous contour. Flagship models (e.g., GPT-5, Gemini 2.5 Pro, Claude Sonnet 4), even those that perform well on prior primitive‑vision benchmarks, fail these tests and barely exceed random accuracy on two variants of our tasks that are trivial for humans. Our structured evaluation suite allows us to test if VLMs can perform similar visual algorithms to humans.  Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new benchmark evaluating nonlocal visual reasoning in VLMs. It is composed of three different tasks designed to test different types of nonlocal reasoning (comparative perception, saccadic search, smooth visual search). Results show that SOTA VLMs mostly fail to perform these tasks, which humans can easily complete, suggesting ample room for improvement in their reasoning capabilities.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths:*\n\nAs VLMs improve, benchmarking their abstract reasoning abilities is becoming more important.\nIt is interesting to see tasks that are inspired by human psychology, which can be a fruitful avenue for finding new ways to test these models.\n\nThe idea of testing VLMs for sequential visual reasoning, particularly decomposed into three different sub-categories, appears to be novel and it is interesting to what degree current VLMs encapsulate them.\n\nThe results on the visual scavenger hunt task seem interesting, since there is a clear differentiation between models that do and do not show non-trivial performance.\n\n*Weaknesses:*\n\nI have some major concerns about the methodology which make me question the degree to which the results in the paper are meaningful.\nMy central concern is that these images depicting abstract shapes are very different from the natural images that these models are normally trained and evaluated on. This has already been explored in prior work (L22). If other works already show that modern VLMs don’t understand basic geometry in abstract shapes, what does this work add? In other words, do these results really say something interesting about different types of reasoning, or are they just another demonstration of this known phenomenon?\n\nI also have concerns with the task images and prompts used (Fig 1). Some of the textual prompts use ambiguous and confusing wording. For example, the first task uses the term “object” in an abstract sense that is not obvious, and the second example in the figure states that the correct answer is YES because the object was scaled and rotated when scaling might not be considered a rigid transformation. In the third task, a phrase like “starting at the teal triangle” might be interpreted either to mean the given geometric shape and color, or the text stating “teal triangle”. It could be that VLMs struggle because they are not as good at interpreting confusing or convoluted prompts while humans manage to do so – this doesn’t necessarily measure higher level reasoning.\n\nSimilarly, the text in the image prompts is small and hard to read. Although you partially tested this in Supp A.2 (on only one of the many texts in images for one of two tasks using visual text), it could still be that results would improve if text resolution were increased.\n\nSome less significant issues are:\n* Some of the presentation can be improved. I found Fig 1 and its caption to be a bit convoluted, and could benefit from numbering the tasks or even from splitting into three figures accompanying each task in the text.\n* L279 seems like an overclaim. Even if poor results are because of a gap in reasoning abilities, it is not clear that explicit structured reasoning is the solution. For example, maybe inference compute scaling methods without explicit structure such as chain-of-thought could help.\n\nQUESTIONS:\nWould SOTA VLMs also mostly perform randomly on these reasoning types when they are required to understand natural images rather than synthetic shapes?\n\nAre the reasoning types (e.g. comparative perception) inspired by studies in psychology? If so, it would be nice to provide references to their use in that field.\n\nDoes the object re-identification task connect to existing work on Image Difference Captioning or other methods to compare two images?\n\nL174 How is the Circuit Connections task different from the existing Subway Connections task?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is an empirical study on sequential and non-local spatial reasoning in VLMs. The paper introduces an evaluation suite for 3 such reasoning tasks and demonstrates that current VLMs generally fail at these tasks even though they are mostly trivial for humans to complete.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe evaluation suite is of great value to the community as well as the main findings around current limitations of VLMs. The empirical analysis is also thorough across many different VLMs to definitively demonstrate the key finding.\n\nWeaknesses:\nThe paper primarily focuses on identifying an existing weakness within VLMs. While this work is useful and important, the analysis towards identifying the root causes and limitations of the weakness is limited. Finally, there is limited discussion towards overcoming the weakness. Note that while the paper is not required to present a complete positive result in fully resolving the issue, it can be strengthened with some reasonable direction towards improvement.\n\nQUESTIONS:\n1. Fig 2, 4, 6 do not have the variants for o3. Is there a reason these results are missing?\n2. The discussion in Section 4.2 around the justifications is insightful, as it shows that these tasks can fundamentally be broken down into smaller subtasks which the model should be able to solve. This suggests potential strategies for mitigation. How do the models perform in the chain length = 1 variant? Can you extensively verify which tasks are feasible for breaking into small parts and solving and which are not?\n3. Similarly, the viability of self-correction is an interesting consideration. Is the model aware of the viability of a self-correction strategy, e.g., is it aware in the prompt that some texts point to non-existent objects?\n4. In Sec 4.3, it is interesting to identify that the model is not performing tracing. Can we hypothesize what approach the model is taking to solve the problem?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthe authors present a 3-task benchmark of visual challenges easy for humans but potentially difficult for machines.\nthese consist of procedurally generated images spanning 3 categories: object re-identification up to rigid transformation, a visual scavenger hunt, and circuit connection following. The authors argue all of these tasks require \"non-local\" visual reasoning, i.e., instead of \nThe authors verify that the tasks are easy for humans (~100% in all cases), and easy to objectively evaluate the accuracy of.\nBut, all models, including frontier models/reasoning models/etc., fall significantly short.\nThus, the released datasets/environments form a new challenge for vision+language models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nI quite liked this paper! It's difficult to find simple tasks where models fail, and the authors have specified three of them. It's hard to argue against.\n\n- I liked that the authors constructed a few versions of the tasks to attempt to isolate/specify which parts are particularly hard. This provides a nice easy-to-hard curriculum, which could inform training.\n- The argument of the paper is very clear/convincing, e.g., human performance is given for all tasks.\n\nWeaknesses:\n\n- I worry that some of the prompts used to specify the tasks are underdetermined. For circuit connections, while I used commonsense to infer that the breadboard is the big gray thing in the middle of the image and the components are the smaller squares with C_#; to make the task even less debatable, it would have been helpful to more precisely specify this either with text, or with a label in the image. A similar concern with the visual scavenger hunt: I didn't know what it meant to \"follow the label\" initially --- it would have been helpful to be more precise, like: \"starting at the teal triangle, read the text ontop of it, and then \"follow the label,\" i.e., move to the shape described by that label. Repeat the process of following the label for 3 steps, in total.\"\n\n- It would have been nice to have some few-shot results in addition to the zero-shot results.\n\n- While I appreciate the sentiment, the authors claim in L126 that the task is purely visual and cannot be simulated in natural language. I don't know if \"ascii art\" constitutes natural language, but I could imagine rendering all of the author's challenges (save for the visual scavenger hunt which relies on color) in that format. (in fact, that would be a potentially cool ablation of whether or not this task is fundamentally bottlenecked by the visual encoder itself).\n\nWould have liked:\n\n- I would have liked to have seen a non-synthetic-image version of these tasks --- while not programatically generate-able, you could imagine annotating versions of these tasks on some photographic images.\n\n- I am curious to see if training on synthetic data like this (whether via sft or RL in an environment: both are supported by the author's work) will \"patch\" this issue, and whether it will generalize to photographic images.\n\n\nOverall, my assessment is positive, but, I would have liked to have seen some of the ideas taken one step further, e.g., with fine-tuning as a pointer as to if this problem is going to be very hard to solve or somewhat patchable with enough post-training data.\n\nQUESTIONS:\nmy wishes for this work, in question form:\n\n- did you try few-shot benchmarks?\n- did you try fine-tuning?\nand maybe most of all:\n- (if fine-tuning or another method works:) does patching this type of reasoning with synthetic images translate into improvements on tasks requiring this type of reasoning on photographic images?\n\nLIMITATIONS:\nsomewhat; I think even the obvious parts of my wishlist (e.g., fine-tuning/few-shot) were not addressed in the limitations section. I think the marginal risk of this work for society is low given that the images are all synthetic constructions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on the observation that human reasoning about images often involves comparing parts of the image that are spatially separated, the authors devise three evaluation tests for VLMs. The three tests are loosely inspired by human visual behaviors, i.e. “comparative perception”, a form of visual search (“saccadic search”), and curve tracing (“smooth visual search”). To this end, the authors provide three procedurally-generated task evaluations, Object Re-Identification, Visual Scavenger Hunt, and Circuit Connections, which are supposedly straightforward for humans, and evaluate 12 current VLMs. The results show that these visual tasks are still challenging for current VLMs.\n\nSTRENGTHS AND WEAKNESSES:\nThe manuscript is reasonably clearly written, and the figures illustrate the tasks and evaluation results well. \n\nEvaluating the capabilities of VLMs is currently a topic of interest to the ML community.\n\nWhile the three tasks clearly seem to capture specific properties of visual tasks that are difficult for current VLMs, it is not clear how important it is to devise an additional benchmark for VLM. As an example, the classic Bongard problems contain images requiring “saccadic search” or “smooth visual search”.\n\nRegarding 3.3, the authors are encouraged to cite relevant literature, such as the “visual routines” idea by Shimon Ullman.\n\nMinor & typos:\nOur structured evaluation suite allows us if VLMs can perform similar visual\nalgorithms to humans. \nspecific environments environments\nPlease reformulate: “We manually observe the first 20 responses of all models”\n… “with high confidence (p≤ 0.05) across almost models on the Standard and Unique Colors trials.”\n\nQUESTIONS:\nCould the behavior of the models described in the section “Failure Modes” be made more quantitative? It is difficult to understand what exactly is meant by “in contrast, higher-performing models possess higher visual capacity but only utilize it selectively. They do not perform well when objects are presented as coherent, connected entities. These models have strong implicit priors about entity analysis.”\nor similarly \n“However, their vision is either too fuzzy to perform the same tracing when wire colors are shared or they do not perform tracing.”\n\nThe authors state that “the object is always kept in-frame and unoccluded”, but Figure 1 suggests that objects are occluded. Could you clarify?\n\nHow unique are the present tasks compared to other VLM benchmarks involving reasoning on diagrams, including ARC, concept ARC, and Ravens progressive matrices?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}