VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs
Authors: Shmuel Berman
Jia Deng
Venue: NeurIPS 2025 spotlight ()
Keywords: vision
vlms
visual understanding
visual reasoning
reasoning
Primary area: evaluation
TL;DR: Our structured dataset allows us to analyze how model vision compares to human perception and to determine whether VLMs perform similar visual reasoning algorithms as humans can.

Abstract: Vision Language Models (VLMs) excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple perceptual tests. We present an evaluation that tests vision-language models’ capacity for non-local visual reasoning— reasoning that requires chaining evidence collected from multiple, possibly distant, regions of an image.  We isolate three distinct forms of non‑local vision: comparative perception, which demands holding two images in working memory and comparing them; saccadic search, which requires making discrete, evidence‑driven jumps to locate successive targets; and smooth visual search, which involves searching smoothly along a continuous contour. Flagship models (e.g., GPT-5, Gemini 2.5 Pro, Claude Sonnet 4), even those that perform well on prior primitive‑vision benchmarks, fail these tests and barely exceed random accuracy on two variants of our tasks that are trivial for humans. Our structured evaluation suite allows us to test if VLMs can perform similar visual algorithms to humans.  Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.

VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs
Shmuel Berman
Jia Deng
Princeton University, Department of Computer Science {sb6870, jiadeng]@prinnetonn.ed
Abstract
Vision- Models (VLMs) excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple perceptual tests. We present an evaluation that tests vision- usioonlanguaage models' capacity for nonlocal visual from multiple, possibly distant, regions of an image. We isolate three distinct forms of nonlocal vision: comparative perception, which demands holding two images in working memory and comparing them; saccadic search, which requires making discrete, evidence-driven jumps to locate successive targets; and smooth visual search, which involves searching smoothly along continuous contour. Flagship models (e.g. GPT-5, Gemini 2.5 Pro, Claude Sonnet 4), even those that perform well on prior primitive-vistiee vision benchmarks, fail these tests and barely exceed random accuracy on two variants of our tasks that are trivial for humans. Our structured evaluation suite allows to test if VLMs can perform similar visual algorithms to humans. Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.
-reaoinnnggg that requires chaining evidence collected
Introduction
Vision-Language Models (VLMs) have demonstrated impressive performance on complex multimodal tasks. These models appear to possess a deep understanding of images and achieve over 90% accuracy on benchmarks like A12D and ChartQA [10] 14 [2]. Such high-level competence would suggest these models have strong primitive visual skills. However, recent work such as VLMs are Blind [16] or HallusionBench (6). shows that VLMs excel answering high-dimensional questions that require background context but fail at recognizing simple geometry. Yet newer models score higher on these adversarial benchmarks, indicating that they possess stronger visual acuity.
While higher performance indicates stronger visual perception, these adversarial benchmarks do not test sequential visual reasoning. Most tasks can be simulated from the image-space to text-space, where each look at the image content is independent and does not require understanding relationships within the image itself. The actual reasoning can then occur in text space, where Large Language Models (LLMs) excel.
series of independent extractions
This strategy works only when combined with known assumptions about an image. Graph compre- hension questions are embedded with strong priors such as standardized layouts (e.g., axes, legends) and may not require the model to reason over nonlocal regions. Models could instead learn to exploit these learned commonalities and only superficially parse graphs. This reliance on convention over direct visual evidence would stunt models from parsing images that defy these norms. Indeed, recent literature supports the hypothesis that VLMs are not as proficient at reading graphs as benchmark scores suggest [13], For example, Masry et al. [33 showed that top models' performance catastrophi- cally declined on their novel ChartQAPr Pro dataset compared to ChartQA, indicating a missing skill set
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

for handling diverse, unseen charts. This, along with findings from benchmarks like HallusionBench (6. suggests VLMs still prioritize background knowledge over the visual evidence presented to them.
We identify three core types of nonlocal visual reasoning. First, comparative perception is the qualitative comparison of visual entities when precise discrepancies are difficult to articulate (c.g., recognizing that two complex shapes are not identical without explicitly itemizing every differing feature). Second, saccadic search, named after the rapid eye movements humans make, is the process of gathering and integrating information from different image regions. Each piece of evidence informs the next step; for example, this process used when consulting chart's legend, then locating the corresponding line, then referencing an axis. Third, smooth visual search describes the continuous tracing of visual elements, such following the outline of an object or tracing a curve to its conclusion. To systematically evaluate these capabilities, we introduce a procedurally grovedurally-genneraeed generated evaluation set comprising three task categories designed be trivial for humans and require minimal prior knowledge. We present three task categories: Object Re- Re-ldentification, Visual Scavenger Hunt, and Circuit Connections. Examples of these tasks appear in Figure I
We design our evaluation to answer the following questions:
When do VLMs err in basic perception or perceive correctly but fail at visual reasoning? Do early perceptual mistakes compound or self-correct during reasoning?
2. Can VLMs perform comparative perception and saccadic search? If so, must these models use natural language judgments to guide these processes, or can they execute these tasks through direct visual analysis?
3. Can VLMMs perform smooth visual search, an operation that involves tracing continuous contour path not easily decomposable into natural language steps? If VLMs find this continuous operation challenging, do they attempt to reframe it as a sequence of discrete operations, use different heuristic?
All tested models perform poorly on at least one variant of a task that is trivial for humans. The performance gradient on Object Re-ldentification shows that modern models selectively choose when to examine an image closely. Our work further shows that fuzzy vision interferes with visual reasoning and the models seem unable to self-correct. They instead rely on their prior natural language judgments over direct evidence in the image. VLMs struggle the most with smooth visual search and our analysis of when they succeed suggests that most models cannot trace lines.
Our major contributions are:
We release three procedural generators for an evaluation suite. Each generator creates synthetic image-question pairs for minimal-context task to probe three distinct facets of nonlocal visual reasoning: comparative perception, saccadic search, and smooth visual search. Our generator, evaluation sets, and evaluation code are available here
We conduct comprehensive evaluation of leading VLMs (including GPT-5, GEMINI 2.5 PRO, and CLAUDE SONNET 4) demonstrating that even flagship models lag far behind humans on trivial visual reasoning tasks, despite advances in primitive vision.
We create several variants of each task to determine why models fail to perform as well as humans. We also determine under which circumstances they look carefully at images and use their perception abilitics.
2 Related Work
Benchmarks on Perception Primitives. VLMs achieve strong performance on many complex tasks, including OCR, image captioning, and scene understanding [17 5 22 15]. However, this high-level competence contrasts with known deficiencies in low-level perception. For instance, VLMs can struggle with recognizing basic shapes or performing simple visual arithmetic (16) 7 20 Research suggests these perceptual limitations may originate in the language decoder, even with adequate image encoder representations [7] [6]. These foundational gaps raise questions about whether VLMs' success always stems from robust visual processing, and critically, when and how they engage in visual algorithms.

---

Object Re-ldentification
Question: "Image shows object made of shapes. Does this SAME object appear the Image up to transformation the ENTIRE object? Respond (yes) or (no).
Image
Image 2
Image
Image 2
(no) (The circle and line moved.)
Answer:
(yes) (The object whole scaled and rolated together)
Visual Scavenger Hunt
Circuit Connections
Question: "Starting at the red square, follow the text labels for 3 steps. What color do you end on?"
Question: "Which component does the wire from port 6 on the breadboard connect to?"
croci
oong pare
lnecicil
papple angie
O
O
Raje aeam
Answer: (C1)
Answer: (green) (Red square, yellow triangle, red cirole and finally the green aircle.)
Figure I: Object Re-ldentification (top): Determine whether the same object that appears in Image 1 also appears Image up to a transform of the entire object but not individual component shapes. Visual Scavenger Hunt (bottom-left): From the indicated shape, follow the written labels for the specified count and report the final shape's 's color. Circuit Connections (bottom-right). From a named port on the central breadboard, trace the wire to its endpoint. The prompts here abbreviated; full instructions are in the appendix.
Benchmarks on Visual Reasoning. To probe the underlying algorithmic visual skills necessary for robust image understanding, we take different approach from existing visual reasoning benchmarks. Evaluations like Bongard problems and ARC [3 4 test complex reasoning and are constructed as abstract puzzles that are challenging humans. Failing such task does not distinguish between a failure in high-level abstract reasoning and failure in fundamental visual processing. Our evaluation isolates these primitive visual skills by stripping away the puzzle-solving layer. While benchmark suites like VisOnlyQA and HallusionBench evaluate specific visual weaknesses, such as hallucination and we evaluate general visual algorithms.
illusion failures under controlled adversarial settings 9 6
Chart and Graph Understanding. The importance of visual data interpretation has motivated numer- ous evaluations and benchmarks such ChartQA and MultiChartQA, which expose VLMs to diverse charts 12 23 8. This area has also spurred development of several models specifically trained for chart understanding [14 11.. However, VLMs' underwhelming performance on more recent benchmarks such as ChartQAPro suggests that they have yet to develop robust graph-understanding capabilities [13]

---

3
Evaluation Design
Our evaluation isolates three core capabilities of nonlocal visual reasoning, each grounded in prin- ciples from classical human vision research. The first, comparatie perception, draws on work in perceptual organization and attention. Classic Gestalt principles such as proximity and connectedness suggest that humans automatically group visual elements into coherent objects (2T), while Treisman's Feature Integration Theory (FIT) 18] shows that attention serves to bind distributed features into unified representation. By comparing Object Re-Identification performance on the Unconnected variant against the Standard variant (where components are contiguous), we directly test whether models are sensitive to the Gestalt principle of connectedness. The second capability, saccadic search, is motivated by Ullman's 's theory of visual routines (19), which proposed that complex visual operations are composed of primitive procedures such as shifting the focus of attention and indexing marked locations. Specifically, it is based on systematic scanning. or using each observation to guide the next. The third capability, smooth visual search, corresponds to the visual routine of boundary and contour tracing within Ullman's 's framework.
3.1 Object Re-ldentification.
We use this task to test comparative perception: the model must hold two views in working memory and compare them under allowed transformations. Each instance of the task involves images, "Image I' and 'Image 2'. 2-6 shapes are shown in 'Image I'. In 'Image 2", this entire object undergoes random transformation (i.e., rotation, translation, and scaling) and ís always kept fully in-frame.
In half of the examples (the negative cases), one or more of the component shapes are also transformed independently to create structurally diffferent object. We avoid nearly imperceptible transformations,, such as small rotations, translations, and re-scalings, as we do not aim to test pixel-level accuracy. Additionally, we render distractor objects in 'Image 2' to reflect the human capacity re-identify objects across different environments. We restrict these distractor objects from occluding any components of the original object. Examples of this task can be found in Figure D
To disentangle VLM success conditions, we present three presentation variants. The Standard variant renders objects with physically contiguous component shapes to test the most intuitive concept of what an object is. The Unconnected variant removes this contiguity requirement to evaluate a more abstract object concept and determine how connectedness affects VLM perception. In the Pixel-Perfec variant, we do not apply any transformation to the object as a whole. Thus, positive examples of 'Image 2' are pixel-for-pixe matches of 'Image 1 (except for the added distractors). Examples of all three tasks can be found in the Appendix.
The examples are generated from a uniform distribution between "Yes" and "No", so the random guessing baseline is 50%.
3.2 Visual Scavenger Hunt.
This task tests saccadic search: the ability of the model make discrete, evidence-driven jumps across the image. In many real-world visual challenges, locating a pixel cluster with semantic content not enough. Instead, the task adapts the agent gathers new clues. For example, an unlit gas stove does not signal a gas leak unless the knob is in the on position. Humans use each observation to decide where look next, and this capability is vital generalist agents.
We design our Visual Scavenger Hunt task to evaluate iterative visual search explicitly. The model is presented with grid of different- collordd shapes, each labeled with a different color-shape pair. The prompt provides a starting shape, and requires the model to follow a straightforward "scavenger hunt" of shapes around the image for specified number of steps (which we call the chain length). The scene is randomly generated, so the entire scene must be searched to find a specific shape. A visual example can be found in Figure [
We evaluate the models on chain-lengths of 2, 3, and to test their performance degrades over long horizons. As we permute over 11 colors, the random chance baseline is approximately 9%.

---

3.3 Circuit Connections.
We frame Circuit Connections as an instance of smooth visual search. The model is shown a procedurally generated circuit diagram with breadboard, several components, and wires. The task is to specify which component a breadboard port is connected to; the model must trace a continuous contour-4 a wire wire-from its source to its terminal point. An example of this appears in Figure |
Inspired by the "Subway Connection" task in BlindTest [16), we refine the task to directly evaluate the ability to follow the wire. Our synthetic generator contains three variants: the Standard, Single Color, and Unique Colors versions. On the Standard trial, each wire one of five colors, selected randomly. The Unique Colors variant is designed the task is solvable by only looking at the ends of the wire. In the Single Color examples, all wires in an image have the same color. This trial is designed to prevent the model from taking shortcuts- by associating colors with locations or reasoning in natural language- rather than following the wire manually.
control; each wire has unique color that
Although our previous tasks are motivated by abilities necessary for real-world tasks, we design Circuit Connections to mirror a practical skill: reading wiring diagrams. We adopt informal conven- tions-wo -wires hold their directions crossings- improve diagram clarity and emphasize the need to trace the actual path. As we do not aim to test schematic-reading priors, we state these conventions explicitly the prompt.
The rendered image contains between 4-10 components, chosen from a uniform distribution. Thus, a random guessing baseline 14.29%.
4 Experiments
We evaluate GEMINI 2.5 PRO, CLAUDE SONNET 4, GPT-5, as well as other closed and open-source models on our benchmark in few-shot setting. All error bars represent standard error. The full evaluation details, including prompts, can be found in the Appendix.
Before running our experiments, we manually self-evaluated 200 examples of each task's main category to find a human baseline. Our evaluators scored 100% on the Object Re-ldentification and Visual Scavenger Hunt tasks and 99.5% on the Circuit Connections trial.
4.1 Object Re-ldentification
The results for Object Re-ldentification are summarized Figure 2 model significantly outperforms random chance, indicating that they cannot perform comparative perception. However, on the other two trials, models such as GPT-5, GEMINI 2.5 PRO, and CLAUDE SONNET 4 score between 12-24 percentage points higher. They remain over 20 percentage points below the human baseline in all trials.
On the Standard variant, no
The F1 (positive) and F1 (negative) scores, displayed in Figure subdivide the models into three classes. Firstly, models that essentially ignore the input and almost always predict the same result (MOLMO 7B and LLAMA 3.2 1 IB). The vast majority of models attempt to answer the question and are not especially biased across all trials, but give poor predictions for both classes. The third class consists of models that improve in the latter two variants.
Failure Modes A group of Londels-MoL.e (7B), LLAMA-3.2 2 VISION (11B), and PHI-4 MULTI- MODAL (14B)- effectively ignore the imput and almost always predict the same result, as evidenced by their Fl scores being 0.00 for one class across multiple variants (Figure 3). This indicates they either cannot perform comparative perception or do not attempt to do so across all trials.
The models highlighted in the orange box in Figure 3 fail in ways that show they have the ability to selectively compare objects only in certain cireumstances. On the Standard variant, their accuracy is near random chance, with the highest score being 60%. However, their performance significantly improves on both the Unconnected (GPT-5: 77%, GEMINI 2.5 PRO: 65%, CLAUDE SONNET 79%) and Pixel-Perfect GPT-5: 71%, GEMINI 2.5 PRO: 65%, CLAUDE SONNET 4: 70%) variants. In the Standard variant, the component shapes of the object are always physically contiguous. According to Treisman's 's Feature Integration Theory, humans bind basic features-cold shape,
5

---

Varient
Standard
Unconnected
Poxtl-Perfect
02
Accuracy (96
56
40
22
Homan
Cleste Streeet
Denisi 2.5 Pro
Cleste 2.7 Sonnes
Genme (278)
Honmal Small 3.248))
Oaerd Set USDD
InarmtE (248)
Phi Muttinettll 44)
Unita 3 Witmm
Maimo
Figure 2: Accuracy on all variants of Object Re-Identification.
orientation- into unified object representations through focused attention. These models' lower accuracy on the Standard variant indicates they do not bind objects the same way humans do. More specifically, their poor performance (<60%) suggests that these models struggle with fine-grained inspection of connected entities. This contrasts with human perception, where Gestalt principles of grouping (e.g., by proximity and connectedness) make connected objects easier to process and compare. Because the Standard variant is subset of the Unconnected variant, we suspect that these models capable of comparing different objects but engage this ability selectively. If two Gestalt-grouped objects are similar enough the model does not inspect them closely. They appear to bypass comparative attention when objects appear similar enough under Gestalt grouping cues.
Standard
9.47
0.53
3.56
0.12
0.37
0.25
0.38
9.65
0.88
1.47
12
8.41
6.74
0.64
8.72
0.66
6/72
0.44
8.71
0.86
9.20
0.
Unconnected
224-
0.78
0.75
6.77
0.75
0.11
6.71
0.31
6.79
0.60
0.54
1.00
0.58
6.28
Pixel-Perfect
F1
0.15
0.2
oPt3
Cluat Scoee-4
Genon k17m
Momee Small 12200
Gowest SM p200
Inteoot 1240
Uleoa 24lesm 1300
Moima 1780
PRou Muttinatte 1200
Omee SN8 2700
Gemii 25
Goode Semmer
Figure 3: for positive and negative classes across all trials Object Re-Identification. Some models predict identically across the majority of trials (red box.) The strong models (orange box) perform poorly on the standard variant, but become better at recognizing similar objects when tested on the other two trials.
Natural Language Strategies The stronger models also perform slightly worse on the Pixel- Perfect as they do on Unconnected.
this trial, GEMINI-2.55PPH scored 100% accuracy on the 11 responses in which it provided more than 10 tokens. We theorize that because the first and second images are so similar, these problems are easier to convert to an instance of natural-language comparison any perturbation disqualifies the second object from being identical the first.

---

Taken together, these results indicate that VLMs are currently incapable of generalist comparative perception and do always examine the images carefully.
4.2 Visual Scavenger Hunt
The results for Visual Scavenger Hunl are shown in Figure 4 GEMINI 2.5 PRO, O4-MINI, and GPT-5 perform well above the random-guess baseline of 9%. 04-MINI shows a clear trend downward as chain length increases.For For all models except GPT-5, 04-MINI, and GEMINT 2.5 PRO, performance is mostly static with respect to the length of the chain. CLAUDE 3.7 SONNET, PHI-4-MULTIMODAL. 03, and GEMMA 3 27B perform few percentage points above this baseline. The rest of the models score at about the accuracy of random guessing.
Variant (Chain Length)
80
Accuracy (%)
40
Bandom Chanos 1053
16
15
16
35
20
Human
Gembal 2.5 Pro
GPE-S
Casde Sonnet
Claude 3.7 Sonnet
Muttmestal (148)
Gemma (278)
Comm2 SVL (78)
Mittral Smatt 3. 3.12448)
Intem (14B)
Meims (7B)
Gunnn SVt coon)
Limme 3.2 vision
Figure 4: Accuracy on the Visual Scavenger Hunt task. Only GPT-5, 04-MINI, and GEMINI 2.5
PRO significantly outperform random chance.
Failure Modes We manually observe the first 20 responses of all models. The weaker models exhibit guessing behavior. They hallucinate paths even for a chain length of 2: an example of this type of response can be found in Figure Even with correct final colors, justifications often cite non-existent shapes or paths. 04-MINI often claims it cannot read the text and refuses to answer the question. We debunk this claim with an ad-hoc study; in isolation, 04- MINI was able extract text out of a randomly selected tile on 20 random examples from our dataset. However, it cannot always accurately extract the entire grid, shown in Figure 5 It successfully extracts every label, but hallucinates when it transcribes the actual shapes and colors. In our empirical analysis, all of GEMINI 2.5 PRO's mistakes occurred mid-search and seemed to be due confusion with nearby slot that had a similar color, label, or shape.
To determine if failures on this task stem from inability to locate the shapes and text, from breakdown in chaining these steps through saceadic search, we performed follow-up experiment. We decomposed the chain length 3 task into three sequential, single-step queries, where the model's answer for one step was used as the starting point for the next. The results are shown in Table [ In addition to overall accuracy metric is calculated only on trials where the model correctly navigated the first two steps, thereby isolating the error rate of the final jump. We use this conditional metric because grading intermediate steps requires verifying both shape and color, whereas the final step, like our main task, only requires identifying the correct color.
the three-step sequence, we report the Final-Step Error Rate. This
These results show that top-performing models are highly capable executing the atomic unit of the task when guided, whereas the other models cannot perceive well enough to perform the task. However, their high single-step accuracy contrasts with their low accuracy over multi-step search. Simple error multiplication suggests 04-MINI should achieve 83% accuracy (0.943), far higher than
7

---

Table Performance on the decomposed Visual Scavenger Hunt task over 75 trials. Accuracy reflects the success rate over the full shree-step interactive sequence, while the Final-Step Error is conditioned on the first two steps being correct.
Model
Accuracy (%)
Final-Step Error ( (%)
04-mini
90.67
6
Gemini 2.5
88.00
8
Llama 3.2 Vision (11B)
24.00
85
Qwen 2.5 VL (32B)
16.00
86
Qwen 2.5 VL (7B)
14.67
85
the 44.5% observed Figure This discrepancy indicates that the primary bottleneck for the strong models is not perception, but the inability perform multiple steps autonomously. Conversely, the other models' poor performance suggests they lack the basic perceptive abilities required to perform single step correctly.
Heuristic Strategies The marginal success of the second group models over random accuracy likely results from simple heuristics, like choosing the most frequent color that appears in the image strategy with 12% accuracy), not task capability. This evidenced by the lack clear trend in their performance chain length increases, which we would expect to decrease if they were performing lossy visual algorithm.
o4-mini: "Here is the printed label grid: Row 1: (1.5) yellow square (3.5) yellow square... Here shapes:
(1.1) green
Row 3: (3.1) square... (3.5) yellow square.. on the red triangle, and go to the the yellow square, which is at (3.5). The label says yellow square
pink square
magenia triarigle
purple circle
yellow square
would
(3.5) again... on (8.5). color you
steps you
on (yellow)"
Gemini 2.5 Pro: Starttart the red trianglo... CB)..
SQUID
triangle
cyeЛ square
squas
"yello
The
color red... located at (P5,C1)... has the label "gray trlangle Follow the label "gray triangle".. the gray located label 'lime triangle".. Follow label trannle"... located (R4, C3).. the label "blue circle"... Follow the labal "blue circle"... ... Final Answer: [blue]"
'ellow square".
C2)..
id square
magenta triangle
red circle
blue square
yellow SQUAR
trannle... label is "yellow square." The label for the ynllow square is square." So, we move to the cyan label for the square
Qwen-2255v 2.5L (32b): "Start
square
circle." pink is square." to the blue square. We end up on blue square. (blue).".
triangle, and then go to the magenta triangle, and then go to the yellow square, and then to the cyan square, then end the green (green)".
(11b);
gray triangle
pink triangle
circle
purple triangle
MUe squara
Figure 5: Example responses for Visual Scavenger Hunt from qualitative analysis. Most models can locate the first shape but have trouble extending the chain from there. 04-MINI and GEMINI 2.5 PRO are both nigh-performing but different strategies.
Self Correction GEMINI 2.5 PRO and 04-MINI lack the ability to self-correct this trial. Although examples with longer chains have more opportunities for error, they also provide additional signal that a mistake has been made. The other shapes the image are labeled randomly, meaning about 30% of the time they lead to shape that does not exist the grid. For human performing the task, this would be a strong signal of a prior mistake, but we do not observe this process across any model.
Most models lack the capability systematically search through novel image type that requires iter- ative visual grounding and evidence accumulation. Though the strongest models clearly outperform random accuracy, they make frequent mistakes, have inconsistent perception, and cannot self-correct.

---

4.3 Circuit Connections
Our results, summarized in Figure 6 indicate that all tested VLMs struggle with smooth visual search. Every model both exceeded random chance on the Standard variant and improved on the Unique Colors variant, with the exception of LLAMA-3.2 The hardest variant was Single Color, with a peak accuracy of 27% by GEMINI 2.5 PRO.
100
Variant
Single
Standard
Colors
Accuracy (%g
60-
40
Nandom Charce DA FRAA
Hommi
Gemest 2.3 Rr
Claste 2.7 Socned
63
Clesso Sprens
IntemiL33 13am
Pis Mutimantal 40)
Greed SVt (221
Owen set (78)
Mestrat Small 3.1248)
Gemena (778)
Maione 170
3 viseen (340)
Figure 6: Results on the Circuit Connections task. Performance rises drastically across the board from Single Color Unique Colors, suggesting that heuristics that do not involve tracing the line play large role in model success.
Table 2 presents the log odds coefficients for two key characteristics: the Euclidean distance of the connecting wire and the number of times this wire crosses other wires. The log-odds analysis allows us to interpret by what factor the odds of a binary outcome (c.g., success vs. failure) change for each unit increase predictor variable, like the distance between the source and target port or the number of wire-crossings.
Table 2: A Log Odds and p-values for Distance (px) and Number of Crossings Effects for Circuit Connections. The log odds coefficient, B, for characteristic indicates that one-unit increase in that characteristic changes the logarithm of the odds of a correct model response by B. The null hypothesis is that the log-odds coefficient of O.
Single
Standaed
Unique Tridl
Bffict
Cronsing Fifect
Crosiigs
Fifiet
Crosings
Log-odls
Log-odds
Log-nils
Log-odis
Log-odls
Logecdss
GPT5
400105
0100311
-110415
0,00235
00000
0100225
-0.0025
0.582
-00025
0.299
0.5477
0.0442
Gemini 2.5 Pho
-01064
0100446
-0.3399
0.0248
-0.0090
B:45 10-2
-0.1656
0.304
-00055
013335
-3.261
0.151
o4-min:
-00186
7.81 x 10-1
-0.2154
0.0122
-0.100
1.1% 10-5
-0.923
0.0142
-4005
0.159
400819
0.782
Clande Scmmat 3.7
-00106
080078
-09305
0.100
-0,090
1.2 10-1
.03021
0.153
-0097
696 x 10-1
2011 10
0.561
Clande Sernec
-0001
6.91 10
-0.5653
0.861
-0.121
7,85 x 10-1
-00420
0.540
-4009
0.00185
407360
0/00533
(144))
-0,0033
0.114
-B0587
0.798
-0,060
0.00363
-0.0722
0.567
-00121
1.56 x 10--
.03154
0.191
Fli-4 Muliinalal (14B)
+-S 10-1
0.791
-6% 10-6
0.990
-0.0070
0.0082
-0.9935
0.595
-00015
0.506
40.0193
0.921
Qwon2 SV1 (521)
0.114
-00387
0.795
-0.139
243 10
1848
0.301
-00080
B0129
408764
0.0099
Qwen2.50 SVL (7B)
.00003
0.0692
00.718
0.308
..0042
0.0839
-0.0236
0.575
.00083
0,2084
.02162
0.386
Mistral Small 3.1 (24B)
0.0000
0.00003
0.2182
0.449
0.0108
4.5 10-1
0.2331
0.188
80008]
0,20915
0.2546
0,309
Gemma 3 (278)
0.033
0000321
0,133
0.136
0.0165
5.01
0.1152
0.509
00121
8.13 10
0.4617
0.139
Motno (1B)
+000055
0.343
0.3027
012)
-0.0000
0.107
-0.2058
0.188
400000
0.388
+00750
0.746
Lim Vision (1185)
-00000
0.521
-0.3200
0.0888
-0.0141
BUT 10-
-0.2815
0.253
-00099
003003
4.1177
0.885
The statistics in Vable@aalse suggest that these models are not performing human-like tracing. The strongest predictor of model performance across all trials is distance, with high confidence (p < 0.05) across almost all models on the Standard and Unique Colors trials. The crossings effect is strongest in the nighest-performing models.
Across all Circuit Connections trials, the models' success rates and our log odds analysis indicate that no model performs contour tracing. They cluster into two groups- one which does slightly better than random on Unique Colors and Standard, and the three models which significantly beat random accuracy.
Failure Modes

---

Most models we tested show complete inability to perform line tracing. These models perform at or below random chance (14.29%) across all three variants of the Circuit Connections task (Standard, Single Color, Unique Colors), as depicted in Figure
This widespread low accuracy suggests these models cannot trace lines, and our log-odds analysis in Table supports this assertion. Many models in this group show significant correlation between Euclidean Distance and success in the Standard or Unique Colors trials when they beat random accuracy. However, the statistical significance of this effect disappears in the Single Color variant (e.g., MOLMO (7B) p=0.343) The wire-crossings effect for this group remains largely insignificant across all variants. This general lack of sensitivity to crossings is evidence they do not perform human-like line tracing intuitively, the task of line-tracing hardest when two lines interfere. We hypothesize they are able to exceed random accuracy by using color-cues and proximity heuristics.
Top performing models, such as GEMINI 2.5 PRO, CLAUDEE 3.7 SONNET, and O4-MINI, form a second group that performs better but still demonstrates significant limitations in tracing. Their accuracy drops significantly when color cues are removed or made uniform. For example. GEMINI 2.5 PRO's accuracy falls from 48% on Unique Colors to 27% on Single Color. While tracing is expected to be harder when wires of the same color interfere, this low performance suggests either lack of, or severely limited, tracing ability. The Euclidean distance of the connecting wire remains a significant negative predictor for these models across all trials. However, what distinguishes them from the first group is that these models do exhibit a statistically significant negative crossing effect in the challenging Single Color trial, where wires often cross other wires of the same color (e.g CLAUDE 3.7 SONNET shows log-odds coefficient of -0.9305 (p=0.0102) for the crossings effect in this variant.) This behavior is consistent with tracing algorithm, where crossings represent visually ambiguous points. However, because these models also do poorly in Unique Colors compared to humans, we suspect these models are possibly performing choppy, saccadic movements to find successive points on the wire rather than a smooth trace. This strategy aligns with the statistics: in the Single Color variant, models get caught in wire crossings and struggle to accurately jump to the next segment, impacting performance.
5 Conclusion
We introduce a suite of targeted tasks to distill nonlocal visual reasoning into three components: comparative perception, saccadic search, and smooth visual search. Most VLMs tested fail each benchmark, even those designed to deal with structured data, Closed-source VLMs achieved the highest accuracy but performed wildly differently under slight permutations of the underlying task. The task with the lowest performance ceiling was Circuit Connections, which requires smooth visual search. This suggests a principal limitation of these models is their difficulty with tasks that resist natural language reformulation.
This suggests that despite gains other visual benchmarks, VLMs lack reliable framework to analyze images. Without these capabilities, the vision of VLMs will remain fundamentally less robust than human vision.
While our synthetic setting isolates primitives, it does not capture the complete gamut of natural images, which may only require visual skims parse correctly. Additionally, we evaluate on only 200 or 125 examples per variant for cost-efficiency reasons.
Limitations.
References
[1] AbdeIrahman Abouelenin et al. "Phi-4-Mini Technical Report: Compact yet Powerful Multi-
modal Language Models via Mixture-of-LoRA.. In: arXiv preprint arXiv:2503.01743 (2025).
[2] Shuai Bai al. "Qwen2.5-VL. Technical Report". In: arXiv preprint 13X1v22502.13923 (2025).
[3] Mikhail Moiseevich Bongard. Pattern recognition. Spartan Books, 1970.
[4] François Chollet. On the Measure of Intelligence. 2019. arXiv: /991..1544 [cs.AI]
URL:
https://arxiv. orgo/o////ba/ abs///b/////
[5] Matt Deitke et al. "Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models", In: arXiv preprint arXiv:2409.17146 (2024).
10

---

[6] Tianrui Guan et al. Beelh: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision- Language Models". In: Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, pp. 14375- 14385.
[7] Kung-Hsiang Huang et al. "Why Vision Language Models Struggle with Visual Arithmetic?
Towards Enhanced Chart and Geometry Understanding" In: arXiv preprint arXiv:2502.11492 (2025).
[8] Mohammed Saidul Islam et al. "Are Large Vision Language Models up to the Challenge of
Chart Comprehension and Reasoning"" ? In: Findings of the Association for Computational
Linguistics: EMNLP 2024. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 3334 3368. DOI: 10 18653 / v1 / 2024 findings emnlp 191 URL: attps://aclanttholl
org/ 2024. findings- emnlp 191
[9] Ryo Kamoi al. "VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information". In: arXiv preprint arXiv:2412.00947 (2024).
[10] Aniruddha Kembhavi et al. "A Diagram is Worth Dozen Images". In: Computer Vision ECCV 20/6. Ed. by Bastian Leibe et al. Cham: Springer International Publishing, 2016,
pp.
ISBN: 978-3-319-46493-0.
[11] Kenton Lee et "Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Under- standing". In: Proceedings of Machine Learming Research 202 (2023), pp. 18893-18912, URL: sttps://proceed///
mlr. www....reeeeee....... wtwpsp/e0e/eeee223 ee23g httml
[12] Ahmed Masry et al. CharlQA: A Benchmark for Question Answering about Charts with Visual
and Logical Reasoning. 2022. arXiv: 2203. 10244
[Cs
URL: https::/taaatt org/
abs/2203 10244
[13] Ahmed Masry et al. "ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering". In: arXiv preprint arXiv:2504.055 05506 (2025).
[14] Ahmed Masry et al. "UniChart: A Universal Vision-languaal Pretrained Model for Chart Comprehension and Reasoning". In: arXiv preprint arXi::2305.14761 (2023).
[15] Mistral AI. Mistral Small 3.1: Available on Hugging Face. //tp:raasaaaa...mm ai ai/news/
mistral-small-3-1 2025.
[16] Pooyan Rahmanzadehgervi et al. "Vision Language Models are Blind: Failing to Translate Detailed Visual Features into Words". In: arXiv preprint (iti:2207.06588 (2024).
[17] Hugo Touvron et al. "LLaMA: Open and Efficient Foundation Language Models". In: arXiv preprint arXiv:2302. 1397/ (2023).
[18] Anne M. Treisman and Garry Gelade, "A feature-integration theory of attention". In: Cognitive Psychology 12.1 (1980), pp. 97-136.
[19] Shimon Ullman. "Visual routines". In: Cognition 18.1-3 (1984), pp. 97-159.
[20] Haoran Wei et al. "Slow Perception: Let's Perceive Geometric Figures Step-by-step". In: arXiv preprint 14
(2024).
[21] Max Wertheimer. "Laws of organization in perceptual forms". In: A Source Book of Gestalt Psychology. Ed. by W. D. Ellis. English translation of Wertheimer (1923). London: Routledge & Kegan Paul, 1938. pp. 71-88.
[22] Jinguo Zhu et al. "Intern Exploring Advanced Training and Test-Time Recipes for SoureSource Multimodal Models". In: arXiv preprint arXiv:2504. 10479 (2025).
[23] Zifeng Zhu et al. "MultCharrQA: Benchmarking Vision-Language Models on Multi-Chart Problems". In: arXiv preprint arXiv:2410.14179 (2024).
A Task Details: Examples and Prompts
This section provides visual examples for each task category, its variants, and the prompts used.
A.1 Object Re-Identification
In this task the model must determine if an object in 'Image 1' is identical to an object in 'Image under allowed transformations, possibly with distractor shapes present. For clarity, we provide examples for its three variants.
11

---

Figure shows examples for the Standard variant where object components are contiguous. Figure illustrates the Unconnected variant where object components are not necessarily connected. Figure displays the Pixel-Perfect variant, where positive examples have no rigid transformation applied to the object between 'Image and 'Image 2' (though distractors may present in Image 2).
Image
Image 2
(a) Standard Variant (Positive Example Objects the same)
Image
Image 2
(b) Standard Variant (Negative Example Objects are different)
Figure 7: Examples of the Object Re-ldentification (Standard Variant) task. Prompt: "The first image shows an object made of connected geometric shapes, which together form an object. Does this SAME object appear in the second image? For example, if component shape were be rotated or translated separately from the entire composite-object, it would be different object. Respond with (yes) (no) (inside the curly brackets). There may be extra shapes in Image 2 that are not part of the original object; as long as the object from Image is present, the answer is yes even there are other shapes present." [Two examples with answers would precede this.]
A.2 Visual Scavenger Hunt
In Visual Scavenger Hunt, the model presented with a grid of different-colored shapes, each labeled with different color-shape pair, as shown in Figure 10 The image generation procedure for the grid identical across variants; the only change is the chain length (number of steps) the model must follow.
12

---

Image
Image
(a) Unconnected Variant (Positive Example)
Image
Image
(b) Unconnected Variant (Negative Example)
Figure 8: Examples of the Object Re- le-lenttfficatonn (Unconnected Variant) task. Prompt: "The first image shows an object made of geometric shapes, which together form an object. Does this SAME object appear the second image up to rigid translation, rotation, and scale of the ENTIRE object as whole? Respond with [yes) or (no). There may be extra shapes in Image 2 that are not part of the original object; as long as the object from Image is present, the answer yes even there are other shapes present." [Two examples with answers would precede this.]
A.3 Circuit Connections
The Circuil Connections task (Section 3.3) assesses smooth visual search, requiring the model trace a wire from a specified port on a central breadboard to its connected component. Examples for its three variants, based wire coloring, are shown in Figure [T] the Standard Variant (Figure) Iaa), where multiple wires can share colors; the Single Color Variant (Figure [[b), where all wires are the same color; and the Unique Colors Variant (Figure IIc). where each wire has distinct color.
B Evaluation Methodology and Supplementary Information
B.1 Evaluation Parameters
Certain models-specifical MOLMO (7B) and LLAAM 3.2 VISTON technillyn accept multiple images input, although their documentation advises against this practice. Accordingly,
13

---

Image
Image
(a) Pixel-Perfect Variant (Positive Example)
Image
Image
(b) Pixel-Perfect Variant (Negative Example)
Figure 9: Examples the Object Re- Re-Identification (Pixel-Perfect Variant) task. Prompt:"The first image shows an object made of geometric shapes, which together form an object. Does this SAME object appear in the second image? For example, if component shape were be rotated or translated, it would be different object. Respond with [yes) or [no) (inside the curly brackets). There may be extra shapes in Image 2 that are not part of the original object; as long as the object from Image present, the answer yes even if there are other shapes present." [Two examples with answers would precede this.]
we ran each of these models twice in the few-shot setting: once with the image-based demonstrations and once without. We report the higher score of the two runs in every case. For LLAMA 3.2 VISION (11B) the inclusion of additional example images yielded higher accuracy on every run. MOLMO (7B) attained the best performance when the demonstrations were omitted.
Secondary metries for o3 were captured incorrectly due bug in the evaluation code. They are thus omitted from Figure 3 and Table 2
B.1.1 Computational Resources
Experiments were conducted using a combination of local high-performance computing resources and commercially available model APIs. For the open-source models, we evaluated locally on a system equipped with three NVIDIA A6000 GPUs (48GB VRAM each) and dual Intel(R) Xeon(R) Gold 5220R CPUs @ 2.20GHz. For other models, accessed them via API. We
14

---

pinksqquare
GEEy EQUaI
yellow CIrC
grantrandte
circle
yellow circlo
green triangle
gancrclle
blue circle
purple square
gray circle
graylrnngle
pur puipetrrang trangle
magentacint
ojan trangle
cynnciicle
graysquare
cysn ciicle
magenta square
green
Figure 10: Example of the Visual Scavenger Hunt task. For chain length the prompt would be: "Starting at the blue triangle, follow the labels for 4 steps. (For instance, in different example of steps, you might start blue triangle, then go to red square, then blue circle, then a magenta triangle, then green circle. The answer would be green.) After those steps, what color are you on? Answer with the color curly braces, e.g. [red)." [A single example tracing a path (accompanied by an image) would precede this, e.g: "We start at the blue square, and then go the purple triangle, and then go to the yellow square, and then go to the red square, and then end at the gray square. [gray]")
(a) Standard Variant
(b) Single Color Variant
Unique Colors Variant
Figure 11: Examples of the Circuit Connections task variants. The prompt: "Which component does the wire from port on the breadboard, which is gray rectangle with numbered ports, connect to? A wire is series of connected, same colored lines that go from the center of a port, represented on the screen as white circle, to another port. Each wire only connects two ports, one at either end. A wire will NEVER turn at the same spot that it intersects another wire, and wires do not change colors. Answer with the component label in curly braces, e.g (CO)." [Two examples would precede this.]
evaluated CLAUDE 3.7 SONNET, GEMINI 2.5 PRO, and CLAUDE SONNET via OpenRouter at model codes anthropic/ haappicc//laudeee claude-3. 7. T-sonnet::: :ent::hhinng thinking, google/ google/gemini- ieemini-2.55 pro-preview, and anthropic/ clludeee sonttet-sonnttttteee respectively. These OpenRouter models were evaluated between May 1, 2025 and May 15, 2025 (for CLAUDE 3.7 SONNET and GEMINI 2.5 PRO) and between September 17, 2025 and September 20, 2025 (for CLAUDE SONNET OpenAl models were evaluated via the OpenAl API at model codes We gathered model responses for these evaluations between April 28, 2025 and May 14, 2025 for 04-mini-2025. -0-6-04-166 between July 24, 2025 and July 27, 2025 for 20----2-0--16 and between September 17, 2025 and September 20, 2025 for gpt-5-2025-08-077
The
03-2025-04-16 and Ept--5-205500-00
15

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction clearly outline the paper's focus on evaluating nonlocal visual reasoning VLMs, the specific capabilities tested (comparative perception, saccadic search, smooth visual search), and the main findings of model underperformance, all of which are substantiated by the experimental results presented.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a Limitations paragraph at the end of Section 5 acknowledging the synthetic nature of the dataset and the limited number of examples per variant.
Guidelines:The paper includes a "LLimitations" paragraph at the end of Section acknowl- edging the synthetic nature of the dataset and the limited number of examples per variant.
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper is primarily an empirical evaluation of Vision Language Models and does not introduce new theoretical results or mathematical proofs requiring formal assumptions and derivations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We release our generator, evaluation sets, and evaluation code in the Introduc- tion, and will provide all evaluation details not currently in the paper in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The Experiments section and the Appendix specify the evaluation settings for the tested VLMs, including temperature, few-shot examples, and the number of examples per task variant. We also provide access to our generator and the specific examples we tested. Our data is fully reproducible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/nips. www.ccccccccccccc.ccccocccoce htt.i///guidde//eeeubbbbiisee sonddeSubiissioPPooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We list some evaluation details in the Experiments section and the rest in the supplemental Appendix. We do not train any models as part of our work.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
18

---

Figures 2 and include error bars in the accuracy plots. Table 2 reports p-values for log odds coefficients, indicating statistical significance of certain effects.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in the supplemental Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification; The research involved evaluating existing Al models on procedurally generated synthetic data and focused on model capabilities and limitations. None of this appears to present ethical concerns conflicting with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
19

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses the potential harms of VLMs that rely more vision in the Limitations section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The assets we release are procedural generators for synthetic evaluation tasks involving abstract visual elements, which do not inherently pose high risk for misuse in the manner large pre-trained models or datasets with sensitive information.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We only use external model assets, and cite them all properly.
20

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release procedural generator and some data in Section I and document it at that repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not crowdsource conduct experiments with outside human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
21

---

Answer: [NA]
Justification: The human performance baseline was established through self-evaluation by author and lab-mates, research with external human subjects that would typically require IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
This research evaluates Vision Language Models (VLMs), which inherently contain LLM components. LLMs are the object of the study, not a methodological tool used by the authors in an original or non-standard way to develop the research itself.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
22


Reviews:
[{"review_id": "E0YAviUQcK", "reviewer": "Reviewer_94Tc", "summary": "Based on the observation that human reasoning about images often involves comparing parts of the image that are spatially separated, the authors devise three evaluation tests for VLMs. The three tests are loosely inspired by human visual behaviors, i.e. “comparative perception”, a form of visual search (“saccadic search”), and curve tracing (“smooth visual search”). To this end, the authors provide three procedurally-generated task evaluations, Object Re-Identification, Visual Scavenger Hunt, and Circuit Connections, which are supposedly straightforward for humans, and evaluate 12 current VLMs. The results show that these visual tasks are still challenging for current VLMs.", "questions": "Could the behavior of the models described in the section “Failure Modes” be made more quantitative? It is difficult to understand what exactly is meant by “in contrast, higher-performing models possess higher visual capacity but only utilize it selectively. They do not perform well when objects are presented as coherent, connected entities. These models have strong implicit priors about entity analysis.”\nor similarly \n“However, their vision is either too fuzzy to perform the same tracing when wire colors are shared or they do not perform tracing.”\n\nThe authors state that “the object is always kept in-frame and unoccluded”, but Figure 1 suggests that objects are occluded. Could you clarify?\n\nHow unique are the present tasks compared to other VLM benchmarks involving reasoning on diagrams, including ARC, concept ARC, and Ravens progressive matrices?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The manuscript is reasonably clearly written, and the figures illustrate the tasks and evaluation results well. \n\nEvaluating the capabilities of VLMs is currently a topic of interest to the ML community.\n\nWhile the three tasks clearly seem to capture specific properties of visual tasks that are difficult for current VLMs, it is not clear how important it is to devise an additional benchmark for VLM. As an example, the classic Bongard problems contain images requiring “saccadic search” or “smooth visual search”.\n\nRegarding 3.3, the authors are encouraged to cite relevant literature, such as the “visual routines” idea by Shimon Ullman.\n\nMinor & typos:\nOur structured evaluation suite allows us if VLMs can perform similar visual\nalgorithms to humans. \nspecific environments environments\nPlease reformulate: “We manually observe the first 20 responses of all models”\n… “with high confidence (p≤ 0.05) across almost models on the Standard and Unique Colors trials.”", "was_revised": true, "final_justification": "I acknowledge the solid answers and additional evaluations by the reviewers and will maintain my increased score, although I am still not convinced that this is a fundamental contribution.", "rebuttal": "We appreciate your review and request to motivate our work against existing benchmarks! We have addressed your feedback below.\n## How important is it to devise an additional benchmark  if Bongard problems contain images requiring these skills?\n\nOur benchmark is important because it is a targeted evaluation of visual reasoning skills; other benchmarks do not tell us if VLMs fail specifically because they lack these skills. Bongard problems require puzzle solving ability, and failing on them doesn’t mean you can’t perform visual algorithms. We  isolate and test visual algorithms like line tracing and object comparison. \n\nThese visual algorithms are important because without robust low‐level visual reasoning, we cannot guarantee that models are using visual evidence instead of making assumptions about the image in front of them. Even the small VLMs we test solve ChartQA at above 80% accuracy and score above 60% on MathVista. Yet their vision is demonstrably brittle: they score below 20% on BlindTest, and frontier models score much worse on ChartQaPro than on ChartQa by 30–60%. This suggests that VLMs lean on pretrained world priors to fill gaps in what they cannot analyze visually (as also shown in Vo et al.’s “VLMs are Biased,” 2025). \n\n## “The authors are encouraged to cite relevant literature, such as the “visual routines” idea by Shimon Ullman.”\n\nWe agree and will add psychological studies into our paper to ground our evaluation. We will cite Ullman’s visual routines framework to explain how our saccadic search task mirrors his postulated “scan” and “trace” routines when following discrete targets and continuous contours, and we will reference Treisman’s Feature Integration Theory to clarify how the VLMs are unable to compare object features under some circumstances.\n\n## “Minor & typos: “Our structured evaluation suite allows us if VLMs can perform similar visual algorithms to humans.” “specific environments environments” Please reformulate: “We manually observe the first 20 responses of all models” … “with high confidence (p≤ 0.05) across almost models on the Standard and Unique Colors trials.””\n\nWe will reformulate these to be clearer and error free.\n\n## Could the behavior of the models described in the section “Failure Modes” be made more quantitative?\n\nYes, we will edit the failure mode section to be more quantitative. For object Re-ID, we will discuss the models’ higher accuracy on Unconnected than Connected, and how this means they process contiguous shapes differently. For Visual Scavenger Hunt, we will edit the paper to discuss the single-step performance of the models across the trials. For Circuit Connections, we will discuss how our log-odds analysis allows us to say which models are possibly performing tracing and hypothesize about what each model is doing.\n\n### “It is difficult to understand “in contrast, higher-performing models possess higher visual capacity… They do not perform well when objects are presented as coherent, connected entities….”\n\nWe will revise our text to make it more clear.  What we wanted to say is that because frontier VLMs score higher on the Unconnected variant than on the Standard variant, this means that they perform worse when objects are connected entities. We will edit this analysis into the paper.\n\nFig. 3’s shows that O4‑mini, Gemini 2.5 Pro, and Claude 3.7 Sonnet demonstrate moderate object‑distinction skills—but only in specific scenarios. On the Standard variant, their accuracy is near random chance (56%, 53%, and 49% respectively, as shown in Fig. 2). However, their performance significantly improves on both the Unconnected (O4-mini: 74%, …) and Pixel-Perfect (O4-mini: 50.4%, …) variants. The only difference is that in the Standard variant, the component shapes of the object are always physically contiguous (L141). \n\nThe models’ low accuracy on physically connected elements shows they fail to carefully inspect connected entities. We will also edit our paper to connect this to Gestalt principles of grouping (e.g., by proximity and connectedness) and how they generally make connected objects easier to process and compare for humans.\n\n### It is hard to understand “However, their vision is either too fuzzy to perform the same tracing when wire colors are shared or they do not perform tracing.”\n\nWe will revise our text to make it more clear. We wanted to say that because the top performing VLMs we tested doubled their accuracy from a setup in which all wire colors were the same to one in which each wire color was different, they are not capable of performing actual tracing.\n\nGemini 2.5 Pro, Claude 3.7 Sonnet, and O4-mini perform better than other models but still demonstrate limited tracing abilities. Their accuracy drops significantly when color cues are removed or made uniform While tracing is expected to be harder when wires of the same color interfere, this low performance suggests either a lack of, or severely limited, tracing ability. \n\nWhat distinguishes these models from the others is that they show a statistically significant negative crossing effect in the Single Color trial, where wires only cross other identically colored wires. Gemini 2.5 Pro shows a log-odds coefficient of -0.5349 (p=0.0244) for the crossings effect in this variant. This behavior is consistent with a tracing algorithm, because crossings are visually ambiguous points. However, because these models also do poorly in Unique Colors as compared to humans, we suspect these models employ a higher-granularity version of the \"visual localization\" heuristic, possibly performing saccadic movements to find successive points instead of smoothly tracing it. \n\n## The paper states ’the object is always kept in-frame and unoccluded’, but Fig. 1 suggests that objects are occluded.\n\nThe object is not occluded because we define the object as the configuration of shapes in Image 1 and do not consider the components of the object occluding each other as the object being occluded. The primary object in our task is composed of component shapes—such as triangles and rectangles—which together form a single object. These components may partially occlude each other, similar to how the intersecting lines in an “X” overlap. However, such self-occlusion is considered intrinsic to the object and does not violate our definition of occlusion.\n\nImage 1 is literally just the primary object itself so it is trivially unoccluded. In Image 2, where we add distractor objects, we guarantee that no part of the object is ever occluded by a distractor object. In negative examples where the object is transformed, we do not guarantee that the component shapes won’t occlude each other in Image 2.\n\n## How unique is this compared to ARC, Ravens progressive matrices(RPMs), etc?\n\nOur benchmark is unique compared to these because we evaluate lower level skills in isolation. Unlike Bongard, ARC, and RPM puzzles, our tasks isolate visual algorithms from abstract reasoning. Solving Bongard problems forces you to think about a conceptual rule across two example sets and even know the rule’s name; ARC puzzles demand hypothesis testing on limited examples to extrapolate abstract patterns; and Raven’s matrices rely on relational reasoning to complete interactive patterns. These are puzzles that can be difficult for humans to solve. Our task strips away the reasoning and the knowledge. Our problems are intuitive— there is no puzzle to solve, and the human solve rate is >=99%. We specifically target primitive visual algorithms, such as holding objects in working memory (comparative perception) or continuous visual tracing (smooth visual search). \n\nSolving Bongard problems does not imply mastery of the basic visual reasoning we identified—searching over images or tracing lines—nor does failing one imply incapacity, since Bongard tasks demand puzzle‑solving skills. The skills we test are very primitive, and to our knowledge no prior evaluation has isolated them. New models like o3, Gemini 2.5 Pro, and Claude 3.7 perform much better on “VLMs Are Blind,” and our central research question is whether those gains translate into consistent execution of the low‑level visual algorithms humans use."}, {"review_id": "FnnrHlNu6b", "reviewer": "Reviewer_wW7w", "summary": "the authors present a 3-task benchmark of visual challenges easy for humans but potentially difficult for machines.\nthese consist of procedurally generated images spanning 3 categories: object re-identification up to rigid transformation, a visual scavenger hunt, and circuit connection following. The authors argue all of these tasks require \"non-local\" visual reasoning, i.e., instead of \nThe authors verify that the tasks are easy for humans (~100% in all cases), and easy to objectively evaluate the accuracy of.\nBut, all models, including frontier models/reasoning models/etc., fall significantly short.\nThus, the released datasets/environments form a new challenge for vision+language models.", "questions": "my wishes for this work, in question form:\n\n- did you try few-shot benchmarks?\n- did you try fine-tuning?\nand maybe most of all:\n- (if fine-tuning or another method works:) does patching this type of reasoning with synthetic images translate into improvements on tasks requiring this type of reasoning on photographic images?", "limitations": "somewhat; I think even the obvious parts of my wishlist (e.g., fine-tuning/few-shot) were not addressed in the limitations section. I think the marginal risk of this work for society is low given that the images are all synthetic constructions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nI quite liked this paper! It's difficult to find simple tasks where models fail, and the authors have specified three of them. It's hard to argue against.\n\n- I liked that the authors constructed a few versions of the tasks to attempt to isolate/specify which parts are particularly hard. This provides a nice easy-to-hard curriculum, which could inform training.\n- The argument of the paper is very clear/convincing, e.g., human performance is given for all tasks.\n\nWeaknesses:\n\n- I worry that some of the prompts used to specify the tasks are underdetermined. For circuit connections, while I used commonsense to infer that the breadboard is the big gray thing in the middle of the image and the components are the smaller squares with C_#; to make the task even less debatable, it would have been helpful to more precisely specify this either with text, or with a label in the image. A similar concern with the visual scavenger hunt: I didn't know what it meant to \"follow the label\" initially --- it would have been helpful to be more precise, like: \"starting at the teal triangle, read the text ontop of it, and then \"follow the label,\" i.e., move to the shape described by that label. Repeat the process of following the label for 3 steps, in total.\"\n\n- It would have been nice to have some few-shot results in addition to the zero-shot results.\n\n- While I appreciate the sentiment, the authors claim in L126 that the task is purely visual and cannot be simulated in natural language. I don't know if \"ascii art\" constitutes natural language, but I could imagine rendering all of the author's challenges (save for the visual scavenger hunt which relies on color) in that format. (in fact, that would be a potentially cool ablation of whether or not this task is fundamentally bottlenecked by the visual encoder itself).\n\nWould have liked:\n\n- I would have liked to have seen a non-synthetic-image version of these tasks --- while not programatically generate-able, you could imagine annotating versions of these tasks on some photographic images.\n\n- I am curious to see if training on synthetic data like this (whether via sft or RL in an environment: both are supported by the author's work) will \"patch\" this issue, and whether it will generalize to photographic images.\n\n\nOverall, my assessment is positive, but, I would have liked to have seen some of the ideas taken one step further, e.g., with fine-tuning as a pointer as to if this problem is going to be very hard to solve or somewhat patchable with enough post-training data.", "was_revised": true, "final_justification": "I maintain my positive rating, the authors added the requested experiments and strengthened their work.", "rebuttal": "Thank you for your insightful review! We are encouraged that you found the paper convincing. We have addressed your feedback below.\n\n## “It would have been nice to have some few-shot results in addition to the zero-shot results.”\nAll tasks and models in the paper were evaluated in a few-shot setting(L191). This was a core part of our experimental setup to mitigate any prompt ambiguity and solely test the tasks themselves. \n\nHowever, for completeness of this response, we evaluated a subset of models in a zero-shot setting as well.  All of these examples were generated independently for each model. These results are not comparable to each other and with the results in the paper (Fig.s 2,4,6), as the examples were generated (via random sampling) per-model, but they are comparable between few-shot and zero-shot. They also have different difficulty levels(such as a reduced maximum number of wires from fifteen to twelve, and a different minimum rotation amount in Object-Reidentification). However, the results between a single model between few-shot and zero-shot are comparable:\n\n### Object Re-Identification\nStandard, zero-shot: Claude 3.7 Sonnet (48.00 %), Gemini 2.5 Pro (57.40 %), o4‑mini (58.20 %), o3 (54.23 %), Molmo 7B (49.20 %), Llama 3.2 VL (11B) (49.00 %), Qwen 2.5 VL (7B) (46.00 %)\n\nStandard, one-shot: Claude 3.7 Sonnet (56.00 %), Gemini 2.5 Pro (54.33 %), o4‑mini (65.60 %), o3 (57.50 %), Molmo 7B (48.00 %), Llama 3.2 VL (11B) (49.00 %), Qwen 2.5 VL (7B) (47.40 %)\n\n### Visual Scavenger Hunt (VSH)\nChain Length = 3, zero-shot: Gemini 2.5 Pro (50.00 %), o4‑mini (32.00 %), Qwen 2.5 VL (7B) (12.80 %)\n\nChain Length = 3, one-shot: Gemini 2.5 Pro (40.00 %), o4‑mini (34.00 %), Qwen 2.5 VL (7B) (13.2%)\n\n### Circuit Connections\nStandard Variant, zero-shot: Claude 3.7 Sonnet (42.67 %), Gemini 2.5 Pro (52.00 %), o4‑mini (56.00 %), Llama 3.2 VL (11B) (18.20 %), Qwen 2.5 VL (7B) (12.80 %), Molmo 7B (20.60 %), \n\nStandard Variant, one-shot: Claude 3.7 Sonnet (40.40 %), Gemini 2.5 Pro (52.00 %), o4‑mini (50.00 %), Llama 3.2 VL (11B) (17.40 %), Qwen 2.5 VL (7B) (13.20 %), Molmo 7B (18.40 %).\n\nBecause the performance between zero-shot and one-shot does not differ that much, using few-shot should also not make a big difference. However, we note that o4-mini and Claude 3.7 Sonnet may be an exception on Object-Reidentification, where their performance increases. This indicates that there is some small amount of ambiguity that is cleared up with the examples. \n\n## The authors claim in L126 that [circuit connections] is purely visual and cannot be simulated in natural language.\n\nWe do not claim this but we agree that the original working in the paper \"This task is purely visual and cannot be effectively simulated in natural language.\" can be misleading and will revise it accordingly. What we meant to say is that the way that humans solve this task cannot be simulated in natural language. While the humanlike algorithm for Object Re-Identification and VSH can be intuitively broken down into natural language statements— “The red triangle was scaled up, so they are not the same object…” or “First we go to the red triangle, and then we see it say blue square…”— contour tracing has no intermediate steps in natural language in the way that humans perform it. It can only be broken down into smaller instances of line-tracing. Psychological research, such as Ullman’s theory of visual routines, confirms the idea that line-tracing is a primitive visual technique for humans.\nWe agree with the reviewer that these tasks could be represented non-visually, such as through ASCII art, and think this is a good direction for future work.\n\n## \"I would have liked to have seen a non-synthetic-image version of these tasks”\n\nWe construct a more natural version of VSH and evaluate models on it.  by taking images from the COCO dataset and rendering labels on the objects which point to other objects in the scene. We use image segmentation data to find the pixels that belong unambiguously to one object and render text labels at those locations. We evaluate several models on this in a two-shot setting with the following prompt (the starting object is different per-image): \n\n\"Starting at the cat in the image, follow the text labels for 2 steps. (For instance, in a different example you might start at the topmost car, then go to the leftmost person, then the rightmost bottle. The answer would be bottle.) After those steps, what object are you on? Answer with the object name in curly braces, e.g. {car}.\"\n\nThese are the results:\n\nClaude 3.7 Sonnet: 33%\n\nGemini 2.5 Pro: 38.67%\n\nQwen 2.5 VL 7b: 26.67%\n\nQwen 2.5 VL 32b: 40.00%\n\nLlama 3.2 11b: 28%\n\nGemma 3 27b: 40%\n\nThese results show that the smaller models are much more performant on the natural version of the task. However, it is important to note that this task is very different from the shape-based version— it may have between 3 and 10 text annotations instead of the consistent 25 in the synthetic version of VSH, but the average number is four.\n\nRandom accuracy is also not well-defined because what is an object in real-life is not well-defined, but a simplified version of random accuracy is picking a label at random which leads to 25% accuracy. Notably, we do not observe reasoning chains among the two models which did poorly on the synthetic variants but did better on this one- Gemma 3 (32b) and Qwen 2.5 (32b). We suspect this is because it is not a true visual search— image context gives strong hints about where objects are. We suspect these models use their internal representations of the image to model the short chain without actually performing saccadic search. Gemini 2.5 and Claude 3.7, in contrast, show saccadic traces on almost every example. We hypothesize this is more difficult, as there are many more spots of ambiguity to jump to on a natural image. \n\n## \"I am curious if training on synthetic data like this will \"patch\" this issue, and whether it will generalize to photographic images.\"\nWe finetune two models on our tasks and find that only Visual Scavenger Hunt can be patched with finetuning. We used a peak learning rate of 1e-4, AdamW optimizer and a cosine learning rate schedule. We trained on each task for a single epoch of 20,000 images.\n\nAfter being trained, only Qwen7b on VSH was able to achieve 100% accuracy on a separate validation set. Furthermore, it was trained on chain lengths of 3 and 4, and achieved only 20.8% accuracy on chains of length 2 and 22% accuracy on chains of length 5. \n\nFor Object Re-Identification, the Qwen 2.5 (7B) reached 77% accuracy and Llama 3.2 (11B) reached 84% accuracy on the Unconnected variant. For Circuit Connections, Qwen 2.5 (7B) reached 77.6% accuracy and Llama 3.2 (11B) reached 70.80% accuracy on the Standard variant.\n\nPreliminary testing of these fine-tuned models on the semi-natural task (see “non-synthetic” section above) was inconclusive; the model output was degenerate. More research is needed to see if reinforcement learning can solve this problem without causing catastrophic forgetting.\n\nWe did not conduct a hyper-parameter search to try to reach higher accuracy. However, these models were tested and trained on data from the same variants. Based on these fine-tuning experiments and the results of frontier models in Fig. 6, we believe it is unlikely that existing architectures are capable of smooth visual search. However, we do not feel that there is enough evidence to say that current VLMs architectures cannot be trained to perform comparative perception and saccadic search. \n\n## The Circuit Connections prompt is not fully determined\n\nThe Circuit Connections task is determined because we write in the full prompt that the breadboard is the grey rectangle. The prompt in Fig. 1 is a truncated version of the full prompt, which can be found in Fig. 5 of the supplemental. We also provide the model with few-shot solved examples, which it can use to determine what the components are.\n\n## The Visual Scavenger Hunt(VSH) prompt is underdetermined\n\nThe VSH task is not undetermined because we included a single solved example with a fully annotated path (e.g., \"We start at the red triangle, which says blue square. We then go to the blue square.... Finally, we end on the {red circle}\") before asking the model to solve a new example. This detailed annotation defines \"follow the label,\"as the label-shape path is unique in the image given. The full prompt can be found in Fig. 4 of the supplemental."}, {"review_id": "87uHdd8oPP", "reviewer": "Reviewer_3qvb", "summary": "This paper is an empirical study on sequential and non-local spatial reasoning in VLMs. The paper introduces an evaluation suite for 3 such reasoning tasks and demonstrates that current VLMs generally fail at these tasks even though they are mostly trivial for humans to complete.", "questions": "1. Fig 2, 4, 6 do not have the variants for o3. Is there a reason these results are missing?\n2. The discussion in Section 4.2 around the justifications is insightful, as it shows that these tasks can fundamentally be broken down into smaller subtasks which the model should be able to solve. This suggests potential strategies for mitigation. How do the models perform in the chain length = 1 variant? Can you extensively verify which tasks are feasible for breaking into small parts and solving and which are not?\n3. Similarly, the viability of self-correction is an interesting consideration. Is the model aware of the viability of a self-correction strategy, e.g., is it aware in the prompt that some texts point to non-existent objects?\n4. In Sec 4.3, it is interesting to identify that the model is not performing tracing. Can we hypothesize what approach the model is taking to solve the problem?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe evaluation suite is of great value to the community as well as the main findings around current limitations of VLMs. The empirical analysis is also thorough across many different VLMs to definitively demonstrate the key finding.\n\nWeaknesses:\nThe paper primarily focuses on identifying an existing weakness within VLMs. While this work is useful and important, the analysis towards identifying the root causes and limitations of the weakness is limited. Finally, there is limited discussion towards overcoming the weakness. Note that while the paper is not required to present a complete positive result in fully resolving the issue, it can be strengthened with some reasonable direction towards improvement.", "was_revised": true, "final_justification": "Most of my questions were answered and resolved. My remaining question is how to overcome poor performance. The authors suggest some ideas around breaking down tasks or separately executing visual steps. I look forward to seeing some of these ideas play out in future work", "rebuttal": "We thank the reviewer for their feedback. We are encouraged that they find our evaluation suite thorough and valuable, and have addressed their concerns below.\n\n## The analysis of identifying the root causes of weakness and ways to overcome the poor performance is limited\n\nWe will add additional analysis. Our experiments suggest VLMs’ poor performance stems from selective perception: when comparing two objects composed of noncontiguous shapes, all frontier models improved by ~30% despite identical setups (L212–213). This gap shows that models only inspect items closely when they are perceived as distinct entities.\n\nTo overcome this, a task could explicitly treat each object as a distinct entity—e.g., “a group of birds” rather than “flock”—to encourage active comparison. We believe reinforcement learning or chain-of-thought prompting would enable this behaviour in all settings, though even in the best “Unconnected” trials, model accuracy remains well below human levels.\n\nMoreover, our stepwise experiments (see “Break down the tasks” above) reveal that model performance degrades far faster than independent error multiplication suggests. Prompting models to execute visual steps separately may curb this error amplification, similar to the success seen in chain-of-thought  (Wei et al., 2022).\n\n\n## o3 Results are missing. What is the reason?\n\no3 was the most expensive model we evaluated, and per-trial it was about as expensive as all of the other models combined. However, OpenAI reduced the cost per-token on June 10th, and so we present the full results for o3 (with previous results repeated):\n\nObject Re-Identification, Standard: 55.2%\n\nObject Re-Identification, Unconnected: 79.2%\n\nObject Re-Identification, Pixel Perfect: 69.60%\n\nVisual Scavenger Hunt (VSH), Chain Length 2: 22.40%\n\nVSH, Chain Length 3: 18.4% (typo in Fig. 4, which says 18.5)\n\nVSH, Chain Length 4:  13.60%\n\nCircuit Connections, Single Color: 20%\n\nCircuit Connections, Standard: 32%\n\nCircuit Connections, Unique Colors: 38.4%\n\no3 follows a similar performance trend across trials as Gemini 2.5, and o4-mini, bolstering our paper’s results. Interestingly, o3 and Claude 3.7— both slightly older models— do worse on VSH. This indicates newer frontier models have made progress on saccadic search.\n\n## In VSH, is the model aware that some text points to non-existent objects? Do models self-correct?\n\nThe models are not aware that the text points to non-existent objects and we did not see any instances of self correction. We examined the first twenty responses of all models across the chain length 3 variant, and we did not see a single instance of self-correction(L193-195). The low-performing models imitate the sentence structure of the one-shot example given but completely hallucinate a path and show no awareness that the shapes they mentioned do not exist on the board. The higher performance models had a much higher success rate of matching text-labels with shapes, but we did not observe a single instance of backtracking. \n\nHowever, these models would frequently say they could not find the next object on the board and give a random guess, such as the following output by Gemini 2.5 where it could not find the next shape (a magenta circle):\n‘We need to find a \"magenta circle\". Looking at all the shapes, there is no circle that is magenta. This suggests a possible error in the label. One interpretation is that the label is a typo for \"magenta square.’ It then proceeded to continue by checking the magenta square.\n\nIn our initial inspection of the twenty first responses per model, we did not observe any instances of models explicitly saying they could not find a shape that truly did not exist. However, a search for the word “exist” revealed the following response by o4-mini after it landed on a cyan triangle by mistake:\n“We start at the cyan triangle. The text-label on this shape is \"green circle\". There are two greenish circles, a lime one and a darker green one. The path from the dark green circle leads to a dead end, as it instructs us to find an \"orange triangle\", which does not exist. Therefore, we must follow the path to the lime circle.” The model justifies its choices instead of backtracking.\n\nThis suggests that the lack of self-correction is a fundamental part of model failure modes in VSH. We have edited the discussion around Fig. 5 to include this analysis.\n\n\n\n## Can you extensively verify which tasks are feasible for breaking into small parts and solving and which are not?\n\nWe posit that VSH is the most naturally decomposable task, and perform an additional experiment to verify its performance in this setting. We also add this analysis to the paper.\n\nWe ran an additional experiment on VSH that breaks down saccadic search. Instead of asking the model to step on its own through the whole chain of length 3, we asked it to go one step at a time (also in a one-shot setting) with a chain length of 1. We did this for three steps, using the model’s answer as the starting point for the next chain.\nThese are the results:\no4-mini: 90.67%\nGemini 2.5: 88%\nLlama 3.2 Vl (11b): 24%\nQwen 2.5 VL 7B: 14.67%\nQwen 2.5 VL 32B: 16.00%\n\nThis experiment also allows us to capture the per-step error rate of each model. We report only the final-step error rate—calculated over trials in which the second‑to‑last step was answered correctly—since intermediate steps require specifying both the next shape and its color. Gemini-2.5 has a final-step error rate of 8% and o4-mini has an error rate of just 6%, whereas the other models all have an error rate above 85%. \n\nGemini 2.5 and o4-mini’s higher performance in this setting shows that they are capable of the atomic unit of the saccadic search, but are just incapable of an autonomous, sustained saccadic search across the image. After all, simple error multiplication suggests this would yield a success rate of 83% for o4-mini and 77% for Gemini 2.5, which neither reach. \n\n The lower performance of the other models is supported by prior work such as VLMs are blind, and suggests that they cannot perform the atomic unit of the task. We have included this analysis in the paper.\n\nIt is not obvious to us how to break down line-tracing and object comparison into steps. Psychological research generally confirms that these are visual primitives (Ullman’s theory of visual routines).\n\n\n\n## What approach is the model taking to solve in Circuit Crossings(4.3)?\n\nWe hypothesize, following Wei et al.’s “Slow Perception” work, that the best‑performing models execute a series of discrete “jumps” along the wire, whereas the poorest performers attempt only a single global guess. Our log-odds analysis in Table 1 corroborates this: models show no evidence of smooth, human‑like contour tracing. We will revise the analysis section accordingly.\nAcross all Circuit Connections trials, no model demonstrates true contour tracing. Models fall into two clusters. The first group—including Molmo (7B), Llama‑3.2 Vision (11B), etc—performs at or below the 14.3% random baseline on Standard, Single Color, and Unique Colors (Fig. 6). Their accuracy correlates only with wire proximity (distance effect) when they exceed chance in Standard or Unique Colors, and they show no sensitivity to crossings (p > 0.1). This indicates reliance on simple localization heuristics rather than actual tracing.\n\nThe second group—Gemini 2.5 Pro, Claude 3.7 Sonnet, O4‑mini, and O3—outperforms random chance, but do not score highly enough on any variant to conclude they can perform contour tracing. When color cues are removed in the Single Color variant, accuracy plummets (e.g., Gemini 2.5 Pro from 47.9% to 27%). The Single Colors is designed to be the only trial where line-tracing is the only working strategy, as in other variants wire colors are often unique. This gap in performance demonstrates an inability to maintain a continuous follow‑the‑line process. The only piece of evidence that suggests that these models do try and trace lines is that they exhibit significant negative crossing coefficients in Single Color (Gemini 2.5 Pro β = –0.535, p = 0.024; O4‑mini β = –0.715, p = 0.031; Claude 3.7 Sonnet β = –0.931, p = 0.010). Intuitively, the task of line-tracing is hardest when two lines interfere, but should not interfere with color-matching heuristics. Only in these models is there a significant effect. However, because of their poor performance, we hypothesize their tracing capabilities consist of a heuristic of discrete, saccadic hops that “get stuck” at ambiguous crossings. This pattern matches Wei et al.’s “Slow Perception” hypothesis: rather than tracing continuously, they perform coarse jumps informed by color and proximity, failing where such cues conflict or vanish."}, {"review_id": "lqzSEwLAU9", "reviewer": "Reviewer_2sTq", "summary": "This paper presents a new benchmark evaluating nonlocal visual reasoning in VLMs. It is composed of three different tasks designed to test different types of nonlocal reasoning (comparative perception, saccadic search, smooth visual search). Results show that SOTA VLMs mostly fail to perform these tasks, which humans can easily complete, suggesting ample room for improvement in their reasoning capabilities.", "questions": "Would SOTA VLMs also mostly perform randomly on these reasoning types when they are required to understand natural images rather than synthetic shapes?\n\nAre the reasoning types (e.g. comparative perception) inspired by studies in psychology? If so, it would be nice to provide references to their use in that field.\n\nDoes the object re-identification task connect to existing work on Image Difference Captioning or other methods to compare two images?\n\nL174 How is the Circuit Connections task different from the existing Subway Connections task?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "*Strengths:*\n\nAs VLMs improve, benchmarking their abstract reasoning abilities is becoming more important.\nIt is interesting to see tasks that are inspired by human psychology, which can be a fruitful avenue for finding new ways to test these models.\n\nThe idea of testing VLMs for sequential visual reasoning, particularly decomposed into three different sub-categories, appears to be novel and it is interesting to what degree current VLMs encapsulate them.\n\nThe results on the visual scavenger hunt task seem interesting, since there is a clear differentiation between models that do and do not show non-trivial performance.\n\n*Weaknesses:*\n\nI have some major concerns about the methodology which make me question the degree to which the results in the paper are meaningful.\nMy central concern is that these images depicting abstract shapes are very different from the natural images that these models are normally trained and evaluated on. This has already been explored in prior work (L22). If other works already show that modern VLMs don’t understand basic geometry in abstract shapes, what does this work add? In other words, do these results really say something interesting about different types of reasoning, or are they just another demonstration of this known phenomenon?\n\nI also have concerns with the task images and prompts used (Fig 1). Some of the textual prompts use ambiguous and confusing wording. For example, the first task uses the term “object” in an abstract sense that is not obvious, and the second example in the figure states that the correct answer is YES because the object was scaled and rotated when scaling might not be considered a rigid transformation. In the third task, a phrase like “starting at the teal triangle” might be interpreted either to mean the given geometric shape and color, or the text stating “teal triangle”. It could be that VLMs struggle because they are not as good at interpreting confusing or convoluted prompts while humans manage to do so – this doesn’t necessarily measure higher level reasoning.\n\nSimilarly, the text in the image prompts is small and hard to read. Although you partially tested this in Supp A.2 (on only one of the many texts in images for one of two tasks using visual text), it could still be that results would improve if text resolution were increased.\n\nSome less significant issues are:\n* Some of the presentation can be improved. I found Fig 1 and its caption to be a bit convoluted, and could benefit from numbering the tasks or even from splitting into three figures accompanying each task in the text.\n* L279 seems like an overclaim. Even if poor results are because of a gap in reasoning abilities, it is not clear that explicit structured reasoning is the solution. For example, maybe inference compute scaling methods without explicit structure such as chain-of-thought could help.", "was_revised": true, "final_justification": "My main concerns were well-addressed in the author response.", "rebuttal": "We appreciate your detailed feedback, and respond below.\n\n## Images depicting abstract shapes are very different from the natural images that these models are normally trained and evaluated on\n\nThese shapes are not different from the training data of these models because they have been trained on charts, and diagrams, which make liberal usage of abstract shapes (See technical reports for Qwen, Phi, etc). We evaluate in this setting because it is an effective way of isolating visual skills from learning natural correlations (e.g a chair is usually near a table).\n\n## If other work shows that VLMs don’t understand basic geometry, what does this work add?\n\nWe test basic visual reasoning which no other benchmark tests. Previous work tested VLMs’ capacity to perceive abstract shapes and showed that older models could not count shapes or tell if they intersected. However, SOTA models (e.g., o4-mini) now achieve 90% accuracy on \"VLMs Are Blind\", which shows they do understand basic geometry. \n\nThese models perform at around random accuracy on some portions of our benchmark, and significantly below humans on all portions. This discrepancy means that our tasks assess something more demanding: the ability to integrate multiple, nonlocal visual cues. Our evaluation allows us to prove that even though models can match humans on higher level tasks such as chart understanding, they cannot use the skills that humans use to solve those tasks (L33-39)\n\n\n## Object Re-ID uses the term “object” in an abstract sense that is not obvious\n\nThe full prompt specifies an object is made of geometric shapes; the reviewer is referring to the prompt in Fig. 1, but this prompt was truncated to simplify the illustration. In Fig. 1,2 and 3 in the supplement, we show the full prompt: \n“The first image shows an object made of connected geometric shapes, which together form an object. Does this SAME object appear in the second image?...” \n\n\n\n## The 2nd example in Fig. 1 states that the correct answer is YES because the object was scaled and rotated when scaling might not be considered a rigid transformation.\n\nThe full prompt (supplement, Figure 2) specifies that scaling counts as a transformation. Additionally, only the Unconnected variant uses the term “rigid transformation\". The full sentence is as follows: “Does this SAME object appear in the second image up to a rigid translation, rotation, and scale of the ENTIRE object as a whole?” \n\n## In the third task, a phrase like “starting at the teal triangle” might be interpreted either to mean the given geometric shape and color, or the text stating “teal triangle”. \n\nThis prompt is unambiguous because it is evaluated with a solved example in its context. On its own, this phrasing is ambiguous, but we also provided the model with a different solved example of VSH, fully annotated with the correct chain (“We start at the red triangle, then go to the blue triangle…”). This chain only matches the image if you start with the shape and then use its text to keep going.\n\nWe ran an additional experiment on VSH that shows that the high performing models understand the prompt but cannot perform saccadic search. We modified the chain length 3 trial to go one jump at a time per model response. We used the model’s answer for the prior step as the starting point for the next request. \n\nGemini 2.5: 88%\n\no4-mini: 90.6%\n\nQwen 2.5 7B: 14.7%\n\nThis much-higher success rate from SOTA VLMs indicates they know to start on the shape.\n\nThis experiment also allows us to capture the per-step error rate of each model. Gemini-2.5 has a final-step failure rate of just 8%, and o4-mini’s is just 6%. Yet, assuming independent errors and using Fig. 4’s overall accuracies (68.5% and 44.5%), we’d predict per-step error rates of 12% (1−0.685¹ᐟ³) and 24%. The discrepancy shows that saccadic search, not individual step comprehension, drives the performance drop.\n\n\n\n## Maybe VLMs struggle because they are not good at interpreting confusing prompts \n\nOur experiment in the teal triangle response above and prior research suggest that it is unlikely the VLMs struggle with our tasks because they don’t understand the prompts. Research about LLM’s instruction following ability is mixed, and some studies show instruction tuned models exceed human performance on ambiguous instructions (AmbiBench, Are Language Models Worse than Humans at Following Prompts? It's Complicated). \n\nAdditionally, understanding the few-shot examples also requires the skills that we test. (L191) In the reviewer’s specific example—whether to start with the teal triangle’s text or its shape—the model could resolve that ambiguity by performing a saccadic search over the solved example in its context. \n\nOur experiment in the section above further strengthens our claim that the models understand the individual steps but cannot search. The models are more accurate when given the same prompt but for a single step— it is search that is difficult.\n\n## The text in the image prompts is small and hard to read, and this could hurt performance \nWe run new experiments and show that text size does not lower performance on Circuit Connections(CC) and does lower performance on Visual Scavenger Hunt(VSH). However, our experiments show that models struggle more with saccadic search than text-reading.\n\nWe generated an additional 75 examples of the Standard Variant of CC and tripled the text size. Here are the results:\n\nGemini 2.5 Pro: 32%\n\nClaude 3.7 Sonnet: 32%\n\nGemma 27B: 17.33%\n\nQwen 7b: 21.33%\n\nNo model performs significantly better than in Fig. 6, suggesting that text size does not hurt performance.\n\nWe also conducted two experiments for VSH, Chain Size 3. In the first one, we doubled the text size. In the second, we replaced the text labels with mini-shapes and modified the prompt to be as follows:\n\n \"This is a grid of LARGE shapes (outlined in black), each containing a small mini shape (outlined in white) inside of it. You need to follow a chain by looking at the mini shapes inside the large shapes… For example: If you start at a large red square that contains a small blue circle inside… The answer would be {yellow}. Always move to the LARGE shape that matches the mini shape you just saw. Starting from the large purple triangle, follow the chain for 3 steps… Answer with the color in curly braces, e.g. {red}.\"\n\nThe results for the large-text trial are as follows:\n\nGemma 3 27B: 17.4%\n\nQwen 7b: 9.33%\n\nGemini 2.5 Pro- 77.3%\n\no4-mini-high: 81.33%\n\nAnd for the no-text version, they are as follows:\nQwen 7b: 18.67%\n\nGemma 3 27b: 5.67%\n\nClaude 3.7: 29.33% \n\nGemini-2.5 Pro: 56%\n\nAs Gemini 2.5 Pro was 68.5% accurate and o4-mini was 44.5% accurate in the original trial (Fig. 4),  high‑performing models perform worse with smaller text size on VSH. In the Unclear Prompts analysis, o4‑mini achieves ~94% one‑step accuracy—so, under independent‐error assumptions (0.94³≈83%), three‑step performance should be ≈83%, but Fig. 4 shows only 44.5%. This shows that they struggle more with saccadic search than text reading.\n\n## “I found Fig. 1 and its caption to be a bit convoluted”\n\nWe will reorganize Fig. 1 to make the individual tasks clearer.\n## “These findings strongly argue for a shift in focus towards models that explicitly support structured and systematic visual reasoning...”(L279) is an overclaim. It is not clear that explicit structured reasoning is the solution. Maybe inference compute scaling methods could help\n\nWe will tone it down. We did not mean that explicit structured reasoning will necessarily be the solution. Other approaches could help too. We will revise accordingly.   \n\n\n## Do SOTA VLMs perform badly on these reasoning tasks over natural images?\n\nWe performed new experiments that show that these models also perform poorly on semi-natural images, which suggest that models may also struggle on fully natural images.  We modified VSH to use natural images from the COCO dataset and render labels on the objects which refer to other objects in the scene. We use image segmentation data to find the pixels that belong unambiguously to one object and render text labels at those locations.  We evaluate several models on this in a two-shot setting.\n\nClaude 3.7: 33%\n\nGemini 2.5: 38.67%\n\nQwen 2.5 7b: 26.67%\n\nQwen 2.5 32b: 40.00%\n\nGemma 3 27b: 40%\n\nOn COCO images with ∼4 labels per image, the VLMs achieve only 26–40% accuracy, barely above the ~25% they would get by randomly picking a label. These results show that the failure persists even in semi-natural images. \n\n## Are the reasoning types inspired by psychology?\n\nYes, our reasoning types are loosely inspired by psychological research, and we will update  the paper to include references. We now cite Ullman’s (1984) visual routines framework—whose “scan” and “trace” routines mirror our saccadic search task—and Treisman’s (1980) Feature Integration Theory to explain VLMs’ failures in object‐feature comparison (L209–214).\n\n## Does object re-id connect to existing work in Image Difference Captioning(IDC)?\n\nObject Re-ID evaluates a sub-task of IDC. Additionally, while IDC aims to describe discrepancies, evaluating if a description is correct is difficult. Evaluating binary correctness instead of a caption makes our evaluation precise.\n\n\n## What is the difference between Circuit Connections and BlindTest’s Subway Connections?\n\nSubway Connections can be solved without smooth search, whereas our benchmark requires smooth search and so can be used to test it. The Subway Connections task can be solved by looking at A and B, seeing which colors leave each of them, and the number of connections is just the intersection between those two. In our Single Colors task, this is impossible because all wires are the same color, so the only consistent way to solve it is to actually trace the line. Only our Unique‐Colors control does the task resemble Subway Connections, and we use this as a control to examine which factors cause success and failure in each setup."}]

Author rebuttal:

Meta-review: The paper received four expert reviews. The authors provided a rebuttal that attempted to address the concerns raised in the reviews. The reviewers read the rebuttal and engaged with the authors.  The reviewers unanimously like the paper and recommended accept. The area chair agreed with the recommendation and decided to accept the paper. Congratulations! Please see the reviews for feedback on the paper to revise the final version of your paper and include any items promised in your rebuttal.
Decision: Accept (spotlight)