Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiation
Authors: Arina Kharlamova
Bowei He
Chen Ma
Xue Liu
Venue: ICLR 2026 Poster (2026)
Keywords: CAPTCHA
multimodal models
spatial reasoning
robustness
evaluation benchmark
Primary area: datasets and benchmarks
TL;DR: Spatial CAPTCHA demonstrates that spatial reasoning remains a decisive weakness for multimodal models, enabling secure and scalable human verification.

Abstract: Online services rely on CAPTCHAs as a first line of defense against automated abuse, yet recent advances in multi-modal large language models (MLLMs) have eroded the effectiveness of conventional designs that focus on text recognition or 2D image understanding. To address this challenge, we present **Spatial CAPTCHA**, a novel human-verification framework that leverages fundamental differences in spatial reasoning between humans and MLLMs. Unlike existing CAPTCHAs that rely on low-level perception tasks vulnerable to modern AI, Spatial CAPTCHA generates dynamic questions requiring geometric reasoning, perspective-taking, occlusion handling, and mental rotation—skills intuitive for humans but difficult for current AI systems. The system employs a procedural generation pipeline with constraint-based difficulty control, automated correctness verification, and human-in-the-loop validation to ensure scalability, robustness, and adaptability. Evaluation on a corresponding benchmark, **Spatial-CAPTCHA-Bench**, demonstrates that humans vastly outperform 10 state-of-the-art MLLMs, with the best model achieving only 31.0\% Pass@1 accuracy. Result comparison with Google reCAPTCHA further confirms the effectiveness of Spatial CAPTCHA as both a security mechanism and a diagnostic tool for spatial reasoning in AI.

Published as a conference paper ICLR 2026
SPATIAL CAPTCHA: GENERATIVELY BENCHMARKING SPATIAL REASONING FOR HUMAN-MACHINE DIFFERENTI- ATION
Arina Kharlamova" Bowei H1*. Chen Ma", Xue Liu MBZUAI, McGill University, City University of Hong Kong [Arina Waiian.aaalamman Kharlamova, Bowei. He) @mbzuai. awwuaiaac.a.. ac.ae
ABSTRACT
Online services rely on CAPTCHAs first line of defense against automated abuse, yet recent advances in multi-modal large language models (MLLMs) have eroded the effec- tiveness of conventional designs that focus on text recognition or 2D image understanding. To address this challenge, we present Spatial CAPTCHA, a novel human- -eraanvveifiicttin framework that leverages fundamental differences in spatial reasoning between humans and MILMs. Unlike existing CAPTCHAs which rely on low-level perception tasks that are vul- nerable to modern AI, Spatial CAPTCHA generates dynamic questions requiring geometric reasoning, perspective-taking, occlusion handling, and mental rotation. These skills are intuitive for humans but difficult for state-of-the-eee (SOTA) AI systems. The system em- ploys procedural generation pipeline with constraint- based difficulty control, automated correctness verification, and human-in ness, and adaptability. Evaluation in-the-loop validation to ensure scalability, robust- corresponding benchmark, Spatial-CAPTCHA. Bench, demonstrates that humans vastly outperform 10 state-of-the-art MIL.Ms, with the best model achieving only 31.0% Pass@ I accuracy. Furthermore, we compare Spatial CAPTCHA with Google reCAPTCHA, which confirms its effectiveness as both security mechanism and diagnostic tool for spatial reasoning in AI.
1 INTRODUCTION
Modern web services face persistent threats from automated abuse, including credential stuffing, content scraping, and spam. To mitigate these risks, CAPTCHAS (Completely Automated Public Turing tests to tell Computers and Humans Apart) pose challenge-response response tests that are easy for humans yet hard for machines, serving as a practical, first-line defense at Internet scale (Von Ahn et al., (003). fact, CAPTCHA technologies have been widely commercialized and deployed across all major web platforms like Google and Facebook, e-commerce services, and security infrastructures (Kumar et al. 2022). Unlike general- purpose evaluation suites such Human's 's Last Exam (Phan et al. (025) and ZeroBench (Roberts et al.) 2025). CAPTCHAS must be automatically and continuously generated, remain unpredictable, and preserve a difficulty gap in the wild.
In the past decades, CAPTCHA mechanisms have evolved from lest- and image-based variants to more sophisticated protocols, including Google reCAPTCHA (BuiltWith) 2024a b). Difi-CAPTCHA (Jiang et al. 2023a), and VideoCAPTCHA (Gurale et al. 2025). However, the rapid progress of advanced machine
"Equal contribution

---

Published as a conference paper ICLR 2026
espeiiaa multi-modal large language models (MLLMs) such as GPT-4o (OpenAI
a). and
tool-using agents (OpenAI, 2025)-hase enabled computers to surpass the human capabilities in many areas, making existing CAPTCHAs not reliable anymore. Especially, CAPTCHA systems that primarily test superficial pattern recognition (e.g., object detection) are increasingly vulnerable (Deng al., 2024); even adversarial hardening can only yields transient robustness with limited generalization (Hitaj et al. 2020).
However, in spite of achieving the great success on many language and 2D perception tasks, such MI.I gents still exhibit significant limitations on spatial understanding and reasoning, largely due to the scarcity of related training data and the constraints of current visual encoder designs (Wu et al. 2025a Xu et al. 2025b). In contrast, humans possess innate 3D perceptual and spatial-reasoning capacities, which arise from genetic predispositions and are further refined by postnatal sensory- motor experience and culturaa/environmemni learning (Mallot & Basten 2009). In other words, humans inherently have an internal spatial model in their minds and thus construct the 3D scenario with only single-perspective image (Land 2014). This motivates us to utilize this characteristic to distinguish human and machines.
To achieve this goal, we first categorize and design seven types of tasks to evaluate spatial capabilities which are easy for humans but challenging for AI (MLLMs), Then, we develop an autonomous pipeline. Spatial CAPTCHA, which can generate unlimited questions corresponding each task, suitable for real-world online service In particular, we have integrated mechanisms including constraint-based difficulty control, automated correctness verification, and human-in comman-in-the-lowe -nothe-lopp validation to ensure scalability, robustness, and adaptability. Further, we collect certain number of generated instances from each task to obtain a benchmark, Spatia--CAPTCHA-BBB
thus evaluating the performance of different testers an offline manner. We evaluate the human and machine performance on this benchmark and also questions from representative CAPTCHAs including Google reCAPTCHA. Experiment results demonstrate advanced MLLM's scores on our benchmark are much lower than those on Google reCAPTCHA, especially for most advanced models (e.g.. 29.0 vs 55.3 for Gemini-2.5-Pro)) Meanwhile, the human scores can keep consistently over 90 similar to other CAPTCHAS, which indicates spatial CAPTCHA can effectively differentiate human and machines.
2 RELATED WORKS
Bot attacks, automated scripts agents that mimic human interactions, abuse online services through content theft, inventory scalping, payment fraud, account takeover, or infrastructure overload, posing serious security and economic risks (Dunham & Melnick 2008 Kumar et al.) 2022). They cause direct financial loss and erode customer trust, with industry reports confirming their prevalence and costliness (Imperva 2025). Recent attacks
Bot Attacks and Defense:
verifictiiiii systems evolve along two axes (Plesner jet al.] 2024): (i) powerful vision and vision-language models (Liu et al. 2023 OpenAT a) generalize across CAPTCHA types and defeat unseen challenges (Teoh et al. 2025): ); (ii) behavioral simulation via
generated mouse, touch, or timing patterns enables bypassing detectors (Liu et al.) 2024). As a result, adversaries now combine solvers (Motoyama et al. 2010 Ye al. 2018]. behavior emulation, and adaptive strategies (Deng et al. 2024), motivating CAPTCHAs that probe reasoning modalities where humans still retain advantage (Hitaj et al. 2020). Therefore, we introduce Spatial CAPTCHAs that require relational and spatial reasoning, which is robust against current multimodal solvers and behavioral mimics.
MLLMs and Agents: Recent years have seen rapid progress in multimodal large language models (MLLMs), spanning open-source efforts (e.g., the BLIP family (Li et al. 2022 2023: Dai et al. 2023). LLaVA series (Liu et al. 2023 Lin et al. 22024), and Qwen- VL (Wang et al. 2024b; Bai et al. (0255 and proprietary systems
(c.g.. Claude 4 (Anthropic bb. Gemini 2.5 (Google DeepMind b). GPT-4o (OpenAl a). and GPT-4o
mini OOpenAl b).. Their canonical modular besign-wherre a vision encoder extracts features that are aligned via projection layer before being fed into
seamless multimodal understanding
and generation. Powered by large-scale pretraining, MLLMS excel on tasks such as visual question answering, OCR, and reasoning over diagrams videos (Liu al. 2023). and they underpin practical agents like GUI
2

---

Published as a conference paper ICLR 2026
agents (Wang et al. 2025; OpenAl 2025) that interpret screen content to exccute actions. Despite these advances, current models remain limited in spatial reasoning (Xu et al. 2025b): unlike humans, who can infer 3D structures and dynamies from partial observations, MLLMs often fail on tasks requiring geometric consistency, physical intuition, or embodied perspective-taking This gap motivates CAPTCHAs that exploit machine weaknesses in spatial reasoning.
3 THEORETICAL BASIS OF SPATIAL CAPTCHA
The Spatial CAPTCHA paradigm is grounded in well-studied human cognitive abilities rather than arbitrary puzzle design. Human spatial cogmition is characterized by several fundamental abilities (Porat & Ccobanu
2024 Freksa et al. 2017), including (I) spatial perception and reference system, (II) spatial orientation and
perspective-taking. (III) mental objects rotation and (IV) spatial visualization involving multiple transfor-
mations. These categories have been identified in psychometric taxonomies (Bar-Hen-Schweiger & Henik
2024; Carroll| 1993 Knaufi 2006) and operationalized in classic instruments (Shepard & Metzler I971
Hegarty & Waller 2004, Duffy et al. 2024).
Spatial CAPTCHA formalizes each spatial ability distinct task category, where the solution is anchored a mathematically well-defined invariant. These invariants include but not limited to topological relations
coordinate transformations (Stevens et al. 2012 Cohn & Renz 2008
Egenhofer & Franzosa 1991),
rotational equivalence in two and three dimensions (Shepard & Metzler 1971
1971,Cohen
Cohen et al. 2018 Cohen &
Welling 2016), and the composition of Euclidean motions (Murray al. 1994 Lynch & Park [017]] Blanco
2010). Their concrete parameterization including covering input variables, rendering constraints, and answer
definition specified in task manifests, described in detail in $4.2 Concretely, instance produced by sampling from a parametric family ~ G(O) with 9 ~ Pe, where the associated query f(z) explicitly targets the intended invariant. This design, combined with constraint- based modulation of visual cues, ensures that
task success depends on genuine spatial reasoning rather than incidental lexical patterns surface textures.
Our contribution is a theory-first framework that maps cognitive constructs onto verifiable invariants, yielding a compositional ensemble of task classes. For completeness, Appendix A presents all four ability categories with representative task instances.
4
SYSTEM FRAMEWORK OF SPATIAL CAPTCHA
4.1 INVARIANT-SPECIHI TASK MANIFESTS AND GROUND- -RUTH CERTIFICATION
Before introducing the detailed procedural mechanics of generation and rendering, we first begin from the declarative level by formally specifying the structure of inverient-specified task manifests and the certification rules that guarantee their validity as ground truth: namely, what can be generated and how it is certified.
What a manifest is (concrete representation) A task manifest is a checkkbbee specification that binds cognitive invariant to a family of renderable items with controlled variability and difficulty. In practice, manifests are canonical JSON objects validated against a JSON Schema; the schema, validators, and CLI tooling which are described in $4.2 Formally, manifest is tuple
M (id, I, Po), T, g. r, V, R),
For clarity and reproducibility, now state the role and type of every field in M: (1) id E ID is the manifest identifier (name, type, version) ensures provenance and reproducibility. (II) I E Inv is the targeted invariant which captures the semantics of the class of tasks and everything else merely serves this check (e.g., left/right allocentricity, rotational congruence, topological adjacency). (III) (0, Po), where @ = (0,), Po E A(O) the concrete parameterization of content variables with sampling prior Pe (counts, angles, poses, occlusions,
3

---

Published as a conference paper ICLR 2026
candidate set size, distractor types) that defines the input space, where each parameter has well-typed domain (ranges, enumerations, or stochastic permutations). (IV) T θ that instantiates the question, candidate set, and correct answer, binding scene semantics (V) g θ + S is the scene function, pseudo-random generator that from @ constructs a candidate world model, produces derived outputs (e.g., answer key), and encodes geometry a coordinate-based structure. (VI) r = (T false: slossss) adds distractor mechanisms, producing false answers or slot fillers from the scene so that multi-option tasks remain nontrivial. (VII) V S + {0, 1 is the validator suite, rejecting invalid scenes (c.g.. intersecting objects, insufficient margins, lack of uniqueness) and ensuring well-posedness. (VIII) R. θ x S + X is the renderer, projecting the validated scene into images or panels with fixed style settings.
5 - (Σ", Ans, a*) is the task function
solvable problem.
Minimal guarantees All components operate in geometric space: constructs scenes by rigid motions (placing objects under explicit spatial relations), produces near-miss candidates via controlled spatial perturbations, and V verifies invariants (non-intersection, adjacency, and separation margins encoded by I').
Consequently:
soundness: the label y is computed from the scene S and is independent of rendering;
uniqueness under margins: if ((S) = 1, exactly candidate in the set returned by TT satisfies the predicate family tied to I;
validity and human legibility: visibility/contrast and margin checks reject ambiguous or visually marginal items; and
spatial necessity: success requires the intended spatial reasoning, since distractors differ only in prohibited relations while superficial appearance alone cannot satisfy the certified invariants.
Difficulty and variability by design Difficulty is controlled at the level of the manifest rather than left rendering accidents. The content space é exposes interpretable knobs listed in details in Appendix [B We define monotone difficulty map
d(0) = w T()),
with features @(0) drawn from these knobs; w is fitted to pilot human response times via isotonic/quantile regression. Sampling uses stratified priors Pe over target bins (easy/medium/hard) with rejection against V to ensure admissibility. This keeps items human-simple (defined in $6.T): ambiguity is excluded by separation margins, legibility guards (visibility/contras)), and symmetry screens, while reasoning load is set by d(0), not by clutter or texture. The detailed procedure described in Appendix B.2
4.2 INSTANCE SYNTIEESS PIPELINE
The instance synthesis pipeline turns high-level manifest specification M into deliverable items with consistent difficulty control and auditability. We factor the pipeline into three macro-stages: (I) Scene Metadata Random Generation, (II) Procedural Generation, and (III) Task Generation, across which the internal steps are distributed: Sampling occurs in Stage 1; Scene Construction, Distractor Synthesis, and Validation occur in Stage 2; Rendering, HoomptanddAAssee Construction, and Assembly occur in Stage 3. This structure isolates responsibilities, lets task families evolve without cross-coupling, and preserves reproducibility across engines and environments.
Operational Setup Inputs: a valid manifest reference; a random seed; access to a scene generator, a distractor mechanism, validator suite, and renderer. Outputs: packaged instance containing rendered panels, prompt, an answer set with one correct choice, and metadata sufficient for re-execution.
Scene Metadata Random Generation Input variables 0 Pe are drawn according the manifest. These knobs encode the semantic degrees of freedom of the task (e.g., counts, layout, base geometry) while stratified priors control difficulty bins.
4

---

Published as conference paper ICLR 2026
4.2.1 PROCEDURAL GENERATION
(1) Scene generation The sampled input 0 is passed to the scene function G, which constructs a candidate world model S' in geometric space. This stage employs con- strained procedural generation: input variables define the admissible complexity of the scene and, later in render- ing, its visual appearance, while validity functions impose additional constraints that guarantee readability and dis- tinguishability between correct and distractor answers.
Tiok Candidate Monifest
Scene Metadata
Objects databust
Procedural generation
(2) Distractor synthesis Given S, distractor mecha- nisms I' [T'tslse: generate near-miss alternatives. These are constrained perturbations of the base scene that yield plausible but incorrect answer candidates (e.g.. wrong viewpoint, mismatched rotation, or inconsistent projection).
Objects' relations
80
Task generation
m
(3) Validation The validator suite V certifies the instance. Validators reject degenerate or ambiguous cases by enforc- ing invariants such as non-intersection, sufficient angular or depth margins, uniqueness of the correct answer, and visibility/contrast checks. Only scenes with V(S) are admitted.
Figure End-to-end synthesis pipeline. Input variables are sampled from the manifest (formal- ized $4.1).
4.2.2 TASK GENERATION
(1) Rendering The validated scene is mapped to images via x -> X, producing one or more rendered panels. Rendering is label-inert: it affects visual style but not the computed answer. In practice, this stage may call external engines such as Blender for high-fidelity 3D output or VTK for lightweight geometric visualization, but the pipeline itself remains agnostic to the rendering backend.
(2) Prompt and answer construction. Input variables and outputs of G are bound into a task template T, producing the natural-language prompt, the candidate set, and the correctness marker. Distractor variants generated in step (3) populate the answer slots.
(3) Assembly All components (such rendered images, task prompt, answer variants, and correctness label) are packaged into a single CAPTCHA instance. Each instance both service-ready (deliverable to end users) and dataset-ready (loggable for evaluation).
This design enforces two layers of constraints. First, input variables & control task complexity and visual load through interpretable knobs. Second, V enforces admissibility constraints to guarantee legibility, uniqueness, and spatial necessity. Because the pipeline defined in terms of declarative manifests, it remains agnostic specific rendering engines or scene implementations. This abstraction enables extensibility across task families and supports runtime instance generation personalized to user history and trust scores, without compromising the certification guarantees.
5 SPATIAL-CAPTCHA--BEN DATASET
Spatial-CAPTeS CBaaial-CAPTCCH-BBeeeee is the first benchmark instantiated from the Spatial-CAPTCHA framework. It comprises K-4 spatial-ability categories (reference systems; riiemnatoonperspectiveeaiigg mental rotation; multi-step spatial visualization), each stratified into D=3 difficulty bins (easy/medium/harl), Across T task
5

---

Published as conference paper ICLR 2026
Table 1: Comparison with CAPTCHA suites and spatial reasoning datasets. Columns: Modality/Eval (dominant imput signal and scoring protocol), Procedural Gen (programmatic instance synthesis), Offline Supervision (public static (input.target) pairs suitable for supervised fine-tuning), Complexity Metric (explicit difficulty/rbuutness measure), Human pass (%), Best MLL.M/Agent (%; strongest pass @ reported by the source under its primary protocol), and Open Date/Code. Percentages are absolute: denotes not reported.
Modality EEal
Procedural
Offline
Complexity Metric
Human pass (%)
Best MLLM/ Agent (%)
Open
Benchmarkk System
Gen
Supervision
Data/Code
CAPTCHA Swites
Tuo et al
2025
Image+Text/ / Agentic
K
93.3
40.0
Wa 2025h
Image+Text Text / Offline
X
X
98.0
99.5
Ding 2025
Image Offline
86.95
Jiang
Imuge/ Ollline
Chandra al
2025
Audion Video Oflline
X 92.8
5.2
Spotial datasets
Ma 2025
Image+Tore T / Offline
/
95.7
52.0
Wang 2024g
Image+Toxt Text / Offline
X
67.1
Du al.
2024
Image+Text Text / Offline
90.3
49.1
Stogianridis.o
Image+ Offline
Comsa & Narayanan
2023
Text Offline
X
93.5
88.3
Rodionev et al 2025
Text Offline
x
85.0
Spotid-CAPPCHA (Owrs)
Spataa-CCPPCCHA-BBL Bench
Image+Text Text / Offlline
99.8
31.0
formulations (currently 7-7; extensible; e.g., Unfolded, Sun Direction, Revolution, Pyramid, Polyomino, Full Views), the dataset contains Nin== 1050 instances with per-formulation counts (150,-. 150) and per- bin counts ENE, NM, = 300, 250). Per-category counts are (N1..... NN) with Σi= 1,500 with both CAPTCHA suites and spatial reasoning datasets. Beyond comparative positioning, the scale and coverage of Mpatial-AAPTCHA--II research directions. The dynamic extensibility of the dataset also enables forward- wword-- experimentation: researchers introduce new spatial invariants, difficulty progressions, and distractor families without breaking comparability.
Table [ contrasts Patiia-CAPTCHA-
open qualitatively new
6 EXPERIMENTS
6.1 EXPERIMENTAL SETUP
We evaluate model and human performance on Caaelal--AACCHH--BBeeC additionally construct Spatia-CAPTCHA--Beno (Tiny) subset of 70 items, stratified by task category and difficulty level. We also conduct experiments on reCAPTCHA-Bench. dataset with 1.50 samples collected from Google reCAPTCHA service (Plesner et al. 2024 BuiltWith 2024b).
$ For human evaluation, we
Models Evaluated We assess a diverse pool of state-of-the-art Large Language and Vision-Language Models, spanning both proprietary (e.g.. GPT-4o (OpenAl a). Claude Sonnet (Anthropic b), Gemini 2.5 Pro (Google DeepMind, b)) and open-source architectures (e.g.. Llama, Mistral). The complete list is provided in Table All models are evaluated zero-shot without fine-tuning chain-of chain-of-thought augmentation.
Evaluation Metrics We adopt multi-faceted evaluation protocol designed to capture both task-level
correctness and cognitively grounded failure patterns. Evaluation metrics grouped by intent into accuracy, human upper bound, and spelitt-specific diagnostics. A detailed Appendix [ summarises the full set of metrics used throughout this study, along with their scope and interpretive roles. Specifically, & is set as 3.
6

---

Published as conference paper ICLR 2026
Table 2: Results on Spatial-CAPTCHA Bench and OCCAPTCHA-Bema Bench. The left 3 columns reports aggregate metrics; the middle 4 columns reports pass@ by specific abilities: spatial perception (reference systems), spatial orientation (perspective-taking) mental object rotation, and multi-step spatial visualization. They are abbreviated as SP, SO, MOR, and SV in the table, respectively. The right 2 columns reports pass @ and pass @+ on reCAPTCHA- Higher score indicates better performance.
SpatialCAAPTCHA-BBLL
reCAPTCHA-Bench
Overall Metries
Per-Ability Pass
Overall Metrics
Methods
Rank
pess
6f
k-
SO
MOR
SV
pass
pass@ k
Baseline
Chance level (Random)
21.4
51.1
1.1
16.7
25.0
16.7
25.0
0.2
0.6
Human Level (Simple)
89.5
95.6
89.6
83.3
86.4
Proprictary Models
gpt-5.1
36.1 1
61.6
18.2
68.3
35.6
42.2
27.5
gemini-3-yp i-pro-imagee-prviiiiii
40.3
32.1
04-mini
3
31.0
56.0
10.3
60.0
35.7
31.6
25.3
36.7
$4.0
gemini-2.5-pro
29.0
48.4
31.7
30.7
23.7
55.3
58.7
chalggt-4o-latest
26.1
38.0
17.7
44.0
23.3
27.1
27.3
52.7
57.3
gemini-2.5-flas
21.6
44.6
6.0
16.7
25.0
16.7
25.7
31.3
40.0
claude-sonnet-4
10
21.4
30.8
11.0
24.0
21.7
18.0
26.3
10.7
claude-opus-4
12
7.1
13.0
2.1
4.7
5.3
2.0
16.7
6.0
7.3
Open-weight Models
qwen2.5-vl
24.0
31.0
16.2
34.7
23.0
21.6
28.7
4.0
6.0
multimoall instrruct
22.7
32.9
11.4
19.3
27.7
20.2
21.3
2.7
2.7
mmavei-maverick
29.9
12.7
28.7
14.7
24.7
2.7
3.3
mistral-medium-
11
20.2
43.5
4.7
14.0
27.7
13.6
22.7
6.7
12.0
Evaluation Process. Each model is evaluated independently per task instance. Prompts are held fixed across all runs and models; no instance-level tuming is permitted. Human annotators (N=60) were instructed to solve each Tiny instance fast possible, simulating the "human-sines simulee"smpe"" requirement. All codes and prompts are provided the Github repository to ensure reproducibility.
6.2 BASELINES
Chance-Level (Random) A trivial baseline selects uniformly random from the candidate answer set. This reflects a calibrated floor of performance for each task formulation. Due to class imbalance and distractor synthesis, random accuracy varies slightly across task types, but remains within 21.4% across the benchmark.
Human-Level (Simple) To approximate a soft upper bound, we report a Human-Simple Pass Rate on a 70-instance subset (Spatial-AAPTCHA--BBee
(Tiny)), annotated by N 60 human raters under 30-second time constraint per item. An item marked as "passed" if at least two annotators select the correct answer. This simulates low-friction human reasoning under minimal supervision.
6.3 EXPERIMENTAL RESULTS AND ANALYSIS
Comparison with reCAPTCHA-Bende From Table relative to reCAPTCHA-Bench it can be observed that the scores reCApTCHA-Bench much higher than that Spatial-CAPTCHAABBBII for advanced MILMs (e.g., 29.0 vs. 55.3 for Gemini-2 22.1ro.))))) As for human evaluation, the score Spatical-CAPPCCAA-Baea (tiny) is even little bit higher than that SCCATCHA--BenC This
7

---

Published as a conference paper ICLR 2026
demonstrates our Spatial CAPTCHA can indeed better differentiates human from machines by identifying larger human-model gap. This also proves the superiority and the potential for large-scale commercial use of our designed Spatial CAPTCHA.
Efficiency and accuracy are only weakly coupled. As shown in Figure 2c latency spans two orders of magnitude across systems, yet slower models are not more accurate: Gemini-2.55 P-roo exhibits the largest median response time (95.4s, IQR [17.6, 160.5) without a commensurate accuracy advantage, while Gemin n 5-F15- ash answers in near real time (1.8s, IQR [1.6. 2.2) with only modest losses. High-lateney models such phi- and 5-ve2..-v11770 -2b similarly fail to convert time into accuracy, suggesting inefficiency rather than deeper reasoning. Moreover, the variance profiles differ sharply: some models (e.g., Gemini- -2..mnn--2..rr fluctuate by over order of magnitude, whereas others (e.g., 04-min n Gemini-Flash) remain stable, indicating that latency is more diagnostic of system implementation and routing overhead than of spatial reasoning capability.
Task characteristics are the cause of systematic differences observed between humans and models. Radar plot Figure show that accuracy peaks on SUN DIRECTION and PYRAMID, where reconstruction based on sequential signals is sufficient, but collapses on UNFOLDED and AGENT SIGHT, which require enforcing adjacency constraints or integrating occluded multi-view geometry. As highlighted in Figure [a humans display near-reflex latencies on SUN DIRECTION (median 2.1s; IQR [1.3, 2.9), consistent with embodied heuristics (e.g., shadow-light light vector decoding), whereas models show no analogous latency drop, which is evidence of missing perceptual grounding. The UNFOLDED family is particularly diagnostic: models answer quickly yet fail often, a pattern consistent with template- based shortcuts that ignore global compatibility. At the level of cognitive class, performance is higher for spatial perception and reference-frame aligmment (27.5% = 16.6 pp) than for multi-step visualisation (24.2% + 5.7 pp). while human accuracy is comparatively stable across abilities (within 7 pp). This consistency, contrasted with the variability observed in models, implies that the system is more sensitive to the depth of transformations rather than the mere complexity of the imagery.
Calibration is uniformly poor. Figure Be illustrates k/k reliability against p ls coverage (with k=3) places every model beneath the identity line: confident sets under-represent the truth. GPT-o4 -mini, for in-
Human
Easy Bin
D Medium Bin
Models
Hard Bin
80
Overall
10a
Answer Time [s]
Score
40
Time (5)
3+0
109
10ab
40
20
109
Agout Sight
Full Views
Poly umina
Pyraiid
Revolution
Sun Direction
Uniolded
o4-mimi
Gemini Pro
4PT40
Qwen2 5.VV
Phu-4
Gemini- Rosh
04-mini
Gemini-Pro Pro
GPT40
Qwen Qwen2.5.VV V.
Phi-4
Gemini- Fladi
(a) Time-
Task Type Dist.
(b) Difficult-AAcc, Dist.
(c) Time- Model Dist.
Figure 2: Distributions of response times and accuracies across task types, difficulty levels, and models.
8

---

Published as conference paper ICLR 2026
Full Views
Pass@k vs, Reliabllity
Agent Sight
Well-calibrated
Reliability 2 Coverage
400
Polyomina
R-of-k Reliability
256
Count
200
126
115
Unfolded
Overconfident
70
187
173
High coverage Jow reliabiity
Pyramid
MOR
Sun Direction
0.2 0.4
Revolation
Pass Coverage
(a) Difficulty distribution within spa-
04 mini
gpt
(c) Pass@k vs. *Reliability
gemini-2.S-pro
4-v-225-v
tial ability categories
(b) Per-task performance profiles by model
Figure 3: Overview of task difficulty, model profiles, and reliability. Colours in (b.c) mark top-5 models from Table where shown results are also consistent with Figure 2
stance, achieves high coverage (56.0) but low reliability (10.3). typifying overconfidence; el claude is more conservative (coverage 13.0; reliability 2.1) yet still uninformative as none approach parity.
This consistent lack of calibration compounds the identified performance shorteomings, models not only overestimate their certainty in individual predictions but also struggle to retain accuracy when faced with escalating task complexity. Difficulty stratification (illustrated in Figure 3a) confirms that our bins capture real complexity gradients rather than noise. Performance curves in Figure 2b indicate that from Easy to Hard, models' pass@1 drops steeply (61.4% 48.7 pp to 12.4% + 33.0 pp: Cohen's h=1.08), while humans decline gradually (slope z 3.0 pp). The combination of steep model slope and shallow human slope implies that what is hard here
low-level vision but compositional constraint satisfaction: chaining local signals under global geometric rules. This aligns with the per-task anomalies above and with the observation that added latency seldom recovers correctness.
Taken together, Spatial CAPTCHA separates humans from MLLMs by diagnosing structural failures in invariant preservation, embodied perception, and calibration. This makes both an effective discriminator and a diagnostic lens into the unresolved challenge of uncertainty-awant constraint- spatial reasoning.
7 CONCLUSIONS AND FUTURE WORKS
In this work, we introduced Spatial CAPTCHA, generative framework for benchmarking and deploying spatial reasoning challenges as new form of human mucann differentiation. By systematically designing seven categories of tasks targeting spatial understanding and reasoning, we demonstrated that our pipeline can instances. Extensive evaluations revealed
continuously generate scalable, verifiable, and difficulty-control a persistent mamaannmachieee performance gap: while humans consistently achieved nearly 100% accuracy, state-of-the--rt multimodal LLMs exhibited significant performance drops, confirming the practicality of our approach. Moreover, the introduction of Spatial-CAPTCHA-BBL benchmark for standardized evaluation of both human and machine capabilities. In the future, plan to design GUI-interactive spatial reasoning challenges, requiring users to manipulate or align objects rather than simply provide answers, thereby enriching the human-machine differentiation space. Besides, extending the CAPTCHA to temporal-spatial challenges (e.g., reasoning across video sequences or dynamic object interactions) could further enhance robustness against automated solvers.
provides reproducible offline
9

---

Published as conference paper ICLR 2026
REPRODUCIBILITY STATEMENT
For implementation details, please refer Appendix The complete codebase and generation scripts quired to reproduce our study are available through the the/https:: thu.. wwwcmomoocoooccoooocooooooo DDlldruss// /tw.dd/ssssstiiio sppatial, captcha The repository includes benchmark construction tools, evaluation pipelines, and configura- tion files with fixed random seeds to ensure deterministic regeneration of all benchmark instances. De- tailed instructions are provided in the main text and appendices for environment setup, model evalu- ation with fixed zero-shot prompts, and difficulty calibration procedures. We also document the hu- man evaluation protocol, ensuring that both machine and human baselines can be reliably reproduced. We release both the full dataset and the Tiny subset used for human evaluation on Hugging Face at https: http:s://hhgiinnee ngw.cinggacc.... co/da date/atttstssss amo r odi Spatial -aaeiial-APPCA--aaaa -APTCHA--eench
ETHICS STATEMENT
This study involved human participants to evaluate Caatia--APPCHAA-BAR -Bench. Participation was entirely voluntary, and no compensation or incentives were tied to outcomes. No personal or identifying information was collected; participants could optionally provide arbitrary display names solely for leaderboard purposes. All responses were used only in aggregate analyses, and no individual-level data are reported. The study conducted in accordance with institutional ethical guidelines, with procedures designed to minimize any potential risks to participants. The tasks involved solving spatial reasoning challenges, which posed no foreseeable risks beyond those encountered in everyday computer All materials, instructions, and protocols are transparently documented to ensure responsible and reproducible human evaluation.
REFERENCES
Anthropic. Claude 4 Opus model card. httpps//www..antttroooo..o/o///p a. (Accessed: July 2025).
Anthropic. Claude Sonnet model card. http::/www..nthhopiccccco//c/lc
b.
(Accessed: July 2025).
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint a3Xiv:2502.13993, 2025.
M. Bar-Hen Schweier and A. Henik. Looking beyond seeing: Components of visual-spatial ability as an overarching process. Acta Psychologica, 251:104577, November 2024. doi: 20.10166j.ccpsyy224..444
hRR.httts:///do
orgg10 1111/1 wttpecttssy..2222...... 2024.104577 104577
José-Luis Blanco. tutorial on SE(3) transformation parameterizations and on-manifold optimization.
Technical report, University of Mälaga, 2010.
BuiltWith. recaptcha v2 usage statistics. reCAPTCHA v2. 2024a. (Accessed: July 2025).
https://trends.t www.bbbbuuiihh..ccc www.ithhcom/w..gett....
BuiltWith. recaptcha v3 usage statistics.
https://treeds... butltwitbbbilwiiii. com www.//idoeee widgets/
recAPTCHA-V3S v3. 2024b. (Accessed: July 2025).
Neil Burgess. Spatial memory: how egocentric and allocentric combine. Trends in Cognitive Sciences,
10(12):551-557, December 2006a. ISSN 1364-6613. doi: 10.s0066j.ices.206.11 2006.10.05. URL http:
7/dx wol. wtt.:oooooorgg1111111111.0 2006. 20.......0.0..0000000.000..1.0.0
10

---

Published as conference paper ICLR 2026
Neil Burgess, Spatial memory: how egocentric and allocentric combine. Trends Cogn. Sci., 10(12):551-557, December 2006b.
John B. Carroll. Human Cognitive Abilities: A Survey of Factor-Analytic Studies. Cambridge University Press, 1993.
Joydeep Chandra, Prabal Manhas, Ramanjot Kaur, and Rashi Sahay. Aura-captcha: A reinforcement learning and gan-enhanced multi-modal captcha system, 2025. URL htttp::/aaxxii www.aiiwwi.o....aa.oooooooooooot abss 2508. 14976
Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of the 33rd International Conference on Machine Learning, pp. 2990-2999, 2016.
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In International Conference on Learning Reprexentations, 2018.
Anthony G. Cohn and Jochen Renz. Reasoning about qualitative spatial relations. Handbook of Knowledge Representation, pp. 551-596, 2008. Preprint with statement on invariance available on arXiv.
Julia Comsa and Srini Narayanan. A benchmark for reasoning with spatial prepositions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 16328-16335, Singapore, December 2023, Association for Computational Linguistics. doi: 10. 101138611102222 www.:223.mmppmmii......m 1015. URL htttps://ccc/ /acl://///caaaohooooooo www.annooooooog////00000 emnlp www.mmiinnn.... 1015/
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250-49261 2023.
Gelei Deng, Haoran Ou, Yi Liu, Jie Zhang, Tianwei Zhang, and Yang Liu. Oedipus: LIm-enhanced reasoning captcha solver. arXiv preprint arXiv:2005.07996 2024.
Ziqi Ding, Gelei Deng, Yi Liu, Junchen Ding, Jieshan Chen, Yulei Sui, and Yuekang Li. Illusioncaptcha: captcha based visual illusion, 2025. URL http:ttt////aiv..oott oww.:s/.rggabb/bb 2502. 05461
Thanh- Do, Thanh- Toan Nguyen, and Ian Reid. Affinity and affordance: Learning multi-object functional relations for robotic manipulation. IEEE Transactions Robotics, 38(6)33753-3769, 2022.
Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial-bench: Benchmarking
spatial understanding for embodied tasks with large vision-language models, 2024. URL https://
arxiv. org. abs 2406. 05756
Gavin Duffy, Sheryl Sorby, and Brian Bowe. Exploring the role of spatial ability the mental representation of word problems in mathematics. Front. Educ., 9, June 2024.
Ken Dunham and Jim Melnick. Malicious bots: an inside look into the cyber-criminal underground of the internet. Auerbach Publications, 2008.
Max J. Egenhofer and Robert D. Franzosa. Point-set topological spatial relations. International Journal of Geographical Information Systems, 5(2):161-174, 1991.
Christian Freksa, Ana-Maria Oltefeanu, Thomas Barkowsky, Jasper van de Ven, and Holger Schultheis. Spatial problem solving in spatial structures. In Lecture Notes in Computer Science, Lecture notes in computer science, pp. 18-29. Springer International Publishing, Cham, 2017.
11

---

Published as a conference paper ICLR 2026
Google DeepMind. Gemini 2.5 Flash. httpss////eepilllooooo/o//o/////
a.
(Accessed: July 2025).
Google DeepMind. Gemini 2.5 Pro. https://aeeenidd.ooo//////////// b. (Ac- cessed: July 2025).
Madhuri Gurale, Komal Borate, Rachana R. Sangitrao, and Garima Tiwari. Enhanced security through honeywords and video captcha: An innovative authentication and tracking mechanism. In 2025 Fifth Inter- national Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT), pp. 1-6. IEEE, 2025.
Mary Hegarty and David Waller. A dissociation between mental rotation and perspective-taking spatial abilities. Intelligence, 12(2)1175-191, 2004, ISSN 0160-2896. doi: (0.10161 URL https: www www.::ssieceeedieeeeeeee. com httpa///c..c.ooooooooooooooooaaa
Dorjan Hitaj, Briland Hitaj, Sushil Jajodia, and Luigi V Mancini. Capture the bot: Using advensarial examples to improve captcha robustness to bot attacks. IEEE Intelligent Systems, 36(5):104 112, 2020.
Imperva. 2025 Imperva Bad Bot Report: The Rapid Rise Bots and the Unseen Risk for Business. 12th annual imperva bad bot report, Imperva, Inc., March 2025. wwwpe//www....ooo...aooo httpp//c.c.ssssseeffeaa///eeeees httpe//aatleaaeee///tttt//////// cowtent/campaaaieeieo wttp://tttppppuotoooooooeeeeoeee sot.recccc.......cccc........... coml Accessed: 2025-09-20.
URL
Laurent Itti and Christof Koch. Computational modelling of visual attention. Nature Reviews Neuroscience, 2:194-203, 203, 2001.
Ran Jiang, Sanfeng Zhang, Linfeng Liu, and Yanbing Peng. Diff-captcha: An image-based captcha with security enhanced by denoising diffusion model. arXiv preprint arXiv:2308.08367, 2023a.
Ran Jiang, Sanfeng Zhang, Linfeng Liu, and Yanbing Peng. Diff-captcha: An image- captcha with security enhanced by denoising diffusion model, 2023b. URL
08367
Markus Knauff. A neuro-cognitive theory of relational reasoning with mental models and visual images. In Mental Models and the Mind Current Developments in Cognitive Psychology, Neuroscience, and Philosophy of Mind, Advances in psychology, pp. 127-152. Elsevier, 2006.
Mohinder Kumar, M. K. Jindal, and Munish Kumar. systematic survey on captcha recognition: types, creation and breaking techniques. Archives of Computational Methods in Engineering, 29(2):1107-1133) 2022.
Michael F Land. Do we have an internal model of the outside world? Philosophical Transactions of the Royal Society B: Biological Sciences, (61(1666)220133
2014.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. International conference on machine learning, pp. 12888-1888 12900. PMLR, 2022.
Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19744 PMLR, 2023.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 59715597559 1-944,, 2024.
12

---

Published as conference paper ICLR 2026
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3499 -3966, 2023.
Jiahua Liu, Zeyuan Cui, Wenhan Ge, and Pengxiang Zhan. Dmtg: A human-like mouse trajectory generation bot based on entropy-contrrolld diffusion networks. arXiv preprint arXiv:2410.18233, 2024.
Yaxin Luo, Zhaoyi Li, Jiacheng Liu, Jiacheng Cui, Xiaohan Zhao, and Zhiqiang Shen. Open captchaworld: comprehensive web-based platform for testing and benchmarking multimodal IIm agents, 2025. URL
Kevin M. Lynch and Frank C. Park. Modern Robotics: Mechanics, Planning, and Control. Cambridge University Press, 2017.
Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Jieneng Chen, Celso M de Melo, and Alan Yuille. 3dsrbench: A comprehensive 3d spatial reasoning benchmark, 2025. URL ltttss://axiv..ort
abss2412.072
Hanspeter A. Mallot and Kai Basten. Embodied spatial cognition: Biological and artificial systems. Image and Vision Computing, 27(11):165-1177, 2009.
Marti Motoyama, Kirill Levchenko, Chris Kanich, Damon McCoy, Geoffrey M Voelker, and Stefan Savage. [CAPTCHA-Solving] services in an economic context. In 19th USENIX Security Symposium (USENIX Security 10), 2010.
Richard M. Murray, Zexiang Li, and S. Shankar Sastry. A Mathematical Introduction to Robotic Manipulation. CRC Press, 1994.
Austin Myers, Ching L. Teo, Cornelia Fermüller, and Yiannis Aloimonos. Affordance detection in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1659-1666, 2015.
OpenAl. GPT-4o: OpenAl's New Multimodal Flagship Model.
https: https://openaa.comm
hello-gpt got--ppttoooooo (Accessed: July 2025).
OpenAL 04 Mini. http:////sa....ccoooo/////t//ttt
b. (Ac-
cessed: July 2025).
OpenAl. Computer-Using Agent. Release on OpenAI website, January 2025. URL /RRpsttps:///paai
com/ ht/p//ndexxx/// http://cppuuttuuseeee usig -ngenttngng///////a Accessed: 2025-09-20.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, et al. Humanity's last exam. arXiv preprint arXiv:2501. 14249 2025.
Andreas Plesner, Tobias Vontobel, and Roger Wattenhofer. Breaking recaptchav2. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 1047-1056. IEEE, 2024.
Ronen Porat and Ciprian Ceobanu. Enhancing spatial ability: A new integrated hybrid training approach for engineering and architecture students. Educ. Sci. (Basel), 14(6):563, May 2024.
Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vlad Bogolin, et al. Zerobench: An impossible visual benchmark for contemporary large multi- modal models. arXiv preprint arXiv:2502.00660 2025.
Fedor Rodionov, Abdelrahman Eldesokey, Michael Birsak, John Femiani, Bernard Ghanem, and Peter
Wonka. Planqa: A benchmark for spatial reasoning in llms using structured representations, 2025. URL rctps://arxin wttp://wwwww..ppp///////...
13

---

Published as a conference paper ICLR 2026
Roger N. Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects. Science 171(3972): 701-703, February 1971. doi: 10.1 10.1120/sciemmce.11139222.. science. 171. 3972. 701]
URL
Joshua Stevens, Jennifer M. Smith, and Raechel A. Bianchetti. Mapping Our Changing World. Department of Geography, The Pennsylvania State University, University Park, PA, 2012.
Ilias Stogiannidis, Steven McDonagh, and Sotirios A. Tsaftaris. Mind the gap: Benchmarking spatial reasoning in vision-language models, 2025. URL
org/ abs/ 2203/250... 19707
Yanlong Sun and Hongbin Wang. Perception of space by multiple intrinsic frames of reference. PLoS ONE, 10442 May 2010. ISSN 1932-6203. doi: 10.137 Ijoraallllalllooeee.000 001044..... URL http
//xx.doi.org/////// 1371 worr/a/ooral..... pone 0010442
Xiwen Teoh, Yun Lin, Siqi Li, Ruofan Liu, Avi Sollomoni, Yaniv Harel, and Jin Song Dong. Are [CAPTCHAs] still bot-hard? generalized visual [CAPTCHA) solving with agentic vision language model. In 34th USENIX Security Symposium (USENIX Security 25), pp. 3747-3766, 2025.
Luis Von Ahn, Manuel Blum, Nicholas J. Hopper, and John Langford. Captcha: Using hard ai problems for security. In International conference on the theory and applications of cryptographic techniques, pp- 294-311. Springer, 2003.
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint wrXiv:2509.02544 2025.
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth thousand words? delving into spatial reasoning for vision language models. In The Thirty-Eight Anmual Conference on Neural Information Processing Systems, 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, et al. Qwen2-vl: Enhanc- ing vision-language model's 's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024b.
Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-mllm: Boosting milm capabilities in visual-based spatial intelligence. arXiv preprint arXiv:2505.23747, 2025a.
Zonglin Wu, Yule Xue, Yaoyao Feng, Xiaolong Wang, and Yiren Song. Mca-bench: A multimodal benchmark for evaluating captcha robustness against vlm-based attacks, 2025b. URL htttpps///aaxi.oootttt /bss/
2506. 05982
Wenrui Xu, Dalin Lyu. Weihang Wang, Jie Feng, Chen Gao, and Yong Li. Defining and evaluating visual language models' basic spatial abilities: A perspective from psychometrics. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1 1571-11599 Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979-8 89779-8-9976-25111 doi: 10. (0.18633/v1 1/w.211222222 h2w.:/aallo2ooo..555555 567.. URL http:httss/eee http:::::/aaaaaaaooaoooeseo wtt.:/y5..g2002202 acl 1oong.56777
Wenrui Xu, Dalin Lyu, Weihang Wang. Jie Feng, Chen Gao, and Yong Li. Defining and evaluating visual lan- guage models' basic spatial abilitics: A perspective from psychometrics. arXiv preprint arXiv.2502.11859, 2025b.
Guixin Ye, Zhanyong Tang, Dingyi Fang. Zhanxing Zhu, Yansong Feng, Pengfei Xu, Xiaojiang Chen, and Zheng Wang. Yet another text captcha solver: A generative adversarial network based approach. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, pp. 332-348, 2018.
14

---

Published as conference paper ICLR 2026
A TASK CLASSES BY SPATIAL ABILITIES
A.1 SPATIAL PERCEPTION AND REFERENCE SYSTEM ABILITY
Figure 4: Illustrative examples of tasks targeting Spatial perception and reference system ability.
Spatial perception refers to the ability to judge the arrangement and orientation of objects relative to one's 's frame of reference Xu al. (2025a): Burgess (2006b). In spatial-cognition taxonomies is treated core sub-ability of visuospatial reasoning. Tasks targeting this ability require the solver to detect how objects align or orient in scene under a fixed coordinate system. Crucially, problems may be posed in egocentric fobserver-centered or allocentric (world-centered) coordinates Burgess] (2006a). Such questions hinge on maintaining a consistent reference frame (e.g. vertical axis) across views.
The key to solving pprcepppprro tasks is identifying invariant geometric relations that survive rigid transformations. particular, collinearity and parallelism are preserved under translation and rotation. For instance, points that lie straight line in one view remain collinear in any rotated or translated view, and any pair of parallel lines stays parallel after rotation or scaling. Humans naturally excel at judging basic alignments and reference-relationshiss. but this skill is difficult for algorithms lacking explicit frame-of- reference reasoning Sun Wang (2010). By contrast, models often fail when surface textures change even though the geometry is unchanged.
figure examples illustrate that the underlying invariant (alignment, orientation, and relative positioning across different views) is explicitly targeted. Each Spatial CAPTCHA instance is generated by sampling rigid transformation (rotation/translatioon of a base scene and asking a question anchored on the invariant relation. Solvers must therefore track the reference axis and preserving orientation, not surface appearance.
15

---

Published as conference paper ICLR 2026
A.2 SPATIAL ORIENTATION AND PERSPECTIVE-TAKING ABILITY
Figure 5: Examples of tasks probing Spatial orientation and perspective-taking Participants must mentally adopt alternative viewpoints to determine relative positions or directions of objects. The design highlights the distinction between object-centered transformations (rotation) and observer-centered transformations (orientation shift).
Spatial orientation and perspective-taking the ability compute where things are relative viewpoint and to mentally adopt alternative viewpoints without physically moving. Cognitive science distinguishes egocentric (viewer-centered) and allocentric moolddccnnteedd encodings, with perspective-takinn requiring systematic transforms between the two Hegarty & Waller (2004); Carroll (I993); Knauff (2006). Classic findings show dissociations between object rotation and perspective-takinng taking: the latter engages navigation- and scene based skills (updating the heading. re-anchoring axes, handling occlusions) that are only weakly predicted by mental rotation performance Hegarty & Waller Waller[(0004) (2004).
In the collage the correct answer is determined by viewer-centered predicate invariant to world-frame rotations and translations, not by object appearance.
A.3 MENTAL OBJECTS ROTATION ABILITY
rotation tasks. Classic studies by Shepard and Metzler [Shepard & Metzler (1971) showed that when subjects decide whether two shapes the under rotation, their reaction time increases lincarly with the angular difference between the shapes. Introspective reports confirm that people "mentally rotate" image align with the other. Similarly, the Vandenberg- Kuse Mental Rotations Test (MRT) presents flat images (often of 3D-based objects or letters) at various orientations, and asks participants identify which candidates the same shape versus mirror reflections. These
Human spatial cognition well-suited
16

---

Published as a conference paper ICLR 2026
Figure 6: Examples of tasks engaging the Menial objects rotation ability. The settings include polyhedral matching, 3D block assemblies and abstract shape comparisons. In all cases successful performance requires mentally rotating objects to establish equivalence or detect mismatch, showing how this core capacity recurs across spatial reasoning challenges.
findings support an analog mental-imagery process: subjects form a mental representation of the base shape and continuously rotate it until it matches target orientation, then make a match/mismatch decision. Representative instances that isolate this ability are shown in Fig. 6
Figure 7: Examples of tasks engaging the spatial visualization ability involving multiple transformations.
17

---

Published as a conference paper ICLR 2026
A.4 SPATIAL VISUALIZATION INVOLVING MULTIPLE TRANSFORMATIONS
Spatial visualization denotes the capacity to manipulate an imagined configuration through sequence of operations (as rotations, reflections, translations, folds, cuts, and recombinations) while keeping track of intermediate states. In psychometries it is treated as factor separable from. though correlated with, mental rotation and spatial orientation Carroll (1993); Hegarty & Waller (2004). Classic instruments such as the Paper Folding Test (PFT) instantiate this ability by requiring subjects to simulate multiple fold fold-punch- punch-unfold unfold steps to predict the final pattern. Unlike single-transform problems, success depends on composing operations and maintaining stable internal representation across steps. Representative instances that refer to this ability are shown in Fig.
B DIFFICULTY MAP CONSTRUCTION AND CALIBRATION
B.1 INTERPRETABLE KNOBS
For each class we vary only factors that change the spatial problem, not appearance:
Perception (reference frame). Number of objects in the scene; polygonal complexity (sides 3-8); tilt magnitude relative to gravity/horizon; minimal gaps da between primitives; number of near-parallel distractors.
Perspective-taking. Camera yaw/pitch/roll ranges; baseline distance to landmarks; number of landmarks; depth layers (near/mid/far) and occlusion fraction; horizon tilt; discrete viewpoint set size m (candidate panels).
Mental rotation (2D). Rotation angle gap A6; presence/absence of mirror alternatives; vertex count/con- cavity of shapes; symmetry order of the base shape (to avoid trivial or ambiguous matches); number of candidates.
Topological relations. Grid size/board extent; number of pieces/regions; hole count and connectivity; mini- mal separation between components; edit distance of distractor graphs (touching vs. strictly inside/outside).
Variability is achieved without compromising label soundness. The scene function G and distractor mecha- nisms I' generate semantic diversity (base shapes, layouts, camera poses, fold sequences) while remaining within the invariant /. Distractors are synthesized as near-misses along the same spatial axes that define d(0) (e.g., angle gaps just above du, mirrored but non-congruent shapes, off-by-one transform sequences), success requires the intended spatial relation rather than superficial cues.
Reproducibility and provenance Every instance carries a manifest identifier and seeds for m). enabling exact regeneration and audit. Changes to a class are diffs to M (versioned), not ad-hoc asset edits.
B.2 DIFFICULTY MAP CONSTRUCTION
By elevating the manifest to the first-class abstraction, we (i) connect each item family to a precise invariant, (ii) guarantec trouh correctness and uniqueness independent of rendering, and (iii) unlock effectively unbounded, auditable, and human-simple item bank (details in Section 5). The detailed procedure for constructing and calibrating the difficulty map, including isotonic and quantile regression fits well binning strategies, is provided Appendix [8. B.3 and B.4 An illustrative example manifest, including the field-tossymbol alignment, is presented in Appendix
[B.2
18

---

Published as a conference paper ICLR 2026
B.3 ISOTONIC AND QUANTILE REGRESSION DETAILS
The objective of this stage is to translate raw human performance statistics, primarily response times and success rates, into a calibrated difficulty signal that is both monotone and globally comparable. We formalise the mapping in two phases: (i) fitting predictive models from task parameters to human outcomes, and (ii) combining these predictions into a single latent difficulty score that can be inverted during item generation.
Data structure For each task family, we collect dataset of N instances. Each instance is annotated with (a) a hyperparameter vector x specifying the generative knobs (c.g., polygon sides, rotation angle, viewpoint set size), and (b) observed human outcomes: mean response time f(x) correct trials, and empirical success rate (x) E [0, 1]. The goal is to characterise how variations x influence human performance.
Monotone regression of response times Response times positive, heavy-tailed, and expected to grow monotonically with task difficulty. We therefore apply isotonic regression to log t(x), fitted separately for each family. The isotonic model y (x) learns a non-decreasing function along axes of known monotonicity (e.g., larger rotation angles, greater occlusion), optionally smoothed to avoid degenerate step functions. This yields a calibrated predictor of expected solution latency.
Quantile modelling of success rates. Success rates lie in [0, 1] and typically exhibit heteroscedastic, non- Gaussian noise with ceiling effects on easier instances and occasional floor effects on harder ones, but not a strict bimodal pattern. To capture this variability without imposing a parametric mean-variance relationship, we fit quantile regressions for s x. The model (x) estimates the conditional median (r=0.5) as a robust central tendency and lower quantile (e.g., 0=225 to characterise fragile regions where non-trivial fraction of participants fail despite similar knobs. Predictions are clipped to [0, 1] and subsequently aligned across families via the global isotonic calibration described above.
Unified difficulty mapping Response time and success rate capture complementary facets of hardness: the former reflects cognitive effort given success, the latter reflects probability of failure. To fuse them, we first apply per- farmll rank normalisation:
(x) QuantileRank(log t(x)),
(x) QuantileRank(11 - ss()).
Both 1 and Ey lie in [0. 1], with larger values corresponding to greater difficulty. To achieve foss-famiiy comparability, we then align these variables globally via isotonic calibration against their pooled empirical CDFs, producing T, E E [0, 1]. The final difficulty score is defined as convex blend
d(x) = T(x) + (1 a a) E(x), a E [o, 1].
In practice we fixed a = 0.6, based on preliminary trials showing that a slight emphasis on response time yiclds smoother difficulty distributions and better separation of adjacent levels, while still preserving discrininability from success rates. This choice is not critical but stabilises the map across heterogeneous task families.
Inverse use in generation During item synthesis, the difficulty map is inverted: given a target difficulty value d' bin, the system searches for hyperparameters x whose predicted difficulty d(x) falls within the desired band. The procedure is as follows:
1. Select target pair. Sample point (T* E+) on the iso-difficulty line a + (1 a) d:, ensuring fcasibility within [0, 11².
2. Map back to family scales. Invert the global calibrators to obtain family-specific targets Tj, Fij. then recover approximate raw values t", s* using per-family inverse CDFs.
3. Solve for knobs. Search for x minimising
X, (x) t-1 + 2g s*1+ ((x),
19

---

Published as a conference paper ICLR 2026
subject admissibility constraints (visibility margins, symmetry screens). Here f2 is diversity regulariser encouraging coverage of the knob space.
4. Verity. Recompute d(x) for the candidate x and accept if d(x) E I and prediction errors are within tolerances (ct, Cz). Otherwise, adjust the target pair along the iso-difficulty line and repeat.
This inversion procedure exploits the fitted forward models T 8y, turning the difficulty score into a generative control knob. It closes the loop: desired bins in difficulty space translate into concrete parameter settings, ensuring principled and reproducible control over task hardness rather than reliance on uncontrolled rendering artefacts.
B.4 BINNING STRATEGIES AND PRIORS
With a scalar difficulty score d(x) E [0, 1J established, we discretise the continuum into bins that support controlled sampling during benchmark construction. Binning ensures that items are evenly distributed across difficulty levels while remaining aligned across task families.
Quantile-based binning We partition d(x) into three bands: easy, medium, and hard, using global quantile thresholds. This ensures that each bin contains approximately equal probability mass, preventing trivial instances from dominating and providing adequate coverage of the hard tail. Applying thresholds globally across all task families keeps the bins comparable, so that easy in one class corresponds similar expected human effort in another. The resulting distributions across bins are shown in Figure 3a
Stratified priors for sampling During synthesis, bins are sampled according to stratified priors P.. These priors control the relative prevalence easy, medium, and hard instances in the generated benchmark and are defined consistently across task families. The priors normalised to preserve global proportions, ensuring that sampling remains balanced while still allowing targeted emphasis (e.g., for stress-testing models).
Rejection and admissibility After sampling from bin, enforce validity by rejecting any instance that violates structural constraints (P) or visual guards (V). This ensures that binning never admits ambiguous or degenerate cases, such overlapping primitives or low-contrast distractors. The final benchmark therefore achieves a stratified and interpretable distribution of difficulty levels that is both reproducible and free of rendering artefacts.
C
INVARIANT-SPECIFED TASK MANIFESTS AND GROUND-TRUTH CERTIFICATION
C.I EXAMPLE MANIFEST.
To make the abstraction concrete, Listing shows minimal ISON manifest instantiating the tuple M
for a viewpoint-matching item; each field maps to id, 7, (0, Pe), T, r, R as defined above, with the field-to-symbol alignment summarized in Table The distractors explicitly encoded alternative agent viewpoints, validated for uniqueness, ensuring the task remains well-posed.
"type": "custom", "script": "generate. py", "name": "Agent Sight", "inpui";
3
5
ESOX_COUNT";
"type": "int",
7
"min": 1,
"max":
20

---

Published as conference paper ICLR 2026
"COLOR MAP": "COLOR_MAP":
"type": "enum", "values": ["Pastell", "Pastel2"]
15
}
"task":
17
"prompt.:: "Imagine you are...",
"answer":
19
"num_ inun_variants": 4,
"variants";
21
"type": "enum",
"values"; ["1", "2", "3", "4"]
}
"correct"; "SCORRECT"
26
Listing 1: Example ISON manifest
D EVALUATION PROCESS DETAILS
This appendix provides a detailed description of the evaluation metrics used throughout the study. The metrics are designed to capture not only task-level correctness but also calibration, coverage, and cognitive plausibility of model behaviour. They are computed consistently across all task families and difficulty bins.
Pass Pass@) measures the proportion of task instances for which the model's top-ranked prediction is correct. This is the most stringent correctness metrie, analogous exact match, and reflects whether
Table 3: Alignment between the canonical JSON manifest and the formal tuple M. Distractors are explicit scene variants (e-g., fake agent locations with unique views) generated alongside the correct answer.
ISON
Example
name version
id
"Agent sight", "custem", *1.2"
Took Semonties
invariant Lask, prompt
"view_match"
T
"imagine are the STaRGET in the above figure, which one of the following scenes will you see?* "SCORRECT*
lakkanswer.corres
Scene & Rendering
socne/script
"genecate.py"
validators
V
""uniqueness" "margint")
renderer
R
"custom"
Sampling& & Distractors
inpatBOX
input BOX COUNT 0, Pe
"min" :1, "nax":5
inpot.COLOR MAP
"values"; ["Pastell", "Paste12")
makkannswer..ariantt
g.g
["fako_agent_A' "eako_agent__"" B",...)
21

---

Published as a conference paper ICLR 2026
Table 4: Generalized invariant families aligned with that certify the intended invariant and the distractor strategies used to generate nontrivial but incorrect alternatiess.
abilities. Each row specifies validators
Invariant family (7)
Validators (V)
Distractor strategy (T)
Spatial perception and reference system
alignment and parallelism checks under rigid transforms; collincarity
near-parallel or collincar but misaligned seg- ments; objects offsct just beyond tolerance
consistency:
uniqueness lests
Spatial orientation and perspective taking
egocentric allocentric consistency; ray visibility; camera transform equivalence: uniqueness andits
fake observer viewpoints yielding plausible incorrect views; near-pose confusions
Mental object rotation
oquivaleeee under (0)2))00(3);
mitror images: rotated near-matches differing by small angular offsets; flipped but similar silhauettes
congruence tests with angular marginns; mir- mrfreflection screens
Spatial visualization with multiple transformations
multi-stpp transformation execution (fold, re- volve, unfold); graph isomorphism checks sleps: tracking of vuxel/projection consistency
partial transformation paths; inconsistent jection sets; solids from alternative operation sequences
the model can reliably prioritise the correct answer without reliance on downstream ranking or sampling. Formally, y the ground truth and ( the top prediction for instance i, then
Pass
k lo (1)
=y}.
Pass@k k Pass relaxes the top-1 requirement by scoring an instance correct if the ground truth appears within the top-k predictions. This metric reflects the model's ability to maintain coverage of the correct answer under uncertainty. For k = 3 as used in our study,
P
1 N r tu E (0))) ..... }}.
Σ
(k)
k-of-h Reliability Beyond coverage, we assess how reliably the model's top-k predictions contain only correct answers. The k-of-k metric computes the fraction of instances where all of the top-k predictions equal the ground truth. This is stricter than Pass@1 k and quantifies whether a high-confidence prediction set is trustworthy, For k 3, this amounts to
N
k-of-k = N
Σ HO = (2) =9 = y1
(3)
Reliability vs. Coverage To diagnose calibration, we plot k-of-k against Pass @k (cf. Figure 3c). Ideally, a well-calibrated model lies near the identity line: if it predicts the answer is within its top-k set, then that set should reliable. Models below the line exhibit overconfidence (claiming coverage without reliability). while those above the line are overly conservative.
Per-ability metries In addition to aggregate metrics, we report @ass@1 stratified by cognitive ability class: spatial perception (SP), spatial orientation (SO), mental object rotation (MOR), and multi-step visualisation (SV). These disaggregated metrics reveal which cognitive primitives are most brittle for models and whether difficulty arises from perceptual compositional factors.
22

---

Published as conference paper ICLR 2026
Human-level reference Human annotators (N=60) provide an empirical soft upper bound. An item considered solved if at least two annotators select the correct answer under time constraints. This yields both pass rate and distributions of human response times, against which model predictions are normalised. Reporting both metrics provides insight into where models deviate most strongly from embodied or time- bounded human reasoning.
iffficulty-stratffee performance Finally, we analyse metries within Easy, Medium, and Hard bins defined by the difficulty map (Appendix [B44). This stratification verifies that accuracy decreases monotonically with difficulty for both humans and models, confirming that d(x) captures substantive cognitive load rather than noise.
Together, these metrics provide multi-faceted view of performance: Pass@1 captures strict correctness, Pass & captures coverage, k-of-k exposes calibration, per-ability scores isolate cognitive bottlenecks, and human- references provide grounding in real-world effort.
Table 5: Summary of evaluation metrics used in this study. Metrics grouped by evaluation intent: correctness, calibration, efficiency, human upper bounds, and cognitive attribution.
Metric
Type
Scope
Purpose and Interpretation
Overall Accuracy Merrics
Pass G
Accuracy
10,1
Top-1 1 correciness under deterministic decoding (7=0.0). Measures default model reliability without sampling.
Pass d'
Accuracy
10,
Success rate with k=3 completions. Probes recoverability under model uncertainty. Fraction of instances where all i sampled ourpats are identioal and correct. Measures model confidence output consistency.
k/k Reliability
Epistemic Stability
[o, 1
Human Upper Bound
Human Simple Pass Rate
Sanity Check
[0,
Fraction of instances correctly solved hy at least 2 of human annotators under 30s time limit. Used establish baseline for "non-trick" solvability.
Per-Abiliyy Pasvi
Spatial Perception
Accuracy
10, Accuracy on tasks requiring recognition of spatial layout, object relationships, and metric adjacency in visual scenes. Accuracy 10,
Spatial Orientation
Accuracy
involving cowpoint transformations egocentric-to-alleeeete
alignment.
Mental Rotation
Accuracy Accuracy
00,1
Accuracy tasks roquiring rigid-body rotation of objects in 2D 3D space.
Spatial Visualisation
[0,
Accuracy tasks requiring multi step spatial transformations, such folding, cutting, or layored movement.
FAILURE ANALYSIS
Despite modest performance on select task types, current models systematically fail generalise spatial reasoning beyond perceptual regularities. This section analyses dominant failure modes through taxonomy of error classes and representative examples. All qualitative patterns are drawn from a held-out evaluation set, with aggregate statistics reported over n=70 Spatial CAPTCHA instances.
E. TAXONOMY OF FAILURE MODES
We categorise model errors into three broad families: (i) Invariant violations, where the predicted output contradicts task-specified geometric relational constraints; (ii) Hallucinated structure, in which the model invents non-existent elements or misattributes spatial relationships, and (iii) Calibration errors, wherein the top-k prediction set fails to reliably include the correct answer despite high predicted likelihood.
Invariant violations This family dominates the error distribution, with approximately 63.5% (94/148) of coded failures. In UNFOLDED, models frequently misplace facets of cube net, breaking adjacency
23

---

Published as conference paper ICLR 2026
which one of clituiing zome Iyou
ww 4
Yoar Answey
Crenc3
Famm8
(a) contraint-based difficulty control mechanism
(b) Question and answering page
Figure Illustration of the agent sight task of our online spatial CAPTCHA service: (a) provide difficulty control flexibility by adjusting box and cylinder counts and color maps: (b) the question and answering page which runs automated correctness verification on the backend while also recording the solving time.
constraints. In PYRAMID, they misalign side-view projections, confusing planar-to-volumerty consistency. Sub-classes include viewpoin//perpeecivve errors (74 cases), rolation mirror misalignments (42 cases), and rare but diagnostic topology/containmet violations. These patterns confirm that models fail to internalise certified invariants and instead resort weakly correlated perceptual cues.
Hallucinated structure Roughly 35.1% (52/148) of failures fall in this family, where models fabricate symmetry, occluded elements, or entire structures absent from the input. This is most evident in FULl VIEWS, where occluded geometry is invented, and in multi-projection tasks, where unsupported symmetries are projected onto irregular shapes. For example, GPT-4o variants tend overgeneralise from canonical forms, inferring staircases or pyramids where no such invariants exist. These errors reveal brittle inductive priors and over-regularisation of spatial patterns.
Calibration errors Though less frequent in natural-languas language rationales, calibration issues remain evident in evaluation metrics. At 0.7% of coded failures, explicit overconfidence rare, but systematically all models show gap between high Pass @ k coverage and low k/k reliability. For instance, distractor options are often included in the top-k set with high likelihood, while the true answer is excluded. This reflects poor uncertainty estimation and suggests that models rely shallow scoring heuristics rather than calibrated spatial reasoning.
ONLINE SPATIAL CAPTCHA SERVICE
To better show our contribution on building CAPTCHA, we show the webpage screenshot of our online spatial CAPTCHA service (laking agent sight task as an example) in Figure
G LLM USAGE STATEMENT
During the preparation of this manuscript, LLMs were utilized exclusively for language refinement and stylistic editing. The technical contributions, experimental design, data analysis, and interpretation of results were not generated by LLMs. All conceptual development, methodological details, coding, and evaluation solely the responsibility of the authors. In accordance with policy, the authors assume full accountability for the accuracy and integrity the content, and any errors or misrepresentations are exclusively their own responsibility.
24

---

Published as a conference paper ICLR 2026
H IMPLEMENTATION DETAILS
Environment All experiments were orchestrated from local development environment running on Mac Studio (Apple M2 Ultra, 128GB unified memory) with Python 3.13. However, no inference was executed locally. All model queries and evaluations were conducted via the OpenRouter API, ensuring consistent inference environment across experiments.
Human studies For the human evaluation component, participant groups were recruited from multiple institutions and diverse demographics. In particular, we included (i) graduate and undergraduate students from two universities, and (iii) broader community participants representing varied nationalities, age groups, and professional backgrounds (recruited through the extended social networks of the authors). This composition ensured both institutional diversity and cultural heterogeneity. All human studies informed consent protocols.
conducted under
Generation pipeline Task generation relied on a combination of open-source 3D and visualization toolchains. Procedural scenes were synthesized using Bl Blender 4.4.. 3.3.3, geometric manipulations and renderings were facilitated by vedo 2025.5. 4, while classical Python libraries such matplot. oooolll1 1ib were used for visualization and plotting. The generation pipeline was fully scripted and released to guarantee reproducibility.
Validation Automated task validation employed both standard libraries and domain-specific packages. In particular, we used scipy==1.. 16. 0 for statistical consistency checks and polyomi 0 7.1 for verifying combinatorial tiling constraints. Additional validation relied on custom Python scripts to enforce task-specific invariants and to audit correctness prior to release.
reCAPTCHA comparison For the comparison against commercial CAPTCHA systems, we clarify that there is no publicly available reCAPTCHA-Ben -Bnchh. Instead, we rely on MCA MCA-Bench, which contains task type explicitly inspired by Google reCAPTCHA but manually created by the authors MCA-Bench. To ensure fairness, we exported only the subset of MCA-Bench corresponding to reCAPTCHA-style items and used this as a proxy benchmark. This choice was motivated by the need to compare our method against the most widely deployed CAPTCHA solution in practice, while preserving as much fidelity possible to the task format encountered in the wild.
I LONGEVITY AND FORWARD-LOOKING RELEVANCE OF SPATIAL CAPTCHA
Statie CAPTCHAS, once effective mechanism for distinguishing human and machine perceptual ca- pabilities, have exhibited rapid degradation in discriminative efficacy as vision and multimodal reasoning models advance. practice, static datasets confer only transient robustness: once their distribution is absorbed into large-scale training corpora, generalisation collapses due to overfitting at the dataset or feature-template level. This phenomenon underscores a fundamental limitation: fixed perceptual benchmarks cannot sustain discriminatory power in an evolving model ecosystem.
SPATIAL CAPTCHA is designed explicitly to mitigate this brittleness through procedural generation. Rather than static dataset, it constitutes parametric generative frumework that models the key dimensions of human spatial cognition. These factors jointly define a continuous control manifold from which puzzles are synthesised. By sampling along this manifold, we can systematically scale perceptual load and reasoning com- plexity. yielding a continuously tunable difficulty curve. Empirically, model accuracy degrades monotonically with increasing compositional depth (Figure 9. demonstrating that the generative parameters induce smooth and controllable adjustment of challenge. This property ensures not only granular difficulty calibration but also sustained adaptability model capabilities evolve.
25

---

Published as conference paper ICLR 2026
Model accuracy versus compositional depth
50
Model accuracy
40
Accuracy (%)
30
20
10
0
3
5
6
Compositional depth
Figure 9: Model accuracy versus compositional depth in Spatial CAPTCHA. Accuracy declines monotonically as relational and occlusion complexity increase, confirming continuously tunable difficulty.
1.1 MODULAR COGNITIVE PRIMITIVES AND EXTENSIBILITY
Beyond immediate tunability, the framework is designed to remain forward-compation with emerging insights from cognitive psychology and ncuro-symbolic Al. Each cognitive primitive is implemented as a modular transformation operator within the generator. This modularity allows principled integration of new perceptual constructs without architectural reconfiguration. For example, as research elucidates mechanisms of dynamic attention shifts and anchoring human perception (Itti & Koch (200]) advances in multi-object affordance modeling in robotics and embodied AI (Myers et al. 2015 Do et al. 2022), corresponding operator modules be introduced to expand the cognitive span of generated tasks.
Table Spatial complexity effects where highest parameter values yield lowest accuracy. Parameters are grouped by cognitive complexity type, confirming systematic difficulty progression despite non- -onmooooo intermediate values.
Parameter
Range
Accuracy (%)
Interpretation and Effect
Boseline
Minimal complexity
simplest configurations (lowest parameter values), Represents baseline spatial reasoning capahility hefore complexity penurrations.
Performance
Virual Complesity
Scene density (2-10 objects)
Low-High
30.3
Accuracy drops from 42.2% (sparse) to 25.8% (dense scenes). Visual clutter and occlusion systematic procesing challenges despite momotonic intermediate
values.
Geometric Complexity
Polygon complexity (3-6 sides)
Low-High
28.0
Performance decrcases from 29.0% (trianglos) to 21.7% (bexagons). Higher-order polygons geometric reasoning demands despite intermediate fluctuations.
Combinstorial Complesity
Spatial arrangement (3-5 tiles)
Low-High
25.8
Performance deelines from 27.8% 23.2% with pieces. Increased combinatorial demands strain spatial working memory and arangement reasoning.
Projection Complexity
Mapping resolution (3x3-144)
Low-High
24.7
Accuracy decrcases from 26.6% to 22.7% with fincr grids. Higher resolation increases 3D-to-2D corresoondence complexity.
Surface Complexity
surface detail (4-8 vertices)
Low-High
21.4
Accuracy drops 22.7% 7% highest counts. Complex surfaces challenge 3D transoormaiion understanding despite non-linear progresion..
26

---

Published as conference paper ICLR 2026
J ASSESSING DATA- VERSUS MODEL-LIMITED PERFORMANCE VIA
To determine whether the observed performance ceiling stems from data scarcity or from architectural limitations in contemporary multimodal transformers, conduct controlled fine-tuning study frontier models. The objective is to assess whether additional task-specific supervision yields meaningful accuracy gains or whether the deficits instead arise from inductive biases that misaligned with the -paaial-relational structure of the benchmark.
J.1 TUNING CONFIGURATION
We fine-tuned the most capable publicly accessible frontier model the time of study (gpt-4o 4o (2024-08-06 base)) via the official OpenAl supervised fine-tuning interface. Table 7 summarises the exact hyperparameters and training budget.
Table 7: Fine-tuning hyperparameters and training budget.
Parameter
Value
Model and Data Configuration
Model
gpt-4c 4 (2024-08-06 base)
Training tokens
3.7M
Epochs
Batch size
Training Dynamics
Learning-ran rate multiplier
Optimisation mode
Supervised fine-tuning (OpenAI API)
Reasoning budget
Fixed; identical to evaluation setting
Prompt scaffold
Unchanged; only task-specific examples added
To reduce confounds related to prompt-format drift or context-length truncation, we retained the original evaluation scaffold, modifying only the demonstrations used during mme-tuning.
J.2 EFFECT OF FINE-TUNING ON ACCURACY
Fine-tuning produced modest improvement: accuracy rose from 38% 57.7%, still far below human performance (89.5%). Table g provides the full comparison, including the stronger gpt-5-1 model.
Despite increased model scale and reasoning capacity, gpt-5. exhibits only marginal gains, mirroring the stagnation observed after fine-tuning.
J.3 IMPLICATIONS
Taken together, the evidence supports a model-limited interpretation: current multimodal transformers appear constrained not by exposure to the task distribution but by representational inadequacies intrinsic to their architccture. These include weak mechanisms for modelling equivariance, non-local geometric dependencies, and viewpoint-consiisent relational structure. Mitigating such limitations likely requires architectural interventions (e.g., explicit spatial modules, equivariant layers, or hybrid neural-symbolic operators) rather than additional data alone.
27

---

Published as conference paper ICLR 2026
Table 8: Accuracy improvements from fine-tuning, compared against stronger baseline models and human performance.
System
Accuracy
Model Performance
got-4o (base)
38.0
9pt-40-2024-0--00 (fine-tuned, ours)
57.7
Comparative Upper Bounds
gpt-5.1
61.5
Human
89.5
REPRODUCIBILITY NOTES
All fine-tuning runs adhered strictly to the standard OpenAl API configuration, without undocumented hyperparameter overrides. Logs, prompts, and training traces appear in the supplementary artefacts.
K PERCEPTUAL CLARITY SAFEGUARDS
To ensure that logically well-formed SPATIALCAPTCHA instances also exhibit robust perceptual unambiguity, we implement a three-stage verification pipeline: (i) pre-deployment human validation, (ii) post-render robustness checks, and (iii) runtime quality control.
K.1 HUMAN PERCEPTUAL VALIDATION
dedicated perceptual-clariin study demonstrates that more than % of generated puzzles achieve at least 95 % inter-purticipunt agreement. The empirical distribution confirms that visually ambiguous instances constitute only a negligible minority, providing a lower bound on perceptual soundness prior to any algorithmic filtering.
K.2 AUTOMATED POST-RENDER VERIFICATION
Puzzle families subject to strong 3D/2D projection artefacts (e.g., viewpoint rotation or perspective transfor- mations undergo an automated similarity- based confusability check). Rendered silhouettes of the target and distractors are compared; instances exceeding calibrated similarity threshold are discarded.
K.3 RUNTIME RELIABILITY- FILTERING
During deployment, SPATIALCAPTCHA continuously monitors online human remmnn-e consistency. Puzzles exhibiting anomalously high disagreement (relative to historical baselines for their category) are automatically flagged and removed. This procedure provides adaptive quality control under distributional drift and long-tail ambiguity, mirroring mechanisms used in large-scale production CAPTCHA systems.
Together, these safeguards ensure that logical correctness is matched by high perceptual clarity, enforced both generation time and under live conditions.
28

---

Published as conference paper ICLR 2026
L POLICY-INDUCED ASYMMETRIES IN CAPTCHA PERFORMANCE
The apparent advantage of open-source models on the Spatial-CAPTCHH benchmark is predominantly a policy-induced abstention urtefact rather than evidence of superior visuospatial reasoning. Contemporary proprictary MLLMs incorporate multi-layered safety and anti-abuse filters that aggressively pattern-mater canonical reCAPTCHA affordances (c.g., characteristic logos, tiled layouts, and lexical cues such as "captcha", "are you robot?", embedded site-key prompts). These filters frequently trigger deterministic refusal templates (e.g.. "I cannot assist with CAPTCHAS"), thereby converting a large fraction of queries into hard abstentions.
By contrast, our Spatial-CAPTCHA are intentionally unbranded and semantically neutral. They lack the visual and textual markers that typically activate CAPTCHA-protection policies, and are therefore interpreted by proprietary systems as generic geometric-reasoning tasks. Open-source models, which usually ship without hard refusal rules for CAPTCHA-adjaceee content, attempt both tasks uniformly. As result, their measured accuracy reflects a greater proportion of unblocked attempts rather than genuine cognitive advantage.
Refusal-Aware Evaluation. To disambiguate reasoning capacity from policy-induced abstention, we report for each model and task (i) the proportion of explicit refusals (REFUSAL and (ii) overall accuracy across all attempts (ACCURACY %). This refusal-aware stratification (see Table (9) enables fairer cross-mode comparisons and reveals that the primary driver of observed performance asymmetry is the refusal layer rather than underlying model competence.
Table 9: Model performance on reCAPTCHA tasks with refusal-aware reporting.
Model
Refusal ( (%)
Accuracy (%)
claude-sonnet-4
10.0
11.3
claude-opus-4
6.7
10.0
qwen2.5-v1-72biinstuu
0.0
2.7
llama 10----mma maverick
0.0
1.3
29


Reviews:
[{"review_id": "lnjuzsBmtz", "reviewer": "Reviewer_kkRJ", "summary": "Spatial Captcha presents a procedural generation pipeline for creating challenging multiple choice visual spatial reasoning captchas. The pipeline is designed to target well studied visual reasoning tasks that are easier for humans than for models. The authors evaluate some of the frontier multimodal models on the generated captchas and find that they only achieve 31% accuracy at best while humans achieve over 99% accuracy.", "questions": "1. How does a frontier system like GPT-5 with a high thinking budget perform on the Spatial CAPTCHA dataset? Does it significantly outperform the state-of-the-art VLMs evaluated in the paper?\n2. Have you tested whether fine-tuning on procedurally generated puzzles significantly improves model performance, or whether the limitation is architectural?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and the motivation for building a more challenging captcha is clear. The authors thoroughly explain the human psychology research that led to the design of the dataset and validate that the puzzles generated are challenging for many state-of-the-art multimodal models.", "weaknesses": "The authors motivate their work by pointing out that current CAPTCHAs are too simple for the best performing models. This is a valid concern but I am not sure if building a slightly more challenging CAPTCHA is the best way to address it. It has become clear that even if a certain type of visual reasoning task is harder for models today, it will be significantly easier for the next generation of frontier models. The authors could strengthen the paper by discussing the expected lifespan of Spatial CAPTCHA as a benchmark and how the framework could adapt to future model advances. I believe a more forward looking approach would be to embrace that as computer-use agents get more capable they will make up a majority of internet traffic and we should adopt more [forward looking mechanisms](https://blog.cloudflare.com/introducing-pay-per-crawl/).\n\nAnother concern is that the spatial CAPTCHA dataset is framed as out-of-distribution data for current models. While I agree that web data lacks these specific reasoning tasks, model developers could readily generate synthetic data targeting this distribution. The paper attributes poor model performance to architectural limitations in VLMs, but this claim would be significantly strengthened by fine-tuning experiments demonstrating that performance is low even with targeted training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cUoQogi0D7", "reviewer": "Reviewer_BLj8", "summary": "This paper introduces Spatial CAPTCHA, a novel framework designed to differentiate humans from machines by leveraging tasks grounded in spatial reasoning—a known weakness of current Multimodal Large Language Models (MLLMs). The authors have developed a sophisticated, automated pipeline that procedurally generates a virtually endless supply of puzzles based on four fundamental human spatial abilities, such as mental rotation and perspective-taking. A key innovation is the use of formal specifications (\"manifests\") to ensure each puzzle is valid, unambiguous, and has a controllable difficulty level.\n\nTo validate their approach, the authors created Spatial-CAPTCHA-Bench, a dataset of 1,050 puzzles, and conducted a large-scale experiment comparing the performance of 10 state-of-the-art MLLMs against 60 human participants. The results reveal a stark performance gap: humans achieved approximately 90% accuracy, while the best-performing AI model only reached 31%. The study also found that MLLMs are poorly calibrated, often expressing high confidence even when providing incorrect answers, further demonstrating the effectiveness of this new CAPTCHA paradigm.", "questions": "The paper does an excellent job of ensuring the logical correctness and unambiguity of the puzzles via the manifest system. However, could you elaborate on how the system guarantees perceptual unambiguity after the 3D scene is rendered into a 2D image? Specifically, what prevents a situation where a mathematically incorrect distractor might, due to rendering artifacts or its specific orientation, be visually confusable with the correct answer? A discussion on this point—perhaps covering renderer fidelity, human validation studies focused on distractor clarity, or potential post-rendering programmatic checks—would significantly bolster the framework's claim of robustness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. **Novel and High-Impact Problem:** The paper addresses a timely and critical real-world problem. With conventional CAPTCHAs becoming increasingly vulnerable to AI, the proposed method of targeting a well-established weakness in MLLMs—spatial reasoning—is an insightful and promising direction for the next generation of human-verification systems.\n\n2. **Rigorous and Scalable Generation Pipeline:** A standout contribution is the procedural generation pipeline. Using formal manifests to define and control puzzle generation is an elegant solution that guarantees the validity and logical unambiguity of each CAPTCHA by design. This makes the system renderer-agnostic and capable of producing an endless supply of challenges, which is essential for a robust security tool.\n\n3. **Methodical and Cognitively Grounded Design:** The work is built on a strong theoretical foundation. Instead of designing arbitrary puzzles, the authors methodically ground their tasks in four well-researched categories of human spatial cognition. This principled, theory-first approach adds significant scientific credibility to the framework.\n\n4. **Convincing and Well-Executed Experiments:** The empirical evaluation is thorough and the results are compelling. The massive, undeniable performance gap between humans and a wide array of top-tier MLLMs provides powerful evidence for the system's effectiveness. The direct comparison to reCAPTCHA further strengthens the claim that this is a superior method for human-machine differentiation.\n\n5. **Exceptional Clarity and Presentation:** The paper is very well-written, engaging, and easy to follow. The motivation, methodology, and results are presented with a clear and logical narrative, making the paper's significant contributions highly accessible.", "weaknesses": "The primary weakness of the paper lies in an important, unaddressed subtlety in its claim of unambiguity.\n\n**Lack of Perceptual Ambiguity Verification:** The framework's core claim is that puzzles are \"unambiguous\" because they are generated from a mathematically precise and logically sound manifest. However, this guarantee of logical unambiguity does not automatically extend to perceptual unambiguity in the final rendered 2D image. The paper does not adequately address the possibility that rendering artifacts, unfortunate camera angles, or subtle visual similarities could cause a mathematically incorrect \"distractor\" option to appear correct, or vice-versa. While the high-fidelity renderer and significant geometric differences between options mitigate this risk, the lack of a formal post-rendering check for perceptual clarity is a notable gap in the validation process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYkOory9PG", "reviewer": "Reviewer_NVWJ", "summary": "This work presents Spatial CAPTCHA, a framework designed to be robust against MLLMs that defeat current CAPTCHAs. It leverages the significant gap between human and AI spatial reasoning abilities by generating dynamic questions requiring geometric reasoning, perspective-taking, and mental rotation. Evaluation on the new Spatial-CAPTCHA-Bench demonstrates that humans vastly outperform 10 state-of-the-art MLLMs, with the best AI model achieving only 31.0% Pass@1 accuracy.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a new and well-founded open-source pipeline that can automatically generate large scale 3d spatial captchas based on spatial cues such as positioning, counting, etc. The performance on the benchmark shows impressive human performance (almost 100%) and decent MLLM performance (lower than all open code/data captchas). The authors also provided extensive analysis to the reliability of their benchmark and the performance of different models under each specified category.", "weaknesses": "1. Why would open-source models perform better on spatial-captcha than on recaptcha, even though proprietary ones do the opposite? The latter discovery demonstrates how difficult spatial-captcha is, but there isn't enough explanation for the open-source models' performance.\n2. Figure 3b's graphs are all stacked together. Without a legend and enough explanation, it is difficult to make meaningful thoughts about this figure.\n3. Typos: All citations in Section 3 have wrong format (no parentheses).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)