GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMs
Authors: Shixian Luo
Zhu zezhou
Yu Yuan
Yuncheng Yang
Lianlei Shan
Yong Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Geometry Spatial Representation
Procedural Geometry Reasoning
Primary area: datasets and benchmarks
TL;DR: We introduce GeoGramBench, a new benchmark probing LLMs’ ability to translate procedural geometry code into internal spatial representations, revealing that current models struggle with this core aspect of spatial-symbolic reasoning.

Abstract: Geometric spatial reasoning forms the foundation of many applications in artificial intelligence, yet the ability of large language models (LLMs) to operate over geometric spatial information expressed in procedural code remains underexplored. In this paper, we address this gap by formalizing the \texttt{Program-to-Geometry} task, which challenges models to translate programmatic drawing code into accurate and abstract geometric reasoning. To evaluate this capability, we present \textbf{GeoGramBench}, a benchmark of 500 carefully refined problems organized by a tailored three-level taxonomy that considers geometric complexity rather than traditional mathematical reasoning complexity. Our comprehensive evaluation of 17 frontier LLMs reveals consistent and pronounced deficiencies: even the most advanced models achieve less than 50\% accuracy at the highest abstraction level. By systematically analyzing model behaviors, our study exposes key limitations in program-driven spatial reasoning and positions GeoGramBench as an important resource for benchmarking and advancing behavioral research in symbolic-to-spatial geometric reasoning.

Published as a conference paper at ICLR 2026
GEOGRAMBENCH: BENCHMARKING THE GEOMETRIC PROGRAM REASONING IN MODERN LLMS
Shixian Luo"i, Zezhou Zhu"t, Yu Yuan Yuncheng Yang', Lianlei Shan', Yong Wu
Li Auto Inc. wuyong5e www.ixix.g.....m com
ABSTRACT
Geometric spatial reasoning forms the foundation of many applications in artificial intelligence, yet the ability of large language models (LLMs) to operate over geo- metric spatial information expressed in procedural code remains underexplored. In this paper, we address this gap by formalizing the Program-to-Ger task, which challenges models to translate programmatic drawing code into ac- curate and abstract geometric reasoning. To evaluate this capability, we present GeoGramBench, benchmark of 500 carefully refined problems organized by a tailored three-level taxonomy that considers geometric complexity rather than traditional mathematical reasoning complexity. Our comprehensive evaluation 19 frontier LLMs reveals consistent and pronounced deficiencies: even the most advanced models achieve less than 50% accuracy the highest abstrac- tion level. By systematically analyzing model behaviors, our study exposes key limitations in program-driven spatial reasoning and positions GeoGramBench an important resource for benchmarking and advancing behavioral research in symbolic-to-spatial geometric reasoning. Project page: https://gihub.com//iAuuu DSR/GeoGrraBennch
ry
INTRODUCTION
Spatial reasoning is fundamental both human cognition and artificial intelligence, supporting applications ranging from robotics and autonomous navigation to automated design (Davis et al. 2011). With the rise of large language models (LLMs), interest has grown in evaluating their ability to interpret geometric transformations and spatial relations in complex environments (Yang et al.
2024b Tang al. 2025).
Mathematical geometric spatial reasoming is specialized subdomain of spatial reasoning, requiring models to comprehend intricate geometrie relationships and perform deep spatial reasoning. Re- searchers have recently developed multiple benchmarks including Mathverse (Zhang et al. (024b). GeoSense (Xu et al.) 2025), and Euclid (Zhang et al.) 2024a) to assess LLMs' capabilities visual geometry comprehension. Another emerging direction leverages procedural geometric code, such as Asymptote code, as a symbolic and structured interface for expressing geometry problems and probing spatial reasoning. While some existing benchmarks (e.g.. AIME24 (MAA 2025). MATH-500 (Zhang et al.) 2024b)) include subsets containing Asymptote code, there is a lack of systematic, dedicated benchmarks specifically designed to evaluate LLMs' ability to perform program- driven spatial geo- metric reasoning. In this work, we formalize this unique setting as the P tnnerram-t--eeee ry task, referring the translation and abstraction process from procedural code internal spatial representations.
Preliminary studies (Muennighoff' al.) 2025) have shown that current LLMs struggle bridge procedural geometry code to spatial reasoning. We expanded these investigations on broader range of models further corroborate these observations, confirming this pronounced deficiency. For example, as shown in Figure [ advanced models such as DeepSeek-RI (Guo et al. 2025) suffer substantial drops accuracy: 23.5% AIME24 and 10.9% MATH-500, when transitioning from text-only problems
"Equal contribution. Cooresponning author.

---

Published as a conference paper at ICLR 2026
(Pr) those with embedded procedural code (Prc). Similar trends are observed for models such as GPT-ol (Jaech et al. (024) and QwQ-32B (Team 2025a), collectively indicating critical limitations in their ability to construct reliable spatial representations from symbolic code. Furthermore, recent work (Albalak et al. 2025) has highlighted the need to explore P roa-taam----aee abstraction promising and under-investigated research direction.
spatial
Motivated by these findings, we introduce GeoGramBench, a dataset of 500 curated problems incorporating programmatic drawing code, designed to systematically assess both spatial-geometric abstraction capabilities and mathematical reasoning in LLMs. Our proposed taxonomy organizes problems into three categories: Primitive Recognition. Local Relation Composition, and Global Abstract Integration, based on the geometric complexity encoded in procedural code rather than traditional reasoning difficulty. Evaluation of 19 frontier L.MMs reveals that even models such as the reasoning- massoning-oreented GPT-ol achieve less than 50% accuracy on the most challenging level, underscoring the unique difficulty of this task and the urgent need for advances in spatial-reasoning model design.
This work makes the following contributions:
We formalize the P Program-to-Geome
translation task as critical and underex- plored capability for LLMs, encompassing not only the interpretation of procedural drawing code but also the downstream geometric reasoning it enables.
We present GeoGramBench, a rigorously curated benchmark of 500 geometry problems with explicit procedural code, organized by three-level laxonomy that enables comprehen- sive and fine-grained assessment of
toottoo-eGrrry ry competence.
We conduct an extensive evaluation of 19 models, providing accuracy metries and detailed behavior analyses aligned with our research questions. Our results highlight persistent weaknesses in geometric program reasoning, establishing GeoGramBench as novel evalua- tion axis and fostering future advancements in spatially-rrounded,, grounded, symbolically-rie model training and analysis.
Profiem onc yne of AXBC it sparn contieters 837 anout
Aoeel (UT, CMPSS -GLtt-ode)
14
LLaINI
$12
laeel 919-1-1--131
Anver:
(a) Example of problem from (h) Accuracy comparison of models (c) Accuracy comparison of models on Pro in MATH-500.
on Py vs. Pro AIME24.
Pr vs. Pre MATH-500.
Figure 1: Overview and performance analysis text-only (PP) and text+code (Prc) geometry problems, (a) The procedural code wrapped with [asy] [/asy] and its geometric figure is visualized to facilitate understanding. (b) and (c) show accuracy comparisons of models Pr
and Pre subsets in AIME24 (Prcl = 5, |rrl = 25) and MATH-500 (PPrel = 42, [Pr| = 458), respectively, In both benchmarks, accuracy consistently drops for problems with procedural code.
2 RELATED WORKS
Visual Geometric Perception To study visual geometric reasoning, several benchmarks such as Euclid ZZhagg et al., 2024a), MM-MMtth (Sun al.) 2024). GeoSense (Xu et al. (2025). Math- Verse (Zhang et al. 2024bb. and Math (Lu et al. (023) have been introduced, each incorporating visual geometric content. These datasets measure large multi modal models' comprehension of visual geometric concepts and their handling of mathematical problems with visual components. Their focus is mainly on diagram interpretation rather than procedural geometric code understanding, which represents different but equally important aspect of geometric spatial reasoning.
Mathematical Reasoning Benchmarks diverse array of benchmarks has been developed to evaluate the mathematical reasoning abilities of large language models (LLMs).. Datasets such as
2

---

Published as a conference paper at ICLR 2026
GSM8K Cobbe et al. 2024), Minerva-MATH (Lewkowycz et al. 2022). CollegeMath (Tang et al.) 2024). MMLU-
2021), MATH-500 Lightman et al.) 2023). Oly OlympiaBBench (He al.
STEM Henddrycks al. 2020). and AIME24
[MAA] 2025 primarily focus algebraic, arithmetic, and word-problem reasoning. Many of these benchmarks target complex multi-step solutions, ranging from advanced high school mathematics the level of international mathematical olympiads.
Symbolic Graphics Perception and Generation A growing line of work targets the symbolic understanding and manipulation of SVG code for LLMs. Benchmarks such as SGP-Bench (Qiu et al. 2024), SVG Taxonomy (Xu & Wall 2024), and SVGenius (Chen et al. (025a) focus SVG code parsing and perception, probing models' abilities to interpret, understand, and reason about symbolic vector graphies. Other efforts, such
Gen-Gehnench (Chen al. 2025b), provide benchmarks for evaluating models' capabilities in SVG code generation, while methods like Chat2SVG (Wu al. 2025) propose approaches for improving SVG generation from natural language instructions. Together, these works lay important foundations for assessing both the comprehension and generative aspects of symbolic graphics in LLMs. However, there is still a lack of benchmarks specifically designed to test the construction of geometric spatial structures and deeper reasoning within geometry, which is the focus of GeoGramBench.
3 PROGRAM-TO-GGEMETTE
3.1 TASK DEFINITION
We define Soogrramtt--eoneee
as the task in which a model interprets procedural code to construct mathematical geometric representations, and subsequently reasons over these represen- tations to solve geometry problems. This paradigm provides a comprehensive assessment of two fundamental capabilities: (a) the ability accurately construct mathematical geometric diagrams from symbolic instructions, and (b) the ability to perform spatial reasoning and mathematical problem solving based on these constructed diagrams.
3.2 TAXONOMY
An effective laxonomy is critical for evaluating benchmark quality and pinpointing capability bot- tlenecks in the Frogram-to-Genne ry task, which requires translating procedural geometric code into diagrammatic understanding. Traditional taxonomies, such as those based on topological complexity (Zhou al.) 2025), logical intricacy (Lin et al., (025), reasoning difficulty (e.g., high school to olympiad levels) (MAA) 2025 Sun et al. 2025 Hendrycks et al. 2021). focus on reasoning steps rather than the geometric structures central to this task. To address this, we propose new taxonomy tailored to the P Progr am- com-too -o--emee zy task, defined by the geometric complexity of diagrams derived from procedural code. This three-level hierarchy reflects the types and number of geometric elements:
Primitive Recognition (Primitive): Problems involving procedural code that specify only one or two geometric primitives (e.g., points, lines, arcs, circles, polygons), focusing on basic mathematical properties such as length, area, angle.
Local Relation Composition (Compesitional); Problems with multiple local geometric elements, requiring the recognition, integration, and composition of spatial relationships among subcomponents of the diagram.
Global Abstract Integration (Abstract): Items demanding spatial direction, parameterization, recursion, 3D objects, composite structures, or advanced geometric operations (e.g., rotation, folding, projection), thus requiring not only the construction of complex diagrams but also global and stepwise spatial reasoning across the entire configuration.
To validate this taxonomy, analyzed the QwQ-32B (Tam 2025a) performance on the MATH- 500 (Lightman et al. 2023) dataset, comparing accuracy across reasoning complexity (per MATH- 500 annotations) and geometry complexity for text-only (Pr) and text+code (Prc) problems (see Figure 2 For Pr, accuracy decreases with increasing reasoning complexity, consistent with existing benchmarks. In contrast, for Prc, accuracy is largely independent of reasoning complexity, with
3

---

Published as a conference paper at ICLR 2026
significant drops tied to geometric intricacy. This trend, validated by clear accuracy decline on MATH-500 as geometric complexity increases, confirms that geometric complexity, rather than reasoning steps, is the primary challenge in this task. Thus, our taxonomy provides a robust framework for evaluating model capabilities Program-to-ae -o-ommtttt task.
3.3 TASK DATA FORMAT
Each problem consists of two parts: textual description and geometric draw- ing code, such as written Asymptote or Matplotlib style. The model receives both the text and the code as input. The target is numeric answer, which may represent a length, area, volume, an- gle, ratio, or count. This setup requires the model to parse the drawing code, form internal geometric representa- tion, and reason through the mathemati- cal question to produce the correct numeric solution.
180
230
16.5
Leva-2
Levi-44
Priccte
Canpetikeal
Altrac
Figure 2: Effect of reasoning and geometric complexity.
3.4 RESEARCH QUESTIONS
Understanding the capabilities of large language models (LLMs) in the Program-to-Geommes task requires a systematic investigation into their ability to process and reason over procedural geometric code. This task demands sequence of essential, hierarchically related skills: from recognizing basic geometric elements to composing complex spatial configurations and leveraging chain-of-thought (CoT) reasoning enhance problem-solving. Investigating these capabilities is motivated by the need to identify specific bottlenecks in LLMs performance, guiding the development of models better equipped for spatial reasoning applications. Based on this task definition and taxonomy, we articulate the following research questions to structure our analysis of LLMs' behavior in the Program-to-Gerr
task:
Is there evidence that LLMs can understand and represent basic geometric elements from program code?
RQI:
RQ2:
How effectively can LLMs compose and abstract geometric elements into coherent spatial configurations as specified by program code?
RQ3: How does CoT reasoning influence LLMs' spatial geometric reasoning abilities with program code?
These research questions are fundamental evaluating the progression of LLMs capabilities in this domain, and their investigation is critical for advancing our understanding of how to enhance symbolic-to-spaal spatial reasoning future models.
BENCHMARK CONSTRUCTION
In this section, we present the systematic construction process of GeoGramBench, a dedicated benchmark for Program-tar -t- -orrrmmttt-----rtrrrrr--- reasoning. We first introduce critical challenge inherent this task domain, namely answer leakage, before detailing our comprehensive data construction pipeline that forms the foundation of our benchmark (more details in Appendix
4.1 ANSWER LEAKAGE CHALLENGES
In the Program-to--eomee leakage within the code itself. The program code that generates geometric figures often contains precise numerical specifications that directly or indirectly reveal the answers sought. Benchmark like Math-500 (Lightman al. 2023), discovered numerous instances where answers were
task, a significant challenge arises from the potential for answer
4

---

Published as a conference paper at ICLR 2026
Proble: ..Mat inches, of Mack bar?
Problem: Calorite
bounding rectangle.
[war]
[oay mol &
(16227)02277
(8.1)(003)
(2r -ontethnn- (20)
poi C 15x---22 482) (0.000 Cy- ((we))
(0277)) [Jass]
do 11.)) (lay)]
Arswer:
2.7
Answer:
Armwer
Arswer: 04-443
Figure 3: Illustration of two types of answer leakage in procedural code, highlighted in red. On the left is direct leakage, where the answer is explicitly given by a coordinate value in the Asymptote code; in this case, we rescale the coordinates to preserve the geometric shape. On the right is indirect leakage, where the answer can be computed from code parameters; here, we modify the procedural code to mask such critical information.
directly embedded in the Asymptote code. Similar issues persist across various open- geometry problem collections we collected. As illustrated in Figure 3 we categorize two types of answer leakage the procedural code. Direct leakage occurs when the answer explicitly encoded as a coordinate value in the Asymptote code (c.g.. circle's s radius segment's length). Indirect leakage occurs when the answer can be computed from code parameters or formulas.
4.2 COLLECTION AND PREPROCESSING
We first aggregated approximately 905K candidate problems from three open-source mathemat- ics datasets, including NuminaMath- 1.5 (Li et al., 2024), HARP (Yue et al. 2024), and Omni- MATH (Gao al. 2024). with focus on sources rich geometry content. We filtered for problems containing embedded Asymptote code by searching for [asy] and [/asy] J tags, resulting in subset comprising about 1% (9,260 problems). We then deduplicated this subset using an n-gram (n = 8) similarity approach (Muennighoff et al., (025). reducing the set to 1,782 unique items. Finally, by following the schema from si (Muennighoff al. (225)) and leveraging GPT-4o (Hurst et al. 2024) for prompt-based classification, we selected only geometry problems, yielding 1,247 geometry-focused items for subsequent curation.
4.3 HUMAN REFINEMENT AND VERIFICATION
To ensure data quality and suitability for geometry code understanding tasks, we implemented two-stage manual verification process, conducted by a team of four experts (each holding a master's degree or higher in mathematics or related fields). The first round aimed standardize problem types and formats, while the second round focused on enhancing overall problem quality.
In the first round, we performed initial screening and format normalization: (a) non-relevant questions (such as hyperlink chains, multi-part items, and proofs) were filtered out according to best practices from BigMath (Albalak et al. 2025);; (b) convertible multiple-choice questions were transformed into open-form computation problems by removing options, while those not amenable to conversion were discarded entirely; and (c) answers were standardized into consistent 18TμX format. At the end of this screening, 547 candidate problems remained.
In the second round, we implemented rigorous three-pronged refinement process to improve problem quality:
Decontamination: To minimize community-sounn sourced contamination, we systematically re- vised problem statements by removing redundant descriptive information that might enable direct textual inference. Additionally, we adjusted problem conditions and modified cor- responding answers maintain mathematical consistency. Furthermore, we adjusted the answer requirements (such as replacing queries about lengths with those about area, volume, ratios) to further reduce the risk of leakage and promote authentic geometric reasoning.
Answer Leakage Prevention: As detailed in Section 4.1 to address this task-specific vul- nerability, we implemented two targeted strategies: systematically rescaling coordinates while preserving geometric relationships for direct leakage, and modifying masking code parameters for indirect leakage. These interventions ensure that answers cannot be derived through mere code inspection (see Figure
5

---

Published as a conference paper at ICLR 2026
Aceuracy Verification: Each answer manually checked for correctness; items with ambiguous, unverifiable, or doubtful solutions were removed.
Through this thorough process. we ultimately obtained 392 high-quality, contamination free geometry problems for augmentation and evaluation.
Primzive Recognition
Local Reatiion Compostion
Global Absraatt Integration
Prodem: -ARC, rupoose ot = B.A.,
Feires
Problem: The patton in figure below continues insond infieidy The the larget miangl Find shudol arc.
value
and weach that de 5. 3.5
boyl Amerian
Find the are of 2272. pasf
[ro]
C=(10.6)
pai Antous)) Bin 5y,0 o Xworggn.
dasw Co
emaent
labal $44 A. SWT
label: 585 SEJ:
label SCS N):
label"
miggo.0)- (1.00)))(--)------- (K.))-(-60))
-oydl, blut:
-yyde whoel:
[aay)]
[Faee]
vinualax
vasalize
Asswer: 122
Nuswor:
Predle: Glder odeeiiio snwess howe on sepect onio of 4::., maaring the with the height s-4 to 3... bey)
Probiam: Square ABCD his sidk leucth 4, and jos]
The ownwrr of the fanm the right metangiar primm chean beow are ronracted to form as actllorion. Whe is the vo solnn of thi aettiweros?
Problem:
rieranne
(0
Creda
[ar]]
-(811)(00.)
craw 41,4%) (irde (.T)):
gry
()
((95))((61))
dadwel): d0on((00000((5000)
Iake ( SMS [2 0yS): labo SPS (366144M) (/an))
[[axy]
vissllee
Asswer: 27
Arewer:
Ansuer:
10
Representative examples from GeoGramBench illustrating the three ascending Program-to-Geomet::: difficulty levels: Primitive Recognition, Local Relation Composition, and Global Abstract Integration. Each category is exemplified by sampled problems, highlighting the increasing spatial complexity and abstraction across levels.
Figure 4:
4.4 BENCHMARK AUGMENTATION
To enhance difficulty balance and problem diversity, supplemented GeoGramBench with additional items: 5 geometry problems
from AIME24 (MAA 2025). 42 from MATH- 500 (Lightman et al. 2023). and 61 geomet- ric problems adapted from Mathverse (Zhang et al. (024b). For the Mathverse subset, we selected representative solid geometry prob- lems and manually transcribed diagrams into matplotlib code to diversify the procedu- ral drawing code within the dataset. Our periments indicate minimal impact from the choice of drawing language (see Appendix A). Altogether, GeoGramBench comprises 500 ge- ometry problems, supporting robust evaluation across a variety of geometric phenomena.
Lxmera
Crant
GeoGramBnncs
Figure 5: Statistics of GeoGramBench.
4.5 DIFFICULTY AND SUBTYPE CATEGORIZATION
Building our theoretical and empirical insights in Section 3.2 we categorize all 500 GeoGram- Bench problems into three ascending difficulty levels: Primitive Recognition, Local Relation Com- position, and Global Abstract Integration, based the type and number of geometric elements and the spatial relationships involved (see Figure 4 The categorization is implemented through a combination of GPT-4o (Hurst al. 2024) assisted classification and thorough human expert review.

---

Published as a conference paper at ICLR 2026
The resulting distribution, detailed in Figure 5 establishes GeoGramBench as the largest and most diverse benchmark for the Progr www-tt---ommeeee task to date.
To facilitate deeper analysis of LLMs performance and failure modes, further categorize these problems based on problem-solving objectives, identifying common challenges across geometric properties. This subtype classification divides the three levels into six task types: angle, length, area, volume, ratio, and count, determined via manual annotation. Figure illustrate the distribution of GeoGramBench's 500 problems, highlighting the diverse representation of multiple task types within each category, including the introduction of volume tasks in the most complex level, enabling targeted investigation into specific reasoning difficulties (e.g.. 3D structures or angle computations) to guide future model improvements. Detailed subtype statistics and definitions are provided in Appendix C.8 underscoring GeoCGraBBench 's value as a diagnostic tool for evaluating L.LMs capabilities across this varied range of problem types.
5 EXPERIMENT
We benchmark 19 popular LLMs on GeoGramBench, providing broad comparative analysis in this section. Section details our evaluation framework and prompt engineering strategies. Section 5.2 introduces the tested models, followed by quantitative comparisons in Section 5.3
5.1 EVALUATION PROTOCOLS
For open-source models, we adopt the Luo et al. (2025) framework for evaluation, while for closed- source models, we utilize official APIs with identical prompt templates (Ler's 's think step by step and output the final answer within Vboxed(J.). All result parsing is standardized using [uo al. (2025), with assistance from GPT-4o when necessary. Each problem evaluated in zero-shot setting: the model input consists strictly of the problem text and the procedural geometry drawing code. For each problem instance, we sample 8 responses using temperature 0,6, and report final accuracy as the mean over these 8 outputs, which balances model stochasticity and answer reliability.
5.2 EVALUATION MODELS
We evaluate a total of 19 mainstream LLMs, including both proprietary APIs and leading open-source
systems. The closed-source models include GPT-5 (OpenAl, 2025). GPT-4o (Hurst et al.) 2024).
GPT-o3-mini (Piiserr& & Jud, 2025), the GPT-ol series (Jaech et al. 2024). and Gemini-Pro-I.5
(Team
et al. 2023). The open-source models cover wide range of scales, including DeepSeek-Ri - 1 Guo
al. 2025). ) DeepSeek-v -3-0324 (Liu et al. 2024), Qwen3 (Team] (025b) and QwQ-32B
Team
2025a), ), as well as other prominent models from 32B down to 1.5B parameters: DeepSeek-RR Distill
variants (Guo et al. 2025), Bespoke-Stratos
(Labs) 2025), s1.1-32B (Muennighoff et al.
2025
Sky-Tl-mini-7B Li ct al. 2025), and DeepScaleR-1.5B --55---reviww (Luo ct al. 2025).
5.3 MAIN RESULTS
Closed-source models outperform open-source models. Table I shows that the latest GPT-5 model achieves state-of-the-art performance, with an overall average accuracy of 75.01 %. GPT-5 also obtains the highest scores in both the Primitive and Compositional levels. Among open-source models, Qwen3-235B-THhe 1255B--hinking-2555
ranks second, surpassing earlier closed-source models like GPT-ol and GPT-o3-mini, with an average accuracy of 74.00%. GPT-ol and GPT-o3-mini follow, each with average accuracies around 70%.
All models perform below 50% in the Abstract level. As the geometric program complexity increases, accuracy drops sharply for all models on the Abstract level, with no model surpassing 50%. For example,
achieves 89.09% accuracy on the Primitive level and 79. 12%% the Compositional level, but only 49.05% on the Abstract level. Similarly, GPT-5 reaches 90.44% on the Primitive level and 84.59% on the Compositional level, but falls to just 39.26% on the Abstract level. This trend underscores significant limitations in current LLMs' ability to construct and reason about complex geometric figures and spatial relationships, particularly when deeper geometric abstraction is required.
7

---

Published as a conference paper at ICLR 2026
Most challenging subtypes: angle and volume. At the Primitive and Compositional levels, which focus on 2D geometric figures, the angle subtype is most challenging, as it requires the model to reconstruct and reason about implicit spatial relationships in 2D, which is difficult with only code input. In contrast, length or count subtypes are typically more straightforward, often solvable with direct algebraic or counting methods. At the Abstract level, dominated by 3D geometric figures, the most difficult subtypes shift to area and especially volume. Area and volume problems require complete 3D spatial understanding, making them especially difficult at the Abstract level. This means models struggle much more with these tasks. Angle problems can sometimes be simplified to 2D scenarios using vector or riangle-based techniques, so their relative difficulty is lower compared to area and volume. The key point is not that angle questions become simple, but that area and volume tasks 3D geometry present much greater challenge for current models. Thus, the stark accuracy drop for area and volume in the Abstract level reflects the greater demand for true 3D spatial integration and highlights the significant challenges that current models face in higher- digher-dmmensinall reasoning.
Madel
ALL
Ang
Angl
Teagth
Ane
Rato:
Cieri
Any.
Apple
Langt
Araa
Corn
Ang
Angl
Tzege
Ame
Wilme
Core
Closed-
Medzlk
53.49
00.31
97.50
02:34
15.47
$3.47
16,10
51.55
19.91
31.37
73.42
79.35
42.53
55.36
59.35
30.14
30.58
99.35
71.32
20.00
30.38
16.1
1003
88.61
61.78
36.38
62.50
79.85
68 16
96:33
81:91
34.%
63.31
63:08
05:00
04.64
47%
16
31.18
05.08
15:01
70.68
31.75
08.11
-6.43
38.25
21.36
9.75
23,00
21.08
4.51
5.52
2.10
25.00
4,50
21400
5181
Orue-wenu Medils
Owwns Trrking 2007
09.00
84:10
50.00
11
$0.07
01.15
19:13
66:75
$1.60
$3:21
100.1
50.05
09.65
60:21
65:35
41.54
31:48
10.00
08:33
76.00
DeraSad-tt 483
$4.08
73.35
30.30
07.08
13.13
06.93
10,33
19.05
74.32
12.35
41.35
50:11
35.35
24.30
00.35
58.33
09.17
Dhwent-130 130
14.84
17.50
81:11
10103
18.13
55.00
51.03
36.51
06.75
13.55
Derptink mottl Dove 37n
67 m1
30.00
$1.05
00:00
47.05
11.08
09.11
09.07
07
15.03
50.00
43
35.50
13.13
(810)
67:08
Sryee 1:30
72.30
02.50
41.30
55.13
41.05
17.02
29.57
26.25
5:97
15.28
46.53
15117
Derstask- (jooe-111
72.75
00.00
12.30
00:35
18.70
4103
15.11
$3.56
$3.08
33.05
41.03
15.13
31.25
34.10
52.10
Decabock Duttil (,
45.0
26.30
25
35/10
21.23
43.19
32.15
11.00
711
19.35
9.20
21.55
21.7
Table 1: Accuracy (%) of selected closed-source and open-souree LLMs on GeoGramBench across three difficulty levels. For each model, the lowest swwest-prffomming subtype within each level is highlighted with background color in bold.
), and for each subtype, the highest accuracy among all models shown
BEHAVIOR ANALYSIS OF LLMS
We address our RQs through both quantitative and qualitative analyses based on benchmarking results and model responses. Furthermore, we summarize several common failure patterns observed across different models.
RQI: Is there evidence that LLMs can understand and represent basic geometric elements from program code?
RQI investigates the fundamental ability of LLMS to recognize basic geometry elements, which can be quantitatively measured by the evaluation results of Primitive Recognition.
As shown in Table most of the models achieve 60% accuracy on the Primitive Recognition level, suggesting that they can effectively parse and build basic geometric scenes from procedural codes. Qualitatively, some of the model responses explicitly reveal the capability to interpret and reconstruct geometric information. As shown in Figure (6) models frequently examine the procedural code for geometry understanding: "Now, looking at the Asymptote code", "Let me parse the Asymptote code a bit", and "maybe should try to visualize this". They can also identify simple geometric relationships according to the procedural code. For example, "c is (2,0), so c/2 is (1,0). So the inner are is between points a/2 and c/2", and "path inner = arc(d, a/2, c/2, CW),...ptth outer = arc(d, c, a, CCW);" These behavior demonstrate that LLMs are intent and capable to map procedural code into internal geometric structures. In conclusion, modern LLMs are able to construct basic geometric representations from procedural code.
8

---

Published as a conference paper at ICLR 2026
How effectively can LLMs compose and abstract geometric elements into coherent spatial configurations as specified by program code?
RQ2:
RQ2 investigates LLMs' capability of the geometry composition and global representation abstraction. According to the results in Table [ all models experience a significant drop in accuracy from Compositional problems Global Abstract Integration. For example, GPT-ol drops from 76.12% to 44.67%, and DeepSeek-RI drops from 73.13% to 40.86%. These results indicate that current LLMs may lack of compositional and spatial abstraction ability solve complex geometry problems. Qualitatively, while models can often parse and assemble some local structures, small errors in local constructions frequently appear, preventing LLMs to construct a complete and coherent global representation, As illustrated in Figure G a model may read piece of code like "path inner arc(d, a/2, c/2, CW)" and reason about directions ("which would be the other direction compared to the inner counterclockwise path before"), but single mistake in local spatial assignment may generate downstream confusion: "maybe got the direction of the angle wrong?... the actual angle between the points is \theta, so the area calculations still hold.". This phenomena suggests that modern LLMs may not good at capturing complex compositional geometry relationships for high level spatial reasoning. In summary, although LLMs have made progress in local geometric parsing, their ability to synthesize and reason over globally consistent spatial structures in P Program- o Geoometry tasks remains limited.
RQ3: How does CoT reasoning inftuence LLMs' spatial geometric reasoning abilities with program code?
Qualitatively, while models frequently perform iterative self-reflection and verification of code ("Let me check again"), and repeatedly parse diagram instructions, their CoT trajectories rarely correct or update internal geometric understanding, as shown in Figure 6 For instance, the model may cycle through algebraic steps and verbalize uncertainty ("Hmm, this is a bit confusing without seeing the actual diagram. Since can't see the diagram, maybe should proceed with the information given."), "), yet consistently fails to resolve spatial relationships or integrate local shapes into a whole. This observation illustrates that CoT may lead LLMs fall into repetitive symbolic reasoning. Such repetitiveness does not beneficial for LLMs to construct high level spatial representations as a whole, even leading to confusion about complex geometry relationships.
Quantitatively, our experimental results show downward trend in accuracy as geometric complexity increases, consistent with our benchmark taxonomy's s emphasis on geometric intricacy rather than reasoning steps. This suggests that most LLMs exhibit weak internal geometric representation capabil- ities, and long CoT reasoning provides limited benefit for improving this aspect. To further investigate this. conducted quantitative experiment using Token Budget Forcing (BF) (Muennighot et al. 2025), detailed in Appendix [E] which extends CoT reasoning and provides a robust validation of its limitations. These findings highlight critical bottleneck: while CoT enhances mathematical reasoning in other domains, its effectiveness in teerrrattt-Geommee ry tasks is constrained by LLMs' weak spatial abstraction capabilities.
Common Failure Patterns Through extensive qualitative analysis of model responses, conducted by manually reviewing a substantial number of failure cases, we distilled several prevalent failure patterns that are shared across multiple LLMs. Due to the current lack of accurate automated assessment methods for this process, our analysis is based on representative examples rather than exhaustive annotation. Nevertheless, these patterns foreground critical limitations in current model behavior. (1) The models exhibit a pronounced preference for algebraic methods, frequently relying on coordinate calculations rather than leveraging geometric constructions, even when the latter would be significantly more efficient. This algebraic bias often results in the neglect of essential geometric heuristies. (2) L.MMs rarely introduce auxiliary lines or points, a classic strategy in geometry that can simplify complex problems, indicating limited flexibility in exploring alternative geometric solutions. (3) Models consistently struggle with instructions that involve spatial orientation, such as distinguishing between clockwise and counterclockwise directions interpreting vertical and horizontal references. These difficulties often lead to misinterpretation or incorrect reasoning steps. (4) A common issue is the confusion in mapping symbolic relationships (e.g., angle vertex labels) to their corresponding geometric elements in the diagram, reflecting an insufficient internal spatial representation and lack of robust grounding for abstract symbols.

---

Published as a conference paper at ICLR 2026
Geetier chaggee
Daos so
dapee,, time, soading poblee
FQJ: Conseruce Erdence
Ue (ie sudley ae1x
mie. Trme lage wa youk for doom gooe colly meme Be shaed rigort
that
coanplits ths makd metinaice to iove free sodo parsing quild
any we retie
dinciis kut dealley 0u41
CoLCEms (u0 dh syole
Hg2 Compostional Completeress
gring butles deaclis compane hew cornt dosiiiiiiee par betons aur this coinuting tock to
The mode donseitval dur Ancenoo enoo mupping sputial inegry.
sctcally pataus and lage
the meybe the right sirrr to ne to
stragding countandoooonn (CCW)
then
hevepees ood weter.
From
(809 Col Revoning
fiw fake fhr bolk tn So
Gaartall ane of lage secur fuom C. mee nendectouss minas the of aaader
monss se f-quetttoiing and eecrriiaiit of the
mutiiple rande
075 (Cw) ty ee reds tewe that ssms ae anigrbowent check spil dhemat only worke on the
Asyrppttt oode, wa oliinatlly unalle arriec the cornec approazh.
Figure Illustrative solution process generated by the QwQ-32B model on a Local Relation Composition problem. The model initially attempts to construct spatial representations from the provided code, then interprets geometric elements such as direction and region, exhibiting behavior aligned with all three research questions and chain-of offthoougtttbeeee based refinement. Multiple rounds of reflection and verification are observed, although these iterative steps do not consistently yield correct or fully integrated solutions.
local construction, compositional integration,
Overall, our behavior analysis offers an in-depth diagnosis of how contemporary LLMs process programmatic geometric information and where their reasoning strategies fall short. By system- atically linking model behaviors to distinct task attributes, we not only clarify the boundaries of current capabilities, but also map out the critical obstacles restricting progress in caabolic-to-spatiie reasoning.
7 CONCLUSION
In this work, we present GeoGramBench, the first large-scale benchmark for evaluating LLMs on the challenging Program-te soen-tto-Gemmeryy task, which connects procedural code with geometric reasoning. Our experiments on 19 leading models show that even the strongest models achieve less than 50% accuracy on complex problems, revealing critical gaps in symbolic to spatial understanding. Through a new taxonomy and comprehensive analysis, we identify major shortcomings and establish GeoGramBench as an important resource for advancing this research direction. We hope this benchmark will encourage further progress toward models with better spatial abstraction and symbolic reasoning, pushing forward the development of AI with stronger spatial intelligence.
ETHICS STATEMENT
This work introduces a mathematical reasoning benchmark and does not involve human subjects, personal data, sensitive information, or real-world deployments. The dataset is constructed purely from publicly available mathematical problem sources and procedural code, with rigorous data cleaning and verification to ensure research integrity. We are not aware of any ethical risks or concerns related to privacy, fairness, safety, discrimination, or legal compliance in this study.
REPRODUCIBILITY STATEMENT
The complete details of GeoGramBench dataset construction are provided in Section 4 and Ap- pendix C We include all evaluation code in the supplementary materials, along with detailed instructions for running experiments. For model evaluation, we access closed-source models via their official APIs, and open-source models are obtained from Hugging Face (httpss//uuggingffcceco// complying with their respective terms of use.
REFERENCES
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, et al. Big-math: A large-scale, high-quality
10

---

Published as a conference paper at ICLR 2026
math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502.17387, 2025.
Siqi Chen, Xinyu Dong, Haolei Xu, Xingyu Wu, Fei Tang, Hang Zhang, Yuchen Yan, Linjuan Wu, Wenqi Zhang, Guiyang Hou, al. Svgenius: Benchmarking Ilms svg understanding, editing and generation. In Proceedings of the 33rd ACM International Conference Multimedia, pp.
13299, 2025a.
Yamei Chen, Haoquan Zhang, Yangyi Huang, Zeju Qiu, Kaipeng Zhang, Yandong Wen, and Weiyang Liu. Symbolic graphies programming with large language models. arXiv preprint NrXii:2509.05208, 2025b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. Cornell Universit- arXiv, arXiv,Comell Comell University arXiv, Oct 2021.
Katie Davis, Joanna Christodoulou, Scott Seider, and Howard Earl Gardner. The theory of multiple intelligences. Davis, K., Christodoulou, J., Seider, S., & Gardner, H.(2011). The theory of multiple intelligences. In RJ Sternberg & SB Kaufmmnn (Eds.), Cambridge Handbook of Intelligence, pp. 485-503, 2011.
Bofei Gao, Feifan Song, The Yang. Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.1294. 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint 2000:24022.1408,, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint wrXiv:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXi::2410.21276 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card, arXiv preprint arX1v:2412.16720, 2024.
Bespoke Labs. Bespoke-stratee
The unreasonable effectiveness of reasoning distil- http://pww...s..oooooeooeeeeaaee
lation. https://www. wwwp://wwwwwool.bb...oommm www.bpppkel.b...illooo/epppppppp reasoning-dissillation. 2025. Accessed: 2025-01-22.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph Gonzalez, and lon Stoica. S*; Test time scaling for code generation. arXiv preprint arXiv:2502. 14382, 2025.
11

---

Published as a conference paper at ICLR 2026
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. The Twelfth International Conference Learning Reprexentations, 2023.
Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of Ilms for logical reasoning. arXiv preprint arXiv:2502.0/ 100, 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1rX1::2412.19437, 2024.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:23 2023.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. DeepScaleR: Surpassing ol -p-eeeve with a 1.5b model by scaling rl, 2025. URL http::/t///p/tttttttttt radio b75 nction. si te DeeDeepSSaale--Bae lec O1 wittrreewwwwww-hh-eeee 1- Model Poy-Scall ng - 1661 Tar.0066666000001006000600003310
MAA. American invitational mathematics examination- in american invitational mathematics
examination aime 2024, February 2025. URL/httts://maa org. competition
s http::/ww.......aaaaaaa.aaa.aaaa
aime
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501.19393 2025.
OpenAl. Gpt-5 system card, August 2025. URL htttp:://gpeaai.commiiiii tem-cards
5 5--sy
Rolf Pfister and Hansueli Jud. Understanding and benchmarking artificial intelligence: Openai's o3 is not agi. arXiv preprini arXiv:2501.07458. 2025.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z Xiao, Katherine M Collins, Joshua B Tenenbaum, Adrian Weller, Michael J Black, and Berhard Schölkopf. Can large language models understand symbolic graphics programs? arXiv preprint arXiv:2408.088: 00300,0.8333..83 2024.
Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, Lei Fang, and Ji-Rong Wen. Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models. arXiv preprint arXiv:2503.2/380, 2025.
Kai Sun, Yushi Bai, Ji Qi, Lei Hou, and Juanzi Li. Mm-math: Advancing multimodal math evaluation with process evaluation and fine-grained classification. arXiv preprint arXiv:2404.0509) 2024.
Kexian Tang, Junyao Gao, Yanhong Zeng, Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kaifeng Lyu, and Kai Chen. Lego-puzzles: How good are milms multi-step spatial reasoning? arXiv preprint arXiv:2503. 19990, 2025.
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02882 022884 2024.
Gemini Team, Rohan Anil. Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katic Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
12

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025a. URL htttps://qwen.m.. githhb. wttbl////ol/gge qwa- 32b/
Qwen Team. Qwen3 technical report, 2025b. URL http:pppp////ax.....oo//////2..2
Ronghuan Wu, Wanchao Su, and Jing Liao. Chat2svg:
Vector graphics generation with large language models and image diffusion models. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 23690-23700 2025.
Liangyu Xu, Yingxiu Zhao, Jingyun Wang, Yingyao Wang. Bu Pi, Chen Wang. Mingliang Zhang, Jihao Gu, Xiang Li, Xiaoyong Zhu, et al. Geosense: Evaluating identification and application of geometric principles in multimodal reasoning. arXiv preprint arXiv:2504. /2597, 2025.
Zhongzheng Xu and Emily Wall. Exploring the capability of Ilms in performing low-level visual analytic tasks on svg data visualizations. In 2024 IEEE Visualization and Visual Analytics (VIS), pp. 126-130. IEEE, 2024.
Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15Xiv:2412.15115, 2024a.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14177 2024b.
Albert S Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K Singh. Harp: A challenging duman-annotater math reasoning benchmark. arXiv preprint arXiv:2412.08819, 2024.
Jiarui Zhang, Ollie Liu, Tianyu Yu, Jinyi Hu, and Willie Neiswanger. Euclid: Supercharging multimodal Ilms with synthetic high-fidelity visual descriptions. arXiv preprint arXiv:2412.08737, 2024a.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Yu Qiao, al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186. Springer, 2024b.
Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your lims behave over infinitely increasing context length and reasoning complexity? arXiv preprint arXiv:2502.05252. 2025.
13

---

Published as a conference paper at ICLR 2026
A EFFECT OF DRAWING LANGUAGE ON PROGRAM-TO-OOE TO-GRMMTT- PERFORMANCE
key motivation for investigation is determine what extent challenges in Poogramtto-Gomeet
reasoning arise from the logic of geometrie construction itself, rather than from surface-level code syntax unfamiliarity with specific drawing languages. To test this, translated 5 geometry questions containing Asymptote code from AIME24 and 42 questions from MATH-500 into equivalent Python matplot code, holding geometric content constant while varying only the programmatic language. As shown in 7 QwQ-32B exhibits less than I % differ- ence in absolute accuracy between the Asymptote and Matplotlib versions on both benchmarks. This minimal gap provides strong evidence that the principal bottleneck in Program-3 Coem-ttt--emee...mmm task performance not due to the choice of drawing language, but rather stems from deeper difficul- ties in spatial abstraction and geometric reasoning from code. This result reinforces conclusion that surface syntax is not the main limiting factor for LLMs in this domain.
p
asy
ppll
7,C
T.C
90
86.5
80
Accuracy (%)
47.5
46 3
40
30
20
AIME24
MATIL-500
Figure 7: Comparison of QwQ-32B accuracy on equivalent geometry problems expressed Asymp- tote versus Matplotlib code (AIME24 and MATH-500), The negligible performance gap demonstrates that toogrammtt-Geome-ee ry capability is independent of drawing language syntax.
Example
Problem Statement:
Rectangles ABCD and F.GH are drawn such that D, F, C, F are collinear. Also, A, D, H. G all lie on a circle. If BC = 16, AB = 107, FG = 17, and FF 184, what is the length of CE?
G
C
B
F
A
Figure 8: Visualization generated from the drawing code
Drawing Code (Asymptote):
import graph;
unitsize (0.1cm)
pair A = (0, 0);
pair B (70,0);
pair C (70,16);
pair D (0,16);
14

---

Published as a conference paper at ICLR 2026
pair (3,16);
pair F (90,16);
pair G (90,33);
pair H = (3,33);
label ("\$A\$", A, 8);
label ("\SB\$", B, S);
label ("\$C\$", C, N);
label ("\$D\$", D, N);
label ("\SE\$", E, 8);
label ("\$F\$", F, S);
label ("\SG\S", G, N)
label (" H, N);
draw (E DE--D--A- -B- ---G--F------
Drawing Code (Matplotlib):
import matplotlib. pyplot plt
A (0, 0)
B = (70, 0)
(70, 16)
D = (0, 16)
(3, 16)
F = (90, 16)
(90, 33)
H = (3, 33)
for pt in [A, B, C, D, E, F, G, H]: plt.plot (pt[0), pt[l), 'ko')
plt.text (A[O), A[1]-1, "\SA\S", ha='center', va= vá='top', fontsize =13)
pet.text (B[O), B[1]-1, "(sBis", ha=' center' va= top', fontsize =13)
pet.text (C[0), C[1]+1, "\Scis", center' va bottom', fontsize-13)
plt.text (D[0), D[1]+1, "(SD\S", ha=' center' va= bottom',
fontsize=13)
plt.text (E[0), B[1]-1, "\sE\s", ha=' center' va= top', fontsize =13)
plt.text (F [0], F[1]-1, "(sFis", center' va= top', fontsize =13)
pet.text (G[O), G[1]+1, "\SG\s", ha-tcrr va bottom', fontsize=13)
plt.tex(([(0) (H[0), H[1]+1, "\SH>s", ha=/ center', va= bottom,, fontsize=13)
[tt.plot[[[[)), D[C], A[0], B[O), c(0), B[O]], [E[1], D[1), A [1], B[1], c(1), E[1]], color=/black
plt.plot ([E[0), H[C], G[0], F[0], c[o]], [E[I], H[1], G[1), F
[1], c[1]], color=/black")
plt-xxim(-5, 95)
plt.ylim(-S, 38)
plt.gea () .set _sse_aspecct ('equal)))
plt.axis ('off')
15

---

Published as a conference paper at ICLR 2026
plt. Bilighht_lluuuttt () plt. aat.show// ()
B PREVENTING INFORMATION LEAKAGE IN PROCEDURAL. GEOMETRY CODE
A critical aspect of dataset curation for
evaluation is the prevention of information leakage through the procedural drawing code. this context, information leakage refers to situations where the answer to geometry problem is either explicitly implicitly encoded in the program, enabling a model (or human) to bypass genuine geometric reasoning and instead extract the solution directly from code inspection.
We identify two primary forms of leakage:
Direct leakage: The answer appears explicitly in the code, for example as a coordinate, length, or parameter value (e.g., circle radius segment described directly in the Asymp- tote code).
Indirect leakage: The answer can be inferred by performing simple calculations extract- ing formula results from the parameters or structure of the code, even though it is not written verbatim.
To mitigate these risks, systematically reviewed all procedural code in the dataset. For direct leakage, critical coordinates and parameters are rescaled or randomized while preserving the diagram' structure. For indirect leakage, problem variables and code formulas modified masked to preclude simple reverse engineering of the answer.
Below present concrete examples comparing original and mitigated code for selected problems. Each example includes its problem statement and paired Asymptote code, annotated as "before" and "after" modification.
Example 1:
Problem Statement:
In AABC, point F' divides side AC' in the ratio 1 : 2. Let E' be the point of intersection of side BC and AG where the midpoint of BF. The length of EC divided by the length of BE is
Answer: 3
Before modification (Leakage present):
After modification (Leakage mitigated):
size (2. 5inch);
size (2. 5inch);
pair A, B, C, E, F, G;
pair A, B, C, E,
(0,3);
A = (0,3);
B = (-1,0);
B (-1,0);
(3,0);
C = (4,0);
= (0,0);
(0,0);
(1,2);
F = (1.14, 2.14);
intersectionpoint (
intersectionpoint (B--F, A--E)
);
draw Braw(A----C--yyall
draw (A--B--C--cycle))
(B--(B-FF))
draw (A--E); draw (B--F);
label (A,AAA'",,,
(abes(""""\\',,,
Labee()""SBS"",,"
(abBs("
label (\"$CS)",, ", C, (00))
label (\"$ES\",,,S"
label (\"$ES\(",E,""
label (100((()58)) $$F$\" "F,,NE)
label ",
label (A SSGS \..5,,\"",, SE);
label (\ "SGS\ G, SE);
Figure 9: Side-by-side comparison of Asymptote code: before (left) and after (right) information leakage mitigation.
16

---

Published as a conference paper at ICLR 2026
Example 2:
Problem Statement:
In rectangle ABCD, point M is the midpoint of AD. The area of LAMC is 12, and the length side AD.
AD
3.2 Find
AB
Answer: 8
Before modification (Leakage present):
After modification (Leakage mitigated):
size(4cm)
size (4cm); ;
draw( ((2
draw (0, -3,
-- (0) 0),
label (\"$A$\", (0,0), S
label ("SA$", (0,0), SW);
label (\"$BS\", (6, 0), SE);
label (\"SB5)", (3, 0), SE);
label (A (>"$C>>,, (6,8), NE);
Label (\"#C$\", (3, NE);
label (\ "SDS\", (0, 8), NW);
Label (\"$DS\", (0, 4), NW)
label (1 "MM (0, 4), W);
label (\ (>MMM (0, 2), W);
Figure 10: bide- comparison of Asymptote code: before (left) and after (right) information leakage mitigation.
C DETAILED BENCHMARK CURATION
We assemble team of four experts (each holding Master's degree higher in mathematics related fields) to ensure data quality. Our team manually verifies and refines samples from three aspects: question reformulation and standardization, decontamination, answer verification and leakage prevention.
C.1 INTRODUCTION TO ASYMPTOTE CODE
Asymptote is vector graphics language designed for technical drawing and geometric visualization. "Asymptote code" refers to structured source code written the Asymptote language, which describes geometric primitives (such as points, lines, circles, polygons, 2D and 3D shapes), spatial relationships. and drawing instructions. The Asymptote syntax supports mathematical expressions, procedural constructs, and graphical commands, enabling precise programmatic construction of geometric diagrams.
C.2 STANDARDIZATION OF PROCEDURAL CODE
Our standardization pipeline for procedural code involves the following key steps:
1. Initial Filtering:
We collect a large set of mathematical problems and filter for those containing embedded Asymptote code by searching for [asy] and [/asy] tags. This step isolates problems that include geometry drawing code.
2. Syntax Validation: We use the Asymptote tool to check for syntax correctness in each drawing code snippet. Problems with invalid syntax or unrenderable diagrams are excluded.
3. Diagram Verification and Adjustment: We confirm that each code snippet produces valid and clear geometric diagram in Asymptote. If necessary, we adjust canvas size and proportions to avoid visual overlap, ambiguity, or unclear geometric elements.
4. Answer Leakage Prevention: As described in Section 4.1] we systematically review each code segment for direct or indirect answer leakage. Direct leakage (where answers appear explicitly the code) is handled by rescaling coordinates or revising geometry. Indirect leakage (where answers or modifying the relevant parts of the code to ensure the answer cannot be obtained by inspection alone.
be inferred from code parameters) is prevented by masking
17

---

Published as a conference paper at ICLR 2026
C.3 QUESTION REFORMULATION AND ANSWER STANDARDIZATION
Question reformulation The formulation of each sample in GeoGramBench should be simple QA pairs for convenient evaluation. To achieve this, we start to deal with multiple choice questions, proof- based questions and multi-part part problems, which are not in QA format. Multiple choice questions can be transformed into open-ended computation problems by preserving the correct choice as the answer and removing all other choices. Some of the proof-based questions can be transformed into computation problems (like "Prove that PA = 4P can be rewrite to "Compute the ratio between PA and PB"), whereas others are not suitable for such transformation (like "Prove that AB 2 3PR). Multi-part problem always consists of several sub-problems which can be simplified into a single question format by retaining of the computable sub-questions. Questions amenable to conversion can be retained and reformulated into new QA samples, while others may be excluded from the benchmark. According the aforementioned rules, our team members carefully assess the formulation of each question and perform corresponding modifications and deletion.
Answer standardization Considering the diversity and complexity of mathematical expressions, answer standardization is crucial for accurately evaluating model-generanl responses. Our team manually modify the answer of each question by removing arithmetic operators (like +, -), letters and characters that irrelevant for computation and evaluation (like text cm²2)) )). and standardize each answer into KTTX format as simple as possible (like simplify \frac (28) (Nsqrt
to 4 Naqrt ( (7)). The above operations successfully ensure the consistency of question formulation and answer standardization, which benefits subsequent data processing and contributes reliable benchmarking. The resulting subset contains 547 candidate samples.
C.4 DECONTA MINATION
Most of the samples we collected originates from public datasets and internet resources, which indicates a high possibility that these data has already been included in the LLMs' pre-training corpora. Besides, current data samples contains certain degree of redundancy and unnecessary information, which may introduce unexpected bias to benchmarking. To mitigate the above influences as much as possible, our team manually perform data decontamination for all the 547 samples from three aspects:
Extraneous information removal We believe hyperlinks and code comments are not only un- necessary information for mathematic geometry spatial reasoning, but also introduce text bias for mathematic geometry problem reasoning. As a result, each member in our team carefully examine and delete all these contents in each question;
Problem statement rephrasing
To prevent samples from being solved solely based question statement, encourage LLMS focus on mathematic geometry spatial reasoning, we reduce some comprehensive and specific mathematical expressions in question text. To minimize the overlap between LLMs' pre-training corpora and benchmarking samples, our team modifies the given condition and question objective of some samples;
Coordinate modification In some samples, the coordinates used to generate pictures are identical to the given conditions in the problem statement, which may enable LLMs to derive answer through algebraic geometry reasoning based on text solely. Such problem solving approach cannot effectively evaluate the mathematic geometry spatial reasoning ability of L.LMs. To decrease the possibility of LLMS using algebraic geometry problem solving approach, we adjust the coordinates in each samples program code, which maintains the geometric shape and relationship of the original picture. The above decontamination methods ensures each item in GeoGramBench is a completely new sample, contributing to valuable and reliable mathematic geometry spatial reasoning benchmarking.
C.5 ANSWER VERIFICATION AND LEAKAGE PREVENTION
Answer verification We observe that some of the original answers are wrong to the corresponding questions after decontamination. To avoid such circumstances, we carefully verify the answer of each sample one by one by both referencing the original question from the Internet and calculate answer by ourselves. The QA pairs that cannot be searched on the Internet are removed.
18

---

Published as a conference paper at ICLR 2026
Answer leakage prevention We find some of the correct of samples during verification. As shown Figure 9 [0 the answer can explicitly equals to the answer, implicitly computed according to the code for generating image. This situation may allow L.MMs access the answer advance, which harm to the evaluation of mathematic geometry spatial reasoning. To prevent answer leakage, our team manually revised the code for all samples once again by rescaling coordinates and masking codes with numbers. Answer verification and leakage prevention guarantee the correctness of all the samples and the fairesss of benchmarking.
already leaked in the code
After human verification and refinement, we ultimately obtained 392 high-quality, contamination-free geometry problems for later augmentation and evaluation.
C.6 AUGMENTATION
We introduce additional samples enhance difficulty and diversity of GeoGramBench: 5 geometry problems from AIME24 MMA (2023) 42 from MATH-500 Lightman et al. (2023), and 61 geometric problems adapted from Mathverse Zhang al. (2024b). The 47 samples from AIME24 and MATH- 500 are retained without modification dur to their high quality. For the Mathverse subset, we first filter 119 samples with two key words: Vision Intensive and Solid Geometry, These samples focus on solid geometry questions, with the majority of problem solving information presented in image. This advantages makes them highly suitable for mathematic geometry spatial reasoning evaluation. However, Mathverse only provides the original images without the plotting code for reproducing the picture. Thus, our team decide to write python matplotlib code with our own to construct new evaluation samples in GemGramBench. Notably, we do not ask for multimodal models (like GPT-4o) for help because such models performs poorly when transforming solid geometry picture to matplotlib code.
Altogether, GeoGramBench comprises 500 hand-crafted geometry problems, which contributes to valuable and reliable mathematic geometry spatial reasoning evaluation.
C.7 TAXONOMY CLASSIFICATION PROMPT DETAILS
In constructing the GeoGramBench taxonomy, categorized all 500 problems into three ascending difficulty levels: Primitive Recognition, Local Relation Composition, and Global Abstract Integration, based primarily the geometric and spatial complexity of each problem. This classification process was conducted through a combination of large language model (GPT-40) assisted clustering and meticulous human expert correction. The initial clustering enabled an efficient, scalable filtering of geometry problems, while human review ensured rigor, consistency, and alignment with the intended definitions of each difficulty level.
To ensure reproducibility and transparency, we provide below the actual prompt used in the taxonomy assignment stage:
19

---

Published as a conference paper at ICLR 2026
Given geometry problem and its drawing code of diagram:
There are three categories of geometry problems:
1. Primitive Recognition
The asy diagram/code contains very few geometric elements (e.g.. one or two basic shapes, or minimal labeled points/lines).
The solution can be reached with direct observation or a single basic calculation; no significant composition, auxiliary constructions, synthesis are required.
Tests only clementary recognition or reading from the diagram.
2. Local Relation Composition
The asy diagram/code includes multiple geometric elements (points, lines, circles, polygons, etc.) combined in finite and explicitly described way. -hhe solution requires synthesizing, coordinating, or combining several local relationships, auxiliary constructions, or properties. The process involves several steps, but remains within standard 2D geometry.
The primary challenge is combining and reasoning locally among elements shown in the diagram.
3. Global Abstract Integration
Thee asy diagram/code may be complex, recursive, or defined by folding, projection, 3D arrangement, or abstract/global spatial processes.
The solution needs global synthesis: either full configuration analysis, recursive processes. or 3D/limiteettreme configuration reasoning.
Tests the model's ability to reconstruct and reason about a highly integrated or abstract global geometrie structure.
Instructions:
Classify the problem into one category: Primitive Recognition, Local Relation Composi- tion, Global Abstract Integration.
2. For geometric elements, consider only what is explicit in the asy code. Judge the solution/reasoningg requirement based on the problem's 's actual goal and what conceptual/coopptttionna effort is needed to reach the answer. Briefly justify your classification: refer relevant features in the diagram and in the problem's 's required reasoning process.
Output format:
Category: [Primitive Recognition Local Relation Composition Global Abstract Integration]
Justification: [A short explanation, citing relevant diagram elements and the level of reasoning/effort required.] required
C.8 SUBTYPE DISTRIBUTION AND DEFINITION
To provide more granular analysis of geometry problem-solving, GeoGramBench includes six distinct task subtypes: Angle, Length, Area, Volume, Ratio, and Count. Each subtype captures different aspects of mathematical reasoning:
Angle: Problems that require determining unknown angles.
Length: Problems involving the calculation or comparison of lengths line segments, perimeters, distances between points.
Area: Tasks focused on finding the area of various geometric shapes or regions (triangles, circles, polygons, composite shapes).
Volume: Problems dedicated to computing the volume of three-dimensional objects such as cubes, spheres, prisms, or their composites.
Ratio: Questions centered on the proportional relationships among lengths. areas, or other geometric quantities, often requiring understanding of similarity, scale, or division.
Count: Problems that entail counting geometric objects or features, such as the number of sides, vertices, faces, qualifying structures within diagram.
20

---

Published as a conference paper at ICLR 2026
Table 2 presents the detailed distribution of these subtypes across the three levels of GeoGramBench. Primitive, Compositional, and Abstract. This diverse coverage facilitates diagnostic evaluation of LLMs' performance across a wide range of geometric reasoning skills.
Table 2: Distribution of GeoGramBench problems by subtype within each complexity category.
Subtype
Primitive
Compositional
Abstract
Angle
22
20
Length
25
88
20
Area
26
89
46
Ratio
14
51
4
Count
15
31
15
Volume
27
Total
102
279
119
D IMPACT OF DOMAIN-SPECIFI DATA ON LLMs' PERFORMANCE
To address the potential influence of data scarcity LLMs' performance in the task, we conducted data ablation experiments to systematically evaluate the impact of adding domain-specific examples from GeoGramBench. These experiments aim to clarify whether the observed performance gaps are primarily due to a lack training data or inher- modeling limitations, while also assessing the benchmark's effectiveness in enhancing spatial reasoning capabilities.
Experimental Setup The baseline model is s1.1-32B (Muennighoff et al. 2025), fine-tuned via supervised learning the s1k dataset (a collection of 1,000 general reasoning examples designed to enhance long chain-of tf-ouhh (CoT) capabilities). Given the limited availability high-quality programmatic geometry data, we partitioned GeoGramBench into test of 200 problems (spanning the Primitive, Compositional, and Abstract levels) and a training set of 300 problems. The training set was enriched with detailed reasoning chains distilled from DeepSeck- R1 (Guo al. (025), creating a high-quality distillation dataset for fine-tuning, We performed ablation studies by incrementally adding 50, 100, 150, and 300 code-based geometry training samples from the GeoGramBench training set to the original s1k data. The combined dataset was used for fine-tuning, following the protocol from (Muennighoff et al. 2025).
Table 3: Data ablation results on 1-3BB fine-tuned with increasing GeoGramBench samples added to s1k. Accuracies (%) are reported 200-problem test subset. A denotes improvement over the baseline (Exp. 1). Bold highlights the best per level.
Exp.
# Added Samples
Primitive
Compositional
Abstract
Avg.
1
59.38
46.00
15.80
38.18
2 50
57.99
49.61
16.18
39.80
+1.62
3 100
61.22
49.66
18.62
41.20
+3.02
4 150
61.11
50.90
18.38
41.73
+3.55
5
300
61.11
51.42
17.83
41.79
+3.60
Experimental Analysis Adding domain-specifi data from GeoGramBench leads to clear im- provement in accuracy. Comparing Exp. 3 (100 samples) to the baseline (Exp. 1). the average aceu- racy increases by 3.02 percentage points, with notable gains across all levels-Primitive (+1.84%). Compositional (+3.66%), and Abstract (+2.82%). This demonstrates that exposure to task-specific examples enhances the model's ability resolve ambiguities in geometric code, improving both logical reasoning and internal geometric representations. However, the gains plateau with further data increases. Tripling the added samples from 100 (Exp. 3) to 300 (Exp. 5) yields only marginal additional improvement of 0.59 percentage points (3.60% vs. 3.02%), with Abstract accuracy even declining slightly from 18.62% to 17.83%. This suggests that while initial data exposure mitigates
21

---

Published as a conference paper at ICLR 2026
unfamiliarity, the model's performance is ultimately constrained by intrinsic weaknesses in geometric spatial representation, which additional data alone cannot fully address.
Implications These results validate GeoGramBench effectiveness resource for improving tasks, with as few as 100 in-domain examples
LLMs' performance on yielding significant benefits. The benchmark's potential generalizability other spatial reasoning tasks is also suggested, given the diverse geometric properties (e.g.. angle, volume) it encompasses. However, the plateau effect highlights the need for architectural training advancements beyond data augmentation to enhance spatial abstraction, positioning GeoGramBench as a valuable testbed for future research in this area.
E TOKEN BUDGET FORCING EXPERIMENT
To deepen understanding of CoT's influence symbolic-to-spatial geometric reasoning. conducted quantitative experiment using Token Budget Forcing (BF) (Muennighoff et al. 2025). technique that extends CoTT by appending "Wait" tokens (N-Ignore) method has previously improved the sl model's performance on AIME24 (MAA 2025) from 50.0% 56. 66,7% prompting its application on the s1. 1-IBB model (Muennighoff et al. 2025), a variant of Owen2.5-33B-Instruu.
delay conclusion. This
the Program wwwaaamttto--eeeeeeeeeeemeee Geomet tr ry task. We implemented BF
(Yang et al.
2024a) optimized for long CoT reasoning, evaluated across GeoGramBench.
Table 4 presents the results, showing accuracy and token counts with N-Ignore values of 0 (baseline), 4, and The experiment extended the CoT length significantly. For example, setting N-Ignore to increased the token count by 77.4% to 18,710, yet yielded only marginal accuracy gains, peaking 54.90% with N-Ignore values 1, 2, and This represents only a 0.30% improvement over the baseline of 54.60%, before accuracy declined slightly at N=6. Per-level analysis reveals modest improvements (e.g.. +0.53% Compositional at N=4), with Abstract showing minimal change (+0.31% % at N=1). This suggests that while BF expands reasoning capacity, it does not substantially enhance the model's 's ability to construct accurate spatial representations from code.
Table Performance of s1.1-32B with Token Budget Foreing (BF) on GeoGramBench. Accuracies (%) are averaged over 8 samples per problem, with token counts reflecting CoT length.
BF
N-Ignore
Token Count
Primitive
Compositional
Abstract
Avg.
No
0
10,544
75.37
58.96
26.58
54.60
Yes
1
11,336
76.47
58.78
26.89
54.90
Yes
2
12,319
76.47
58.78
26.89
54.90
Yes
4
15,245
75.49
59.49
26.89
54.90
Yes
6
18,710
74.50
59.13
26.89
54.40
The plateau in performance despite increased token counts indicates that the limitation lies not in the length of reasoning but the model's ability to update internal geometric models. This reinforces the qualitative observation that CoT's symbolic focus hinders effective spatial abstraction.
F ANALYSIS OF PREDICTION VARIABILITY
To further assess the reliability and consistency of model performance on our benchmark, we provide a detailed analysis of prediction variability using standard deviation as a measure of uncertainty.
For each of the 500 benchmark questions, sampled 8 responses at temperature of 0.6 for each model. For every question, we computed the standard deviation of these binary outcomes (correct or incorrect), and then averaged these standard deviation values over all questions for each model. We report the results both overall and by difficulty level (Primitive, Compositional, Abstract).
We find that for most models and difficulty levels, the overall variance is low (typically below 0.03 overall), suggesting that models produce consistent results across multiple independent samples. However, variance is notably higher in the Abstract category for certain models, with maximum deviation values reaching 0. This elevated variability typically occurs on harder spatial tasks, where model confidence is lower outputs are less stable, highlighting the increased uncertainty when
22

---

Published as a conference paper at ICLR 2026
Table Standard deviation of model predictions (across samples per question) by difficulty level.
Model
Overall
Primitive
Compositional
Abstract
GPT-o3-mini
0.033
0.036
0.020
0.093
GPT-ol
0.033
0.034
0,030
0.058
GPT-ol-preview
0.025
0.061
0.033
0.029
Gemini-Pro-I.5
0.015
0.030
0.026
0.011
GPT-ol-mini
0.038
0.050
0.031
0.088
GPT-ol-preview
0.027
0.042
0.035
0.045
Beee3-225BB-Thinking--2
0.007
0.011
0.007
0.023
DeepSeck-R1
0.008
0.008
0.010
0.017
DecpSeck-v3-0324
0.046
0.060
0.032
0.103
solving complex geometric problems. For instance, relatively advanced models such as Qwen3- 2351B-Thinking-2500 and DeepSeek-RI show consistently low standard deviation values across all levels, indicating strong and stable reasoning ability. In contrast, models like DeepSeek-v3-0322. or GPT-ol-mini can exhibit substantially higher variance, especially in the Abstract category, reflecting greater instability and inconsistency for challenging problems.
23

---

Published as a conference paper at ICLR 2026
G MORE BEHAVIOR ANALYSIS OF LLMS
Problem statement:
In quadrilateral ABCD, angle BAD and angle CDA are trisected as shown. What is the degree measure of angle AF Answer:
Geometric Code:
size (150); pair A C, Di
A = (0 B (2, 4); c = (7,4); D = (7, -2); draw 0) (2,
(7,4) (7, -2) -ycle); ;
label ("SAS", A, SW);
label ("$BS", B, NW);
label ("SCS", C, NE);
label ("$DS", D, SE); pair E, Fi
15.5-.211-.23 22
F = (5, 3); draw (A--E--D);
draw (A--F--D);
label ("$ES", E, N);
label ("$F$", F, NW);
(B) (C); ;ot (D) ; dot (E) ; dot (F)
1"bx1'" (1, 1.5), S);
label ("$x5", (2, 1), StW);
label ("$x$", (2, -1), N+N+N+W);
label ("$y5",
.5-.3), S) label ("$y$", (6.5+.3, ;))
label ("Sys", (5+.5, -1.5+.3));
label (1srio"(\eirrrs", (2.5, label ("5100~ ("5100=(\\circ)) (Micrr))
(6.5-2233.5)
B
C
110
F
100%
E
x
y
A
C
D
Figure 11: Visualization generated from the drawing code
24

---

Published as a conference paper at ICLR 2026
Response from QwQ-32B
<think> Okay, so need to find the measure of angle AFD this quadrilateral ABCD where angles BAD and CDA are trisected
First, let me note down the coordinates again to get a better idea: A: (0,0) B: (2,4) C: (7,4) D: (7,-2) So quadrilateral ABCD has vertices at these points.Let me sketch this mentally. Starting at A (0,0), going to B (2,4), then to (7,4), then to D (7.-2), and back to A. So AB is a line going up to the right, BC is horizontal line to the right, CD goes down to the right, and DA goes straight left down to A. Wait, actually, line from the lower right back to the origin. Angles BAD and CDA being trisected Alternatively, since both E and F are connected from to D, but through different paths? Maybe on one trisection and F on another? Hmm, this getting bit confusing without seeing the actual figure. Let me try to reconstruct based on coordinates. First, let's compute angles BAD and CDA see their measures, which can then be trisected. Starting with angle BAD. This is the angle at vertex A between points B, A, and D. D.Coordinates: Point B: (2,4) Point A: (0,0) Point D: (7,-2) vectors AB and AD can be calculated Now, arccos(O. 1592). Let me use calculator for better precision. Using calculator, arccos(O. I 592): cos(80 u00bO) u2248 0.17 cos(81 u00b0) u2248 0.1564 So 0.1592 is between 80 and 81 degrees. Let compute the difference: At 80 u00b0: 0.1736 At 81 u00b0: 1565 Difference between 80 and 81 u00b0 is u00bO, and the difference in cos is -0.0172 per degree... Alternatively, maybe angle B is 110 u00b0, angle C is 100 u00bO, then angles at A and D would need to be 360 210 = 150 u00b0, so angle BAD + angle CDA 150 u00b0, so if angle BAD is trisected into 3x and angle CDA into 3y, then 3x + 3y 150 uO0b0, so + = 50 u00bO, then angle AFD would be 180 - X y 130 uO0b0? Wait, no, in triangle AFD, angles at A and D would be and y, so angle AFD 180 (x + y) 180 -50=130 u00b0, but that contradicts coordinate. Alternatively, maybe angle at B is 110 u00b0, angle at C is 100 u00b0, then angles A and would need to be 360 210 150 u00b0, so angle BAD) + angle CDA 150 u00b0, so if angle BAD is trisected into 3x and angle CDA into 3y, then 3x + 3y = 150 u00b0, SO x y 50 u00b0, then angle AFD would be 180 -:00x- X y 130 u00b0? Wait, and y, so angle AFD = 180 - (x + y) = 180-500130 u00bO, but that contradicts coordinate. Alternatively, maybe angle AFD is external angle or something else which is 80.8 u00bo, so the closest integer is 81 u00b0. Since the problem likely expects an integer, I'll go with 81 as the answer. **Final Answer* 81 To find the degree measure of angle LAFD quadrilateral ABCD where angles ZBAD and LCDA are trisected, we start by analyzing the given coordinates and using vector calculations. The coordinates of the points are:
at (7--2), so DA would go from (7,-2) back to (0,0). Hmm, so DA
in triangle AFD, angles A and D would be
(0, 0) B (2. 4) C (7,4) D (7, 2) F (5, 3) We need to determine the angle at point F between points A, F, and D. This angle is formed by the vectors F and FD. **Calculate vectors:** FA
0 3) (-5.
FFD = - F = (7 5, - - 3) = (2, 5) 2. **Compute the dot product:**
FÀ. = (25)(2) + (-3) (-5) - 15
** Compute the magnitudes of the vectors:** * IF
(-3
/31-
FDD
2
/4+ /9 4. * **allulltee the cosine of the angle:**
F FD
5
5
cos(0)
F FD)
/
/986
5. Determine the angle using the arccosine function:** *s
5
f = arccos
/986
Using a calculator, we find that:
0 80.8
Rounding to the nearest degree, the measure of angle LAFD is approximately 81%, Thus, the final answer is 81
25

---

Published as a conference paper at ICLR 2026
Problem Statement: In the figure below, quadrilateral CDEG is square with CD 3, and quadrilateral BEFH is rectangle. If BE 5, how many units is BH? Express your answer as mixed number.
Answer: 1
Geometric Code:
(amtiizeSmm))
pair A= B= C= (6, D= Ep= 3), G= 36,
pair 1 #0-bisectorppoint (Ep, B+B-Bp)
pair [
pair F
["be(("" ("SB"
label ("SCS",, ;,
label (",DS",,,))>
label ("SES",Ep,E)
label F
label "PSGS",G,NY" NW)
label ("SHS",H, NW) ;
F
G
E
H
A
B
C
D
Figure 12: Visualization generated from the drawing code
26

---

Published as a conference paper at ICLR 2026
Response from P-QQB2B
Okay, let's see. I need to find the length of BH in the figure provided... should start by trying to understand the coordinates of the points based on the Asymptote code given. First, looking at the Asymptote code, they define point A at (0,0), B at (3,0), C at (6,0), D at (9.0), E is labeled as Ep which is (9,3), and G is at (6,3). So, let me note down these coordinates:
A: (0, 0)
B: (3, 0)
C: (6, 0)
-D: (9, 0)
-E: 3)
G: (6.
Alternatively, maybe the square is oriented differently? But since the problem didn't specify orientation, but the Asymptote code had coordinates with C at (6,0), D (9,0), E at (9.3), G at (6.3). Wait, but maybe it's better to compute coordinates step by step... Alternatively, since BEFH a rectangle, then H must be such that BH is equal and parallel to EF, and FH is equal and parallel to BE. Alternatively, since BE is from B to E, then to get to from E, we move in the direction perpendicular to BE ... Alternatively, perhaps chose direction incorrectly. Let me try with the other perpendicular direction. If instead of (3,4), we take (-3,-4). Then vector EF would be (-3,-4). Then point F would be E(3.3) + (-3,-4) -0.- Then vector FH would be equal to BE? Wait, no. Wait, in rectangle, opposite sides are equal. So FH should be equal and parallel to BE. Hmm, perhaps need to think differently. Alternatively, since BE is one side, and EF is another side, then FH should be equal and parallel to BE, and HB should be equal and parallel to EF.
So if vector BE is (-4,3), then vector FH should also be (-4,3). Starting from F, adding vector BE gives H F+ BE. Wait, but maybe I need to think of the rectangle as B - E - F F> H -> B.. In the Asymptote figure. points are: A(O,0), B(3,0), C(6.0), D(9,0), E(9.3), G(6.3) We need to find H's coordinates.H is extension(B, HO, A, G). Let's try compute HO and then the line. Which point So line BHO the line from B(3,0) HO(3,0). which is a point. Not helpful... Vector Ep B = (9-3, 3-,)=(6,3) Vector Ep (2 Ep) = Ep (-3,-3) = (9- (-3), (+3)) (12,6) So the two vectors are (6,3) and (12,6), which are in the same direction (since (12,6)=2*6,3)), So the angle between them is zero, the bisector is the same line. Hence, the angle bisector undefined, or the line itself. So HO is along that line. Ah, yes! Because vector BE is (6,3), and vector HB is (3/5, -6/5). Their dot product is indeed zero. So the calculation is correct. Therefore, the answer is
#*Final Answer** 1 Given quadrilateral (CDEG) is square with (CD 3) and quadrilateral (BEFH) a rectangle with (BE = 5), we need find the length of (BH).
1. **Cordinatess and Setup*#; In the Asymptote code, points (C), (D), (E), and (G) form a square with (CD 3). The coordinates are (C(6,0)), (D(9.0)), (E(9,3)), and (G(),3)). The problem states (BE 5), but in the Asymptote coordinates, (BE) is calculated as
(v(9- 3)-0-322-0 (3 0)2 45 3v5). To scale this to (BE 5), the scaling factor is J/
5
355
3
2. ** Finnding Coordinates of (+)** In the Asymptote figure, line (AG) is from (A(0.0) to (G(6.3)). The parametric form (AG) is ((6t, 3t)). -Point (F) on (AG) is found by ensuring (EF) is perpendicular to (BE). Solving, (F) is (422 21 Point (H) is determined by the rectangle properties, resulting in ( 1 g)
**Calculating (BH) in Asymptote Coordinates **; Coordinates of (B(3.)) and ((2,
Distance (BH) is (v (3 ! + )
+
(- 9)
3/5
**Scaling to Problem's 's Coordinates Scaling factor (vá)) is applied to the Asymptote 1).
length of (BH), resulting in (3/3 v/s 3
Thus, the length of (BH) is 1
27

---

Published as a conference paper at ICLR 2026
H A HYPOTHESIS ON INTERNAL GEOMETRIC REPRESENTATIONS IN LLMs
Drawing on both quantitative results and behavior analyses, we hypothesize that large language models confronted with procedural geometry code engage in multi-stage internal reasoning process closely aligned with the pipeline illustrated in Figure 13
The process begins with the extraction of local geometric features substructures ((Z), Z2. ... from the input text and code ((T, C)), corresponding to the abilities probed in RQ1. Our evidence shows that models are generally able to parse and represent these local primitives with high accuracy in simpler cases.
The next critical stage involves integrating these local elements into a coherent, global representation (Z'), reflecting the compositional reasoning explored in RQ2. This is where we observe a pronounced bottleneck: small errors or ambiguities in local geometry can disrupt subsequent steps, making difficult for models to build structurally correct and complete diagram as complexity increases.
Subsequently, models iteratively attempt to update and refine their global geometric understanding. often through shin-of-thouggh (CoT) reasoning or self-reflective steps, in hopes of reconciling inconsistencies and clarifying spatial relationships. Despite such iterative efforts, our analysis of model outputs indicates that most fail to achieve robust global integration, as highlighted by the continued drop in accuracy and recurring spatial confusion on the most complex tasks (RQ3).
Finally, the model produces answer (A), leveraging whatever spatial structure has been suc- cessfully constructed and refined. Our overall findings suggest that while LLMs can recognize and extract local geometric information, and remain significant limitations aggregating and refining these components into a globally consistent geometric representation for accurate problem solving. Overcoming these integration and synthesis difficulties is likely to be a key research frontier for closing the gap in Program to co--eometrry spatial reasoning.
extent initiate the integration process, there
These findings point to the need for future research on more robust scene composition and iterative spatial integration mechanisms in LLMs, as well as the development of benchmarks and training strategies tailored to these specific bottlenecks.
(T.C)
[Z1, Z2,
Z
2
A
RQI
RQ2
RQ3
Figure 13: Illustration of the hypothesized multi-stage internal geometry representations process in tasks. The model first extracts local geometric substructures (121, 22, ...)) from the problem statement ((T, Ct), then integrates these into coherent global structure (2-), which further iteratively refined and updated (22 .), before finally predicting the answer (A). Each stage corresponds core research question: RQI (local construction), RQ2 (compositional integration), and RQ3 (global abstraction and reasoning). Dashed arrows indicate how both input information and intermediate representations propagate throughout the process.
LLMs for
I LIMITATION AND FUTURE WORK
While GeoGramBench provides a rigorous assessment of LLMs' abilities on mathematical geometry problems described by procedural code, it does not address reasoning in weal-world 3D scenarios. In addition, our analysis failure patterns remains largely qualitative due to the lack of robust automated tools for systematic error diagnosis. Current approaches failure mode analysis often
28

---

Published as a conference paper at ICLR 2026
rely on LLM-based evaluation, but their reliability is questionable- these methods require LLMs with very high reasoning capabilities, and the faithfulness their thoiuht processes is still open research problem. Moreover, although GeoGramBench can improve LLM performance on P Prooram-to-Ger ry tasks, the evaluation of its effectiveness broader range of spatial reasoning challenges is still prefiminary and warrants more thorough investigation.
supervised fine-tuning experiments indicate that
In future work, plan extend GeoGramBench include real-world 3D scenarios. We also intend to explore the potential of this benchmark for guiding model training on more diverse spatial tasks, such those encountered research to build on GeoGramBench, develop more advanced evaluation and probing techniques, and systematically investigate model behavior a variety of procedural and spatial contexts, ultimately advancing our understanding of spatial reasoning in large language models.
robotics and other applied domains. We encourage further
J THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this work, GPT-4o was utilized as general-purpose aid for polishing writing, specifically for grammar correction and refining sentence expression. No content was generated by LLMs for research ideation, experimental design, data analysis, or substantive scientific contribution. We take full responsibility for the final content.
29


Reviews:
[{"review_id": "E486dTBU9w", "reviewer": "Reviewer_btiM", "summary": "The paper formalizes Program‑to‑Geometry: reading procedural drawing code (Asymptote/Matplotlib‑style) and performing downstream geometric reasoning. It releases GeoGramBench, 500 curated problems with a three‑level taxonomy 1)Primitive Recognition, 2) Local Relation Composition, and 3) Global Abstract Integration, organized by geometric complexity rather than typical “reasoning complexity,” plus six subtypes from \"Angle\", to \"Count\". Across 17 models, accuracy drops sharply at the highest abstraction level, with no model exceeding ~50% there.", "questions": "I would like know the authors' thoughts and opinions regarding the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Code-based spatial reasoning is an interesting and under-explored domain.\n2. Clear task definition and a taxonomy grounded in geometric (not step-count) complexity; the benchmark isolates program-to-diagram understanding from general math difficulty. \n3. Broad evaluation over 17 models with consistent prompting and multi-sample reporting; the “<50% on Abstract” result is a useful negative finding that signals a genuine capability gap.\n4. The ANSWER LEAKAGE CHALLENGES is sound and interesting.", "weaknesses": "1. Positioning vs. prior art (major). The paper does not cite SGP-Bench (ICLR 2025), which evaluates LLMs’ understanding of symbolic graphics programs without rendering as well, at broader scope (SVG for 2D; CAD for 2D/3D) and with systematic invariance tests under program transformations. While GeoGram focuses more on the 2D geometry math aspect, I think SGP-Bench is closely related and predates this submission, it should be discussed and contrasted.\n2. GeoGramBench stays within Asymptote/Matplotlib-style code and contest-style numeric targets. That makes results crisp, but limits to the mathematical geometry domain which is a fraction within the broad discussion of geometric spatial capability.\n3. While the paper's goal is to evaluate the program-to-geometry / geo-spatial capability of the LLMs, the questions also involve mathematic reasoning capability to solve the planar geometry tasks. Therefore, the final accuracy is kind of inconclusive for the geo-spatial one itself.\n4. I think the DSL choice (Asymptote/Matplotlib‑style) can significant affect the evaluation results since various LLMs could be exposed to different amount of DSL code training. few-shot CoT experiments are important to justify the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZrjUIh9nN", "reviewer": "Reviewer_QZFV", "summary": "The authors of this paper formalizes this novel task called Program-to-Geometry, which tests the model's abilities make geometric reasoning based on programmatic drawing code. The proposed GeoGramBench, consists of 500 curated problems incorporating programmatic drawing code and evaluated the performance of the 17 current state-of-the-art models on their proposed benchmark.", "questions": "- benchmark is important and interesting for the community to push the boundary of current AI systems, but there are many capabilities that we could that test, the more interesting problems are what are the implications of this benchmark? how can we use this benchmark to improve the current models? if we find a way to improve the models to be better in the proposed benchmarks, are they domain-specific capabilities or can they benefit other reasoning abilities? \n- the authors seem to be discovering that the CoT might not be beneficial for this task, which is fine per se, however, since CoT or different variants are standard approaches for improving the reasoning performance, why does it fail on GeoGramBench? might it be an indication that the tested Program-to-Geometry ability to be a very specific and not generic ability?\n- what does the level 1-5 in Figure 2 mean?\n- I am a bit confused with figure 1: math-500 and aime 24 are text-only benchmarks, where you getting the visualizations? let's say you constructed it, but you are suggesting in this diagram, that by providing additional visualization to the already existing text and code for geometric shapes, the performance actually gets worse?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- interesting insights have been provided in the section 6 \"behavior analysis of LLMs\"\n- detailed description of the data curation process (pre-selection, deduplication, human verification and answer leakage prevention)", "weaknesses": "- a more detailed analysis for the main results table in Section 5.3 is required, it is interesting that they struggle with angle and volumne, but lacks a bit of insights (for example why is this not the most challenging questions for abstract?)\n- I am not entirely sure when the experiments have been drafted, but maybe add one most recent model for evaluation?\n- The tasks seems undoubtly interesting, and one could see the implication, but testing the model to make geometric reasoning based on procedural code (albeit might have different syntax, but the geometric properties of these DSL are similar, information about the 2D/3D properties are stored somewhere in the code) has been studied for example in SGPBench (similar to the tested ability b in section 3.1 to make spatial and geometric reasoing) and SGP-GenBench (similar to ability a in section 3.1 generate geometric diagrams) and other similar works, in fact a lot of works has been dedicated for that, I would want to have some discussions with benchmarks from that area in the discussion / related work sections\n- in general I am open for discussion and modification of my ratings", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N5ZmWAOJvG", "reviewer": "Reviewer_XpCL", "summary": "This paper introduces GeoGramBench, a benchmark to evaluate large language models' ability to perform geometric spatial reasoning from procedural drawing code (e.g., Asymptote), formalizing it as the Program-to-Geometry task. It highlights LLMs' underexplored weaknesses in translating symbolic code into spatial representations. The benchmark comprises 500 curated geometry problems, organized by a novel three-level taxonomy based on geometric complexity (Primitive Recognition, Local Relation Composition, Global Abstract Integration). Drawing from sources like NuminaMath and MATH-500, the dataset addresses challenges like answer leakage through human refinement and decontamination. Evaluations on 17 frontier LLMs are comprehensive and do reveal the persistent deficiencies.", "questions": "- Could the paper include a formal definition of \"Asymptote code\" to provide clarity for readers? Additionally, it would be helpful to elaborate on the process of standardizing procedural code from diverse data sources into a consistent format.\n- Although benchmarks in this niche are scarce, it would strengthen the discussion to explicitly compare the proposed work with related efforts, such as [1].\n\n[1] Qiu Z, Liu W, Feng H, et al. Can large language models understand symbolic graphics programs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The authors effectively address prevalent issues in similar datasets, such as answer leakage, through meticulous processing techniques that significantly elevate the benchmark's overall quality.\n- The evaluation spans 17 open- and closed-source models of varying scales, uncovering both the capabilities and shortcomings of existing LLMs in this domain.\n- The paper is generally well-written, with smooth and coherent flow throughout.", "weaknesses": "- It would be valuable to incorporate evaluations using vision models that directly interpret the rendered images, alongside results from multimodal models, to serve as a reference baseline. This would help disentangle whether the observed performance gaps arise from deficiencies in reasoning capabilities or challenges in parsing Asymptote code.\n- There is a discrepancy between Table 1 and the corresponding text on page 8: \"For example, GPT-o1 drops from 76.02% to 43.35%, and DeepSeek-R1 drops from 75.27% to 40.38%.\" Please verify and ensure alignment between the visual and textual elements.\n- The paper would benefit from a standardized definition of the dataset's input-output format, including an accessible, introductory explanation of Asymptote code to enhance clarity for readers unfamiliar with procedural drawing languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MNbjBHJBTv", "reviewer": "Reviewer_Dim1", "summary": "How well can models engage in geometric reasoning? How well can models engage jointly over structured symbolic and spatial representations? The authors formalize a new evaluation task (Program-to-Geometry), collate a new testbed for this task, and apply that testbed to assess current models.", "questions": "- How much variance is there in model performance (see question above, re: error bars)? \n- For future work, not this revision :) have the authors looked at reasoning token usage across the difficulty levels in the benchmark?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated and engages with an interesting reasoning challenge (understanding geometric reasoning as it relates to symbolic representations of those problems). The authors' benchmark is quite well-structured in my opinion --- particularly in having multiple difficulty levels and coding by concept. That kind of granular decomposition will surely help other researchers and model developers better isolate and probe model capacities! \n\nI also appreciated the diversity of model analyses the authors engaged in (both quantitative and more qualitative, e.g., Section 6). \n\nI also found the authors' formalization and explication of two kinds of answer leakage very important and a valuable contribution in its own right!", "weaknesses": "While the empirical work is wide-ranging, I think the paper could be improved in clarity and rigour of the empirical analyses. For instance, the authors sample with a moderately high temp (0.6), to my understanding. However, no error bars are reported? How variable are the results? It would be good for the authors to present some sense of performance variation across tasks (e.g., 95% CIs around the mean; stdev, etc). \n\nI also found the phrasing of Program-to-Geometry in Section 3.1 somewhat confusing. It seems like there are two capabilities the authors are interested in (as they lay out in that 3.1). But is Program-to-Geometry as a task both? Just one? It's worth clarifying (and I found that confusing...) \n\nIt also would be good for the authors to clarify how their work relates to https://arxiv.org/pdf/2408.08313, which seems quite similar, especially related to RQ1 (which was at ICLR last year and is not cited).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)