Can Large Language Models Understand Symbolic Graphics Programs?
Authors: Zeju Qiu
Weiyang Liu
Haiwen Feng
Zhen Liu
Tim Z. Xiao
Katherine M. Collins
Joshua B. Tenenbaum
Adrian Weller
Michael J. Black
Bernhard Schölkopf
Venue: ICLR 2025 Spotlight (2025)
Keywords: Large Language Models
Symbolic Graphics Programs
Primary area: foundation or frontier models, including LLMs
TL;DR: Assessing Large Language Model Reasoning over Symbolic Graphics Programs

Abstract: Against the backdrop of enthusiasm for large language models (LLMs), there is a growing need to scientifically assess their capabilities and shortcomings. This is nontrivial in part because it is difficult to find tasks which the models have not encountered during training. Utilizing symbolic graphics programs, we propose a domain well-suited to test multiple spatial-semantic reasoning skills of LLMs. Popular in computer graphics, these programs procedurally generate visual data. While LLMs exhibit impressive skills in general program synthesis and analysis, symbolic graphics programs offer a new layer of evaluation: they allow us to test an LLM's ability to answer semantic questions about the images or 3D geometries without a vision encoder. To semantically understand the symbolic programs, LLMs would need to possess the ability to "imagine" and reason how the corresponding graphics content would look with only the symbolic description of the local curvatures and strokes. We use this task to evaluate LLMs by creating a large benchmark for the semantic visual understanding of symbolic graphics programs, built procedurally with minimal human effort. Particular emphasis is placed on transformations of images that leave the image level semantics invariant while introducing significant changes to the underlying program. We evaluate commercial and open-source LLMs on our benchmark to assess their ability to reason about visual output of programs, finding that LLMs considered stronger at reasoning generally perform better. Lastly, we introduce a novel method to improve this ability -- Symbolic Instruction Tuning (SIT), in which the LLM is finetuned with pre-collected instruction data on symbolic graphics programs. Interestingly, we find that SIT not only improves LLM's understanding on symbolic programs, but it also improves general reasoning ability on various other benchmarks.

Published as a conference paper at ICLR 2025
CAN LARGE LANGUAGE MODELS UNDERSTAND SYMBOLIC GRAPHICS PROGRAMS?
Zeju Qiu Weiyang Liu Haiwen Feng"* Zhen Liu Tim Z. Xiao Katherine M. Collins
Joshua B. Tenenbaum' Adrian Weller' Michael J. Black'
Bernhard Schölkopf
Max Planck Institute for Intelligent Systems, Tubingen 'University of Cambridge 'MIT I Joint first author *Joint second author Project lead sgp-bench. ceienchhgithu..... github.io
ABSTRACT
Against the backdrop of enthusiasm for large language models (LLMs), there is a growing need to scientifically assess their capabilities and shortcomings. This is nontrivial in part because
difficult to find tasks which the models have not encountered during training. Utilizing symbolic graphics programs, we propose domain well-suited to test multiple spataal-semanti reasoning skills of LLMs. Popular in computer graphics, these programs procedurally generate visual data. While LLMs exhibit impressive skills in general program synthesis and analy- sis, symbolic graphics programs offer a new layer of evaluation: they allow us to test an I.LM's ability to answer semantic questions about the images or 3D geometries without vision encoder. To semantically understand the symbolic programs, LLMs would need to possess the ability to "imagine" and reason how the corresponding graphies content would look with only the symbolic description of the local curvatures and strokes. We use this task to evaluate LLMs by creating large benchmark for the semantic visual understanding of symbolic graphics programs, built procedurally with minimal human effort. Particular emphasis is placed on transformations of images that leave the image level semantics invariant while introducing significant changes to the underlying program. We evaluate commercial and open-source LLMs on our benchmark to assess their ability reason about visual output of programs, finding that LLMs considered stronger at reasoning generally perform better. Lastly, we introduce a novel method improve this ability Symbolic Instruction Tuning (SIT), in which the LLM is finetuned with pre-collected instruction data on symbolic graphics programs. Interestingly, we find that SIT not only improves LLM's understanding on symbolic programs, but it also improves general reasoning ability on various other benchmarks.
INTRODUCTION
What are large language models (LLMs) capable of? Recent studies [5, 58] have shown that LLMs are able to generate generic computer programs, indicating a degree of pragmatic understanding of the symbolic structure of programs. Motivated by this progress, we focus another important family of computer programs, called symbolic graphics programs, where a graphics content (e.g., image, 3D asset) be generated by running a program. We interested in the following question: Can large language models "understand" symbolic graphics programs?
Before trying answer this question, start by defining what consider "understanding" of symbolic graphics programs, in the context of this work. Because a (deterministic) graphics program can be uniquely rendered to an image (the graphics programs LLMs' understanding of the graphics program as the semantic understanding of the corresponding rendered image. More specifically, we approximate such semantic visual understanding by the ability to correctly answer semantic questions only based on the raw program input. These semantic questions are generated based on the rendered image, such that they are easy to answer given the image and yet challenging given only the program as text prompt. Guided by this insight, we propose a generic pipeline for creating benchmarks that can evaluate this particular ability of LLMS to understand symbolic graphics programs, while requiring minimal human effort. While we of course recognize that there are other elements of visual reasoning that characterize understanding in humans, and that we ought evaluate in machine intelligence, we believe that our benchmark provides insight into one element of "understanding" of symbolic graphics programs that helps assay what current L.MSs are (and not) capable of.
consider here), we characterize

---

Published as a conference paper at ICLR 2025
Henderng
LAAA
Endeze Celesetlesawer sanea inriinne usks rosted goubelk prrgatt
Contee
CouTTO uer teral lel
ArOua R. Box DiPoo:
EVI (irgu to LX)
Recstze parurtaicc of grebolk progras
d2 Wast woppchileneddeeeeeooo the of etjon? Alsppr it Trank Lins Bilkcaee
Quay GPT4: axxaric quidien
Cancell
Resiwal
bettee
Mhat priuury cdlar Brtehladdd theinspe!
AcOe:
Dibor:
x Red it Bike V.Yck Gown
Whar-kkp the hand in laau???
Endazies
ge Wast wygoofattl degioe leage?
A. 2: H-lapkal C:S.bll Dil.bury
Sqaa
Text goane
What the presoncie the tough?
ailia 2. C Gren D. Vilko
What bpe afddect dapitioii the imgee
LLM
usde translated oprbhlli
Trandation pertatistzn syyhhelli prograns
(3) Somanic Understanding Symbokic Graphacs Programs
(b) Consistency of Symbolac Graphics Program Uiderstanding
Figure I: Our beachmark assesses LLMs' undorstanding of symbolic graphics programs in semantic understanding and prediction consistency. Note that the LLM can only see symbolic graphies programs and the corresponding questions. The rendered images are not input to the LLM.
The semantic understanding of symbolic graphics programs is particularly interesting in the following aspects. First, such semantic understanding may necessitate a form of "visual imagination" ability that enables the LLM to "imagine" how the produced graphics content visually looks like. If a LLM can perfectly answer every possible semantic question of a symbolic graphics program (with sufficient knowledge, the graphics content can be reconstructed), then we can say that such a LLM may have good "visual imagination" of this program. Second, symbolic programs represent a procedural way to generate the graphics content, and hence the semantic understanding also requires the oangee sequential reasoning of the program. The order of the symbolic operations may substantially affect its semantic meaning, making the problem quite challenging. Third, many semantic questions involve an accurate grounding of semantic components, and such a grounding in the symbolic program requires a fine-grained understanding of the program structure. This motivates us study whether LLMs have the ability to semantically understand symbolic graphics programs, and furthermore, how to improve this ability. In general, correctly answering semantic questions about symbolic graphics programs requires a combination of multiple sophisticated reasoning abilities from LLMs, which therefore makes the task of symbolic graphies program understanding an ideal benchmark to contribute towards evaluating the holistic reasoning capabilities of LLMs. Reasoning over symbolic programs is of particular interest from a cognitive perspective as well [103, 84, 25]. To what extent LLMs can operate over such representation with rich structures remains open problem.
Motivated by the significance of symbolic graphics program understanding, we build a benchmark. called SGP- for two important variants of symbolic graphics programs: scalable vector graphics (SVG) generic language for representing 2D vector graphics, and customized computer- aided design (CAD) as a domain-speciff language (DSL) for representing 2D/3D objects. Our benchmark consists of two types of evaluations. (1) Semantic understanding: We construct number of semantic questions (i.e., multiple-choice questions with 4 options) from a set of images (from multiple different categories). These questions are fed LLMs along with the symbolic program to evaluate the semantic understanding. (2) Semantic consistency: To evaluate the robustness of LLM's semantic understanding. we perform random translation and rotation to the original symbolic programs and then test the same semantic questions based on the perturbed programs. We evaluate the consistency of the answers from LLMs using these perturbed symbolic programs with identical semantic meaning. This evaluation can also help lower the possibility of test data leakage, because the randomly perturbed programs are unlikely to be seen during pretraining. An overview of SGP-Bench is given in Figure 1. We further validate our automated labels via human study (see Appendix B).
In addition to performing evaluation under the common in-context setting wherein LLMs are used and not finetuned, we also evaluate whether finetuning LLMs on curated dataset can boost performance. To this end, we propose Symbolic Instruction Tuning (SIT). The key idea is to collect an instruction dataset based on the rendered images. Because the semantic questions of interest are usually easy to answer from the visual input, we take advantage of the rendered images (that correspond to symbolic programs) and query powerful aanguage-vision model (e.g., GPT-4o) for detailed captioning. This leads to a scalable way to collect an instruction dataset for symbolic graphies programs. Then, we simply finetune open-source LLMs on this dataset. Our experiments demonstrate that SIT can improve a model's semantic understanding of symbolic programs, and more importantly, its general reasoning ability. Our contributions are summarized below:
"out-offthe
2

---

Published as a conference paper at ICLR 2025
We introduce a new task of symbolic graphics program understanding and propose generic yet highly scalable benchmark creation pipeline for this task.
We build a large benchmark, SGP-Bench, for comprehensively evaluating LLM's semantic under- standing and consistency of symbolic graphics programs. In SGP-Bench, we consider two types of symbolic graphics programs: SVG for 2D vector graphics and CAD for 2D/3D objects.
To improve the symbolic program understanding, we collect an instruction-folloiingg dataset and propose symbolic instruction tuning, which can also improve general reasoning performance.
Finally, we introduce symbolic MNIST dataset where the symbolic program is so challenging for LLMs to understand that GPT-4o can only achieve a chance-level performance, while the rendered image is easily recognizable by humans.
2 SEMANTIC UNDERSTANDING OF SYMBOLIC GRAPHICS PROGRAMS
We introduce the task of seman- tic symbolic graphies program un- derstanding. Our goal is to as- sess to what extent a LLM is able to "understand" a symbolic graph- ies program, which may begin to belie some latent capability to "visually imagine". Specifically, leverage the correspondence between deterministic symbolic graphics programs and rendered images, and then we character- ize the understanding of symbolic graphics programs as the semantic understanding of the correspond- ing rendered image. To do so, we
Rendering
Semantic understanding
Qi: Bow of are thebwn!?
A: Tyes Ie Too Coe Diter
Of Wake letto snop as the rie-i
tall
Croan Eillls
Of: Hen
Ok: BoeWaarope oparopeent of dowert is theen in Inage? Cata BoCaa CiPu D: Puilinng
Spulis te prog
Quedia
Question answering based on the program
Figure 2: Illustration of the symbolic graphics progran understanding task,
use the performance on arrestion-aswwerte to evaluate the semantic understanding of images. The same set of questions, along with the corresponding symbolic graphics programs, are then used to evaluate the symbolic program understanding of L.LMs (the rendered image will not be used here). Figure 2 gives an illustration of symbolic graphics program understanding. The intuition behind this evaluation is that, if an LLM has a good sense of the symbolic graphics and implicit de-rendering. then the LLM should have rough understanding about its rendered image such that it is able to answer arbitrary semantic questions about the rendered image.
(111 for How rag
Symbolic graphics program understanding can be viewed as form of visual question answering in the sense that visual input is represented by a symbolic program representation. Compared to current vision-languse images with a text-aligned encoder [76]. we consider the case where the visual input are encoded by a symbolic program that can exactly recover the graphics content. From this perspec- tive, our task aims to uncover the potential of using symbolic programs as a representation to perform visual reasoning.
models [56, 55, 121] that encodes
The
3 WHY IS UNDERSTANDING SYMBOLIC GRAPHICS PROGRAMS INTERESTING?
To showcase why semantic understanding of symbolic graphics problems require multiple sophisticated reasoning abilities, we provide a few qualitative examples of LLM's output in Figure 3 and Figure 4. Figure 3 shows a qualitative example of how OpenAl-ol reasons over the CAD program. The reasoning process is highly nontrivial, as it requires multiple reasoning abilities, such as numeric perception, spatial reasoning, geometric understanding, long-range planning and common sense. This is also part of the reason that SGP-Bench can well evaluate the general reasoning ability of LLMs. In Figure 4, we query different L.LMs (from weak to strong: Llama-3. 1-8B, Llama-3.1 1-00B, OpenAl-ol) by asking
Figure A qualitative example of CAD reasoning.
3

---

Published as a conference paper at ICLR 2025
S
To Jcoriic what hor onpeeeere proddct Strtienss an aeid
carfd coumiaay oVGo dgi sppiscedd is lmugs
ais worldy was out oddel
Slade deaic dee SV0 onde
ectecc
Calki Buter Cars ICS Doov Doe caree spectlid
linvert specchort begicial inaited
ania cooedaaaa curic contacds Vie dluing eeplli uny me om whal mevin and livel Alh plating lu himm
Ray Cardlaatin and Aer
Puad 190
Carer (179
220000
prea Piiis and litta
Emeer (10
pooe thiil and the
tne las boy dep euco
11.
Carr 033 p-ttus yos cuere potto (itel th and (1trr
Theomn thre eerm deveeetio weme apen digh fireeeeen with
Tooug tong the figtt by mecciity oppoomnd te insptte. My
Sturc
ovetol partar D004
lnggonal Comalowegte che deoe mny the digi 'I".
for Edlveing
Stp Viuuibble Frib
"To sads Be and ip Anronnnnn
The dadiac caned wascs The ao poppant depytai conse everal paas propacioe
baoge lndird
TTe
C:arpie dhow kuy aeplias tet
the dip fvoell
Mtp Morth Digh
lie ter de leatt'
pat's choonmtteers
tats intuidrveent and ichvort aovomon, de becy st boters and apecticame
Higure 4: Qualitative examples of bow LLMs reason over the symbolic program and obtain their answers.
which digit this given SVG program represents. We can observe that all LLMs start to reason from the low-level curves and gradually build up its understanding from local components to semantic
elements. Specifically, LLMs understand the symbolic program through line-by-line reasoning and then combine the results into an overall semantic understanding. This process is intriguing because it shows that LLMs understand symbolic programs through reasoning rather than memorization. More interestingly, the more powerful LLMss e.g., OpenAT-ol, show better general understanding of the symbolic program and its fine-grained grounding, which is consistent with the results on SGP-Bench.
Uler
Ukt
the innge. which ceange sinck leger? Think skplyy aup befiec enivering
Hose fisst tansloe SVG Bosel SVG
whic onrege circl is loge, left right? Thank sep before answeiing
luge far smoondin cneller bas both right
Atthoggh
aoud
OparAlie
circles are actually the oser sie Tho wren garae of the surondding cincle
steal contnn that soric peropiion realing one oraage ciell boaic
onk, This roeak apsoorration sonaning lnfount cinkes sealler and shout lowazd the kil ads the can and Ess right act lager and placed berust be right okd: The cokea and layoed takes froes amaged orange varsannde ovenly speced
bigge
though they
x
Openkiool
Reesoning through imaae
Use
ti oggooir whlch omage ierileis lrrgrr lint onnnghe Thask arey
Joloer axtioring
cattue adi dicke Adiadd aer you crle ettiut zaagee
Took de oeror
QueAliol
soacle deen (0 of suA onrp
The 5VG ende in lake heu raás cenage sindes The lell umage cickoi deital with radka The rgg araep nage ciccl deftad todic of Comppeing
Sky -top reauniing demeiiss onich e cicleli loger
New beenet har acaiters
fib of sierloo. rick
nigh egriticarly the
begar the let.
oranga
V
Coclede: The
Resening thruuh pybblle grophic pregaan droctly
Renuning through symbolic graphics program inchnorly
Figure 5: OpenAl-ol still suffers from the spurious correlation from the Ebbinghaus illusion while reasoning over images (a). In contrast, OpeaAl-ol works perfectly fine while reasoning over symbolic graphics programs directly (b) or indirectly (c).
To highlight the importance of symbolic graphics programs, we give another example in Figure where powerful LLM like OpenAI-ol can suffer from the visual spurious correlation, and in contrast, this spurious correlation can be avoided when using the symbolic graphies program as the visual representation. Specifically, we construct a visual example that resembles the Ebbinghaus illusion, but the conclusion differs from the classic Ebbinghaus illusion (i.e., two orange circles are of the same size, although they look different) as we intentionally make one of the orange circle obviously larger than the other (shown in Figure 5). Then we feed this curated example to OpenAl-ol and ask which orange circle is larger. We compare three cases: (a) image input; (b) symbolic program input; and (c) image input with indirect symbolic program prompting. Once we first ask the LLM to translate the image into symbolic program, the L.M will no longer suffer from the spurious correlation. The same
4

---

Published as a conference paper at ICLR 2025
Lirr
lier
with lp beazx." Think sup bexx
Plav
cok winch lin a kings kop or bebeeee Think aughy sightctet bet anrverng.
hdp meacis 5X0 repoueraaaeee of trage and hevandaer Ihe legibs
Clask 3.5
C
weee
lara
Reasoting though
pingo 137 4814 218
speck
arhald 318 L330 ETCh anee inwtbcctation asandee
Ler
Lecinng 1SC500 solc
dv SVG cods, wiich la lange, up benour? Thin upp dap far swrveiig.
Burtoe aree veeli Eai foce a-50 s0 (ragh - uaing)
Tap kae 30t (nn- 300 wall
Basses lac 53 -30 unds
-Basss C4C enart loquer than (200 uxls)
Claske 3.5
59G oade dais does that de bokas lise space 200 viile lepline spana only 300
Ressoting through synbboli groptics program disectly
(c) Reasening trruugh synbolic graplics program indivettly
Higure 6: Claude 3.S-onnee also
spurious correlation from Müller-Lyer illusion.
phenomenon also appears for the Müller-Lyer illusion using Claude 3.tele--.-SSomie S-Sonnet (Figure 6). These examples validate the advantages of reasoning the visual world through symbolic graphics programs.
4 A BENCHMARK FOR SYMBOLIC GRAPHICS PROGRAM UNDERSTANDING
4.1 DATASET CREATION PIPELINE
Symbolic programs
por Braleation daa
UK4
To construct our benchmark, need questions about a symbolic program, based on its rendered image. To build large bench- mark, is essential to consider how we can scale up the question collection effectively, with minimal human effort. To this end, we use a powerful vision-language model (e.g.. GPT-4o) to generate semantic questions based on the rendered images, and then we inspect them manually to make sure that these questions are rea- sonable and the answer to them is correct. We also run human study over randomized set of 500 of the automatically generated questions along with the corresponding images, and find high agreement (see Appendix B). The overall procedure for dataset creation is given in Figure 7. In this pipeline, the rendering of symbolic programs and the GPT--o querying are both scalable and can be done with minimal human involvement. Human annotators then inspect the generated question-answe based on the rendered image, which requires much less efforts than manually writing questions and answers. We emphasize that this works for most of the symbolic graphics programs. SVG and 2D CAD programs can directly produce 2D images, so it is straightforward models and we first render them into 2D images with few fixed camera positions. These rendered images are used to query GPT-4o, and the following procedures are identical to the SVG case.
torpectio
Rendesia
Iespection
Rendered images
Query GPT-4o
Semandic questions & answers
Figure 7:
Dataset construction procedure.
pairs
-uestion-answer triplet creation method is general, as
use this pipeline. For 3D CAD programs, they produce 3D
Nachee coaaaa
(a) SVG ditedt satbika Pansinay
BAD datact (3D, 3D strssass Aateaaty
Figure Key datasct statistics for both SVG and CAD programs. We show the distribation the operation number program for both SVG and CAD data, together with the number of examples of each category in the SVG dataset.
4.2 BENCHMARKING SEMANTIC UNDERSTANDING
SVG dataset statistics. We collect 1, 085 SVG programs covering 19 categories, and each program has 4 semantic multiple-choice questions (with 4 options), resulting in a total of 4, 340 questions. We ensure that answers evenly distributed across options. Dataset statistics are given in Figure 8(a). Our SVG benchmark consists of types of questions, including "Semantic": 1, 085 questions, "Color": 864 questions, "Shape": 1, 1,217 questions, "Count": 819 questions, and "Reasoning":
5

---

Published as a conference paper at ICLR 2025
Semantie
Caler
Shape
Count
Renssning
Bow caulls anc the obcleet
the abjeet riprouert
color the atjeet?
boandary if sigent
Hely usd far?
A. lobeni
B.Du Bar
Rol B. Blac
Suoree C.Ciind
Triange
A: chez tt tik C.T.bxx
A: Duisking Liokrry Cantaiing
B B.Bual
Ch.liiis Looez
D: Fear
What is depited in the truge?
What is the primury cilee tip part the abjeet?
What shaps located the button of the imags?
Hee scaaps of creare on coue? A One Bi Two
What the object prinurtly und her
A Nedl'eo a. Buacid
Ai Buzzs
Blle
Catle B: Syare
A. Blrrratt Cattis
svo cumnio
Whst pansen issags detgg.
x: Karsing setrming C. Cyslin H.Hara Fizz
tis apper wanes the abjeet:
A: Red
Yollow
shaps lie objeat!
S.uacce
Cirde
It Iranste Recasali
Hove many logs dars the objjct base?
A,
B.
H+
A-
nuthrmatical operattin in
innge?
It-5
Wist is the the post likdy repeocalig"
What the leariin insge?
What (be>lepe the whjed catar? D Ttanne
Ban assae, whech doo abject lave?
Whst is the exprasion of skgee??
Scacore Worck
B
Bi the A:
B. Aupyy
What type aniend is reprosmned by enjeet?
What is the printery color object the imags? Biar
What is the thr-lage the betteen part onject Clessir It Syar
Hew many wings does atject have?
What noteria the hend of snject llutt made of?
Fish
But
Goeer
A Tio Ohe
x. Wired 8t
CAD DayyCAD
Bioe mane bakes are vaiible the C&D objest?
Wakch letter the abject catvort? B.M
Wher the shops the cantral CAbaajjrri Callskkr
Which the followring best doscribo to base clamettrr?
Oaz B: Iwo
Theee Di Four
C.A DrD
Dr Prien
Stzdkz
Eoeger Dr. Trddeemiiaee
B: Equal
What is the seaered profie skjeet? Japeaal
Hoee many etralghe procee lesale circeler
What goomutric shops atteched to the larger lerarr-shjeed C. Pyreria
Elnn many disticct es colidtiical
abject?
Beeea
D: Lrangaia
C. Tlece
D: Foor
Di Cylindee
Thoz
Dr
CAD (Fiin))))
What festrre shille sn abjeer?
Hure usny viside there the CAD abjeer?
pcaseciric shape grivenrily unkis hend the Imagr?
sbjject on the kil??
A.Bl
B. iaketaiin pritiii featur
A Cylleerr Splan C.Hlcaass Octaga
A- Ou:
B.Taw
X. Case
B. Sylasr
How mane bake are there
How suan distiact tiers or akjeet
Whet shope ls contral jaint C: Crinaugsar
Blww muny bale are sbibic L: Tiree
enject?
hano?
Thece
Dereez
A Ouc C Theze Foor
Tea
Dr NeNe
CAD cass
How rean clrcles are there isage?
What shape is the esinn culline
Whereis the dat place in coch C: Retsvo comee Di olge
What shape the miin body the Trasiine
As3
C. Ciicle
D Squrt
Redangle
pestinsed withis the lerger spuarc?
What is the prisses shaps ehDeee inoge? space limit, we cenil the programss and only show the rendered images.
Whatbark granetri dupe the reaks buty altte C-b sbject
retermae paiats the comers ahe restengic?
Cestare
Lefi
A Palasss
DiCide
A. Ciics
Reetingl
3. Triegis U-tasa
Ceer
D.Fise
Figure 9: Example questions SVG and CAD programs.
355 questions. "Semantic" tests the global semantic meaning of the object represented by SVG codes, while the other four question types focus on detailed, local understanding of the object. "Color" color-related questions about specific object parts, which evaluates the localization of the corresponding semantic part. "Count" is about counting the occurrences of certain patterns or semantic parts. "Shape" is about the shape of certain parts of the object, which is to find geometric shapes that closely resemble the object part. Figure 9 gives some SVG examples.
CAD dataset statistics. We collect 2, 400 CAD programs from three different datasets [101, 105, 86). The CAD dataset consists of 1000 programs from DeepCAD [105], which forms the 3D subset; 700 programs from the Fusion360 Reconstruction Dataset [101]. which constitutes the 3Doompiex subset; and 700 programs from SketchGraphs [86], which makes up the 2/) subset (as shown in Table 1). Different from SVG, there is no generally established syntax for building CAD models from graphics codes; each of our 3 CAD subsets follows a different language syntax, with varying levels of complexity. When benchmarking LLMs for CAD tasks, we include domain-specific language syntax rules as part of the input prompt. The LLM is required apply in-context learning of this syntax and answer the test questions. Then we feed the renderings to GPT-4o and generate one semantic multiple-choic question (with options) and its answer. This gives us 2, 400 questions in total. We make sure that ground truth answers are evenly distributed across 4 options. Detailed dataset statistics are given in Figure 8(b). Some examples from our CAD dataset are provided in Figure 9.
Experimental results and discussion. We find that graphics program understanding, as we opera- tionalize it here, is challenging. The average accuracy of all models (proprietary and open-source) is below 70% (ranging from 30% to 67%) on SVG and below 75% (ranging from 28% to 74%) on CAD. Among these, SVG makes it more difficult for the models to understand as these 2D graphics contain richer semantics. Significant performance improvements are observed in line with scaling laws [1 13], as larger model sizes consistently lead to gains across various open-source LLMs. For example, Llama-3's score is improved from 0.429 0.548 on SVG and from 0.633 to 0.694 when

---

Published as a conference paper at ICLR 2025
its size increased from 8B to 70B, Qwen-1.5's from 0.376 to 0.499 on SVG and 0.486 to 0.632 on CAD with the size from 7B to 110B. We also notice consistent improvements given the same model size and model family but from different generations. For example. Qwen-72B from 0.466 to 0.537, LJama-8B from 0.429 to 0,465 and Llama-70B from 0.548 to 0.574. The consistent performance gain both SVG and CAD indicates that semantic understanding of symbolic graphies programs is a fundamental capability that is aligned with the scaling law of LLMs.
Compared the open-sourced LLMs that we consider here, proprietary models (GPTs) and (Claudes) outperform most of them by a large margin. Within the family of current most popular GPTs, we see a performance improvement with 27% boost (from GPT-3.5's 0.498 to GPT-4's 0.633) when evaluating these GPT variants on our SGP-Bench. This result is aligned with the seeming improvement of reasoning ability in the GPT family, validating that SGP-Bench can well distinguish different LLMs. The overall best-perfone best-performing model in both SVG and CAD is Claude 3.5 Sonnet.
The semantic understanding of graphies programs can be probed across different aspects, ranging from attribute-level investigations of "color" and "shape" to higher-level discussions of "semantics" "counting" " and "reasoning". Our benchmark is designed to cover these investigations for LLMs. Most LLMs perform well on color-related questions, followed by shape-related questions, with "count" and "semantic" questions showing progressively lower performance. This consistency is intriguing, as resembles the coarse-to-fin fine structure of visual information processing. "Color" is the most visually salient feature, "shape" understanding requires a finer grasp of global and local structures, and "count" and "semantic" questions demand deeper comprehension and knowledge. The difficulty curve is evident, with most open-source models achieving roughly half the accuracy on semantic questions compared to color questions. For instance, the best-performing open-source model, Llama3. 1-405B, achieves 37.6% accuracy on semantics and 81.6% accuracy on color grounding tasks. While open- source models struggle with "semantic" questions, ChatGPT performs quite well, with semantics being their second-best category after color grounding.
Symbolic programs
4.3 BENCHMARKING SEMANTIC CONSISTENCY
LLMs exposed to vast amounts of online SVG data. To investigate whether their semantic understanding abil- ity due potential data leakage, we propose a semantic consistency test by introducing global translations or ro- tations to SVG graphics. ensuring SE(2) invariance. Such spatial interventions greatly alter the code representation, as SVG graphics consist of lines and Bezier curves with anchor points, and SE(2) operations change all numerical values in the code. However, the SVG's semantics- such as shape or color-remain unaffected by this perturba- tion. This allows us to examine how LLMS behave when the same vector graphics are presented with drastic code- numerical changes (see Appendix 1) We perform non-trivial coordinate-level perturbations to the code, rather than using SVG transformation functions, to prevent shortcut learning by LLMs. Due to the nested structure of the tested SVG code, we visually inspect the perturbed renderings ensure that the semantics remain unchanged after perturbation. the model performs consistently under these perturbations, it suggests that its semantic understanding stems from a fundamental level of comprehension rather than trivial memorization.
Translaion poturbation
Senaantic questions &
Whether snewers ard consistenc?
Symbolic progr ringrams
Rotztion perurtattion
Figure 10: semantic consistency tie understanding remains the same when the program is perturbed without semantically changing rendered con- tent. Image perturbations result in significant code-level changes, as symbolic programs use absolute coordinates.
if
Dataset specifics. We use our SVG dataset to evaluate the semantic consistency with respect to translation and rotation. For each SVG sample, we randomly choose 5 different translations (T) and rotations plus translations (SE(2), harder case), resulting in a visually small amount of spatial shifts of the rendered object, meaning nearly no changes semantics, but complete change in SVG code numeric given the shift of SVG's anchor points and curves. Then evaluate all the LLMs with the same question set of the Underttnttnnnn benchmark but with these perturbed code inputs.
Evaluation. We measure the semantic consistency with two metrics: 1) the average accuracy of all perturbed SVG inputs'
accuracy, showing the overall accuracy once the samples intervened; and 2) the proposed "consisteney score" that counts the average frequency of the most selected answer to each question for all groups of perturbed samples (where they were translated or rotated from the same SVG program). This score indicates how much the L.LMs being consistent (no
7

---

Published as a conference paper at ICLR 2025
Model
SVG Unnlestandingg
SVG Inveriince
Aog
Semantics
Count
Color
Shape
Resson
T Avg.
SD(2) Ang.
Cons.
S0x2) Cons.
Arg
30
30pmpt
2D>
Open-munce
gennricc LLM
Gemm--1.1-28
0.317
0:321
0.333
0.25
0.356
0.287
0,312
0.270
0.954
0.920
0.278
0.204
0.251
0.281
Gemma-1.1- 1.178
0.993
0.347
0.275
0.453
0,523
0.299
0,403
0.350
0.917
0.854
0.476
0.497
0.464
0.400
IstemLM2
0,382
0.279
0.324
0.431
0.299
0
0
0.788
0.772
0.480
0.551
0.411
LatemLMM2200
0,424
0.255
0,579
0.623
02,433
0,276
0,426
0.407
0.777
0.727
0.525
0,586
0.49
0.474
Isterdl M2.5-78
0.421
0.273
0.317
0.598
0.515
0.282
0.419
0.404
0.809
0.778
0.562
0.639
0.506
0,500
Yi-1.5-9B
0.355
0.309
0.404
0.493
0.297
0.501
0.372
0.374
0.947
0.947
0.489
0.581
0416
0,361
Yi-1.5 34B
0.643
0.308
0.364
0.644
0.523
0.234
0.546
0,423
0.845
0.819
0.583
0.649
0.563
0,510
23-8B
0,250
0.244
0,255
0.326
0.259
0.250
0.233
0.442
0,8%
0.428
0,508
0.359
Aya-233358
0.442
0.307
0.354
0.648
0,511
0.318
0.451
0.434
0.898
0.857
0.488
0
0.499
0.457
Commond R-35R
0.461
0:311
0.406
0:54
0.478
0.536
0.579
0.504
Command R: 104B
0,500
0.339
0.449
0.727
0,565
0.341
0,521
0.477
0.917
0.875
0.5K3
0.634
0,570
0.524
Qwen-1 1557B
0.376
0:226
0.317
0.234
0.371
0.382
0.792
0.486
0.500
0.443
Qwen 1.5:320
0,491
0.307
0,501
0.713
0,552
0.310
0,512
0.492
0.972
0.928
0.575
0.664
0.567
0.456
Owen: L.:72B
0.406
0.299
0.319
0.598
0.265
0.574
0.461
0.853
0.854
0.600
0.659
0.526
Qwen-1 LS-1100
0,499
0.324
0.431
0.734
0.560
0.332
0,486
0.470
0.839
01.21
0.632
0,711
0.607
0,546
Qwen-2-728
0,537
0.373
0.466
0.770
0.610
0.572
0,530
0.491
0.869
0.852
0.692
0.759
0.660
0.630
Mistral7B: v0.3
0,417
0.304
0.324
0.624
0.470
0.296
0,434
0,417
0.919
0.895
0.495
0,551
0.481
0.429
Mistal Nebdo 12B
0.649
0.296
0.355
0.652
0,548
0.296
0.430
0,443
0.894
0.855
0.558
0.623
0.549
0.510
Mazeul-Lage2-1388
0,572
0.389
0.55M
0.635
0.404
0.577
0.540
0.889
0.710
0,755
0.640
11ama3-88
0.429
0.304
0.572
0.626
0.484
0.203
0.426
0.410
0.905
0.873
0.550
0.633
0.422
0,512
11ama3-70B
0,548
0.364
0.496
0,559
0.525
0.694
0.566
LLama3.1 1.1.1.00 KB
0,465
0.339
0.385
0.667
0,533
0,268
0.464
0.448
0.821
0,906
0.574
0,626
0.539
0,534
L1ama3.1-708
0.574
0,400
0.543
0.785
03659
0.411
0.584
0.554
0.856
0.825
0.688
0.739
0.663
0.641
[. 1-4258
0,580
0.376
0.584
0.816
0.647
0.589
0,570
0.548
0.840
0.817
0.717
0.767
0.700
0.661
Open rource code LLM
CodeQweat.5
0,301
0.245
0,262
0.387
0.245
0,527
0.324
0.883
0,859
0.376
0.419
0.350
0.540
DespSeck-ChskeeV2....
0.451
0.309
0.579
0.637
0.548
0.268
0,456
0.476
0,902
0.867
0.547
0.611
0.521
0.451
0.491
0:309
0.446
0.901
0.602
0.659
0.547
Proprictory models
GPT:3 Tubo
0
0.319
0:451
0.729
0.577
00338
0.509
0.492
0.897
0.870
0.576
0.654
0.530
0,510
GPT-4 Turbo
0.609
0.764
0,539
0.832
0.687
0.412
0.606
0.576
0.867
0.835
0.716
0,762
0.694
0.674
GPPT-4 mini
0,585
0.398
0.504
0.791
0.709
0.414
0.995
0,561
0.881
0.852
0.659
0.717
0.61
0.594
4PT-4o
0.632
0,787
0.555
0.832
0.696
0.671
0.625
0.586
0.878
0,844
0.732
0,782
0.696
Cluude 3 Halka
0.456
0.264
0.398
0.750
0.610
0
0.496
0.476
0.902
0,367
0.612
0.677
0.581
0.549
Cladk: 3
0,565
0.375
0.503
0.657
0.305
0.582
0.566
0.644
0.673
0.509
Chude 3.5 Sonnet
0.674
0.505
0.584
0.891
0.758
0.527
0.670
0.649
0.903
0.870
0.742
0.760
0.727
0.717
Table I: Performance of various L.MMs on SGP-Bench. This tahle evaluates how well models understand SVG inputs ('SVG- Understanding") and their behavior under random perturbations of these inputs ('VVG- Invariance") ). It also assesses 3D 2D semantic understanding of CAD code. We found the results demenstrate the "scaling law". with larger LLMs in the same family showing superior performance. Bold texts rank.
indicates performance with 1st rank, and underlined lexts indicates performance with 2nd rank
matter right or wrong) regardless of the drastic program change. If the score is close to 1, it means all the predictions are the same even with totally different input codes.
TED
T Cons.
SEX() Cons.
Experimental results and discussion. Our experiments with the SVG- Invariance benchmark demonstrate that most L.Ms exhibit robust semantic understanding of graphics programs under translation (T) and translation + rotation (SE(2) perturbations. In Table 1, most of the LLMs achieve over 80% consistency in both perturbation settings, with half of the models exceeding 90% consistency. Not only do the models remain consistent in their predictions under perturbations, but their performance on perturbed inputs also shows minimal fluctuation compared to their performance on the indicates that the semantic understanding ability that we evaluate of L.MSs unlikely due to data leakage, but rather, could stem from potential foundational capability to interpret the semantics of deterministic, symbolic graphics programs. Additionally, we assess the structural alterations introduced by our perturbation operation by calculating the tree edit distance between the original and perturbed code. Our findings indicate that the perturbation leads to varying levels of structural changes in the code. However, we observe no significant correlation between the degree of structural modification and the consistency performance (see Table 2).
5-10
0.890
0.833
20-25
0.882
0.822
Table 2: Consistency with varying tree edit distance (TED) between original and modified codos.
benchmark. We posit that this
300
Husean frendesal innge tepel tyyobolk prggoe iqua
4.4 PREDICTION ENTROPY OF LLMS AND HUMANS
Erequeccy
To study the consensus of LLMs, we compute the average predic- tion entropy on 500 symbolic programs using GPT-4o, LLama3-
100
8B, -LLama3770B Mistral-7B, Yi-1.5-34B, Gemma-1.1- 1.1-7B and 1.5--B.. We conduct a human experiment on the rendered images of these programs and collect the answers (each question has at least 5 participants, see Appendix B). Figure 11 shows that humans have strong consensus when answering questions based
Prediction entragy
Figure 11: Compurison of prediction entropy.
8

---

Published as a conference paper at ICLR 2025
on images, while LLMs show low consensus when answering questions based symbolic programs. This implies that LLMs may have different inner working mechanisms to understand symbolic programs. We are excited by future work to better investigate this difference.
5 IMPROVING LLMS WITH SYMBOLIC INSTRUCTION TUNING
Generating symbolic instruction data. Inspired by how visual instruction tun- ing [56] enables large vision-language models understand images with visual- crestion-answering (VQA) data, we de- sign new method to perform symbolic instruction tuning for LLMs to better bridge the gap between the semantic understanding and symbolic reasoning over the graphies programs. To our knowledge, there exist no semantic in- struction datasets directly over symbolic graphics programs, After rendering these symbolic graphics programs into images, we can easily query powerful vision-
insruction dearrhou the based SVG given, with appoopeately completes and gounetric faaures The corsiparalig SVG sode: [SV
provize farther Whie
[lastraction)
cols
cher dooemvnd mesk cooppatzed tockgnand Tooe inviipe Acceije
and per we quete
The hel kajae-- tf Bas dectam fo fight ad dentwrr
ne paloe doencei
doster
"Maae"" out rggenst slupe pluem tren instungh
language models (e.g. GPT-4o is used
in our case) to obtain a detailed seman-
"Maaa dol Chule
demcse san porkon dle mu olt lokke tre wnros wan.
whisk
tic captioning based on the rendered im-
the dtis come, an, hore iide whl fer
age. The intuition is straightforward, we want to build an intrinsic connection between semantic description and sym- bolic programs. The instruction data is created in a similar spirit to our bench- mark. We leverage the correspondence between symbolic problems and graph- ics content, and then use the rendered im- ages to obtain semantically rich descrip- tion. Following this idea, we construct the first semantic description dataset for symbolic graphies programs. Specifically, for each image rendered from symbolic graphics program, we prompt GPT-4o to produce a semantically-rict deseription. Finally, we collect a dataset that contains detailed semantic descriptions for 72K symbolic programs. Moreover, our SIT data can also be used in a reverse fashion (rev-SIT), i.e., rephrasing the answer as the new question and the question the new answer. Figure 12 shows the comparison between original and reverse SIT data.
and reals
Oiord lerparnde::
dec deing nend urag baok pnd solan
Exasppe Ouigisal
Doserbo the obyor from the presided SVG code by Socaing os sensaiis and aconetic atuisuees The SVG cook: (SYG)
Crate SVG care that securablly renders an image beaod oz lie gives description. Tae conreganding dencription: (Imaage Descripttinn
[Inerartion]
[Image Description)
[SV
[Reegenne]
(b) Strusture of origital data
Stuuctrr of revess
Figure 12: Comparison between original and reverse SIT data.
Supervised finetuning with symbolic instruction data. We generally follow the standard instruction fine-tuning procedure (including the default hyperparameter settings) from Alpaca [93] and use supervised finetuning train open-source LLMSs with our own symbolic instruction data. To facilitate future research, our symbolic instruction data is also made publicly available.
Dataset Size
Avcuracy
Experimental results. We perform supervised finetuning on Llama-3. 3.BB with orthogonal finetuning [60, 75] to demonstrate the effectiveness of SIT. Here we use the original SIT data (no rev-SIT data is used). In Appendix E.2,
Original
46.5
S0T-10k
48.0 (+1.3)
SIT 25k
50.3 (+3.6)
provide results on both Llama-3-8B and Gemma-7B to show that the performance gain is agnostic to the base LLM. The performance of LoRA [36] also given in Appendix E.2 (only slightly worse than OFT). From the experimental results in Table 3, we observe that SIT has consistently improved the semantic graphics program understanding of LLMs, increasing the performance of Llama-3. 1-B8B from 46.7% to 51.4% with 55K instruction question-answer answer pairs. With more instruction data (from 10K to 55K), the performance also increases. We note that Llama-3.1-88 achieves competitive performance among all open-source I.MMs after being finetuned with SIT. The performance is already better than GPT-3.5t. The finetuning results demonstrates that the ability of symbolic graphics program understanding can be improved with SIT. However, the improved performance of Llama-3.1 1- 8B remains worse than Llama-3.1 indicating that the tested ability is fundamental and differences between models of varying scales cannot be easily leveled by finetuning on benchmark- data.
SIT 40k
51.2 (+4.5)
S1T-55k
51.4 (14.7)
Table 3: Performance of SIT.

---

Published as a conference paper at ICLR 2025
6 SIT CAN IMPROVE GENERAL REASONING ABILITY
Since Figure 4 shows that the ability to understand symbolic graphics programs is associated with some fundamental reasoning abilities of LLMs, we are interested in whether symbolic graphics programs can be used as a novel data source for building better instruction tuning datasets, which can help to improve the general reasoning ability of LLMs. To verify this, we test the instruction- tuned models on a variety of popular LLM benchmarks, including benchmarks focusing on natural language understanding (XNLI [17], IFEval [119], HellaSwag [117], C-Eval [39], CoQA [80],
MMLU [34], SQuAD2.0 (77]), generic reasoning (BigBenchHard [92]. PIQA [7], AGlEval [118))
and mathematical reasoning (Arithmetic [9], MathQA [3], GSM8k [14], ASDiv [71)..
Experimental results and discussion. We use the Llama-3.1-88 model (without instruction tuning), and the baseline is finetuned with Open-Instruct (OI) [97] that contains 143K question-answer pairs (details in Appendix I). We evaluate whether finetuning with
Benchmarkk
01
OU-SIT
Ol-rev-IIT
Ol-mixed-SII
XNLI
41.8
(4..)
(+1.3)
(+1.I+
IFEvalgmepe
14.8
(+1.5)
(43.5)
16.6 (+1.8)
IFFval
24.9 28.9
(+5.6)
29.6 (+4.7)
HellnSwag
60.0
(+0.2)
60.5 (40.5)
60.4 (+0.4)
C-Eval
46.4
(+1.5)
(+1.6)
48.1 (+1.7)
SIT data can improve general reasoning by testing three ways of using SIT data: (1) mixing original SIT data into OI; (2) mixing the reverse SIT data into OI; (3) mixing both original and reverse SIT data into OL The results are given in Table 4. We can observe that mixing SIT data can generally improve the instruction following and the reverse usage of SIT data (i.e., sym- bolic graphics program generation) can improve a set of reasoning abilities that are complementary to sym- bolic graphics program understanding. The mixture of both original and reverse SIT data often achieves better performance than the OI baseline, the OI SIT baseline and the OI + rev-SIT baseline. These results are consistent with recent findings that training on code can enhance reasoning ability [66, 4] and mathematical understanding [88]. Symbolic graphics programs, specialized form of code, are used to generate visual graphics content. Like traditional code, they possess a hierarchical structure, but unlike typical programs that produce numerical outputs, symbolic graphics programs generate outputs rich in semantic information, encompassing multiple challenging reasoning tasks such as component localization, color identification, affordance prediction, and semantic and geometric understanding. For instance, answering a question like "What is the object primarily used for?" requires LLMs to first semantically identify the object and then determine its usage. This process involves multiple interconnected reasoning steps. where an error in any one of them leads to an incorrect final answer. SIT enhances reasoning abilities by interpreting low-level graphics programs through high-level natural language descriptions. From Figure 12(a), we see that symbolic graphics program descriptions are highly detailed and semantic qualiticsqallities often lacking in general programs.
MMLU
60.4
(+0.6)
61.1 (+0.7)
(+1.2)
SQuAD2.0
28.9 (-..)
(42.7)
29.9 (+1.0)
BBH
59.5 60.7 (+1.2) 60.2 (+0.7)
(+1.7)
PIQA
79.9 (+0.4) 80.3 (+0.4)
80.4 (+0.5)
AGIEval
23.7
(46.6) (47.9)
29.2 (+5.5)
Arithmetic
89.8
(+2.0) 90.1 (40.3)
91.8 (+2.0)
MathQA
40.3 (+1.)
40.7 (+1.4)
GSM8k
48.2
(42.5)
(4.0))
51.5 (+3.3)
CoQA
67.9
69.1 (+1.2) 68.7 (+0.8)
69.1 (+1.2)
ASDiv
18.5
21.8
(+3.3)
20.1 (+1.6)
21.3 (+2.8)
Table 4: Results on variety of popular LLM evaluation bench- marks when performing instruction tuning with without SIT. The Open-Instruet (Ol) datasct serves as our baseline.
7 A CRITICAL VIEW ON CURRENT LLM'S CAPABILITY
Method
Accuracy
am
Qwen-1.5-1106
Despite the observed remarkable capability of LLMSs to perform complex, multi-step reasoning over symbolic programs, it is evident that there remains substantial potential for further advancements. We provide an intriguing experiment to demonstrate that SVG programs can be quite difficult for LLMS to understand such that even if the corresponding rendered images are fairly easy for humans to recognize, all these powerful LLMs still fail dramatically.
Qwen 2-70b
11.3
GPT-3.5:
10.2
GPT-4t
10.6
4PT-40
13.0
Table 5: Accuracy of L.L.M on SGP-MNIST.
Specifically. we construct symbolic graphics programs that can produce MNIST-like images, as shown in Figure 4 (and Appendix A. Our SGP-MNIST dataset contains 1,000 symbolic graphics programs (100 per digit), each asking which digit the SVG program represents. The results are given in Table 5. Even the powerful GPT-4o can only achieve an accuracy slightly higher than the chance-level. The MNIST-like symbolic program presents a unique challenge due to the absence of semantic components for LLMs to reason upon. Instead, it comprises convoluted, irregular path trajectories that resemble handwritten digits. Additionally, the program contains not only single paths but enclosed loops to represent the "thickness" of digits, demanding precise path planning by the LLMs. For instance, the digit 1 is not represented as a single "line" but rather as an clongated loop, which must be distinguished from more oval-shaped loops, such those representing the digit O.
10

---

Published as a conference paper at ICLR 2025
Without prior knowledge of digit "thickness." the LLM must infer this distinction through detailed reasoning over the loop structures, further elevating the complexity of the task. The chance-level performance suggests that how LLMs understand SVG programs is very different from how humans understand images; better understanding similarities and differences in human and machine reasoning is important if we are to build systems that can appropriately work with us [16]. There many exciting yet totally unexplored problems in this task, and our benchmark can serve as a stepping stone to improving symbolic graphies program understanding for LLMs.
8 RELATED WORK AND ACKNOWLEDGMENT
Symbolic graphics programs. Generating visual data by procedural modeling with symbolic programs has been essential to computer graphics since its inception, particularly for 2D shapes and 3D geometry. See [83] for an overview. Common program types include constructive-solie geometry (CSG) [21, 45, 81, 89, 1 112], computer-aided design (CAD) [30, 50, 51, 87, 108], vector graphics (e.g., SVG) [78, 79]. L-systems [33], and customized domains [22, 95, 18, 37, 24, 69). Among these, SVGs constructed from primitive shapes like vector paths, curves, or polygons. Central to SVGs is the vector path, providing detailed control over graphics and geometry primitives. Similarly, procedural 3D geometric modeling, particularly in CAD applications, involves parameterized operations to produce geometry. Datasets like the ABC [47] and Fusion 360 Gallery [102] offer hierarchical decomposition, joints, contact surfaces, construction sequences, and shape segmentation based on modeling operations. Our paper focuses on graphics programs of SVG and CAD by introducing a new semantic understanding task that requires a challenging reasoning over the programs.
Graphics program understanding and generation. As graphics programs often provide compact, scalable and potentially more semantic descriptions compared to raw pixels and voxels, it has been widely explored to discover graphics programs for 2D images like 2D hand drawings and synthetic patterns [98, 99, 89, 23, 82, 29, 87], for 3D objects represented in voxels and meshes [29, 43, 95, 102, 89, 23] and for 3D scenes represented by multi-view images [68, 49, 104, 111, 69, 61, 53, 31, 30, 46, 20]. [104] infers gustom-designed markup code from images that can be easily translated to renderer-friendly inputs. In follow-up work, [111] explore how graphics programs can be used for visual question answering (VQA). Recently, [48] has advanced this direction by examining large language models (LLMs) for synthesizing graphics programs to reconstruct visual input. In contrast, we benchmark LLMs to perform semantic-leve question answering, similar to VQA, but use graphics programs input without relying any visual modality.
Large language models. LLMs have demonstrated growing potential in many applications, ranging from mathematical problem solving and theorem proving assistance [65, 114, 116, 15] to aiding biological discovery [67, 27, 94, 59]. Applying LLMs for programming tasks is also a popular direction of research. Specifically, many works have explored topics such as code retrieval [26], automated testing [19, 62, 109], repairing [44, 106, 41, 42, 100), documentation [13, 2], and genera- tion [11, 5, 54, 28, 72]. These abilities of understanding and generating programs are usually gained from pretraining or finetuning on large datasets of code. Our work investigates LLMs' capability of understanding symbolic graphics programs, which differs significantly from the prior works since the semantic meaning of graphics programs are often defined visually by their corresponding graphics.
Relevant benchmarks and datasets. Many benchmarks have evaluated different aspects of LLMss AI safety/ethics [107, 38], out-of-distribution performance 15, 110], API/tool usage [52], code generation [35], etc. Perhaps the most relevant aspect of LLM evaluation to our task is (non-graphics) program understanding abilities 196, 44, 35, 64, 57, 63, 90, 34, 11]. As graphics programs can be rendered into images, it is also highly relevant to investigate how vision- Languala models capable of visual understanding [12, 1, 74, 40, 70, 32, 6, 91, 85, 120]. For SVG programs, [10] studies whether LLMs can understand them and [122] introduces concurrent benchmark for this purpose. Different from existing benchmarks, SGP-Bench is one of the first benchmarks to evaluate the semantic understanding of general graphics programs.
A Acknowledgement. The authors would like to thank Yao Feng and Yandong Wen for helpful suggestions. Additionally, HF would like to thank Hanqi Zhou for her support throughout the project, during which time they became engaged (Fun fact: HF and Hanqi Zhou got married a few hours before the submission deadline). The diamond ring featured in Figure 1 symbolizes this joyous personal milestone and is courtesy of the entire team. This work was supported in part by the German Federal Ministry of Education and Research (BMBF): Tubingen AI Center, FKZ: 011S1 18339, and
11

---

Published as a conference paper at ICLR 2025
by the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. WL was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP XX. project number: 276693517. KMC acknowledges support from the Marshall Scholarship and Cambridge Trust. AW acknowledges support from a Turing Al Fellowship under grant EP/VO25279/1, The Alan Turing Institute, and the Leverhulme Trust via CFI. MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB is consultant for Meshcapade, his research in this project was performed solely at, and funded solely by, the Max Planck Society.
REFERENCES
[1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In ICCV, 2019. 11
[2] Toufique Ahmed and Premkumar T. Devanbu. Few-shot training 1lms for project-specific code-summarization. ASE, 2022. 11
[3] Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorde Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. arXiv preprint 1905. 2019. 10
[4] Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. To code, or not to code? exploring impact of code in pre-training. arXiv preprint arXiv:2408. 10914, 2024. 10
[5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint
2021. 1, 11
[6] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In UIST, 2010. 11
[7] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020. 10
[8] Martin Bokeloh, Michael Wand, and Hans-Peter Seidel. A connection between partial symme- try and inverse procedural modeling. In SIGGRAPH, 2010. 11
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020. 10
[10] Mu Cai, Zeyi Huang, Yuheng Li, Haohan Wang, and Yong Jae Lee. Leveraging large language models for scalable vector graphies-driven image understanding. arXiv preprint arXiv:2306.0660 2023. 11
[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 11
[12] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zimick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 3xXi::554400325 2015. 11
[13] Colin B Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundare- san. Pymt5: multi-mode translation of natural language and python code with transformers. arXiv preprint arXiv:2010.031500 2020. 11
12

---

Published as a conference paper at ICLR 2025
[14] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. 10
[15] Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. Evaluating language models for mathematics through interactions. PNAS, 2024. 11
[16] Katherine M Collins, lli Sucholutsky, Umang Bhatt, Kartik Chandra, Lionel Wong, Mina Lee, Cedegao E Zhang. Tan Zhi-Xuan, Mark Ho, Vikash Mansinghka, et al. Building machines that learn and think with people. Nature human behaviour, 2024. 11
[17] Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:/809.05033 2018. 10
[18] Boyang Deng, Sumith Kulal, Zhengyang Dong, Congyue Deng. Yonglong Tian, and Jiajun Wu. Unsupervised learning of shape programs with repeatable implicit parts. In NeurIPS, 2022. 11
[19] Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. arXiv preprini arXiv:2304.02014 2023. 11
[20] Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta-Sim2: Unsupervised learning of scene structure for synthetic data generation. In ECCV, 2020. 11
[21] Tao Du, Jeevana Priya Inala, Yewen Pu, Andrew Spielberg, Adriana Schulz, Daniela Rus, Armando Solar-Lezama, and Wojeiech Matusik. InverseCSG: Automatic conversion of 3D models to CSG trees. ACM Transactions on Graphics, 2018. 11
[22] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning infer graphics programs from hand-drawn images. In NeurlPS, 2018. 11
[23] Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar- Lezama. Write, execute, assess: Program synthesis with a REPL. NeurlPS, 2019.
[24] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Lue Cary, Armando Solar-Lezama, and Joshua Tenenbaum. Dreameoder: Boot- strapping inductive program synthesis with wake-sleep library learning. In PLDI, 2021. 11
[25] Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: growing gener- alizable, interpretable knowledge with wake-sleep bayesian program learning. Philosophical Transactions of the Royal Society A, 2023.
[26] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 08/500 2020. 11
[27] Noelia Ferruz and Birte Höcker. Controllable protein design with language models. Nature Machine Intelligence, 2022. 11
[28] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. In ICLR, 2023. 11
[29] Aditya Ganeshan, R. Kenny Jones, and Daniel Ritchie. Improving unsupervised visual program inference with code rewriting families. In ICCV, 2023.
[30] Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S.M. Ali Eslami, and Oriol Vinyals. Synthe- sizing programs for images using reinforced adversarial learning. In ICMI, 2018. 11
13

---

Published as a conference paper at ICLR 2025
[31] Nishad Gothoskar, Marco Sesumano-Townee Ben Zinberg, Matin Ghavamizadeh, Falk Pollok, Austin Garrett, Josh Tenenbaum, Dan Gutfreund, and Vikash Mansinghka. 3DP3: 3D scene perception via probabilistic programming. In NeurIPS, 2021. 11
[32] Yash Goyal, Tejas Khot, Douglas Summers-Star Stany, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
[33] Jianwei Guo, Haiyong Jiang, Bedrich Benes. Oliver Deussen, Xiaopeng Zhang, Dani Lischin- ski, and Hui Huang. Inverse procedural modeling of branching structures by inferring I-systems. ACM Transactions on Graphics, 2020. 11
[34] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0332
2020. 10, 11,22
[35] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In NeurIPS, 2021. 11
[36] Edward J. Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. 9, 39
[37] Yiwei Hu, Chengan He, Valentin Deschaintre, Julie Dorsey, and Holly Rushmeier, An inverse procedural modeling pipeline for SVBRDF maps. ACM Transactions on Graphics, 2022. 11
[38] Yue Huang, Qihui Zhang, Lichao Sun, et al. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306. / 1507, 2023. 11
[39] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level 0rXiv:2355.83322 2023. 10
chinese evaluation suite for foundation models. arXiv preprint
[40] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 11
[41] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. Impact of code language models on automated program repair. In ICSE, 2023. 11
[42] Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. Inferfix: End-to-end program repair with llms. In ESEC/FSE, 2023. 11
[43] Kenny Jones, Homer Walke, and Daniel Ritchie. PLAD: Learning to infer shape programs with pseudo-labels and approximate distributions. In CVPR, 2022. 11
[44] Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In ICML, 2020. 11
[45] Kacper Kania, Maciej Zieba, and Tomasz Kajdanowicz. ing of constructive solid geometry tree. In NeurlPS, 2020. 11
discover-
[46] Amlan Kar, Aayush Prakash, Ming- Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-Sim: Learning to generate synthetic datasets. In ICCV, 2019. 11
[47] Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Mare Alexa, Denis Zorin, and Daniele Panozzo. ABC: A big CAD model dataset for geometric deep learning. In CVPR, 2019. 11, 21
[48] Peter Kulits, Haiwen Feng. Weiyang Liu, Victoria Abrevaya, and Michael Black. Re-thinking inverse graphics with large language models. arXiv preprint arXiv:2404. 15228, 2024. 11
14

---

Published as a conference paper at ICLR 2025
[49] Tejas D. Kulkamni, Pushmeet Kohli, Joshua B. Tenenbaum, and Vikash Mansinghka. Picture:
A probabilistic programming language for scene perception. In CVPR, 2015. 11
[50] Changjian Li, Hao Pan, Adrien Bousseau, and Niloy J. Mitra. Sketch2CAD: Sequential CAD
modeling by sketching in context. ACM Transactions on Graphies, 2020. 11
[51] Changjian Li, Hao Pan, Adrien Bousseau, and Niloy J. Mitra. Free2CAD: Parsing freehand drawings into CAD commands. ACM Transactions Graphics, 2022. 11
[52] Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yonghin Li. Api-bank: A benchmark for tool-augmented llms. arXiv preprint arXiv:2304.082 2023. 11
[53] Yikai Li, Jiayuan Mao, Xiuming Zhang. Bill Freeman, Josh Tenenbaum, Noah Snavely, and Jiajun Wu. Multi-planc program induction with 3D box priors. In NeurlPS, 2020. 11
[54] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-leve code generation with alphacode. Science, 2022. 11
[55] Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023. 3
[56] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 3,9
[57] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In NeurlPS, 2023. 11
[58] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In NeurIPS, 2024.
[59] Shengehao Liu, Jiongxiao Wang, Yijin Yang, Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. Conversational drug editing using retrieval and domain feedback. In ICLR, 2024. 11
[60] Weiyang Liu, Zeju Qiu, Yao Feng. Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In ICLR, 2024. 9, 39
[61] Yunchao Liu, Jiajun Wu, Zheng Wu, Daniel Ritchie, William T. Freeman, and Joshua B. Tenenbaum. Learning describe scenes with programs. In ICLR, 2019. 11
[62] Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. Fill in the blank: Context-aware automated text input generation for mobile GUI testing. In ICSE, 2023.
[63] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wer Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. 11
[64] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. In NeurlPS, 2021. 11
[65] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering math- ematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.0955 2023. 11
15

---

Published as a conference paper at ICLR 2025
[66] Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help ilms reasoning? arXiv preprint arXiv:2309.1629. 2023. 10
[67] Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotech- nology, 2023. 11
[68] Vikash K. Mansinghka, Tejas D. Kulkarni, Yura N. Perov, and Josh Tenenbaum. Approximate Bayesian image interpretation using generative probabilistic graphics programs. In NIPS, 2013.
[69] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019. 11
[70] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. 11
[71] Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers, 2021. 10
[72] Erik Nijkamp. Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In ICLR, 2023. 11
[73] Stefan Palan and Christian Schitter. Prolific.ac subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17:22-27, 2018. 26
[74] Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer Srage-to-sentene models. In ICCV, 2015. 11
[75] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. NeurIPS, 2023. 9, 39
[76] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021. 3
[77] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint
2018. 10
[78] Pradyumna Reddy, Michael Gharbi, Michal Lukac, and Niloy J. Mitra. Im2Vec: Synthesizing vector graphics without vector supervision. In CVPR, 2021. 11
[79] Pradyumna Reddy, Zhifei Zhang, Zhaowen Wang, Matthew Fisher, Hailin Jin, and Niloy Mitra.
A multi-implicit neural representation for fonts. In NeurlPS, 2021. 11
[80] Siva Reddy, Danqi Chen, and Christopher Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguisties, 2019. 10
[81] Daxuan Ren, Jianmin Zheng, Jianfei Cai, Jiatong Li, and Junzhe Zhang. ExtrudeNet: Unsuper- vised inverse
for shape parsing. In ECCV, 2022. 11
[82] Marzia Riso, Davide Sforza, and Fabio Pellacini. pOp: Parameter optimization of differentiable vector patterns. Computer Graphics Forum, 2022. 11
[83] Daniel Ritchie, Paul Guerrero, R. Kenny Jones, Niloy J. Mitra, Adriana Schulz, Karl D. D. Willis, and Jiajun Wu. Neurosymbolic models for computer graphics. Computer Graphics Forum, 2023. 11
16

---

Published as a conference paper at ICLR 2025
[84] Joshua S Rule, Joshua B Tenenbaum, and Steven T Piantadosi. The child as hacker. Trends in cognitive sciences, (A(11):900 915, 2020.
[85] Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Sciencequ: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 2022. 11
[86] Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P. Adams. SketchGraphs: A large-scale dataset for modeling relational geometry in computer-aided design. In ICML 2020 Workshop Object- Orieett-r Learning, 2020. 6, 21
[87] Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P. Adams. Vitruvion: A generative model of parametric CAD sketches. In ICLR, 2022. 11
[88] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint (rXxi:2402.033000 2024. 10
[89] Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. CSGNet: Neural shape parser for constructive solid geometry. In CVPR, 2018. 11
[90] Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners. arXiv preprint 2rXiv:2210.03000 2022. 11
[91] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that read. In CVPR, 2019. 11
[92] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.092611 2022. 10
[93] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cearuuctionfolloviiii llama model. https: htt///gtttuuee
staaforrdaaaaaaaaaaaa alaca, 2023, 9
[94] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nuture Medicine, 2023. 11
[95] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning to infer and execute 3D shape programs. In ICLR, 2019.
11
[96] Priyan Vaithilingam, Tianyi Zhang, and Elena Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI, 2022. 11
[97] VMware AI Labs. Open-instruct. Huggingface.co, 2023. URL https::pp///cccccct httpll::://hggnnnie hhggg/hggagg.......///////// co/ adlatasstts///aaaaa
Accessed: 2024-10-02. 10
[98] O. Št'ava, B. Beneš, R. Měch, G. Aliaga, and P. Krištof. Inverse procedural modeling by automatic generation of L-systems. Computer Graphics Forum, 2010. 11
[99] O. Št' ava, S. Pirk, J. Kratt, B. Chen, R. Měch, O. Deussen, and B. Benes. Inverse procedural modelling of trees. Computer Gruphics Forum, 2014. 11
[100] Yuxiang Wei, Chunqiu Steven Xia. and Lingming Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. In Satish Chandra, Kelly Blincoe, and Paolo Tonella (eds.), ESEC/FSE, 2023. 11
17

---

Published as a conference paper at ICLR 2025
[101] Karl D. D. Willis, Yewen Pu, Jieliang Hang Chu, Tao Du, Joseph G. Lamboume, Armando Solar- LezaraLezama, and Wojeiech Matusik. Fusion 360 gallery: A dataset and environment for programmatic cad construction from human design sequences. ACM Transactions on Graphics, 2021. 6, 21
[102] Karl DD Willis, Yewen Pu, Jieliang Luo, Hang Chu, Tao Du, Joseph G. Lambourne, Armando Solar-Lezama, and Wojeiech Matusik. Fusion 360 gallery: dataset and environment for programmatic CAD construction from human design sequences. ACM Transactions on Graphics, 2021.
[103] Lionel Wong, Gabriel Grand, Alexander K Lew, Noah D Goodman, Vikash K Mansinghka, Ja- cob Andreas, and Joshua B Tenenbaum. From word models to world models: Translating from natural language to the probabilistic language of thought. arXiv preprint crXiv:2306.12677 2023. 2
[104] Jiajun Wu, Joshua Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In CVPR, 2017. 11
[105] Rundi Wu, Chang Xiao, and Changxi Zheng. Deepead: A deep generative network for computer-aided design models. In ICCV, 2021. 6, 21
[106] Chunqiu Steven Xia and Lingming Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. ESEC/FSE, 2022.
[107] Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, al. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705 2023. 11
[108]
Xianghao Xu, Wenzhe Peng, Chin-Yr Yi Cheng, Karl D.D. Willis, and Daniel Ritchie. Inferring CAD modeling sequences using zone graphs. In CVPR, 2021. 11
[109] Chenyuan Yang. Yinlin Deng, Runyu Lu, Jiayi Yao, Jiawei Liu, Reyhaneh Jabbarvand, and Lingming Zhang. White-box compiler fuzzing empowered by large language models. arXiv preprint arXiv:2310.15991 2023. 11
[110] Linyi Yang. Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang. Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. arXiv preprint arXiv:2211.0888 2022. 11
[111] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic VQA: Disentangling reasoning from vision and language understanding, In NeurIPS, 2018. 11
[112] Fenggen Yu, Zhiqin Chen, Manyi Li, Aditya Sanghi, Hooman Shayani, Ali Mahdavi-Amin and Hao Zhang. CAPRI-Net: Learning compact CAD shapes with adaptive primitive assembly. In CVPR, 2022. 11
[113] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022.
[114] Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. In ICLR, 2024. 11
[115] Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting out-of distrriiint robustness in nlp: Benchmarks. analysis, and lms evaluations. In NeurIPS, 2024. 11
[116] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309. 05653, 2023. 11
18

---

Published as a conference paper at ICLR 2025
[117] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In ACL, 2019. 10
[118] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu,, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304. 06364, 2023. 10
[119] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. cetrucctio-followiing evaluation for large language models. arXiv preprint urXiv:2311.07911 2023. 10
[120] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
[121] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models. ICLR. 2024. 3
[122] Bocheng Zou, Mu Cai, Jianrui Zhang, and Yong Jae Lee. Vgbench: Evaluating large language models on vector graphics understanding and generation, arXiv preprint arXiv:2407. 10972, 2024. 11
19

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Benchmark Details
21
A.1
Data preparation
21
A.2
Evaluation protocol
22
A.3
Evaluated model specs
22
B Human Study Details
26
C SVG Invariance Illustration
29
D More Examples in SGP-Bench
30
D.I SVG Data
30
D.2 CAD Data
................................
34
D.3
Symbolic Instruction rnstruction-oolloiingg Data (SVG)
38
E
Details More Results of Symbolic Instruction Tuning
39
E.1
Implementation Details
................................
39
E.2
More Experiments in Symbolic Instruction Tuning
................................
40
F
Text Prompt Template
F.I
Template for benchmark construction
41
F22
Template for evaluating models on SGP-Bench
45
20

---

Published as a conference paper at ICLR 2025
A BENCHMARK DETAILS
We adopt implementations from other projects to build our SGP-Bench. We follow the implemntation of Math Vista' for querying GPT or open-sourced Llama3.1-8B and perform LLM-based answer extraction, vLLM for efficient model inference and simple-evals' for a unified benchmarking framework.
Our data license follows the license of the original license of our data source.
A.1 DATA PREPARATION
SGP-Bench (SVG). Our SVG data are sampled from kaggle SVG Icons dataser' and we build our SGP-Bench (SVG) using text prompts from F.I. The original data from kaggle SVG Icons is crawled from SVGrepo*. The website is merely an aggregator, so it follows that any content on it must at least be licensed permissively enough for SVGrepo to distribute it, and therefore it is acceptable to distribute as part of a collection in our benchmark. Refer to the SVGrepo license for further details.
We use beautifulsoup and SvgLib to process the SVG XML code perform translation and rotation perturbations for the invariance investigation, a visual sample can be found in Fig. 17. Specifically, as we assume that all XML elements in each SVG figure do possess any "transform" " attribute as it complicates the augmentation process. For elements that can be fully specified by coordinates (e.g., <rect>, <polygon>), we perform augmentation by perturbing these coordinates. For <path> elements in which path information is fully specified in "d" attributes, we first turn all relative operations (e.g.. "I 23", meaning that draw a line from the current position (x, y) to the position (x + 2, + 3)) into absolute ones, and later perturb the coordinates but not other path attributes. As mentioned in the main paper, small spatial perturbations can drastically change the numerics of the SVG XML code (See Section C for more details).
SVG Invariance:
SGP-Bench (CAD). Our CAD (3D) sequences data are sampled from DeepCAD [105] datasets, which contains around 180k manually constructed CAD sequences that originally from the ABC dataset [47]. We manually sample 1000 sequences and use pythonocc (OpenCASCADE) to verify and normalize these CAD sequences, then render the front and back view of the 3D CAD models. Since all the CAD models are from the OnShape platform, the copyright of the CAD models is owned by their creators. For licensing details, see Onshape Terms of Use I.g.ii. Our CAD (3Deomplex) sequences are sampled from Fusion360 Reconstruction Dataset [101] datasets, which contains 8,625 sequences, with more complex curve operations for constructing sketches. Our CAD (2D) sequences are sampled from SketchGraphs [86] dataset, which consists of 15 million sketches extracted from real-world CAD models. The major difference is it consists of 2D CAD sketches without Extrusion operations.
2
3
4
S
8
9
Figure 13: Examples of MGP-M challenge, hand-writlen digit constructed SVG programs.
SGP-MNIST. The MNIST SVG data is sampled from the kaggle MNIST-SVG dataset We
randomly sample 1000 samples from MNIST-SVG (100 samples per digit category) to build our SPG-MNIST benchmark. The data comes with CC BY-SA 3.0 license.
hhtps///gituu..oo//upppa com/lupantech/Math Vista
hhups://githubbcom
www.e/coooommmoeeeeee fllm-project/lllm
hhtpp://wwww. kaggle.
"https://www
htttp://www.svgrepppt httpppp.ooomaa///ccccnnng///////
"hutps://www wwwwwwwwwwww.....wwwwwwwwwwww..w www.a//ecco//aaaaatee/eeeeeeeeee www..//wwwooodaeeee/////py/aeeee
21

---

Published as a conference paper at ICLR 2025
A.2 EVALUATION PROTOCOL
Inference. To conduct massive-scale evaluation (8000+ question samples with 16 models), we leverage vLLM to perform high-throughput and memory-effiiiet inference for all the open-source LLMs. And we use OpenAl API for evaluating different variants of the GPT models. We deploy the vI.LM inference engine as server that also uses the same format as OpenAl API to achieve a unified testing framework for both GPT and all open-source models. The vLLM inference engine is deployed on a node with 8 NVIDIA H100 80G GPUs.
Evaluation. We benchmark the performance of all the models via the question answering accuracy. Following the common protocol [34], we ask the model to generate the answer sentence in a formatted way (see text prompt examples in F.2). Then, we extract the target answer from the output sentence by parsing the answer according to its position. If the extracted answer matches the ground truth, will count as 1, otherwise i's O. We use the average accuracies for the results in Table 1.
Enhanced answer extraction with LLM: In our experiment, we found the Symbolic Instruction Tuning makes the model less capable in following the formatting instruction. This is likely due to our fine-tuning only uses symbolic graphies description, which causes the model to forget its instruction following skill. Therefore, the model after SIT often answers questions correctly but in a different format. This affects the aforementioned format-based answer extraction. For example, given a color-grounding question of the input subject, the formatted answer should be "The answer is A) Yellow.", however, the model outputs "The car is yellow". To mitigate this issue, we follow the GPT-enhanced answer extraction of Mathvista", where we present both the question options and model's output to GPT4 to extract the answer in the formatted way. A 5-shot CoT is also applied here to augment the robustness of extraction process (More details in F.2). The results on Table 3 obtained with the enhanced answer extraction. More details of the SIT evaluation can be found in Appendix E.
A.3 EVALUATED MODEL SPECS
Here we list the model details of all LLMs we evaluated in the CGP-Bench, their performance are demonstrated in the table 3. Generally, evaluated 3 types of LLMs, the representative open-soureed LLMs from tech giants and start-ups, the code-specific LLMs that were built for code generation and understanding and the strongest proprietary models from the GPT and Claude family.
A.3.1 OPEN SOURCED LLMS
Demma--11-2B// Gemma is a suite of lightweight, advanced open models created by Google DeepMind and other teams across Google, it's the best performing model class when it's released on Feb 21, 2024. Primarily designed for text generation, Gemma models come in multiple sizes, i.c. 2B 7B, to fit various computing resources and deployment needs. The models are trained on 3T (2B) 6T (7B) tokens of Coinarily-Englisse data from web, mathematics, and code. It is based on a transformer decoder with context length of 8192 tokens. It leverages Multi-Query Attention, RoPE Embeddings, GeGLU Activations and RMSNorm. The Gemma models are using architectures. data and training recipes inspired by the Gemini model family. The models are available in two versions: pretrained version and an Instruction tuned version, the latter refined through human language interactions to perform well in conversational roles, similar to a chat bot. We only test and perform the symbolic instruction tuning on the Instruction tuned version.
Mistral-0.3-7B The Mistral-0.1-78 from Mistral AI is released September 27, 2023 and marked as the best 7B model at the time. The 0.1 version model features 8k context window, with Grouped-query attention (GQA) for faster inference and SWA for handling longer sequences more effectively at a reduced computational cost. The model is further updated to 0.3 version in May 21, 2024, upgrading its context length to 32k, its vocabulery size and RoPE theta, but the SWA is removed in this version.
Mistral-NeMo and Mistral-Large2 Mistral NeMo is a 12B large language model built by Mistral AI with a context window size of up to 128k tokens. Mistral NeMo is trained with quantization
hhttp://giiuub.com/llnnnn hhaps///gtthh..cm///paeetttt/t/t
22

---

Published as a conference paper at ICLR 2025
awareness, allowing FP8 inference without any loss in performance. Mistral NeMo uses new tokenizer, Tekken, based on Tiktoken, which enables more efficient compression of natural language text and source code, compared with previous Mistral model series.
Mistral Large 2 is the new generation of Mistral AI's flagship model, with a model size of 123 billion parameters. Especially, Mistral Large is trained on very large proportion of code data, resulting in state-of-the-art performance, on par with proprietary models like GPT-4o or Clause Opus.
Yi-1.5-9B/34B The Yi model family, developed by -ocussd startup 01.AI, includes 6B and 34B pretrained language models. Their performance is attributed to high-quality data from meticulous data-engineering efforts. For pretraining. 3.1 trillion tokens of English and Chinese corpora were constructed using cascaded data deduplication and quality filtering pipeline. Finetuning involved a carefully refined instruction dataset of fewer than 10K instances, each verified by dedicated machine learning engineers. Built on Transformer architecture, Yi models feature Grouped- -uppedd--rrrruuree Attention (GQA), SwiGLU activation, and RoPE with an adjusted base frequency (RoPE ABF). The Yi-6B base model, with 32 layers, was scaled up to the Yi-9B model, which has 48 layers, by duplicating the original 16 middle layers (layers 12-28). We hence test the Yi-9B model together with the 34B version in SGP-Bench.
InternI.M2-7H5 M22-1///ntemn--M222000000 W0002220////eee.MMMMM. MM2.5-BB InternI.M2 is open-sourced L.M model series developed by Shanghai Al laboratory, with context window length of 200K.
InternLM2.5 is an open-sourced, billion parameter base and chat model with a context window size of 1M. It supports gathering information from more than 100 web pages and has in general a very strong capability in tool utilization.
Aya-23-8B/35B Aya 23 is open-source LLM model series developed by C4AI, featuring advanced multilingual capabilities. Aya-23 is fine-tuned (IFT) to follow human instructions and supports a context window length of 8 8192 tokens.
Command R-35B/104B C4AI Command-R R is a research release of a 35B large language model with open weights, optimized for variety of use cases including reasoning, summarization, and question answering. Command-R has the capability for multilingual generation evaluated in 10 languages and highly performant RAG capabilities. It supports context length of 128K.
C4AT Command-Re is an open-source multilingual L.LM with enhanced features, including Retrieval Augmented Generation (RAG) and tool usage for automating complex tasks. Command excels in multi-step tool usage, allowing the model to combine various tools across multiple steps to complete sophisticated tasks.
Qwen, released in April 2024, developed by Alibaba Cloud, is series of transformer- based large language models pre-trained on diverse data, including web texts, books, code, and more, over 2.2 trillion tokens. The Qwen1.5 5 series includes various sizes of decoder models, each available as a base and aligned chat model, supporting long context lengths (8K tokens for 1.8B, 7B, and 14B models, and 32K tokens for the 72B model). It outperforms similar-scale open-source models on various Chinese and English tasks and even exceeds some larger models in benchmarks. These models feature the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, and a mix of sliding window and full attention mechanisms. They also include an advanced tokenizer for multiple languages and coding languages. Qwen's 's extensive vocabulary of over 150K tokens enhances compatibility with multiple languages, allowing for improved capabilities without expanding the vocabulary.
Bwen=1..-7B//22B/22BBBBB
Qwen-2-72B Qwen2, is the newest series of large language models, developed by Alibaba, which surpasses its previous released Qwen1.5 series, yielding state-of-the-art performance across different benchmarks. Qwen2-72B-Inse -nnstruttttttt has an extended context length of up to 128K and is instruction- aligned with both supervised finetuning and direct preference optimization.
-B773 Meta's Llama 3 is the latest generation of Ilama family, release in April 18, 2024, featuring pretrained and Instruction-fine fine-tuned versions with 8B and 70B parameters. Designed with standard decoder-only transformer architecture, Llama 3 models demonstrate state-of-the-an performance across various industry benchmarks and show improved reasoning capabilities. Key enhancements include a tokenizer with 128K token vocabulary for efficient language encoding and grouped query attention (GQA) for better inference efficiency.
23

---

Published as a conference paper at ICLR 2025
Llama 3 models pretrained on extensive dataset of over 15T tokens from publicly available sources, including a significant increase in code and high-quality non-Em -on-English data covering 30+ languages. This dataset is seven times larger than that used for Llama 2, ensuring superior model performance.
For Instruction-tunie Llama 3 employs combination of supervised fine-tuning (SFT), rejection sampling, proximal policy optimization (PPO), and direct preference optimization (DPO). This approach, coupled with meticulously curated data and multiple quality assurance rounds, significantly enhances model alignment and response diversity.
Llama3. 100.33.1-8/70//00000 Introduced in July 2024, Llama 3.1 pretrained 15 trillion tokens of data from publicly available sources as well as over 25M synthetically generated examples. The intruction-tuned variants are post-trained using supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences to guarantee safety and helpfulness. The Llama 3.1 405B demonstrates competitive performance across 150 benchmarks with leading foundation models, including GPT-4o and Claude 3.5 Sonnet.
A.3.2 CODE-SPECIFIC LLMS
CodeQwenl.5577 CodeQwenl 1.5-BB is based on Qwen 1.5-BB. It is further trained on 3T tokens of code data, and it also includes group query attention (GQA) for efficient inference.
is an tixture-of-Eppert (MoE) code language model, that demonstrates comparable performance to GPT-4 Turbo code- related tasks. Specifically. PrepSeek-Coderf V2 is continued to pre-train on intermediate checkpoint of DeepSeek-V2 with 6 trillion additional tokens, substantially enhance its reasoning capabilities in code and mathematical related tasks. DeepSeek-Code -odeee--ooorVV V2 supports 338 programming languages and has a context length of 128K.
DeepSeek-Codee -aw.eee---oooo----------e- 12-16---stttt DeepSeek-Codent Cooeee--ooor---------
2odestrall222B-1e Codestral-22B-vo is the code-specific variant of Mistral-O.1 0.1-22B, it's trained on a diverse dataset of 80+ programming languages, including the most popular ones, such as Python, Java, C, C++, JavaScript, and Bash.
A.3.3 GPT FAMILY
GPT-3.5t (GPT-3.5 Turbo) is text only language model released by OpenAl on November 2022. The specific version of the model we are using is gpt-3.5-turbo-00 September 2021 and context window with length of 16K tokens.
It has knowledge cutoff of
GPT-4t (GPT-4 Turbo) is vision language model launched by OpenAl on March 2023. The specific It has updated knowledge cutoff of April 2023 and context window with length of 128K tokens. It is more powerful than GPT-3.5.
version of the model we are using is 111-4-uurb--00240044
GPT-4o (GPT44 Omni) is multimodal model released by OpenAl on May 2024, which support data types such as audio, vision, and text. The specific version the model we are using gpt-4o- 2024-05-13. It has similar performance as GPT-4t on English text and code, but with significant improvement on non-English text, í.e., over 50 languages. At the same time, is able to reason with vision input. GPT-4o has knowledge up October 2023 and supports context window with length of 128K tokens.
GPT-4o mini is multimodal model released by OpenAl in July 2024, which is more cost-efficient and smaller modal than GPT-4. has a context window size of 128K tokens and the has knowledge up October 2023.
A.3.4 CLAUDE FAMILY
Claude is a multimodal, multilingual, proprietary model series developed by Anthropic. The Claude series includes different models: Haiku, the fastest and most lightweight model; Sonnet, the best balanced model between performance and speed; and Opus, the highest-performing model. We did not evaluate Claude 3 Opus because, in June 2024, Anthropic released Claude 3.5 Sonnet, the newest berf-performing model.
24

---

Published as a conference paper at ICLR 2025
Specifically, using 20240229 for Claude 3 Sonnet, and -oadde--aaiku-2000 for Claude Haiku for benchmark evaluation.
for Claude 3.5 Sonnet, (laude-3-sonnee
25

---

Published as a conference paper at ICLR 2025
B HUMAN STUDY DETAILS
We ran a human study to verify the labels produced by GPT-4o for the benchmark over a subset of 500 stimuli. We recruited 55 participants from the crowdsourcing platform, Prolific [73]. Stimuli were batched into 10 sets of 50 stimuli each. Each participant was randomly assigned batch of 50 stimuli; stimuli were presented in a random shuffled.. On each trial, participants saw the question, original image, and set of multiple choice options. Participants selected an option by clicking button. We include an example screenshot of trial in Figure 14. Participants were paid at a base rate of $12.50/hr. They were informed that they could receive bonus up to $15/hr based on the amount of correct answers they achieved. All participants received the full bonus. Our study was approved by our institutional ethics review board, and all participants provided informed consent. We include the set of instructions and sample screenshots in Figures 15 and 16, respectively. We found high inter-annotator agreement (participants in the same batch had between 0.7 0.85 Fleiss Kappa's alpha agreement, where higher implies higher agreement). We find that humans' mode response matched GPT-4o 90% of the examples (450 of the 500 stimuli).
What shape are the windows on the building?
Crcular
Triangular
Hexagonal
Figure 14: Example survey question.
26

---

Published as a conference paper at ICLR 2025
mecoomal
ponductignn an thenr anderand Your getamand horaa sbour riuut Your waardh coghtts roenos aarizaery
sapasiment, you arowering muliple cheice questions
Mease indicato resporie the tutitttto your chocoe.
Sorea ounnt or very eaay may horder: Take much you nead aach quastion. Raase sure not
There optional boruaup so 515thour for qui resporaes.
xfrelte
Nor
Figure Experiment instructions.
27

---

Published as a conference paper at ICLR 2025
your tnceledge yoforey tight you dont lrow mery shonyes Prttotattorooo again.
riaal
hrvide oeptore for ebual somees
Now will poupa your fina anove
agai aos
Chicing
slake.
Getina
New JAN TOROy bight
Figure 16: Experiment instructions (continued).
28

---

Published as a conference paper at ICLR 2025
C SVG INVARIANCE ILLUSTRATION
Our SVG - Invariance test is essential for testing whether model has a fundamental understanding of the code, it is able to pass the benchmark due memorizing the SVG code samples, since we built our SVG-Bench using public available SVG datasets. In Figure 17 we see two SVG codes, illustrating two samples, that are semantically identical. The rotated sample is generated by ourself by applying SE(2) transformation on the original sample (from SVG Icons). We can see that semantically these two samples are identical, the code changed drastically.
<agg Cd-"Capa. (eayleeessblle background
298,66;"" versioe"11.1 ViewBox-"0 298.6667
298 xm 1-10pt
-puth
20..0.0..44711111. 1.4833.342 0,157.264.5
(1)
11,.24441......03..... 1.200.70033.00.77 154122
095, 145 399--1
167,)
440
858
19,2.416
5 3, 792
63.7922148.044 047
63,238.066322-
377-3.766. 3663.1 wwW.33.83333.6
633-5,33,22
K27-8 ---55-333333
192.
-08-999-0000000 000.090--50---1088 445--445333 35922 406. 900
199-116
02.043,2.9335.9000 0...6655...00
9.011
198-0 477-12.0-111
1-7.28.5.889
e-3.519, 333519.166 (6.39.66666630
527 12. 456
43.046
0.738-
5588999
-7.25:6 5,899 25,61880
Original
22.677,0..40 (0-04-4455.. 323.600.966 999--79900, 1.900 9,209,3.49
19--176636, 111111120000 211,222000044
<igg
<log?
Stee alpnCape 1" abde-" enuble beckspound 295.657 298 6677 veruume" 1.1" viewboo- 10 298.667 208.660 andopuece proveive y-"0ps"
der 155001
PETEMOA0ASRR- R04 THE TNEOWSTONA, TWOT 6AHROATKK T17A04555TTT TRE
434426 05 HESSWOO JWAN TINTSTT S TTMAWN CEETAS ARGST 1902TER O7 PPORNTECS748 (0)
45 3501 2832 FOROS 3467 T00C ST DONARESSIEELE ST CONEEETT 14205 SEORE SO60ETETT 200 (00SE28560L
71054139938
1162009 124 249 AKESI
DDALLSS 37422200 1ST 1S252 5140013 1797346
259.33022 0140 222
2442355 ISOOUR 401 LOOS USSE 2OR 11922500 DES 114ESTO RESTSOOT LOON, FO0
3443895
1,242 545576905758L
AERICO PERCSSSOONS CONESCTNETIIOO ACCOSETE TIN OON
10550 THLSROO ISS
305901 254 160T BEO0S, DETOOTIACNA TECCAS TOI TIGAES CETTETORE SESTONS TO LOSK SECEES ATSTIZI (919
177 STT0S44 STTATTAATOONE TONESTTI ACROE SOOEEO BOTAED TRUY RECORR THE 1OCT4TED
Rotated
10TERNA:
SATHANDEETT
CETEEAUI SAUIISS STTA #TWIRESUSTT
TON6 T
FOR TAOTR RASRRAAEE TTAMAS 1SAANET
20 IIRETTSAR DACTT DENCTMERRHHATT PATT TO TIDEN TR6TST AM TAO I7TOWATTE THA AER PAINST TI AROWACS
012 1112271 72966700 2000E 7466 169520 4263 207776777480 (6774423333 15, 18264745500222 S4.60 1239 148001120
Figure 17: Illustrtioon of the SVG Invariance test.
29

---

Published as a conference paper at ICLR 2025
D MORE EXAMPLES IN SGP-BENCH
D.1 SVG DATA
wnd oud
(TS20
How hales visible the objuct??
What the handle
of the object?
Are
Square Trimple
Cirde U-itaze
Three
DoFosr
Animals
Animals
What unimal dees the objuct represnt!
What unimal daes the object repreant?
Aclion
It: Exploon Di
A: Hepporr
R:Ginilk.
C. Dog
CiLin
D. Mackey
Book
Book
Sheg "oee, Phople Agpondd waw
1994
What po sbject
What oalor the telephan recelver on the object?
Waayy agdln "illi 1007
depicted in the image?
(7,44 42.
Siniidook B:
Di
Green Yellow
Clothing
Clething
What the color the collar af the shirt the image?
What type sbject
depicted in the image?
Red Blue
Vellow
Mal B.Stce.
Glove
dffic Austore
Figure 18: SVG examples SGP-Bench.
30

---

Published as a conference paper at ICLR 2025
Musical Instrument
Time Clock
M
renay objeet??
mano object?
A:5
B
Rt:2
C:7
D
D:44
Acrial Crafi Crafhs
Beverage
object
primuary
A. Ainplaee
B: Best
A. Yellow
B: Blaa
C.
D: Train
C. Red
D: Green
Building
Computer
What shupe
What type objjctt shoma the imnge??
windows buildin??
stsuurt socurt coc.aaa
Lapup Smamppons
A: Clouler
Trinngll
C. Rectasasiar
D: Hexagomal
Entertainment
Land Crafts
type abjoct
imagy?
object
B: Dice
A:
Cwiss
D. Doninaes
C. Three
Foo
Figure examples
31

---

Published as a conference paper at ICLR 2025
Dairy
Food
manyy bales abjecc?
What shape promizently
C.4
A:3
D:5
R:3
C: C:Tningl
At Star
Il: Circle
Food
Food
Spain
opuarta
1963
www.aaaaaaaaaaeeeo..o.oo can soot--oooo couua soud skae 500d (448
What's the abject
How many distinct wertias
amgs?
abject's main body?
C. bayele
Aupopeideccn
mack D: ban
and SAA entu tew sous pocke
A:
D.5
Purniture
Furnlture
many joints aem object have?
many vieible legs abject the have?
velurra AEARATI Affic
One Turee
BeSs D. Four
Ao C. Four
B: TTrce
D. Five
sioge
preniieet object?
Imsago?
Gloess Boed M:Mutthhe
Green
puse 24
Figure 20: SVG examples in our SGP Bench.
32

---

Published as a conference paper at ICLR 2025
Stience
What the object in Image?
What the lipaad
AuElask
A. Blse
B. Velilis
C: Green
Red
221113
grometric shape the rraser the pencil?
Hew many rows buttoss are there the abjject the lmaage?
Rouste Cielll
Triangg
That" (omteroenet
A:
D:
It:3
Figure SVG examples SGP-Bench.
33

---

Published as a conference paper at ICLR 2025
D.2 CAD DATA
Operations
SOL.
And (A (hi pott (TSUEU
Areal 153.2/55.2
Liner 1553,
S00:
25,,
22211228.11.1222
SOB
Linert 1783:
SO0:
SOL:
SOEL
SOL,
Cledler
Hew visible cylindrical CAD abject have?
How visible or screw holes en CAD abject? Two Four
Extil 92,64, 192,1100011000000 18,55444
Like 211 ATT
110 2012
Newbbdy Kine- sidedd
FOS
C: Throe
SOB
Euc (R2, (R2PP 22, TAL 100.224 satee--iaaaa siattttio
Throe
Operations
Operations
SOI:
Line: (/nc:/23..222
and Cidercti
Line:1 (133,153%
Line: (128.128
boddy Ohoosidllll SOI..
Clicke 138,44
Exi (192,64. 99,399999,1888 20,
How many holes visiblc on the flange of the CAD abject?
What type of feature the side of object?
EOS
A: Two
B: Three
indemationn hole
protrusion bump
C: Tour
D: Five
Operations
SOL:
4 Operations
Line (213, 12500t lneotto 200 139g
Cirde 12RUA))
Cirdle 126249)
Extr 64,4 192, 12U22220.
Newbudy One sidedt.
Nowebaayy
Cirdee
Bake 112 112.119 10.
Extx (4, 192, 12 12
Newbady Onee sided)
SOL:
Clnderi 178,
Extr 64, 197, 178.00 99,178
CaLLDaee-allel di.
What feature
Cus One-addddl.
What the primary body object?
Cledet 1775, 128,
object?
Clrder 178,
Bar'l TOU (09,00
1:
D: surfice
1 190, 190,194 (4,,7K, 171 12
A: Square
Tringge
2005
Juin Oha--edadl;;
C. CyLander D. Sphire
Operations
Operations
Liwortti shookh fer afti vig aed the tity
Liaud (1 (Si (171 (001t1
Daii coups ro, coppeeted salccs
DIC DS uou 1o ChUCh thene dek
2002 Uuka frae dete
What the shape the base of the CAD object?
Bass 28 120. 221. LSALS 3.14 www.uaauuaaaaaoooo Dee-didddd
supports does CAD object have?
many
Liwostty E20ce tee 1213 2011 1o 126.1205
A: Trinngle
B: Square
Two
B:
C: Rectingle
Cirels
C: Six
D. Eight
Higure 22: CAD examples (3D) in our Sen-Bench.
34

---

Published as a conference paper at ICLR 2025
Operations
Operations
Rorrti th en Namberr santaetyfimnaattt demmaantic
Chr
Chodle H9.
500
Buc ROAA 34 15 271 Sortwbyyrreeedeee
cype geumetric feature is the contral hole of the CAD abjeet?
(111
2it 237 www.200220 pokae www.lllllee
What the primary shape of the CAD object?
Senciii and 19 you you is you attemeeting
C: Chicular
Square
D: Triangulte
Hesagonal
C: Rettingutrr
A: Trimnsauu
shapdd
Opeerations
SOLL
OOperations
2015 auethiy seeptry
Frectii the COO 20050000 2r by select
Eor 10 MUNAAOS mo ctuke dravidult
3128, I2EAND 1241 Eiuuts
Awautti, shirns
Rap 122
208:
Arc 125, Eaaed 120, 130s
2044
SOEE
What shope primarily makes up the botty of the CAD object?
SOLL
slww)
What primary feature
CAD abjeet?
Low (21a 22g 2o1 sew (33 co
Square Triangle Cyllinder
A. Cylinder
B. Syhare
C: Cube
D: Cone
Operationn
Operations
SOL:
Bestth LoL.E 21 Linoee
SOL:
Clicle 776
Newbudy Oneskdddd
SOL:
Clide (17 A9g
Newbady One-skded);;
SOL,
Extx192,44, 222._/44..2..200.2222....8 20..222...22,,.....22,
shage
Cut Saldlll
How many arms does the CAD object have extending from central body?
main vertical frature
SOL,
CAD abjeet?
Cube
B: Sphere
Jotn Oue--4ieed
A: One
B: Two
C-Cylinder
D: Cone
T.
Four
SDN
Operations
Operations
120 2227
Ious it rebunn INL. Nemboah ikled
11 1n0 10 1011
Exg pn 182 Mi 201 (Ruili (il.eae siell
SOK.
ubols
SOlLAko 12822322 DE Lan (2L.aat
10 21..-112220
S01,
2ww. (128,2204
Kati 14 Lakke 1304, FEL (20 LSS SOl, Mo
1o U2S (26, 101, Diiuhe Diee saledh
A 11LA 126 200
MiNe222 2o (12.2222
How cylindrical rods perpendicalar cylinder CAD image?
SOE Carderd
What shape is the main body the CAD object?
Int 188 (My (M thi siledt
SOL, Condect 22122221111 21.111 Preeeded:
123 2laad (128,2004
(211)
A. Square
B: Cylinder
C: Sphere D: Cone
806
Figure 23: CAD examples (3D) SGP-Bench.
35

---

Published as a conference paper at ICLR 2025
visible
muny are visible on the CAD object?
abjecst
Ate
Two
II: Two
C C.TTase
D: Four
Three Four
What the oricntation of the curved boom is rebstion to the base?
smaller cytinder rolative to larger cylinder is the CAD object?
A: Pespendicllrr C: Deginal
Pupendicaler Punllel angle
Inclinal
2000
Figure 24: CAD examples (3Demples) SGP Bench.
36

---

Published as a conference paper at ICLR 2025
What is the shape the outermost figure in the image?
How many cireles are showa the image?
Thangle
N:Thee
B::
Square
Rectingle
smaller outside of the larger circle?
What type angles are formed at the corners the inner shape in image?
Striggt
Figure 25: CAD examples (2D) in our SGP-Bench.
37

---

Published as a conference paper at ICLR 2025
D.3 SYMBOLIC INSTRUCTION-POLLOWNNG DATA (SVG)
Balor batter desrlee piued ipperthtt SVG cods
Rakos jus instarion desa plied pcsiiles Sar
cngy ae te uppiy uadd repar Give SVG calioo
(SVG)
The dust dnneraaaee
and ber
-Cie saacn Thes cap capes tlc whick choness pale wi suald sarsec and
"H
maueese on kolker de uneiwii poueuut Cmearos coles
offonee Busarder.
Belee intiactio firt desorer tell, piknol will appethid futler conted Widea
Bdde in Eattutuee rat doxito parce wntsus ope thi provida crnics Moxca
proenii Gutare The g SVG wade (SVG)
Lighightig big ao coneeii The SVG onde (LVG)
slecpae moppre
aprul
20l
repoest eo thit igropnaa peaddyyyyy crepie in the pep Huse gave 2010 eacs, expire the cyjer
repe no te sarrlllllyy cerp repan the object ao gedergeess with
dasicaiually bad ai rosal icatis
tha Iely rightg wake das
Brrnn
"Syalidee coogle acolt
devern hordd
Lokee ser corting alsc
Figure 26: SVG examples of our symbolic instruction. following data.
38

---

Published as a conference paper at ICLR 2025
E DETAILS AND MORE RESULTS OF SYMBOLIC INSTRUCTION TUNING
E IMPLEMENTATION DETAILS
We use the unsloth framwork to finetune the base models (lamaa-Sb-insteer and Gemma-1.1- 1.1-7b-it For both models, we use the exact same training setting: we finetune the base models with LoRA [36] on NVIDIA H100 80GB gpu with learning rate 2e-4, batch size of 2 and for epoch.
We use the PEFT framework to test different fine-tuning methods when performing SIT. We choose two common fine-tuning methods LoRA [36] and orthogonal finetuning [75, 60] to fine-tune the base model 1.lam3.1 1-aaaa.---b-Inmsee --bb-nnstruc.. For both fine-tuning methods, we train on 8 NVIDIA H100 80GB gpus with learning rate Ic-4, per device batch size of I and for epoch.
As introduced in Section 6, also use PEFT to test if SIT can improve generic instruction tuning, by mixing our curated SIT data into the publicly available instruction tuning dataset open-instrucr" We use LoRA to fine-tune the base model Llama3. 1-8b on 8 NVIDIA H100 80GB gpus with learning rate lc-4, per device batch size of 1 and for epoch. We test mixing with different SIT data splits, including 10K, 25K, 40K, 55K, and 72K. For example, Open- Open-Instrrn Ins--nnttttt----.- 10K, Open- Instruct-rev-STT--O0 and Openn-nntructt-iixed-S----
are constructed by mixing Open-Instruct with SIT-1OK, rev-SIT-10K and mixed-SIT- 10K. More specifically, rev-SIT-10K is constructed from SIT-10K according to Figure 12, while mixed-SIT- 10K uniformly samples exactly 5K of the SIT instructionn-fllooiing pairs and convert them to rev-SIT, while the rest 5K is kept unchanged. The best result is reported in the Table 4. We employ the widely-used mn-evaluation-hamme 12 to obtain the results on a variety of LLM benchmarks.
10 /htpp:://ithhub.o httpps//gthiib.oo///ggggggg com/huggingffac//pcftt
"htps://hugiingfacc http:/gnnnnnnaacocaaatttt///////
Phups://githu..co www.////oeeeea/////ee//aa www.thherr//mmmeecaaatnnnnnaaaaa
39

---

Published as a conference paper at ICLR 2025
E.2 MORE EXPERIMENTS IN SYMBOLIC INSTRUCTION TUNING
We additionally provide an ablation study of using different-size SIT data to finetune the base LLMs and measure their performance after SIT the SGP-Bench. We uniformly sample 72k SVG programs from the SVG Icons dataset to build an instruction-following dataset using the text prompt examples in F.1 query GPT-4v. The SIT-25K dataset is built by choosing the samples with the shortest code length out of the original 72k instruction following pairs. The SIT-10K dataset is subset of SIT-25k, by uniformly sampling from the SIT-25k dataset. For SIT-40k and SIT-55k, we additionally sample more data with short code length from the SVG Icons dataset and mix it with SIT-25k. In this way, we can ensure that the smaller SIT dataset is always a subset of the bigger one.
We use the LLM-based evaluation because noticed that after SIT, the generic instruction- ability the finetuned model degerates compared to the original model. We want to eliminate the cases where the finetuned model answers the questions correctly but do not follow the answer template, so that a matching-basee evaluation will fail to extract meaningful answers. The results shown in the Table 6. We notice that for Llama3-8B-instee semantic understanding up-to some SIT data size, afterwards the performance degenerates, while for Gemma-1..17b6 the overall semantic understanding improves significantly without noticeable degeneration.
the SIT will improve the generic
Dataset Size
Ilamm3-8B
Gemma-7B
Original
43,20
39.33
STT 10k
48.16 (+4.96)
45.60 (+6.27)
SIT-25k
51.43 (+8.23)
46.87 (+7.54)
SIT-40k
45.62 (+2.42)
45.21 (+5.88)
SIT:55k
40.99 (:2.21)
47.28 (+7.95)
Table 6: Ablation study of studying the effect of different-sized SIT data on the model's performance on the SPG-Bench (SVG-Undesttanding) using based answer extraction.
We also conducted an ablation study to determine whether the finetuning method affects the SIT results. Our findings indicate that the enhancement in the model's ability to understand symbolic programs is independent of the finetuning approach. Both OFT and LoRA significantly improve the model's 's understanding of symbolic programs, shown in Table 7.
LoRA
Llama 3.1-8B
46.7
46.7
47.9 (+1.2)
(+1.3)
STT-25k
49.8 (43.1)
50.3 (43.6)
SIT-40K
$1.0 (+4.3)
51.2 (+4.5)
STT-55k
51.3 (+4.6)
51.4 (44.7)
Table 7: Ablation study of studying the effect of different-sized SIT data on the model's performance the SPG-Bench (SVG-Undestaadding using LLM-based answer extraction and different fine-tuning methods, TThe value differs from the value main table, because we use LLM-based evaluation guarantee consistency.
40

---

Published as a conference paper at ICLR 2025
F
TEXT PROMPT TEMPLATE
F.1 TEMPLATE FOR BENCHMARK CONSTRUCTION
We randomly sample from the following 20 prompts generate the Symbolic Instruction Tuning (SIT) data:
"Describe in detail the semantic or geometric characteristics of the object shown in the image."
"Offer a detailed description. of the geometric or semantic attributes of the object in this image.
"Can you provide detailed account of the geometric or semantic features of the object in the image?" "Give a comprehensive description of the semantic or geometric properties the object depicted in the image." "
"Elaborate on the geometric semantic features of the object the image."
"Provide an in-depth description of the semantic or geometric aspects of the object shown in the image." "Detail the semantic or geometric features of the object in the image."
"Explain in detail the semantic geometric characteristies of the object displayed in the image."
"Could you detail the geometric or semantic features of the object in the image?"
need a detailed description attributes of the object in the image."
the geometric or semantic
"Please describe the semantic or geometric features of the object in the image comprehensively."
"Provide a thorough description of the geometric or semantic properties of the object in this image."
"Can you elaborate on the semantic or geometric features of the object in the image?"
"Describe precisely the semantic or geometric characteristies of the object shown in the image." "Give a detailed explanation of the geometric or semantic features of the object in the image."
"Offer a complete description of the semantic or geometric aspects of the object in the image."
"Detail the geometric or semantic properties of the object depicted in the image."
"Explain the semantic geometric features of the object in the image in detail."
"Provide detailed analysis of the geometric semantic features of the object in this image." "Elaborate the semantic and geometric characteristics of the object shown in the image."
41

---

Published as a conference paper at ICLR 2025
We use the following prompt to query GPT to construct the SGP-Bench (SVG) question- answer pairs.
Given the image, contruct in total 4 miltiple-choice question- answer pairs, with answer choices A, B, C, D, that concentrate on the semantics or geometric features of the object in the ing. The first three questions are random.
The forth question should ask about the semantic of the whole object.
Note: the format of the question-answer pairs should be as follows:
===
Question: what is the capital of Germany?
Options: A) Rome; B) Beijing; C) Paris; D) Berlin
Answer:
Question: What is the color the sky on clear day?
Options: A) Gray; B) Blue; C) Orange; D) Green
Answer: B
We use the following prompt to query GPT construct the SGP-Bench (CAD) question-answer pairs:
Construct five multiple-choice question-answer pairs, with answer choices A, B, C, D, that concentrate the geometzy of the CAD object in the image.
Note: the format of the question-answer pairs should be as follows:
Question: How did Spider-Man get his powers?
Options: A) Bitten by a radioactive spider; B) Born with them; C) Military experiment gone awry; D) Woke with them after strange dream
Answer: D
Question: What is the color of the sky clear day?
Options: A) Gray; B) Blue; C) Orange; D) Green
Answer: B
We randomly sample from the following 20 prompts to construct the questions for the SGP-MNIST benchmark. We do not need to query GPT because we have the ground truth label for every SGP. MNIST code.
"What number between 0 and 9 is shown in this picture?" "Identify the digit from to depicted this image."
is illustrated in this image?" "Can you tell which digit (0-9). this image represents?"
"Which number from 0 through
"What is the digit, from 0 to 9, that appears in this image?"
"Determine the digit between and 9 displayed in this image."
"Spot the digit (0-9) thar this image portrays."
"Which of the digits 0-9 does this image il illustrate?"
"Recognize the digit from 0-9 shown in this picture."
"From 0 to 9, what digit is shown here in this image?"
"What single digit from 0-9 is presented in this image?"
"Specify the digit (0-9) that is represented by this image."
"What digit, ranging from to 9, does this image show?"
"Identify which one of the digits 0-9 is depicted in this img."
42

---

Published as a conference paper at ICLR 2025
"Name the digit between 0 and that this image represents."
"Which digit, through 9, is displayed in this picture?" "Tell which digit from 0 to is shown in this image."
"Pinpoint the digit from 0-9 represented in this image. "What digit from the range 0-9 is depicted in this image?"
"Indicate which digit (0-9) is illustrated in this image."
Given the question-anse andtwer pairs, either through querying GPT (SVG) randomly sample from pre-defined prompt pool, we use the following question template to construct the questions for our SGP-Bench (SVG):
Examine the following SVG code carefully and answer the question based on your interpretation of the rendered image.
(SVG)
Question: (Question)
Given the question-answer pairs use the following question template to construct the questions for our SGP-Bench (CAD):
Examine the following CAD code carefully understand 3D object it generates and answer the question based on your interpretation the rendered image of that object.
(CAD)
Hint: the CAD code has the following syntax:
(Hint)
Question:
(Question)
When constructing the SGP-Bench (CAD), also provide the syntax of CAD code:
CAD code consists of a sequencee of CAD commands that describe
object.
The comnands fall into two categories: sketch and extrusion. Sketch commands are used to specify closed curves on in 3D space. Each closed curve is referred as a loop, and one or more loops form a closed region called profile. always starts with an indicator command <SOL> followed by series of curve commands. All the curves on the loop are in counterclockwise order, beginning with the curve whose starting point is at the most bottom-left.
plane
loop
total, there are three possible curve commands: Line, Arc, and Circle.
line y): line, with x, y line end-point.
Arc (x, u, f): an arc, with x,y as are end-point, u as sweep angle and counter-clockvise, f=1 means it is not counter-clockvise. circle (x, y, r): a circle, with x,y as the center point the radius.
whether it is counter-clockisse, f-0 means it is
The extrusion command has two purposes:
1) It extrudes sketch profile from 2D plane into 3D body, and the extrusion type can be either one-sided, symmetric, two-sided with respect to the profile's sketch plane. 2) The command also specifies (through the parameter b in Ext) how to merge the newly extruded 3D body with the previously
43

---

Published as a conference paper at ICLR 2025
created shape by one of the boolean operations: either creating a new body, or joining, cutting, or intersecting with the existing body.
(x, 2, o, Pr the sketch plane orientation, o, P, q as the sketch plane origin, as the scale of the associated sketch profile, e, as the extrude distances towards both sides, b as the type of merge operation (could be New-body operation, join operation, cut operation and intersect operation) and as the extrude type (could be one-sided, symmetric or two-sided). <EOS> means the end of the code.
e, f, b, u): extrude operation, with
X, Y,
CAD code consists of sequence of CAD commands that describe 3D object.
The commands fall into two categories: sketch and extrusion. Sketch commands are used to specify closed curves on a 2D plane in 3D space. Each closed curve is referred as loop, and one or more loops form a closed region called profile. A loop always starts with an indicator command LOOP followed by series of curve commands.
Possible primitive types are defined with the following parameters: Arc start_ cicclleecnntee.ppii...raaaaaooo: Line (start_ppoon,, rational,
center_point, normul,orraal,
Circle eng.opoiittil.niussccrrrreeeeeer
( _ normall))) normal), Blljppaccakkrrrmmmooooorrssseser
The extrusion command ExtrudeFeaturee (operation, start_extent, extent_type, extent_one, extent_two) has two purposes: 1) It extrudes sketch profile from a 2D plane into 3D bedy, and the extrusion operation can be either one-sided, synmetric, two-sided with respect to the profile's sketch plane.
2) The command also specifies (extent_type) how to merge the newly extruded 3D body with the previously created shape by one of the boolean operations: either creating a new body, or joining, cutting intersecting with the existing body.
CAD code consists of a sequence of CAD commands that describe 2D object.
primitive and constraint total, there are five possible primitive types: Point (x, y), Line (dirX, dirY, patX, pntY, startParam, endParam) Circle (xCenter, yCenter, xDir, yDir, radius, clockwise), Are (xCenter, yCenter, xDir, yDir, radius, clockwise, startParam endParam), and Ellipse (xXeetee, (xCenter, yCenter, xDir, yDir, radius, minorRadius, clockwise).
The commands fall into two categories:
x, y: the point coordinates. dirX, dirY: the unit direction vector.
xCenter, yCenter: the coordinates the center point. clockwise: boolean value that indicates the orientation of the unit direction vector.
pntX, pntY (Line): the coordinates of a point on the line. startParam, endParam (Line): signed start/end point distances to the point (pntX, pntY).
startParam, endParam (Arc) start/end angles to the unit
44

---

Published as a conference paper at ICLR 2025
direction vector.
Al1 primitives have an isConstruction boolean parameter
indicating if primitive is to be physically realized simply serve as a reference for other primitives. All constraints act on at least primitive, indicated by the corresponding number.
F.2 TEMPLATE FOR EVALUATING MODELS ON SGP-BENCH
When evaluating different models on our SGP-Bench, we use the following evaluation template (multiple choice):
Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: SLETTER' (without quotes) where LETTER is of ABCD. Think step by step before answering.
(Question)
A) (A)
B) (B)
C) (C)
D) (D)
Important, the last line of your response must be of the following format:
'Answer: SLETTER' (without quotes) where LETTER must be one of A, B, C or D.
When evaluating different models on our SGP-Bench, we use the following evaluation template (generation):
Solve the fol following problem step by step. The last line of your response should be of the form Answer: $ANSWER (without quotes) where $ANSWER is the answer to the problem.
(Question)
Important, put your answer on its own line after "Answer:", and you do not need to use Vboxed command.
We use the following template, when we perform LLM-based evaluation:
Please read the following example. Then extract the answer from the model response and type it at the end of the prompt.
Hint: The last line of your response should be of the following format: 'Answer: SLETTER' (without quotes) where LETTER
one of ABCD.
Question: What the primary color of the object in the image?
A) Red
B) Blue
C) Black
D) Green
Model response: **Step 1: Examine the image* AnAnThe image
45

---

Published as a conference paper at ICLR 2025
consists of various shapes filled with different colors. We need to identify the primary color of the object in the image. Sn\n 2: Focus the dominant color++\n\nfh, shapes that cover the largest area in the object are filled with shades of blue and its variations, An\ne=Answer: B**
Extracted answer: B
Hint: The last line of your response should be of the following format: 'Answer: SLETTER' (without quotes) where LETTER is one of ABCD.
Question: What is the background color of the image?
A) Red
B) Green
C) Blue
D) Yellow
Model response: Answer: The background color is blue.
Extracted answer: C
Hint: The last line of your response should be of the following
format: 'Answer: SLETTER' (without quotes) where LETTER one of ABCD.
Question: What is the shape of the buckle on the jbcc??
A) Circle
B) Triangle
C) Square
D) Hexagon
Model response: Answer: D) Hexagon.
Extracted answer: D
Hint: The last line of your response should be of the following
format: 'Answer: SLETTER' (without quotes) where LETTER is one of ABCD.
Question: What type of object is shown in the image?
A) Watch
B) Belt
C) Bracelet
D) Necklace
Model response: The object in the code is watch.
Extracted answer: A
Hint: The last line of your response should be of the following
format: 'Answer: SLETTER' (without quotes) where LETTER is one of ABCD,
Question: What is the primary color of the object in the image?
A) Blue
B) Yellow
C) Green
D) Red
46

---

Published as a conference paper at ICLR 2025
Model response: The primary color of the object in the code is yellow.
Extracted answer: B
47


Reviews:
[{"review_id": "W3MydB19f7", "reviewer": "Reviewer_zWHc", "summary": "The paper assesses large language models (LLMs) for their ability to \"understand\" raw symbolic graphics programs like SVG and CAD without direct visual input. By creating a benchmark (SGP-Bench), the authors test this through tasks requiring semantic-level answers and transformation invariance, providing insight into how LLMs interpret non-image-based graphics data. The authors also introduce Symbolic Instruction Tuning (SIT), a method to fine-tune LLMs with specific symbolic graphics data, show that its enhancing both specialized and general reasoning abilities.", "questions": "1. Regarding Weakness 1, it appears the model learns from structured data without evidence of internal image rendering, likely due to lack of instruction. It would be interesting to test whether explicitly prompting for “internal rendering” as a Chain-of-Thought (CoT) step impacts performance.\n2. I'm curious whether any experiments have been conducted using rendered images as input in multi-modal LLMs to compare performance with raw SVG code inputs, providing insight into how well models perform with visual representations.\n3. In Appendix E.2, it’s noted that fine-tuning SIT-55k on Llama3-8B actually degrades performance. Is this due to answers not following the specified template, or are the answers correctly formatted but simply inaccurate?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative in presenting a benchmark and dataset that adds value to multi-modal LLM and vision foundation model research.  \n2. Extensive experimentation with well-designed elements, particularly in the fine-grained categorization of SVG-Understanding. While categories like \"color\" and \"shape\" might be addressed through subpattern matching in SVG codes, \"Semantics\" and \"Reasoning\" better assess true reasoning capability.  \n3. Clear presentation and valuable insights.", "weaknesses": "1. The SGP-MNIST experiments show limited spatial reasoning in LLMs, suggesting that some claims about models’ spatial reasoning or “visual imagery” abilities may be overstated.  \n2. The conclusion is somewhat underwhelming, reiterating known principles such as scaling laws and fine-tuning effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5Y41Ayn3u", "reviewer": "Reviewer_ESu8", "summary": "This paper introduces a benchmark designed to assess LLMs'  comprehension of symbolic graphics programs in SVG format, two 3D CAD formats (DeepCAD and Fusion360), and one 2D CAD format (SketchGraphs). The authors present evaluation results across several LLMs and introduce a symbolic instruction dataset, showing that fine-tuning on this dataset can enhance LLM reasoning capabilities in general.", "questions": "1. In L76-L77, the authors claim that \"The order of symbolic operations may substantially affect semantic meaning\". However, this does not seem applicable to SVG, where objects are defined by coordinates and unaffected by drawing order. Could this be clarified?\n2. In Figure 6, what qualifies as an \"operation\" in vector graphics? For example, does a polyline with multiple vertices count as one operation or several?\n3. Inconsistencies in notation between Figure 6(b) (symbol $3D_{Recon}$) and the text (symbol $3D_{complex}$ in L257-L258) need clarification.\n4. During fine-tuning, was there any overlap between training and test datasets? Additionally, could more detail be provided on the construction of the symbolic instruction dataset such as whether it includes both SVG and CAD formats?\n5. If the enhanced answer extraction with LLM is applied consistently across all experiments or is only applied to models trained on SIT data?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The benchmark dataset is large, with 4,340 questions in the SVG set and 2,400 in the CAD set.\n2. Both 2D and 3D objects are evaluated.\n3. The paper addresses data leakage issues by utilizing global transformations like translations and rotations.\n4. The proposed instruction tuning dataset improves general reasoning ability in LLMs.\n5. The paper provides a critical view on current LLMs' capability.", "weaknesses": "1. The human study reports 90% labeling accuracy, leaving 10% mislabeled data, which could significantly impact the results given limited variation of the accuracy in LLM's evaluation. I suggest doing a human filtering across the whole benchmark to improve the accuracy of the ground truth answer.\n2. The method for generating questions on 3D dataset from limited rendered views may lead to errors, especially in counting tasks; for example, the second part Figure 7 (row 3, column 3) is confusing. The question \"How many visible cylindrical pins are there on the CAD object?\" doesn't make sense since \"visible\" doesn't have a meaning given a code representing a 3D object without perspective information.\n3. Several parts of the writing need further clarification (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yEb0RZUuh0", "reviewer": "Reviewer_rpKP", "summary": "The authors explore the use of symbolic graphical programs (SGPs) as a means to assess the reasoning abilities of large language models (LLMs). SGPs are defined using visual markup languages and can be compiled by symbolic programs to produce images, such as SVGs in 2D or CAD models in 2D/3D. The central hypothesis is that the high-level reasoning skills of an LLM are linked to its capacity to (1) identify complex objects from primitive shapes and answer questions about them (Semantic Understanding) and (2) recognize when different sets of primitive shapes produce the same overall program (Consistency). To test this hypothesis quantitatively, the authors compiled approximately 4,000 program-question pairs for SVG images and 5,000 pairs for CAD programs. Their findings reveal that (1) open-weight models significantly lag behind proprietary ones in performance, (2) fine-tuning using an intermediate representation substantially improves reasoning performance, and (3) all models face challenges in distinguishing MNIST digits presented as SVG schemas.", "questions": "(Merged with the Weaknesses Section)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- SGPs offer an intriguing middle ground between perception and reasoning. While humans can intuitively link rendered images to semantic concepts, associating these same concepts with the underlying SVG schema may demand additional time and coding knowledge.\n- The dataset presented is highly scalable, as millions of SVG images and CAD programs with permissive licenses are available online for use in this benchmark.\n- The proposed experiments are both well-motivated and effectively executed.\n- The presentation is clear, consistent, and transparent.", "weaknesses": "* L108: \"highly scalable benchmark.\" While I agree that SGPs have this potential, the paper does not demonstrate specific contributions toward achieving scalability. For example, benchmarks like [MineDojo](https://minedojo.org/) achieve high scalability by developing tools to efficiently scrape relevant data from platforms like YouTube, Reddit, and Wikia, resulting in a dataset of around 700k Minecraft videos and 6M+ comments. In contrast, the SGP dataset in this paper only contains ~5,000 data points. This is surprising given the abundance of online SVG images (for example, [SVGRepo](https://www.svgrepo.com/) claims to have over 500k SVGs). If SGP-Bench were a \"highly scalable\" benchmark, I would have expected an order of magnitude more data points or an explanation of the barriers preventing larger-scale sampling. Although this does not diminish the significance of the benchmark (Multimodal reasoning benchmark do not have to be \"internet-scale\" to be influential: https://arxiv.org/abs/2401.06209), the scalability claims should be adjusted accordingly\n* The current definition of symbolic graphical programs might be overly broad. For instance, HTML could technically qualify as an SGP within the given definition since it uses a consistent markup language and rendering engines (like Chrome, Safari, or Firefox). While this isn’t inherently a weakness, it could impact the paper's novelty claims. If the SGP definition admits HTML as well, the authors might need to expand their related work and adjust their novelty claims to include efforts like https://arxiv.org/pdf/2210.03347, https://mbzuai-llm.github.io/webpage2code/, https://arxiv.org/abs/2403.09029 (and other relevant insights from work on action transformers for webpage understanding).  \n\n- [Figure 4] the comparison between OpenAI-o1 (a model likely using Chain of Thought reasoning) and standard zero-shot LLM queries using Llama3.1 70B seems problematic. However, this qualitative example showcases how  CoT reasoning might be a natural way for increasing LLM performance on this benchmark. Is there a particular reason the benchmarked models (Table 1)  do not evaluate the respective models under the CoT (or similar multi-step reasoning) paradigm?  \n\n- What are some potential \"reasoning shortcuts\" that could enable an LLM to perform well on this dataset without possessing true high-level reasoning abilities? For instance:\n  - Data Leakage: Since the SVGs were sourced from the internet, it’s possible that proprietary LLMs were trained on some SVG/caption pairs. The authors propose the semantic consistency experiments to evaluate this. I believe that this partially mitigate this issue (as the jittered images likely wouldn't appear in the training set), but I'm concerned that jittering the image might not modify the underlying XML structure enough to conclusively reject the influence of data leakage. There are many ways to further validate this.\n    - Calculate the tree edit distance (The Zhang Shasha algorithm is implemented in `pip install zss`) between the original and jittered SVG. Is the tree-edit-distance above a reasonable minimum threshold? Is the tree-edit-distance correlated with the relative performance difference? If low absolute correlation is observed, then the performance is invariant to the SVG structure, and data leakage might be absent.\n    - Evaluate the performance of open-sourced models on obfuscated SVG files (Tools like https://github.com/svg/svgo will minify SVGs). If the minifier significantly alters the XML structure and performance remains unchanged, it could indicate no data leakage. \n    - Of course, the authors may have other effective methods to test this. This isn’t necessarily a criticism of the paper, but rather a recognition that  the benchmark offers a unique opportunity to measure performance gains in LLM reasoning attributed to data leakage.\n  \n- L396: \"No semantic instruction dataset directly over symbolic graphical programs.\" This paper https://mikewangwzhl.github.io/VDLM/ seems to propose an alternative semantically meaningful intermediate representation. I'd be interested in the authors' opinions on this paper.\n- Generally, this paper is missing a discussion on the limitations of this benchmark. Here are some prompting questions: \n  - Are all natural images expressible as SVGs? What does this mean for the comprehensiveness of this benchmark?\n  -  Are SVGs / CAD programs representative of all types of symbolic graphical programs? What about visual specification languages like OpenGL, Taichi, etc? What about programming languages like Turtle/LOGO which admit recursive structures?\n  - What are the limitations of SGP-Bench's scalability?\n\nIn conclusion, I'm currently recommending this paper for a **Weak Rejection**. While the benchmark is undoubtedly interesting to the ICLR community, it is crucial to reconcile the claims in this manuscript with those from the existing literature before the paper is accepted for publication. I am open to discussing these points with the authors and potentially revising my recommendation during the discussion period.\n\n----\n\nAfter discussions with the authors, I'm revising my recommendation to **Accept**. Overall, SGPs offer a promising avenue to assess the reasoning abilities of LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)