Can Large Language Models Understand Symbolic Graphics Programs?

## Abstract

Abstract Assessing the capabilities of large language models (LLMs) is important to be able to characterize what they know (and don’t) and how they can be appropriately used. Yet, capability assessments are often challenging, in part, because it is hard to find tasks to which they have not been exposed during training.
We take one step to address this challenge by turning to a new task: focusing in on symbolic graphics programs, which are a popular representation for graphics content that procedurally generates visual data.
Large language models (LLMs) have shown exciting promise towards program synthesis, but do they “understand” symbolic graphics programs?
Unlike conventional programs, symbolic graphics programs can be translated to graphics content ( e.g. , 2D images, 3D geometry).
Here, we characterize an LLM’s “understanding” of symbolic programs in terms of their ability to answer questions related to the graphics (spatial) content.
This task is challenging as the questions are difficult to answer from the symbolic programs alone – yet, they would be easy to answer from the corresponding graphics content as we verify through a human experiment.
To understand symbolic programs, LLMs may need to possess the ability to “imagine” how the corresponding graphics content would look without directly accessing the rendered visual content.
We use this task to evaluate LLMs by creating a large benchmark for the semantic understanding of symbolic graphics programs. This benchmark is built via a novel usage of program-graphics correspondence, hence requiring minimal human efforts.
We evaluate both commercial and open-source LLMs on our benchmark to elucidate a preliminary assessment of their ability to reason about visual scenes from programs.
We find that this task well distinguishes existing LLMs and models that are considered good at reasoning perform better.
Lastly, we introduce a way to improve this ability – Symbolic Instruction Tuning (SIT).
Specifically, we query powerful vision-language models ( e.g. , GPT-4o) with questions and images generated by symbolic programs. These program-question pairs are collected as our instruction dataset which is then used to finetune an LLM. With a small amount of data, we find that SIT can improve the understanding of LLMs regarding symbolic graphics programs.
Assessing how well models “understand” symbolic graphics programs offers new possibilities for LLMs to perform visual reasoning. Finally, we showcase such possibilities in generic instruction tuning.

### 1 Introduction

What are large language models (LLMs) capable of? Recent studies [, ] have shown that LLMs are able to generate generic computer programs, indicating a certain level of understanding about the symbolic structure of programs. Motivated by such progress, we focus on another important family of computer programs, called symbolic graphics programs, where a graphics content (*e.g.*, image, 3D asset) can be generated by running a program. We are interested in the following question: *Can large language models “understand” symbolic graphics programs?*

Before trying to answer this question, we start by defining what we consider “understanding” of symbolic graphics programs, in the context of this work. Because a (deterministic) graphics program can be uniquely rendered to an image (the graphics programs we consider here), we characterize LLMs’ understanding of the graphics program as the semantic understanding of the corresponding rendered image. More specifically, we approximate such a semantic visual understanding by the ability to correctly answer semantic questions only based on the raw program input. These semantic questions are generated based on the rendered image, such that they are easy to answer given the image and yet challenging given only the program as text prompt. Guided by this insight, we propose a generic pipeline for creating benchmarks that can evaluate this particular ability of LLMs to understand symbolic graphics programs, while requiring minimal human efforts. While we of course recognize that there are other elements of visual reasoning that characterize understanding in humans, and that we want to evaluate in machine intelligences, we believe that our benchmark provides insight into one element of “understanding” over symbolic graphics programs that enables deeper understanding of what current LLMs are (and are not) capable of.

Figure: Figure 1: Our benchmark assesses LLMs’ understanding of symbolic graphics programs in semantic understanding and prediction consistency.
Refer to caption: /html/2408.08313/assets/x1.png

The semantic understanding of symbolic graphics programs is particularly interesting in the following aspects. First, such a semantic understanding may necessitate a form of “visual imagination” ability that enables the LLM to “imagine” how the produced graphics content visually looks like. If a LLM can perfectly answer every possible semantic question of a symbolic graphics program (with sufficient knowledge, the graphics content can be reconstructed), then we can say that such a LLM may have a good “visual imagination” of this program. Second, symbolic programs represent a procedural way to generate the graphics content, and hence the semantic understanding also requires the long-range sequential reasoning of the program. The order of the symbolic operations may substantially affect its semantic meaning, making the problem quite challenging.
Third, many semantic questions involve an accurate grounding of semantic components, and such a grounding in the symbolic program requires a fine-grained understanding of the program structure. This motivates us to study whether LLMs have the ability to semantically understand symbolic graphics programs, and furthermore, how to improve this ability. In general, correctly answering semantic questions about symbolic graphics programs requires a combination of multiple sophisticated reasoning abilities from LLMs, which therefore makes the task of symbolic graphics program understanding an ideal benchmark to contribute towards evaluating the holistic reasoning capabilities of LLMs. Reasoning over symbolic programs is of particular interest from a cognitive perspective as well [, , ]. To what extent LLMs can operate over such a representation with rich structures remains an open problem.

Motivated by the significance of symbolic graphics program understanding, we build a benchmark, called *SGP-Bench*, for two important variants of symbolic graphics programs: scalable vector graphics (SVG) as a generic language for representing 2D vector graphics, and customized computer-aided design (CAD) as a domain-specific language (DSL) for representing 2D/3D objects. Our benchmark consists of two types of evaluations. (1) *Semantic understanding*: We construct a number of semantic questions (*i.e.*, multiple-choice questions with 4 options) from a set of images (from multiple different categories). These questions are fed to LLMs along with the symbolic program to evaluate the semantic understanding. (2) *Semantic consistency*: To evaluate the robustness of LLM’s semantic understanding, we perform random translation and rotation to the original symbolic programs and then test the same semantic questions based on the perturbed programs. We evaluate the consistency of the answers from LLMs using these perturbed symbolic programs with identical semantic meaning. This evaluation can also help lower the possibility of test data leakage, because the randomly perturbed programs are unlikely to be seen during pretraining. An overview of SGP-Bench is given in Figure [1](#S1.F1). We further validate our automated labels via a human study (see Appendix [B](#A2)).

In addition to performing evaluation under the common in-context setting wherein LLMs are used “out-of-the-box” and not finetuned, we also evaluate whether finetuning LLMs on a curated dataset can boost performance. To this end, we propose a new method, *Symbolic Instruction Tuning* (SIT) to improve the semantic understanding of symbolic programs. The key idea is to collect an instruction-following dataset based on the rendered images. Because the semantic questions of interest are usually easy to answer from the visual input, we take advantage of the rendered images (that correspond to symbolic programs) and query a powerful language-vision model (*e.g.*, GPT-4o) for detailed captioning and question-answering. This leads to a scalable way to collect an instruction-following dataset for symbolic graphics programs. Then, we simply finetune open-source LLMs on this dataset. Our experiments demonstrate that SIT can indeed improve a model’s semantic understanding of symbolic programs. We are excited about future work, motivated by our benchmark, to improve this element of symbolic graphics program understanding. Our contributions are summarized below:

- •
We introduce a new task of symbolic graphics program understanding and propose a generic yet highly scalable benchmark creation pipeline for this task.
- •
We build a large benchmark, SGP-Bench, for comprehensively evaluating LLM’s semantic understanding and consistency of symbolic graphics programs. In SGP-Bench, we consider two types of symbolic graphics programs: SVG for 2D vector graphics and CAD for 2D/3D objects.
- •
To improve the symbolic program understanding, we collect an instruction-following dataset and propose a new finetuning method, called symbolic instruction tuning.
- •
We introduce a symbolic MNIST dataset, where the symbolic problem understanding can be extremely challenging so that even the most powerful LLM (*i.e.*, GPT-4o) can only obtain nearly a chance-level performance while the rendered image remains very easy to recognize. This result suggests that how LLMs understand symbolic graphics programs is quite different from human.
- •
Finally, we find that SIT can also improve generic instruction following performance.

### 2 Related Work

Symbolic graphics programs. Generating visual data by procedural modeling with symbolic programs has been essential to computer graphics since its inception, particularly for 2D shapes and 3D geometry. They offer a concise, structured, and interpretable representation, attracting significant attention in the field — refer to [] for an overview. Common program types include constructive-solid geometry (CSG) [, , , , ], computer-aided design (CAD) [, , , , ], vector graphics (*e.g.*, SVG) [, ], L-systems [], and customized domains [, , , , , ]. Among these, SVGs, constructed from primitive shapes like vector paths, curves, or polygons defined by mathematical equations, are widely used in artistic applications such as icon creation and typography.
Central to SVGs is the vector path, providing detailed control over graphics and including primitives like rectangles, ellipses, and text.
Similarly, procedural 3D geometric modeling, particularly in CAD applications, involves parameterized operations that can be adjusted and re-executed to produce updated geometry. Datasets like the ABC dataset [] and Fusion 360 Gallery [] aid CAD research and machine learning, offering hierarchical decomposition, joints, contact surfaces, construction sequences, and shape segmentation based on modeling operations. Our paper focuses on graphics programs of SVG and CAD by introducing a new semantic understanding task that requires a challenging reasoning with the programs.

Graphics program understanding and generation.
As graphics programs often provide compact, scalable and potentially more semantic descriptions compared to raw pixels and voxels, it has been widely explored to discover graphics programs for 2D images like 2D hand drawings and synthetic patterns [, , , , , , ], for 3D objects represented in voxels and meshes [, , , , , , ] and for 3D scenes represented by multi-view images [, , , , , , , , , , ].
[] infers custom-designed markup code from images that can be easily translated to renderer-friendly inputs. In follow-up work, [] explore how graphics programs can be used for visual question answering (VQA). Recently, [] has advanced this direction by examining large language models (LLMs) for synthesizing graphics programs to reconstruct visual input. In contrast, we benchmark LLMs to perform semantic-level question answering, similar to VQA, but use graphics programs as input without relying on any visual modality.

Large language models. LLMs have demonstrated growing potential in many applications, ranging from mathematical problem solving and theorem proving assistance [, , , ] to aiding biological discovery [, , , ].
Applying LLMs for programming tasks is also a popular direction of research.
Specifically, many works have explored topics such as code retrieval [], automated testing [, , ], repairing [, , , , ], documentation [, ], and generation [, , , , ].
These abilities of understanding and generating programs are usually gained from pretraining or finetuning on large datasets of code.
Our work investigates LLMs’ capability of understanding symbolic graphics programs, which differs significantly from the prior works since the semantic meaning of graphics programs are often defined visually by their corresponding graphics.

Relevant benchmarks and datasets. There exist many benchmarks in evaluating different aspects of LLMs: AI safety/ethics [, ], out-of-distribution performance [, ],
API/tool usage [], code generation [], *etc*. Perhaps the most relevant aspect of LLM evaluation to our task is (non-graphics) program understanding abilities, for which a number of papers have explored [, , , , , , , , , ].
As graphics programs can be turned into images via rendering, it is also highly relevant to our tasks to investigate how vision-language models are capable of visual understanding. To evaluate the performance of different models for this related task, numerous types of public multi-modal datasets on generic domains are used, such as human-like image-caption pair datasets (*e.g.*, COCO Caption [], Nocaps [], and Flickr30k []) and visual question answering datasets (*e.g.*, GQA [], OK-VQA [], VQAv2 [], and Vizwiz []). Some specific domain datasets exist as well, such as TextVQA [] that involves questions on OCR text in images, ScienceQA [] that involves commonsense questions on scientific topics and Youcook2 [] that provides temporal information in the form of video clips. These datasets are heterogeneous in both data formats and evaluation metrics and as a result performing a comprehensive evaluation for visual understaning of VLMs is highly tedious. It is also worth pointing out that some more recent datasets like OwlEval [] and MME [] are proposed to alleviate this issue with more unified and comprehensive evaluation protocols. Different from all existing benchmarks, SGP-Bench is one of the first benchmarks to evaluate the semantic understanding of symbolic graphics programs.

### 3 Semantic Understanding of Symbolic Graphics Programs

Figure: Figure 2: Illustration of the symbolic graphics program understanding task.
Refer to caption: /html/2408.08313/assets/x2.png

We introduce the task of semantic symbolic graphics program understanding. Our goal is to assess to what extent a LLM is able to “understand” a symbolic graphics program, which may begin to belie some latent capability to “visually imagine”. Specifically, we leverage the correspondence between deterministic symbolic graphics programs and rendered images, and then we characterize the understanding of symbolic graphics programs as the semantic understanding of the corresponding rendered image. To do so, we use the performance on question-answering to evaluate the semantic understanding of images. The same set of questions, along with the corresponding symbolic graphics programs, are then used to evaluate the symbolic program understanding of LLMs (the rendered image will not be used here). Figure [2](#S3.F2) gives an illustration of symbolic graphics program understanding. The intuition behind this evaluation is that, if an LLM has a good sense of the symbolic graphics and implicit de-rendering, then the LLM should have a rough understanding about its rendered image such that it is able to answer arbitrary semantic questions regarding this rendered image.

Symbolic graphics program understanding can also be viewed as a form of visual question answering in the sense that visual input is represented by a symbolic program representation. Compared to existing vision-language models [, , ] that encodes images with a text-aligned neural encoder [], our paper considers the case where the visual input are encoded by a symbolic program that can exactly recover the graphics content. From this perspective, our task aims to study and uncover the potential of using symbolic programs as a representation to perform visual reasoning.

### 4 SGP-Bench: A Benchmark for Symbolic Graphics Program Understanding

#### 4.1 Dataset Creation Pipeline

Figure: Figure 3: Dataset construction procedure.
Refer to caption: /html/2408.08313/assets/x3.png

To construct our benchmark, we need questions about a symbolic program based on its rendered image. To build a large benchmark, it is essential to consider how we can effectively scale up the question collection with minimal human efforts. To this end, we use a powerful vision-language model (*e.g.*, GPT-4o) to generate semantic questions based on the rendered images, and then we inspect them manually to make sure that these questions are reasonable and the answer to them is correct. We also run a human study over a randomized set of $500$ of the automatically generated questions along with the corresponding images, and find high agreement (see Appendix [B](#A2)). The overall procedure for our dataset creation is given in Figure [3](#S4.F3). In this pipeline, the rendering of symbolic programs and the GPT-4o querying are both scalable and can be done with minimal human involvement. Human annotators then inspect the generated question-answer pairs based on the rendered image, which requires much less efforts than manually writing questions and answers. We emphasize that this program-question-answer triplet creation method is general, as it works for most of the symbolic graphics programs. SVG and 2D CAD programs can directly produce 2D images, so it is straightforward to use this pipeline. For 3D CAD programs, they produce 3D models and we first render them into 2D images with a few fixed camera positions. These rendered images from different views are then combined together to query GPT-4o, and the following procedures are identical to the SVG case.

Figure: Figure 4: Key dataset statistics for both SVG and CAD programs. We show the distribution of the operation number in a program for both SVG and CAD data, together with the number of examples of each category in the SVG dataset.
Refer to caption: /html/2408.08313/assets/x4.png

#### 4.2 Benchmarking Semantic Understanding

SVG dataset statistics. We collect in total $1,085$ SVG programs covering $19$ categories, and each program has $4$ semantic multiple-choice questions (with $4$ options). We make sure that answers are evenly distributed across $4$ options. Detailed dataset statistics are in Figure [4](#S4.F4)(a). Our SVG benchmark consists of $5$ different types of questions, including “Semantic”: $1,085$ questions, “Color”: $864$ questions, “Shape”: $1,217$ questions, “Count”: $819$ questions, and “Reasoning”: $355$ questions. “Semantic” tests the global semantic meaning of the object represented by SVG codes, while the other four question types focus on detailed, local understanding of the object. “Color” is color-related questions about specific object parts, which evaluates the localization of the corresponding semantic part. “Count” is about counting the occurrences of certain patterns or semantic parts. “Shape” is about the shape of certain parts of the object, either finding the exact definition of the geometric shape or finding geometric shapes that resemble the object part. Figure [5](#S4.F5) gives some SVG examples.

CAD dataset statistics. We collect $2,400$ CAD programs from three different datasets [, , ]. The CAD dataset consists of $1000$ programs from DeepCAD [], which forms the 3D subset; $700$ programs from the Fusion360 Reconstruction Dataset [], which constitutes the 3Drecon. subset; and $700$ programs from SketchGraphs [], which makes up the 2D subset (as shown in Table [1](#S4.T1)). Depending on the data, we either render one image or two images from two opposite isometric views (*i.e.*, front and back). Then we feed the images to GPT-4o and generate one semantic multiple-choice question (with 4 options) and its answer. This gives us $2,400$ questions in total. We make sure that ground truth answers are evenly distributed across $4$ options. Detailed dataset statistics are given in Figure [4](#S4.F4)(b). Some examples from our CAD dataset are provided in Figure [5](#S4.F5).

Figure: Figure 5: Example questions for SVG and CAD programs. Due to the space limit, we omit the programs and only show the rendered images.
Refer to caption: /html/2408.08313/assets/x5.png

Experimental results and discussion.
We find that graphics program “understanding”, as we operationalize it here, is challenging. The average accuracy of all models (proprietary and open-source) is below 65% (ranging from 31% to 63%) on SVG and below 80% (ranging from 29% to 78%) on CAD.
Among this, SVG makes it even more difficult for the models to understand as these 2D graphics contain richer semantics.
However, the models have substantial performance gains in accordance with the scaling law [], and this increase, with larger model size, is observed across various open-source LLMs. For example, Llama-3’s score is improved from 0.429 to 0.548 on SVG and from 0.633 to 0.694 when its size increased from 8B to 70B, Qwen-1.5’s from 0.376 to 0.499 on SVG and 0.560 to 0.694 on CAD with the size from 7B to 110B. We also notice consistent improvements given the same model size and model family but from different generations. For example, Qwen-72B from 0.466 to 0.537, Llama-8B from 0.429 to 0.465 and Llama-70B from 0.548 to 0.574. The consistent performance gain on both the SVG and CAD programs indicates that the semantic understanding of symbolic graphics programs, including both 2D semantic understanding and 3D geometry understanding, is a fundamental capability that is aligned with the scaling law of LLMs, even if these models have not been specifically trained for this task.

Compared to the open-sourced LLMs that we consider here, proprietary models (GPTs) and (Claudes) outperform most of them by a large margin. Within the family of current most popular GPTs, we see a performance improvement with a 27% boost (from GPT-3.5’s 0.498 to GPT-4’s 0.633) when evaluating these GPT variants on our SGP-Bench. This result is aligned with the seeming improvement of reasoning ability in the GPT family, validating that SGP-Bench can well distinguish different LLMs. The overall best-performing model in both SVG and CAD is Claude 3.5 Sonnet.

The semantic understanding of graphics programs can be probed across different aspects, ranging from attribute-level investigations of “color” and “shape” to higher-level discussions of “semantics”, “counting” and “reasoning”. Our benchmark is designed to cover these investigations for LLMs.
Most LLMs perform well on color-related questions, followed by shape-related questions, with “count” and “semantic” questions showing progressively lower performance. This consistency is intriguing, as it resembles the coarse-to-fine structure of visual information processing. “Color” is the most visually salient feature, “shape” understanding requires a finer grasp of global and local structures, and “count” and “semantic” questions demand deeper comprehension and knowledge. The difficulty curve is evident, with most open-source models achieving roughly half the accuracy on semantic questions compared to color questions. For instance, the best-performing open-source model, Llama3-405B, achieves 37.6% accuracy on semantics, while reaching 81.6% accuracy on color grounding tasks. While open-source models struggle with “semantic” questions, GPT models perform surprisingly well, with semantics being their second-best category after color grounding.

**Table 1: Performance of various LLMs on SGP-Bench. This table evaluates how well models understand SVG inputs (’SVG - Understanding’) and their behavior under random perturbations of these inputs (’SVG - Invariance’). It also assesses 3D & 2D semantic understanding of CAD code. We found the results demonstrate the "scaling law", with larger LLMs in the same family showing superior performance. Bold texts indicates performance with 1st rank, and underlined texts indicates performance with 2nd rank and 3rd rank.**
| Model | SVG - Understanding | SVG - Invariance | CAD |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Avg | Semantics | Count | Color | Shape | Reason | T Avg. | SE(2) Avg. | T Cons. | SE(2) Cons. | Avg | 3D | 3Drecon. | 2D |  |
| Open-source generic LLM |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Gemma-1.1-2B | 0.317 | 0.321 | 0.333 | 0.25 | 0.356 | 0.287 | 0.312 | 0.270 | 0.954 | 0.920 | 0.278 | 0.294 | 0.253 | 0.281 |
| Gemma-1.1-7B | 0.393 | 0.347 | 0.275 | 0.453 | 0.523 | 0.299 | 0.403 | 0.390 | 0.917 | 0.894 | 0.476 | 0.497 | 0.464 | 0.460 |
| InternLM2-7B | 0.382 | 0.279 | 0.324 | 0.570 | 0.431 | 0.299 | 0.381 | 0.381 | 0.788 | 0.772 | 0.480 | 0.551 | 0.446 | 0.411 |
| InternLM2-20B | 0.424 | 0.255 | 0.379 | 0.623 | 0.483 | 0.276 | 0.426 | 0.407 | 0.777 | 0.727 | 0.525 | 0.586 | 0.490 | 0.474 |
| InternLM2.5-7B | 0.421 | 0.273 | 0.317 | 0.598 | 0.515 | 0.282 | 0.419 | 0.404 | 0.809 | 0.778 | 0.562 | 0.639 | 0.506 | 0.509 |
| Yi-1.5-9B | 0.355 | 0.309 | 0.404 | 0.493 | 0.297 | 0.301 | 0.372 | 0.374 | 0.947 | 0.947 | 0.469 | 0.581 | 0.416 | 0.361 |
| Yi-1.5-34B | 0.443 | 0.308 | 0.364 | 0.644 | 0.523 | 0.234 | 0.446 | 0.423 | 0.845 | 0.819 | 0.583 | 0.649 | 0.563 | 0.510 |
| Aya-23-8B | 0.290 | 0.244 | 0.255 | 0.343 | 0.326 | 0.259 | 0.290 | 0.273 | 0.942 | 0.896 | 0.428 | 0.508 | 0.384 | 0.359 |
| Aya-23-35B | 0.442 | 0.307 | 0.354 | 0.648 | 0.511 | 0.318 | 0.451 | 0.434 | 0.898 | 0.857 | 0.488 | 0.551 | 0.429 | 0.457 |
| Command R-35B | 0.461 | 0.311 | 0.442 | 0.676 | 0.495 | 0.341 | 0.478 | 0.443 | 0.833 | 0.803 | 0.536 | 0.579 | 0.509 | 0.504 |
| Command R-104B | 0.500 | 0.339 | 0.449 | 0.727 | 0.565 | 0.341 | 0.521 | 0.477 | 0.917 | 0.875 | 0.583 | 0.634 | 0.570 | 0.524 |
| Qwen-1.5-7B | 0.376 | 0.226 | 0.317 | 0.563 | 0.471 | 0.234 | 0.371 | 0.382 | 0.792 | 0.780 | 0.486 | 0.560 | 0.426 | 0.443 |
| Qwen-1.5-32B | 0.494 | 0.307 | 0.501 | 0.713 | 0.552 | 0.310 | 0.512 | 0.492 | 0.972 | 0.938 | 0.575 | 0.664 | 0.567 | 0.456 |
| Qwen-1.5-72B | 0.466 | 0.299 | 0.319 | 0.698 | 0.598 | 0.265 | 0.474 | 0.461 | 0.883 | 0.854 | 0.600 | 0.658 | 0.590 | 0.526 |
| Qwen-1.5-110B | 0.499 | 0.324 | 0.431 | 0.734 | 0.560 | 0.332 | 0.486 | 0.470 | 0.839 | 0.821 | 0.632 | 0.711 | 0.607 | 0.546 |
| Qwen-2-72B | 0.537 | 0.373 | 0.426 | 0.770 | 0.630 | 0.372 | 0.520 | 0.491 | 0.869 | 0.852 | 0.692 | 0.753 | 0.669 | 0.630 |
| Mistral-7B v0.3 | 0.417 | 0.304 | 0.324 | 0.624 | 0.470 | 0.296 | 0.434 | 0.417 | 0.919 | 0.895 | 0.495 | 0.551 | 0.481 | 0.429 |
| Mistral-NeMo-12B | 0.449 | 0.296 | 0.355 | 0.652 | 0.548 | 0.296 | 0.480 | 0.443 | 0.894 | 0.855 | 0.568 | 0.623 | 0.549 | 0.510 |
| Mistral-Large2-123B | 0.572 | 0.389 | 0.558 | 0.814 | 0.635 | 0.408 | 0.577 | 0.540 | 0.889 | 0.847 | 0.710 | 0.755 | 0.716 | 0.640 |
| LLama3-8B | 0.429 | 0.304 | 0.372 | 0.626 | 0.484 | 0.293 | 0.426 | 0.410 | 0.905 | 0.873 | 0.550 | 0.633 | 0.472 | 0.512 |
| LLama3-70B | 0.548 | 0.364 | 0.496 | 0.749 | 0.645 | 0.369 | 0.559 | 0.525 | 0.905 | 0.874 | 0.634 | 0.694 | 0.619 | 0.566 |
| LLama3.1-8B | 0.465 | 0.339 | 0.385 | 0.667 | 0.533 | 0.268 | 0.464 | 0.448 | 0.821 | 0.806 | 0.574 | 0.626 | 0.539 | 0.534 |
| LLama3.1-70B | 0.574 | 0.400 | 0.543 | 0.788 | 0.659 | 0.411 | 0.584 | 0.554 | 0.856 | 0.825 | 0.688 | 0.739 | 0.663 | 0.641 |
| LLama3.1-405B | 0.580 | 0.376 | 0.584 | 0.816 | 0.647 | 0.389 | 0.570 | 0.548 | 0.840 | 0.817 | 0.717 | 0.767 | 0.700 | 0.661 |
| Open-source code LLM |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| CodeQwen1.5-7B | 0.301 | 0.245 | 0.262 | 0.344 | 0.387 | 0.245 | 0.327 | 0.324 | 0.883 | 0.859 | 0.376 | 0.419 | 0.350 | 0.340 |
| DeepSeek-Coder-V2-16B-Instruct | 0.451 | 0.309 | 0.379 | 0.637 | 0.548 | 0.268 | 0.496 | 0.476 | 0.902 | 0.867 | 0.547 | 0.611 | 0.521 | 0.481 |
| Mistral-Codestral-22b-v0.1 | 0.491 | 0.309 | 0.446 | 0.698 | 0.581 | 0.321 | 0.503 | 0.470 | 0.860 | 0.816 | 0.602 | 0.659 | 0.577 | 0.547 |
| Proprietary models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-3.5 Turbo | 0.498 | 0.319 | 0.451 | 0.729 | 0.577 | 0.338 | 0.509 | 0.492 | 0.897 | 0.870 | 0.576 | 0.654 | 0.530 | 0.510 |
| GPT-4 Turbo | 0.609 | 0.764 | 0.539 | 0.832 | 0.687 | 0.412 | 0.606 | 0.576 | 0.867 | 0.835 | 0.716 | 0.762 | 0.694 | 0.674 |
| GPT-4o mini | 0.585 | 0.398 | 0.504 | 0.791 | 0.709 | 0.414 | 0.595 | 0.561 | 0.881 | 0.852 | 0.659 | 0.737 | 0.612 | 0.594 |
| GPT-4o | 0.633 | 0.787 | 0.553 | 0.832 | 0.696 | 0.471 | 0.625 | 0.586 | 0.878 | 0.844 | 0.733 | 0.782 | 0.711 | 0.686 |
| Claude 3 Haiku | 0.486 | 0.264 | 0.398 | 0.750 | 0.610 | 0.301 | 0.496 | 0.476 | 0.902 | 0.867 | 0.612 | 0.677 | 0.581 | 0.549 |
| Claude 3 Sonnet | 0.565 | 0.375 | 0.503 | 0.803 | 0.657 | 0.395 | 0.582 | 0.566 | 0.875 | 0.838 | 0.644 | 0.673 | 0.649 | 0.599 |
| Claude 3.5 Sonnet | 0.674 | 0.505 | 0.584 | 0.891 | 0.758 | 0.527 | 0.670 | 0.649 | 0.903 | 0.870 | 0.742 | 0.769 | 0.727 | 0.717 |

#### 4.3 Benchmarking Semantic Consistency

Figure: Figure 6: The semantic consistency test assesses if semantic understanding remains the same when the program is perturbed without semantically changing the image content.
Refer to caption: /html/2408.08313/assets/x6.png

Large language models (LLMs) are exposed to vast amounts of online data, including open-access SVG data. To investigate whether their semantic understanding ability is due to potential data leakage, we propose a semantic consistency test by introducing global translations or rotations to SVG graphics, ensuring SE(2) invariance. Such spatial interventions fundamentally alter the code representation, as SVG graphics consist of lines and Bezier curves with anchor points, and SE(2) operations change all numerical values in the code. However, the SVG’s semantics—such as shape or color—remain unaffected by this perturbation. This allows us to examine how LLMs behave when the same vector graphics are presented with drastic code-numerical changes (see Appendix [A.1](#A1.SS1) for more details). If the model maintains the performance consistency under these perturbations, it may suggest that the semantic understanding is based on a fundamental level of comprehension or visual imagery rather than a trivial memorization of the code.

Dataset specifics. We use our SVG dataset to evaluate the semantic consistency with respect to translation and rotation. For each SVG sample, we randomly choose 5 different translations (T) and rotations plus translations (SE(2), harder case), resulting in a visually small amount of spatial shifts of the rendered object, meaning nearly no changes in semantics, but complete change in SVG code numeric given the shift of SVG’s anchor points and curves. Then we evaluate all the LLMs with the same question set of the SVG-Understanding benchmark but with these perturbed code inputs.

Evaluation. We measure the semantic consistency with two metrics: 1) the average accuracy of all perturbed SVG inputs’ question-answering accuracy, showing the overall accuracy once the samples are intervened; and 2) the proposed “consistency score” that first counts the most common selection for each group of perturbed samples (where they were translated or rotated from the same SVG program), then divided by the number of perturbations. This score indicates how much the LLMs being consistent (no matter right or wrong) regardless of the drastic program change. If the score is close to 1, it means all the predictions are the same even with totally different input codes, vice versa.

Experimental results and discussion.
Our experiments with the SVG-Invariance benchmark demonstrate that most LLMs exhibit robust semantic understanding of graphics programs under translation (T) and translation + rotation (SE(2)) perturbations. In Table [1](#S4.T1), most of the LLMs achieve over 80% consistency in both perturbation settings, with half of the models exceeding 90% consistency. Not only do the models remain consistent in their predictions under perturbations, but their performance on perturbed inputs also shows minimal fluctuation compared to their performance on the SVG-Understanding benchmark. We posit that this indicates that the semantic understanding ability that we evaluate of LLMs is unlikely due to data leakage, but rather, could stem from a potential foundational capability to interpret the semantics of deterministic, symbolic graphics programs.

#### 4.4 Prediction Entropy of LLMs and Humans

Figure: Figure 7: Comparison of prediction entropy.
Refer to caption: /html/2408.08313/assets/x7.png

To evaluate the consensus of different LLMs, we compute the average prediction entropy on $500$ symbolic programs using GPT-4o, LLama3-8B, LLama3-70B, Mistral-7B, Yi-1.5-34B, Gemma-1.1-7B and Qwen-1.5-72B. We also conduct a human experiment on the rendered images of these programs and collect the answers (each question has at least 5 participants, see Appendix [B](#A2)). Figure [7](#S4.F7) show that humans have strong consensus when answering questions based on images. In comparison, LLMs show low consensus when answering questions based on symbolic programs. This implies that LLMs might have different inner working mechanisms to understand symbolic programs. We are excited by future work to better investigate these differences.

Figure: Figure 8: Examples of our instruction data: SVG code, text prompt and paired response (the rendered images are not used for finetuning).
Refer to caption: /html/2408.08313/assets/x8.png

### 5 Symbolic Instruction Tuning: Boosting Symbolic Program Understanding

Generating symbolic instruction data. Inspired by how visual instruction tuning [] enables large vision-language models to understand images with visual-question-answering (VQA) data, we design a new method to perform symbolic instruction tuning for LLMs to take steps to better bridge the gap between the semantic understanding and symbolic reasoning over the graphics programs. While to our knowledge there does not exist any semantic instruction-following datasets directly over symbolic programs, these symbolic graphics programs can be rendered into 2D images. With these images in hand, we can easily query powerful vision-language models (*e.g.*, GPT-4o is used in our case) to obtain a detailed semantic captioning based on the rendered image. The intuition is straightforward, as we want to build an intrinsic connection between semantic natural language description and symbolic programs. The instruction data is created in a similar spirit to our benchmark. We leverage the correspondence between symbolic problems and graphics content, and then use the rendered images to obtain semantically rich description. Following this idea, we construct the first semantic description dataset for symbolic graphics programs. Specifically, for each image rendered from a symbolic graphics program, we prompt GPT-4o to produce a semantically-rich description. Finally, we collect a dataset that contains detailed semantic descriptions for 55K symbolic programs.

Supervised finetuning with symbolic instruction data.
We generally follow the standard instruction fine-tuning procedure (including the default hyperparameter settings) from Alpaca [] and use supervised finetuning to train open-source LLMs with our own symbolic instruction data. To facilitate future research, our symbolic instruction data is also made available at [Hugging Face](https://huggingface.co/sgp-bench).

**Table 2: Performance of SIT.**
| Dataset Size | Accuracy |
| --- | --- |
| Original | 46.7 |
| SIT-10k | 48.0 (+1.3) |
| SIT-25k | 50.3 (+3.6) |
| SIT-40k | 51.2 (+4.5) |
| SIT-55k | 51.4 (+4.7) |

Experimental results.
We perform supervised finetuning on Llama-3.1-8B with orthogonal finetuning [, ] to demonstrate the effectiveness of SIT. In Appendix [E.2](#A5.SS2), we provide results on both Llama-3-8B and Gemma-7B to show that the performance gain is agnostic to the base LLM. The performance of LoRA [] is also given in Appendix [E.2](#A5.SS2) (only slightly worse than OFT). From the experimental results in Table [2](#S5.T2), we observe that SIT has consistently improved the semantic graphics program understanding of LLMs, increasing the performance of Llama-3.1-8B from 46.7% to 51.4% with 55K instruction question-answer pairs. With more instruction data (from 10K to 55K), the performance also increases. We note that Llama-3.1-8B achieves competitive performance among all open-source LLMs after being finetuned with our instruction dataset. The performance is already better than GPT-3.5t. The finetuning results demonstrates that the ability of symbolic graphics program understanding can be improved with SIT. However, boosting the performance of open-source LLMs beyond the capability of GPT-4o (the source of our instruction data) remains an open challenge.

### 6 A Challenge for SVG Program Understanding

Figure: Figure 9: Examples of our SGP-MNIST challenge, hand-written digit constructed by SVG programs.
Refer to caption: /html/2408.08313/assets/x9.png

Is it really easy to answer semantic reasoning questions over symbolic graphics programs? We provide an intriguing experiment to demonstrate that SVG programs can be quite difficult for LLMs to understand such that even if the corresponding rendered images are fairly easy for humans to recognize, all these powerful LLMs still fail dramatically, only reaching a chance-level accuracy.

**Table 3: Accuracy of LLMs on SGP-MNIST.**
| Method | Accuracy |
| --- | --- |
| Yi-1.5-34B | 5.5 |
| LLama3-70B | 10.0 |
| Qwen-1.5-110b | 10.0 |
| Qwen-2-70b | 11.3 |
| GPT-3.5t | 10.2 |
| GPT-4t | 10.6 |
| GPT-4o | 13.0 |

Specifically, we construct symbolic graphics programs that can produce MNIST-like images, as shown in Figure [9](#S6.F9). The symbolic programs that can render MNIST-like images are quite simple, typically only containing 1 or 2 path operations. We introduce the SGP-MNIST challenge where we have 100 symbolic graphics programs per digit (in total 10 digits: 0-9). Therefore, we have 1,000 symbolic graphics programs in total. For each problem, we ask one multiple-choice question (with 10 options of digit 0-9): which digit does the SVG program represent? The results are given in Table [3](#S6.T3).
Despite the fact that these rendered images are quite easy for human to recognize, the corresponding symbolic programs turn out to be difficult for LLMs. Even the powerful GPT-4o can only achieve an accuracy slightly higher than the chance-level. This experiment demonstrates that the kind of symbolic graphics program understanding we explore here can be quite challenging in general, even if some LLMs might seem to work well in some cases. The results suggest that how LLMs understand SVG programs is very different from how humans understand images; better understanding similarities and differences in human and machine reasoning is important if we are to build systems that can appropriately work with us []. There are many exciting yet totally unexplored problems in this task, and we hope that our benchmark can serve as a stepping stone to improving the understanding of symbolic graphics problems for LLMs.

### 7 Improving Generic Instruction Tuning with Symbolic Graphics Programs

Figure: Figure 10: Comparison between original and reverse SIT data.
Refer to caption: /html/2408.08313/assets/x10.png

We test the instruction-tuned models on a variety of popular LLM benchmarks, including benchmarks focusing on language understanding (XNLI [], IFEval [], HellaSwag [], C-Eval [], CoQA [], MMLU [], SQuAD2.0 []), generic reasoning (BigBenchHard [], PIQA [], AGIEval []) and mathematical problem-sovling (Arithmetic [], MathQA [], GSM8k [], ASDiv []). Specifically, we use the Llama-3.1-8B base model (without any instruction tuning), and then the baseline is finetuned with the Open-Instruct dataset that contains 143K question-answer pairs (details are given in Appendix [E.1](#A5.SS1)). We evaluate the effectiveness of using SIT data to improve generic instruction tuning performance by mixing additional SIT data into the Open-Instruct data. Moreover, our SIT data can also be used in a reverse fashion (rev-SIT), *i.e.*, rephrasing the answer as the new question and the question as the new answer. See the comparison between original and reverse SIT data in Figure [10](#S7.F10). We test three ways to using SIT data: (1) mixing original SIT data into the Open-Instruct data; (2) mixing the reverse SIT data into the Open-Instruct data; (3) mixing both original and reverse SIT data into the Open-Instruct data. The results are given in Table [4](#S7.T4). We can observe that mixing SIT data can generally improve the instruction following ability, suggesting that understanding symbolic graphics programs requires a combination of multiple reasoning abilities and training on SIT data can improve these abilities. More interestingly, we find that the reverse usage of SIT data (*i.e.*, symbolic graphics program generation) can improve a set of reasoning abilities that are complementary to symbolic graphics program understanding. The mixture of both original and reverse SIT data can greatly improve the instruction following ability of LLMs and achieve better performance than the Open-Instruct baseline, the Open-Instruct + SIT baseline and the Open-Instruct + rev-SIT baseline.

**Table 4: Results on a variety of popular LLM evaluation benchmarks with or without SIT.**
| Benchmark | Open-Instruct | Open-Instruct-SIT | Open-Instruct-rev-SIT | Open-Instruct-mixed-SIT |
| --- | --- | --- | --- | --- |
| XNLI [] | 41.8 | 43.3 (+1.5) | 43.1 (+1.3) | 42.9 (+1.1) |
| IFEval [] | 14.8 / 24.9 | 16.3 (+1.5) / 28.9 (+4.0) | 18.3 (+3.5) / 30.5 (+5.6) | 16.6 (+1.8) / 29.6 (+4.7) |
| HellaSwag [] | 60.0 | 60.2 (+0.2) | 60.5 (+0.5) | 60.4 (+0.4) |
| AGIEval [] | 23.7 | 30.3 (+6.6) | 31.6 (+7.9) | 29.2 (+5.5) |
| C-Eval [] | 46.4 | 47.9 (+1.5) | 48.0 (+1.6) | 48.1 (+1.7) |
| BigBenchHard [] | 59.5 | 60.7 (+1.2) | 60.2 (+0.7) | 61.2 (+1.7) |
| Arithmetic [] | 89.8 | 91.8 (+2.0) | 90.1 (+0.3) | 91.8 (+2.0) |
| GSM8k [] | 48.2 | 50.7 (+2.5) | 51.0 (+2.8) | 51.5 (+3.3) |
| CoQA [] | 67.9 | 69.1 (+1.2) | 68.7 (+0.8) | 69.1 (+1.2) |
| PIQA [] | 79.9 | 80.3 (+0.4) | 80.3 (+0.4) | 80.4 (+0.5) |
| MMLU [] | 60.4 | 61.0 (+0.6) | 61.1 (+0.7) | 61.6 (+1.2) |
| MathQA [] | 39.3 | 40.4 (+1.1) | 40.3 (+1.0) | 40.7 (+1.4) |
| SQuAD2.0 [] | 28.9 | 28.7 (-0.2) | 31.6 (+2.7) | 29.9 (+1.0) |
| ASDiv [] | 18.5 | 21.8 (+3.3) | 20.1 (+1.6) | 21.3 (+2.8) |

### 8 Concluding Remarks

We propose a new task over which to evaluate LLMs to contribute towards a multi-faceted assessment of their capabilities: the task of “understand” images – without image; that is, directly from graphics programs. For such a task, we create SGP-Bench, a benchmark that shows distinguishable results between LLMs and introduce the method of Symbolic Instruction Finetuning (SIT) that improves LLMs’ capability of understanding graphics programs. Our work helps deepen our collective understanding of what LLMs can and cannot do, and hopefully inspires a broader diversity of evaluation tasks. However, our work only represents the beginning of symbolic graphics program understanding for LLMs. The inner working mechanisms of such semantic understanding of LLMs and how to improve this ability remain to be explored.

### Acknowledgement

The authors would like to thank Yao Feng and Yandong Wen for helpful suggestions. Additionally, HF would like to thank Hanqi Zhou for her support throughout the project, during which time they became engaged (Fun fact: HF and Hanqi Zhou got married a few hours before the submission deadline). The diamond ring featured in Figure [1](#S1.F1) symbolizes this joyous personal milestone and is courtesy of the entire team.
This work was supported in part by the German Federal Ministry of Education and Research (BMBF): Tubingen AI Center, FKZ: 01IS18039B, and by the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. WL was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP XX, project number: 276693517. KMC acknowledges support from the Marshall Scholarship and Cambridge Trust. AW acknowledges support from a Turing AI Fellowship under grant EP/V025279/1, The Alan Turing Institute, and the Leverhulme Trust via CFI. MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB is a consultant for Meshcapade, his research in this project was performed solely at, and funded solely by, the Max Planck Society.

## Appendix

### Appendix A Benchmark Details

We adopt implementations from other projects to build our SGP-Bench. We follow the implemntation of MathVista(^1^11[https://github.com/lupantech/MathVista](https://github.com/lupantech/MathVista)) for querying GPT or open-sourced Llama3.1-8B and perform LLM-based answer extraction, vLLM(^2^22[https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)) for efficient model inference and simple-evals(^3^33[https://github.com/openai/simple-evals](https://github.com/openai/simple-evals)) for a unified benchmarking framework.

Our data license follows the license of the original license of our data source.

#### A.1 Data preparation

###### SGP-Bench (SVG).

Our SVG data are sampled from kaggle SVG Icons dataset(^4^44[https://www.kaggle.com/datasets/victorcondino/svgicons](https://www.kaggle.com/datasets/victorcondino/svgicons)) and we build our SGP-Bench (SVG) using text prompts from [F.1](#A6.SS1). The original data from kaggle SVG Icons is crawled from SVGrepo(^5^55[https://www.svgrepo.com/page/licensing/](https://www.svgrepo.com/page/licensing/)). The website is merely an aggregator, so it follows that any content on it must at least be licensed permissively enough for SVGrepo to distribute it, and therefore it is acceptable to distribute as part of a collection in our benchmark. Refer to the SVGrepo license for further details.

SVG Invariance: We use beautifulsoup and SvgLib to process the SVG XML code to perform translation and rotation perturbations for the invariance investigation, a visual sample can be found in Fig. [14](#A3.F14). Specifically, as we assume that all XML elements in each SVG figure do not possess any "transform" attribute as it complicates the augmentation process. For elements that can be fully specified by coordinates (*e.g.*, <rect>, <polygon>), we perform augmentation by perturbing these coordinates. For <path> elements in which path information is fully specified in "d" attributes, we first turn all relative operations (*e.g.*, "l 2 3", meaning that draw a line from the current position $(x,y)$ to the position $(x+2,y+3)$) into absolute ones, and later perturb the coordinates but not other path attributes. As mentioned in the main paper, small spatial perturbations can drastically change the numerics of the SVG XML code (See Section C for more details).

###### SGP-Bench (CAD).

Our CAD (3D) sequences data are sampled from DeepCAD [] datasets, which contains around 180k manually constructed CAD sequences that are originally from the ABC dataset []. We manually sample 1000 sequences and use pythonocc (OpenCASCADE) to verify and normalize these CAD sequences, then render the front and back view of the 3D CAD models. Since all the CAD models are from the OnShape platform, the copyright of the CAD models is owned by their creators. For licensing details, see Onshape Terms of Use 1.g.ii. Our CAD (3Drecon.) sequences are sampled from Fusion360 Reconstruction Dataset [] datasets, which contains 8,625 sequences, with more complex curve operations for constructing sketches. Our CAD (2D) sequences are sampled from SketchGraphs [] dataset, which consists of 15 million sketches extracted from real-world CAD models. The major difference is it consists of 2D CAD sketches without Extrusion operations.

###### SGP-MNIST.

The MNIST SVG data is sampled from the kaggle MNIST-SVG dataset(^6^66[https://www.kaggle.com/datasets/jacekpardyak/mnist-svg](https://www.kaggle.com/datasets/jacekpardyak/mnist-svg)). We randomly sample 1000 samples from MNIST-SVG (100 samples per digit category) to build our SPG-MNIST benchmark. The data comes with CC BY-SA 3.0 license.

#### A.2 Evaluation protocol

###### Inference.

To conduct massive-scale evaluation (8000+ question samples with 16 models), we leverage vLLM(^7^77[https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)) to perform high-throughput and memory-efficient inference for all the open-source LLMs.
And we use OpenAI API for evaluating different variants of the GPT models.
We deploy the vLLM inference engine as a server that also uses the same format as OpenAI API to achieve a unified testing framework for both GPT and all open-source models. The vLLM inference engine is deployed on a node with 8 NVIDIA H100 80G GPUs.

###### Evaluation.

We benchmark the performance of all the models via the question answering accuracy. Following the common protocol [], we ask the model to generate the answer sentence in a formatted way (see text prompt examples in [F.2](#A6.SS2)).
Then, we extract the target answer from the output sentence by parsing the answer according to its position. If the extracted answer matches the ground truth, it will count as 1, otherwise it’s 0.
We use the average accuracies for the results in Table [1](#S4.T1).

Enhanced answer extraction with LLM: In our experiment, we found the Symbolic Instruction Tuning makes the model less capable in following the formatting instruction.
This is likely due to our fine-tuning only uses symbolic graphics description, which causes the model to forget its instruction following skill.
Therefore, the model after SIT often answers questions correctly but in a different format.
This affects the aforementioned format-based answer extraction. For example, given a color-grounding question of the input subject, the formatted answer should be "The answer is A) Yellow.", however, the model outputs "The car is yellow". To mitigate this issue, we follow the GPT-enhanced answer extraction of Mathvista(^8^88[https://github.com/lupantech/MathVista](https://github.com/lupantech/MathVista)), where we present both the question options and model’s output to GPT4 to extract the answer in the formatted way. A 5-shot CoT is also applied here to augment the robustness of extraction process (More details in [F.2](#A6.SS2)). The results on Table [2](#S5.T2) are obtained with the enhanced answer extraction. More details of the SIT evaluation can be found in [Appendix E](#A5).

#### A.3 Evaluated model specs

Here we list the model details of all LLMs we evaluated in the SGP-Bench, their performance are demonstrated in the table [2](#S5.T2). Generally, we evaluated 3 types of LLMs, the representative open-sourced LLMs from tech giants and start-ups, the code-specific LLMs that were built for code generation and understanding and the strongest proprietary models from the GPT and Claude family.

##### A.3.1 Open-sourced LLMs

Gemma-1.1-2B/7B
Gemma is a suite of lightweight, advanced open models created by Google DeepMind and other teams across Google, it’s the best performing model at it class when it’s released on Feb 21, 2024.
Primarily designed for text generation, Gemma models come in multiple sizes, i.e. 2B / 7B, to fit various computing resources and deployment needs. The models are trained on 3T (2B) / 6T (7B) tokens of primarily-English data from web, mathematics, and code. It is based on a transformer decoder with context length of 8192 tokens. It leverages Multi-Query Attention, RoPE Embeddings, GeGLU Activations and RMSNorm. The Gemma models are using architectures. data and training recipes inspired by the Gemini model family. The models are available in two versions: a pretrained version and an Instruction-tuned version, the latter refined through human language interactions to perform well in conversational roles, similar to a chat bot. We only test and perform the symbolic instruction tuning on the Instruction tuned version.

Mistral-0.3-7B
The Mistral-0.1-7B from Mistral AI is released September 27, 2023 and marked as the best 7B model at the time. The 0.1 version model features a 8k context window, with Grouped-query attention (GQA) for faster inference and SWA for handling longer sequences more effectively at a reduced computational cost. The model is further updated to 0.3 version in May 21, 2024, upgrading its context length to 32k, its vocabulery size and RoPE theta, but the SWA is removed in this version.

Mistral-NeMo and Mistral-Large2
Mistral NeMo is a 12B large language model built by Mistral AI with a context window size of up to 128k tokens. Mistral NeMo is trained with quantization awareness, allowing FP8 inference without any loss in performance. Mistral NeMo uses a new tokenizer, Tekken, based on Tiktoken, which enables more efficient compression of natural language text and source code, compared with previous Mistral model series.

Mistral Large 2 is the new generation of Mistral AI’s flagship model, with a model size of 123 billion parameters. Especially, Mistral Large 2 is trained on a very large proportion of code data, resulting in state-of-the-art performance, on par with proprietary models like GPT-4o or Clause Opus.

Yi-1.5-9B/34B
The Yi model family, developed by LLM-focused startup 01.AI, includes 6B and 34B pretrained language models. Their performance is attributed to high-quality data from meticulous data-engineering efforts. For pretraining, 3.1 trillion tokens of English and Chinese corpora were constructed using a cascaded data deduplication and quality filtering pipeline. Finetuning involved a carefully refined instruction dataset of fewer than 10K instances, each verified by dedicated machine learning engineers. Built on Transformer architecture, Yi models feature Grouped-Query Attention (GQA), SwiGLU activation, and RoPE with an adjusted base frequency (RoPE ABF). The Yi-6B base model, with 32 layers, was scaled up to the Yi-9B model, which has 48 layers, by duplicating the original 16 middle layers (layers 12-28). We hence test the Yi-9B model together with the 34B version in SGP-Bench.

InternLM2-7B/InternLM2-20B/InternLM2.5-7B
InternLM2 is a open-sourced LLM model series developed by Shanghai AI laboratory, with a context window length of 200K.

InternLM2.5 is an open-sourced, 7 billion parameter base and chat model with a context window size of 1M. It supports gathering information from more than 100 web pages and has in general a very strong capability in tool utilization.

Aya-23-8B/35B
Aya 23 is an open-source LLM model series developed by C4AI, featuring advanced multilingual capabilities. Aya-23 is fine-tuned (IFT) to follow human instructions and supports a context window length of 8192 tokens.

Command R-35B/104B
C4AI Command-R is a research release of a 35B large language model with open weights, optimized for a variety of use cases including reasoning, summarization, and question answering. Command-R has the capability for multilingual generation evaluated in 10 languages and highly performant RAG capabilities. It supports a context length of 128K.

C4AI Command-R+ is an open-source multilingual LLM with enhanced features, including Retrieval Augmented Generation (RAG) and tool usage for automating complex tasks. Command-R+ excels in multi-step tool usage, allowing the model to combine various tools across multiple steps to complete sophisticated tasks.

Qwen-1.5-7B/32B/72B/110B
Qwen, released in April 2024, developed by Alibaba Cloud, is a series of transformer-based large language models pre-trained on diverse data, including web texts, books, code, and more, over 2.2 trillion tokens. The Qwen1.5 series includes various sizes of decoder models, each available as a base and aligned chat model, supporting long context lengths (8K tokens for 1.8B, 7B, and 14B models, and 32K tokens for the 72B model). It outperforms similar-scale open-source models on various Chinese and English tasks and even exceeds some larger models in benchmarks. These models feature the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, and a mix of sliding window and full attention mechanisms. They also include an advanced tokenizer for multiple languages and coding languages. Qwen’s extensive vocabulary of over 150K tokens enhances compatibility with multiple languages, allowing for improved capabilities without expanding the vocabulary.

Qwen-2-72B Qwen2, is the newest series of large language models, developed by Alibaba, which surpasses its previous released Qwen1.5 series, yielding state-of-the-art performance across different benchmarks. Qwen2-72B-Instruct has an extended context length of up to 128K and is instruction-aligned with both supervised finetuning and direct preference optimization.

Llama3-8B/70B
Meta’s Llama 3 is the latest generation of llama family, release in April 18, 2024, featuring pretrained and instruction-fine-tuned versions with 8B and 70B parameters. Designed with a standard decoder-only transformer architecture, Llama 3 models demonstrate state-of-the-art performance across various industry benchmarks and show improved reasoning capabilities. Key enhancements include a tokenizer with a 128K token vocabulary for efficient language encoding and grouped query attention (GQA) for better inference efficiency.

Llama 3 models are pretrained on an extensive dataset of over 15T tokens from publicly available sources, including a significant increase in code and high-quality non-English data covering 30+ languages. This dataset is seven times larger than that used for Llama 2, ensuring superior model performance.

For instruction-tuning, Llama 3 employs a combination of supervised fine-tuning (SFT), rejection sampling, proximal policy optimization (PPO), and direct preference optimization (DPO). This approach, coupled with meticulously curated data and multiple quality assurance rounds, significantly enhances model alignment and response diversity.

Llama3.1-8B/70B/405B
Introduced in July 2024, Llama 3.1 was pretrained on  15 trillion tokens of data from publicly available sources as well as over 25M synthetically generated examples. The intruction-tuned variants are post-trained using supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences to guarantee safety and helpfulness. The Llama 3.1 405B demonstrates competitive performance across 150 benchmarks with leading foundation models, including GPT-4o and Claude 3.5 Sonnet.

##### A.3.2 Code-specific LLMs

CodeQwen1.5-7B
CodeQwen1.5-7B is based on Qwen1.5-7B. It is further trained on 3T tokens of code data, and it also includes group query attention (GQA) for efficient inference.

DeepSeek-Coder-V2-16B-Instruct
DeepSeek-Coder-V2-16B-Instruct is an Mixture-of-Experts (MoE) code language model, that demonstrates comparable performance to GPT-4 Turbo in code-related tasks. Specifically, DeepSeek-Coder-V2 is continued to pre-train on intermediate checkpoint of DeepSeek-V2 with 6 trillion additional tokens, substantially enhance its reasoning capabilities in code and mathematical related tasks. DeepSeek-Coder-V2 supports 338 programming languages and has a context length of 128K.

Codestral-22b-v0.1
Codestral-22B-v0.1 is the code-specific variant of Mistral-0.1-22B, it’s trained on a diverse dataset of 80+ programming languages, including the most popular ones, such as Python, Java, C, C++, JavaScript, and Bash.

##### A.3.3 GPT family

GPT-3.5t (GPT-3.5 Turbo) is a text only language model released by OpenAI on November 2022.
The specific version of the model we are using is gpt-3.5-turbo-0125.
It has a knowledge cutoff of September 2021 and a context window with length of 16K tokens.

GPT-4t (GPT-4 Turbo) is vision language model launched by OpenAI on March 2023.
The specific version of the model we are using is gpt-4-turbo-2024-04-09.
It has an updated knowledge cutoff of April 2023 and a context window with length of 128K tokens.
It is more powerful than GPT-3.5.

GPT-4o (GPT-4 Omni) is a multimodal model released by OpenAI on May 2024, which support data types such as audio, vision, and text.
The specific version of the model we are using is gpt-4o-2024-05-13.
It has similar performance as GPT-4t on English text and code, but with significant improvement on non-English text, *i.e.*, over 50 languages.
At the same time, it is able to reason with vision input.
GPT-4o has knowledge up to October 2023 and supports context window with length of 128K tokens.

GPT-4o mini is a multimodal model released by OpenAI in July 2024, which is a more cost-efficient and smaller modal than GPT-4. It has a context window size of 128K tokens and the has knowledge up to October 2023.

##### A.3.4 Claude family

Claude is a multimodal, multilingual, proprietary model series developed by Anthropic. The Claude series includes different models: Haiku, the fastest and most lightweight model; Sonnet, the best balanced model between performance and speed; and Opus, the highest-performing model. We did not evaluate Claude 3 Opus because, in June 2024, Anthropic released Claude 3.5 Sonnet, the newest best-performing model.

Specifically, we are using claude-3-5-sonnet-20240620 for Claude 3.5 Sonnet, claude-3-sonnet-20240229 for Claude 3 Sonnet, and claude-3-haiku-20240307 for Claude 3 Haiku for benchmark evaluation.

### Appendix B Human Study Details

We ran a human study to verify the labels produced by GPT-4o for the benchmark over a subset of $500$ stimuli. We recruited $55$ participants from the crowdsourcing platform, Prolific []. Stimuli were batched into $10$ sets of $50$ stimuli each. Each participant was randomly assigned a batch of $50$ stimuli; stimuli were presented in a random shuffled. On each trial, participants saw the question, original image, and set of multiple choice options. Participants selected an option by clicking a button. We include an example screenshot of a trial in Figure [11](#A2.F11). Participants were paid at a base rate of $\$12.50$/hr. They were informed that they could receive a bonus up to $\$15/hr$ based on the amount of correct answers they achieved. All participants received the full bonus. Our study was approved by our institutional ethics review board, and all participants provided informed consent. We include the set of instructions and sample screenshots in Figures [12](#A2.F12) and [13](#A2.F13), respectively. We found high inter-annotator agreement (participants in the same batch had between $0.7-0.85$ Fleiss Kappa’s alpha agreement, where higher implies higher agreement). We find that humans’ mode response matched GPT-4o on 90% of the examples (450 of the 500 stimuli).

Figure: Figure 11: Example survey question.
Refer to caption: /html/2408.08313/assets/user_study_figs/sample_trial2.png

Figure: Figure 12: Experiment instructions.
Refer to caption: /html/2408.08313/assets/user_study_figs/starter_instructions.png

Figure: Figure 13: Experiment instructions (continued).
Refer to caption: /html/2408.08313/assets/user_study_figs/instruct5.png

### Appendix C SVG - Invariance Illustration

Our SVG - Invariance test is essential for testing whether a model has a fundamental understanding of the code, or it is able to pass the benchmark tests due to memorizing the SVG code samples, since we built our SVG-Bench using public available SVG datasets. In Figure [14](#A3.F14) we see two SVG codes, illustrating two samples, that are semantically identical. The rotated sample is generated by ourself by applying a SE(2) transformation on the original sample (from SVG Icons). We can see that semantically these two samples are identical, the code changed drastically.

Figure: Figure 14: Illustration of the SVG - Invariance test.
Refer to caption: /html/2408.08313/assets/x11.png

### Appendix D More Examples in SGP-Bench

#### D.1 SVG Data

Figure: Figure 15: SVG examples in our SGP-Bench.
Refer to caption: /html/2408.08313/assets/x12.png

Figure: Figure 16: SVG examples in our SGP-Bench.
Refer to caption: /html/2408.08313/assets/x13.png

Figure: Figure 17: SVG examples in our SGP-Bench.
Refer to caption: /html/2408.08313/assets/x14.png

Figure: Figure 18: SVG examples in our SGP-Bench.
Refer to caption: /html/2408.08313/assets/x15.png

#### D.2 CAD Data

Figure: Figure 19: CAD examples (3D) in our SGP-Bench.
Refer to caption: /html/2408.08313/assets/x16.png

Figure: Figure 20: CAD examples (3D) in our SGP-Bench.
Refer to caption: /html/2408.08313/assets/x17.png

Figure: Figure 21: CAD examples (3Drecon.) in our SGP-Bench.
Refer to caption: /html/2408.08313/assets/x18.png

Figure: Figure 22: CAD examples (2D) in our SGP-Bench.
Refer to caption: /html/2408.08313/assets/x19.png

#### D.3 Symbolic Instruction-following Data (SVG)

Figure: Figure 23: SVG examples of our symbolic instruction-following data.
Refer to caption: /html/2408.08313/assets/x20.png

### Appendix E Details and More Results of Symbolic Instruction Tuning

#### E.1 Implementation Details

We use the unsloth(^9^99[https://github.com/unslothai/unsloth](https://github.com/unslothai/unsloth)) framwork to finetune the base models Llama3-8b-instruct and Gemma-1.1-7b-it. For both models, we use the exact same training setting: we finetune the base models with LoRA [] on 1 NVIDIA H100 80GB gpu with learning rate 2e-4, batch size of 2 and for 1 epoch.

We use the PEFT(^10^1010[https://github.com/huggingface/peft](https://github.com/huggingface/peft)) framework to test different fine-tuning methods when performing SIT. We choose two common fine-tuning methods LoRA [] and orthogonal finetuning [, ] to fine-tune the base model Llama3.1-8b-Instruct. For both fine-tuning methods, we train on 8 NVIDIA H100 80GB gpus with learning rate 1e-4, per device batch size of 1 and for 1 epoch.

As introduced in Section [7](#S7), we also use PEFT to test if SIT can improve generic instruction tuning, by mixing our curated SIT data into the publicly available instruction tuning dataset open-instruct(^11^1111[https://huggingface.co/datasets/VMware/open-instruct](https://huggingface.co/datasets/VMware/open-instruct)). We use LoRA to fine-tune the base model Llama3.1-8b on 8 NVIDIA H100 80GB gpus with learning rate 1e-4, per device batch size of 1 and for 1 epoch. We test mixing with different SIT data splits, including 10K, 25K, 40K, 55K, and 72K. For example, Open-Instruct-SIT-10K, Open-Instruct-rev-SIT-10K and Open-Instruct-mixed-SIT-10K are constructed by mixing Open-Instruct with SIT-10K, rev-SIT-10K and mixed-SIT-10K. More specifically, rev-SIT-10K is constructed from SIT-10K according to Figure [10](#S7.F10), while mixed-SIT-10K uniformly samples exactly 5K of the SIT instruction-following pairs and convert them to rev-SIT, while the rest 5K is kept unchanged. The best result is reported in the Table [4](#S7.T4). We employ the widely-used lm-evaluation-harness(^12^1212[https://github.com/EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)) to obtain the results on a variety of LLM benchmarks.

#### E.2 More Experiments in Symbolic Instruction Tuning

We additionally provide an ablation study of using different-size SIT data to finetune the base LLMs and measure their performance after SIT on the SGP-Bench. We uniformly sample 72k SVG programs from the SVG Icons dataset to build an instruction-following dataset using the text prompt examples in [F.1](#A6.SS1) to query GPT-4v. The SIT-25k dataset is built by choosing the samples with the shortest code length out of the original 72k instruction following pairs. The SIT-10k dataset is a subset of SIT-25k, by uniformly sampling from the SIT-25k dataset. For SIT-40k and SIT-55k, we additionally sample more data with short code length from the SVG Icons dataset and mix it with SIT-25k. In this way, we can ensure that the smaller SIT dataset is always a subset of the bigger one.

We use the LLM-based evaluation because we noticed that after SIT, the generic instruction-following ability of the finetuned model degerates compared to the original model. We want to eliminate the cases where the finetuned model answers the questions correctly but do not follow the answer template, so that a matching-based evaluation will fail to extract meaningful answers. The results are shown in the Table [5](#A5.T5). We notice that for Llama3-8B-instruct, the SIT will improve the generic semantic understanding up-to some SIT data size, afterwards the performance degenerates, while for Gemma-1.1-7b-it, the overall semantic understanding improves significantly without noticeable degeneration.

**Table 5: Ablation study of studying the effect of different-sized SIT data on the model’s performance on the SPG-Bench (SVG-Understanding) using LLM-based answer extraction.**
| Dataset Size | Llama3-8B | Gemma-7B |
| --- | --- | --- |
| Original | 43.20 | 39.33 |
| SIT-10k | 48.16 (+4.96) | 45.60 (+6.27) |
| SIT-25k | 51.43 (+8.23) | 46.87 (+7.54) |
| SIT-40k | 45.62 (+2.42) | 45.21 (+5.88) |
| SIT-55k | 40.99 (-2.21) | 47.28 (+7.95) |

We also conducted an ablation study to determine whether the finetuning method affects the SIT results. Our findings indicate that the enhancement in the model’s ability to understand symbolic programs is independent of the finetuning approach. Both OFT and LoRA significantly improve the model’s understanding of symbolic programs, as shown in Table [6](#A5.T6).

**Table 6: Ablation study of studying the effect of different-sized SIT data on the model’s performance on the SPG-Bench (SVG-Understanding) using LLM-based answer extraction and different fine-tuning methods. * The value differs from the value in the main table, because we use LLM-based evaluation to guarantee consistency.**
| Dataset Size | LoRA | OFT |
| --- | --- | --- |
| Llama 3.1-8B* | 46.7 | 46.7 |
| SIT-10k | 47.9 (+1.2) | 48.0 (+1.3) |
| SIT-25k | 49.8 (+3.1) | 50.3 (+3.6) |
| SIT-40k | 51.0 (+4.3) | 51.2 (+4.5) |
| SIT-55k | 51.3 (+4.6) | 51.4 (+4.7) |

### Appendix F Text Prompt Template

#### F.1 Template for benchmark construction

We randomly sample from the following 20 prompts to generate the Symbolic Instruction Tuning (SIT) data:

We use the following prompt to query GPT to construct the SGP-Bench (SVG) question-answer pairs.

We use the following prompt to query GPT to construct the SGP-Bench (CAD) question-answer pairs:

We randomly sample from the following 20 prompts to construct the questions for the SGP-MNIST benchmark. We do not need to query GPT because we have the ground truth label for every SGP-MNIST code.

Given the question-answer pairs, either through querying GPT (SVG) or randomly sample from a pre-defined prompt pool, we use the following question template to construct the questions for our SGP-Bench (SVG):

Given the question-answer pairs we use the following question template to construct the questions for our SGP-Bench (CAD):

When constructing the SGP-Bench (CAD), we also provide the syntax of CAD code:

#### F.2 Template for evaluating models on SGP-Bench

When evaluating different models on our SGP-Bench, we use the following evaluation template (multiple choice):

When evaluating different models on our SGP-Bench, we use the following evaluation template (generation):

We use the following template, when we perform LLM-based evaluation:

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Regarding Weakness 1, it appears the model learns from structured data without evidence of internal image rendering, likely due to lack of instruction. It would be interesting to test whether explicitly prompting for “internal rendering” as a Chain-of-Thought (CoT) step impacts performance.
2. I'm curious whether any experiments have been conducted using rendered images as input in multi-modal LLMs to compare performance with raw SVG code inputs, providing insight into how well models perform with visual representations.
3. In Appendix E.2, it’s noted that fine-tuning SIT-55k on Llama3-8B actually degrades performance. Is this due to answers not following the specified template, or are the answers correctly formatted but simply inaccurate?
soundness: 3
strengths: 1. Innovative in presenting a benchmark and dataset that adds value to multi-modal LLM and vision foundation model research.  
2. Extensive experimentation with well-designed elements, particularly in the fine-grained categorization of SVG-Understanding. While categories like "color" and "shape" might be addressed through subpattern matching in SVG codes, "Semantics" and "Reasoning" better assess true reasoning capability.  
3. Clear presentation and valuable insights.
summary: The paper assesses large language models (LLMs) for their ability to "understand" raw symbolic graphics programs like SVG and CAD without direct visual input. By creating a benchmark (SGP-Bench), the authors test this through tasks requiring semantic-level answers and transformation invariance, providing insight into how LLMs interpret non-image-based graphics data. The authors also introduce Symbolic Instruction Tuning (SIT), a method to fine-tune LLMs with specific symbolic graphics data, show that its enhancing both specialized and general reasoning abilities.
weaknesses: 1. The SGP-MNIST experiments show limited spatial reasoning in LLMs, suggesting that some claims about models’ spatial reasoning or “visual imagery” abilities may be overstated.  
2. The conclusion is somewhat underwhelming, reiterating known principles such as scaling laws and fine-tuning effects.

---

confidence: 5
contribution: 4
initial_rating: 5
presentation: 2
questions: 1. In L76-L77, the authors claim that "The order of symbolic operations may substantially affect semantic meaning". However, this does not seem applicable to SVG, where objects are defined by coordinates and unaffected by drawing order. Could this be clarified?
2. In Figure 6, what qualifies as an "operation" in vector graphics? For example, does a polyline with multiple vertices count as one operation or several?
3. Inconsistencies in notation between Figure 6(b) (symbol $3D_{Recon}$) and the text (symbol $3D_{complex}$ in L257-L258) need clarification.
4. During fine-tuning, was there any overlap between training and test datasets? Additionally, could more detail be provided on the construction of the symbolic instruction dataset such as whether it includes both SVG and CAD formats?
5. If the enhanced answer extraction with LLM is applied consistently across all experiments or is only applied to models trained on SIT data?
soundness: 3
strengths: 1. The benchmark dataset is large, with 4,340 questions in the SVG set and 2,400 in the CAD set.
2. Both 2D and 3D objects are evaluated.
3. The paper addresses data leakage issues by utilizing global transformations like translations and rotations.
4. The proposed instruction tuning dataset improves general reasoning ability in LLMs.
5. The paper provides a critical view on current LLMs' capability.
summary: This paper introduces a benchmark designed to assess LLMs'  comprehension of symbolic graphics programs in SVG format, two 3D CAD formats (DeepCAD and Fusion360), and one 2D CAD format (SketchGraphs). The authors present evaluation results across several LLMs and introduce a symbolic instruction dataset, showing that fine-tuning on this dataset can enhance LLM reasoning capabilities in general.
weaknesses: 1. The human study reports 90% labeling accuracy, leaving 10% mislabeled data, which could significantly impact the results given limited variation of the accuracy in LLM's evaluation. I suggest doing a human filtering across the whole benchmark to improve the accuracy of the ground truth answer.
2. The method for generating questions on 3D dataset from limited rendered views may lead to errors, especially in counting tasks; for example, the second part Figure 7 (row 3, column 3) is confusing. The question "How many visible cylindrical pins are there on the CAD object?" doesn't make sense since "visible" doesn't have a meaning given a code representing a 3D object without perspective information.
3. Several parts of the writing need further clarification (see questions below).

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: (Merged with the Weaknesses Section)
soundness: 3
strengths: - SGPs offer an intriguing middle ground between perception and reasoning. While humans can intuitively link rendered images to semantic concepts, associating these same concepts with the underlying SVG schema may demand additional time and coding knowledge.
- The dataset presented is highly scalable, as millions of SVG images and CAD programs with permissive licenses are available online for use in this benchmark.
- The proposed experiments are both well-motivated and effectively executed.
- The presentation is clear, consistent, and transparent.
summary: The authors explore the use of symbolic graphical programs (SGPs) as a means to assess the reasoning abilities of large language models (LLMs). SGPs are defined using visual markup languages and can be compiled by symbolic programs to produce images, such as SVGs in 2D or CAD models in 2D/3D. The central hypothesis is that the high-level reasoning skills of an LLM are linked to its capacity to (1) identify complex objects from primitive shapes and answer questions about them (Semantic Understanding) and (2) recognize when different sets of primitive shapes produce the same overall program (Consistency). To test this hypothesis quantitatively, the authors compiled approximately 4,000 program-question pairs for SVG images and 5,000 pairs for CAD programs. Their findings reveal that (1) open-weight models significantly lag behind proprietary ones in performance, (2) fine-tuning using an intermediate representation substantially improves reasoning performance, and (3) all models face challenges in distinguishing MNIST digits presented as SVG schemas.
weaknesses: * L108: "highly scalable benchmark." While I agree that SGPs have this potential, the paper does not demonstrate specific contributions toward achieving scalability. For example, benchmarks like [MineDojo](https://minedojo.org/) achieve high scalability by developing tools to efficiently scrape relevant data from platforms like YouTube, Reddit, and Wikia, resulting in a dataset of around 700k Minecraft videos and 6M+ comments. In contrast, the SGP dataset in this paper only contains ~5,000 data points. This is surprising given the abundance of online SVG images (for example, [SVGRepo](https://www.svgrepo.com/) claims to have over 500k SVGs). If SGP-Bench were a "highly scalable" benchmark, I would have expected an order of magnitude more data points or an explanation of the barriers preventing larger-scale sampling. Although this does not diminish the significance of the benchmark (Multimodal reasoning benchmark do not have to be "internet-scale" to be influential: https://arxiv.org/abs/2401.06209), the scalability claims should be adjusted accordingly
* The current definition of symbolic graphical programs might be overly broad. For instance, HTML could technically qualify as an SGP within the given definition since it uses a consistent markup language and rendering engines (like Chrome, Safari, or Firefox). While this isn’t inherently a weakness, it could impact the paper's novelty claims. If the SGP definition admits HTML as well, the authors might need to expand their related work and adjust their novelty claims to include efforts like https://arxiv.org/pdf/2210.03347, https://mbzuai-llm.github.io/webpage2code/, https://arxiv.org/abs/2403.09029 (and other relevant insights from work on action transformers for webpage understanding).  

- [Figure 4] the comparison between OpenAI-o1 (a model likely using Chain of Thought reasoning) and standard zero-shot LLM queries using Llama3.1 70B seems problematic. However, this qualitative example showcases how  CoT reasoning might be a natural way for increasing LLM performance on this benchmark. Is there a particular reason the benchmarked models (Table 1)  do not evaluate the respective models under the CoT (or similar multi-step reasoning) paradigm?  

- What are some potential "reasoning shortcuts" that could enable an LLM to perform well on this dataset without possessing true high-level reasoning abilities? For instance:
  - Data Leakage: Since the SVGs were sourced from the internet, it’s possible that proprietary LLMs were trained on some SVG/caption pairs. The authors propose the semantic consistency experiments to evaluate this. I believe that this partially mitigate this issue (as the jittered images likely wouldn't appear in the training set), but I'm concerned that jittering the image might not modify the underlying XML structure enough to conclusively reject the influence of data leakage. There are many ways to further validate this.
    - Calculate the tree edit distance (The Zhang Shasha algorithm is implemented in `pip install zss`) between the original and jittered SVG. Is the tree-edit-distance above a reasonable minimum threshold? Is the tree-edit-distance correlated with the relative performance difference? If low absolute correlation is observed, then the performance is invariant to the SVG structure, and data leakage might be absent.
    - Evaluate the performance of open-sourced models on obfuscated SVG files (Tools like https://github.com/svg/svgo will minify SVGs). If the minifier significantly alters the XML structure and performance remains unchanged, it could indicate no data leakage. 
    - Of course, the authors may have other effective methods to test this. This isn’t necessarily a criticism of the paper, but rather a recognition that  the benchmark offers a unique opportunity to measure performance gains in LLM reasoning attributed to data leakage.
  
- L396: "No semantic instruction dataset directly over symbolic graphical programs." This paper https://mikewangwzhl.github.io/VDLM/ seems to propose an alternative semantically meaningful intermediate representation. I'd be interested in the authors' opinions on this paper.
- Generally, this paper is missing a discussion on the limitations of this benchmark. Here are some prompting questions: 
  - Are all natural images expressible as SVGs? What does this mean for the comprehensiveness of this benchmark?
  -  Are SVGs / CAD programs representative of all types of symbolic graphical programs? What about visual specification languages like OpenGL, Taichi, etc? What about programming languages like Turtle/LOGO which admit recursive structures?
  - What are the limitations of SGP-Bench's scalability?

In conclusion, I'm currently recommending this paper for a **Weak Rejection**. While the benchmark is undoubtedly interesting to the ICLR community, it is crucial to reconcile the claims in this manuscript with those from the existing literature before the paper is accepted for publication. I am open to discussing these points with the authors and potentially revising my recommendation during the discussion period.

----

After discussions with the authors, I'm revising my recommendation to **Accept**. Overall, SGPs offer a promising avenue to assess the reasoning abilities of LLMs.

Meta-review: This paper studies exactly the question in the title: Can LLMs understand symbolic graphics programs. The goal is to see whether LLMs can answer semantics questions (of the rendered image) based on the raw program input. The authors introduce a new benchmark (SGP-Bench) covering SVG and CAD, evaluating on semantic understanding (annotated with GPT-4o with some human validation) as well as semantic consistency (does answers remain consistent with random translation/rotation). The authors evaluate existing LLMs as well as with instruction tuning to find that this can not only improve performance on SGP-Bench but also other general reasoning datasets. Finally, the authors demonstrate the difficulty for LLMs to understand SVGs through showing their poor performance on SGP-MNIST (classifying MNIST-like digits). There is unanimous decision from reviewers to accept the paper - the AC agrees that the paper proposes an interesting new benchmark, performs well-designed experiments, and the paper is well-written. Reviewers have brought up valuable points to improve the paper, including a discussion on limitations of the benchmark, connections to VLMs for solving these problems (besides it being used to generate annotations), potential training data leakage from captioned SVGs / other graphics programs in the pre-training set of existing LLMs. Also, multiple reviewers have brought up concerns on the 90% labeling accuracy for the questions - it would be good for the authors to also add this discussion in the main paper on question difficulty and any annotator variability.

Decision: Accept (Spotlight)