Visually Descriptive Language Model for Vector Graphics Reasoning
Authors: Zhenhailong Wang
Joy Hsu
Xingyao Wang
Kuan-Hao Huang
Manling Li
Jiajun Wu
Heng Ji
Venue: Accepted by TMLR ()

Abstract: Despite significant advancements, current large multimodal models (LMMs) struggle to bridge the gap between low-level visual perception—focusing on shapes, sizes, and layouts—and high-level language reasoning involving semantics, events, and logic. This limitation becomes evident in tasks requiring precise visual perception, such as comparing geometric properties or solving visual algorithmic reasoning problems. To study this failure mode, we focus on an important visual domain: vector graphics —images composed purely of 2D objects and shapes, which are prevalent in Web, PC, and Mobile environments. Importantly, we consider rasterized vector graphics without assuming access to their underlying vector code. We identify two key research questions: how can we enable precise visual perception, and how can we facilitate high-level reasoning based on such low-level perceptions? To accurately capture low-level visual details, we explore using SVG for the precise encoding of visual scenes. However, SVGs are not readily interpretable by LLMs or LMMs in a zero-shot manner. To address this challenge, we propose the Visually Descriptive Language Model (VDLM) to build a bridge between low-level visual perception and high-level language reasoning. VDLM learns an intermediate symbolic representation called Primal Visual Description (PVD), which translates raw SVGs into a higher-level abstraction comprising primitive attributes. This abstraction allows for direct interpretation by foundation models for zero-shot generalization to different reasoning tasks. Without any human-annotated data, VDLM leads to significant improvements in state-of-the-art LMMs, such as GPT-4o, across various low-level multimodal perception and reasoning tasks on rasterized vector graphics. Additionally, we provide extensive analyses of VDLM’s performance, showing that our framework offers improved interpretability due to its disentangled perception and reasoning processes. As the first attempt to construct a descriptive intermediate representation for low-level visual reasoning, we also conduct an in-depth error analysis, highlighting remaining limitations and suggesting directions for future research.

Published in Transactions on Machine Learning Research (05/2025)
Visually Descriptive Language Model for Vector Graphics Reasoning
Zhenhailong Wang', Joy Hsu', Xingyao Wang', Kuan-Hao Huang Manling Li
Jiajun Wu', Heng Ji
University of Illinois Urbana-Champaign, 2ttnnford University, TTexas A&M University, *Northwestern University
{wangz3, xingyao6, khhuang, hengjt)@iillinoss.od
[joyej, manling!] @stanford.edu, jagjuvvu@esstaaffrr.eeeu
Reviewed on OpenReview:
Image
Question
Visual Features
(a) Existing
GPT-4o
No
LMM
LMM
following statement about scene
correct? There is one tower with black block the basc blue block top
Answer
low-level
spach d= 120,00 520,29 10,29 10000000000
Image
SVG
trann tranxforeet frensaaat 90,
SVG
Accurate and higher-level
+
(b) VDLM
Question
VF
PVD
Is the following statement about the scene correct? There is one tower black block at the base and a blue block at the top
Doo. 'objwct 41z (P'sppe's
"reciangie's "vectiooa's 11190.0,
100.0" raaly toolor's 10, 6, 01, shape"
(ours)
"style';
PVD
LMM
nox piidse bow) cortains objesta tobjer 4'.... the colors of these objecte are black, black, black, and hise, pospectivelly, matahes the desoription as it has black block et the base and bice black at the 20p. Fical Ansver
Answer
Yes
Existing monolithie LMMs rely solely on pretrained vision encoders, such as CLIP (Radford et al. 2021), for perception, which often fail accurately capture low-level visual details in vector graphies-style images. In contrast, VDLM enables precise visual reasoning by introducing SVG encoding and learned intermediate symbolic representation, Primal Visual Description (PVD), that bridges low-level SVG perception with high-level language reasoning. "VF" in (b) refers to "visual features".
Figure 1:
Abstract
Despite significant advancements, current large multimodal models (LMMs) struggle to bridge the gap between low-level visual cnrception-focusiii on shapes, sizes, and layouts- and high-level language reasoning involving semantics, events, and logic. This limitation becomes evident in tasks requiring precise visual perception, such comparing geometric properties solving visual algorithmic reasoning problems. To study this failure mode, we focus on an important visual domain: vector graphics-imagen composed purely of 2D objects and shapes, which are prevalent in web and mobile environments. Importantly, we consider rasterized vector graphics without assuming access to their underlying vector code. We identify two key research questions: how can we enable precise visual perception, and how can we facilitate high-level reasoning based on such low-level perceptions? To accurately capture low-level visual details, we explore using SVG for the precise encoding of

---

Published in Transactions on Machine Learning Research (05/2025)
visual scenes. However, SVGs are not readily interpretable by LLMs or LMMs in zero-shot manner. To address this challenge, we propose the Visually Descriptive Language Model (VDLM) to build a bridge between low-level visual perception and high-level language reasoning. VDLM learns an intermediate symbolic representation called Primal Visual Description (PVD), which translates raw SVGs into a higher-level abstraction ion comprising primitive attributes. This abstraction allows for direct interpretation by foundation models for zero-shot generalization to different reasoning tasks. As an initial step to construct a descriptive intermediate representation for low-level visual reasoning, the SVG-to-PVD model is currently limited to simple compositions of primitive shapes, for which synthetic data can be generated without human annotation. Nevertheless, empirical experiments show that VDLM leads to significant. improvements in state-of-the-art LMMs, such as GPT-4o, across various low-level multimodal perception and reasoning tasks on rasterized vector graphics. Additionally, we provide extensive analyses of VDLM's performance, showing that our framework offers improved interpretability due to its disentangled perception and reasoning processes. We also conduct an in-depth error analysis, highlighting remaining limitations and suggesting directions for future research.
1
Introduction
In recent years, large multimodal models (LMMs) OpenAl 2023b Anil et al. 2023
Liu et al. 2023b Chen
et al. 2023b Bai et al.) 2023) have achieved impressive performance across wide spectrum of general
vision- language benchmarks [Goyal et al. 2017 Fu et al. 2023 Liu et al. 2023d Yu et al. 2023 Li et al.
2023a However, these monolithic LMMs still struggle with soomingly simple tasks that require preciso perception of low-level visual details. In particular, we empirically observe that LMMs frequently exhibit this failure mode in vector graphics, which are images composed purely of 2D objects and shapes. For example, a stato-of-thh-art LMM like GPT-4o (OpenAT, 2024) can still fail 43% of the time when comparing the lengths of two line segments, and 54% of the time when solving a simple 2x2 maze. LMMs' ability to understand vector graphics is largely underexplored compared to natural images but is essential for growing downstream applications in web, visual design, and OS environments (Zhou et al. 2023b Liu et al. 2024 Xie al. 2024 Rawles et al. 2024 Zheng et al. 2024 Lå et al. 2024). In this work, we focus on the fundamental aspects of low-level visual reasoning involving vector graphics including measurements, spatial relations, counting, and logical reasoning. It is important to note that in this paper, "vector graphics" refers to rasterized images in JPEG or PNG format, without assuming access to their underlying vector code. This reflects a more realistic setting for visual reasoning in real-world scenarios, such as web Zhou et al. (2023b); Lu et al. (2024) and mobile Wang et al. (2024 2025). To address the aforementioned challenge, we identify two main research questions. First, how can we enable precise visual perception in LMMs? Second, how can we bridge the gap between low-level perception and high-level reasoning?
For our initial question, we explore encoding a rasterized image via vectorization with SVG representation, which describes a scene with paths (c.g., polygons and splines) and their corresponding measurements and positions. SVG representations, by nature, are unbiased towards high-level semantics and can capture low-level visual details in text. The vectorization process can be faithfully accomplished with an rule-based raster-to-vector algorithm. However, such machino-generated SVG is often noisy and far from natural language, making it insufficient for language reasoning. Our preliminary experiments ($A) demonstrate that existing foundation models are unable to interpret waahine-gemerated SVG codes in zero-shot settings. Another key challenge lies in the scarcity of end-to-end instruction tuning data containing (SVG, question, answer) triplets, making direct fine-tuning infeasible.
To bridge this perception-reasonigg gap and address data scarcity, we propose translating the low-level SVG paths to a higher-level intermediate symbolic representation, referred as Primal Visual Description (PVD), which can directly be leveraged by foundation models for multimodal reasoning. Specifically, we learn an LLM- based (Jiang et al. 2023) SVG-to-PVD model, which transforms the raw SVG paths into a set of primitive attributes (c.g., shape, position) with corresponding predicted values (o.g., rectangle, pixel coordinates of the vertices). See Figure in the blue box for an example. Notably, the PVD representation
2

---

Published in Transactions on Machine Learning Research (05/2025)
Maypets "grid" "vestioua";
"equa" (11s, 181, [15, 164).. HI toolork (6, 01, Fine
Inputs
sorth
so MC1.11 1:1,148 1138, 143 1588, I.1.115 141 1235,
96) noaduuen 1. "esiort; 1255, 4), "olyle"; alape"
transform- cranslaace 1/)
Img to SVG
SVG
to PVD
>/>
11880, 2351. 1227 2561, (253,
.25, (eope" 2581 trilles
"triangle "varticen": ....
Input Image
("oypat: "rectangie",
356.01, 18:4, C.dl, [336.0 5.81, 133..6
"coloc*, 252], "atyls": *eiiled shapa*)
Given a 2x2 maze, where the walls depicted by a grid of black lines...
Incrementally Decomposed
PVD Representation in JSON (plots
Single SVG Paths
only visualization)
PVD Perception Module
The start position is marked by a red circle, and the position by a red star
#82 Play is codee Lie Comvetaviity of dl culls Vereical connertiene: "Llffue, tros]." the the start position c.. (0, 01",
[iFa.sel, cositionn
Solve the maze
Input Text
LMM
the end position at (,, 21" H8 step 31 doive the Kane "I13,
131%)
Output
Reasoning Module
Figure An example of VDLM during inference. First, VDLM extracts individual SVG paths from the input image and then transforms them into Primal Visual Description (PVD) using trained language model. These PVD perception results, along with the imput text querics and the original input image, are subsequently fed into an LMM for reasoning. It is worth noting that although "star" (*) is not explicitly part of the PVD primitive ontology (see Figure 3), the SVG-to-PVD model can approximate the "star" by composing two triangles (+). A strong off-the-shelf reasoner, such as GPT-4 (OpenAl 2023a), can accurately deduce that this composition corresponds to the "star," " which is the target end position of the maze. For the complete response, refer to Figure
consists of primitive attributes that serve fundamental building blocks for vector graphics, enabling learning from procedurally synthesized (SVG, PVD) pairs without requiring lask-specific annotations. PVD filters out unnecessary noise in SVGs, enhancing the semantics of perception and facilitating subsequent reasoning. Currently, we limit the scope of the SVG-to-PVD model to simple compositions of geometric primitives, for which synthetic data can be generated without human annotation. We leave generalization to entirely open-domain images as future work.
Comprising SaG-based image perception and PVD abstractions, present the Visually Descriptive Language Model (VDLM), which contains three components: a rule-based visual encoder that converts images to SVG to capture precise visual details, learned language model that translates SVG to PVD, and an inference-only reasoner that conducts zero-shot reasoning about downstream tasks with the PVD Experimental results demonstrate that VDLM achieves strong zero-shot performance in various visual reasoning tasks, outperforming V.-v (Liu ct al. 2023a), G-LLaVA (Gao et al., 2023), GPT-4V (OpenAT 2023b), GPT-4o (OpenA] 2024). and Visual Programming approaches such ViperGPT (Surís al. 2023). Moreover, VDLM also enhances interpretability through better disentanglement of perception and reasoning processes.
representation. An overview of VDLM provided Figure I
To summarize, the key contributions of our work are throefold: First, we identify a critical failure mode of LMMs when reasoning about tasks that require precise, low-level perception. Second, we introduce VDLM, the first attempt to enhance LMMs' the-grained visual reasoning capabilities with vectorized and symbolic visual descriptions- SVG representations and learned Primal Visual Description. Finally, we present an in-depth analysis of the emergent patterns, the disentangled impact of PVD quality on end-task performance, and the remaining limitations, highlighting directions for future work.
3

---

Published in Transactions on Machine Learning Research (05/2025)
PVD Primitives Ontology
Circle
Ellipsee
Rectangle
Triangle
Polygon
Segment
Grid
Graph (line drawing)
Composition-file
Composition-outings
Figure 3: Ontology of the primitives in PVD. Composition of fundamental building blocks for vector graphics, as shown in Figure
2 VDLM Framework
We introduce the VDLM framework, which consists of three main components. First, rule-based perception module converts images into SVG format, capturing low-level visual details ($ 2.1 that are highly comple- mentary to CLIP-like image features. Second, a trained language model aligns SVGs with intermediate visual descriptions ($ 2.2 Third, an inference-only foundational model reasons about downstream tasks using both visual and textual perception results ($ 2.3 Refer to Figure for an overview of VDLM.
2.1 Precise Visual Perception with SVG Encoding
Prior work (Krojrr et al. 2022, Tong al. 2024) has demonstrated Chat, although CLIP-based (Radford et al. 2021 vision encoders are effective at capturing high-level visual semantics, they can fall short in prescrving fine-grained visual details. We propose extracting an SVG representation that more accurately captures detailed measurements and is highly complementary to the widely used visual features. This can be achieved using rule-based image-to-SVG converters, such as Vtracer VTracer (2024), for which we empirically observe near-perfect reconstruction quality on rasterized vector graphic images (see detailed analysis in Appendix D). SVG describes shapes, lines, and colors using mathematical expressions and paths with precise coordinates.
We conduct suite of preliminary experiments ($ A) to investigate the potential of using SVG for representing visual inputs. We empirically observe that SVG representation outperforms CLIP-based features on low-level vector graphics reasoning tasks given sufficient task-specific fine-tuning data. However, two key challenges remain (8 ) First, off-the-shelf foundation models, such as GPT-4 (OpenAl 2023a), have limited zero-shot reasoning abilities when dealing with raw SVG code. Second, obtaining task-specific (SVG, question, answer) data for finetuning is extremely challenging, which restricts generalization to unseen tasks and domains. We discuss how address these challenges by learning intermediate abstraction.
2.2 Bridging Low-Level Visual Perception with High- Language Reasoning
Primal Visual Description (PVD). We propose Primal Visual Description, a higher-level abstraction that transforms low-level SVG paths to more structured primitives required for reasoning. PVD is a text- based visual description that consists of a set of primitive geometry objects, e.g,, circles and line segments. Each PVD element contains the primitives' attributes (e.g., color, shape, position, size) with corresponding predicted values (e.g., blue, circle, pixel coordinates of the center, length of the radius). An example of the PVD representation is follows (See Figure 14 for full definitions):
{"type": "circle", "center": [252, 315],
"radius": 202, "color": [175, 155, 98].
"style": "filled shape").
4

---

Published in Transactions on Machine Learning Research (05/2025)
Unlike raw SVG code, PVD can be directly reasoned about by strong off-the-shelf foundation models to generalize across various downstream tasks. Moreover, PVD is sulficient to serve as a unified visual description across different types of vector graphics, as complex concepts can be composed of multiple primitive shapes. For example, a "cross" can be composed of two "roctangles." As shown in Figure 3 the ontology of the Primal Visual Description contains canonical primitive shape types that can be composed to cover various vector graphics in the wild. The primitive shapes include circles, ellipses, rectangles, triangles, polygons, line segments, grids, paths, and graphs. A path in PVD is defined a non-intersecting polyline. Graphs and grids are defined as set of vertices connected by set of edges. As an initial step to build visually descriptive intermediate representation, we focus on demonstrating proof of concept; extension to a more comprehensive ontology will be left for future work.
Learning SVG-to-PVD alignment with a language model. We then train a language model to generate PVD outputs from SVG inputs. The input is a single SVG path depicting a visual concept, and the output is the predicted one or more primitives in the defined PVD ontology. During inference, given an arbitrary raster image, we first convert it into raw SVG file, which may contain a large number of SVG paths, including noise and speckles. To denoise the raw SVG file and extract salient shapes, we propose an incremental decomposition algorithm. Specifically, we incrementally include SVG paths while checking the difference between the partially rendered image of currently chosen paths and the fully rendered image of the original raw SVG filc. We compute the summation of the absolute pixcl-by-pixel difference between the two images and set an empirical threshold. If the difference after adding a new path is below this threshold, i.e, if the added path does not bring much additional visual information to the scene, we will skip that path. For the ordering of the path selection, we follow the default ordering from |Tracer (2024) that heuristically places the paths with a larger area the front. The paths that come afterward will be stacked on top of previous paths during rendering. Upon obtaining the decomposed single SVG paths, we first generate their PVD representation individually. We then aggregate the individual PVD predictions into holistic perception of the entire image using this JSON template: ["object_0": <PVD output for path 0>, "object_1" <PVD output for path 1>, ...].
Importantly, since PVD is task-agnostic, the data for training the SVG-to-PVD model can be procodurally generated without human annotation. We develop a data generator leveraging PlL.IIageDDaw and VTracer (2024), which creates a large-scale (SVG, PVD) paired dataset containing randomly generated primitives. In some real-world tasks, such as geometry problems, multiple primitive shapes with the same color can overlap. When converted to SVG, these shapes tend to be parsed into one merged SVG path. To enable the SVG-to-PVD model to learn to decode individual primitives from such compositional concepts, we additionally generate data instances with randomly overlapped shapes. The target PVD representation, in this context, is a list of primitive PVD JSON objects. We ensure that each generated image contains only one unicolor object, single or composed, so that the converted SVG contains a single SVG path. This facilitat es a language model in effectively learning the alignment between SVG and PVD.
To improve the robustness to unseen inference images, randomize the image sizes, the positions and ronations of the shapes, as well as the styles of the shapes (filled outlined). We additionally use two data augmentation methods, Gaussian Blur and Pixel Noise, to add variance to the training SVG paths. Our final dataset contains 160K (SVG, PVD) pairs. More details can be found in Appendix C
We fine-tune a pretrained Mistral-7b (Jiang et al. 2023) model on the synthesized PVD 160K dataset to perform SVG-to-PVD generation. We conduct full-parameter fine-tuning for 3 epochs with a learning rate of le-5. The training objective is a standard Language Modeling loss on the generated PVD tokens follows:
N Σ
L =
log P(d: 1o
(1)
where s and d refer to the input SVG tokens and the generated PVD tokens respectively. We use the Megatron- LLM (Cano al.) 2023) library for efficient LLM fino-tining and the entire training process can be done in 16 hours on 4 NVIDIA A A100-40GB GPUs.
5

---

Published in Transactions on Machine Learning Research (05/2025)
Low-Level Visual Reasoning Tasks
Is this an acule angle an obtuse angle?
Iwo lines of equal longth?
yellow semieircle to the lefl of rectangle. True
A cyan ellipse the right of semicircle.
rightnus! shupe magenta cross
Acute
No
False
False
Angle Classification
Length Comparison
Shapeworld Spatial Reasoning (2Ob) MMuttiOJ Superlative)
Determine test example (bottom) depicts the concepl the two reference examples (top)
maze
Target Ouptut
(2*2); [(0,0), (0.1), (I.D)
There
with 2 yellow blocks.
Truc
Target Ouptut: False
NLVR
Geoclidean 2-Shot Learning
Maze Solving (2x2 (3x3)
High-Level Visual Reasoning Tasks (VGBench-QA)
Q: What element of the euvironment depicted by this SVG image?
Q: What is the color of face the SVG image?
Q: What action does this SVG image most commonly represent in the context of food- related applications?
B: River C: Tree
Mountain
A: Pink B: Yellow
A: Cutting or slicing B: Stirring or mixing C: Grilling frying D: Measuring weighing
D: Cloud
C: Blue D: Green
Category
Color
Usage
Figure 4: Our full evaluation benchmark with focus low-level visual reasoning about vector graphics (detailed in 3.1). We additionally include high-level reasoning tasks with rendered SVG images from VGBench- QA Zou el al. 2024). All lasks are evaluated in a zero-shot setting.
2.3
Enhancing Low-Level Visual Reasoning with Primal Visual Descriptions
PVD provides precise visual details that are highly complementary to the semantic-centric visual features from pretrained visual encoders, such as CLIP. Its textual nature further facilitates direct integration into an inference-only LLM or LMM reasoner.
We explore variants of VDLM: VDLM-mm and VDLM-txt, based on the type of reasoner applied. By default, VDLM utilizes a multimodal LMM as the reasoner, referred to VDLM-mm, which takes both the original image and the PVD perception as input. To examine the efficacy of using PVD to represent visual information, we also consider VDLM-txt, which uses text-only LLM as the reasoner. A detailed execution Irace of the VDLM functions is illustrated in Figure 2 We observe that a strong reasoner, such as GPT-4 [OpenAl 2023a),, without any fine-tuning, can effectively perform various types of task-specific reasoning based on the PVD representation. This includes identifying higher-level concepts, computing measurements, examining spatial relations, and performing multi-step reasoning. The reasoning procedure is also more explainable and transparent compared to the output of existing monolithic LMMs.
3 Experiments
3.1
Tasks
Low-level visual reasoning tasks. Evaluating LMMs in tasks that require precise visual perception about vector graphics is a highly underexplored research area and has limited existing resources. To this end, construct new evaluation benchmark that comprises 9 tasks which cover important aspects of low-level visual perception and reasoning, including measurements, spatial relations, counting, logical reasoning, and complex reasoning problems such maze solving. The description of each task is as follows:
(1) Angle Classification: Identify whether an angle is acute or obtuse. (2) Length Comparison: Determine whether two line segments are of equal length. (3-4) Shapeworld Spatial Reasoning: The Shapeworld (Kuhnle &
Copestake 2017) dataset on spatial relations with images containing exactly two objects or multiple objects. (5) Shapeworld Superlative: The Shapeworld datasct on superlative statements. (6) NLVR: The Natural
Language for Visual Reasoning dataset (Suhr et al. (017) which contains diverse counting, spatial reasoning, and logical reasoning queries. (7) Geoclidean 2-shot Learning: A repurposed Geoclidean (Hsu et. al.

---

Published in Transactions on Machine Learning Research (05/2025)
Low-level Visual Reasoning Vector Graphics
SW-S
SW-S
SW
Maze 2x2
Maze
Tools
AC
LC
NLVR
Geo
All
20bj
mObj
Sup
Monolithic Large Multimodal Models
Llava-1.5-7b
0.53
0.49
0.48
0.55
0.35
0.53
0.50
0.00
0.00
0.381
Llava-1.5-13b
0.53
0.51
0.51
0.47
0.61
0.48
0.50
0.00
0.00
0.401
Gllava-7h
0.59
0.50
0.43
0.54
0.43
0.49
0.58
0.00
0.00
0.396
GPT-4V
0.58
0.64
0.77
0.60
0.61
0.63
0.64
0.28
0.02
0.530
GPT-4o
0.63
0.57
0.97
0.92
0.81
0.71
0.46
0.08
0.663
Visual Programming with LLM (text-onlyy reasoner
ViperGPT (w/ GPT-4)
Cl
0.11
0.67
0.61
0.47
0.53
0.43
0.02
0.03
0.00
0.319
VDLM with LLM (text-only)
reasoners
VDLM-txt (w/ GPT-4)
0.89
0.95
0.78
0.80
0.68
0.63
0.40
0.19
0.661
VDLM-txt (w/ GPT-4)
0.73
0.95
0.89
0.68
0.72
0.72
0.64
0.40
0.26
0.666
VDLM with LMM (multimodal) reasoners
VDLM-mm (w/ GPT-4V)
0.53
0.94
0.84
0.62
0.72
0.71
0.69
0.60
0.20
0.652
VDLM-mm (w/ GPT-4o)
0.90
0.95
0.91
0.82
0.82
0.86
0.71
0.61
0.34
0.769
Table Zero-shot accuracy on low-level visual reasoning tasks. Task abbreviations: AC (Angle Classification) LC (Length Comparison), SW-S-20bj/mObj (Shapeworld Spatial Reasoning with two objects or multiple objects), SW-Sup (Shapeworld Superlative), Geo (Geoclidcan 2-shot Learning). "CI" refers to Code Interpreter. VDLM-mm brings consistent overall improvements to GPT-4V and GPT-4o, as indicated by the comparison within the blue and orange rows. Detailed analysis is presented in $3.3 and 44
High-level Visual Reasoning Vector Graphics
VGBench-QQA Color
Category
Usage
Llava-v1.5
0.26
0.32
0.27
0.283
Mlva-x115-133
0.32
0.43
0.39
0.380
Gllava-7b
0.16
0.33
0.21
0.233
GPT-4o
0.58
0.84
0.76
0.726
VDLM-mm (w/ GPT-40)
0.62
0.86
0.75
0.743
Table 2: Zero-shot accuracy on high-level visual reasoning tasks. We show that VDLM-mm preserves the LMM's capability on semantic-centric reasoning that does not require precise low-level perception.
2022 datasct requiring the model to understand compositional geometric concept with only two reference examples. (8-9) Maze Solving: Solve 2x2 or 3x3 maze, given the starting and ending positions. Among these lasks, Angle Classification, Length Comparison, and Maze Solving are newly created from scratch (See Appendix K for more details).
High-level visual reasoning tasks. Although the focus of this work is on low-level visual reasoning, we additionally include a set of high-level tasks to investigate the impact VDLM on knowledge reasoning tasks. These lasks rarely require precise perception of the locations and measurements of the primitives. We leverage VGBench (Zou et al., 2024); benchmark originally proposed for evaluating LLMs in understanding and generating vector graphics codes. In this work, we evaluate LMMs and VDLM-mm for question-answering based the rasterized VGBench SVG images.
Figure shows simplified input and output examples for cach task. Full prompts can be found in Appendix J To reduce the cost of evaluating proprietary models, we randomly sample a subset of 100 instances for each task. We consider a zero-shot evaluation setting for all tasks. Note that the SVG-to-PVD model in VDLM is trained purely on synthesized kask-agnostic data.
3.2 Models
We compare our work with strong baselines, including both stato-of-the-art monolithic large multimodal models (LMMss, i.e., LLaVA-v1.5 (Liu et al. 2023a GLLaVA Gao et al. (0033 GPT-4V (OpenAT

---

Published in Transactions on Machine Learning Research (05/2025)
2023a GPTT40 (OpenAl 2024 well as visual programming agents, e.g., ViperGPT (Surís et al. 2023 Viper GPT employs an L.M to generate code, which can call external vision models, such as GLIP LTi et al. 2022) and BLIP2 Li et al. 2023b), to process the image and generate the final output. Given that ViperGPT--syyle models successfully separate perception from reasoning, we seck to investigate whether the existing perception tools adequately recognize low-level primitives in vector graphics. For VDLM, we explore two variants, namely VDLM-mm with GPT-4V, GPT-4o and VDLM-txt with GPT-4 (text-only).. | We also experiment with applying weaker LMM reasoners, such as LLaVA, to VDLM-mm. We find that interpreting PVD requires certain level of text reasoning capability, and the benefits only emerge with strong LMMs, shown in Figure 5 However, recent strong open-source LMMs, such as Qwen-2.5-VI--725 (Bai et al. 2025
can also benefit from PVD features (see Appendix F for additional experiments). To obtain more insights in comparing with ViperGPT, we further investigale augmenting VDLM-(xI with a Code Interpreter (CT). We employ the GPT-4 Assistant [ for our experiments, designating the code interpreter as the sole tool available. We use the same set of prompts for both VDLM-txt and VDLM-mm. Soe dctails about prompt design Appendix J]
3.3 Results
Table shows the zero-shot accuracy for the evaluation tasks. We outline the key findings follows:
VDLM significantly improves LMMs low-
Emergent Benefit of VDLM on Strong LMMs
level reasoning tasks, while preserving their
Avg. Acc. on Low-level Reasoning Tasks
0.8
capabilities in high-level reasoning. The fact that VDLM-txt performs competitively even with a text-only reasoner highlights the efficacy of the intermediate PVD representation for precise low- level visual perception. Without any task-specific fine-tuning, strong LMM reasoners can effectively in- corporate the additional information provided PVD alongside the image input. Figure further demon-- strates that this benefit only emerges when the LMM has a certain level of text-reasoning ability and per- sists state-of-the-art LMMs. For high-level rea- more
Without VDLM
0.7
With VDLM
0.5
0.3
0.2
soning tasks (Table 2), the improvement subtle, as the tasks focus on the semantics of the vector graphics, such as "what can this be used for?", which rarely require precise location measurements of visual elements.
0.0
Llava-1.5-7B 11538-1.5-13B
GPT4V
GPT-4o
Figure The direct improvements brought by VDLM to LMMs emerge when the LMM possesses sufficient text reasoning capabilities. These improvements are consistent with stronger LMMs, such as GPT-4o, which have enhanced spatial reasoning performance.
QA performance on complex math problems does not necessarily reflect a faithful under- standing of low-level visual concepts. We ob- serve that G-LLaVA (Gao al. 2023). model demonstrating strong performance on geometrie problems, such MathVista (Lu al. 2023), still struggles with understanding basic lines and angles, which are prerequisites for solving geometric math problems.
Existing vision-language models, such as GLIP and BLIP2, are ineffective low-level visual preceptors. This is evidenced by the unsalisfactory performance of ViperGPT, even when equipped with strong planner like GPT-4. On the other hand, we observe that augmenting the reasoning model in VDLM-txt with code interpreters can be particularly holpful for tasks requiring algorithmic reasoning, such as 3x3 maze solving.
*GPT-4V model version: gpt-4- 100-4--100--vsioo--rreeee
1GPT-4o model version: gpt-4o- 2pt-44-2024405553 0124-05-13
tGPT-4 (text-only) model version: gpt-4-0125-previow,
8

---

Published in Transactions on Machine Learning Research (05/2025)
Instance Correctness KDE w.r.t Perception Quality
Task: All
Task Performance w.r.t Perception Quality
1.0
Incorrect
SSIM
7
Correct
DINOv2 Score
6
Median (Incorrect)
CLIP Score
Task Accurarcy
SSIM + DINOv2 Score CLIP Score
Density
5
Median (Correct)
0.6
4-
3-
0.4
2-
1-
0.2
0
0.6
0.7
0.8
0.9
1.0
1.1
0.0
Perception Score
0.4
0.5
0.6
0.7
0.8
0.9
1.0
(SSIM DINOv2 Score CLIP Score)
Perception Score
Figure 7: Instance-level correlation between PVD perception quality and end-task performance, We observe consistent positive correlation in task- level.
Figure 6: Task-level correlation between PVD percep- tion quality and end-task performance.
Impact of errors in PVD perception. In certain tasks, such as Shapeworld Spatial Reasoning, GPT-4o achieves better performance than VDLM-mm. The reason lies in the imperfect perception results of the SVG- to-PVD model, despite the image-to-SVG step achieving near-perfect reconstruction as shown in Appendix D Since the SVG-to-PVD model trained with purely synthetic data, is not yet perfect when generalizing diverse domains. In Appendix E we present an additional ablation study on the design choices of the PVD model, including alternative LLM selections and comparison to the PNG-to-PVD approach. We also carefully analyze the remaining errors in $4.2 and demonstrate the impact of improving perception we discuss future work for developing more general and expressive
end-task performance ($ 4.1 In $
PVD representation.
4 Analysis
4.1
Primal Visual Description Quality End- Task Performance
One advantage of a modular system that enhancing an individual module can leads to improvements in the overall system. In this section, we explore whether a positive correlation exists between the quality of the intermediate perception representation and end-task performance. To investigate this, we first define metrics to reflect the quality of the PVD perception. Upon gencrating PVD perception result, render it back into raster image using procedural image generator. We then compute similarity score between the reconstructed image and the original input image as measure of the perception performance. For measuring the similarity, we consider both pixel-based and ambedding-basee metrics. We adopt the Structural Similarity (SSIM) Index (Wang et al. 2004) score to assess pixel-level similarity. Additionally, to account for semantic similarity, we adopt a CT.IP-scor (Radford et al.) 2021) and a DINOv2-score (Oquab al., 2023), which are calculated as the cosine similarity of the flattened CLIP and DINOv2 embeddings, respectively.
In Figures 0 and 7 visualize the impact of the perception quality, the 9 low-level reasoning tasks with VDLM-txt, at both the task and instance levels. In Figure 0 each point denotes the accuracy of task, with different colors representing different similarity metrics. The dashed lines depict lincar regression results of the points, revealing consistent positive correlation between perception quality and task accuracy across the metrics. Since the task-level accuracy may not be directly comparable across different tasks, we additionally perform an instance-level analysis using Kernel Density Estimation (KDE) on the correctness of all task instances with respect to their perception scores. As shown in Figure 7 the "correct" distribution visibly skews to the area of higher perception scores, indicating that better perception tends to result in correct.
9

---

Published in Transactions on Machine Learning Research (05/2025)
final answer. This finding is promising, suggesting that enhancing the intermediate PVD representation, even with a fixed reasoning model, can effectively boost downstream task performance.
4.2 Interpretable Error Analysis
The improved interpretability, resulting from PVD's disentangled perception and reasoning, allows us to conduct an in-depth analysis of the failure modes of VDLM. We find that both the perception step (SVG- to-PVD) and the reasoning step (PVD-to-answer) can contribute to errors. On tasks that require complex multistep reasoning, such Maze Solving, reasoning errors become more prevalent; otherwise, perception errors most directly contribute to poor performance. Details and illustrative examples of these errors are provided in Appendix B along with a distribution of perception and reasoning errors from human analyses. The prevalent error types for both perception and reasoning steps are summarized as follows.
Common perception crrors include failures in faithfully perceiving novel shapes that are not covered by or cannot be composed within the PVD ontology, and failures capturing intentional constraints between primitives, such as a line exactly segmenting a circle, due to the random nature of the data generation on the positioning of objects. In Table [ improves PVD perception. Common reasoning errors over the PVD perception include failures in discovering intentional constraints without being explicitly asked, such as automatically recognizing that rhombus is not the same concept as a general quadrilateral; failure in handling ambiguous instructions; and failure in complex multi-step reasoning tasks like solving mazes.
we show that the proposed augmentation during synthetic data generation
5 Related Work
Visual shortcomings in large multimodal models. While state-of-the-art LMMs achieve strong
performance on existing multimodal benchmarks (Goyal et al.
2017 Fu et al.
2023 Liu et al.] 2023b d Yu
et al.
2023 Li et al.) 2023a), which primarily focus on natural images, recent work (Lu et al.) 2023 Yue
et al.
2023 Huang et al. 2023 Zhou et al. 2023a Hsu et al.
2022 Gao et al. 2023 has shown that they
struggle with charts, geometric diagrams, and abstract scenes, This observation aligns with recent studies
investigating visual shortcomings in LMMs. Tong et al. (2024) suggests that current LMMs struggle with
visual details because the image-text contrastive pretraining of the CLIP visual backbone does not encourage the preservation of fine- ane-grained visual features, such as orientation and quantity. To address this issue, recent studies have either leveraged the mixture-of-experts approach (Tong al., 2024 Fan et al. 2024; Lu et al, 2024 Jain et al. 2023b), incorporating various types of vision encoders, such as SAM (Kirillov et al. 2023) DINOv2 (Oquab el al. 2023), roducee auxiliary losses that emphasize local details during multinodal pretraining McKinziee et al. (2024): Bica et al. (2024] ; Varma et al. (2023). In this work, we propose a novel perspective for addressing this visual deficiency in vector graphics with an intermediate percoption representation.
Image vectorization and program synthesis. Generating vectorized or symbolic representations of visual concepts has been topic of interest in both the NLP and computer vision communities. Recent
work Vinker et al. 2022 Lee et al.
2023 Ma et al. 2022 Rodriguez et al. 2023 Jain et al. 2023a Tang et al.
2024 Xing et al. 2024 Hu et al. 2024 has investigated generating vector graphics codes from raster images or text prompts. We focus on the reverse problem of understanding and reasoning about vector graphics as
visual inputs. We find that vector graphics reasoning serves as a challenging testbed to evaluate low-level
visual reasoning abilities large multimodal models (IMMs). Although Bubecck et al. (2023): Cai al.
(2023) Zou et al. (2024): Qiu al.) (2024) have demonstrated the potential of text-only large language models (LLMs) in understanding the semantics of vector graphics codes, it remains unclear how to enhance the ability of large multimodal models to process rasterized vector graphics without access to the underlying code, which is more common in real-world scenarios. Therefore, we propose the intermediate Primal Visual Description representation to further enhance low-level perception and reasoning, without sacrificing the performance of semantic understanding. This work is also heavily inspired by related work in neural-symbolic models (Ritchic et al. 2016 Wu et al. [2017] Yi et al. 2018 Mao et al.] 2019 Hsu al. 2023 Zhang et al., 2023 Trinh et al.) 2024). This paradigm aims to de-render visual scenes into structured representations, retrieve programs from the input text, and execute these programs on the image representations. However,
10

---

Published in Transactions on Machine Learning Research (05/2025)
most neural-symbolic work not be applied to recent LMMs and is limited to specific tasks. We aim to learn task-agnostic, visual description that can be directly reasoned about by off-the-shelf LMMss
Disentangling perception and reasoning in large multimodal models. Another closely related line of work has invostigated discntangling visual perception and reasoning with visual programming (Gupta &
al. 2023 Ge et al. 2023 Wu & Xie 2023) and tool-using (Wu et. al. 2023 Liu Kembhavi 2023 Surís
et al. 2023c Qiao et al. 2024.. These models leverage the code generation capabilities of LLMs to compose
and employ set of lanion-language or vision-only models, such as object detection and image caption models,
as subroutines for solving visual reasoning tasks. Despite promising performance on natural images, as shown
in $ 3 we find that these models are still limited by the existing vision-language models' inability to process low-level primitives effectively.
6 Limitations and Future Work
VDLM represents an imitial step to constructing descriptive intermediate representation for low-level visual reasoning. While our results strongly support this proof of concept, several limitations remain, highlighting directions for future work.
First, due the scarcity of end-to-end instructional tuning data (discussed in 22.1). the current SVG-to-PVD model is constrained compositions of simple primitives, for which annotation. To enhance PVD's generalization across more diverse domains, we identify several possible improvements. For instance, incorporating human-created vector graphics with procedural annotations could enrich the ontology and training dataset. Additionally, integrating visual search Wu & Xie (2023) during inference could facilitate the conversion of focused regions or parts into PVD, enabling multi-step perception and reasoning. Second, SVG and PVD are inherently designed for representing 2D vector graphics. Future research is needed develop a more general intermediate representation that extends beyond 2D vector graphics to encompass 3D structures and natural images.
synthesize data without requiring
Acknowledgments
This rescarch is based upon work supported by U.S. DARPA ECOLE Program No. #HR0011239000 AFOSR YIP FA9550-23-1-0127, ONR N00014-23-1-2355, ONR YIP N00014-24-1-2117, ONR MURI N00014-24-1-2745 and the Stanford Institute for CemannCenteeed AI (HAI). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, loannis Antonoglou, Julian Schrittwicser, Amelia Glacse, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca. Roelofs, Anaïs White, Anders Andreassen, Tamara Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv prepring arXiv:2312. 11805, 2023. URL https://doi orgooogg//// 10/1000444
2312. 11805
Jinze Bai, Shuai Bai, Shusheng Yang, Shijic Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-V Owen-VL: A Frontier Large LasiunnL Model with Versatile Abilities. arXiv preprint ar arXiv:2308.122 12966, 2023. URL httpps://doi.org/100. arxvv. 2308. 12966 12966
11

---

Published in Transactions on Machine Learning Research (05/2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arxiv preprint or arXiv:200.. 13923, 2025.
Toana Bica, Anastasija Nic, Matthias Bauer, Goker Erdogan, Matko Bosnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascamu, and Jovana Mitrovic. Improving Fine-Grained Understanding in Image-Text Pre- Tre-Trainnn. arXiv prepring arXiv:2401.09865, 2024. URL
/ttps://doi.org// 10. 40850//....55//22111111 arr.v./rr/vvvvv. 04401099866
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tílio Ribeiro, and Yi Zhang. Sparks of Artificial General Intelligence: Early Experiments with GPT-4. arXiv prepring ar 23v: 222112 2023. URL htttp:://doi.org///
arXiv. 2303. 12712
Mu Cai, Zeyi Huang, Yuheng Li, Haohan Wang, and Yong Jae Tee. Leveraging Large Language Models for Scalable Vector Graphics- Drrvhen Image Understanding. arXiv prepring arXiv:2306. 06094, 2023. URL https://doi. www.oogg//...... 48550/55551330 2306.06994
Alejandro Hernández Cano, Matteo Pagliardini, Andreas Köpf, Kyle Matoba, Amirkeivan Mohtashami, Xingyao Wang, Olivia Simin Fan, Axel Marmet, Deniz Bayazit, Tgor Krawczuk, Zeming Chen, Francesco Salvi, Antoine Bosselut, and Martin Jaggi. EpfLLM Megatron-LLM, 2023. URL https://github com/
epfLLM/Megatron LLM
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. ShareGPT4V: Improving Large Multi-Modal Models with Better Captions, 2023a. URL https://doi
org/10. 485500 http://aXvvv.....
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistcc Tasks. arXio prepring Xrxi:23312. 111122..1221 14238, 2023b. URL https: org/ 2xgg/10.// 48540/555555/ arxiv. 2312 14238
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023. URL https://lmsys blogb
2023-03-30-vicumm vinaa/
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. InstructBLIP: Towards General-purpose Vision-Languuae Models with Instruction Tuning. Tn Advances in Newral Information Processing Systems 36: Annual Conference on Neurol Information Processing Systems 2023 (NeurIPS), 2023. URL http:: .niss. cc/paper html
Abstract-
Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang, Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. MouSi: Poly-Visual-Be Visual-Expert Vision-Language Models. arXiv prepring or 24t1:1401.12200 2024. URL https://don org/ 10. 48550/arXiv 1401.17222
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. ar prepring arXic:2306. 13394, 2023. URL /ttps://doi.org// 10. 1.8500/axiiv 2306. 13394
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model. arXiv prepring orXi::2312.11970 2023. URL URLhttps:: httph::t/p/att org/1o.rg111.... 48550/ /arvvv 2312. 11370
12

---

Published in Transactions on Machine Learning Research (05/2025)
Jiaxin Ge, Sanjay Subramanian, Baifeng Shi, Roci Herzig, and Trevor Darrell. Recursive Visual Programming. arviv prepring or arXiv:2312.02249 2023. httllhttps://dd// http::::///ii.o////1//// wttp 2312. 02249
Yash Goyal, Tejas Khot, Douglas Summer--Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In 2017 IEEE Conference on Computer Vision and Paltern Recognilion (CVPR), 2017. URL https://doi htt.///ooooo/////////////////// 1ttp///...10///P..22000 670
Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional Visual Reasoning without Training. In HEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. URL
https://doi. org/ 10. 10w.1//55///P/22...... 20217329..22........ 2023.01436
Joy Hsu, Jiajun Wu, and Noah D. Goodman. Geoclidean: Few-Shot Generalization in Euclidean Geometry. In Advances in Neural Information Processing Systems 35: Annual Conference on Newral Information
Processing Systems 2022 (NeurIPS), 2022. URL http::/pppppprrr.ttcm/ catpppers.iiiis.... hecepaper_fileee hallles/papr//0222) Sattro---Abtrac---attttttee and Benchmarks.
Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum, and Jiajun Wu. What's Left? Concept Grounding with Logic-Enhanced Foundation Models. In Advonces in Neural Information Processing Systems 36: Annual Con- ference Neural Information Processing Systems 2023 (NewrIPS), 2023. URL htttp://papers... .npee.iip...... cc/ paper
html
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations (ICLR), 2022. URL https://openreviee wttp::/www..eeoooommmmm//////eee
Juncheng Hu, Ximing Xing, Zhengqi Zhang, Jing Zhang, and Qian Yu. Vectorpainter: A novel approach stylized vector graphics synthesis with vectorized strokes. arXiv preprint Sttk::2405022663 2024.
Kung-Hsiang Huang, Mingyang Zhou, Hon Pong Chan, Yi R. Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning. Xixiv prepring arXiv:2312. 2023. URL
2312. 10160
Ajay Jain, Amber Xie, and Pieter Abbeel. VectorFusion: Text-to-SVG by Abstracting Fixel-Based Diffusion Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023a. URL https://doi. org/ 10. 10t.1111//CVPP229.... 202.:/299..222.001100
Jitesh Jain, Jianwei Yang, and Humphrey Shi. VCoder: Versatile Vision Encoders for Multimodal Large Language Models. arXiv prepring arx:::2312. 2023b. URL httpp://ddott
2312. 14233
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Télio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayod. Mistral 7B. arXiv prepring arXiv:2310. 23..0635, 2023. URL https: //doi.org/10 48550/855K..
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment Anything. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023. URL https
//doi.org//t org/10. 10. 1109/ICCV51070. 2023.00371
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. Int. J. Comput. Vis., 123(1):32 -3, 2017. URT http:////....ooooo.....////1110/
13

---

Published in Transactions on Machine Learning Research (05/2025)
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Maria Ponti, and Siva. Reddy. Image Retrieval from Contextual Descriptions. In Proceedings of the 60th Annual Meeting of the Associalion for Computational Linguistics (ACL), 2022. URL https://doi.org: http:://ooo.rrr0000..6//////1120 105.00/666///100222221-1.ggg 241
Joseph B Kruskal. On the Shortest Spanning Subtree of Graph and the Traveling Salesman Problem. Proceedings of the American Mathematical society, 7(1):48-50 1956.
Alexander Kuhnle and Ann A. Copestake. ShapeWorld- A New Test Methodology for Multimodal Language Understanding, Xrv prepring or arXiv:1004.04111 2017. URL httt:://aaxii.org/bbb//////
04517
Hyundo Lee, Inwoo Hwang, Hyunsung Go, Won-Seok Choi, Kibeom Kim, and Byoung-Tas Taa Zhang. Learning Geometry-Awan Representations by Sketching. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. URL https://doi.orrg org/ 1109/CVPR527// 2ww./22222..22........ 2023. 02233
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. arXiv prepring arXiw:2307.16199, 2023a. URL
https://doi.... org/ 10. 48550/ 48550/88X5i......./ www.a...vv.............. 16125
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping Langu Pre-training with Frozen Image Encoders and Large Language Models. In International Conference Machine Learning (ICML), 2023b. URL https://proceeddngggg mlr. press/v202/11123g //10////........rrs/////////3133 html
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Tu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded Language-Imag Pre-training. In JEEE/CVE /CV Conference on Computer Vision and Pattern Recognition (CVPR), 2022. URL
1019//////109/// 2022.01063
Tsung-Yr Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In Computer Vision - ECCV 2014- 13th European Conference, 2014. URL https://doi.org/// wwt.:///....110001900 1007
Haotian Liu, Chunyuan Li, Yuheng Ti, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. Xixiv prepring arXiv:2310.099: 10.03744. 2023a. URL http:: ottps:///oi.orr//0 http://....85///aii....... 2310 03744
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, In Advances in Newral Information Processing Systems 36: Annual Conference on Neural Information Process- ing Systems 2023 (NewrIPS), 2023b. URL http::/paperr: nips wttpppceettllee/eeeeeeeeeeecceee abstraaaaattttttetttattaaaaaa--- Abstract- Confeeeetttoooooeceeeeee html
haahh/
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents. arv prepring Xiv:2311.05437, 2023c. URL https://doi.oorg100 otg//1.448500/r2ii...... 48550/arXiv
2311.054311
Xiao Liu, Tianjie Zhang, Yu Gu, Tat Long Tong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint orXiv:2408.00: 2024.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBBnch: Is Your Multi-modal Model an All-around Player? arXiv prepring orXio:2307. 06281, 2023d. URL htttps///doi: http:///oooooo/////// org/10 48550/ arxiv 2307.0622
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xic, and Chong Ruan. DeepSock- VL: Towards Real-World LanguanL Understanding. arXiv prepring 0ate2440..5552. 2024. URL
htttp:://bi..rr//10.855555 wtw.:85.50002222.2..00.055 200...440..5550
14

---

Published in Transactions on Machine Learning Research (05/2025)
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V, Bard, and Other Large Multimodal Models. arXie prepring S.k0:2310002255: 2023. URL
https://doi.ows org. 48550/ arxiv. 2325.02255
Xing Han Lú, Zdeněk Kasner, and Siva Reddy, Weblinx: Real-world website navigation with multi-turn dialogue. arviv preprint arXiv:2402.05088 2024.
Xu Ma, Yuqian Zhou, Xingqian Xu, Bin Sun, Valerii Filev, Nikita Orlov, Yun Fu, and Humphrey Shi. Towards Layer-wise Image Veclorization. In JEEE/CVE Conference on Compuler Vision and Pallern Recognation (CVPR), 2022. URL
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Ncuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In 7th Interna- tional Conference on Learning Representations (ICLR), 2019. URL https:/ mettfforumiie rJgMihRetm
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: Visual Question A Answering Benchmark Requiring External Knowledge. In IEFE Conference on Computer Vision and
Pattern Recognition (CVPR), 2019. URL http:://ppenaccess. thecvf. coom/oonent cttp///tteetttcte_____ee ______CCR________ 201,200199hhll Marino _atooo_KKKV_____ A_ Vissalllueetauuuaaaaaaaeaaaaaeee Question Answering Benchmark, ReeuiriiggExterraal Externateeaaa..... Knowledge httrrgeteeeeeaaaaaoaeeeeeeee 2019 _as__papperh_tt
CVPR_
Brandon McKinzie, Zhe Gan, Phin-phil Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanject Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. MM1: Methods, Analysis & Insights from Multimodal LLM Pre- training. arXiv preprint arXiv:2403. 200.0660,, 2024. URL hattps://dot //t.///////////..////////////// 48550/ arXiv. 2403. 2403.09611
09611,
OpenAl. GPT-4 Technical Report, 2023a.
OpenAl. GPT-4V(ision) System Card, 2023b. URL http:p////chhh.opaaaaa..////pppp Card pdf
OpenAl. GPT-40 System Card, 2024. URL httt::///oaa.ppeaa.co////oooooop
pdf
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wo- jeiech Galuba, Russell Howes, Po-Yao Huang, Shang-Waa Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. arXiv prepring 27Xiv:2304.07198, 2023. URL htttps://dot 08851...800////
48550/ /tto//rxiv.... 2304.07193
Yuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang, Lin Chen, Songyang Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Prism: framework for decoupling and asscssing the capabilitios of vlms. arXiv preprinl X4v: 14544, 2024.
Zeju Qiu, Weiyang Liu, Haiwen Feng Zhen Liu, Tim Z Xiao, Katherine M Collins, Joshua B Tenenbaum, Adrian Weller, Michael J Black, and Bernhard Schölkopf. Can large language models understand symbolic graphics programs? arXiv preprint arXiv:2408.. 08..0800.8.... 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning Transfcrable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference mlr. Machine Learning (ICML), 2021. URL http:///occcaaan......eeeeeesees
15

---

Published in Transactions on Machine Learning Research (05/2025)
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Ti, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.1577, 2024.
Daniel Ritchie, Anna Thomas, Pat Hanrahan, and Noah D. Goodman. Neurally- Procedural Models: Amortized Inference for Procedural Graphics Programs using Neural Networks. In Ad- vances in Neural Information Processing Systems 29: Annual Conference on Newral Information Pro- cessing Systems 2016 (NewrIPS), 2016, URL https: http:///pppceddnggeeeeeeaaa.c//s hwtpc/ppp..cpppape////////aaaaaa .ttll
Juan A Rodrigucz, Shubham Agarwal, Issam H Laradji, Pau Rodrigucz, David Vazquez, Christopher Pal, and Marco Pedersoli. Star Vector: Generating Scalable Vector Graphics Code from Images. arXiv preprint arXiu:2312.11530 2023.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- OKVQA: A Benchmark for Visual Question Answering Using World Knowledge. In Computer Vision ECCV 2022 17th European Conference, 2022. URL https: http://i...gg///////
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. TextCaps: A Dataset for Image Captioning with Reading Comprchension. In Computer Vision ECCV 2020 16th European Conference, 2020. URL ntttpa://di..oo 10. 11tp:/000000000--00000--0-----00
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi A Corpus of Natural Language for Visual Reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017. URL 1tt1//111119////0111000000 2023.
18653/11/P17-2023
Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for
Reasoning. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023. URL https:
Zecheng Tang, Chenfei Wu, Zekai Zhang, Mingheng Ni, Shengming Yin, Yu Liu, Zhengyuan Yang, Lijuan Wang, Zicheng Liu, Juntao Li, et al. Strokemuwa: Tokenizing strokes for vector graphic synthesis. arXiv preprint ar 2iv:22401. 17098, 2024.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xic. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. ar prepring ar xrXiv224010660 062000 2024. URL https://doi. org/ 10///10..0 48550/ http//rxiii... 2401. 06209
Trieu H. Trinh, Yuhuai Wu, Quoe V. Le, He He, and Thang Luong, Solving Olympiad Geometry without Human Demonstrations. Nat., (25(7995):476 (1995))4764482 482, 2024. URL /ttps:///ab..org111100833
841586-023-06747--
Maya Varma, BenntB Delbrouck, Sarah M. Hooper, Akshay Chaudhari, and Curtis P. Langlotz. ViLLA: Vision- Language Representation Learning from World Data. In JEEE/CVF International Conference Computer Vision (ICCV), 2023. URL https://doi.org/// 21tp://0.001///110000000022..2.3
Yacl Vinker, Ehsan Pajouheshgar, Jessica Y. Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. CLIPasso: Semantically-aware Object Sketching. ACM Trons. Graph., 41 (4):86:118861111 2022. URL
h1tp://...1111/52222222 3530068.
VTracer. Vtracer, 2024. URL https: htttps:/www..visitte http://www.ssoooottterrrrr/////s
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.01011, 2024.
Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. ar Xio preprint 25Xi::2001.111 11733, 2025.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Ecro P. Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. JEEE Trans. Image Procelss., 13(4):600-0) 612, 2004. URL
attps://aoi.orr/111 wtt..///............... .00.. 819861
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models. arXiv prepring arXiv:2308.04671, 2023.
URL https://doi. org/10. 48550/arXiv 2303.04671
Jiajun Wu, Joshua Tenenbaum, and Pushmect Kohli, Neural Scene De-rendering, In 2017 IEEE Conference Compuler Vision and Pallern Recognition (CVPR), 2017. hot.:/ttpeee//iii.or////111////1
744
Penghao Wu and Saining Xie. V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs, arXiv prepring urXiu:2812.14135, 2023. URL
14135
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:240,.09912 2024.
Ximing Xing, Haitao Zhou, Chuang Wang, Jing Zhang, Dong Xu, and Qian Yu. Sygdreamer: Text guided svg generation with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4546 4555, 2024.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In Advances in Newral Information Processing Systems 31: Annual Conference on Newral Information Pro- cessing Systems 2018 (NewrIPS), 2018, URL https::/pproceddiiii wwt.igg...rri..... www.nes.../pappeeeeeeeeeee cc/paperr2011//aatl 5e388103a391daabeee wwtp:/dddddd667/67309333 Abstract html
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MetVet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv prepring arXiv:2305.02/ 02490, 2023. URL https:///oi. www/oo..//../.. 480000.85558880 http:/rXivv... 2308. 2308.02490
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A Massive Multi-disccillnne Multimodal Understanding and Reasoning Benchmark for Expert AGI. arXiv prepring 2rX1v:2311.16500 2023. URL htttss://doi.ooo http///oooorro//////// 48550/ arxiv. 2311. 16502
Sharon Zhang, Jiaju Ma, Jiajum Wu, Daniel Ritchie, and Maneesh Agrawala. Editing Motion Graphics Video via Motion Vectorization and Transformation. ACM Trans. Graph., :20:):220:-222:11 2023. URL
https://doi. org/10. 1145/3618316
Boyuan Zheng, Boyu Gou, Jibyung Kil, Huan Sun, and Yu Su. Gpt-Av(ision) is a generalist web agent, if grounded. In Forty-first International Conference Machine Learning, 2024. URL https://openrevierr htw.fformmmidppieckkJ000
Mingyang Zhou, Yi Ren Fung, Long Chen, Christopher Thomas, Heng Ji, and Shih- Chang. Euhanced Chart Understanding via Visual Language Pre-training on Plot Table Pairs. In Findings of the Association for Computational Linguistics: ACL 2023, 2023a. URL httpp:://ooi: org/
(trgg/00.000 10. 10.18653/1122023 http://///......oooooo....o findings-acl
85
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint XrXi:2307.1388 13854, 2023b.
Bocheng Zou, Mu Cai, Jianuui Zhang, and Yong Jae Lee. Vgbench: Evaluating large language models on vector graphics understanding and generation. arxiv preprint arXiv:2477.10972, 2024.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Appendix
The appendix is organized as follows: In Appendix wc present preliminary experiments comparing SVG and image-based representations. In Appendix B we include details on error analyses, and in Appendix C present additional experiments on Vtracer
A
we describe Primal Visual Description details. Appendices D
visual encoding quality, PVD model variants, and open-source LMM reasoner variants. Appendix G provides additional qualitative examples of PVD parsing novel concepts via composition. Appendix H presents further efforts in refining prompts for integrating PVD into LMMs. Appendix I
shows the full input. and output from Task prompts and newly constructed downstream task datasets can be found in Appendices and respectively. In Appendix L include detailed statistics for all of the datasets we used.
GPT-4 for the maze-solving example depicted in Figure 2
A Preliminary Experiments on SVG Representations
We introduce a suite of probing tasks to evaluate current LMMs' capabilities in performing tasks with vector graphics. The results show that even state-of-tho-art LMMs, such as GPT-4V, struggle with tasks that require precise perception of low-level primitives, such as comparing the lengths of two lines. We then investigate where this deficiency originates and propose an alternative representation, Scalable Vector Graphics (SVG), for representing such precise low-level features. We find that, compared to image-based representations, SVG reprosentations can be more efficient for visual reasoning vector graphics. However, they are not without their own limilations, which we will elaborate on in A.3
A.1 Image and SVG Representations
In the probing tasks, we include both discriminative and generative tasks, each with varying levels of emphasis low-level visual details. Illustrations of the input and output examples are available in Figure S We additionally include a non-vector- vector-graphics task, Clevr QA, which consists of realistic 3D rendered seenes. This is to test the limits of SVG representations in encoding 3D objects within realistic images. Detailed statistics of these tasks can be found in Table 13
For each task, we consider two evaluation settings: zero-shot and fine-tuning. We explore two types of representations for the input image: (1) direct use of the image pixels, encoding them into patch embeddings with image encoder, c.g., CLIP (Radford et al. 2021 (2) conversion of the image into SVG code using
rule-based raster-to-SVG converter (VTracer) 2024
For fine-tuning with the image input, we instruction-finetune Llava-v1.5-7b (including the LLM-backbong and the projection layer) using Lora (Hu et al., 2022) on the training set for one epoch. For fine-tuning with the SVG input, we only fine-tune the LLM backbone of Llava-v1.5, Vicuna (Chiang et al., 2023 using Lora for one epoch, with the input image's SVG code concatenated in the context. The results are shown in Table 3 Key observations include:
(1) The SOTA oounesource LMM, Llava-v1.5, struggles to achieve non-trivial performance most probing tasks even with dedicated fine-tuning. On tasks with binary choices, Llava tends to predict homogeneous answers, disregarding differences in the input image.
(2) The SOTA closed-source LMM, GPT-4V, excels on task Line or Angle, which focuses on querying the high-level semantics of the primitive concept ("what's in the image"). However, its performance significantly decreases on tasks requiring more precise low-level perception, e.g., Angle Classification and Length Comparison.
(3) Fine-tuning the LLM backbone, Vicuna, with SVG inputs consistently outperforms fine-tuning the entire Llava model with image inputs. This highlights the potential of using SVG as an alternative representation in vector graphics.
(4) We note that SVG may inherently be inefficient representing rendered 3D scenes and realistic images due to factors like camera perspectives, lighting, and shadows. While our focus in this work is on vector graphics, leave the extension to other domains for future exploration.
18

---

Published in Transactions on Machine Learning Research (05/2025)
Probing Tasks
How many big things either green objects or spheres?
Is this a line angle? Line
Are the two lines of equal length?
an ucute angle or an obtuse angle?
No
Acute
Angle
Length Comparison
Angle Classification
Clevr QA
Task: Scene Description (JSON) Cimeraation
Tusk: Maze Configuration (JSON) Generution
Target Ouptut: [fccllor: I'name': 'eyan', 'rgh': [0.0, 1.0, LO0, 'shade; 0.28), shape': 'craps' 'size'; ['x': 0,08, y: 0.083, 'rotation': 0,52, "texture': solidd, position': f'x': 0.71, yy: 0.09)...... ...
Tirget Ouptut: ('size:: [2, 2). 'connection list' [[[Tre,,
True], [False, False]], [[Fas]] False], [Tme, False]], 'start position': [0, 0J, 'end pod_tioiiion"; [l. 1J. 'solation:: [[0. [L. 0]. L]]
Shapeworld Scene
Maze Scene
Figure 8: Illustration of the probing tasks. The four tasks at the top are question-answerii tasks, while the two tasks at the bottom are ccenegenneattin tasks. The goal of the scene-generation tasks is generate the entire structured scene description following predefined schema.
Input Type
Line or Angle
Angle Classification
Length Comparison Clevr QA
GPT-4V
Image SVG
1.00
0.58
0.64
0.57
Zero-Shot
GPT-4
0.45
0.47
0.60
0.36
Llava-v1.5-7b
Image
0.50
0.50
0.45
Finetuned
Vieuna
SVG
0.93
0.70
0.99
0.54
Shapeworld Scene
Maze Scene start-pos (acef)
Input Type
shape (aect)
position (124)
connoctivity (acet)
end-pos (acet)
Zero-Shot
GPT-4V
Image
0.33
0.27
0.27
Llava-v1.5-bb
Image
0.04
0.67
0.26
0.03
0.03
Finetuned
Vieuna
SVG
0.15
0.07
0.54
0.08
0.09
Table 3: Probing task results. We report the accuracy for the four question- answering tasks at the top. At the bottom, use different metrics for different fields in the predicted scene description JSON. "acc" refers to accuracy (larger is better) while "12" refers to the Euclidean distance between the predicted and ground truth [x, y] coordinates (lower is better). Scores with blue background denote the better finc-tuned method compared to the SVG and Image representation. Scores with a red background denote tasks where fne-tuned methods cannot outperform zero-shot GPT-4V. Detailed analysis can be found in $ A.I
A.2 Llava's Failure Mode in Visual Reasoning with Vector Graphics
We further invest igate whether the difficulty in understanding low-level visual features of Llava models stems from (1) the visual backbone itself, i.e., CLIP, or (2) the bridge between the visual backbone and the LLM backbone. We include a set of Linear Probing experiments on three binary classification probing tasks, where we train a simple linear classifier based on the visual backbone features (before and after projection) of the Llava model. As shown Figure D
(1) In tasks requiring more precise low-level perception, such as Angle Classification and Length Comparison, CLIP embeddings are inherently less effective at capturing relevant features. Furthermore, shown in Figure 10 in some tasks, e.g., Length Comparison, linear regression even fails to achieve 90%++ training accuracy after 10 epochs of training, struggling to converge.
(2) When connected to an LLM using the projection layer, the visual features in Llava become less effective for low-level visual reasoning. Additionally, there a significant gap between linear probing and instruction fine-tuning performance. These results suggest that even the backbone does preserve useful features, the LLM cannot effectively leverage those visual tokens after projection.
19

---

Published in Transactions on Machine Learning Research (05/2025)
Linear Probing on Llava's Visual Backbone
0.99
1.0
0.96
0.93
0.90
0.8
0.67
0.66
Accuracy
0.6
0.50
0.50
0.50
0.4
Linear Probing: CLIP Emb
0.2
Linear Probing: CLIP Emb + Projection
Instruction Finetuning: Llava
0.0
Line or Angle
Angle Classification Length Comparison Tasks
The average accuracy of linear probing, computed across ten epochs. Detailed training and testing scores for each epoch be found in Figure 10 The results demonstrate that (1) CLIP embeddings are less effective for tasks requiring precise perception, such Length Comparison, in comparison to tasks that emphasize on higher-level semanties, such as Line or Angle; (2) connecting to an LLM through the widely-used Llava-style architecture results in further diminished performance on tasks involving low-level visual details.
Figure 9:
We hypothesize that the failure mode likely stems from the multimodal pretraining and instruction-tuning paradigm, where the tasks are biased towards high-level semantics, such as image captioning (Lin 2014 4
Sidorov al. 20200) and natural-imager based VQA (Goyal et al. 2017 Krishna et. al. 2017 Marino et al.
2019 Schwenk et al. 2022). The training mixtures (Liu et al. 2023b Dai et al. 2023 Chen et al. 2023a for current LMMs predominantly focus on high-level fcatures of images, providing little incentive for models to retain low-level visual details. For example, the caption of an image containing a 2D maze, such as the one shown in Figure 2 is likely to be "A 2x2 maze with black lines, a red circle and a star." and may not include detailed configurations the mazes, such as the precise locations of the walls, the red circle, and the red star.
A.3 Remaining Challenges of Using SVG Representations
Although have demonstrated that SVG can serve promising alternative representation for rcasoning about vector graphics, we identify several remaining challenges:
(1) Pretrained LLMs, including the most capable ones such as GPT-4 (OpenAl 2023a possess limited out-of-the-box understanding of SVG code. This limitation is evidenced by the low zero-shot performance of GPT-4 with SVG input (see row in Table B
(2) Even after finetuning, the SVG-basod LLM may still underperform zero-shot GPT-4V on certain tasks, particularly those involving complex scenes, such as Shapeworld Scene and Maze Scene. In these instances, the SVG code becomes excessively verbose. These findings suggest that learning model to directly comprehend the raw SVG code of an entire image poses significant challenges.
(3) A fundamental challenge, irrespective of the chosen representation for visual input, is the lack of generalization capability to unseen tasks and various vector graphics image domains. we rely on existing LMM training mixtures, even any image can be converted into SVG code, the tasks remain biased towards high-level semantics. In addition, it is infeasible directly mannally construct and annotate (SVG, question, answer) pairs covering diverse tasks with vector graphics.
20

---

Published in Transactions on Machine Learning Research (05/2025)
1.0
Train Loss (Log Scale)
0.9
Train Accuracy
0.8
0.7
0.6
2
2
Epochs
Epochs
1.0
Test Loss (Log Scale)
0.9
Test Accuracy
0.7
0.6
Epochs
Epochs
CLIP Embedding Projection: Line or Angle
CLIP Embedding: Line or Angle
CLIP Embedding Projection: Length Comparison
CLIP Embedding: Length Compurison
CLIP Embedding
Projection: Angle Classification
CLIP Embedding: Angle Classification
Figure 10: Linear probing training details: Different line styles represent different tasks, while different colors refer to different visual embeddings used for training the linear classifier. The training loss (top-left) shows that the projected embedding (orange lines) learns slower pace compared the original CLIP embedding (blue lines). The training accuracy (top-right) reveals that for certain tasks, such as Length Comparison, the model continues to struggle with overfitting the training set even after 10 epochs.
These challenges motivated us propose another layer of abstraction, the Primal Visual Description, aimed bridging the gap between low-level perception and high-level language reasoning on downstream tasks.
B Error Analysis Details
As introduced in $ the proposed VDLM consists of two stages focused on perception- namelly, Image-to-SVG and SVG-Lo-PVD, and one stage focused on reasoning, i.e., PVD-to-final answer. We aim to investigate the errors in both the perception and reasoning modules.
For cach task, we mamually examine orror cases and determine whether the crror primarily stems from the perception stage or the reasoning stage. We task a human with reviewing the reconstructed image from the PVD representation and assessing the question of the task instance. Tf, for a human, the reconstructed image is still insufficient for solving the task, we classify this error as perception error. Otherwise, it categorized reasoning error. Figure II illustrates the distribution of errors between perception and reasoning stages. We further identify some typical categories of perception and reasoning errors as follows:
Common perception errors. (1) Novel shapes not covered by the Primal Visual Description (PVD): For example, as visualized Figure [2 the Shapeworld dataset includes a "semicircle" shape type which not in the PVD ontology; we see that the Icarned SVG-to-PVD model tends to predict it as an ellipse. This perception error directly contributes to the inferior performance of VDLM-mm compared GPT-4o the Shapeworld tasks, as shown in Table 1
21

---

Published in Transactions on Machine Learning Research (05/2025)
Perception Reasoning Errors
10
Error Distribution
Perception
Reasoning
Perception Reasoning
Brror Count
2
49.4%
50.6%
Length Comparison
Angle Classification
Shapeworld Spatial (2Obj)
Shapeworld Spatial (MultiOb))
Shapeworld Supertative
Geoelidean S-hoo
NLVR
2x2 Maze Solving
3x3 Maze Solving
Tasks
Figure 11: Error distribution by VDLM-txt between perception and reasoning on low-level vector graphics reasoning.
PVD Perception Visualization
Error Type
Input Image
PVD Perception
[... 'objec=_2" 2 [('type's 'ellipse' 'center'; 199, 90], 'major axis lergth': 'minor_axis__ae_
Novel shape (semicircle)
21, 'minor axis axis_Iength'
10, 'rotation"s 'color': [249, 249, 621, 'style'; 'filled shape" }]]
('object_00 0': [] [''type::
Accurate constraints (circle segment)
'triangle' l'type':
ciicle''
"object_"" 1': [ t'type':
'ellipse' ....) iemissing
line segment in the circle
an the right>]
Figure 12: Perception eтror examples. The example at the top illustrates error wherein the SVG model predicts a semicircle as an ellipse. The example at the bottom demonstrates that the SVG-to-PVD model struggles to decode overlapping primitives with accurate constraints, such as a segment of a circle.
(2) Accurate constraints between primitives: Although the PVD accommodates scenarios where multiple objects of the same color overlap, the attributes, e.g., position, of each object are decided independently and randomly. Thus, the SVG-to-PVD model often fails to capture intentional constraints between objects; for example, a line that perfectly segments a circle. These constraints are particularly emphasized in the Geoclidean 2-shot Learning task (Figure 12 where VDLM struggles to outperform GPT-4V and GPT-4o.
(3) Very small objects: During inference, the iterative decomposition process heuristically ignores SVG paths that only contribute only minor differences to the reconstructed image. This method effectively reduces
22

---

Published in Transactions on Machine Learning Research (05/2025)
PVD Perception Visualization
Error Type
Input Image
Reasoning Model (GPT-4) Response
To analyae the given. ISON and identify the patsen and constraints of reference and test concepts, ler' break down the information provided for each object:
Discovering constraints
Objecct (**h (Test Example):
Type: Polygon
(Rhombus
Vortiee:: [3304, 416), (395, 527), (433, 665), (295, 629)]
Quadrilateral)
Chimclensties: Outlined shape, black color, polygon with verties.
**Ojjcct 1* (Reference Example Type: Polygon ...
**Objest 2* (Reference Example 2):
Type: Polygon ...
Given the descriptions, the reference examples (Objject and Object 2) and the test example (Object all polygons with vertices, suggesting they quadrilaterals...
Given information and assuming the concept involives quadrilaterale without requiring them to have uniform angles or sides (since we cannor precisely measure them from the given data), the example appears follow the same broad concept examples.
being quadrilateral
By cxaminiig the connections. However, we can deduce this from absence gnd between cells.
where walls ihus inker dizectly specify walls between cells,
Algorithmic reasoning (Inferring maze connectivity)
For simplicity, let's issume a fully connected grid without internal walls ISON doesn't spoeify walls between cells explicitly), ), and then adjust based typical maze patterns:
Verticcal Connections:** All trae, indicating no vertical walls between cells.
**Horizontal Connections:** All mue, indicating horizontal walls herween cellss.
("solution": "[(), 1), (0, 0g""
Figure 13: Reasoning error examples: The example at the top demonstrates that the reasoning model fails to uncover the deeper constraints within the perceived polygons. It is able recognize that they are all quadrilaterals but unable to further discern that the reference concepts rhombuses with four equal sides, while the test concept is not. The example at the bottom illustrates that the reasoning model struggles to infer connectivity based on the perceived grid, thus failing to provide the correct solution.
noise from the rule-based image-to-Se SVG converter but may omit very small objects in some cases. Adjusting this threshold is necossary for specific scenarios.
Common reasoning errors. (1) Discovering intentional constraints: Without specific queries, the reasoning model can fail to identify intentional constraints. For example, differentiating rhombus from general quadrilateral, as shown in Figure 13
(2) Handling ambiguity: Visual inputs sometimes provide useful inductive biases that can help the model better understand the task or make reasonable assumptions when the instructions are ambignous. For instance, when presenting an angle an image and asking whether is an acute or obtuse angle, as Figure $ it is visually straightforward assume that the angle defined by the middle point as the vertex with rays extending outwards. However, without such visual cues, reasoning over pure symbolic representations makes it challenging to infer which angle the question refers to among the detected undirected edges. To mitigate ambiguity, adding more precise instructions for VDLM-txt is necessary in some tasks.
(3) A hmic reasoning: Language-based reasoners can struggle with complex multi-step reasoning tasks, such as inferring the connectivity (Figure 13) of a maze using the vertices and edges the grid in pixel coordinates, or counting the number of objects located within certain box.
C Primal Visual Description (PVD) Details
PVD JSON schema definition:
See Figure 14
23

---

Published in Transactions on Machine Learning Research (05/2025)
Types
Schema
Example
"type": Moirrlen e",
"type"; "cirole",
"cenler": [x,
"cenLec": [205, 210],
"radius":
*radius": 117,
Circle
"color": [r, bl,
*color": [193, 190, 165), Tatyle"; *outlined shape",
"style"; *filled shape" "outlined shape",
"line_width" width"d d (if style ia "outlined")
"line
*line_wtt width":
"tyoe": "ellipse", "center": [x, yl,
TLype"; "ellipae",
"najor axis length"; 11,
"center"; [258, 168),
"minor axia axcg__lnnth";
"major length": 147,
Ellipsc
"minor ax1a longth"; "eegeeghh 60, "colaLion"; : 16,
"rotation";
"color": le, bl,
"color": [85, 220, 981,
"style"; *filled shape" or "outlined shape", "line_width": d (if style is "outlined")
"style": "Filled shape"
Rectangle
"type": "rectangle" "triangle" or "polygon",
"type"; "triangle", "vertices" [[452, 418), (298, 113), (266, 255]], tcolor": [165, 130, 141), "slyle": "Cilled shape",
"vertices": [[x1, yl,, [x2, y2), ...) "color": [e 9r bl,
Triangle
"style"; "filled shape" or "outlized shape", "line widch"; style "outlined")
Polygon
"sype": "line segment", "vertices": ([x1, yl), [x2, y2])
"type"; "line segment", "verticea": ([822, 114), (93, 20]1, "color"; [166, 32, 971, Sine_width": 10
Line Segment
"color"; Tr, bl.
"1 "line width":
"Lype"; "grid",
"vertices"s [[73, 214], [33, 640),
"veztices": " [xx,, yll, [x2, y21, "edges": [[x,, yil, (x2, 3211, .::;!/
"type": "grid",
*ddges": [[[73, 214], [73, 6d01l, [[215. 214), [215, 640]].
(215, 2141, [215, 6401),
Grid
"color": JE, gr bl.
"line "lino_widthe width":
[173, 640], [215, 6401l]
*color": [23, 120],
*line *line_widlhe widlh";
"type": "path", "verLices; [199, 691, [17, 330), (61,
" "Lyge": "palh",
"vertioes": [[x1, yl), [22, y2l,
"edges": [[[x, yll, (x2, y211, ..!!
Path
Tedges": [1[59, 691, [17, 3301), [[17, 2301, [61, 77111
"color"; lE, bl.
"line widch":
"color": 138, 0),
*line width":
"typa": "lire drawing*. "vertieee" [1399,
[433, 823]
(483, 570], [531, 4131, [534, 5781
"sype": *line drawing",
"vectices": [[xl, yll, [x2, y2l, "edges": [[x,, yil, (x2, y211. .::s) "color"; [r, bl.
"edgea": [[[393, 497], 1483, 570]1,
[1531, 4431, (534, 5761],
Graph
[1483, 570], (534, $36]1.
"line width":
[1483, 570], [433,
(1534, 578], (433, 823]]]
"olor" [254, 230, 139],
line width":
Figure 14: PVD JSON schema definition.
Generation procedures (Single Object):
Circle: Randomly sample center and radius to draw circle within the canvas.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Style
Concept
Instances
Circle
10K
Ellipse
10K
Rectangle
10K
Filled
Triangle
10K
Single Object
Polygon
20K
Outlined
Line Segment
10K
Grid
10K
Path
10K
Graph
10K
Cirole
5K
Rectangle
5K
Filled
Triangle
5K
Line Segment
5K
Composition
Circle
10K
Rectangle
10K
Outlined
Triaugle
10K
Line Segment
10K
Total
160K
Table 4: PVD 160K dataset statistics.
Ellipse: Randomly sample center, a major axis, and a minor axis, then randomly rotate by an angle. Verify if the ellipse is largely within the canvas; if not, try again.
Rectangle: Randomly sample top-left corner, a width, and a height, then randomly rotate by an angle. Verify if the rectangle is largely within the canvas; if not, try again.
Triangle: Randomly sample three points as vertices to draw a triangle. Check if the area is larger than a threshold; if not, try again.
Polygon: Randomly sample N E [5, 20] points. Order the points with respect to the centroid that no intersections will happen when connected with polyline. Draw polygon with the sampled points. Check if the polygon has an area larger than threshold; if not, try again.
Path: Randomly and iteratively sample N E [3, 16) points, connect the newly sampled point with the previous point form a line segment. Verify if the newly added line segment does not intersect with any of the previous line segments; if yes, resample the point.
Grid: Sample a grid of points with a size M x N where M, N E [2, First, use Depth First Search
(DFS) algorithm to connect all grid vertices into a connected graph. Then randomly add additional edges between adjacent vertices.
Randomly sample N E [4, 16] points. First, use Kruskal's algorithm (Kruskal 1956 to find a Minimum Spanning Tree that connects all the points. Then randomly add additional edges to the graph.
Graph:
Generation procedures (Composition): Tteratively draw shapes the canvas chosen from the following set of object types: ["circle", "rectangle", "triangle", "line segment"]. After the first shape is drawn, at each iteration, the later shapes are constrained to have the same color the previous shapes. We ensure overlap between the newly added shape and the previous shapes, while making sure that the intersection ratio does not exceed predefined threshold. This prevents cases where one shape entirely contains another, making it impossible to decode into individual Primal Visual Description elements.
PVD 160K datasct: Using the aforementioned generation procedure, we generate a largo-scale dataset con- taining 160K (SVG, PVD) pairs for training the LLMM-based SVG-to-PVD model. The detailed configuration be found in Table
25

---

Published in Transactions on Machine Learning Research (05/2025)
SSIM
DINOv2 Score
CLIP Score
w/o aug w/ aug
0.892
0,874 0.893
0.886
0.895
0.893
Table 5: Impact of the data augmentation (Gaussian Blur and Pixel Noise detailed in 2.2 on SVG-to-PVD model perception performance.
Data augmentation details: To enhance the robustness of the SVG-to-PVD model to images with various sizes and quality, we introduce the following randomized data augmentation during data generation.
Random pixel noise: Probability (how often to apply the augmentation): 0.1; Ratio range (what portion of the selected area will be filled with noise pixels): (0.01, 0.05); Intensity range (the intensity of the noise pixels): (0.1, 1.0); Dilate range (how many pixels will the selection area be extended from the boundary): (1, 3) in pixels; Noisc size: (1, 3) in pixels.
Gaussian blur: Probability (how often to apply the augmentation): 0.1; Radius: (0.1, 0.5).
Table shows the ablation study with and without the data augmentations.
Image-to-SVe SVG Visual Encoding Quality
VTracer Encoding Quality. As introduced in Section 2.1 the first step of our proposed perception
module to encode a raster image into an SVG representation using VTracer, rule-based converter that we found yields near-perfect reconstructions for vector-graphic style inputs. To verify this, we convert the VTracer-encoded SVGs back into PNG images and compare them with the originals. The results are presented
in Table 6
Task
SSIM
DINOv2 Score
CLIP Score
Augle Classification
0.990
0.991
0.997
Length Comparison
0.994
0.970
0.986
Shapeworld Spatial 20bj
0.997
0.984
0.995
Shapeworld Spatial mObj
0.991
0.984
0.988
Shapeworld Superlative
0.991
0.982
0.987
NLVR
0.994
0.989
0.993
Geoclidean
0.648
0.994
0.995
Maze 2x2
0.998
0.906
0.775
Maze 3x3
0.998
0.998
0.959
ALL
0.956
0.978
0.967
Table 6: VTracer SVG encoding reconstruction metrics. The exceptionally low SSIM score Geoclidean is due to the fact that most pixels are transparent. Since compute SSTM only on non-transparent regions, the score becomes more sensitive to slight pixel differences. However, the reconstruction appears visually near-perfect across all tasks, as reflected by the high DINOv2 score.
We observe near- near-perfect reconstruction quality from VTracer. For reference, the average reconstruction quality from the PVD representation (after SVG-to-PVD conversion) is SSTM 0.895, DINOv2 Score 0.880, and CLIP Score 0.893, indicating that most perception errors arise during the SVG-to-PVD step. However, as discussed in Section 2.2, although SVG preserves all low-level features, it is extremely verbose and noisy making unintelligible to LLMs and LMMs. This observation motivates the development of the intermodiate PVD representation.
Impact of VTracor Error the End-task Performance. To further investigate whether imperfoctions in SVG encoding significantly affect end-task performance. We sort the instances for each task with respect to VTracer reconstruction quality, using the aggregated metric: SSIM DINOv2 score + CLIP score. The
26

---

Published in Transactions on Machine Learning Research (05/2025)
VTracer Reconstruction Quality vs. End-Task Performanec
Low Quality
Medium Quality
High Quality
Task Performance (%)
80
60
40
20
Angle Classffication
Length Comparison
Shapeworld Spatial 20bj
Shapeworld Spatial mOhj
Shapeworld Superlative
NLVR
Geoelidean
Maze 2x2
Maze 3x3
ALL
Tasks
Figure 15: Impact of SVG encoding quality end-task performance.
instances then grouped into three bins corresponding to low, medium, and high VTracer quality, We examine the end-task performance (VDLM-mm + GPT-40) of these groups to investigate whether performance is sensitive to VTracer errors. In Figure we show the average accuracy on each task for each bin. Note that each bin has about 33 instances.
Overall, we observe that end-task performance fairly robust vectorization errors, while also exhibiting a trend similar what we reported in Section i.c., positive correlation between perception quality and end-task performance.
E
Additional Ablations on PVD Perception
PVD Model
SSIM
DINOv2 Score
CLIP Score
SVG-to-PVD (Mistral-7B)
0.895 0.889
0,880
0.893 0.880
SVG-to-PVD (Qwen2.5-7B)
0l876
Table 7: Perception metries of different SVG-to-PVD models.
PVD
SW-S 20bj
SW-S mObj
SW Sup
Maze
Maze
Reasoner
AC
LC
NLVR
Geo
All
Model
2x2
3x3
GPT-4o
0.63
0.57
0.97
0.82
0.92
0.81
0.71
0.46
0.08
0.663
GPT-40
Mistral-7B
0.90
0.95
0.91
0.86
0.71
0.769
GPT-4o
Qwen2.5-7B
0.82
0.98
0.99
0.79
0.89
0.83
0.68
0.70
0.29
0.774
Table 8: End-task performance with different. SVG-to-PVD models.
Different LLM choices for SVG-to-PVD model. We investigate the effect of using different LLM for SVG-to-PVD translation. Specifically, train Qwen-2.5-71B base model on the PVD-160k dataset using the same number of epochs our original model. We observe comparable perception and end-task performance to our original PVD model, which is based on Mistral-7B. Table and Table g report the perception scores and end-task performance of SVG-to-PVD models trained with different LLM backbones. These results suggest that the SVG-to-PVD translation process robust to the choice of LLM.
http:://buuginnssssccc////ee/e//
27

---

Published in Transactions on Machine Learning Research (05/2025)
PVD Model
Training Epochs
Final Loss 4
SVG-ta-PVD (Qwen2.5-7BB
3 epochs
0.052
SVG-to-PVD (Mistral-7B)
3 epochs
0.051
PNG-to-PVD (Qwen2.5-VLL VL-7B)
3 epochs
0.243
Table Comparison of training loss between the PNG-1o-PVD and SVG-Io-PVD models.
PVD Model
SSIM t
DINOv2 Score
CLIP Score f
SVG-to-PVD (Mistral-7B)
0.895
0.880
0.893
SVG-to-PVD (Qwen2.5-7B)
0.889
0.876
0.880
PNG-ta-PVD (Qwen2.5-V1L-7) VL.-7B)
0.262
0.320
0.385
Table 10: Comparison of perception metrics between the PNG-to-PVD and SVG-to-PVD models.
Ablation with PNG-to-PVD model. To further validate the design choice of using SVG for initial visual encoding, we compare it with directly training PNG-to-PVD model using a large multimodal model. Specifically, we train -.v on the same PVD-160k dataset, where the inputs are png images and the target outputs are PVD strings. Table and Table [O] the comparison between the PNG-to-PVD and SVG-to-PVD models terms of final training loss and perception metrics. Using identical hyperparametters, find that directly translating PNG PVD is significantly less effective, as evidenced by much higher loss and substantially lower perception performance. We observe that the PNG-to-PVD model rarely produces valid PVD JSON outputs and yields significantly worse reconstruction performance (entirely not usable for downstream reasoning).
F
Additional Experiments Using Open-Source LMMs as Reasoners
PVD Model
SW-S
SW-S mObj
SW Sup
Maze
Maze
Reasoner
AC
LC
NLVR
Geo
All
20bj
2x2
3x3
Qwen2.5-VL-72B
0.78
0.75
0.98
0.75
0.92
0.72
0.67
0.64
0.15
0.707
Qwen2.5- VL-72B Mistral-7B
0.59
0.96
0.96
0.78
0.89
0.69
0.73
0.66
0.28
0.727
Table 11: End-task performance with open-source LMM reasoners.
We added new experimental results demonstrating that VDLM can also work effectively with recent open- source LMMs. Specifically, we use 2.wn-2255VVL-721"" as the reasoner to compare performance with and without the inclusion of PVD representations. Across 9 tasks, we observe a 2% overall improvement.
G Additional Qualitative Examples on PVD Parsing Novel Concepts
We highlight that our PVD ontology constitutes minimal but powerful set of primitives for exprossing shapes in vector graphics. We demonstrate that the current PVD model shows promise in approximating novel shapes through composition. In Figure 16 we show several examples of the PVD model parsing novel concopts such as "star," "cross," and "circle segment."
Further Exploration in Prompt Engineering
Observing that the PVD representation can sometimes be inaccurate and may degrade performance (i.e., on ShapeWorld tasks), we explore introducing verification step the prompt. Specifically, we instruct the model to first "double-check whether the objects in the PVD perception match the objects in the image." We find that this step can enhance performance on tasks where the initial image-only bascline is already reasonably strong. Table 12 shows the performance comparison on World tasks with the same LMM reasoner and PVD representation.
httper//aagggngsseeeeee///////ep Instruct
28

---

Published in Transactions on Machine Learning Research (05/2025)
Novel Input Shape
PVD Perception Visualization
PVD Perception
[["type": "triangle", "vertices"; 262, 2361, [222, 236], [253, 259]], "color"; [255, 7, 7}, "style": "filled shape"), ("type": "triangle", "vertices", [1242, 221], [231, 2591, [250, 23611, "color": [255, 7, 71, "style": "filled shape")]
Star
[["type": "zectangle", "vertices"; [1273.0, 279.01, [311.0, 246.0 [321.0, 260.01, 1283.0, 293.011, "cclor": [69, 69, 250],
"style": "filled shape";,
("type": "rectangle" "vertices", [,275.0, 256.0l,
[287.0, 243.0 1320.0, 282.0], (308.0, 293.011,
"color": [59, 69, 250), "style": "Fil ed shape";]
Cross
[["type": "circle", "center": [364, 544], "radius": 128, "cclor": [0, 0, 0], "line width": 4, "style": "outlIned shape"), h, ("type"; "line segment", "vertices": [[489, 540], [326, 663]1, "color": : [0, 0, 01, "line width": 4]]
Cirele Segment
Figure 16: Qualitative examples on PVD model approximating novel concepts.
SW-S
SW-S
SW
Prompt Version
Reasoner
PVD Model
20bj
mObj
Sup
GPT-4o
0.97
0.81
0.92
Original
GPT-4o
Mistral-7B
0.91
0.82
0.82
New
GPT-4o
Mistral-7B
0.98
0.81
0.88
Table 12: ShapeWorke task performance with new prompt.
Full Response of the Example in Figure
See Figure 17 for the full input prompt and the generated response from GPT-4 on the 2x2 maze-solving task shown in Figure
J Task Prompts
Figure 18 shows the prompts for models with only image representations as visual inputs.
Figures 19 27 show the prompts for VDLM, where (perception) will be filled with the aggregated Primal
Visual Description perception result, and the orange text are instanco-specific inputs such the question.
For VDLM-mm, the original image input will be preserved and feed to the LMM reasoner along with the
29

---

Published in Transactions on Machine Learning Research (05/2025)
filled prompt. Since the reasoning in VDLM-txt is based solely the PVD representation which is purely textual, task instructions that assume visual inputs can become ambignous. For example, in the task Angle Classification, it is unclear which angle the question is referring to if we are only given the coordinates of two undirected edges. Therefore, we design task-specific prompts that remove such ambiguity. Another noteworthy point that, in contrast to visual inputs that naturally accommodate a degree of imprecision, symbolic representations lack such inherent leniency. For instance, even if two line segments differ by only one pixel in length, they might be considered identical in visual representations, but symbolie representations would likely identify them different. To reintroduce level of tolerance in tasks that involve arithmetic reasoning, such as length comparison, we incorporate task-specific instructions to account for reasonable margin of error, like 5%.
K
Newly Constructed Downstream Task Datasets
Angle Classification. We use the Geoclidean data generator to generate images containing a single acute or obtuse angle with randomized orientations and ray lengths. The domain-specific language for generating the two concepts are shown as follows:
Acute Angle:
"11- line(p1(), p2()",
"c1" cprcle(p1)), p20)",
"c2- circle(p2(), p1O)", "12* line c2), p4(c1, c2))",
"14 1pne(p5 12), p7(11))"
"15 = 1p6)(1)))),, p7(11))"
Obtuse Angle:
"11* line(p1 p2()",
"c1* circle(p1(), p20)",
"c2* circle(p2(), p1()",
"12* 1ine(p c2), p4(c1, c2))",
"13* 1ine(p 12), p6(12))",
"14* 1ine(p5(11, 12), p7(11))",
"15* line(p)))2), p7(11))",
"16- 1ine(p8(13, 14), p9(15))",
"1100* line(p c2), p10(16))",
"c101* circle(p5(c1, c2), p10(16))",
"c102* circle(p10(16), p5(c1, c2))",
"1101* (p10011(0111 c102),
p101 (c101, c102))"
"17 (p11 1101), p6(12))",
"18 = 1ine 101), p7(1)))
Length Comparison. We use matplouit plot two non-intersecting line segments on a canvas. These line segments may either be of identical length of differing lengths. scenarios where the lengths vary, we ensure the discrepancy substantial (exceeding 15% relative the length of the shorter line segment) to ensure perceptibility, The orientation of each line segment is determined independently and randomly, being either horizontal or vertical.
1f /attpe///etaaa.coo////////////ap I httppr//aatpleelib.orr/tttattt
30

---

Published in Transactions on Machine Learning Research (05/2025)
Maze Solving.
We leverage the maze-dataset package to generate 2D unsolved mazes along with their corresponding ground truth solutions. We use "circle" shape to denote the start position and "star' shape to denote the end position. We generate two subsets featuring 2x2 and 3x3 maze configurations.
L
Dataset Statistics
Training Instances
# Eval Instances
Line or Angle
10K
1K
Angle Classification
10K
1000
Length Comparison
10K
1000
Probing Tasks
Clevr QA
36K
1000
Shapeworld Scene
15K
100
Maze Scene
10K
600
Angle Classification
100
Length Comparison
100
Shapeworld Spatial Reasoning (2Obj)
100
Shapeworld Spatial Reasoning (MultiObj)
100
Zero-Shot
Shapeworld Superlative
100
Downstream Tasks
NLVR
100
Geoclidean 2-shot Learning
100
2x2 Maze Solving
100
3x3 Maze Solving
100
VGBench-QAA Category
100
VGBench- QA Color
100
VGBench-QA Usage
100
Table 13: Statistics of the probing tasks ($ A.1 and the downstream tasks ($ $3 The GPT-4(V) zero-shot
results on probing lasks are reported on 100 randomly sub-sampled instances from the entire eval split.
Detailed statistics of the probing tasks used in $ A and the zero-shot downstream tasks mentioned in can
found Table
31

---

Published in Transactions on Machine Learning Research (05/2025)
Input Prompt
The following JSON cuntains approximated perception o 2x2 mace. Each object (puleniaalyy including the background) represented list of geonetric shopes. object contains multiple shapes, composite object, The (K: y) coordinates for the vertices and edges correspond the width height position in pixels, and (0, 0) is the top-leil corner of the image.
perception
Pobject L''ype: 'rectangle, "vertices:: 110.0, 336.01, 10.0, G.LL (3360), O.UJ, 1336.0, 336.0) 3]6. 'color:: [253, 253, 2531, style: 'Glledd shape') Vojject 1% (yype: 'grid,
'verices': [[16, 16), [16, 1643, [16, 312), (172, 16), (172, 164), [172, 312], [328, 166, [328, (328, 312]], 'edges": [[[16, 16), [16, 164]], [[16, 16), (172, 16]. [[16, 164). 116, 3121J, (176, 1641,
164]). 11172, 1328, Téll. 1116, 1172, 31211, 11328,
1328, 164 I1172, 3121, 1328, 312(], [1328, 1646, 31211I, 'ollo' 10, 0).
'limo widhh: 10:1. 'objest 2% [Plype: 'cincle", cennter: (94, 94), 'radiuu:: 14, 'celor': [255, 4, 4). Ssyle: "iilled shape') J. lojject 3: ([lpe: 'rrnnglee Veerticss [[262, 236].
(222, 236), (253, 259]], toolo: [255, J]. 'style: 'iilled shape"). l'tpe': 'riangle', 'vertieest; [[42, 2213, [231, 259). (250, 256]]. 'color: (255, 7, 'syle": 'illed shape')])
2x2 mare, walls depicted by geid black hetween cells and along the perimeter. The maze cells defined within the grid. start positien marked by rod circle, and the end position by red star. The pusition of each cell can be representedd by coordinaties frow, column), beginning from the lop-lell comee as (0, 0). Here, "row" corresoonds
verticll dimension (height) the image, 'oolume' the horizontal dimension (width).
Perform the fallowing steps solve the maze:
(1) Infer the connectivity of the cells using a connection list. For example, a 2x2 maze should have a 'counection, list containing two sublisits with dimension 1x2 and 2xl.. For in mingt00, 1) and rangelO, 23, connection [ist[U] HUl is 'True' ieelll "G,3 vertically connected to cell (i+ without being seperated by wall. Similarly, forj in rang((, and range(0,, 2), cconecction [[[D] "Tree'
'i,,)" horizontally connected to (i. (i,j without being seperated by a wall.
(2) Infer the start position and and position of the maze the row-column format. by finding path You MUST provide your final answer, and the answer should follow this format: ("solution": "a list of (row, column) coordinates").
start position
position.
GPT-4 Respouse
solve the maze based on provided JSON data, let's break down stops as required:
HiB Step I: Infer Connectivity of Cells
Given the grid lines and their coordinates, we can infer the connectivity between cells, The grid lines form the boundaries of 2x2 maze, meaning there are vertical lines and 3 horizontal lines that define the cells. By examining the edges the JSON, we ean determine where the walls are and thus infer which cells are connected.
Vertical connections ( ('comncctinn Jixx[0]) y: There are no vertical walls inside the maze, so both cells the first column are connected to their respective cells in the second column,
Horizontal connections ('comncction Not( I"): There horizontal wall hetween the two cells in the first row, meaning there's no direct connection between them.
However, the wells in the second row are connocted.
Therefore, the connection list' can be represented as:
Vertical connections: [[True, True]]
Hortrontal connections: [[False], [Tree]]
MHV Step 2: Infer the Start and End Positions
The start position marked red circle, which is located at approsimately '[94, 94). This places the top-left cell, '(0, 0)', The end posiliinn is marked composite shape resembling star, located the bottom--rgght cell, which inferred from the coordinates of the triangles forming the star shape. This places the position
M# Stop 3: Solve the Maze
the connectivity and the start/end positions, we can path from the the The puth, considering the connectivity and the positions, would be: Start 0):
2. Mave right 1Y' Mave down (1, 1y:
3.
Thus, the solution path is: 1(0, 0). (0, 1), DJ,
HW Final Answer
"json
("solution:: " "(00. 0). (0. D), 1)")
Figure 17: Full input prompt and GPT-4 response of the 2x2 maze solving example in Figure 2 The blue part in the input prompt indicates the generated Primal Visual Description (PVD) of the entire image.
32

---

Published in Transactions on Machine Learning Research (05/2025)
scene containing objects with
following attributes:
-- Attribute Ontology -..
Is this an acute ungle i or an obtuse ingle? Answer with 'Acute" or "Obtuse".
Are these two lines of equal lengt?? Choose "Yes" or "No".
color; ('red], 'greea', 'bluee, 'yellow', 'magental,
'eyan', 'gray]]
shape: ('square', 'rectangle,, 'trinngle, bentagon',
'oross', 'cirele', 'semicirole', 'ellipse1
Acute-or-Obtuse
Length-Comparison
Think stop by stop and answer the following. question: (question)
Given an image containing three boxes with light grey background, horizuntally layed sepernted by two dark grey rectangles placed vertically, which are referred to as walls. An "odge" is referred odge" is referred to as the bottom boundary. an object's boundary is located very close to edge or wall (e.g. within 10 pisels),
The top part of the seene provides (n shot) reference examples of Euclideun geometry concept Faelidean geometry concept consists of composition of one or more primitive goometric shapes, such circles and line segments, with constraints, These constraints include but are not limited lengths, angles, and spotial relationships between the primitive shapes, for example, two perpandicular line segments or an equilareral triangle, etc.
MUST provide your final answer, the answer should follow this format: ("answer": "yes" or "no")]
boxes
boundary of the image. A "base
Shapeworld
Given inlx [n] maze, where the walls depicted by a grid of black lines between cells and along perimeter.
considered "touching" Each box contains set of shapes. There are types l'iaages: "Tower', "Scatter", In "Tower" images, euch box contains only squares stacked towers with up to squares. In "Scatter" images, each box contuins scattered objects of different sizes and shapes.
defined within the grid. The position marked red cirele, and the end position by red star. The position cach cell coordinates (row, column)) beginting from the top- left corner as (0, D). Here, 'row' corresponds the vertical dinccasion (beightt) of image, and 'column' to the horizontal dimension (width).
maze
The bottom purt of the scene presents test esample, separated from the top part by red borizontal line.
represented by
First, identify the pattern and constraints of the reference and test concepts.
Think step by step answer the following question: [question)
Then, determine the test example depicts the same concept as the reference examples.
Solve the maze by finding path from the start position to the end position.
You MUST provide your final answer, and the answer should follow this format: {I (lanswer": "yos" "
You MUST provide your final answer, and the answer shoold follow this format: {f ("answer": "yes" or "no")).
You MUST provide your final answer, and the answer should follow this format: (I [""oolutinn: "a list of (row, columnn) coordinates") I].
Geoclidcan-2Shhr
Maze-Solve
Given an image containing a SVG graphic, step by step answer the following question: [question)
[options]
You MUST provide your final answer, and the answer should follow this format: {( (("nswer": choose from "A", "B", "D"))
VGBench QA
Figure 18: Prompts for zero-shot downstream tasks with image input.
The following JSON contains an upproximated perception of 2d scene. Each object (potentially including the background) is represented by list of geometric shapes. If the object contain multiple shapes, composite object. The (x. y) coordinates are in pixels, and (0. 0) is the top-left cornrr of the image.
perception (perception)
Answer the following question: this acute angle obtuse ungle?
Note that if the pereeption result includes a path, the angle is defined with the midpoint as the vertex, and the rays extend from the midpoint to both the head and end points. For example, a path from A to B to C is perceived, the angle is defined as the angle between the vector BA and BC. You MUST provide your final answer, and the unswer should follow this format: ( ("answer": "acute" or "obtuse"]: 1).
Figure 19: Prompt for task Angle Classification with Primal Visual Description perception input.
33

---

Published in Transactions on Machine Learning Research (05/2025)
The following JSON contains an upproximated pereeption of 2d scene. Each object (potentially including the background) is represented by list of geometric shapes. object contain multiple shapes, composite object. The y) coordinates pixels, and top-left of the image.
perception
(perception]
Answer the following question: Are these two lines equal length?
Note that perception can be Doisy, A 5% offsct in the measurements is acceptable. You MUST provide your final answer, and the answer should follow this format: {I anaswee": "yes" or "no"]]
Figure 20: Prompt for task Length Comparison with Primal Visual Description perception input.
Given 2d scene containing objects with the following attributes:
-- Attribute Ontology --
color: ['red', 'green', 'blue', 'yellow', 'magenta', 'syan', 'gray"] shhpe: l'square', 'rectangle', 'triangle', "pentagon", 'cross', 'circle', 'semicircte', 'ellipse']
The following JSON contains approximated pereeption of the scene. Each object (potentially including the background) is represented by list of geometric shapes. If the object contain multiple shapes, composite object. The (x. y) coordinates are in pixels, and (0, 0) is the top-left corner of the image.
.- perception- iperception)
Note that the perception can be noisy. First identify the best matching shape type and the color type from the ontology for each perceived object For composite objects, please match the entire composition to one of the most probuble objects the ontology. Make educated guesses necessary. Then, think stop by step and answer following question:
(question]
You MUST provide your final answer, and the answer should follow this format: (("answer:: "yes" "no")]
Figure 21: Prompt for task Shapeworld Spatial Reasoning (2Obj) with Primal Visual Description perception input.
Given 2d scene containing objects with the following attributes:
--- Attribute Ontology
color: ['red,, 'green', 'blue', 'yellow', 'magenta', 'syan', 'gray]]
shape: ['square', 'rctangle', 'triangle,, "pentagon', 'cross', 'circle', 'semicincle', 'ellipse']
The following ISON contains approximated perception of the scene. Each object (potentially including the background) is represented by a list of geometric shapes. If the object contain multiple shapes, composite object. The (x. y) coordinates are in pixels, and (0, 0) is the top-left corner of the image. If two objects overlap, the one with the larger index is considered to be front the other.
perception (perception)
Note the perception can be noisy. First identify the best matching shape type and the color type from the antology for each perceived object. For composite objects, please match the entire composition of the most probable objects ontology. educated guesses necessary, Then, think step step and answer following question:
(question)
You MUST provide your final answer, and the answer should follow this format: (( (f"answer": "yes" or "no")3
Figure 22: Prompt for task Shapeworld Spatial Reasoning (MultiObj) with Primal Visual Description
perception input.
34

---

Published in Transactions on Machine Learning Research (05/2025)
Given a 2d scene containing objects with the following attributes:
--- Attribute Ontology
color: ['red', 'green', 'blue', 'yellow', 'magenta', 'syan', 'gray"] shape: ['square', 'rectangle', 'triangle', "pentagon" 'cross', 'circle', 'semicircle, 'ellipse']
The following JSON contains an upproximated perception of the scene. Each object (potentially including the background) is represented by a list of geometric shapes. If the object contain multiple shapes, composite object. The (x. y) coordinates are in pixels, and (0. 0) is the top-left corner of the image. The lowermost object has the lurgest y-coordinate, and rightmost object has the largest x-coordinate.
perception (perception]
Note the perception can be noisy. First identify the best matching shape type the color type from the ontelogy for each perceived object. For composite objects, plense match the entire composition of the most probuble objects the ontology. Make educated guesses nevessary, Then, think step step and answer following question:
(question]
You MUST provide your final answer, and the unswer should follow this format: (1 (!"answer": "yes" or "no")]3
Figure 23: Prompt for task Shapeworld Superlative with Primal Visual Description perception input.
Given an image containing three boxes with light grey background, horizontally laid out. boxes are separated by two dark grey rectangles placed vertically, which referred as walls. An "odge" is referred to the boundary the image. "base edge" is referred the bottom boundary, object's boundary located very close edge or wall (e.g. within 10 pixels), considered as "touching". Each box contains set of shapes, There are two types of images: "Tower", and "Scatter". In "Tower" images, each box contains only squares stacked in towers with to 4 squares. In "Scatter" images, euch box contains scattered objects of different sizes and shapes.
The following JSON contains an approximated perception image. Each object (potentially including the background) is represented by geometric shupes. If the object contain multiple shapes, composite object. The (x, y) coordinates are pixels, and (0, 0) is top-left corner of the image.
perception iperception]
Now, identify content in each box bused on the perception result, and then think step by step answer the following question: (question]
You MUST provide your final answer, and the answer should follow this format: ("answer"; "yes" or "no")]
Figure 24: Prompt for task NLVR with Primal Visual Description perception input.
The following JSON contains an approximated perception image. Each object (potentially including the background) represented by geometric shapes. If the object contain multiple shapes, composite object. The y) coordinates are in pixels, and (0. the top-lefi corner of the image.
perception iperception)
The top part of scene provides (a. shot) reference examples Euclidean geometry concept. Euclidean geometry concept consists of composition of onne limited to lengths, angles, and spatial relationships between the primitive shapes, for example, two perpendicular line segments or equilateral triangle. etc.
primitive geometric shupes, such circles and line segments, with construints. These construints include but not
The bottom part of the scene presents a test example, separated from the top part by a red borizontal line. First, identify the pattern und constraints of the reference and test concepts based on the perception result, Note that the perception can be noisy. Make educated guesses if necessary.
Then, determine the test example depicts same concept as the reference examples.
You MUST provide your final answer, and the answer should follow this format: (("anwerr: "yes" or "no")].
Figure 25: Prompt for task Geoclidean 2-shot Learning with Primal Visual Description perception input.
35

---

Published in Transactions on Machine Learning Research (05/2025)
The following JSON contains approximated pereeption a [n]x(n) Fach object (potentially including the background) represented list geometric shapes. If the object contains multiple shapes, it is composite object. The coordinates for the vertices and edges correspond to the width and height position in pixels, and (0, 0) is the lop-left corner of the image.
.-. perception iperception]
In the [u]x (n] maze, walls depicted by grid of black lines between cells and along the perimeter. The maze cells are defined within the grid. The start position is marked by red circle, and the end position by red star. The position of cach cell can be represented by coordinates (row, column), beginning from the top-left corner as (0, 0). Here, 'row corresponds to the vertical dimension (height) of the image, and 'column' to the borizontal dimension (width).
Perform the following steps solve the maze:
(1) Infer the connectivity of the cells using connection list. For example, [n]x(n] should have "conecction list containing sublisits with dimension x/]] and [m)] For in runge(O, (m]) and j in runge(O, (n]), "conncction list[O][i]li] is "True' if cell '6. is vertically connected to cell (i++l, without being seperated by a wall. Similarly, for in range(0, [m]) and in range(O, [n]), connection [is[]]][] is "True' if'ell (i.j) is horizontally connected cell 'i. j+1) without being seperated by wall.
(2) Infer start position end position the maze the row-column format.
(3) the maze by finding path from You MUST provide your final answer, and the answer should follow this format: [ ("solution": "a list of (row, column) coordinates")
position the end position.
Figure 26: Prompt for task Maze Solving with Primal Visual Description perception input.
Given image containing SVG graphic, think by step and answer the following question: [question]
[options]
The following JSON contains approximated reference perception of the image. Each object (potentially including background) represented by list of geometric shapes. If the object contain multiple shapes, composite object. The coordinates are in pixels. and (0. 0) is the top-left corner of the image.
reference perception [perception]
Note that the reference perception can be noisy. Refer to the reference perception when necessary for answering the question.
You MUST provide your final answer, and the answer should follow this format: [ [["anwwer": choose from "A", "B", "C", "D"))
Figure 27: Prompt for VGBench-Q2 QA tasks with Primal Visual Description perception input.
36


Reviews:
[{"review_id": "CtY84YSkPW", "reviewer": "Reviewer_GtF7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- First work to propose an intermediate symbolic representation (PVD) for low-level visual reasoning in LMMs, addressing a clear gap in the field.\n- Strong empirical results on carefully constructed benchmarks.\n- The modular design (SVG → PVD → reasoning) enables transparent error analysis (e.g., perception vs. reasoning failures).\n- PVD is task-agnostic and compatible with off-the-shelf LMMs (e.g., GPT-4).\n\nWeaknesses\n- VDLM has limited generalization capability, as PVD’s ontology is restricted to 9 primitives, which may cause failures on novel shapes or complex constraints.\n- VDLM depends on strong reasoning LMMs as depicted in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses a limitation of current MLLMs that they often struggle to bridge low-level visual details (shapes, sizes, layouts…) with high-level reasoning (semantics, logic…). The authors propose a method Visually Descriptive Language Model (VDLM), which introduces an intermediate symbolic representation called Primal Visual Description (PVD). The result outperforms SOTA LMMs on low-level reasoning tasks and maintains performance on high-level semantic tasks (e.g. VGBench-QA).\n\nMore technically, VDLM consists of:\n- A rule-based visual encoder converting images to SVG.\n- A fine-tuned LLM for SVG-to-PVD conversion, which abstracts SVG paths into structured primitives (e.g., circles, rectangles).\n-  An inference-only LLM that uses PVD for downstream tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "[Major] The ablation of SVG-to-PVD translation is limited, which is a critical component in VDLM.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YVUAt45KLO", "reviewer": "Reviewer_6KFk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The paper addresses a known challenge in visual AI by enhancing LLMs' capability to understand and analyze low-level image details.\n- The methodology is elegantly straightforward, making the approach accessible and potentially adaptable.\n- The paper's clear writing and logical structure facilitate easy comprehension of complex concepts.\n\n### Weaknesses:\n\n- Limited scope: The proposed PVD framework struggles with highly complex visual scenarios. The authors should have explored and discussed more general and complicated graphic reasoning tasks.\n\n- Performance limitations: Table 1 reveals significant constraints in pure text-based reasoning. The method only achieves substantial improvement when combined with original image features, yet the paper inadequately discusses the underlying reasons for this limitation.\n\n- Limited transformation pipeline: The two-stage transformation process (PNG → SVG → PVD) creates optimization challenges and prevents end-to-end enhancement. This design choice lacks sufficient justification through empirical comparisons or experimental evidence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel approach to enhance LLMs visual graph understanding by transforming visual graphs into a structured textual format called Primal Visual Description (PVD). The PVD framework converts visual information into a primitive-based ontology that allows LLMs to leverage their text-based reasoning capabilities while preserving the original visual features. The methodology was evaluated across several low-level visual graph reasoning tasks, demonstrating significant improvements compared to baseline LLMs performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Suggested Changes:\n\n- Suggest that the authors test their approach on standardized complex visual reasoning benchmarks.\n\n- Recommend extending PVD to incorporate hierarchical representations (other than simple combination) that can better capture complex visual relationships\n\n- Recommend exploring different prompt engineering techniques that might enhance the text-only performance\n\n- Propose analyzing the error patterns to identify systematic weaknesses in the text representation\n\n- Suggest to try experimenting with direct PNG-to-PVD transformation using computer vision techniques", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0vyM7573jW", "reviewer": "Reviewer_KNeD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The performance improvements mainly come from converting a visual input to a symbolic, textual representation that can be directly leveraged by a powerful LLM, which bridges the gap between pixel data and language reasoning. This transformation into text is particularly effective in the domain of vector graphics because vector graphics inherently lose less visual information during the conversion compared to natural images.\n- VDLM is quite scalable and modular: VDLM is capable of zero-shot inference across diverse downstream tasks and requires no training except for the SVG-to-PVD translator. Furthermore, each stage of the pipeline is independent, allowing the SVG extractor, PVD translator and reasoner to be improved individually.\n- Disentangled perception and reasoning improves model's reasoning interpretability\n- Overall, the paper is well-written and easy to read: Figure 1 and 2 show the difference of VDLM and a traditional LMM pipeline well. The inclusion of full prompts in the appendix makes the work reproducible. The detailed error analysis in Section 4 and Section B provides a clear understanding of the limitations of the proposed model.\n\nWeaknesses\n\nEven though the authors emphasize that this work is a proof of concept and the first attempt to convert vector graphics into textual descriptions, I believe that the proposed method suffers from severely limited applicability. I think this stems primarily from the following three factors:\n1. The PVD ontology is overly simplistic and fails to cover the wide range of real-world use cases.\n2. Without access to very powerful text-only or multimodal reasoners such as GPT-4 or GPT-4o, it is difficult to expect meaningful performance gains from PVD.\n3. To enable precise visual perception, the authors focus solely on effectively translating visual inputs into textual descriptions, without making any effort to improve the perception capabilities of the visual encoder itself. This is particularly limiting for domains where VDLM might underperform, such as natural images and 3d-rendered objects.\n\nFurthermore, the proposed model is overly reliant on external tools, including a raster-to-SVG algorithm (VTracer) and a very powerful and proprietary LLM or LMM. The authors should have demonstrated the potential for performance improvement using open-source and relatively smaller LMMs or LLMs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the well-known issue of large multimodal models (LMMs), that is, relatively poor performance on tasks requiring low-level visual perception. The authors propose a novel framework called Visually Descriptive Language Model (VDLM) that converts an input image into a textual description of its primitives and uses a LMM or large language model (LLM) to reason over this description. The proposed pipeline consists of three stages: (1) use a rule-based vectorization (VTracer) to extract SVG code, (2) use a learned model to translate the SVG code into a structured text description called Primal Visual Description (PVD), and (3) feed this PVD into a LMM or LLM reasoner to answer questions. VDLM shows substantial performance improvements over using the backbone LLM or LMM alone on nine low-level visual reasoning tasks while achieving comparable or slightly better performance on high-level reasoning tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the aforementioned weaknesses, as well as the following questions:\n\nThe proposed pipeline first uses VTracer to convert images into SVG code.\n1. Is the pipeline's success largely gated by the quality of this vectorization, or is it forgiving to vectorization errors?\n2. Did the authors observe failure cases for this stage? If VTracer produces an imperfect or noisy SVG, how much does the learned SVG-to-PVD model denoise it?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In accordance with the points above, and the comments from the reviews, I've recommended a minor revision to make sure the limitations are more clearly communicated throughout.
Decision: Accept (poster)