Visually Descriptive Language Model for Vector Graphics Reasoning

Authors: Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji
Primary area: foundation or frontier models, including LLMs
Keywords: Large Multimodal Model, Large Language Model, Vector Graphics, Low-level Perception, Low-level Visual Reasoning

Abstract:
Despite significant advancements, current large multimodal models (LMMs) struggle to bridge the gap between low-level visual perception—focusing on shapes, sizes and layouts—and high-level language reasoning involving semantics, events and logic. This limitation becomes evident in tasks requiring precise visual perception, such as comparing geometric properties or solving visual algorithmic reasoning problems. To study this failure mode, we focus on an important visual domain: vector graphics—images composed purely of 2D objects and shapes, which are prevalent in various LMM-based agent tasks in web, visual design, and OS environments. We identify two key research questions: how can we enable precise visual perception, and how can we facilitate high-level reasoning based on such low-level perceptions? To accurately capture low-level visual details, we utilize Scalable Vector Graphics (SVG) for precise encoding of visual scenes. However, SVGs are not readily interpretable by LLMs or LMMs in a zero-shot manner. To address this challenge, we propose the Visually Descriptive Language Model (VDLM), which introduces an intermediate textual representation called Primal Visual Description (PVD). PVD translates SVGs into a text-based abstraction comprising primitive attributes (e.g., shape, position, measurement) along with their corresponding values. PVD can be learned with task-agnostic synthesized data and represents visual primitives that are universal across various vector graphics. This abstraction is more structured, allowing for direct interpretation by foundation models for zero-shot generalization to different reasoning tasks. Without any human-annotated data, empirical results demonstrate that VDLM leads to significant improvements in state-of-the-art LMMs, such as GPT-4o, across various low-level multimodal perception and reasoning tasks on vector graphics. Additionally, we provide extensive analyses of VDLM’s performance, showing that our framework offers improved interpretability due to its disentangled perception and reasoning processes. Finally, we demonstrate the promise of this representation by showing a positive correlation between the quality of the PVD perception and the end-task performance.

Decision: reject
Reviewer scores: 5, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730224008950

thread_type:
official_review

<thread_object>
creator: reviewer_hw2c

<content>
summary:
This paper proposes a method called VDLM, designed to improve large vision-language models capabilities to understand and reason about fine-grained visual concepts, specifically those well-suited to vector graphic representations. VDLM first converts an input image into a SVG format using a rule-based encoder. This SVG is then preprocessed and translated by a fine-tuned language model (e.g., Mistral-7b) into a structured JSON-like format called "Primal Visual Description" (PVD), which serves as an intermediate representation. Finally, this PVD is input into large vision-language models (e.g., GPT-4). The authors demonstrate that this pipeline improves performance of the base model (e.g. GPT4o) vs using a raw image or a raw/unprocessed SVG encoding.

strengths:
* The paper focuses on an area of significant impact: improving visual reasoning/understanding in large vision-language models.

* The writing is clear and well-structured, making the methodology easy to follow.

* The proposed method achieves notable improvements in several evaluated tasks.

weaknesses:
In my view, the main weakness of the paper is that, while it aims to "enable precise  visual perception and facilitate high level reasoning", the approach does not actually relies on visual perception. Instead, it sidesteps the need for genuine visual comprehension by essentially translating an image into text and performing all reasoning in text, limiting effectiveness for complex imagery requiring true visual inspection. Figure 5 supports this: only the GPT models show improvement, suggesting limited applicability unless the model’s text-based understanding is SoTA.  

SVG Limitations:  While the approach may work on some SVG, the claim in the abstract that SVG can accurately capture detailed visual scenes seems overstated, particularly within the context of this study.

Questionable Fine-Tuning Claim: The authors claim in line 085 “finetuning a model to reason about raw SVG codes can be inefficient and infeasible without corresponding task-specific annotations.”  This claim appears unsubstantiated, as fine-tuning an existing large multimodal model (e.g., LLaMA 3.2 11B, LLaVA) on the synthetic data generated for the PVD translator could potentially improve visual perception on such imagery without bypassing the need for vision.

Key questions and suggestions:

1. Can the SVG-PVD translator be replaced by a model like GPT-4o using few-shot in-context learning on synthetic data, avoiding the need for extensive fine-tuning? This could serve as a viable baseline.

2. Given the authors opt for fine-tuning, a comparison where a model similar to Mistral-7b is fine-tuned to solve the task end-to-end rather than translating into the PVD language should be shown. 

Additionally, in Table 1, presenting GPT-4o’s performance with text only would help isolate the effect of model power versus the impact of image integration, rather than conflating both under a single comparison (e.g. VDLM-txt GPT4 vs. VDLM-txt GPT4o). Based on recent findings in the literature, I might guess that the image is adding very little given the current content and most of the information needed available in text.

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730292416626

thread_type:
official_review

<thread_object>
creator: reviewer_dbww

<content>
summary:
This work focuses on vector graphics and proposes primal visual description (PVD) to better perceive and reasoning about 2D objects and shapes. Specifically, they propose PVD technique translates SVGs into a text-bassed abstraction aomprising primitive attributes along with their corresponding values. In this way, the PVD can replace the visual embeddings and directly participate in the LLM's reasoning process, improving the interpretability due to the disentangled perception and reasnoing processes.

strengths:
1. The writing in this work is excellent, with clear logic that makes it easy to follow.

2. The experimental and analysis sections in this work are thorough and well-developed.

3. The concept of PVD is intuitive, simple, and effective.

weaknesses:
1. The scope of application domains and evaluation data considered by the authors is too narrow. Although the authors mention web and OS environments as application scenarios in the abstract, the experimental evaluation is limited to simple SVG images. This may restrict the impact and significance of the work. In real-world applications, we cannot always assume that SVG-format images will be available.

2. The evaluation of high-level visual reasoning tasks is insufficient. First, only one benchmark was included in the validation. Secondly, the study only examined improvements for GPT-4V with VDLM support, which raises questions about whether other open-source, less capable LMMs (e.g., LLaVA-1.5, Qwen-VL) could benefit from VDLM as well.

questions:
1. What does the "VF" mean in Figure 1(b)? Does it denote the visual features?

2. Lacking discussion about related work [A]. It also highlights the disentangling of perception and reasoning for LMMs.

[A] Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs. NeurIPS 2024

I will consider raising the score if my concerns can be well addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730432702412

thread_type:
official_review

<thread_object>
creator: reviewer_673u

<content>
summary:
This paper introduces a novel approach to enhance fine-grained image understanding by transforming raster images into a new vector graphics format called PVD, which consists of 9 distinct primitives. The authors propose a two-stage process: first, converting the raster image to an SVG format using a rule-based model, and second, converting the SVG to the proposed PVD format using a self-trained LLM. By incorporating the PVD representation into the LLM’s context, the authors report improved performance in fine-grained image understanding tasks. The paper also provides a comparative analysis with baseline approaches that input either raw images into VLMs or SVG representations into LLMs, highlighting the advantages of their method.

strengths:
1. This work appears to be the first attempt to enhance foundation models' fine-grained image understanding capabilities by converting images into vector-based formats.
2. The introduction of the PVD vector graphics format offers a structure that seems better aligned with large language models.
3. Significant performance gains are observed when using the proposed PVD-based approach compared to directly using VLMs.

weaknesses:
1. The rationale behind the two-stage transformation process (from PNG to SVG via a rule-based algorithm, and then from SVG to PVD via a self-trained LLM) requires further clarification. Specifically: (a) What fundamental distinctions, other than syntax, does PVD offer compared to SVG? (b) Why is an LLM necessary for the second step, given that SVG is a structured, well-defined format that could potentially be converted to PVD with rule-based transformations? (c) Why not directly modify the rule-based raster-to-vector algorithm to output PVD instead of SVG?
2. The proposed PVD format’s definition could benefit from further clarity, as there appears to be overlap among the 9 primitives (e.g., "composition-filled" could overlap with "polygon"; "path" and "grid" might overlap with "line segment"). Introducing a hierarchical structure might provide a clearer and more effective organization.
3. The approach relies on synthesized data for training the LLM to convert SVG to PVD, which might limit the diversity and generalizability of the model’s performance.
4. Some terms in the paper could be refined for accuracy. For instance, the term "vector graphics" should refer to images defined by geometric primitives, such as SVG and PVD. In contrast, this work use that term "vector graphics" to refer to images rendered by SVG (which is the raster image in fact) (e.g., in the title and L91). Additionally, conflating "vector graphics" and "SVG" may be misleading (e.g., in L21–L22), as SVG is just one of several vector graphics formats.

questions:
1. The results in Table 3 (Appendix) seem inconsistent with those in Table 1. If I understand correctly, Table 1 reports that GPT-4V achieved 58% in Angle Classification (AC) and 64% in Length Comparison (LC) with image inputs. However, Table 3 lists GPT-4V as achieving 58% in Angle Classification (AC) and only 57% in Length Comparison (LC). Could you clarify this discrepancy?
2. See questions raised in the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730947977213

thread_type:
official_review

<thread_object>
creator: reviewer_8s4n

<content>
summary:
The paper introduces VDLM that aims to decrease inaccuracies associated with modern VLMs by USING an intermediate representation Primal Visual Description (PVD), primitive geometry object descriptor that can help ground the interpretation of the image and increase the usability of the model on perception and reasoning tasks downstream tasks involving vector graphics. More specifically, the paper identifies Low-level visual reasoning tasks and High-level visual reasoning tasks, introducing benchmarks for evaluating the former and using VGBench-QA for the latter, and showing performance gains for both text and multimodal versions of VDLM.

strengths:
- Interesting issue: The VLMs are known to struggle with low-level details in image analysis, making a low-resource solution like VDLM impactful.
- Benchmarking dataset: To the best of my knowledge the dataset is unique and is a significant contribution that could serve as a good benchmark for quantitative evaluation of current and future VLMs.
- PVD approach: Is intuitive and encodes a lot of information about the object in a relatively compact manner.

weaknesses:
- Practical usability: The Ontology of PVD is limited and seems like it would be hard to expand to more general use cases, furthermore it seems like PVD use would strain the context length significantly in any more complex use cases, thus while PVDs might be learnable in task-agnostic setting I am concerned about the generalizability of VDLM.
- Limited baselines: The paper reports numbers for Variations of GPT, LLaMA, ViperGPT models, however there are no comparisons with any symbolic approaches, making the positioning of the results hard to place.

questions:
- How would the method extend to SVG complexity found in websites and most usual use cases?
- How well could the method handle occlusion between distinct shapes?
- Could the authors provide further reasoning behind their choice of baselines? Why were symbolic reasoning approaches not included in the comparison?
</content>
</thread_object>

</thread>


Metareview: