Title: VGBench: Evaluating Large Language Models on Vector Graphics Understanding and Generation
Benchmark: VGBench
Summary: The paper introduces VGBench, a comprehensive benchmark for vector graphics images that tests both visual understanding and generation. Formats like SVG, TikZ, and Graphviz are included, and performance is generally strong, though LLMs do worse with the lower-level SVG format.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: processing vector graphics
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: For the QA task (VGQA), models are given a vector graphics representation (in textual format) and a multiple choice question about a high-level feature of the image, like the colour of a depicted entity. For the generation task (VGen), models must generate vector graphics code from a textual description.
Task item definition: For VGQA: a snippet of vector graphics code, a question with multiple choice answers, and a correct answer.
For VGen: a textual description, the desired output format (e.g. SVG), and some ground truth vector graphics code.
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 10,124
Dataset size notes: 4,279 examples in VGQA, 5,845 examples in VGen
Dataset metadata available: Yes
Dataset metadata detail: vector graphic format
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice, Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: vector graphics format and question subtype (e.g. "Domain", "Layout", "Relation" questions)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Instruction Following / 
New taxonomy: General Capability / Instruction Following