StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs
Authors: Jialin Yang
Dongfu Jiang
Tony He
Sherman Siu
Yuxuan Zhang
Disen Liao
Zhuofeng Li
Huaye Zeng
Yiming Jia
Haozhe Wang
Benjamin Schneider
Chi Ruan
Wentao Ma
Zhiheng Lyu
Yifei Wang
Yi Lu
Quy Duc Do
Ziyan Jiang
Ping Nie
Wenhu Chen
Venue: Accepted by TMLR ()

Abstract: As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce $\textbf{StructEval}$, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: $\textbf{1)}$ generation tasks, producing structured output from natural language prompts, and $\textbf{2)}$ conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps—even state-of-the-art models like o1-mini achieve only $75.58$ average score, with open-source alternatives lagging approximately $10$ points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.

Published in Transactions on Machine Learning Research (01/2026)
SvructEval: Benchmarking LLMs' Capabilities to Generate
Structural Outputs
Jialin Yang* 11
Dongfu Jiang* 116
Lipeng He'
Sherman Siu' Haozhe Wang
Yuxuan Zhang' Disen
Liao' Zhuofeng Li
Huaye Zeng'
Yiming Jia'
Benjamin Schneider
Chi Ruan Wentao Ma
Zhiheng Lyu'
Yifei Wang' Yi Lu
Quy Duc Do
Ziyan
Jiang' Ping Nie!
Wenhu Chen
University of Waterloo, University of Toronto, HHKUST, 'Shanghai University, 'Independent Contributor, 'Vector Institute University of British Columbia "Equal Contribution
T(j586yang, longfu.jian, wenhuchen, ping. nie)@uwaterrrlo.cc
Reviewed on OpenReview: http:///ooeerviiie.eeee//oouuuut
Abstract
As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce SvructEval, comprehensive benchmark for evalusting LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompls, and 2) conversion lasks, translating between struc- tured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant perfor- mance gaps- even state-of-the-an thert models like ol-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than genorating text-only structures. Please see our project page for more detail https://tiger--ii wwt.ulaa...ittbbiiii..o... itttuub.o//Strutttvvall
1 Introduction
In recent years, there has been a significant surge in the capabilities of large language models (LLMs) in generating human-like text and performing wide range of natural language processing tasks. State-of- the-art models like GPT-4o (Hurst. et al. 2024) OpenAl ol/o3 (Contributors et. al. 2024). and Google's
Gemini TTam et al. 2023 have achieved superior performance in knowledge QA (Hendrycks et al. 2020
Wang et al. 2024), instrucioon-ollooinng (Chiang et al. 2024 Zhou et al. 2023 and code generation (Zhuo
et al. 2024
Jain et al. 2024
Despite recent advances, many real-world applications require not only fluency in the content of the output but also precise control over its structure. This includes lasks where the expected output must follow specific formats such JSON, XML, LaTeX, HTML, code in frameworks like React Vue. Additionally, in these tasks, we also want the code to render a page that correctly places elements according to the requirements. These types of structured output are essential in domains like software development, data pipelines, user interface generation, and scientific publishing, where incorrect formatting can lead to disrupted pipelines or non-functional outputs.
However, most existing benchmarks focus on the semantic quality (Wang al. 2024 or reasoning ability of LLMs (Hendrycks et al.) 2021 He et. al.] 2024), with limited emphasis their ability to produce

---

Published in Transactions on Machine Learning Research (01/2026)
StructEval-T
StructEval-V
[T]
M+
SVG
HTML
YAML
JSON
CSV
XML
Angular
LaTlex
Markdown
TOML
CANVAS
TikZ
C
Marpiomb
Typst
Vae
Vega
React
STRUCTEVAL evaluates the LLM's capability generate structured outputs, including text-only tasks like JSON, TOML, etc, and visual rendering tasks like HTML, React, Latex, etc.
Figure
Task Prompt
2
Query & Metric
3
Expert Review
System: You a prompt design
Task Prompt
Query
Metric
Generate
assistant...
GPT-4.1
Query Metric
Instruction:
You are designing new LLM query based on the following input/ output types...
Query
2 Rounds
Please output HIML code...
Manual
Label Studio
Review
Metric
Example:
You generate new query that is structured like the example below...
VQA Q: size?
Keywords Syntax Rules
A: 16px
Query
Metric
Figure 2: The overall designed annotation pipeline of STRUCTEVAL dataset
format-conforming structured outputs. Some recently proposed benchmarks aim evaluate the quality of structured outputs tend to target specific modalitics, such code generation (Zhuo et al. 2024 text-only structures (Gu et al.) 2024 Tang et al. 2023), rather than offering comprehensive evaluations across diverse structured formats. As existing benchmarks gradually become more saturated, it is still unknown how the current stato-of-thc-art models perform structured generation tasks. We argue that effectively evaluating the models' performance on such tasks inherently challenging due to the following issues:
(1) Data Collection Challenges: Gathering diverse structured tasks and corresponding examples requires domain expertise across multiple formats, with high-quality annotations demanding significant effort and specialized knowledge.
(2) Evaluation Metric Complexity: Designing reasonable metrics in a unified form for both text-only structures (JSON, YAML) and visual outputs (HTML, SVG) is difficult, as they require different assessment approaches for structural correctness and visual fidelity.
(3) Technical Implementation Barriers: Building a framework that supports exocution and evaluation across numerous rendering environments requires complex integration of multiple language interpreters and visualization tools.
To address these challenges, we introduce STRUCTEVAL, a comprchensive benchmark that systematically evaluates LLMs' abilities to produce highly struct output. Our benchmark encompasses 21 distinct formats and 44 task types organized into two complementary subsets: StructEval-T, which assesses the generation of text-only structures such as JSON and TOML, and StructEval- V, which evaluates the quality of visually rendered outputs from code such as HTML and SVG. Both subsets include generation tasks (converting natural language to structured outputs) and conversion tasks (transforming between two structured
2

---

Published in Transactions on Machine Learning Research (01/2026)
# Total Tasks
# Total Examples
# Avg Keywords
# Avg VQA pairs
Subset
SE-T-gen
5
250
7.9
SE-T-com
14
700
17.5
SE-V-gen
13
650
11.1
7.9
SE- V-conv
12
435
22.2
9.0
StructEval
44
2035
14.7
8.5
Table 1: The overall statistics of the STRUCTEVAL dataset. Here "SE" denotes StructEval. "T" and 'V' represents the StructEval- T and StructEval- V subsets respectively. "gen" and "conv" represent the 'generation' and "conversion' lask types respectively.
formats), Soe Figure for example formats. To ensure robust evaluation across these diverse formats, we have developed a novel assessment framework that integrales syntactic validity checking, keyword matching, and visual question answering, providing a holistic measure of both structural correctness and output fidelity.
Our comprchensive evaluation reveals significant performance gaps across models and tasks. Even stato-of- the-art commercial models like ol-mini achieve only an average score of 75.58, while the best open-source model, such as Llama-3-SB- 3--SB-Instrurt. lags 10 points behind, underscoring the performance gap between commercial and open-source LLMs. We observe that generation tasks generally pose greater challenges than conversion tasks, and producing code capable rendering correct visual content proves more difficult than generating text-only structured outputs. Task difficulty varies considerably across formats: while some tasks are effectively solved by all LLMs with scores exceeding 0.95 (such Text-Markdoww and THxt-H others remain particularly challenging with all models scoring below 0.5 (including Text-Merraid and Matplotlib--Tikke Through this systematic analysis, aim to drive progress in structured output generation capabilitics that are increasingly crucial for the real-world applications of language models.
2 StructEval Dataset
In this section, we first present an overview of our STRUCTEVAL dataset and statistical analysis in subsectionn 2.1 Next, we elaborate on how we design the whole pipeline for annotation and quality review in subsection 2.2 We will introduce how we design the evaluation metrics for each task in our dataset in section
2.1 Overview
As shown in Table I our STRUCTEVAL dataset comprises a total 2,035 examples, covering 44 unique structure generation tasks across 18 structured output formats. The datasct organized into two main subsets: StruclEval-T and StructFual-V.
StructEval-T is designed to evaluate an LLM's ability to generate structured outputs directly from natural language prompts without rendering. Supported formats include JSON, XML, YAML, Markdown, CSV, TOML, among others. These are highly useful formats in many downstream applications.
StructEval- assesses an LLM's ability to generate executable code for visual rendering that fulfills a specified visual requirement. This subset includes formats such as HTML, React, Matplotlib, Canvas, LaTeX, SVG, Mermaid, and more. These are widely adopted formats for various applications.
Each example in the dataset is categorized as either generation or conversion. In generation tasks, the model is required to produce structured output based on a natural language description with detailed specifications. In conversion tasks, the model must translate structured content from one format another (e.g.. JSON YAML, HTML to React).
3

---

Published in Transactions on Machine Learning Research (01/2026)
Rule Type
Example
Description
Literal key access
plaeet.name
Checks key name exists as a child of object planet.
Nested lists with index
alanetmmons(00) name planet.moon moneettmoos.K 8, name data. "key. wi th. dots'
Verifies first item in moons list has name field. Confirms that name exists for any moon in the list. Treats entire quoted token
Wildcard lists
Backtick quoting
single key, useful for
special characters.
CSV header check
csv: :sv::discovery. location
Ensures CSV output has column named discovery locatioon..
XML attribute fallback
Looks for id attribute, using to indicate XML format.
Table 2: Supported rule types in our path-based evaluation.
EvalctEval-T Question, Key Words
Please output JSON code.
Task:
Summarize metadata about fictional scientific article. Feature Requirements:
1. Top-level field "title" is string containing the article title.
2. Field "authors" is a list of exactly two items.
Each element of "authors" contains "name" (string) and "affiliation" (string).
4. Field "publication.y year" is an integer.
Field "keywords" " list. strings.
Keywords:
title
authors[0].nes name
authors[1].affiliaee
publication
keywords[2]
Figure 3: Example question and key words of the StructEval-T gencration task
Human Evaluation of VQA Questions
Unfair
Fair
Total
Fair Proportion (%)
Correct
347
352
98.58%
Wrong
39
6
45
13.33%
Total
44
353
397
88.92%
Accuracy (%)
13.64%
98.30%
88.66%
Table 3: Human evaluation results of sampled VQA questions used in StructEval-V. Each question is annotated fair or unfair, and correctness is measured by VLM judge performance.
Formally, each example is represented as a triplet (q, K, Q²), where a denotes the structure genera- tion question, K "hh...,,kii is set of keywords expected to appear in the output, and Q" {(T",a)) ai),..,, ( )) is set of visual question-answer (VQA) pairs used for evaluating examples in the StructEval-V subset(An example StrletEval-V task with keywords and VQA pairs is shown in Figure 4h. In contrast, for StructEval- T, Q" is empty and not used during evaluation (An example Struct Eval-T question and its keywords are shown in Figure 3). To ensure comprehensive evaluation, each example in the dataset. contains on average 14.7 keywords and 8.5 VQA pairs, as detailed in Table 1
To further assess the quality and fairness of the VQA pairs used StructEval-I V, we conduct a human expert evaluation. Each VQA question is judged as either fair, meaning it can be reasonably answered by VLM judge using only the rendered image, unfair, typically involving information not visually accessible, such
4

---

Published in Transactions on Machine Learning Research (01/2026)
precise numeric values interactive UI elements. Table 397 sampled VQA pairs, 88.92% were considered fair, and 98.58% of the correct VQA questions were judged fair. Overall, 98.30% of all fair questions could be correctly answered by our VLM judge (GPT-4.I-mini), supporting the validity our automated evaluation process.
presents the results of this evaluation. Among
The dataset encompasses a wide spectrum of structured output formats, ranging from widely-used data serialization types like JSON and YAML risually-renderable formats such SVG, Mermaid, and TikZ. This diverse format coverage enables more holistic evaluation of LLMs' capabilities in both structured data modeling and visual code generation. Notably, the inclusion of niche yet expressive formats- -unnats-suuhhh Typst for typesetting, Mermaid for diagram specification, and Tik2 for LaTeX-basod graphics- evaluative scope beyond conventional tasks. These formats collectively span domains including web front-end development, data exchange, scientific visualization, and technical documentation. The distribution of tasks across these formats shown Table 7 highlighting the balanced composition of generation and conversion tasks across both textual and visual modalities.
the
2.2 Annotation Pipeline
To construct high-quality and diverse benchmark, we design multi-stage annotation pipeline consisting of three key components: 1) task curation, 2) LLM-based synthesis, and 3) expert review (see Figure 2 for overview of this pipeline). This pipeline ensures both the scalability and accuracy of the STRUCTEVAL dataset.
Task Prompt We begin by identifying broad spectrum of structure generation and conversion tasks that span both text-based and executable visual formats. These tasks are selected to reflect practical use
StructEval-V Question, Keywords Matching, VQA Pairs
Please output HTML code.
Task:
Design webpage that presents user's travel itinerary. Feature Requirements:
Include a centered <h1> header with the text "Trip Summary",
Use <table> to list destinations; include 3 rows and 2 columns.
Apply a class "highlight" to the second row.
Add <button> labeled "Export PDF" at the bottom of the page.
Keywords:
Trip Summary
highlight
<h1>
Export PDF
VQA Pairs:
Q: What text is displayed in the <h1> header?
A: Trip Summary
Q: How many rows in the table?
A: 3
Q: What class is applied to the second table row?
A: highlight
Q: What text the button at the bottom?
A: Export PDF
Figure 4: Example question, keywords, and VQA pairs for STRUCTEVAL-Y V generation task

---

Published in Transactions on Machine Learning Research (01/2026)
cases and diverse real-world scenarios, covering 18 target formats and 44 distinct task types (also shown in Table 7 Each task specification includes format constraints, input-output expectations, and, where applicable, conversion rules. Please refer to subsection A.4 for a sample task prompt.
Query /Merric Generation Given the high cost of fully manual annotation, we loverage a large language model to synthesize an initial pool of candidate examples. Each example consists of task query and set of associated evaluation metrics, including keywords for text outputs and visual question-answer (VQA) pairs for visual outputs. This step allows to rapidly generate large and varied collection of plausible instances that serve as drafts for human refinement.
Expert Review To ensure quality and correct we employ a two-pass human review process. Annolators first validate and refine the generated task queries and associated metrics. They are allowed to freely modify, add, remove any part of the synthesized content to ensure task clarity, completeness, and ovaluability. In the second pass, separate reviewer verifies the consistency and correctness of each example. All annotation is conducted using LabelStudio (Tkachenko et al., 2025), open-source collaborative annotation tool designed for structured data. The final dataset contains 2035 curated examples, carefully reviewed to support robust evaluation across both StructEval-T and StructEval- V settings.
3 StructEval Evaluation
Before the evaluation, we feed the LLM with the questions q in the datasets with the corresponding prompt template defined in Table 4. We require the LLM to output the desired structured outputs between <
 and "<|END_COOD" "OOND__DDEE>" so we can correctly parse the structured outputs for evaluation. For the StructEval-Ve V, parsod outputs will be additionally sent to our rendering engines acquire the rendcred visual oulputs (see examples in subsection A.3) We then evaluate model outputs using an automatic evaluation pipeline that captures both structural correctness and semantic fidelity. Specifically, have designed core metrics depending on the task format: 1) Syntax Score, 2) Keyword Matching Score, and 3) Visual Question Answering (VQA) Score.
(StructEval Question)
IMPORTANT: Only output the required output format. You must start the format/code with [|BEGIN_CODEI] and end the format/code with [oDEE_ No other text output (explanation, comments, etc.) are allowed.
Do not use markdown code fences.
Table 4: Prompt template used for LLM inference before the evaluation
Syntax Score. The Syntax Score verifies the structural correctness of the generated output. For text-based formats such JSON, YAML, and CSV, this involves parsing the output using format-specific Python parser. For executable visual formats like HTML, LaTeX, or SVG, the code is rendered using headless renderer to determine whether it executes successfully. A score of is assigned if the output is syntactically valid or successfully renderod; otherwisc, the score is Soe the subsection A.3 for some correctly rendcred images, code produced by the lested LLMs.
Keyword Matching Score This metric evaluates whether the generated output contains the required structural elements. Given the reference set of expected keywords K = (ki,...,kik] kik for a given task, we assess their presence using exact matching or regular expression rules.
For the tasks of StructEval-T T such as JSON or XML, keyword matching is performed over field names and values using dot-path references to account for nested hierarchies. The score computed as the proportion of expected keywords correctly matched in the model's output. Our evaluation supports a variety of path formats as shown in Table 2 The way dot-path rules are created differs depending on the task type.

---

Published in Transactions on Machine Learning Research (01/2026)
VQA Prompt Template
You are given an image and list of question-ansee answer pairs.
For each pair, verify if the image content supports the expected answer based on corresponding question.
Base your judgment solely on the visual content the provided image, and the question.
Do not use any external information or common-sense reasoning beyond what is visible.
Respond with a JSON object mapping each question number to true or false (e.g,, ('1': true, "2': false)).
If' the image is unclear or does not contain enough information to answer, use null for that question.
Here are the question-answer pairs: (qa list}
Figure 5: Prompt template used for VQA evaluation. We GPT-4.1-mini in the benchmark evaluation.
For generation tasks, each task prompt includes feature requirements stated in natural language. These requirements define target keys and their relationships one another (e-g., nesting depth, list membership). Annotators translate each requirement into concrete dot-path rule using the syntax rules shown in Table 2 For conversion lasks, the input is itself a structured format (e-g., YAML XML). We use an LLM to parse the structural schema of the input-idenne iidentifying key names, nesting levels, and list structures and convert them into target dot-path rules that the generated output must preserve.
This approach ensures that models are not only producing syntactically valid outputs, but also preserving the expected structural relationships.
For the tasks of StructEval-V V such HTML, and Matplotlib, we simply detect whether the annotated keyword is in the structured outpuls and give scores accordingly.
VQA Score This score is used exclusively for tasks in the StructEool- V subset, where the output is expected to be visually rendered. After rendering the output, GPT-4.1-mini (Hurst et al. 2024), a vision-langunage model (VL.M), is employed to answer a set of visual questions Q = [(g) a af),..., ( Poy (orr))) The VLM will be given both the questions and answers and required decide whether the VQA pair matches this rendered image. The VQA score is computed as the proportion of correctly answered questions.
Final task scores calculated weighted combinations of these metrics, with weights adjusted based on whether the task is renderable. Let Sk, E [0, 1] denotes the syntax, keyword matching, and VQA score respectively. For the StructEval-T lask, the final score is computed as:
0.2 + 0.8
(1)
For StructEval- V, the final score is computed as:
s = 0.2 s, + 0.1 Sy + 0.7 Sp
(2)
This evaluation framework provides a unified, fine-grained view of model performance across both structured data generation and visual code synthesis tasks, supporting deeper insights into LLM capabilities across modalitics.
4 Experiments
4.1
Experimental Setup
Evaluation Models. We evaluate a range of open-source and commercial large language models (LLMs)
using our benchmark. For open-source models, Metta-llama-3-SB-nntteee (n--nnttted(Grraaaafiii
Grattafiori et al. (2024) Phi-

---

Published in Transactions on Machine Learning Research (01/2026)
3-mini-128k-inse 12kk-isstuttt Abdin et al.
(2024a), Phi-4-mini-insse innssructtttttiii Abdin et al. (024b) Swen2.5-7B-Instruue Yang et al. (2024). and Qwen3-4B Yang et al. (2025). For commercial models, we use Gemini-1.5-pro and Gemini-2 O-f-lssh Team et al. (2023). GPT-441-mins and GPT-44o Hurst et al. (0024), GP-T-4--mini, and
ol-mini Contributors et al. 2024). All tasks are evaluated in a zero-shot sctting using consistent prompts
and parameters.
Inference Setup. All model generations are performed using LLM-Engine Jiang (2024), unified inference framework that supports both open-source backends (e.g., VLLM, SGLang, Together), and commercial APIs (e.g., OpenAI, Claude, Gemini). For spun-s models, specifically utilize the vLLM engine for efficiency Kwon et al. 2023). For closed source models, we simply call the APIs. As shown in Table 5 we use greedy decoding by default. All tasks are evaluated zero-shot using uniform task prompts defined in Table 4. When performing the VQA evaluation, we select GPT-4.1-mini as the VLM due to its superior multimodal abilities (OpenAl 2025). We apply the VQA prompt template defined Figure and ask the VLM decide whether each VQA pair matches the rendered visual image once.
Parameter
Value
Max tokens
Unlimited
Temperature
0.0 (deterministic)
mum proc
32
time_ out
None
num workers
num gpu per__err_tteeeeee worker
Cache usage
Disabled
Batch API
Disabled
Hardware
NVIDIA RTX A6000 GPU
Table 5: Inference configuration
Output generations are automatically scored using the evaluation pipeline described in section 3 including syntactic validity checking, keyword matching, and VQA accuracy. GPT-4.1-mini used as the vision-language model for all VQA-based evaluations.
Evaluation.
HHustt et al,
2024
4.2
Main Results
StruutEvall- generation
StructEval-V conversion
Models
Average
conversion
generation
Open Source
Llama-3.1-8B-nsttee Grattafiori
2024
60.22
71.26
54.44
61.15
61.77
Moua-Llaama---S--Inssttte Craftaliori
al
2024
49.18
56.91
51.59
1mni--125k-nnstmett Abdin al. 2024
47.39
29.78
44.77
41.23
40.79
Phi-4-mini-instruce Abdin ct p 202
51.38
72.30
51.62
52.48
56.97
Qwen2.5 Swwn2.5-7B-Inntreee Tnsstrue
Team 2024
59.21
62.18
53.28
61.43
59.03
Qwen3-49 4B
Yang el al. 2025
64.95
81.13
57.00
65.08
67.04
Close Source
Gemini- 1aainii-15-ppp 1.5prro
Team et al.
2023
88.07
74.24
58.11
66.59
71.75
Gemini-2200flaan 2.0-fash Tram el ml
2123
72.42
72.20
53.62
51.97
62.55
GPT-4.1-mint OpenAT
2025
92.57
75.63
64.30
70.04
75.64
GPT-4u Huss
2024
91.52
73.95
65.39
73.20
76.02
GPT-4o-min Hurst
2024
79.86
75.57
60.77
76.54
73.19
ol-mini Coutributors ct
2024
88.12
81.82
75.58
A (al-mini Qwen3-4B)
23.17
0.70
4.99
5.32
8.54
Table 6: Main evaluation results of STRUCTEVAL
8

---

Published in Transactions on Machine Learning Research (01/2026)
TedL-TOML
90%
80%
Text-Memmidd
80%
50%
Matplotii TIKZ
40%
30%
YAML-XML
CSV-
10%
0%
0%
20%
40%
60%
80%
100%
V-Gen
(a) Avg. score over all models based on most challenging (b) Avg. score over all models based on the four task types subtasks
Overall Performance Table 6 summarizes the performance of all evaluated models across the two main task groups: StructEval-T and StructEval- V,, each further divided into generation and conversion subtasks. Overall, GPT-4o achieves the highest average score of 76.02% among all 12 models. The best-performing open-source model Qwen3-4B, with score of 67.04%, trailing GPT-4o by approximately 10 percentage points. While GPT-4o excels particularly in the generation tasks within the StructEval- category, Qwen3-4B demonstrates consistently strong performance across all task types among open-source models. This likely reflects Qwen3-4B's robust reasoning capabilities relative to other open-source alternatives.
In contrast, the lowest-performing model is Although one might attribute this to its relatively small size of 3.8 billion parameters, model size alone does not fully explain the poor results. For example, phi-3-mini underperforms even compared to similarly sized models such as phi-4-mini-instree tasks, a category where models with strong reasoning abilities such ol-mini (81.8%%) and Qwen3-4B (81.13%) Lend to perform well.
waimiiii-1k--iitttrrte with an average score of only 40. 40.79%.
Notably, it achieves the lowest score in StructFaal-7 conversion
Error analysis reveals two key failure modes for conversion task, the model frequently produces malformed closing tags, outputting |<|END_CODE| |> instead of the correct WEEND_COE|>> which significantly penalizes its score. Second, in the CSV-Lo-JSON conversion task, the model fails to capture hierarchical relationships (e.g., parent-child) specified in the CSV headers, leading to structurally incorrect JSON outputs. These recurring structural errors in StructEval- conversion tasks substantially contribute to the model's overall low performance.
First, in the TOML-to- YAML
Open-Source vs. Closed-Source Models When comparing open-source models and commercial models, that by A (closeavg openavg) value, which is the difference between the average score of commercial source model and open model, that commercial model's score is consistently higher than open-source models, this makes sense given the much larger parameters of commercial models by scaling law. We see that commercial models exceed open-source models on average the most on generation tasks in StructEval-T setting, and the performance gap is smallest. on generation lasks in StructEval-V setting.
Generation vs. Conversion As shown in Figure 6b a comparison between generation and conversion lasks in both StructEval-T and StructEval- V settings reveals that, in general, models perform better on conversion tasks than on generation tasks. An exception to this trend occurs in the StructEval-T setting, where commercial models tend to outperform on generation tasks, while open-source models show the opposite behavior schieving higher scores on conversion tasks.
Under temperature setting of 1, commercial models attain average score of 75.78% StructEval-T T generation tasks. In contrast, open-source models average only 8.58% on the same tasks for the TOML format. This considerable disparity in TOML generation performance partly explains why commercial models perform better on StructEval-T generation tasks overall. However, the performance gap is not confined to TOML- commercial models also lead in the other four generation formats within StructEval-I1
In the StructEval- setting, commercial models significantly outperform open-source counterparts on generation tasks involving complex visual formats such as Mermaid and TikZ. These tasks require advanced

---

Published in Transactions on Machine Learning Research (01/2026)
visual reasoning capabilities, which are more prevalent in multimodal commercial LLMs like GPT-4o and GPT-4o-mini.
Subtasks Analysis Mcanwhile, several tasks in both in generation and conversion types appear to be saturated, with most models achieving scores exceeding 90%. These include generation tasks for common formats such as JSON, HTML, CSV, Markdown, and YAML, as well as conversion tasks like YAML-to-JSON, Heact-to-HTML, TOML-to-JSON, and Markdown-to-HTMM.. Such results indicate that LLMs have already mastered many structurally straightforward format transformations.
There remain several challenging tasks where all models struggle significantly (shown in Figure 6a), including generation tasks like TeXI- Text-+VVG, Text->Mermale and Tega-> as well as conversion tasks like YAML-XML, CAMM, Matplotiib->Tike and Markkoown-Angular((se scores in subsection A.2). Both closed-source and open-source models achieve low scores on these tasks, which typically require complex structural or visual reasoning. Notably, the performance gap between closed-source and open-source models is even wider these challenging subtasks, suggesting that proprietary models may have advantages in handling more complex structural representations and transformation logic.
5 Related Work
5.1 Large Language Models
Large Language Models (LLMs) have demonstrated remarkable capabilities and gained surging popularity in recent years, ever since the release ChatGPT (OpenAl 2023 Over the years, open-source models like Llama (Grattafiori al. 2024), Phi (Abdin al. a)) and Qwen (Yang et al. 2024 2025) developed by companies like Meta, Microsoft, and Alibaba further facilitated widespread integration of AI into diverse workflows and everyday applications. Leveraging their large parameter sizes and extensive post-training, LLMs are capable of performing diverse array of Natural Language Processing (NLP) tasks (Wan et al., 2023). One of the key aspects of the generative capabilities of these models is their ability to generate structured data and transform data from one type to another while maintaining strict adherence to specified formats (Guo et al., 2024). In this paper, we design a new and comprehensive benchmark that evaluates the capability L.LMs to understand, generate, and manipulate structured data across a range of complex, real-world tasks.
5.2 Evaluation of LLMs
Evaluating structured output has become a focal point for understanding LLM's limitations (Ning al,, 2025). SoEval (Liu et al. 2024 offers fast, rule-based check for JSON and XML, but its flat schemas lail to reveal errors in deeper hierarchies. StrucText-FFi (Gu et al. 2024) shifts the task to reasoning over structure-rich text (JSON, YAML, LaTeX) rather than generating the structures themselves, while FOFO Xia et al.) 2024) extends to domains such as law and finance yet covers only a few formats and still relies on human verification. Developer-focusee suites like StackEval (Shah et al., 2024) for HTML, CSS, and plotting libraries, and CodeXGLUF (Lu et al. 2021) for multilingual code tasks remain limited to programming artifacts, and Struc-Bench (Tang al. 2023) concentrates on tabular generation with bespoke metrics. Each benchmark highlights part of the challenge- be it format adherence, domain coverage, table fidelity. However, none simultaneously demands broad format coverage, automated grading, and robust transformation capabilities. StructEval addresses these gaps by spanning 18 code and non-code formats, unilying generation, completion, and conversion (asks, and scoring outputs with fully automated structural and rision-based metrics, offering a comprehensive lens on how well LLMs respect and manipulate complex schemas.
5.3 Structured Output Generation
The ability to generate structured outputs is central to many real-world applications of LLMs (Gu al., 2024 Tang et al., (023). These outputs are not only expected to be semantically coherent but must also adhere strictly to syntactic and structural constraints constraints-violations of which can lead to parsing failures,
10

---

Published in Transactions on Machine Learning Research (01/2026)
rendering errors, broken downstream applications. Common tasks include generating JSON for API responses (Geng et al. 2025 YAMI or TOMT, for configuration files (Peddiredyy) 2024), HTML or React for UI components (Si et al. 2024), and LaTeX or Markdown for technical writing (Wen et al. 2024). Moreover, in data science, models are used to transform unstructured descriptions into structured formats like CSV or tables for integration into analysis pipelines (Li et al. 2023 Su et al. 2024). In publishing and education, tools that convert textual prompts into diagrams (c.g.. using TikZ, SVG, or Mermaid) help automate visualization
generation (Lee et al. 2025 Rodriguez et al. 2025 Ku al. 2025). Despite its significance, structured output generation remains challenging due to the need for models to internalize both syntax rules and hierarchical schema relationships across a wide variety of formats. Our STRUCTEVAL. first conducts a comprehensive evaluation of existing LLMs on both renderable and non-renderable tasks, showing that they still struggle to correctly generate some data formats including TOML, SVG, and Mermaid.
6 Conclusion
In this paper, we have comprehensively studied LLMs' abilities to generale highly structured content. Having the ability to generate fully structured content is highly useful for many downstream tasks. Our paper is among the first few to provide evaluation suite for that. Our results indicate that current models are still lagging on the renderable structured content, especially on less frequent format. We advocate that the future models should invest more time to optimize their abilities to generate highly structured output.
Limitations
Non-interactive formats Our benchmark focuses on evaluating LLMs' ability to generate static visual rendering formats such as HTML, React, Mermaid, etc. While this approach effectively assesses the model's capacity to produce well-structured and visually coherent outputs, it is currently limited to single-page page, non-interactive formats. The evaluation does not account for dynamic behaviors such as button interactions, page transitions, animations, scroll events, which are essential to many real world user interfaces. Future work could extend the benchmark to include dynamic rendering tasks, enabling more comprehensive assessment of LLM capabilities in producing fully interactive and responsive user experiences.
Expert Review While our dataset underwent a two-pass expert review process to ensure correctness, diversity, and minimize polential biases, the initial content was still generated by large language models. Despite expert oversight, residual biases inherent the model outputs may persist, particularly in subtle or context-dependent scenarios that are challenging detect through manual review. Moreover, expert validation, while thorough, may not fully capture the wide range of cultural, social, contextual sensitivities relevant to diverse user populations. Future work could incorporate broader multi-annotatas audits or automated bias detection techniques to further enhance dataset reliability and inclusiveness.
Acknowledgement
This research was supported in part by the Google Cloud Research Credits Program.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mci Gao, Min Gao, Amit Garg, Allic Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewell, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengehen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro
11

---

Published in Transactions on Machine Learning Research (01/2026)
Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Beckers Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yclong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddananu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jishang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024a. URL https://ar wwwp:t/..rrg/bb///0/000 14219
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024b. URL https: http::://rrii.org/ab////// wwtp:////.abb/./..../.// 08905
Wei-Lin Chiang, Lianmin Zhong, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating 1lms by human preference. ArXiv, abs/2403.04132, 2024. URL https:
s/api.ssmanticccholaa.... http:/ccchhoollooogrrrrrrooooooa wwwp/oorrssD:/eeeeeooo 268264113
Foundational Contributors, Ahmed El-Kishky, Daniel Selsam, Francis Song, Giambattista Parascandolo, Hongyu Ren, Hunter Lightman, Hyung Won, Ilge Akkaya, Πya Sutskever, Jason Wei, Jonathan Gordon, Karl Cobbe, Kevin Yu, Lukasz Kondraciuk, Max Schwarzer, Mostafa Rohaninejad, Noam Brown, Shengjia Zhao, Trapit Bansal, Vineet Kosaraju, Wenda Zhou Leadership, Jakub W. Pachocki, Jerry Tworek, Liam Fedus, Lukasz Kaiser, Mark Chen, Szymon Sidor, Wojeiech Zaremba, Alex Karpenko, Alexander Wei, Allison Tam, Ananya Kumar, Andre Saraiva, Andrew Kondrich, An drey Mishchenko, Ashvin Nair, B. Ghorbani, Brandon McKinzie, Chak Bry don Eastman, Ming Li, Chris Koch, Dan Roberts, David Dohan, David Mély, Dimitris Tsipras, Enoch Cheung, Eric Wallace, Hadi Salman, Haim ing Bao, Hessam Bagher-inezhale Ily Kostrikov, Jiacheng Feng, John Rizzo, Karina Nguyen, Kevin Lu, Kevin R. Stone, Lorenz Kuhn, Mason Meyer, Mikhail Pavlov, Nat McAleese, Oleg Boiko, Oleg Murk, Peter Zhokhov, Randall Lin, Raz Gaon, Rhythm Garg, Roshan James, Rui Shu, Scott McKinncy, Shibani Santurkar, Suchir Balaji, Taylor Gordon, Thomas Dimson, Weiyi Zheng, Aaron Jaech, Adam Lerer, Aiden Low, Alex Carney, Alexander Neitz, Alexander Prokofiev, Benjamin Sokolowsky, Boaz Barak, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Camillo Lugaresi, Chelsca Voss, Chen Shen, Chris Orsinger, Daniel Kappler, Danicl Levy, Doug Li, Eben Freeman, Edmund Wong, Fan Wang, Felipe Petroski Such, Foivos Tsimpourlas, Geoff Salmon, Gildas Chabot, Guillaume Leclere, Hart Andrin, Ian O'Connell, Ignasi Tan Osband, Clavera Gilaberte, Jean Harb, Jiahui Yu, Jiayi Weng, Joe Palermo, John Hallman, Jonathan Ward, Julie Wang, Kai Chen, Katy Shi, Keren Gu-Lemberg, Kevin Liu, Leo Liu, Linden Li, Luke Metz, Maja Trebacz, Manas R. Joglekar, Marko Tintor, Melody Y. Guan, Mengyuan Yan, Mia Glaese, Michael Malek, Michelle Fradin, Mo Bavarian, Nikolas A. Tezak, Ofir Nachum, Paul Ashbourne, Pavel Izmailov, Raphael Gontijo Lopes, Reah Miyara, Reimar H. Leike, Robin Brown, Ryan Cheu, Ryan Greene, Saachi Jain, Scottie Yan, Shengli Hu, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Suvansh Sanjeev, Tao Wang, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Tianhao Zheng, T. Garipov, Valerie Qi, Vitchyr H. Pong, Vlad Fomenko, Yinghai Lu, Yining Chen, Yu Bai, Yuchen He, Yuchen Zhang, Zheng Shao, Zhuohan Li, Lauren Yang, Mianna Chen, Aidan Clark, Jieqi Yu, Kai Xiao, Sam Toizer, Sandhini Agarwal, Safety Research, Andrea Vallone, Chong Zhang, lan D. Kivlichan, Meghan Shah, Sam Toyer, Shraman Ray Chaudhuri, Stephanie Lin, Adam Richardson, Andrew Duberstein, Charles de Bourcy, Dragos Oprica, Florencia Lconi, Made laine Boyd, Malt Jones, Matt Kaufer, Mehmet Ali Yatbaz, Mengyuan Xu, Mike McClay, Mingxuan Wang, Trevor Creech, Vinnie Monaco, Erik Ritter, Evan Mays, Joel Parish, Jonathan Uesato, Leon Maksin, Michele Wang, Miles Wang, Neil Chowdhury, Olivia Watkins, Patrick Chao, Rachel Dias, Samuel Miscrendino, Red Teaming, Lama Ahmad, Michael Lampe, Troy Peterson, and Joost Huizinga. Openai ol system card. Ar Xio, abs/2412. 16720, 2024. URL https: www.mntniiicccooo.oooooooooooooo
12

---

Published in Transactions on Machine Learning Research (01/2026)
Saibo Geng, Hudson Cooper, Michal Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. Generating structured outputs from language models: Benchmark and studies. arXiv preprint arXio:2501.1066 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Alunad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Amirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliann Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Tvan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia. Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgion, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedamuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenholer, Cynthia Gao, Damon Civin, Dana Bealy, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Lo, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco
13

---

Published in Transactions on Machine Learning Research (01/2026)
Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, This Leontiadis, Frina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- -aan-Baptsste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Ti, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Kencally, Miao Liu, Michaol L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyalskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mchta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanic Max, Stephen Chen, Steve Kehoc, Steve Satterficld, Sudarshan Govindaprasad, Sumit Gupla, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zel Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The Ilama herd of models, 2024. URL /tttps//arxxv..orr wwtp:oo///////////// 21783
Zhouhong Gu, Haoning Ye, Xingzhou Chen, Zeyang Zhou, Hongwei Feng, and Yanghua Xiao. Struetext-eval: Evaluating large language model's reasoning ability in structure-rich text. arXiv preprint 1rXiv.2406.1062,, 2024. URL https://arxiv..rrg wwt.:////..b.///...... 10621
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. anXiv preprint Xrxi::400..0 010.006668 01680, 2024.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Annual Meeting of the Association for Compulational Linguistics, 2024. URL httph:ttt//at semanticscholast org// :
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs abs/2009. /20090030000 2020. URL https://app. semanticscholar wwtp://oooo/SS.///551...
14

---

Published in Transactions on Machine Learning Research (01/2026)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXie, abs/2103.03874, 2021. URL
oot.mmttttccccoooo....ooo//ooooo 232134851
OpenAl Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Adilya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mkadry, Alex Baker-Whitcome Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alexander Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alexandre Passos, Alexander Kirillov, Alexi Christakis, Alexis Comneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, An drey Mishchenko, Angela Back, Angela Jiang, An toine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, B. Ghorbani, Ben Leimberger, Ben Rossen, Benjamin Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll L.. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Chris Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Danc Sherburn, Danicl Kappler, Daniel Levin, Danicl Levy, David Carr, David Farhi, David Mély, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Phong Duc Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Prochl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Hai-Biao Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Pondé de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian D. Kivlichan, Ian O' O'Connell, Ian Osband, Ian Silber, Ian Sohl, 1brahim Cihangir Okuyucu, Ikai Lan, Πya Kostrikov, Πlya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub W. Pachocki, James Aung, James Betker, James Crooks, James Lemuon, Jamie Ryan Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quiñonero Candela, Joe Beutler, Joc Landers, Joel Parish, Johannes Heidocke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Joshua Gross, Josh Kaplan, Josh Snyder, Josh Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Thow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Ouyang Long, Louis Feuvrier, Lu Zhang, Lukasz Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metx, Lyric Doshi, Mada Aflak, Maddie Simens, Made laine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Ma teusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Ali Yatbaz, Mengxue Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Mina Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Na talie Cone, Natalie Staudacher, Natalic Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nikolas A. Tezak, Niko Felix, Nithanth Kudige, Nitish Shirish Keskar, Noah Deutsch, Nocl Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jalle, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Phil Tillet, Prafulla Dhariwal, Qim ing Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Raphael Gontijo Lopes, Raul Puri, Reah Miyara, Reimar H. Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith,
15

---

Published in Transactions on Machine Learning Research (01/2026)
Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Ramilevich Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Colley, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Floundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-40 system card. ArXiv, abs/2410.21276, 2024. URL
wwwpecchoo.a..ooorooouuutttt sem.nniicohmmttcccchoooooooao wacelaarrrrrrppoooeeee httg:/CorpuSD::23666 273662196
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. ArXiv, hts/24033.0994,, 2024. URL. http:: wtt.:ppiisemaaiiccollaa...oo.m org/
CorpusID: 268379113
Dongfu Jiang. Lim-cngines: A unified and parallel inference engine for large language models. 2024.
https:
//gi figithub.com
Max W.F. Ku, Thomas Chong, Jonathan Leung, Krish Shah, Alvin Yu, and Wenhu Chen. Theoremexplaina- gent: Towards multimodal explanations for Ilm theorem understanding. ArXiv, abs/2502. 19400 2025. URL
httpss///pp..smmmtiiccccc http://aaaaaatttaoooorrooooooooa
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Jacwook Lee, Jeongah Lec, Wanyong Feng, and Andrew Lan. From text to visuals: Using Ilms to generate math diagrams with vector graphics. ArXiv, abs/ abs/2503.07449, 2025. URL http:/pppi..smmmniiccchchooaa hBs/CorppSID:276928442
Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table-tuned gpt for diverse table tasks. ArXiv, 2023. URL https://app semanticscholar wwwpeccooaar.orr/ooppsssss..e wwwp//rppr/D PBg//CorpuSID:244127777 264127877
Jian Liu, Jian Wang, Wei Zhang, and Ming Li. Are 11ms good at structueed outputs? a benchmark for evaluating structured output generation. Information Processing & Management, 61(55:003809, 2024. doi: 121016//.pp..224.11000
URL https: http:::///oooorr/////////// 1016/ 202. 103809
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and gencration. arXiv preprint (xXiu:210204466 2021. URL htttp:://arxxw.rr//bb/////
04664
Kun-Peng Ning, Shuo Yang, Yu-Yang Liu, Jia- Yu Yao, Zhen-Hui Liu, Yong-Hong Tian, Yibing Song, and Li Yuan. Pico: Peer review in 1ms based on the consistency optimization, 2025. URL https:
7/arxiv.
//arxiv.org/abs/2444 org//bs//402..0/ 01830
OpenAl. Chat generative pre-trained transformer (chatgpt). https://www.opeeai.coom 2023.
OP Introducing gpt-4.1 1.1 in the api, April 2025. URL http:://agenaa.ccm/mndeeeeggg-- Accessod: 2025-05-20.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Abhiram Reddy Peddireddy. Effective workflow automation in github: Leveraging bash and yaml. Journal of Artificial Intelligence. & Cloud Computing, 2024. URL https://api. pwwaaaniiiiccool..oor/r//rrrrrpp
271141990
Juan A Rodriguez, Abhay Puri, Shubham Agarwal, Tssam H Laradji, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. Starvector: Generating scalable vector graphics code from images and text. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 29691-29693, 2025.
Nidhish Shah, Zulkuf Gene, and Dogu Araci. Stackeval: Benchmarking Ilms in coding assistance. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, 2024.
URL
Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: Benchmarking multimodal code generation for automated front-end engineering. ar preprint or orXiv:2433. 03166,, 2024.
Aofeng Su, Aowen Wang, Chaonan Ye, Chengcheng Zhou, Ga Zhang, Gang Chen, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, Haoxuan Lan, Jiaming Tian, Jing Yuan, Junbo Zhao, Junlin Zhou, Kaizhe Shou, Liangyu Zha, Lin Long, Liyao Li, Peng Wu, Qi Zhang, Qingyi Huang, Sa Yang, Tao Zhang, Yun Ye, Wufang Zhu, Xiaomeng Hu, Xijun Gu, Xinjie Sun, Xiang Li, Yuhang Yang, and Zhiqing Xiao. Tablegpt2: A large multimodal model with tabular data integration. ArXiv, abs/2411.02000 2024.
URL https://apl semanticscholaa wwashhollrrrrg/CCCo org/CorpusII: 273812244
Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, and Mark Gerstein. Struc-bench: Are large language models really good at generating complex structured data?
arXiv preprint arXic:2309.0880 2023. URL httpp:///axxi http:///aaxiiir/aa////// abb//339.00863
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Sorieut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805, 2023.
Qwen Team. Qwen2.5: A party of foundation models, 2024. URL http:httt////nnleeeee http://wwennm..ggtu.bbi...////// git.b.tthhhho. hotthbbbo//blog/weeen 5/
Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Studio: Data labeling software, 2020-2025, URL hhttp::t/ttttiiiii cowm/Humaniignalllaaaaaaaa available from https://githhub.co http://////ttttboooooooaaaaaaaaa SatimaanSSggal/lbbb-sttdddd
Open source software
Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. Gpt-re: In-context learning for relation extraction using large language models, 2023. URL httpss///rxxv.cr
abs/2305.02105
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max W.F. Ku, Kai Wang, Alex Zhuang, Rongqi 'Richard' Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. ArXio, abs/ 0bs/2406.0153 2024. URL https://api. http/pppp..mmmtiicscoooo.aar wwtp://ooo/oo 270210486
Haomin Wen, Zhenjie Wei, Yan Lin, Jiyuan Wang, Yuxuan Liang, and Huaiyu Wan. Overleafcopilot: Empowering academic writing in overleaf with large language models. ArXiv, 240//2243..973,, 2024. URL
http::///ap..smmtiiissioo wtw.schoo......ggrrruusssiiaaaaa 268511055
Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. FOFO: A benchmark to evaluate LLMs' format-following capability. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 680 699, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10. 18.13533/vl a2l21//222lllllloo
URL https://aclaantholog. www..//............/ acl acl.long............ long.on...0 40/
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang,
17

---

Published in Transactions on Machine Learning Research (01/2026)
Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhíhao Fan. Qwen2 technical report, 2024. URL https://arxiv, wtt.orgg////eeeeee//eeeeee
abs 2407.10671
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujje Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Kegin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingleng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL httts:://aakaeee www. 2tt///bb/2255..000 09388
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloviii evaluation for large language models. ArXiv, abs/2311.07911, 2023. URL https://api. www.sseemannnchol....... .rrg http:/CoopSI::::1111500
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiao ke Hong, Wen-Ding Li, Jean Kaddour, Minglian Xu, Zhihan Zhang, Prateck Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiao-Nan Du, Harm de Vries, and Leandro von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. ArXiv, abs/2406.1552 2024. URL https: / //ppi. htwpaa/iiiiisioaoooo//o//ssssooo 200
18

---

Published in Transactions on Machine Learning Research (01/2026)
A Example Appendix
A.1 Task Distributions
Subset
Tasks
Examples
Generation
Text JSON
50
Text CSV
StructEval-T
Text - TOML
50
Text - XML
50
Text - YAML
50
Text + Angular
Text + Canvas
50
Text - HTML
50
Text + LaTTeX
50
Text + Markdown
Text + Matplotlib
StructEval-V
Text + Mermaid
50
Text + React
50
Text + SVG
50
Text TikZ
50
Text + Typst
Text + Vega
50
Text -> Vue
50
Conversion
CSV + JSON
50
JSON - CSV
50
XML JSON
50
JSON XML
YAML + JSON
JSON + YAML
50
XMI. 7 CSV
50
StructEval-T
CSV + XML
50
XML YAML
50
MA XML
YAML +
TOML + JSON
50
CSV + YAML
50
TOML YAMI
50
Marplotlib TikZ
100
Markdown + HTML
50
HTML + React
45
React HTML
45
Vue + HTML
HTML +
40
StructEval-V
Markdown -> React
30
HTML > Augular
30
Markdown Viee
25
Vue React
Markdown + Angular
React + Angular
Table 7: Statistics of number examples for each task in all the 4 subsets of STrUCTEVAL.
19

---

Published in Transactions on Machine Learning Research (01/2026)
A.2 Subtask Performance
T-JISON
T-TOML
T-XAMI
T-CSV
T-XMM
Model
Avg.
78.82
81.68
6.76
59.38
74.44
60.22
Micta-llama-3-8--nnsiiie
69.08
45.04
7.94
45.30
78.54
49.18
Phi-3-mini-
1--mmnn-228k-mstruet
68.84
93.50
37.68
36.92
47.39
Phi-f-miniiInstrutt
51.50
82.56
16.12
66.54
Qwen-2.5-1
Bwee-2.5-7B--ae 2.5-7B-nntt
84.40
90.62
13.22
61.30
46.52
59.21
Qwen-3-BB
90.96
76.44
7.44
71.16
78.74
64.95
Gemini-1.5-pro
94.06
100.00
75.38
73.32
97.08
88.07
Gemini-2.0-flash
48.88
98.40
78.78
44.60
91.44
72.42
GPT-411-mini
99.26
99.92
91.34
77.06
95.26
92.57
GPT-40
99.36
90.22
70.32
97.68
91.52
GPT-4o-mini
97.88
99.90
29.56
75.10
96.84
79.86
ol-mini
92.56
99.24
89.40
71.12
88.28
83.12
Table 8: StructFval-T Generation Scores
T-AAng
T-ASTEX
T--M
T-MMPL
T--React
T-SVG
T-TTE
Model
1lama-3.--B-Insttuuc
87.31
80.52
64.30
44.18
46.92
Meta-Llama-33-6 3-8B- Instructtt
48.92
68.40
72.06
56.34
55.24
40.16
28.04
Phi-3-mimi-1255 288--tmmettt
Phi-4-mini-Instruct
02.60
72.92
88.90
71.30
58.46
39.72
35.28
Qwen-2.5-7B-lnstruo
63.08
66.68
81.02
74.70
65.48
17.30
48.88
Qwen-3-4B
48.80
72.60
92.80
89.34
77.06
53.44
55.38
Gemini-1.5-pp
90.62
76.94
94.00
84.96
33.68
54.72
69.44
Gemini-2.0-BB
61.24
C.T-4.1-mini
84.52
76.20
91.80
96.34
69.58
58.74
69.74
GPT-4o
62.32
GPT-to-mini
86.72
78.44
94.36
95.36
75.46
53.98
60.76
ol-mini
89.30
49.24
92.08
96.06
71.98
58.12
71.86
Table 9: StructEval-V Generation Scores (Part 1)
T-Merraid
T-HTML
T--Canvaa
T--Typst
T-Aega
T-Nue
Model
Avg.
Llama-3.1
95.96
9.02
23.38
28.36
$7.90
30.56
54.44
Mita-Llama-3-8B--1e
72.52
29.46
30.74
31.28
Phi-3-mini-1220
92.10
11.12
22.90
35.56
39.84
32.50
44.77
Phi-4-mini-Instruct
97.24
9.30
42
34.72
29.48
28.90
51.62
Qwen-225-7B-lostrrucc
92.92
33.44
30.56
37,90
44.52
53.28
Qwen-3-1B
98.50
13.62
9.92
45.28
29.
54.28
57,00
Gemini-1.5-pro
58.11
Cemini-2.0-flach
99.26
9.66
45.28
29.74
32.46
29.16
53.62
GPT-4.1-mini
99.30
48.46
9.96
48.28
38.44
49.60
64.30
GPT-4o
99.22
36.00
23.91
72.20
40.01
33.51
65.39
Gii--o-mini
99.02
30.50
9.96
41.28
33.66
30.50
60.77
ol-mini
99.41
27.76
65.08
33.52
61.98
Table 10: StructEval-V Generation Scores (Part. 2)
20

---

Published in Transactions on Machine Learning Research (01/2026)
C-JSON
X-JISO
V-JSON
J-YAMU
J.CSV
J-XXL
Model
-Jama-3.1-8B-Imme
92.52
Meta-Llama- Llama-3-3 1aama-3--8-Inssteee InStuen
31.40
48.00
69.24
55.40
90.00
74.00
48.26
Phi-3-mini-L 128k--lnstree
Phi---minn--ussioue
15.42
97.62
89.56
61.90
100.00
100.00
90.70
Qwen-2.5-7PB neettt
31.36
Qwen-3-4B
55.28
100.00
92.84
65.98
100.00
98.00
99.78
Gemini-1.5-ppn
48.14
100.00
40.14
67.14
98.00
100.00
99.78
Gemini-2.00-B
25.72
100.00
69.76
100.00
100.00
99.78
GPT--411miini
55.52
100,00
38.08
69.76
100.00
100.00
99.78
GPT-40
38.56
99.78
GPT-4o-mini
38.52
100.00
73 26
65.98
98.00 00.00
100.00
98.22
o1-mini
58.46
100.00
82.70
100.00
100.00
99.78
Table 11: StructEval-T Conversion Scores (Part 1)
Tomb Toml-JSON
Tomt- PoMt AML
C-XML
X--YAML
K-NML
V-CSVV
C-YAML
Avg.
Mama-3.1-8B-Instrruad
Meta-Llama- Llama-3.a 3-----------stttee
17.28
54.48
38.12
61.90
63.38
36.50
63.
53.65
Phi-3-mini-LB 128k Instruct
Phi-4-mini--Iuee Inntrrit
21.72
60.00
48.28
84.14
86.02
66.22
61.841
72.39
Qwen-2.5-77 2...-22.----nstuue
18.12
81,62
24.16
97,62
78.22
70.86
65.68
62.18
Qwen-3-4B
24.82
94.10
18.68
98.94
96.92
65.08
95.36
81.13
Gemini-1.5-pro
27.14
42.96
47.56
100.00
90.76
71.40
97.36
74.24
Geminni-2.0-faahh
97.36
GPT--4.-mmin 4.1-mini
29.36
59.18
48.36
100.00
100.00
60.82
97.36
75.63
GPT-4o
GPT-4o-mini
29 62
40.20
48.76
98-10
100.00
50.00
97.36
75.37
o1-mini
29.26
88.62
48.36
100.00
100.00
72.40
97.36
81.82
Table 12: StructEval-T Conversion Scores (Part 2)
R-HTML
V-NITM
MD MD-RRact Reaac
ITTL-AA
MD Vue
MPL--TiZZ
Model
Llama-3.1-8 1.iaa-.,1-BB-aaae B.sstuutttt
88.36
81.65
43.23
60.90
36.36
16.26
Meia--lama-3--8 Instruct
86.82
85.23
33.73
52.83
29.52
8.29
Phi-3-mini- Instructt
70.73
73.85
30.80
32.77
27.32
17.15
Phi-4-mini- Instruct
Qwen-2.5-P7 2.--7B--nstsstee
89.29
79.53
31.70
68.67
33,50
26.32
Qwen-3-4B
51.23
Deminni-1---pr
95. 24
91.27
34.83
86 43
30.96
38.89
Cemini-220-faaat
93.02
88.67
32.37
29.30
32.00
17.46
GPT--411miii
95.22
90.12
52.87
81.97
31.96
36.80
CPT-40
95.36
90.55
74.20
87.17
37.56
39.69
ol-mini
95.09
89.65
58.37
87.90
36,50
40.60
Table 13: StructEval-V Conversion Scores (Part 1)
21

---

Published in Transactions on Machine Learning Research (01/2026)
HTML Rlssct
MD MD-HITML
HTML--Vo
V-RReact
MD
R-AAng
Model
Avg.
Llama-3.1-88 1-serrrur
58.28
55.02
72.93
75.73
26.90
85.20
61.15
Meta-Tlama- Maama----B--e Instrrett
84.52
73.11
75.28
62.73
33.10
57.00
56.91
Phi-3-mini- Inlt-Insttuct
Phi-4-mini- Instruct
92.44
57.11
41.05
55.87
26.50
71.20
52.48
2uen-22557BB-nnsttee
85.16
50.87
35.00
84.60
61.43
Qwen-3-4B
90.20
65.31
83.05
68.13
34.50
85.00
65.08
Gemini-11.5ppo
95.28
10.62
86.65
61.00
49.80
85.20
66.39
Gemiti-22.0-laal
96.60
41.04
67.77
28.20
29.20
51.97
CPT-4.1-mini
96.40
88.09
46.28
86.47
49.10
85.20
70.04
GPT-40
GPTT-oo-mini
93.14
86.42
79.75
81.20
19.20
97.80
76.54
Table StruetEval-V Conversion Scores (Part 2)
T Text, C CSV, J JSON, XML, Y YAML, Ang. Angular, MD Markdown, MPL Matplotlib, React, V
Vue.
22

---

Published in Transactions on Machine Learning Research (01/2026)
A.3 Examples of rendered image
3p503 Rotede
Project Snapshor
Oivenkev
leary
(a) Angular
(b) Matplotlib
(c) Mermaid
Newsletter Signup
Enior your email
Lages racelve weekdy updanss
Subscribe
(d) React
(e) SVG
(I) TikZ
mageaay Suepecosee
OuT LWW
co CON COC
0C0 CN
Vega
Figure 7: Example images rendered in STRUCTEVAL. tasks.
23

---

Published in Transactions on Machine Learning Research (01/2026)
A.4 Task Generation Prompt
Sample Prompt
You prompt-design assistant building benchmark items for conversion lasks.
Input Format: (iuput_lypa
Output Format: {output__yy [trpptt___ytt
Your task: Think silently through the checklist and then output a single JSON object with:
"raw_output_,merric"" dot-paths for the expected keys/attributes in the foutput_typee structure
"query": generated imput format (imput_lype) code inside <code>...</cot lags.
Assumed Mapping Rule (state implicitly the paths):
No XML attributes unless absolutely necessary.
If an allribute is required, map it to key prefixed with "e", and include that in dot-paths.
CHECKLIST (INTERNAL DO NOT OUTPUT)
Pick super creative and random domain.
Generate {input pypuut__yppttttt code with:
least two levels of nesting
At least one list inside an object/tlement
Avoid XML attribules where possible; prefer child clements.
Wrap the code <code>.. </code> tags.
Dot-path rules:
JSON / YAML TOML: parent. child, list[o].chit
XML: element.child element. eattr (only if used)
CSV: csv: (not used here)
OUTPUT FORMAT
"raw_ www.outtput_meriic"" ["<dot_pathl>>, "<dot_path2>", ...].
"query": fecode?..<</cdkee
Figure 8: Example task generation prompt
24


Reviews:
[{"review_id": "9VRxuEnzBL", "reviewer": "Reviewer_hMWu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes StructEval, a benchmark for LLM structured data generation (e.g., JSON, HTML). Previous benchmarks focused more on LLM reasoning capabilities and the semantic quality of the generated outputs without much emphasis on whether the generated output conforms to the required format. The authors argue the importance of this benchmark as LLM is being increasingly used in software development. State-of-the-art models like o1-mini still produce subpar performance on this benchmark, and open source counterparts lagging 10 points behind.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides appropriate and convincing evidence regarding the following aspects of the benchmark:\n1. Data curation\n2. Evaluation criteria and metric design\n3. Annotation pipeline\n4. Model evaluation on the benchmark", "audience_yn": "Yes", "audience_explanation": "I am convinced by the importance of the benchmark: LLM must be able to accurately generate structured outputs in order to assist in software engineering.", "requested_changes": "minor in writing:\n- intro paragraph 2 line 4 “in these tasks” repeated twice\n\nrequested additional details and experiments:\n- how was the human experts for VQA questions evaluation gathered? can we have more details about the evaluators?\n- If budget permits, would it be possible to add currect SOTA models from GPT series and Claude series?", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "phGqcL9byn", "reviewer": "Reviewer_N7n6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a new benchmark dataset for large language models, focusing on their holistic ability to deal with structural outputs. The dataset targets two different types of task: generation of structured data from unstructured text and conversion between structured formats. The dataset further consists of two subsets of data modalities: StructEval-T, for text only structures and StructEval-V, for visual outputs.  The authors generate the dataset through a multi-step annotation pipeline, first using a large language model to generate a pool of candidate examples for each data subset, and then refining examples through two pools of manual curation. Each example consists of a question, keywords expected to appear in the output, and (for the StructEval-V dataset) visual question-answer pairs. Language models are evaluated automatically, with scores given by a three-part metric with each part normalized between 0 and 1. The final score is a weighted sum of these individual scores:\n\n- A syntax score evaluates structural correctness of the generated input, and is binary 0 or 1 based on successful execution/rendering of the output. \n- A keyword matching score reflects the proportion of required keywords included in the LLM output. \n- A VQA score for StructEval-V reflects the proportion of correctly answered questions. \n\t\nThe authors show evaluate a range of LLMs on their benchmark, including both open and closed source models. Their results demonstrate significant heterogeneity across different subsets of the task, but show that in general, open source models are outperformed by closed source models, and that conversion tasks are easier than generation tasks (except for closed source models on StructEval-T). Results demonstrate that there is also significant heterogeneity in the difficulty of specific subtasks, with high performance  for generation or conversion tasks between common formats, and generally lower performance when working with less common formats.", "claims_and_evidence_yn": "No", "claims_explanation": "The authors argue that StructEval addresses several key issues which make the performance of LLMs difficult to evaluate on structured data: 1) Data collection Challenges, 2) Evaluation Metric Complexity, and 3) Technical Implementation Barriers. I think that StructEval comes close to addressing these challenges, but I have several concerns around these claims. \n### Benchmark properties\n- One concern I have is that the number of examples within the benchmark is quite small, considering that examples are split between at four separate subtasks with order ~100s of examples per subset. This size is understandable given the complexity of the benchmark, but is around one order of magnitude smaller than some other benchmarks which effectively focus on just one of the subtasks, such as StrucText-Eval (Gu et al. 2024) or SoEval (Liu et al. 2024). It would be useful to know how performance of (even a subset) of LLMs on larger text based benchmarks relates to performance on StructEval. \n- Equations $1,2$ describe one possible weighting of $s_{s,k,v}$ to generate a single score. This choice of weighting appears arbitrary, and downweights syntactic errors relative to others. One could imagine cases where syntactic errors are much more important than keyword errors, and it is not clear if the conclusions of the experiment would hold if a different scoring were to be used. \n\n### Accessibility \n- As stated in the introduction, one of the main challenges of designing a benchmark like StructEval is building a framework which can support evaluation across rendering environments. It is not clear how the authors have ensured that practitioners will be able to evaluate models on the benchmark. I understand the need for anonymity in the review phase, but it is especially important to explicitly address the question of how StructEval can be used in the main text of the paper.", "audience_yn": "Yes", "audience_explanation": "- The variety of task types considered does indeed make for a more holistic evaluation of LLM performance than many existing benchmarks. It is useful to highlight heterogeneity in the performance of LLMs on different types of tasks, which could be a useful avenue for further investigation. \n- Evaluation of structured renderable output via VQA is interesting, and contrasts with many previous approaches which perform evaluation of renderable code artefacts at level of structured output, but not directly in the space of rendered visuals.", "requested_changes": "## Requested changes\nThe following points are critical to securing my recommendation for acceptance of the paper. \n\n### Benchmark properties.\n- SoEval is quite similar to the StructEval-T generation subdataset (without the requirement of flat schemas). Given that SoEval consists of roughly $\\sim3700$ examples, it would be useful to know how well performance of even a subset of the models evaluated on SE-T-gen compares to performance of models on SoEval. Likewise, it would be interesting to know how LLM performance on different subsets of StructEval (or all of StructEval) relate to other structured data benchmarks which are not as holistic in their evaluation. I believe such an analysis would be quite useful to strengthen the case for StructEval as a general-purpose benchmark for structured data. \n- How correlated are subscores $s_{s,k,v}$? Are there many examples where the syntax is incorrect, but the keywords are? If these different subscores are highly correlated, one could imagine that the specific weighting of the different subscores matters very little. \n\n### Description of VQA component. \n- In section 2.1, there is a paragraph describing QA and fairness of the VQA pairs with human experts. What model is used to generate/conver the structured output for the \"correct\" subset of VQA pairs? My understanding is the the VLM judge (GPT-4.1-mini) is responsible for answering questions based on the rendered image. Is it also responsible for generating the image in this case? \n- Questions which are judged unfair should plausibly be removed from the benchmark, but it is not clear that this is the case. Please make this clear. \n- Please include a specific discussion of the benefits and drawbacks of VQA as an evaluation framework vs. evaluation of renderable programming artifacts. \n\n### Accessibility. \n- Please include a clear discussion of how practitioners can evaluate new LLMs on StructEval. I imagine that the paper will eventually link to a code repository, but special care should be taken to ensure that the need to work with different rendering environments (as discussed in the introduction) is not a barrier to use. \n\nThe following points would improve my evaluation of the paper, but are not critical to ensuring my recommendation. \n### Error analysis\n- How correlated are errors on individual samples across different LLMs? Within a certain task (e.g. $\\text{Text}\\rightarrow \\text{TOML}$) are there much harder and easier examples which drive performance, or is there a great deal of variance in errors across LLMs? \n### Minor points\n- Second paragraph:\"in these tasks\" x2\n- Figure 3 formatting: \"Feature Requirements\" should have a newline\n- Please be careful of the use of \\citet vs. \\citep throughout the paper. \n- In Table 6, it appears that delta is the difference between o1-min and Qwen3-4b. The paragraph **Open-Source vs. Closed-Source Models** describes the quantity $\\Delta(\\text{close}_{avg}-\\text{open}_{avg})$. Is this meant  to be described in the table as well? \n- Appendix A is titled \"Example Appedix\".", "broader_impact_concerns": "Given that evaluation of large language models on structured data is immediately relevant to many large scale industrial applications, I believe that a broader impact statement around the implications of this work for (for example) data privacy, climate, and fairness would be useful.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wfx6GJlDXq", "reviewer": "Reviewer_Ljyw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "StructEval assembles a two-part benchmark—StructEval-T for text-only structured formats and StructEval-V for visually rendered code—covering a wide range of structure generation tasks and output formats, with ~2,000 total examples. The paper formalizes a scoring scheme that blends a binary syntax check, a keyword-path structural check, and—only for renderable tasks—a VQA-style visual fidelity check answered by a VLM judge (GPT-4.1-mini). Final scores weight these components differently for text vs. visual tasks, with StructEval-T emphasizing keyword matching, and StructEval-V emphasizing VQA performance. The main experimental result is that strong commercial models still leave headroom on structured outputs, with the paper stating that GPT-4o averages 76.02% and that the best open-source baseline is roughly ten points worse. The dataset is produced through an LLM-assisted synthesis of tasks and metrics followed by human review, and the VQA question set is audited for fairness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "A few elements of the design are genuinely compelling. The VQA judge template is explicit and conservative, and the paper’s “path-rule” structural metric is comprehensive in allowing for dot-paths, wildcards, CSV-column predicates, and XML attribute fallbacks. Most of the core claims are well-supported, although some elements could benefit from additional study:\n\nThe evaluation uses GPT-4.1-mini as both an evaluated model in the leaderboard and as the VQA judge for StructEval-V. Even with a template restricting the judge, this introduces potential bias: if the judge systematically favors some stylistic or structural choices that correlate with its own generations, models that emulate or align with that style could be advantaged. The authors partly mitigate this by auditing VQA items for fairness, and the audit statistics are encouraging, but the circularity risk remains conceptually. It would strengthen the claim if the authors re-ran VQA on an orthogonal VLM and reported agreement metrics, or if they included a small human-scored study for end-to-end outputs. The paper’s own reporting shows 88.92% of sampled VQA questions marked “fair” by human evaluators, and 98.30% accuracy on the fair subset, which is a good start but does not fully close the loop on judge dependence.    \n\nWhile using temperature = 0.0 and greedy decoding is standard for benchmarking, and ensures strict comparability, the structured-output focus makes it plausible that mild sampling (e.g., temperature 0.1–0.5) or pass@k evaluation—common in code benchmarks—could expose variance and robustness issues that greedy decoding masks. Including a small ablation would provide stronger evidence that rankings are stable across reasonable decoding policies.\n\nIn terms of prior work and positioning, the paper is correct that most benchmarks emphasize semantic or reasoning quality rather than strict structural fidelity, and that recent efforts tend to be modality-specific (code-only or text-only). The related-work framing cites such trends directly and motivates the unified metric design that spans both text and visual outputs. That said, the paper could engage more directly with the literature and tooling around constrained decoding and structure-aware generation, because benchmarked performance is tightly coupled to whether one uses schema constraints at inference time. There are now widely used approaches for JSON/XML/YAML conformance (e.g., JSON-schema-guided decoding; regex-constrained sampling; grammar-constrained decoders in libraries like Outlines/Guidance/LMQL; tokenizer-level constraints for brackets and quotes), and there is a recent wave of function-calling / tool-use evaluators that measure exact key-path and type fidelity. Including at least one constrained-decoding baseline would raise the ceiling and tell us whether the observed difficulty is a “model knowledge” gap or a “decoding/control” gap. Even a limited ablation on a text-only subset (say, JSON and TOML generation) using grammar-constrained decoding would be extremely informative to practitioners deciding between model scaling and better inference control for structured outputs. The authors’ own motivation section acknowledges that format conformance is crucial in production pipelines—this is precisely where constrained decoding shines.", "audience_yn": "Yes", "audience_explanation": "I believe this paper would interest a general ML audience, though its core value will be highest for researchers and engineers working on code-as-policy, structure-aware generation, UI synthesis, and data-engineering pipelines. The reason is that it cleanly operationalizes reliable measurement of schema conformance and renderable fidelity under constrained output regimes. The dataset scale is modest but thoughtful, the metric design is pragmatic, and the failure-mode analysis is compelling. The two most important ideas that a broad audience should take away are captured well by these lines: “Having the ability to generate fully structured content is highly useful for many downstream tasks,” and “current models are still lagging on the renderable structured content, especially on less frequent format.” Those are accurate and consequential claims for anyone deploying LLMs in production systems where structured outputs are non-negotiable. I would, however, consider the paper “conditionally impactful”: the ultimate influence will hinge on addressing judge dependence, decoding sensitivity, and the constrained-decoding baseline. If those are strengthened, StructEval can sit alongside reasoning and code benchmarks as a standard battery for model releases and ablations.", "requested_changes": "* See above for experimental suggestions.\n* The introduction states “21 distinct formats,” while the dataset overview and table summarize 18. If both are correct, please explain the discrepancy. \n* Introduction, paragraph 2: duplicate phrase “in these tasks, in these tasks”\n* End of section 3: typo ”The for StructEval-T task…”\n* End of section 3: typo “the final score s in computed”\n* Section 4.1, paragraph 2: typo “close-source”", "broader_impact_concerns": "I do not see major ethical risks beyond standard concerns with LLMs or benchmarks that may encourage over-reliance on scores without addressing semantic correctness.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)