jqBench: a benchmark for reading and editing JSON from natural language and/or examples
Authors: Gust Verbruggen
Chris Parnin
Vu Le
Sumit Gulwani
Venue: ICLR 2026 Poster (2026)
Keywords: JSON
benchmark
code generation
nl-to-code
programming-by-example
Primary area: datasets and benchmarks
TL;DR: We introduce a benchmark for reading and editing JSON data using natural language and/or examples, with a focus on the `jq` tool.

Abstract: We introduce jqBench, a new benchmark for evaluating language models on JSON querying and transformation tasks, where the intent can be given specified using natural language and/or examples.
Whereas jqBench is mainly aimed at using the jq tool, it can be used to evaluate other programming languages that query and/or transform JSON.
Benchmarks are automatically created from two rich sources of data: Stack Overflow discussions (1496 instances with instructions and examples, called jqStack) and the Spider dataset for SQL generation from natural language (859 instances with instructions and JSON Schema, called jqSpider).
We describe and analyze the automated pipeline for benchmark creation, and perform extensive baseline experiments on different models to analyze the complexity and failure modes.
Using implicit feedback, the best model (Opus 4.1) scores 76% on the jqStack benchmarks and 81% on the jqSpider benchmarks.
Additionally, we show (1) that access to the documentation surprisingly does not help, (2) jq lags behind Python, and (3) that automatic feedback (and therefore examples) is crucial.
Besides the challenging benchmarks, we release 13K converted but filtered cases for training purposes.

Published as a conference paper at ICLR 2026
JQBENCH: A BENCHMARK FOR READING AND EDITING JSON FROM NATURAL LANGUAGE AND/OR EXAMPLES
Gust Verbruggen
Chris Parnin
Microsoft Keerbergen, Belgium
Microsoft Ralcigh, USA
Evertruggendmicroosff..coomm
cparninémicrossoft.com
Vu Le & Sumit Gulwani
Microsoft
Redmond, USA
{levu, sumi (g) @mi emicrosoft.com
ABSTRACT
We introduce JQBENCH, new benchmark for evaluating language models on JSON querying and transformation tasks, where the intent can be given specified using natural language and/or examples. Whereas JQBENCII is mainly aimed using the jq tool, can be used to evaluate other programming languages that query and/or transform JSON. Benchmarks
automatically created from two rich sources of data: Stack Overflow discussions (1496 instances with instructions and examples, called JQSTAcK) and the Spider dataset for SQL generation from natural language (859 instances with instructions and ISON Schema, called JQSPIDeR). We describe and analyze the automated pipeline for benchmark creation, and perform extensive baseline experiments on different models to analyze the complexity and failure modes. Using implicit feedback, the best model (Opus 4.1) scores 76% on the JQSTACK benchmarks and 81 % on the JQSPIDER benchmarks. Additionally, we show (1) that access to documentation surprisingly does not help, (2) jq lags behind Python, and (3) that automatic feedback (and therefore examples) is crucial. Besides the challenging benchmarks, we release 13K converted but filtered cases for training purposes.
INTRODUCTION
ISON has become the de facto standard for structured data exchange-poweringg web APIs, databases, event streams, and configuration files-and now underpins modern Al workflows, serving common input and output representation for lange-lannagemmodde (LLM) inference and agentic workflows. common subset tasks involves queries and transformations of ISON representations. which can be performed with tools such as jsonpath or jq.
Consider, for example, jq expression (left) that operates on social media dataset (right) and selects all users with more than 100 followers and extracts the titles of their posts:
.users[]
("users": [["name": "A", "followers"; 42,
select foolowwesss 100)
{name, posts: [. postst[] title]]
"posts": t["title": "X"}]}]}
This short query filters numerical attribute and simultaneously traverses nested arrays while yielding new structure. When given the simple task to "find all elements that are present in both the arrays" and three input-output examples like [C1, 2, 3, 4], [2, 4, 6, 8, 10]] C2, 4], models struggle to generate correct expressions. GPT-5 arrives at
[C. [o][]
Sitem
select(. index(Sitem)]]]
which does not come close to the simple solution [[]] [.[0] .[1]).

---

Published as a conference paper at ICLR 2026
This illustrates both the expressive power of jq and the challenges of generating such transformations from natural language, especially as the constraints grow more complex. Surprisingly, there is no benchmark that jointly captures natural-language prompts and executable JSON queries and transformations.
In this paper, we propose JQBENCH, benchmark for JSON querying, filtering, and transformation from natural language and/or examples, with a specific focus on the jq query language. The flexible and potentially complex nature of JSON data, the variety of signals that can be part of the input specification-llik natural language, examples and JSON schemas-and the expressive yet concise and relatively uncommon nature of jq make JQBENCH an interesting benchmark for different research directions: (1) prompting and agentic workflows using small and large language models, (2) fine-tuning of small language models, or even (3) symbolic inductive programming.
To this end, we collected Stack Overflow questions tagged with jq, as well as NL-to-SQL tasks from an improved version of the Spider dataset, and use automated pipelines to convert them into JQSTACK and JQSPIDER, respectively. From Stack Overflow, our pipeline distills realistic developer problems into machine-checkable tasks by extracting natural-language context, compiling candidate jq expressions and input-output examples, and uses agent to generate and validate multiple test cases. From Spider, we automatically transform relational databases into JSON databases with associated JSON schemas and derive equivalent jq programs, enabling benchmarks that require reasoning over thousands of nested records. Together these sources yield a diverse corpus of 1496 (JQSTACK) and 859 (IOSPIDER) JSON querying and transformation tasks that combine authentic language, rich structure, and automatically verifiable solutions. Additionally, from the JQSTACK creation process, we release 3641 easier tasks and their solutions that can be used in fine-tuning research, both directly and as seed for synthetic data generation.
Experiments on different models reveals that J is sufficiently challenging: Highest baseline of 76% for Opus 4.1 on JQSTACK and 75% on JQSPIDER. Furthermore, the novelty of jq and the unique -rooessoes setting challenge weaker models, while complex JSON operations remain difficult even for stronger ones. We show interesting lessons learned from JQBENCH, including the potential of jq. the importance of implicit feedback based on examples, the "documentation trap" for capable models in agentic loops, and feasibility of JOBENCII as an interesting PBE benchmark.
In summary, we make the following contributions:
1. We present JQBENCH JQSTACK U JQSPIDER, benchmark for generating jq expressions that query, filter and transform from natural language and/or examples that covers complex JSON operations over diverse real-world scenarios.
2. We develop an automated pipeline that extracts authentic tasks from Stack Overflow and Spider, synthesizes and executes jq programs and input examples, and verifies correctness through execution feedback.
3. We perform baseline evaluations on different large and smaller models, understand and analyze properties of both IQBENCH and the current state of language models on jq. Among other experiments, we compare implicit feedback versus explicit feedback (tools), we compare the uncommon jq language versus the very common Python language, and we study the importance of the natural language instruction.
The dataset and evaluation scripts can be found www. www.gt.thhh..... www.ggtthh...c////dddeeeeeutttte The training data is at
2 JQBENCH
This section describes the two collections of problems that make up JQBENCH: JQSTACK (diverse problems from Stack Overflow with input and output examples) and JQSPIDER (problems adapted from the Spider dataset with large inputs and schema). For a primer on jq. we recommend reviewing the manual (jg 2025) and formal specification (Färber 2024).
All prompts and agent tool signatures used in this section shown in Appendix A We use OpenAl's GPT-4.11 1, GPT-5 and Anthropic's 's Claude Opus 4.6 for data generation. Besides the final dataset, release the generation pipeline and all intermediate artifacts, including candidates without
2

---

Published as a conference paper at ICLR 2026
successful conversion and cached model responses, to facilitate further research and development JQBENCH.
2.1 JQSTACK
Each data point (u, 1, 13 in JQSTACK consists of an instruction u, two or more inputs I and one or more jq expressions E.
Conversation
Extract
Convert
Converting comma delimited
D Iistractionn cardidaee esamplo..
"Using odd, and find
achieve the
deliited
intendal
Ado puaugg Koue theaeiig
"Ourpet each element of the top-level amay, compuct
uny of rev deievii wereee sorv
artide to use ueu the oprioo, eq
suporate line."
tod taat2", (T4": 93', Sherss
Pan, codoode O3'
-El Roonn))
".[3" "-e .[]"
(DvA), 1, (n
[[1, 2], 3, (14": 5)]
teees
[C4, 2)x
(G,2) 3
[tt 2).
Ut 21 99 trehetis (10% "twert)
[1,2]
test2
(14%
shate R3
Per:33
(*4":5)
test")
rame
ouggut
Figure I: Overview of extraction and conversion of Stack Overflow conversations to JQSTACK tasks. Components tagged with a star (*)) are part of the benchmark.
2.1.1 CREATION
We create JQSTACK from all Stack Overflow posts tagged with jq four steps: (1) information extraction, (2) annotation, (3) conversion to test cases and (4) filtering. An overview of the extraction and conversion steps the core creation process- shown in Figure
Extraction From each Stack Overflow discussion (question answers) we instruct the model (GPT-5) to generate a jq task by extracting (I) direct quotes from the discussion that capture the key problem and solutions, (2) concise and precise description of the user's intent, (3) all candidate jq expressions that satisfy this intent, and (4) optionally, any provided input and output examples. This yields 15860 raw extractions.
Conversion Given the intent, the candidate expressions and the candidate examples, we then instruct a model (GPT-5 into Opus 4.1) in an agentic loop write jq expression that satisfies the intent. The agent is given two tools: add test(n, i, o) adds a named test n where input i is expected to produce output and run sun_tests(e) (e) runs jq expression compilation and execution results. Running add_test tool twice with the same name will update the test, which the agent can use correct test outputs after reflecting on their results (which happened on 7.5% of conversions). All candidate jq expressions are executed on the final tests to determine successes. This yields 14620 valid conversions with GPT-5 and an additional 923 with Opus 4.6.
all tests, providing
Filtering Many tasks are too trivial to challenge large and capable models, for example, a question to print the value of a field named "text" with a target expression text. We therefore use simple filter to remove tasks solvable by jq generation prompt without interaction at temperature O using GPT-5 (filters 1 11066 cases) into Opus (filters 1913 more cases) keep 1496 challenging cases.
Review Finally, we manually review the dataset, focusing on instances that were never solved by any baseline across any evaluation. These problems were often underspecified, but conversion agent was able to brute-force them by having access to all examples (and target solution). Common types of underspecification are hard-coded constants, adversarial edge cases, JSON schema changes (like ("a": .... ) in all seen examples and ("b": ("a": ...)) in the unseen one), and ambiguity in the utterance. Ambiguity was often fixed (~ 25), other problems were skipped as they became too trivial with proper specification (~ 43).
3

---

Published as a conference paper at ICLR 2026
gpt-5 (r 0.392)
# of ilerations
6 8
Count
Kept
10
10'
6570
3523
False
0 2
-10
True
12 15
18 21 # of pipes
25
28
32
10
544
621
760
412
363
164
opus-4.6 6 (r 0.165)
106
# of ittrrtions
10
38
4 6
100
Count
Extract
Complex
Subset
Transform
Superset
Augment
Keys
10
12 15
18 21
24
29
32
37 43
# of pipes
Figure 2: Distribution of failed, filtered or kept
Figure 3: Relation between of iterations needed to create the task and of the query.
(log scale) per type of task.
# itcrations
8
gpt-5 (r -0.338)
gpt-5
2 6
Count
10²
Similarity Solution
$100
0.0
0.2
0.4
0.6
0.8
1.0
Candidate similarity
opus-4.6
# iteratatoo
opus-4.6 (r -0.287)
2 4 6
Count
10
1000
0.2
0.4
0.6
Candidate similarity
Iteration
Figure 4: Similarity between candidate query and final query in function of # of iterations taken convert.
Figure 5: With each iteration, queries suggested by the conversion argent diverge further from the original candidates.
2.1.2 ANALYSIS
Each task can be classified based on propertics of the input and output JSON. The output can subset (removing values) superset (adding values) of the input, it can be JSON object using only leaf values extracted from the input or using all leaf nodes and some augmented values, it can have the same structure but with some transformed leaf nodes, that fits into none of these boxes. Complex transformation typically extract nodes, transform them, and then build a new JSON object. Based on the suggested tests, Figure shows the number of failed conversions, number of filtered and number of kept tasks. Most tasks involve extracting or filtering (subset) from the input JSON. Tasks that involve changing the structure (extract and complex) are notably harder, as more of them are kept.
be complex transformation
Figure shows the number of iterations needed successfully convert the task and the number of pipe characters in the query. Other proxies for complexity (like rarity of functions, control flow, and nesting depth) show similar relations: is harder to create more complicated benchmarks, even presence of candidate solutions. This result is (partially) explained by Figure 4 which shows how the number of iterations required to create task is lower if the final solution is closer to the candidates. Finally, Figure shows that suggested queries diverge further from the extracted candidates with each iteration: does not only make small syntactic fixes, but really tries to rewrite the expression to obtain correct solution.
4

---

Published as a conference paper at ICLR 2026
2.2 JQSPIDER
Each data point (u, e, d) in JQSPIDER consists of an instruction u, the JSON Schema 8, expression e, and the dataset d.
2.2.1 CREATION
We create JQSPIDER in three steps from the repaired Spider dataset (Yang et al., 2025): : (1) converting each database schema to ISON Schema, (2) converting SQL databases to JSON databases that adhere to the generated JSON schema, (3) converting SQL queries to jq expressions.
SQL Database
JSON Schema
JSON Database
leves aeoiiing
"sailors": I
Sailors
Reserves sid bid day
"sid": by
sid
"salTors": [[ "sid": int, "name": string. "rating": int, "age"t int, "resorves": [[
schema feedback
"nave": "Fugane",
1 syntax feeilhack
*.x "reserves";
"day": "0/12", "boat": : ( (...)
rating age
Boats
"day": string, "boat":
bid
"bid": int, "name": string,
def f(dh):
name
"colon": string
color
}1
Figure 6: Overview of SQL Database to JSON Database conversion. Components tagged with star (*) are part of the benchmark.
JSON Schemas We use the model to convert a SQL schema (columns, column types and foreign keys for each table) to JSON schema. We instruct the model to choose appropriate root table and to leverage nested objects and arrays to represent one-to-one and one-to-many relations. The jsonschema Python package] is used to provide validation feedback. During small-scale empirical testing, we found that LLMs are able to generate more interesting schemas than symbolic heuristics.
JSON Databases We then the model to convert the SQL database to JSON file that adheres to the generated schema. First, we symbolically generate a trivial JSON format that encodes each table separately. Second, we iteratively instruct the model generate Python function that converts this ISON file into new JSON file that adheres to the given schema, using compilation and execution feedback in each iteration. Figure shows overview of this conversion. Out 202 databases, this process succeeds for 197 of them.
Queries Finally, given the natural language instruction, SQL query, JSON Schema and expected output of the SQL query on the original database, we iteratively instruct the model to write jq expression, again using compilation and execution feedback in each iteration. This process is very similar to the conversion step in Figure original SQL queries. This yields 893 benchmark tasks.
[ without the add test tool-the tests are provided by the
2.2.2 ANALYSIS
Figure 7 shows that SQL databases with more tables yield more deeply nested JSON, generally requiring chained jq expressions. Figure g shows that more tables and nesting depth are both indicators of lower conversion rate. 9 that more SQL JOINs correspond more pipe characters ja, confirming the need for long chains.
EXPERIMENTS
We perform extensive experiments to evaluate both our dataset(s) and the current performance of LLMs on writing jq expressions.
5

---

Published as a conference paper at ICLR 2026
13
Nesting depth
11
E- 0.326
Conversion succese nite
9
Conversiont success
15
7
pipes pipes
0.5
0.6
10
5
10
9.4
0.4
4 2
6
8 10
12
14
16
26
0.2
0.2
# of tables
of tables
Nesting depth
0.0
#f JOINS
2.5
Figure 7: # of tables versus nesting depth.
Figure 8: Conversion rate versus # of tables and nesting depth.
Figure 9: JOIN versus pipes.
3.1 IMPLEMENTATION DETAILS AND METRICS
We use the jq Python binding execute expressions as jq. all 1). This causes all results to be wrapped in list, for example, jq. simple examples the prompt (including the one above) and also consider a successful evaluation each prediction all i) == [o] (or vice versa) for the expected output o.
("foo": 1)) == [1]. We therefore provide four
On JQSPIDER, the keys of record-style outputs are ignored, meaning that [["a": 1, "b": 2}] == [[ "x": 1, "y": 2)]. Unless the original query mentions ORDER BY, we also ignore order.
We use all but one of the tests as input-output examples. This strategy is common in programming- by-example (Li & Ellis 2024).
3.2 BASELINES
We evaluate different models on JQBENCH in different (agentic) settings that leverage possible reward signals for feedback. The default setting (V) is based SELF-DEBUG (Chen et al. 2023) and simply provides feedback based on the available information. Compilation feedback is always provided, exccution feedback is provided when inputs are available, and test results provided when outputs are available. Additionally, we use the following tools in baseline experiments:
</> ce:e(e: str, i: str) runs expression output. This tool can be used even when inputs are not available, as the agent can synthesize its inputs that it (thinks it) knows the output to. On JQSPIDER, for example, when only a JSON schema is provided because the whole file is too large and there is therefore no way validate some outputs, it can synthesize smaller examples test hypotheses.
serialized JSON object and prints the
Q search docs (k: str searches (parsed) version of the documentation for keywords k and returns the name of all documentation sections that matches any of the keywords.
print docs(s: str[]) prints the sections of the documentation in Markdown format. Examples of documentation sections are shown in Appendix and we release the parsed documentation.
In addition to the jq solvers, we compare its performance against using Python to solve JQBENCH.
The model is instructed return Python function that accepts single argument (the JSON object). On JQSPIDER, we instruct to return a value or a list of records to match the output format of the expected jq expression. The structure of the Python prompt mimics that of the jq prompt as close as possible (see Appendix A).
We all setups over maximum of eight iterations temperature 0.
3.3 RESULTS
Table I and Table 2 show performance and configuration statistics of different configurations on JQSTACK and JQSPIDER, respectively. The following paragraphs describe these results in more detail.
httpss//ppppi.og///ojje/////

---

Published as a conference paper at ICLR 2026
MODEL
CONFIG
# FEEDBACK
TOOLS
PERFORMANCE
Q
Q
B
</>
c?
opus-4.1
Jq
5.66
0.97
0.90
0.54
gpt-5
Jq
0.64
0.15
gpt-5-mini
Jq
2.94
0.85
0.69
0.22
gpt-5
Jq
0,48
0.80
1,31
0.98
0.92
0.68
0.11
opus-4,1
Jq
0.09
0.30
0.67
0.99
0.95
0.76
0.30
gpt-S-mini
Jq
0.69
1.12
1.60
0.96
0.86
0.59
0.10
phi-4
Jq
5.06
1.26
0.83
0.40
0.24
0.10
0.02
opus-4.1
Jq
V
V
0.52
0.11
0.30
3.60
2.76
0.51
0.46
0.31
gpt-5
Jq
V
V
0.42
0.68
1.11
0.68
0.18
0.98
0.91
0.68
gpt-5-mini
Jq
0.61
1.03
1,49
0.74
0.54
0.95
opus-4,1
py
0.00
0.04
0.25
1.00
0.98
0.83²
0.69
gpt-5
Py
0.00
0.06
0.38
1.00
0.99
0.82
0.66
gpt-5-mini
Py
0.01
0.20
0.70
0.79
0.58
4
Py
0.20
1.13
1.74
0.95
0.82
0.54
0.34
language,
QB documentation mode,
implicit mode, (c?) compile, (e?) execution, (v?) value, </> test
expression.
Best overall jq performance.
Best overall performance.
Table 1: Results for JQSTACK under different configurations.
MODEL
CONFIG
FEEDBACK
TOOLS
PERFORMANCE
c?
c7
C?
c?
V?
opus-4.1
Jq
1.72
1.00
0.96
0.77
gpt-4,1
Jq
0.83
0.99
0.94
0.75
gpt-5
Jq
0.00
0.98
0.91
0.72
opus-4.1
Jq
0.03
0.08
1.00
1.00
0.81
gpt-4.1
Jq
0.06
0.12
1.00
1.00
0.79
g-t-4.1-mmni
Jq
0.01
0.20
1.00
1.00
0.77
gpt-5
Jq
0.01
0.11
1.00
1.00
0.78
phi-4
Jq
2.00
1.28
0.84
0.74
0.43
opus-4.1
Python
0.00
1.00
0.99
0.78
gpt-4.1
Python
1.00
0.99
0.79
8-t-4.1-mini
Python
1.00
0.99
0.76
gpt-5
Python
1.00
0.99
0.80
A language, expression.
implicit mode, (c?) compile, (e?) execution, (v?) value, </> test
Best overall performance.
Table Results on JOSPIDER.
Contrasting JQSTACK and JQSPIDER. The benchmarks reveal notably different behavior and therefore facilitate different areas of research. JQSTACK, which draws from real Stack Overflow questions, exhibits wider spread of model performance: value-match scores range from roughly 10% (Phi 4) to 76% (Opus 4.1). In contrast, strikingly low variability: all models except for Phi 4 are in the 72%-814 range, and even Phi achieves 43%. JQSTACK relies more on deep knowledge of jq. including many built-in functions (116 versus 56 different operators and functions in solutions) and the ability define custom operators (99 versus 2 tasks where solution defines custom function). JQSPIDER relies less on deep knowleds of different jq operations and more on longer chains of piping map and select operations together (median of 4 pipes per task versus 3 for JQSTACK).
from Spider database queries-showe
Language novelty as a performance bottleneck. The difference in performance between Python and jq JQSTACK highlights that models often understand the task, but some do not know how to express solution using jq. This is especially visible for smaller models (54% - 10% on Phi 4) but GPT-5 suffers from the language bottleneck (82% - 68%). Phi 4 getting lower value feedback rates (v?) compared to GPT models is due to the fact that it does not obtain an executable expression within 8 iterations. Opus 4.1, which is known for its strong coding performance, suffers barely any
7

---

Published as a conference paper at ICLR 2026
MODEL
CONHIGURATION
FEEDBACK
# TOOLS
PERFORMANCE
QB
C?
e?
V?
Q
B
</>
c?
e?
V?
v@I
opus-4.1
Jq
0.08
0.31
0.71
0.99
0.95
0.71
0.33
gpt-5
Jq
V
0.38
0.74
1.37
0.98
0.91
0.63
0.15
gpt-5-mini
Jq
V
0.53
1.10
1.87
0.96
0.85
0.53
0.14
opus-4.1
Py
0.00
0.04
0.27
1.00
0.97
0.76
0.64
gpt-5
Py
0.01
0.04
0.48
1.00
0.98
0.76
0.60
gpt-5-mini
Py
0.01
0.15
1.09
1.00
0.97
0.69
0.50
оe language,
QB documentation mode, implicit mode, (c?) compile, (c?) exccution, (v?) value, </> test
expression.
Table 3: Results for JQSTACK using only examples (no NL) under different configurations.
Control Flow
Cumulative Value Match
1.00
Function Definitions
0:11
0.11
6:34
0.10
0.2
opus-4.1
gpt-5
Function Rarity
accuracy
0.75
Objects Arrays
Nesting Depth
0:1
4.00
6.11
6:8
0.12
0.12
0.1
0.50
phi-4
gpt-5-mini
Pipes
Unique Functions
0:02
0.09
0.17
0.11
-0.0
0.25
Variable Bindings
opus-4.1
gpt-5
gpt-5 S-mini
phi-4
0.00
2
3
4
7
8
Iteration
Figure 10: Heatmap of performance drops for different complexity proxies.
Figure 11: Increase of cumulative value match over iterations with implicit feedback.
performance loss (-5%). This is reinforced by its low feedback rates: only 1% of jq expressions failed to compile, and only 5% of compilable expressions failed to execute. These results highlight how JQBENCH is useful testbed for studying how models learn unfamiliar grammars (Cassano et al.) 2024
Zhang et al. 2025).
Freedom to test is fatal. Moving from implicit feedback to letting the model use the run_com code tool sees stark decline in performance: -53% GPT-5 and even -22% on Opus 4.1. We identify four key reasons why letting models explicitly ask for feedback (based on the examples it sees in the prompt) is not working as expected. (1) Not using any tool calls: GPT did not use any tools in 78% of tasks. (2) not (properly) leveraging the provided examples: GPT-5 27.8%) and GPT-5 mini (28.2% 2%) have significantly lower input coverage than Opus 4.1 (68.5%). (3) Ignoring incorrect outputs: GPT 5 (33.8%), GPT 5 mini (44.3%) and Opus 4.1 (19.9%) all use an example and get value feedback, yet propose wrong solutions. (4) Failing to generate correctly serialized ISON object for inputs or fabricates inputs: GPT-5 (20.2%). Examples are predicting ["foo": "bar" or repetitively predicting the target string "1, "1,2N33,4 4" instead of the escaped version "\"1, 2
Feedback is crucial. Table shows the value match after iteration 1, and Figure shows the cumulative value match after each iteration. All models require a significant amount of feedback. the effect of which diminishes after 4 iterations (Opus 4.1) or 7-8 iterations (all others). Figure further details the type feedback that is given after each iteration. Opus 4.1 understand the syntax better and is therefore able focus the values much sooner. Stagnating performance is due the main failure mode: overfitting on the examples (even if the utterance is explicit about generality, such "objects at any depr"). The smaller Phi model struggles to generate queries that compile, even after eight iterations, indicating promising direction for fine-tuning on the 13K filtered cases.
What makes complexity? Figure I0 shows the performance drop for different complexity proxies: # of (control flow, function definitions, object and array initializations, pipe characters, variable bindings, unique functions), average function rarity and nesting depth. The performance drop is computed by diving result across the median complexity proxy and computing the difference
8

---

Published as a conference paper at ICLR 2026
Fraction of Problens
opus-4.1
gpt-5
gpt-S-mini
Compile
Exccute
Valtes
Correct
0.0
Iteration
Iteration
Itemation
Iteration
Figure 12: Feedback over iterations for different models.
Value Match Rate
1.0
Model
gpt-5
0.5
gpt-5-mini
opus-4.1
0.0
phi-4
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q2 Q3 Q4 Dist. from original
Iterations (gpt-5)
Itcrations (claude)
Figure 13: Value match in function of number of iterations needed to create task (left and middle) and in function of distance from the original candidate solution (right).
between the value match for "simple" and "complex" side. Opus 4.1 only affected by the # of function definitions, which are typically required in challenging and nested problems. Function rarity not a problem.
Contamination or complexity? Figure 13 shows the value match in function of the iterations taken to build each task for GPT 5 (left) and Opus 4.6 (middle), well as the distance of the solution query to the candidate query (right). Tasks that are harder to create are harder to solve. Except for the completely changed sieriesswhich were also harder create and thus likely hard tasks in general-ther there is no relation between the distance to the candidate solution, indicating that contamination not an issue.
The documentation trap. Opus 4.1, which the most proficient at jq according to JQBENCH, performs worse when having access to the documentation (76% - 31%). The primary cause is getting stuck loop of requesting documentation, rather than solving the problem. This is can be observed by the 5x documentation request rate by Opus 4.1 compared to next most eager model (GPT mini). After a certain level of proficiency, using SELF-DEBUG pays off more than doing retrieval-augmented generation over the documentation (Zhou et al. 2023
JSON by example. Omitting the natural language instruction converts each task into pogramming--y--examee
(PBE) task, where the goal is to learn program such that p(i) = o for all example pairs (i, o). PBE is popular area of research
both symbolic (Gulwani 2011 Cropper
2019) and neural fronts (Chen et al., 2018 Shi et al. 2022) with recent attention to LLMs (Li & Ellis
2024). When instruction simply states that "the query should match the following (input JSON
outpul JSON) examples," the value match changes as 76% + 71% (Opus 4.1). 68% - 63% (GPT 5) and 59% -> 53% (GPT 5 mini). Whereas significant decrease- the instruction expected to
provide strong signal-these results indicate that JQSTACK poses an interesting PBE benchmark.
RELATED WORK
4.1
NL-TO-CODE BENCHMARKS
The closest related benchmarks to JOBENCH DoCSPIDER Özer al. 2025) and JSON-
SCHEMABENCH (Geng et al.) 2025), which both target langual interfaces for document or schema-centric ISON tasks, but with different emphases. DOCSPIDER adapts the Spider text-to-SQI.

---

Published as a conference paper at ICLR 2026
dataset document databases by converting relational data into MongoDB collections and pairing natural-language questions with MongoDB queries. Like DOCSPIDER, we also translate SQL to another representation (iq expressions). However, JQBENCH additionally draws real-world Stack Overflow questions and a wide variety of organically shaped JSON inputs paired with jq expressions. This yields tasks that span ad-hoc filtering, restructuring, and rich transformations across highly diverse and variably nested JSON, going far beyond the relatively homogeneous datasets that underpin POCEPIDER, JSONSCHEMA A ENCH evaluates constrained decoding methods for reliably generating ISON outputs that comply with complex schemas. Like JQBENCH, it can be used to assess whether language models respect schema constraints and reason about schema adherence.
Other benchmarks combine natural language with domain-specific expression languages, such as MONGOSH query expressions (MongoDB Education Al 2025) or Vega-Lite visualization specifica- tions (Luo et al.
2021
Finally, several benchmarks target NL-to-CLA programs (jq is first and foremost a CLI tool). Terminal- Bench (Team] 2025) is benchmark for evaluating the ability of agents to operate in terminal environments, including tasks such as building an initramfs for a kernel, but targets a small number of complex, multi-step aamen-dadiiittttionn tasks, with limited data manipulation tasks. NL2Bash (Lin et al. 2018) contains 12K one-line Linux shell commands such as top -p $ (pgrep -d',' http) mined from Stack Overflow posts. It only contains jq commands, however.
4.2 BENCHMARK CONSTRUCTION
Constructing high-quality benchmarks is challenging. For example, Yang et al. (0255) found that the widely used Spider dataset (Yu et al. 2018) contains over 30% incorrect NL-to-SQL mappings, highlighting how manual curation can introduce substantial errors. It tempting to avoid these issues by using purely synthetic data; however, Fürst et (2024) finds model robustness can suffer when datasets omit authentic user queries.
Rescarchers have drawn on several approaches for mitigating issues in benchmark construction. Benchmarks, such as ODEX, draw upon natural language queries from Stack Overflow to build NL-python benchmark, but with manual test case construction DonstuuctionWWang et al. 2023). BIGCODEB ENCH (Zhuo et al. (025) used ODEX seed, to synthetically generate more (instruction, code) solutions in an LLM + human annotation loop. Early work used weak supervision, learning logical forms from question-answar pairs without full annotations (Berant et al. 2013). More recently, researchers have leveraged techniques that include program execution-basee filtering, self-consistency checks (Wang
et al. 2023), and LLM-driven self-refinement (Madaan et al. 2023
5 LIMITATIONS
JQBENCH has few limitations. We currently supports only a single JSON input per task, and do not handle multiple-input or multi-file jq programs, though future extensions could relax this restriction better capture real-world multi-source workflows. The automatic nature of JQBENCH construction, which relies on an LLM to generate solution from rich input signal, may have limited the number of challenging tasks in the dataset. We can expand any remaining failed automatic conversions with human annotations and stronger models. Based on the English data sources, JQB ENCH is available in English only.
CONCLUSION
We present JQBENCH, an automatically constructed benchmark for evaluating language models on JSON querying and transformation tasks. The benchmark includes diverse and challenging real-world problems drawn from Stack Overflow (JQSTACK) and Spider (JQSPIDER) datasets, and supports both NL and PBE tasks in low-resource language setting. A baseline set of experiments reveal novel insights into effects of output language used for inference, potentially adverse effects of tools in agentic workflows, struggles with novel languages in small language models. Finally, our benchmark will enable future research in execution- -uideed decoding, exploratory data analysis, language novelty, and tool proficiency.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from snestion-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013. URL htttps://aclanthoolce org/ 1160
Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Car- olyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha. Knowledge transfer from high-resource to low-resource programming languages for code llms. Proceedings of the ACM on Programming Languages, 8(00PSLA2):677-008, 2024.
Xinyun Chen, Chang Liu, and Dawn Song. Execution -uideed neural program synthesis. In Interna- tional Conference on Learning Representations, 2018.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. The Twelfth International Conference on Learning Representations, 2023.
Andrew Cropper. Playgol: learning programs through play. In International Joint Conference on Artificial Intelligence, 2019.
Michael Färber. formal specification of the jq language, 2024. (tL/ittps//arrvv.or//aa
2403. 2403.20132
Jonathan Fürst, Catherine Kosten, Farhad Nooralahzadeh, Yi Zhang, and Kurt Stockinger. Evaluating the data model robustness of text-to-sql systems based on real user queries, 2024. URL. https wwwplv.orggbbb////// 08349.
Saibo Geng, Hudson Cooper, Michal Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. Jsonschemabench: A rigorous benchmark of structured outputs for language models, 2025. URL
10868
Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. In Principles of Programming Languages, pp. 317-330, 2011.
jq. jq manual (version 1.8).
2025. Accessed: 2025-09-24.
Wen-Ding Li and Kevin Ellis. Is programming by example solved by lmms? Advances in Neural Information Processing Systems, 37:44761-44790, 2024.
Xi Victoria Lin, Chenglong Wang. Luke Zettlemoyer, and Michael Ernst. NL2Bash: corpus and semantic parser for natural language interface to the linux operating system. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (cds.), Proceedings of the Eleventh International Conference Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European or
Language Resources Association (ELRA). URL
Yuyu Luo, Nan Tang, Guoliang Li, Chengliang Chai, Wenho Li, and Xuedi Qin. Synthesizing natural language visualization (nl2vis) benchmarks from n12sql benchmarks. In Proceedings of the 202/ / International Conference on Management of Data, SIGMOD '21, pp. 1235-1247, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 1781440333431. doi: 10. 10.1145/344016.3445222
URL /https://dol: org/1e. 0011/14/44455 34485/344116.... 3457261
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023. URL /ttpsstss///rxiv http:/wwwwwwror///////ee/ 17651
MongoDB Education AI Natural language to mongodb shell (mongosh) benchmark. ct/
Hugging Face Dataset, 2025. URL: http::///gggggaeeeeoooaatttt//// hobteee/e/aaatttttoonnooo-----e
naturall -onguage-----moonee
Kensen Shi, Hanjun Dai, Kevin Ellis, and Charles Sutton. Crossbeam: Learning to search in
bottom-up program synthesis. In International Conference Learning Representations, 2022.
11

---

Published as a conference paper at ICLR 2026
The Verminal-Benchs Team. Terminal-bench: A benchmark for ai agents in terminal environments. April 2025.
https://github. htttss///gthubb..cm//uueeeeessss Latluudd-inssttttttttmmmnalnlll
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution- based evaluation for open-domain code generation. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023, U /tttps://ppenreeee
Yieun Yang, Zhaoguo Wang. Yu Xia, Zhuoran Wei, Haoran Ding, Ruzica Piskac, Haibo Chen, and Jinyang Li. Automated validating and fixing of text-to-sql translation with execution consistency. Proc. ACM Manag. Data, 3(3), June 2025, doi: 10. 10.1145/3725271. URL https:://ci..... org/10 1145/3725271
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
Jipeng Zhang, Jianshu Zhang, Yuanzhe Li, Renjie Pi, Rui Pan, Runtao Liu, Zheng Ziqiang, and Tong Zhang. Bridge-coder: Transferring model capabilities from high-resource to low-rest low-resource programming language. Findings of the Association for Computational Linguistics: ACL 2025, pp. 10865-10882, 2025.
Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig. Doeprompting: Gen- erating code by retrieving the docs. In The Eleventh International Conference on Learning Representations, 2023.
Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari. Imam Nur Bani Yusuf, Haolan Zhan, Junda He. Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng. Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations. 2025. / wtt.:ne///oommmiddy/yrrcoooo
Arif Görkem Özer, Recep Firat Cekinel, Ismail Hakki Toroslu, and Pinar Karagoz. Docspider: dataset of cross-domain natural language querying for mongodb. Natural Language Processing, pp. 1-32, 2025. doi: 20..7//l..2224..33
12

---

**Command**; jqq foo? Published as a conference paper at ICLR 2026
A PROMPT
All prompts are a minor variation on the following. The examples are specifically kept trivial: we are testing the model and not our ability to write prompt.
You are a helpful assistant that helps users write jq queries. Wrap the suggested jq query in <jq></jq> tags without surrounding quotes.
jq Details
The jq will be executed using the Python bindings for jq with the input)" function.
This causes the output be wrapped in JSON 1ist, even if the result is single value.
Note that you should not provide this code: you should only provide the jq expression itself.
Some examples:
<jjq. foo</jq> is executed as """.all("......."""""""""" foo", ["foo": : 13) [1]
<jq>(b: a + 1}</jq> is executed as
[j<</[0]</ja> is executed as "jq.: .a 13", ("a": 13) = =[[bb: 2}]"
[["a": 1}, {"b": 2)]) == [["a": 1)]
(Cg)map(((0)<</pp) is executed "qq.all"mapp. [[1, 2], [3, 41]) [[1, 311
B DOCUMENTATION
B.
addpath.me md
# "adppathpstthrrayy"
Ensures path exists, creating objects/arrays; returns modified input.
## Example 1
**Command**: bjq "addpath(["ar" "b"])"
**Input*:: "(" **Output** "(""":[(""""uu111""
B.2 ortiionll-bjeee--ideeiiiere-eee-
foo?"
Optional object identifier index. Like "oo""" but suppresses errors when the input is object; produces "null" instead (and when iterated inside arrays, missing + values simply vanish if further filtered).
#2 Example 1
**Input**; "("foo": 42)*
**Output** "22"
## Example
**Command**: bjq '. foo?
**Input**: "("bar": 1)"
*#Output*:; "null"
13

---

Published as a conference paper at ICLR 2026
Example 3
**Command**; bjq '[.0] .a?]"*
**Input**; '[O. ["a":1}]"
**Output**: [[1]]
14


Reviews:
[{"review_id": "c7bFm8ZexL", "reviewer": "Reviewer_JYjj", "summary": "This paper introduces JQBench, a new benchmark for evaluating systems that translate natural-language descriptions and examples into data-transformation programs written in `jq` (a lightweight yet expressive JSON processor). Given the ubiquity of JSON as a structured data format, the release of such a benchmark is both timely and valuable for research at the intersection of program synthesis, code generation, and natural-language interfaces.\n\nJQBench comprises two complementary datasets: \n\n- JQStack, created by curating transformation problems from Stack Overflow posts, where realistic developer questions are distilled into machine-checkable tasks. The authors extract natural-language context, compile candidate `jq` expressions and input–output examples, and automatically validate them using an agent-based test generator.\n- JQSpider, adapted from the Spider text-to-SQL dataset, reformulated here to express natural-language–to–JSON-transformation problems rather than SQL query generation.\n\nIn total, JQBench includes 751 JQStack and 893 JQSpider tasks, plus 3,641 easier subtasks derived from JQStack examples. Empirical evaluation shows the benchmark’s difficulty: even the strongest models, such as Claude 4.1, achieve only 77 % accuracy on JQStack and 81 % on JQSpider, highlighting substantial room for progress. The authors also report interesting diagnostic findings such as reduced performance when models have access to external tool feedback, and the observation that the novelty of the `jq` language itself poses a primary bottleneck.", "questions": "1. How is task difficulty distributed within JQBench? Are there systematic ways to characterize easy vs. hard transformations?\n2. Given that JQBench currently supports only single-JSON inputs, have you considered extending it to include multi-step or compositional transformations that require reasoning over multiple JSON structures or files?\n3. Do you plan to extend JQBench to other transformation languages or to cross-format tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- JQBench fills a clear gap, as despite JSON’s dominance as a structured-data representation, there are currently no benchmarks that target program synthesis for JSON transformations.\n    \n- The pipeline for JQStack, based on curated Stack Overflow problems with natural-language context and executable test cases, makes the benchmark realistic and reproducible.\n    \n- The paper documents the data-collection pipeline, validation process, and statistics, supporting reproducibility and future extensibility.\n    \n- The authors evaluate a number of models and provide thoughtful analyses of performance bottlenecks, including surprising effects such as decreased performance when models use external tools.\n    \n- The dataset’s release is likely to stimulate follow-up work on code synthesis and model grounding for low-resource programming languages.", "weaknesses": "- The paper’s primary contribution is the benchmark itself; there is little methodological or theoretical innovation beyond dataset construction.\n    \n- While model comparisons are informative, the analysis could be strengthened by deeper ablations, e.g., performance versus task complexity, or qualitative error analyses that categorize common reasoning failures.\n    \n- Although JQBench is diverse, the total number of problems remains modest compared to large-scale synthesis benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SmE32QlbYC", "reviewer": "Reviewer_1PiN", "summary": "The paper introduces jqBench, a new benchmark for testing language models on JSON querying and transformation tasks using natural language or examples. jqBench is automatically generated using Stack Overflow and Spider dataset.", "questions": "Q1. What are the main failures observed during benchmarking?\n\nQ2. Any human checks on the quality of the data?\n\nQ3. How will it change if using BIRD dataset which is more complex?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Strengths \n\nS1. The standardization using JSON schema + jq can be very significant. And this paper propose such a benchmark in the area \n\nS2. The pipeline automates the dataset generation with task translation+ react style verification.", "weaknesses": "Weaknesses\n\n- The justification of the significance of jq operations is not enough. The comparison between the methods using table schema + SQL queries and those using JSON schema + jq is missing. Will jq outperform or is on-par with SQL generation while have the benefits of universal representation? What are the benefits of this translation.\n\n- The tasks are mostly code generation. How about other tasks that highly rely on doc understanding, like RAG and natural language Q&A, how general the idea could be. \n\n- The volume is not a big, less than 2000 tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gaVyfiBo93", "reviewer": "Reviewer_Vcr8", "summary": "This paper introduces JQBENCH, a new benchmark designed to evaluate the ability of large language models (LLMs) to query and transform JSON data. The tasks can be specified using natural language, input/output examples, or both. The benchmark is novel in its focus on the jq query language, a powerful but specialized tool.", "questions": "1. The paper mentions that \"In contrast, JQSPIDER—derived from Spider database queries—shows strikingly low variability: all models except for Phi 4 are in the 72%–81% range.\" For my own suggestion, I think the Spider dataset has had potential data contamination issues since its creation. Therefore, I recommend using the BIRD [1] dataset and even its LiveSQLBench [2] variants instead. This switch might bring about some new insights.\n2. In the Spider dataset, certain databases contain tables that lack \"value\" (i.e., have no actual data entries). How would you handle this scenario?\n\n\n[1] https://bird-bench.github.io/\n[2] https://livesqlbench.ai/", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a highly relevant and practical problem. As LLMs are increasingly used in agentic workflows and data processing pipelines, their ability to handle JSON—the de-facto standard for data exchange—is critical. Focusing on jq is a novel choice that tests a model's ability to learn a concise, powerful, and non-mainstream domain-specific language.\n2. The creation of two separate benchmarks (JQSTACK and JQSPIDER) is a major strength. JQSTACK provides \"in-the-wild\" problems from real developers, ensuring the tasks are diverse and practically relevant. JQSPIDER provides structured, complex, and deeply-nested queries, effectively testing a model's logical reasoning and ability to chain operations (mapping JOINs to jq pipes).", "weaknesses": "1. How do you handle complex multi-table reference scenarios in a database when building jq-spider set. For example, what is the core principle for selecting the root table? \n2. A key challenge with JSON objects lies in their nested structure—an issue particularly prominent in real-world use cases like agent function calls. However, the paper does not provide detailed analysis of this aspect. In particular, it fails to explore critical questions: whether the depth of nested structures impacts model performance, and if so, how.\n3. The presentation of Tables 1 and 2 is confusing, with several unclear elements. For example, some accuracy values exceed 100%—a result that requires explanation, as accuracy is typically constrained to a 0%–100% range. Additionally, the entry for \"gpt-4.1-mini\" is split across two lines, which disrupts readability. Further questions arise from the data itself: why does GPT-5 exhibit performance numbers similar to GPT-4? And under certain configurations, why does GPT-4.1-mini outperform both GPT-5 and GPT-4.1?\n4. The automated pipeline relies heavily on GPT-4.1 to generate JSON schemas from SQL and to translate SQL queries into jq expressions. This could introduce a bias where the \"correct\" jq solutions in JQSPIDER reflect GPT-4.1's specific style of jq programming, which may not be the most optimal or human-idiomatic solution.\n5. This paper lacks a detailed error analysis of different models. For example, the paper states the cause is the model \"getting stuck in a loop of requesting documentation,\" but doesn't deeply analyze why. Is this a failure of the agent's reasoning, a problem with the RAG prompt, or an issue with the quality/structure of the provided documentation? A few qualitative examples of these failure loops would significantly strengthen this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)