SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables
Authors: Sungho Park
Jueun Kim
Wook-Shin Han
Venue: ICLR 2026 Poster (2026)
Keywords: Table-Text Question Answering
Multi-hop Question Answering
Benchmark Generation
Large Language Model
SQL Query Generation
Provenance-based Refinement
Primary area: datasets and benchmarks

Abstract: Real-world Table–Text question answering (QA) tasks require models that can reason across long text and source tables, traversing multiple hops and executing complex operations such as aggregation. Yet existing benchmarks are small, manually curated—and therefore error-prone—and contain shallow questions that seldom demand more than two hops or invoke aggregations, grouping, or other advanced analytical operations expressible in natural-language queries. We present SPARTA, an end-to-end construction framework that automatically generates large-scale Table–Text QA benchmarks with lightweight human validation, requiring only one quarter of the annotation time of HybridQA. The framework first constructs a reference fact database by enriching each source table with grounding tables whose tuples are atomic facts automatically extracted from the accompanying unstructured passages, then synthesizes nested queries whose number of nested predicates matches the desired hop count. To ensure that every SQL statement is executable and that its verbalization yields a fluent, human-sounding question, we propose two novel techniques: provenance-based refinement, which rewrites any syntactically valid query that returns a non-empty result, and realistic-structure enforcement, which confines generation to post-order traversals of the query graph. The resulting pipeline produces thousands of high-fidelity question–answer pairs covering aggregations, grouping, and deep multi-hop reasoning across text and tables. On SPARTA, state-of-the-art models that reach over 70 F1 on HybridQA or over 50 F1 on OTT-QA drop by more than 30 F1 points, exposing fundamental weaknesses in current cross-modal reasoning. We will release the benchmark, construction code, and baseline results to spur progress toward robust, realistic Table–Text QA models.

Published as conference paper ICLR 2026
SPARTA: SCALABLE AND PRINCIPLED BENCHMARK OF TREE-STRUCTURED MULTI-HOP QA OVER TEXT AND TA- BLES
Sungho Park, Jueun Kim, Wook-Shin Han
POSTECH, Pohang, Republic of Korea {shpark, jekim, wshan @cblab. ccblab.postech. ac. kr
ABSTRACT
Real-world Table-Text question answering (QA) tasks require models that can reason across long text and source tables. traversing multiple hops and executing complex operations such as aggregation. Yet existing benchmarks are small, manually curated- and therefore error-prone- and contain shallow questions that seldom demand more than two hops invoke aggregations, grouping, or other advanced analytical operations expressible in natural-language queries. We present SPARTA, end-to-end construction framework that automatically generates large-scale Table-Text QA benchmarks with lightweight human validation, requiring only one quarter of the annotation time of HybridQA. The framework first constructs a reference fact database by enriching each source table with grounding tables whose tuples are atomic facts automatically extracted from the accompanying unstructured passages, then synthesizes nested queries whose number of nested predicates matches the desired hop count. To ensure that every SQI. statement is executable and that its verbalization yields fluent, human-sounding question, we propose two novel techniques: provenance-ben based refinement, which rewrites any syntactically valid query that returns non-empty result, and realistic-structur enforcement, which confines generation to post-order traversals of the query graph. The resulting pipeline produces thousands of question answer pairs covering aggregations, grouping. and deep multi-hop reasoning across text and tables. On SPARTA, state-of-the-art models that reach over 70 Fl on HybridQA over 50 Fl on OTT-QA drop by more than 30 F1 points, exposing fundamental weaknesses in current cross-modal reasoning, Our benchmark, construction code, and baseline models are available
INTRODUCTION
Table-Tex: Text QA has emerged fundamental challenge in building robust question answering (QA) systems capable of operating across heterogeneous data modalities (i.e., text and tables)(Chen al. (2020a b 2021 ;
Zhao al. )2 Zhu et al. (2021). Such task is particularly evident in scenarios where textual descriptions and table entries originate from one or more sources (e.g., textual information and tables multiple Wikipedia pages) and must be jointly analyzed to arrive at the correct answer. While single Wikipedia page often contains both text and tables, it is not unusual for relevant information to span multiple pages or documents, necessitating cross drossmdt retrieval and the effective integration of disparate information.
significant limitation of existing Table-Tex. QA benchmarks is that human annotators manually construct them Chen al. (2020a b) 2021) 1): Zhao et al. (2022); Zhu et al. (202T), resulting in fundamentally flawed benchmark designs that hinder comprehensive system evaluation.
"Corresponding author.

---

Published as a conference paper ICLR 2026
height sereng playee
XD
you Lemun polce cos ou up Diaae 17700 S.M.P. ETPPL
New
hige
Miren Lepaer
Farts lrealo slrved 33 mái and fed point...
fonds
(4) Sta-dtructuudd Beasoning
playns
pertoeed 1990,
garee??
Pactan
281
Heght
Hele
Feethe
Heyer hame
(lected recorted ronding rimerdt and sldt
Allerled
Silleded) Mrcidd dobbl dodbl of peres rowernti
Chan Ressoning
[Ouetters aital noxton Leneros enouge serviion more thet 300 rain lercaaa lsol one moee wth enchersding Fusl entrgerainggrsae than
Nor'
1972
34
Ancmdl Bullo
03
The Tert Mert boer
Coudty
Operation Large Scale
Figure 1: Representative examples of our SPARTA benchmark (see Appendix M for more examples).
Benchmack
Fatie Size
Question
GroppiggHevigg
Query Shape Supported
Malti-hop Beasnding
Ansotativo Ernur Rate (over songite
[Col
(-3--opp)
Closs modil
Un-modal
2AL-0AZhuzzal 2021
4.0
9.6
Morusi Morual
30%
4.6
15.7
Marail
21%
GTT GTT@NCRRT
4.4
15.7
Marual
21%
SPARTA (NB4)
SPARTA
(Melicaa)
100540
3,20LS
Lishtwiight Validutioon
Auto (LLM)
Table I: Comparison of Taxle-Text QA benchmarks (see Appendix A for detailed annotation audit results).
(1) Limited question types and shallow reasoning. Existing Texte-Text QA benchmarks, constrained by manual annotation complexities, feature a restricted range of shallow questions. These typically require only direct information extraction (such pinpointing a fact within single textual passage locating a specific entry in a table). Even for questions that go beyond this simple extraction, the reasoning depth remains shallow, seldom demanding more than two hops or involving advanced analytical operations like aggregation or grouping. This is despite such operations being common in real-world natural language queries yet underrepresented in benchmarks. This deficiency hinders the thorough evaluation of a system's 's deep, multi-step inference capabilities. Furthermore, current multi-hop hop questions usually follow simplistic linear chains, rather than the expressive, tree-structured reasoning (e.g., multi-brance paths, longer chains, or uni-modality hops) crucial for assessing systems on complex inference tasks, as exemplified in Figure |
(2) A Annotation noise. Our quality audit uncovers numerous annotation errors that undermine the reliability of the benchmark. Re-inspecting 100 randomly sampled dev examples from HYBRIDQA, we find that 21 % contain at least one error, which we classify into three categories: (1) Redundant modality (52.4 (6): table and passage encode the same fact, yet the instance is tagged modality suffices; (2) Incomplete answer set (23.8 %): several answers are correct but only one is recorded, distorting recall; (3) Incorrect or unanswerable (23.8 %): the labelled answer is wrong cannot be derived from the provided data, revealing a lapse in quality control. Our audits on other henchmarks reveal similar error patterns (see Appendix A)).
cross-modal question even though single
(3) Reliance on single, small-scale web tables. Current benchmarks almost exclusively draw on compact web tables cille--typically scraped from Wikipedia corporate thereby providing only toy-scale scenarios.
2

---

Published as a conference paper at ICLR 2026
As Table I shows, tasks either involve single table or, when multiple tables are present, the mean table cardinality hovers around 15 rows, far short of the thousands of rows found in real-world databases. This simplification is largely pragmatic: reasoning over larger tables dramatically increases annotator effort and error rates Chen et al. (2020b). Consequently, existing benchmarks cannot meaningfully evaluate QA systems realistic, high-complexity settings that demand reasoning over large, heterogeneous relational data.
and unstructured- erence fact database. Each original relation (e.g., a web table or a financial ledger) remains intact as a source table. Grounding tables, which store atomic propositions as tuples for SQL-addressable access, are populated using two complementary methods (detailed in Section 3.2): (1) utilizing validated corpora such as ROTOWIRE| Wu et al. (2022); and (2) employing table-to-text strategy that generates atomic facts directly from structured data. With textual facts now addressable via SQL, queries over this combined store freely mix modalities; no pointer to the original span is needed as answers are returned directly by query execution.
SPARTA unifies all evidence-strucee
a single relational store called the ref-
Stage 1 - Reference fact database construction. Source and grounding tables are merged into the reference fact database, making all facts uniformly queryable.
Stage - Query generation. A large language model (LLM) receives the schema and sample rows and emits SQL whose number of nested predicates matches a target hop count. Note that SPARTA synthesizes queries that instantiate the four representative nesting patterns-Tyyps N, A, J, and JA-outinee in Appendix B Two safeguards ensure that only realistic, executable statements survive: (1) Provenance-bused refinement loops provenance feedback- unnetchhed joins or overly selective predicates- bacaattees-aaaaaaeeeeeeeee to the LLM until the query returns non-empty result. (2) Realistic-strucet strructrr enforcement confines generation to post-order traversals of query graph, yielding human-like join orders and enabling early pruning of infeasible subqueries.
Stage - Question verbalisation. Each validated query is paired with its execution result, then a second LLM rewrites the SQL into a fluent natural-language question, producing high-fidelity pair (question, answer) that span aggregation, grouping, and deep multi-hop joins across large tables. Here, the final correctness-iie the validity of the question-answer pair-is checked via lightweight human verification; unlike HybridQA, our pipeline does not require re- re-performing full multi-hop reasoning, thereby keeping audit costs low (see Section 3.4).
This SQL-centric pipeline yields a large, diverse, and rigorously validated benchmark that corrects the size, noise, and logical shallowness of previous Table-Text QA resources. On SPARTA, state-of oftte-of-the-art models that exceed 70 F1 on HybridQA or exceed 50 F1 on OTT-QA drop by more than 30 F1 points, revealing fundamental weaknesses in current cross-modes reasoning and highlighting directions for future research.
2 RELATED WORK Table-Text QA Benchmark. Table-Text QA benchmarks evaluate a model's ability to jointly reason over structured tables and unstructured passages. HybridQA Chen al. (2020b) introduced the task, and OTT-QA Chen et al. (2020a) extended it to open-domain settings, but both suffer from annotation noise, shallow reasoning depth, and lack of support for advanced analytical operations. Specifically, they do not support GROUP BY or HAVING clauses, and only 1.1% of questions involve aggregation. Their multi-hop reasoning is confined to short, linear chains and fails to capture tree-structured or uni-moda reasoning paths. Other Beachhaaarss-------
[hhu et al.] (2021), FinQA Chen et al.] (202T). and MultiHiertt Zhao narrowly on numerical reasoning in financial contexts rather than multi-hop reasoning.
eet al. further limiting coverage Zhang et al. (2023), Additionally, all existing Table-Text QA datasets rely on small, manually annotated web tables, which hinders scalability and realism. SPARTA addresses these gaps with an SQL-centric pipeline that constructs large-scale benchmark of executable, compositional questions over hybrid corpora, offering a principled testbed for multi-hop QA across text and tables.
Synthetic Benchmark Generation. Recent synthetic benchmark generation scales QA pairs from pre-existing sources, but most are single-modal: relying on knowledge graphs (unn et al. (2024); Omar et al. (2025):
Orogat & Ro-yoby 2003 2022) or text corpora Bonifacio et al. (2022); Jeronymo et al. (002). ignoring
3

---

Published as a conference paper ICLR 2026
Reference Fact Database Construction
2- Queryy Generation
Question Verbalisation
Rost-orter
Reskltt
Query Generator
Generattin
SQL2NL
ST
Translator
Texts
Grounding Tables
SQL Query Graph
NL Question
Provenance Based Fixer
Reference Fact
Empty
SCL Esccution
Answer
Saurce Tables
Database
Figure 2: Overview SPARTA: (1) Reference Fact Database Construction, (2) Query Generation, (3)
Question Verbalisation. ST and GT denote source table and grounding table, respectively.
cross-modal reasoning, ERBench Oh ct al. (2024) uses relational databases. yet its questions are binary or multiple-choice, based on shallow templates excluding analytical operators like GROUP BY, HAVING, and aggregations; it also lacks table-text interplay. Similarly, TDBench Kim et al. (2026) leverages temporal databases to automate time-sensitive QA, but is confined to temporal reasoning within structured tables. In contrast, SP ARTA generates multi-hop questions bridging tables and passages, mirroring complex nested SQL patterns to provide a rigorous cross-modal benchmark for Table-TexI QA. Beyond QA, benchmarks in other domains impose commain-specffc constraints: database performance benchmarks Nambiar & Poess (2006);
Erling et al. (2015) use fixed schemas and templates for reproducible profiling, unlearning benchmarks Maini
al. (2024) Zhong al. (2024) create forget/retain partitions for selective forgetting: and PHEL Kim (0024)
employs template-based generation of NL--Nested SQL. pairs to guarantee the executability of structurally complex queries. SPARTA's constraint fundamentally different: every synthetic example must encode tree-structured multi-hop reasoning grounding semantically sound, executable SQL and natural-languaa questions, requiring analytical operations and table-text alignment. Our provenance based refinement and realistic-strueture enforcement address this, producing semantically rich, executable queries.
3 SPARTA
3.1 TABLE-TER QA TASK AND BENCHMARK GENERATION
Given natural-language question QNI., set of source tables (T=(T(W,...) T(m)), and set of passages Cp=(p()), }. a QA system fo must return the answer fo(gNL. Sr, Cp). Each passage in Cp decomposed into atomic facts and stored as tuples in grounding tables Gr. Merging these with the original source tables yields unified reference fact database D. An LLM then: (i) generates executable SQL queries on D that vary in depth (selection, aggregation, nesting, etc.), and (ii) verbalises each query into fluent natural-language question 9NL. The resulting pairs (gNL. a) constitute a scalable benchmark for Table-Text QA. An overview of the entire pipeline is provided in Figure
3.2 REFERENCE FACT DATABASE CONSTRUCTION
We use the ROTOWIRE dataset as part of our reference fact database, whose structured tables are widely used gold supervision for text-to-table and have been verified by the authors of Wu al. (2022) for consistency with the accompanying game reports. Each NBA game report in this corpus is decomposed into atomic facts, which are stored as tuples Gr, guaranteeing perfect alignment between text relational data. To construct Sr, integrate six public NBA datasets covering salaries, awards, draft data, and team histories sourced from Kaggle and dat a, world c. kag i a b dat Shared entity attributes such as PLAYER NAME and primary-forent
TEAM NAME enforced
key pairs, yielding connected schema in which every tuple from Gr can be joined to at least one table in Sy. The resulting database contains three grounding tables and six source tables (see Appendix
4

---

Published as a conference paper at ICLR 2026
While our construction uses NBA data for illustration, SPARTA is inherently domain-agnostic. From any relational database, one designates subset of relations as Sy and treats the remaining relations as Gr- Applying table-to-text generation to Gr yields a companion set of textual passages Cp. forming the reference- fact database D = ST U GT with no information overlap between the two sets. The query-generation pipeline then applies unchanged. yielding a portable recipe for building large-scale Teble-Text QA benchmarks in any domain with relational data. To demonstrate this, we extended our pipeline to two new momainss-oviin and using Kaggle datasets kag d) with configurations identical to the NBA domain (see Appendix [E]. For these datasets, we start from existing structured tables and convert subset into grounding tables using rule-based templates. This table-to-text transformation is deterministic and template-driven with templates manually designed and verified to prevent spurious facts emrrs.
3.3 QUERY GENERATION
For non-nested queries, SPARTA builds the statement clause-by by-clause: the LLM emits each clause in canonical SQL order, conditioned on the schema and previously written clauses, and immediately executes the partial query. If the result is empty, the execution outcome is fed back the LLM can revise the offending clause, ensuring the query remains executable and semantically meaningful at every step.
The next step is to synthesise nested SQL queries that act as faithful logical forms for multi-hop reasoning. A generated query must satisfy two criteria: (i) it should resemble a query that a human analyst would plausibly write, avoiding degenerate template artifacts, and (ii) it must execute over 'D without error and return a non-empty result. These guarantees ensure that every (INL, a) pair is both natural and answerable.
Template-based generation fills fixed slots with ad-hoc limits or auxiliary predicates to guarantee execution, yet the resulting SQL is often semantically unsound. For instance, SELECT birthplace FROM les___llye__infrmmttee
WHERE bi birthplace <> 'Chicago, Illinois' birthplace <> 'Dallas, Texas' runs without error but expresses a vacuous intent ("... not born Chicago or not born in Dallas," matching everyone). Conversely, one-shot LLM prompting produces natural queries, but these frequently yield empty results and show limited diversity (see 3) We therefore introduce a stal-stage framework: (i) realistie-structue enforcement and (ii) provenance- refinement. 3 REALISTIC-STRUCTUS
ENFORCEMENT
A nested SQL query can be modeled as a query graph G -VV E) where each node U; V corresponds to a distinct query block-name outermost watementt-will each (directed) edge eiy E E denotes nested predicate that correlates blocks Qi and 2 through a shared attribute reference, thus capturing the dependency structure of the original nested query graph form (see Appendix B for representative nested query patterns). Based on this representation, we measure query complexity by the number of edges in the query tree, each representing reasoning hop.
every SELECT
FROM
WHERE subquery including the
For nested-query generation, SPARTA adopts Pest-Order+Prot as the default. That is, to preserve realistic structure, we force the LLM to build the query tree in post-order: compose each leaf subquery first, then wrap it with successively higher-level blocks exactly how analysts craft nested SQL. We choose post-order traversal over alternatives like breadth-first top-down, because the latter require validating incomplete queries before inner subqueries are constructed. In contrast, post-order ensures that each intermediate block is executable by validating subqueries first and then composing higher-level predicates. In Post-Ordert Prov, leaf nodes are generated clause-by-clause For the target question type we pick the relevant clauses (WHERE, GROUP BY, ORDER BY, ...) in canonical order, and let the LLM fill each one using (i) the schema, (ii) earlier clauses, and (iii) partial results. If clause yields an empty result, we roll back to the last valid subquery, sparing redundant LLM calls. Internal nodes arise by recursively enclosing validated subqueries. At every step the LLM selects child query, picks joinable table, and emits connecting predicate (AND/OR, etc.). Empty outputs trigger novenance-guidee repair ($3.3.2) otherwise the predicate is kept. The loop iterates until the query graph grows to the specified target size.
3.3.2 PROVENANCE- BASED REFINEMENT
5

---

Published as a conference paper at ICLR 2026
FROM
WHERE
[Orginal Query C
playe
(SLLECT player mame
FROM nas soger inforoatoo
obber
bitt yeu
160
130
130
inforatee
dlw pans
plar pare
Jane
Tar
Tye
Jam
saary
WIGI:
MUSTT
play Jocen
(S0J0C pieee
EWN parms In
WHEAE bettuverr (590)
184
Ian
Jocan
100
Mrr
(MARCT pkayr
SSEECT
FROM 18y glecr alliew
of op 0)
oafppctded Roulh
WHERE saary 200 200
The Sub Query of with scoverett hout
FFltrdoatty Salayy 500,000'
Stop3 LLM Promptin
(Resbed Duary O*)
[Orignal Query
iPro lemati Pradicated]
FRON 18. fir, ehemetin
of
warld
jsfry Provenance)
atterics
(WIRT (laye
E/7 20
pleas
Mmn
AMD
SSIECC
WHir lete 1990)
FROM dbc_pkye__ Jillacce
Figure 3: Overview of provenance- based refinement.
The LLM builds the query graph in vost-order-validatedd leaves first, then one outer predicate at a time. If an evolving query returns no rows, a provenance-based refinement process is initiated repair the query. The refinement process leverages "why-not provenance," a database technique used to identify which predicates in a query are responsible for filtering out expected tuples Bidoit et al. (2014); Chapman & Jagadish| (2009); [Lee et al. (017). While traditional why-not provenance often relies on user- eser-provided examples of the missing tuples, our approach dynamically derives the expected tuples from intermediate query results.
The process unfolds in three steps. First, when query yields an empty result, we peel off predicates in reverse order until the query yields result. Second, we sample tuple from this non-empty result set. Finally, we run why-not provenance tool Dietrich et al. (2022) to identify the blocking predicate and provide this provenance report to the LLM, instructing it to rewrite only the problematic clause.
Ablations are (i) One-Shot-k; which inserts all k predicates in a single pass with no checks, and (ii) Post-Order (no provenance), which follows the same construction but skips the repair loop. Figure 3 illustrates the overall process of provenance-based refinement. Provenance feedback relaxes the predicate from salary > 800000 to salary > 600000.
3.4 QUESTION VERBALISATION
For each executable SQL query QSQL. we generate a corresponding natural aatural-language question qNL using AST-ICL Al Lawati et al. (2025). a SOTA basedd SQL-to-text model. We adopted the LLM-based model over template-based methods, which are limited by rigidity and reliance on handcrafted templates, as documented in prior work Iyer et al. (2066); Xu et al. (2018). In AST-ICI,, the SQL abstract syntax tree supplied as in-context exemplar, and the model emits fluent question 9NL whose semantics align with the query. Executing (SQL on D yields the answer a, completing the benchmark pair (gNL. a). Every instance is thus interpretable, executable, and suitable for probing multi-hop reasoning over hybrid (table text) data.
The verbalized questions were validated and corrected by three CS graduate students with SQL/schema literacy ensure factuality and meaningfulness. This process is lightweight, requiring substantially less effort than full manual annotation. Specifically, validating 3,300 queries takes about 1,493 minutes of total worker time, whereas HybridQA required roughly 6,600 minutes create the same number of queries from scratch.
4 EXPERIMENTS
4.1 EVALUATION SETUP
Hardware and Software Settings. We conducted our experiments on a machine with Intel()) Xeon(R) Gold 6230 CPU @ 2. 10GHz and 1.5 TB of RAM running Ubuntu 22.04.4 and 4 RTX A6000 GPUs, with LLM inference managed via the SGI Lang Zheng et al. 2024) inference engine. We used -l0maa-.1-70B-Insttte Dubey et al. (024) the L.M..
Query Generation Methods. For non-nested query generation, SPARTA's default is Edecutiin-Guidee generation: the LLM writes each clause in canonical order, executes the partial query, and immediately edits
6

---

Published as a conference paper at ICLR 2026
any clause that empties the result. As an ablation we also evaluate (i) One-Shot, which emits the whole query from schema only, and (ii) Clause, which builds the query sequentially without execution feedback.
For nested-query generation, SPARTA's default is Post-Order+ Prov: validated leaves are wrapped one predicate at time (post-order); each new predicate is executed immediately and, when empty, repaired with provenance feedback. Ablations include (i) One-Shot-ki which inserts all k predicates in single pass with no intermediate checks, and (ii) Post-Order (no provenance), which follows the same post-order construction without provenance- repair. We generate 500 non-nested and 600 nested SQL. queries per method on the NBA domain (configuration in Table 10), so that quality and cost can be compared equal footing,
Table-Tex QA Methods. To gauge how current state-of-the-art systems break down under SPARTA'S deeper hops, larger tables, and advanced analytical operations, we evaluate SOTA Teble-Text QA methods, including methods based on prompting LLMs such as ODYSSEY Agarwal et al. (2025) and HProPro (Shi et al. (2024). These models have shown strong results on HybridQA, where models reason over provided tables and linked documents. ODYSSEY constructs a graph from the table and linked documents, enabling the L.LM traverse the graph for query answers. HProPro generates and executes program via the LLM to produce query responses. Since existing Table-Text QA methods are not originally designed to support uni modal hops, we apply minimal extensions to enable such behavior during evaluation on SPARTA. Specifically, for ODYSSEY, we augment the hybrid graph by adding edges between matching cells of columns that share a join relationship. For HProPro, we adapt the prompt format by replacing the input table with a list of relevant tables. For fully end-to-ent scenario in which no oracle is provided, we pair the Table Text QA methods with HELIOS Park et al. (2055)) top retriever on OTT-QA-SO the model must both retrieve evidence and reason over it. We also run every method with GPT-5 and GPT-3.5-turbo backbones to test LLM sensitivity.
4.2 BENCHMARK GENERATION CoST AND QUERY NATURALNESS
A scalable benchmark must maximise useful queries while minimising LLM calls and wall time. We therefore track seven complementary metrics in Table 2
Table 2: Cost metrics used for benchmark generation.
Metric
Delinition
Success-Q Q
# of non-nested queries that execute without error and return least one row.
Exec-Err
statements that fail parse cuntime. revealing schema logic errors. syntactically valid queries retumn zero rows because predicates are too restrictive.
Empty-Q
Duplicate-Q
(essclin # of LLM invocations required every step succeds on the first attempt (baseling cost).
of queries whose result duplicates a previously generated query. reducing diversity:
Ideal Calls
Total Calls
# of actual LLM invocations, ie.. Ideal Calls plus extra calls for provenance-guiie fixes or other retries.
Wall Time
Total wall-clock lime obtain all successful queriee.
Table B summarizes generation overheads for both non-nested and nested SQL. For non-nested queries, Execulion-Guidee is most economical, needing only 1,134 total LLM calls-just 7.2% above the ideal 1,058 and finishing in 2,466s. In contrast, One-Shof begins with the lowest ideal budget (500 calls) but produces 60 empty and 1,265 duplicate outputs, inflating usage to 1,830 real calls (266% of ideal) and incurring the highest latency: Clause mitigates these failures yet still exceeds its ideal by 24.9%. For nested queries, Post-Order+ Prov is most cost-effective, completing with 4,722 calls in 26.278s-cuttin call volume by 42.8% versus vanilla post-order and by 66.2% versus Ohe-Shot-k. These results show that disciplined post-order construction combined with provenance-driver repair minimizes redundant generations while ensuring executable, semantically plausible SQL; detailed analysis of generation overheads across varying query graph shapes and sizes is provided in Appendix F
To assess the realism of the generated SQL queries, we employ scoring-based evaluation framework combining automatic and human assessments. Each query is rated from (least natural) to 5 (most natural) across three dimensions: Relevance, which measures alignment with the genuine curiosity of a typical person; Speci f ficity information need; and Overall Naturalness which combines the above criteria to decide whether the query is likely to be asked by real person. For comprehensive assessment, conduct automatic
Clarity, which assesses whether the query expresses clear and well-scoped
7

---

Published as conference paper ICLR 2026
Table 3: Generation Cost Comparison of Query Generation Methods.
Method
Succee-Q
Emply-Q
Duplicate-
Exte-Rer
Idral Call
Total Calls
Wall Time (c)
Now nested Query Generation
Ciaue
Fawaninn-Cuiided
2466
Nested Generation
One Shot-
2064
11962
118316.67
Fos Order tso provenance)
600
0
3104
8253
38967.40 7.40
Pon
600
0
M004
4723
26277
Win (MM)
Nedred (eutts eved
Meneviteed Duuan eve
hested huuma--vall
Figure 4: Comparison of Query Naturalness for Different Generation Methods.
evaluation (auto-evall eval) using ChatGPT-4o OpenAl and independent human evaluation (human-eval) by three external CS graduate students with SQL/schema literacy. As a baseline for comparison, we also evaluate queries generated by template filling with randomly sampled column-value pairs. This dual approach, integrating M---ee auto-evaluation with human judgment, yields a robust, multi-perspective measure of how convincingly the generated queries mirror real user intent.
and
Figure reports the naturalness scores of queries generated by different methods, evaluated across three criteria. Among the non-nested query generation methods, Execution-Guidee Generation achieved the highest scores consistently across both automatic and human evaluations. Specifically, in terms of overall naturalness, it outperformed Clause-by-Clen -CCuuee, One-shot, and Template-brul based generation by 1.3%, 11.4%, and 37.5%, respectively, in auto-eval; and by 6.0% and 36.7% over One-shot and Template- based methods in human- For nested query generation, Post-order Generation with Execution Guidance achieved the top scores across all three metrics. Compared to Post-order, One-shot Nested, and Template-basee auto-eval improvements of 1.7%, 8. %, and 123.2%, and human-eval gains of 2. %, 12.5%, and 117.8% respectively. These results confirm that LLM-based generation strategies- especially those leveraging clause- wise generation and post-order traversal-ane significantly more effective at producing realistic and fluent SQL queries than template- based approaches.
generation, it yielded
4.3 TABLE- MABLE-TEXT QA EVALUATION RESULTS
Table Table 3 report the Table Text QA performance of representative methods across eight benchmarks, revealing the increased difficulty posed by SPARTA. We evaluate SPARTA under two configurations: (1) SPARTA (Oracle), where models are given ground-truth tables and linked passages; and (2) SPARTA (Retrieval), where models must retrieve relevant content from the entire corpus. On SPARTA (Oracle), ODYSSEY with GPT-5 achieves an average score of 35.6% across all domains, representing a sharp 33.9-point drop compared to its performance on HybridQA (69.5%). Similarly, HProPro with GPT-5 achieves an average Fl score of 40.4%, a 30.1it drop from its HybridQA performance (70.5%). These results reveal the limitations of existing methods when scaled to larger, more complex queries. Interestingly, HProPro with GPT-5 outperforms ODYSSEY on the NBA and Movie domains, which feature tables with thousands of (as shown in Table owing its ability to generate executable programs that directly operate over
8

---

Published as a conference paper ICLR 2026
Table 4: Table-Tex QA Accuracy on the SPARTA (Oracle) across multiple domains.
IlytndQQ
NBA
Molicl
EM
EM
F1
EM
OUYSSEY GPP33
9.0
2688
148
2399
33,6
6.7
22.9
33.2
213
12.0
206
313
20.3
32.7
42.2
42.6
4422
HProPro S-r-s.--tarbe
11.0
13.6
164
138
22.2
278
29.1
29.2
15.
19.5
202
19.
16.
20.
218
20,9
31
25.5
257
26.1
ODYSSEY GIPSS
31.2
28.4
38.4
18.1
204
34 3
33.9
24.
47.5
54.2
68.3
54.2
19.7
35.6
43.
85.5
58.3
885
69.1
1A,5
HProPro o/ GPTS
23.6
33.1
362
34,0
36.6
47.1
49.2
48.
28.1
41.
43.2
41.6
29.5
40.4
42.9
41.5
89.7
70,5
71.1
73.1
Table 5: Table-Text QA Accuracy the SPARTA (Retrieval) across multiple domains.
SPARTA
OTTQA
Method
NBA
Maric
Medical
Ang.
EM
F1
P
R
EM
F1
P
EM
P1
R
EM
P
R
EM
F1
Reader
4.6
17.6
64
160
12.9
6.5
65.2
667
HOS-HP (PTS-5
14.5
18.6 21.6
28.6
21.7
13.7
273
15.3 28.0
22.5
47.7 56.0
tables. This result highlights the limitations of ODYSSEY when applied to large-scale tables and aligns with the broader observation that larger table sizes increase the difficulty of table-QA for LLMs Patnaik et al.] The performance gap between GPT-5 and GPT-3.5-turbe (35.6 vs. 20.6 F1 for ODYSSEY and 40.4 vs. 20.3 Fi for HProPro) underscores the importance of advanced LLM reasoning capabilities in handling such challenges. In the retrieval setting, where gold tables provided, performance degrades further: the best method (HELIOS + HProPro with GPT-5) attains only 22.6 F1. This sharp decline illustrates the compounded challenge of retrieval and reasoning
heterogeneous corpora. We additionally evaluate the FiE Reader Ma et al. (2023), the state-of-the-as fine-tuned reader model on OTT-QA. While FiE Reader surpasses HELIOS + HProPro w/ GPT-5 by 9.2 points on OTT-QA, lags behind on SPARTA by 11.0 points, showing fine-tuned models fail to generalize to SPARTA's more complex, out-of-domalt settings.
4.4 ANALYSIS
We conducted a comprehensive analysis of the models' execution results on the SPARTA benchmark. This investigation uncovers several fundamental vulnerabilities in current table-text QA models, pointing to critical directions for future work.
Models struggle to handle complex multi-hop query structures. We evaluate Table-Text QA models under various tree-structured query configurations, fixing the number of edges four: (Depth Breadth 3), (Depth 2, Breadth 2), and (Depth 3, Breadth 1). We also included intermediate shapes with three edges, such (Depth Breadth 2) and (Depth Breadth 1), to further validate the trend.
As shown in Figure 5a model performance degrades sharply as either depth or breadth increases. At fixed depth, expanding breadth from (Depth Breadth 1) to (Depth 1, Breadth 3) reduces HProPro and ODYSSEY by 25.2% and 27.5%. respectively. At fixed breadth, increasing depth from (Depth 1, Breadth 1) to (Depth 3, Breadth 1) yields 47.2% and 49.9% declines. Additional comparisons (Depth 2, Breadth 1) (Depth 2, Breadth 2). and (Depth 1, Breadth 2) to (Depth 2, Breadth show consistent degradation, further confirming that both deeper and broader queries cause substantial F1 drops. These findings suggest that existing methods are fundamentally limited in performing rreestructured reasoning over multiple relational paths, regardless of whether complexity arises from depth or breadth.
Models struggle with analytical operations such grouping and ordering. As shown in Figure 5b both ODYSSEY and HProPro exhibit consistent performance degradation when advanced analytical clauses present. For queries that include GROUP BY and HAVING clauses, ODYSSEY attains an Fl score of 35.4, whereas HProPro attains 27.1. When ORDER BY and LIMIT are present, the scores are 31.2 for ODYSSEY and 21.4 for HProPro. Aggregation queries show similar pattern, yielding 28.4 for ODYSSEY and 37.2 for HProPro. Compared with each model's average F1, these analytical scores are markedly lower, indicating weak numerical reasoning, filtering, and ranking capabilities and exposing fundamental limitations
9

---

Published as a conference paper ICLR 2026
ODYSSEY
ODYSSEY wt GPTS
HProPro wr GPTS
71.5core
60:407
09
100
Agregiic
Groyddy
Orberty
(a) Fl scores acTOss tree configurations (D: Depth, B: Breadth)
(b) Fl scores across analytical operations
Figure 5: Comparison of F1 scores across different configurations.
in addressing real-world table-text questions. Notably, ODYSSEY performs worst on aggregation queries, whereas HProPro struggles most with ORDER BY and LIMIT.
Table 6: Performance Comparison With and Without Text Data in Table-Texx QA.
Method
Setting
EM
GPT-5
Table-Tent Cross Rensoning
23.9
28.6
363
28.4
Table-only Ressoming
32,0
39.2
49.5
38.8
Table-Test Remsoning
11.9
17.2
Table-only Reasoming
29.2
45.2
46.9
46.4
Performance drops sharply when unstructured text is required. As shown in Table 6 the inclusion
of table-text cross reasoning leads to a significant decline in performance. HProPro's Fl score drops by 63.9% (from 45.2 to 16.3), while ODYSSEY experiences an even steeper drop of 23.0% (from 39.2 to 28.6). This sharp contrast highlights the difficulty of reasoning over unstructured passages in conjunction with structured tables. Although both models perform moderately well when queries rely only on tabular data, they consistently fail to retrieve and integrate relevant textual spans when external context is present. These failures indicate that current cable-Tex QA models lack robust cross modal alignment and semantic grounding, limiting their effectiveness in real-world scenarios that demand joint reasoning over heterogeneous data sources.
To further support our findings, we include supplementary analyses in the appendix: an ablation study on nesting types (Appendix G) and error case analysis (Appendix [
5 CONCLUSION
In summary, we present SPARTA, a benchmark generation framework that rectifies the three critical short- comings of existing Texle-Text QA resources shallow question design, annotation noise, and toy-scale tables-by (i) unifying heterogeneous evidence inside a reference fact database, (ii) generating logically deep, human-like SQL nested queries whose hop count and analytical operations are explicitly controlled through a provenance-guider LLM pipeline, and (iii) verbalising them into natural-language questions using an LLM- SQL-to-texx model, with lightweight human validation for fluency and correctness. On SPARTA, state-of-ue state-of-the-att the-art models that reach over 70 F1 on HybridQA or over 50 Fl on OTT-QA drop by more than 30 F1 points, exposing fundamental weaknesses in current cross-modal reasoning.
6 LIMITATIONS AND FUTURE WORK
SPARTA currently focuses on the Table-Text setting. Future work will extend it to multimodal inputs like images and videos by using vision-language models to summarize visuals into atomic statements, normalizing them into grounding tables, and merging with the existing fact database. Since these tuples follow the same schema as D, the query-generation pipeline ($3.3) applies unchanged. A complete multimodal extension, including dataset collection, schema design, and evaluation, is planned for future research.
10

---

Published as a conference paper ICLR 2026
7 ACKNOWLEDGMENTS
This work was partly supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government (RS-2025-00517736 30%), Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea governmen(MMSTT) (No.
Global Developing Vector DB for Long- Term Memory Storage of Hyperscale AT Models, 10%), and Basic Science Research Program through the National Research Foundation of Korea Ministry of Education (No. RSS2002-00411560 10%)
AI Frontier Lab, (No. RS-
Reproducibility Statement We include prompt examples for provenance-basel refinement, realistic- structure enforcement, and automatic naturalness evaluation in Appendix N Details of the experimental setup provided in Section 4.1
REFERENCES
Nba salaries.
Accessed: 2025-05-15.
Nba games. 2025-05-15.
http://pwwww.ccccooco/c////////a a. Accessed:
Historical nba finals and
historical -aal--ba---ii finals
mvp mvp
Accessed: 2025-05-15.
Imdb movies analysis
sql.
imdb-movies-date
Hospital management dataset.
hospital - hatelll-maangmmeeett-ee -atasseee----
Nba dataset project
httpps//www.krrgieeccoo///eeeeee http://www.aaaelaaac////aaaa//ar nba data-dateet---- prro-pooject
Accessed: 2025-05-15.
1991-2021 nba stats.
19912021-nbasster f. Accessed: 2025-05-15.
Nba/aba/baa team stats per game
aba-aba-baa-stale htt.://wwaaaaae...eeeeeaaaaaaaaa
csv
Accessed: 2025-05-15.
Ankush Agarwal, Chaitanya Devaguptapu, and Ganesh S. Hybrid graphs for and-eet based question answering using LLMs. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pp. 858-875, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176- 189-6. URL hctps://
aclanthology.org 2025.0025... naacl long. 397
Ali Al Lawati, Jason Lucas, and Prasenjit Mitra. Semantic captioning: Benchmark dataset and graph- aware few-shot in-context learning for sq12text. In Proceedings of the 3 International Conference on Computational Linguistics, pp. 1026-8042 2025.
Nicole Bidoit, Melanie Herschel, and Katerina Tzompanaki. Query-based why-not provenance with nedex- plain. In Extending database technology (EDBT), 2014.

---

Published as a conference paper ICLR 2026
Luiz Bonifacio, Hugo Abonizio, Marzich Fadaee, and Rodrigo Nogueira. Inpars: Unsupervised dataset generation for information retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 2387 2387-2392 2022.
Adriane Chapman and HV Jagadish. Why not? In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, pp. 523-534, 2009.
Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Wang, and William W Cohen. Open question answering over tables and text. arXiv preprini arXiv:2010.1044 10439, 2020a.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. arXiv preprint arXiv:2004.07347, 2020b.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3697-3711, 2021.
Benjamin Dietrich, Tobias Müller, and Torsten Grust. Data provenance for recursive sqi queries. In Proceedings of the 14th International Workshop on the Theory and Practice of Provenance, pp. 1-8, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Orri Erling, Alex Averbuch, Josep Larriba-Pey, Hassan Chafi, Andrey Gubichev, Arnau Prat, Minh-Duc Pham, and Peter Boncz. The ldbe social network benchmark: Interactive workload. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD '15, pp. 619-630, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 978 1500327777,. doi: 10. 2723372.2742786. URL http:pts///po...////////////// wtt.://........111/1333//....232
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Summarizing source code using a neural attention model. In 54th Anmual Meeting of the Association for Computational Linguistics 2016, pp. 2073-2083. Association for Computational Linguistics, 2016.
Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprint arXiv:2301.01826, 2023.
Hyeonji Kim. A Systematic Data Augmentation and Cleaning Technique for Improving Text-to-SQI. Ph.d. dissertation, Pohang University of Science and Technology, August 2024. URL (tllht //ttt:://www /ww dcollection. net www.eetiiooc.et/aaaaeeeee//ooaoo http:/ssteec///000002222200
Ph.D. Thesis.
Soyeon Kim, Jindong Wang, Xing Xie, and Steven Euijong Whang. Harnessing temporal databases for systematic evaluation of factual time-sensitive question-answering in LLMs. In The Fourleenth Interna- lional Conference on Learning Representations, 2026. URL https: http:::ss///ppevvvieeeee net/ formm?
id Sd-W7RNxsTKR
Won Kim. On optimizing an sql-like nested query. ACM Transactions on Database Systems (TODS), 7(3): 443-469, 1982.
Seokki Lee, Sven Köhler, Bertram Ludäscher, and Boris Glavic. A sql-middleware unifying why and why-not provenance for first-order queries. In 20/7 IEEE 33rd International Conference on Data Engineering (ICDE). pp. 185-496 IEEE, 2017.
12

---

Published as a conference paper ICLR 2026
Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. Chain-of-skille configurable model for open- question answering. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pp. 1599-1618, 2023.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for Ilms, 2024. URL petltts://arxivv.
abs 2401.06112
Raghunath Othayoth Nambiar and Meikel Poess. The making of ipe-ds. In Proceedings of the 32nd International Conference on Very Large Data Bases, VLDB '06. pp. 1049-1058. VLDB Endowment, 2006.
Jio Oh, Soyeon Kim, Junseok Seo, Jindong Wang, Ruochen Xu, Xing Xie, and Steven Whang. Erbench: An relatioielt based automatically verifiable hallucination benchmark for large language models. Advances in Neural Information Processing Systems, 37:53064-53100 2024.
Reham Omar, Omij Mangukiya, and Essam Mansour. Dialogue benehmark generation from knowledge graphs with cost-effective sttrivall-ugmeent llms. Proceedings of the ACM on Management of Data, 3 (1):1-26, 2025.
OpenAL Chatgpt via chat completions URL wtt.lltttte//ptttccco....aaaoaaa chat.gpt-4o-1ate
Abdelghny Orogat and Ahmed El-Roby, Smartbench: demonstrating automatic generation of comprehensive benchmarks for question answering over knowledge graphs. Proceedings of the VLDB Endowment, 15(12); 3662-3665, 2022.
Abdelghny Orogat and Ahmed El-Roby. Maestro: Automatic generation of comprehensive benchmarks for question answering over knowledge graphs. Proceedings of the ACM on Management of Data, 1(2):1224, 2023.
Sungho Park, Joohyung Yun, Jongwuk Lee, and Wook-Shin Han. HELIOS: Harmonizing early fusion, late fusion, and L.M reasoning for multi-gran galtilgranular table-text retrieval. In Wanxiang Che, Joyce Nabende, Elkaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 424444, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN
-0. doi: 10.18553/11/2025. 1559/
acI-long. 1559. URL Bttp:/ttse///ccaaroooooooooo////
Sohan Patnaik, Heril Changwal, Milan Aggarwal, Sumit Bhatia, Yaman Kumar, and Balaji Krishnamurthy. Cabinet: Content relevance-based noise reduction for table question answering. In The Twelfth International Conference on Learning Representations.
Qi Shi, Han Cui, Haofeng Wang, Qingfu Zhu, Wanxiang Che, and Ting Liu. Exploring hybrid question answering via program- based prompting. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1 11035-110000 1104,, 2024.
Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Hcad-to-tail: How knowledgeable are large language models (Imss)? aka will I1ms replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 311-325, 2024.
Xueqing Wu, Jiacheng Zhang, and Hang Li. Text-to-table: A way of information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2518-2533, 2022.
13

---

Published as a conference paper ICLR 2026
Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, and Vadim Sheinin. Sql-to-text generation with graph- to-sequence model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 931 931-936, 2018.
Lingxi Zhang, Jing Zhang, Xirui Ke, Haoyang Li, Xinmei Huang, Zhonghui Shao, Shulin Cao, and Xin Lv. A survey complex factual question answering. Al Open, 4:1-12, 2023.
Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. Multihiertt: Numerical reasoning multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. -888,6600, 2022.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37:62557-62588 2024.
Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions, 2024. URL https://arxiv
14795
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pp.
3277-3287, 2021.
14

---

Published as conference paper ICLR 2026
A CROSS-DATASET ANNOTATION AUDIT
Any Redundant modality
Error breakdown (within erroneous samples)
Dataset
% of all samples
Incomplete answer set
Incorrect / unanswerable
HYBRIDQA
21%
52.4%
23.8%
23.8%
MULTIHIERITT
26%
15.4%
30.8%
53.8%
TAT-QA
30%
96.7%
0.0%
3.3%
FINQA
17%
41.2%
0.0%
58.8%
SPARTA
0%
0.0%
0.0%
0.0%
Table 7: Audit of 100 randomly sampled dev examples from each dataset. "Any error" shows the fraction of all samples containing at least one erro. The breakdown columns report the relative distribution among erroneous samples.
B SUPPORTED NESTED QUERY PATTERNS
SPARTA synthesizes queries for each of the four primary nesting patterns Kim (1922) commonly observed in real-world SQL, illustrated Fig
Table 8: Nested dested-query query patterns.
Type
Inner aggreg.
Correlation
Typical intent example
Type-N
X
X
Pure set membership. Outer block tests whether value belongs the returned by non-correlated subquery (e.g., WHERE IN (SELECT )..
Type-A
Aggregate comparison. Inner block computes an aggregate such AVG or MAX and the result is compared with each outer tuple (c.g.. salary (SELECT AVG (salaryy FROM .).
Type-J
Correlated filtering. Inner query references at- tributes of the outer block without aggregation (e.g., EXISTS (SELECT 1 FROM Items i WHERE i.order_id = -.id)).
Type-JA
Correlated aggregate comparison. query both correlates with the outer block and aggregates its own rows before the com- parison (e.g., EXISTS (SELECT FROM Items i WHERE i. o.ordrr_id o.id GROUP BY HAVING SUM (i. limit))..
Inner
Type-N N(C,)
Type-A A(G))
Type-J
Type-JA
NCC)
AIG)
T. C,=TT C,
T, C.=TT. C,
Figure 6: Four primary nesting patterns type (N, A, JA) queries of depth Each consists of outer block (T;) and an inner block (T)), Ares labeled 'A' indicate aggregation in the inner SELECT; straight ares 'N' denote set-inclusion predicates; curved ares denote join predicates.
15

---

Published as conference paper ICLR 2026
C SCHEMA OF REFERENCE-PACE DATABASE
Lang
(a) NBA
relemagping
movie
143U4
ratings
15g14
director mapping
tignt
(b) Movie
billing
patlents
bigitt
patient_
patient home
bigint
nome
bightt
Treatmant_e
bigint
gendar
bighet
treatments
bil_citte
bigitt
datecof_brr biimt
bighnt
bigitt
hightt
apocintment.js
paymeni_atatus
Insurancee
treatment Lype
descr ption
big mt
appointments
cost
big mt
appointment
bigint
doctors
bigitt
appcintmentt date
bigint
special Izadon
bighnt
apocintment nime
bigitt
yearl_ experiennce
bigh-t
reason_for..reet
bigint
Lapannh
bight
bigint
(c) Medical
Figure 7: Schemas of the reference-fact databases used in SPARTA across three domains. Each database consists of two complementary types of tables: souree tables S7 (orange) from public datasets (e.g., NBA player salaries, movie metadata, medical records) and grounding tables Gr (green) encoding atomic facts extracted from textual passages.
16

---

Published as conference paper ICLR 2026
D TABLE-LEVEL STATISTICS OF THE REFERENCE-FACT DATABASE
Table 9: Row/Columm Statistics of All Tables in the Sparta Benchmark
Domain
Table Name
Columns
# Rows
nba_ draft coanftt_ommne____ _taat
35
772
nba_playes bba_player_inoommtion information
14
4596
nba_player_awwe
236
nba aba_champion history
69
nba
nba player affiliation plaaeerraffilliaiionn
13980
nba_team_infore bba_tcam_informatin _njifoooooooonn
30
nba_player_gam _arme_aam___ __gm__mtaetss__ _sstts
45640
Baaiammgameesteeee stats
12
12750
nanba game informtaion
6665
genre
14418
ratings movie
7872
7872
imdb
role _ee_mmaping
15336
director_ director_mapping mapping
3800
names
25617
treatments
200
billing
200
medical
appointments
200
patients
doctors
10
Table provides overview of the structural statistics for all tables in the SPARTA benchmark, including the number of columns and rows per table, grouped by domain (NBA, IMDB, and Medical), These metrics highlight the scale and diversity of the reference-fact database used for evaluation.
E
BENCHMARK CONFIGURATION
Table 10: Benchmark Configuration: SQL Operator Coverage, Query- Serry-Shape/Size Distribution.
Query Shape
Non-matad
(Depth Bevoothh I)
(Depth Breadth 2)
(Depth Breatth 3)
(Depth 2, Broothh 1)
(Tspth Breadth 2)
(Depth Recatth
Total
45.5
9.1
0.1
9.1
9.1
9.1
100.0
SQL Operatur Prinence
WHERE
GROUP
HAVING
AGGREGATION
100.0
15.3
3,4
7.2
4.5
50.0
Nasted Predicode Type Prosence in Nostal Query (%)
Typa-N
Typ-A
Typ-J
TypoJA
32.4
F GENERATION COST ANALYSIS
F. COST ANALYSIS ACROSS LLM SCALES
17

---

Published as conference paper ICLR 2026
To demonstrate that our provenance- refinement is effective regardless of the LLM's size, we conducted additional experiments comparing generation costs across LLMs of varying sizes. Specifically, in addition the A.lam--31--1 1..1-00B-Insrrue model evaluated in the manuscript, we measured generation costs using a smaller-parameter LLM (gpt-oss-20B) and larger-parameter LLM (gpt-oss-120B),
Table 11 shows that Post-Order + Prov is the most cost-effective approach across all LLM variants, completing with 4854, 4,722, and 3831 calls for the respective models, while cutting call volume by 18.8%, 42.8%, and 54.5% versus vanilla Post-Order, and by 64.7%, 66.2%, and 65.7% versus One-Shot-k. These results indicate that disciplined post-order construction combined with provenance-driven repair minimizes redundant generations independent of the LLM's s scale.
Note that the "Ideal Calls" metric represents the number of LLM calls required if every step succeeds. It varies slightly due to probabilistic clause inclusion in SQL query generation (based on predefined per-clauss probabilities). As shown in the table, the variance is very small when repeat experiments three times per method.
LLM
Method
Ideal Calls
Total Calls
One-Shot-k
2661
13736
gpt-oss-
Post-Order (no provenance)
3073
5977
Post-Order+ + Prov
3146
4854
One-Shot-k
2664
13962
Llama-3,1 11-70B-Innstrue
Post-Order (no provenance)
3104
8253
Post-Orde Post-Order+ + Prov
3074
4722
One-Shot-k
2621
11225
ept-oss-120B 120B
Post- Post-Order (no provenance)
3152
8425
Post-Order + Prov
3145
3831
Table 11: Generation cost comparison across LLM sizes for three refinement strategies.
F.2 COST ANALYSIS ON QUERY SHAPE AND SIZE
Figure contrasts LLM usage for star and chain query trees their size grows from one three nested predicates. Here, star queries fix depth to I while increasing breadth (number of branches), whereas chain queries fix breadth
while increasing depth (number of nested levels). The size thus reflects how many nested predicates are added along either the breadth or depth dimension. For star shapes, one-shot generation quickly becomes prohibitive, ballooning to 17x the ideal call count when the hub size reaches three. Building the same queries in post-order slashes that overhead to 3.2x; provenance repair trims it further to 1.6x. Chains tell a different story: because their natural construction order already matches post-order, one-shot and post-order costs are similar, yet provenance still removes 30-40 % of redundant calls at every depth. Branching structures profit most from post-order generation, while provenance-guided repair a universally cheap "insurance policy" that cuts waste regardless of query shape.
Beyond structural complexity, we also measured the average number of LLM calls required to generate a single query as the number of accessed tables increases. Table 12 shows increases of +2.6 calls (from 1 to 2 tables), +3.9 (from 2 to 3), and +2.2 (from 3 to 4), indicating ncar-linear growth.
G ABLATION STUDY ON NESTING TYPES
For deeper analysis, we examined the best-performing system on the SPARTA Senchmark-HProPP with GPT-5-by by breaking down its performance across query-nesting types: plain nesting (N), nesting with aggregates (A), nesting with correlated joins (J), and nesting that combines correlated joins + aggregates (JA). Results shown below.
18

---

Published as conference paper ICLR 2026
Fout-ardar provanarce) Pows-crder- Prgy
7000
Foazionder provenarce) Porz-onder Prov
-32%%
1890
+-22%
+113%
390
+.22% +112%
Querv size
Chain Query Size
(a) LLM Call Count Star- Star-Query Size Increases.
(b) LLM Call Count Chain-Query Size Increases.
Figure 8: Generation cost for varying (a) star-query size and (b) chain-query size. Sky-blue bars mark ideal LLM calls, and the labels above each bar represent the actual excess percentage.
Table 12: Average LLM calls per query as the number of accessed tables increases.
Accessed Tables
Average LLM Calls per Query
2.3 2.6)
8.8 (+ 3.9) 11.0 (+ 2.2)
As shown in Table 13 F1 falls steadily structural (correlated joins) and analytical (aggregates) complexity increases, with the largest drop when both factors are present (Type JA). This ablation study underscores that correlated joins and aggregates the model's primary pain points.
H ANALYSIS ON NEGATION AND RANGE REASONING
To better understand which logical operators and conditions most challenge SPARTA models, we conduct an ablation study focusing on two key query categories: negation and numeric range conditions. These categories capture a large portion of structural breadth in SPARTA and represent frequent sources of model errors.
We evaluate HProPro with GPT-5 on queries containing explicit negation operators (NOT LIKE, NOT EXISTS, NOT IN, <>) as well numeric range operators (>, <, >= <=). Table presents the results.
Both categories show notable degradation compared to the overall SPARTA score, with negation queries dropping by 28.3% and range queries by 18.6%. These findings indicate that logical negation and numeric range reasoning remain significant bottlenecks.
addition this quantitative breakdown, we selected representative samples from both query types and conducted qualitative error analysis, shown in Figure 9 As illustrated in Tigure[[a) negation reasoning presents a recurring challenge. The example query requires identifying players for whom there no record indicating that they (i) are Center with height >> 90, (ii) were born after 1970, and (iii) were drafted "9th overall." All three criteria fall under the scope of single negated condition. However, the model applics negation only the first clause ("Center with height 90")) while incorrectly treating the remaining clauses ("born after 1970," "drafted 9th overall") as independent positive filters. This partial scoping of negation leads the model misinterpret the logical structure and include players who should have been excluded.
19

---

Published as conference paper ICLR 2026
Table 13: scores of HProPro (w/ GPT-5) across different nesting types. Percentage values indicate relative change from the overall average (34.5).
Nesting Type
F1 (HProPro w/ GPT-5)
Type-N
40.0
(+15.9%)
Type-A
33.7
(-2.3%)
Type-J
30.8
(017.)%)
Type-JA
25.6
(-25.8%)
Total
34.5
Table 14: Fl scores of HProPro (w/ GPT-5) on negation and range queries.
Query Category
(HProPro w/ GPT-5)
Negation (NOT LIKE, NOT EXISTS, NOT IN, <>)
28.7 (2..3%)
Range <=>
32.9 18.6%)
SPARTA (Oracle)
40.4
A similar pattern is observed for range reasoning, as shown in Figure The gold query requires identifying teams whose arena capacity exceeds the maximum capacity among teams that (i) were founded before 1970 and (ii) have capacities between 20,000 and 21,711. Although the model correctly computes the maximum capacity among pre-1970 teams, it fails to apply the upper bound constraint (arena_cagg) < 21, 71 711) during the final filtering stage. As result, the predicted code returns teams that satisfy the lower-bound conditions and the dynamic threshold but violate the required upper-bound range condition, producing an incorrect answer.
ty
I ABLATION STUDY ON ROBUSTNESS TO LINGUISTIC VARIABILITY
We further study the robustness of table-text qa models to linguistic variability by evaluating performance under human-verified verified rephrased questions. Specifically, we sampled 100 queries from our benchmark and had them manually rephrased by human annotators, ensuring the core semantic meaning and the correct answer were preserved. We then evaluated the HProPro model with GPT-5 on both the original and the rephrased sets of queries.
Table 15: F1 scores of HProPro (w/ GPT-5) on original and human-verified rephrased queries.
Query Set
(HProPro w/ GPT-5)
Original Questions Rephrased Questions
45.22
45.02 (-0.44%)
As shown in Table 15| the Fl score dropped from 45.22 on the original queries to 45.02 on the rephrased
versions, amounting to negligible decrease of 0.44%. This finding indicates that the model is highly robust to surface-level linguistic variations. We have incorporated the details of this new experiment and its results into the Appendix I of our revised manuscript to strengthen robustness analysis.
20

---

Published as conference paper ICLR 2026
What the masimum rebounds and more game, than and there no indlicoinn that Center height
Question:
who,
Quedtion: What the highest team
team
1970, with cepacity
caperity
capacity
owned by somcone other than Allen,
greater born after 1970, and drafted overall?? Answer: 19
with arana capacity below 21711, and which soared more than 100 points and rebounded mere than 40 times in game? Answer: 12?
solve(table)-
def solvetsblee str:
for row table:
max, before 1970 1970=Noree None
if not (row//wight]])) < 1100)
for row in teble:
continue
[Touddod year 19) 1970] and frow owner [[0] t- "Pac Allen)] and
If ('Center row]postion]]]] and wwghht[ig]]]])--- > and
(Dow arena, capacity 20000 and rowi rowi'arena, sspacity[[0] 21711]:
mu]bbrthywar([0]
and rowjdraft pisk]jo] Iveralf)
Frooffrrm caparity|[j]
belore,
before
coml'stena capestyy
passage_ pwwplpyyr__nmmellle.
table:
estrect the points with 25 rebounds and steals...
farenn papacity a max before 1970:
retum mox_ points
passage Iikk towmteom,_ nome [[]]
* estroct the points with =100 points and -40 resounds...
return max points before 1970
Predicted Code:
Predicted Code:
def solvettable) -t17
def solvejtable) 2.t
points =
max, before 1970 Noow
for row in table:
for row in table:
ifrot [rowfweighs](0]
(foowi/ year|[0) 1570) and (row) wwwee//eee1l)) Allen') and (ruel'arena capacilyyyyyylyyyyyyaoyy
cowfjareee paperclyyyylyy 2:1318)
[T('Centar [ooppostioo]]]] (ow[higght)))
Ihooofarena capecity (10) max, 1970:
missing!
continue
max before_ffreee. 1970 row brenn, papacityy [[0]
io (row(birtiverr])) > 1970 and row[droftpock]]00/ 9thh overalf):
for row in table:
continue
Mtarnna capacityy > max, wwworeeeooe______eeeeeee___e 1970:
passage lirks rowfolayer, name][1]
passage Jink - row]Twam, name [[A]
ixtract the points with as rebounds and stewis
ectract the points with 5-00 points and 240 rebounds...
Figure 9: Illustration of representative error cases where the model fails correctly answer. (a) Negation reasoning error. (b) Range reasoning error.
J ERROR CASE ANALYSIS
We conduct an analysis of the errors encountered by Table-Text QA models on randomly sampled sets of 100 examples each for SPARTA (Oracle) and SPARTA (Retrieval), as illustrated in Fig I0 Representative error types, along with their frequencies and causal interpretations, are summarized below.
Relevant data missing. This the most frequent category of failure, where the model failed to identify all the necessary information to correctly answer the question. SPARTA poses increased demands for multi-hop reasoning across table and text sources, which existing methods often struggle with:
The model identifies only subset of the necessary sources, resulting in incomplete answers. As illustrated in Figure [1] the model was expected return both 62 and 53 as the field goal percentages for the Dallas Haverrick and New York Knicks, respectively, but failed to do so.
Partial retrieval of relevant data:
Failure to identify relevant data: The model does not identify crucial supporting data, leading to either no answer or an incorrect one. For example, in questions requiring information from both abb__laaeer... informat iin and nba_ salp__plae___aaee former, overlooking the award records, and consequently returning an incorrect answer.
the model may access only the
Erroneous data analysis. Compared to prior benchmarks, SPARTA introduces more complex analytical requirements that reveal limitations in model capabilities:
Failure to perform advanced analytical operations: The model struggles with applying operations such as aggregations (e.g., COUNT, MAX) or executing multi-table joins correctly. These operations require precise alignment of relational structures and logic, which is frequently mishandled.
21

---

Published as conference paper ICLR 2026
Portial netraval relevant disa
Purfi:
migurde insunderase niderstanding 20 0%
Questton misunder peenstan ratandnn 18 0%
Question
Fallure idsentlly rolovan deta 33.00 33.0%
Relevant data missing agl
Ree Rellevant nt miss mg 0%.0%
Reeding amiinatign comee
palure idemnily relpeue relppient dara 3.
Reading compp eleeniinn wrrors
6.0%
Erroneous data analpals 31.0%
18 6.6%%
Erroneous data anaty sis 42
Fallure perform advancad
Fatlure perform advanced analy fical operation 1..0%
analy Eicll operation 13.0%
(a) Oracle Setting
(b) Retrieval Setting
Figure 10: Statistics of errors. For detailed descriptions and examples of each emor category, see Appendix
The model incorrectly interprets textual information, leading to erroneous answers. For instance, in where the question asks for the Nuggets' field goal percentage, the model erroneously extracts "37%" from the sentence "Nuggets held Sacramento to just percent from the field, " misattributing Sacramento's statistic to the Nuggets. See Figure 11 for a detailed example of this error.
Reading comprehension errors:
Question misunderstanding. These errors arise from incorrect interpretation of the question intent or con- straints. Representative cases include failing restrict answers to players who played only as point_ggurdd and instead including players who played point _guard along with other positions, misidentifying the relevant time frame (e.g., using 2017 instead of 2016-17), introducing constraints not specified in the question, or omitting key conditions necessary to derive the correct answer.
Schema linking errors. This category involves incorrect associations between the question and the schema elements, such as tables or columns. For instance, when asked to retrieve the name of the head coach, the model fails to identify the headcoach column in the nba_tttttt sttatttt_____aeeeeeeeeeeee info raat on table relevant, thereby omitting necessary information from the final prediction.
22

---

Published as conference paper ICLR 2026
[Question]
What are team fleld goal percentages of the teams that have an arena capacity between 19,000 and 20,000 and scored more than 100 points in a game? [Golden Answer]
42,
Mirneseta inberwativss defeated Portiand they
MBA Team Information
flaptorx cereated Charlette floretts, - managed pore gercent giolding
game
Name
Capacity
figure
impresotel
Trail Biosers
10,983
Homets viating
connecting on
Reptors
Nagpets
Chlcge
Nuggets
10,009
Mavericks
19,400
the Dallas Mavericks 110 3 thefateed the Los Angoles
The New York Knids 20 201 defeated the Boston Cobiis 115 19:120 134 Tuesstyy Knikks firishe the night
Lakers 3- 140 106 Friday They lowed
Mavs to shoot an popping 62 feld...
percent from the
with outstanding S3 percent shooting from the feeld
[LLM-Predicted Answer]: 55, 57
37
Correct
(Type Erroneous data analysis)
Error (Type Relevant data missing)
Figure 11: Illustration of representative error case where the model fails to correctly answer.
K
SOFTWARE AND DATA LICENSES
The licenses for the software and datasets used in this paper as follows:
LLaMA 3. Instuttnnttrut:: LLaMA 3.1
OTT-QA: MIT License
HybridQA: MIT License
All software and datasets used strictly for research purposes and were not utilized in any non-research contexts, particularly for commercial applications.
L AI ASSISTANTS
We used ChatGPT-4o OpenAl to debug code efficiently, quickly identifying and resolving errors in our implementations. Additionally, used it for rephrasing sentences our writing improve clarity and readability.
23

---

Published as conference paper ICLR 2026
M REPRESENTATIVE EXAMPLES FROM OUR SPARTA BENCHMARK
Table 16: 20 representative examples from SPARTA, each consisting of a domain, a natural language question, its corresponding SQL query, and the answer.
Row Type
Content
Domain
NBA
Question
Which player won the NBA MVP award in the 1986 scason?
SQL
SELECT player_nane _amee__nameeeee__ FROM nba_player_award award WHERE season 1986
AND award 'nba mvp'
Answer
Larry Bird
Domain
NBA
Question
What are names the players who scored more than points rebounded more than times game?
SQL
SELECT player_na name FROM nba player_game stats WHERE number_of_pote
15 AND jumber_of_rebeee > Langston Galloway, Quincy Acy, Larry Nance Jr....
Answer
Domain
Movie
Question
In which movies did Riteish Deshmukh act?
SQL
SELECT movie_title FROM role_mapping WHERE category 'actor' AND name = 'Riteish Deshmukh'
Answer
Marjaavaan, Mauli
Domain Question
Movie
What is the total number of movies with median rating greater than 5 and average rating greater than 5.5?
SQL
SELECT COUNT (movie_title) AS total_movies FROM ratings WHERE median_rating
AND avg_rating 5.5
Answer
4877
Domain
NBA
Question
Which Western Conference teams faced the Celties more than once in the Finals?
SQL
FROM nba__champion_histor WHERE nba _ra_champion champion_name _ame 'Ceticc'' GROUP BY western western_champion name
SELECT pestern_champion.
HAVING COUNT wwestern_hammpinn _ Rockets, Lakers
Answer
Domain
Medical
Question
What is the maximum years of experience of pediatrician at Central Hospital?
SQL
SELECT MAX ayears_epeerie FROM doctors WHERE
hospital_branch Central Hospital' AND specialization = Pediatrics' 28
Answer
Domain
NBA
Question SQL
What is the highest salary of Kevin McHale while playing for the Celtics?
SELECT MAX (salary) FROM nba_pt player_afff liation WHERE
player_paa name Keviin McHale' AND team_ name "Cetics'
(continued on next page)
24

---

Published as conference paper ICLR 2026
Row Type
Content (continued)
Answer
3,500,000
Domain
NBA
Question
Which Point Guards, drafted between 2000 and 2005, had more than three-pointers, more than 8 field goals and more than steal in game?
SQL
SELECT player_nawe _ame FROM nba Lal___plye______ gameeaamm___ Laie__aatats_aaaaaaaaaaaaiaaaaae stats WHERE position
WHERE player_name name IN (SELECT [aaee_nnane FROM
'Point Guard' AND draft_year BETWEEN 2000 AND 2005) AND
hba_player_infooree
Shumber_of_thee_pootee goall goals aooals_made _made > 4 AND
Luumbee__o_fiiel__ooalssssms > 8 AND number_of_steal > 1 Chris Paul
Answer
Domain
NBA
Question
Which NBA players who were drafted in the first round and play the center position have salary of over million in the 2016-17 scason?
SQL
SELECT player_ name FROM nba wwayerlla.r...mm information
eolaye__information
WHERE player_ name IN (SELECT player_naa _ame FROM
nba wwwpppaaaeraaaa affiliatioon WHERE salary > 1000000 AND season
= ='2016-17') AND draft_round = 1st round' AND position =
'Center'
Answer
Alex Len, Al Horford, Andre Drummond,
Domain
Medical
Question
What are the names of patients who have an appointment with a doctor who works at the central hospital and has more than 20 years of experience?
SQL
SELECT patient_nes
FROM appointments WHERE doctor_name IN (SELECT name FROM doctors WHERE hospital_branch Centtal Hospital' AND years_ _eaxs_eeeeeeeceeeeee 20)
Answer
Alex Smith, Alex Aiden Moore, Emily Miller,
Domain
NBA
Question
What are the years of birth of the players who have a lane agility time of more than 11.5 seconds, a three quarter sprint of less than 3.35 seconds, more than 10 field goals made and more than rebounds in game?
SQL
SELECT birthyear FROM nba_ laa_pllle___foraiiiio information
WHERE player_naal name IN (SELECT player_nanc FROM
WHERE lane teee_gggltty__tee
11.5
AND
< 3. 3.35) AND player_naa IN gaae_sttattesssseeeeeeeassaaaaee WHERE (SELECT player_name FROM nba_player: wwwpppllaaae_stteeeeeee number_of ltt_=ooffelld_gaal_______ GROUP BY birthyear 1984, 1985, 1989,
10 AND
Answer
Domain Question
Movie
Which movies, starring Vincent D Onofrio as actor, have average rating greater than 5 and median rating of 6, excluding 'Kolonya Cumhuriyeti"?
(continued on next page)
25

---

Published as conference paper ICLR 2026
Row Type
Content (continued)
SQL
SELECT title FROM movie WHERE title IN (SELECT movie_title FROM role_mme @ell__mapping WHERE category 'actor' AND name 'Vincent Onofrio' AND movie_title = title) AND title IN (SELECT movie_title FROM ratings WHERE avg_rating > 5 AND 6 AND novie_title <> 'Kolonya Cunhuriyeti')
Answer
CHIPS, In Dubious Battle
Domain
NBA
Question
Who are the top centers drafted in the 1st round. who have won the dpoy award after 2000, and who have earned more than 2 million dollars in the 2004-05 season, sorted by their draft year in descending order and birth year in ascending order?
SQL
SELECT player_name
FROM mba_ppaaerr_infomatha
WHERE player_name IN (SELECT player_name FROM
WHERE salary 2000000 AND season = 22004-05') AND player_name IN (SELECT player_name FROM mba_playee_awwrd WHERE season 2000 AND award 'dpoy') AND position = Ceeterr' AND draft_round = '1st round' ORDER BY draft year DESC, birthyear ASC LIMIT 5
nba wwppppllaaallllleeeep
Answer
Dwight Howard, Ben Wallace,
Domain
Medical
Question
Find the addresses of male patients born after January 1, 1980, who have MedCare Plus insurance and have made payments that exceed the average failed payments greater than 2500.
SQL
SELECT address FROM patients WHERE name IN (SELECT
patient_name FROM billing WHERE amount > (SELECT AVG (amount)
FROM billing WHERE payment_statul stauuussstuuuss 'Failed' AND amount > 2500) AND date of birth > '1980-01-01' AND gender "M" AND insurance_prov der 'MedCare Plus' 123 Elm St, 789 Pine Rd,
Answer
Domain
NBA
Question
Which NBA players, who centers and taller than the average height of point guards drafted after 1990, have than 8 rehounds in game?
SQL
SELECT player_nard _ame FROM
Stayerrgame_stat..
WHERE player_name IN (SELECT player_name FROM
aba_player_informatiie
WHERE height > (SELECT AVG (height)
FROM nba_ abb__plaerriifoomat..e Guard' AND draft_ year > 1990) AND position = 'Center') AND number of rebound
WHERE position 'Point
Answer
Alex Len, Al Horford, Andre Drummond,
Domain Question
Medical
What are the names of female patients who registered after 2021-09-02 and have billed amounts greater than the average amount of failed payments over 2500?
SQL
SELECT name FROM patients WHERE name (SELECT patient_namd FROM billing WHERE amount > (SELECT AVG (amount) FROM billing WHERE payment_statio gender "F' AND registration_date > "2021-09-0 Emily Jones, Laura Aiden Davis,
'Failed' AND amount 2500) AND
Answer
(continued on next page)
26

---

Published as conference paper ICLR 2026
Row Type
Content (continued)
Domain
Movie
Question
How many movies starring John Abraham have median rating above and average rating 4?
SQL
SELECT COUNT (title) AS number _f_ooiies FROM movie WHERE
title (SELECT movie_title FROM role_man @ole_mapping WHERE
category = 'actor' AND name = John Abraham' GROUP BY movie_title) AND title IN (SELECT movie_title FROM ratings WHERE median_rating > 5 AND avg_rating > 4)
Answer
Domain
NBA
Question
What is maximum height the Lakers players who play as center. were drafted after and salary greater than the highest salary the Suns and greater than 20,000,000?
SQL
SELECT MAX (height) FROM nba player pwwyerriinormmeeeeeeeeeeeeeaeeee informationn WHERE player_name IN (SELECT player_name
FROM
nba Lba_ppayer___ hwlppple__aflliattteee affiliaioon WHERE salary (SELECT MAX (salary) WHERE team_neaaa name = Suns'") AND salary 20000000 AND team_name 'Lakers') AND position 'enter' AND draft_year 1995
FROM nba_
Answer
84
Domain
NBA
Question
What the names of the teams that scored more than the highest points scored by the Thunder when they scored than 25 points in the first quarter and scored teams that scored more than 100 points and had three point field goal pereentage more 30 have an arena capacity of more than 20,000 and are not the Pistons?
than the highest points scored by
SQL
SELECT team_ name FROM nba nba_team, (SELECT MAX (team_popg pattammppontts)__ FROM 'Thunder' AND ream_points_in__uaaee 25) AND team_points > (SELECT MAX (tteamppoints))
stats WHERE team_pol nts
WHERE team_name
FROM nba_team_gemm__ttn
WHERE team_points 100 AND @eam_percenteee of Lattiaage_____heeeeeeeee
felll_eeello_lfl haad__oal_maaa _aal_madee__ 30) AND
team name IN (SELECT team name FROM nba_ team information
WHERE arena_capacity > 20000 AND team_name <> 'Pistons') Bulls
Answer
Domain
Movie
Question
Which movies directed Vivek Athreya have median rating greater than 5 with more than 100 total votes, and not feature Matt Smith as an actor?
SQL
SELECT title FROM movie WHERE title IN (SELECT
T2.
FROM = Vivek Athreya' AND movie.title T2. (2..oiee_tttee) AND NOT title IN (SELECT movie_title FROM role_mapping WHERE category 'actor' AND name 'Matt Smith') AND title IN (SELECT movie_title FROM ratings WHERE median_ran median_rating > 5 AND total_votes > 100 GROUP BY movie_title)
peector_mapping AS T2 WHERE T2 name
Answer
Brochevarevanura, Mental Madhilo
27

---

Published as conference paper ICLR 2026
N PROMPT TEMPLATES
We define suite of prompt templates that guide LLMs to generate executable, semantically coherent SQL queries. Prompts are organized into three categories, with an NBA domain example provided; for other domains, only domain-specific tokens swapped (c.g., replacing "NBA" with "Movie").
Clause-Level Generation. Templates for generating individual SQL clauses in canonical order:
SELECT (non-aggregate, aggregate)
FROM
WHERE
GROUP BY
HAVING
ORDER BY
LIMIT
Nested Predicate Construction. Templates for building multi-hop queries via nested predicates:
Inner Query Selection
FROM Clause for Outer Block
Nested Predicate Generation: Type-N, Type-A, Type-J, Type-JA
Refinement and Evaluation. Templates to improve query validity and assess realism:
Provenance-Based Refinement for repairing empty-result queries
Naturalness Evaluation to assess relevance and intent clarity
28

---

Published as conference paper ICLR 2026
WHERE Clause Generation
You are both NBA fan and an SQL expert. Given the provided database and the generated clauses, generate WHERE clause that reflects authentic NBA-related curiosity.
Ensure the following requirements:
Output Structure: Return JSON object containing a single key, "where", with its value being WHERE clause.
Ensure NBA Fan Relevance: Generate the WHERE clause that aligns naturally with realistic and meaningful queries that NBA fans are likely ask.
Maintain Specificity and Clarity of Intent: Generate the WHERE clause that is well-defined, avoiding overly vague or artificially complex queries.
Align with Generated Clauses: Ensure that WHERE clause maintains logical consistency with previously generated clauses, preserving semantic coherence.
Ensure Synthetic Correctness: Generate the WHERE clause that is syntactically correct and executable on the provided database.
IMPORTANT: Do generate conditions for NULL None values, Also, avoid generating filter conditions that duplicate any existing filters.
Database: (database)
Generated Clauses: (generated_clauses]
Return the results in FLAT JSON format. DO NOT include any explanations or notes the output. ONLY return JSON.
29

---

Published as conference paper ICLR 2026
GROUP BY Clause Generation
You are both NBA fan and an SQL expert. Given the provided database and the generated clauses, generate a GROUP BY clause that reflects authentic NBA-related curiosity.
Ensure the following requirements:
Output Structure: Return a JSON object containing a single key, "group", with its value being GROUP BY clause. The GROUP BY clause should include single column.
Ensure NBA Fan Relevance: Generate the GROUP BY clause that aligns naturally with realistic and meaningful queries that NBA fans are likely to ask.
Align with Generated Clauses: Ensure that the GROUP BY clause maintains logical consis- tency with previously generated clauses, preserving semantic coherence.
Ensure Synthetic Correctness: Generate the GROUP BY clause that is syntactically correct and executable the provided database.
Do not group by any column whose value is fixed by equality (=) condition in the WHERE clause.
IMPORTANT:
Database: (database)
Generated Clauses: (generated_clausee)
Return the results in FLAT ISON format. DO NOT include any explanations notes the output. ONLY return JSON.
30

---

Published as conference paper ICLR 2026
HAVING Clause Generation
You are both NBA fan and an SQL expert. Given the provided database and the generated clauses, generate HAVING clause that reflects authentic NBA-related curiosity.
Ensure the following requirements:
Output Structure: Return a JSON object containing single key, "having", with its value being HAVING clause.
Ensure NBA Fan Relevance: Generate the HAVING clause that aligns naturally with realistic and meaningful queries that NBA fans are likely ask.
Maintain Specificity and Clarity of Intent: Generate well-defined and clear HAVING clause without making it overly narrow or contrived.
Align with Generated Clauses: Ensure that the HAVING clause maintains logical consistency with previously generated clauses, preserving semantic coherence.
Ensure Synthetic Correctness: Generate the HAVING clause that is syntactically correct and executable on the provided database.
Database:
(database)
Generated Clauses:
(generated_clause]]
Return the results in FLAT JSON format. DO NOT include any explanations notes in the output. ONLY return JSON.
ORDER BY Clause Generation
You are both NBA fan and an SQL expert. Given the provided database and the generated clauses, generate an ORDER BY clause that reflects authentic NBA-related curiosity.
Ensure the following requirements:
Output Structure: Return a ISON object containing a single key, "order", with its value being an ORDER BY clause.
Ensure NBA Fan Relevance: Generate the ORDER BY clause that aligns naturally with realistic and meaningful queries that NBA fans are likely ask.
Align with Generated Clauses: Ensure that the ORDER BY clause maintains logical consis- tency with previously generated clauses, preserving semantic coherence.
Ensure Synthetic Correctness: Generate the ORDER BY clause that is syntactically correct and executable the provided database.
Database:
(database)
Generated Clauses:
(generated_clausee
Return the results in FLAT JSON format. DO NOT include any explanations or notes the output. ONLY return JSON.
31

---

Published as conference paper ICLR 2026
LIMIT Clause Generation
You are both NBA fan and an SQL. expert. Given the provided database and the generated clauses, generate LIMIT clause that reflects authentic NBA-related curiosity.
Ensure the following requirements:
Output Structure: Return a JSON object containing a single key, "limit with its value being LIMIT clause.
Ensure NBA Fan Relevance: Generate the LIMIT clause that aligns naturally with realistic and meaningful queries that NBA fans are likely to ask.
Align with Generated Clauses: Ensure that the LIMIT clause maintains logical consistency with previously generated clauses, preserving semantic coherence.
Ensure Synthetic Correctness: Generate the LIMIT clause that is syntactically correct and executable on the provided database.
Database:
(database)
Generated Clauses:
(generated_cllases))
Return the results in a FLAT ISON format. DO NOT include any explanations notes in the output. ONLY return JSON.
32

---

Published as conference paper ICLR 2026
SELECT Clause (Non-Aggregate
You are both an NBA fan and an SQL expert. Given the provided database and the generated clauses, generate a SELECT clause that specifies necessary field for retrieving meaningful NBA-related data.
Ensure the following requirements:
Output Structure: Return a JSON object containing a single key, "select", ", with its value being a SELECT clause that projects single column without aggregation function meaningfully.
Ensure NBA Fan Relevance: Generate the SELECT clause that aligns naturally with realistic and meaningful queries that NBA fans are likely to ask.
Align with Generated Clauses: Ensure that the SELECT clause maintains logical consistency with previously generated clauses, preserving semantic coherence.
Ensure Synthetic Correctness: Generate the SELECT clause that syntactically correct and executable on the provided database.
IMPORTANT:
Do not project columns used in the WHERE clause.
Database: (database)
Generated Clauses: (generated
Return the results in FLAT JSON format. DO NOT include any explanations or notes the output. ONLY return JSON,
33

---

Published as conference paper ICLR 2026
SELECT Clause (Aggregate)
You are both NBA fan and an SQL expert. Given the provided database and the generated clauses, generate a SELECT clause that aggregates a single column for retrieving meaningful NBA-related statistics.
Ensure the following requirements:
Output Structure: Return JSON object containing single key, "select" " with its value being a SELECT clause that aggregates (MAX, MIN, AVG, or COUNT, etc.) a single column meaningfully.
Ensure NBA Fan Relevance: Generate the SELECT clause that aligns naturally with realistic and meaningful queries that NBA fans are likely to ask.
Align with Generated Clauses: Ensure that the SELECT clause maintains logical consistency with previously generated clauses, preserving semantic coherence.
Ensure Synthetic Correctness: Generate the SELECT clause that is syntactically correct and executable on the provided database.
Database:
(database)
Generated Clauses:
ppenerated_clauset
Return the results in FLAT JSON format. DO NOT include any explanations notes in the output. ONLY return JSON.
34

---

Published as conference paper ICLR 2026
Inner Query Block Selection
You are both an NBA fan and an SQL expert. Given the provided database, generated clauses, and the candidate inner query blocks, select the most appropriate inner query block for generating nested predicate that reflects authentic NBA-related curiosity.
Select the most appropriate inner query block to generate a nested predicate that aligns naturally with realistic and meaningful multi-hop queries NBA fans are likely ask.
Your output must be in ISON format with the key:
inner_quert _ew_euuuurlocckee block". ": Select the most appropriate inner query block from the Candi- date Inner Query Blocks.
IMPORTANT:
Do not select the inner query block that has already been used in the generated clauses and is not included in the candidate inner query blocks.
Database: (schema)
Generated FROM Clause: (yenerated_fron_clluus
Generated WHERE Clause: (generated_where_claar))
Candidate Inner Query Blocks:
Return the results in FLAT JSON format. DO NOT include any explanations or notes the output. ONLY return JSON,
35

---

Published as conference paper ICLR 2026
FROM Clause Generation
You are both NBA fan and an SQL expert. Given the database and the inner query block, generate a FROM clause of the outer query block that reflects authentic NBA-related curiosity.
Ensure the following requirements:
Output Structure: Return a JSON object containing a single key, "from", with its value being single-table FROM clause of the outer query block from the provided database (i.e., do not include any sub-selects or nested queries directly in the FROM clause).
Ensure NBA Fan Relevance: Generate the FROM clause that aligns naturally with realistic and meaningful multi-hop queries that NBA fans likely to ask.
Ensure Synthetic Correctness: Generate the FROM clause that is syntactically correct and executable on the provided database.
Separate Inner Query:
The inner query block must remain separate; it should later be incorporated into the WHERE clause, not nested in the FROM clause.
Ensure Natural Connection: Choose an outer table whose columns can be naturally referenced filtered against the results of the inner query block.
IMPORTANT: If the inner query block performs aggregation in the SELECT clause and outer table includes the aggregated columns, reuse the table referenced in the inner query as the outer table.
Database: (schema) }
Inner Query Block: (subquery)
Return the results in a FLAT ISON format. DO NOT include any explanations notes the output. ONLY return JSON.
36

---

Published as conference paper ICLR 2026
Type-N Nested Predicate Generation
You are both an NBA fan and an SQL expert. Based on the given database, generated clauses, selected inner query block Q and its execution result, generate type-n nested predicate that reflects authentic NBA-related curiosity.
Ensure the following requirements:
Ensure type-n Nesting: The inner query block Q must not contain a join predicate that references the relation of the outer query block, and its SELECT clause must project column without an aggregate function.
Ensure NBA Fan Relevance: Generate the nested predicate that aligns naturally with realistic and meaningful multi-hop queries that NBA fans likely ask.
Ensure Synthetic Correctness: Generate the nested predicate that is syntactically correct and executable on the provided database.
Ensure Semantic Alignment: If the inner query's 's SELECT column does not semantically match any column in the outer query's table, revise it for consistency.
The type-n nested predicate must be in the form: OuterTable column [IN NOT IN] (
Your output must JSON format with the keys:
peesteed_redicctee" Only the type-n nested predicate based the selected inner query block.
"Logical_operate" Ifa WHERE clause exists, return 'AND' or 'OR'.
IMPORTANT:
Ensure that the nesting level of the inner query block correctly preserved. The expected nesting level is (height).
Do not modify the nesting level of the provided inner query block. (schema)
Database:
Generated FROM Clause of the Outer Query: (generateed_fromcchl
Generated WHERE Clause the Outer Query: [ggneratted_where_clausss
Selected Inner Query Block Q:
Return the results in a FLAT ISON format. DO NOT include any explanations notes the output. ONLY return JSON.
37

---

Published as conference paper ICLR 2026
Type-A Nested Predicate Generation
You are both an NBA fan and SQL expert. Based on the given database, generated clauses, selected inner query block, and its execution result, generate type-a nested predicate that reflects authentic NBA-related curiosity.
Ensure the following requirements:
The inner query block Q must not contain a join predicate referencing the outer query's s relation, and SELECT clause must contain an aggregate function associated with column.
Ensure type-a Nesting:
Ensure NBA Fan Relevance: Generate the nested predicate that aligns naturally with realistic and meaningful multi-hop queries that NBA fans likely ask.
Ensure Synthetic Correctness: The predicate must executable and logically valid over the schema.
The type-a nested predicate must follow the form: OuterTable. collum columm [=
<=
>=
with aggregate
function
Your output must be in ISON format with the keys:
chested_predicatte Only the type-a nested predicate based on the selected inner query block.
"Logical_opprrtoor" If WHERE clause exists, return 'AND' 'OR'.
IMPORTANT:
Do not revise the SELECT clause the Q.
Ensure that the nesting level remains theight!;
Database: (schema)
Generated FROM Clause of the Outer Query: (generated_frommcclause
Generated WHERE Clause Outer Query: [gemeraated_wweere_cllussss
Selected Inner Query Block Q: (selected (selected_imerr_query) htwpeeeeeeeeeeeeeeeeeeeeee
Return the results in a FLAT ISON format. DO NOT include any explanations notes in the output. ONLY return JSON.
38

---

Published as conference paper ICLR 2026
Type-J Nested Predicate Generation
You are both an NBA fan and an SQL. expert. Based on the given database, generated clauses, selected inner query block, and its execution result, generate type-j nested predicate that reflects authentic NBA-related curiosity.
Ensure the following requirements:
Ensure type-j Nesting: Revise the inner query block Q to ensure it includes join predicate in its WHERE clause that references the outer query's relation, and its SELECT clause project a column without an aggregate function.
Ensure NBA Fan Relevance: Generate the nested predicate that aligns naturally with realistic and meaningful multi-hop queries that NBA fans likely ask.
Ensure Synthetic Correctness: Generate the nested predicate that is syntactically correct and executable on the provided database.
If the inner query's 's SELECT column does not semantically match any column in the outer query's table, revise it for consistency.
Ensure Semantic Alignment:
The type-j nested predicate must be in one of the following forms: OuterTable, column (IN NOT IN] (SELECT [join predicate] ...)
FROM
WHERE
[EXISTS NOT EXISTS] (SELECT predicate] ...)
FROM
WHERE
[join
Your output must be in JSON format with the keys:
"nested praericcepeeicccte.. Only the type-j nested predicate based on the selected inner query block.
laggcccllpperator" If WHERE clause exists, return 'AND' or 'OR'.
IMPORTANT:
The join predicate involving the outer query's 's relation must appear in the WHERE clause of Q, not its FROM clause.
The expected nesting level is (height), Do not modify it.
Database: (schema)
Generated FROM Clause of the Outer Query:
(generatee_frrmm_cl
Generated WHERE Clause of the Outer Query: [ggneratte__wherecclluss
Selected Inner Query Block Q: [selectee__imer__ueeybbbllle
Return the results in FLAT ISON format. DO NOT include any explanations notes the output. ONLY return JSON.
39

---

Published as conference paper ICLR 2026
Type-JA Nested Predicate Generation
You are both an NBA fan and an SQL. expert. Based on the given database, generated clauses, selected inner query block, and its execution result, generate type-ja nested predicate that reflects authentic NBA-related curiosity.
Ensure the following requirements:
Ensure type-ja Nesting: Revise the inner query block Q to include join predicate in its WHERE clause that references the outer query's relation and ensure its SELECT clause contains an aggregate function.
Generate the nested predicate that aligns naturally with realistic and meaningful multi-hop queries that NBA fans are likely to ask.
Ensure NBA Fan Relevance:
Ensure Synthetic Correctness: The resulting predicate must be executable and valid over the database schema.
The type-ja nested predicate must follow of the forms: OuterTable column [= !=
>=] (SELECT [agg] [join predicate] ...)
<= |
FROM WHERE
or [EXISTS NOT EXISTS] (SELECT [agg] [join predicate] ...)
FROM
WHERE
Your output must be in ISON format with the keys:
"nested pressted_redicctee" Only the type-ja nested predicate based on the selected inner query block.
If WHERE clause exists, return 'AND' 'OR'.
IMPORTANT:
The join predicate involving the outer query's 's relation must appear in the WHERE clause, not the FROM clause.
Do not revise the SELECT clause of the Q.
Do not modify the nesting level ((height))
Database: (schema)
Generated FROM Clause of the Outer Query:
Generated WHERE Clause of the Outer Query: (ggnnerated_whereecllusss
Selected Inner Query Block Q:
Return the results in FLAT ISON format. DO NOT include any explanations or notes the output. ONLY return JSON.
40

---

Do not delete the join predicate in the WHERE clause of the problem- atic subquery Q (e.g., WHERE outer_table name. cowumn_naamnnnnmmmm inner_table_rame.com Published as conference paper ICLR 2026
Provenance-based Refinement
You are both NBA fan and SQL expert. Based the given original SQL query, provenance analysis results, and problematic subquery or condition which filters out all the rows, fix the original query's problematic subquery or condition so that it retrieves some results from the database.
Ensure the following requirements:
1) Output Structure: Return a JSON object containing single key, "corrrecee__qeer"" with its value being the corrected SQL query.
2) Ensure NBA Fan Relevance: Maintain the original query's NBA-related curiosity and focus on realistic and meaningful queries that NBA fans likely ask.
IMPORTANT:
You may add an additional predicate the inner query or adjust the filtering threshold within the problematic subquery Q to intentionally include the important rows or exclude outlier rows (e.g., those with extremely high or low values) that overly constrain the outer query.
You may also adjust the comparison operator (e.g., > >=, <=) the value of the problematic condition to relax the filtering criteria.
Original SQL Query: (query)
Problematic Condition: (problematic_conditte
Problematic Subquery Q: grroblematiccsubqueere
Execution Result of the Subquery Q:
Provenance Analysis Results: (proveeaance_aaallysarrrrrrueeee
Return the results in FLAT JSON. NEVER include ANY EXPLANATION or NOTE in the output, ONLY OUTPUT JSON.
41

---

Published as conference paper ICLR 2026
SQL Query Naturalness Evaluation
You have set of evaluation criteria to judge whether given SQL query reflects question that is likely to be asked by typical person.
When evaluating the query, refer to the following points:
1. Relevance:
Definition: Measures how likely it is that a real person would be interested in the query.
Low Score (1): The query covers obscure or highly technical aspects unrelated to typical person discussions (e.g., internal database IDs rarely discussed statistics). High Score (5): The query reflects common, popular interest among people (e.g.. game stats, player/team information, draft results, etc.).
2. Specificity & Clarity of Intent:
Definition: Evaluates whether the question clearly targeted and sufficiently detailed reveal genuine NBA-related therrestwithhou being narrow to be contrived. Low Score (1): The query too vague ("Show me some NBA data") or overly convoluted/contrived.
High Score (5): The query clearly captures plausible question (e.g.. "Which NBA player scored the most points in home games last month?").
3. Overall Naturalness:
Combine the above criteria and decide if the query is "natural" (likely be asked by real person) or "unnatural".
The query is considered natural if its overall score is 3 or higher.
Your output must be in ISON format with the following keys:
"relevance_score" sccoee: Integer from to
Integer from to 5.
" "overral Integer from to 5.
"reason": Explanation referencing the scores and justifying whether the query is consid- ered natural or unnatural.
Database Schema: (database_schent
SQL. Query Template: (question)
Return the results in a FLAT JSON format. NEVER include ANY EXPLANATION NOTE in the output, ONLY OUTPUT JSON.
42


Reviews:
[{"review_id": "3nGiK1pKJS", "reviewer": "Reviewer_JYtH", "summary": "This paper proposes SPARTA, a Text-Table Q&A dataset which is primarily generated by LLM prompts, with light human evaluation. This allows the authors to generate a large corpus of Q&A problems, over varying aspects of the problem instance. The authors provide extensive and high-quality evaluation of the dataset, including qualitative and ablation results.", "questions": "1. Could you more succinctly summarize the aspects which are tested by this dataset, providing toy examples? (lengthy) examples are deep in the appendix. Can this be promoted to the main text in a better way than Table 1? (Perhaps the text around Table 1 can be more contrastive), or a taxonomic figure can be added. \n\n2. Are there aspects which were omitted from the dataset at design-time?\na. robustness as described above\nb. chain of thought reasoning\nc. user evaluation to evaluate naturalness and scope of Q&A \nd. grounded attribution and/or explanation quality\ne. potential bias in evaluation, e.g. evaluation over generated sensitive attribute subgroups or subgroup-perturbations (curation scope r.e. Weakness 3 above)\n\nCould you summarize the challenges or considerations in scoping the evaluation as you did?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, this paper is very well written and I see great value in the proposed dataset. \n\n1. The scope and scale of the dataset is seemingly quite novel. There is great care to the tested aspects of the dataset, and the overall size is sufficiently large for high-quality model comparison. \n\n2. Each aspect of the dataset design is well motivated, and easy to interpret. This paper is interpretable for the general AI research reader. \n\n3. The dataset evaluation is sufficient and provides several insightful, qualitative results.  e.g. Fig 5, Table 6. \n\n4. The authors give a very extensive appendix with examples.", "weaknesses": "1. A primary weakness is the lack human-written and human-curated Q&A instances. While the synthetic generation methodology is impressive and well-evaluated, the authors could provide small curated data subsets for testing certain aspects of reasoning. For example, reasoning over ranges or negations, data inconsistencies (e.g. where text and tables have differing values for a specific fact), etc. \n\n2. Similarly, While the scope of SPARTA is impressive and there is variation over structural, breadth, and clause diversity, more qualitative evaluation could be provided related to breadth in weakness (1). It is difficult to understand *what* model aspects are being tested (and not tested) over the entirety of the dataset. \n\n3. The dataset doesn't include human-verified perturbation groups for robustness analysis. Since machine-guided perturbation models often change the result of complex queries, it would be valuable to validate over a subset of human-verified perturbations (e.g. query rephrasing that retains the same output). \n\n4. Table 6 is somewhat concerning. Given poor text performance, is this problem primarily Tabular Q&A? Does SPARTA have a view where users can consistently use it for Tabular Q&A evaluation?(e.g. no comparability errors from users).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fK1jACrYMk", "reviewer": "Reviewer_kiAT", "summary": "This paper proposed a framework to scale up synthetic data generation for multi-hop question answering over table+text contexts.\nIt creates new tables by merging existing tables with atomic facts extracted from texts, then use a subset to verbalize as the text context.\nQuestions are generated using LLMs to build nested SQL queries bottom-up, then verbalize the queries that will yield result after execution. It also introduced an automated provenance-based approach to fix SQL query.\n\nContributions\n1. The approach seems reasonable for constructing multi-hop type questions\n2. Benchmark is challenging and shows there's still significant headroom for improvements.", "questions": "Mostly minor suggestions on presentations, since this paper has a lot of details and analysis\n\n1. What is the why-not based provenance approach (referenced paper only has why-provenance)? I can kind of infer from Figure 3 but maybe have some textual description would help?\n2. How is provenance-based approach used to help fix the error => seems like the provenance analysis is added to LLM prompt to ask LLM to fix the query/relax the offending constraints, but it is unclear to me how this is used & what the inferior alternatives are doing when I first read through it -- especially in section 3.3.2 -- and Figure 3 only covers Provenance-based approach, so at first it was confusing how the alternatives are doing the fixes (plus figure 3 is small). Maybe adding some textual descriptions to clarify could help", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Paper is well written and clear. Results demonstrate effectiveness of the dataset construction approach\n2. Automated fixing and provenance-based fixing is a pretty creative idea\n3. Substantial analysis of model failures on this dataset is also presented", "weaknesses": "1. seems to be missing the constructed dataset statistics? e.g. how many reference tables are constructed per source dataset?\n2. scalability of this approach seems to be bottlenecked by number of source tables?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gTf5nmUgEy", "reviewer": "Reviewer_iHeT", "summary": "This paper introduces SPARTA, a large-scale Table–Text Question Answering (QA) benchmarks. Unlike existing small and shallow datasets, SPARTA generates complex multi-hop and aggregation-based QA pairs by synthesizing executable SQL queries and corresponding natural-language questions. Through provenance-based refinement and realistic-structure enforcement, it produces thousands of high-quality question–answer pairs with only a quarter of HybridQA’s annotation cost, revealing significant performance drops in state-of-the-art models and highlighting current limitations in cross-modal reasoning.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clear and easy to follow.\n2. The analysis of existing benmarks is comprehensive.\n3. Several experiments along with analysis on this benchmark.", "weaknesses": "1. **Excessive Dependence on High-Capacity LLMs for Pipeline Efficiency**.\nThe efficiency of the Provenance-based Refinement loop relies heavily on the advanced reasoning capability of a large LLM (Llama-3.1-70B-Instruct) to accurately diagnose and correct erroneous SQL predicates. However, the paper does not analyze how the framework’s quality and cost-effectiveness (central to its “Scalable” claim) would be affected when using smaller or less capable LLMs. This raises concerns about the robustness, reproducibility, and general accessibility of the proposed generation pipeline.\n\n2. **Lack of Fine-Grained Error Attribution in End-to-End Settings.**\nThe error analysis highlights major failure categories such as “Relevant data missing” and “Erroneous data analysis”. However, in the end-to-end setting, it remains unclear whether these failures stem from inherent weaknesses in the reasoning and program execution components or from compounded upstream retrieval errors. A more fine-grained breakdown of error contributions across the Retrieval and Reasoning stages is essential to accurately diagnose the true limitations revealed by SPARTA.\n\n3. **Limited Rigor in Query Naturalness Evaluation.**\nThe evaluation of question naturalness relies on verbalizing SQL queries using LLMs (AST-ICL and ChatGPT-4o). This top-down SQL → NL generation process may not faithfully reflect the organic intent or linguistic variability of real user questions. As a result, the claimed “realism” of the complex, tree-structured questions could be biased or overstated.\n\n4. **Oversimplification of Reference Fact Database Construction.** The foundational step of the pipeline involves merging source tables with Grounding Tables, which are created by automatically extracting atomic facts from unstructured text passages. However, the paper primarily relies on a pre-processed corpus (ROTOWIRE) within the NBA domain, and provides insufficient technical detail on the general and robust mechanisms for extracting and normalizing atomic facts from arbitrary text into structured Grounding Tables. This omission raises concerns about the reliability and domain-agnostic portability of the proposed pipeline.\n\n5. This dataset contains three subsets only, which could be expanded in to larger range of QAs.\n\n\nOverall, I believe there is still considerable room for improvement in this paper, and the weaknesses outweigh the strengths. Therefore, I assign a final rating of borderline reject.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RAjZ7l85FV", "reviewer": "Reviewer_N2ch", "summary": "SPARTA introduces a large-scale, principled benchmark for Table-Text Question Answering (QA) by addressing critical limitations—shallow reasoning, small data scale, and annotation noise—inherent in prior manual datasets like HybridQA. The core methodology involves constructing a unified Reference Fact Database by incorporating text-derived atomic facts into structured Grounding Tables, enabling an LLM-guided pipeline to synthesize highly complex, executable SQL queries that model multi-hop, tree-structured reasoning across modalities. Utilizing novel techniques like provenance-based refinement and post-order structural enforcement, SPARTA efficiently generates thousands of high-fidelity question-answer pairs covering advanced analytical operations such as grouping and aggregation, leading to a profound performance collapse (over 30 F1 points) in state-of-the-art QA models, thereby establishing a new, rigorous standard for cross-modal reasoning evaluation.", "questions": "same as weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strength 1: High-Fidelity Scaling and Data Rigor SPARTA dramatically increases the realism of Table-Text QA by operating over large-scale databases featuring thousands of rows (e.g., 3,280.5 mean cardinality), moving well beyond the small, toy-scale web tables typically used in previous benchmarks. This substantial increase in data size rigorously evaluates model performance in environments demanding true large-database reasoning and efficient indexing.   \n\n\nStrength 2: Comprehensive Coverage of Tree-Structured Multi-Hop and Analytical Reasoning The framework systematically generates questions covering advanced analytical operations such as aggregation and grouping, which were largely absent from prior benchmarks. Furthermore, it features deep, tree-structured multi-hop reasoning, which goes beyond the simplistic linear chains of inference found in existing QA resources.   \n\n\nStrength 3: Cost-Effective Rigor via Provenance The generation pipeline achieves high data quality and efficiency by employing Provenance-based Refinement, guaranteeing every synthesized SQL query is executable and returns a non-empty result. This mechanism ensures semantic validity while drastically reducing the cost of generation, requiring only one quarter of the annotation time compared to manual benchmarks like HybridQA.", "weaknesses": "Weakness 1: Undocumented Quality of Atomic Fact Extraction The crucial first step of automatically extracting atomic facts from unstructured text into grounding tables lacks reported validation or error analysis. If errors in this unquantified automated extraction process propagate, they could undermine the benchmark's fundamental data grounding and rigor.   \n\n\nWeakness 2: Acute Cross-Modal Fragility in Program-Based Models The inclusion of unstructured text causes a significant F1 score drop (e.g., 63.9% for HProPro), indicating fundamental brittleness in program-based reasoning when integrating facts from relational tables and textual grounding tables. This massive performance collapse suggests inadequate logical bridging and factual integration capabilities, especially for complex analytical queries that span the heterogeneous data store.   \n\n\nWeakness 3: Vulnerability to Correlated Joins and Aggregation Models demonstrate pronounced vulnerability to the synergistic complexity introduced by combining correlated joins and aggregation (Type-JA queries), yielding the lowest reported F1 scores. This quantitative failure highlights that state-of-the-art models struggle with highly constrained, context-sensitive analytical calculations typical of advanced database logic.   \n\n\nWeakness 4: Scaling Limitations Across Query Structural Dimensions Empirical results show model performance degrades consistently and sharply as query complexity increases along either the depth (chain length) or breadth (branching structure) dimensions of the query graph. This suggests current methods fail to generalize tree-structured reasoning, exhibiting fundamental computational limitations in managing multiple, concurrent relational paths.   \n\n\nWeakness 5: Costly Dependence on LLM Corrective Feedback Despite efficiency efforts, the nested query generation pipeline is inherently reliant on numerous expensive error-correction loops, requiring 53.6% more LLM calls than the theoretical ideal. This high overhead dedicated solely to provenance-based repair reveals the intrinsic unreliability of LLMs in autonomously generating semantically sound SQL without database execution feedback.   \n\n\nWeakness 6: Limited Scope and Insufficient Validation Granularity The benchmark is confined strictly to the Table-Text modality, neglecting other real-world data sources like images or video, thus limiting its applicability in the evolving multimodal QA landscape. Furthermore, the human validation is explicitly \"lightweight,\" avoiding verification of the complex multi-hop reasoning path itself, which risks overlooking subtle semantic misalignment between the generated SQL and the final natural language question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)