BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation via Lens of Dynamic Interactions
Authors: Nan Huo
Xiaohan Xu
Jinyang Li
Per Jacobsson
Shipei Lin
Bowen Qin
Binyuan Hui
Xiaolong Li
Ge Qu
Shuzheng Si
Linheng Han
Edward Alexander
Xintong Zhu
Rui Qin
Ruihan Yu
Yiyao Jin
Feige Zhou
Weihao Zhong
Yun Chen
Hongyu Liu
Chenhao Ma
Fatma Ozcan
Yannis Papakonstantinou
Reynold Cheng
Venue: ICLR 2026 Oral (2026)
Keywords: Interactive
Text-to-SQL
LLM
Code Generation
Primary area: datasets and benchmarks

Abstract: Large language models (LLMs) have demonstrated remarkable performance on single-turn text-to-SQL tasks, but real-world database applications predominantly require multi-turn interactions to handle ambiguous queries, execution errors, and evolving user requirements. Existing multi-turn benchmarks fall short of capturing this complexity, either by treating conversation histories as static context or by limiting evaluation to narrow, read-only (SELECT-ONLY) operations, thereby potentially failing to reflect the challenges encountered in production-grade database assistant. In this work, we introduce BIRD-INTERACT, a benchmark that restores this missing realism through: (1) a **comprehensive interaction environment** that couples each database with a hierarchical knowledge base, metadata files, and a function-driven user simulator, enabling models to solicit clarifications, retrieve knowledge, and recover from execution errors without human supervision; (2) two **evaluation settings** reflecting real-world interaction settings which contain a pre-defined conversational protocol (c-Interact) and a more open-ended agentic setting (a-Interact) in which the model autonomously decides when to query the user simulator or explore the DB environment; (3) a **challenging task suite** that covers the full CRUD spectrum for both business-intelligence and operational use cases, guarded by executable test cases. Each task features ambiguous and follow-up sub-tasks, requiring LLMs to engage in dynamic interaction. The suite is organized into two sets: a full set (BIRD-INTERACT-FULL) of 600 tasks which unfold up to 11,796 dynamic interactions for a comprehensive overview of performance and a lite set (BIRD-INTERACT-LITE) of 300 tasks, with simplified databases for detailed behavioral analysis of interactions, and fast development of methods. Our empirical results highlight the difficulty of BIRD-INTERACT: the most recent flagship model GPT-5 completes only 8.67% of tasks in the c-Interact setting and 17.00% in the a-Interact setting on the full task suite. Further analysis via memory grafting and Interaction Test-time Scaling (ITS) validates the importance of effective interaction for achieving success in dynamic text-to-SQL tasks.

Published as a conference paper at ICLR 2026
BIRD-INTERACT: RE-IMAGINING TEXT-TO-SQL EVAL- UATION VIA LENS OF DYNAMIC INTERACTIONS
Nan Huo
Xiaohan Xu
Jinyang Li
Per Jacobsson'
Shipei Lin
Bowen Qin"
Binyuan Hui
Xiaolong Li
Ge Qu
Shuzheng Si
Linheng Han
Edward Alexander
Xintong Zhu?
Rui Qin
Ruihan Yu"
Yiyao Jin"
Feige Zhou"
Weihao Zhong"
Yun Chen
Hongyu Liu?
Chenhao Ma
Fatma Ozean
Yannis Papakonstantinon
Reynold Cheng
"The University of Hong Kong
8 Google Cloud
"The BIRD Team
bird.bench25egmall com
ABSTRACT
Large language models (LLMs) have demonstrated remarkable performance on single-turn text-to-SQL tasks, but real-world database applications predominantly require multi-turn interactions to handle ambiguous queries, execution errors, and evolving requirements. Existing multi-turn benchmarks fall short of capturing this complexity, either by treating conversation histories as static context or by limiting evaluation narrow, read-only DIEECCT-ONLY operations, thereby po- tentially failing to reflect the challenges encountered in production -gade database assistant. In this work, we introduce BIRD-INTERACTS benchmark that restores this missing realism through: (1) comprehensive interaction environment that couples each database with hierarchical knowledge base, metadata files, and a cniction-driven user simulator, enabling models to solicit clarifications, retrieve knowledge, and recover from execution errors without human supervision; (2) two evaluation settings reflecting real-world interaction settings which contain pre- defined conversational protocol (c-Interact) and a more open-ended agentic setting (a- Interact) in which the model autonomously decides when query the user simulator or explore the DB environment; (3) a challenging task suite that covers the full CRUD spectrum for both business-intelligene and operational use cases, guarded by executable test cases. Each task features ambiguous and follow-up sub-tasks. requiring LLMs engage dynamic interaction. The suite organized into two sets: a full set BIRD-INTERACT-FULAA of 600 tasks which unfold up 11,796 dynamic interactions for comprehensive overview of performance and lite set (BIDD-INTERAC--LTE) of 300 tasks, with simplified databases for detailed behavioral analysis of interactions, and fast development methods. Our empirical results highlight the difficulty of BIRD-INTERACH the most recent flagship model GPT-5 completes only 8.67% % of tasks in the c-Interact setting and 17.00% % in the a-Interact setting the full task suite. Further analysis via memory grafting and Interaction Test-time Scaling (ITS) validates the importance of effective interaction for achieving success in dynamic text-to-SQL. tasks.
INTRODUCTION
Data-driven Decision-making has become indispensable across modern enterprises, prompting surge of interest Natural Language Interfaces to Databases (NLIDB) that empower non-technical users to extract insights from relational databases using natural language (Shi et al., 2024). Motivated by this vision, a wave of methods (Pourreza et al., b025a;b; Pourreza & Rafiei, 2023; Liu et al., 2025; Qu al., 2024; Li et al., 2025b; Maamari et al., 2024; Sheng & Shuai, 2025; Li al., 2025a; Talaei
"Equal contribution.
1 "Correppnniigg authors Reynold Cheng and Chenhao Ma.

---

Published as a conference paper at ICLR 2026
User Task
Corfiuration
Iocerectics
User Agent Interaction Exceple
febrss 6. pautto ao rou af sestemntee sectiyy wleo coe
"aatd e foretion to cataabo rane al onjjentrt
T
uy: lad and conpeeatvv pobrtty soree"
(eeealy need orgent care."
"oour Si rocenaalee
feading 00 schoma, and column meanings.
LIE
Preabrok
Annottated Ambh and Real intentions
CCr ass wher meeite zcorig metre should ssed to derervise Surgere core""
"resdurpeeccomgg >
"aus" (Artjer
Interaction
Wilrrrbilly Scord"
wrgor cave detornatttion spowed by the
Simulator 24
Anjoct iaitenabiity Store
chocked by arnotated cachig
Searching "AUS" from but find defiitionn
urpee"
System S Starrations
Tuee. Con you clarty the formalc for the AYSF"
crsercment wwweoolllllyppyy
DB Environment E
con
'015 colculated by
Metadata M
Artivstiiingg "kuciirno
kosuach bolletig- oding masiro Anttentiinneel Dy
inrare SaRInOa IST, esSOin, wortery
wish are arooeforeeee ouar reptleene
Knowlodge Bose Dos
semoaiataa) 181 de
Insoruction
SOL dreddie annotatd
Iedlancc Fartee (93 ricoic oo oovee ('iilee Vollweliii feaw Wivl wlidky?
"Gond soturd of my quintoru"" [Termination
Figure I: Task overview of BIRD-INTERACT showing the evaluated system interacting with DB Environment and User Simulator to complete the user task with a sequence of sub-tasks.
et al., 2024; Caferoğlu & Ulusoy, 2024; Cao et al., 2024; Lee et al., 2025) based on large language models (LLMs) has recently achieved impressive text-to-SQL performance on popular single-turn benchmarks such as Spider (Yu et al., 2018) and BIRD (Li et al., 2023b).
However, real-world data interaction is rarely a single, perfectly-formed query (Li et al., 2025e; Dinan et al., 2019). It is an iterative, stateful dialogue characterized by ambiguity (Chen et al., 2025b) and evolving goals ( (Wu et al., 2025). The task in Figure exemplifies this complexity. To succeed, the text-to-SQL system must first engage the user to resolve the ambiguity of the term urgent care. Only with this clarified context can it generate the correct SQL. If its initial code fails an execution test, LLM must debug and revise its SQL solution based on the error feedback. After the user confirms the SQL is correct, they may proceed with follow-up question that depends on its intermediate results. Therefore, evaluation on true practical utility LLMs with these multi-faceted aspects requires a benchmark containing complete interactive problem-solving process, rather than isolated, single-turn SQL generation.
Although existing interactive text-to-SQL datasets (Yu et al., 2019b;a; Chen et al., 2025b; Guo et al., 2021; Dahl et al.) have been developed, they inadequately model this reality for two primary reasons. First, most multi-turn text-to-SQL benchmarks rely on static conversation transcripts (Yu et al., 2019a; Chen et al., 2025b; Yu et al., 2019b; Guo et al., 2021). They present models with a clean interaction history without recording the failed attempts, digressions, and clarifications that occur in practice. This design may introduce a fundamental limitation: every LLM is evaluated against the same predetermined dialogue trajectory, regardless of how it would have naturally guided the interaction. This setup fails to reward intelligent interaction strategies and cannot effectively penalize conversational mess up. Second, existing benchmarks suffer from narrow task scope, focusing on read-only (SELECT-only) queries typical of business intelligence (BI) reporting. This ignores a vast and critical range of database management (DM) operations, including data manipulation (INSERT, UPDATE, DELETE), schema modifications (ALTER TABLE), and transactional control, which are also common operations in the normal DBA cycle (Chen al., 2024).
To address these critical limitations, we introduce BIRD-INTERACT. a new benchmark designed to evaluate LLMs in dynamic text-to-SQL environment. Our work makes the following contributions: (1) A High-Fidelity Interactive Environment: We develop a comprehensive sandbox upon an open-souree project LIVESQL (BIRD-Team, 2025) for each task, including hierarchical knowledge base (HKB) with domain-specific facts, metadata files, an executable database environ- ment, and most critically, an interactive user simulator as recent research (Wu et al., 2025; Yao et al., 2025; Wang et al., 2024). This simulator can respond to clarification questions, provide feedback on proposed actions, and guide the model through complex tasks, enabling end-to-end evaluation without human intervention. However, recognizing that traditional simulators, those powered by advanced models like GPT TTT4. exhibit unfair behaviors such as ground-trum leakage, we propose a novel two-stage function-driver approach that maps model questions to constrained symbolic actions
2

---

Published as a conference paper at ICLR 2026
before generating controlled simulator responses. (2) Two Evaluation Settings: We propose two popular evaluation settings. c-Interact prrotooo-guudedd presents tasks with a clear conversational protocol, testing model's ability to follow structured conversation with the user. In contrast, a-Interact (agentic) provides only a high-level goal, requiring the model to autonomously plan a strategy, decide when to query the database, consult documentation, ask the user simulator for help. (3) A Comprehensive and Challenging Task Suite: BIRD-INTERACT expands the scope of evaluation to include the full spectrum of CRUD operations. Tasks are drawn from both analytical and operational domains and are accompanied by executable test cases that verify functional correct- ness. Each task features an ambiguous initial priority sub-task, dynamic clarification requirements, follow-up sub-tasks, and environmental uncertainties, which can only be resolved through dynamic interaction. The suite consists of two parts: a full set (BIRD-INTERACT-FUUII folding up to 11,796 dynamic interactions for a comprehensive evaluation of performance, and lite set (BIRD-ING INTERACCTELLLEEE analysis and faster deployment.
of 600 tasks, un-
of 300 tasks with cleaner databases, enabling finer-grained behavioral
Our experiments show that state-of-the-art models struggle with BIRD-INTERACT, with GPT-5 achieving only 8.67% % success in c-Interact and 17% in o-Interact. We identify distinct challenges across interaction modes: communication effectiveness often determines success c-Interact, while a-Interact suffers from bias toward costly trial-and-error over strategic resource exploration. We also observe Interaction Test-time Scaling (ITS), where performance improves monotonically with additional interaction opportunities across multiple models. These findings support our hypothesis that developing strategic interaction capabilities is key to improving LLM performance on complex database reasoning.
2 PROBLEM DEFINITION
Task Definition. We formalize interactive text-to-SQL as a multi-turn collaboration between text-to-SQL system Sn and user simulator U. operating over database environment & = (D.M,, MM,.J, KC}, where DD the executable database, .M contains schema metadata, and K. represents external knowledge (Lee et al., 2021; Dou et al., 2022; Li et al., 2023b). Given a sequence of related sub-tasks Q = {41, ....... }. the goal for S to generate SQL solutions (.i...., Jn through interactions. For each sub-task gi the interaction proceeds through interaction turn t = 1,2,... until completion:
u/ = -u E), of Sh
- + (u,,
where if represents the interaction history up turn and @ denotes concatenation in prompt. The user simulator Ur manages the interaction by presenting sub-tasks, answering clarification questions for ambiguous queries, and providing feedback on submitted SQL. Critically, subsequent sub-tasks are released only after successful completion of first sub-tasks.
Metrics. Each sub-task qi is annotated with ground-truth SQL of and executable test cases T that define correctness. A predicted solution o; correct if passes all associated test cases, ensuring functional equivalence with o;. In our implementation, each task consists of two related sub-tasks (n 2): an initial priority sub-task qı containing ambiguities requiring resolution, and (2) subsequent followuup sub-task (2. We evaluate system performance using: (1) Success Rate (SR): The proportion of sub-tasks completed successfully, with each sub-task scored 1. We report SR separately for sub-task and sub-task 2 as online evaluation during interaction. (2) Normalized Reward: Defined as normalized scoring according to priority weighting as designed in Appendix to [0, 1] for analyzing system behaviors after interaction (offline evaluation) (Yao al., 2022).
BENCHMARK CONSTRUCTION
This section details the methodology for the construction of BIRD-INTERACT benchmark. We begin by outlining the overall benchmark setup (Section 1), and then elaborate on how we convert clear single-turn tasks into ones requiring interactions (Section 3.2).
3

---

Published as a conference paper at ICLR 2026
3.1 SETUP AND RESOURCES
We build our benchmark on the text-to-SQL tasks and infrastructure of LIVESQLBENCH (BIRD- Team, 2025). We select this foundation due several key advantages. First, LIVESQLBENCH provides a comprehensive evaluation environment. It supports the full spectrum of SQL operations, including DMI, and DDL, which allows for dynamic database states that reflect real-world usage. Furthermore, its permissive license and ready-to-use artifacts, including an executable database sandbox and metadata files, facilitate extension and reproducibility, Third, features a Hicrarchical Knowledge Base (HKB) that organizes external knowledge as nodes in directed acyclic graph (DAG), as shown Figure I, where "AVS" depends on "IF" and "CP!". This structure explicitly models dependencies between facts that require multi hhop reasoning to connect isolated information. Despite these strengths, LIVESQ! BLENC is fundamentally a single-turn benchmark. This design fails to capture the interactive and often ambiguous nature of real-world data analysis scenarios. Our primary contribution is to convert this static benchmark into a dynamic, interactive setting.
3.2 INTERACTIVE TASK ANNOTATION
To maintain the integrity and quality of our benchmark, we recruit 12 expert annotators through rigorous multi-stage selection process detailed in Appendix C. We describe systematically the conversion from single-turn tasks of LIVES SEVESOLBENCH into multi-turn interactive scenarios through two key annotation strategies: ambiguity injection and follow-up sub-task generation:
Ambiguity Injection. Ambiguities in daily life require interactions seek clarification. To make annotation and evaluation controllable. we design methods to inject ambiguities into single-turn queries and the environment from LIVESQLBENCH, pairing each with a unique clarification.
(1) Superficial user query ambiguities: we target surface-level ambiguity in the user request. These include intent-level ambiguities. where the user language is vague (e.g., "elderly people"), and implementattinn-lvee ambiguities, where the user's 's intent is clear but the implementation details (e.g., decimal precision) are under-specific. (2) Knowledge ambiguities: we inject incom-
Knowledge Chain
urgent care
pleteness into the external knowledge. This category includes two subtypes: (i) one-shot knowledge ambiguity, where isolated knowledge entries are removed. (ii) knowledge chain breaking, where intermediate nodes in multi-hop knowledge chains are masked. For example, consider the chain "urgent care "AVS" " IF "I/CPI" in Figure 2. By masking the intermedi- ate node, i.e., the fact "AVS" in HKB, we deliberately break the inferential chain, rendering knowledge ambiguous and requiring user clarification to proceed. (3) Environmental ambiguities: LIVESQLBENCH databases already contain natural noise, such as NULL in critical fields, which further introduces uncertainty in how these cases should be handled.
masked
J207""
Figure 2: Knowledge chain break- ing ambiguity.
Each injected ambiguity is paired with a corresponding SQL snippet from the ground- -ound-truthha query as a clarification source, which guides our user simulator in generating consistent and contextually appropriate clarifications Quality control ensures that ambiguous queries are unsolvable without clarification yet fully reconstructable once clarifications are provided. Complete details are given in Appendix H.
Follow-Up Upp Sub-tasks Annotation. User intents frequently evolve throughout an interactive session (Taylor, 20) with users modifying, filtering conditions, or exploring related aspects of their queries. Therefore, we also extend each initial priority sub-task with one additional follow-up sub-task to resonate with this scenario.
These follow-up sub-tasks are designed carefully using a principled 5-category taxonomy detailed in Appendix H.5. A key contribution of our benchmark is the introduction of state dependency between sub-tasks, different with other datasets (Yu al., 2019a;b; Lee et al., 2021; Zhong et al., 2017; Li et al., 2025d). System models must reason over modified database states or the newly created objects (e.g. tables) from preceding queries write SQLs for follow-up sub-tasks.
4

---

Published as a conference paper at ICLR 2026
3.3 FUNCTION- DRIVEN USER SIMULATOR
Evaluating interactive text-to-QL systems requires user interactions, such as multi-turn requests and responses to clarification questions. Conducting such human-in-the-lopp evaluations scale is impractical. To make large-scale evaluations feasible, recent interactive benchmarks, such as MINT (Wang et al., 2024), employ LLMs to simulate human users (Li et al., 2025c; Yu al., 2019a;b). However, we observe that there are two major issues among these simulators: (1) they sometimes leak information from ground-truth SQL query, and (2) they may deviate from the original task requirements (Barres et al., 2025; Kazi et al., 2024).
Two-Stage Strategy. To ensure a more robust evaluation, we intro- duce two-stage funetion-driven user simulator, illustrated in Figure 3(c). the first stage, an LLM functions as a semantic parser. It maps the system's clarification request into one of three predefined allowed actions: AMB (), LOC () , or UNA (). AMB () is invoked for queries related to ambiguities that have been pre-annotated with the key SQL snippet. LOC () handles reasonable clarification requests
Table I: Data Statistics
STATISTIC
LITE
FULL
Total Tasks
300
600
# BI tasks
195
410
# DM tasks
105
190
Distinet Test Cases
191
that fall outside our pre-annotated ambiguities, such as questions about SQL formatting or specific sub sub-components. In these cases, the simulator uses an AST-based retrieval step locate the relevant SQL fragment (detailed in Appendix N). Finally, UNA () rejects any inappropriate requests, such as attempts elicit ground-truth answers. In the second stage, the user simulator generates final response based on the chosen action and the annotated GT SQL with clarification source. This two-stage approach ensures the simulator's 's behavior remains predictable and controllable, while still permitting diverse and context-awan interactions. Detailed prompts are provided in Appendix R.
Tokens User Query
40.22
32.95
# Tokens / /SL
361.52
252.21
Ambiguities Task
5.16
# sub-tasks/ Task
2
2
# Interactions / Task
13:04
13.64
Agreement
93.33
93.50
3.4
DATA STATISTICS
Table reports key properties of BIRD-INTERACT. The resulting benchmark comprises total of 900 interactive text-to-SQL tasks, each featuring an ambiguous initial priority sub-task, dynamic clarification requirements, follow-up sub-tasks, and environmental uncertainties, collectively spanning the full CRUD spectrum (Create, Read, Update, and Delete). In Appendix E, we also conduct comprehensive comparison against other relevant benchmarks, showing that BIRD- ACT among the most open, challenging, and long-horizon interactive benchmarks in text-to-SQL. scenarios.
4 EVALUATION SETTINGS
Two Evaluation Settings. The interactive framework of BIRD-INTERACT supports evaluation in two scenarios: LLMs as conversational assistants (c-Interact) (Dinan et al., 2019) and as agents (a-Interact) (Schluntz & Zhang, 2024).
Badget-Constrained Awareness Testing. The application of LLMs is limited by computational resources and user patience (Wen et al., 2025; Li et al., 2025c). We introduce ondget-constrained awareness mechanism to both evaluation settings, where interactions are capped by an adaptive budget and systems are informed of the remaining budget. This enables evaluation under varying budgets, including stress-testing (Ahmad et al., 2025; Zhang et al., 2025) in low-budget conditions to assess the system's is ability ask the right questions and plan effectively. The specific budget settings are detailed in the following sections.
4.1 C- INITEERCTT EVALUATION
Interaction Setup. The cInerercc evaluation establishes a multi-turn dialogue between user simu- lator U and system S. The session unfolds in two sequential phases of sub-tasks: First, E presents an underspecified sub-task 91 alongside database metadata M and knowledge base K. System S may engage in clarification dialogue before generating SQL 01- Upon successful validation against test cases Ti, 24 issues a contextually coherent follow-up sub-task 92. prompting S to respond with SQL d2 Each sub-task incorporates single debugging opportunity: following query failure, S may
5

---

Published as a conference paper at ICLR 2026
(a) Databases
(b) System Actions
(c) User Simulator
Environment
DB Metadata Files
DBs
x.5
Retritee Neoninn
Knowledge
Showe
dser
Stage Clarifications
cots mesninggjso. jsoni
jsanssnn
x1 Set ALl Knowiedges
>
BoUues SQ LDCT
Salnnd Qpansss
Dav
Knowledge Base
Arch
x1
Get ALL Columo Mranings
TchelagAns Ars
Lopprns sineee diss welt
Robots
08 Scheeo
Kabe Inttiol Request
Clarify Ashiguity
News
Ash Question
Molke Fal Low-lou Request
Subrit sQU
the Suteittee Sot
Interaction
Ambiguity Resolution
Sveeanho
Debugging
Environment
Request
Dudget Contralned
Action
reward-8.7
reward-B.5
Follow-Up Question Suetwsto,
Oeservation
roward--8.2
Database Env.
Debugging
Reward
c-Interact
Ø-Interact
Figure 3: Two evaluation settings for BIRD-INTERACH e-Interact, where the system engages in conversation with the user, and a-Interact, where the system interacts flexibly, At the end of the task, the system will receive a reward r E [0,1]. 1].
submit one revised query after receiving execution feedback from U. Each debugging attempt incurs a reward penalty to account for the additional computational cost, details can be found in Figure 3. The evaluation episode concludes when both sub-tasks successfully completed or all attempts exhausted. Notably, failure in the initial priority sub-task immediately terminates the entire session.
Budget Constraints. The budget is implemented constraint the number of clarification turns. The total allowed turns, Telar, are calculated as follows: Telar = mmmb + Apat-
Here, Mamb represents the minimum budget required to resolve the ambiguities, which is equal to the number annotated ambiguities the user task. The parameter A a tunable variable that simulates different levels of user patience, granting the evaluated system extra for clarification.
4.2 a-INTERACT EVALUATION
Interaction Setup. The a-Interact provides LLMs with autonomous planning and execution within a pre-defined action space, following REACT paradigm (Yao et al., 2023). We model the complete database environment User Simulator, allowing the agent to determine optimal invocation strategies dynamically. this work, we summarize and define 9 discrete actions common to text-to-SQL with details in Appendix J. BIRD-INTERACT also supports customized scaffolds, details can be found in Appendix J.2.
set of callable tools, containing the target database, metadata, HKB, and
Budget Constraints. To reflect the varying computational costs of different actions, we implement budget-constrained evaluation framework where each action consumes predetermined amount of budget, encouraging cost-effecctive action sequences. The total budget for each task B Brsse 2 mamb + 2 >ppa where Boase 6 is the base budget, Mamb is the number of annotated ambiguity points, and Apat is the user patience parameter, maintaining consistency with the c-Interact framework. This setting evaluates the agent's ability to achieve high performance under resource constraints while balancing thoroughness with efficiency. Further details of action costs are provided in Appendix
This setting can evaluate agent performance under realistic constraints that present practical database interaction scenarios, where users have limited patience and computational resources finite.

---

Published as a conference paper at ICLR 2026
Table Success Rate and Final Normalized Reward of different models on BIRD-INTERACT-FULL The success rate is cumulative; Reward* is the normalized reward. The values reported in c-Interact are after debugging phase, and (+n) means the performance gained via debugging. Avg. Cost the cost for one task average in USD. Our user simulator has avg. cost of 0.03 USD. BI = Business Intelligence User Queries, DM = Data Management User Queries.
Priority Question (Success Rate %))
Follow Ups (Success Rate %)
Avg. Cost 1
Model
Reward
B1
DM
Overall
B1
DM
Overall
e-Interact Text-to-QQ.
GPT-
9.49 1-0000
25.40
+2.12)
14.50
-4687)
5.84 (0.20)
14.81 (+05))
8.67
(40.30
12.58
0.08
Claude-Sonnet-311
10.71
(14,2)
33.86
(1.4)
18.00
(5.50)
4.62 (1044)
16.40 (3.7)
8.33
(1.35)
13.87
0.229
DeepsekkChaat
Chhat-V3.1
(0.77)
-1.50)
(1024)
(100))
Boon-3-Coder-80B
16.30
34.39
22.00
8.03
16.93
10.83
(1200
17.75
$
Claude-Sonnet-4
16.06 (487)
35.98
22.33
10:46
22.22 (3.20)
14.17
(1.00)
18.35
0.29
03-Mini
17.76 (22.00
37.57
(-11.30)
24.00
(45,50)
11.44 (4073)
25.40 (442))
15.83 (41.3)
20.27
0.007
Gemini-2.5-Pr
18:73 +4.389
38.62
(-10.05
25.00
(+6.77)
12.41 (+1.2))
24.87 (+5.20
16.33
(+250)
20.92
0.04
a-Interact Texi-to-SQI.
8.05
24.74
13.33
3.90
4.74
4.17
10.58
$.007
Deepseck-hat-VEl
10.49
31.58
17.17
4.63
4,83
13:47
Gemini-2.5-Pro
10.49
41.58
5.85
20.00
10.33
$.222
11.46
41.58
21.00
5.61
16.84
9.17
17:45
0.60
Blaude-Sonnet..
15.85
53.68
27.83
8.05
22.63
12.67
23.28
0.51
GPT-5
15.61
58.42
29.17
10.98
30,00
17.00
25.52
0.224
5 EXPERIMENT
We benchmark 7 recent and powerful LLMs (2 open-source, closed-source) as system models via fresh PoL Docker instance for more stable evaluation. We set the user patience by default and Intercttt base budget of 6. All models use temperature=O) and top_p=1, with default reasoning settings, conducting single runs due to cost (full details Appendix 1.2 and 1.3).
5.1 MAIN RESULTS
Table 2 summarizes the success rate (SR) and normalized reward (NR) obtained by 7 representative frontier L.LMs on BIRD-INTERACT-FUIT The full experimental results of BIRD-INTERACT-LLITT can be found in Table 10. We can observe:
BIRD-INTERACT Remains Challenging, Leaving Ample Room for Future Improvement. Even the strongest models in our study, Gemin n 1 5--5Prrr ro and GPT-5, capture only 20.92% and 25.52% of the available reward respectively, in the c-Interact and a-Interact mode. Absolute success rates reveal similar limitations: more than 16.33% of tasks are solved end-to-end in c-Interact and 17.00% in a- Interact, with most models falling in substantially lower rates.
Evolving User Intent is Challenge in Online Assessment. Follow-up sub-tasks are noticeably more challenging, likely because the longer, concatenated context in these turns remains a bottleneck for LLMs in interactive text-to-SQL. tasks.
Offline Reward v.s. Online SR Evaluation. Table 2 shows that offline normalized reward (NR) and online success rate (SR) generally correlate positively, though notable divergences occur due to the reward structure allocating 70% the primary sub-task and 30% follow-up sub-tasks. These complementary metrics capture different aspects of model performance. Success rate measures holistic task completion across multi-turn interactions, relevant when users prioritize successful outcomes regardless of path. Normalized reward assesses performance on users' critical initial objectives while crediting challenging follow-up sub-tasks. Together, they provide comprehensive evaluation of the distinct capabilities required for advanced interactive text-to-SQL. systems.
Business Intelligence versus Data Management. Business intelligence (BI) queries pose signifi- cantly greater challenges for LLMs compared to data management (DM) tasks since DM operations typically follow standardized, predictable patterns that L.LMs can effectively learn (Li al., 2025d),
7

---

Published as a conference paper at ICLR 2026
- Insrrativv Performnce
Idveflood Pe formance
30
90
Success Rate (%)
20
20
20
20
10
3
3
5
Claude-3.7-Sonme (-4)
03-Mini (-c)
GPT-4o (-4)
Qwen-3 (-c)
fermannce
40
40
40
30
30
30
30
10
10
10
Claude-3.7-Somee (-a)
03-Mini (:a)
GPT-4o (-m)
Qwen-3 (-a)
User Patience
Figure 4: The performance of different LLMs with different user patience on BIRD-INTERACT--LTEE The red line denotes a-tnttract mode (-a); the blue line denotes c-Interact mode (-c). And the dotted line (Idealized Performance) denotes the performance under ambiguity-fiee single-turn text-to-SQL.
whereas BI queries demand nuanced understanding of complex, domain-specific business logic and analytical reasoning that varies substantially across contexts.
Interaction Mode Emerged as the Decisive Factor for a Successful Outcome. Furthermore, we observe that different models demonstrate varying aptitudes for different interaction paradigms, with each model showing relative strengths in specific modes. For example, GPT-5 performs poorly in the constrained, predefined flow designed personally of the c-Interact mode by achieving only 14.50% SR (worst) but excels in the a-Interact setting with 29.17% SR (best), which affords more flexible and exploratory space. This evidence demonstrates the critical importance of matching interaction modes to model-specific capabilities, which we hypothesize stem from differences in training data distributions and architectural inductive biases (Liu et al., 2024; Gao et al., 2024b).
5.2 INTERACTION ANALYSIS
The Impact of Communication Task Success in c-Intrract. A notable finding is the underperformance of the flagship model, GPT- 5, on the c-Interact, despite its strong performance on many single-turn tasks (Phan et al., 2025; Glazer et al., 2024; Rein et al., 2024). There- fore, we hypothesize that this stems from deficiency in its interac- tive communication abilities rather than its core generation capability. To test this hypothesis, we conduct an experiment termed Memory Grafting. In this setup. provide GPT-5 with the ambiguity reso- lution histories from two other better models, Qwen 3oCoer and 03 03-mini, before asking it to generate the final SQL query. The re- sults, presented in Figure 5, show that GPT-5's performance improves significantly when leveraging the interaction history from either model. This finding indicates that while GPT-5 5 possesses robust SQL gener- ation capabilities. more effective communication schema is required to help it achieve satisfactory outcomes for user tasks. We also further analyze the patterns for effective communication in Appendix P.
With Homory Crattin W thou Mamory ManoryCraling Graling
28-
20.8%
18.8%
Success Rate
16-
Owean Ordee
Oben chader o92-g
Mode 097d
Figure 5: SR of GPT-5 with memory grafting.
Interaction Test-Time Scaling. To investigate the relationship between interaction frequency and model performance, we conduct an Interaction Test-Time Time Scaling (ITS) experiment in BIRD- INTERACT-LITE where results are shown in Figure 4. We simulate varying levels of user patience by allowing different numbers of interaction turns for both c-Inerrct and a- I-Interact. As a baseline, include single-turn task performance for each model, where all necessary context is provided to
8

---

Published as a conference paper at ICLR 2026
create unambiguous tasks. This single-turn condition represents idealized scenario that, while potentially requiring significant user effort to ensure complete information provided (Li et al., 2025d), eliminates the need for further clarification. As demonstrated in the figure. Claude-3 7 exhibits clear scaling behavior with respect to increasing interaction opportunities.
This pattern shows that the model steadily improve by transforming additional interaction chances into valuable information gains through efficient interaction.
ITS Law: A model satisfies this law if, given enough interactive turns, its performance match surpass that of the idealized single-turn task.
Action Distribution Patterns in a-Interact. We analyze action distributions across 7 system models and find concentration two primary actions: submit (direct code execution with error feedback) and ask (user clarification requests), which together comprise 60.87% of all actions. Despite being the most computationally expensive actions (Figure 3), models favor these over systematic exploration behaviors like knowledge and schema retrieval. This suggests LLMs prefer direct trial-and-errov execution over comprehensive environment exploration, likely due to pre- training biases. Future work should incentivize broader tool utilization for complex interactive tasks. Additional analysis on the FULL set appears in Appendix
USER SIMULATOR ANALYSIS
This section presents comprehensive evaluation our function-driven user simulator compared to conventional user simulators and their respective impacts on dynamic interactive text-to-SQI. benchmarks through both objective and subjective experiments.
Evaluation on USERSIM-GUARD. To provide an ob- jective and comprchensive observation of different user simulator mechanisms, we construct a static dataset called USER GUARD, comprising 2,100 questions with ref- erence actions labeled by human experts. Detailed in-
100%
EasySrand
90%
Ascurwey
80%
70%
formation regarding the distribution and annotation pro- cedures can be found in Appendix O. We employed an LLM-as-Judge (Zheng al., 2023) evaluation framework using -5tee33-33BB-222 pendent evaluators. Our analysis reveals significant reli- ability concerns with conventional user simulator designs. Specifically, as shown in Figure 6, when confronted with Unanswerable (UNA) questions, baseline user simulators consistently fail to implement safeguards, resulting in unfair or inappropriate feedback generation with failure rates reaching up to 67.4% depending the backbone model. In contrast, our proposed function-die driven approach demonstrates substantially improved reliability, reducing the failure rate to low 2.7%. This represents dramatic improvement in user simulator robustness and reliability compared to baseline approaches.
60%
50%
40%
as inde-
AMD
LOC
UNA
Figure 6: The accuracy of different user simulators on USERSIM-GUARD.
Alignment with Human User. We evaluate alignment between our user simulators and actual human behavior by having hu- man experts interact with 7 system models on 100 randomly sampled tasks across BI and DM domains. We then compute correlations (Ivey et al., 2024; Kong et al., 2024) between success rates (SR) achieved by human users versus our simulators across the same tasks. As shown in Table 3, function-driven simula- tors demonstrate significantly stronger alignment with human behavior: GPT-4o with function calling achieves 0.84 Pearson correlation (p= 0.02) compared to 0.61 without function calling (p= 0. 14), while Gemini-2.0 2.0-Fila shows similar improvements.
Table 3: Correlation analysis be- tween Al and human users.
Pearson (p (p-value)
GPT--0
ww Pure: (Ourz)
0.84 (p 0.02)
Gemini- 2.0-lt
Punce
0.79 (p= 0.00)
GPT--4
Baseline
0.61 (p= 0.14)
Gemini 2.0 Flash
Boseline
0.54 (p.21))

---

Published as a conference paper at ICLR 2026
These results confirm that incorporating our designed mechanism produces more realistic user simulators that better reflect actual human-Al interaction patterns (detailed analysis in Appendix O).
RELATED WORK
Text-to-SQL. Text-to-SQL has emerged an attractive interface relational databases because it frees users from learning intricate schema details and SQL syntax. The advent of large language models (LLMs) (OpenAI, 2025; Team et al., 2023; Team, 2024: Guo et al., 2025; et al., 2023a; Qu et al., 2025) with strong reasoning and cross-domain generalization has accelerated this progress. Few-shot systems such DIN-SQL. (Pourreza & Rafiei, 2023) and DAIL-SQL (Gao et al., 2024a) exploit in in-context learning to decouple the task into schema-linking and SQL-generation stages, while methods like CodeS (Li et al., 2024a) and DTS-SQL (Pourreza & Rafiei, 2024) improve smaller mod- els through carefully curated, high-quality training subsets. Concurrently, agent-based frameworks that interleave thought, action, and observation, which are exemplified by MAC-SQL. (Wang et al., 2025), demonstrate that iterative interaction with the environment can further raise SQL accuracy. Despite these advances, virtually all existing systems are evaluated only in single-turn settings; their effectiveness in conversational, multi-turn text-to-SQI. scenarios remains open question.
Multi-turn Text-to-SQL addresses the reality that user queries are often ambiguous underspecifted; without clarification the system may return incorrect or empty results. Benchmarks such as CLSQL& and LEARN-TO-CLL CE-N-TO--LARIT extend the Spider (Yu et al., 2018) dataset with dialogue turns to probe this challenge (Yu al., 2019a; Chen et al., 2025b; Li et al., 2024b). However, these resources presuppose a static, noise-free dialogue history shared by all models, ignoring that different systems might ask different follow-up questions (Yao et al., 2025; Barres et al., 2025). More recent evaluations of autonomous agents, for example, MINT, introduce dynamic interaction histories (Wang et al., 2024), yet they have not been adapted to the text-to-SQL setting. Constructing realistic user simulator for databases is non-trivial because it must respect complex schema constraints while keeping the answer space fair and controllable (Zhou et al., 2025; Barres al., 2025). In this work, we fill this gap by proposing interactive benchmark that is implemented with an optimized user simulator, new databases, and knowledge, and we analyze the behaviour of state-of-the-arr reasoning models rigorously to make contributions for realistic and uncertain text-to-SQL. systems.
Multi-turn Text-to-SQL.
FUTURE WORK
While INRD-INTERCCT establishes a comprehensive framework for evaluating interactive text- to-SQL systems, several directions remain for future investigation. First, we plan to develop post-trained, human-aligned local user simulator via post-training, aiming to capture more reliable response patterns while maintaining controllability and reducing API cost. Second, our current a-Interact setting imposes strict budget constraints that create a stress-mode evaluation environment, placing considerable pressure LLM agents to make optimal decisions under resource scarcity. To complement these findings, we will conduct experiments a free-mode setting without the budget- constrained awareness testing (Section 4). This could allow us observe natural interaction strategies when models are unconstrained, identify whether more sophisticated exploration patterns emerge, and characterize the relationship between interaction thoroughness and task success. Comparing stress-moot and free-mode performance will provide deeper insights into officiinncy-cffectiveess trade-off's in interactive text-to-SQL systems.
CONCLUSION
We present BIRD- INTERCT,,, benchmark for evaluating interactive text-to-SQL. systems through dynamic, multi-turn interactions that better reflect woaldworld usage scenarios. Our benchmark features a function-driven user simulator, dual evaluation settings for conversational and autonomous planning modes, and totally 900 challenging tasks designed to test LLM abilities to handle ambiguities and maintain state across turns. Comprehensive evaluation demonstrates a critical gap between existing SQL generation capabilities and the strategic interaction skills required for effective human-Al collaboration in database querying.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
Reynold Cheng, Nan Huo, Xiaohan Xu, Jinyang Li, Xiaolong Li, and Ge Qu were supported by the Research Grant Council of Hong Kong (RGC Project HKU 17202325), the University of Hong Kong (Project 2409 1991009)) and the HKU Faculty Exchange Award 2024 (Faculty of Engineering). And we would like to express our sincere gratitude to Irina Saparina, Mohammadreza Pourreza, Mehdi Bouzouina, Hailong Li, Jiatong Shi, and Professor Shinji Watanabe for their fruitful discussions and valuable insights that helped improve this work.
ETHICS STATEMENT
This research complies with the ICLR Code of Ethics. We have carefully reviewed the guidelines and ensured that our work aligns with the stated ethical standards, including considerations of data privacy, fairness, and responsible use of released datasets and code. Justification: This work does not involve crowdsourcing or research with human subjects. All annotation and task creation were carried out by the authors themselves.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility our work. First, for the evaluated systems, all experimental settings, including model parameters, temperature (= 0), and budget configurations, are clearly documented in Appendix I, ensuring that our evaluation can be replicated under the same conditions. Each task is executed in freshly re-initialized PostgreSQL 14 instance (Docker). The Docker image contains the database engine and benchmark code, and is restarted for every guarantee clean and consistent state. This setup makes experiments deterministic, isolated across runs, and easy to reproduce by rebuilding the environment from scratch. Second, for the user simulator, we validate its robustness Section 6, and detail in Section how we designed and annotated the benchmark to guarantee reliability, uniqueness of clarifications, and consistency of responses from the user simulator. This safeguards reproducibility across different runs and systems. Third, for the benchmark suite, we will publicly release all components under permissive license, including databases, tasks, hierarchical knowledge bases, documentation, interaction logs, and the source code for both evaluation settings and the user simulator. This full release ensures transparency and faithful replication of our experiments. We also provide the prompts used across the whole experiment, which can be found in Appendix R, from Figure 19 Figure 26. Due to the dynamic nature of our interaction evaluation, we will also open-source our interaction trajectories upon publication for better reproducibility.
Experiment Configuration. All experiments CIRD-INTERACE conducted via API, except for the LLM-as-Judge evaluations of different user simulators, which are run on 4 NVIDIA A100 80G GPUs. The estimated cost for each model under BIRD- INTERACT-FRR is shown in Table and the estimated cost for STRD-INTERACT-LLTT is shown Table 10.
REFERENCES
Lama Ahmad, Sandhini Agarwal, Michael Lampe, and Pamela Mishkin. Openai's approach to external red teaming for ai models and systems. arXiv preprint arXiv:2503. 16431, 2025.
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. tau2-bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506. 07982, 2025.
Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. Benchmarking and improving text-to-SQL generation under ambiguity. In Proceedings of the 2023 Conference Empirical Methods Natural Language Processing, Singapore, December 2023.
BIRD-Team Livesqlbench: A dynamic and contamination-fres benchmark for evaluating llms on real-world text-to-sql tasks. https:/ggttub..com//t http:///.....bbbbbece//sssllllln 2025. Accessed: 2025-05-22.
11

---

Published as a conference paper at ICLR 2026
Hasan Alp Caferoğlu and Özgür Ulusoy. E-sql: Direct schema linking via question enrichment in text-to-sql. arXiv preprint arXiv:2409.16551 2024.
Zhenbiao Cao, Yuanlei Zheng, Zhihao Fan, Xiaojin Zhang, Wei Chen, and Xiang Bai. Rsl-sql: Robust schema linking in text-to-sql generation. arXiv preprint arXiv:2411.00073 2024.
Chongyan Chen, Yu- Yun Tseng, Zhuoheng Li, Anush Venkatesh, and Danna Gurari. Accounting for focus ambiguity in visual questions. arXiv preprint arXiv:2501.0223 022201, 2025a.
Maximillian Chen, Ruoxi Sun, Tomas Pfister, and Sercan O Arik. Learning to clarify: Multi- turn conversations with action-based contrastive self-training. In The Thirteenth International Conference on Learning Representations, 2025b.
Xi Chen, Jinguo You, Kun Li, and Xiang Li. Beyond read-only: Crafting a comprehensive Chinese text-to-SQL dataset for database manipulation and query. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 2024.
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Humicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. Expanding the scope of the ATIS task: The ATIS-3 corpus. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
Bryan I. M. de Oliveira, Luana G. B. Martins, Bruno Brandão, and Luckeciano C. Melo. Infoquest: Evaluating multi-turn dialogue agents for open-ended conversations with hidden context. arXiv preprint arXiv:2502.12255 2025.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-poweed conversational agents. In International Conference on Learning Representations, 2019.
Zhongjun Ding, Yin Lin, and Tianjing Zeng. Ambisql: Interactive ambiguity detection and resolution for text-to-sql. arXiv preprint arXiv:2508. /5276, 2025.
Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung- Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, al. Practiq: A practical conversational text-to-sql dataset with ambiguous and unanswerable queries. In Proceedings of the 2025 Con- ference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies, 2025.
Longxu Dou, Yan Gao, Xuqi Liu, Mingyang Pan, Dingzirui Wang, Wanxiang Che, Dechen Zhan, Min- Yen Kan, and Jian- Guang Lou. Towards snowledge-intensiee text-to-SQL semantic parsing with formulaic knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, December 2022.
Avrilia Floratou, Fotis Psallidas, Fuheng Zhao, Shaleen Deep, Gunther Hagleither, Wangda Tan, Joyce Cahoon, Rana Alotaibi, Jordan Henkel, Abhik Singla, Alex Van Grootel, Brandon Chow, Kai Deng, Katherine Lin, Marcos Campos, K. Venkatesh Emani, Vivek Pandit, Victor Shnayder, Wenjing Wang, and Carlo Curino. NL2SQL is solved problem... not! In Conference on Innovative Data Systems Research, 2024.
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. Proc. VLDB Endow., January 2024a.
Jie Gao, Simret Araya Gebreegziabher, Kenny Tsu Wei Choo, Toby Jia-Jun Li, Simon Tangi Perrault, and Thomas W Malone. A taxonomy for human-llm interaction modes: An initial exploration. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA '24, New York, NY, USA, 2024b.
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caro- line Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla, and Mark Wildon. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint, arXiv:2411 1..02722, 2024.
12

---

Published as a conference paper at ICLR 2026
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on ilm-s--ajudgee arXiv preprint arXiv:2411.15594 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Jiaqi Guo, Ziliang Si, Yu Wang, Qian Liu, Ming Fan, Jian-Guang Lou, Zijiang Yang, and Ting Liu. Chase: A large-scale and pragmatic Chinese dataset for cross-database context-dependent text-to-SQL. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume
1: Long Papers), August 2021.
Moshe Hazoom, Vibhor Malik, and Ben Bogin. Text-to-SQL in the wild: A naturally dataset based on stack exchange data. In Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), August 2021.
Zezhou Huang, Pavan Kalyan Damalapati, and Eugene Wu. Data ambiguity strikes back: How documentation improves gpt's text-to-sql. arXiv preprint arXiv:2310. 18742, 2023.
Nan Huo, Reynold Cheng. Ben Kao, Wentao Ning, Nur Al Hasan Haldar, Xiaodong Li, Jinyang Li, Mohammad Matin Najafi, Tian Li, and Ge Qu. Zeroea: a zero eero-training entity alignment framework via pre-trained language model. Proceedings of the VLDB Endowment, 2024.
Nan Huo, Jinyang Li, Bowen Qin, Ge Qu, Xiaolong Li, Xiaodong Li, Chenhao Ma, and Reynold Cheng. Micro-act: Mitigate knowledge conflict in question answering via actionable self-reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), Vienna, Austria, July 2025.
Jonathan Ivey, Shivani Kumar, Jiayu Liu, Hua Shen, Sushrita Rakshit, Rohan Raju, Haotian Zhang, Aparna Ananthasubramanian, Junghwan Kim, Bowen Yi, et al. Real or robotic? assessing whether iims accurately simulate qualities of human responses in dialogue. arXiv preprint 200.002409.083330, 2024.
Taaha Kazi, Ruiliang Lyu, Sizhe Zhou, Dilck Hakkani-Tun and Gokhan Tur. Large language models as user-agents for evaluating wwwoorrneeedddillguuee systems. 2024 IEEE Spoken Language Technology Workshop (SLT), 2024.
Chuyi Kong, Yaxin Fan, Xiang Wan, Feng Jiang, and Benyou Wang. Platolm: Teaching Ilms in multi-round dialogue via user simulator. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
Maya Larbi, Amal Akli, Mike Papadakis, Rihab Bouyousfi, Maxime Cordy, Federica Sarro, and Yves Le Traon. When prompts go wrong: Evaluating code model robustness to ambiguous, contradictory, and incomplete task descriptions. arXiv preprint arXiv:2507.20433 2025.
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. Kaggledbqa: Realistic evaluation of text-to-sql parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
Dongjun Lee, Choongwon Park, Jaehyuk Kim, and Heesoo Park. MCS-SQL: Leveraging multiple prompts and multiple-choice selection for text-to-SQL generation. In Proceedings of the 31 International Conference on Computational Linguistics, Abu Dhabi, UAE January 2025.
Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, SU Hongjin, ZHAOQING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, et al. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. In The Thirteenth International Conference on Learning Representations, 2025.
Boyan Li, Jiayi Zhang, Ju Fan. Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. Alpha-SQL: Zero-shot text-to-SQL. using monte carlo tree search. In Forty-second International Conference on Machine Learning, 2025a.
13

---

Published as a conference paper at ICLR 2026
Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. Codes: Towards building open-source language models for text-to-sql. Proceedings of the ACM on Management of Data (PACMMOD), 2024a.
Haoyang Li. Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. Omnisql: Synthesizing high-quality text-to-sql data scale. Proc. VLDB Endow., September 2025b.
Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. In Proceedings of the AAAI conference on artificial intelligence, 2023a.
Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM already serve as a database interface? a Blg bench for large-scale database grounded text-to-SLLs. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b.
Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Yurong Wu, Chenhao Ma, Jian- Guang Lou, and Reynold Cheng. Tapilot-crossing: Benchmarking and evolving Ilms towards interactive data analysis agents. arXiv preprint 2XXi::2403005007 2024b.
Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Bowen Qin, Yurong Wu, Xiaodong Li, Chenhao Ma, Jian-Guang Lou, and Reynold Cheng. Are large language models ready for multi-turn tabular data analysis? In Forty-second International Conference on Machine Learning, 2025c.
Jinyang Li, Xiaolong Li, Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, Ziwei Tang, Yuanshuai Li, Florensia Widjaja, Xintong Zhu, Feige Zhou, Yongfeng Huang, Yannis Papakonstantinou, Fatma Ozcan, Chenhao Ma. and Reynold Cheng. SWE-SQL: Illuminating LLM pathways to solve user SQL issues in real-world applications. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025d.
Junyan Li, Wenshuo Zhao, Yang Zhang, and Chuang Gan. Steering I1m thinking with budget guidance. arXiv preprint arXiv:2506. 13752, 2025e.
Zongxi Li, Yang Li, Haoran Xie, and S. Joe Qin. CondAmbigQA: A benchmark and dataset for conditional ambiguous question answering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, November 2025f.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang. Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations, 2024.
Yifu Liu, Yin Zhu, Yingqi Gao, Zhiling Luo, Xiaoxia Li, Xiaorong Shi, Yuntao Hong. Jinyang Gao, framework for text-to-sql. arXiv
Yu Li, Bolin Ding, et al. Xiyan-sql: A novel preprint Xiiv:2507.04701, 2025.
Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi. The death of schema linking? text-to-sql in the age of well-reasoned language models. arXiv preprint arXiv:2408.07703 2024.
Sewon Min Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
OpenAl. Openai o3 and 04-mini system card, 2025. Accessed: 2025-05-15.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's s last exam. arXiv preprint arXiv:2501. 14249, 2025.
14

---

Published as a conference paper at ICLR 2026
Mohammadreza Pourreza and Davood Rafiei. DIN-SQL: Decomposed in-context learning of text-to- SQL with self-correction. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Mohammadreza Pourreza and Davood Rafiei. DTS-SQL:
Decomposed text-to-SQL. with small large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 2024.
Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaci, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozean, and Sercan O Arik. CHASE-SQL: Multi-path reason- ing and preference optimized candidate selection in text-to-SQL. In The Thirteenth International Conference on Learning Representations, 2025a.
Mohammadreza Pourreza, Shayan Talaei, Ruoxi Sun, Xingchen Wan, Hailong Li, Azalia Mirhoseini, Amin Saberi, Sercan Arik, et al. Reasoning-say Reinforcement learning with sql tailored partial rewards for
enhanced text-to-sql. arXiv preprint arXiv:2503.23135 2025b.
Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. Before generation, align it! a novel and effective strategy for mitigating hallucinations in text-to-SQL generation. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, August 2024.
Ge Qu, Jinyang Li, Bowen Qin, Xiaolong Li, Nan Huo, Chenhao Ma, and Reynold Cheng. SHARE: An SLM-based hierarchical action CorREction assistant for text-to-SQL. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, July 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Irina Saparina and Mirella Lapata. AMBROSIA: A benchmark for parsing ambiguous questions into database queries. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Erik Schluntz and Barry Zhang. Building effective agents, December 2024. Engineering at Anthropic.
Lei Sheng and Xu Shuai Shuai. SLM-SQL: An exploration of small language models for text-to-SQL. In Proceedings of the 14th International Joint Conference Natural Language Processing and the 4th Conference of the Asia -sii-Pacifii Chapter of the Association for Computational Linguistics, Mumbai, India, December 2025.
Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and 7hi Yang. A survey on employing large language models for text-to-sql tasks. ACM Computing Surveys, 2024.
Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. Chess: Contextual hamnessing for efficient sql synthesis. arXiv preprint PXXv:2405.16655. 2024.
Robert S Taylor. Question-negotiation and information seeking in libraries. College & Research Libraries, 76(3):251-267, 2015.
DeepSeek-AI Team. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Rat-sql: Relation-awant schema encoding and linking for text-to-sql parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang. Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian- Wen Zhang, Di Yin. Xing Sun, and Zhoujun Li. Mac A multi agent collaborative framework for text-to-sql. In Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025), 2025.
15

---

Published as a conference paper at ICLR 2026
Xingyao Wang, Zihan Wang. Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating lims in multi-tur interaction with tools and language feedback. In The Twelfth International Conference Learning Representations, 2024.
Hao Wen, Xinrui Wu, Yi Sun, Feifei Zhang, Liye Chen, Jie Wang, Yunxin Liu, Ya-Qin Zhang, and Yuanchun Li. Budgetthinker: Empowering budget-aware Ilm reasoning with control tokens. arXiv preprint arXiv:2508. /7196, 2025.
Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. Collabllm: From passive responders to active collaborators. arXiv preprint arXiv:2502.00644 2025.
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402. 13111111616 2024a.
Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian- Ban-Guang Lou, and Shuai Ma. Re-reading improves reasoning in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, November 2024b.
John Yang, Akshara Prabhakar, Karthik R Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimban. Webshop: Towards scalable
real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Representations, 2023.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. (SitauS) ) -bench: benehmark for Lunderline[T]p Peoooolluneelinn ( A (A) geent-uunderliee (U)) ser interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural language to code generation in interactive data science notebooks. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: LLong Papers), Toronto, Canada, July 2023.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao. Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross doos-d semantic parsing and text-to-SQI. task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October-November 2018.
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang. Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li. Youxuan Jiang, Michihiro Yasunaga. Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. CoSQL: A conversational text-to-SQL challenge towards cross-domane natural language interfaces to databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, November 2019a.
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong. Richard Socher, and Dragomir Radev. SParC: semantic parsing in context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019b.
16

---

Published as a conference paper at ICLR 2026
Jifan Zhang, Henry Sleight, Andi Peng, John Schulman, and Esin Durmus. Stress-testing model specs reveals character differences among language models. arXiv preprint arXiv:5510.07686, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv: 2017.
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn Ilm agents collaborative reasoning tasks, arXiv preprint arXiv 2025.
17

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
A Limitations
20
The Use of LLM Statement
20
Annotation Group Details
20
C.1 Annotator Entrance Test
20
C.2 Training Tutorials
20
C,3 Qualification Test
21
Benchmark Design Principles
21
Comparison with Related Benchmarks
22
E.1 Task Comparison
22
E,2 Database Comparison
23
F Evaluation Metrics
23
F.1 Success Rate (SR)..
23
F2 Normalized Reward
23
G Test Scripts
24
G.1 BI Queries
24
G.2 DM Queries
24
Ambiguity and Follow-up Annotation Details
25
H.1 User Query Ambiguity Annotation
25
................................
H.2 Knowledge and Environmental Ambiguity Annotation
25
H.3
Ambiguity Chain
H.4 User Query Ambiguity Taxonomy
26
H.5 Follow-up Sub-Task Taxonomy
I Experiment Details
29
L1 Choice of PostgreSQL as the Evaluation Database System
................................
29
1.2 Model Alias
29
1.3 Experiment Setup
30
J Action Space and Selection Patterns in a-Interact
30
J.1 Action Space in a-Interact
30
J.2 Universal Cost Scheme for Custom Agents
33
J.3 Action Selection Patterns and Their Impact (Full Set)
33
18

---

Published as a conference paper at ICLR 2026
K Performance Different Ambiguity Types
34
Experiments on BIRD-INTERACT-LITTE
35
M Error Analysis
35
N User Simulator Design Details
36
Evaluating the Function-Driver User Simulator
37
0.1 UserSim-Guard: A Benchmark for Simulator Robustness
..t.Ii..........................
37
0.2 Experimental Setup
37
0.3 Results and Analysis
38
P Pathways to Effective Communication
38
Human Evaluation of Dataset Quality
40
Prompts
R.1 System Prompts
41
R.2 User Simulator Prompts
41
19

---

Published as a conference paper at ICLR 2026
Preparation: Code, Env, Data
Data Annotation Instruction
In th's sutoril, will komn how annotato the eRD intrect Tisk.
Doeker Instal
Outlinea:
@ Dopior lustal
duptinuctiio single
commpree
Arbigufiie
(Tavonorr, Clarfiiction Annotation)
Expected cata format and Roumpee
Ambigulty Taconomy
Download the creating szriot
You also replace them Coratriction using the the mawarr dumso and lio.
Imporeetation
Dubld docker
Sronledge Arroiguity
enviommenn they oest compost
previous dinectory, directly docket software
Proofroad
consainer evaluction container)
Su rullow Up subtlskk
Expested saca format Eoomple
Basterr cenpose -ddul(
Tracnomy
[Later We give you new DBg boter When you are glven new BBs
Solution SOL and Text Cones Annotation
you need
Data Checking Sufte
containngg droctory, change nome (elk
Data Cross Checking Procsdure
Figure 7: Examples of training materials by screenshots for BIRD-Interact annotators.
A LIMITATIONS
Our work has centered on the text-to-SQL domain, but we believe our proposed interaction evaluation not inherently limited Instead, can cover generalizable human-AI collaboration. Exploring the adaptation of this framework to other generative domains, such Python code synthesis or API call generation, is a promising direction for future research. But at this time, we think it's representative scenario since it also features long-context, hierarchical knowledge, and Al coding problem.
B THE USE OF LLM STATEMENT
Large Language Models (LLMs) were used only for light post-editing of the paper (i.e., reducing syntax errors and performing minor grammar checks). LLMs were not involved in any part of the research discussions, analyses, or idea generation. All insights, contributions, and intellectual content are entirely the authors' own.
C ANNOTATION GROUP DETAILS
To ensure the high quality of annotations for the multi-stage process for annotator selection, training, and qualification. This process aimed to ensure that all annotators possessed strong SQL expertise and followed consistent, reproducible workflow.
benchmark, we designed a rigorous,
C.I ANNOTATOR ENTRANCE TEST
All potential annotators were required to complete structured training program before contributing to the benchmark. We began by recruiting a pool of 33 candidates, including students, engineers, and text-to-SQU. researchers with prior database experience. Each candidate underwent a week- long training period consisting of tutorials and guided exercises (detailed below), followed by qualification exam. This exam tested proficiency annotation of interactive tasks. Only candidates who achieved passing score of at least 90% were admitted as official annotators, resulting in a final team 12 highly qualified contributors.
SQL. generation, schema understanding, and
C.2 TRAINING TUTORIALS
Candidates participated in intensive tutorial program covering essential aspects of interactive text-to-SQI, including:
Database environment setup
20

---

Published as a conference paper at ICLR 2026
Database schema analysis and comprehension
Reproduction of single-tum text-to-SQL. examples from LIVESQLBENCH
Ambiguity taxonomy, injection procedures, and clarification annotation
Follow-up sub-task taxonomy and construction, with solution SQL. and test scripts
Solution validation and evaluation script development
The tutorials contain the DB sandbox, code suite, detailed procedures, examples, and hands-on exercises that mirror the interactive feature of real-world SQL tasks. Some parts of the tutorials are shown in Figure 7. Annotators were introduced to the full annotation workflow required for the creation of the BIRD-INTERACT benchmark.
C.3 QUALIFICATION TEST
Following the tutorial phase, candidates were required complete a qualification assignment con- sisting of 20 representative interactive text-to-SQL tasks. For each task, candidates were asked to:
1. Reproduce the environment and baseline single-turn text-to-SQL task.
2. Inject ambiguity into the task and annotate the corresponding unique clarification, ensuring that with clarification the original clear task could be recovered.
3. Create follow-up sub-task and annotate with solution SQL and test scripts.
4. Validate that the solution SQLs passed all annotated test scripts sequence across sub-tasks.
5. Document their approach and provide validation log.
Only candidates who successfully completed the assignment with satisfactory quality were approved as annotators. This stringent qualification process ensured that all annotators met the high standards required for building a robust and trustworthy benchmark. The overall success rate was approximately 90%, demonstrating the effectiveness of the tutorial materials and training program in preparing candidates for interactive text-to-SQI. annotation. All annotators contributing the final release of BIRD-INTERACE passed this qualification process.
D BENCHMARK DESIGN PRINCIPLES
Our design philosophy for BIRD-INTERACT is guided by two core principles: incorporating realistic interaction challenges and ensuring robust, reproducible evaluation.
Realistic Interaction Challenges. To mirror the complexity of real-world data analysis, tablish scenarios where interaction is indispensable for task completion. This is achieved through two mechanisms. (1) Ambiguity: We deliberately inject different types of ambiguity- -ppanning user queries, knowledge bases, and database environments- that tasks cannot be solved cor- rectly without clarification. Resolving these ambiguities often requires multi-turn exchanges, forcing systems to decide when to query the user, consult the HKB, or explore the database. This design captures the iterative, source-dependent nature of ambiguity resolution. (2) Contextual Follow-ups: Every task includes a subsequent, related query that requires the system to reason over the preceding conversation, the interaction history, and, critically, potentially changed database state.
Reliable and Reproducible Evaluation. We ensure the reliability and reproducibility evaluation from two key aspects. (1) Reference-basee disambiguation: avoid cases where certain ambiguities lack explicit annotations, the simulator additionally provided with the reference SQL, allowing it to generate accurate clarifications when necessary. While in real-world scenarios, real users may only have vague initial goals without an answer when making a request, this pragmatic design choice enhances evaluation reliability. (2) Simulator robustness and reproducibility: employ a two-stage function- driven design to safeguard against adversarial manipulation and ground-trute leakage.
21

---

Published as a conference paper at ICLR 2026
Table 4: Data statistics of features in BIRD-INTERACT compared the evaluation set of related benchmarks. # Avg Turns: Number of User-System interactions by unfolding the model's interaction trajectory. # /ooss/Output: Average number of tokens in the reference output; "/" indicates bench- marks without reference output. Dynamic User: Whether the benchmark supports real-time user interaction (vs, static offline datasets). Dynamic Env State: Whether the database or environment state can be modified during interaction. Amb. Sources: Sources of ambiguity in user queries or environments. LLM + Guard means LLM as user simulator with Guard mechanism to make actions more controllable. [+] Results taken from publicly available Spider 2.0 Lite Gold SQL. All statisties are computed on the test set; if unavailable, we use the dev set instead.
Dutose
Take. Dynaric User
Dynarii Foy
Anh Starces
Fat, Knowlelge
KarglateggA (Lar at, 2001)
212
26.28
X
X
Waediqg Gouou 2017)
15,578
15.39
X
X
Speder (lu 2008
2,147
30.18
X
X
Spatee n-00-- aiartaa 2009)
511
413.77
X
X
Spriee 2. D01 at 20259
766
X
HIBD- stg- (La 210009
1.554
3001
X
X
BUKD Crite tta et 20224)
1.341
100.00
X
X
BUXD-Mrn Der dh 2023g)
1.500
65.55
X
X
AMBROSIA (Sepa lis & Layan, 3009
1.277
15.35
X
User
AIHHOT (Btader 2003)
3.000
31.73
X
User
Antiguult Handliu
Whes Procegu Ca Whong at. 20719
300
5531
X
Deserpiie
Intvjaea dl Eiheer at. 33219
1.000
1.55
1IM
Lur + Perom
CoecArrtegjA (1: 2000g
200
41.01
Qarry Boo.
V
vig Ng-hhcnsnmnggrryy (Chee al, 200000
5,500
1.5
X
Yoeal
X
SperC (Yu et 2019
422
2.85
24.55
Oflic:
X
Conversatioa
Cx al.
292
1.44
36.34
Ofline
X
CHKSE (G 2001)
755
1.30
43.11
Office
X
Beach (TThee (7heagot ot (001)
(003)
30
3.00
37.58
Office
X
X
MIN' (Wang 2028)
1.02
LLM
Ieterantive Beackmork
IntrOwde (Yang stal, 2021)
2
r-baad (Yao stil., 2023)
185
Winshop (Yis en 2007)
900
Our Beachnnork
Binn &
BueD INTrR Lim
600
300
1.83
1,05
252.31
365
LLM Guard
LLM Guard
User
Usr Bay
Table 5: Comparison of released databases across benchmarks.
Deuffin fe-l
Benchmark
DBs
CoL/DB
KB
License
(Li at. 20230)
54.2
2013)
27.1
WikisQl (Thong
5230
KigckDBQA (Lee et al. 2021)
23.4
Free
SEDE fllaroom 2021)
212
Apache Laconse
Spoder (Le al. 2025)
Restricted
May 1DC
BIND-NNTRATT-L.I
BY-SA
Fee
Figure 8: Distribution of advanced SQL features in INTERATT
DIRD INTERACT PULL
91.4
CCBY-N
E COMPARISON WITH RELATED BENCHMARKS
E. TASK COMPARISON
Table 4 compares
with existing text-to-SQL and interactive benchmarks across multiple dimensions. We categorize related work into four groups: SQL Generation, Ambiguity Handling, Static Conversation, and Interactive Benchmarks. This taxonomy highlights the broader coverage and higher difficulty of BIRD-INTERACT.
First, unlike most SQL generation benchmarks that evaluate single-tum queries or pre-collect static conversation history, BIRD-INTERACT integrates ambiguity handling, dynamic multi-turn interac- tions, and dynamic environments in a unified framework. Our tasks require systems not only to generate SQL but also actively engage in clarification and reasoning with both user and environ- ment. Second, the # Avg Turns of BIRD- INTERATTT is around 7.5 per task, significantly higher than most prior benchmarks, which typically unfold into one or a few turns. Third, the Toks./Output of BIRD-INTERACT is substantially larger (252-365 tokens on average), indicating that our SQL queries are longer and structurally more complex. Fourth, unlike static conversational benchmarks with offline conversation transcripts, BIRD-INTERACT features a Dynamic User during evaluation. Our two-stage function-drive driven user simulator ensures robustness by mapping clarification requests into symbolic actions before generating responses. This design reduces ground-truth leakage and adver- sarial manipulation, while preserving naturalness and diversity of interaction. Fifth, BIRD-INTERACT introduces multiple ambiguity sources. Whereas most prior datasets only consider ambiguity the
22

---

Published as a conference paper at ICLR 2026
query level, we additionally inject knowledge and environmental ambiguities. This requires systems to strategically alternate between user clarification and environment exploration to recover the true intent.
Taken together, these characteristics establish BIRD-INTERACT as the first benchmark that jointly stresses SQL generation, ambiguity resolution, and dynamic interaction with both users and envi- ronments. Compared to existing work, it sets higher bar for evaluating interactive text-to-SQL. systems.
E.2 DATABASE COMPARISON
Table 5 compares the databases used in BIRD-INTERACE with those of other widely used text-to-SQI. benchmarks. Compared to most prior benchmarks, our databases span diverse domains and contain more columns per database, resulting in more complex and richer schemas. All databases are paired with knowledge base documents. In terms of licensing, BIRD-INTERACT builds on the spen-source LIVESQLBER BENCH (BIRD-T Team, 2025) datasets released under CC-Y-SA 4.0, ensuring unrestricted academic and industrial use. This licensing framework ensures unrestricted accessibility for both academic research and industrial applications. Spider 2.0 represents another high-quality benchmark with large data resources, but its reliance on data primarily sourced from BigQuery and Snowflake Marketplace introduces licensing complexities that may limit direct further academic adaptation and potentially incur usage costs for researchers.
F EVALUATION METRICS
F.1 SUCCESS RATE (SR)
The Success Rate (SR %) is our primary online evaluation metric, measuring whether each sub-task is solved correctly during interaction. Let N denote the total number of tasks, where each task i in BIRD-INTERACY consists exactly two sub-tasks, denoted q;,1 and q122 Each sub-task qı, is annotated with ground-truth SQL solution of and set of executable test cases Tı.- A predicted SQL oi.j is considered correct if it passes all test cases in Tı.j. The success rate for the j-th sub-task across all tasks is defined
N Σ (Tss(os)) = True], SR, N
where I[-] the indicator function that equals if the prediction correct and otherwise. In reporting, provide SR separately for the two sub-tasks: (1) g. the ambiguous priority sub-task, and (2) f.2,, the follow-up sub-task. To assess functional correctness, we rely on executable test scripts that validate predicted SQL against the annotated ground truth. Details of the test scripts provided in Appendix G.
F.2 NORMALIZED REWARD
To capture the relative importance of different sub-tasks (e.g.. success on the initial ambiguous sub- task is critical for continuing the interaction) and to distinguish system behaviors such as frst-attempt success versus post-debugging success, we propose a Normalized Reward metric. It is calculated by the average reward across all tasks. This metric is reported in addition to the sub-task-level success rates described in Section
Formally, with N total tasks, the normalized reward is caleulated
R E 100
Σ. Ejen1 Tigj 100,
N
where the T., rij is the reward of the task and the sub-task task i. In the c-Intrract setting, to distinguish first-attempt and post-debugging solutions, the reward is defined by:
23

---

Published as a conference paper at ICLR 2026
0.7 1st sub-task is solved without debugging
T1,1 =
0.5 if 1st sub-task is solved with debugging otherwise
0.3 if 2nd sub-task solved without debugging
T12
0.2 2nd sub-task solved with debugging
0
otherwise
In the a- a-Interact setting, since the interaction flow is not fixed, e.g. the debugging times, the reward only considers the pass or fail of each sub-task:
1.0
if both sub-tasks are passed
ri =
0.7
if only the 1st sub-task is passed
0
otherwise
G TEST SCRIPTS
We check sub-task correctness using executable test scripts. For BI sub-tasks (analytical queries), we use default seft exact-match (EM) script that normalizes benign SQL differences (e.g., removing comments, redundant DISTINCT, rounding) and compares execution results between the predicted SQL and the annotated solution SQL under task-specific conditions. For DM sub-tasks (data manipulation or state-changing operations), scripts that assert task-specific postconditions of the database.
manually annotated, case-by-case verification
G.1
BI QUERIES
The default test script cleans predictions/solution (e.g., remove comments, DISTINCT, ROUND wrappers) and then compares execution results between the predicted SQL the annotated solution SQL via a configurable comparator ex_base with conditi tinitiioss map (e.g., order: false to ignore row ordering if the task does not require ordering):
def test fest__case___e cest__case_defaule default (pred_sqls, sqls, scl_sqls, db__dd____ee_ name, cenn, conditions=None))
***Default test_case: pytest-style assertion. """
pred_sqls remove_comments (pred_sqls)
sol_sqls remove_ccomeet (scl_sqis)
prea_aqis = remove_distingg (pred_sqis)
preé_aqls remove_round (pree___ss)
sol_sqls remove_distinct (scl (anove_rounddsol__1e0
sol_sqls
result ex base (pred_ss__q, sqls, sol sql_sqls, db name, conn, conditions) assert result == 1, base returned (result) but expected 1."
return result
G.2 DM QUERIES
DM sub-tasks may involve DML/DDL, stored procedures, functions and do not always return a result set. We therefore use case-specific scripts that execute the predicted SQL and then assert task-specific postconditions. Depending the sub-task, the test script may (i) check the return value of verification query (e.g., calling a created function/view) (ii) inspect the presence/shape/conton of created artifacts (tables, indexes, constraints), or (iii) compare targeted state properties (e.g., row counts, key invariants). For example, this one test case for the user sub-task Figure 1:
def test_case (pred_sqls, sol_sqls, db name, conn) execute_queries (pred_sqls, db_nane, conn)
fy_sq1
"SELECT FROM rank_urgent_care((
24

---

Published as a conference paper at ICLR 2026
prec querr result execute queries (verify sql, db_nare, conn] actual pred_
expected =
(101, 'Ancient Scroll', Decimal ((7.200)
(102, 'Bronze Vase',
Decimal (1 6.65)) 1),
(103, 'Stone Tablet',
Decimal (16.00 50/)0/ )
assert (entactual) == len (expected)
assert actual expected return True
H AMBIGUITY AND FOLLOW-UP ANNOTATION DETAILS
H.1 USER QUERY AMBIGUITY ANNOTATION
core step in constructing interactive scenarios is the deliberate introduction of ambiguity into originally unambiguous single-turn user queries. Our annotation process ensures that systems cannot succeed without active clarification,, thereby reflecting the uncertainties inherent in real-world human- database interactions (Saparina & Lapata, 2024; Dong et al., 2025; Min et al., 2020; Bhaskar et al., 2023). Figure shows the distribution of annotated ambiguities across the dataset.
Two basic ambiguity categories. We distinguish between two fundamental categories that guide annotation:
Intent-level ambiguity arises directly from user language, where the request is vague, un- derspecified, or missing critical details (e.g., "find elderly people" without defining the age threshold). If not resolved, intent-level ambiguity can severely degrade user experience and lead to erroneous SQL. Clarifying such ambiguities the primary requirement for an LLM to faithfully capture user intent.
Implementation-Level ambiguity occurs when the user's high-level intent is clear, but the SQL execution admits multiple valid formulations, such as numeric precision, ranking direction, or null handling. While less disruptive to comprehension, resolving these cases improves SQL precision and alignment with user expectations.
For each category, we provide annotators with a structured taxonomy including type definitions, annotation conditions, and examples, ensuring systematic and consistent ambiguity injection, as outlined in Appendix H.4.
Ambiguity and clarification sources. Each injected ambiguity is paired with unique clarification represented by a key (qUppet from the ground-truth SQL rather than natural language text. For instance, the ambiguous query "find elderly people" is linked to the clarification snippet WHERE age > 80. This design guarantees reproducibility: the user simulator can reliably ground clarifications in SQL semanties, while still generating diverse natural-language paraphrases during interaction.
Quality control. To maintain benchmark reliability, annotators follow strict checklist: (1) Necessity of clarification: each ambiguous query must be unsolvable without clarification, ensuring genuine reliance on interaction. (2) Completeness after clarification: once clarification is provided, the information must suffice for an expert reconstruct the exact solution SQL. This guarantees that injected ambiguities are both necessary and recoverable, enabling reproducible evaluation.
H.2 KNOWLEDGE AND ENVIRONMENTAL AMBIGUITY ANNOTATION
In addition to user query modifications, we also introduce ambiguities that arise from missing or noisy external resources. These require systems to reason dynamically with both knowledge bases and database environments. We annotate them in two categories: knowledge ambiguities and environmental ambiguities (Saparina Lapata, 2024; Dong et al., 2025; Min et al., 2020; Huo al., 2025; Bhaskar et al., 2023).
25

---

Published as a conference paper at ICLR 2026
Knowledge Ambiguities. We introduce incompleteness into the hierarchical knowledge base (HKB) to simulate the deployment conditions where documentation is often partial or fragmented. We distinguish two subtypes:
One-shot knowledge ambiguity: individual knowledge entries are masked without involving dependent chains. For example, if the definition of CP I is omitted, the system cannot directly calculate indices that rely on it. These isolated gaps require the system explicitly ask the user for missing facts.
Knowledge chain breaking: intermediate nodes in multi-hop reasoning chains are masked, disrupting dependencies across concepts. Consider the chain "urgent care" + "AVS" + " "IF/ / CP I" shown in Figure 2. By masking the intermediate node AVS, the inferential link is broken: the query becomes ambiguous, and the system must first request clarification from the user before proceeding the knowledge IF/CPI.
Database Inconsistencies. LIVESQLBENCE databases already contain noise, including string fields mixing numeric values with units, inconsistent column naming across related tables, and NULL values in critical fields. Moreover, their SQL tasks already involve this database noise, providing foundation for data quality challenges. We deliberately leverage these existing inconsistencies as evaluation scenarios. When constructing subsequent sub-tasks, also intentionally involve these noisy columns to increase the complexity of multi-turn interactions. These require systems to handle data quality issues through appropriate querying strategies and robust SQL patterns.
As in user query ambiguities, each ambiguity is also paired with a ground-truth SQL fragment that acts the clarification source.
H.3 AMBIGUITY CHAIN
We combine those individual ambiguities with different types into ambiguity chains that require Multi-Hop Ambiguity Resolution, which integrates three aspects:
1. Nested ambiguities. Clarifications themselves may require further explanation, requiring multi-stage resolution. Not all ambiguities visible the surface level of the query; some unfold only when earlier uncertainties are addressed.
2. Multiple clarification sources. Each ambiguity may require information from different sources. In particular, the system must decide whether seek clarification from the user or to consult the environment (e.g., knowledge base, schema, or documentation).
3. Clarification flows. We define three canonical transition types that characterize how clarifi- cation flows across sources:
User + User: an initial user clarification still requires a further follow-up inquiry to
the user.
User - Environment: the user's clarification points auxiliary information that must be retrieved from the environment, e.g. KB.
Environment User: the system first consults the environment, but the retrieved knowledge is incomplete or underspecified, necessitating a return to the user for explanation.
These transitions can compose into multi-hop clarification sequences such as User Enviroment
+ User. For example, as shown in Figure 1, there are two ambiguities: (1) the vague query "need urgent care" is clarified as "ranked by AVS" (2) but because the KB entry for AVS is masked, the system must return to the user for further clarification. To implement such cases, (1) annotated clarification snippets are intentionally underspecified. and (2) some KB nodes in HKB masked to simulate missing documentation. Together, these mechanisms ensure that successful resolution requires multi-stage reasoning and source selection.
H.4 USER QUERY AMBIGUITY TAXONOMY
We distinguish between two fundamental categories of user query ambiguity that guide annotation:
26

---

Published as a conference paper at ICLR 2026
intent-lovel knowledge ambiguities
implemeenaation-iveell ambiguities
22.6
230
eg.. Jinding gualy
ammiguuityy
snwwedge.mmb gutu
1A-15
LamLambiuutt
mient amb gu
join ambiguiky
scterre innkng_embigun
die net jemhiguity
divide zoro porouamolguity
other
Figure 9; Ambiguity types distribution.
Intent-Level Ambiguity Types. Intent-level ambiguity arises directly from user language, where the request vague, underspecified, or missing critical details (e.g., "find elderly people" without defining the age threshold). If not resolved, intent-level ambiguity can severely degrade user expe- rience and lead erroneous SQL (Saparina Lapata, 2024; Li al., 2025f). We summarize six types of user query ambiguity in Table 6, according related works (Saparina & Lapata, 2024; Li et al., 2025f; Dong et al., 2025; Wang et al., 2020; Min et al., 2020; Bhaskar et al., 2023; Huo et al., 2024; Floratou et al., 2024; Huang et al., 2023; Ding et al., 2025; Xu et al., 2024b;a) and guide the annotators to inject them into unambiguous user queries: (1) Lexical Ambiguity from tokens with multiple meanings, (2) Syntactic Ambiguity from multiple valid grammatical structures, (3) Semantic Ambiguity from vague phrasing (e.g., "recent") ) (4) Schema Linking Ambiguity from unclear schema references. (5) Query Intent Ambiguity where
goals (e.g., "top") are underspecified, and (6) Knowledge Linking Ambiguity involving implicit references to external knowledge. The performance of different types shown in Figure 14.
Table 6: Intent-Level User Query Ambiguity Taxonomy in BIRD-INTERACT
Ambiguity Type
Definition
Example
Lexical Ambiguity
token has multiple meanings senses within the query context.
"Show bills" "Bills" could voices, legislation, or billing records.
Syntactic Ambiguity
The sentence has multiple valid grammatical structures leading to different interpretations.
"Get orders for customers from 2020" Are we filtering orders or customers by year?
Semantic Ambiguity
The query is grammatically correct but mantically vague, lacking details necessary for precise interpretation.
"Recent Iransactions" The time frame for "recent" unspecified.
Schema Ambiguity
Linking
Ambiguity in mapping query term to the correct schema element due to multiple plau- sible candidates.
"List users by status" "Status" could refer to account_statio login_stat etc.
Query Intent Ambi- guity
Uncertainty about the user's intended opera- tion ranking criterion.
"Show the top customers" "TTop may refer to revenue, numher of orders, fre- quency.
Knowledge Linking
referenced concept exists the external knowledge hase, the query's 's link to the knowledge is implicit or unclear.
"Get Impacr Score" "Impact Score" refers to "Artist Impact Score" in the KB.
Ambiguity
Implemennatiion-Leel Ambiguity Types. Implementation-Ievee ambiguity occurs when the user's s high-level intent is clear, but the SQL execution admits multiple valid formulations, such as numeric precision, ranking direction, or null handling. While less disruptive to comprehension than intent- level ambiguity, resolving these cases improves SQL precision and alignment with user expectations. annotated conditionally, i.e., only when the corresponding SQL operations
These ambiguities present in the ground-truth -ruth SQL. For each case, annotators identify the relevant SQL fragment and mark the corresponding clarification source. We summarize the following types:
27

---

Published as a conference paper at ICLR 2026
Decimal ambiguity. Annotated when the solution SQL applies rounding numeric format- ting. Example: ambiguous query "show average score." " clarified query "show average score in two decimals," with the solution SQL using ROUND (AVG (score), , 2)
Join ambiguity. Annotated when the solution SQL requires non non-default join semantics (e.g.. LEFT JOIN, FULL OUTER JO).). Example: ambiguous query "list all customers and their orders," clarified query "list all customers and their orders, even if they have records," with the solution SQL. using LEFT JOIN.
Distinct ambiguity. Annotated when the SQL solution contains the DISTINCT keyword. Example: ambiguous query "get all product names," clarified query "get all different product names," with solution SQL SELECT DISTINCT product_name
Sort ambiguity. Annotated when the SQL solution applies an ORDER BY clause without LIMIT. Example: ambiguous query "show recent purchases," clarified query "show recent purchases sorted by time," with solution SQL including ORDER BY purchase_timm time DESC.
Null ambiguity. Annotated when the SQL. solution contains null-handling opera- tions (e.g.. COALESCE, ISNULL). Example: ambiguous query "count users by re- gion," " clarified query "count users by region, treating null as 0," with solution SQL COUNT (COALESCE (region, ))).
Rank ambiguity. Annotated when ranking functions are applied in the solution SQL (e.g., ROW_ WOW_NNMBER, DENSE_ DENSE_RANK). Example: ambiguous query "show top customers with ranks of revenue," clarified query "show top customers with ranks of revenue; if tied, assign the same rank," with SQL using DENSE_RANE ().
Divide-by zeroo ambiguity. Annotated when the SQL solution explicitly handles the dividing by zero. Example: ambiguous query "show the ratio of passed to total exams," " clarified query "show the ratio of passed to total exams, treating cases with total as 0," with solution SQL using CASE WHEN total=0 THEN O ELSE passed/total END.
These annotations ensure that implementatiion-leeel ambiguities reproducible and systematically linked to concrete SQL constructs. By marking such cases only when relevant SQL operations are present, preserve annotation consistency while enriching the benchmark with the challenges of SQL details in implementation.
Table 7: Implementation-Leew User Query Ambiguity Types
Ambiguity Type
Annotation Condition
Example Transformation
Decimal Ambiguity
ROUND function used in solution SQL.
"Show average score in 2 decimal"
"Show average score"
Join Ambiguity
Non-default join (e.g., LEFT JOIN) is used in solution SQL
"Show all customers and their orders even though they don't I have records" "Show all customers and their orders"
Distinct Ambiguity
DISTINCT keyword is used in solution
"Get all different product names"
SQL
"Get all product names"
Sort Ambiguity
ORDER BY is used without I.IMIT in
"Show recent purchases sorted by time"
solution SQL
+ "Show recent purchases"
Null Ambiguity
Solution contains null handling op- erations (e.g., COALESCE, ISNULL)
"Count users by region. treat null as O" "Count users by region"
Rank Ambiguity
Solution SQL uses ranking func- tions (c.g., ROW_NUMBER, RANK,
"Show top customers with ranks of rev- enue. the tied, give them the same rank number." "Show top customers with ranks of revenue."
DENSE RANK)
Divide-by-zero Am- biguity
Solution SQL must handle division by zero explicitly
"Show the ratio of passed to total exams, treating cases with zero total as 0" "Show the ratio of passed to total ex- ams"
28

---

Published as a conference paper at ICLR 2026
H.5 FOLLOW-UP SUB- -USK TAXONOMY
In addition to initial ambiguities, interactive scenarios require systems to handle diverse follow-up requests that extend refine the analytical chain. We categorize follow-ups into six types (Table 8) according to related works (Yu et al., 2019b;a; Yin et al., 2023), covering constraint adjustments, topic pivots, attribute modifications, result-driven drill-downs, aggregation-basee summarizations, and state-dependent follow-ups based on newly created objects. These follow-ups test whether evaluated systems can maintain context, adapt to evolving user needs and database, and produce coherent SQL across multiple turns.
Table 8: Follow-up Bub-Task Taxonomy in BIRD-INTERACT
Follow-up Type
Description
First Query Example
Follow-up Example
Constraint Change
Tighten or relax filtering conditions.
"List employees hired 2024."
"Only engineers."
"Include
2023 well.
Topic Pivot
Compare or switch entity values to explore
"Saley of Producc A in 2023"
"What aboud Product B?"
alternatiees.
Attribute Change
Modify the requested attributes, metrics, columns.
"Departments with >50 staff"
"Give their average salory."
Result-hased
Drill down, regroup, nest, or reformat hased the previous result
"List projects finished in 2023.
"For Apolio, show its hudget."
Aggregation
Request statistics, concatenations, cnants, Boolean checks (e.g., AVG, STRING_AS AGG, FILTER,
"Show the top-19 10 artists
"Give their joined
by track count."
into single comma-separated string."
LGGILIMI.. EXISTS). Final output is typically scalar, single row, or compact table.
State-Dependent
First query creates modifies dalabase object (e.g., table, view), therehy changing the database state, follow-up query operates on
that table, list only engi- employees with salary neers." above 100k."
sable
I EXPERIMENT DETAILS
I.1 CHOICE OF POSTGRESQL. AS THE EVALUATION DATABASE SYSTEM
BIRD- INDDINTERACT adopts PostgreSQL the underlying database management system for evaluation. This choice is motivated by several key considerations:
Enterprise Adoption and Feature Richness. PostgreSQL is among the most widely deployed open- source database systems in production environments, supporting advanced SQL features essential for complex analyties including window functions, CTEs, recursive queries, JSON processing, and user-defined functions. This enables evaluation on realistic, production- queries rather than basic patterns.
Accessibility and Reproducibility. As an open-source system, PostgreSQL eliminates licensing costs and access barriers. Unlike proprietary cloud platforms (e.g.. BigQuery, Snowflake) that may incur usage fees, PostgreSQL ensures any researcher can replicate our evaluation environment without financial constraints, enhancing long-term benchmark sustainability.
Standards Compliance and Transferability. PostgreSQL maintains strong SQL standards adher- ence (SQL:2016) while providing well-documented extensions. This ensures evaluation results remain broadly applicable across database systems and that skills learned generalize beyond vendor-specific implementations.
In summary, PostgreSQL's combination of real-world relevance, feature completeness, and unre- stricted accessibility makes it optimal for evaluating interactive text-to-SQL. systems under production- like conditions.
I.2
MODEL ALIAS
The following aliases used for the models in this work:
29

---

Published as a conference paper at ICLR 2026
Gemini-2.0-Flash: gemini-2-0-flash--00
PeepSeek-Rl:
GPT-4o: 2tttooooptt-00022--111-00
DeepSeck-V3: dheessekcctt
O3-Mini: 00-
Qwen-3-Coder-480B: orw.:coo---oo......eeeee-ee-----
DeepSeek-Chat-V31. 13
Gemini-2.5-Pro- gemini-2-5-pro
Claude-Sonnet-4:
GPT-5: gpt-5
1.3 EXPERIMENT SETUP
All experiments were conducted under deterministic decoding ensure reproducibility. Specifically, we set t temperat ure=0 and top_ 1_ for models. Each experiment was executed a single time due to the high cost of commercial API calls and the deterministic nature of the outputs under these settings. For both e-Interact and a-Interact, the default user patience budget was set to 3, in addition to the required turns for ambiguity resolution, which equals the number of annotated ambiguities. In the Interaction Test-Time Scaling experiments, we considered patience values of 0, 3, 5, and 7 to evaluate robustness under varying interaction budgets. For I-Iteeract, the base budget systems sufficient capacity to explore the environment and execute SQL queries before submitting. All model inferences were obtained directly from their official APIs or released checkpoints ensure authenticity and consistency. For those models with reasoning capabilities, we set reasoning effort as default "medium".
6 allow
J ACTION SPACE AND SELECTION PATTERNS IN a-INTERACT
Table Action space for the agent showing available actions, their environments, arguments, return values (as observation), and associated costs.
Action
Eny.
Arguments
Return Value
Cost
execute
DB
sql
Query Result
get_schema
DB
Database Schema
get_all__colummmmenn
DB
All Columns' Meanings
get_ccolumm_mae meaning
DB
table, column
Column Mcaning
get_all_extere
wwwpalalooowwwweeeeeeeee names
Knowledge Names
qet_ htw__toowledge_deefiiiieeee
knowl edge
Knowledge Definition
aet__al__ee htw.llloioolleeeeeededlriittoo.e
DB
All Knowledge Definitions
ask
User quest for
User Clarification
submit
User
sql
User Feedback
J.1 ACTION SPACE IN U-INTERACT
Table 9 lists the nine actions agent may invoke during the a-Interact evaluation. They naturally cluster into two families:
Environment-only probes (cost < 1). Seven low-cost calls let the agent inspect the database and hierarchical knowledge base (HKB) without engaging the user:
execute: run a candidate SQL statement and receive the result set;
get_ scc_sch cet__all_coiumee
get_colume wtw.loooommmmaannnnageeeee expose
structural and semantic metadata;
30

---

Published as a conference paper at ICLR 2026
Lluude-37-Sonmm (P1: 33.8%, P2: 17.4%)
03-mini (PI: 25.4%, P2: 11.7%)
DeepSeek-V3 (P1: 23.4%, P2: 9.7%)
GPT-4o (P1: 23.4%, P2: 9.0%)
9%
15%
10%
30%
64%
32%
Owen3 (P1: 21. 17,,% P2: 13.7%)
Gemini 2.0 flash (P1: 21.1%, P2: 10.4%)
DeepSock R1 (P1: 20.7%, P2: 12.4%)
execule
8%
get_Ni jet_al_colum_________________e__ wr___cum_mmeainnggg meanings
285
39%
5%
get extemal knwwledge. namess
get, all knoweloge difinitions
meaning
get, www.:oolleggeeddeeeeeeee
10%
3%
get schema
aweid_action
Figure 10: System action distribution of systems under default setting (patience=3) on LITE set. P1 and P2 indicate the success rate for the first sub-task and the second sub-task.
GPT:S (P1: 29.2%, P2: 17.0%)
Claude-Sonnet-1 (P1: 27.8%, P2; 12.7%)
Claude-3, 7-Sonnet (P1: 10.0% P2: 9.2%)
Gemini-2.5-P (P1: 20.3%, P2: 10.3%)
11%
14%
41%
30%
35%
17%
10%
03-Mini (P1: 19.8%, P2: 8.5%)
Peepseek-Chet-V3... (P1: 17.2%, P2: 4.8%)
Qwen-3-Coden Soarn---Code--4BB0 (P1: 13.3%, P2: 4.2%)
ask
10%
execule
get_ jet_aal_coumn___o meenings
55%
41%
16's
40%
13%
get all extemal knowledge names
columi, meaning
get www.:ooolleeeeeeeeeeeee
get schema
invaid action
Figure 11: System action distribution of systems under default setting (patience=3) on FULL set.. Pl and P2 indicate the success rate for the first sub-task and the second sub-task.
31

---

Published as a conference paper at ICLR 2026
Action Distribution Heatmap
Models vs Action Types Average per Sample
Dagg 20.061
1.14
0.62
0.24
1.72
9.05
4.11
Sluure Sorrec
lang, 24.091
3.22
5.26
0.38
0.06
0.86
0.81
2.72
1.04
0.01
2.40
Ang Actiono por Sammll:
Models
Gemini2
23.971
0.01
0.64
0.00
0.24
0.74
4.10
C2 Miri Duyg
0.00
Deepowek
lavg: 26:77)
3.73
6.45
0.54
0.66
0.37
6
2.48
1.04
0.01
1.72
Owen Coder 4905
colorve mesaminge
antomal Inpelette. nainter
pet al Fromtrige ariiaaions
calove seeeniing
ped intratedies desivationn
get artenno
Irsald artive
Action Types
Figure 12: System action distribution systems under default setting (patience=3) in heatmap on FULL set.
Action Groups Over Models
CP13
Clazze-Srrnd-4
Cl Stmmal
200
200
200
Gamili
Deapseek-CratV1
460
600
400
420
400
400
312
200
10
15
25
20
15
25
33
12
15
520
Socuuee
300
Koow edge
Invll
300
7pm
Figure 13: The interaction pattern of systems: action groups over under default setting (pa- tience=3) on FULL set.
32

---

Published as a conference paper at ICLR 2026
all oet_al__extemmo_ _aae_eeeee____eessee__________ get_ wtt.kkowolllge.... definition _ http://ddyeeeeeeiitiioooo retrieve business concepts from the HKB.
get _
Graduated costs (0.5-1) reflect the different levels of environmental resources consumed by these actions. Actions with smaller input and shorter output (get_ umn umn__eant ng, etc.) assigned a lower cost (0.5), while broader retrievals that return substantially longer responses aal_aal_ooluu_mmaaaaa
get_ schema, etc.) cost 1.0.
User-mediated interactions (cost 2 2). When autonomous reasoning is insufficient, the agent can
ask (cost 2): pose a clarifying question to the user simulator;
submit (cost 3): submit full SQL candidate to the user. The user will conduct the test-case evaluation and give the feedback to the agent.
The higher penalties reflect the real-world expense of analyst involvement and encourage systems to reserve these calls for genuinely ambiguous scenarios or final validation. Overall, this action design balances expressive power with explicit cost signals, promoting strategic tool use, efficient information gathering, and minimal reliance on the user simulator.
J.2 UNIVERSAL COST SCHEME FOR CUSTOM AGENTS
We encourage users of our benchmark to develop their agents with customized action spaces under the budget-constrained awareness testing evaluation. However, the action costs defined in our default setup may not directly apply these actions. To ensure fair and reproducible evaluation across agents with potentially different action spaces, we propose a unified two-tier cost scheme within the a-Interact framework, which all customized agents expected follow when assigning costs to their actions.
(1) Fixed-cost actions: If custom agent involves the actions of asking user, submitting SQL. or executing SQL, it should assign them the same costs as defined in our setup. User-side actions (ask=2, submi t=3) are assigned globally fixed costs reflect the intrinsic expense of human involvement, while execute is assigned a fixed cost of 1.0 regardless of result size, since all agents interact with the same database interface and execution engine, ensuring consistent computational overhead and I/O behavior across implementations.
(2) Token-aware actions: for custom environment actions that differ from those our default setup (e.g., new action get_al lis_al_ttal L__taallennameee costs are determined dynamically based on the number of input and output tokens generated when calling the action, reflecting the relative amount of euvironmental resources consumed. According rule applicable to all agents: an environment action call incurs input tokens < 250 and output tokens 1000, its cost should be set to 0.5; otherwise, it should be assigned cost of 1.0. This universal policy ensures fairness for agents using different action spaces.
empirical statistics, define token-aware
J.3 ACTION SELECTION PATTERNS AND THEIR IMPACT (FULL SET)
Figure and Figure show how seven systems distribute their calls across the nine available actions (Table 9) on the FULL set. We summarize three observations:
Balanced strategies outperform extremes. The strongest performers. GPT-5 (29.2%) and Claude-Sonnet-4 (27.8%), adopt relatively balanced strategies. GPT-5 splits its budget almost evenly between environment probes (47%) and user involvement (ask+ submit: t:: 52%). Claude-Sonnet- Sonnet-4 follows a similar pattern, but with heavier emphasis on execute e (29.9%) and lighter use of submit (20.0%). By contrast, O3-Mini expends an extreme 91 % of its budget on user calls (36% ask, 55% submit) t and allocates only 4% to execute, passing fewer than one-fifth of the first sub- tasks. On the other side, Qwen-3-Car -- (48% execute) e) and DeepSeek- Chat (41% execute) are strongly execution-heavy and likewise underperform (P1 13.3% and 17.2%). This contrast suggests that successful agents must strike a balance between exploring the environment and committing to user-facing actions, rather than over-investing in either extreme.
33

---

Published as a conference paper at ICLR 2026
cinteraction
30
25
Success Rate
10
5
0
intent
knowledge
knowledge
linking linking semantic
schema
Figure 14: Success Rate LLMs on different ambiguity types over e and a-Interact Modes.
Submitting selectively helps, brute execution hurts. Across systems, the proportion of submis calls correlates positively with P1 (Pearson r=0.41, Spearman p=00.54,, while the proportion of execute calls correlates negatively (Pearson r 0.52, Spearman pee 0.54). In practice, this means that repeatedly probing the database with tentative execute calls without consolidation tends waste budget, whereas converging on a grounded hypothesis and committing to submi improves success rates by getting the feedback from the user. For example, Claude-3.7-Sonne and DeepSeek- -hat each keep s submit usage below 17% and 11% instead relying heavily on execute e. At the other extreme, O3-Mini's indiscriminate strategy of submitting more than half of all turns also underperforms, confirming that it is not the absolute amount of submission that matters ignoring the information from the user and environment.
3. Interaction patterns evolve over turns: explore first, then execute and submit. As shown in Figure 13, stronger systems (e.g., GPT-5, Claude-Sonnet-4) follow clear turn-by-turn strategy: in early turns they combine environment exploration with user clarifications gather information, while in mid and later turns, they increase execute e and s submit calls to test and refine SQL. In contrast, weaker systems either submit too early (O3-Mini) overuse execution without consolidation (Qwen- 3-Coder), Icading to poorer performance. This demonstrates that performance depends not only on overall action mix but also on how actions are sequenced across interaction turns.
Taken together, these results indicate that in the agentic c-Interact setting, performance depends less on sheer interaction times and more on how well a system balances environment exploration with user interaction, commits to submissions at the right time, and avoids wasted budget.
K PERFORMANCE ON DIFFERENT AMBIGUITY TYPES
Which knowledge missing type lead to more ambiguity? Linear or High-order? Figure 15 compares tasks where (1) the missing fact lies a simple, "linear" chain of the hierarchy with (2) those where the gap occurs within the chain-what we term a higher-order ambiguity. Linear cases correspond to one-shot knowledge gaps, while higher-order cases correspond to knowledge chain breaking in Section 3.2. In the scripted Interattt setting, every model finds linear gaps easier: once the prerequisite nodes are supplied, the remaining hop almost mechanical. Insert break within the chain, however, and success drops sharply because the model must now infer which intermediate concept is still unknown before can even formulate a clarification. When we switch to the agentic Interaact the story changes only for Claude-Sonnet-3.77 whose planning policy manages to erase the gap between the two categories; O3-Mini and Qwen-3 still stumble on higher-order cases. The trend suggests that the fundamental obstacle is not retrieval per but the metacognitive step of localising the missing link in a multi-step reasoning path sohh--onn only the most disciplined agent manages do reliably.
34

---

Published as a conference paper at ICLR 2026
High Order
Lincare
High Order-
Lincare
30
Success Rate (%)
25
20
15
10
Claude-3 7-Sonnet
DeepSeok-V3
DeepSeck-Ri
Gemini-2.8 Gemini-2.0-Flaan F-Flash
GPT-No
O3-Mini
Qwen-3
Figure 15: Success Rate of LLMs on linear and higher-order ambiguity over e and a-Interact Modes.
Table 10: Success Rate and Final Normalized Reward of different models BIRD-INTERACT-LIER The success rate is cumulative; Reward* is the normalized reward. The values reported in c-Interact after the debugging phase, and (+n) means the performance gained via debugging. Avg. Cost is the cost for one task on average in USD.
Priority Question (Success Rate %)
Follow Ups (Suceess Rate
Model
Reward
B1
DM
Overall
B1
DM
Overall
Cost J
c-Interact Text-to-SQL
DeepSeck-V3
(1.59
40.95
(1687)
20.33
(1330)
(4154)
(1.00)
(+1.5)
$
Qwen-3
14.36
44.76
25.00 (1267)
7.18
28.57
14.67
21.17
$0.03
DeepSeek R1
16.92
43.81
26.33
9.74
27.62
16.00
22.10
$0.08
.laud-Sonnett3.7
17.44 (13.9)
59.05
32.00
9.23 (1205)
27.62
15.67
26.10
$.32
Gemini-2.0 2.0-
16.92 (+1.0)
60.95 (x767)
32.33
(+500)
9.74 (+1
40.95 (+3.533
20.67 (-2.000
27.63
$ 0.004
GPT-4o
26.15 (-7.88)
54.29 (46.0))
36.00
(40000
14.36 (4000)
30.48
(.1.00)
20.00 (+133)
29.67
$ 0.32
O3-Mini
22.56 (1.55)
64.76
(1381)
37.33
(233)
12.31
46.67
(1005)
24.33
(0.33)
32.93
$ 0.09
a-Interaci Text-to-SQL.
Gemini 2.0. Flash
8.21
44.76
21.00
4.10
21.90
10.33
17,80
$ 0.03
DeepSeck-R1
6.67
47.62
21.00
3.59
28.57
12.33
18.40
$ 0.09
GPT-4o
12.31
43.81
23.33
4.62
17.14
9.00
19.03
$ 0.44
DeepSeek V3
11.79
44.76
23.33
6.15
16.19
9.67
19.23
S 0.06
Qwen-3
49.52
22.00
5.64
29.52
14.00
19.60
0.03
45.71
21.57
Claude Somnet-3.7
56.19
34.00
10.77
29.10
EXPERIMENTS ON BIRD-INTERACT-LITE
Table 10 reports results on BIRD-INTERACTTLITE.. We observe patterns consistent with those on the Full set: overall success rates and normalized rewards remain low, confirming the difficulty of interactive text-to-SQL even with simpler databases. Models that balance clarification with environment exploration, such as Claude-Sonnet-3.77 achieve higher SR and NR, while those relying too heavily on either execution or submission lag behind. Follow-up sub-tasks continue to pose greater challenge than priority queries, highlighting the difficulty of maintaining context across interactions.
M ERROR ANALYSIS
We conducted an error analysis by sampling 50 failed cases from our evaluation. We found that over 80% of the errors caused by incomplete ambiguity resolution. In many cases, systems either asked too few clarification questions, asked none at all, or failed to detect the correct ambiguity and request the appropriate clarification. On average, each task in benchmark contains around four
35

---

Published as a conference paper at ICLR 2026
40
20--bahh (thns)
Heras uerr
CIPT-4 (sosin))
Performance
30
Gesaio Ceniio-2..bae Eo.Bus (haelind
20
10
DeepSock-V
Qwen-3
DeepSock-R1
Gemmi-2 O-F-aash
GPI-40
Claude-3.1 3.7-Sonnet
03-Mini
Figure 16: The performance under our proposed two-stage user simulator and baseline user simulator compared with human users on 100 sampled tasks.
SQL
SELECT name FROM employee WHERE age 30
name
WHERE
age
>0
Figure 17: An example of an Abstract Syntax Tree (AST) for SQL query.
ambiguities (Table 1), but systems asked for clarification only about once per task (Figure 12). As result, most tasks were attempted with insufficient information, making it difficult reach the correct solution. This highlights the current limitations of LLMs in human-Al collaborative ability. The remaining errors stem from common issues text-to-SQL generation, such as SQL syntax mistakes, incorrect column selection, or misunderstanding of database constraints.
N
USER SIMULATOR DESIGN DETAILS
The main text describes our function-driven user simulator, which invokes the LOC () action to handle reasonable clarification questions that are not covered by pre-annotated ambiguities. This appendix details the Abstract Syntax Tree (AST)-based retrieval mechanism that allows the simulator to locate the relevant SQL fragment from the ground-trute (GT) query to answer such questions precisely. And the average cost for our function-driven
simulator is 0.03 USD per data.
The primary challenge for the LOC () action is to find the specific part of the GT SQL that corresponds to the system's 's question without resorting to brittle keyword matching on the raw SQL string. An AST provides a structured, hierarchical representation of the SQL query that is ideal for this task. Our retrieval process consists of three main steps: Parsing, Node Matching, and Contextual Snippet Extraction.
1. SQL Parsing into an AST. As first step, the ground-truth SQL query is processed by robust SQL parser (e.g., based on libraries like 'sqiglot') to generate an AST. As illustrated in Figure 17, this tree deconstructs the query into its fundamental syntactic components. Each node in the tree represents a part of the query, such as a clause (SELECT, FROM, WHERE), a function (COUNT (), AVG () ), an identifier (column or table names), an operator (=, >). literal value ('USA', 2023). This hierarchical structure makes every component of the query individually addressable.
2. Node Matching via Semantie Search of LLMs. With the AST generated, the next step is to identify the node(s) most relevant to the system's clarification question. To achieve this, we flatten the AST by traversing it and creating list of all its nodes. This approach is far robust than
36

---

Published as a conference paper at ICLR 2026
simple keyword matching, it can capture relationships like "how many" matching COUNT () or "most recent" matching an ORDER BY
DESC clause.
This AST-based method ensures that the LOC () function reliably ground its responses in the GT SQL, providing accurate and contextually relevant information without leaking the entire query.
EVALUATING THE FUNCTION-DRIVED USER SIMULATOR
To empirically validate the effectiveness of our proposed function-driven user simulator, we conduct comprehensive evaluation focused on its robustness and reliability. We first introduce new benchmark,
Sim Seeerim--uarra specifically designed to challenge user simulators. We then present our experimental setup and report the results, comparing our approach against standard baseline.
0.1 USERSIM-GUARD: A BENCHMARK FOR SIMULATOR ROBUSTNESS
To enable a systematic evaluation of simulator performance, we constructed User Saaessim-Guardd a manually curated dataset containing 2,100 challenging questions.
Construction Methodology. The construction of EiserSim-Gaadd was carried out by team of 7 trained annotators with expertise in SQL and natural language. To ensure data quality and diversity, we implemented a rigorous annotation protocol. The dataset is structured around three categories of system clarification requests, designed to probe different aspects of simulator's capabilities:
AMB (Annotated Ambiguity): For this category, annotators were tasked with formulating natural language questions based on the pre-annotated ambiguities present in the Bird- Interact-Lite benchmark. These questions directly test the simulator's s ability to correctly leverage the provided ambiguity annotations.
LOC (Localizable Information): This category contains reasonable clarification questions that are not covered by the pre-annotated ambiguities. Annotators were instructed to carefully examine the ground-truth SQL query and identify potential points of confusion (e.g., specific column choices, formatting preferences, or sub-component logic) and craft questions accordingly. The answers to these questions can be located and inferred from the ground-truth SQL.
To test the simulator's safety and adherence to its role, this category includes questions that are intentionally inappropriate or attempt to solicit privileged infor- mation. Annotators were prompted to formulate queries that directly ask for the ground-trun SQL,, the database schema, or step-by-step guidance for solving the problem. A robust simulator should refuse to answer such questions.
UNA (Unanswerable):
Furthermore, to investigate the simulator's sensitivity to different interaction styles, we instructed annotators to phrase each question in three distinct styles: Concise (terse and keyword- Normal (standard conversational language), and Verbose (descriptive and context-rich).
Quality Control. To ensure the highest data quality, employed a multi-stage quality control process. Each question-action pair in Users im-Guard rd was annotated using double-blind, "back- to-back" annotation scheme. Specifically, each data point was independently created by one annotator and then validated by a second annotator. Any disagreements between the two annotators were resolved by third, senior annotator who made the final adjudication. This process minimizes individual bias and errors. We measured the inter-annotator agreement (IAA) using Fleiss' Kappa, achieving score of 0.93, which indicates substantial agreement among our annotators and confirms the reliability of our labels.
O.2 EXPERIMENTAL SETUP
Models and Baselines. We evaluate our function-driven user simulator against a baseline simulator that directly generates responses using single-pass LLM prompt. To ensure a fair comparison, both our method and the baseline are implemented using two state-of-the-an large language models backbones: Gemi ni 22. 0-Flash, GPT-4o and C1 allaude-Hi k.
37

---

Published as a conference paper at ICLR 2026
Evaluation Framework. To provide objective and comprehensive observation of different user simulator mechanisms, we designed a robust evaluation framework using LLMss-asJudge. This approach allows for nuanced assessment of response quality beyond simple string matching. We employed the Qwen3 -2233---3335-- 235B-A22BB Innss--trrttt 2507 as evaluator.
For each generated response from simulator, the LLM judges were asked to perform multiple- choice classification task. This format was chosen to mitigate bias of ML------ddgee (Gu et al., 2024), reduce ambiguity, and create more differentiated assessments compared to open-ended feedback. The options were:
A. Perfect: The response correctly and accurately answers the question without revealing any inappropriate information. It is helpful and natural.
B. Acceptable: The response is functionally correct and does not leak information, but it might be slightly unnatural, too brief, or could be phrased more helpfully.
C. Incorrect: The response is factually wrong, fails to answer the question, leaks ground- truth information (especially for UNA questions), or is otherwise inappropriate.
A response is considered failure only if it is classified as 'C'. For reporting purposes, we consider both 'A' and 'B' as correct. To ensure the reliability of our results, we adopt a strict consistency-be evaluation: response is marked as correct only if both LLM judges independently classify it as either 'A' or 'B'. We report the final Accuracy, which is the proportion of responses deemed correct under this consistency rule.
O.3 RESULTS AND ANALYSIS
Our analysis reveals significant reliability concerns with conventional user simulator designs, which are substantially mitigated by our function-drivee approach.
As shown in Figure 6, the contrast is most striking when handling UNA (Unanswerable) questions. Baseline user simulators consistently fail to implement necessary safeguards, often leaking ground- truth details or providing improper guidance. This leads to extremely high failure rates, reaching up to 67. 4% depending on the backbone model. In contrast, our proposed function-driven approach demonstrates substantially improved reliability. By first classifying the intent of the request and invoking the UNA () function, it correctly rejects inappropriate questions, reducing the failure rate to as low as 2.7%. This represents a dramatic improvement in user simulator robustness.
Table 11 presents more detailed breakdown of accuracy across all question categories. We observe that LLMs themselves already perform strongly on the ambiguity-relate categories, achieving be- tween 87.7% and 92.3% aceuracy with the baseline approach. However, they struggle significantly with UNA (unanswerable) questions, where baseline accuracy drops as low as 32.6% for some back- bones. In contrast, our function-driven approach substantially mitigates this weakness, consistently achieving between 87.3% and 97.3% accuracy on UNA questions. These improvements confirm the observations from Figure 6. This demonstrates that while LLMS naturally handle straightforward clarification tasks. they require explicit structural constraints to avoid inappropriately answering questions that should be refused. Our two-stage design enforces such constraints by first identifying the question type before generating response, ensuring the simulator's 's behavior remains predictable, controllable, and aligned with the goal of providing fair and realistic user feedback without leaking ground-truth information."
P PATHWAYS TO EFFECTIVE COMMUNICATION
Motivated by the Memory Grafting results, which highlight the importance of communication skills for interactive text-to-SQU systems, we proceed to a deeper analysis. In this section, we investigate the specific communication patterns and dialogue strategies that lead to improved task performance. Through an in-depth analysis of high-quality interaction data, we identify a recurring and highly effective pattern we term the "funnel effect." This is characterized by series of progressively
We consider user simulator to he reliable if it achieves least 90% accuracy on both labeled and unlabeled ambiguity categories, and at least 80% accuracy on unanswerable (UNA) questions.
38

---

Published as a conference paper at ICLR 2026
Table 11: Accuracy (%) of user simulators on the AverSim-Guara benchmark using updated evaluation results. Accuracy is reported based on the consistency of two independent LLM judges.
Backbone
Simulator
AMB Acc.
LOC Acc.
UNA Acc.
Baseline
89.6
89.3
40.3
Gemini-2. 1arrii-200-PP- 0-Flash
Ours (Function-Drivn)
94.9
93.6
87.3
Baseline
87.7
89.4
77.3
GP
Ours (Function-Driven)
94.6
94.1
97.3
Baseline
92.3
92.0
32.6
Claude-Haikus
Ours Cinntion-Drrvent
95.0
95.0
95,4
Normal Sub-task:
Interaction3:
User: faciliiies 20%, calculate the Resource Demand Differential List the ficiliy
Prevalence
System: How many facilities should show the results? And how shoul! order them by the highest RDD values? User: The results should show the facilities, ordered by the highest RDD values in deseending arder.
RDD ordered RDD from highest lowest, showing the top 100 ficilities, This helps identify resource unkical environments.
Interaction 4:
Interaction I:
System: colculating Parient Functional Impaiment Scoce (PFIS) and Facility Resouree Adequey Index (F FRAI), should use the exact formulas from the external knowlodge...
System: noed to clarify what the Resource Demand Differenrial (RDD) is Rased the extemal knowledge,, I definition
you explait RDD should he calculated this concest?
User: The formulas provided for colenlating PFIS and FRAI conrect and should used deseribed for this task.
User: Resource- Demund Differenial (RDD)
calculated as the difference between the Patient Functional Facility Adouacy
Impuiment Index (FRAI)..
SELECT
Fac key AS Faciliiy sttyty_dd, fi.pfis, Fa.frsi, (fi.pfic ra.irai)
Interaction 2:
FROM fact lity teftt__ulffle cide _iiide_rsskk risk
System:
What does "high-iisk
prevalenee
faci werelityyuunceeeo functionnnnn___oc.. srrrroooonmmmmmemeetttt di fa. key fi. key
20%6 refor specifieslly" referring the Suicide Prevalence (SRP) mentioned the external knowledgee User: Yes, hhigh-risk eases prevalence over 20%6 refors to Suicide Risk Prevalence (SRP) as munsiond in knowladge.
COTN Fact ity resourceeeee.. reeeeourr__adeeeee
key fac cre_adequecy ra ON
MEERS fi.pfis I3 NOT HULL AND fao IS
HOT NULL
LIMI7 100;
Figure 18: Case study of effective communication under c-Interact.
deepening inquiries that begin with user's relatively broad and ambiguous initial intent, then gradually narrow the scope and clarify key details, and ultimately converge into clear and executable analysis plan. We deconstruct this pattern into three primary phases.
Initial Interaction Phase: Concept Clarification and Scoping, In the initial stage of high-quality dialogues, the Large Language Model (LLM) tends to pose questions aimed clarifying core concepts. This allows it to quickly identify ambiguous areas within the user's s query and proactively initiate dialogue for disambiguation. Such questions are highly targeted and efficient, for example: "How would you like to define the "interference score" for euch telescope?" or "Could you clarify what you mean by "machines that are always breaking down"?"
Concurrently, the model does not passively await precise descriptions from the user. Instead, it proactively offers specific options to guide the user toward a more explicit definition, thereby preventing further vague statements from the user, for example: "Should it be based specific columns like atmospheric interference, RFI status, or combination of factors?"
Furthermore, the model effectively integrate external knowledge quantify the user's subjective descriptions into actionable data criteria, for example: "Could you clarify what criteria should be used identify "good quality" scans? Should use the Premium Quality Sean definition from the
39

---

Published as a conference paper at ICLR 2026
external knowledge (SQS > 7.5, comprehensive coverage with Coverage 2 95% and Overlap 2 30%)?"
Mid-term Interaction Phase: Inquiring about Computational Logic and Implementation Details. As the dialogue progresses, the model's focus shifts to implementation details, concentrating on computational logic and operational steps. Given that user queries often involve complex calculations or business logic, such clarification is crucial for ensuring analytical accuracy. This includes precise confirmation of formulas, weight allocation, and the mapping between query variables and specific data fields, for example: "For the repair cost, should I use the muintenance cost (MainICost) or the replacement cost (ReplCost)...?"
The model also demonstrates a forward-looking capability for error detection, anticipating and mitigating potential data processing errors through questioning, for example: "I notice that 'recvDay' and 'beginDay' have different formats. Could you confirm how these dates are formatted so I can correctly calculate the time difference between them?"
A significant finding is the model's 's ability to uncover analytical dimensions that the user may not have considered, effectively asking questions the user didn't 't know to ask. This expands the depth and breadth of the analysis, for example: "Do you want to see the count of collectors for each idol genre, or do you want to see the distribution of idol genres that collectors interact with (which could include multiple genres per collector if they interact with different idols)?"
To ensure the accuracy of complex calculations, the model breaks them down into smaller, more easily verifiable steps and confirms each one with the user, for example: "To calculate Achievement Density (AD), need membership duration in days..."
Final Interaction Phase: Formatting and Final Confirmation. In the final stage, the dialogue's focus shifts to the formatting and presentation of the results. This typically involves a final confir- mation of the output fields, sorting rules, and numerical precision (such the number of decimal places) to ensure the final deliverable fully aligns with the user's expectations, for example: "For the output format, would you like the results to be ordered in any specific way...? Also, should I round the average BFR and standard deviation values to a specific number of decimal places?"
The example illustrated in Figure 18 exemplifies this high-quality interaction flow. The process begins with the clarification of the ambiguous concepts "RDD" and "high-risk cases with prevalence over 20%" It then delves into inquiries about calculation details and determines the presentation and sorting method for the results. Finally, by re-confirming the calculation formula, it ensures the rigor and accuracy of the entire analysis process.
Q HUMAN EVALUATION OF DATASET QUALITY
To rigorously assess the quality and reliability of our BIRD-INTERACT benchmark, we conducted a thorough human evaluation. We randomly selected 300 data points from the dataset and invited 10 experts with significant experience in SQL and database systems to serve as reviewers. Each data point, consisting of a user question, ground-truth -ruth SQL query, and its ambiguity annotations, was evaluated against a set of core quality metrics. The evaluation was performed using binary scoring system (1 for Accept, 0 for Reject) for each metric (Li et al., 2025c).
Evaluation Metrics. The metrics were designed to cover the three primary components of our dataset: the natural language question, the SQL solution, and the ambiguity annotations.
User Query Quality: This metric assesses if the user's natural language query is clear, fluent, and reasonable. The question must be logically sound and fundamentally answerable given the provided database schema. A question that is vague, unnatural, or impossible to answer based on the schema would be rejected.
SQL Correctness and Quality: This evaluates whether the ground-truth SQL query accurately and efficiently fulfills the user's 's request. The query must be both semantically correct (i.e., it logically answers the question) and syntactically valid. We also encouraged reviewers to reject queries that were unnecessarily complex or highly inefficient, ensuring a high standard for the solutions.
40

---

Published as a conference paper at ICLR 2026
Ambiguity Annotation Quality: This metric checks if the pre-annotated ambiguities valid and relevant. A high-quality annotation must represent a genuine point confusion that text-to-SQL system might plausibly encounter (c.g., ambiguity in column selection, grouping logic, or filter conditions). The associated SQL fragment must also accurately correspond the ambiguity it aims elarify.
Ethics and Safety: This assesses whether the content of the user question and the data context are free from any harmful, biased, or unethical content, ensuring the dataset is safe for use.
Evaluation Results. The human evaluation process confirmed the high quality of our dataset. Across all evaluated samples, we achieved an overall acceptance rate of 97.3%, indicating strong agreement from the experts on the dataset's validity. In particular, the SQL Correctness and Quality metric received an acceptance rate of 98. 98.7%, underscoring the technical reliability of our benchmark. The Ambiguity Annotation Quality was also highly rated at 95.3%, confirming that our annotations capture meaningful and realistic interaction challenges. These strong results validate that BIRD- INTERACT is robust and high-quality resource for developing and evaluating interactive text-to-SQL systems.
R PROMPTS
R. SYSTEM PROMPTS
Figure 19 shows the system prompt used under the c-Interact (conversational) setting, and Fig- ures 20-22 show the system prompts used under the a-Interact (agentic) setting.
R.2 USER SIMULATOR PROMPTS
Figure 23 shows the baseline user simulator, and Figure 24-25 show the our proposed two-stage function driven user simulator, containing parser and a generator.
41

---

Published as a conference paper at ICLR 2026
The prompt of system under e-Interact
"""You good data scientist with great PostgreSQL writing ability, You have 08 called "[DDB namne]"", You given Text-
SQL task.
#el Input Information:
You will provided with:
Task Description: The type of task you are trying accomplish.
Schema Informaion: detailed DB scherna with data examples.
Column Meanings: The detailed DB column meanings explanation.
External Knowledges: related External Knowledges about Text-to-SQL
Text to SQL Question: The Text to SQL question of this Text to SQL task.
Inputs:
< Start
Description
You are good data scientist who is tasked with generating PostgreSaL to solve the user query. However, the user's query may clear enough. you ask for clarification about these ambigulty query. only [[max_turn]] to ask for clarification, each turn you can only ask one question with few sentences. After using up all turns or you are clear enough, you can provide
PostgreSQL.
following
1. Ask for Clarification *** You can only ask **ONE** question each timel Then you MUST enclose your question between
and "s/s>", example
2. *Generate Final SQU * Then you MUST endoos your final PostgreSQU between "<t>" "<t>"">postspp"" pootgresql" and ""</t>", for example "pestgrrrrss [FILL-YOUR-SAL] "</t>".
NOTE: think you have enough questions used up turns, you MUST provide Postgre5CY, Text-to- SQL task!
<
Description|>
< Start Schema information|> >
[[D8_sc schemal]
< The Schoma Information|>
<I <he Start DB Colurn Meanings
"isson
[[column, meanings]l
<| <he DB Column Meanings >>
<he Start External Knowledge >>
"ison
[[exterall_ kg]]
<| |hhe End External Knowladge |>
<I The Start Text tQ- Question|]> >
[[user_quenn _query]]
<I The End Text to SQL Question|> >>
UHR Turn 1 [[[max, turn]] turns left):
"<s-|YOUR--ONI-OO SUUII--ONE--USSTONN///ss you
"<t>"" poottgrssl [FIL-YOUU--QL)
""</t>" if you choose to generate final SQL.
aen.
Figure 19: System prompt under Interactt.
42

---

Published as a conference paper at ICLR 2026
The prompt of system under a-Interact (1/tr)
""You helpful PostgreSCI agent interacts
to user's question.
Task Description
Your goal understand
ambiguous question inwolving the knowledge retrieval generate correct SQQI.
query solve You can:
1. Interact with the user to ask clarifying questions understand their request better or submit the SQL query the user. The
user will test SQL correctness and you feedback.
2, interact with the (self.eetting) environment (postgresq) clb colurn meaning file, external knowledge, and so on) to explore database and
relevant information.
Termination condition: The interaction will end when you submit the correct SQL query or the user patience runs out.
Cast your action: your action cost certain amount You are ReAct (Reasoning and then Acting) agent
patience.
according current observation, then action, and then get observation from the enviranment or user. You can repeat this process, like "Observation' >> "Thought" "Action" "Obsenvation >s "Thought" "Action" >> "Observation"
This means
first think about what to
se Interaction Format (Response Format)
previous interaction history, at the beginning), you should respand using the following format:
observation
your
user) request
<thought> the agent's thought about the current state </thought> <interaction objects interaction oboject object </interaction objects <action> action s/action>
IN The interaction object and action space with cost interaction_objeet "Envirommen"
action: eeecuue(ssall to interact with PostgreSQL database. inputs:
-sal: string, PSQL command to execute. Could contain multiple commands separated by semicolonn MUST BE Y"VY"YOUR HEREV"V'V.
STRING, ENCLOSED
output: fetched result from PortgreSQJ. database.
action: "igg_gccheee schema(["
the schema the database. output: string of database cost: 1 cost
action: 'get_ all_Loot columnn
(|l_oounnmmmninng//l meaningss)) to get the meaning columnns database. output: string of all column meanings. cost: 1 cost
action: columnn meaning(table
column, name)' get the meaning of column.
Inputs:
string, of the table get column meaning. -olummn @elummnnaameeeee name: string, name of the column to get meaning.
table
output: string of column meaning.
cost: 0.5 cost
action: all, external knowledge, nanes(() get external knowledge output: list string of external knowledge names.
cost: 0.5
action: get knowledge definition(nowledde name]' to get external knowledge by name.
Inputs:
knowledge, name: string, name of the external knowledge to get definition.
output: string of external knowledge definition.
cost: 0.5 cost
action: "gtt_n _a__al____ knllennwlddgg definitions]) all external knowledge with definitions.
outputt: string of all external knowledge names with definitions.
coss:: cost
interaction_objee "User"
user's question ambiguous, you darrfication to figure out the user's real intent. TO REDUCE COST, YOU ARE ONLY ALLOWED TO ASK ONE QUESTION AT TIME.
action: "sk(question)"
inputs:
question: string, question to ask user for clarification.
output: string
reply. clarify ambiguties his/her
cost: 2 cost
submit|sqll"
the give feedback.
inputs:
multiple commands separated by semicolon. STRING, ENCLOSED BY TWO QUOTES OR \"V") "YORR SQL "V"..
sql: string
output: feedback from about submitted
cost: 3 cost
NOTE) showing how patience remains Je.g. "[SYSTEM Remaining patience: 7/10)"), Pay close attention to this note as indicates how many more interactions make. If patience and you'll submit your answer.
each action, you'll
Figure 20: System prompt under a-Interact (part 1).
43

---

Published as a conference paper at ICLR 2026
The prompt of system under a-Interact (2/3)
# Important Strategy Tips
explore database schema, column meaning and knowledge to understand available tables, columnns user query's involved external knowledge,
FIGURE OUT THE USER'S REAL INTENT BY ASKING CLARIFYING QUESTIONS! IF YOU CANNOT FIGURE OUT THE USER'S REAL INTENT, YOU WILL PRODUCE WRONG SOL AND CAUSE MILUION DOLLARS LOSS OUR COMPANY, THEN YOU WILL BE FIRED!!!
ITO REDUCE COST OF USER PATIENCE, YOU ARE ONLY ALLOWED TO ASK ONE QUESTION AT TIME) FIGURE OUT THE USER'S REAL INTENT BY ASKING CLARIFYING QUESTIONS! IF YOU CANNOT FIGURE OUT THE USER'S REAL INTENT, YOU WILL PRODUCE WRONG SQL AND CAUSE MILLION DOLLARS LOSS OUR COMPANY, THEN YOU WILL BE FIRED!!!
REDUCE COST OF USER PATIENCE, YOU ONLY ALLOWED TO ASK ONE QUESTION AT TIME)
efficient with your actions to conserve user patience
Make your submitted SQJ, valid and addresses all of the question
Keep track of your remaining user patience and prioritiee your actions accordingly careful with the action frequently to you.
column meanings and external knowledge definitions. It
context
#df TOY Example Interaction
User's Question: Information.
calibrated
detected the observatory during phase. Show their
<thought>
This query seems ambiguous. I need to understand:
Which database tables
information about alien signals observatories
2. How "full moon phese" is represented in the data
"calibrated signals"
measure their strength
4. What does "several" mean, or more? and what information should show? database schema.
</thought>
<interaction object>
Environment
<finteraction, _bbjcct>
<action>
get schemal)
</action>
Observation: Database schema showing tables: Observatories(bbeent. ObservStation, ObservDate, LunarStage, Signal), Signalt(Signalt,, SignalType, SignalStrength)...
<thought>
can see there's an Observatories table LunarStage field Signal field. There's also Signals table more details about signal strength. need to understand what values the LunarStage field contains and how to identify "full moon" phase. </thought>>
<interaction object>
>
Environment
</interaction, objjct>>
<action>
columom maciinng_mannnggeee Observetrrrse 'LunarStagge)
</action>
Obxervation: Full name: 'Lunar Explanation: Current lunar during observation. Values include: 'Waving Crescent' First Quarter', 'Waxing Gibbous", 'Full', 'Waning Gibbous", 'Last Quarter, 'Waning Crescent'.
<thought>
Great,
the Lunarštage column. Now need understand what "calibrated alien signals" means and how their strength is measured. Let me check there's any external knowledge that could help. </thought>>
"Full" represents full
<interaction_daae _object
Environment
<finteraction_ohee _object>>
<action>
get_all all jet_all_extemal Lexeeeallaa....... eeteenall.nowweeeee..... knowledge_naneee names[]
</action>
Observation: l'Alien Signal Classification', "Lunar Phase Effects", "Calibrated Signal"]
<thought>
knowledge about "Callbrated Signal" seems directly relevant question "calibrated allen signals".
Let me check that information.
</thought>>
<interaction object>
Environment
</Interaction, _object>>>>>
get_knnn
wwwwwedde_deeflnailiiee peffinntionnaalllrrttttt Signal")
</action>
Observation:
'id":
"knowledge": "Callbrated Signal",
"description": 'Definition of calibrated signal",
"definition": "Those signal with strength 11."
Figure 21: System prompt under a-Interact (part 2).
44

---

Published as a conference paper at ICLR 2026
The prompt system under e-Interact (3/3)
<thought>
signals" signals strength many
user want. need ask the user for clanfication.
</thought>>
<interaction oojject>
User
</Interaction, objects>
<actions
Hsk results do you want? What do about "s"veral"??)
</action>
Observation: User's Response: three results sorted signal strength.
<thought>
What output field should show? need to ask the user for clarification.
</thought>
<interaction object>
User
</interaction, object>
<action>
ask/Doo you want to get the signal name and signal strength?")
<faction>
Observation: User's Response: I just want the signal
<thought>
Now have all the information needed write the query.
1. Join the Observatories and Signals tables
2. Filter observations during the Full
3. Filter for signals with strength > 10
signal strength
5. The user want the top 3 results with just the signal
</thought>>
<interaction_obuue _oobject>
</interaction object>
<action>
execute("SELECT s.SignalID FROM Observatories JDIN Signalslyn ON a.Signal s.SignalID WHERE o.LunarStage VFully AND s.Sinnaltteength DESC 3")
5.SignalStrength
</action>
Observation: (11,))
(11,). (10,)
<thought>
found the signal ids of the tap calibrated alien signals detected during full moon phases. Let me submit this SQL to the user.
</thought>>
<interaction_oopeee oobject>>
Vinteractioo_oojeee
submiI((SELECT SiginaliD FROM Observatories o JOIN Signals o.Signal s.SignalID WHERE o.LunarStage | \/Fuff' AND s. SiignalStrength 10 ORDER BY SiggnalStrength DESC LIMIT 3') </actions
Observation: Your SQL correct! Now, here's follow-up question from user: Can you also count how many distinct celibrated signals were detected during full moon phases?
UAR END OF TOY EXAMPLE INTERACTION HAR
---TAASK START--.
Now let's start with the user's question that may exist ambiguities and require external knowledge understanding solve, IEACH
TIME GIVE ONE ROUND RESPONSE, END YOUR RESPONSE 'e/action>' OTHERWISE YOU WILL BE FIRED!!!)
User's Question: (query):
[user. query])
[SYSTEM NOTE: You have total action budget of [[total] budget]] units. Each action consumes budget. the budget runs out, you submit.]
[[interaction [nnteration_historry]
Figure 22: System prompt under a-Interact (part 3).
45

---

Published as a conference paper at ICLR 2026
User Simulator Base prompt
You are good data scientist with great SQL writing ability. You have a DB called [108_name]]"" You are given the DB schema creation information below:
Here the DB schema information about Text task:
... D8 Schema Info: ---
[IDB_schema]]
User Question:
[[user_query]]
Ambiguitt points:
"ison [lambiguities_sson]
Correct SQL:
"sql [[correct_sq]]]
You are the user from company who asked the question above. And an All assistant is not clear about question. asks clarification below. have to answer those qustions mentioned the "Ambiguity points:" section above. the questian is mentioned above, you MUST tell that can answer, You can refer the correct SQI above to help answer. If you answer any unanswerable questions, your task will be falled and you will fired by your company!
NOTE:
1. Only "Your Answer" visible the Al the front part Ask for Clarification, Your mentions,
2. For each Al's question, you should only focus on rather than leaking information about other clarifications.
Interaction Process Starts:
Turn You should enclose your answer between "os>" and "c/s>"
Asks for Clarification: [[asked question]]
Your answer to Al:ss>>
Figure 23: The prompt of baseline user simulator.
46

---

Published as a conference paper at ICLR 2026
LLM as Parser prompt
USER interacting with collahorator complete Text tack. The Al collaborator may ask ane question about this task. Your goal to generate one re realistic, natural response that a user might give this scenario.
""You rale playing
AB Input Information:
provided
Task Description: The type of task you are trying to accomplish.
Ambiguity Alllebeled ambiguity points
for Text-to-SQL
Ground truth SQJ. Segments: All ground- truth SQJ segments.
Question from Collaborator: The question from collsborator to for clarificatoon ambiguity the Text SQL task.
Inputs:
Description
All|>
The question from Al collaborator maybe releted existing Labeled Ambiguitty Points or re related to unlabeled ambiguity or even irrelevant, So, you should
Action Choices:
1. "labeleditermm stri" When the question about existing labeled Ambiguity Points, use this action and fill the relevant term that ambigulty, Format: ""labeled"" Amb"'**
"unlabeled(segmean stri" When question
existing Ambiguity Points
important ambiguity that needs to be addressed, use this action and fill in the relevant SQL segment. Format: "unlabeledd" ALTER""**
3. "unanswerablel)" When think this question is neither related to labeled Ambiguity Points nor necessary to address, use action. Format: unanseeerbblll"
<) the End of Tack Description]>
<| The Start All Labeled Ambiguity Points (Not visible the A "joon
[[amb_json) [so]]]
End Labeled Ambiguity P
The Ground-truth SOL Segments (Not visible the AI]|>
[[SQI_Glo]]]
The End of Ground-truth SQL Segments|>
<| The Start of Question from Al Collaborator
[lelariiiation
<| The End of Question from Collaborator|>
at Guidelines:
You MUST choose only *Mone action"* listed above.
should NOT tell any thoughts about solution nor any ground-truth SQL information.
If you well, will get 10 thousand USD bonus!
ABl Output Format:
should your step-by-step thought "ethinks"" "s/thinke", "ega" "s/s"". Format example:
Thought:
<thinie(Step-by Thoughe)-/think>
Action:
YourYour Action)

Al Your Response:
Thought:
<thinio
Figure 24: Our proposed two-stage function-driven User Simulator: the prompt of User Simulator stage 1: LLM as Parser.
47

---

Published as a conference paper at ICLR 2026
LLM as Gonorator prompt
"*oo are role-playing es human USER interacting with an callaborrtor to complete a Text-to-SC), task. The collaborator aik one question about this task, Your gosl gererate one realistic, natural response that user might give in this srenario.
Input Information:
will be provided with:
Taak Desrriptio:: The type of task you are trying to arcomplish.
DB Schema informaion: The detailed DB achema with data soamples.
Labeled Ambiguity Points: labeled ambigutty
about the user's question for the lext-to-StI task.
Original
question
SCL Ground-truut SCL Segments: All ground-tnuu SCL segments of this lext to-S0L task.
Ground truth
ground
Queestion from al Collaborator: The question from colleborator to ask for denification on the ambiguity in the Tect-to-SK3 task.
Action Used: The selected action from given action space, where you shoud generate response based on this action!
<| The
Description visible A0) The question from collaborator maybe related to existing Labeled Ambigulty Points or reloted to unlabeled ambigutty even inelevant. So, one action was chosen at previcus turn.
Action
*"abbelditerm: stf"" the question about labeled Ambiguity Points, this action fil relevant tonn ambiiuutty Format: "Mabeldd"AAmb""""
*unnbbeeddiegmmnt str)** When the question NOT about existing labeled ombigulty Points BUT is still valuable and important ambiguily that needs be addressed, use this action and fill in the relevent SOt segmant. Format:
"unnense soraalee **: When think this question is nethher related labeled Ambiguity Points nor necessary address, this Format: "unanseerablef"*
Your You should generate respanse to answer the Collaborstor's question based the action and original clear teat-to-SC question below. You can NOT direclly give the original clear tect-to-SRL questton but can help you to answer question when not sure.
the and Task Description
information]=
[1O8, schemal)
<] The find of bis Schema information|>
the start Labeled Ambiguity Points viable the All
"Json
[[amb_ [soni]
The End of Labelee Ambiguity Points]>
The Original Text to Question)>
[[clear
The Original SCL Question)>
<) The Start Ground-truth SQL (Nor visille the Al]|> postgresel
[[GT_ (.85)
Ground-cou cruth SQL
<) The Start Ground-truth SQJ. Segments (Not visble to the AlJ>
[[SOL Glod]
The find (Ground-euuh SCU Segments]>
Question Collsboratorl>
[eelaniiiation (ill
<) The find of Question from Collsbarator(>
The Start Action Chosen (Not vesibe to AII
[[Actio]]]
Chosen]
es Guidslines:
"Remember"** if de following points well, 10 thousand USD bonus! You should generate directly give the original quostion above. not sure.
Collaborato'ss
to question action and
help you answer question you
You should NOT give any unfair information for esample: can ""NOT" sell any thought steps leading to final solution nor any ground cruth SQL sogments, You can "*NOT" change or adjust any setting of the text to SOL question when answering questions
should You should any question.
Dutgut
Your sponse must follow the format "csofFill-in Your Aesponni)cco for esamole, the action "unanswerablee]" you should respond: lep-Sor, this question is out of scops, so can not answer your questio.e/ss""
Figure 25: Our proposed two-stage funetion-driven User Simulator: the prompt of User Simulator stage 2: LLM Generator.
48

---

Published as a conference paper at ICLR 2026
Judge Prompt
<I The Start of Task Description
reviewing an Al-User conversation about Text-to-SQL task.
"Te Al may ask clarification questions to resolve ambiguily this is allowed. might attempt directly these are not allowed, and the user should refuse to answer such requests. for example, the questions like "Which table or "Which column" usually unanswerable.
ground-truth
step-by-tte by-bbyy-tepppp solution
Your job to check whether the user's response is appropriate: whether
properly: question answerable that the user answered correctly wittht providing ground truth SQL, schema details, or solution sndps"*;and (2). if unarswerable
First, analyze
should "reeuse
your reasoning "<t>..s/t"".
Then, give the user rating from:
**A, perfect**
the handled everything correctly,
**g, normal** + the user's response is accoptable but not ideal,
**C. bad** + if the acted incorrectly.
Only provide the rating letter enclosed in "<sp..s/s>".
<] The End of Task Description |>
<I the Start the Ask for Clarification Question from All>
[|clanification questioni]
<| End of Ask for Clarification Question from All>
User|>
[luser. answerl]
End
** Expected Output Format:**
"<t>/YOUR-THOR WWTTGHHTOUURRREEEECC BEFORE- YOUR CHOICE)s/t;; TUU QUALIY////>
Generation:
You: <t>
Figure 26: LLM-as-judge prompt evaluate the performance of user simulators.
49


Reviews:
[{"review_id": "gVZpu3hFMN", "reviewer": "Reviewer_4t8u", "summary": "This paper introduces BIRD-INTERACT, a new benchmark for Text-to-SQL designed to address the limitations of existing static, single-turn, and read-only datasets. It provides a dynamic, multi-turn evaluation framework where models must handle ambiguity, execution errors, and evolving user goals. The benchmark's core components include a task suite covering the full CRUD spectrum, an interactive environment with a knowledge base, and a novel function-driven user simulator that allows models to ask clarification questions without human supervision. The paper introduces two evaluation settings: c-Interact (a structured, protocol-guided conversation) and a-Interact (an open-ended agentic setting). Empirical results demonstrate the benchmark's significant difficulty, with top-tier models like GPT-5 achieving success rates below 18%, highlighting a substantial gap between current LLM capabilities and the demands of real-world dynamic database tasks.", "questions": "- **Q1. On \"Implementation-Level Ambiguity\":** Could the authors elaborate on the reasoning for classifying under-specification as \"ambiguity\"? Using the example from Appendix H.4, \"Show recent purchases\" is a valid query. Why is a model penalized for not clarifying a specific sort order (ORDER BY purchase_time DESC) that was never requested? Does this not create a benchmark that tests for a specific, \"over-specified\" ground truth rather than functional correctness?\n- **Q2. On \"Decimal Ambiguity\":** Following up on Q1, why is a query like \"Show average score\" considered ambiguous? Is the full-precision answer (e.g., 85.12345) considered incorrect? It seems that requesting rounding (e.g., ROUND(AVG(score), 2)) is an additional formatting instruction, and its absence doesn't make the original query ambiguous.\n- **Q3. On \"Knowledge Ambiguity\" Distinction:** Could you please clarify the distinction between \"one-shot knowledge ambiguity\" and \"knowledge chain breaking\"? The paper's example for chain breaking is to mask the 'AVS' node. How is this functionally different from removing the 'AVS' node under the \"one-shot\" definition? Both actions seem to result in the same missing node. Could you provide an example where these two ambiguity types are mutually exclusive and force different system behaviors?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **S1. Addresses a Critical Research Gap:** The paper correctly identifies a major limitation in existing Text-to-SQL research. The field has largely focused on single-turn SELECT query generation, which does not reflect the ambiguous, iterative, and stateful nature of real-world data analysis. This benchmark shifts the focus to a more realistic and challenging interactive paradigm.\n- **S2. Robust User Simulator:** The two-stage, function-driven user simulator is a significant contribution. By first mapping a model's question to a symbolic action (AMB, LOC, UNA) before generating a response, it avoids the common pitfalls of naive LLM-based simulators, such as ground-truth leakage and un-controlled-generation.\n- **S3. Comprehensive and Realistic Task Design:** The benchmark's design is high-quality. It expands the task scope beyond SELECT to include the full CRUD spectrum (DML/DDL), which is a crucial step towards evaluating real-world database assistants. It also introduces state dependency, where follow-up sub-tasks depend on the state changes (e.g., a newly created table) from a previous sub-task. Finally, the principled injection of ambiguities (in queries, knowledge, and the environment) makes interaction a necessity for success.", "weaknesses": "- **W1. Unclear Knowledge Ambiguity Taxonomy:** The distinction between \"one-shot knowledge ambiguity\" and \"knowledge chain breaking\" in Section 3.2 is unconvincing. The paper's own example for \"knowledge chain breaking\" is to mask the intermediate node \"AVS\". However, masking \"AVS\" is functionally identical to removing it as a \"one-shot\" ambiguity (defined as removing an \"isolated knowledge entry\"). Both actions result in the \"AVS\" node being missing. This conceptual overlap makes the taxonomy feel redundant and fails to delineate a meaningful difference between the two ambiguity types.\n- **W2. Questionable Premise of \"Implementation-Level Ambiguity\":** The paper categorizes user under-specification as \"implementation-level ambiguity\" (Section 3.2 and Appendix H.4), which seems to be a conceptual flaw. For example, the query \"Show recent purchases\" is presented as ambiguous, with the \"clarified\" version being \"Show recent purchases sorted by time\". A query that is merely less specific is not inherently ambiguous, as any result set of recent purchases, regardless of sort order, would be a correct answer. Similarly, \"Show average score\" is treated as ambiguous, with the \"clarified\" version being \"Show average score in 2 decimal\". The un-rounded, full-precision answer is also correct. This design choice seems to conflate ambiguity with a lack of specific formatting instructions, risking unfair penalization of models that make reasonable default assumptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "if3MwOb1hx", "reviewer": "Reviewer_h895", "summary": "This paper proposes BIRD-INTERACT, a benchmark for evaluating Text-to-SQL models in realistic, multi-turn interaction settings. It integrates a hierarchical knowledge base, metadata, and an autonomous user simulator to support dynamic clarification and error recovery. The benchmark includes two evaluation modes and full CRUD tasks with injected ambiguities, revealing a significant gap between current LLM capabilities and real-world interactive SQL reasoning.", "questions": "1. Generalization of the User Simulator (LOC() action): The success of the `LOC()` action depends on retrieving relevant SQL fragments via AST matching. Have the authors considered how their two-stage system would handle an out-of-distribution clarification request where the intent is reasonable, but the answer is not easily localizable to a specific SQL fragment (e.g., asking for an abstract policy not represented directly in the ground-truth SQL)? This would better test the generalizability vs. ground-truth grounding trade-off.\n2. Balance of Budget and Efficiency (for a-Interact): The analysis shows models favor costly `submit` and `ask` actions over efficient exploration tools. In designing the action costs, did the authors consider adaptive or dynamic cost schedules? For example, penalizing models that consecutively choose low-value `execute` actions might more strongly incentivize strategic resource exploration and planning.\n3. DM Task Distribution: The task suite includes the full CRUD spectrum (DM tasks). To better assess model capabilities in this crucial non-SELECT domain, could the authors provide a breakdown of the distribution of specific DM operations (`INSERT`, `UPDATE`, `DELETE`, `ALTER TABLE`) within the benchmark, along with the performance metrics for models on each of these types? This is vital for evaluating model readiness for operational database tasks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The work re-imagines Text-to-SQL evaluation by modeling a full, high-fidelity human-AI collaborative loop, moving beyond the static constraints of previous datasets. The combination of CRUD support, HKB reasoning, and dynamic interaction is unique.\n2. The rigorous task annotation (e.g., SQL snippets as clarification sources), the two-stage simulator with proven robustness on USERSIM-GUARD, and the comprehensive experimental methodology attest to the high quality of the benchmark construction.\n3. The paper is clearly written, and the analysis provides a deep, multi-faceted understanding of why current models struggle with the interactive paradigm.", "weaknesses": "1. Limitation on Simulator Creativity: While the function-driven approach ensures robustness against ground-truth leakage, its reliance on pre-annotated ambiguities and structured retrieval for `LOC()` actions fundamentally limits the model's ability to handle truly novel, unforeseen ambiguities that a real human might introduce. The simulator is robust but sacrifices some degree of open-ended human creativity. Future research should strive to achieve both robustness and broader generalization in the simulation.\n2. High Barrier to Entry: The main results rely heavily on expensive, closed-source models (e.g., GPT-5, Claude-Sonnet-4, Gemini-2.5-Pro). The high average cost per task (up to $0.67 for Claude-Sonnet-3.7) creates a significant financial barrier for broad academic community adoption, particularly for large-scale training and iterative development, even with the smaller LITE set. A more prominent emphasis on developing and benchmarking cost-effective, open-source models capable of effective interaction is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "flMmEOsg4x", "reviewer": "Reviewer_nKK2", "summary": "The paper introduces BIRD-INTERACT, a new benchmark designed to evaluate large language models (LLMs) for *interactive* text-to-SQL tasks. Unlike prior static, single-turn datasets (e.g., Spider, BIRD), BIRD-INTERACT simulates realistic multi-turn interactions between a system and a user simulator, capturing the iterative clarification, debugging, and follow-up behavior that occurs in real-world database applications.", "questions": "-  A key question is whether performance on *BIRD-INTERACT-LITE* reliably predicts results on *BIRD-INTERACT-FULL*. Quantifying this relationship—such as through rank correlation or regression analysis—could clarify whether the Lite version serves as an efficient proxy for large-scale evaluation, enabling faster model iteration without sacrificing benchmarking fidelity.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-  The paper addresses an important gap between single-turn text-to-SQL evaluation and real interactive database usage, providing a benchmark with clear practical relevance.\n- The inclusion of knowledge bases, metadata, and executable DBs offers a realistic simulation environment; the two-stage user simulator design is technically sound and well-motivated.\n- The paper situates its contribution well within prior work (e.g., COSQL, LEARN-TO-CLARIFY, LIVESQLBENCH) and justifies why static multi-turn datasets are insufficient.", "weaknesses": "- Limited discussion of annotation cost and human validation. While expert annotators are mentioned, clearer statistics on time, inter-annotator reliability per ambiguity type, or annotation guidelines would strengthen credibility. \n- The experimental setup reveals that each model was executed **only once** because of the *high cost of commercial API calls*, even though the authors used deterministic decoding (temperature=0, top_p=1) to justify single-run reproducibility. While this design ensures consistency, it also raises concerns about the **statistical robustness and variance** of results—especially given the benchmark’s complexity and multi-turn stochastic nature. Moreover, the reported per-model expenses for BIRD-INTERACT-FULL and -LITE (Table 2 and Table 10) suggest that large-scale evaluation is economically prohibitive, potentially hindering broader adoption and replication by the community", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u0SzVc0rJC", "reviewer": "Reviewer_gEbn", "summary": "BIRD-INTERACT proposes a new benchmark for text-to-SQL that captures the complexity of real-world, multi-turn database interactions. It introduces a dynamic environment with a novel function-driven user simulator to handle ambiguous queries and provide realistic feedback without human supervision. The benchmark features two evaluation settings—a structured c-Interact (conversational) and an open-ended a-Interact (agentic) mode—and includes tasks covering the full spectrum of CRUD operations. Experiments show that even state-of-the-art models struggle, highlighting that effective interaction, not just SQL generation, is a critical challenge.", "questions": "Refer to Weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. First, the paper makes text-to-SQL evaluation much more realistic. Instead of simple, one-shot questions, it creates a messy, back-and-forth conversation that forces models to ask clarifying questions and handle ambiguity, which is how people actually interact with databases. This is a huge and necessary step forward for the field.\n2. Second, the way they built the user simulator is incredibly smart. They didn't just use a standard chatbot that might accidentally leak the answer. They designed a clever two-stage system to make the simulator's responses controlled and reliable, which makes the whole benchmark much more trustworthy and fair.\n3. Finally, the results are a major wake-up call. [cite_start]The paper clearly shows that even the best models today, like GPT-5, fail spectacularly at these tasks, with success rates as low as 8.67%[cite: 2935]. This proves there's a massive gap between just generating code and being able to strategically interact to solve a real problem, which is a super important insight.", "weaknesses": "First, the user simulator, while cleverly designed for consistency, is probably too perfect. Real users are messy—they get confused, change their minds, or give bad information. By making the simulator so rational and predictable, the benchmark might be testing models against an idealized user, not a real one, which could hide how well they'd perform in the wild.\n\nSecond, the pass/fail success metric doesn't tell you why a model failed. When an agent gets a task wrong, it's hard to know if the issue was a bad plan, poor communication, or just a simple syntax error in the final SQL. The evaluation shows that it failed, but not where the breakdown happened, which makes it less useful for diagnosing the core problem.\n\nFinally, the rules of the agent game might be pushing models into unnatural strategies. For example, making it computationally \"expensive\" to ask the user a question could discourage communication and encourage a brute-force, trial-and-error approach. It’s possible the models are just learning how to beat the benchmark’s specific point system rather than learning how to solve database problems effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)