From Conversation to Query Execution: Benchmarking User and Tool Interactions for EHR Database Agents
Authors: Gyubok Lee
Woosog Chay
Heeyoung Kwak
Yeong Hwa Kim
Haanju Yoo
Oksoon Jeong
Meong Hi Son
Edward Choi
Venue: ICLR 2026 Poster (2026)
Keywords: Database Agents
LLM Agents
EHR-QA
DB-QA
Primary area: datasets and benchmarks

Abstract: Despite the impressive performance of LLM-powered agents, their adoption for Electronic Health Record (EHR) data access remains limited by the absence of benchmarks that adequately capture real-world clinical data access flows. In practice, two core challenges hinder deployment: query ambiguity from vague user questions and value mismatch between user terminology and database entries. To address this, we introduce EHR-ChatQA, an interactive database question answering benchmark that evaluates the end-to-end workflow of database agents: clarifying user questions, using tools to resolve value mismatches, and generating correct SQL to deliver accurate answers. To cover diverse patterns of query ambiguity and value mismatch, EHR-ChatQA assesses agents in a simulated environment with an LLM-based user across two interaction flows: Incremental Query Refinement (IncreQA), where users add constraints to existing queries, and Adaptive Query Refinement (AdaptQA), where users adjust their search goals mid-conversation. Experiments with state-of-the-art LLMs (e.g., o4-mini and Gemini-2.5-Flash) over five i.i.d. trials show that while the best-performing agents achieve Pass@5 of over 90% (at least one of five trials) on IncreQA and 60–70% on AdaptQA, their Pass^5 (consistent success across all five trials) is substantially lower, with gaps of up to about 60%. These results underscore the need to build agents that are not only performant but also robust for the safety-critical EHR domain. Finally, we provide diagnostic insights into common failure modes to guide future agent development. Our code and data are publicly available at https://github.com/glee4810/EHR-ChatQA.

Published as a conference paper at ICLR 2026
FROM CONVERSATION TO QUERY EXECUTION: BENCHMARKING USER AND TOOL INTERACTIONS FOR EHR DATABASE AGENTS
Gyubok Leet; Woosog Chay', Heeyoung Kwak², Yeong Hwa Kim² Haanju Yoo?, Oksoon Jeong'. Meong Hi Son', Edward Choi
KAIST' NAVER Cloud Samsung Medical Center?
fgyubok. lee, edwardchoi wttphh/iiiiaiittaaaa..aaa..a
ABSTRACT
Despite the impressive performance LLM-powered agents, their adoption for Electronic Health Record (EHR) data access remains limited by the absence of benchmarks that adequately capture real-world clinical data access flows. In prac- tice, two challenges hinder deployment: query ambiguity from vague user questions and value mismatch between user terminology and database entries. To address this, introduce EHR-ChatQQ ing benchmark that evaluates the end-to-end workflow of database agents: clar- ifying user questions, using tools to resolve value mismatches, and generating correct SQL to deliver accurate answers. To cover diverse patterns of query ambi- guity and value mismatch, EHR- C assesses agents in a simulated environ- ment with an LLM-based user across two interaction flows: Incremental Query Refinement (IncreQA), where users add constraints to existing queries, and Adap- tive Query Refinement (AdaptQA), where users adjust their search goals mid- conversation. Experiments with state-of-the-art LLMS (e.g., 04-mini and Gemini- 2.5-Flash) over five i.i.d. trials show that while the best-performing agents achieve Pass @5 of over 90% (at least one of five trials) on IncreQA and 60-70% on AdaptQA, their Pass (consistent success across all five trials) is substantially lower, with gaps of up to about 60%. These results underscore the need to build agents that not only performant but also robust for the safety-critical EHR domain. Finally, we provide diagnostic insights into common failure modes to guide future agent development. Our code and data publicly available com /ttt::///cttttt....ooo//eeeeeeep Chat
interactive database question answer-
INTRODUCTION
Large Language Models (LLMs) are increasingly operating autonomous agents, interacting with external environments to solve complex tasks. One key application interfacing with structured databases, which can substantially enhance data accessibility for non-technical users. This capa- bility is particularly impactful in high-stakes domains such as Electronic Health Records (EHRs), where enabling natural language queries over vast patient data repositories has the potential to fun- damentally transform both clinical research and patient care (Ohno-Machado 2011 Yang et al., 2022). To assess such capabilities, the prevailing evaluation paradigm has relied on texL-to-SQL benchmarks (Yu et al.) 2018 Wang et al. 2020 Lee et al. 2022 Li et al.) 2023). which measure a model's ability to translate natural language questions into SQL queries. However, these bench- marks primarily emphasize the isolated task of mapping single, well-formed question to SQL, setting that fails to capture the complexities of how clinicians interact with EHRs.
To further investigate the gap between this evaluation paradigm and real-world clinical needs, we collaborated with a major academic medical center and reviewed internal text-to-SQL query logs. This analysis revealed that existing benchmarks not suitable for capturing two core roal-world
"This work mostly done during his internship NAVER Cloud.

---

Published as a conference paper at ICLR 2026
Setup
Interactive QA Simulation with Tools
QAA Tasks & Databases
& Simukited User
Find recent hemoglobin value for patient 1001
Query Ambiguity
("hemoglobin"
value search
Value Mismatch
Agant
Simulated User
What you mean by rreent"? ?
hgb
Datebresn
Tools value search(), executee),
8 Simubbted User
In the past month
Agent
O
("SELECT resultvalue
sql execute
itemname = 'hgb'
Pgent
Patient 1001's recent hemoglobin value is 10.2
[10.22)
Simulation Validator
Agent
Datelhoen
Rule based Evaluation
Simulation Validator
Figure 1: Overview of EHR-ChaIQA. Our benchmark places an agent in a simulated environment
with an LLM- powered user and tools. The agent must manage the entire workflow autonomously: clarifying vague user questions, resolving terminology mismatches, generating and executing accu- rate SQI., and returning the final answer. When the interaction is over, each trace is verified by simulation validator, and only traces that pass proceed to rule-based evaluation.
challenges, which significantly hinders the adoption of these models: (1) Query Ambiguity Users often pose vague questions that do not fully capture their true intent (e.g.. a request such as "Show me recent labs" lacks specifics such test types or time ranges) (Wang et al. 2023 Saparina & Lapata 2024 Dong et al. 2025). (2) Value Mismatch: Clinical terms in EHRs vary due to specialized database naming conventions and often differr from everyday language (e.g.. "Lopressor" may not match the database entry "metoprolol tartrate"), creating a terminological gap (Holmes et al. 2021
De Mello al. 2022) that must be bridged for accurate SQL generation.
Addressing these challenges requires moving beyond static, single-turn SQL generation to an envi- ronment where an agent can clarify user's intent, invoke necessary tools navigate complex EHR schemas and clinical values, and synthesize all relevant information to generate an accurate SQL query. To bridge this gap, we introduce EHR- Chaa-C, an interactive database question answering (QA) benchmark designed to assess this end-to-end agentic workflow, from conversation to query execution, in the EHR domain. By placing agents in a simulated environment with both an LLM- based user and a suite of tools, EHR-ChatQM provides a holistic evaluation of agent capabilities in interactive clinical data access flows. Each interaction trace further verified by a dedicated valida- tor to ensure evaluation reliability. Our benchmark is grounded in real-world clinical QA scenarios and two publicly available EHR databases (MIMIC-IV and eICU). consists of tasks categorized into two interaction flows designed for various query ambiguities and value mismatches: Incremen- tal Query Refinement (IncreQA), where users add new constraints to a query, and Adaptive Query Refinement (AdaptQA), where users adjust their search goals mid-conversation.
Our evaluation of various state-of-the-ann art LLMs on EHR-ChatQA reveals a critical lack of robustness under diverse conversation paths. While agents often succeed in at least one of five attempts task (Pass 5), their ability to succeed consistently in all five attempts (Pass"5) 5) is substantially lower. This performance gap exceeds 38% for IncreQA and 36% for AdaptQA, reaching up about 60%. This inconsistency highlights a crucial lack of reliability in current agents, raising significant concerns for their deployment in safety-critice domains such as EHRs and pointing to key areas for future research.
The main contributions of our work are summarized as follows:
We propose CHR-ChatQQ the first interactive benchmark for EHR QA that holistically evaluates agents' interactive, end-to-end workflows using simulated users and set of cus- tomizable tools for schema exploration, value exploration, and web search.
Grounded in real-world clinical QA scenarios and two publicly available EHR databases, the benchmark contains two interaction flows to reflect various query ambiguity and value mismatch patterns.
While linguistics distinguishes it from vagueness (a lack of specificity), we adopt the common NI.P con- vention of using "ambiguity" to refer any query requiring clarification.
2

---

Published as a conference paper at ICLR 2026
Our evaluation of various LLMs reveals critical performance gap between agent' 's opti- mistic success (Pass (asss()) and its consistent success (Pass* providing diagnostic insights for developing more performant and reliable agents in interactive EHR QA.
RELATED WORK
Text-fo-SQL Benchmarks Text-to-SQL research has largely focused on translating single, well- defined question into an SQL query. Benchmarks such as Spider (Yu et al.) (018)) and BIRD (Li et al. 2023) are prominent examples that have shaped this paradigm. While other benchmarks such as SParC (Yu al. 2019b), CoSQL. (Yu et al.] (019a), and PRACTIQ (Dong al.) 2025) introduced conversational context, their evaluation scope is often limited by predefined interaction patterns, such as requiring the model to generate an SQL query each turn or to expect a type of responses based on the script. This evaluation setting prevents them from capturing the open-ended o and exploratory nature of realistic user interactions, thereby falling short testing agents' ability to resolve query ambiguity and value mismatch. EHRRChatQA is benchmark dedicated bridging this gap by explicitly evaluating agents' exploratory and interactive capabilities through simulations.
EHR Question Answering The unique challenges of the medical domain have inspired several QA benchmarks for structured EHR data. Text-to-QLL benchmarks, from single-turn settings such as MIMICSQL (Wang et al.) 2020) and EHRSQL (Lee et al. 2022) to the multi-turn EHR -eqsSL (Ryu al. 2024), have advanced querying complex EHR schemas but lack support for resolving query ambiguity and value mismatch. More recently, agent-based benchmarks for EHRs such as MedAgentBench (Jiang al. 2025) and MedAgentGymm (Xu et al., 2025) tackle broad range of clinical and biomedical tasks. However, their reliance on initial non-ambiguous task instructions still bypasses the need for dynamic, interactive resolution of query ambiguity and value mismatch, which are essential for real-world, interactive clinical QA. EHR-ChatQA aims to bridge this gap by explicitly requiring the simulated user start from a vague question and allowing the agent to autonomously resolve the task.
Conversational and Tool-Using Agent Evaluation Evaluating L.M agents in dynamic environ- ments has spurred progress in two key areas: tool-using agent benchmarks (Yao al. 2022 Liu
et al.),, which assess instrumental competence, and frameworks for evaluating conversational skills
task-oriented dialogues, from large-scale curated datasets (Budzianowski
al. 2018 Rastogi
et al. (2020) to dynamic user simulations (Sekulić et al. 2024). Recent works such as Tau-Bench Yao et al. 2025) and ToolDial (Shim et al.) (222) combine these paradigms to evaluate the crit- ical dual interaction loop between agents, users, and tools. EHR- -haR-ChaQAA extends this framework to question answering over EHR databases. As shown in Table T although prior works focus on resolving query ambiguity and leveraging general tools for complex tasks, they do not specifically capture the challenges of interactive clinical data access, particularly value mismatch (Holmes et al.)
2021 De Mello et al. 2022). EHR-ChatQA is designed to holistically address these challenges, QA scenarios and EHR databases.
grounded in real-world
THE EHR-CHATQA BENCHMARK
3.1 TASK FORMULATION
The task instances in EHR-ChatQA can be formulated in the POMDP framework as AS T, R, ?), capturing the inherent uncertainty and sequential nature of translating an ambigu- ous user request into an executable query. The state E is latent and includes the user's true but unobserved intent (i.c., user instruction), the full conversation history, the contents of the EHR database, and any accessible external knowledge. The agent's 's action space A models the dual in- teraction loop: (1) user interaction, including asking clarifying questions or providing answers, and (2) tool interaction, including invoking calls to explore the database schema, search relevant values, or access external web knowledge when necessary. After taking an action E A, the agent receives an observation o E Ω which is either a natural language response from the user or the output from tool. The transition dynamics T combine deterministic and stochastic processes: tool interactions with the database are deterministic, while the -asedd user's s responses and web search results stochastic. The reward function R is binary, defined as 1 if the agent's answer matches the
3

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of recent benchmarks categorized by core agent capabilities. EHR-ChatQA is the first benchmark to comprehensively evaluate database agents in the aspects ranging from con- versational ability to effective tool use in the EHR domain. "Value Explor" indicates the mapping of user terminology to database entries (e.g., "WBC" - "white blood cell count"). A denotes reso- lution of ambiguity through SQL suggestions instead of user clarification.
Conversational Ability
Tool-Using Ability
Domain
Benchmark
User Multi-turn
Query Ambiguity
Tool Use
Value Explor.
EHR
Spider Yu et al.
2018
X
X
SParC
Yu et
2019b
X
x
X
CoSQL
Yu al.
2019a
V
X
X
EHRSQL
Loe ct
2022
X
X
X
X
BIRD Li
2023
X
X
X
AgentBench (Lu ct ml.
X
X
V
X
X
EHR-SeqSOL [Ryu al. 2024
V
PRACTIQ Dong et al. 2025
A
X
X
X
Tau- Tau-Bench Yao 2025
V
ToolDial (Shim et al.
2025
V
V
X
MedAgenlBench Jiane al.
2025
X
X
MedAgentGyym Xu al. 2025
EHR-ChatQA (Ours)
ground-truth (GT) and 0 otherwise. In our evaluation, the reward is assigned in rule-based manner against deterministic GT SQL queries and answers (see Appendix [.66 for details).
3.2 BENCHMARK COMPONENTS
Task Instances and EHR Databases Each task instance defines a unique interactive QA sce- nario, containing a user instruction that specifies the LLM-based user's query goal and outlines the intended conversation flow (e.g.. "You want to know the number of patients prescribed Lopressor... no patients are found, you want to count the patients for its generic equivalent...)) Each instance also includes the answer to the interaction goal and the GT SQL queries for evaluation purposes. All user instructions and answers are grounded in patient data stored in two EHR database schemas (MIMIC-IV and eICU), and clinical knowledge is essential for locating, filtering, and reasoning over the retrieved information. The benchmark consists of 366 task instances, distributed across the two databases and the two interaction flows, as shown in Table 2
Tools Default tools provided in this benchmark include functionalitiis for schema explo- ration (table-search) value www.mmmartttssrrreeeeeee
wellumnneaaccii value exploration vvalue-substry wwubstriggsser... wng.seardee ch,
external knowledge retrieval (web_seal search), and final query execution (sqleexecute). Since all SQL annotations are done using interface restricted to these schema and value exploration tools (see Section 4.2). they provide sufficient resources for constructing correct SQL queries when the agent is supplemented with SQL expertise and physician- level clinical knowledge. The web_search () tool is optionally provided to the agent to supple- ment confirm clinical knowledge, accurate handling of such knowledge is critical in the EHR domain. More details on tool specifications are provided in Appendix A.2
Simulated User To evaluate user interaction at scale and measure agent performance across di- verse conversation paths, we use an L..M-based user simulator that leverages certain level of stochasticity in user utterances. We use Gemini-2.0-Fii 2.0--lshh at temperature of 1.0. The simulated user initialized with system prompt (see Appendix (A.322) containing a user instruction and set of behavioral rules. These rules include intentionally starting with vague initial query, which forces the agent to engage dialogue for clarification. The rules also define the conditions for ending the conversation: either the agent successfully retrieves and provides the information requested by the user, or the agent repeatedly fails to retrieve relevant information and shows no sign of progress (see more details in Appendix I). To improve the simulator's instruction-follovin ity, we employ a nested cerification-rellectie mechanism inspired by Shinn et al. (2023), where a turn-level verifier (at temperature 0.0) and chained reflection loop check each response before it is committed to the conversation (see Appendix A.3.4.. Crucially, throughout this process, the user is
reliabil-
Text columns are pre-indexed. We use OpenAl's eext-emhedding---laggee
4

---

Published as a conference paper at ICLR 2026
User instruction: Your to explore medication prescription statistccs. Specifically. you want to know the percentage of patients diagnosed with atrial fibriliaion who were prescribed a medicatinn recorded exactly as 'opressor" o. patients were prescribed medication under the exact name "Lopressor'' you want repeat the same question using medications containing its active ingredient.
Slimulated User
S Tool
Database Agent
User Interaction
Tool Interactionn
I'd like get some statisties on Lapressor prescriptions among pabentt with strial fibillation.
Since there such patients, you check medications with same active Ingredient as Lopressor?
web search (query ingredient of Lopressor) )
- "aclive
Lopressor contains the active drug ingredient metoprolol fartrate,
help with that, Could you be more specific about statistics you're interested in?
There patients diagnosed with atrial fibrllationn prescribed Lopressor. you need any other help?
50
1
eol execule "SELECT COUNTIDISTINCT palientid
want to know the percentage of patients with atrial fibrilation who prescribed Lopressor.
8
50
[O.]]
medicationneme 'metoprolol tartrate '7]
Okay, need to the table and the column where the diagnosis of atrial fibillation recorded.
sq execule(quuery = "SELECT COUNTIDISTINCT patientid...
|109 5852.))
+
medicationname 'Lopressar""'
percentage palients atrial fibillation who received metaprokol is 69.5652%
idenographics, hoepitaladmiesionn, diggnasisodds, proceduresodes, ...]
103
[ebbe_seeech))
Figure 2: Example interaction trace in EHR- CHR-ChatQA (AdapiQA), Given vague initial user ques- tion, the agent clarifies the request (1) by refining "some statisties" into the specific goal of "the percentage." When the initial query yields guides the agent to search for its generic name instead (2). The agent then incorporates clinical knowledge retrieved via web-search () to reformulate the query (3), and finally executes the re- vised query to produce the correct answer (4).
matching records due value mismatch, the user
isolated from the database content and the GT SQL and answers, preventing it from "tipping off" the agent with information they should not possess.
Simulation Validator There occasions where LLM-bbsed simulations deviate from the in- tended behavior, not because of the agent's failure but because the user simulator itself deviates from its instructions. To mitigate this and ensure reliable evaluation of agents, we implement an LLM-as-a-judge classifier (Zheng et al. 2023) as a validator After each completed simulation, the validator reviews the entire dialogue trajectory. it determines that the simulated user has vio- lated its given instruction or rules, the simulation trace is considered invalid and subsequently rerun, regardless of the task's 's outcome. When a user error is detected, the validator's error explanation is accumulated and fed into subsequent re-runs, enabling subsequent re-runs to avoid previously observed mistakes. The prompt used in the validator is provided in Appendix A.4.
4 BENCHMARK CONSTRUCTION
4.1 INTERACTION FLOWS
To capture wide range of query ambiguity and value mismatch patterns grounded in various clinical QA scenarios, we include two different interaction flows for the simulated user to follow:
Incremental Query Refinement (InereQA): This flow tests an agent's 's ability to maintain conversational context the user incrementally constructs query by adding new con- straints toward a single linear goal. The agent must integrate new details into the existing context without losing prior information. Examples of such constraints include specifying event timing (c.g., "diagnosed within the past 180 days"), narrowing to a specific patient cohort (e.g., "among patients diagnosed with diabetes"), or adding related clinical criteria (e.g., "who did not have an HbAlc test recorded") (see Appendix A.I.I for sample task instance).
Adaptive Query Refinement (AdaptQA): Unlike IncreQA's linear goal structure, this flow tests an agent's ability adapt its query plan conditional branching points, where
We use Gemini-2.5- Flash at temperature of 0.0.
5

---

Published as a conference paper at ICLR 2026
Table 2: EHR-ChaIQA task statisties. indicates preprocessed databases with renamed schemas.
IncreQA
AdaptQA
Total
MIMIC-IV*
145
40
185
eICUx
40
181
Total
286
80
366
the user's next question depends on the agent's intermediate response (e.g., searching for a medication within the same or different drug classes if the initially requested one is not found, rolling back when partial information missing, or adopting fallback strate- gies when no relevant data is available). By design, these tasks require more advanced value mismatch resolution than IncreQA, often going beyond synonym matching (see Ap- pendix A.1.2).
4.2 ANNOTATION PROCESS
core team of three annotators (two graduate-Ievel computer science students and one physician) led the initial development, from drafting through internal quality checks. This phase was followed by beta testing with 38 graduate- contributors, whose feedback informed the final revisions.
4.2.1 EHR DATABASES
To evaluate agents' generalizability to different EHR structures, we use two publicly available EHR databases with distinct schemas and data recording practices: MIMIC-IV (Johnson et al. 2023). which contains detailed ICU data from Beth Israel Deaconess Medical Center, and eICU (Pollard) et al. 2018), which includes ICU data from multiple U.S. hospitals. For instance, eICU stores pre- scription records splits it across separate columns (drug name, dosage, unit of measurement) as "clopidogrel." "75.0," and "mg." " As the basis for task instances, demo versions of these databases. Although they contain fewer patients, these demos retain the same schema complexity as the originals
single string such as "clopidogrel bisulfate 75 mg po tabs," whereas MIMIC-IV
subset of records from the privacy-safe
A key challenge we identified is that SOTA LLMs often memorize the original schemas of these popular databases, allowing them to generate SQL without genuine schema exploration. To ensure that our evaluation truly tests agent's 's ability to navigate arbitrary databases, we rename all table and column names (e.g., "patients" cICU+, compel agents to rely on schema exploration tools rather than their prior knowledge. Further details on this process are provided in Appendix [..4
"demographies") The resulting databases. MIMIC-IV+ and
4.2.2 INCREQA ANNOTATION
To create IncreQA tasks, we first curate and adapt clinically relevant queries from two primary sources: the EHRSQL dataset (Lee et al.) 2022) and internal logs from our collaborating medical center. We then annotate the corresponding GT SQL queries and answers. Next, convert these SQL queries into narrative user instructions using a SQL-to-text approach. By grounding each instruction in specific SQL constraint, minimize the ambiguity inherent in natural language. For example, "How many emergency patients are there?" is ambiguous (admission type='urgent" admit jource='emergency room'), but instruction derived from WHERE admit source "emergency room' is not. We manually review the resulting instructions to detect any missing SQL. details residual ambiguities. Finally, we rephrase database values into everyday language to introduce value mismatch challenges (e.g., mapping "malignant neoplasm" to "cancer"). Details of the IncreQA annotation process are provided in Appendix B.I.I
"The annotated SQL queries and GT answers are tailored the demo databases; running them the full versions may yield different results due to differences patient populations and record counts.

---

Published as a conference paper at ICLR 2026
IncreQA
AdapiQA
Model
SR-5
Pass@5 (t)
(t)) Gap-5 (1)
SR-5 (1)
(
Pass"S (t)
Gup-5 (U)
Closed-source Models
Gemini-2.5-Flast
73.3
91.3
47.6
43.7
30.0
65.0
3.8
61.2
Gemini-2.0-Faan
59.4
83.9
29.0
54.9
16.2
40.0
2.5
37.5
04-mini
81.3
96.5
57.7
38.8
35.8
66.2
7.5
58.7
GPT-4o
64.5
87.1
35.3
51.8
28.5
53.8
11.2
42.6
GPT-4o-mini
47.8
73.8
21.3
52.5
19.0
40.0
36.2
Open-source Models
Llama 3,3-70B
40.5
68.9
16.4
52.5
14.2
41.2
1.2
40.0
Qwen3-32B
58.7
87.4
25.5
61.9
21.2
50.0
2.5
47.5
Table 3: Overall results on EHR-ChatQA across two different interaction flows: Incremental Query Refinement (IncreQA) and Adaptive Query Refinement (AdapiQA), averaged over combined MIMIC-IV* and eICU* samples. Metrics include: SR-5 (average success rate over trials), Pass @ (success in least one of 5 trials), Pass (success in all 5 trials), and Gap-5 (Pass@5 Pass"
4.2.3 ADAPTQA ANNOTATION
AdaptQA tasks focus on scenarios requiring goal adjustments during interaction. Unlike IncreQA, begin by defining eight query modification categories (Table [00 representing adjustments due to data absence or schema structures, A key design principle is that each instruction must withhold information that the agent is expected to discover on its own. This enforces a deterministic, decision- tree-like workflow where the user's 's next action depends on the agent's 's intermediate response. We then annotate instructions that implement these conditional workflows based on specific patient data or database structures. For instance, to create task in the "brand-generie name resolution" category, we select patient lacking records for a brand-name drug (e.g., Lipitor) but having records for its generic equivalent (c.g., atorvastatin). The instruction directs the user to ask for the brand name first and, upon failure, to check for the generic name. Crucially, the user in the interaction does not know the underlying database state, so the agent must first report the absence of the brand-namd name record and then, when the user follows up, map the brand name to its generic counterpart using clinical knowledge that the user does not explicitly provide. Following physician verification of these logical flows, we annotate the corresponding GT SQL queries and answers. Details of the AdaptQA annotation process provided in Appendix B.1.2
4.3 QUALITY ASSURANCE
Our quality assurance procedure consists of an internal validation process followed by an external beta-testing phase. The internal validation targets both task instances and the simulation environ- ment. For task instances, we employ an iterative refinement loop: preliminary simulations flag tasks that repeatedly cause agent failures for manual review, which focuses the alignment between annotated SQL and user instructions and on resolving any residual ambiguities. For the simulation environment, manually review failed dialogue trajectories, especially the agent's value-linking logic (e.g., when a user instruction specifies hemoglobin, we check whether the instruction is clear enough to guide only to "Hb," not to other similar terms such as "Hb C²). Following this internal val- idation, the benchmark underwent beta testing over two months with 38 graduate-level contributors, whose feedback on instruction clarity and user behavior informed the final revisions. The contrib- utor demographics and testing details are provided in Appendix A.7 The validator's 's reliability is further verified through manual inspection Appendix(A.441)
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Models
We evaluate leading closed-source and open-source LLMs with strong function-calling
capabilitic. For closed-source models, we use OpenAl's o4-mini, GPT-4o, and GPT-4o-mini,
'We use standard function-calling setup with a shared system prompt and identical tool definitions as uniform haseline for all models, rather than optimizing agent architectures for peak performance.

---

Published as a conference paper at ICLR 2026
well Google's Gemini-2.5-Flas and Gemini-2.0-Flas For open-source models, we eval- uate Llama 3.B-70B and Qwen3-32B, served on four NVIDIA A6000 GPUs using the vLLM li- brary (Kwon et al. 2023). All implemented agents provided with a set of behavioral rules (see Appendix C. database-specific SQL generation rules (see Appendix A.5), and evaluation rules (see Appendix A.6). The temperature for all agent LLMss is set to 0.0, meaning that performance variation across trials for the same task arises solely from the stochasticity of the LLM-based user simulator (temperature 1.0). Each simulation limited to maximum of 30 agent actions. The agent implementation details are provided in Appendix C
Evaluation Metrics We evaluate agent performance using four metrics: SR-k, Pass@k, Pass k. and Gap-k. SR-k measures the average success rate across k i.i.d. trials for each task. Pass@k (Chen et al. 2021). representing an agent's optimistic performance, is the proportion of tasks solved in at least one of these trials. Conversely, Pass"k (Yao al., 2025) assesses consistent and reliable performance by measuring the proportion of tasks solved in all k trials. The final metric, Gap-k, the difference between @ass@k k and k. While SR-k serves as stable measure of overall performance due to its lower sensitivity to the number of trials, k, the other three metrics vary with k. In particular, Gap-k indicates an agent's robustness across diverse conversation paths and an important indicator of potential degradation in performance over multiple runs, which must be avoided the safety-critical EHR domain. We set k=5 throughout the experiments.
5.2
MAIN RESULTS
Overall Result Table 3 summarizes the performance of various state-of-the-art LLMs on EHR-
ChatQA. Closed-source models generally outperform their open-source counterparts. In terms of SR-5, 04-mini achieves the highest overall performance, with scores of 81.3% on IncreQA and 35.8% on AdapiQA, followed by Gemini-2.5-Flashh Open-source models such as Llama 3.3-70B and Qwen3-32B show clear limitations, particularly struggling with AdaptQA (14.2% and 21.2% SR-5, respectively). While the top closed-source models perform strongly on IncreQA, with SR-5 scores of up to 81%, their performance drops to the 16 16-36% range on AdaprQA. This indicates that AdaptQA requires more advanced adaptive query refinement and sophisticated value explo- ration. The most salient finding is the substantial discrepancy between the optimistic performance (Pass@5) and the consistent performance (5. For instance, o4-mini's Gap-5 reaches 38.8% on IncreQA and 58.7% on AdaptQA, indicating that even the best-performing agent lacks the con- sistency required for safety-critical deployment.
Interaction and Cost Analysis Averaged across all models and trials, agents engage in 5.5 user interactions and 7.8 tool interactions per IncreQA task, and 5.8 and 12.1 per AdaptQA task, re- spectively, indicating greater complexity for AdapiQA. The environment setup, which employs a simulated user and validator, costs approximately $0.004 per IncreQA task and $0.007 per Adap- IQA task. The total cost of rumning the full benchmark over five runs, using 04-mini as the database agent, is approximately $95 across all task instances, whereas the cost using Gemini-2.5-FFaash is about $52.
Further Performance Analysis A deeper analysis of the top model, 04-mini, reveals key factors behind its performance inconsistency. As expected, Gap-k widens with more trials (k), since the condition for consistent success becomes stricter (Figure 3a). Furthermore, trials involving more user interactions tend to have lower overall success rates as well as higher inconsistency (Figure 3b). This trend holds even for individual trials, where a trial's success is negatively correlated with its relative number of user-agent interactions even within the same task (Figure Bc).
5.3 COMMON ERROR CASES IN INTERACTIVE DATABASE AGENTS
We conduct a detailed error analysis by manually inspecting failed trajectories from 04-mini and Gemini-2.5-Flash across both IncreQA and AdapiQA to understand the root causes of failures and the factors driving the performance gap. We categorize failures into two distinct groups: (1) Con- sistent Failures, where the agent fails across all five trials, indicating fundamental limitations; and (2) Inconsistent Failures, where the agent succeeds in at least one trial but not all, highlighting brittleness to variations in conversation paths.
8

---

Published as a conference paper at ICLR 2026
100
Performance
20
00.4
Gep
80
20
Performance (%6
0-11%
66-10%%
60
interaction longth by percentile
(b) @ass@5, Pass"5, and Gap-5 user interaction length.
Gap-k
Ress@ik
9-10%
11-69%
66- 100%
User interaction length by percentiee
(c) Success rate by user interaction length (IncreQA vs. AdaptQA).
(a) Performance vs. number of trials (k).
Figure Detailed performance of the 14-mini-powered agent.
Consistent Failures Consistent failures are dominated by difficulties in handling value mismatch and in generating complex SQL. The largest category, Value Linking Errors (51.9% arises when an agent fails to retrieve all relevant database terms. This includes overlooking drug brand names (e.g.. "Coumadin" for warfarin), failing to match textual variations (c.g., finding "essential (primary) hypertension"), and being unable resolve synonyms abbreviations (c.g.. "wbc" for "white blood cells" and "leukocytes"). The second major category. SQL Generation Errors (25.9%) involves subtle yet critical flaws query logic. For instance, agents often misinterpret the timing of events, querying patient's 's overall first hospital visit instead of their first visit for specific diagnosis. The remaining failures include Limited Clinical Knowledge (14.8%), such as failing to identify cardiovascular complications after surgical procedure, and Rule Violations (7.4%), where agents disregard explicit scope constraints like querying all admissions instead of only the current hospital stay.
Inconsistent Failures Inconsistent failures reveal the fragility of current agents. Small variations the dialogue trajectory, stemming from stochastic user responses, can lead to drastically different outcomes. The dominant category, SQL Generation Issues (75.0%), arises when slight variations in user phrasing disrupt the agent's context tracking, leading to SQL that omits crucial context from previous turns the same context handled correctly in successful trials. For instance, in an IncreQA task, a user might first ask for "patients with diabetes". In a successful trial, the follow-up "How many of those over 65?" correctly maintains both constraints. However, in failed trial, subtle rephrasing such as "And what about their age, specifically over 65?" can cause the agent to drop the initial "diabetes" constraint, erroneously querying the age of the entire patient population. The remaining eтrors include incomplete value retrieval (15.0%). limited clinical knowledge (5.0%), and occasional rule violations (5.0%).
Diagnostic Insights To address consistent failures, which are largely driven by value mismatch, future work should improve the agent's exploration strategies. Agents must comprehensively find relevant database entries. To mitigate the performance gap across multiple runs reflected in incon- sistent failures, the priority should be to improve context management. This requires developing techniques that enforce state-trad tratketracking consistency across linguistic variations, such explicit query state representation or specialized fine-tuning focused on context-dependent query refinement.
CHALLENGES IN SIMULATION-BASEE EVALUATION
Simulation-baar benchmarks present unique challenges due to the stochastic nature of LLM- generated dialogues. In our experiments, the common error types that cause simulation re-runs include: omitting conditions (41.8%), where users fail provide specific details before concluding

---

Published as a conference paper at ICLR 2026
the conversation; premature ending (19.5%), where users send the termination signal before reveal- ing all the necessary information specified in the instruction instruction deviation (15.6%), where users perform actions not specified in the instruction; conversation style violations (15,4%), such as using AI-like phrases or revealing all goals at once; and other miscellaneous errors (7.7%), such knowledge leakage incorrect value confirmation. Despite careful instruction design and nested perification-relecttin mechanism, many cost-effective LLMs, including open-source mod- els and Gemini-2.0-Fllash are not yet fully reliable at following detailed or unfamiliar instructions. In particular, correcting one error type through reflection can introduce different one, simulta- neously satisfying multiple constraints often exceeds the cerruuctioo-oolowinng capacity of current models. We expect this limitation to diminish as more powerful and cost-effective LLMs become available for use in user simulation.
CONCLUSION
We introduce EHR-ChaiQA: the first conversational benchmark for end-to-end evaluation of database agents in the safety-critical EHR domain. Moving beyond static text-to-SQI, EHR- ChatQA assesses an agent's ability to resolve query ambiguity and value mismatch through user conversation and active tool use. Our evaluation of state-of-the-arr LLMs two interaction flows, Incremental (IncreQA) and Adaptive (AdaptQA), reveals a critical robustness gap: the difference between succeeding in of five independent trials (Pass @5)) 5) and all five (Pass" 5) exceeds 38% on IncreQA and 36% on AdaptQA, reaching up to about 60%. This gap is mainly rooted in the agent's failures in accurate context management and SQL generation. We believe EHR-ChatQA can serve as a valuable resource for advancing database agents in interactive question answering over EHRs.
8 ETHICS STATEMENT
We adhere to the ICLR Code of Ethics and prioritize ethical considerations in this research. To pro- tect patient privacy, EHR-ChatQA utilizes publicly available, de-identified EHR datasets (MIMIC- IV-demo and eICU-demo), which are free of Protected Health Information (PHI), allowing for the safe evaluation of LLMs.. Our findings highlight significant performance gap in current state- of-the-art models between optimistic and consistent agent performances across QA tasks, raising concerns about the premature deployment of these agents safety-critical clinical environments. We also acknowledge that the source EHR databases may contain inherent biases reflecting the demographics and clinical practices of the originating U.S. hospitals.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our experiments, we provide comprehensive resources, including the complete EHR-ChatQA benchmark with all 366 task instances and the EHR databases, an eval- uation framework. tool suite, and simulation environment, which will be available in our GitHub repository. Detailed methodologies are provided in the appendices. covering database preprocessing and schema renaming (Appendix B.4). implementation details for the simulated user and validator (Appendix (.3)), and the implementation of the database agents (Appendix C). Due to the stochas- ticity of the simulated users, the exact numbers reported in the experiments may not be perfectly experime
reproduced, and occasional user-side errors may persist as these LLMs are not perfectly instruction- following. However, we have introduced a simulation validator to mitigate such issues, and we expect that their occurrence will decrease as LLM steerability continues to improve.
ACKNOWLEDGMENTS
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (ITPP) grants
RAS-2019-I110000 No. (0S-2005--0204967)) the Korea Health In- dustry Development Institute (KHIDI) grant No.RRS2025-00222 National Research Foundation of Korea (NRF) grant ( funded by the Korea government (MSIT, MOHW).
No.RS-2002-00223335 and
Thiis termination rule is intentionally included in user instructions to prevent open-ended questions from excessively deviating and leading to non-terminating conversations.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Pawet Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz-a large-seale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026, 2018
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.055: 033344, 2021.
Blanda Helena De Mello, Sandro José Rigo, Cristiano André Da Costa, Rodrigo da Rosa Righi, Bruna Donida, Marta Rosecler Bez, and Luana Carina Schunke. Semantic interoperability in health records standards: a systematic literature review. Health and technology, 12(2):255-272, 2022.
Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung- Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, Patrick Ng, and Zhiguo Wang. PRAC- TIQ: A practical conversational text-to-SQL. dataset with ambiguous and unanswerable queries. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.). Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (Volume I: Long Papers), pp. 255-273. Albuquerque, New Mex- ico, April 2025. Association for Computational Linguistics. ISBN 19198-9876--18666 URL 137]
John H Holmes, James Beinlich, Mary Boland, Kathryn Bowles, Yong Chen, Tessa Cook, George Demiris, Michael Draugelis, Laura Fluharty, Peter E Gabriel, et al. Why is the electronic health record so challenging for research and clinical care? Methods of information in medicine, 60(01/02))032-0488 2021.
Yixing Jiang. Kameron C Black, Gloria Geng, Danny Park, Andrew Y Ng, and Jonathan H Chen. Medagentbench: Dataset for benchmarking ilms agents in medical applications. arXiv preprint 1rX1v:2501.14654, 2025.
Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao. Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Gyubok Lee, Hyeonji Hwang, Seongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang, Min- joon Seo, Jong-Yeur Kim, and Edward Choi. Ehrsql: A practical text-to-sql benchmark for electronic health records. Advances Neural Information Processing Systems, 15:15599-1560, 2022.
Jinyang Li, Binyuan Hui, Ge Qu Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chen- Chuan Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can Ilm already serve as a database interface? big bench for large-scale database grounded text-to-sqls. In Advances in Neural Information Processing Systems, volume 36, 2023.
Xiao Liu, Hao Yu. Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In The Twelfth Inter- national Conference on Learning Representations.
Lucila Ohno-Machado. Realizing the full potential of electronic health records: the role of natural language processing. Journal of the American Medical Informatics Association, 18(5):539-533 2011.
11

---

Published as a conference paper at ICLR 2026
Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5(1):1-13, 2018.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceed- ings of the AAAI conference on artificial intelligence, volume 34, pp. 8689- 8099-866,,-00 2020.
Jaehee Ryu, Seonhee Cho, Gyubok Lee, and Edward Choi. Ehr-seqsql: A sequential text-to- sql dataset for interactively exploring electronic health records. In Findings of the Associa- tion for Computational Linguistics: ACL 2024, pp. 16388-164000 Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/w1/0244
http://////22..nnnnnnnaalll//// findings-acl.971. URL htw.:or//........nn.go...o----e. Findings-aci.977)
Irina Saparina and Mirella Lapata. Ambrosia: A benchmark for parsing ambiguous questions into database queries. Advances in Neural Information Processing Systems, 37:90600-90628, 2024,
Ivan Sekulić, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, André Ferreira Manso, and Roland Mathis. Reliable IIm-based user simulator for task-oriented dialogue systems, arXiv preprint FrxXiv:2402.133744 2024.
Jeonghoon Shim, Gyuhyeon Seo, Cheongsu Lim, and Yohan Jo. Tooldial: Multi-turn dialogue generation method for aug-eneteete language models. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: httt:::://operevvieee net. fttt//orummm d=
[IJ5eGJsKZ
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8652. 2023.
Bing Wang. Yan Gao, Zhoujun Li, and Jian GuanG Lou. Know what I don't know: Handling ambiguous and unknown questions for text-to-SQL. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 5701-5714, Toronto, Canada, July 2023. Association for Computational Linguis- findings-acl.3355 URL/https: batttpss//s..aatccoooooor///////
tics. doi: 10. 10.18653/v htt.://///.22.....nngna........
Findings-ard act.sacc.3555
Ping Wang, Tian Shi, and Chandan K Reddy. Text-to-sql generation for question answering on electronic medical records. In Proceedings of The Web Conference 2020, pp. 350-361, 2020.
XAI. Grok beta - the age of reasoning agents. https::/x..a//eew//////
February
2025. Accessed: 2025-09-20.
Ran Xu, Yuchen Zhuang, Yishan Zhong, Yue Yu, Xiangru Tang, Hang Wu, May D Wang, Peifeng Ruan, Donghan Yang, Tao Wang, al. Medagentgym: Training Ilm agents for code-based medi- cal reasoning at scale. arXiv preprint arXiv:2506. 04405, 2025.
Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, et al. A large language model for electronic health records. NPJ digital medicine, 5(1):194, 2022.
Shunyu Yao. Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744 35:20744-20757, 2022.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 7-bench: A benchmark for tool-agent-user interaction in real-world domains. In The Twelfth International Conference on Learning Representations, 2025.
Tao Yu. Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao. Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3911-3921, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/ v1/D18- 11DD1--125... URI httpt:tpp........... / //ac accl anthan ogy or.
12

---

Published as a conference paper at ICLR 2026
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chem Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga. Sungrok Shim, Tao Chen, Alexan- der Fabbri, Zifan Li. Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. CoSQL: A conversational text-to-SQI. challenge towards cross-domain natural language interfaces to databases. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1962-1979, Hong Kong, China, Novem- ber 2019a. Association for Computational Linguistics. doi: 10.18653/v1/D19-1 1204. URL http://awwaaooooooooooooooooeeae 1204/
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. SParC: Cross- domain semantic parsing in context. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedingy the 57th Annual Meeting of the Association for Computational Linguis- fics, pp. 1511--523, Florence, Italy, July 2019b. Association for Computational Linguistics. doi: 10.18653/v 1/P19- 1443. URL http://///aaaarttooooooooooo//// 1443/
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bencl and chatbot arena. Advances Neural Information Processing Systems, 36:46595-46622 2023.
13

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY CONTENTS
Benchmark Details
16
A.1
Sample Task Instances
16
A.1.I
IncreQA Sample
16
A.I. 1.2
AdaptQA Sample
16
A.2
Tool Specifications
18
A.3
User Implementation
19
A.3.J
Simulated User
19
A.3.2
User System Prompt
19
A.3.3
Example Variability in Simulated User Turns
20
A.3.4
Nested
Mechanism
A.4
Simulation Validator Implementation
22
A.4.1
Validator Reliability
22
A.5
Database-Specific Rules
24
A.5.I
MIMIC-IV* Rules
24
A.5.2
eICUx Rules
25
6
Evaluation Details
26
A.6.1
IncreQA
26
A.6.2 AdaptQA
26
A.7
Contributor Demographics and Beta Testing Process
26
Annotation Details
27
B.1
IncreQA and AdaptQA Annotation
27
B.1.1 IncreQA Annotation
27
................................
B.1.2 AdaptQA Annotation
27
B.2
AdapiQA Categories
28
SQL Annotation Style
29
B.4
EHR Database Preprocessing
30
B.4.1
MIMIC-IV Renaming
30
B.4.2
eICU Renaming
31
Agent Implementation
32
Agent System Prompt
32
Agent Rules
32
Agent Rules for Evaluation
33
D Sample Trajectories
34
D.I Incremental Query Refinement
34
D.1.1 Successful Case (IncreQA)
34
14

---

Published as a conference paper at ICLR 2026
D.1.2 Failed Case (IncreQA)
38
D.2 Adaptive Query Refinement
44
D.2.1
Successful Case (AdapIQA)
44
D.2.2
Failed Case (AdapiQA)
49
D.3
Invalidated Trajectories by the Validator
59
D.3.I
Violation of No Final Check Rule
59
D.3.2
Violation of Missing conditions Rule
59
D.3.3
Violation of Performing Agent- like Tasks Rule
60
15

---

Published as a conference paper at ICLR 2026
A BENCHMARK DETAILS
A.1 SAMPLE TASK INSTANCES
In this section, we present sample task instances for IncreQA and AdaptQA.
A.1.1 INCREQA SAMPLE
"task id": "6",
"task type": "incre",
"db_id": "mimic_iv.star"
"instruction":
"Your goal is find the number of patients admitted the hospital who meet specific criteria. Specifically, you want know how many patients admitted in the past 90 days have family history of breast cancer.", "gold sql":
"SELECT COUNT(DISTINCT ha.patientid) AS patient_count FROM hospitaladmissions ha JOIN admissiondiagnoses ON ha.admissionid ad.admissionid JOIN diagnosiscodes ON ad.icdcode dc.icdcode AND ad.codeversion dc.
codeversion WHERE dc.description 'family history of malignant neoplasm of breast' AND daterimelhaa.dmmiddareemm)) >= datetime((2100-122- 23:59:00', days')",
"gold [[1]]
A.1.2 ADAPTQA SAMPLE
"task_id": "12",
"task type": "adapt",
"db id": "mimic iv star",
"instruction":
"Your goal is explore patient data related medication prescriptions. Specifically, you are interested in patient ID 10008287 and want to know when they were prescribed carbamazepine for epilepsy treatment. If the patient was prescribed carbamazepine, you want to know the most recent prescription time after the first epilepsy diagnosis. the patient was not prescribed carbamazepine, you want to identify whether any other class of medication used for epilepsy treatment was prescribed. If such medications are found, you want know the most recent prescription time after the first epilepsy diagnosis. If such medications are found, end the conversation. You want the answer in the exact time format recorded in the database.",
16

---

Published as a conference paper at ICLR 2026
"gold sql":
"WITH valid_epilepsy.codee AS SELECT DISTINCT d.icdcode, d.codeversion, di. description FROM admissiondiagnoses d JOIN diagnosiscodes di ON d.icdcode = di. icdcode AND d.codeversion di.codeversion WHERE di.description LIKE "%epilepsy epilepsy diagnoses AS SELECT d.patientid, MIN/d.recordeddateettmm) AS firstepilepsyytiimg FROM admissiondiaggnosss d JOIN valid valideepilepsyyccodes ON d. icdcode c.icdcode AND
WHERE d.patientid
10008287 GROUP BY d.patientid ), http://mmaaapiieeeperriiiiiiiiie AS SELECT patientid, p.admissionid, p.startdatetime FROM medicationorders p JOIN dillepsy.diagoses ed ON p.patientid ed.patientid WHERE p.medicationname=
carbamazepine' AND p.startdatetime ed.first epilepsy time )
allermaiwe..pplleyyyppesrrrproos AS SELECT p.patíentid, p.admissionid,
startdatetime FROM medicationorderss JOIN epilepsy ed ON p.patientid
edd.patientid WHERE p.medicationname IN ('levetiaccetam; 'valproate',"
topiramate') AND p.startdatetime > ed.first.epilepsy.tte time
combined,.resccritionnsss SELECT patientid, admissionid, startdatetime FROM
carbamaazeinneepreccriiioons WHERE EXISTS (SELECT FROM
carbamazepine_prescritttoott UNION
UNION ALL SELECT patientid, admissionid,
startdatetime FROM alteemmtttivveeppllpppyeeeerooss WHERE NOT EXISTS (SELECT
SELECT FROM carbamazepine ) SELECT startdatetime FROM
uumbined__pescriptiin ORDER BY startdatetime DESC LIMIT 1;",
"gold_answer": [[2100-10-12 20:000000000
17

---

Published as a conference paper at ICLR 2026
A.2 TOOL SPECIFICATIONS
Table 4 presents the six default tools, categorized by their purposes. These tools serve as channels for access database content and external clinical knowledge to solve question-answering tasks in EHR-ChatQA, For each tool, the equals sign (=) denotes its default arguments.
Table 4: Definition of six default tools provided in EHR- CHR-ChatQA.
Tool name
Input
Output
Description
Schemaa exploration
tabl eusearch
None
List of tables
Lists all available tables in the datahase.
col volumnssearch
Table name
Columa namnss with sample
Shows the columns of specified table along with sample data.
Value exploration
value.subata wuwatring.ae.. searchh
Table column
k values containing the substring
Finds values that contain the specified substring in the given column.
name, value, k=100
simi iimilarity... search
name, column
k similar values
Finds values similar input value based semantic similarity""
name, value, k=100
External knowledge retrieual
xab_search
Keyword
Web search results
Retrieves relevant external clinical knowledge from the web.
SOL execution
sql.execuce
SQL query, k=100
SQL result
Executes the provided SQL query and returs
up
we use FAISS wrapped in the LangChain library. Text columns frequently used in QA tasks are pre-indexed, as listed below (table names with their corre- sponding column names in brackets).
For
For MIMIC-IV*, the pre-indexed columns include:
hospitaladmiissons: [admissiontype,, admitsource, dischargedestination]
diagnosiscodes: [description]
procedurecodes: [description]
medicationorders: [medicationname]
clinicalitemtypes: [itemname]
labtesttypes: [itemname]
microbiologyresllt::: [specimentype, testname, organismname]
For eICU*, the pre-indexed columns include:
allergy reretrieaciinn [drug_nand] name, allergy_nann
condition: [condition_nanl _nmme]
fluid balance: [fluid label]
lab: [lab_name] name]
prescription: [drug_nad [arue
icupatient: [ethnicity, hospital aalaaaamiionssuuure
treatment: (treameet_nname
18

---

Published as a conference paper at ICLR 2026
A.3
USER IMPLEMENTATION
A.3.1
SIMULATED USER
For the implementation of simulated users, use Gemini-2.00Flaan due to its low cost and effective cornucttoo-followiin capabilities. To simulate user stochasticity in natural language utterances, the temperature of the user simulator is set to 1.0. Table 3 shows the system prompt for the user simulator.
A.3.2 USER SYSTEM PROMPT
Table 5: User system prompt teach behavioral rules.
Your task is to simulate a user with no knowledge SQL or database management systems, who needs specific information from an EHR database and relies on the D8 agent for help.
Instruction: (user_insee inser_instruction)
Rules:
The current time is 2100-12-31 23:59:00.
Start with short, broad question that reflects the overall goals from the instruction.
Use your own words to describe your goals for the DB agent. Do not reveal all your goals at once. Instead, share them gradually, one two sentences at time.
Speak casually and directly, without functionally unnecessary phrases (like "please" or "thank you") that make the tone sound like an AI assistant.
Do not generate SOL, code snippets, empty messages, assistant-Like outputs. Stay in the role of user, If the DB agent requests speci fic tables or column names, instruct it to locate them. independently (unless the instruction says otherwise).
AI-
DB agent.
If the DE agent requests writing reviewing SQL queries, summarizing the everall goal, instruct it complete the task independently.
If the DB agent gives an intermediate answer, don't complete it yourself. Instead, instruct it to finalize it (e.g., performing calculations like time differences If the DB agent's answer seems satisfactory (even though you do not know whether it is correct or whether the requested data actually exists), ask the D8 agent to double check that their final answer covers all goals raised. If not, request any missing parts. If the DE agent's answer covers all goals raised, generate only "##+END#** to end the conversation. Do not add it after every reply Use it only once in the final message.
rephrasing answers).
Do not deviate from what is specified in the instruction, such as failing to mentien the top ranked tied results to retrieve, requesting medication order records or prescription records instead admini records, requesting months of data instead 180 days, asking follow-up questions when they are not specified in the instruction, er revealing disalloved information before the DB agent mentions it.
19

---

Published as a conference paper at ICLR 2026
A.3.3 EXAMPLE VARIABILITY IN SIMULATED USER TURNS
Using the annotated user instructions and the user system prompt in Table 5 the LLM-powered user generates diverse dialogue trajectories with the agent even under the same instruction. Figure shows sample variability in user turns for both IncreQA and AdaptQA.
62
23
40
flrence turn acroes mak (morigN)
Masemin diffeennce counts across train (AdiptOOA)
(a) IncreQA
(b) AdaptQA
Figure 4: Distribution of the difference between the maximum and minimum number of user turns for each unique task in IncreQA and AdaptQA. A unique task is defined by db_id, task_type, and task_id. Most tasks have a difference of to turns, which indicates that the user simulator generates stochastic conversations rather than fixed scripts.
20

---

Published as a conference paper at ICLR 2026
A.3.4 NESTED VERIFICATION- REFLECTION
ERRRECATION-REELECTION MECHANISM
To improve the reliability of the simulated user, we employ two-level nested mechanism that operates at both the turn level and the conversation level. After the user LLM generates candi- date response for each turn, an inline verifier-a a separate LLM call at temperature 0.0 evaluates whether the response adheres to the given instruction and behavioral rules. If the verifier rejects the response, a chained reflection loop triggered: reflection L.M generates a revised response along with an explanation of the identified issue, and the verifier re-evaluates the revision. This process repeats iteratively, with each iteration receiving the previous iteration's output as input, enabling progressive refinement. When the simulation validator (Appendix A..4) detects a user error after the conversation concludes, its explanation is appended to an accumulated error list and injected into both the verifier and reflection prompts as "previous user mistakes" in subsequent re-runs.
21

---

Published as a conference paper at ICLR 2026
A.4 SIMULATION VALIDATOR IMPLEMENTATION
Tables 7 the system and input prompts used by the simulation validator. the validator outputs "user_error" instead of "no_error," the conversation is flagged and the simulation is re-run.
A.4.1 VALIDATOR RELIABILITY
To assess the reliability of the simulation validator, manually inspected 100 randomly sampled trajectories. The inspection identified two types of validator errors:
False rejection )):: Valid behaviors were incorrectly flagged. This primarily occurs due to information asymmetry: while the user does not see the ground truth SQL, the validator references check whether the user has omitted details required for accurate SQL generation. For instance, a natural user paraphrasing (e.g., asking for "hypertension" in general) may be flagged as a deviation if the ground truth SQL requires specific ICD codes. These false rejections only trigger trajectory regeneration and do not affect agent performance evaluation.
False acceptance (9%): Minor deviations were missed by the validator. Specifically, 7% involved behavioral issues (e.g., skipping double-check before ending the conversation) and 2% involved edge cases in task instructions not explicitly covered by database specific rules. These missed errors did not involve critical failures that would compromise task validity.
Table 6: System prompt for the simulation validator.
Your task is to determine whether [USER] accurately followed the provided rules and user instruction during their conversation with DB AGENT] Errors are defined any deviations from the rules or user instruction. You must carefully review the rules, user instruction, conversation between [USER] and [DB AGENT), and the gold SQL query to identify any errors made by [USER]
Table 7: Input prompt for the simulation validator.
[user sysee_system_oom p
Conversation:
|conversation)
Gold SQl:: Igold_sq] sql)
Types of common user errors:
The user gives away their goals all at once in the same turn. The user acts like a DB agent or AI assistant instead of the user (e.g., writing, reviewing, or executing SOL queries, calling external APIs, or responding to the DB agent in a machine assistant way)
The asks for information that is slightly different from what is specified in the instruction (e.g., requesting nedication order records prescription records instead of administered records, requesting 6 months of data instead of 180 days).
The user confirms values that differ from those the gold sQL, unless specified otherwise for just "diabetes" when the gold SQL uses LIKE "$dlabetes*").
the instruction (e.g., requesting data
The user menticns information beyond the instruction, including related or unrelated details not specified (e.g., asking follow-up questions not in the instruction).
The user does not provide all the detailed conditions specified in the instructicn before ending the conversation. These conditions may include, for example, retrieving all tied ranked results, specifying the top "N" resuits to retrieve, handling duplicate patient records, or indicating keywords to include or exclude when
22

---

Published as a conference paper at ICLR 2026
searching for data. However, if the D2 agent retrieves no relevant data, these conditions are not required.
The user does not provide all the detailed conditions specified in the predicates of the gold sQI, either explicitly or implicitly, before ending the conversation.
The user does not double-check with the DE agent to see if the agent's s final answer satisfies all the information the user provided before ending the conversation.
The user viclates any other rules speci fied in the rules or the user instruction.
You must respond in JSON format with the following fields: explanation: Provide a clear and concise explanation of why you made the decisicn.
broken_rules If a user error is found, provide the exact rule or instruction that the user violated. If no error is found, provide an empty string.
evidence: If a user error is found, provide the exact user response that caused the error. If no error is found, provide an empty string.
result: Answer "user_error" if a user error is found. Anawer no_error" if no user error is found.
23

---

Published as a conference paper at ICLR 2026
A.5 DATABASE-SPECIER RULES
Database-specific rules guide the agent in referencing database contents and generating SQL queries. These rules complement the general agent rules (Section C.Ip. covering time-related operations, database-specific schema structures (e.g., the hierarchy linking patient records to admission records, and admission records to ICU records), and hints for complex numerical operations, such as survival rate calculations. The complete database-specifie rules are detailed in Sections A.5.1 and A.5.2
A.5.1 MIMIC-IV* RULES
Table 8: SQL assumptions for MIMIC-IV*
Below are the SQL generation rules:
Use SQLite for SQL query generation.
The current time is 12100-12-31 23:59:00'. When referring time, do not use SQLite's native functions like now. Instead, use 22100-12-31 23:59:00' for 'now", '2100-12-31' for 'today', '2100-12' for this month', and '2100' for 'this year'.
Use DENSE_RANK RANK() for questions involving ranked results (e.g., the most or the top rost common/Frequen events) to retrieve values from the specifies fied column (e.g., diagnosis names). exclude counts or ranks unless the user explicitly requests them. Do not use DENSE RANK () for questions without ranked result requests. For cost-related questions, the event type (' ('adrisssooniianooses'' 'adhilssooppooredrrer'',, labresults', 'medicationorders when specifically retrieving costs for diagnoses, procedures, lab results, or medications, respectively
costrecords eveenttyee
specify
When retrieving costs for diagnoses, join costrecords.cost. with admissi admmissioonroceddrs..e.oo.m
For procedures, join with For lab results, join with labresults. recordid. For medications, join with medicationorderzs www.dddessrecoridd... recordid.
The medicationerden table stores ordered or prescribed medications, while the intakerecords table records administered drugs or fluids
when asked to retrieve procedures, diagnoses, or lab tests, return their names instead of their codes.
All values stored in the database are in lovercase.
days ago, use datetime 2'100-22-11 23:59:00', --N days') ), instead of 2A100'2100-1--3 23:59:00', '-N days') when handling "within N days/hours, include the boundarics inclusively.
When calculating
For questions involving the timing of diagnoses conditions relative to other events, you must the first diagnosis time for each patient unless directed otherwise.
when searching for specific medi cation names in the database, use pattern like smorphinet instead of exact matches like morphine unless directed ctherwise.
As clinical and lab events often share identical names but have different codes (e.g., codes 50902 and 52535 both represent chloride ,, use the names if grouping them in SQL.
If the results contain numerical values (e.g., tine differences in days or hours, or survival rates), round them to four decimal places.
24

---

Published as a conference paper at ICLR 2026
A.5.2 EICU* RULES
Table 9: SQL assumptions for elCUx.
Below are the SQL generation rules:
Use SQLite for SoL query generation.
The current time is 22100-12-31 23:59:00'. When referring to time, do not use SQLite's native functions like now. Instead, use 22100-12-31 23:59:00' for 'now', '2100-12-31' for 'today', '2100-12' for 'this month', and '2100' for 'this year'.
Use DENSE RANK() for questions involving ranked results (e.g,, the most or the top N rost common/frequent events) to retrieve values from the speci fied column (e.g., diagnosis names) exclude counts or ranks unless the user explicitly requests them.
DENSE (ANK() for questions without ranked result requests.
The patient identifiers patientjid, id, hosp_ ii, and unit_ia represent the unique patient ID, hospital admission ID, and ICU admission ID, respectively. The hierarchy of them is patient hospital icu. when retricving specific hospital or ICU admission records, use their admission IDs rather than admission or discharge times.
For cost-related questions, use cost cost.event_typ: to specify the event type ('cendition', 'treatment', 'lab', 'prescription') when specifically retrieving costs for conditions, treatnents, lab results, or prescriptions, respectively. For example, when retrieving costs for conditions, join cost.event_ie id with condition. condition id with event eren__type-a = 'condition'.
Use fluid balance for both input and output events. Specity input events using fluid_balance. fluid_path events using fluid_balance, fluid_path LIKE 'Soutput''.
balanee
" and output
The prescripticn table stores ordered or prescribed medications,
while the Bluid_balance table records administered drugs fluids when fluid www.ballaaaaaaeaa..oo.eeeee. fluid path LIKE "Sintake8'.
All values stored the database are in lovercase.
Patients with no records of death are considered to have survived when dealing with dcath-related questions.
When calculating days ago, use datetime (22100-12-31 23:59:00', days'), instead of DATR((2100-12-3) 23:59:00', *-N days') When handling "within days/hours, include the boundaries inclusively.
For questions involving the timing of diagnoses conditions relative to other events, you must the first diagnosis time for each patient unless directed otherwise.
when searching for specific medication names in the database, use pattern like Smorphineš instead of exact matches like morphine unless directed ctherwise.
If the results contain numerical values (e.g., tine differences in days or hours, or survival rates), round them to four decimal places.
25

---

Published as a conference paper at ICLR 2026
A.6 EVALUATION DETAILS
The evaluation methods for IncreQA and AdaptQA differ, and the appropriate method is applied based on the interaction flow type.
A.6.1 INCREQA
For IncreQA tasks, evaluate the agent by executing the SQL query it generates (parsed from the input to the sql execute tool). We compare the resulting output to the ground truth (GT) SQL output to ensure accuracy even for queries that return many rows. Up 100 results are checked. Evaluation logic: The GT SQL is constructed to satisfy all constraints introduced throughout the interaction. Example: If user first asks for "patients with diabetes" and then refines it to "among them, who are over 65?", the GT SQL filters for both conditions. If the agent executes a query checking only for "age 65" (ignoring the diabetes context), the result will not match the GT, leading to failure.
A.6.2 ADAPTQA
For AdaptQA tasks, correctness is evaluated based on the content within the (enswr><</nssee> tags of the agent's response the user. This method better suited for tasks that require clinical reasoning beyond simple SQL retrieval. For instance, after retrieving medication data, the agent might need to identify drugs within the same class or those with similar purposes but different mechanisms. For tasks requiring numerical answers (e.g., patient counts), the instructions explicitly ask the agent respond in words rather than numerals (e.g.. "ten" instead of "10"). This avoids false positives from numeral-word mismatches during evaluation. Evaluation logie: Since the simulator follows a fixed procedure, the correct answer is deterministic. We evaluate success by checking if the content within the <answer> tags strictly matches the annotated GT answer, which reflects the user's ' final intent after all refinements. Example: Consider asking for "AAspirin" (Count: 0) and then refining to "acetylsalicylic acid" (Count: 10). The GT answer is explicitly set to "ten". If the agent answers <answer>ten /answer>, succeeds. If it fails update the search term and answers "zero", it fails.
A.7 CONTRIBUTOR DEMOGRAPHICS AND BETA TESTING PROCESS
To ensure the robustness and usability of the EHR- CHR-ChatQA benchmark, we conducted a beta-testing phase involving 38 graduate- contributors.
Demographics and Background The cohort consisted of Master's and Ph.D. students specializ- ing in Computer Science and Artificial Intelligence. All contributors were participants in a graduate- level course focused Healthcare AI. Their strong technical background ensured that they ef- fectively represented the target audience of this benchmark, namely researchers and developers of database agents.
Training and Setup Contributors were not trained to act as simulated users but were instead onboarded as "agent developers." To facilitate this, we provided them with standardized devel- opment environment, which included: (I) the complete benchmark codebase and the simulation environment, (2) detailed documentation defining the task objectives and evaluation metrics, and (3) a baseline agent implementation to serve as a reference point for tool usage and API interaction.
Task and Feedback Mechanism Over a duration of two months, contributors were tasked with developing and optimizing their own database agents solve the benchmark tasks. During this process, they were instructed to report qualitative failures in the simulation environment via Google Form. Specifically, they identified edge cases where the LLM-based user simulator deviated from intended behaviors, such as revealing constraints too early or terminating conversations prematurely.
This human-in-the-loop testing phase was critical for quality assurance. Based on the reported issues, we iteratively refined the user instructions and the simulation validator to better detect and filter out invalid dialogue traces.
Outcome
26

---

Published as a conference paper at ICLR 2026
B ANNOTATION DETAILS
B.1 INCREQA AND ADAPTQA ANNOTATION
Each task instance (examples in Appendix A.I consists of an "instruction," "gold_sql." "gold answer," and metadata (c.g., task id, db id). Below, we detail the construction pipeline for both interaction flows.
B.1.1 INCREQA ANNOTATION
Source Selection IncreQA tasks are designed to mimic a user who incrementally adds constraints through multi-turn interactions. We curate and adapt clinically relevant queries from two primary sources: the EHRSQL dataset (Lee et al. (2022), originally collected from over 200 hospital profes- sionals, and of question-SQL. pairs internally stored our collaborating medical center.
SQL/Answer Annotation After collecting set of candidate queries, we perform value sampling ensure the queries return non-empty results our specific EHR databases (MIMIC-IV and eICU Demo). For query like "patients diagnosed with [Diagnosis] during [TTmme Range]," selecting ran- dom values often yields zero results due to data sparsity. We manually explore the database to sample valid value pairs (e.g., specific diagnosis codes and overlapping admission dates). We then xAI's Grok (xAI 2025) to generate initial SQL drafts, explicitly prioritizing Common Table Expressions (CTEs) for readability and maintainability (see Appendix [B.3). Finally, we manually verify that the SQL execution produces the intended result based on the corresponding natural language question. The final SQL and its execution result become "gold_sq) and "gold -osweer"ser."
Instruction Creation Once the SQL queries are finalized, we create user instructions through a SQL-to-text approach. This method is crucial for preserving fine- fine-grained SQL conditions within the narrative instructions. By decomposing complex SQL. logic into discrete semantic components, we ensure that every constraint corresponds to a specific part of the user's goal. For example, SQL query containing ORDER BY chartting time DESC LIMIT 1 encodes strict operational constraint: fetching only the latest record, A natural language question often glosses over this detail (e.g., "What is the patient's creatinine level?"), which could be interpreted requesting average, a full history, or the peak value. However, our approach captures this SQL operation explicitly the narrative: "You want to check the creatinine level. Specifically, you are interested only in the most recent value," reducing errors in omitting details required for the agent to generate correct SQL. We also introduce value mismatches by manually replacing exact database values with lay terms (e.g., mapping "malignant neoplasm of breast" to "breast cancer").
B.1.2 ADAPTQA ANNOTATION
Source Selection AdapiQA tasks are designed around conversational flows that mimic clinical interactions requiring query goal adjustments. To facilitate this, we define eight categories of query modification, including interaction flows such resolving brand-generic name mappings (e.g.. brand < generic) and switching a primary lab test to its clinical alternative for a condition (e.g.. hemoglobin -> hematocrit). (see Appendix B.2 for the full list of categories).
Instruction Creation Unlike IncreQA, the annotation process for AdaptQA begins with instruc- tion creation, followed by SQL/Answer annotation. This reverse order is necessary because Adap- IQA scenarios are highly dependent on the patient data stored in the database and the database schema. Using each of the defined categories, we prompt Grok generate various scenarios that fit under the corresponding category. For example, for "brand-genee -eneeric name resolution," the prompt generates instructions such as: "First search for Lasix. If not found, pivot to search for its generic name (Furosemide)." We then manually verify these scenarios against the actual database to find patient records where the condition holds true (e.g., patient who has no record for "Lasix" but does have record for "Furosemide") ) A physician verifies the clinical validity of these goal adjustments. resolution strategies than IncreQA,
By design, these tasks require more advanced often requiring domain knowledge beyond simple synonym matching.
27

---

Published as a conference paper at ICLR 2026
SQL/Answer Annotation Since AdapiQA instructions imply a conditional step-by-step flow, the order of operations is critical. We annotate the SQL queries to reflect how a user proceeds through the logical pivot. For logic based on data existence, we utilize conditional expressions in the GT SQL such as EXISTS (SELECT defined. However, fully capturing the pivot logic in single statement not always feasible. For patterns involving schema unavailability (for example, asking for table that does not exist) or complex clinical reasoning that can only be inferred from the retrieved tool output, we bypass the intermediate logical pivoting and annotate the SQL straightforwardly so that it corresponds to the ground truth answer.
FROM lasix x pat i ent.s) assuming that specific CTEs
B.2 ADAPTQA CATEGORIES
Table 10: Eight categories of query goal modification in AdaptQA.
Categury
Description
Brand-generie name resolution
The user queries medication by its brand name (e.g., (e.g..Lipitoc) If the database lacks records under that the agent must ideatify and search for generic equivalout stoevastatin), or vice versa. This tests the agent's 's ability to resolve drug name allases without explicit user puidance.
Within class drug adaptation
When a specific drug is unavailable, the user adapts the search within the same therapeutic class. This involve expanding fimm a single drug the entire class (eg. flaoxetine -> all SSR1s), narrowing from a broad class subclass, excluding a particular medication from class-level search.
Cross-class drug subslitution
When the entire initial drug class yickds no results, the user's query broadens to different thorapeutic class for the same condition. For instance, if no statin was preserihed for hyperlipidemia management, the search expands found for hyperteasion, the search breadeus to other untihyyertensive classes.
lipid-lowering drug classes (eg-. fibrates), if ACE inhihitor was
Clinically related condition lookup
Starting from a primary diagnasis, procedure, or medicatine, the user pivots to identifying clinically lated conditions when the initial query yickds results, Examples include scarching for post-surgical complications (egg, partial hepatectomy - pulmonury embolism). cardionscull lar comorbidiries of autaimmane diseases (e.g., psoriasis myncandial infarction), or known side effects of prescribed medication.
Alterativve lah test for condirion
When a specific lab test not found in the patient's is recends, the agent must pivat to elinically equivalent alsemativve used assess the same condition (c.g.. troponin 1+ troponin T for myocardial damage). This mirrors clinical reasoning where a preferred diagnostic test may be unavailabic.
Alternatvve procedure for treaiment
When the primary procedure or treatment is not recorded for patient, the user directs the search loward an alternative clinical procedure for the same condition. The agent must recognize clinically appropriste substitutes without explicit guidanes from the user.
Compound criteria with fallback
The user comstructs queries integrating multiple clinical events diagneses, results, and vital combiied via logical operators (ANDIOR) to identify spocilic paticat cohort. When par- criteria are not met, query requires selective fallback specific conditions while maintaining others, testing the agent's ability manage complex inclusion and exclusion logic.
Schema fullhack handling
The user's 's query targets specific datahase table that does not exist in the schemn (e dedicated cuncer registry table). agent must recognize the table's absence fall back to altemative data source (e.g.. the general diagnosis table) to fultill the user's 's intent.
28

---

Published as a conference paper at ICLR 2026
B.3 SQL ANNOTATION STYLE
SQL annotations in EHR-ChaiQA use Common Table Expressions (CTEs) to enhance readability and maintainability. A sample user instruction and its corresponding gold SQL query are shown in Table IT
Table 11: Sample user instruction and its GT SQL query.
User Instruction
Gold SQL.
WITH LastAdmission AS SELECT admissionid
FROM hospitalladmissions. WHRRE patlentíd 10108445 ORDER BY admitdatet ime DESC LIMIT 1
Your goal to find information related to specific patient's lab tests. Specifically, you are interested patient ID 10018845 and want to know all timestamps when the Hb value was 8 or lower during the patient's last hospital visit. You want to search for Hb specif- ically, not other similar lab tests like "Hb C" or "Hb A2." When querying the DB agent, since you do not know how Hb is stored, use common terms like "Hb" "Hgb" when referring to it, and let the DB agent find it for you.
HbTest AS (
SELECT itemcode FROM labtesttypes WHERE itenname - 'hemoglobin'
SELECT 1r. dr.resultdatettmee FROM labresults 1 JOTN AdstAdnssionn 1a ON 1r addimsssionns = Ia.admissionn JOIN HbTest ht
ON Ir.itemcode = Itt. itemcode
WHERE 1r. pa.pntientid = 10019845
AND 1r. resultvalue <=
AND 1r. resultvalue IS NOT NULL
ORDER BY tr.resul=datetimme
29

---

Published as a conference paper at ICLR 2026
B.4 EHR DATABASE PREPROCESSING
Our preliminary analysis revealed that many LLMs memorize the original MIMIC-IV and eICu schemas, which leads to SQL generation without actual schema exploration. To prevent this, rename the schema so that generating SQL without using the provided schema tools inevitably results in errors. The detailed schema mappings are provided in Table and Table
B.4.1 MIMIC-IV RENAMING
Table 12: Table and column renaming mappings for MIMIC-IV.
Original Table
Mapped Table
Column Mappings (MIMIC-IV to MIMIC-IV-
patients
demographics
row - recordid. subject.id + patientid, gender + gender, dub dateof- hirth, dod datenkleath
admissions
hospitaladmissios
rowidd recordid, subject.id + patientid, ladm id adnissionid, admit- time admitdatetime, dischtime - dischurgedatetime, admission_type - admissiontype, admission location admitsource, discharge.location dis- chargedesinatioon, insurance - insurancetype, language language, mari- tal stl_sta - maritalstatus, age + age
diagnoses
diagnosiscodes
id recordid, ied - description
iadende, ied - codeversion, long title
ied procedures
procedurecodes
row id - recordid, ied code - iedende, ied version - codeversion, long title - description
labitems items
labtesttypes
recordid,
itemoode, label itemname
clinicaliemttyps
id recordid, itemid itemeode, label itemname, abbreviation abbreviation, linksto - itemtype
diagnoses ied
admissiondaaguosss
row id + recordid, subject id patientid, hadm id - admissionid, ied code - iedeode, version codeversion, charttime - recordedaateimme
procedures
admissionpoocdures
id + recordid, subject patientid, hadm - iedcode, ied.version codeversion, charttime recordddanetimee
admissionid, ied code
labevents
labresults
row id recordid, subject id patientid. hadm id + admissionid, itemid - itemcode, charttime resultdatetime, valuenum + resultvalue, valueuom resultunit
prescriptions
medicationorders
rowid + recordid, subject patientid, hadm + admissionid, start- time + startdatetime. stoptime + eaddatetime, drug + medicationnane, dose_val_rs dosevalue, dose_unit_r doseunit, tienroute
administra-
cost
costrecords
admissionid, event_type eventtype, eventid costid, chargetime costdatetime, cost costa- mount
row recordid, subject - patientid, hadm_d +
chartevents
clinicalevents
row id recordid, subjectid patientid. hadm - admissionid, stay _id icuadmissionid, itemid itemcade, charttime recondeddatetimee, valuenum - value, valueuom - unit
inputevents
intakerecords
row jd - recordid, subject_id patientid, hadm_id - admissionid, stay jd ieundmissionid, starttime standatetime, itemid - itemcode, totalamount totalvalume, totalamountuom volumeunit
outputevents
outputrecords
row jd recordid, subject.id patientid, hadm_id .id - admissionid, stay iid icuadmissionid, charttime + reconleddattttime, itemid + itemcode, value + volume, valueuom volumeunit
microhiologyeevens
micmsiiologyresllts
id recordid, suhject id patientid, hadm id admissionid, chartrime - collecteddaterime,, spoc_rye - specimentype. test_name - testname, org name organismname
icustays
icucpisodes
recordid, subject patientid, hadm .d admissionid, stay icuadmissionid, first careunit - initialgareunit, last careunit finalcareunit, intime admirdatetime, outtime - dischargedaterime
transfers
patientrransfers
recordid, subject -2 patientid, hadm - admissionid, + transferid, eventtype - trunsfertype, careunit - careunit, intime trans- ferindatetime, outtime + trasferoutdatetime
30

---

Published as a conference paper at ICLR 2026
B.4.2 EICU RENAMING
Table 13: Table and column renaming mappings for eICU.
Original Table
Mapped Table
Column Mappings (eICU to elCU+)
patient
icupatient
hosp. id, patientunil- stayid unit.i, gender gender, age - age, ethnicity ethnicity, hospi- talid - bospital id, wardid - ward id, admissoonheihht + height admission. admissionweight + weight.admissonn dischargeweight + wischa..ischarge.. hospitaladmitimee hospital admit time. hospialadmiisouree + hospi- tal Mll_dmmissionn source, unitadmitimme + admi time, uniidischargetime + unit.dischuree.iing bospinadischareeime taldischargestttuss + hospital discharge sttttu
uniquepid patient id, pateenteaallaystenstayyi
hospital.dischare..e hospi-
diagnosis
condition
diagnosisid - condition patientunisstayid + unitid. diagnosisname condition name, diagmosisimme comdition time, ie19code icd9 code
trealmentid ueatmentid, patientunilstayid unitid. treatment name, treatmentlmme - treatment
lab
lab
labid + lab id. patientunilslayid ++ uniLid, labuame + name labresult - lah result, labresultiime + lab ressuelssiimsss
medication
prescription
medicationid prescripion.id. patientunissaaid unitid, drugname drug.name, dosage + dasage, routeadmin - administratinn.runt drugstart time - medication.statt.u drugstoptime + medication medication.stop.iin stop
cost
enst_id. paiienheellhsystemttaddd - hesp.d, patientunitstayid unit.id, eventlype event.lype, eventid eventid. chargetime cost.time, cost - cost_amount
allleryy
allergy.nen reaction
allergyid + allergy.id .id, patientuniltaaid + unit.id, drugname + drug name. allergyname - allergy name, allergytime allergy_ime
intakcoutput
fluid balance
intakcoutpurid fluid balance id, patientunisstayid unit cellpath fluid path, celllabel - fluid label, celivaluenmerric - fhiid value numeric, in- takcourputtmme fluid sltid_balance.fim
microlab
miembiology
microlabid microhiolngy id, patientunistaaiid unit id, culturestte cul- ture site, organism - organism, culturtakentime - culture.taken time
vitalperiodic
vital signs
vitalperiodicid - vital. vital_sign id, patientunisstayid - unit id, temperature + temperature. sao2 s902. heartrate tion rate, systemissytolic - systolic hp, systemicdiatollic + diastolic hp. systemiemean - mean.bp. observatoontine - vital time
respiration respira-
hospital
hospital
hospitalid - hospital numbedscategory ingstatus teaching status, region region
capacity category, teach-
31

---

Published as a conference paper at ICLR 2026
C AGENT IMPLEMENTATION
For selecting the backbone LLMs for the agents, small models (c.g., 7B or 13B), DeepSeek-RI, and the Gemma series are excluded due their limited performance in tool invocation tasks. Owing to budget constraints, we also exclude Anthropic models (e.g., Opus 4 and Sonnet) as well as Gemini-2.5-Pro.
Table 14 presents the system prompt used for our agent baselines. In addition to the system prompt, the agent's input includes three other components: (1) agent behavioral rules detailing interaction behavior (Section C.2p; (2) evaluation rules for IncreQA and AdaptQA to ensure accurate evaluation (Section A.6); and (3) database-specific rules outlining the SQL annotation assumptions for MIMIC-
IV and eICU* (Section A.5).
C.I AGENT SYSTEM PROMPT
Table 14: Agent system prompt.
Instruction:
You are a 18 agent that helps users by answering their questions natural language based on information from database.
You are currently engaged statistics from EER database. If the user's request is ambiguous or missing details (e.g., filtering criteria), ask clarifying questions to better understand the request.
conversation with user who wants
retrieve some
You have access to a set of tools to assist the user: search: search for tables in the database columo_search: search for columns within a table value_ seauue_substring_serrehh search for values a column by substring match
column by sql_execute: run SQL queries the database web web_search: retrieve externsl clinica- knowledge not in the database
search for values semantic similarity (embedding-bbssed)
value_ sarbbu__smmlrrtt_eeeeeee
Use table_search and column_search to explore the database schema. to explore
Use batuuu_subssringgeeeee stored values.
and malne_similarity_eeeee
Clinical concepts (e.g., diagnoses, procedures, nedications, lab tests) the database may not exactly match the user' words. Use the value search tools find relevant entries.
To search for verify clinical knowledge not in the database g., drug's mechanism of action),
web_search.
Never invent or assume information that is not provided by the user or retrieved using the tools.
Make only one tool call at a time. Do not send a user-facing response in the same turn as a tool call.
After gathering all necessary information, use sqi exe_uee to write and run a single valid SQL query that fully answers the user's latest request.
When you write an SQL query, always execute it with exl_uxe and return the results to the user along with your explanation.
Sigent_rules
[database_rule]
C.2
AGENT RULES
Table 15; Agent rules.
Below the general rules for the DB agent:
32

---

Published as a conference paper at ICLR 2026
The DB agent must assume the user has no knowledge of sQb, databases, or stered values, and cannot execute queries. The DB agent must interact with the user only in natural language and must not show raw SQL queries.
The DB agent must not modify the database schema contents. The following commands are forbidden: INSERT, UPDATE, DELETE, DROP, ALTER.
The DB agent must. write queries that finish within 60 seconds; otherwise, the query results will be invalid.
The DR agent must limit each conversation to interactions including user exchanges and tool calls) and seconds total. The DB agent must always explain answers in natural language, including the zeasening or conditions used to arrive at those answers. If SQL references are necessary, the DB agent must explain them in terms understandable to someone with no SQL knowledge.
The DB agent must clearly explain when a question cannot be answered (e.g., due to limitations of SQL or empty results) and ask the user to rephrase or modify the request.
The DB agent must generate a non-empty response, which must include either message or tool call.
C.3 AGENT RULES FOR EVALUATION
Table Tnn Table 17 present the prompts used for IncreQA and AdaptQA tasks, respectively. These prompts are appended to the agent rules to guide the agent toward behavior that aligns with accurate evaluation criteria. Note that for both task types, the agent's 's responses evaluated each time they are generated, as correct answers may emerge during the conversation due to the iterative nature of open QA dialogues.
Table 16: Scoring rules for InereQA.
Below the grading rules:
The DB agent' performance is evaluated based on the generated SQL queries, requiring at least one SQL query (via sqi_execute) to retrieve answers during the interaction with the database. For accurate assessment, when the user revises their question, the agent
write a new sQl query from scratch the latest request, without relying on previous query results.
fully address
For questions that involve calculations (such as time differences survival rates) data manipulation/ ingggggggtgtgggoonn the DB agent must SQL Language to compute the results rather than relying capabilities.
Table 17: Scoring nules for AdaptQA.
Below the grading rules:
The DB agent' performance is evaluated based on the results in its natural language response to the user.
When providing answers to the user, the DB agent must enclose the final answer in <answer></answer>> tags (e.g., <answer>42</anaeer>>)) All other content, including intermediate results, explanations, units, and any additional details, must be placed outside these tags
when answers are textual data (e.g., timestamps or di diagnosis names them exactly stored in the database. For numerical answers round them to four decimal places.
33

---

Published as a conference paper at ICLR 2026
D SAMPLE TRAJECTORIES
In this section, we present sample task instances along with one successful and one failed conversa- tion trajectory for IncreQA and AdaptQA.
D.1 INCREMENTAL QUERY REFINEMENT
D.1.1 SUCCESSFUL CASE (INCREQA)
The agent successfully retrieves the number of patients diagnosed with breast cancer by querying values the description column the diagnosiscodes table.
Instruction: Your goal to find the number of patients admitted to the hospital who meet specific criteria. Specifically, you want know how many patients admitted in the past 90 days have family history of breast cancer.
=22
User: I'm trying to figure out how many patients admitted the hospital have certain conditions.
Agent: table_ tabeleeseaaccllaguueeeerrttooot"
Tool: admissiondiagnoses, admisionproccedures,, clnncaeeetss,,,,,neeee clinicalitemtypes, costrecords, demographics, diagnosiscodes, hospitaladmissions, icuepisodes, intakerecords, labresults, labtesttypes, medicationorders, mictooo outputrecords, patienttransfers, proceucccss
Agent:
Tool: CREATE TABLE admissiondiagnoses
recordid INTEGER NOT NULL,
patientid INTEGER NOT NULL, admissionid INTEGER NOT NULL, icdcode TEXT NOT NULL, codeversion INTEGER NOT NULL, recordeddatetime TEXT NOT NULL,
PRIMARY KEY (recordid), FOREIGN KEY (codeversion) REFERENCES diagnosiscodes (codeversion), FOREIGN KEY (icdcode) REFERENCES diagnosiscodes (icdcode),
FOREIGN KEY (admissionid) REFERENCES hospitaladmissions (admissionid)
/+
3 rows from admissiondiagnoses table:
recordid patientid admissionid icdcode codeversion recordeddatetime
10035185 22580999 4139 9 2100-05-17 12:53:00
34

---

Published as a conference paper at ICLR 2026
110035185 22580999 v707 9 2100-05-17 12:53:00
2 10035185 22580999 41401 2100-05-17 12:53:00
*
CREATE TABLE diagnosiscodes
recordid INTEGER NOT NULL, icdcode TEXT NOT NULL, codeversion INTEGER NOT NULL, description TEXT NOT NULL, PRIMARY KEY (recordid)
/*
rows from diagnosiscodes table:
recordid icdcode codeversion description
0090 infectious colitis, enteritis, and gastroenteritis
01160 tuberculous pneumonia [any form], unspecified
01186 other specified pulmonary tuberculosis, tubercle bacilli found by
bacteriological histological examination, but tuberculosis confirmed other methods inoculation of animals]
*/
Could you clarify which specific conditions you'd like to count? For example, are you interested in diabetes, heart failure, COPD, etc.? Also, should these diagnoses be those recorded the time of hospital admission?
Agent:
User: want to know how many patients admitted in the last 90 days have family history of breast cancer.
Agent:
value substring ortp:///meeeestoroocrrcioioiiioe
"faaily"""ammiyy")
Tool: Values in diagnosiscodes.deecriittoo containing 'family': ['family history of polycystic
kidney', 'nom-lammly member, perpetrator of maltreatment and neglect', 'family history of unspecified malignant neoplasm', 'aamlly history eye disorders', 'family disruption due death family member'].
Agent: Saluummsaaccfaaruueensssstteeeer "nspssaaaaaaaalaasssiissnn""
Tool: CREATE TABLE hospitaladmissions
recordid INTEGER NOT NULL,
patientid INTEGER NOT NULL,
admissionid INTEGER NOT NULL,
35

---

Published as a conference paper at ICLR 2026
admitdatetime TEXT NOT NULL, dischargedatetime TEXT, admissiontype TEXT NOT NULL, admitsource TEXT NOT NULL,
dischargedestination TEXT, insurancetype TEXT NOT NULL, language TEXT, maritalstatus TEXT, age INTEGER NOT NULL,
PRIMARY KEY (recordid),
FOREIGN (patientid) REFERENCES demographics (patientid)
/*
rows from hospitaladmissions table:
recordid patientid admissionid admitdatetime dischargedatetime admissiontype
admitsource dischargedestination insurancetype language maritalstatus age
10004235 24181354 2100-03-19 14:3::00 2100-03-28 14:02:00 urgent transfer from hospital
skilled nursing facility medicaid english single
10009628 25926192 2100-09-30 17:08:00 2100-10-08 13:20:00 urgent transfer from
hospital home health care medicaid ? married 58
2 10006053 22942076 2100-11-15 23:39:00 2100-11-17 17:20:00 urgent transfer from hospital
died medicaid english None 52
Agent:
"cgpr/aaaaraaannnrrttedddeddddrd "olpriddeeeeooouummmddddrrriiiii ""value"""'rreas" breast"))
Tool:
Values in diagnosiscodes.descriiioon containing 'breast': l'other and unspecified disorder of breast associated with childbirth, postpartum condition or complication', malignant neoplasm of axillary tail of female breast', 'full-thickness skin loss [third degree, not otherwise specified] of chest wall, excluding breast and nipple', 'personal history of malignant neoplasm of breast', 'open wound of breast, without mention of complication'..
Agent: sql_ ex "SELECT COUNT(DISTINCT ha.patientid) num_ jum_patients FROM hospitaladmissions AS ha JOIN admissiondiagnoses AS ad ON ha.
admissionid = ad.admissionid JOIN diagnosiscodes AS de ON ad.icdcode dc.icdcode AND
ad.codeversion = dc.codeversion WHERE dc.description = 'family history of malignant
neoplasm of breast' AND haaadmitdatetime >= datetime(221o0-122 23:59:00', '-90 days') (}})
Tool: [(1.)]
36

---

Published as a conference paper at ICLR 2026
Agent: There have been unique patient admitted in the last 90 days with diagnosis of family history of malignant neoplasm of breast.
User: Can you double check that the answer covers my goals?
Agent: Yes-here's how the query addresses each part of your goal:
Last days: We filtered admissions with
ha.dmmitdatetime > datetime(22100-1222 23:59:00", -90 days')
Family history of breast cancer: We joined admissions -> diagnoses ->> codes and used dc.description "family history of malignant neoplasm of breast'
Unique patients: We counted DISTINCT ha.patientid so repeat admissions aren't double- counted.
The result was patient. Let me know if you'd like any further breakdown or checks!
User: ###:ENDmm
37

---

Published as a conference paper at ICLR 2026
D.1.2 FAILED CASE (INCREQA)
The agent fails to retrieve information breast cancer, it queries unrelated tables and columns (e.g., clinicalevents and clinicalitemtype) instead of the diagnosiscodes table. Due to the use of FAISS. wrapped in the LangChain library, values with semantic similarity below the default thresh- old (0.8) are not retrieved. In the example below, the agent applies the value similarity tool to the itemname column of the clinicalitemtypes table for "cancer," but the tool returns no result. Note that itemname corresponds to the label in the d ittm table in MIMIC-IV, which covers a wide range of clinical practices, vital signs, and measurements.
Instruction: Your goal is to find the number of patients admitted to the hospital who meet specific criteria. Specifically, you want to know how many patients admitted the past 9o days have a family history of breast cancer.
User: want to find out how many patients have been admitted to the hospital recently and also have family history of breast cancer.
Agent: To assist you with your request, need to clarify couple of things:
What do you mean by "recently"? Are you looking for admissions within a specific time frame (e.g., the last month, last year)?
2. How is the family history of breast cancer recorded in the database? there a specific table column that indicates this information?
Once have this information, can proceed to gather the necessary data from the database
User: I'm interested in patients admitted in the last 9o days. As for the family history, could you check the database to find where that info is stored?
Agent: table fabelssaacccaaeemmmmms//roooo"""
Tool: admissiondiagnoses, admissionprocedures, clnicaleveets,,, clinicalitemtypes, costrecords, demographics, diagnosiscodes, hopittaadmmiisisnn icuepisodes, intakerecords, labresults, labtesttypes, medicationorders, mittioooestt outputrecords, patienttransfers, proceuddcdss
Agent: ouw.m./aecccggggemeeeee///a/aaee
Tool:
CREATE TABLE demographics recordid INTEGER NOT NULL, patientid INTEGER NOT NULL, gender TEXT NOT NULL, dateofbirth TEXT NOT NULL, dateofdeath TEXT,
38

---

Published as a conference paper at ICLR 2026
PRIMARY KEY (recordid)
/*
3 rows from demographics table:
recordid patientid gender dateofbirth dateofdeath
1 2079-07-22 00:00:00 None
10003400 f 2028-07-22 00:00:00 None
10002428 2020000030 00:00:00 None
*/
Agent:
column searchlaguueens--(tabbe names". ""iiiiicacciiaeeets"""))
Tool:
CREATE TABLE clinicalevents recordid INTEGER NOT NULL, patientid INTEGER NOT NULL, admissionid INTEGER NOT NULL, icuadmissionid INTEGER NOT NULL, itemcode INTEGER NOT NULL, recordeddatetime TEXT NOT NULL, value DOUBLE PRECISION, unit TEXT,
PRIMARY KEY (recordid),
FOREIGN (itemcode) REFERENCES clinicalitemtypes (itemcode), FOREIGN KEY (icuadmissionid) REFERENCES icuepisodes (icuadmissionid),
FOREIGN KEY (admissionid) REFERENCES hospitaladmissions (admissionid)
y*
3 rows from clinicalevents table:
recordid patientid admissionid icuadmissionid itemcode recordeddatetime value unit
10005817 20626031 32604416 220210 2100-12-24 00:00:00 19.0 insp/ min
10005817 20626031 32604416 220051 2100-12-24 00:00:00 37.0 mmhg
2 10005817 20626031 32504416 223762 2100-12-24, 00:00:00 3
*/
Agent: Cllummsecccharruuueesssstttteeee "iincattts"""iialatettttss""")
Tool: CREATE TABLE clinicalitemtypes! (
recordid INTEGER NOT NULL,
itemcode INTEGER NOT NULL, itemname TEXT NULL, abbreviation TEXT NOT NULL, itemtype TEXT NOT NULL,
39

---

Published as a conference paper at ICLR 2026
PRIMARY KEY (recordid)
/*
3 rows from clinicalitemtypes table:
recordid itemcode itemname abbreviation itemtype
226228 gender gender clinicalevents
226545 race race clinicalevents
229877 suction events (ch) suction events (ch) clinicalevents
*/
Agent: value substring petnciiiiggnntttttaaaarrriiiiiie "fvalue"."family history"))
Tool: No values in clincclittettypes.teemmmee contain 'family history'.
Agent: value httpliiiii..iiiiiioooorrrrrrrrii
"f"mill"."family history"))
Tool: No matches found in cinicalitemtyyes.itemnnee for 'family history'.
Agent: ("uumnssaccchagguueesstttaaeeee otapsa/ssssrrrrcooo.oooooooiiiir
"hospitaladmissions"))
Tool:
CREATE TABLE hospitaladmissions recordid INTEGER NOT NULL, patientid INTEGER NOT NULL, admissionid INTEGER NOT NULL, admitdatetime TEXT NOT NULL,
dischargedatetime TEXT, admissiontype TEXT NOT NULL,
admitsource TEXT NOT NULL,
dischargedestination TEXT, insurancetype TEXT NOT NULL, language TEXT, maritalstatus TEXT, age INTEGER NOT NULL,
PRIMARY KEY (recordid),
FOREIGN KEY (patientid) REFERENCES demographics (patientid)
/+
3 rows from hospitaladmissions table:
recordid patientid admissionid admitdatetime dischargedatetime admissiontype
admitsource dischargedestination insurancetype language maritalstatus age
40

---

Published as a conference paper at ICLR 2026
10004235 24181354 2100-03-19 14:38:00 2100-03-28 14:02:00 urgent transfer from hospital
skilled nursing facility medicaid english single
10009628 25926192 2100-09-30 17:08:00 2100-10-08 13:20:00 urgent transfer from
hospital home health care medicaid ? married 58
10006053 22942076 2100-11-15 23:39:00 2100-11-17 17:20:00 urgent transfer from hospital died medicaid english None 52
*
Agent: columms.errcddrruuuennittttteeed httr:/aaaieeammmmmmmmmemarrrrra
Tool:
CREATE TABLE demographics
recordid INTEGER NOT NULL, patientid INTEGER NOT NULL, gender TEXT NOT NULL, dateofbirth TEXT NOT NULL, dateofdeath TEXT, PRIMARY KEY (recordid)
/*
rows from demographics table:
recordid patientid gender dateofbirth dateofdeath
10014729 2079-07-22 00:00:00 None
10003400 2028-07-22220000 00:00:00 None
10002428 2020-07-30 00:00:00 None
*/
Agent: column Colummssarccarguueetttttaaaalll "aines"tdiniccaleeenss"")
Tool:
CREATE TABLE clinicalevents recordid INTEGER NOT NULL, patientid INTEGER NOT NULL, admissionid INTEGER NOT NULL, icuadmissionid INTEGER NOT NULL, itemcode INTEGER NOT NULL, recordeddatetime TEXT NOT NULL, value DOUBLE PRECISION, unit TEXT,
PRIMARY KEY (recordid),
FOREIGN (itemcode) REFERENCES clinicalitemtypes (itemcode),
FOREIGN KEY (icuadmissionid) REFERENCES icuepisodes (icuadmissionid),
FOREIGN KEY (admissionid) REFERENCES hospitaladmissionss (admissionid)
41

---

Published as a conference paper at ICLR 2026
/*
rows from clinicalevents table:
recordid patientid admissionid icuadmissionid itemcode recordeddatetime value unit
10005817 20626031 32604416 220210 2100-12-24 00:00:00 19.0 insp/min
10005817 20626031 32604416 220051 2100-12-24 00:00:00 37.0 mmhg
10005817 20626031 32604416 223762 2100-12-24 00:00:00 37.0
*/
Agent: value substring oumpliircrrccooooooororrrrrrrain ","value":" "cancerr))
Tool: No values in dinicalitemttypes.ieemmmee contain 'cancer'.
Agent: value
""vacuee""cancer))
Tool: No matches found in clincalitemtypes.iteemmaee for 'cancer'.
Agent: It seems that I couldn't find any specific entries related to "family history of breast
cancer" the database. This could mean that the information either stored a different format or not recorded the tables checked.
To proceed, can look for any general family history cancer-related entries in the clinical events demographics tables. Would you like me to do that?
User: Yes, please check for any general family history or cancer-related info.
Agent: value substring seachhargumentss=(tabe":: "clinicalitemtypes",, "column":"
itemname", "value": "family history")))
Tool: No values in clincalitemtyppssitemmmae contain 'family history'.
Agent: value ouupsrgggeegggooooooorrrrrrrrrrm
""value"""cancer""
Tool: No values in clincalltemttyes.ttemmaaa contain 'cancer'.
Agent: have searched for general family history and cancer-related information in the database, but unfortunately, couldn't find any relevant entries the clinical item types. appears that the information regarding family history of breast cancer may not be explicitly recorded in the database, or it might be stored in different format table that
42

---

Published as a conference paper at ICLR 2026
haven't checked. If you have any other specific criteria different information you would like to explore, please let me know!
User:
Okay, can you at least tell me how many patients have been admitted in the last 90 days?
Agent: "ggLeexeuteelrruuettt==uuueeee "SELECT COUNT(DISTINCT patientid) AS
patient.coume count FROM hospitaladmissions WHERE admitdatetime > '2100-09-30"))
Tool: [(8,)]
Agent: There have been patients admitted to the hospital in the last 90 days. you have any other questions need further assistance, feel free to ask!
User: Okay, so you couldn't find the family history info. Just to confirm, you were able to find the number of patients admitted in the last 90 days?
Agent: Yes, that's correct! able find that there have been 38 patients admitted to the hospital in the last days. However, couldn't find any specific information regarding family history of breast cancer in the database. If you have any other questions or need further assistance, please let me know!
User:
Double check that you've covered my goals: the number of patients admitted in the last 90 days with family history of breast cancer.
Agent: understand your request clearly now. You want to know the number of patients who were admitted in the last 90 days and also have a family history of breast cancer. Unfortunately, was unable to find any specific entries related to family history of breast cancer in the database. This means cannot confirm how many patients meet both criteria. you have any other specific information different criteria you would like explore, please let me know!
User: ###EN DD###
43

---

Published as a conference paper at ICLR 2026
D.2 ADAPTIVE QUERY REFINEMENT
D.2.1 SUCCESSFUL CASE (ADAPTQA)
The user initially asks whether specific patient was prescribed carbamazepine for epilepsy treat- ment, but the agent finds that no such records exist for the patient. The user then requests information about other medications for epilepsy. The agent searches the database using clinical knowledge with keywords such as levetiracetam, phenytoin, and phenobarbital. The agent checks whether the patient was prescribed any of these medications and successfully retrieves the most recent prescription date for levetiracetam, as requested.
Instruction: Your goal is to explore patient data related to medication prescriptions.
Specifically, you are interested patient ID 10008287 and want know when they were prescribed carbamazepine for epilepsy treatment. the patient was prescribed carbamazepine, you want to know the most recent prescription time after the first epilepsy diagnosis. If the patient was not prescribed carbamazepine, you want to identify whether any other class of medication used for epilepsy treatment was prescribed. If such medications are found, you want know the most recent prescription time after the first epilepsy diagnosis. no such medications are found, end the conversation. You want the answer in the exact time format recorded in the database.
===
User: I'm looking into patient prescriptions. Can you help me find some info about the medications prescribed patient ID 10008287?
Agent: taaleeeeercchaaggemmemetoroooon"
Tool: admissiondiagnoses, admissionprocedures, cinicaeeensss,,,,,, clinicalitemtypes, costrecords, demographics, diagnosiscodes, hospisaassaaaaassssssonnns,, icueiisdees, intakerecords, labresults, labtesttypes, medicationorders, miccoooooet.mtrooooooorrssstsss outputrecords, patienttransfers, proceede
Agent: column seachhargummett-<"tablee names":" l "nniesaccmmiiciiooorrrr""))
Tool:
CREATE TABLE medicationorders ( recordid INTEGER NOT NULL, patientid INTEGER NOT NULL, admissionid INTEGER NOT NULL, startdatetime TEXT NOT NULL, enddatetime TEXT,
medicationname TEXT NOT NULL, dosevalue TEXT NOT NULL, doseunit TEXT NOT NULL, administrationroute TEXT NOT NULL, PRIMARY KEY (recordid),
44

---

Published as a conference paper at ICLR 2026
FOREIGN KEY (admissionid) REFERENCES hospitaladmissionn (arp
/*
3 rows from medicationorders table:
recordid patientid admissionid startdatetime enddatetime medicationname dosevalue
doseunit administrationroutt
10020740 23831430 2100-04-19 11:00:00 2100-04-20 22:00:00 insulin unit
02:00000000000000000000000000000 10005817 20625031 2100-12-20 02:00:00 2100-12-21 210-12211 12:00:00 insulin 0.0 unit sc
2 10005817 20626031
2100-12-27 20000
10:00:00 2100-12-28 19:00:00 insulin o.0 unit sc
*/
Agent:
Could you clarify which details you'd like to for patient 10008287's prescriptions For example, are you interested in the medication names only, the start/end dates, dosages, administration routes, or all of those?
User: I'm particularly interested in whether patient 10008287 was prescribed carbamazepine for epilepsy. If so, want know the most recent time it was prescribed, but only after their first epilepsy diagnosis.
Agent: value aarcchhgeeeeettttaaeeiiieiittttr medicationname", "value" "valuu" "aatluurrcaaammemeee""""""""""""
Tool: Values in medicatioonnderrseeiiitttommmm g 'carbamazepine': [' carbamazepine')
Agent: column ("uummseerccddruuueettttttleee.r oomaas/sssssiri.o.mommommooossss
Tool:
CREATE TABLE admissiondiagnoses recordid INTEGER NOT NULL, patientid INTEGER NOT NULL, admissionid INTEGER NOT NULL, icdcode TEXT NOT NULL, codeversion INTEGER NOT NULL, recordeddatetime TEXT NOT NULL,
PRIMARY KEY (recordid), FOREIGN KEY (codeversion) REFERENCES diagnosiscodes (codeversion),
FOREIGN KEY (icdcode) REFERENCES diagnosiscodes (icdcode),
FOREIGN KEY (admissionid) REFERENCES hospitaladmissions (admissionid)
/*
rows from admissiondiagnoses table:
recordid patientid admissionid icdcode codeversion recordeddatetime
45

---

Published as a conference paper at ICLR 2026
10035185 22580999 4139 9 2100-05-17 12:53:00
10005585 22580999 v707 9 2100-05-17 12:53:00
2 10005185 22580999 41401 2100-05-17 12:53:00
*/
Agent: ccl.mmsesacchaaggueeerrrrlrleeee "digoosssss""
Tool:
CREATE TABLE diagnosiscodes (
recordid INTEGER NOT NULL,
icdcode TEXT NOT NULL, codeversion INTEGER NOT NULL, description TEXT NOT NULL, PRIMARY KEY (recordid)
)
/*
3 rows from diagnosiscodes table:
recordid icdcode codeversion description
0090 infectious colitis, enteritis, and gastroenteritis
01160 tuberculous pneumonia (any form], unspecified
0116 other specified pulmonary tuberculosis, tubercle bacilli not found by
bacteriological or histological examination, but tuberculosis confirmed by other methods inoculation animals]
osaars/mmeenssstlleddrrddiooooes Agent: epilepsy""" ","alue""""
Tool: Values in diagnosiscodes.descriiioon containing 'epilepsy': ['epilepsy complicating pregnancy, childbirth, or the puerperium, delivered, with mention postpartum complication', 'epilepsy, unspecified, without mention of intractable epilepsy', 'epilepsia partialis continua, with intractable epilepsy', 'epilepsy complicating pregnancy, childbirth, or the puerperium, unspecified as to episode of care or not applicable', 'localization- related (focal) (partial) epilepsy and epileptic syndromes with complex partial seizures, with intractable epilepsy', 'generalized convulsive epilepsy, with intractable epilepsy', localization-related (focal) (partial) epilepsy and epileptic syndromes with complex partial seizures, without mention of intractable epilepsy', 'localization-relttee (focal) (partial) epilepsy and epileptic syndromes with simple partial seizures, without mention of intractable epilepsy', 'epilepsia partialis continua, without mention of intractable epilepsy', 'epilepsy complicating pregnancy, childbirth, or the puerperium, postpartum condition or complication"..
46

---

Published as a conference paper at ICLR 2026
Agent: sqL. wwweeuttaaruumeesssuuee"e""""""" "SELEC MAX(mo.startdatetime)) AS
most_recent..rrecrppiioo FROM medicationorders mo WHERE mo.patientid 10008287 AND mo.medicationname = 'carbamazepiine AND mo.startdatetime >> (SELECT MIN(ad.
recordeddatetime) FROM admissiondiagnoses ad JOIN diagnosiscodes dc ON ad.icdcode
dc,icdcode AND ad.codeversion dc.codeversion WHERE ad. ad.ppaienttd 10008287 AND dc. description LIKE "Kepilepsy%);")
Tool: [Nonee)]
Agent: sql uqgL_eeeetttlagmeeet=/eeey "SELECT ad.recordeddatetime AS diagnosis time FROM admissiondiagnose ad JOIN diagnosiscodes dc ad.icdcode dc.icdcode AND ad. codeversion dc.codeversion WHERE ad.patientid 10008287 AND dc.description LIKE '% epilepsy% ORDER BY ad.recordeddateetim ASC LIMIT 1;"})
Tool: (02100-10-09 01:17:000))
Agent: sol.uveeuuddrrrieetstssueerr"""" "SELECT FROM medicationorders WHERR patientid 10008287;"))
Tool: [(354, 10008287, 22168393, 22100-10-10 00000000000000000000000000000000 'bag', '1.0',
bag', 'iv'), (356, 10008287, htt.://.................../...or
20000000000000000000000000000000 'bag', '1.0',
'iv'), (655, 10008287, 22168393, 22100-10009 20000000000000000000000000000000 cefazolin
cefazllll
'1.0', "g', 'iv'), ..., (6797, 10008287, 22168393, 2000 '0.9% sodium chloride', '1000.0', 'ml', 'iv), (6804, 10008287, 22168393, '2100-10-09 2000 22:0000000 '2100-10-10 10:00:00', 'o.9% sodium chloride', '1000.0', 'ml', 'iv)]
Agent: sggLexeeatelaruuent=="uuuueee "SELECT COUNT(*) AS count.prescriptions FROM medicationorders WHERE patientid 10008287 AND medicationname 'carbamazepine;"")
Tool: [(o,)]
Agent: The answer is <answer:>No, patient 10008287 was not prescribed carbamazepine after their first epilepsy diagnosis.</answer>
User: Ok, if carbamazepine wasn't prescribed, can you check if they were prescribed any other medications for epilepsy after their first epilepsy diagnosis? If so, give me the most recent prescription time.
Agent: sql execttelagummnts==uueey::: "WITH first epi AS (SELECT MIN(ad. recordeddatetime) AS dt FROM admissiondiagnoses JOIN diagnosiscodes de ON ad.
icdcode dc.icdcode AND ad.codeversion dc.codeversion WHERE ad.ppaient 10008287 AND dc.description LIKE '%epilepsy'' SELECT MAX(m.startdatetimm) AS
47

---

Published as a conference paper at ICLR 2026
most.recent prescription FROM medicationorders JOIN first epi ON m.startdatetime f.
dt WHERE m.patientid 10008287 AND (m.medicationname LIKE "%levetiracetam% OR m.
medicationname LIKE '%phenytoin% OR m.medicationname LIKE "Kphenobarbita%% OR m.
medicationname LIKE '%valpro% OR m.medicationname LIKE "%lamotr%' OR m.
medicationname LIKE "Kgabapentin% OR m.medicationname "%topir%' OR m.
medicationname LIKE "%pregabat% OR m.medicationname LIKE '%ethosux% OR medicationname LIKE "%lacosamidee%))")
Tool: [02100-10-12 20:00:0000]
Agent: The answer <ann0er222001002220000000aaaaa2
User: Great! Just to be sure, this time represents the most recent time patient 10008287
prescribed any medication for epilepsy after their first diagnosis? I need the answer in the exact time format recorded in the database.
Yes, that is correct. The most recent epilepsy medication prescription time after their first epilepsy diagnosis <answer>2100- 20:00:00</answer>>>
Agent:
User: Perfect, thanks!
###EN D###
48

---

Published as a conference paper at ICLR 2026
D.2.2 FAILED CASE (ADAPTQA)
After the user initially asks about carbamazepine for epilepsy treatment for a specific patient, the agent finds relevant data. The user then asks about other medications for epilepsy. The agent uses a web search tool to retrieve a list of medications, including levetiracetam. The agent writes an SQL query using the diagnosis "epilepsy, unspecified" and the medication list for the patient. However, since the patient was diagnosed with "epilepsy, unspecified, not intractable, without status epilepticus," the agent fails to retrieve the specific data needed, despite having the list of epilepsy medications.
Instruction: Your goal is to explore patient data related to medication prescriptions. Specifically, you interested patient ID 10008287 and want know when they were prescribed carbamazepine for epilepsy treatment. If the patient was prescribed carbamazepine, you want to know the most recent prescription time after the first epilepsy diagnosis. If the patient was not prescribed carbamazepine, want to identify whether any other class medication used for epilepsy treatment was prescribed. such medications are found, you want know the most recent prescription time after the first epilepsy diagnosis. If such medications found, end the conversation. You want the answer in the exact time format recorded in the database.
=22
User: want to know about the prescriptions for patient 10008287.
Agent:
Could you please specify what information about the prescriptions you interested in? For example, are you looking for the medication names, dosages, dates, or something else?
User:
I'm interested in finding out if patient 10008287 was prescribed carbamazepine for epilepsy. If they were, need the most recent prescription time after their first epilepsy diagnosis.
Agent: Okay, need to find out if patient 10008287 has been diagnosed with epilepsy and prescribed carbamazepine. Then, both conditions are met, need to find the most recent prescription time for carbamazepine after the first epilepsy diagnosis. First, need to find the relevant tables containing diagnoses and medication orders.
Agent: table searchlarguments-(ttod input": ""})
Tool: admissiondiagnoses, admissonprrocedures clinicalevents, clinicalitemtypes, costrecords, demographics, diagnosiscodes, hospitaladmissions, icuepisodes, intakerecords, labresults, labtesttypes, medicationorders, microbiologyresults, outputrecords, patienttransfers, procedurecodes
49

---

Published as a conference paper at ICLR 2026
Agent: Okay, the tables admissiondiagnoses and medicationorders seem relevant. I need to explore these tables to see how diagnoses and medication orders are recorded.
Agent: column searchlarguments- names": "admissionndiagnosss, medicationorders "})
Tool:
CREATE TABLE admissiondiagnoses
recordid INTEGER NOT NULL,
patientid INTEGER NOT NULL, admissionid INTEGER NOT NULL, icdcode TEXT NOT NULL, codeversion INTEGER NOT NULL,
recordeddatetime TEXT NOT NULL,
PRIMARY KEY (recordid),
FOREIGN KEY (codeversion) REFERENCES diagnosiscodes (codeversion), FOREIGN KEY (icdcode) REFERENCES diagnosiscodes (icdcode),
FOREIGN KEY (admissionid) REFERENCES hospitaladmissions (admissionid)
/*
rows from admissiondiagnoses table:
recordid patientid admissionid icdcode codeversion recordeddatetime
10035185 22580999 4139 9 2100-05-17 12:53:00
10035185 22580999 v707 9 2100-05-17 12:53:00
10035185 22580999 41401 2100-05-17 12:53:00
*/
CREATE TABLE medicationorders recordid INTEGER NOT NULL, patientid INTEGER NOT NULL, admissionid INTEGER NOT NULL, startdatetime TEXT NOT NULL, enddatetime TEXT, medicationname TEXT NOT NULL, dosevalue TEXT NOT NULL, doseunit TEXT NOT NULL, administrationroute TEXT NOT NULL,
PRIMARY KEY (recordid),
FOREIGN KEY (admissionid) REFERENCES hospitaladmission (admissionid)
/+
rows from medicationorders table:
50

---

Published as a conference paper at ICLR 2026
recordid patientid admissionid startdatetime enddatetime medicationname dosevalue doseunit administrationroute
10020740 23831430 2100-04-19 11:00:00 2100-04-20 22:0000000 22:00:00 insulin o.0 unit sc
10005817 20626031 2100-12-20 02:00:00 2100-12-21 12:00:00 insulin 0.0 unit sc
10005817 20626031 2100-12-27 10:00:00 2100-12-28 19:00:00 insulin o.o unit
*
Now need to find the icdcode for epilepsy in the diagnosiscodes table. will use calue_similarity.seae search to find similar values to "epilepsy" in the description column of the diagnosiscodes table.
Agent:
Agent: value similarity (oooaiicyeeecccccaeeeeesssooor "description", "table":" diagnosiscodes", "value": "epilepsy"))
Tool: found close matches dianoosiscodes..escrriiin:: l'epilepsy and recurrent seizures 'epilepsy, unspecified", 'other epilepsy', 'epileptic spasms', 'epilepsy, unspecified, not intractable', 'epilepsy, unspecified, without mention intractable epilepsy', 'epilepsy, unspecified, with intractable epilepsy', 'epilepsy, unspecified, intractable', 'other epilepsy and recurrent seizures', 'epilepsy, unspecified, intractable, without status epilepticus', epilepsy, unspecified, not intractable, without status epilepticus', 'other epilepsy, intractable, without status epilepticus', 'epilepsy, unspecified, intractable, with status epilepticus', 'other epilepsy, intractable, with status epilepticus', 'epilepsy, unspecifted, not intractable, with status epilepticus', 'generalized idiopathic epilepsy and epileptic syndromes', 'other epilepsy, not intractable, without status epilepticus', 'generalized idiopathic epilepsy and epileptic syndromes, intractable').
Agent: The value similarity search returned several matches. To proceed, will use 'epilepsy unspecified' as the diagnosis for epilepsy. Now need to check patient 10008287 was diagnosed with 'epilepsy, unspecified' and prescribed 'carbamazepine', If so, need find the most recent prescription time after the first epilepsy diagnosis.
Agent: value bolluessubsringgseeee saaarrnggsaaccchruuueeesstroouuu " "medicationname", "table":" medicationorders" "value": "carbamazepine"))
Tool:
Values in melicaaionnndersmeeiiitttooaaaa conting 'carbamazepine': ['
carbamazepine').
Agent:
Okay, the medication name is 'carbamazepinn', Now have all the information need to construct the SQL query.
Agent: sg "SELECT medicationorders JOIN admissiondiagnoses ON m.patientid a.patientid WHERE m.
"SEEEC MAX(msstarddatetime) FROM
51

---

Published as a conference paper at ICLR 2026
patientid = 10008287 AND m.medicationname "carbamazepine' AND a.icdcode IN (SELECT icdcode FROM diagnosiscodes WHERE description = 'epilepsy, unspecified") AND m.
startdatetime (SELECT MINrecordeddattetim)) FROM admissiondiagnoses a2 WHERE a2. patientid 10008287 AND a2.icdcode IN (SELECT icdcode FROM diagnosiscodes WHERE description 'epilepsy, )nspecifted'))")
Tool: [Nonee))
Agent:
The query returned None, which means either the patient was not diagnosed with epilepsy, was not prescribed carbamazepine, or the prescription occurred before the diagnosis.
Agent: www.e/wccuueearruunsssssssusseee "SELECT MIN(recordeeddatetim)) FROM admissiondiagnoses a WHERE a.patientid = 10008287 AND a.icdcode IN (SELECT icdcode FROM diagnosiscodes WHERE description = 'epilepsy, unspecified""))
Tool: [(None,)]
Agent:
Since the query to find the first epilepsy diagnosis returned None, it means the patient was never diagnosed with epilepsy. Therefore, cannot find the most recent carbamazepine prescription time after the first epilepsy diagnosis.
Agent: seL.exeuueeargummntsss''uuuee "SELECT DISTINCT patientid FROM
medicationorders WHERE medicationname 'carbamazepine' AND patientid 10008287")
Tool: []
Agent: The patient 10008287 was not prescribed carbamazepine. Therefore, cannot find the most recent carbamazepinn prescription time after the first epilepsy diagnosis.
answer:-Patient 10008287 not diagnosed with epilepsy and was not prescribed carbamazepine<<</nsserr
User: Okay, if they weren't prescribed carbamazepine, I'd like to know if they were prescribed any other medications for epilepsy. so, want the most recent prescription time after their first epilepsy diagnosis, just like asked for carbamazepine.
Agent: Okay, I need to find out if patient 10008287 has been diagnosed with epilepsy. If they have, need to find any medications prescribed to them that are used to treat epilepsy and then find the most recent prescription time for those medications after the first epilepsy diagnosis. First, need identify medications used treat epilepsy. will web search for this.
52

---

Published as a conference paper at ICLR 2026
Agent:
web searchaaggueentss""uueeee "medications used to treat epilepsy"))
Tool:
"Below is a list some of the most common bramd-name name drugs currently used treat epilepsy. Your doctor may prefer that you take the brand name of anticonvulsant and not the generic substitution. Talk with your doctor about this important issue.)
nBrivaracetam (Briviact) ( CCCanaaiiiol nCarbamazepiin (tppaiooee/// (Carbatrol or oaaaaaaatteeeoppprr//oooaaaappp (Valium), lorazepam (Ativan) and similar benzodiazepine tranquilizers such as clonazepam
(Aptiom (Fintepla))nn)
[...] Ethosuximide ( FFelbamatee
nLacosamide nlaamoorinn (Lamiccal))nthnneeettiaaaaaaa (Keppra)),n
nOxcarbazepine (Oxtellar XR, httlapa/hh../laanoooooo/////ayya (tapriinnnnnnnnPnaaaaaaaa nPregabalin
( Topamax) (nlpas))nnnnnnlnnnnnana /Valproate, valproic acid (Depakene, (. (Zonegran)
nPhenytoin (Dilantin)',
[..] The most commonly used medications are benzodiazepines because they get into the bloodstream quickly to start working on the brain to stop the seizure. They include:),nny Epilepsy Drug Guidelines),n (lt may take several months before the best drug and dosage determined for you. During this adjustment period, you will be carefully monitored through frequent blood tests to measure your response to the medication.", "Below is list of some of the most common brand-name drugs currently used to treat epilepsy. Your doctor may prefer that you take the brand name of anticonvulsant and not the generic substitution. Talk with your doctor about this important issue.),n n) nBrivaracetam (Briviact) nnaanaaadi (Epidiolex); n Carbamazeeiee (Carbatrol Tegretol)) n' [Cenobamate(xcoopr)) , |/Dnaazepamm (Valium), lorazepam (Ativan) and similar benzodiazepine tranquilizers such as clonazepam htt.oopiiinnnnnESiiicraaaaaaaa
(Aptiom [...] Ethosuximide hww.oontiiinnnnnFeelaaaan ( (felbatto)),,.feefuuuammie (Fintepla)),n
nLacosamide (VIMPAT) nLamotrigine nlLamorrinnn (Lamiccal),,,,nLeeettiaacaaat (Keppr))) n)
nOxcarbazepine (Oxtellar XR, Trileptal)nn (ty.ae/aaaaaaaeellyyyyyaaaaaaaaa
nPhenytoin (Dilantin)\,n |\ (Lyrica)) (nkcca))n,,TTTaabiinn (G ( (i (Zonegran) Topamax) n |/Valproatte, valproic acid (Depakene,
[..] The most commonly used medications are benzodiazepines because they get into the bloodstream quickly to start working on the brain to stop the seizure. They include:),n)n ## Epilepsy Drug Guidelines), nlt may take several months before the best drug and dosage are determined for you. During this adjustment period, you will be carefully monitored through frequent blood tests measure your response the medication.", "Medications used to treat epilepsy called antiepileptic drugs (AEDs). There more than prescription AEDs on the market, and they're mostly available as oral tablets or capsules. n' nYour options depend on:) n \ Types of antiepileptic drugs (nD nThere are two types of medications for epilepsy: n nSome people may need to take more than one medication to prevent seizures. [...] ### Levetiracetam (Elepsia XR, Keppra, Keppra XR, Spritam) nLevetiracetam (Elepsia XR, Keppra, Keppra XR, Spritam) may treat a range of epileptic seizures, including) n nit's available pill, an IV solution,
53

---

Published as a conference paper at ICLR 2026
oral solution, and injection. nLevetiracetam may cause fewer side effects than other medications used for epilepsy. It's safe to take during pregnancy, according to experts such as the United Kingdom's Commission on Human Medicines. (n) n Lorazepam (Ativan) [..]] Lorazepam (Ativan) a benzodiazepine that's used treat all types seizures. also used to treat status epilepticus. Status epilepctícus is a prolonged, critical seizure that's regarded as medical emergency. n nnl's available pill, oral concentrate, and an injection. (n) n Methsuximide (Celontin)\(n nMethsuximide (Celontin) is used for absence seizures. It's prescribed when other treatments don't work in treating your seizures.",
"Donate n nPopular searches: Diagnosing Epilepsy Treatments and Therapies what is epilepsy mMaee an
and Seizure Medicationsi,n
....aaeeaeeeeee................. nftititt Learn about FDA-approved medications to
treat epilepsy and seizures., 7: Epilepsy and Seizure mi nFilter Alphabetically Select Clear) nLoading ... (,) n* Acetazolamide nMore info( Brand names: Diamox, Diamox Sequels, generics) [...] Acetazolamide (a SEET ZOLE a mide) has been FDA-approved the treatment of, along with other drugs, centrencephalic epilepsies (absence, generalized seizures). n' (лнннн Brivaracetam ()) nMore info(Brand names: Briviact) , BBrivaraceta (briv RA se tam) has been approved by the FDA to treat focal (partial) onset seizures in patients month of age and inder.)nn,,ntiitte n info(Brand names: Epidiolex) [...] More info(Brand names:
acetate (ES li Kar BAZ e peen) has been approved by the FDA to treat focal onset seizures in patients years of age and older.) in n Ethosuximide nMore info(Brand names: Zarontin, generics) n nεthhouuimmde (ETH oh SUX mide) has been approved by the FDA control absence (petit mal) epilepsy. *:: Ethotoin MMore info(Brand names: Peganone, generics)",
"Healthcare providers prescribe antiseizure medications to treat epilepsy and symptomatic seizures. They also prescribe these medications to prevent and/or treat seizures that happen during or following brain surgery. [..]] Gray gradient gradient) nSearch Icon Blue nCleeellnd Clinic logo n n# Antiseizure Medications (Formerly Known Anticonvulsants)) n nAntiseizure medications (anticonvulsants) help treat epilepsy and other causes of seizures. They can treat other conditions as well, like anxiety and neuropathic pain. There are several different types of antiseizure medications. You and your healthcare provider will work together to find the best one for you. \/) nAdvertisement [...] Cleveland Clinic non-profit academic medical center. Advertising on site helps support our mission. We do not endorse non-Cleveland Clinic products services. Policy' n' n## Overview') n n#### What are antiseizure medications ( anticonvulsants)2, n' nAntiseizure medications (previously known as antiepileptic or anticonvulsant medications) are prescription medications that help treat and prevent seizures. Healthcare providers may prescribe these medications to treat other conditions well.",
"Epilepsy Website Logo') n' (ntt List of Anti-Seizure Medications (ASMs) n) nit!!
Understanding Epilepsy nAnti-epileptic drugs (ASMs) are the main form of treatment
54

---

Published as a conference paper at ICLR 2026
for people living with epilepsy, with up to 70% (7 in people) having their seizures controlled through this medication. |n),n Australia there are over 20 ASMsare used to treat seizures. The ASMsprescribed are often selected on the basis of the seizure type/s, age, gender and side effects. ASMsmay be prescribed as tablets, syrups and liquids.", "One large randomized trial, the Standard and New Antiepileptic Drugs (SANAD) trial, demonstrated some comparative advantages of certain AEDs when treating focal generalized epilepsy. In the end, when comparing valproate, lamotrigine, topiramate for generalized seizures, they recommended valproic acid as their first-line choice. Additionally, when comparing carbamazepine, gabapentin, lamotrigine, oxcarbazepine, and topiramate for focal seizures, lamotrigine was cited as the first-line [..]] In summary, it is now abundantly clear that anti-seizure medications wield disparate mechanistic profiles, but they all effectively suppress epileptic seizures in one way another. Accordingly, grouping the drugs together by mechanism is very helpful organizing principle. From this viewpoint, it may become easier to appreciate that some drugs have different efficacy profiles for different seizures types and epilepsy syndromes. Ethosuximide is an exception with its specific limited use with [...] Phenytoin one the oldest anti-seizure medications and is still widely used for focal and generalized seizures. is also administered for status epilepticus. addition, practitioners may invoke phenytoin as a second-line agent for patients with mixed seizure types (e.g, tonic -clonic and myoclonic). As mentioned, phenytoin blocks voltage-gated sodium channels, but other possible mechanisms revolve around decreased synaptic transmission, smaller changes in ionic gradients involving the",
"Medicines.t n Surgery-)n Therapies that stimulate the brain using device.), n A ketogenic diet.),n nttmt Medication n noss people with epilepsy can become seizure- free by taking one anti-seizure medicine, which is also called anti-epileptic medicine. Others may be able to decrease the number and intensity of their seizures by taking more than one medicine. [...] Tell your healthcare professional immediately if you notice new or increased feelings of depression or suicidal thoughts. Also contact your healthcare professional right away if you have changes in your mood behaviors. n Tell your healthcare professional if you have migraines. You may need anti-seizure medicine that can prevent your migraines and treat epilepsy. [...] ###/ Potential future treatments n .nResearchers are studying many potential new treatments for epilepsy, including n Continuous stimulation of the seizure onset zone, known as subthreshold stimulation. Subthreshold stimulation continuous stimulation to an area of the brain below level that's physically noticeable. This type of therapy appears improve seizure outcomes and quality of life for some people with seizures. Subthreshold stimulation helps stop a seizure before it happens.",
"I stiripentol| Diacomite Image 50: Medline unavailable Image 51: Medline unavailable
|||] sulthiame Ospolot Image 52: Medline available Image 53: Medline unavailable In
tiagabine Gabitril Image 54: Medline available Image 55: Medline available [\n]
topiramate Topamax Image 56: Medline available Image 57: Medline available [\v]]
vigabatrin Sabril Image 58: Medline available Image 59: Medline available [...]
55

---

Published as a conference paper at ICLR 2026
ethosuximide
Zarontin Image Medline available Image Medline available [|/]]
felbamate Felbatol Image 22: Mims unavailable Image 23: Medline available [\n]
gabapentin Neurontin Image 24; Medline available Image 25: Medline available l'
lacosamide Vimpat Image 26: Medline available Image 27: Medline available [),n]
lamotrigine Lamictal Image 28: Medline available Image 29: Medline available [..]]
phenobarbitone Phenobarbitone Image 40: Medline available Image 41: Medline
unavailable [|] phenytoin Dilantin Image Medline available Image 43: Medline
available || potassium bromide Dibro-Be mono Image 44; Medline unavailable
Image 45: Medline unavailable ||] primidone Mysoline Image 46: Medline available
Image 47: Medline available |\ sodium valproate Epilim Image 48: Medline available
Image 49: Medline available
"Sodium valproate is epilepsy drug prescribed for all seizure types including absence, myoclonus and tonic clonic seizures. New regulations have been introduced by the Medicines and Healthcare products Regulatory Agency (MHRA) around the way in which the drug, sodium valproate is prescribed to women and girls of childbearing age. ( nLET S STAY TOUCH... [..]] The decision whether start taking anti-epileptic drugs (AEDs) can be difficult, and there a think about. Here we look at the benefits and risks of taking, or not taking, AEDs. n InSodium valproate [...] Epilepsy Society',n' nSite Navigation n nUtilitiy Nav) nSite Search nbreadcrumb navigation: n nList of anti- seizure pnndicattion,,n,,nullllii nAuthored by A nAllo in this section' n' of anti-seizure medication),n) nHere is list of anti-seizure medication (ASM), previously called anti-epileptic drugs or AEDs."
on 8 October innpdaeee; April 2025),n) n))
Based on the web search results, here's a list of medications commonly used to treat epilepsy:
Agent:
Brivaracetam
Cannabidiol
Carbamazepine
Cenobamate
Diazepam
Lorazepam
Clonazepam
Eslicarbazepine
Ethosuximide
Felbamate
Fenfluramine
Lacosamide
Lamotrigine
Levetiracetam
Oxcarbazepine
56

---

Published as a conference paper at ICLR 2026
Perampanel
Phenobarbitol
Phenytoin
Pregabalin
Tiagabine
Topiramate
Valproate, valproic acid
Zonisamide
Acetazolamide
Methsuximide
Stiripentol
Sulthiame
Gabapentin
Primidone
Sodium valproate
will now construct query to find the most recent prescription time for any of these medications after the first epilepsy diagnosis.
Agent: st..exeeuudddrrgueetss/ueeeeee "SELECT MAX(m.startdatetime) FROM medicationorders m JOIN admissiondiagnoses ON m.patientid = a.patientid WHERE m.
patientid = 10008287 AND m.medicationname IN ('brivaracetam', 'cannabidiol,,
carbamazepine', 'cenobamate', |enbomaaaepee,,,eeennamamte, 'diazepam' 'lorazepam', 'clonazepam', 'eslicarbazepinej ethosuximide', 'felbamate', 'fenfluramine', 'lacosamide', 'lamotrigine', 'levetiracetam", oxcarbazepine', 'perampanel,, pprrmppal 'phenobarbitol; 'phenytoin', 'pregabali,,, 'tiagabine', topiramate', 'valproate', 'valproic acid', 'zonisamide', 'acetazolamide; 'methsuximide'" stiripentol', 'sulthiame', 'gabapentin', 'primidone', 'sodium valproate') AND a.icdcode IN SELECT icdcode FROM diagnosiscodes WHERE description rerprippiooieeelllllsssssueuu unspecified) AND m. startdatetime > (SELECT MIN(recordeddatetime) FROM admissiondiagnoses a2 WHERE a2. patientid 10008287 AND a2.icdcode IN (SELECT icdcode FROM diagnosiscodes WHERE description 'epilepsy, unspecifted')"))
Tool: [Nonne))
The query returned None, which means either the patient was not diagnosed with epilepsy, not prescribed any the listed medications, the prescriptions occurred before the diagnosis. Since already determined that the patient not diagnosed with epilepsy, the result is None. <answer> Patient 10008287 was not diagnosed with epilepsy and was not prescribed any anti-epileptic medicatiooss:</nnserrr
Agent:
User: Could you double check that your answer covers everything? wanted know if patient 10008287 prescribed carbamazepine and, if not, if they were prescribed any
57

---

Published as a conference paper at ICLR 2026
other epilepsy meds. For any epilepsy meds, need the most recent prescription time after their first epilepsy diagnosis.
Agent: Yes, have confirmed that patient 10008287 was neither diagnosed with epilepsy nor prescribed carbamazepine or any other anti-epileptic medications from the list generated using web search. Therefore, cannot provide prescription time after their first epilepsy diagnosis. <answer>Patient 10008287 not diagnosed with epilepsy and was not prescribed carbamazepine or any other anti-epileptic medications.</answer
User: ###END##!
58

---

Published as a conference paper at ICLR 2026
D.3 INVALIDATED TRAJECTORIES BY THE VALIDATOR
Below we show sample trajectories that were invalidated by our simulation validator along with the specific rules they violated.
D.3.1
VIOLATION OF No FINAL CHECK RULE
Instruction: Your goal is to find the administration routes of specific ordered medications. Specifically, you are interested in the administration routes of ordered medications containing ISDN. You want to search for isosorbide dinitrate specifically, not other similar medications like ""isosorbide mononitratey When querying the DB agent, since you do not know how isosorbide dinitrate is stored, use common terms like ("ISDN)" when referring to it, and let the DB agent find it for you.
User: I'm trying figure out how certain meds are given to patients. Can you help?
Agent: can help with that. Could you please specify which medications you are interested in and what information about their administration you would like to know? For example, are you interested the dosage, frequency, or route of administration?
(...omitted...)
Agent: The administration route for isosorbide dinitrate is 'po/ng'. This likely refers to oral or nasogastric administration.
User: #*HENDн## (RULE VIOLATION)
Before ending the conversation, ask the DB agent double check that their final answer covers all goals raised. If not, request any missing parts.
D.3.2 VIOLATION OF MISSING CONDITIONS RULE
Instruction: Your goal is to find information related to patient's diagnosis. Specifically, you are interested in patient ID 10004733 and you want to know how many days have passed since the patient was diagnosed with cerebral artery occlusion during the current hospital admission. If the patient has multiple records of the same diagnosis, you want to know the number of days elapsed since the earliest diagnosis. You want diagnosis names containing cerebral artery occlusion.
59

---

Published as a conference paper at ICLR 2026
User: I'm trying find out some information about patient's diagnosis.
Agent: can help with that. What specific information are you looking for regarding a patient's diagnosis? For example, are you trying to find:
patient's diagnosis history?
The most common diagnoses?
Diagnoses associated with certain procedures medications?
The cost associated with particular diagnosis?
Something else entirely?
Please provide more details so can narrow down my search.
User: I'm interested in patient ID 10004733. want to know how long ago they were diagnosed with something.
Agent: Okay, I can help you find out how long ago patient ID 10004733 was diagnosed. Let me first check the admissiondiagnoss table to see contains the diagnosis date and patient
User: ###END### (RULE VIOLATION)
+ The user must provide all detailed conditions specified in the instruction before ending the conversation.
D.3.3 VIOLATION OF PERFORMING AGENT-LIKE TASKS RULE
Your goal to find the number of blood culture tests performed on a specific patient. Specifically, you want know how many culture tests were collected from venipuncture blood for patient ID 031-10300 this month.
Instruction:
==2
User: need to figure out how many blood culture tests a particular patient has had.
Agent: To determine the number of blood culture tests, need the patient's ID. Could you please provide the patient ID?
User: The patient ID is 031-10300.
(...omitted...)
Agent:
The patient with ID 031-10300 had 2 blood culture tests collected from venipuncture blood.
60

---

Published as a conference paper at ICLR 2026
User: Okay, have the patient ID, the time frame (this month), and the collection method venipuncture blood). Does your request include any other conditions? (RULE VIOLATION)
+ The user acts like DB agent or assistant instead of the user (e.g., writing, reviewing, executing SQL queries, calling external APIs, responding the DB agent in machine assistant way).
61


Reviews:
[{"review_id": "JlMjM3ib63", "reviewer": "Reviewer_6mwk", "summary": "The paper proposes a new benchmark to better capture the real-world complexities of text-to-sql queries on EHR databases. They propose an environment leveraging LLMs that simulate real user interactions. By going beyond static benchmarks, the paper argues to better validate the query ambiguity and value mismatch, and their interactive resolution that are common in practice. \n\nDisclaimer: I am not well aware of text-to-sql datasets let alone EHR-specific ones. My assessment is based only on the paper's content. It is possible that I missed any important related work.", "questions": "Please comment on the questions mentioned in the Weakness section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The motivation for EHR text-to-sql is clear and the need for going beyond static benchmarks is also imparted well. \n\nThe paper carefully controlled contamination by changing the names of tables and columns in the database. They also sourced database and queries from real hospitals, which makes the dataset interesting.", "weaknesses": "W1) Details on how they simulated the two user interaction workflows: IncreQA and AdaptQA in Sections 4.2.2, 4.2.3 are not clear to me. The authors should consider elaborating or elucidating with an example. \n\nW2) I also could not follow how they validated the final response from the interaction. Since the queries can be altered mid-way (in AdaptQA) somewhat arbitrarily, how's the evaluation done? \n\nW3) In order to establish the reliability of dataset, it is required that the authors report some validation to ensure that the numbers reported in Table 3 are not inflated. In other words, how many of the interactions are misjudged due to failures of user-simulator, user validator, or response validator?  \n\nW4) The proposed benchmark has a heavy system component because they are proposing an environment for validation. But I did not find system setup instructions in the supplementary folder of main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3AK9Pw90wS", "reviewer": "Reviewer_WcY6", "summary": "This paper builds a benchmark called EHR-ChatQA to test how well LLM agents can answer questions over EHR databases in a real conversation, not just single direct questions by focusing on cases where the user starts vague, clarifies over time, or changes their question mid-way — like how clinicians actually talk in real world. The benchmark requires the model to interpret the question, ask clarifying questions, write SQL, and return an answer. The contribution is mostly - a realistic test setup + showing there’s still a big gap in making LLM agents dependable for EHR querying.", "questions": "Couuld the authors clarify how the proposed method handles edge cases or scenarios where the stated assumptions do not hold?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "I believe the authors thought well and paper is original in framing EHR QA as a conversational, iterative task rather than single-turn SQL. The quality of the benchmark construction is solid, though it could be better validated. The writing is clear, making the setup and results easy to follow. I do think the significance is moderate — it highlights an important reliability gap, but the contribution is mainly diagnostic rather than solution-oriented.", "weaknesses": "I believe the analysis focuses mostly on pass/fail metrics, without deeper breakdowns of failure modes or model behavior, which limits insights into improving performance. Finally, the work emphasizes problem framing rather than proposing strategies to address the reliability gap, so the contribution feels diagnostic rather than forward-moving.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BfhrRGN9yS", "reviewer": "Reviewer_FdSS", "summary": "The authors seek to improve the benchmarking of language model agents in EHR-relevant database tasks, such as clarifying questions, solving value mismatches, and generating accurate SQL. They find that the current LLMs they test struggle with consistent success in this regard. They describe the method of benchmark creation as well as testing, including the use of LLM-based validation.", "questions": "1. Who were these 38 graduate-level contributors? How were they trained? What is their background? What was their specific input?\nHow was the \"LLM-as-judge\" validator system itself validated? What were the impacts of implementing this system?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I commend the authors for engaging with this particular topic within the Health AI field - optimizing the nuts and bolts of back-end EHR agent interaction will be very important to ensuring these tools are practically useful. Their recognition of the inherent ambiguity and clinical context-specificity of clinician questions is also important. \n- The authors' introduction includes a reasonable summary of the relevant literature in the field and the purpose of their benchmark. \n- The figures are clear and add to the quality of the work. The overall workflow and structure is quite clear. \n- The core QA tasks are reasonable, and the level of human annotation appears overall strong (although I would love further detail on the \"38 graduate-level contributors\" and what was actually done). \n- I greatly appreciate the depth of analysis the authors offer regarding the nature and presumed causes of errors. This strongly supports the validity of this system as a useful benchmark.\n- In general, this paper appears to make a strong contribution to the broader literature.", "weaknesses": "- It is always difficult, when presenting such a benchmark, to know the extent to which the failures reported by the authors (e.g. the worse Pass^5 vs Pass@5) are inherent to the model vs related to flaws in the authors' implementation of that model. While this is inevitable to benchmarking, I do think it should clearly be acknowledged that these results represent a plausible floor, but not at all a ceiling for that which the authors seek to evaluate. \n- Benchmarks using LLMs as user-agents have the same set of concerns, regarding the prospect for the questioner-LLM to inadvertently \"tip off\" the answerer-LLM, but that does not render them useless in this regard. The authors overall do a good job of highlighting these errors in their section 6, but perhaps this should also be discussed. That is, there are failures which make the test too difficult, but there also may be those which make it too easy.  \n- Perhaps more a direction for fture work, but as a clinician I feel that these systems can be further elevated through the use of clinical context itself to improve the questioning process, rather than just general clinical knowledge.\n- I am quite concerned, however, about the authors' use of the LLM-as-judge paradigm without any human annotated gold-standard set for whether the validator actually achieves its goals. How are we to know, for example, whether these invalidations are correct, or whether this \"validator\" is itself discarding valid behaviors? Quis custodiet ipsos custodes? This use of unvalidated LLM-as-judge is very concerning here, and I recommend that either further validation of this approach is offered, or the section is removed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rcmg1M310Q", "reviewer": "Reviewer_Rw7P", "summary": "The paper introduces EHR-ChatQA, a new interactive benchmark for evaluating Large Language Model (LLM) agents accessing Electronic Health Record (EHR) databases, specifically addressing the practical challenges of query ambiguity and value mismatch common in clinical settings. EHR-ChatQA assesses the full end-to-end agentic workflow, including conversational refinement using an LLM-based user simulator, active schema and value tool use, and accurate SQL generation, across two distinct interactive flows: Incremental Query Refinement (IncreQA) and the more adaptive AdaptQA. Evaluations conducted across state-of-the-art closed- and open-source models reveal a critical robustness gap (Pass@5 versus Pass^5), exceeding 35-60% in the challenging AdaptQA flow, demonstrating that current agents, while occasionally capable, are fundamentally unreliable for safety-critical EHR tasks.", "questions": "- Given the importance of AdaptQA in revealing the robustness gap, what efforts are planned to expand its size to improve the statistical reliability of results for this difficult and novel task type?\n- Could authors provide a more detailed, quantitative breakdown of the error types (Value Linking vs. SQL Generation) specifically correlated with the Gap-k metric, differentiating brittle failures from consistent failures?\n- Since simulation validity is crucial, please provide a few detailed examples of dialogues deemed \"invalid\" by the LLM-as-a-judge validator to better illustrate how the validator enforces the specified user behavior rules.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The benchmark is the first interactive evaluation specifically designed for EHR question answering, holistically assessing the full agentic workflow including tool use and conversational refinement.\n- Construction of the task instances is rigorously grounded in real-world clinical QA scenarios, utilizing publicly available EHR databases with renamed schemas to enforce genuine exploration rather than memorization.\n- A sophisticated simulation environment employs a stochastic LLM user and a dedicated LLM-as-a-judge validator to mitigate simulation noise and ensure evaluation fidelity.\n- Evaluation metrics centered on consistent success (Pass^k) and the resulting robustness gap (Gap-k) provide vital diagnostic insights into agent reliability, critical for safety-sensitive applications.", "weaknesses": "- The Adaptive Query Refinement (AdaptQA) flow, which represents the more challenging and novel adaptation scenario, comprises only 64 instances, potentially limiting the statistical robustness of evaluations in this critical area.\n- Although the user is simulated by an LLM, its conversational spontaneity is heavily constrained by strict behavioral rules (Table 5), raising questions about the fidelity of the simulation to real clinical dialogue complexity.\n- Evaluation relies significantly on extensive, database-specific SQL generation rules (Tables 8 and 9) provided to the agent, suggesting the benchmark may test adherence to prompting constraints rather than intrinsic database reasoning ability.\n- Performance of open-source models on AdaptQA is critically low (0.0% Pass^5), which diminishes the utility of the benchmark for diagnosing failures in current open LLM architectures.\n- More quantitative diagnostic insights are needed, detailing which specific types of user utterances or tool outputs most frequently trigger the observed dramatic drop in consistent success (Gap-k).\n- Lack of transparency regarding the hyperparameters and thresholds used for the `value_similarity_search` tool makes replication or external validation of agent interaction logic difficult.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)