EHR-SeqSQL : A Sequential Text-to-SQL Dataset For Interactively Exploring Electronic Health Records (2024)

Abstract: In this paper, we introduce EHR-SeqSQL, a novel sequential text-to-SQL dataset for Electronic Health Record (EHR) databases. EHRSeqSQL is designed to address critical yet underexplored aspects in text-to-SQL parsing: interactivity, compositionality, and efficiency. To the best of our knowledge, EHR-SeqSQL is not only the largest but also the first medical text-toSQL dataset benchmark to include sequential and contextual questions. We provide a data split and the new test set designed to assess compositional generalization ability. Our experiments demonstrate the superiority of a multiturn approach over a single-turn approach in learning compositionality. Additionally, our dataset integrates specially crafted tokens into SQL queries to improve execution efficiency. With EHR-SeqSQL, we aim to bridge the gap between practical needs and academic research in the text-to-SQL domain.

heading: 1 Introduction
text: Text-to-SQL provides a practical opportunity for non-experts to explore databases, even without prior knowledge of the database operations. Electronic Health Records (EHRs) are large-scale relational databases (RDBs) storing vast and comprehensive patient data (Johnson et al., 2016; Pollard et al., 2018). Medical experts often ask questions that require highly complex reasoning across multiple tables and access to a vast number of records within a single query (Lee et al., 2022). Handling such complexity in large-scale databases remains a significant challenge in the current text-to-SQL research (Li et al., 2023).
Several efforts have been made to construct textto-SQL datasets for EHRs. MIMIC-SQL (Wang et al., 2020) is the first text-to-SQL dataset that targets a subset of MIMIC-III (Johnson et al., 2016),
*These authors contributed equally to this work.
one of the widely-used open-source EHR databases, consisting of 26 tables. DrugEHRQA (Bardhan et al., 2022) provides the first question answering dataset that incorporates both structured tables and unstructured notes from EHRs. EHRSQL (Lee et al., 2022) is a dataset curated based on a survey
16388
from various medical experts, reflecting the diverse information needs of the actual medical field.
Still, there are important yet underexplored objectives for the practical application of text-to-SQL models. First, the text-to-SQL system should incorporate interactivity. Most of the existing text-to-SQL research assumes a single-turn scenario, whereas the process of exploring information in real-world situations is often continuous (Iyyer et al., 2017; Yu et al., 2019a,b). For better user usability, it is desirable for the system to be interactive as well. Second, the text-to-SQL system should learn compositionality. Realistically, datasets cannot capture all the diverse needs of users. Therefore, it is crucial that the model can handle a wider range of queries especially when these unseen queries comprise components (i.e. sub-queries) that the model has explicitly seen during training. This is the compositional generalization ability, which is challenging even for the highly proficient language models (Qiu et al., 2022). Lastly, the text-to-SQL system should consider efficiency. Real-world databases are often significantly larger than academic ones (Hazoom et al., 2021; Zhou et al., 2021). Efficient query execution is crucial, given the magnitude of the databases in actual hospitals (e.g. MIMIC-III dataset includes the medical history of 46k patients). This becomes even more significant when we consider a real-time interaction scenario between a text-to-SQL model and a user (Li et al., 2023).
In this work, we introduce EHR-SeqSQL which addresses all three objectives. Our contribution is as follows:
• Construction of a sequential text-to-SQL dataset for exploring EHR data: As illustrated in Figure 1, EHR-SeqSQL is designed for multi-step interactions, built by decomposing the diverse and complex queries of EHRSQL and setting each question as the interaction goal. To the best of our knowledge, EHR-SeqSQL is the first multi-turn text-to-SQL dataset targeting structured medical records. We make our dataset public to encourage future research.
• Validating the effectiveness of EHR-SeqSQL in the compositional generalization: We designed two experiments to evaluate the compositional generalization ability. The results show that decomposing questions enables the model to better generalize to unseen interaction goals
during training. Furthermore, it demonstrated the potential for generalization on longer sequences not encountered during training.
• Proposing special tokens for SQL: We introduce a highly effective way to enhance query execution efficiency in multi-step interactions with the use of the novel special tokens for SQL. These tokens significantly increase time efficiency during query execution as well as improve the performance of text-to-SQL models. These tokens are not only domain-agnostic but also more effective as the database size increases.

heading: 2 Related Work

heading: 2.1 Multi-turn Text-to-SQL
text: There are only a few datasets that are specifically designed for context-dependent text-to-SQL tasks. ATIS (Hemphill et al., 1990) was the first to include a series of user questions aimed at interacting with a flight database. Recently, Yu et al. (2019a,b) proposed cross-domain context-dependent text-toSQL datasets, based on the questions from Spider (Yu et al., 2018) serving as interaction goals. While their motivation aligns closely with ours, their questions and the grounding databases are predominantly simplistic. In real-world scenarios, however, users typically have much more complex requirements, and databases often contain a significantly larger number of rows (Lee et al., 2022; Li et al., 2023). EHR-SeqSQL is designed upon this objective, based on the questions collected from an actual survey and grounding the real-world database, MIMIC-III (Johnson et al., 2016). To the best of our knowledge, this is the first multi-turn text-to-SQL dataset in the healthcare domain.

heading: 2.2 Compositional Generalization
text: Compositional generalization, a major challenge in semantic parsing, enables a model to manage diverse and unfamiliar user queries using components recognized from training. It is typically achieved by creating different training and test splits of interest. Lake and Baroni (2018) discovered that sequenceto-sequence models struggle to learn compositional structures, as demonstrated through their primitive and length splits. More recently, template splits (Finegan-Dollak et al., 2018) and TMCD splits (Shaw et al., 2020) were proposed to evaluate compositional generalization on text-to-SQL
datasets. Notably, recent efforts (Gan et al., 2022a; Liu et al., 2023b) have proposed compositional generalization in multi-turn settings, which aligns with the concepts we use in this paper. However, we aim to extend their approach by proposing and experimenting with two different challenges in multi-turn compositional generalization: unseen (Section 5.1) and longer (Section 5.2) interactions.

heading: 3 Data Construction
text: We aim to convert text-SQL pairs of EHRSQL to multi-turn text-SQL pairs that embrace interactivity and query execution efficiency. Note that EHRSQL contains text-SQL samples for two different EHR sources, namely MIMIC-III (Johnson et al., 2016) and eICU (Pollard et al., 2018). We use the MIMICIII version of EHRSQL, given its complex schema and wider adoption by the NLP community. Questions for eICU and the questions that cannot be transformed into a proper SQL query are not the scope of this dataset but can be addressed in the future work.
Our data construction process is divided into two stages: SQL decomposition and natural language
question (NLQ) generation. Initially, we break down each SQL query of EHRSQL into a sequence of subqueries to create more granular meanings of queries. Next, we create a corresponding NLQ for each subquery. Compared to NLQ decomposition, SQL decomposition enables systematic breakdown of a question without any loss of information, due to the strict, rule-based nature of SQL syntax. The overall process is shown in Figure 2 and we provide detailed explanations in the following sections.

heading: 3.1 SQL Decomposition
text: Stage 1. Decomposing Nested Query Since most SQL queries in EHRSQL have a nested structure, we decompose the queries based on their nesting levels. This method allows each subquery to contain more granular meanings of the original query and facilitate the multiple turns of interaction. We start by asking the inner query first, and then the outer query is asked while referring to the result of the inner query. This decomposition approach guarantees the executability of queries at each turn and allows anaphoric expressions in the subsequent turns.
In addition, we devised two special tokens for
SQL to leverage the previous turn’s subqueries and results: prev_query and prev_result. These tokens are accompanied by a specific turn index (e.g. prev_query1, prev_result2). Specifically, prev_query token refers to the generated query of the specified turn, whereas the prev_result token refers to the execution result of the specified turn. Two tokens are substituted with either the referred query or its execution result before execution. The effectiveness of these tokens is later discussed in Section 5.3.
Stage 2. Decomposing SQL Clauses After Stage 1, multi-turn questions are obtained, where each can be asked with reference to the answer to previous questions. Still, these questions often convey multiple conditions that users might ask in separate questions. For example, the question “What was the last hospitalized admission time that patient 17694 was admitted via transfer from emergency room?” contains the specific conditions about the admission time, the patient, and the admission route. Therefore, we further decompose queries on the clause-level: WHERE, ORDER BY, and HAVING clauses and aggregation functions (e.g. MAX, MIN, SUM) in the SELECT clause. In cases where multiple clauses appear together, we parse in the order of SQL execution. Exceptions were made only when the original semantics of the query became ambiguous after being decomposed. More details are in Appendix A.1.
Stage 3. Merging Subqueries by Frequency We noted that frequently appearing consecutive subqueries are most likely decomposed due to the specific database schema characteristics or SQL structures rather than each being queries users would naturally ask. In Stage 3, we employ a recursive application of the Byte Pair Encoding (BPE) algorithm to merge frequent consecutive subquery pairs. This recursive approach allows for the amal-
gamation of not only two but up to three or more subqueries, thereby simplifying complex patterns and enhancing query interpretability. Further details are given in Appendix A.2.

heading: 3.2 NLQ Generation
text: Stage 1. Rule-based SQL-to-NLQ Generation While previous SQL-to-NLQ studies (Gan et al., 2022b; Wu et al., 2021) decomposed SQL at the clause level for sub-questions and concatenated clauses later, such simple concatenation has a risk of unnaturalness. Thus we create NLQ templates for each corresponding SQL subqueries for the quality of NLQ. To efficiently annotate NLQs for each corresponding SQL subqueries, we first normalize subqueries by replacing specific table names, column names, and condition values to abstract terms such as table and col (see Figure 2). We then manually create NLQ templates for each normalized subqueries. These templates include slots for table names, column names, condition values, SQL functions (e.g. GROUP BY, AVG), and time expressions. Then the actual question is generated from the NLQ templates by the slot-filling process. More details are in Appendix A.3.
Stage 2. Paraphrase Generation We further paraphrase the NLQ templates to enhance linguistic variability within the dataset, using ChatGPT for its superior performance in understanding and generating text (Guo et al., 2023). We ensured that all the slot values and turn indices were preserved after paraphrasing. Additionally, we employed the self-consistency method and a duplicate question detection model to ensure the quality of the paraphrases, following Lee et al. (2022). On average, we obtained 10.39 paraphrases for each NLQ template. Finally, we randomly assign the paraphrased template to each question and fill the slots with the original condition values.
Quality Check The authors meticulously conducted two-step quality check at both the turn level and the interaction level. All question templates used during the generation process were checked against the following criteria: 1) Completeness: All information in the SQL is also explicitly stated in the NLQ, and 2) Naturalness: Questions are formulated to be as natural as possible after the template masks are realized. Every question and SQL template was crafted to meet these standards. We then conducted quality checks on 1,000 randomly selected interactions from the final dataset at both the turn and interaction level. At the turn level, we checked for clarity of each question on the turn level, since they contain coreferences. At the interaction-level, whether the interactions accurately represent the original intent of the EHRSQL questions is evaluated. We revised the templates or introduced more templates wherever issues were identified. This process was repeated until every sampled interaction met our stringent quality standards at both the turn and interaction levels.

heading: 3.3 Data Statistics
text: Table 1 presents the statistics of multi-turn textto-SQL datasets. EHR-SeqSQL has the largest number of interactions and turns compared to all existing multi-turn text-to-SQL datasets. Notably, this is the first work to introduce special tokens within SQL queries to improve execution efficiency. Furthermore, we also provide a new split and additional test set to evaluate the compositional generalization, as detailed in Section 5.
Questions in our dataset are categorized into four types: independent, referential, filtering, and modifying. These categories are not mutually exclusive and a question may belong to more than one category. Independent questions are those without any prior context. Referential questions refer to previous questions or answers while filtering questions narrow the previous question’s scope by adding conditions. Modifying questions are particularly challenging as they only mention the altered condition and omit all the same conditions. Table 7 shows the distribution of the questions.

heading: 4 Experimental Setup

heading: 4.1 Task
text: The objective of the model is to generate a SQL query given the interaction history and current question. We experiment with two versions of interac-
tion history: one that includes only the previous questions, denoted as QQ, and another that includes both the previous questions and their corresponding SQL queries, denoted as QS. For the QS setting, the model is trained with the interaction history using the ground-truth queries, while during the inference phase, the model’s own predictions are used in order to simulate a real-world application.

heading: 4.2 Baselines

heading: 4.2.1 Fine-tuning Models
text: We employ both a fine-tuning approach and an incontext learning approach for our baselines. For the fine-tuning approach, we use the T5 models (Raffel et al., 2020), the general-purpose sequenceto-sequence models as our baseline models. We did not use the state-of-the-art models (Cai et al., 2022; Xiao et al., 2022; Qi et al., 2022; Zheng et al., 2022; Scholak et al., 2021; Li et al., 2021) for SParC or CoSQL due to their SQL grammar being confined to Spider or standard SQL which is not compatible with our dataset.

heading: 4.2.2 In-context Learning Models
text: We use ChatGPT, LLaMA-7B (Touvron et al., 2023), and Code-LLaMA-7B (Roziere et al., 2023) for our in-context learning models. Instead of a zero-shot approach, we employ few-shot prompting. We retrieve similar exemplars for each test instance using the BM25 algorithm to use as prompt. For every experiment with in-context learning setting, we use 20-shot. For EHRSQL, which is singleturn, computing the similarity for each question is sufficient. However, for EHR-SeqSQL, which is multi-turn and context-dependent, it’s necessary to consider both the current question and the interaction history. Therefore, we use 10 retrieved examples related to the current question and another 10 related to the entire interaction history. More details such as prompt and few-shot retrieval in our in-context learning baseline are in Appendix E.

heading: 4.3 Evaluation
text: The two commonly used metrics in text-to-SQL are Exact Match Accuracy (EM) and Execution Accuracy (EX). However, the EM metric can sometimes be overly strict because it doesn’t take into account predictions that have different SQL syntax but yield the same execution result as the groundtruth query. Thus we use the EX score to measure the model performance based on query execution
results. Before execution, the special tokens are replaced with the generated queries or their execution result through post-processing, thus any errors from the referenced turn will be propagated.
Additionally, for the multi-turn setting, we adopt Interaction Match (IM) and Question Match (QM) following Yu et al. (2019b) on EX. IM measures the accuracy of the entire interaction while QM measures the accuracy of each turn. We utilize different metrics based on the experiment purpose.

heading: 5 Experiments
text: We now empirically demonstrate the benefits of EHR-SeqSQL in terms of two types of compositional generalization (§ 5.1, § 5.2), as well as the effectiveness of the special tokens (§ 5.3).

heading: 5.1 Generalization to Unseen Interaction Goals
text: Compositional Split We first aim to evaluate whether training models in a multi-turn setting can lead to the acquisition of compositional generalization abilities. To explore this aspect, we split our dataset in a compositional manner, namely compositional split. This split differs from the random split in EHRSQL where the distributions of SQL structures in the training and test sets are nearly identical. In contrast, compositional split includes unseen SQL structures in the test set, though these structures can be decomposed into smaller parts that are all present in the training data.
More concretely, we first define the terms compositions and components and use the concepts to automatically split the dataset. Compositions represent SQL templates in EHRSQL, where condition values, SQL functions (i.e. SUM, AVG, etc), time expressions (i.e. subqueries constraining last year, in 2023, etc) are masked. Components refer to the decomposed SQL template clauses derived from Stage 2, as explained in Section 3.1. Each com-
position, which corresponds to an interaction goal, contains a set of components that exist as individual SQL subqueries in the dataset. Table 11 provides concrete examples of components and compositions. We employ a greedy algorithm to split the dataset, similar to Shaw et al. (2020). Table 2 provides the statistics of the random and compositional splits. More details are given in Appendix B.
Metric We only measure IM to compare the performance of a model trained on EHRSQL and a model trained on EHR-SeqSQL. Given that an interaction in EHR-SeqSQL corresponds to a question in EHRSQL, correctly predicting every question within an interaction is equal to correctly predicting a single question in EHRSQL.
Result The experimental results are shown in Table 3. In a random split, all models exhibit strong performance with both EHRSQL and EHRSeqSQL. Still, there is a noticeable performance difference between models specifically trained for SQL generation and those that are not, such as LLaMA and Code-LLaMA, which show relatively worse performance.
However, in the compositional split, models generally perform better with the EHR-SeqSQL dataset, which indicates superior generalization ability in multi-turn setting. T5 models, in particular, show a significant performance increase, ranging from 7.63%p (from 67.8% to 75.43%), to 30.34%p (from 52.15% to 82.49%). T5-base finetuned in QQ setting has the best performance. This suggests that a task-specific fine-tuning enables effective extraction of necessary information only from NLQs without being distracted by other contextual factors. Interestingly, T5-3B shows lower performance than T5-base, and this performance gap is particularly pronounced in our proposed compositional split. This might be due to the compositional split having significantly less training
data and the evaluation data being perceived as out-of-distribution due to its compositional nature. This result is consistent with previous observations (Kale and Rastogi, 2020; Zhang et al., 2023; Shaw et al., 2020) which found that smaller models (T5-base or T5-large) can outperform their larger counterpart, T5-3B, outside the general domain or when dealing with out-of-distribution data.
On the other hand, ChatGPT suffers a performance drop in the QQ setting, which is suspected to be due to the absence of predicted SQL queries within the prompt, which leads ChatGPT to have less information on target representation. Still, a multi-turn setting leads to a significant performance increase in the QS setting. It’s worth noting that ChatGPT has better compositional generalization ability in a single-turn setting than fine-tuned models. LLaMA shows very low performance in every setting, indicating it is not well-suited for generating SQL. Code-Llama performs relatively better and demonstrates clearly better performance with our dataset. Overall, our experiments show that training with EHR-SeqSQL allows the models to generalize well even with unseen questions.

heading: 5.2 Generalization to Longer Interactions
text: Longer Interaction Generation In this section, we aim to explore whether the models trained with EHR-SeqSQL can comprehend longer interactions, reflecting another form of compositional generalization ability. To test this, we created a new test set (TestL) composed of much longer interactions with multiple follow-up questions. This setup is intended to replicate real-world scenarios where users ask a series of questions to incrementally explore a topic of interest (Yu et al., 2019b). For each test set of random split and compositional split, we created 100 longer interaction instances by connecting related interactions. Details about the longer
interaction generation process can be found in Appendix C.1. The statistics for TestL for each test set can be found in Table 2. These interactions are, on average, five times longer than those in the training data. The TestL with a random split challenges generalization for longer sequences, whereas the TestL with a compositional split further complicates this by testing generalization to unseen questions, presenting a scenario that is not only more challenging but also more reflective of practical applications.
Metric In this experiment, we investigate whether a model trained on the interactions with three turns on average can generalize to interactions with an average of 14 turns. Given this significant increase in complexity, IM metric is deemed excessively strict. Instead, we report QM to measure the ratio of correctly answered turns. We further measure the Index of the First Failure (IFF), to capture where the model first generates an incorrect response. The methodology for calculating the IFF score is detailed in Appendix C.2.
Result The experiment results are presented in Table 4. Similarly with the findings in Section 5.1, scores from TestL with random split outperform that from TestL with compositional split. This is
likely because the model should generalize to the unseen interaction goals as well as longer interactions in TestL with compositional split. Interestingly, IFF scores from all models are higher than the average length of interactions in the training data. This suggests that the models can learn interaction-level compositionality and generalize to longer turns after being trained with EHR-SeqSQL. However, for the QS setting, T5-base model had a disadvantage in taking long interactions due to its input length constraint, which led to the lowest IFF score. Contrasting with fine-tuned T5 models, ChatGPT notably excelled in QM and IFF scores, achieving 12.21 in the QS setting, near the ideal 15.17, demonstrating its adaptability to longer interactions.

heading: 5.3 Effects of Special Tokens for SQL
text: In this section, we evaluate the effect of two special tokens–prev_query and prev_result. These two tokens allow models to easily reference either the previous query or its execution result and thereby alleviate the decoding burden. Additionally, prev_result can further reduce the execution overhead by preventing duplicated subqueries from being executed multiple times. We assess the utility of these tokens from two aspects: model performance and query execution efficiency.

heading: 5.3.1 Effects on Model Performance
text: Details First, we evaluate the impact of the special tokens on model performance. We compare a model trained with queries with the original EHRSeqSQL, which include the special tokens, to another trained with its standard SQL query version. We ensure that all other training factors, such as model architecture and hyperparameters remain consistent between the two models. To prevent the test set from being too simple, which could potentially undermine the impact of the special tokens, we use a compositional split.
Result As shown in Table 5, the incorporation of special tokens consistently enhances the performance across all settings, regardless of the model variant or the type of interaction history. In the QQ setting, where the absence of prior query history makes the contextual questions more challenging, the special tokens contribute to a substantial performance increase. This is because those tokens simplify referencing previous questions. Specifically, at the question level, the use of these special
tokens leads to a performance increase of 14.6%p (from 75.37 to 89.97), and 30%p (from 52.49 to 82.49) at the interaction level. In the QS setting, on the other hand, the reference is more straightforward because the previous SQL information is given. Still, special tokens enhance model performance by reducing the complexity of the target representation. ChatGPT demonstrates a robust performance even without the special tokens in the QS setting. We speculate this is because ChatGPT is trained on diverse data, which likely includes the standard SQL. Therefore, it might be familiar with standard text-to-SQL tasks, which is also consistent with the recent finding (Liu et al., 2023a).

heading: 5.3.2 Effects on Execution time
text: Details We further evaluate the impact of our special tokens with respect to query execution efficiency. Specifically, we compare the execution time of the queries from EHR-SeqSQL and that of their standard SQL version. The original database consists of the medical records of 1,000 patients and has a size of 95MB. Following the DB construction process in Lee et al. (2022), we designed two additional larger databases, one with the medical records of 10,000 patients and another with all 46,520 patients that MIMIC-III provides. The size of these databases is 921MB and 5.06GB respectively. The results are reported on the queries from the test set in a random split, which covers all
types of SQL queries in EHR-SeqSQL1. We only considered the subset of queries that contain the special token, prev_result.
Result The result is shown in Table 6. We observed an 18% decrease in average execution time, in the original database. On a single query basis, the execution time is reduced at most to 99.89%, where the original SQL query has five nested queries inside. The effectiveness of the special token tends to increase with the size of the database. In the largest database, the special token yielded an average time reduction of 81.0%. Given that realworld databases typically contain huge amounts of data, we anticipate that the special token will yield a significant practical impact in multi-turn text-to-SQL environments.

heading: 6 Conclusion
text: We present EHR-SeqSQL, the first and the largest sequential text-to-SQL dataset designed for EHR databases, aimed at improving interactivity, compositionality, and efficiency in text-to-SQL parsing. Our experiments show that the question decomposition significantly enhances the model’s generalization capabilities for unseen interaction goals and extends its applicability to longer, untrained interactions. Moreover, the introduction of novel SQL-like tokens, which reuse query and execution results from prior queries, enhances execution efficiency and improves the model performance. We believe our dataset will serve as a valuable testbed for assessing contextual and interactive text-to-SQL tasks and bridge the gap between industry needs and academic research.
Limitations
Our dataset is designed to exclude ambiguous or unanswerable questions, focusing instead on the primary objectives of evaluating the model’s performance in the text-to-SQL task. It includes converting natural language utterances into appropriate SQL queries and assessing the model’s capacity for compositional generalization, generating unseen combinations of subqueries observed during training. This design choice allows our dataset to rigorously test the model’s performance. However, we recognize that ambiguous or unanswerable questions are also significant in real-world scenarios. Future research could expand into this area,
1We excluded the execution time that is nearly zero.
potentially enhancing the model’s robustness.
Ethics Statement
Our main concerns are patient privacy and ensuring that the proposed dataset as well as the construction process are both compliant with the privacy regulations. The MIMIC-III database, which requires credentialed access via PhysioNet, is a database that contains de-identified medical records of intensive care patients. Lee et al. (2022) further deidentified MIMIC-III for EHRSQL by shifting time and values across the database. We use the database constructed from EHRSQL, which carefully prevents the re-identification of medical records from the questions. We utilized the ChatGPT api to paraphrase our manually generated question templates. Each question template corresponds to the decomposed queries from EHRSQL and would unlikely contain the sensitive medical information.

heading: Acknowledgements
text: This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant (No.RS-2019II190075), National Research Foundation of Korea (NRF) grant (NRF-2020H1D3A2A03100945), and the Korea Health Industry Development Institute (KHIDI) grant (No.HR21C0198), funded by the Korea government (MSIT, MOHW).

heading: A Data Construction Details
text: Category Example # of questions
Independent Q1. What is the specific item id of the hemoglobin lab test? 14,005(44.22%) Dependent
A.1 Details in Stage 2 in Section 3.1
In stage 2, we decompose the subqueries based on the SQL clauses. Decomposed clauses are parsed according to the logical order of execution of an SQL statement - WHERE, ORDER BY, HAVING, and SELECT. However, there are certain cases that are excluded from this decomposition. First, if a WHERE clause contains any aliases of table or column names without specifying the original name, it is not decomposed for clarity. For example, see SELECT DISTINCT T1.C1 FROM ( PREV_QUERY5 ) AS T1 WHERE T1.valuenum = 73.0. If decomposed, the meaning of T1 would be ambiguous since we do not know the meaning of T1, which is not a column name in MIMIC-III. Thus we ensured the conditions or clauses with table aliases are always used with the subquery where the alias is defined. Second, we do not decompose the clauses for optional data cleansing SQL clauses. For example, see SELECT microbiology events.org_name FROM microbiologyevents WHERE microbiologyevents.spec_type_desc = ‘foot culture’ AND microbiologyevents.org_name IS NOT NULL. In this SQL query, the bolded subquery is intended to ensure all the selected rows have contents, excluding any NULL values. It is an optional condition and a SQL writing style choice which is not included in the original question, so we choose not to decompose such clauses. Also, the ten shortest question templates from EHRSQL are maintained without further splitting or modifications, which are not likely to be asked through multiple sentences. These include questions asking about the drug intake method, the cost of a lab test, or the number of current patients. We do not decompose GROUP BY but combine it either with SELECT or HAVING, since they are not explicitly expressed in the natural language questions (Guo et al., 2019; Wu et al., 2021).
A.2 Details in Stage 3 in Section 3.1
The BPE algorithm was applied to merge the subqueries from both Stage 1 and Stage 2. Specifically, we first derived SQL templates by masking condition values in the SQL queries obtained in Stage 2. Each template was treated as a token in the BPE algorithm, and the syntactically mergeable pairs among the most frequent token bigrams were repeatedly merged. A syntactically mergeable pair refers to a pair of queries that were originally a single SQL query but were separated due to our decomposition strategy. Since BPE algorithm calculates frequency without considering the relationship of token bigrams, we added this constraint to make every resultant SQL query to be executable. Then, we sampled half of each randomly to maximize the complexity of each subquery and the diversity of each interaction. Considering the total number of turns in each stage, we merged bigrams that appear more than 100 times in the subqueries from Stage 1 and bigrams that appear more than 150 times in the subqueries from Stage 2. The final SQL queries for EHR-SeqSQL are acquired throughout three stages.
A.3 List of NLQ templates In Section 3.2, we mentioned the process for rule-based SQL-to-NLQ generation. Detailed steps of this process can be found in Table 8. We used specific SQL templates paired with NLQ templates for this process, as presented in Table 9. Each placeholder in the NLQ template is determined by the DB schema and the values present in the actual SQL query. A few examples showcasing the pairing of DB schema and NL expressions are available in Table 10. The placeholders related to value, operation, and time are used in the same way as described by Lee et al. (2022).
A.4 Prompt for paraphrasing Figure 3 shows the prompt for paraphrasing the questions at an interaction level. For each NLQ template, key expressions (such as condition values, reference indices, etc.) that must be strictly preserved were indicated in order to maintain consistency. During the data construction process, we utilized the ChatGPT API to paraphrase template questions. Prior to paraphrasing, all specific values within each template are replaced with representative, generic values for their respective slots. Once paraphrased, these generic values are then realized back to their original form. This process naturally prevents any sensitive information from being sent to the ChatGPT server.

heading: B Details on the Compositional Split
text: The concept of compositions and components is based on the SQL query since it is the common factor across an EHRSQL instance and its corresponding EHR-SeqSQL instance. Table 11 provides three examples of composition and components. There are EHRSQL questions which is an interaction goal of
the corresponding interaction in EHR-SeqSQL, and their templates where the same anonymizing logic in deriving composition is applied.
We use a greedy algorithm to automatically split our data into training and test sets. Starting with all compositions assigned to the training set, we iteratively allocate a composition that has the maximum number of unique components to the test set while constraining all the components in the composition to exist in the training set, until no composition can be further assigned to the test set. You can find that the components in the last row are all present in the components of the first two rows. Thus, according to our split algorithm, if both the first two compositions are in the training set, the last composition can be assigned to the test set.

heading: C Longer Interaction
text: C.1 Generation Process
As given in Figure 4, we define the concept of related interaction goal by using the context graph. Each interaction goal has its own context graph, whose nodes are defined by the specific condition values (e.g. conditions for patient, drug, or lab tests) in the original EHRSQL. Two questions are deemed related if they have overlapping condition values. Q1 and Q2 are related because they share the same patient 85895. However, Q1 and Q3 are not directly related without Q2 as a bridge. While we used three independent questions from the EHRSQL to demonstrate this concept in Figure 4, a longer and context-dependent interaction used in Section 5.2 is depicted in Figure 5. In this example for longer interaction, five EHRSQL questions are connected, each paired with an original SQL query. In contrast, EHR-SeqSQL involves context-dependent interaction spanning twelve turns, each paired with a query with our special tokens for SQL.
C.2 IFF Score Calculation
IFF = { n+ 1, if all turns are correct k, otherwise
k denotes the specific turn number in the interaction where the first incorrect response occurs. n represents the total number of turns in the interaction. The final IFF score for a test set is calculated as the average IFF score across all interactions. Hence, if the model achieves a perfect score for every interaction, the IFF score becomes one plus the average number of turns of all interactions.
Note that the perfect IFF score for QQ and QS setting is different in Table 4. This is because we removed the test sample that exceeds the maximum token length of T5 when the interaction history is concatenated with the current question for a fair comparison.

heading: D Fine-tuning Baseline
text: D.1 Configuration We fine-tuned the T5-Base and T5-3B using the Adam optimizer, with a global batch size of 32. The learning rate were set to 1e-4 for T5-Base and 5e-5 for T5-3B, respectively. For the other hypterparameter configurations, we followed the settings used in EHRSQL. All experiments were carried out on either a single A100 80G GPU or a A6000 48G GPU. The training process typically took around 10 hours for T5-Base and around 24 hours for T5-3B.
E In-context Learning Baseline
E.1 Prompt Configuration Figure 6 demonstrates how we configured the prompt of ChatGPT for both the EHRSQL and EHRSeqSQL. Unlike EHRSQL where the target representation is standard SQL, EHR-SeqSQL include the special tokens, which are first introduced in this work. Thus, we include a simple description for PREV_QUERY and PREV_RESULT tokens in the prompt. In the QS setting during the inference process, for the current turn, the interaction history includes both previous questions and queries generated by ChatGPT.
E.2 Few-Shot Learning Approach for EHR-SeqSQL We developed a new method for few-shot learning for the multi-turn, context-dependent setting of EHRSeqSQL. This method is detailed in Figure 7. To begin, we created two corpora: one for interaction-level training data and another for turn-level training data. For our few-shot learning approach, we retrieved examples, with half based on the interaction history and the other half based on the current question. In our experiments, we used a total of 20 examples for few-shot learning.

heading: F Use Cases of the Special Tokens
text: Table 12 illustrates an example of the interactions in EHR-SeqSQL where the special token significantly reduce the length of target representation as well as the execution time. Due to readability and spatial issues, the standard SQL version of each turn has been omitted from the table. You can see the standard SQL would be quite lengthy by looking at the SQL† that the used special token refers to.

heading: Future Work
text: Future research could expand into this area,
1We excluded the execution time that is nearly zero. potentially enhancing the model’s robustness