Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: null
review_content: No ethics issue. Dataset is collected by conducting a poll explicitly for the purpose of this dataset.
review_title: Ethics review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper presents EHRSQL, a new text-to-SQL dataset for question answering (QA) from electronic health records (EHRs). The paper provides a comprhensive description of the methodology used to create the dataset. In addition, the paper proposes a new semantic parsing task ("trustworthy emantic parsing") and provides benchmark results on this task.
strengths: This paper and its associated dataset (EHRSQL) are a significant contribution to NLP for health care. A key strength of EHRSQL is that the seed questions used to build the dataset (1,742 utterances) were collected through consultation with clinical professionals and other stakeholders in the health care system. Another strength of this dataset is that the authors incorporated different types of time granularity in the questions, which is of particular importance for QA in health care. Finally, another strength of this dataset is that the authors included unanswerable questions in the dataset with the purpose of testing the trustworthiness of QA systems and developed a benchmarking task to test the trustworthiness of semantic parsing.
weaknesses: I could't identify significant weaknesses in this paper.
correctness: The dataset is designed with a sound methodology, which is especially suitable for the domain of this dataset (health care). The proposed evaluations for the task on trustworthy semantic parsing are adequate for the scope of this paper.

clarity: The paper is well written and the methodology for the data collection and benchmarking is clearly explained.
relation to prior work: The authors acknowledge prior work and they clearly explain how their work is an advancement compared to previous contributions. 
additional feedback: No additional feedback.
review_title: An intreresting paper and a valuable contribution to datasets for QA in health care

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: summary and contributions: This paper has introduced a novel text-to-SQL dataset (EHRSQL) for two publicly available EHRs- MIMIC-III and eICU. The questions are collected using a poll conducted at a hospital. They address real-world questions. The questions include time-sensitive questions and also include unanswerable questions from the poll results to test the reliability of the system.
strengths: The following are the strengths of the papers:
1. The dataset would be a promising contribution to the EHR question-answering community. It is the first text-to-SQL dataset for the eICU database. Since they address real-world questions, so they should be quite relevant to the EHR research community.
2. The dataset will help to test trustworthiness of QA systems. Also, the dataset is time-shifted to keep the time span of the two EHR databases (i.e. MIMIC-III and eICU) realistic.
3. The effort made in the paraphrase generation module was very impressive. The steps involve using human paraphrasing along with use of T5 paraphrasers.
4. There does not seem to be any ethical and social implications. 


weaknesses: Some of the weaknesses of this paper are as follows:
1. Section 3.2 mentions the concept of "cost sampling procedure". The authors need to provide a more elaborate and clear explanation of how the cost is computed from the MIMIC-III database. Some examples should be used to explain the idea.
2. Even though the paper has experimented with models like T5 and GAP for the EHRSQL dataset, the authors should have also evaluated its performance on the "TREQS model" (TRANSLATE-EDIT MODEL FOR QUESTION-TO-SQL QUERY GENERATION) which was used originally for the MIMICSQL dataset.  Or at least, the authors should have mentioned the reason for not evaluating using the TREQS model.

correctness: The authors have provided a well-detailed description of the dataset creation. They have listed all the templates in the Appendix section. The claims in the submission seem to be correct. The paper clearly mentions the models used to evaluate the dataset.
clarity: The paper is very well written and is easy to understand.
relation to prior work: The paper was successful in discussing its contributions and its relevance in comparison to previous works.
additional feedback: Even though there are a few minute weaknesses of the paper, they can be overlooked since it is a significantly useful dataset for the EHR question-answering community.
review_title: Official review of the Paper 260

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Similar to emrKBQA, this work created a repository of medical questions framed around the MIMIC and eICU-CRD datasets, and assembled them into templates. The authors generated a total of 24,000 queries. To increase the diversity of the questions, the authors employed a collection of models (i.e., T5, Roberta, GPT-Neo) to generate, on average, 21 paraphrases per question template. The authors then trained a T5 model with schema serialization on their data and reported an execution accuracy of around 86% on both the MIMIC and eICU-CRD splits.
The claim of presenting a corpus of realistic questions is misleading, as the questions were taken from a poll, rather than from a real clinical setting such as while rounding. . The authors failed to provide issues with slot-filling.
Further, more comparison to prior work, i.e. emrKBQA, is necessary in order to ground the data and the results presented.

strengths: The work presented employed a poll to source questions, and applied more advanced models as compared to emrKBQA. It offers additional slot-filled templates and paraphrases on top of emrKBQA dataset. And although the methodology presented is not new, the modeling, specifically the T5 model, is solid. A strong baseline model was used to evaluate the test set.
weaknesses: The authors recruited non-medical annotators to determine whether or not the paraphrases generated are similar or distinct from the original questions. However, questions may mean different to clinicians versus non-experts.
The performance of a zero-shot setting is slightly misleading. An ablation study is more appropriate to truly assess zero-shot learning as slot filling often only leads to redundant training data. For example, analysis of the emrKBQA dataset showed that training a model on only 5% of the data was equivalent to training the model on 100% of the data. 
How does the model perform when it encounters paraphrases it has never seen before? A paraphrase level split of performance should be presented. The breakdown of the results is extremely weak. Statistics of where the model is wrong is needed, i.e., percent of time the model fails on X).
Finally, what is the point of paraphrase generation if there is a need for three annotators to assess them? 
correctness: While the questions generated were technically written by medical experts, the claim that they accurately “reflect user interests” or are “naturally-occuring” is misleading, as the questions were taken from a poll, rather than from a real clinical setting such as while rounding. Further, by templatizing and then slot-filling these questions, the distribution of the questions changes. This leads to questions that are structurally realistic, but may not be relevant. Testing on this data likely gives researchers a false sense of confidence, especially if the questions are not broken down into different categories.


clarity: The main claim is that the EHRSQL dataset is more diverse, but this is not substantiated. There are significant gaps in the description of the data collection. For example, the slot filling of the templates is lacking in detail. Much more analysis of the questions generated is needed.
●	Why are there unanswerable questions in the EHRSQL if the instruction was specifically not to ask questions that require “external knowledge, ambiguous or qualitative statements, or asking for reasons behind some clinical decisions”? How many of the 1,742 questions fall under this category?
●	How frequently did medical experts ask about “groups of patients”?
●	How were “conditions” chosen in stage 3 of slot filling? by ICD-9 codes?
●	What is the point of the de-identification section? The patient IDs are already random.



relation to prior work: The authors correctly pointed out that the emrKBQA dataset is not diverse, as it mostly asks about test results. However, there is no analysis to demonstrate that the dataset presented is more diverse than the emrKBQA dataset. A breakdown of the types of questions asked is necessary in order to support the claim that the questions generated are more diverse. Moreover, there was no breakdown of model performance with respect to the different types of questions.
additional feedback: Although the submission provides a unique dataset, but it is currently unclear how much it differs from the emrKBQA dataset. The questions are presented in the Appendix but no analysis was performed to compare with emrKBQA.
review_title: EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors propose an extension of the capabilities of the well-known Text-To-SQL datasets MIMIC-III and eICU by introducing 24,000 more realistic utterances, manually collected from the 222 participants of the Konyang University hospital and obtained using paraphrasing augmentation on the base 1,742 utterances.

The collection of real-world intentions is the dataset's major contribution. Then, the dataset is structured around the concept of trustworthiness, which was generally forgotten is the previous works despite the fact that it's a huge concern in medical applications. Finally, they employ paraphrase to do cutting-edge data augmentation and increase the dataset's possibilities.

The authors also evaluate this dataset performance using off the shelves state-of-the-art semantic parsing models and compare them to more general-purpose state-of-the-art sequence to sequence models in various scenarios for highlighting their limits in more realistic uses-cases.
strengths: At the time of writing, this publicly available dataset is not the largest one available for this task and domain in terms of size. But it proposes numerous improvements, which are particularly important for real-world usage of the tools. The authors invest a lot of time and effort in the evaluation and detection of the trustworthiness of the systems, and none of the previously released datasets really tackle the issue of unanswerable questions in the same way that this team does. Compare to previous works, they invest a lot of human labor to come up with much more realistic data : from the technical side point by offering more and bigger SQL tables and from the linguistic side point, since the utterances are way closer to real natural language, if we compare it directly to the MIMICSQL dataset. Adding one more layer of de-identification during the generation process, by randomly shuffling values across patients to make the values untraceable and be able to publicly release the dataset without any more requirement, is something not negligible. It's also reinforcing this idea of ​​privacy protection in machine learning applied to healthcare. The semi-automatic paraphrasing pipeline seem to provide good quality outputs and is sufficiently explained to be replicated by ourselves using the code you are providing freely on the GitHub repository of the project.
weaknesses: The emrKBQA dataset contains only twice as many utterances after the collection and cleaning process, but have 39 time more utterances than yours after data-augmentation / paraphrasing. You put a lot of effort to obtains high quality paraphrasing, which is a very good point, but you haven't invested time to augment the data by filling up slots with "random" medications and diseases to artificially obtains more data.

If I take for example the template "what is the intake method of {drug_name}?", we got the following paraphrase variations :

- what are the methods of consumption of potassium chloride 20 meq/50 ml iv piggy back 50 ml bag?
- tell me the intake method for potassium chloride tabs?
- what is the ingesting method of levothyroxine?

But none of them are duplicated sentence with variations only for medications and diseases. Unfortunately, you didn't do it, however the corpus could have easily grown as a result. It also could be interesting to learn about it if you have tried this strategy, whatever the results could have been.

I also don't understand why when I am counting the number of elements in the train.json and valid.json files for MIMIC-III and eICU, I only got 20,837 elements when you are advertising 24K examples in the table 4 ? Is it a typo ? Or the GitHub repository isn't up-to-date ?
correctness: Both metrics : execution accuracy (EX) and exact string match (ESM), are generally used to evaluate the task for similar datasets (MIMIC-III and eICU). Since the authors have also put their effort in evaluating unanswerable questions, they introduce another metric called false-negative rate (FNR). Experiments in the other hand are based on three architectures : GAP, T5 and T5+Schema which represent correctly the approaches that could be used to perform the task at a state-of-the-art level.
clarity: Well organized, written with a good english. The only thing I would like to have is a detailed description of the text-to-SQL task itself, since the paper have to be open to a broad range of peoples, sometimes not familiar with the task. It was a little confusing at the start for me and require reading the paper a few times to capture the meaning of it. A simple diagram of the followed pipeline at the beginning of the paper could dramatically improve readability of it.
relation to prior work: Absolutely clear, the authors take their time to describe the datasets available for the task and the domain during the tables 4. However, authors could also refer to the corpus built on top of Spider like SParC (https://arxiv.org/abs/1906.02285) and SPLASH (https://www.aclweb.org/anthology/2020.acl-main.187/) to demonstrate that some other similar initiatives has also been experimented on datasets other than MIMIC-III.
additional feedback: A tremendous amount of work was put in place to collect such utterances and extend current dataset capabilities. They put many efforts into understanding errors in the state-of-the-art T5 model by integrating unanswerable questions during the evaluation phase. Lot of works as been invested in the paraphrase generation system to produce high quality variations and all their works has been generously open-sourced on GitHub to encourage reproducibility.

It will also be very interesting to get our hand on the human annotations used to rate the quality of the paraphrases, since it can be useful to set up an automated system for healthcare.
review_title:  A more realistic Text-to-SQL dataset for medical domain applications.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes EHRSQL, a text-to-SQL dataset for semantic parsing of electronic health records (EHRs). The authors collected 1,742 utterances from 222 medical professionals from the Konyang University hospital to build 174 question templates and 56 time templates; 
statements that were ambiguous, required external knowledge, or were outside the scope of the schema were filtered out and used as "unanswerable" questions to simulate potential queries that may occur in real-world medical settings. SQL annotation is then performed for each template; then a sampling process is used to generate all text-to-SQL pairs.

Human and machine-generated paraphrases of the question templates were also included to add linguistic variety to the dataset. Overall, EHRSQL contains 24,211 text-to-SQL pairs linking two medical databases (MIMIC-III and eICU); EHRSQL includes multi-table queries, time queries, and unanswerable questions to further bridge the gap between existing text-to-SQL datasets and real-world medical query settings.
strengths: The paper contributes a text-to-SQL dataset using text statements that reflect real-world medical settings.

The dataset includes a large collection of 174 question templates; EHRSQL also includes an additional 56 time-based templates, resulting in significantly more time-based text-to-SQL pairs than existing datasets.

EHRSQL contains "unanswerable" questions, further simulating real-world conditions, something that no existing dataset contains. Based on the addition of these instances, the authors introduce the task of "trustworthy semantic parsing", in which one must consider the validity of the model's answer as well as its ability to refuse questions beyond its capability.

The authors provide paraphrased question templates to add linguistic variety to the dataset. These paraphrased questions are filtered using human evaluation.

The paper is well-written and easy to read and follow.

The dataset has the potential to positively impact an important area of research.
weaknesses: One of my main concerns is the comparison of EHRSQL to emrKBQA, since emrKBQA seems to be a similar but much larger dataset than EHRSQL (940K vs. 24K text-to-SQL pairs for emrKBQA and EHRSQL, respectively), potentially limiting the impact of the proposed EHRSQL dataset.

How extensible is the EHRSQL dataset? It was not clear to me how one could extend this dataset, and if so, how easy or difficult that process would be. 

62% of the responses are from the nursing staff vs. 22% for physicians. Are there observable differences between the medical statements made by these two medical groups?

MINOR WEAKNESSES

There are multiple grammatical errors, I recommend using a service like Grammarly to fix these issues.

line 156: No space between "...revisions.Space..."
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: No.
review_title: Review

Meta-review: metareview: This work builds a new text-to-SQL dataset, EHRSQL, using two widely studied medical record databases, namely, MIMIC-III and eICU. The questions are collected using a poll conducted with medical experts and hence are realistic (and not synthetic ones). Another interesting aspect of this work is that the questions collected are time-sensitive in nature and the database also includes unanswerable questions from the poll results. This certainly advances the reliability and generalizability of the system.

It would be helpful for the authors to include a discussion o the limitations of this dataset and potential directions one can extend this dataset. The scope of the unanswerable questions is restrictive. Overall, though the work is extremely rigorous, the scope of this research is limited to a relatively restricted setting. Given the specific focus of the work on the healthcare domain, a poster presentation is recommended since the work will be interesting to only a subset of the audience.

confidence: 5: The area chair is absolutely certain
Decision: Accept