SlowGuess/ACL_2020_OCR huggingface dataset:
Ahmed Elgohary*
elgohary@cs.umd.edu
Singhar Hosseini, Ahmed Hassan Awadallah
Microsoft Research, Redmond, WA
{sahoss, hassanam}@microsoft.com
We study the task of semantic parse correction with natural language feedback. Given a natural language utterance, most semantic parsing systems pose the problem as one-shot translation where the utterance is mapped to a corresponding logical form. In this paper, we investigate a more interactive scenario where humans can further interact with the system by providing free-form natural language feedback to correct the system when it generates an inaccurate interpretation of an initial utterance. We focus on natural language to SQL systems and construct, SPLASH, a dataset of utterances, incorrect SQL interpretations and the corresponding natural language feedback. We compare various reference models for the correction task and show that incorporating such a rich form of feedback can significantly improve the overall semantic parsing accuracy while retaining the flexibility of natural language interaction. While we estimated human correction accuracy is  $81.5\%$ , our best model achieves only  $25.1\%$ , which leaves a large gap for improvement in future research. SPLASH is publicly available at https://aka.ms/Splash_dataset.
Natural language interfaces (NLIs) have been the "holy grail" of natural language understating and human-computer interaction for decades (Woods et al., 1972; Codd, 1974; Hendrix et al., 1978; Zettlemoyer and Collins, 2005). However, early attempts in building NLIs to databases did not achieve the expected success due to limitations in language understanding capability, among other reasons (Androutsopoulos et al., 1995; Jones and Galliers, 1995). NLIs have been receiving increasing attention recently motivated by interest in developing virtual assistants, dialogue systems, and
Find all the locations whose names contain the word "film"
finding the Address of Locations table for which Location_Name contains "film"
770 Edd Lane Apt. 098
14034 Kohler Drive
Address is wrong. I want the name of the locations
finding the Location_Name of Locations table for which Location_Name contains "film"
Location Name
Film Castle
Figure 1: An example of human interaction with a Text-to-SQL system to correct the interpretation of an input utterance. The system generates an initial SQL parse, explains it in natural language, and displays the execution result. Then, the system uses the human-provided natural language feedback to correct the initial parse.
semantic parsing systems. NLIs to databases were at the forefront of this wave with several studies focusing on parsing natural language utterances into an executable SQL queries (Text-to-SQL parsing).
Most of the work addressing the Text-to-SQL problem (and semantic parsing in general) frames it as a one-shot mapping problem. We establish (Section 4.1) that the majority of parsing mistakes that recent neural text-to-SQL parsers make are minor. Hence, it is often feasible for humans to detect and suggest fixes for such mistakes. Su et al. (2018) make a similar observation about parsing text to API calls (Su et al., 2017) and show that parsing mistakes could be easily corrected if humans are afforded a means of providing precise feedback. Likewise, an input utterance might be under- or mis-specified, thus extra interactions may be required to generate the desired output similarly to query refinements in information retrieval systems (Dang and Croft, 2010).
*Most work was done while the first author was an intern at Microsoft Research.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2065-2077
Humans have the ability to learn new concepts or correct others based on natural language description or feedback. Similarly, previous work has explored how machines can learn from language in tasks such as playing games (Branavan et al., 2012), robot navigation (Karamcheti et al., 2017), concept learning (e.g., shape, size, etc.) classifiers (Srivastava et al., 2018), etc. Figure 1 shows an example of a text-to-SQL system that offers humans the affordance to provide feedback in natural language when the system misinterprets an input utterance. To enable this type of interactions, the system needs to: (1) provide an explanation of the underlying generated SQL, (2) provide a means for humans to provide feedback and (3) use the feedback, along with the original question, to come up with a more accurate interpretation.
In this work, we study the task of SQL parse correction with natural language feedback to enable text-to-SQL systems to seek and leverage human feedback to further improve the overall performance and user experience. Towards that goal, we make the following contributions: (1) we define the task of SQL parse correction with natural language feedback; (2) We create a framework for explaining SQL parse in natural language to allow text-to-SQL users (who may have a good idea of what kind of information resides on their databases but are not proficient in SQL Hendrix et al. (1978)) to provide feedback to correct inaccurate SQL parses; (3) we construct SPLASH—Semantic Parsing with Language Assistance from Humans—a new dataset of natural language questions that a recent neural text-to-SQL parser failed to generate correct interpretation for together with corresponding human-provided natural language feedback describing how the interpretation should be corrected; and (4) we establish several baseline models for the correction task and show that the task is challenging for state-of-the-art semantic parsing models.
We formally define the task of SQL parse correction with natural language feedback. Given a question  $q$ , a database schema  $s$ , a mispredicted parse  $p'$ , a natural language feedback  $f$  on  $p'$ , the task is to generate a corrected parse  $p$  (Figure 2). Following Yu et al. (2018),  $s$  is defined as the set of tables, columns in each table and the primary and foreign keys of each table.
Figure 2: An example from our SQL parse correction task (DB Name: cre主題 parked and Table Name: Locations). Given a question, initial predicted parse and natural language feedback on the predicted parse, the task is to predict a corrected parse that matches the gold parse.
Models are trained with tuples  $q$ ,  $s$ ,  $p'$ ,  $f$  and gold parse  $p$ . At evaluation time, a model takes as input tuples in the form  $q$ ,  $s$ ,  $p'$ ,  $f$  and hypothesizes a corrected parse  $\hat{p}$ . We compare  $\hat{p}$  and the gold parse  $p$  in terms of their exact set match (Yu et al., 2018) and report the average matching accuracy across the testing examples as the model's correction accuracy.
In this section, we describe our approach for collecting training data for the SQL parse correction task. We first generate pairs of natural language utterances and the corresponding erroneous SQL parses (Section 3.1). We then employ crowd workers (with no SQL knowledge) to provide feedback, in natural language, to correct the erroneous SQL (Section 3.3). To enable such workers to provide feedback, we show them an explanation of the generated SQL in natural language (Section 3.2). Finally, to improve the diversity of the natural language feedback, we ask a different set of annotators to paraphrase each feedback. We describe the process in detail in the remainder of this section.
3.1 Generating Questions and Incorrect SQL Pairs
We use the Spider dataset (Yu et al., 2018) as our source of questions. Spider has several advantages over other datasets. Compared to ATIS (Price,
SQL:
Step 1: Find the number of rows of each value of id in browser table.
Step 2: Find id, name of browser table with largest value in the results of step 1.
Figure 3: An example of a SQL query, the corresponding template and the generated explanation.
1990) and GeoQuery (Zelle and Mooney, 1996), Spider is much larger in scale (200 databases vs. one database, and thousands vs. hundreds of SQL parses). Compared to WikiSQL (Zhong et al., 2017), Spider questions require inducing parses of complex structures (requiring multiple tables, joining, nesting, etc.). Spider also adopts a cross-domain evaluation setup in which databases used at testing time are never seen at training time.
To generate erroneous SQL interpretations of questions in Spider, we opted for using the output of a text-to-SQL parser to ensure that our dataset reflects the actual distribution of errors that contemporary parsers make. This is a more realistic setup than artificially infusing errors in the gold SQL. We use the Seq2Struct parser (Shin, 2019)<sup>1</sup> to generate erroneous SQL interpretations. Seq2Struct combines grammar-based decoder of Yin and Neubig (2017) with a self-attention-based schema encoding and it reaches a parsing accuracy of  $42.94\%$  on the development set of Spider.<sup>2</sup>
Note that we make no explicit dependencies on the model used for this step and hence other models could be used as well (Section 6.3).
We train Seq2Struct on  $80\%$  of Spider's training set and apply it to the remaining  $20\%$ , keeping
only cases where the generated parses do not match the gold parse (we use the exact set match of Yu et al. (2018)). Following the by-database splitting scheme of Spider, we repeat the 80-20 training and evaluation process for three times with different examples in the evaluation set at each run. This results in 3,183 pairs of questions and an erroneous SQL interpretation. To further increase the size of the dataset, we also ignore the top prediction in the decoder beam<sup>3</sup> and use the following predictions. We only include cases where the difference in probability between the top and second to top SQLs is below a threshold of 0.2. The intuition here is that those are predictions that the model was about to make and hence represent errors that the model could have made. That adds 1,192 pairs to our dataset.
3.2 Explaining SQL
In one of the earliest work on natural language interfaces to databases, Hendrix et al. (1978) note that many business executives, government officials and other decision makers have a good idea of what kind of information residing on their databases. Yet to obtain an answer to a particular question, they cannot use the database themselves and instead need to employ the help of someone who can. As such, in order to support an interactive Text-to-SQL system, we need to be able to explain the incorrect generated SQL in a way that humans who are not proficient in SQL can understand.
We take a template-based approach to explain SQL queries in natural language. We define a template as follows: Given a SQL query, we replace literals, table and columns names and aggregation and comparison operations with generic placeholders. We also assume that all joins are inner joins (true for all Spider queries) whose join conditions are based on primary and foreign key equivalence (true for more than  $96\%$  of Spider queries). A query that consists of two subqueries combined with an intersection, union or except operations is split into two templates that are processed independently and we add an intersection/union/except part to the explanation at the end. We apply the same process to the limit operation—generate an explanation of the query without limit, then add a limit-related step at the end.
We select the most frequent 57 templates used in Spider training set which cover  $85\%$  of Spider
https://github.com/rshin/seq2struct
2When we started the dataset construction at the beginning of June 2019, we were able to achieve a parsing accuracy of  $41.30\%$  on Spider's development set which was the state-ofthe-art accuracy at the time. It is worth noting that unlike current state-of-the-art models, Seq2Struct does not use pretrained language models. It was further developed into a new model called RAT-SQL (Wang et al., 2020) which achieved a new state-of-the-art accuracy as of April 2020.
3We used a beam of size 20.
queries. For each SQL template, we wrote down a corresponding explanation template in the form of steps (e.g., join step, aggregation step, selection step) that we populate for each query. Figure 3 shows an example of a SQL queries, its corresponding template and generated explanations. We also implemented a set of rules for compressing the steps based on SQL semantics. For instance, an ordering step following by a "limit 1" is replaced with "find largest/smallest" where "largest" or "smallest" is decided according to the ordering direction.
3.3 Crowdsourcing Feedback
We use an internal crowd-sourcing platform similar to Amazon Mechanical Turk to recruit annotators. Annotators are only selected based on their performance on other crowd-sourcing tasks and command of English. Before working on the task, annotators go through a brief set of guidelines explaining the task. We collect the dataset in batches of around 500 examples each. After each batch is completed, we manually review a sample of the examples submitted by each annotator and exclude those who do not provide accurate inputs from the annotators pool and redo all their annotations.
Annotators are shown the original question, the explanation of the generated SQL and asked to: (1) decide whether the generated SQL satisfies the information need in the question and (2) if not, then provide feedback in natural language. The first step is necessary since it helps identifying false negative parses (e.g., another correct parse that does not match the gold parse provided in Spider). We also use the annotations of that step to assess the extent to which our interface enables target users to interact with the underlying system. As per our assumption that target users are familiar with the kind of information that is in the database (Hendrix et al., 1978), we show the annotators an overview of the tables in the database corresponding to the question that includes the table and column names together with examples (first 2 rows) of the content. We limit the maximum feedback length to 15 tokens to encourage annotators to provide a correcting feedback based on the initial parse (that focuses on the edit to be made) rather than describing how the question should be answered.
A total of 10 annotators participated in this task. They were compensated based on an hourly rate
Number of	Train	Dev	Test
Examples	7,481	871	962
Databases	111	9	20
Uniq. Questions	2,775	290	506
Uniq. Wrong Parsing	2,840	383	325
Uniq. Gold Parsing	1,781	305	194
Uniq. Feedbacks	7,350	860	948
Feedback tokens (Avg.)	13.9	13.8	13.1
Table 1: SPLASH summary
(as opposed to per annotation) to encourage them to optimize for quality and not quantity. They took an average of 6 minutes per annotation.
To improve the diversity of the feedback we collect, we ask a separate set of annotators to generate a paraphrase of each feedback utterance. We follow the same annotators quality control measures as in the feedback collection task. An example instance from the dataset is shown in Figure 2.
3.4 Dataset Summary
Overall, we ask the annotators to annotate 5409 example (427 of which had the correct SQL parse and the remaining had an incorrect SQL parse). Examples with correct parse are included to test whether the annotators are able to identify correct SQL parses given their explanation and the question. Annotators are able to identify the correct parses as correct  $96.4\%$  of the time. For the examples whose predicted SQL did not match the gold SQL, annotators still marked 279 of them as correct. Upon manual examinations, we found that annotators were indeed correct in doing so  $95.5\%$  of the time. Even though the predicted and gold SQLs did not match exactly, they were equivalent (e.g., 'price between 10 and  $20^{\prime}$  vs. 'price  $\geq 10$  and price  $\leq 20^{\prime}$ ).
After paraphrasing, we ended up with 9,314 question-feedback pairs, 8352 of which correspond to questions in the Spider training split and 962 from the spider development split. We use all the data from the Spider development split as our test data. We hold out  $10\%$  of the remaining data (split by database) to use as our development set and use the rest as the training set. Table 1 provides a summary of the final dataset.
We conduct a more thorough analysis of SPLASH in this section. We study the characteristics of the mistakes made by the parser as well as characteristics of the natural language feedback.
4We provide the data collection instructions and a screenshot of the data collection interface in the appendix.
4.1 Error Characteristics
We start by characterizing the nature of errors usually made by the models in parsing the original utterance to SQL. To understand the relation between the gold and the predicted SQL, we measure the edit distance between them for all cases for which the model made a mistake in the SQL prediction. We measure the edit distance by the number of edit segments (delete, insert, replace) between both parses. We find the minimal sequence of token-level edits using the levenshtein distance algorithm. Then, we combine edits of the same type (delete, insert, replace) applied to consecutive positions in the predicted parse in one segment. Figure 4 shows a frequency histogram of different values of edit distance. We observe that most inaccurate predictions lie within a short distance from the correct SQL  $(78\% +$  within a distance of 3 or less).
In addition to the number of edits, we also characterize the types of edits needed to convert the predicted SQL to the gold one. Our edit distance calculations support three operations replace, insert and delete. Those correspond to  $58\%$ ,  $31\%$  and  $11\%$  of the edit operations respectively. Most of the edits are rather simple and require replacing, inserting or deleting a single token (68.1% of the edits). The vast majority of those correspond to editing a schema item (table or column name):  $89.2\%$ , a SQL keyword (e.g., order direction, aggregation, count, distinct, etc.):  $7.4\%$ , an operator (greater than, less than, etc.):  $2.2\%$  or a number (e.g. for a limit operator):  $1.2\%$ .
The edits between the predicted and generated SQL spanned multiple SQL keywords. The distribution of different SQL keywords appearing in edits and their distribution across edit types (replace, insert or delete) is shown in Figure 5. Note that a single edit could involve multiple keywords (e.g., multiple joins, a join and a where clause, etc.). Interestingly, many of the edits involve a join highlighting that handling utterances that require a join is harder and more error prone. Following join, most edits involve where clauses (making the query more or less specific), aggregation operators, counting and selecting unique values.
The error analysis demonstrates that many of the errors made by the model are in fact not significant and hence it is reasonable to seek human feedback to correct them.
Figure 4: A histogram of the distance between the gold and the predicted SQL.
Figure 5: A histogram of different SQL keywords appearing in edits (between the gold and predicted SQL) and their distribution across edit types (replace, insert or delete).
4.2 Feedback Characteristics
To better understand the different types of feedback our annotators provided, we sample 200 examples from the dataset, and annotate them with the type of the feedback. We start by assigning the feedback to one of three categories: (1) Complete: the feedback fully describes how the predicted SQL can be corrected, (2) Partial: the feedback describes a way to correct the predicted SQL but only partially and (3) Paraphrase: the feedback is a paraphrase of the original question. The sample had  $81.5\%$  for Complete,  $13.5\%$  for Partial and  $5.0\%$  for Paraphrase feedback. Examples of each type of feedback are shown in Table 2. Upon further inspection of the partial and paraphrase feedback, we observe that they mostly happen when the distance between the predicted and gold SQL is high (major parsing errors). As such, annotators opt for providing partial feedback (that would at least correct some of the mistakes) or decide to rewrite the question in a different way.
We also annotate and present the types of feedback, in terms of changes the feedback is suggesting, in Table 3. Note that the same feedback may suggest multiple changes at the same time. The
Complete Feedback: [81.5%]
Question:	Show the types of schools that have two schools.
Pred. SQL:	SELECT TYPE FROM school GROUP BY TYPE HAVING count(*) >= 2
Feedback:	You should not use greater than.
Partial Feedback: [13.5%]
Question:	What are the names of all races held between 2009 and 2011?
Pred. SQL:	SELECT country FROM circuits WHERE lat BETWEEN 2009 AND 2011
Feedback:	You should use races table.
Paraphrase Feedback: [5.0%]
Question:	What zip codes have a station with a max temperature greater than or equal to 80 and when did it reach that temperature?
Pred. SQL:	SELECT zip_code FROM weather WHERE min_temperature_f > 80 OR min_sea_level_pressure_inches > 80
Feedback:	Find date,zip code whose max temperature f greater than or equals 80.
Table 2: Examples (question, predicted SQL and feedback) of complete, partial and paraphrase feedback
table shows that the feedback covers a broad range of types, which matches our initial analysis of error types. We find that a majority of feedback is referencing the retrieved information. In many such cases, the correct information has not been retrieved because the corresponding table was not used in the query. This typically corresponds to a missing inner one-to-one join operation and agrees with our earlier analysis on edit distance between the gold and predicted SQL. The second most popular category is incorrect conditions or filters followed by aggregation and ordering errors. We split the first two categories by whether the information/conditions are missing, need to be replaced or need to be removed. We observe that most of the time the information or condition needs to be replaced. This is followed by missing information that needs to be inserted and then unnecessary ones that need to be removed.
We heuristically identify feedback patterns for each collected feedback. To identify the feedback pattern, we first locate the central predicate in the feedback sentence using a semantic role labeler (He et al., 2015). Since some feedback sentences can be broken into multiple sentence fragments, a single feedback may contain more than one central predicate. For each predicate, we identify its main arguments. We represent every argument with its first non stopword token. To reduce the vocabulary, we heuristically identify column mentions and replace them with the token 'item'.
We visualize the distribution of feedback patterns for the top 60 most frequent patterns in Figure 6, and label the ones shared among multiple patterns. As is shown, our dataset covers a diverse variety of feedback patterns centered around key operations to edit the predicted SQL that reference
Figure 6: Patterns of feedback covered in our dataset. Patterns are extracted heuristically using predicates and arguments extracted from the feedback sentence. The figure shows the top 60 frequent patterns.
operations, column names and values.
Our work is linked to multiple existing research lines including semantic parsing, learning through interaction (Li et al., 2017a; Hancock et al., 2019; Li et al., 2017b, inter alia) and learning from natural language supervision (Srivastava et al., 2017; Co-Reyes et al., 2019; Srivastava et al., 2018; Hancock et al., 2018; Ling and Fidler, 2017, inter alia). We discuss connections to the most relevant works.
Text-to-SQL Parsing: Natural language to SQL (natural language interfaces to databases) has been an active field of study for several decades (Woods et al., 1972; Hendrix et al., 1978; Warren and Pereira, 1982; Popescu et al., 2003; Li
Feedback Type	%	Example
Information
- Missing	13%	I also need the number of different services
- Wrong	36%	Return capacity in place of height
- Unnecessary	4%	No need to return email address
Conditions
- Missing	10%	ensure they are FDA approved
- Wrong	19%	need to filter on open year not register year
- Unnecessary	7%	return results for all majors
Aggregation	6%	I wanted the smallest ones not the largest
Order/Uniq	5%	only return unique values