SlowGuess/NAACL_2021_OCR huggingface dataset:
NL-EDIT:
Correcting Semantic Parse Errors through Natural Language Interaction
Ahmed Elgohary*, Christopher Meek*, Matthew Richardson*, Adam Fourney*, Gonzalo Ramos*, Ahmed Hassan Awadallah*
$\star$  University of Maryland
elgohary@cs.umd.edu {meek,matri,adamfo,goramos,hassanam}@microsoft.com
We study semantic parsing in an interactive setting in which users correct errors with natural language feedback. We present NL-EDIT, a model for interpreting natural language feedback in the interaction context to generate a sequence of edits that can be applied to the initial parse to correct its errors. We show that NL-EDIT can boost the accuracy of existing text-to-SQL parsers by up to  $20\%$  with only one round of correction. We analyze the limitations of the model and discuss directions for improvement and evaluation. The code and datasets used in this paper are publicly available at http://aka.ms/NLEdit.
Major progress in natural language processing has been made towards fully automating challenging tasks such as question answering, translation, and summarization. On the other hand, several studies have argued that machine learning systems that can explain their own predictions (Doshi-Velez and Kim, 2017) and learn interactively from their end-users (Amershi et al., 2014) can result in better user experiences and more effective learning systems. We develop NL-EDIT—an approach that employs both explanations and interaction in the context of semantic parsing.
Most existing systems frame semantic parsing as a one-shot translation from a natural language question to the corresponding logical form (e.g., SQL query) (Yu et al., 2018a; Guo et al., 2019; Wang et al., 2020, inter alia). A growing body of recent work demonstrates that semantic parsing systems can be improved by including users in the parsing loop—giving them the affordance to examine the parses, judge their correctness, and provide feedback accordingly. The feedback often comes in the form of a binary correct/incorrect
Semantic Parsing:
Vot
Vote_ID	State	Candidate_ID
4235	NC	1
2355	CA	2
1894	PA	2
Candidates
Candidate_ID	First_Name	Last_Name
1	Donald	Trump
2	Joe	Biden
What is the full name of the candidate with the most votes?
SQL: SELECT first_name FROM candidates  
JOIN votes ON candidates.candidate_id = votes.candidate_id Group BY voter_Id  
ORDER BY COUNT(*) DESC LIMIT 1
Answer: Donald
Explanation:
1. For each vote id, find the number of rows in Votes
2. Find first_name with the largest value in the result of step 1
It should be candidate id in step 1. Also find last name in step 2.
SQL:SELECT first_name, last_name FROM candidates JOIN votes ON candidates.candidate_id = votes.candidate_id GROUP BY candidate_Id ORDER BY COUNT(\* GR DESC LIMIT 1
removevote_id
GROOP-BI: add candidate_id
SELECT:add last_name
Answer: Joe Biden
Explanation:
1. For each candidate id, find the number of rows in Votes
2. Find first name, last name with the largest value in the result of step 1
Figure 1: Example human interaction with NL-EDIT to correct an initial parse through natural language feedback. In the Semantic Parsing Phase (top), an off-the-shelf parser generates an initial SQL query and provides an answer paired with an explanation of the generated SQL. In the Correction Phase (bottom), the user reviews the explanation and provides feedback that describes how the explanation should be corrected. The system parses the feedback as a set of edits that are applied to the initial parse to generate a corrected SQL.
signal (Iyer et al., 2017), answers to a multiple-choice question posed by the system (Gur et al., 2018; Yao et al., 2019), or suggestions of edits that can be applied to the parse (Su et al., 2018).
Unlike other frameworks for interactive semantic parsing that typically expect users to judge the correctness of the execution result or induced logical form, Elgohary et al. (2020) introduced a framework for interactive text-to-SQL in which induced SQL queries are fully explained in natural lan
Association for Computational Linguistics: Human Language Technologies, pages 5599-5610
guage to users, who in turn, can correct such parses through natural language feedback (Figure 1). They construct the SPLASH dataset and use it to evaluate baselines for the semantic parse correction with natural language feedback task they introduce.
We present a detailed analysis of the feedback and the differences between the initial (incorrect) and the correct parse. We argue that a correction model should be able to interpret the feedback in the context of other elements of the interaction (the original question, the schema, and the explanation of the initial parse). We observe from SPLASH that most feedback utterances tend to describe a few edits that the user desires to apply to the initial parse. As such, we pose the correction task as a semantic parsing problem that aims to convert natural language feedback to a sequence of edits that can be deterministically applied to the initial parse to correct it. We use the edit-based modeling framework to show that we can effectively generate synthetic data to pre-train the correction model leading to clear performance gains.
We make the following contributions: (1) We present a scheme for representing SQL query Edits that benefits both the modeling and the analysis of the correction task, (2) we present NL-EDIT, an edit-based model for interactive text-to-SQL with natural language feedback. We show that NL-EDIT outperforms baselines in (Elgohary et al., 2020) by more than 16 points, (3) We demonstrate that we can generate synthetic data through the edit-based framing and that the model can effectively use this data to improve its accuracy and (4) We present a detailed analysis of the model performance including studying the effect of different components, generalization to errors of state-of-the-art parsers, and outline directions for future research.
In the task of text-to-SQL parsing, the objective is given a database schema (tables, columns, and primary-foreign key relations) and a natural language question, generate a SQL query that answers the question when executed against the database. Several recent text-to-SQL models have been introduced (Yu et al., 2018a; Zhang et al., 2019; Guo et al., 2019; Wang et al., 2020, inter alia) as a result of the availability of SPIDER (Yu et al., 2018b), a large dataset of schema, questions and gold parses spanning several databases in different domains.
The task of SQL parse correction with natural
language feedback (Elgohary et al., 2020) aims to correct an erroneous parse based on natural language feedback collected from the user. Given a question, a database schema, an incorrect initial parse, natural language feedback on the initial parse, the task is to generate a corrected parse.
To study this problem, Elgohary et al. (2020) introduced the SPLASH dataset. SPLASH was created by showing annotators questions and a natural language explanation of incorrect parses and asking them to provide feedback, in natural language, to correct the parse. The dataset contained 9,314 question-feedback pairs. Like the SPIDER dataset, it was split into train-dev-test sets by database to encourage the models to generalize to new unseen databases. They contrast the task with conversational semantic parsing (Suhr et al., 2018; Yu et al., 2019b,a; Andreas et al., 2020) and show that the two tasks are distinct and are addressing different aspects of utilizing context. They establish several baseline models and show that the task is challenging for state-of-the-art semantic parsing models. We use these as baselines for this work.
3 SQL Edits
We define a scheme for representing the edits required to transform one SQL query to another. We use that scheme both in our model and analysis. Our goal is to balance the granularity of the edits—too fine-grained edits result in complex structures that are challenging for models to learn, and too coarse-grained edits result in less compact structures that are harder for models to generate.
We view a SQL query as a set of clauses (e.g., SELECT, FROM, WHERE), each clause has a sequence of arguments (Figure 2). We mirror the SQL clauses SELECT, FROM, WHERE, GROUP-BY, ORDER-BY, HAVING, and LIMIT. For subqueries, we define a clause SUBS whose arguments are recursively defined as sets of clauses. Subqueries can be linked to the main query in two ways: either through an IEU clause (mirrors SQL INTERSECT/EXCEPT/UNION) whose first argument is one of the keywords INTERSECT, EXCEPT, UNION and its second argument is a pointer to a subquery in SUBS. The second is through nested queries where the arguments of some of the clauses (e.g., WHERE) can point at subqueries in SUBS (e.g., "id NOT IN  $\text{SUBS}_1$ ").
With such view of two queries  $\mathcal{P}_{source}$  and  $\mathcal{P}_{target}$ , we define their edit  $\mathcal{D}_{source\rightarrow target}$  as
Figure 2: Edit for transforming the source query "SELECT id, MAX(grade) FROM assignments WHERE grade > 20 AND id NOT IN (SELECT id from graduates) GROUP BY id" to the target "SELECT id, AVG(grade) FROM assignment WHERE grade > 20 GROUP BY id ORDER BY id". The source and target are represented as sets of clauses (left and middle). The set of edits and its linearized form (Section 4) are shown on the right. Removing the condition "id NOT IN SUBS<sub>1</sub>" makes the subquery unreferenced, hence pruned from the edit.
the set of clause-level edits  $\{\mathcal{D}_{source\rightarrow target}^c\}$  for all types of clauses  $c$  that appear in  $\mathcal{P}_{source}$  or  $\mathcal{P}_{target}$  (Figure 2). To compare two clauses of type  $c$ , we simply exact-match their arguments: unmatched arguments in the source (e.g., MAX(grade) in SELECT) are added as to remove arguments to the corresponding edit clause, and unmatched arguments in the target (e.g., "id" in the ORDER-BY) are added as to-add arguments.
Our current implementation follows SPIDER's assumption that the number of subqueries is at most one which implies that computing edits for different clauses can be done independently even for the clauses that reference a subquery (e.g., WHERE in Figure 2). The edit of the SUBS clause is recursively computed as the edit between two queries (any of them can be empty); the subquery of source and the subquery of target, i.e.,  $\mathcal{D}_{source\rightarrow target}^{SUBS} = \mathcal{D}_{source:SUBS_1\rightarrow target:SUBS_1}$ . We keep track of the edits to the arguments that reference the subquery. After all edit clauses are computed, we prune the edits of the SUBS clause if the subquery will no longer be referenced (SUBS $_1$  in Figure 2). We follow the SPIDER evaluation and discard the values in WHERE/HAVING clauses.
Throughout this paper, we refer to the number of add/remove operations in an edit as the Edit Size, and we denote it as  $|\mathcal{D}_{\text{source} \rightarrow \text{target}}|$ . For example, the edit in Figure 2 is of size four.
We follow the task description in Section 2: the inputs to the model are the elements of the interaction—question, schema, an initial parse  $\tilde{\mathcal{P}}$ , and feedback. The model predicts a corrected  $\bar{\mathcal{P}}$ . The gold parse  $\hat{\mathcal{P}}$  is available for training. Our model is based on integrating two key ideas in an encoder-decoder architecture. We start with a discussion of the intuitions behind the two ideas followed by the model details.
4.1 Intuitions
Interpreting feedback in context: The feedback is expected to link to all the other elements of the interaction (Figure 1). The feedback is provided in the context of the explanation of the initial parse, as a proxy to the parse itself. As such, the feedback tends to use the same terminology as the explanation. For example, the SQL explanations of (Elgohary et al., 2020) express "group by" in simple language "for each vote_id, find ..." As a result, human-provided feedback never uses "group by". We also notice that in several SPLASH examples, the feedback refers to particular steps in the explanation as in the examples in Figure 1. Unlike existing models (Elgohary et al., 2020), we replace the initial parse with its natural language explanation. Additionally, the feedback usually refers to columns/tables in the schema, and could often be ambiguous when examined in isolation. Such ambiguities can be usually resolved by relying on the context provided by the question. For example, "find last name" in Figure 1 is interpreted as "find last name besides first name" rather than "replace first name with last name" because the question asks for the "full name". Our first key idea is based on grounding the elements of the interaction by combining self-learned relations by transformer models (Vaswani et al., 2017) and hard-coded relations that we define according to the possible ways different elements can link to each other.
Feedback describes a set of edits: The difference between the erroneous parse and the correct one can mostly be described as a few edits that need to be applied to the initial parse to correct its errors (Section 7). Also, the feedback often only describes the edits to be made (Elgohary et al., 2020). As such, we can pose the task of correction with NL feedback as a semantic parsing task where we convert a natural language deception of
Figure 3: The Encoder of NL-EDIT grounds the feedback into the explanation, the question, and the schema by (1) passing the concatenation of their tokens through BERT, then (2) combining self-learned and hard-coded relations in a relation-aware transformer. Three types of relations (Interaction Relations) link the individual tokens of the inputs. Question-Schema and Schema-Schema relations are not shown.
the edits to a canonical form that can be applied deterministically to the initial parse to generate the corrected one. We train our model to generate SQL Edits (Section 3) rather than SQL queries.
Our encoder (Figure 3) starts with passing the concatenation of the feedback, explanation, question, and schema through BERT (Devlin et al., 2019). Following (Wang et al., 2020; Suhr et al., 2018; Scholak et al., 2020), we tokenize the column/table names and concatenate them in one sequence (Schema) starting with the tokens of the tables followed by the tokens of the columns. Then, we average the BERT embeddings of the tokens corresponding to each column (table) to obtain one representation for the column (table).
Wang et al. (2020) study the text-to-SQL problem using the SPIDER dataset and show the benefit of injecting preexisting relations within the schema (column exists in a table, primary foreign key), and between the question and schema items (column and table names) by: (1) name linking: link a question token to a column/table if the token and the item name match and (2) value linking: link a question token to a column if the token appears as a value under that column. To incorporate such relations in their model, they use the relation-aware self-attention formulation presented in (Shaw et al., 2018). The relation-aware transformer (Shaw et al., 2018) assigns a learned embedding for each relation type and combines such embeddings with the self-attention of the original transformer model (Vaswani et al., 2017): If a pre-existing relation  $r$  holds between two tokens, the embedding of  $r$  is added as a bias term to the self
attention computation between the two tokens.
In addition to those relations, we define a new set of relations that aim at contextualizing the feedback with respect to the other elements of the interaction in our setup: (1) [Feedback-Schema] We link the feedback to the schema the same way the question is linked to the schema via both name and value linking, (2) [Explanation-Schema] Columns and tables are mentioned with their exact names in the explanation. We link the explanation to the schema only through exact name matching, (3) [Feedback-Question] We use partial (at the lemma level) and exact matching to link tokens in the feedback and the question, (4) [Feedback-Explanation] We link tokens in the feedback to tokens in the explanation through partial and exact token matching. Since the feedback often refers to particular steps, we link the feedback tokens to explanation tokens that occur in steps that are referred to in the feedback with a separate relation type that indicates step reference in the feedback, and (5) [Explanation-Explanation] We link explanation tokens that occur within the same step. We use the same formulation of relation-aware self-attention as (Wang et al., 2020) and add the relation-aware layers on top of BERT to integrate all relations into the model (Figure 3).
Using a standard teacher-forced cross-entropy loss, we train our model to generate linearized SQL Edits (Figure 2). At training time, we compute the reference SQL Edit  $\mathcal{D}_{\tilde{\mathcal{P}}\rightarrow \hat{\mathcal{P}}}$  of the initial parse  $\tilde{\mathcal{P}}$  and the gold parse  $\hat{\mathcal{P}}$  (Section 3). Then we linearize  $\mathcal{D}_{\tilde{\mathcal{P}}\rightarrow \hat{\mathcal{P}}}$  by listing the clause edits in a fixed order (FROM, WHERE, GROUP-BY, ... etc.). The argument of each clause—representing one add or remove operation—is formatted as <CLAUSE> ADD/REMOVE ARG </CLAUSE>. We express SQL operators in ARG with natural language explanation as in (Elgohary et al., 2020). For example, the argument "AVG (grade)" is expressed as "average grade". At inference time, we generate a corrected parse  $\bar{\mathcal{P}}$  by applying the produced edit to the initial parse  $\tilde{\mathcal{P}}$ .
We use a standard transformer decoder that either generates tokens from the output vocab or copies columns and tables from the encoder output. Since all editing operations should be directed by the feedback, we tried splitting the attention to the encoder into two phases: First, we attend to the feedback only and update the decoder state
Table 1: Example SQL Editors with corresponding feedback templates. The synthesized feedback is reversing the edit applied to a correct SQL as our synthesis process starts with the gold SQL and reaches an initial SQL after applying the edit.
accordingly. Then, we use the updated decoder state to attend to the other inputs. With that, we only observed a marginal improvement of  $0.5\%$  in the accuracy. We conduct all our experiments with standard decoder-encoder attention and plan to investigate other attention patterns in the future.
5 Synthetic Feedback
In this section, we describe our process for automatically synthesizing additional examples for training the correction model. Recall that each example consists of a question about a given schema paired with a gold parse, an initial erroneous parse, and feedback. Starting with a seed of questions and their corresponding gold parses from SPIDER's training set (8,099 pairs), our synthesis process applies a sequence of SQL editing operations to the gold parse to reach an altered parse that we use as the initial parse (Algorithm 1).
By manually inspecting the edits (Section 3) we induce for the initial and gold parses in SPLASH training set, we define 26 SQL editors and pair each editor with their most frequent corresponding feedback template(s) (Examples in Table 1). We also associate each editor with a set of constraints that determines whether it can be applied to a given SQL query (e.g., the "Remove-Limit" editor can only be applied to a query that has a limit clause).
Algorithm 1 summarizes the synthesis process. We start by creating  $N$  (controls the size of the dataset) clones of each seed example. Elgohary et al. (2020)'s analysis of SPLASH shows that multiple mistakes might be present in the initial SQL, hence we allow our synthesis process to introduce up to four edits (randomly decided in line:4) to each clone  $p$ . For each editing step, we sample a feasible edit for the current parse (line:5) with man
ually set probabilities for each edit to balance the number of times each editor is applied in the final dataset. Applying an edit (line:6) involves sampling columns/tables from the current parse and/or the schema, sampling operators and values for altering conditions, and populating the corresponding feedback template. We combine the feedback of all the applied editors into one string and use it as the feedback of the synthesized example.
Setup: We conduct our experiments using SPLASH (Elghary et al., 2020) (Section 2) whose train, dev, and test sets are of sizes 7481, 871, and 962, respectively. Using our feedback synthesis process (Section 5), we generate 50,000 additional synthetic training examples. In our preliminary experiments, We found that training the model on the synthetic dataset first then continuing on SPLASH outperforms mixing the synthetic and real examples and training on both of them simultaneously. We train the model on the synthetic examples for 20,000 steps and continue training on the real examples until reaching 100,000 steps in total. We choose the best checkpoint based on the development set accuracy. We varied the number of training steps on the synthetic examples and 20,000 steps achieved the highest accuracy on the dev set.
We use BERT-base-uncased (Devlin et al., 2019) in all our experiments. We set the number of layers in the relational-aware transformer to eight (Wang et al., 2020) and the number of decoder layers to two. We train with batches of size 24. We use the Adam optimizer (Kingma and Ba, 2015) for training. We freeze BERT parameters during the first 5,000 warm-up steps and update the rest of the parameters with a linearly increasing learning rate from zero to  $5 \times 10^{-4}$ . Then, we linearly decrease the learning rates from  $5 \times 10^{-5}$  for BERT and
<sup>1</sup>We ensure there is no overlap between examples in the seed and the dev set of SPLASH.
Correction Acc. (%)	Edit ↓ (%)	Edit ↑ (%)	Progress (%)
Rule-based Re-ranking	16.63	38.35	32.81	-15.67
EditSQL+Feedback	25.16	47.44	23.51	7.71
NL-EDIT (Ours)	41.17	72.41	16.93	36.99
Oracle Re-ranking	36.38	34.69	1.04	31.22
Table 2: Comparing NL-EDIT to baselines in (Elgohary et al., 2020): Rule-based Re-ranking and Edit-SQL+Feedback and to the beam re-ranking upper-bound. Edit ↓ (Edit ↑) is the percentage of examples on which the number of edits/errors strictly decreased (increased). Progress is the average relative reduction in the number of edits (Section 6). Elgohary et al. (2020) estimate the upper-bound on the correction accuracy as  $81.5\%$ .
$5 \times 10^{-4}$  for the other parameters to zero. We use beam search with a beam of size 20 and take the top-ranked beam that results in a valid SQL after applying the inferred edit.
Evaluation: We follow (Elgohary et al., 2020) and use the correction accuracy as our main evaluation measure: each example in SPLASH test set contains an initial parse  $\hat{\mathcal{P}}$  and a gold parse  $\hat{\mathcal{P}}$ . With a predicted (corrected) parse by a correction model  $\bar{\mathcal{P}}$ , they compute the correction accuracy using the exact-set-match (Yu et al., 2018b) between  $\bar{\mathcal{P}}$  and  $\hat{\mathcal{P}}$  averaged over all test examples. While useful, correction accuracy also has limitations. It expects models to be able to fully correct an erroneous parse with only one utterance of feedback as such, it is defined in terms of the exact match between the corrected and the gold parse. We find (Table 2) that in several cases, models were still able to make progress by reducing the number of errors as measured by the edit size (Section 3) after correction. As such, we define another set of metrics to measure partial progress. We report (Edit  $\downarrow$  and Edit  $\uparrow$  in Table 2) the percentage of examples on which the size of the edit set strictly decreased/increased. To combine Edit  $\downarrow$  and Edit  $\uparrow$  in one measure and account for the relative reduction (increase) in the number of edits, we define
$$
\mathrm {P r o g r e s s} (\mathcal {S}) = \frac {1}{| S |} \sum_ {\tilde {\mathcal {P}}, \bar {\mathcal {P}}, \hat {\mathcal {P}} \in \mathcal {S}} \frac {| \mathcal {D} _ {\tilde {\mathcal {P}} \to \hat {\mathcal {P}}} | - | \mathcal {D} _ {\bar {\mathcal {P}} \to \hat {\mathcal {P}}} |}{| \mathcal {D} _ {\tilde {\mathcal {P}} \to \hat {\mathcal {P}}} |}.
$$