SlowGuess/NAACL_2021_OCR huggingface dataset:
Given a test set  $\mathcal{S}$ , the Progress of a correction model is computed as the average relative edit reduction between the initial parse  $\tilde{\mathcal{P}}$  and the gold parse  $\hat{\mathcal{P}}$  by predicting a correction  $\bar{\mathcal{P}}$  of  $\tilde{\mathcal{P}}$ . A perfect model that can fully correct all errors in the initial parse would achieve a 100% progress. A
model can have a negative progress (e.g., Rule-based re-ranking in Table 2) when it frequently predicts corrections with more errors than those in the initial parse. Unlike correction accuracy, Progress is more aligned with user experience in an interactive environment (Su et al., 2018) as it assigns partial credit for fixing a subset of the errors and also, it penalizes models that predict an even more erroneous parse after receiving feedback.
Results: We compare (Table 2) NL-EDIT to the two top-performing baselines in (Elgohary et al., 2020) and also to the beam re-ranking upper-bound they report. NL-EDIT significantly increases the correction accuracy over the top baseline (Edit-SQL+Feedback) by more than  $16\%$  and it also outperforms oracle re-ranking by around  $5\%$ . We also note that in  $72.4\%$  of the test examples, NL-EDIT was able to strictly reduce the number of errors in the initial parse (Edit  $\downarrow$ ) which potentially indicates a more positive user experience than the other models. NL-EDIT achieves  $37\%$  Progress which indicates faster convergence to the fully corrected parse than all the other models.
Following the same experimental setup in Section 6, we compare NL-EDIT to other variants with one ablated component at a time (Table 3). We ablate the feedback, the explanation, and the question from the encoder input. We also ablate the interaction relations (Section 4.2) that we incorporate in the relation-aware transformer module. We only ablate the new relations we introduce to model the interaction (shown in Figure 3), but we keep the Question-Schema and Schema-Schema relations introduced in (Wang et al., 2020). For each such variant, we train for 20,000 steps on the synthetic dataset then continue training on SPLASH until step 100,000. We also train an ablated variant that does not use the synthetic feedback where we
2The learning rate schedule is only dependent on the step number regardless of whether we are training on the synthetic data or SPLASH. We tried resetting the learning rates back to their maximum values after switching to SPLASH, but did not observe any improvement in accuracy.
(a) Feedback Length
(b) Explanation Length
(c) Reference Edit Size
(d) Transition of Edit Size
Figure 4: a-c: Breakdown of the correction accuracy on SPLASH test set by (a) feedback length, (b) explanation length, and (c) size of the reference edit (number of add or remove operations). The number of examples in each group is shown on top of the bars. d: Transitions in edit size after correction. For each edit size of the initial parse (rows), we show the distribution of the edit size after correction.
NL-EDIT	41.17
- Feedback	19.81
- Explanation	26.80
- Question	38.27
- Interaction Relations	35.35
- Synthetic Feedback	35.01
Table 3: Correction accuracy on SPLASH Test of NL-EDIT versus variants with one ablated component each.
train for 100,000 steps only on SPLASH. For all variants, we choose the checkpoint with the largest correction accuracy on the dev set and report the accuracy on the SPLASH test set.
The results in Table 3 confirm the effectiveness of each component in our model. We find that the model is able to correct  $19.8\%$  of the examples without the feedback. We noticed that the ablated-feedback model almost reaches that accuracy only after training on the synthetic data with very minor improvement  $(< 1\%)$  after training on SPLASH. Only using the question and the explanation, the model is able to learn about a set of systematic errors that parsers make and how they can be corrected (Gupta et al., 2017; Yin and Neubig, 2019).
7.2 Error Analysis
In Figure 4, we breakdown the correction accuracy by the feedback and explanation lengths (in number of tokens) and by the reference edit size (number of required edit operations to fully correct the initial parse). The accuracy drops significantly when the reference edit size exceeds two (Figure 4c), while it declines more gradually as the feedback and explanation increase in length. We manually (Examples in Table 4) inspected the examples with longer feedback than 24, and found that  $8\%$  of them the feedback is long because it describes how to rewrite the whole query rather than being lim
Long Feedback Not Describing an Edit:
"you should determine the major record format from the orchestra table and make sure it is arranged in ascending order of number of rows that appear for each major record format."
Long Feedback Describing an Edit:
"replace course id (both) with degree program id, first courses with student enrolment, course description with degree summary name, second courses with degree programs."
Table 4: Example long feedback that NL-EDIT struggles with. Top: The feedback describes a rewriting of the query rather than how to edit it. Bottom: The initial query has several errors and the feedback enumerates how to edit all of them.
ited to only the edits to be made. In the remaining  $92\%$ , the initial query had several errors (edit size of 5.5 on average) with the corresponding feedback enumerating all of them.
Figure 4d shows how the number of errors (measured in edit size) changes after correction. The figure shows that even for examples with a large number of errors (four and five), the model is still able to reduce the number of errors in most cases. We manually inspected the examples with only one error that the model failed to correct. We found  $15\%$  of them have either wrong or non-editing feedback and in  $29\%$  the model produced the correct edit but with additional irrelevant ones. The dominant source of error in the remaining examples is because of failures with linking the feedback to the schema (Examples in Table 5).
7.3 Cross-Parser Generalization
So far, we have been using SPLASH for both training and testing. The erroneous parses (and corresponding feedback) in SPLASH are based on the Seq2Struct parser (Shin, 2019). Recent progress
Adding extra edits:
Ques.: Which city and country is the Alton airport at?
Initial: SELECT City, Country FROM airports WHERE AirportName = 'Alton' AND Country = 'USA'
Feedback: remove "and country equals USA" phrase.
Predicted: <where> remove AirportName equals </where> <where> remove Country equals </where>
Gold: <where> remove AirportName equals </where>
Failing to link feedback and schema:
Ques.: What are the full names of all left handed players, in order of birth date?
Initial: SELECT first_name, last_name FROM players ORDER BY birth_date Asc
Feedback: make sure that player are left handed.
Predicted: <where> add birth_date equals </where>
Gold: <where> add hand equals </where>
Table 5: Example failure cases of NL-EDIT.
Figure 5: Distribution of Edit Size per example in SPLASH compared to the generalization test sets constructed based on/EditSQL,TaBERT,and RAT-SQL.
in model architectures (Wang et al., 2020) and pretraining (Yin et al., 2020; Yu et al., 2021a) has led to parsers that already outperform Seq2Struct by more than  $30\%$  in parsing accuracy. Here, we ask whether NL-EDIT that we train on SPLASH (and synthetic feedback) can generalize to parsing errors made by more recent parsers without additional parser-specific training data.
We follow the same crowdsourcing process used to construct SPLASH (Section 2) to collect three new test sets based on three recent text-to-SQL parsers: EditSQL (Zhang et al., 2019), TaBERT (Yin et al., 2020) and RAT-SQL (Wang et al., 2020). Following Elgohary et al. (2020), we run each parser on SPIDER dev set and only collect feedback for the examples with incorrect parses that can be explained using their SQL explanation
framework. Table 6 (Top) summarizes the three new test sets and compares them to SPLASH test set. We note that the four datasets are based on the same set of questions and databases (SPIDER dev).
Table 6 (Bottom) compares the parsing accuracy (measure by exact query match (Yu et al., 2018b)) of each parser when used by itself (No Interaction) to integrating it with NL-EDIT. We report both the accuracy on the examples provided to NL-EDIT (Error Correction) and the End-to-End accuracy on the full SPIDER dev set. NL-EDIT significantly boosts the accuracy of all parsers, but with a notable drop in the gains as the accuracy of the parser improves. To explain that, in Figure 5 we compare the distribution of reference edit size across the four datasets. The figure does not show any significant differences in the distributions that would lead to such a drop in accuracy gain. Likewise, the distributions of the feedback lengths are very similar (the mean is shown in Table 6). As parsers improve in accuracy, they tend to make most of their errors on complex SQL queries. Although the number of errors with each query does not significantly change (Figure 5), we hypothesize that localizing the errors in a complex initial parse, with a long explanation (Table 6), is the main generalization bottleneck that future work needs to address.
8 Related Work and Discussion
Natural language to SQL: Natural language interfaces to databases have been an active field of study for many years (Woods et al., 1972; Warren and Pereira, 1982; Popescu et al., 2003; Li and Jagadish, 2014). The development of new large scale datasets, such as WikiSQL (Zhong et al., 2017) and SPIDER (Yu et al., 2018b), has reignited the interest in this area with several new models introduced recently (Choi et al., 2020; Wang et al., 2020; Scholak et al., 2020). Another related line of work has focused on conversation semantic parsing, e.g. SParC (Yu et al., 2019b), CoSQL (Yu et al., 2019a), and SMCalFlow (Andreas et al., 2020), where parsers aim at modeling utterance sequentially and in context of previous utterances.
Interactive Semantic Parsing: Several previous studies have looked at the problem of improving semantic parser with feedback or human interactions (Clarke et al., 2010; Artzi and Zettlemoyer, 2013). Interactions are supported in multiple ways including binary correct/incorrect signal (Iyer et al., 2017), answers to a yes/no or a multiple-choice
<sup>3</sup>https://yale-lily.github.io/spider
Seq2Struct (SPLASH)	EditSQL	TaBERT	RAT-SQL
Correction Test Sets Summary
Number of Examples	962	330	267	208
Average Feedback Length	13.1	13.5	12.9	12.2
Average Explanation Length	26.4	28.3	32.2.9	34.0
Semantic Parsing Accuracy (%)
Error Correction	41.1	28.0	22.7	21.3
No Interaction	41.3	57.6	65.2	69.7
End-to-End	61.6	66.6	71.1	74.0
Δ w/ Interaction	+20.3	+8.9	+5.9	+4.3
Table 6: Evaluating the zero-shot generalization of NL-EDIT to different parsers (EditSQL, TaBERT, and RAT-SQL) after training on SPLASH that is constructed based on the Seq2Struct parser. Top: Summary of the dataset constructed based on each parser. Feedback and explanation length is the number of tokens. Bottom: The Error Correction accuracy on each test set and the end-to-end accuracy of each parser on the full SPIDER dev set with and without interaction.  $\Delta$  w/ Interaction is the gain in end-to-end accuracy with the interaction added.
question posed by the system (Yao et al., 2019; Gur et al., 2018) or suggestions of edits that can be applied to the parse (Su et al., 2018).
Yao et al. (2019) and Gur et al. (2018) ask yes/no and multiple-choice questions and use the answers in generating the pars. Elgohary et al. (2020) introduce SPLASH (Section 2), a dataset for correcting semantic parsing with natural language feedback. Using language as a medium for providing feedback enables the human to provide rich open-form feedback in their natural way of communication giving them control and flexibility specifying what is wrong and how it should be corrected. Our work uses SPLASH and proposes to pose the problem of semantic parse correction as a parser editing problem with natural language feedback input. This is also related to recent work on casting text generation (e.g. summarization, grammatical error correction, sentence splitting, etc.) as a text editing task (Malmi et al., 2019; Panthaplackel et al., 2020; Stahlberg and Kumar, 2020) where target texts are reconstructed from inputs using several edit operations.
Semantic Parsing with Synthetic Data: Semantic parsing systems have frequently used synthesized data to alleviate the challenge of labeled data scarcity. In their semantic parser overnight work, Wang et al. (2015) proposed a method for training semantic parsers quickly in a new domain using synthetic data. They generate logical forms and canonical utterances and then paraphrase the canonical utterances via crowd-sourcing. Several other approaches have demonstrated the benefit of adopting this approach to train semantic parsers in low-resource settings (Su et al., 2017; Zhong
et al., 2017; Cheng et al., 2018; Xu et al., 2020). Most recently, synthetic data was used to continue to pre-train language models for semantic parsing tasks (Herzig et al., 2020; Yu et al., 2021a,b). We build on this line work by showing that we can generate synthetic data automatically without human involvement to simulate edits between an erroneous parse and a correct one.
9 Conclusions and Future Work
We introduced a model, a data augmentation method, and analysis tools for correcting semantic parse errors in text-to-SQL through natural language feedback. Compared to previous models, our model improves the correction accuracy by  $16\%$  and boosts the end-to-end parsing accuracy by up to  $20\%$  with only one turn of feedback. Our work creates several avenues for future work: (1) improving the model by better modeling the interaction between the inputs and exploring different patterns for decoder-encoder attention, (2) evaluating existing methods for training with synthetic data (e.g., curriculum learning (Bengio et al., 2009)), (3) optimizing the correction model for better user experience using the progress measure we introduce, and (4) using the SQL edits scheme in other related tasks such as conversational text-to-SQL parsing.
This work has benefited greatly from discussions with Xiang Deng, Alex Polozov, Tao Yu, and Guoqing Zheng. We thank Pengcheng Yin for sharing TaBERT predictions before the official code release. We are very grateful to our reviewers for their insightful feedback and suggestions.
Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine, 35.
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics.
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the International Conference of Machine Learning.
Jianpeng Cheng, Siva Reddy, and Mirella Lapata. 2018. Building a neural semantic parser from a domain ontology. ArXiv, abs/1812.10037.
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. RYANSQL: Recursively applying sketch-based slot fillings for complex text-to-SQL in cross-domain databases. arXiv preprint arXiv:2004.03125.
James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world's response. In *Conference on Computational Natural Language Learning*.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics.
Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
Ahmed Elgohary, Saghar Hosseini, and Ahmed Hassan Awadallah. 2020. Speak to your parser: Interactive text-to-SQL with natural language feedback. In Proceedings of the Association for Computational Linguistics.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the Association for Computational Linguistics.
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. Deepfix: Fixing common c language errors by deep learning. In Association for the Advancement of Artificial Intelligence.
Izzeddin Gur, Semih Yavuz, Yu Su, and Xifeng Yan. 2018. DialSQL: Dialogue based structured query generation. In Proceedings of the Association for Computational Linguistics.
Jonathan Herzig, P. Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020. TAPAS: Weakly supervised table parsing via pretraining. In Proceedings of the Association for Computational Linguistics.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations.
Fei Li and HV Jagadish. 2014. Constructing an interactive natural language interface for relational databases. In Proceedings of the VLDB Endowment.
Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, tag, realize: High-precision text editing. In Proceedings of Empirical Methods in Natural Language Processing.
Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, and Raymond Mooney. 2020. Learning to update natural language comments based on code changes. In Proceedings of the Association for Computational Linguistics.
Ana-Maria Popescu, Oren Etzioni, and Henry Kautz. 2003. Towards a theory of natural language interfaces to databases. In International Conference on Intelligent User Interfaces.
Torsten Scholak, Raymond Li, Dzmitry Bahdanau, Harm de Vries, and Chris Pal. 2020. DuoRAT: Towards simpler text-to-SQL models. ArXiv preprint arXiv:2010.11119.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Conference of the North American Chapter of the Association for Computational Linguistics.
Richard Shin. 2019. Encoding database schemas with relation-aware self-attention for text-to-SQL parsers. arXiv preprint arXiv:1906.11790.
Felix Stahlberg and Shankar Kumar. 2020. Seq2Edits: Sequence transduction using span-level edit operations. In Proceedings of Empirical Methods in Natural Language Processing.
Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, P. Pantel, M. Gamon, and Mark J. Encarnacion. 2017. Building natural language interfaces to web APIs. In Proceedings of the ACM International Conference on Information and Knowledge Management.
Yu Su, Ahmed Hassan Awadallah, Miaosen Wang, and Ryen W White. 2018. Natural language interfaces with fine-grained user interaction: A case study on web APIs. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval.
Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to executable formal queries. In *Conference of the North American Chapter of the Association for Computational Linguistics*.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the Association for Computational Linguistics.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the Association for Computational Linguistics.
David H.D. Warren and Fernando C.N. Pereira. 1982. An efficient easily adaptable system for interpreting natural language queries. American Journal of Computational Linguistics, 8.
W. A. Woods, Ronald M Kaplan, and Bonnie L. Webber. 1972. The lunar sciences natural language information system: Final report. BBN Report 2378.
Silei Xu, Sina Semnani, Giovanni Campagna, and Monica Lam. 2020. AutoQA: From databases to QA semantic parsers with only synthetic training data. In Proceedings of Empirical Methods in Natural Language Processing.
Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based interactive semantic parsing: A unified framework and a text-to-SQL case study. In Proceedings of Empirical Methods in Natural Language Processing.
Pengcheng Yin and Graham Neubig. 2019. Reranking for neural semantic parsing. In Proceedings of the Association for Computational Linguistics.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2021a. GraPPa: Grammar-augmented pre-training for table semantic parsing. In Proceedings of the International Conference on Learning Representations.
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018a. SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task. In Proceedings of Empirical Methods in Natural Language Processing.
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. 2019a. CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases. In Proceedings of Empirical Methods in Natural Language Processing.
Tao Yu, Rui Zhang, Alex Polozov, Christopher Meek, and Ahmed Hassan Awadallah. 2021b. SCoRe: Pretraining for context representation in conversational semantic parsing. In Proceedings of the International Conference on Learning Representations.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of Empirical Methods in Natural Language Processing.
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Irene Li Heyang Er, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Vincent Zhang Jonathan Kraft, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. Sparc: Cross-domain semantic parsing in context. In Proceedings of the Association for Computational Linguistics.
Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. Editing-based sql query generation for cross-domain context-dependent questions. In Proceedings of Empirical Methods in Natural Language Processing.