Did You Mean...? Confidence-based Trade-offs in Semantic Parsing
Authors: Elias Stengel-Eskin
Benjamin Van Durme
Venue: EMNLP 2023 Main (2023)
Keywords: calibration
semantic parsing
safety
paraphrasing
TL;DR: We show how model confidence can be used to balance trade-offs in semantic parsing, both for improving safety in user-facing applications and for reducing annotation load.

Abstract: We illustrate how a calibrated model can help balance common trade-offs in task-oriented parsing. 
In a simulated annotator-in-the-loop experiment, we show that well-calibrated confidence scores allow us to balance cost with annotator load, improving accuracy with a small number of interactions. 
We then examine how confidence scores can help optimize the trade-off between usability and safety. 
We show that confidence-based thresholding can substantially reduce the number of incorrect low-confidence programs executed; however, this comes at a cost to usability.  
We propose the DidYouMean system which better balances usability and safety by rephrasing low-confidence inputs.

Did You Mean...? Confidence-based Trade-offs in Semantic Parsing
Elias Stengel-Eskin' UNC Chapel Hill est estenggcs.... unc. edu
Benjamin Van Durme Johns Hopkins University
vandurme jhu. edu
Abstract
Model Prob.
We illustrate how a calibrated model can help balance common trade-offs task-oriented parsing. In simulated annotator-in-the-loy experiment, we show that well-calibrated con- fidence scores allow us balance cost with an- notator load, improving accuracy with small number of interactions. We then examine how confidence scores can help optimize the trade- off between usability and safety. We show that confidence-based thresholding can sub- stantially reduce the number of incorrect low- confidence programs executed; however, this comes a cost to usability. We propose the Yea system (cf. Fig. I) which bet- ter balances usability and safety by rephrasing low-confidence inputs.
Accuracy
Do have anything going on tonight?
Prediction:
(Yield (< (<<(EventOn thwinekagee..........
Execute
Confi
p(correct
Do have any events that end at the end of the day Saturday ?
Prodiction: (Vield (< (-EEventonoottaafaarTiee -}33)
Con/Lsdence-0.eu
Gloss and confirm
pi correeccj=a..1
accape
Execute
Did you mean:
1 Introduction
reject
Retry
Do I have anything Saturday that ends the end the day?
Task-oriented dialogue systems (Gupta al., 2018; Cheng et al., 2020; Semantic Machines et al., 2020) represent one path towards achieving the long- standing goal of using natural language as an API for controlling real-world systems by transforming user requests into executable programs, i.c. trans- lating natural language to code. Central to the systems' success is the ability to take rational ac- tions under uncertainty (Russell and Norvig, 2010). When model confidence is low and the system is un- likely to succeed, we would prefer it defer actions and request clarification, while at high confidence, clarification requests may annoy user. Relying on model confidence requires it to be well-correlated with accuracy, i.c. it requires a calibrated model.
The DidYouuean system. At high confi- dences, simply execute the predicted parse. At low confidences, DidYouMean rephrases the query based on the predicted program and asks user confirm the paraphrase. The program is executed if the user accepts.
Figure I:
models, we first examine how they could be used in annotation interface, with view balancing the trade-off between annotation cost and correct- ness. We simulate a human-in in-the-loop human--n-the-llog (HITL) experiment where high-confidence tokens are au- tomatically annotated and low-confidence tokens trigger dialogue with an oracle annotator who either picks the correct token from top-K list or manually inserts it. With small number of inter- actions substantially boost annotator accuracy.
Recent work has focused on the calibration of se- mantic parsing models. Specifically, Beengel-Eskinng and Van Durme (2022) benchmarked the calibra- tion characteristics of a variety of semantic parsing models, finding some of them to be well-calibrated, especially on parsing for task-oriented dialogue. Given the relatively cell-calibrated nature of these "Work done while at Johns Hopkins University.
A similar trade-off exists between usability and safety in task-oriented user interfaces. We exam- ine how sequence-level model confidence scores can be used to balance this trade-off by reducing the number of incorrect programs executed while also minimizing the number of follow-up user in- teractions and their cognitive burden. We find that thresholding outputs based on model confidence

---

(i.e. rejecting outputs falling below a tuned thresh- old) reduces the number of incorrect programs exe- cuted by 76% compared to the baseline. However, this comes at a cost to usability, as roughly half the correctly-prediceed parses are also rejected. To strike a balance between safety and usability, we introduce the YiuMonn system (ef. Fig- 1), which rephrases the input conditioned on the pre- dicted parse and asks users to confirm the accuracy of the paraphrase. In a user study, we obtain an 36% improvement in usability over the thresholded system while maintaining a 58% reduction in the number incorrect programs executed.
Models We use MISO (Zhang et al., 2019a.b), well-calibrated model from Stengel-Eskin and Van Durme (2022). Rather than predict the SM- CalFlow surface form, including syntactic tokens like parentheses, MISO directly predicts the under- lying execution graph. The graph can deterministi- cally be "de-compiled" into its surface form, and vere-versa. The fact that MISO predicts an underly- ing graph makes it attractive for applications which require confidence scores, as only predicts con- tent tokens (i.e. functions, arguments) rather than structure markers, like parentheses. For details on MISO's architecture, see Zhang et al. (2019b) and Stengel-Eskin et al. (2022).
2 Related Work
For token confidence estimation, we use the maximum probability across the output vocabu- lary at each timestep. This has been shown be relatively robust confidence estimator classi- fication (Hendrycks and Gimpel, 2016; Varshney et al., 2022). For sequence-level scores, we follow Stengel-Eskin and Van Durme (2022) and take the minimum over token-level confidence scores.
Our experiments in Section 4 involve predictive model for human-in-the -a-tn--i-th---oyy coding: similar mod- els have been integrated into IDEs, e.g. Chen et al. (2021). Yia relates to the interactive se- mantic parsing domain (Li and Jagadish, 2014; Chaurasia and Mooney, 2017; Su et al., 2018), where humans are included in the semantic parsing loop. In this domain, Yao et al. (2019) introduce a confidence-basee interactive system in which a parsing agent can ask users for clarification. Our work follows in this spirit, but asks the user to confirm parse rather than generating questions for the user answer; we also operate in dif- ferent parsing domain, preventing us from using Yao et al.'s is system as a baseline. DidYouMean also relates broadly to selective prediction, where a model expected to abstain from making de- cisions at low confidence (Chow, 1957; El-Yaniv et al., 2010; Varshney et al., 2022; Xin et al., 2021; Whitehead et al., 2022). Our system extends selec- tive prediction's 's setting by including human-in- the-loop. Finally, YouMan shares a motivation with Fang et al. (2022), who introduce a method for reliably summarizing programs. Their work pro- vides post-hoe hoc action explanations while we focus on resolving misunderstandings before execution.
4 Human-in-the-Loop Simulation
Production datasets like SMCalFlow are constantly evolving as new functionalities are added. The ex- pensive and consuming nature of annotating data can be mitigated by the use of predictive pars- ing models which suggest speculative parses for new utterances. However, the model's 's output can be incorrect, especially given distr--ditti inputs. We need to ensure that annotators are not introducing errors by overly trusting the model. If the model is well-calibrated, we can use its confidence to reduce such errors. For example, we can alert annotators to low confidence predictions and ask them to intervene (Lewis and Gale, 1994). Using threshold, we can prioritize time or cor- rectness: higher threshold would result in more annotator-model interactions, decreasing the speed but increasing program correctness (reducing the need for debugging) while lower threshold would increase speed but also lower the accuracy.
3 Methods
Since we do not have access to expert SM- CalFlow annotators, we simulate an oracle human- in-the-loop (HITL. annotator who always provides correct answer by using the gold annotations pro- vided in the dataset. Specifically, for given input, we decode the output tokens of a predicted pro- gram 0g ... On, normally as long as predictions are confident (above a given threshold). If at time the confidence p(o:) falls the threshold, we attempt to
Datasets Our data is drawn from the SMCalFlow (Semantic Machines et al., 2020) trik-trie dia- logue dataset, which contains Lisp-like programs (cf. Appendix A). We follow the same prepro- cessing as Platanios al. (2021), and use the SM- CalFlow data splits given by Roy et al. (2022): 108,753 753 training, 12,271 validation, and 13,496 testing dialogue turns.

---

match the decoded prefix 00....... Or prefix 9o..... 91 If the prefixes do not match, we count the example as incorrect. If they do match, we replace Or with Gt. the gold prediction from our oracle annotator, and continue decoding. We con- sider three metrics in this experiment: (1) The exact match accuracy of the decoded programs (higher is better). (2) The percentage of total tokens for which we have to query an annotator (lower is bet- ter, as each query increases annotator load). (3) The percentage of uncertain tokens (below the thresh- old) for which the gold token gt is in the top predictions at timestep t. Here, higher is better, as selecting a token from a candidate list is typically faster than producing the token.
the gold
(2021); Lynch al. (2022); Nair et al. (2022)). In this setting, actions may have irreversible conse- quences, so determining safety before execution is key. Safety considerations need to be balanced with usability of the system: an unplugged agent would be very safe but unusable. To increase us- ability, an agent might make follow-up requests to user, like asking for clarification or confirmation. The types of requests the agent makes have varying cognitive load on the user: for example, providing confirmation takes less effort than rephrasing.
We measure how well we can reject incorrect programs before executing them. Following past work in selective prediction (EI- Yaniv et al., 2010; Varshney et al., 2022, i.a.), we measure success by coverage and risk, as well as F score w.r.t. program correctness. Coverage is the percentage of inputs for which program is executed and risk is the per- centage of executed programs that were incorrect. Precision is inverse risk, and recall the percent- age of correct programs which were accepted. addition to F1, we consider FO.5, which upweights precision (safety) by a factor of 2. A low-coverage, low-risk system may be safer but have more false negatives, i.e. reject more correct programs, de- creasing its usability. A high-coverage, high-risk system is more usable the cost of false positives, i.e. executing incorrect programs. We do not com- mit to setting an optimal threshold for this trade-ofi since it is task-specific.
Results and Analysis Fig. shows our three met- rics as the threshold is increased in increments of 0.1. We see that accuracy grows exponentially with a higher threshold, and that the percentage of tokens for which an annotator intervention is required grows at roughly the same rate. The ex- ponential growth reflects the distribution of token confidences, with most tokens having high confi- dence. Finally, we see that while at low confidence, most tokens must be manually inserted, the rate which they are chosen from the top 5 list rapidly increases with the threshold. Thus, the increased number of annotator interactions required at higher thresholds may be offset by the fact that many of these interactions are choice from the top-5 list.
Exact Match
% User Interactions
% Top 5
90.0
We consider systems. As a baseline, we con- sider a system that exccutes everything predicts (accept); this will result in the erage, but also high risk. We can also use MISO's calibrated nature to improve safety outcomes by tuning a sequence-level confidence threshold for rejecting programs (tuned). We tune the full val- idation set using Fl; we explore the range [0.0, 1.0) in increments of 0.01, finding 0.40 to be optimal. Finally, we introduce the YouMann system for filtering low-confidence programs. For a given utterance, DidYouuean shows the user a para- phrase of the input; the user then decides to accept the parse based on this paraphrase. This allows perrectly-predicee
75-
87.5
50
85.0
cov-
82.5
80.0
0
0.1
0.5
0.9
0.1
0.9
0.1
0.5
Threshold
Figure 2: Simulated ponpaator-inthe-lloooo results across
increasing confidence thresholds.
5 User Correction via DidYouMean
Section 4 showed that token-level confidence scores can be used to balance speed and correctness in an annotation interface. We see a similar trade- off between safety and usability in user interfaces using semantic parsing models. Here, we define safety rejecting unsuccessful programs before executing them. This strict definition is motivated by physical domains: imagine that rather than con- trolling digital assistant, user guiding a robot via language commands (c.g. Winograd (1972); Lynch and Sermanet (2020); Stengel-Eskin et al.
low-confidence programs to be accepted and executed, while reducing the user load: making a binary choice to accept paraphrase is a receptive task, while rephrasing an instruction is a more costly productive task. Details of DidY- ouMean's 's components are given below.

---

Glossing Model Since users are typically unfa- miliar with formats like Lisp, we need present the user with a natural language paraphrase - or gloss - of the candidate parse. To train a gloss- ing model, we modify Roy et al. (2022)'s seq2seq BenchCLAMP semantic parsing framework: rather than using the user utterance with the previous turn's 's context (Lo, (1) as input and program take the context and program (Mo, P) as the input and user instruction U4 as the output. We use BART-large (Lewis et al., 2020) and fine-tune it on the SMCalFlow training split. See Appendix A for model details.
Setting
Cov.
Risk 4
FP
FTT
R0.5 1
Accept
1.00
0.67
67
0.50
0.38
Tuned
0.32
0.50
16
0.49
0.50
Chosen
0.68
0.54
37
0.61
0.51
Re- Re-parsed
0.68
0.41
28
0.66
0.62
Table I: Coverage, risk, number of false positives (FP),
and F measures for accepting correct parses and reject- ing incorrect parses.
P output,
After majority voting, annotators accepted 68/100 glosses and rejected 32.
Results Table 1 shows the results of the user study. In addition to standard selective prediction metrics like coverage (the percentage of inputs for which program is executed) and risk (the per- centage of executed programs that are incorrect) we report the number of false positives (incorrect programs executed) and Fl and FO.5 scores. Tun- ing a threshold yiclds better safety outcomes than accepting everything, with lower risk. However, this safety comes at cost to the usability of the system; a coverage of only 0.32 indicates that only 32% of inputs have their programs executed. The "tuned" system's low usability is reflected in the Fl and FO.5 scores, which balance precision and recall, The "chosen" system, while better in F1, com- parable to the "tuned" system FO.5, which takes both usability and safety into account but priori- tizes safety at a 2:1 ratio. Users are able to recover some usability (as measured by coverage) in this setting but also add to the risk, which is higher for "chosen" than "tuned". The number of incorrect programs executed increases when glosses are cho- sen (as compared to the tuned threshold). When the accepted glosses are re- re-parsed, we see a shift back towards a system favoring safety, with fewer incorrect programs being executed than in the "cho- sen" setting; this is reflected in lower risk score. For both Fl and FO.5, the "re-parsed"" system best balances usability and safety.
YouMoune System When a low-confidence parse is detected, YidMou triggers dialogue with the user in order to recover some usability over simply rejecting all low-confidence parses. Fig. I shows the system workflow. Did shows the original utterance Ui and the gloss 2i to the user, who determines whether they are identical or not. If they accept the gloss, optionally re-parse the gloss it rather than the original utterance U4; this can remove typos and other idiosyncracies. We call this the re-parsed setting, while choosing the original prediction L7 is the chosen setting. We pre- dict that allowing users to accept and reject glosses will improve the balance between safety and usabil- ity (i.e. F1) over the threshold system by allowing them to accept correct low-confidence parses. In other words, adding human interaction will allow us achieve balance which cannot be attained given the tradeoffs resulting from thresholding.
User Study We conduct a static user study of DidYouMean with examples from the SMCalFlow validation set. We sample 100 MISO predictions with confidence below 0.6 (to ensure that the set contains a number of mistakes). This sample is stratified across 10 equally-spaced bins with 10 samples per bin. MTurk annotators were shown the dialogue history, the user utterance, and the gloss, and asked to confirm that the gloss matched the utterance's ' intent. The template and instructions can be seen in Appendix B. We obtained three judg- ments from three different annotators per example.
These results show that a calibrated model can be used with a threshold to greatly improve safety, re- ducing the number of incorrect programs accepted by 76%. DidYouMean allows users to recover some low-confidence programs by accepting and rejecting programs based on their glosses, result- ing in the best aggregated scores. Note also that the threshold was tuned on Fl score on the en- tire dev set. This means that the Fl performance of that tuned system is as high as possible for meafidence-threshodd-eee
Annotation Statisties In total eight annotators participated in the task, with four completing the majority of tasks. For each example, all three an- notators agreed on 79% examples, indicating the task is forl-lorm For the remaining 21%, we use the majority decision to accept or reject.
system. Thus, DidY- ouMean achieves a balance outside what can be

---

achieved by tuning: simply increasing the thresh- old would decrease safety and result in lower F! than the current threshold of 0.40.
Human correction via choosing
10
Human-correctee
Nocleus
Exaet Match Accuracy
0.8
6 Selection Study
0.6
In Section 5 we presented the YouMann sys- tem, which allowed users to confirm reject potential gloss. The gloss was chosen from larger set of candidates; this naturally raises the ques- tion of whether users can directly choose a gloss from the candidate list, rather than accepting or rejecting single gloss at a time. Here, we exam- ine to what extent it is helpful to users to choose glosses from list of options. We take sample of validation programs for SMCalFlow, stratified across 10 evenly-spaced confidence bins from 0.0 1.0. Note that this differs from the setting in Section 5, where the maximum bin was 0.6. We then present the top predictions decoded with nu- cleus sampling (Holtzman et al., 20;9); the number of predictions depends on the confidence of the model. Specifically, we set cutoff of 0.85 and add predictions until the sum of the sequence-level con- fidence scores is greater than the cutoff; this could be achieved by a single very confident prediction multiple low-confidence predictions. To reduce the load on the annotators, we limit the number of predictions seen to a max of 10, even if the cutoff is not reached.
0.2
0.4
0.6
0.8
Confidence
Figure Selection experiment. Annotators sometimes select accurate glosses, but often have trouble deciding between seemingly invariant glosses, lowering perfor- mance especially at medium confidences.
to lower performance. Qualitatively, many incor- rect choices are driven by glosses that appear to be paraphrases, but in fact correspond to subtly dif- ferent Lisp programs. These results suggest that accepting/rejecting glosses more promising than choosing them.
7 Conclusion
We examine two common trade-offs semantic parsing, and how a well-calibrated model can be used to balance them. In Section 4 we illustrated how token-level model confidences could be used in simulated HITL task-oriented parsing annota- tion task. Our experiments in Section 5 extended these results sequence-level confidences and non- expert users; we found that model confidence could be used to improve the usability-safee trade-off' and introduced YouMan, which improved us- ability by asking users to accept predictions.
Annotators were asked to choose one of the top predictions or to manually rewrite the query if none of the predictions were adequate; the cho- sen or rewritten query was then re-parsed and com- pared to the gold parse. Further annotation de- tails are given in Appendix C. Of the 100 exam- ples sampled, annotators manually rewrote 7 and chose from the top-k list for the other 93, Ignor- ing the rewritten examples, 39 model predictions were incorrect and 54 were correct; by choosing glosses, annotators correct 5 incorrect predictions. However, they also inadvertently changed cor- rect predictions to incorrect. Figure Fig. 3 shows the exact match accuracy before and after annota- tors selected glosses at each confidence bin. At low confidence, we see very minor increases on the order of single program being corrected. At high confidence, annotators generally have only one or two options, and are able to choose the correct one, resulting in similar performance nu- cleus decoding. However, at medium confidence, annotators often choose the wrong gloss, leading
Limitations
Our study is limited by the models, datasets, and languages we consider. Firstly, we examine only English datasets, limiting the impact of our results. We also only consider one task-oriented parsing dataset, and focus on one model architecture.
We make several limiting assumptions in Sec- tion 4 and Section 5. Foremost amongst these is the assumption of access to oracle annotator in Section 4; clearly, no such annotator exists. Our results may vary when real annotators are brought into the loop. For one, we do not know exactly how choosing from the top-k list will compare to insertion w.r.t. speed. We also do not know how

---

automation bias (Cummings, 2004) would affect the top-k list: given that the correct answer is of- ten in the list, real annotators might overly rely on the list and prefer it to token insertion, resulting in incorrect programs.
Hong Yu. Diarmuid Ó Séaghdha, and Anders Johannsen. 2020. Conversational semantic parsing for dialog state tracking. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 8107-8117 17, Online. Association for Computational Linguistics.
The experiments in Section 5 rely on a glossing model to translate predicted programs into natural language (NL). We approached with neural Lisp- to-NL. model; this has several limitations. Neural text generation models often hallucinate outputs, i.e. generated glosses may not be faithful to their corresponding programs, Unlike Fang et al. (2022), who use a grammar- grammaar-based approach for response generation, we do not assume access to a gram- mar but note that our method is compatible with grammar-based constraints. Our annotators in Sec- tion 5 face the additional challenge of interpreting and choosing glosses. SMCalFlow programs are nuanced and slight input variations can result in dif- ferent programs. These nuances are often obscured by the glossing model, resulting in two different programs glossing to semantically equivalent ut- terances; we explore this further in Appendix C. Annotators might mistakenly accept glosses from incorrect programs reject correct glosses; this would be difficult address even with faithful translation method.
Chi Chi-Keung Chow. 1957. An optimum character recog- nition system using decision functions. In IRE Trans- actions on Electronic Computers, 4, pages 244-254. IERE.
Mary Cummings. 2004, Automation bias intelligent time critical decision support systems. AIAA 1st Intelligent Systems Technical Conference.
Ran El- Yamiv et al. 2010. On the foundations of noise- free selective classification. Journal of Machine Learning Research, 11(5).
Hao Fang. Anusha Balakrishnan. Harsh Jhamtani, John Bufe, Jean Crawford, Jayant Krishnamurthy, Adam Pauls, Jason Eisner, Jacob Andreas, and Dan Klein. 2022. The whole truth and nothing but the truth: Faithful and controllable dialogue response genera- tion with dataflow transduction and constrained de- coding. arXiv preprint arXiv:2209.0780 07000.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Ku- mar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representa- tions. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2787-2792.
Acknowledgements
Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples neural networks. arXiv preprint arXiv:1610.02136.
We would like to thank Anthony Platanios, Subhro Roy, Zhengping Jiang, Kate Sanders, Yu Su, and Daniel Khashabi for their feedback on an earlier draft. This work supported by NSF #179907, and NSF Graduate Research Fellowship to the first author.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious of neural text de- generation. In International Conference on Learning Representations.
David Lewis and William Gale. 1994, A sequential al algorithm for training text classifiers. SIGIR 94:
References
Proceedings of the Seventeenth Annual International ACM-SIGLA Conference on Research and Develop- ment in Information Retrieval, organised by Dublin City University, pages 3-12 12. Springer.
Shobhit Chaurasia and Raymond Mooney. 2017. Dialog for language to code. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 175- 180.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising requence-to-sequene pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Mark Chen, Jemry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
Jianpeng Cheng, Devang Agrawal, Héctor Martinez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Dain Kaplan, Dimitri Kartsaklis, Lin Li, Dhivya Piraviperumal, Jason D. Williams,
Fei Li and Hosagrahar V Jagadish. 2014. Constructing interactive natural language interface for relational databases. Proceedings of the VLDB Endowment. 8(1):73-84.

---

Corey Lynch and Pierre Sermanet. 2020. Language conditioned imitation learning over unstructured data. arXiv preprint
Elias Stengel-Eskin, Emmanouil Antonios Platanios, Adam Pauls, Sam Thomson, Hao Fang. Benjamin Van Durme, Jason Fisner, and Yu Su. 2022. When more data hurts: troubling quirk in developing broad-coverage natural language understanding sys- tems. Proceedings of the 2022 Conference on Lm- pirical Methods Natural Language Processing (EMNLP).
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Arm- strong, and Pete Florence. 2022. Interactive lan- guage: Talking robots in real time. arXiv preprint arXiv:2210.66-00
Elias Stengel-Eskin and Benjamin Van Durme. 2022. Calibrated interpretation: Confidence estimation in semantic parsing, arXiv preprint arxiv:2211.07443.
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. 2022. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315, PMLR.
Yu Su, Ahmed Hassan Awadallah, Miaosen Wang, and Ryen W White. 2018. Natural language interfaces with fine-grained user interaction: A case study web apis. In The 41 International ACM SIGIR Con- ference on Research & Development in Information Retrieval, pages 855-864.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 1-111
Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022. Investigating selective prediction approaches across several tasks in iid, ood, and adversarial set- tings. In Findings of the Association for Computa- tional Linguistics: ACL 2022, pages 1995-2002.
Emmanouil Antonios Platanios, Adam Pauls, Subhro Roy. Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, and Dan Klein. 2.21. Value-agnostic con- versational semantic parsing. In Proceedings of the 59th Annual Meeting of the Association for Compu- lational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Vol- ume Long Papers), pages 3666-3681, Online. As- sociation for Computational Linguistics.
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Mareus Rohrbach. 2022. Reliable visual ques- tion answering: Abstain rather than answer incor- rectly. In Proceedings of the European Conference on Computer Vision (ECCV).
Subhro Roy. Sam Thomson, Tongfei Chen, Richard Shin, Adam Pauls, Jason Fisner, and Benjamin Van Durme. 2022. Benchelamp: A benchmark for evaluating language models on semantic parsing. arXiv preprimt arXiv:2206. 10668.
Terry Winograd. 1972. Understanding natural language. Cognitive psychology. 3(1)
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the A Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 1040-1051.
Stuart Russell and Peter Norvig. 2010. Artificial in- telligence a modern apprach. Pearson Education,
Inc.
Semantic Machines, Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Craw- ford, Kate Crim, Jordan DeLoach, Leah Dorner, Ja- son Eisner, Hao Fang. Alan Guo. David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk. Sm- riti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lan- man, Perey Liang, Christopher H. Lin, Tlya Lints- bakh, Andy McGovem, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Sny- der, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson. Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov, 2020. Trik-triented dialogue as dataflow syn- thesis. Transactions of the Association for Computa- tional Linguistics, 8:556-571.
Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based interactive semantic parsing: A unified framework and a text-to-sql case study. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing
pages 5447-5458
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019a. AMR parsing as sequence-to- graph transduction. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 80-94, Florence, Italy. Association for Computational Linguistics.
Elias Stengel-Eskin, Andrew Hundt, Zhuohong He. Aditya Murali, Nakul Gopalan, Matthew Gombolay, and Gregory D. Hager. 2021. Guiding multi-step re- arrangement tasks with natural language instructions. In 5th Annual Conference on Robot Learning.
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme, 2019b. Broad-coverage semantic pars- ing as transduction. Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language

---

Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 3786-3798, Hong Kong, China. As- sociation for Computational Linguistics.
A Model and Data
Data We use version 2.0 of SMCalFlow, which contains language inputs paired with Lisp-like pro- grams. Fig. 4 shows example input and output for SMCalFlow. Unlike many other task-oriented formalisms, SMCalFlow contains salience oper- ators like refer, which allow for deixis in the input.
Input
Output
Remove the even for the 23rd from the calendar.
(Yield
Coodatecomnitteentiieee (UpdatarruE11gh) Eventtnaaper
(Event. .d (singleton
I didn't find any events Tuesday the 23rd.
SeeryyEvetRReppone results ionndveenttsspeettttooeolllll
(EventOrDate (nextDay(Aitnth
(Today) 23L) (Event.subject.
No meant the annual general meeting, change the dates from the 23rd to 7th may
(?- general
necting" nesting")))))) Eveett.start_'
(DateTine, date_? (?= (MO
(May)))))))
Figure 4: Input-output example for SMCalFlow
Input Representation We follow Stengel- et al. (2022) and Roy al. (2022) using the previ- ous dialogue turn as input if available. Thus, each datapoint consists an input X = (Mo, Ao, and an output program P. where Uo is the previous user utterance (if it exists), Ao an automatically- generated agent response to the previous utterance, and 44 is the current user utterance.
Paraphrasing To find good paraphrase of each predicted parse, we generate N glosses Ul via beam search and take the one that yields the highest probability of the predicted parse P. i.e. ú¹ argmax; PMIsO(P (o, Ao.
Evaluating the glossing model Instead of evalu- ating the glossing model using string metrics such as BLEU (Papineni al., 2002) which can be noisy, we choose evaluate the output programs using cycle-consistency. Specifically, we evalu- ated trained models by glossing programs the gold test set and then parsing those glosses with a fixed MISO parser. We explore T5-base, T5- large, BART-base, and BART-large architectures. All accuracy scores are reported in Fig. 5 along with the baseline accuracy obtained by parsing the gold test inputs. The best-performing gloss model is BART-large. Note that all glossing mod- els outperform MISO without glosses. This can be explained by the fact that we gloss the inputs from the gold program, which we then evaluate on,

---

allowing for information leakage. We also hypoth- esize that the gloss model, having been trained on the entire dataset of utterances, averages out many unnotator-specifit idiosyncracies which may make inputs hard to parse. This result does not imply that glosses generated from predicted programs would yield better performance when parsed than the user input.
this was only to be done in cases where none of the options reflected the intended meaning of the original utterance.
Tredes
Exact Match
82,99
82.78
84.00
Figure Template the selection HIT.
large
BART-base
BART-large
Model
Figure 5: Exact match for programs parsed from glossed inputs from the SMCalFlow out glosses is overlaid in black.
Performance with-
B User study details
Fig. 6 shows the template for the confirmation HIT. The instructions asked users to read the paraphrase produced by the agent and determine whether the agent had correctly understood the user. Annotators
Usen: yes, confimm that.
Chattie: ''e put that on colenderr.
User: Can you canterence roste
miink
rghh??
Figure 6: Template for the confirmation HIT.
were recruited from a list of qualified annotators on Amazon Mechanical Turk and paid $0.05 per example, averaging to roughly $16 per hour.
C Selection study details
The template for the selection study is shown in Fig. 7: each example gives the gloss as well as a rounded confidence score for the predicted pro- gram. Annotation was run with the same group of annotators as the experiments in Section 5; annota- tors were paid $0.11 per example, or about $16 per hour of annotation. Each example was annotated by a single annotator, all of whom had been vetted in pilot annotation task. Annotators were instructed help a robot named SvenBot, who had recently learned English and was not confident about its un- derstanding, disambiguate between several options. The interface contained a text box where annota- tors could optionally manually re- we-irite the input;


Reviews:
[{"review_id": "rYU9dnClEe", "reviewer": "Reviewer_11rs", "summary": "", "questions": "A. Could you justify your current baseline selection better? Why are other HITL or automated baselines missing? ", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ A new HITL system that improves task performance at a small cost of human involvement.", "reasons_to_reject": "- Missing baselines: The paper presents a system that relies on human feedback (confirming rephrasing) and shows that system outperforms a naive system that accepts all predictions and a system with confidence thresholds tuned for F1. However, the paper does not compare to previous work (e.g., \"Yao et al. (2019) introduce a confidence-based interactive system in which a parsing agent can ask users for clarification.\" in line 79) or other obvious baselines (e.g.,  a system with another model to select the most likely rephrase). ", "questions_to_authors": "A. Could you justify your current baseline selection better? Why are other HITL or automated baselines missing? ", "paper_topic_and_main_contributions": "This paper presents a new system for semantic parsing that rephrases low-confidence inputs and asks users to confirm the paraphrases. By leveraging the relatively cheap human signal (answering Yes/No questions), the system strikes a good usability-safety trade-off for semantic parsing.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Wrong format: Limitations should not be a numbered section \n"}, "extra_text": {}}, {"review_id": "RHyWyS4HM1", "reviewer": "Reviewer_DWn7", "summary": "", "questions": "How come real annotators were not brought into the loop, as indicated in the limitations section?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "For the most part a well-written and well-rounded short-paper contribution on an intriguing topics which makes for a rather compelling read. The results are believable and well-presented.", "reasons_to_reject": "The paper is at times a bit dense and would definitely benefit from an extra page to unpack some of the experiment setup and results discussion paragraphs. The content as it is may be a bit hard to follow for readers with whom the topic falls outside core areas of expertise.", "questions_to_authors": "How come real annotators were not brought into the loop, as indicated in the limitations section?", "paper_topic_and_main_contributions": "Topic:\n\nThe paper investigates the effects of model calibration in task-oriented parsing through a simulated annotator-in-the-loop experiment.\n\nMain contributions:\n\n- Shows how well-calibrated confidence scores can allow for balancing cost with annotator load, increasing accuracy through a small number of interactions.\n- Confidence-based thresholding reduces the number of incorrect executions at a cost to usability, whereby applying the proposed system (DidYouMean) improves that balance by rephrasing low-confidence inputs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MbTaWXpPaz", "reviewer": "Reviewer_q8g5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is very well-written and well-organized.  The experiments are sound and the results are reasonable.  Multiple annotators were employed with good inter-annotator agreement.  The Limitations section is thorough and addresses my concern about the use of machine-generated \"glosses\" in lieu of Lisp code presented to the annotators.\n\nI think this paper will get other researchers thinking about generated response confidence and safety vs. usefulness.  I wonder if, in larger production systems, a user-specific confidence threshold couldn't be learned.", "reasons_to_reject": "Some reviewers may not think this work is novel enough.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes experiments in balancing safety vs. usefulness of a dialogue system that generates and executed code based on the input.  First, the authors determine confidence scores on the generated code and conduct simulated human-in-the-loop studies setting different confidence score thresholds.  The higher the confidence score, the fewer generated programs will execute.  Then, the authors developed the DidYouMean system which, for generated code below the confidence score threshold, will try to rephrase the user's query in an effort to better understand the ask and generate safer (higher confidence) code.  DidYouMean is evaluated using eight MTurk annotators.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "On line 046, the comma after \"annotator\" is unnecessary.\n\nOn line 289, you first mention four annotators but then say \"All 3\".  Should this be four, or is additional information needed to discuss a subset of three annotators?\n\nIn that same paragraph, it fits better with the formal style of conference papers to write numbers ten and under using words.\n\nOn line 301, I think \"results\" isn't necessary before \"yields\"."}}, {"review_id": "yByg0ZYTkH", "reviewer": "Reviewer_k4cA", "summary": "", "questions": "Question A: Why did you choose the three metrics you used in Section 4?\n\nQuestion B: Who were the annotators in Section 5 and what specific content were they shown (were they also seen the original utterance and the gloss per how the DidYouMean system would actually be used)?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work here is well-motivated and provides sufficient details and results for a short paper. The writing is clear and the experimentation is thorough. Efforts towards mitigating the need for expert annotators are valuable and this is an interesting application of calibration, being that the confidence score is used to filter high-risk predictions (which is possibly innovative for semantic parsing). ", "reasons_to_reject": "The human-in-the-loop experiments are simulated using an annotator who is always correct, as the authors did not hire expert annotators. The assumption that any given annotator is 100% accurate on even a simple task is flawed and this study would be more rigorous if (1) either real annotators were used, or (2) disagreement analysis was considered in the simulation, rather than relying on one simulated accurate annotator. It is also not clear why the three employed metrics were chosen, i.e. if their use is motivated by prior work or what information they capture. ", "questions_to_authors": "Question A: Why did you choose the three metrics you used in Section 4?\n\nQuestion B: Who were the annotators in Section 5 and what specific content were they shown (were they also seen the original utterance and the gloss per how the DidYouMean system would actually be used)?", "paper_topic_and_main_contributions": "This work develops a system to use calibration of semantic parsers to reduce the amount of annotation needed to train these models. Specifically, the authors craft a simulated annotator-in-the-loop experiment which uses calibration scores to determine whether a specific query needs to be rephrased and checked with the user. This can then be used to reduce errors in system output, which the authors argue reduces some of the need for a high amount of annotated data. Specifically, the authors demonstrate that token-level confidence scores (by proxy of three metrics) can be used to balance usability and safety; the \"DidYouMean\" system allows low-confidence parses to have a second chance at resulting in a correct response by triggering a dialogue which rephrases the question back to the user. The work also presents a user study of the DidYouMean system which shows relatively high inter-annotator agreement, and also shows that ensuring safety comes at a high cost of usability, so a threshold will need to be used to effectively implement the system.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The \"Semantic Machines et al.,\" 2020 citation is not correct\n\nFigure 1 is slightly hard to understand at first-- I think it would help if the graph went underneath the dialogue so that the reader's first inclination isn't to see where the orange arrow is going."}}]

Author rebuttal:

Meta-review: This paper investigates the use of calibrated models to address two trade-offs for dialogue systems that generate and execute code based on user input: (1) cost/speed vs. accuracy in annotation, and (2) usability vs. safety in task-oriented user interfaces. 

- The first question is addressed by showing the benefits of manually annotating only low confidence samples, although the experiments rely on a simulated annotator, which is always correct, thus limiting the generalizability of the results.

- For the second question, safety is viewed as rejecting an unsuccessful program before executing it, while usability is not clearly defined but is operationalized as the percentage of inputs for which a program is executed. Here, the paper shows that confidence-based thresholding reduces the number of incorrect executions at the cost of coverage, and that rephrasing a user's query for low confidence outputs yields safer (ie higher confidence) outputs. This system is evaluated in a user study with 8 MTurk annotators.

Overall, the paper represents an interesting focused contribution that is appropriate for a short paper.

I will note that it was not very clear to me what is gained by looking at these two problems jointly in a single short paper. I would recommend either addressing that point more directly, or focusing the paper on the second study.
Decision: Accept-Main