ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models
Authors: Dheeraj Mekala
Jason Andrew Wolfe
Subhro Roy
Venue: EMNLP 2023 Main (2023)
Keywords: zero-shot
semantic parsing
mtop
large language models
llm
QA datasets
TL;DR: LLMs fail in zero-shot semantic parsing because they cannot detect unanswerable questions and we improve it by training a LLM that can abstain.

Abstract: We explore the use of large language models (LLMs) for zero-shot semantic parsing. Semantic parsing involves mapping natural language utterances to task-specific meaning representations. LLMs are generally trained on publicly available text and code and cannot be expected to directly generalize to domain-specific parsing tasks in a zero-shot setting. In this work, we propose ZEROTOP, a zero-shot task-oriented parsing method that decomposes semantic parsing problem into a set of abstractive and extractive question-answering (QA) problems. For each utterance, we prompt the LLM with questions corresponding to its top-level intent and a set of slots and use the LLM generations to construct the target meaning representation. We observe that current LLMs fail to detect unanswerable questions; and as a result, cannot handle questions corresponding to missing slots. We address this by fine-tuning a language model on public QA datasets using synthetic negative samples. Experimental results show that our QA-based decomposition paired with the fine-tuned LLM can zero-shot parse ≈ 16% of utterances in the MTOP dataset.

ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using
Large Language Models
Dheeraj Mekala*
Jason Wolfe
Subhro Roy
UC San Diego
OpenAl
Microsoft Semantic Machines
dmekala@uesd.eem
jascnkolfe@opensi.com
subbro subbro. royeerrromeeiiccoee.......o.o com
Abstract
to its top-level intent and of relevant slots, and leverage the LLM's ability to zero-shot answer reading comprehension questions.
We explore the use of large language models (LLMs) for zero-shot semantic parsing. Seman- tic parsing involves mapping natural language utterances to task-specific meaning representa- tions. LLMs are generally trained on publicly available text and code and cannot be expected to directly generalize to domain- -ppecffc pars- ing tasks a zero-shot setting. this work, we propose ZEROTOP, zero-shot task-oriented parsing method that decomposes semantic pars- ing problem into of abstractive and extrac- tive question-answering (QA) problems. For each utterance, we prompt the LLM with ques- tions corresponding its top-level intent and set of slots and use the LLM generations to con- struct the target meaning representation. We ob- that current LLMs fail to detect unanswer- able questions; and as a result, cannot handle questions corresponding to missing slots. We address this by fine-tuning a language model on public QA datasets using synthetic nega- tive samples. Experimental results show that our QA-based decomposition paired with the fine-tuned LLM can zero-shot parse ne 16% of utterances in the MTOP dataset.
As illustrated in Figure 1, we cast top-level in- tent classification as an abstractive QA task. To address LLMs' bias towards predicting labels com- mon in the pretraining data (Zhao et al., 2021), propose to generate an intent description in an unconstrained manner and infer the intent label most similar to the generated description. We view slot value prediction as an extractive QA problem. Most utterances do not mention all the slots. It is therefore essential for the model to abstain from prediction when corresponding slots are not men- tioned. Through our analyses, we observe that most LLMss frequently hallucinate text for miss- ing slots with high confidence, resulting in poor performance. To address this, we fine-tune an LM on a collection of public QA datasets augmented with synthetic unanswerable samples. We call our trained model Abstainer, as it is capable of identi- fying unanswerable questions and abstaining from prediction. We hierarchically prompt for nested slots using the Abstainer, and infer nested intents if their corresponding slots detected. We empir- ically show that this QA based decomposition of ZEROTOP is an effective way to leverage LLMs for domain specific semantic parsing, outperform- ing several strong baselines in the zero shot setting.
1 Introduction
Large language models (LLMs) are trained on pub- licly available text (Raffel et al., 2020; Sanh et al., 2021; Brown et al., 2020) and code (Chen et al., 2021) and have been shown to attain reasonable zero-shot generalization
2 Related Work
a diverse set of NLP tasks (Wang et al., 2019). However, they are not expected to generalize to domain-specific semantic parsing tasks in a similar way, where the inductive bias from pre-training is less helpful. In this work, we propose ZEROTOP that decomposes the seman- tic parsing task into one of answering series of extractive and abstractive questions, corresponding
LLMs are increasingly used for semantic parsing in low-data scenarios utilizing canonical representa- tions (Shin et al., 2021; Yang et al., 2022), and prompt-tuning (Schucher et al., 2022; Drozdov et al., 2022). The closest work to ours is Zhao et al. (2022) where they decompose parsing into QA tasks. However, they assume access to some an- notated data whereas we focus on a strict zero-shot setting where only the schema information is avail- able along with some natural language prompts for
Work done during an internship at Microsoft Semantic Machines. Work done while at Microsoft Semantic Machines.

---

"please send that message to my mom WhatsApp"
Slots
What the intend do?
What the relation?
Abstainer
C1
mom
Possible Nested Intents
Send Message
GET_ CONTACT
Cn
Ordinal Which one?
Name Name_App Which app?
Recipient Who the recipient?
Where the school located?
Abslainer
DET_SCHOOL
WhatsApp
mom
[IN:SEND MESSAGE RSCRIIIEE [INGEET CONTACT [SL:CONTACT RELATED my | (SLTYPE RELATION mom ]1]]
ZEPOTOP hierarchically prompts LLMs for identifying intents, slots, and nested intents, and combines LLM generations to create meaning representation. First, we identify top-level intent. Next, we prompt for each slot. If slot can accommodate nested intents (in red), we prompt for nested slots and infer nested intents.
Figure I:
schema entities. Our work is also related to ap- proaches towards zero-shot dialog state tracking using LLMs (Gao et al., 2020; Lin et al., 2021a.b). Specifically, Lin et al. (2021a) uses an Abstainer handle missing slots. Our method differs in that, we focus on semantic parsing where the Abstainer needs be applied multiple times along with intent detection to create nested meaning representations.
Algorithm 1: ZEROTOP: Our proposed
Zero-shot semantic parsing method.
Input:
Set of intents I, Set of slots S, Slot questions Q. intent-to-slot mapping 12S, slot-to-intent mapping $21, llo-toocaandddate-nnettdee mapping S2NI, Intent-model ,M,, Abstainer Mabss and Utterance u
Output: Predicted meaning representations MR
intent = -M,
slot Values ()
for slot S; E I2S(intent) do
3 ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing
slot = Mazu(u Qs.)
for candidate N.intent I, E S2NI(S,) do for slot S, I2S(I,) I do if Maaaa (alot Valuess,,, Qs is not
Problem Formulation We focus on task- oriented parsing with hierarchical intent-slot schema. Let I [I,, IIn and S (2,, )..). be the set of all possible top- level intents and slots respectively. Each intent Ij has set of slots S (S...s)..... that can filled. Possible slots in an intent are rep- resented by the intent-to-sie tlt mapping 12S: I + P(S), where P(-) is the powerset operator. Sim- ilarly, the inverse slot-to-intent mapping is repre- sented by S2I: S I. The input in our setting consists of I2S and S2I, but no annotated data. ZEROTOP requires users provide a question per slot (Qs,, Qs....., )s.). that represents their purpose. In a real-life setting, this can be obtained from a domain developer.
NONE then Update Valueel with nested intent I, Mabs Values Values 2s,)
MR Construct representation with intent, slot Values
Return MR
call) as intent for 92% of the data in the call do- main. Therefore, we propose to first generate an intent description in an unconstrained fashion by priming the LM with the following prompt.
Answer the following question depending on the context. context: A user said, (uiterance). question: What did the user intend to do? answer:
Then, we choose the intent label that is most simi- lar to generated answer using RoBERTa sentence similarity (Reimers and Gurevych, 2019). Unlike Zhao et al. (2022), our approach does not require enumerating choices in the prompt allowing us to handle a large number of intents and slots as found in datasets like MTOP
Unconstrained Generation for Zero-Shot Intent Classification We view zero-shot intent classi- fication as an abstractive QA problem. One intu- itive way is to prime the LLM with QA prompt and then constrain the generation to search over only valid intent labels (Shin et al., 2021). How- ever, LLMs are known be biased towards text sequences (Zhao et al., 2021) more common pre- training data. For example, in the MTOP dataset, the TO-3B model predicts CREATE_ CALL (make
Leveraging QA datasets for Slot Value Predic- tion Slot value prediction involves extracting phrases for a slot from the user utterance. We cast this as an extractive QA problem. All slots might

---

not be mentioned in an input utterance. For exam- ple, in the MTOP dataset, on average, only one- third of possible slots are mentioned per utterance.
ZEROTOP: Putting it all together The pseudo- code of ZEROTOP mentioned in Algorithm-11 ZEROTOP employs a top-down, greedy prompting strategy, where we first prompt for intent and then, its respective slots. First, we obtain the top-level in- tent using the intent model. Based on the predicted intent, we prime the Abstainer for corresponding slots using their respective questions as prompts. For each identified slot value, we prompt the Ab- stainer for slots of candidate nested intents. We use the same prompt format for this step with the identified slot value now considered as the input utterance. Finally, we combine predicted intent, identified slot values, and nested intents to create the meaning representation.
The QA model needs to abstain from prediction for such missing slots. To analyze the abstaining capability of pre-trained QA models, we consider a few nop-performing zero-shot LLMs TO-3B, GPT- 3, and Codex with their corresponding prompts and experiment on 500 sample subset of unanswer- able questions from the SQuAD dataset (Rajpurkar et al., 2018). We observe the accuracy of all mod- els to be < 5% and notice that they frequently hallucinate and generate answers for unanswerable questions. In section 4, we also consider log- likelihood-based threshold for abstaining and show that this threshold is difficult to tune using public QA datasets.
4 Experiments
To address this challenge. we leverage multiple publicly available QA datasets' to train Abstainer, a QA model capable of abstaining from prediction. Specifically, we generate synthetic unanswerable training samples by modifying existing QA data, and train QA model jointly on existing datasets and synthetic unanswerable questions. For every (question, answer, context) triplet, we generate syn- thetic unanswerable questions by either (1) remov- ing the sentence containing the answer span from the context, or (2) randomly sampling context that doesn't 't have the same question. After training the Abstainer, we prompt for each slot with its corresponding question for slot value prediction, in the following format:
We experiment on the English language subset of MTOP (Li et al., 2021) dataset. MTOP is multi- lingual task-oriented semantic parsing dataset com- prising data from languages and 11 domains. The test set has 4386 samples with 113 distinct intents and 74 slots. On average, each intent has 3.6 slots and 33% of possible slots are filled per utterance. Experiment Settings. We evaluate the zero- shot setting, therefore we have no training data, We manually create questions for slots using one example per slot. For training Abstainer, we fine- tune TO-3B on the extractive and abstractive QA datasets for epoch with a constant learning rate of 10-4. We use complete meaning representation match accuracy as the performance metric. More details in Appendix A.2.
Answer the following question depending on the context.
context: A user said, [uttenance). question: [slot question)
Baselines. We compare ZEROTOP with con- strained TO-3B, GPT-3 and Codex as intent mod- els where they are primed with intent generation prompt and are constrained to search over valid intent labels. We also compare with calibrated constrained TO-3B (Zhao et al., 2021) whose log- its are adjusted to counter LM biases. We con- sider an ablation of ZEROTOP where assign in- tent labels based only on their similarity with user utterance using RoBERTa sentence transformer. ZEROTOP-Intent represents our proposed intent prediction method.
arswer:
To identify nested intents, we as- sume knowledge of candidate nested intents that be accommodated by each slot, represented by the stot-to-candidatt-nneee intentt mapping S2NI: S PZ).. Our method assumes that depth of out- put representations at most 4 i.e. nested intents cannot further have more nested intents. One intu- itive way is to prompt the LLM for nested intent with the intent prediction prompt. However, our un- constrained generation- based intent model would predict many false positive nested intents. We in- stead use Abstainer prompt for their respective slots. If any slot value is identified, we consider its corresponding intent via S2I to be present as well.
Nested Intents
We compare with constrained T0-3B, GPT-3, and Codex as slot models as well, however, when primed with a question corresponding to a slot, the output constrained to be either from the utter- ance or from their corresponding phrases indicat- ing that question cannot be answered. We com-
'The QA datasets details are mentioned in Appendix A.1

---

Intent Model
Accuracy(S)
perior performance than alternative combinations. We observe that TO-3B, GPT-3, and Codex fail abstain frequently. The TO-3B model abstains only for 38% of unanswerable slot questions whereas Abstainer does for 89%. As a result, we observe notable performance gain by plugging in Ab- stainer as the slot model for each intent model base- line. Moreover, we observe MTQA and SEQZERO prompting methods, which originally proposed for custom- finottndd few-shot models, offer lit- tle assistance in zero-shot settings. For instance, when applying MTQA prompting to the TO-3B model, we observe marginal improvements. This indicates that the pre-trained TO-3B is unable to accurately identify fillable slots, demonstrating the necessity of Abstainer. Similarly, while SEQZERO prompting improves the performance of TO-3B, its effectiveness remains significantly inferior Abstainer. Finally, we see similar performance of
TO-3B constrained
34.02
TO-3B constrained calibrated
36.64
GPT-3 constrained
10.
Codex constrained
48.02
RoBERTa sentence
17.14
ZEROTOP-Intent
49.58
Table 1: Top-level intent classification results.
Intent Model
Stot Model
Acc(%)
GPT-3 constrained
GPT-3 constrained
3.00"
Codex constrained
Codex constraincd
TO:3B constrained Abstainer
2.42
TO-3B constrained
11.81
TO-3B constrained Abstainer
3.88
RoBERTa sentence
TO-3B constrained
4.10
TO-3B constrained-MTOR
4.63
TO-3B constrained-SEQZEROO
8.39
ZEROTOP-Intent
Abstaincr-MTQA
13.12
Abstainer-SEQZERO
13.68
MTQA and SEQZERO prompting with Abstainer. However, our approach of independently prompt- ing for each slot outperforms them.
Abstaincr
15.89
Table 2: Complete meaning representation match eval- uation. To limit API cost, we limit GPT-3 and Codex evaluation on 500-example subset, and hence their results are not directly comparable.
Annotation Effort Analysis We use 74 samples i.e. one per slot to design questions for slots. To analyze annotation effort, we train an utterance-to- meaning representation T5-3B (Raffel et al., 2020) parser using these 74 samples and compare with our method. The match accuracy of T5-3B parser on the MTOP dataset 8.19% and of ZEROTOP is 15. 89% justifying our annotation effort.
pare with two kinds of prompting for slot values. MTQA (Zhao et al., 2022) propose prompting an LLM to identify filled slots and then prime for their respective values. SEQZERO (Yang et al., 2022) introduce prompting each slot sequentially and using the previously identified slot value for prompting the next one. Abstainer is our finetuned TO-3B that abstains from prediction. We prompt for each slot independently.
Greedy vs Beam search ZEROTOP follows a greedy strategy where we hierarchically prompt for top-level intent and for its corresponding slots. We compare it with the beam search strategy with beam size 3. Specifically, we consider 3 top-level intents and prompt for their corresponding slots, consider top-3 slot values for every slot and finally compute the best meaning representation based on their ag- gregated NLL scores. The NLL score of intent Imm its slots S E I2S(Im), and their corresponding slot values slot Valuos(S]] is computed:
Results and Discussion From zero-shot intent classification results in Table 1, we observe that ZERoTOP-Intent performs significantly bet- ter than constrained TO-3B, GPT-3, and Codex. We found that constrained TO-3B is biased to- wards certain labels. For example, it predicts CREATE CALL (make call), SEND MESSAGE (send message), CREATE_REMINEE minder), as intent for more than 90% of the data in call, message, reminder domains respectively. Our proposed unconstrained formulation lets the model freely express the intent and, computing similarity later with the intent labels addresses this bias.
log (1 a) log Values S,I
REMINDER (create re-
where a is tumed on a held-out validation set. Note that p(slot Values|Zzn is computed recursively for its nested intents. The complete match accuracy of the greedy prompting strategy on MTOP dataset is 15.89% and of beam search strategy is 16.86%. This demonstrates that beam search can improve performance with validation data. Without valida- tion data and setting a to 0.5, performance drops
As shown in Table 2, the combination of ZEROTOP-Inee and Abstainer demonstrates su-

---

FI score vs NLL Threshold on MTOP-EN
no threshold that consistently results in good per- formance for all LLMs, which implies that this has to individually tuned for each LLM. Finally, we observe Codex performs better than Abstainer for some thresholds. As our problem setting includes no annotated data, we investigate whether we can infer the optimal threshold for Codex using public QA datasets. Specifically, we consider 500 answer- able and 500 unanswerable QA pairs from SQuAD dataset and plot F1 scores with range of confi- dence thresholds Figure 2(b). We can observe that the performance on answerable and unanswer- able subsets is mutually exclusive i.c. there no threshold where the performance on both answer- able and unanswerable subsets is high. The range of thresholds that result in the best performance the whole set (highlighted in green) does not transfer MTOP and achieved at the cost of unanswerable set where the Fl score is less than 5%. Given the difficulty tuning threshold and the API costs of Codex, we believe using Abstainer as the slot model to be a better choice.
35
30
20
TO-38
15
Codex
5
Anstainer
1
2
NLL Threshold
(a) F1 score vs NLL threshold on MTOP dataset
F1 vs NLL Threshald SQUAD
25
score
20
Answerable
15
Unanswerable
10
3
4
5
NLL Threshold
(b) F1 score vs NLL threshold on SQuAD dataset
Figure 2: We consider negative log- lekiiiiehhodd (NLL) as confidence and vary the threshold to abstain from prediction and plot Fl scores on the MTOP dataset. We show that this NI.I threshold is difficult to tune using public QA datasets such SQuAD performance on answerable and unanswerable subsets is mutually exclusive.
Conclusion
In this paper, propose ZEROTOP that decom- poses semantic parsing into abstractive and extrac- tive QA tasks. ZEROTOP identifies top-level intent by generating in an unconstrained fashion and infer- ring the intent label most similar to the generated description. We train Abstainer using public QA datasets, that is capable of identifying unanswer- able questions and abstaining from prediction.
to 12.36% i.e. 3% less than greedy. Therefore, we believe greedy prompting is better choice.
Confidence score-based Abstainer study We can alternatively have LLMs abstain from predic- tion based on a confidence score based threshold. We consider negative log likelihood (NLL) of the predicted slot value the confidence score and abstain from prediction if it is greater than the threshold. We experiment on slot value predic- tion task with TO-3B, Codex, and GPT3 as LLMs and plot macro scores for multiple NLL thresh- olds on a randomly sampled subset of 500 samples from MTOP dataset in Figure 2(a). Specifically, we consider the gold intent of each sample and prime LLM for extracting slot values for each slot of the gold intent. We consider Fl score as the met- ric due the label imbalance across possible slot values. We present the Fl-score of the Abstainer for reference. First, we observe that Abstainer is significantly better than TO-3B and GPT3 for all confidence thresholds, Second, we notice that there
6 Limitations
ZEROTOP assumes that the meaning representa- tions are of a limited depth i.e. nested intents can- not further have more nested intents and this is one of the limitations. Moreover, we also assume that it is possible write natural questions corre- sponding to slots. A slot for which natural ques- tion cannot be expressed, the LLM can't handle it without additional supervision. Finally, we believe there is huge scope for improvement in the perfor- mance of LLMs and ZEROTOP in domain-specific tasks such as zero-shot semantic parsing and on the MTOP dataset.
7 Ethics Statement
This paper proposes a zero-shot semantic parsing method using large language models. The aim of the paper is to minimize the human effort

---

annotation by leveraging language models. The oulput of our method is meaning representation that doesn't contain any harmful content. Hence, we do not anticipate any major ethical concerns.
In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 643-653, Lishon, Portugal. Association for Compu- tational Linguistics.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021, MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 2950-2962, Online, Association for Computa- tional Linguisties.
8 Acknowledgments
We thank the anonymous reviewers and our col- leagues from Microsoft Semantic Machines, espe- cially Hao Fang, Richard Shin, Adam Pauls, Matt Gardner, and Jason Eisner for their helpful feed- back.
Kevin Lin, Oyvind Tafjord. Peter Clark, and Matt Gard- ner. 2019. Reasoning over paragraph effects in situa- tions. ArXiv, 0bs/1908.00585
References
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebas- tian Riedel, and Pontus Stenetorp. 2020. Beat the ai: Investigating adversarial human annotation for read- ing comprehension. Transactions of the Association for Computational Linguistics, 8:662-678.
Zhaojiang Lin, Bing Liu, Andrea Madotto. Seungwhan Moon, Zhenpeng Zhou, Paul A Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, et al. 2021a. Zero-shot dialogue state tracking via cross- task transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 7890-7900
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul A Crook, Zhenpeng Zhou, Zhiguang Wang. Zhou Yu, Andrea Madotto, Eunjoon Cho, and Rajen Subba. 202 Leveraging slot descriptions for zero-shot cross- dialogue statetracking. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages
Dheeraj Mekala, Chengyu Dong, and Jingho Shang. 2022a. LOPS: Learning order inspired pscudo-label selection for weakly supervised text classification. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4894-4908, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv.2107.03374.
Dheeraj Mekala, Varun Gangal, and Jingbo Shang. 2021. Coarse2Fine: Fine- tine-grained text classification on coarsely-grainee annotated data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 583-594, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Pradeep Dasigi. Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019. Quoref: read- ing comprehension dataset with questions requiring coreferential reasoning. In EMNLP.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022. Compositional semantic parsing with large language models. arXiv preprint 2209::220 15003.
Dheeraj Mekala and Jingbo Shang. 2020. Contextu- alized weak supervision for text classification. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 323- 333.
Shuyang Gao, Sanchit Agarwal, Tagyoung Chung, Di Jin, and Dilek Hakkani-Tur. 2020. From machine reading comprehension to dialogue state tracking: bridging the gap. In ACL 2020 Workshop on NLP for Conversational AL.
Dheeraj Mekala, Tu Vu, Timo Schick, and Jingbo Shang. 2022b. Leveraging qa datasets to improve generative data augmentation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Us- ing natural language to annotate natural language.
Dheeraj Mekala, Xinyang Zhang, and Jingbo Shang. 2020. Meta: Metadata-empooeeee weak supervi- sion for text classification. In Proceedings of the

---

2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. Quartz: An open-domain dataset of qualitative relationship questions. arXiv preprint arXiv:1909.0355.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):166)
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- benchmark for general- purpose language under- standing systems. Advances in neural information processing systems, 32.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulka- mi, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. TWEETQA: A social media focused question answering dataset. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5020- 5031, Florence, Italy, Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguisties.
Jingfeng Yang, Haoming Jiang. Qingyu Yin, Danqing Zhang, Bing Yin, and Diyi Yang. 2022. SEQZERO:
Few-shot compositional semantic parsing with quential prompts and zero-shot models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 49 60. Seattle, United States. Association for Computational Linguisties.
Joshua Robinson and David Wingate. 2023. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations.
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Samkaramarayanan. 2018. DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension. In Meeting of the Associa- tion for Computational Linguistics (ACL).
Wenting Zhao, Konstantine Arkoudas, Weiqi Sun, and Claire Cardie, 2022. Compositional task-oriented parsing abstractive question answering. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4418 04277, Seattle, United States. Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training en- ables zero-shot task generalization. arXiv preprint 10.00221...8800..
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR
Nathan Schucher, Siva Reddy, and Harm de Vries. 2022. The power of prompt tuning for low-resource seman- tic parsing. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume Short Papers), pages 148-156, Dublin, Ireland. Association for Computational Linguisties.
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kai Sun, Dian Yu. Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. Dream: A challenge data set and models for dialogue-based reading compre- hension. Transactions of the Association for Compu- lational Linguistics, 7:217-231.

---

Type
Dataset
Samples
increases (e.g. 74 in MTOP), demands significant manual effort from users, which our paper aims reduce. Moreover, such rules would generally pre- dict many false positives/negatives, and classifying or identifying the appropriate ones accurately re- quires training data, which is not available in our zero-shot setting. Therefore, we compare against 10197
Adversarial QA (Bartolo et al.. 2020)
36000
QA-SRL (He et al., 2015)
8397
DuoRC (Saha
2018)
189089
Extractive
ROPES (Lin al, 2019)
14000
SQuADv2 (Rajpurkar al., 2018)
150000
Quoref (Dasigi al., 2019)
24000
ReCoRD WWang
2019)
121000
DREAM (Sun et al., 2019)
Abstractive
QuaRTz (Tafjord 2019)
3861
strong LLMs that are known for their impressive
Tweet-QA (Xiong 2019)
10692
zero-shot performance such GPT3, Codex, TO.
Table 3: Relevant statistics of the QA dataset used to train Abstainer.
The baseline LLMs such as GPT3 and Codex are not trained for semantic parsing. Wouldn't this make the performance improvement us- ing ZEROTOP less significant? We compare against the TO model which is fine-tuned several QA datasets like our Abstainer model. We cannot fine-tune the GPT3 and Codex models on these datasets separately. However, these instruction- tuned GPT3 and Codex are known to perform well on several dnuestion-asswernn & reading com- prehension benchmarks (Robinson and Wingate, 2023). Therefore, we consider them competitive baselines. In this paper, we show that these per- form worse zero-shot task-oriented parsing even when it is converted into QA task, for which they are known to perform well. The reason behind their poor performance because (1.) they are biased to- ward predicting labels common in the pre-training data, and (2.) they frequently hallucinate text for unanswerable questions. Through our work, we present this shortcoming, analyze the cause, and propose method to fix it.
A Appendix
A.1 QA Datasets for Training Abstainer
Publicly available QA datasets have been previously leveraged for generating synthetic data (Mekala et al.. 2022b) in weakly (Mekala and Shang, 2020; Mekala et al., 2020) and minimally supervised settings (Mekala et al., 2021, 2022a). In this paper, we use multiple extractive and abstrac- tive QA datasets to generate synthetic unanswer- able samples and train Abstainer. The details about datasets are mentioned in Table 3.
A.2 Experimental Settings
We use the OpenAl API text fext-daavici-on0 for GPT-3 and code deee-ddvinniiiii 002 for Codex. The Abstainer is fine-tuned on 411732 answerable and 435898 unanswerable samples. The batch size is 32 and each batch contains equal number of answerable and unanswerable samples. We used NVIDIA Tesla V100 for our experiments.
Why did you choose one prompt per slot and not multiple? We can possibly consider multi- ple prompts per slot and ensemble the predictions, which would intuitively boost the performance. However, multiple prompts per slot imply more annotations. Our motivation behind this work is to minimize the human annotations, thus we chose single prompt per slot.
A.3 Frequently Asked Questions
What is the scope of the presented ideas? We believe our idea can be easily extended to any se- mantic parsing tasks involving natural language interfaces; we considered Task-oriented parsing as a first step because of its simpler representa- tion. Through this work, we wanted to highlight an important real-life task that LLMs such as GPT3, be-
Codex, and TO underperform. Therefore, lieve this is useful and hope our work motivates more rescarchers focus on this shortcoming.
Why didn't you extract all syntactic phrases of certain type in the tree for slot-value detection? Syntactic phrase-basee based extraction slot values re- quires users to manually enter rules (a.k.a labeling functions) for each slot. When the number of slots


Reviews:
[{"review_id": "VTXKFfzuYE", "reviewer": "Reviewer_7287", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review. Below, we address the reviewer’s main concerns:\n\n__More test sets__\nWe wanted to consider the dataset with general-purpose intents and slots that align with the LLM’s pre-trained data. Therefore, MTOP perfectly fits this criteria and we show that it is unable to abstain in such a familiar setting. For other datasets with complex representation such as CalFlow, we expect it to hallucinate more and it requires more involved work. We leave this for future work.\n\n__Long Paper Suggestion__\nThank you for your suggestion! We appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clearly written, concise, and easy to follow.\n\nThe authors describe some interesting solutions to problems faced when using LLMs for zero-shot parsing (and other zero-shot labeling and recommendation tasks).\n\nThe authors show good results using their proposed method.\n\nI think many in the community would find this paper interesting.\n\nThe paper shows the continued utility of some of the smaller LLMs in CL which I think some have lost focus of in the GPT frenzy.", "reasons_to_reject": "The authors could have done more testing of their approach on alternate datasets, and could have tested using alternate models for their abstainer fine-tuning.\n\nI think that this work could easily be restructured into a long paper with additional experiments. If the paper is not accepted, the authors should consider this.\n\nOverall, though, these are minor considerations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel method zero-shot semantic parsing using a fine-tuned LLM. The authors describe several issues with using prompting strategies for predicting intent and slot values from text using LLMs. First, the fact that the LLMs tend to hallucinate values for slot values missing from the text. Second, that LLMs tend to be strongly biased toward labels seen more frequently in training data when asked directly to predict an intent label. \n\n1) to deal with the latter, the authors simply ask the model a general question for intent prediction, like \"what did the user intend to do\" and allow the model to freely generate a response. They then use a similarity scorer to match the generated text to the most similar intent label.\n\n2) to deal with the former issue, they fine-tune an LLM to create an \"abstainer\" model which is specifically trained on questions to which no answer is present in the provided text. The model learns to abstain from answering such questions.\n\nThe authors pipeline involves first predicting intent using the method described in 1) and then repeatedly prompting the abstainer model to fill with questions associated with the intent's slot values to try to fill in those for which values are available. The authors write these questions for all of the slot values manually, somewhat adding to the overhead of using their proposed approach.\n\nThe authors show significant zero-shot parsing improvement over several competitive zero-shot semantic parsers in tests on a sample of the MTOP dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aJJMmEQwYK", "reviewer": "Reviewer_i51D", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review. Below, we address the reviewer’s main concerns:\n\n__More test sets__\nWe wanted to consider the dataset with general-purpose intents and slots that align with the LLM’s pre-trained data. Therefore, MTOP perfectly fits this criteria and we show that it is unable to abstain in such a familiar setting. For other datasets with complex representation such as CalFlow, we expect it to hallucinate more and it requires more involved work. We leave this for future work.\n\n__Conclusion section__\nDue to the space limit, we removed the \"Conclusion\" section. We will add it back in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written and easy to follow. The details of the method are clearly explained.\n2. The proposed ZEROTOP that combines abstractive and extractive question-answering, along with fine-tuning on public QA datasets using negative samples, is well-motivated. \n3. The paper provides empirical evidence of the effectiveness of the proposed method, showing promising results on the MTOP dataset. ", "reasons_to_reject": "1. Experiments are only conducted on a single dataset\n2. The paper doesn't have a \"conclusion\" section, which makes the paper somehow incomplete.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores the use of LLM for zero-shot semantic parsing, proposing the ZEROTOP method, which leverages large language models for zero-shot semantic parsing by decomposing the problem into a set of question-answering tasks. Experimental results on MTOP shows the advantages of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X8AUUh3I6o", "reviewer": "Reviewer_h26o", "summary": "", "questions": "Could you add the questions crafted for each slot to the appendix?\n\nIn this line of research it may be important to decide when not to parse an intent, i.e. decide when a user utterance is not relevant for the given schema. This way we could have a truly zero-shot agent capable of parsing only relevant inputs. Sure, it is a topic for a different paper, but it would be nice to read some considerations on that.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review. Below, we address the reviewer’s main concerns:\n\n__More test sets__\nWe wanted to consider the dataset with general-purpose intents and slots that align with the LLM’s pre-trained data. Therefore, MTOP perfectly fits this criteria and we show that it is unable to abstain in such a familiar setting. For other datasets with complex representation such as CalFlow, we expect it to hallucinate more and it requires more involved work. We leave this for future work.\n\n__Abstaining via Prompting__\nWe tried zero-shot prompting since it is a zero-shot setting. GPT3/Codex/T0-3B constrained in Table 2 refers to zero-shot prompting results for the slot with very intuitive questions. For example, *what is the music genre?* is the question for the MUSIC_GENRE slot. As mentioned in lines 226-229, the output is constrained to be either from the utterance or from their corresponding phrases indicating that the question cannot be answered. We observed LLM’s incapability of abstaining from prediction when there is no slot in a zero-shot setting. Therefore, we used auxiliary publicly available QA datasets to train specifically for abstaining. Individual hand-tuning of prompts might help a little, however, given the amount of manual effort in hand-tuning and the zero-shot setting, we believe fine-tuning on data from a different task is a better choice. We agree with your point that this would help in motivating our design.\n\n__Questions for slot__\nYes, we will make these questions for the slots public.\n\n__Abstaining for intent__\nYes, that’s a great suggestion. Thank you! Abstaining at the intent level is very important for a dialogue agent and it is widely studied as \"Out-of-distribution detection\". We leave this for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper shows that an LM can be used to parse sentences starting from a schema, and that it is important to add abstaining capabilities to the LM so that it knows when a slot is not present in a utterance.\n\nThis area of research is interesting since it enables parsing unseen schemas.", "reasons_to_reject": "Experiments cover a single dataset.\n\nDetermining if a slot is fillable could potentially still be done by prompting the LM, avoiding fine-tuning altogether. It would probably require crafting a prompt for each slot, but would help in motivating the need of fine-tuning the model on data that it is not super-related to the parsing task.", "questions_to_authors": "Could you add the questions crafted for each slot to the appendix?\n\nIn this line of research it may be important to decide when not to parse an intent, i.e. decide when a user utterance is not relevant for the given schema. This way we could have a truly zero-shot agent capable of parsing only relevant inputs. Sure, it is a topic for a different paper, but it would be nice to read some considerations on that.", "paper_topic_and_main_contributions": "This paper proposes a prompting method for semantic parsing in a zero-shot fashion. A language model is prompted to perform intent classification and slot detection, supporting an additional level of nesting, i.e. a nested intent. No training data is used: the model has access to schema information and the only manually crafted data are questions associated with slot names.\n\nFirst, the LM is prompted to describe the intent of a user utterance. Then, a sentence encoder is used to compute the similarity between the generated description and the available intents. The most similar intent is selected. This works better than constraining generation to intent labels and computing generation likelihood, since probabilities favor words that occur frequently in the LM pretraining corpora.\n\nSlot filling is performed by asking the LM to answer a question for each slot associated with the predicted intent. The question is manually crafted for each slot. Since not all slots have a corresponding value in any given user utterance, the authors train an abstainer model (T0-3B) on available QA data that contain the not-answerable annotation. This way, the LM can abstain from filling a slot.\n\nIf a predicted slot can have a nested intent, the abstainer is prompted with the possible slots of the nested intent. If one of those is filled with a value, the nested intent is added to the interpretation.\n\nThe paper shows that being able to decide when not to fill a slot with a fine-tuned model is very important since LLM will always try to generate/hallucinate values.\n\nGenerating descriptions for the intents, slightly improves the similarity between utterances and descriptions compared to utterances and intent labels.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the use of LLMs for zero-shot semantic parsing where the main idea is to adapt the formulation of the task (semantic parsing) into an (extractive or abstractive) QA problem, which is a very natural thing to do. The LLMs also need the capability to figure out when a specific slot is missing and the authors show how it is possible to teach the model (i.e., tune it) that functionality via using synthetic negative examples. All the reviewers have detected a good deal of strengths of the paper, and I align with the overall sentiment of the reviewers.

The results are reported on a single dataset: MTOP, and I agree that the main issue of the current work is the fact that it evalutes on a single dataset, a wider coverage of other related semantic parsing problems and datasets would make the main findings more generalisable and more impactful eventually.
Decision: Accept-Main