SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Prompting
Authors: Xiaoying Zhang
Baolin Peng
Kun LI
Jingyan Zhou
Helen M. Meng
Venue: EMNLP 2023 Findings (2023)
Keywords: schema-guided LLM prompting
task bot
zero-shot dialog generation
TL;DR: We introduce SGP-TOD, Schema-Guided Prompting for building task bots effortlessly based on large language models, utilizing merely the predefined task schema.

Abstract: Building and maintaining end-to-end task bots using minimal human effort is a long-standing challenge in dialog research. In this work, we introduce SGP-TOD, Schema-Guided Prompting for building Task-Oriented Dialog systems effortlessly based on large language models (LLMs). Utilizing the predefined task schema, i.e., belief instruction and dialog policy, we instruct fixed LLMs to generate appropriate responses on novel tasks, without the need for training data. Specifically, SGP-TOD comprises three components: an LLM for interacting with users, a Dialog State Tracking (DST) Prompter to aid the LLM in tracking dialog states with the given belief instruction, and a Policy Prompter to direct the LLM to generate proper responses adhering to the provided dialog policy. Experimental results on Multiwoz, RADDLE, and STAR datasets show that our training-free strategy, SGP-TOD, yields state-of-the-art (SOTA) zero-shot performance, significantly surpassing the few-shot approaches. In a domain-extension setting, SGP-TOD aptly adapts to new functionalities by merely adding supplementary schema rules. We make our code and data publicly available.

SGP-TOD:
Building Task Bots Effortlessly via Schema-Guided LLM Prompting
Xiaoying Zhang', Baolin Peng', Kun Li', Jingyan Zhou', Helen Meng
'The Chinese University of Hong Kong, Hong Kong
TTencent AI Lab, Bellevue
Centre for Perceptual and Interactive Intelligence, Hong Kong (zhangxy, kunli, jyzhou, hmmeng)
(baolinpeng) ) @llobal..encent.coo
Abstract
User: am looking reetzurant with moderaale price range serving modeem Furopssn food.
Building and maintaining end-to-end task bots using minimal human effort is a long- standing challenge in dialog research. In this work, we introduce SGP-TOD, Schema- Guided Prompting for building Task-Qriented Dialog systems effortlessly based on large lan- guage models (LL.ss.. Utilizing the predefined task schema, ie.. belief instruction and dialog policy, we instruct fixed LLMs generate ap- propriate responses novel tasks, without the need for training data, Specifically, SGP-TOP comprises three components: LLM for in- leracting with users, a Dialog State Tracking (DST) Prompter aid the LLM in tracking dia- log states with the given belief instruction, and a Policy Prompter to direct the LLM to generate proper responses adhering to the provided dia- log policy. Experimental results on Multiwoz, RADDLE, and STAR datasets show that our training-frre strategy, SGP-TOD, yields state- of-the-art (SOTA) zero-shot performance, sig- nificantly surpassing the few-shot approaches. In a domain-extensioo setting, SGP-TOD aptly adapts to new functionalities by merely adding supplementary schema rules. We make our code and data publicly available.!
ocslauratt pizza goleer dliieeso, pifkorenge n donicaree cheap, mo moderate. expensive, area
Frompter
Betiet Instruction
SOL: se lect" from
resiaurntt serves [ealue_food
pricaranno ge modorato;
price range?
Reqqustt price range
No
Database
Resiaurant Swo match
Preferred area?
Foecuaot arew
No
Folicy Prompter
Action: restburant (irorm (chokces). requastt (anse))
Diálog Polley
Task Schema
System: least choices. do you have prefemed area?
The proposed SGP-TOD is depicted with dialog example, where the prompters integrate the task schema (right) to assist the frozen LLM in generating an appropriate response (left).
Figure 1:
2023), have revolutionized natural language pro- cessing (NLP) applications (Wei al., 2022; Wang et al., 2023), owing to their remarkable conversa- tional skills (Qin et al., 2023), instruction-following abilities (Ouyang et al., 2022) and zero-shot gener- alization capabilities (Chowdhery et al., 2022a; Hu et al., 2022). This raises a research question: can LLMs be effectively utilized for building task bots with minimum human effort?
1 Introduction
Building task-oriented dialog (TOD) systems has been a long-standing challenge in artificial intelli- gence. The prevailing approach for creating task bots (Hosseini-Asl et al., 2020; Peng et al., 2021a; la; Sun et al., 2022) is to fine-tune pre-trained language models (PLMs), such as T5 (Raffel et al., 2020) and GPT-2 (Radford et al., 2019). Despite their great success, developing and maintaining such task bots generally requires adequate annotated data and ex- tensive finettunig/rrettriiing
A contemporary study (Hudecek and Dusek, 2023) explores the potential of LLMs for rapidly building task bots via few-shot prompting, a.k.a. in-context learning (ICL) paradigm (Brown et al., 2020; Madotto et al., 2021). Though demonstrably effective, the ICL performance is highly influenced by the quality of the in-context exemplars (Zhao et al., 2021; Liu al., 2022; Dong et al., 2023), they struggle to provide comprehensive informa- tion for dialog task completion.
Recently, large Language Models (LLMs), such ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI,
'https://githhin htt.ilu..ccm/aaaaaaa cww/zhangxy--aeeeaooaaapaaaaaaae 281 s/spp-to

---

In this work, we introduce symbolic knowledge (Nye al., 2021: Cheng al., 2023), i.e. the task schema into LLMs, for creating task bots. Task schema (Mosig et al., 2020; Mehri and Eskenazi, 2021) encompasses a concise symbolic represen- tation of a task, supplying LLMs with a compre- hensive blueprint. It comprises (i) task-specific ontology containing all slots and their appropri- ate values (Budzianowski et al., 2018); and (ii) dialog flow explicitly outlining fundamental inter- action patterns (Peng et al., 2021b). Specifically, we propose SGP-TOD (as depicted in Figure 1), a Sathema-guided prompting method for rapidly build- ing task bots. We integrate the predefined task schema and dialog context into prompts through the use of two specifically-designed prompters, namely a DST Prompter and Policy Prompter. Utilizing these prompters, we adeptly guide fixed LLMs to track dialog states, retrieve database en- tries, and generate appropriate responses for novel tasks in a zero-shot manner, without the need for additional training or fine-tuning. By incorporat- ing task- sessifpcciiic symbolic knowledge into LLMs, SGP-TOD provides knowledge-basen coherent and human-like responses. Moreover, this training- free design empowers developers to flexibly proto- type dialog systems on new tasks, while seamlessly extending system functionalities through modify- ing the task schema.
LLM prompting strategy that facilitates in ef- fortlessly creating task bots, climinating the necessity for task-specific data or fine-tuning.
We integrate symbolic knowledge - task schema into LLMs, allowing them to generate schema-compliant responses and adaptively expand their functionalities to tackle new tasks by solely modifying the task schema.
We demonstrate the effectiveness of SGP- TOD Multiwoz, RADDLE, STAR datasets in zero-shot settings using both automatic and human evaluations. SGP-TOD notably ele- vates the SOTA zero-shot performance.
2 Related work
Zero-Shot Task-Oriented Dialog Modeling. Zero-shot generalization is an essential yet chal- lenging task in TOD research. A comprehensive study is shown in Appendix A. In this paper, we focus on zero-shot end-to and-to-end dialog modeling, in- cluding policy management and dialog generation.
The works by Zhao and Eskenazi (2018) and Qian and Yu (2019) utilize ontology and response templates train dialog models, enabling the dis- covery shared dialog policies between the source and target domains. To enable broader adapta- tion to diverse dialog policies, Mosig et al. (2020); Mehri and Eskenazi (2021) implement task policy skeletons, training dialog models to adhere to novel policies. Furthermore, Zhao et al. (2022) employs a neural language model (LM) for tracking dialog states and user actions using slot and action descriptions; subsequently, policy program is de- ployed to facilitate an LM in generating system actions and responses. Despite the effectiveness of previous approaches, they still require ample fine-tuning and copious annotated dialog corpora on source or heterogeneous domains/tasks.
We perform empirical automatic evaluations on two multi-domain datasets, namely, Multiwoz 2.0 and 2.2 (Budzianowski et al., 2018: Zang et al., 2020), as well as two single-domain/tsk/ datasets, RADDLE (Peng al., 2021a) and STAR (Mosig al., 2020), within zero-shot scenarios. Addition- ally, we complement these assessments with inter- active human evaluations. The results indicate that SGP-TOD, employing merely task schema devoid of any training or fine-tuning, substantially boosts the SOTA zero-shot results, markedly outperform- ing few-shot prompting/fne-ttniin methods, and even attaining competitive results of full-shot fine- tuning approaches. In domain-extension context, SGP-TOD proficiently adapts new functional- ities by simply adding a handful of schema rules without necessitating further data collection, sig- nificantly exceeding the few-shot prompting/fine- tuning methods reinforced by machine teaching (Williams and Liden, 2017).
A concurrent study to ours is Hudecek and Dusek (2023), which employs a prompting strat- egy - -I--TOO (instruction-guided TOD) to guide frozen LLMS in generating suitable responses. Specifically, IG-TOD first tracks belief states by utilizing slot descriptions as prompts, then retrieves database entries, and generates responses. Our SGP-TOD differs in that: (i) we employ slot names and value examples, rather than slot de- scriptions, as prompts to facilitate frozen LLMs in generating belief states, thereby reducing hu- man effort; (ii) we offer a policy skeleton to guide
In summary, our contributions three-fold:
We propose SGP-TOD, a schema-guided

---

LLMs in producing appropriate responses. In addi- tion, experimental results indicate that SGP-TOD substantially outperforms IG-TOD.
rameterized by 0) in generating a belief state b, (Equation 1). The belief state is then used query a database and obtain the database (DB) state Ct (Equation 2). (ii) System Action Determination the Policy Prompter incorporates a policy skeleton PS, assisting the LLM in generating a system ac- tion a,, based on h4, b. and ca (Equation 3). (ii)) Dialog Response Generation grounded in the dialog history h4, belief state b,, DB state Cr sys- tem action a,, the Policy Prompter aids the LLM in generating delexicalized response by providing the policy skeleton PS (Equation 4). Ultimately, the delexicalized response is automatically post- processed to generate system response in natural language. Detailed illustration with dialog exam- ple is shown in Appendix L.
Leveraging LLMs for Dialog Tasks. LLMs (Chowdhery et al., 2022b; OpenAl, 2023) have exhibited unparalleled mastery of natural language understanding, reasoning and generation (Wei et al., 2022: Bubeck et al., 2023). Three primary re- search directions have obtained substantial success in numerous dialog tasks by utilizing LLMs. (i) Few-shot prompting (Brown et al., 2020) has show- cased remarkable performance in intent classifi- cation (Yu et al., 2021), semantic parsing (Shin and Van Durme, 2022), dialog state tracking (Hu al., 2022; Xie et al., 2022), and response gener- ation (Madotto al.. 2021). (ii) Li al. (2022); Mehri et al. (2022); Dai et al. (2023) employ LLMs for data augmentation, i.e., generating synthetic task- frik-triented dialogs to train smaller models for inference. (iti) Recently, several studies endeavor to support LLMs in specialized tasks by incorpo- rating external knowledge. Peng et al. (2023) advo- cates for enhancing LLMs' responses with external knowledge and automated feedback to reduce hal- lucination. Liang et al. (2023) suggests connecting LLMS with millions of APIs to accomplish diverse tasks. Different from the aforementioned works, we aim to employ LLMs in building task bots in a zero-shot manner using pre-defined task schema.
b. LLLM((M BI)
(1)
C. (B.,,)
(2)
a, LLL ((n bi. ct, PS)
(3)
T1 LLM ol (hu, bi, ct a,, PS)
(4)
3.2 LLM
An LLM is responsible for following task task-specific instructions and generating appropriate responses. Many off- LLMs, e.g., ChatGPT, Codex (Chen et al., 2021), are pre-trained on massive corpora of text data and/or code data. In addi- tion, they are trained to follow instructions in the prompts (Ouyang et al., 2022) and provide perti- nent responses. Exhibiting remarkable proficien- cies natural language processing, instruction compliance, and zero-shot generalization across di- verse downstream dialog tasks, these LLMs serve as valuable foundation models for our approach.
3 SGP-TOD
3.1 Overview
The overall architecture of the proposed SGP- TOD (Figure I) consists of three key components: (i) an LLM, responsible for adhering to instruc- tions, comprehending user queries, and generat- ing coherent responses for user interaction: (ii) DST Prompter, tasked with supporting the LLM in tracking dialogue states using the belief instruc- tion; (iii) a Policy Prompter, guiding the LLM to adhere to the predefined task policy for providing suitable system actions and responses.
3.3 DST Prompter
Given the dialog history hr, the DST prompter aims to guide the LLM in predicting the belief state b, at each turn i, using the belief instruction BI. The belief state b, is defined as the concatenation of the domain/task (i.e., user intent) d, and a set of slot-value pairs {(s), vf); i = 1,...,,,, ny is the total number of pairs in the set.
Me }. where
At each dialog turn 1, the end-to-end generation task is systematically divided into three subsequent sub-tasks: (i) Belief State Prediction given the dialog history up to current dialog turn h1, which is a sequence of utterances alternating between the user and the system he [u1, 71, U2, ......, Ug] (where u and r denote user and system utterances, respectively), the DST Prompter embeds the be- lief instruction BI to direct the frozen LLM (pa-
As shown in Figure 2, the proposed DST prompter contains four parts: (i) task instruc- tion that offers general guidance on belief state prediction; (ii) belief instructions BI of all do- mains/tasks; (iii) a formatting example illustrating
"W assess several task instructions written by different authors, yiclding minor performance disparities.

---

DST Prompter
Policy Prompter
Task Instruction
Task instruction
Following the instructons, predict belief state bused the heelory.
Folleing instruction, generate appropriate based on history.
Bellef instructions
Formatting example (from other taskidomain)
rastaurant nama = pizza hut city golden wak, care, chinese, meditrrancnn, brtish, etc: pricorange = dontcare, chesp, moderate, expensive; area dont care, centre, nest, north, south, west, booking_dke day monday, lueeday, wednesday; thursday, miday, saturday, sunday; booking booking_people 1.2.3445 3,4,6,66) 8,77 booking time 12:15, 13:30, etc.
dont
Policy skeleton (on target taskidomain)
(1) user: I'm looking for restaurant that offers |value, in [value, priaernnee) price range. action: restaurant (inform (choices), require (area)) system: have over [value_ count] restaurant choose from, you preferred (sos]
[value_food] food] food
amtactionn type swimmingooal, theatre, architecture, museum, rightclun, boet, park, college, conperthall, entertainment multiple sports, cinema; area dont care, centre, east, north, south. west:
(2) user: need restaurant that serves Ivalue, faod] [value [velluepprieeanng] pricerange] the (value srea] side town, action: restaurant (recommend [namme, inform (food, pricerange, area)) system: how about [restaurant nemem] serves [valle__ food] food
Formatting example
pricerange) range (value, area).
you tell whether any scods verues in the center of lown?
[eod]
(17) restaurant one action: restaurant (recommend
SQL: select attraction where type muttiple sports; area centre
(name)) system: How does Irestaurant, name) sounc?
Test (on target task/domain)
Test (on target task/domain)
history
history
user: I am looking for a restaurant the moderate price range serving moderm Euppeen food. system: have least choices, you preferred
user: am looking for restaurant in the modderate price range cholces, you
modern Europsan food. system: mind? I'd
least 2
mind? user. like end town, please.
restaurant
restaurant the south town, please.
SQL: solect from restaurant where pricerange moderate: food modern Europeen: aree south DB: Reetaurant one matcch
LLM
LLM
SQL: restaurant pricerange moderate; -modem Europsan; area south
action restaurant (recommend (name), inform (food, pricerange. [value
[msttaurant,
Figure 2: Illustration of belief state prediction utiliz- ing DST Prompter. The predicted belief state is high- lighted.
food] [sakue_food] [value [ealue_anss] side town. [ans]
the [value_priieeeeee pricerangel price rainge on the
Figure 3: Illustration of system action determina- tion and response generation employing the Policy Prompter. The pertinent template turns previously
the anticipated output format to direct the LLM, in addition, we follow Hu et al. (2022) and adopt SQL state to represent the dialog state b;3; and (iv) the test imput, i.e., the given dialog history h, Since the prompt is fixed and no labeled data from the tar- get task domain is used, refer to this setting "zero-shot", following Wang al. (2022b).
predicted belief state retrieved DB state within the input, alongside the generated system action and generated response in the output are accentuated.
3.4 Policy Prompter
Belief Instruction. For each task/domain, the belief instruction contains the task/domain name, all potential slot names, and their possible val- ues (Figure 2). Regarding categorical slots, such as the "price range" in the restaurant domain, all plausible values are included, i.e., "don't 't care". "cheap", "moderate", and "expensive"; whereas, for non-categorical slots, such as "name", only a few value examples are injected, e.g., Pizza Hut City, Golden Wok. etc.4 Detailed belief instructions for all tasks/domains can be found in Appendix B.
Dialog policy, governing the behavior of task bots, plays a crucial role in task-oriented dialogs. To rep- resent the dialog policy for given task, we utilize policy skeleton, which delineates interaction pat- terns and encompasses business logic in the form of template dialog flows (Peng al., 2021b). The Policy Prompter is devised guide the static LLM in adhering to the policy skeleton PS, enabling the sequential generation of appropriate system actions a, and responses Tt-
Analogous the DST Prompter, the Policy Prompter (Figure 3) comprises four components: (i) a task instruction; (ii) a formatting example de- rived from another task/domain, consisting of a par-
"SQL: select from de where = v/ vii... s of". 'Wo assess helief instructions with diverse slot value ex- amples, revealing minor performance variations.

---

tial policy skeleton and its associated dialogue turn exemplar (in Appendix C); (iii) a policy skeleton for the previously predicted domain/task; and (iv) the test input, i.e., the dialog history he, generated belief state by. and obtained DB state Ct-
(Inform + Success) x 0.5 + BLEU. Additionally, we utilize BERTScore (Zhang al., 2020). Following Mehri and Eskenazi (2021), we per- form the next action prediction task on STAR (wherein the system actions and response templates are mapped one to one), which predicts next sys- tem action given the dialog history, We report the results using Fiscore (%) and accuracy(%)). Human Evaluation Metries. We conduct inter- active human evaluations (by five student helpers), following the evaluation protocol in the DSTC9 Track 1 challenge (Gunasckara et al., 2020). For each dialog session, students are mandated to in- teract with a dialog agent via natural language and assess the overall dialog quality employ- ing these five metrics: (i) Success w/o g(%), (ti) Success w/ g(%). (ii)) Understanding(1-
Policy Skeleton. Given that user behaviors and DB results jointly determine system actions and responses, policy skeleton is designed to cover all fundamental user behaviors and characteristic DB results, along with their corresponding system ac- tions and responses. 5 Considering the infeasibility of developing a multi-task/domain policy skeleton for every possible combination of tasks and do- mains, we opt to develop distinct policy skeleton tailored to each specific task and domain.
Following Mehri and Eskenazi (2021), our strat- egy converts the established dialog policy into series of template dialog turns X' that are logically arranged and concentrate on task completion:
(iv) Appropriateness((-S)) and (v) Turns. Full details are elaborated in Appendix
Compared Methods. We compare the proposed SGP-TOD with SOTA zero-shot transfer methods and zero-shot/few-sho prompting strategies. (We report the mean results of three different rums.)
X = (x: IN a', (u'=""""'''''''r (')))rre,
(5)
2;
Zero-shot transfer methods:
where æ, is a template dialog turn, which contains a user utterance u² or a DB state c², matching sys- tem action a', and system response r². N denotes the total number of template turns within the pol- icy skeleton (around 10-20 template turns depend- ing on the task complexity). In order to equip the frozen LLM with new capabilities or modify cur- rent ones, we only need insert, amend, or eliminate a few template turns within the policy skeleton.
BERT+S (Mosig et al., 2020) augments a BERT-base classifier (Devlin et al., 2019) with a system-side schema to predict system action.
SAM (Mehri and Eskenazi, 2021) is based on BERT-base, which uses a user-aware schema to predict the next system action.
ANYTOD-XXL (Zhao al., 2022) adopts T5-XXL. (Roberts al., 2022) to generate sys- tem actions and responses utilizing slot/action descriptions and a policy program. It is pre- trained on SGD dataset (Rastogi et al., 2020a).
4 Experiments
4.1 Experimental Setup
Datasets. (i) Two multi-domain dialog datasets:
Prompting methods:
Multiwoz 2.0 (Budzianowski et al., 2018) and
T-TO-CHHATGPT (Hudecek and Dusek, 2023) a prompting approach based on Chat- GPT, exploiting slot descriptions for tracking dialog states, fetching DB entries, and gener- ating responses. (G-TOD-CHAIGPT- and are in the zero-shot and few-shot settings, respectively.
Multiwoz 2.2 (Zang et al., 2020). (ii) Two
single-domain/task datasets: RADDLE (Peng et al.. 2021a,c) and STAR (Mosig et al., 2020) (single- task dialogs from the corpus, following the "happy path"). Details are elaborated in Appendix D.
Automatic Evaluation Metrics. We evaluate the end-to-end dialog generation performance using the same metrics as those listed in Budzianowski et al. (2018): Inform (%)) Success(%) (%), (%) (Papineni et al., 2002) and Combined(%) (%) judges the overall quality, defined as Combined
SOW-SHOT-CHATTE CHATGPT is a seotshot prompt- ing approach applied to ChatGPT, utilizing a few (i.e., k) training dialog turns as prompts. Optimal results are achieved with & - 15 on Multiwoz and A 10 on RADDLE.
'We do not enumerate every conceivable combination of user behaviors or potential database results, as schema engi- necring is not the primary focus of this study.
SGP-TOD (Ours) is compatible with any off- the-shelf LLMs. In this paper, we employ

---

Multiwoz
Multiwoz
Model
Inform
Succeea
ELEU
Combined
Inform
Success
ELEU
Combined
Fudl-she shol jine-toning (with 8.4k+ maiiiing didloge):
DAMD (Zhang el al., 2020)
76.33
60.40
16.60
84.97
SIMPLETOD (Hosseini-All et 2020)
84,40
70.10
15.01
92.26
SOLOIST (Peng et al., 2021a)
85.50
72.90
16.54
95.74
81.70
67.10
13.60
88.00
PPTOD (Su et al., 2022)
89.20
79.40
18.62
102.92
MARS (Sun et al., 2022)
88.90
78.00
19.90
103.35
88.90
78.00
19.60
103.05
Zeno-shot transter method (pre-trained on SGD);
ANYTOD-XXI
73.90
24.40
3.40
52.55
Few-shot prompting:
IG-TOD-CHAA -DDD-HATGPP--SS
20.00
7.17
SEw-SHOT-CHATEE
44,74
24.32
7.88
42.41
45.40
24.50
7.72
42.67
Zenosshot prompting:
IG-TOD-CHAN TTODDCCHATGPT-2
15.00
3.58
T-P-TTD--HATTPP (Ours)
64.56
54.05
7.17
66,48
64.70
54.70
6,96
66 66
SGP-TOD-CODEL (Ours)
71.67
52.55
7,91
70.02
75.50
52,30
6.62
70.53
SGP-TOD-GPT3.J (Ours)
83.88
69.87
9.09
82.00
72.50
9.22
86.47
Table I: End-to-end dialog generation evaluation results on tiwoz. Results SOLOIST, MARS, ANYTOD- XXL Multiwoz 2. are cited from Zhao al. (2022). Results of GG-TO--CHATGPP cited from Hudecek and Dusek (2023). Other results of the full-shot tinettnning methods are cited from He et al. (2022) and Sun et al. (2022). (Difference mean is significant with pc0.01.)
Model
Attraction
Train
Hotel
Restaurant
Info.
Succ.
BLED
Comtb.
Info.
Succ.
BLEU
Comb.
Tnfo.
Suce.
BLED
Comb.
Info.
Bucc.
BLED
Couh.
Few-shot fine- nuning (wink 30 traiiing dialogs):
SIMPLETOD SOLO157
65.66
46.97
5.85
62.17
59.00
44.00
7.07
58.57
33.50
40.00
7.70
$8.95
75.50
44.50
11.00
71.00
86.00
65.00
12.90
88.40
80.81
64.65
9.96
82.69
74.50
43,50
8.12
67.12
81.00
$5.50
12.80
81,50
Few shot prompring:
TEW SHOT CHAIGPT
75.00
67.00
8.22
79.23
79.80
65.66
8.12
80.85
$1.00
26.50
5.80
44.55
80.00
55.50
7.71
75.46
Zho-shott prmppting:
SGP TOD CHATGPT
95.00
94.00
7.13
101.65
76.77
74.24
6.75
82,26
76. 50
57.00
5.16
71.91
90.00
82.50
6.72
92.97
SGP-TOD-CO TODD-CODRX
98.00
93.00
10.45
105.95
78.79
70.20
8.56
83.50
69.
7.86
84.36
91.00
85.00
10.50
98.50
3GP-TOD-
96.00
82.
Table End-to-end dialog generation evaluation results RADDLE. The few-shot fine-tuning results are cited from Peng et al. (2021a). (Difference in mean is significant with p<0.01.)
ChatGPT, GPT-3.5 and Codex. Implementa- tion details are provided in Appendix E.
considerably outperforms the sho-shot prompting methods, i.e. IG-TOD-CHARGPI--S and FEW- SHOr-CHAIGPT (e.g., about 30 points improve- ment over Success). This suggests that provid- ing explicit and concise task instructions via task schema is preferable to imparting implicit task guid- ance through the selected dialog turns.
4.2 End-to-End Evaluation on Multiwoz
Results. We present the evaluation results in multi- domain contexts on Multiwoz Table 1. In addi- tion to the aforementioned methods, we include the results of SOTA full-shot fine-tuning approaches to facilitate a more comprehensive comparison. SGP- TOD obtains SOTA zero-shot performance, sub- stantially outperforming few-shot prompting ap- proaches across all metries, while even exhibiting competitive results in comparison to full-shot fine- tuning methods concerning Success and Inf orm. This confirms the effectiveness of integrating the task schema with the LLMs' proficient language processing capabilities. Comparison with Prompting Methods. SGP- TOD-CHATGPT distinctly surpasses the zero- shot prompting approach IG-TOD-CHAIGPT-ZZ with respect to Success (surpassing by 40%) and BLEU (exceeding by 3%). Morcover, SGP-TOD- CHATGPT, without requiring task-specific data,
Comparison with Zero-Shot Transfer Methods. Our SGP-TOD demonstrates a substantial advan- tage over ANYTOD-XXL, which necessitates task- specific pre-training and additional annotations, e.g. slot and action descriptions, over all the met- rics. This exemplifies the potency of SGP-TOD, which markedly reduces the necessity for human labor and computational resources.
Comparison with Full-Shot Fine-Tuning Meth- ods. SGP-TOD exhibits competitive performance over Inform and Success. The lower BLEU is due to a lack of linguistic variations of the tem- plate utterances, which is acceptable considering the trade-off between human effort and efficacy.

---

Model
Task transfer
Donain transfer
Model
FI/ES/ZS
Restaur ant--uranttEx Extt
Fi
Accuracy
Accuracy
Info. Succ.
BLBV BERTS.
Zem-shot mansfer
Without domainn-eelle relevant know lodge
Heave-one [une-taning with 2.56 training dialogs):
BERT+S
24.25
24,89
25.70
28,56
ChuiGPT
44.00
6,00
4.31
85.96
49.82
51.30
3.T-3.5
zs
34.00)
16.00
8,70
84.31
Zero-shon shot prompiing:
With prior knowledge on Restaur and
SGP-TOD WODDCODEX-INI
45.18
SGP-TOD-GPT3.5
47.67
48.27
49.76
50.39
SOLOIST
FT
78.00
0.00
10.62
87.24
SGP-TOD CODEX
49.78
51.01
52.72
53.66
CCP-TOD-CHNIGPT
88.00
34.00
5.45
86.11
SOP-TOD-GPT3.C --3.-E--..-EEEE
50.84
50.74
53.50
53.21
78
94,00
30.00
10.68
87.30
With knowledge on Restaurants ant Exxt
Table 3: Zero-shot end-to-end next action prediction evaluation results on STAR. (Difference in mean is sig-
SOLOIST+TEACH
FT
82.00
38.00
10.99
87.66
FEW-SHOT-GPT3..A TEACH
88.00
54.00
12.95
88.90
nificant with p=0.01.)
SGP-TOD-CHAR CONTCH
78
88.00
78.00
6.25
86.15
ACP-TOD-GPT3.5-ER
ZS
96.00
86,00
14.57
89.01
4.3 End-to-End End Evaluation on RADDLE
Table 4: End-to-end evaluation results domain
tension. FT: fine-tuning, FS: few-shot prompting, 7S:
Results. Table reports the results in single- domain settings on RADDLE. On all four dialog tasks, SGP-TOD demonstrates remarkable zero- shot performance that consistently surpasses both few-shot prompting and fine-tuning approaches. This results in substantial improvements of up to 12% in Inform, 45% in Success, and 19% in Combined metrics, while maintaining competitive BLEU scores. This evidence further substantiates the efficacy of SGP-TOD.
zero-shot prompting, Info.: Inform, Succ.: Success, BERTS.: BERTScore. (Difference in mean is signifi- cant with p<0.01.)
data, demonstrates enhanced proficiency compared to GPT-3.5 in interpreting action labels. In addition, appending template responses is presumed to facil- itate the explication of action labels for GPT-3.5. Impact of Different Task Schemas. SGP-TOD- CODEX-INI, utilizing an identical task schema as employed in training SAM, manifests commend- able performance. This result highlights that SGP- TOD as a flexible prompting strategy, compatible with any manually-crafted task schema.
4.4 End-to-End Evaluation on STAR
Setup. BERT+S, SAM are fine-tuned on source lasks/domains then zero-shot the held-out task/domain. SGP-TOD is presented with two formatting turns from the source tasks/domains. Results. Following Mchri and Eskenazi (2021), we report the zero-shot evaluation results in two set- tings, i.e., task transfer and domain transfer in Table 3. SGP-TOD, merely with two formatiing sample turns, demonstrates exceptional performance, passing or rivaling SOTA zero-shot transfer meth- ods in both settings. This outcome signifies that, even when faced with complicated business logic and system actions in dialog policies, the proposed SGP-TOD continues to exhibit commendable per- formance. Additionally, we investigate the impact of changing the number of training dialogs and formatting example turns in Appendix F.
4.5
End-to-End Evaluation on Domain Extension
Setup. We conduct experiments in domain exten- sion setting (Gasic et al., 2014; Lipton al., 2018) to assess the efficacy of SGP-TOD in adapting deployed task bots to incorporate novel function- alities. Following Zhang et al. (2022), we con- struct the Restaurant-exx corpus by extending the Restaurant in RADDLE with four new slots: [restaurant_diie _ishunt_dd_sh____ooei [value [piicee [start_time], and [end_ fond_time. Details are shown in Appendix J. Compared Methods.
ChatGPT, GPT-3.5 denote zero-shot prompt- ing that receive two formatting examples.
Impact of Different LLMs and Prompting For- mats. SGP-TOD-Come surpasses SGP-TOD- GPT3.5 while rivaling (with template responses affixed to action labels in the policy prompt, demonstrated in Figure 10 in Appendix M). We conjecture that Codex, bene- fiting from extensive pre-training copious code
CGP-TOD-CHAICEE represent our SGP-TOD implementation. with the Restaurant policy skeleton.
SGP-TOD-GPT3.
SOLOIST is trained with 50 training dialogs in Restaurant domain (reported in Table 2).
S is fine-tuning method en- hanced with machine teaching (Simard et al., 2017). We deploy SOLOIST to converse with
"ANYTOD-XX XXLL requires additional annotations, egg. be- lief descriptions, which makes it not suitable for STAR.

---

real users, then implement machine teach- ing obtain 10/50/50 annotated dialogs in Restaurant-ext for training, validating, and testing. We fine-tune SOLOIST with the gath- ered 10 training dialogs covering new slots.
Restaurant
Model
8u/o g t
3 g
Vod.
App.
34.00
30.00
2.18
2.10
PEW-SHOT- -HITICCHIIPTT
94,00
74.00
4.58
4.72
8.32
SGP-TOD-CHNIGH 100.00
92.00
4.86
4.88
7.28
Table Human evaluation results. S w/o g. S w/ g
PEW-SHOT-GPPT.55TEEA-S is the few-shot prompting strategy augmented with machine teaching. We use 10 randomly selected dialog turss from the collected 10 training dialogs as prompts (with peak performance at 10).
Success without with grounding; Und.: Understanding; App.: Appropriateness: T.: Turns.
Multiwez 2.2
Model
Inform
Succass
BLEU
Combined
SP-TOD-GPT3.5
82.00
72.50
9.22
86.47
SCP-TOD-CCAATGTEEXX,, SGP-TOD- GP3.5-EXT refer SGP-TOD with Restaurant-Ext policy skeleton, where we only add four template turns about four new slots to the policy skeleton of Restaurant.
policy policy DDB
81.80
56.20
6.63
75.63
52,30
73.42
policy -DB -beliel"
38.60
33,90
6.29
42.54
Table 6: Ablation study results the impact of the three components in the proposed SGP-TOD and the database expertise on Multiwoz 2.2 using GPT-3.5. -policy: removing Policy Prompter, -DB: removing database expertise, -helief: removing DST Prompter.
Results. In Table 4, 5-EXX surpasses and notably SGP-TOD-GPT3.55 all other evaluated approaches by a substantial mar- gin over all the metrics. This demonstrates the strong adaptability of our SGP-TOD in accommo- dating novel functionalities, revealing its immense potential for lifelong learning. Two interactive dia- log examples are supplied in Appendix K. Comparison with Approaches Augmented by Ma- chine Teaching. SOLOIST yields zero Success, a predictable result given its lack of awareness regarding the new features. Augmented by ma- chine teaching, SOLOIST+TEACH substantially im- proves SOLOIST in terms of Inform and Success. Nevertheless, relying solely on prior Restaurant knowledge, both SGP- TOD-CHATGPT and SGP- TOD-GP3.5 exhibit performance on par with SOLOIST+TEACH, demonstrating that SGP-TOD provides enhanced robustness in zero-shot gen- cralization. Moreover, SGP-TOD-GP3.5-EX. obtains substantially higher Success rates than SOLOIST+TEACH (a rise of 48%) and FEW-SHOT- GPT3.5+TEACH (an increase of 32%). Compared fin-tuuing/prompping strategies utilizing addi- tional dialogs corrected through machine teaching, SGP-TOD facilitates a more agile adaptation to novel functionalities by merely modifying template turns within the task schema.
Results. In Table 5, our proposed SGP-TOD- CHATGPT attains remarkably high performance in zero-shot context, consistently outpacing SOLOIST and FEW-SHOT-CHATGPT across all metrics. Particularly, regarding Success w/ g. SGP TOP-CHAGPPT significantly surpasses FEW-SHOT-CHATGPT (by 18%) and SOLOIST (by 62%), illustrating its proficiency in accomplishing tasks within real-world scenarios. Furthermore, CGP-TOD-CHATGPE exhibits formance (demonstrated in Appendix I). A detailed analysis is provided in Appendix I.
stable per-
4.7 Ablation Study
In Table 6, we study the impact of the three com- ponents SGP-TOD (namely, Policy Prompter. DST Prompter, and LLM) as well as the databasc expertise, on Mul tiwoz 2.2 utlizing GPT-3355. Com- bining the three elements in SGP-TOD with the database expertise produces the optimal result, un- derscoring the value of enhancing the LLM with the task schema and external database information. Detailed analyses are provided in Appendix G.
Conclusion
We present SGP-TOD, a schema schema-guided prompt- ing strategy aimed at the expeditious construction of end-to-end task bots. relying exclusively on LLMs and the corresponding task schema. Em- ploying the symbolic knowledge task schema. SGP-TOD guides fixed LLMs to generate suitable responses for novel tasks in a zero-shot fashion. Empirical findings on four well-studied datasets
4.6 Interactive Human Evaluation
Setup. We conduct interactive human evalua- tions Restaurant domain to evaluate the perfor- mance of SOLOIST, WWW-SHOT-CHAIGPT. SGP- TOD-CHATGPT (reported in Table 2), with 50 dialogs gathered for analysis, respectively. Details can be found in Appendix H.

---

reveal that SGP-TOD attains remarkable SOTA zero-shot performance, using both automatic and human evaluations. For future work, we plan to ex- plore the use SGP-TOD to develop personalized chatbots by utilizing pertinent task schema.
individuals or groups (Zhou et al., 2022) and avoid any negative impact on users' well-being (Bergman et al.. 2022). Instances of these Prompters are pro- vided Appendix M. Furthermore, subsequent research endeavors may consider utilizing the Ope- nAI moderation API in conjunction with other related APIs to systematically filter out unsuitable user inputs and system responses.
Limitations
This work is accompanied by two primary limita- tions. (i) Data contamination (Brown et al., 2020; Madotto et al., 2021) in prompt-based zero-shot learning pertains to the potential presence of test samples during the LLM pre-training phase. Given that data utilized for pre-training LLMs, such as ChatGPT and GPT-4, remains undisclosed and con- tinuously expands, verifying data contamination presents a formidable challenge. Consequently, our research cannot preclude data contamination in the experimental process, deferring more com- prehensive investigation to future endeavors. Nev- ertheless, undertake domain-extension exper- iments (Table 4 in Section 4.5), subjecting our proposed SGP-TOD evaluation on a novel test set (currently not publicly available), encompass- ing recently obtained and annotated human-bot di- alogs. The remarkable zero-shot performance of SGP-TOD demonstrates its substantial potential for adeptly adapting innovative functionalities. without reliance task-specific data.
A Acknowledgements
This Project is partially supported by the HKSARG General Research Fund (Ref No. 14207619). We would like to express our gratitude to Xiaohan Feng and Haohan Guo for their valuable comments.
(ii We employ the manually-craficn task schema as prompts to steer the LLMs towards gen- erating suitable responses on novel tasks. As illus- trated Table 3 of Section 4.4, SGP-TOD exhibits minor performance discrepancies when implement- ing disparate task schema formulated by various authors. Notwithstanding such variations, our ob- jective is to offer foundational basis for schema- guided LLM prompting: future research may inves- tigate approaches designing more efficient task schema, i.e., diverse formats and coverage.
Ethics Statement
Throughout the interactive human evaluations and domain-extension experiments, all participating student helpers were informed of the research ob- jectives prior to the collection and annotation of human-bot dialog logs. Their privacy was ensured remain protected and undisclosed during the re- search period. Each participant received equitable remuncration.
The Prompters utilized in this research incorpo- rate no language that discriminates against specific
/nttps://platform. wwwnaa..co//ooeeeooooopop wtoo///oc////eooooooooooo htt////oo/g/g/dee./// /es
moderatiinn/verriiew

---

References
Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brenman Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022a. Palm: Scaling language modeling with pathways.
A. Stevie Bergman, Gavin Abercrombie, Shannon Spruit, Dirk Hovy, Emily Dinan, Y-Lan Boureau, and Verena Rieser. 2022. Guiding the release of safer E2E conversational Al through value sensitive de- sign. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dia- logue, pages 39-52, Edinburgh, UK. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901 Curran Associates, Inc.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eek, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022b. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311
Sébastien Bubeck, Varun Chandrasekaran, Ronen Fl- dan, Johannes Gehrke, Eric Horvitz. Ece Kamar, Pe- ter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang, 2023. Sparks of artificial general in- telligence: Early experiments with gpt-4.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Ultes Stefan, Ramadan Os- man, and Milica Gašié. 2018. Multiwoz a large- scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Giovanni Campagna, Agata Foryciarz, Mehrad Morad- shahi, and Monica Lam. 2020. Zero-shot transfer learning with synthesized data for multi-domain dia- logue state tracking. In Proceedings of the 58th An- nual Meeting of the Association for Computational Online. Association for
Linguistics, pages Computational Linguistics.
Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao. Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023. Auggpt: Leveraging chatgpt for text data augmentation.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto. Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
Jacob Devlin, Ming-Wai Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre- pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Minneupolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages Linguisties.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Binding language models in symbolic languages. ICLR, abs/2210.02875.
Association for Computational
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts,
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong

---

Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey on in-context learning.
Zhaojiang Lin, Bing Liu, Andrea Madotto. Seungwhan Moon, Zhenpeng Zhou, Paul A Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, et al. 2021a. Zero-shot dialogue state tracking via cross- task transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7890-7900.
Milica Gasic, Dongho Kim, Pirros Tsiakoulis, Cather- ine Breslin, Matthew Henderson, Martin Szummer, Blaise Thomson, and Steve J. Young. 2014, In- cremental on-line adaptation of pomdp-hased dia- logue managers to extended domains. In INTER- SPEECH 2014, 15th Annual Conference of the Inter- national Speech Communication Association, Singa- pore, September 14-18, 2014, pages 140-144. ISCA.
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul A Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, and Rajen Subba. 202 Leveraging slot descriptions for zero-shot cross cooss-domain dialogue statetracking. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5640-5648.
Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D'Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric. Behnam Hedayatnia, Karthik Gopalakrishnan. Yang Liu, Chao- Wei Huang, et al. 2020. Overview of the ninth dialog system technology challenge: Dste9, arXiv preprint Sidi::201.064855
7achary Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, and Li Deng, 2018. Bbq-networks: Efficient exploration in deep reinforcement learning for task-oriented dialogue systems. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 32.
Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang. Min Yang, Fei Huang, Luo Si, et al. 2022. Galaxy: A generative pre-trained model for task-oriented dialog with semi- supervised learning and explicit policy injection. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100- 1 11, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Ehsan Hosscini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33:20179- 20191.
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie. Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In- context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627-2643, Abu Dhabi, United Arab Emirates. Association for Com- putational I. Linguistics.
Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt- based learning for dialogue systems. arXiv preprint arXiv:2110.08113
Shikib Mchri, Yasemin Altum, and Maxine Eskenazi. 2022. LAD: Language models as data for zero-shot dialog. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dia- logue, pages 595- 995, Edinburgh, UK. Association for Computational Linguistics.
Vojtech Hudecek and Ondrej Dusek. 2023. Are Iims all you need for task-oriented dialogue? CoRR, abs/2304.06556.
Mihir Kale and Abhinav Rastogi. 2020. Template guided text generation for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 6505- 6520. Association for Computational Linguistics.
Shikib Mehri and Maxine Eskenazi. 2021. Schema- guided paradigm for zero-shot dialog. arXiv preprint arXiv:2106.0708
Johannes EM Mosig, Shikib Mehri, and Thomas Kober. 2020. Star: A schema- schema-guided guided dialog dataset for trans- fer learning. arXiv preprint arXiv:2010.11 18853.
Zekun Li, Wenhu Chen, Shiyang Li, Hong Wang, Jing Qian, and Xifeng Yan. 2022. Controllable dialogue simulation with in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4330-4347, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Maxwell I. Nye, Michael Henry Tessler, Joshua B. Tenenbaum, and Brenden M. Lake. 2021. Improv- ing coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In Advances in Neural information Processing Sys- tems 34: Anmual Conference on Neural Information Processing Systems 2021, NewrlPS 2021, December 6-14, 2021, virtual, pages 25192-25204.
Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji. Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, and Nan Duan. 2023. Taskmatrix.at ai: Com- pleting tasks by connecting foundation models with millions of apis.
OpenAI. 2022. large-scale generative pre-training model for conversation. OpenAl blog.
OpenAI. 2023. Gpt-4 technical report.

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages Curran Associates, Inc.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020a. To- wards scalable multi-domain conversational agents: The schema-guide dialogue dataset. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Whirty-Second Innovative Applica- tions of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8689-8696. AAAI Press.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 31 1-3188, Philadelphia, Pennsylvania, USA, Association for Computational Linguistics.
Abhinav Rastogi. Xiaoxue Zang. Srinivas Sunkara. Raghav Gupta, and Pranav Khaitan. 2020b. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 5689-8696,
Baolin Peng. Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie. Yu Hu, Qiuyuan Huang. Lars Liden, Zhou Yu. Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.1 12813.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin. Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Mare van Zee, Jacob Austin, Sebas- tian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jas- mijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Thorph Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard. Noah Fiedel, Mark Omernick, Brennan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scaling up models and data with t5x and seqio.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2021a. Soloist: Building task bots at scale with transfer learning and machine teaching. Trunsactions of the Association for Computational Linguistics, 9:807-824.
Baolin Peng, Chunyuan Li, Zhu Zhang, Jinchao Li, Chenguang Zhu and Jianfeng Gao. 2021b. Syn- ergy: Building task bots scale using symbolic knowledge and machine teaching. arXiv preprint arXiv:21 10. 11514.
Baolin Peng. Chunyuan Li. Zhu Zhang, Chenguang Zhu, Jinchao Li, and Jianfeng Gao. 2021c. RADDLE: an evaluation benchmark and analysis platform for ro- bust task-oriented dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ACI/IJCNIP 2021, (Volume 1: Long Pupers), Virtual Event, August 1-6, 2021, pages 4418-4429. Associa- tion for Computational Linguistics.
Darsh J Shah, Raghav Gupta, Amir A Fayazi, and Dilek Hakkani-Tur 2019. Robust zero-shot cross-domain slot filling with example values. arXiv preprint MxXiv:996.00870
Richard Shin and Benjamin Van Durme. 2022. Few- shot semantic parsing with language models trained on code. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, pages 5417-5425, Seattle, United States. Association for Computational Linguisties.
Kun Qian and Zhou Yu. 2019. Domain adaptive dialog generation via meta learning. Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 2639-2649, Florence, Italy. Association for Computational Linguistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language process- ing task solver?
Swadheen Shukla, Lars Liden, Shahin Shayandeh, Es- lam Kamal, Jinchao Li, Matt Mazzola, Thomas Park. Baolin Peng, and Jianfeng Gao. 2020. Conversa- tion Learner machine teaching tool for building dialog managers for task-oriented dialog systems. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties: System Demonstrations, pages 343-349, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 18)::

---

Patrice Y. Simard, Saleema Amershi, David Maxwell Chickering, Alicia Edelman Pelton, Soroush Gho- rashi, Christopher Meek, Gonzalo A. Ramos, Jina Suh, Johan Verwey, Mo Wang, and John Werns ing. 2017. Machine teaching: A new paradigm for building machine learning systems. CoRR, abs/1 hls/1707.0674..
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang. Vic- tor Zhong, Bailin Wang. Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer and Tao Yu. 2022. UnifiedSKG: Unifying and multi salli-t structured knowledge grounding with text-to-text language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 602-631, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2022. Multi-task pre pre-training for plug-and and--lay task-oriented dialogue system. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
Haipeng Sun, Junwei Bao, Youzheng Wu, and Xi- aodong He. 2022. Mars: Semantic-aware contrastive learning for end-to-end task-oriented dialog. arXiv preprint arXiv: 120.08977
Dian Yu, Luheng He, Yuan Zhang, Xinya Du, Panupong Pasupat, and Qi Li. 2021. Few-shot intent classifi- cation and slot filling with retrieved examples. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 734-749. Association for Computational Linguistics.
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang. Haojun Huang. Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie. 2023. On the robustness of chatgpt: An adversarial and out-of-distribution perspective.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online. Association for Computa- tional Linguistics.
Qingyue Wang, Yanan Cao, Piji Li, Yanhe Fu, Zheng Lin, and Li Guo. 2022a. Slot dependency model- ing for zero-shot cross-domain dialogue state track- ing. In Proceedings of the 29th International Confer- ence on Computational Linguistics, pages 510-520, Gyeongju, Republic of Korea. International Commit- tee on Computational Linguistics.
Tianyi Zhang*, Varsha Kishore Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi. Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, et al. 2022b. Super- naturalinstructions:: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference Empirical Methods in Natu- ral Language Processing, pages 5085-5109.
Xiaoying Zhang, Baolin Peng, Jianfeng Gao, and He- len Meng. 2022. Toward self-learning end-to-end task-oriented dialog systems. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 516-530.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals. Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilitics of large language models.
Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. Task- oriented dialog systems that consider multiple appro- priate responses under the same context. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9604-9611.
Jeffrey Zhao, Yuan Cao, Raghav Gupta, Harrison Lee, Abhinav Rastogi, Mingqiu Wang. Hagen Soltau, Izhak Shafran, and Yonghui Wu. 2022. Anytod: A programmable trie-tri dialog system. arXiv preprint
Jason D. Williams and Lars Liden. 2017. Demonstra- tion of interactive teaching for end-to-end dialog con- trol with hybrid code networks. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dia- logue, Saarbrücken, Germany, August 15-17, 2017, pages 82-85, Association for Computational Linguis- tics.
Tiancheng Zhao and Maxine Eskenazi. 2018. Zero-shot dialog generation with cross-domain latent actions. In Proceedings of the 19th Anmual SIGdial Meeting on Discourse and Dialogue, pages 1-10, Melhourne, Australia, Association for Computational Linguistics.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini- Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable muuli--omaan state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Compu- lational Linguistics, pages 808-819. Florence, Italy. Association for Computational Linguistics.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance language models. Proceedings of the 38th International Conference

---

on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706.
DST Prompter
Task Instruction
PMLR.
Fallowing the instructions, predict the ballef state based on the history.
Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng. 2022. Towards identifying social bias in di- alog systems: Framework, dataset, and benchmark. Findings the Association for Computational Linguistics: EMNLP 2022, pages 3576-3591, Abu Dhabi, United Arab Emirates, Association for Com- putational Linguistics.
Belief instructions
resiaurant name: pizza hut tht,, golden wak, etc.; food dont care, chinese, mediterranean, bittish, etc:; pricerange dantcore, cheap, moderate, expensive; area
centre, north, south, west booking day monday, tuesday, wednesday, thureday, friday, saturday, sunday, booking_ people m 1,2.3,445,6.7 booking_tme
13:30,
attraction attraction lype swimmingpoo. theetre, archilecture, museum, nightclub, boat, park. college, concerthall, enteriainment, multiple sports, cinema; vilaage
the cherry hinton rightclub, etc:; area easl, north, south, wesl
centre,
hotel name huntingdoo marrittt hotel, and guest house, pricerange donl cheap,
expensive; dont care, centre, east, north, south, west stars don' care, 0,1,2,3,4.5; parking dont care, yes, no: intamet don' care, yes, no; type hotel, guest house: booking day monday, lueeday, elc.; booking people 1,2.3, e.5. booking_ slay 1,2.
10:45, 12:06,
cambridge, etc.; day = monday, tunsday, wadnesday, thursday. friday saturday, sunday arriveby 21:00, 09:45, etc; departure cambridge, stansted airport, etc. booking people 1,2,3.4.5,B
tnxi leaveat 08:45, 16:15, atc. destination saint john's college, kettle's yard, galleria, etc. departure hunlingdon cineworld cinema, bridge guest etc. arriebyy 17:15, 17:30, etc.
Figure Detailed belief instructions in DST Prompter.
A Zero-Shot Task-Oriented Dialog Modeling.
Table summarizes four main research directions in zero-shot task-oriented dialog modeling: slot filling (SF), dialog state tracking (DST), end-to-end policy management (E2E policy) and end-to-end dialog generation (E2E dialog).
B Detailed Belief Instructions in DST Prompter
Figure 4 shows the detailed belief instructions DST Prompter.
C A Formatting Example in Policy Prompter
Figure 5 presents formatting example in Policy Prompter.
D Experimental Setup
Datasets.
Multiwoz 2.0 (Budzianowski et al., 2018) is
a multi-domain task-oriented dataset, which

---

Model
Task
Schema types
Training strategy
Eine-tuning
Pre-training
Prompting
ROBUSTSP (Shalh sl., 2019)
SF
slot names/value examples
TRADE (Wo al, 2019)
namestvalue examples
2STL-SD (Campagna al., 2020)
DST
ontology, dialng templates
V (asynthesized data)
S-DST (Rastogi et ol., 2020h)
DST
Herviie,. intent names descriptooss
slot namesiiescripionn
TSDST
numevdescriptions
TRANSPERQA (Lin et 2021a)
DST
slot names/valur examples
V (QA tasks)
(C- (Ilu et al. 2022)
DST
slot names/value examples
SDM-DST (Wang et al., 2022a)
DST
slot mames/value examples
BERT+S (Mosig
2020)
E2E policy
system-side policy skeletons user-awane policy skeletons
SAM (Mehzi and Eskenazi, 2021)
E2E policy
7SDG (Zhao and Eskenazi, 2018)
F2E dialog
ontology, response templates
DAML (Qian and Yu, 2019)
E2E dialog
ontology, response templates
policy program
ANYTOD (Zhao et al., 2022)
E2E dialog
+slot narmes/value examples +slot descriptions
/
V Cheterogencous tasks)
action samesshllles/eeseee
IG-TOD (Hudccok and Dusek, 2023)
E2E dialog
slot names
+slot descriptions
SGP-TOD (ours)
F2F dialog
user aware pulicy skeletons (esler names/value examples)
Table 7: Zero-shot task-oriented dialog modeling. (Schema items enclosed in parentheses are required only when accessible.)
contains 4.438/1.000/1.00 dialogs for train- ing/validating/estingg spanning seven do- mains: restaurant, attraction, train, hotel, taxi, police, and hospital. Multiwoz 2.0 is anno- tated with belief states and system actions.
tributes should be asked (for example, ask for the user's s name before asking for the hotel name), how to query a database, etc.
Automatic Evaluation Metrics. We evaluate the end-to-end dialog generation performance using the same metrics as those listed Budzianowski et al. (2018): (i) Inform(%) assesses whether the agent returns an acceptable entity. (ii) Success (%) determines if the agent appropriately responds to each attribute request. (iii BLEU(%) (Papineni et al., 2002) measures the word over- lap of the generated response against the human response in the corpus. (iv) Combined(%) judges the overall quality, which is defined as Combined = (Inf orm + Success) x 0.5 + BLEU. Additionally, we utilize BERTScore (%) (Zhang* et al., 2020), which focuses on computing semantic similarity between the generated responses and the ground truth, and correlates better with human judgments. Following Mehri and Eskenazi (2021), we per- form the next action prediction task on STAR, which predicts next system action based on the dialog his- tory. Since the system actions and deterministic response templates
Multiwoz 2.2 (Zang et al., 2020) is im- proved version of Multiwoz 2.0. encompass- ing refined belief state annotations, slot de- scriptions, user action annotations, etc.
RADDLE (Peng et al., 2021a,c) consists of four single-domain dialog datasets derived from Multiwoz 2.0 (i.e., restaurant, train, hotel, attraction), reorganized by Peng al. (2021a). Each corpus contains 50/50/200 dialogs for training/validaaingttstinn expect for 100 test- ing dialogs attraction domain.
STAR (Mosig al., 2020) includes 24 tasks in 13 domains (e.g., "apartment" domain comprises "apartment- and "apartment- schedule"), requiring the dialog model to con- form to the provided task schema. We use 2,688 single-task dialogs from the corpus, which follow a "happy path", i.e. the user is not instructed to execute any action excced- ing the schema's expectations. Without ad- ditional annotations, STAR only provides flow chart diagram that outlines the dialog policy for each task. The flow chart outlines the task, including the sequence in which at-
mapped one to one in STAR corpus, we believe the end-to-end next action pre- diction task falls within end-to-end dialog model- ing, following Mosig et al. (2020); Mehri and Eske- nazi (2021). 1). In addition, we report the results using weighted Fiscore (%) and mean accuracy(%) (%).

---

Policy Prompter
fers matched slot values compared the database record. (ii) Understandingg(-S) quantifies the accuracy with which the agent comprehends user utterances. (iv) Appropriateness((1. signifies the naturalness, appropriateness and fluency of an agent response. (v) Turns denotes the average number of dialog turns within successful dialog sessions.
Task instruction
Following the instructons, genarate appropriate response based on the history.
Formatting example (from other task/domain)
Palisy skeleton
(1) user: Im looking for information |attraction, name). action: attraction (infomm (name, address, area, entrance fee)) system: [attraction name) is in the (value area] and their address [attraction address). the entrance fee is (value cound] pounds. [eos]
E Implementation Details
(2) user: What the phone number? action: sttraction (inform (phone), require (more)) system: [ettraction [pooee] is there anything else can for you? [sos]
Regarding SGP-TOD:
LLMs: We employ ChatGPT ("gpt-3.5- turbo"), GPT-3.5 ("exxtdavinci-003") and Codex ("code-davinci-0022) as the fixed LLMs to implement the proposed SGP-TOD. Throughout the evaluation, we set temperature to 0.5.
nead the areallocation and postcoda. action:
attraction (inform (area, postcode)) system: The lattraction name] is the (value__ area), their postal code jottraction postoode]. [eos]
(4) user:
That is all need loday. thank you! action: attrection
(ganeral (byel) system: You welcome. thank you for calling the (value_ _pacc_] town info [value__nt [vallu__area] have a great day, [sos]
(5) DB:
attraction
match. action: attraction (inform results matching your query.
(nane)) system: try a different area or lype? [eos]
DST Prompter belief instruction: In the con- text of multi-domain scenarios, the belief in- structions encompassing all domains are incor- porated, while solely the target domain's be- lief instruction is introduced in single-domain settings.
(6) altraction system: We have jattraction [acectionnnmme)___e (oos)
(name))
(?) attraction
action: attraction (inform (choices),
request (area)] system: There are [valuee_rounf] which area were you looking at? [eos]
history
user: Hello, I'm looking for the museum of classical archaeology. Certainly. the
Policy Prompter - policy skeleton: For the Multiwoz datasets, we manually construct the policy skeleton through observing a few di- alogs in the training corpus, following Mosig et al. (2020); Mehri and Eskenazi (2021). In the case of the STAR corpus, we employ flow chart diagrams and several dialogs to develop the policy skeleton, following the guidelines set forth by Mehri and Eskenazi (2021). We integrate the relevant user template utterance and the system action into the policy skeleton, thereby augmenting the LLM's understanding of directives, the absence of belief anno- tations. The prompt examples for the STAR dataset are shown in Appendix M.
could you tell more about it? (attraction name] located at (attraction, address] and entrance, phone (altraciion phone). do you also need the postcode Yes, I'd lke that. thank you much!
attraction archseology DB: Attraction one match
SQL:
museum of classical
attraction (inform (postcode), require (more))
system: The posicode for the [attraction name) is [attraction
datraction_postoade] postoade]. Is there anything else can help you with
Figure A formatting example Policy Prompter.
Human Evaluation Metries. We employ inter- active human evaluations to assess the quality of dialog agents, following the evaluation protocol in the DSTC9 Track 1 challenge (G (Gunasekara al., 2020). We recruit student helpers to help with eval- uations. For each dialog session, student helpers are provided with a goal and accompanying in- structions, subsequently necessitating a discourse with the agent to achieve the goal via natural lan- guage. Upon the conclusion of each dialog sion, students are mandated assess the overall dialog quality employing these five metrics: (i) Success w/o g(%) evaluates whether the agent ac- complishes the task. (ii) Success w/ g(%) judges whether the agent accomplishes the task and of-
Formatting example: Following the zero-shot scenario in Wang et al. (2022b), we insert one formatting example from different tasks (fixed through the experimental procedure) into the prompt. The formatting example employed within DST Prompter/Policy Prompter is ran- domly chosen from the training corpus of dif- ferent tasks/domains, conforming to zero-shot scenario proposed by Wang et al. (2022b). We appraise multiple randomly selected format- ting examples, the evaluation results reveal mi- nor deviations. In the experiments on domain

---

extension (Section 4.5) and ablation analy- sis (Section 4.7), we employ the same (two) formatting exemplar turns originating from other domains within the RADDLE corpus for all prompting techniques.
various configurations of k. the optimal re- sults manifest with 15 on Multiwoz (2.0 and 2.2), and on RADDLE, exhibiting no further substantial enhancements.
SGP-TOD (Ours) is schema schema-guided guided prompt- ing strategy, which is compatible with any off-the-shelf LLMs. In this paper, we employ ChatGPT ("spt-3.5-turbo"), GPT- 3.5 ("lext-davinci-0033) and Codex ("code- davinci-002") as the fixed LLMs. Following the zero-shot scenario in Wang et al. (2022b), we insert one formatting example from dif- ferent tasks (fixed through the experimental procedure) into the prompt. More implemen- tation details are provided in Appendix E.
Regarding compared methods:
(i) Zero-shot transfer methods:
BERT+S (Mosig et al., 2020) is a schema- guided method that augments BERT- classifier (Devlin et al., 2019) with provided system-side schema predict the next system action.
SAM (Mehri and Eskenazi, 2021) represents schema-guides model based on BERT-base. which aligns the dialog context a user-aware schema to predict the next system action.
F Zero-Shot End-to-End Evaluation Results STAR
Figure 6 exhibits the zero-shot evaluation results on STAR, utilizing varying amounts of training di- alogs (ranging from to 1,000) and formatting example turns (spanning from 1 to 10) from source domains/tasks. SGP-TOD, merely with two for- matting sample turns, achieves superior compa- rable performance compared BERT+S, SAM, which are fine-tuned on adequate source data. We observe that SGP-TOD, employing only two formatting sample turns, attains superior or com- mensurate performance in terms of both Fiscore and Accuracy, when compared to SAM trained with 1,000 dialogs. Given that single dialog tains more than 10 dialog turns, this result suggests that SGP-TOD diminishes labeling expenses by a minimum factor of 1,000. Furthermore, it note- worthy that augmenting the quantity of formatting exemplar turns exerts a negligible influence on the performance of SGP-TOD.
ANYTOD-XXL (Zhao et al., 2022) adopts neural LM to track dialog states and user actions utilizing slot and action descriptions. Then a program that outlines a predefined task policy executed to recommend appropri- atc system actions. Upon considering these system actions, an LM generates the ultimate system action and formulates the correspond- ing template response using the approach pro- posed by Kale and Rastogi (2020). ANYTOD- XXL is implemented on T5-XXL (Roberts et al., 2022) and pre-trained on SGD dataset (Rastogi et al., 2020a)
Prompting methods:
IG-TOD-CHATGPT (Hudecek and Dusek. 2023) a prompting approach based on ChatGPT that leverages the dialog context and manually-crafted slot descriptions as the prompt, to track dialog states, fetch DB entries, and produce responses. IG-TOD- CHATGPT-ZS and IG-TOD-CHATGPT- are in the zero-shot and few-shot settings, re- spectively.
G Ablation Study
Table 8 exhibits the findings from an ablation in- vestigation, addressing the effects of the three inte- gral aspects of SGP-TOD in conjunction with the database expertise, implemented on Multiwoz 2.0 and 2.2, employing GPT-3.5.9 Combining the three elements in SGP-TOD with the database expertise produces optimal results across both datasets. The removal of the Policy Prompter, database knowl- edge, and DST Prompter leads to consistent de- clines in all evaluation metrics, underscoring the
FEW-SHI WWW-SSOT-COE CATTT-CHTTPPTT is few-shot prompt- ing strategy implemented on ChatGPT, where use few (i.e., k) dialog turns, randomly sampled from the training corpus to instruct ChatGPT on task execution. Upon evaluating
"The Schema-Gmided Dialog (SGD) dataset constitutes a comprehensive, large-scale, multi-domain corpus encompass- ing over 16,000 dialogs that span across 16 distinct domains.
"We inject the same two formatting example turns into the prompt throughout the evaluation.

---

50
40
40
Maan F1
Mean F1
30
30
20
SAM
SAM
UGP-TOD-GPT
SGP-TOD-GPT3
SGP-TOD-Coder
10
SGP-TOD-Cadex
BGP-TOD-GPTS-E22
SGP-TOD-GPT3
10
100
1000
10
100
1000
dia dialoos/eeample turns
dialogs/example tums
Task transfer
Domain transfer
50
50
45
Mean Accuracy
40
Mean Accuracy
35
30
30
25
20
BERT+S
20
BERT-S
SCP-TOD-GPTS
SGP-TOD-GPTS
15
SGP TOD E2
SGP TOD GPTD-EEE
1000
# dia dlallogs/example turns
dialogs/example tums
(c) Task transfer
(d) Domain transfer
Figure 6: Zero-shot end- to-end evaluation results on STAR with different numbers of training dialogs (1, 10, 100. 1,000) / demonstration example turns (1, 10) from source domain/tasks.
value of enhancing the fixed LLM with the task schema and external database information.
H Human Evaluation Details
Specifically, GPT-3,5 (in the final row) exhibits commendable zero-shot performance, highlighting the need of exploiting its superior zero-shot gen- eralization capabilities in dialog generation tasks. Additionally, Disabling the Policy Prompter in- curs a discernible decline in performance regarding Success (approximately 16%) and BLEU (roughly 3%), as the Policy Prompter's 's primary function is to provide task completion guidelines and interaction patterns. Eliminating the database expertise pri- marily reduces Success (by approximately 4%), implying that incorporating database information contributes task completion. Lastly, excising the DST Prompter engenders considerable diminu- tion in performance concerning Inform (around 43%) and Success (nearly 18%), due to the DST Prompter's intended purpose of assisting the frozen LLM in apprehending the dialog context.
We enlisted 5 student helpers (i.e., undergraduate students possessing basic proficiency in English communication) to participate in the evaluations. For each dialog agent, we collected 50 dialogs for analysis. Followed the methodology proposed by Li al. (2022), we generated user goals through the subsequent techniques: (i) Randomly selecting slots and slot values within the Restaurant do- main from RADDLE corpus to construct a user goal; (ii) Replacing the slot values of the user goals randomly chosen dialogs from the Restaurant cor- pus with corresponding new values from randomly sampled database entries, thus forming new user goal; (iii) Merging the user goals of several domly selected dialogs from the Restaurant cor- pus to create a composite user goal. Lastly, we randomly chose 50 distinct user goals from these newly generated goals.

---

Multiwoz 2.0
Mul tiwoi 2.2
Model
Inform
Succees
BLEU
Combined
Inforn
Success
BLEU
Combined
SP-TOD-GPT3.5
83.88
69.87
9.09
85.97
82.00
72.50
9.22
86.47
-policy
82.28
55.65
6.51
75.48
81.80
56.20
6.63
75.63
-policy -DB
81.20
50.95
6.48
72.56
81.40
52.30
6.57
73.42
-policy -DB -belicf
38.74
33.13
6.18
42.12
38.60
33.90
6.29
42.54
Table 8: Ablation study on the impact of the three components in the proposed SGP-TOD and the database expertise on Multiwoz using GPT-3.5. -policy: removing Policy Prompter, -DB: removing database information, -belief: removing DST Prompter.
I Human Evaluation Results
SOLOIST is trained with 50 training dialogs in the Restaurant domain (previously reported in Table 2).
Figure 7 shows the interactive human evaluation results. SGP-TOD- WOD-CCATGPPT exhibits more stable performance. In contrast to the automated evaluation results shown in Table 2, FEW-SHOT- CHATGPT significantly outperforms SOLOIST over all metrics. This indicates that corpus- corpus-based evaluations might be biased, given that real user inputs tend to be more dynamic, complex, even with noise. Notably, sistently excels compared to the other methods in both evaluations, implying its robustness in han- dling diverse user inputs.
SOLOIST+ TEACH is fine-tuning method en- hanced with machine teaching (Simard al., 2017). Machine teaching an efficient ap- proach to equip deployed task bots with the ability to handle new functions by correct- ing representative failed human-bot dialogs. We deploy SOLOIST to converse with real users, then implement machine teaching via Conversational learner (Shukla al., 2020). an effective machine teaching tool, obtain 10/50/50 examples in Restaurant-ex for training, validating, and testing. Finally, we fine-tune SOLOIST with gathered 10 training dialogs covering four new slots, resulting in dialog agent SOL.OISTTEAACH
con-
J More Details and Results on Domain Extension
Setup. Following Zhang al. (2022), con- struct the Restaurant-ext corpus by extending the pre-existing Restaurant RADDLE (Peng et al., 2021c) with additional functions. Specif- ically, we introduce four new slots: [restau- rani_dish],
FEW-SHOT-GPT3. GPTH5TT-PTT..++TAATTT the few-shot prompting strategy augmented with machine teaching. Based on GPT-3.5, we utilize 10 randomly selected dialog turns from the col- lected 10 training dialogs the prompt (with peak performance at 10), resulting in FEW- SHOT-GPT3.5+TEACH
[start_lime], and The initial slot pertains to recommen- dations for signature restaurant meals, while the final three concern delivery service details. All database entries are updated with corresponding values. Table 10 exhibits a dialog example on do- main extension. The associated pestaurant-EEr database entry is illustrated Table 9.
SGP-TOD-CHAIGPTEEN SGP-TOD- GP3.5-EXT refer to SGP-TOD with Restaurant-Exx policy skeleton, where we only add four template turns about four new slots to the policy skeleton of Restaurant,
Compared Methods.
Results. Comparison with Base LLMs. The sub- stantial improvement of SGP-TOD-CHATGPT. EXT and SEP-TOD-GPTT Exx over ChatGPT and GPT-3.5 illustrates SGP-TOD's efficiency in supplying task-specifie knowledge in zero-shot way.
ChatGPT, GPT-3.5 denote zero-shot prompt- ing with base LLMs that receive merely two formatting example turns from other domains in RADDLE."" 10
SGP-TOD CHATGPT, CGG-TODDCHAIGPT. SGP-TOD-GPT3.5 represent our SGP-TOD implementation, with the Restaurant policy skeleton.
Impact of Different LLMs. SGP-TOD- CHATGPT-EXT attains a lower BLEU yet a compa- rable BERTScore, suggesting that ChatGPT gener-
WWe utilize the same formatting example turns in all zero- shot prompting methods.
ates more diverse responses.

---

100
100
Success wi groundiing (1%
10
Success grounding
80
Appr ristormant
60
Turns
Uneess U
40
20
20
SOLO
F8 Model
SGP.
8OLO.
F8. Model
SGR
SOLO.
F5.
SGP.
SOLO.
F8.
SGR
SCLO.
FS. Modal
SGR
Model
Model
(a) Success w/o
(b) Success w/ g
(c) Understanding
(d) Appropriateness
(c) Turns
Figure 7: Interactive human evaluation results. SOLO.: SOLOIST, FS.: FEW-SHOT-CHATGPT, SGP: SGP-TOD-
CHATGPT.
Slot
Value
the desired price range despite the user's indiffer- ence. We argue that SGP-TOD assists the frozen LLM in discerning user intentions. In Turn 4, Chat- GPT continues to furmish fabricated details (i.e., the restaurant and phone number) concerning the nonexistent eatery, while HHR-TTD-CHAATGH
"adrress"
"21 24 Northampton Street" "west"
areaa"
"food"
"british"
"id"
"14810"
"location"
[52.21031. 0.11381]
nume".
"saint johns chop house"
phone"
"01223353110"
posteode"
"ch30ad"
EXT identifies a suitable Korean restaurant and the corresponding factual information. In contrast with ChatGPT, SEP-TTOO-CHHTGP-EEEN adeptly ad- dresses inquiries about the delivery service (Turn 6), indicating that SGP-TOD is capable of endow- ing the frozen LLM with novel functionalities.
pricerange"
"moderate"
"type"
"restaurant"
"dllivery"
"yes"
delivery fee"
"6 pounds"
dissh"
"Beef Wellington"
start. time"
"10:30 am"
end time"
"22:40 pm"
Table 9:
An example of Restaurant-Ext DB entry. The newly introduced slot-value pairs relevant to the extended functionality are highlighted.
L
Our Proposed SGP-TOD with Dialog Example
Figure 8 illustrates the proposed SGP-TOD with dialog example.
K
Case Study
Despite the superior performance of the proposed SGP-TOD on GPT-3.5, showcase interac- tive examples utilizing ChatGPT, a renowned and potent chatbot. In Table 11, a user engages with ChatGPT (left) and SGP-TOD-CHTTGPT EXT (right) to complete the identical task on Restaurant-Ext.. The user initiates the conver- sation by seeking recommendations for Tuscan restaurant with price range preference. Lack- ing external database information, ChatGPT con- veys inaccurate details (Turn 2), whereas SGP- TOD-CHATGPT-EXX informs users of the ab- sence of matching restaurants and proposes alter- natives (Turn 2). This exemplifies the benefits of integrating real-world expertise into the fixed LLM. Furthermore, ChatGPT persistently inquires about
M Prompt Examples for STAR Dataset
Two exemplars of prompts for STAR dataset are presented Figure 9 and Figure 10.
II ChatGGP and SGP-TOD-CHATPT-ENN are previously reported in Table 4. The same two formatting example turns are incorporated into the prompt for both zero-shot strategics.

---

Tum
Utterance
User: I want a restaurant on the west of town.
System: Ok. Are you looking for any particular type of food?
User: I don't It care.
System: Are
looking for a particular price range?
User: A moderately priced restaurant would be good.
System: How ahout Saint Johns Chop House? They serve moderately priced British food on the west side of town.
User: Can please have their address and phone numher?
System: Suint Johns Chup House's address is 21-24 Northumpton Street, their phone number 01223353110.
User: Does the restaurant offers delivery service? If it does, what's the delivery fee?
10
System: Yes, it does, and the delivery charges pounds.
User: Thank you, goodbye.
12
System: Thank you for using system. Goodhye.
Table 10: dialog example on domain extension. Task bots need learn to provide pertinent responses concerning the expanded delivery service in subsequent interaction turns, adapting to the continually evolving user/enterprise demands.
Turn
Interaction between a user and ChatGPT
Turn Interaction between a user and COOD-CCaGGPT-E
User: Hi, i'm looking for place with Tascon food in any price range.
Hi, i'm looking for a place with Tuscon food in any price range.
User:
System: am sorry, there are no Tuscan restaurants. other options: Korean, Itaiian Chinese. Which one would you like more information on?
2 System: Certuinly, there are Tuscon restaurants
Here
in town. What is your preferred price range?
3 User: don't It care about the price. Pleuse recommend
User: How about uny Korean restaurants?
one for me, i also need the phone number please.
also need the phone number please.
System: The phone number for Pizza Hut City Centre
System: Little Seoul is a Korean restaurant.
4 is 00123323737 Is there anything else can help you with today?
Their phone number 01223305681.
5 User: Does restaurant offer delivery service?
S User: Does restaurant offer delivery service
How much does the delivery charge? System: I'm sorry, but I could not find any information on delivery fee in our database. You may want to contact the restaurant directly anything else I cun ussist you with today?
How much does the delivery charge?
System: Yes, they offer delivery service and the delivery charge pounds. Can help you with anything clse?
6
6
inquire about this. there
7 User: No. Thank you, goodbye. System: Glad could assist you. Have a good day.
7
User: No. Thank you, goodbye.
8
System: You's 'e welcome, have a great meal! Goodbye!
Table 11: Illustration of two interactive examples between user and CHATGPT (left), (right). Appropriate responses and inappropriate responses highlighted.
SQL select from Restaurant where pricorange = moderato: food - moderm Europeart area south
System: Irestauront name] sound?
Restaurant (moommend//amm)) inform (food. pricerange. aree))
serves (valuw lood] food in [value_pricet ppwperanneeee price range on the [valuo area] side of town.
Bellef State
System Action
System Response
DB
Large Language Model
Dialog History
DST Prompter
DB State
Policy Prompter
looking for restaurant in the moderate price range serving
Restaurant
moderm Eurocean food.
System:
choices, you have
preferred area in mind?
User: I'd like restaurant on the soult end of please.
Figure& 8: Illustration of the proposed SGP-TOD with dialog example. Note that the belief state in the represented in the SQL format, the details of which are descrihed in Section 3.3.

---

Policy Prompter
Task instruction
Policy Prompter
appropriate eystem the most relevant task rule.
history, following
Task instruction
Formatting example (from other task/domainn)
Generate appropriate system actions based on the history, following the most relevant task rule.
Task schema
(1) user: Hello [sep] system action: hello system: Hello, how cen help? [ece]
Formatting example (from task/domain)
Task schema
(2) user: action: ask name system: Could you give me your name, please? [eos]
out
my (sep) system
(1) user: Hello system action: hello [eos]
find action: nsk name [nos]
my [sep) system
(3) user:
[NAME] [eep] system Booking_number number system: Can get your booking ID,
ride ask booking please? [sos]
(3) user: My name [NAME] [sep] system action: Kooking_numbet [eos]
History
History
user: Hi, need to check my ride status.
user: H., need check my ride status.
Answer
user: i'd like find the status (sep) system
action: esk_ name system: Could you give your name, pleass? [cos)
(2) user: action: name [eos]
status my ride (sep) system
Test (on target task/domain)
Test (on target taskidomain)
Task schema
(1) user: hello [sep] system action: hello (eos]
(1) user: hello [sap) system action: hello system: Hello, how can help? [sos]
(2) user: what is the weather lkk? (sep) system action: [eos]
(2) user: what the weather like [sep] system action: weather ask, ask__ay day system: For what day would you like the weather (ecs)
(3) user: [DAY WEEK] (sep) system action: weether aakk Jocation [e0s]
(3) user: on [DAY WEEK] [sep] system wealher ask localion system: For what localion would you like the weather forecass? [eos]
History
Tuesday.
History
user: i want to know the weather for Tuesday.
LLM
LLM
Answer
(3) user: on [DAY OF WEEK] [sep] system action:
weether [90s]
Answer
(3) user: on (DAY OF WEEK] (sep) system action: wealher aek location syslem: For what location would like the wealher forecast? [eos]
Figure 9: Policy Prompter of SGP-TOD on STAR. The relevant template turn within the input, the generated
template utterance and the system action in the output are accentuated.
Figure 10: Policy Prompter SGG-TOO-- 22DEE22 on STAR. The relevant template turn in the input, the generated user template utterance. the system action and the system response in the output are highlighted.


Reviews:
[{"review_id": "ALdY72GG8r", "reviewer": "Reviewer_5kY2", "summary": "", "questions": "1. How are the dialog flows in Policy Prompter different from few-shot exemplars? Is exclusion of generated SQL query the only difference? Can Policy Prompter be truly called zero-shot?\n2. How are the dialog flow outlines selected? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reply-to-Reasons-To-Reject and Q1:**\n\nThank you for your questions. We answer your questions from the following two perspectives:\n\n**\"1) What is the difference between dialog flows and few-shot exemplars?\"**\n\n- *Dialog flows* are abstract representations of dialogue policies. They explicitly describe anticipated interaction patterns (Peng et al., 2021b) and are typically depicted using graphs (such as the flowcharts in the STAR corpus (Mosig et al., 2020). However, in the *few-shot setting*, the few-shot exemplars consist of examples sampled from the training data. These exemplars encode very limited interaction patterns, illustrating partial dialogue policies. \n\n**\"2) Why is SGP-TOD a zero-shot model?\"**\n\n- As illustrated in Figures 3 and 5 and explained in Section 3.4, the policy prompter takes dialog flows as input, which sketch the dialog policies. Each element within these dialog flows is textualized, appearing as a delexicalized user utterance/database state, a system action, and a system response. Since the policy prompter takes *zero* examples from the dataset as input, we note it as a *zero-shot prompting* method, following Mosig et al. (2020) and Mehri and Eskenazi (2021).\n\n**Reply-to-Q1:**\n\n- Please see the detailed response provided above, under “Reply-to-Reasons-To-Reject and Q1”.\n\n**Reply-to-Q2: “Selection of dialog flow outlines.”**\n\n- Dialog flows that describe the functions of dialog systems in an abstractive way are typically pre-built when developing task-oriented dialogs systems.  For the particular datasets used in this paper, similar to Mosig et al. (2020) and Mehri and Eskenazi (2021), we manually create the outline by examining the task descriptions, provided by the dataset. More details are provided in Appendix E.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. SGP-TOD demonstrates how to systematically leverage LLMs for TOD task. Based on the benchmarking, SGP-TOD (GPT3.5) achieve SOTA zero-shot performance.\n2. Prompt design in SGP-TOD can be relevant for other tasks with complex reasoning. ", "reasons_to_reject": "SGP-TOD uses dialog flow guidelines which are partial dialog context with gold delexicalized responses. It is unclear how these dialog flows are different from the few-shot setting. This raises a question on SGP-TOD to be a zero-shot model. ", "questions_to_authors": "1. How are the dialog flows in Policy Prompter different from few-shot exemplars? Is exclusion of generated SQL query the only difference? Can Policy Prompter be truly called zero-shot?\n2. How are the dialog flow outlines selected? ", "paper_topic_and_main_contributions": "Goal: To leverage LLMs for developing task-oriented dialog (TOD) systems in zero-shot setting.\n\nMain Idea: Prompt LLM with task schema and dialog context to generate DST and policy outputs.\n\nDetails\n1. The paper proposes SGP-TOD that prompts LLMs using task-specific ontology (possible slot-value pairs) and flow outlines.\n2. SGP-TOD has two prompters. DST Prompter queries LLM for DST in SQL format for a given dialog history. Policy Prompter then utilizes flow outlines, DST, DB state and dialog history and queries LLM for action and response.\n3. Authors benchmark SGP-TOD on Multiwoz 2.0/2.2, RADDLE and STAR datasets and show that SGP-TOD has significant gains against few-shot methods and sets new zero-shot SOTA.\n4. Authors also perform human evaluation across models which is a good addition.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1Wr3srriaE", "reviewer": "Reviewer_t1ZP", "summary": "", "questions": "- It may be useful to discuss parameter counts when comparing results. Even though prompting and fine-tuning are vastly different learning mechanisms, the models being used differ greatly in size from some of the baselines, which makes at least a mention of parameter counts worthwhile.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reply-to-Reasons-To-Reject: \"Human evaluation results.\"**\n\n- Thank you for the comment. Our primary focus of this paper is on zero-shot and few-shot prompting evaluation. While we acknowledge that SOLOIST is somewhat dated, it remains a prominent model for MultiWoZ dataset. Furthermore, human evaluation results have shown that SGP-TOD significantly outperforms Few-shot-ChatGPT and SOLOIST, with success rates surpassing them by 18 and 62 points, respectively (Table 5). These findings align with automatic evaluation results and show the effectiveness of Schema-Guided Prompting.\n\n**Reply-to-Q1: \"Parameter counts.\"**\n\n- Thank you for the valuable suggestion! We will add the details on parameter counts  in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents a sound, simple, flexible prompting-based technique to take up all TOD sub-tasks in E2E fashion\n- Results look fairly strong, comparing favorably with nearly all the finetuned baselines. The non-finetuning baselines compared against are all quite recent and strong ones as well.\n- The paper is clearly written and the modeling approach is explained in sufficient detail.", "reasons_to_reject": "- Human evaluation results compare only to SOLOIST, which is a relatively old paper at this point. It is unclear just how convincing the human evaluation is, for that reason.", "questions_to_authors": "- It may be useful to discuss parameter counts when comparing results. Even though prompting and fine-tuning are vastly different learning mechanisms, the models being used differ greatly in size from some of the baselines, which makes at least a mention of parameter counts worthwhile.", "paper_topic_and_main_contributions": "The paper describes an LLM prompting-based approach to task oriented dialogue. Using exemplars and prompts for the state tracking, dialogue policy, and response generation tasks, the model uses LLM prompting to perform all those tasks in an E2E fashion. Results are shown on MultiWoZ 2.0 and 2.2, and also the RADDLE and STAR benchmarks - both the regular variants of these datasets as well as the domain extension setting on STAR.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QWIJu8s9VJ", "reviewer": "Reviewer_msez", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reply-to-Reasons-To-Reject:**\n\n**\"1) Effectiveness of the two prompters.\"**\n\n- In the submitted manuscript, we reported our main results in Table 1 and explicated them in Section 4.2. The proposed SGP-TOD-ChatGPT notably surpasses IG-TOD-ChatGPT-zs (Hudecek and Dusek, 2023) with respect to the Success score (54.70\\% vs 20.00\\%), a state-of-the-art prompting technique employing slot descriptions for tracking dialog states, fetching DB entries, and generating responses (detailed in Section 4.1). This underscores the indispensability of our proposed policy prompter in SGP-TOD for guiding LLMs towards generating appropriate responses for new tasks. To answer this specific question, we also conducted an experiment on SGP-TOD-ChatGPT by excluding the Policy Prompter and report the result in below table. The result indicates that, with only the DST prompter, SGP-TOD-ChatGPT-w/o Policy still substantially surpasses IG-TOD-ChatGPT-zs in terms of Success (41.60\\% vs 15.00\\%). This superior performance additionally substantiates the efficacy of the proposed DST prompter.\n- In Table 6, the DST Prompter significantly contributes to the performance of SGP-TOD-GPT3.5 (approximately 31\\% increase over Combined Score), which demonstrates the efficacy of the proposed DST Prompter. In addition, the Policy prompter further advances the Combined score by nearly 11\\%. This notable advancement is remarkable given the intricate nature of the well-established Multiwoz 2.2 dataset (Zang et al., 2020), the challenging research objective of zero-shot end-to-end dialog generation, and the robust baseline exemplified by GPT3.5 (the last row in Table 6).\n\nWe would greatly appreciate it if you could provide more detailed comments.\n\n|            Model           | Multiwoz 2.2 |         |      |          |\n|:--------------------------:|:------------:|---------|------|----------|\n|                            | Inform       | Success | BLEU | Combined |\n| IG-TOD-ChatGPT-zs          | -            | 15.00   | 3.58 | -        |\n| SGP-TOD-ChatGPT-w/o Policy | 68.80        | 41.60   | 3.75 | 58.95    |\n\n**\"2) Comparison to meta-instructions.\"**\n \n- Thank you for your insightful comments. Exploring meta-instructions to empower LLMs for decision-making could lead to a highly intriguing direction. The concept of the proposed prompters essentially represents a form of meta-instruction. Through experiments, we have demonstrated that our approach outperforms existing prompting methods for Task-Oriented Dialogue. Nevertheless, we do acknowledge that there might exist other uncomplicated and easily comprehensible formats for LLMs to track states and generate actions.  We will leave it as future work.\n\n**Reply-to-Reproducibility:**\n\n- Parameter settings: Comprehensive experimental details have been provided in Appendix E. \n\n- Training/evaluation data: All utilized datasets, which are well-studied and publicly accessible, have their details delineated in Appendix D.\n\n- Prompter examples pertaining to our proposed approach can be found in Appendices C and M. Furthermore, as stated in the abstract, we intend to release our code and data upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method has strong performance on various benchmark datasets.\n2. This paper demonstrates how previous techniques for TOD like dialog state tracking can effectively help design prompts for LLM-based chatbots.", "reasons_to_reject": "1. Analyses are not sufficient to support the effectiveness of the two prompters. (1) Tab. 6 suggests that the improvement is mainly from the DST prompter. (2) It is unclear whether the two complex prompter is more effective than simply introducing meta-instructions to let the LLM track states and make policies.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes schema-guided prompting to build task-oriented dialog systems (TOD) with large language models (LLM) in zero-shot settings. It introduces a Dialog State Tracking Prompter to aid LLM to track dialog state explicitly and a Policy Prompter to direct the generation of responses. Experimental results show strong zero-shot performance on various datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper benchmarks prompting for zero-shot TOD using DST and Policy prompt mechanisms. Using ChatGPT, the paper demonstrates SOTA performance on zero-shot TOD using schema-guided prompting. 

This is primarily a benchmarking paper and does not introduce any core technical innovation. However, reviewers appreciated the overall prompting framework. There were some concerns around complexity of the overall prompting setup, towards which, the authors clarified on the necessities of the modules in certain scenarios. 

The paper argues schema-guidance to be better than few-shot prompting. As such, discussing about the paper - "Show, Don’t Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue" might be useful.
Decision: Accept-Findings