Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking
Authors: Yuxiang Wu
Guanting Dong
Weiran Xu
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialogue State Tracking
Zero-Shot
Large Language Models
In-Context Learning
Semantic Parsing

Abstract: Zero-shot Dialogue State Tracking (DST) addresses the challenge of acquiring and annotating task-oriented dialogues, which can be time-consuming and costly. 
However, DST extends beyond simple slot-filling and requires effective updating strategies for tracking dialogue state as conversations progress. 
In this paper, we propose ParsingDST, a new In-Context Learning (ICL) method, to introduce additional intricate updating strategies in zero-shot DST. 
Our approach reformulates the DST task by leveraging powerful Large Language Models (LLMs) and translating the original dialogue text to JSON through semantic parsing as an intermediate state. 
We also design a novel framework that includes more modules to ensure the effectiveness of updating strategies in the text-to-JSON process. 
Experimental results demonstrate that our approach outperforms existing zero-shot DST methods on MultiWOZ, exhibiting significant improvements in Joint Goal Accuracy (JGA) and slot accuracy compared to existing ICL methods.

Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking
Yuxiang Wul", Guanting Dong' Weiran Xu
Beijing University of Posts and Telecommunications Beijing, China (yuxiangw, donggaatinggg edu. edu. cn
Abstract
Pronyt (lastructioo Schana descriptia + Examplo) Cootertsblee Horel
Curment turn dillagan
Zero-shot Dialogue State Tracking (DST) ad- dresses the challenge of acquiring and annotat- ing task-oriented dialogues, which can be time- consuming and costly. However, DST extends beyond simple slot-filling and requires effective updating strategies for tracking dialogue state conversations progress. In this paper, pro- pose ParsingDST, a new In-Context Learning (ICL) method, introduce additional intricate updating strategies in zero-shot DST. Our ap- proach reformulates the DST task by leveraging powerful Large Language Models (LLMs) and translating the original dialogue text to JSON through semantic parsing state. We also design a novel framework that includes more modules to ensure the effective- ness of updating strategies in the text-to- text-to-JSON process. Experimental results demonstrate that our approach outperforms existing zero-shot DST methods on MultiWOZ, exhibiting sig- nificant improvements in Joint Goal Accuracy (JGA) and slot accuracy compared to existing ICL methods.
The dinne
hatel with low
User:
No. ssan an sétreciion XXN
Transate SO0N
Decade State directly
Info: [Rase:
Plye
Midako
luotet namu:: X.X bottel Piize nagg cheep Atrrzbon ramti: XXX
Mtcres stree]]
schema
Blodi box
jette Perge Cheap)
Not an eatit!
Ouly achirve slot hard understand intricute vlate updating
(Uzen
strategtee tn DS
Rejaes (Rond:
Foml:: [XX Intel] [dlak]
intermediate
turt:: IXXV]]);
Brggunt (4 line lnd
Controliable intergretable
State Updatted Strategles
came: XXX
Golden labe!
Figure I:
An example of the comparison between the previous ICL method and ours.
performance. Recent studies have showcased the impressive and adaptable expertise possessed by large language models (Raff) et al.. 2020; Ouyang al., 2022; Liu et al., 2023). However, with the discovery of the emergent ability of large language models in downstream tasks (Cobbe et al., 2021; Wei et al., 2022a,b), the In-Context Learning (ICL) method (Brown et al., 2020) has garnered more attention in DST research. ICL offers more flexible and cost-effective approach as it eliminates the need for retraining when new slots domains are introduced. Some recent studies have explored the effectiveness of ICL in DST (Hu et al., 2022; Madotto et al., 2021; Xie et al., 2022) such as the IC-DST method, which has demonstrated remark- able performance surpassing previous fine-tuning methods.
1 Introduction
Dialogue State Tracking (DST) is crucial in Task- Oriented Dialogue (TOD) systems to understand and manage user intentions(Wi (Wu et al., 2019; Hosseini-Asl et al., 2020; Heck et al., 2020; Lee et al.. 2021; Zhao et al., 2022). Collecting and annotating dialogue states at the turn level is chal- lenging and expensive (Budzianowski al., 2018), and commercial applications often need to expand the schema and include new domains. Therefore, is vital to develop DST learning strategies that can adapt and scale effectively with minimal data.
Most of the existing fine-tuning methods for zero-shot DST have primarily focused on domain- transfer approaches (Hosscini-Asl et al., 2020; Lin al., 2021b.a), which have not yielded satisfactory
Nevertheless, all the previous methods directly generate the dialogue state or its change, which only achieves a simple slot-filling task that extracts the slot's value from dialogue but does not explain
'The first two authors contribute equally. Weiran Xu is the corresponding author.

---

RESOMERE Done XXX
Trundata 150N
Reckendccckk 19:10
Conkinl
(:S0n Fojes (I. (Rediened piae nage 11:33
Rosuee [Botaass::
Contect Uiber:
SON
Strieg Sssed 1pdating
Based an Coskestadze
Losk lasc (260000))
Receat (eestamant: (XXX) book Irm 1TR081::21
Snjer
Ney indiaate (ileruenet
bana lanc (18:000)
Resaare book tea: 20.0
Tempanary rbil
Irto (Baemapet
naae (XX32))
Not pnsilablee
Stratog--ooe spdatigg
Hlausass: tinn (o
Message
Syston nod be for H0ue wow dontt
Ic: O3)
Sex Nat undbble (). Iet: (2100))
Hessag
User Foredi-ooo. oull petkal
Seaalssute Resturatt x Rediened- boak 28:10
Tampprnry
20108 redernet
A4 lar (1)
Moddl
Figure Our framework of ParsingDST that includes filter module.
how the state updating strategies work during DST which makes the DST process a black box. And we find the main obstacle the current zero-shot DST method that the large language models (LLMs)' understanding of updating strategies have not aligned with the DST task.
ods with the powerful LLMs by OpenAT²: gpt-3.5- turbo-0301 and text-davinci-003 (Brown et al., 2020) on MultiWoz 2. 2..4 dataset.
In summary, our work makes the following con- tributions: (1) To our best knowledge, we are the first to apply text-to-JSON by semantic parsing as the transition between dialogue to state, making it possible to tool additional state updating strategies and other modules, making the process of DST more controllable and interpretable. (2) We pro- posed a novel framework with modules in the pro- cess of text-to-JSON to maintain the effectiveness of updating strategics. (3) We experiment with different ICL methods with various LLMs on Mul- tiWOZ 2. &2.44, proving our ICL method outper- forms the previous in the same model and achieves statc-of-art in zero-shot DST task.
In this work, we proposed new ICL method named ParsingDST for zero-shot DST. Specifically, we reformulate the DST task by semantic parsing that translates the origin dialogue text into the designed JSON format (Text-to-JSON) as the in- termediate state through the LLMs The translated JSON data will include the information about the speaker, interaction action, domain, slot, and cor- responding values in formatted and structured manner, which enables the introduction of manual state updating strategies later.
We illustrated the significance of updating strate- gies by presenting an intricate scenario in Figure 1. Previous ICL approaches typically decode the dia- loguc to directly extract the values of all slots, such as "hotel-name" "price-range", and "attraction- name" in the example, without considering the up- dating strategies, which leads errors in the result. In contrast, our method applies additional updating strategies after the text-to-JSON conversion. We delete non-entity slots like "price_range" when appears in the system but not in the context, and we set the value of the "hotel name" to "[Delete]" when it is mentioned in the user's "reject" action, which makes our result aligns with the golden label.
DST System
2.1 Data format design
As mentioned before, we translate the origin dialogue text the ISON format which includes interactive actions. The domain, slot, and value JSON are inferred from the utterance. And the formats of User JSON and System JSON are different. More details about ISON format and our prompt are in Appendix A.3.
2.2 Dialogue context representation
In order to maintain semantic consistency, we still use JSON format as context representation. When translating system's utterance, we put all domain- slot-value pairs from the context state into the "re- quest" action of the data that is User ISON's format as context representation. Then When translating user's utterance, merge the previous context representation and the System JSON the new context representation.
Based our observations, we noticed that when merging context, system, and user utterances as di- alogue input like the previous methods, the L.MMs will often exhibit confusion between these differ- ent types of information, which will invalidate the updating strategies. To address this, we also de- signed companion framework with more process modules to ensure the effectiveness of updating strategies and conducted experiments validate them. We choose to test and compare ICL meth-
"More details about models httpps///on poppp:/ppppaa.....o http/pppaa.coo........oooo com/ 3 "Snapshot of gpt-3.5-turbo from March 1st, 2023, this model will not receive updates.
*Codes in thub. www.hh..co/oool.o..oo wwt.e//ooigggullpTeeeee///rrrrir

---

2ulliWo...1
Restaurant
Supersised fine-issdddiettiiini fine-tuning method
SimpleTOD++ (Hosscini-Asl et al., 2020)
28.01
17.69
15.57
59.22
27.75
29.65
TSDST a description (I.in et al., 2021b)
33.09
21.21
21.65
64.62
35.42
35.20
TransferQA (Lin al., 2021a)
31,25
22.72
26.28
61.87
36.72
35.77
Previous ICL method with different LLMs
Codex (deprecated)
46.69
57.28
IC-DST Text-davinci-003
38.55
43,98
71.25
45.99
50.09
IC-DST (pt-3.5-turbo--ii
59.32
40.20
46.50
68.32
52.87
53.13
Our method with difforent L.Mss
PST Text-davinci-003
61.60
39.92,137
62.56.418.5
80.45...20
51 1..89....0
59.88_..7
PT G,5-335 1ptt-3.-ttrbb--553 turbo-0301
64.95
46.76.0.56
67.04.20.54
80.266,1144
62.78,
63.36,
filter module
63.15,320
15.35,5.15
G6 200.0.00000
.0.155
11-6.56
62. 9..1
frumework
64.02,...0
66.63,.643
66.95
75.42 7.10
0.24
61.71
MultiWoZ. 2.4
Attraction
Hotel
Reslauratt
Taxi
Train
AVG
IC-DST Gpt-3.5-tuto-0000
59.81
40.20
46.80
68.32
52.87
53.60
ParsingDST Gpt-3.5-turho-025
65.63
46.76.6.56
67.67.20.87
80.59
62.59,9.72
64.65 64.65,11IE
Table Zero-shot per-domain JGA on MultWWOZ 2.14 we calculate the average of all per-domain JGA as the measure of overall performance, and we use subscripts to indicate the JGA changes of our methods compared to the previous sota ICL method under the same model.
2.3 Updating Strategies
dividedly. As shown in figure The details of the steps in our framework pipeline are as follows: Stepl. In the turn of dialogue, convert the context state S,-1 to JSON format Ct as context represen- tation (mentioned in section 2.2) in equation (1), then merge it with the system utterance Mit the dialogue input for LLM, generate the System ISON AJsON equation (2):
Through the observation of the dataset, we have formulated some status update strategies, which are based on rulers and will influenced by the speaker, interactive action, and entity or not. As for slots in generated User JSON: (1) The slot in "reject" action will be deleted later; (2) The slot in "request" action will be updated. As for System JSON: (1) The slot in "ask_for' or "not__valiaaa" avaliable" for
C State2/SON(S
(1)
actions wil be ignored, these actions reducing the mistakes of interaction classification in semantic parsing; (2) The slot in "info" action will be updated when it is an entity or it also in context state, but in other cases, we will ignore it
ASSON Trans2/SOO(( Alae)) (2)
Step2. Then the context state is updated by updat- ing strategies and the System JSON and saved as the temporary state Stemp as equation (3):
Stemp = Update(S, 1, A'Joo)
(3)
2.4 Modules
Step3. Equation (4) will filter the generated System JSON with the module mentioned in section 2.4:
The introduction of the new framework makes it possible to use additional modules correct the generated previous content. In this paper, tried simple module that filters System JSON to make it only include the updated slot or entity slot in our framework.
A'JSON = Filter(A'sson)
(4)
Step4. Equation (5) combines context JSON and filtered System ISON as the new context represen- tation, then equation (6) inputs the new context with user's utterance U together and generate User JSON UJSON with LLM:
2.5 Framework design
C - C. + A'SONN
(5)
The motivation for designing this framework is to avoid harmful information from the history state or previous speaker being mixed into the ISON gen- crated by the subsequent speaker and affecting the effectiveness of the update strategy. Our framework modifies the shared input-output method which pro- cesses all together. We make asynchronous and add more modules in the pipeline: process the con- text, the system utterance, and the user utterance
USSON Trans2JSON U
(6)
Step5. Finally, the temporary state is updated by the updating strategies and the User JSON to get our dialogue state of this turn S; in quation (7):
S = Update (St UJSox)
(7)
If it is the first turn, the process will start from step4 and ASSOON will be empty.

---

ParsinDSST
IC-DST
ID: MULLIG jeo Tune
Contert dain
Slot Acrauaayy (%)
98.52
97.94
98.3
9896
97.78
98.23
Dialogas:
mestaam food Treach "retburzth ond'ls "oeetre')
97.16
Use reedd bille frddy 17,00, pleco
96.67
PariagOST:
97.95
"rotacreiionn"" "lontee"" "sotaaraiioaaa"" Tromaammn
people
94,73
IC-DST:
("wnerrrt food" "trechh indermot "cantee" "indemeeiin
expeadive" "poltaratt Lead'
books Lay's
"risaay "oszaran
MUT yec Term
Castonlstale:
prael-ptoeegge "chepp"
Attraction
Hotel
Restaurant
Tax
Train
Itlkoyae:
Sasver: lave all thess un pueit housss and of then have froe interss and palling.
Would you Lik ce la gloz your txtridforaaionn
Figure 3: The average slot accuracy in 5 different do- mains of MultiWOZ 2.1 with GPT -3.5 model.
Ule sam theanards
PoumingliM::
Compliiee:
""ondaa"" (aatt andiate:: (1). "tato" ('loggogt (iocptag tope: grest house'
"atoree
porking "'ess 113, ("ooggng" IPI
("lokgrghh
"Sinetoo"
3 Experiment
Resul:
Tbotd peteaange "chegg" "takd ans" "eas""
PandegisT (ate founmwekks
3.1 Dataset, Metric, and Evaluation
("goter (*as avatlabe" "Into" ("odggro" ('Iddgin ('poett "intrmet
yee'l pprking:: yyes 11) "atl oor: (13 "coe" rrrjett (). regueat ("odgreg"
MultiWOZ (Budzianowski et al., 2018) is multi- domain human-humma written dialogue dataset, the labels and utterances have been refined in subse- quent versions, c.g., MultWWZZ 2.1 (Eric et al., 2019) and MultiWOZ2.4 (Ye et al., 2021) is a cleaner version. We use Joint Goal Accuracy (JGA) which is the average accuracy of predicting all slot assignments for given service in turn correctly evaluate the main results of models. Regarding text preprocessing and labeling, we followed most of the previous work of IC-DST (Hu al., 2022) on preprocessing and labeling data. The only change is that we renamed some names of the slots and do- mains. More details about baselines and zero-shot settings are in Appendix A.1 and Appendix A.2.
(0030 nece' (chag "Maciiee" [ ["aarr] (isaaaa 2 you)) "aaking" - Lass ('ae"ll1))
Real::
("ete prowmnge" "chep
"antérmmmee "yee" (hoti-prrting 'ye")
Figure 4: Two representative samples in the test set of MultiWOZ 2.1.
influence of updating strategies in zero-shot DST. Ablation Studies. We conduct an ablation study to better prove the effectiveness of our modules and new framework. In the situation without the frame- work, we follow the traditional setting to merge context, user, and system utterance as dialogue in- put, then output JSON all at once. As shown in Ta- ble 1, the overall performance degrades whether the filter module or our framework is discarded. How- ever, observed that the JGA in certain domains remained similar to the previous performance dur- ing our ablation studies, we believe that is because our modules and updating strategies still need be optimized.
3.2 Results and Analysis
Main Result. Table shows the zero-shot per- domain JGA result on MuliWOZ 2.1, and AVG JGA as the measure of overall performance. Be- cause IC-DST's best performance model Codex- Davinci (Chen et al., 2021) has been deprecated by openAl, we supplement two experiment results with GPT-3.5 and Text-davinci models. We can find from the result that the supervised fine-tuning method is far inferior to the ICL method. And no matter in which domain, our method always out- performs the previous sota ICL method IC-DST by a large-scale margin in the same model. For exam- ple, using the same GPT-3.5 model, we achieved a 20.54% increase in JGA in the restaurant domain. Although the supplemented experiments of IC-DST show worse performance compared the Codex model, our ParsingDST method still outperforms the IC-DST Codex in all five domains with the same model of supplemented experiments, which shows the strong ability of our method and the great
Slot Accuracy Analysis. Figure 3 shows the slot accuracy of models using PasingDST and IC-DST with the Gpt-3.5 model. It can be seen that our method achieves better results on all 5 domains compared to the IC-DST (e.g., 2.88% improvement in the restaurant domain), which further proves the effectiveness of our method. We speculate that it is crucial for DST to introduce an appropriate dialog state update strategy into the model.
Case Studies. To further illustrate the effective- ness of our framework, Figure 4 shows two rep- resentative samples of different methods' predic- tions. In the first case, the IC-DST method makes wrong prediction that includes the non-entity slot "pricerange" from system utterance that does not appear in context, however, our method can handle the situation well because of the introduction of updating strategies. In the second case, parsing-

---

DST makes a wrong prediction when removing its framework, because its User JSON included slots "internet" and "parking" from system utter- ance, which makes the updating strategies invalid because these slots will be updated rather than be ignored. In our method, we can process System JSON and context JSON with modules that are included in the framework before the harmful in- formation influences the latter User ISON.
Pawef Budzianowski, Tsung- Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašié. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint urXiv:1810.000
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained code. arXiv preprint arXiv:2107.0334..
4
Conclusion
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert. Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv: 10. 14168.
this paper, we reformulate DST to semantic pars- ing with LLMs that translate the dialogue text into JSON as the intermediate state, which enables us to introduce updating strategies and makes DST process more controllable and interpretable. Fur- thermore, present novel framework that in- cludes more modules within the text-to-JSON con- version process to ensure the effectiveness of up- dating strategies. Through experiments on the Mul- tiWOZ 2. 1.44 dataset, our system achieves sota performance in zero-shot settings. Our analyses showed that our method surpasses the previous ICL method and the innovative modules and frame- work benefit the overall performance. In future work, we plan explore and evaluate more variety of updating strategies and modules to further en- hance our framework and we will test our method more open-source LLMs like Llama (Touvron et al., 2023).
Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, San- chit Agarwal, Shuyang Gao, and Dilek Hakkani- Tur. 2019. Multiwoz 2.1: A consolidated multi- momti-domain dialogue dataset with state conrections and state track- ing baselines. arXiv preprint 1907,v:100001660
Michael Heck, Carel van Niekerk, Nurul Lubis, Chris- tian Geishauser, Hsien-Chia Lin, Marco Moresi, and Milica Gašié. 2020. Trippy: A triple copy strategy for value independent neural dialog state tracking. arXiv preprint 2005::2055.02877
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33:201 79- 20191.
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022. In-context learning for few-shot dialogue state tracking. arXiv preprint arXiv: 2203.0800
Limitations
This work has two main limitations: (1) The per- formance of our framework is highly dependent on the inference language model, which may limit the framework's usage. For example, it depends on the ability of LLMSs that are pre- pre-trained in JSON and natural language data and can understand both. (2) Lack the detailed comparison of various updating strategies and modules in the framework. We will design and test more kinds of updating strategies and modules for our framework in future work.
Chia-Hsuan Lee, Hao Cheng. and Mari Ostendorf. 2021. Dialogue state tracking with language model using schema-driven prompting. arXiv preprint wrXiv:2109.07506
Zhaojiang Lin, Bing Liu, Andrea Madotto. Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang. Zhou Yu, Eunjoon Cho, Rajen Subba, et al. 202 la. Zero-shot dialogue state tracking via cross-task trans- fer. arXiv preprint 2Xiv:210090440
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, and Rajen Subba. 2021b. Leveraging slot descriptions for zero-shot cross-domain dialogue state tracking. arXiv preprint arXiv:2105.04222
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877-
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, al. 2023. Summary of chatgpt/gpt-4 research and perspective towards

---

the future of large language models. arXiv preprint arXiv:2304.01852
Jeffrey Zhao, Raghav Gupta, Yuan Cao. Dian Yu, Mingqiu Wang. Harrison Lee, Abhinav Rastogi. Izhak Shafran, and Yonghui Wu, 2022. Description- driven trientre dialog modeling, arXiv preprint urXiv:2201.09900
Andrea Madotto, Zhaojiang Lin. Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt- based learning for dialogue systems. arXiv preprint 1r.0v:21
A
Appendix
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744
A.1
Baseline
SimpleTOD (Hosseini-Asl et al., 2020) Success- fully leverage the pretrained language model GPT- for the end-to-end TOD modeling in the unified way.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020, Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(4):588--5511
TSDST (Lin et al., 2021b) slot description en- hanced approach for zero-shot & few-shot cross- domain DST based on T5.
TransferQA (Lin et al., 2021a) Reformulated DST as QA problem. The model is pre-trained with large amount of QA data. At inference time, the model predicts slot values by taking synthesized extractive questions as input IC-DST (Hu et al., 2022) An in-context learn- ing method for zero-shot & few-shot DST by text- to-SQL with LLMs. It's the previous state-of-art method that outperforms the supervised fine-tuning methods in zero-shot DST.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne I Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint
arXiv:2302.13971.
Yizhong Wang, Swaroop Mishra, Pegah Alipoor- molabashi, Yeganch Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Benchmarking generalization via in-context instruc- tions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705.
A.2 Zero-shot setting of ICL frameworks
There are no labeled examples to retrieve, but for- matting examples included, following previous zero-shot learning work. (Wang al., 2022), and we set the temperature parameter of LLMs for a stable result.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models, arXiv preprint arXiv:2206.007 07682.
A.3 Prompt design
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le. Denny Zhou, et al. 2022b. Chain-of Saain-of-tie ofothh prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
Below are the template of the user and system's 's prompt in zero-shot settings. The last example is the test instance that needs to be completed. We find the example is more useful than the task-description prompt in the in-context learning, so we use simple example that includes all slots of the domain to map the relationship between slot and value the dialogue rather than directly describe the slot and its value. Furthermore, for LLMs to understand the meaning of action in JSON format, we also formulate some examples to demonstrate some interaction typical scenarios.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini- Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state genera- tor for task-trie dialogue systems. arXiv preprint arXiv:1905.055
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Shi Sheng Wu. Ming Zhong, Pengcheng Yin, Sida I Wang. al. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text lan- guage models. arXiv preprint arXiv: 00566.6
System prompt template
translate system message JSON:
data format of ISON:
inpat message:
Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. 2021. Multiwoz 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. arXiv preprint arXiv:2104.00775
system: ".."
output ISON:
["system" availabbll (domsin: (slot: [value])). "info" (domsin: (slot: [valne]] ]], 'ask. far": (domain: [slce]]]] [END]
example:

---

Input message:
inpul message: user: "no, not picky as long as book en sunday" ourpur JSON:
system: "the booking for restaurant 10:00 on sunday was snccessfal"
output
("uses": I'reject": [). "request": ("restauran"": ("price rangee ("any"
I"systom": ("not available"" (b. "info": I"restaurant": toclock_book":
"cuisine"; ["any"], "muum poople": ["4"], "wook_day") ["sunday [133] [END)
['10:00"], "week day": ['sunday"]] "I ssk_fort: ())) [END]
exsmupe::
example:
input message:
ingut message:
user: " want to be in the east of town can i get their phone number and please"
system:: is chinese restaurant centre"
ourput ISON:
ourpur JSON:
("system": wailaabll' [}. "info": ("cuisine": ['chinese"] "direction':
("user": ("reject": "request": ("restauran": ("direction":
["east"].
I"centre"] II,. "or", (11)
"bbon_ppubbrr" II. "address"" 113111
[END]
[END]
evample:
example:
input message:
input message:
system: "how about abe restaurent the city centre"
user: "musha noe restourant an attraction"
oulput ISON:
ISON:
I"systen": ("not ("not__vvillle available": 1b. "info": I'restaurat": nntull l'abe
("user": ("rejcer": ("restaurant": ['Toll_name"] ].. "roquest": I"straction":
restaurant"]. J." "direction"; l'oentre"]),, IH,, "ask_for" (H [END]
["Tull_name" ['nushn"]ll [END]
example:
[M]
input message:
[EXM]
system: "how about the the and food for the restaurant"
[ST]
ourpur ISON:
[W]
I"system": l'nor available":
[b. "info": (). "nsk for":
"restamraan":
('dinection,, "cuisime" 'I]1]
example:
[END]
contest:
[PREDIC]
example::
inpul message:
inpot
[IALOG]
system: 'do you certain price range or part
ourput
ourput JSON:
("system's availablee:
"info":
"or""for":
("restuuaat":
Special tokens in prompt
('price range". ddrrction"]]]]
[END]
The prompt has some special tokens which will
example:
be replace by something else in the data-bulding
input message:
system: "sorty can not book restaurant musa for you can only find nandos"
process:
output ISON:
"[DM]" is the domain list.
("system"; available': I"restaarant": ("fall, name": ("musha") ]] "info":
("restaurami": ("full_ name": ('nandos "or""for" (}3)
"[EXMJ" is the domain example which is user
[END]
utterance that includes all slots of the domain and its translated JSON.
[ST]
"[ST]" is the slot list of the domain. "[KW]" is the possible choice of some slot.
example:
"[PREDIC]" is the context ISON. "[DIALOG]" is the utterance that should be trans- lated to JSON.
[PREDIC]
input message: [DIALOG]
ourput JSON:
User prompt template
"[END]" is the stop token.
translate
JSON:
dalu Format of ISON:
input message:
ourpur ISON:
("user"": ('rejoet:: [domain: (slot)), "request": [domain: (slot: [valuc]]
[[ND]
example: input message:
user: wani a place to sak. in the city centre. with cheup price"
ourput ISON:
("reject": []. "request":
("restaurant": l'direction": ["centre"].
"price sange": ("cheap'])1)) [END]
example:
inpul message:
user: "eo particular food rype"
output JSON:
("user": ("reject": []. "request": ("restaurant": ("cuisine": ["an)"II!!]
[END]
example:
["system": ('not avatiable":
(b. "into": (), "or"_for:: ""restuuratt":
["pricerrange" range", "cuisine")) '[]]]


Reviews:
[{"review_id": "mUeLsVLJgw", "reviewer": "Reviewer_9Gsy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestions and questions. Below is our detailed response:\n\n**Q**: Code can be released for other users to work on top of the current work. \n\n**A**: Thank you for acknowledging our work. As we stated in section 3.1 of our paper, we will release our complete code upon acceptance. We have also provided Supplementary Materials for reference, which include the main content of our code. We hope this will be helpful to you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tResults are presented well with detailed ablation study, The ablation analysis and case studies provide additional insight into the effectiveness of the approach.\n•\tIt presents a novel approach with strong performance, and clear motivation. \n•\tThis is a very well written paper, coherent and easy to understand.\n•\tArchitecture of the model is well-defined in figure 2. \n•\tOverall, this is an effective paper with strong results.\n", "reasons_to_reject": "Code can be released for other users to work on top of the current work", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes ParsingDST, a new In context learning method that translates the original dialogue to json representation as an intermediate state.  Experiment results shows this method outperformance current SOTA on zero shot DST. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JzkxPdJPYJ", "reviewer": "Reviewer_Zu2U", "summary": "", "questions": "- How did you preprocess the text?\n- How did you preprocess the labels?\n\nBonus:\n- Nothing wrong with using closed-source models; would be nice (i.e., useful for practitioners in this field) to have results with open-source LLMs.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestions and questions. Below is our detailed response:\n\n**Q1**: Only evaluated on the 2.1 version of MultiWoz which is known to have noisy annotations. \n\n**A1**: Hello, we used the settings that performed best on Moz 2.1 and supplemented our method with JGA results on Moz 2.4 (the performance improvements compared to IC-DST are shown in parentheses below): \n\nPer-domain JGA of **IC-DST Gpt-3.5-turbo-0301** and **ParsingDST Gpt-3.5-turbo-0301** in Moz 2.4: \n\n|Model| attraction | hotel | restaurant | taxi | train | avg |\n| :---- | :---- | :----: | :----: | :----: | :----: | :----: | \n|IC-DST Gpt-3.5-turbo| 59.81 | 40.20 | 46.80 | 68.32 | 52.87 | 53.60 |\n|ParsingDST Gpt-3.5-turbo| 65.63(+5.81) | 46.76 (+6.56) | 67.67(+20.87) | 80.58(+12.26) | 62.59(+9.72) | 64.65(+11.05)|\n\n\nAs we can see, our method still achieves very satisfactory results and has significantly improved compared to the IC-DST method. We will supplement our results on MultiWOZ 2.4 in the future to demonstrate our generalization ability. \n\n**Q2**: Nothing wrong with using closed-source models; would be nice (i.e., useful for practitioners in this field) to have results with open-source LLMs. \n\n**A2**: Thank you for your affirmation. We believe your opinion is very valuable. As we mentioned in the Limitations section, this in-context method relies on the capabilities of LLM, which are trained in JSON format data and natural language. In future work, we will further demonstrate the effectiveness of our method using models such as Llama and ChatGLM etc. \n\n**Q3**: The updating strategy section is a bit unclear. Would be good to expand on that in the camera-ready if the paper is accepted. Again, I would also stress the data preprocessing part: I would strongly suggest adding a section in the appendix. \n\n**A3**: We apologize for any confusion caused by our presentation. In section 2.3, we mentioned the update strategy for representing the current slot-value form of the state based on the content of the JSON generated by LLM. This is a rule-based processing strategy. We used the if-else expression to make the expression of the rules more concise and accurate, in accordance with our code-writing logic. The JSON forms generated by user and system utterances were mentioned in section 2.1. We will sort out the expression in the main text and add a detailed data processing section in the appendix to make the relevant parts easier to understand. \n\n**Q4**: The grammar is a bit problematic in some points with obvious mistakes (e.g., appendix section title \"Explains\" to mean \"More details\"?) and some parts even if not wrong are not written. The paper would need a thorough revision of the exposition. \n\n**A4**: Alright, we will double-check our grammar expression. However, we believe that these minor differences in expression do not have much impact on the paper. \n\n**Q5**: How did you preprocess the text? How did you preprocess the labels? \n\n**A5**: Regarding text preprocessing and labeling, we followed most of the previous work of IC-DST (https://github.com/Yushi-Hu/IC-DST) on preprocessing and labeling data. The only change is that we renamed some of the slot and domain names. The specific mapping rule dictionary is included in the self.kw2schema variable on line 26 of prompt_utils.py in the Supplementary Materials we submitted, which we hope will be helpful to you. We will consider explaining this in the appendix in future revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting and useful method", "reasons_to_reject": "- Only evaluated on the 2.1 version of MultiWoz which is known to have noisy annotations. The paper would benefit from evaluation of other versions (e.g, 2.2 and 2.4)\n- The data preprocessing and evaluation are not explained in depth. Since results are very much dependent on these factors, the paper needs to add more details about it to improve its soundness (happy to raise the soundness score once addressed).", "questions_to_authors": "- How did you preprocess the text?\n- How did you preprocess the labels?\n\nBonus:\n- Nothing wrong with using closed-source models; would be nice (i.e., useful for practitioners in this field) to have results with open-source LLMs.", "paper_topic_and_main_contributions": "This paper introduces ParsingDST, a new In-Context Learning (ICL) method for zero-shot Dialogue State Tracking (DST) that reformulates DST as a text-to-JSON task. Previous ICL approaches typically decode the dialogue to directly extract the values of all slots without considering the updating strategies. In contrast, this method applies additional updating strategies after the text-to-JSON conversion. This allows for manual updates, too.\n\nThe main contributions are: reformulating DST as a text-to-JSON task and proposing new state-updating strategies (given the extracted JSON).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Overall the manuscript is not clearly written in some parts. For example:\n\n- The updating strategy section is a bit unclear. Would be good to expand on that in the camera-ready if the paper is accepted. Again, I would also stress the data preprocessing part: I would strongly suggest adding a section in the appendix.\n\n- The grammar is a bit problematic in some points with obvious mistakes (e.g., appendix section title \"Explains\" to mean \"More details\"?) and some parts even if not wrong are not written. The paper would need a thorough revision of the exposition."}}, {"review_id": "YzwHUg6xYc", "reviewer": "Reviewer_Qxvg", "summary": "", "questions": "Can you elaborate on what exactly is the framework, and what it means to evaluate your approach with and without the framework?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestions and questions. Below is our detailed response:\n\n**Q1**: The paper only evaluates on MultiWOZ 2.1. \n\n**A1**: Hello, we used the settings that performed best on Moz 2.1 and supplemented our method with JGA results on Moz 2.4 (the performance improvements compared to IC-DST are shown in parentheses below):\n\nPer-domain JGA of **IC-DST Gpt-3.5-turbo-0301** and **ParsingDST Gpt-3.5-turbo-0301** in Moz 2.4: \n\n|Model| attraction | hotel | restaurant | taxi | train | avg |\n| :---- | :---- | :----: | :----: | :----: | :----: | :----: | \n|IC-DST Gpt-3.5-turbo| 59.81 | 40.20 | 46.80 | 68.32 | 52.87 | 53.60 |\n|ParsingDST Gpt-3.5-turbo| 65.63(+5.81) | 46.76 (+6.56) | 67.67(+20.87) | 80.58(+12.26) | 62.59(+9.72) | 64.65(+11.05)|\n\nAs we can see, our method still achieves very satisfactory results and has significantly improved compared to the IC-DST method. We will supplement our results on MultiWOZ 2.4 in the future to demonstrate our generalization ability.\n\n**Q2**: Can you elaborate on what exactly is the framework, and what it means to evaluate your approach with and without the framework? \n\n**A2**: As mentioned in the introduction, after receiving the JSON output of the LLM (mentioned in section 2.1), we refer to the information in this result and update the current dialogue state using the updating strategy (mentioned in section 2.3). The updating strategies adopted for the JSON generated by different speakers are also different. \n\nTherefore, the motivation for designing this framework is to avoid harmful information from the history state or previous speaker being mixed into the JSON generated by the subsequent speaker and affecting the effectiveness of the update strategy. \n\nUsually, when we obtain the dialogue state, we input the history state and messages from both the user and the system to LLM and obtain the outputs of both together. Our framework modifies this shared input-output to be asynchronous. In each turn, we first input the history state and message of the previous speaker to LLM to obtain the JSON result, and then use additional modules (such as a simple filter module mentioned in section 2.3 of the paper to process the JSON from the system) to make rule-based modifications before using the update strategy to obtain a temporary dialogue state, which is used to guide the LLM's generation of the JSON result from the subsequent speaker. The update strategy is then used again to obtain the final dialogue state of the current turn. This asynchronous input-output framework avoids harmful information from previous context or speaker affecting the content generated later in the decoding process with more processing module and ensures the effectiveness of the update strategy.\n\nIn the absence of a framework, it means that the messages from both the user and the system in the turn are jointly inputted to LLM to obtain the JSON output of both at once. Afterwards, the final dialogue state of the current turn is obtained by accepting rule-based updating strategy processing. However, because the updating strategies adopted by different speakers are different, in the decoding process, the harmful information of the history state or the previous speaker generated may impact the information of the later speaker (such as confusing non-entity slot values from the system into the generated user JSON), which affects the results obtained through the updating strategy.\n\nWe have already described the part about our framework in the Introduction section and Framework design section (Section 2.4), and the above content is a more detailed explanation. We hope it can help you further understand our framework more easily.\n\n**Q3**: Section 2.4 was very hard to follow. It would help to clarify which functions are LLMs and which functions are rule-based. \n\n**A3**: Our section 2.4 is a further explanation of the framework diagram in Figure 2, where the order corresponds to the process of gradually obtaining the final dialogue state from left to right in the figure. The function state2JSON for converting slot-value type state to JSON format context is mentioned in section 2.2, and the rule-based update and filter functions are mentioned in the paper as corresponding respectively to the updating strategies and modules in section 2.3. If other functions need to use LLM, we have explained it in the paper such as Trans2JSON. Thank you for your feedback. We will consider adding references to previous sections in section 2.4 to make the article easier to understand, as well as adding a data preprocessing chapter in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper achieves a significant improvement over the state of the art zero-shot performance on the well-known MultiWOZ 2.1 benchmark.", "reasons_to_reject": "The paper only evaluates on MultiWOZ 2.1. It is well-known that MultiWOZ, and MultiWOZ 2.1 specifically, is plagued by annotation issues, which subsequent revisions have attempted to address (See for example Ye et al https://arxiv.org/abs/2104.00773, Campagna et al https://arxiv.org/abs/2009.07968 ). In particular, previous work has noted that MultiWOZ 2.1 does not have a well-defined and consistently-followed rule of whether a mentioned slot should be included in the \"gold\" belief state.\nGiven the description of the errors provided by the authors themselves in the introduction, the most likely explanation of the improvement is not that the model is measurably better, but that the update strategy tracks better the quirks of MultiWOZ's poor and inconsistent annotation scheme.", "questions_to_authors": "Can you elaborate on what exactly is the framework, and what it means to evaluate your approach with and without the framework?", "paper_topic_and_main_contributions": "The paper proposes a novel approach to use large language models for dialogue state tracking, in the in-context learning framework. Rather than directly proposing the target belief state, they propose to generate a structured representation that accounts for the user's acceptance or rejection of a suggestion, and accounts for whether the assistant was making a suggestion or asking for clarification. The formal representation is updated throughout the dialogue with a combination of a LLM prediction and a rule-based state update based on the previous formal state.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 2.4 was very hard to follow. It would help to clarify which functions are LLMs and which functions are rule-based."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the zero-shot dialogue state training problem. The authors introduce a novel in-context learning approach, ParsingDST, which utilizes large language models. This method translates DST into JSON via semantic parsing, acting as an intermediary state. The tests indicate that the proposed approach sets a new state-of-the-art for zero-shot DST settings.

The soundness scores for this paper stand at (4, 3, 2). In the initial submission, a prevalent concern among reviewers was the outdated evaluation benchmarks. However, the authors addressed this in their rebuttal by presenting results from the most recent benchmark. Consequently, there's a consensus among reviewers on the soundness of the experiments. The score of 2 for soundness primarily reflects the paper's methodological section, which could benefit from clearer exposition. We advise the authors to enhance the clarity of the technical details.

Regarding excitement, the scores are uniformly (3, 3, 3), indicating unanimous agreement among reviewers on this metric.
Decision: Accept-Findings