Towards LLM-driven Dialogue State Tracking
Authors: Yujie Feng
ZEXIN LU
Bo LIU
Li-Ming Zhan
Xiao-Ming Wu
Venue: EMNLP 2023 Main (2023)
Keywords: dialogue state tracking
large language models

Abstract: Dialogue State Tracking (DST) is of paramount importance in ensuring accurate tracking of user goals and system actions within task-oriented dialogue systems. The emergence of large language models (LLMs) such as GPT3 and ChatGPT has sparked considerable interest in assessing their efficacy across diverse applications. In this study, we conduct an initial examination of ChatGPT's capabilities in DST. Our evaluation uncovers the exceptional performance of ChatGPT in this task, offering valuable insights to researchers regarding its capabilities and providing useful directions for designing and enhancing dialogue systems. Despite its impressive performance, ChatGPT has significant limitations including its closed-source nature, request restrictions, raising data privacy concerns, and lacking local deployment capabilities. To address these concerns, we present LDST, an LLM-driven DST framework based on smaller, open-source foundation models. By utilizing a novel domain-slot instruction tuning method, LDST achieves performance on par with ChatGPT. Comprehensive evaluations across three distinct experimental settings, we find that LDST exhibits remarkable performance improvements in both zero-shot and few-shot setting compared to previous SOTA methods. The source code is provided for reproducibility.

Towards LLM-driven Dialogue State Tracking
Yujie Feng Zexin Lu Bo Liu Liming Zhan Xiao-Ming Wu" Department of Computing, The Hong Kong Polytechnic University, Hong Kong S.A.R. (yujie.feng, zexin.lu, bokelvin.liu, Imzhan.zhan @ connnett.ooyu..kk xiao-ming.wus wei.g..@@@@@lllee poupooyu.cdu.hk
Abstract
Dialogue
State
Dialogue State Tracking (DST) is of paramount importance ensuring accurate tracking of user goals and system actions within task- oriented dialogue systems. The emergence of large language models (LLMs) such GPT3 and ChatGPT has sparked considerable inter- est in assessing their efficacy across diverse applications. In this study, conduct ini- tial examination of ChatGPT's capabilities in DST. Our evaluation uncovers the exceptional performance of ChatGPT in this task, offer- ing valuable insights to researchers regarding its capahilities and providing useful directions for designing and enhancing dialogue systems. Despite its impressive performance, ChatGPT has significant limitations including its closed- source nature, request restrictions, raising data privacy concerns, and lacking local deployment capabilities. To address these concerns, present LDST, an LLM-driven DST framework based on smaller, open-source foundation mod- els. By utilizing novel domain-slot instruction tuning method, LDST achieves performance par with ChatGPT. Comprehensive evalua- tions across three distinct experimental settings, we find that LDST exhibits remarkable per- formance improvements in both zero-shot and few-shot setting compared to previous SOTA methods. The source code! provided for re- producibility.
Clood morning, what cas help you!
Restaurant:
prlceranae expersive:
Please find me an expensive place side of'nnm..
SrOa: WorEl
Indion, Iualns, Mesicun you could choese from
Error propegation
Restaurant:
Riceting:: expersivee
100: mortis
fro chinese;
Okay, how about the good luck chinase faed? Will that work for
Restaurant:
20x3rtngg expensive:
mm21 sontig
feed: chinesc;
Yes, that sounds good!
lack chinose food;
theme amything can help you with?
Rostaurant: 2 Co-neferance
riccrane eagersive
also need hotel in the same
Dame: good
restaurntt
Hotel
nights.
peicerange: expensive
Figure I: Example of multi-domain dialogue. The slots "hotel- "hotel-pricerange" and cestaurrnt-pricerange have co-reference relationship, where the value of the former is inferred from the latter. The slot "restaurant- area" demonstrates error propagation behavior.
the evolving user goals and system actions during a conversation. In general, the multi-domain di- alogue state is represented as list of triplets in the form of (domain, slot, value), e.g., "crestau- rant, area, cast>"' These predefined slot pairs extracted from the dialogue context at each turn.
A plethora of models have been proposed address the challenges of multi-domain DST, documented recent studies (Qixiang et al., 2022; Zhou et al., 2022: Feng et al., 2022b; Guo et al.. 2022a; Yang et al., 2022; Ma et al., 2023; Xu et al., 2023a). These models primarily focus on effective transfer and generalization across diverse domains, addressing the crucial challenges of co-reference (Feng et al., 2022a) and error propagation prob- lem (Wang and Xin, 2022) depicted in Figure 1. The co-reference challenge poses a significant hur- dle in enhancing DST performance, as it arises from the linguistic variations in multi-turn dia- logues where slots and values are often indirectly
1 Introduction
Task-oriented dialogue systems have emerged as powerful tools for assisting users in accomplishing a wide range of tasks (Huang al., 2020). These systems, such as Apple Siri and Microsoft Cortana, function as virtual personal assistants, providing support for tasks like flight reservations, appoint- ment scheduling, and hotel bookings. Dialogue State Tracking (DST) plays a crucial role in task- oriented dialogue systems by accurately tracking
*Conresponding author.
'https://gittee" com/WaadSSeee/LLDST

---

expressed. Moreover, the error propagation issue emerges when the model fails recognize and rectify errors the previously predicted dialogue state, leading to the persistence of errors in subse- quent turns. Despite significant efforts to address these issues, they persist as ongoing challenges.
contributions are concluded follows:
We present the first evaluation of ChatGPT in DST task. highlighting its superior perfor- mance over prior methods and providing valu- able insights for advancing dialogue systems.
In recent days, the emergence of large-scale pre- trained language models has revolutionized the field of natural language processing (NLP). Models like ChatGPT2 have shown excellent performance, sparking significant interest in evaluating their ef- fectiveness across different dimensions (Tan et al.. 2023; Wang et al., 2023; Jiao et al., 2023; Yang et al., 2023a; Gao et al., 2023; Liu et al., 2023). De- spite the significant advancements made by large language models (LLMs), their performance in multi-domain DST remains relatively unexplored. To bridge this research gap, we conduct an evalua- tion of ChatGPT's capabilities for DST. The evalu- ation unveils ChatGPT's exceptional performance in the DST task, offering valuable insights to re- searchers and providing useful directions for fur- ther exploration.
We propose LLM-driven DST (LDST) based on smaller, open-source foundation models. LDST achieves comparable performance to ChatGPT by employing an innovative assem- bled domain-slot instruction tuning technique.
We extensively evaluate LDST on three bench- mark datasets across various experimental set- tings, revealing significant performance im- provements over previous approaches. In the zero-shot scenario, LDST boosts the JGA score by 16.9%, elevating from 65.3% to outstanding 82.2%. In the few-shot scenario, LDST improves the JGA score by 7.5%, rais- ing it from 47.7% to a notable 55.2%.
2 Assessing the Capabilities of ChatGPT for DST
While ChatGPT demonstrates superb perfor- mance, it has significant limitations (Zhou et al., 2023; Yang et al., 2023a; Cao et al., 2023). Firstly. it is not open source, so the underlying code and model parameters cannot be modified by users. Second, it is subject to request limitations, which restrict its usage in high-demand scenarios. Fur- thermore, there are concerns regarding strong data privacy protection, as the system may collect and store user data. Lastly, ChatGPT cannot be de- ployed locally, limiting its availability and control. These limitations hinder the applicability and adop- tion of ChatGPT in various practical scenarios for building task-oriented dialogue systems.
In this section, we evaluate the effectiveness of ChatGPT in addressing the DST task. Before going into detail, we first formally define the problem.
DST: Problem Formulation In task-oriented di- alogue systems, a dialogue with T turns of conver- sations between the system and the user can be rep- resented as {(A1, U1) (2), (Ar, Ur)}, where A represents system response and rep- resents user input. A predefined slot set S = [St....,JJ] Sy} is given, where J is the total number of slots. The dialogue context at turn / includes previous turns of interactions, denoted as X₂- - {(1, U1), (A2, (2),U2). ... (A,, U.)). The dialogue state at turn is represented as a set of (slot, value) pairs, denoted as B, { {S), V1) ....., (S,, VJ) where VJ is the value of slot S,. For multi-domain DST, following previous works (Lee et al., 2019), a slot defined as the concatenation of the specific domain and the slot, e.g., " "crestaurant-areas" If no information is provided in the dialogue about spe- cific slot, the value associated with that slot is set to "NONE". Essentially, the DST problem is defined as learning dialogue state tracker F X + Be.
To overcome the limitations of ChatGPT, we in- troduce LDST, a DST framework driven by LLMs but based on smaller, open-source foundation mod- els. LDST employs a novel assembled domain- slot instruction tuning method and parameter effi- cient tuning technique, enabling it to achieve per- formance comparable to ChatGPT while utilizing much smaller model and limited computational resources. LDST demonstrates exceptional perfor- mance across three different experimental settings, surpassing prior state-of-the-ar methods by a large margin and demonstrating its remarkable adapt- ability and generalization capabilities. Our main
Leveraging ChatGPT for DST We evaluate the performance of ChatGPT (using the gpt-3.5-turbo API service) on three multi-domain DST bench- marks, using the JGA and AGA evaluation metrics
#haps://chat.ppenai.com

---

the expected value is "14:00", ChatGPT may in- correctly predict "NONE", resulting in lower pre- diction accuracy. Secondly, the addition of a demo to the input has reduced effect, which may seem counter-intuitive. However, our analysis of the ror results suggests that ChatGPT also analyzes the dialogue context within the demo, even when the demo and tested sample are clearly differentiated in the input. Therefore, we chose the first prompt as the best template for the subsequent evaluation. Prompt Type
Input
Expected Output
AGA
Single return
No domo
Perfoom the task of nulti-dommin dialogue sute tracking
hotel pricerange: Cheap
95.92
The hnllowing the dizingue you need to lest: (daloguu context)
value of shooel pricenunges. (the description of the slot and possible valse lis]
So value slo sheeel pricenange
2
Muli return
No demo
Perfoom the task of mult domain dialogue state tricking.
botchpricmmgg Cheap. tniin-ouugpopper
B1,50
And the slot schema in this lis:
hotel pricermgei 'mmi bookpeople'. 'trite lotvent, "trin destination, "rstummctrrrel, x0x). following the dialogue you
traiskkneet: 14:00,
lest: (dialguue contect]
Please roturn the waher of slo lst pootel-riccemmyy Perform the lask of mulii-ommiin dislogue state trickin.
boekpookpspps...... troin-korvett
Single return
One
butckpriiznunge: Chep 91.93
will show
eompll Please nouenn the state of slot.
exampie dinlog [dladogue contestl
So value slot <helckgrionnnger
Ouigul resalt: cheap
The hillowing is the dislogue you need test: [daioome contest
value of slat shass---oss priezranges [Tee deseription the and it's possible value lid]
So value slt cheeekgriennrgee
Mult retur
One dema
Perform the task of nulti domain dillouue stae tracking.
hotel pricerange: Cheap. traizbookpeppe:: train-kovveat 14:00,
73.35
will show you an example. Please return the state the slot tst: horrl-griceraage, trait- bookpeople' Xxx esample dialog
(dalogue costext)
Ourput result: Tran DenoDeepparreeereeee The following the dialogue you need to jest: (dialogue context)
"Irin-Amm Arriaal: Cambridge, hotel pricerange: Cheap, XXx
Please return the value of slor ist (hotel priceronge. train bookpeuple. truit kavest. ....
Figure 2: Illustraioon of different prompt templates used and the corresponding results on the MultiWOZ. 2.2 test set.
(for detailed descriptions of the datasets and met-
Brsdlox
rics, refer to Section 4. As shown in Figure 2, we explore various prompt templates and utilize
163
the MultiWOZ 2.2 dataset (Zang et al., 2020) to select the optimal prompt.
In Figure 2. "single return" and "multi return" refer to the number of slot values returned in each ChatGPT API request. "Single return" involves requesting and receiving values for one slot at time, while "multi return" entails requesting and receiving values for all slots simultaneously. For instance, in the MultiWOZ 2.2 dataset which has 49 different slots, "multi return" retrieves values for all 49 slots in a single request. This causes a signif- icant increase API requests for "single return" but simplifies the model's 's task, resulting in improved performance. Conversely, "multi return" reduces API requests but increases token count per request. "No/one demo" denotes whether an example is pro- vided in the prompt as a demonstration to aid the model's comprehension of the task. Selecting "one demo" is similar to adopting the in-context learn- ing concept. Detailed prompt template design is provided in the Appendix A.I.
Stamwoz ShartWoz
(a) JGA score
(h) AGA score
Figure 3: The results of the best baseline and ChatGPT on various datasets. The higher the values of the JGA and AGA metrics, the better. SOTA results for Multi- woz 2.2, Multiwoz 2.4, JGA score for SGD datasets, and AGA score for SGD datasets were obtained from previous works (Bang et al., 2023a; Ye et al.. 2022a; Zhao al., 2022; Feng al., 2022a), respectively.
Performance of ChatGPT As can be seen from Figure 2, the first prompt, which retrieves the value of single slot in each request without including a demo in the input, achieves the highest AGA score. This is attributed to the inherent difficulty of the task that necessitates the model to provide multiple slot values in a single request. We have observed that ChatGPT tends to predict "NONE" for slots that should have a specific value. For in- stance, in the case of the slot "lavel-leaveat" where
The full evaluation results of ChatGPT on the three datasets are shown in Figure 3. Firstly, on the SGD dataset, the AGA score of ChatGPT is sig- nificantly superior than the previous SOTA method (Feng et al., 2022a), and achieves 7.46% ab-
'The evaluation of the MultiWOZ 2.2 dataset were con- ducted between April 15th and 18th 2023. The evaluations of MultiWOZ. 2.4 occurred between June 10th and 12th, 2023. The SGD was assessed hetween June 14th and 17th, 2023.

---

solute imporvement in AGA score. In addition, the average improvement on the three datasets is 0.73% JGA score and 2.34% in AGA score. Secondly, ChatGPT's performance on the Multi- WOZ 2.2 dataset is slightly worse than the previ- ous SOTA method (Bang ct al., 2023a). However, through careful analysis of the errors, we found that 70% of them were due to annotation errors in the original dataset. Thus, on the MultiWOZ. 2.4 dataset which has fixed the annotation errors, ChatGPT outperforms the best baseline method (Ye et al., 2022a).
parameter-cfficien fine-tuning (PEFT) technique to train the foundation model with the instruction dataset. PEFT enables the training of a foundation model with limited computational resources.
3.1 Instruction Tuning
Unlike prompt tuning, instruction tuning (Chung et al., 2022) provides more explicit and detailed guidance to the model through task-specific instruc- tions. This allows for finer control over the model's behavior and leads to improved performance com- pared to prompt tuning. The core idea of instruction tuning is designing the instruction dataset, typically including instruction, input, and output fields. Usu- ally, different instructions are assigned for different tasks. However, employing a fixed instruction tem- plate for multi-domain DST may limit the model's robustness, as emphasized by Wang et al. (2023), which highlights the crucial influence of prompt design model performance.
Limitations of ChatGPT In summary, ChatGPT exhibits comparable performance when solving the DST task compared to the previous SOTA meth- ods. This highlights the ability of current LLMS capture and comprehend complex linguistic pat- terns and dependencies within multi-turn dialogues. However, ChatGPT has several significant limita- tions that impede its applicability and adoption in various practical scenarios. Firstly, we observed that ChatGPT often provides responses with sig- nificant amount of explanatory content, or it may not align perfectly with our expected answer for- mat. For instance, when the ground truth value is "2 pm," ChatGPT might return "I "14:00." While both are essentially correct answers, such variations can affect the accuracy of the final metrics. And Chat- GPT is not open source, which restricts the ability of developers and researchers to modify and cus- tomize the model. Secondly, ChatGPT is subject to request limitations, which may impact real-time or high-volume applications that rely on prompt and efficient responses. Furthermore, ChatGPT oper- ates in a cloud- environment and lacks strong data privacy protection measures, which raises con- cerns about the privacy and security of sensitive information shared during the dialogue sessions. Lastly, ChatGPT relies on an internet connection for operation and cannot be deployed locally.
To address this challenge, propose novel Assembled Domain-Slot Instruction Generation ap- proach for the DST task. This approach generates diverse instruction samples by randomly combin- ing different instruction and input templates, expos- ing the model to a rich variety of instruction types during the fine-tuning process to reduce the model's sensitivity to prompts. As shown by the provided example in Figure 4, for each sample in the original dataset, it consists of the dialogue context X; at turn f, the current requested slot S, and its correspond- ing state Vj. The raw data is then passed through the Instruction Data Generation module to generate instruction samples. The detailed template settings for each field are introduced as follows.
Instruction Prompt Specifically, two types of instruction templates are defined: (1) Standard Slot Tracking Instruction and (2) Customized Slot Tracking Instruction. The difference between them is that the Customized Slot Tracking Instruction provides a more specific domain-slot information. And the instruction field each sample is ran- domly selected from these two templates.
3 Fine-tuning Smaller Foundation Models with Instructions for DST
To overcome the aforementioned limitations of ChatGPT, we introduce LDST, an LLM- driven DST framework that leverages fine-tuning smaller, open-source foundation models such as LLaMa (Touvron al., 2023) with instructions specially tailored for DST. We first outline the process of constructing an instruction dataset for the multi-domane DST task. Next, we utilize a
Input Prompt For the input field, the prompt template is composed of four main parts: (1) the di- alogue context, (2) domain- description prompt, (3) Possible Value List (PVL) prompt and (4) the query prompt. The green, purple, blue and orange text the example in Figure refers to these four prompts respectively. In particular, we concate- nate all sub-sequences with special segment tokens,

---

Instruction Data Generation
Frozen
Outpur
Instruction: (Randemly choose from the two templates below) (I) "Standard Slot Tracking Instraction>
Pine-Trane
f
8.4M Leamable parameters
eg, Track the state of the slot the input dialogue.
LLaMa
training on *RTX3090
(2) <Customized Slat Trucking Instruction>
Comparable ChasGPT
eg. Track the state of the slot <domain- slot> the input dialogue.
00 Assembled instructions
Low-Rank Adaptation
Input: [dialogue context] domain-slot description (randomly added) + pussible value list frandomly added). if the slot not mentioned in the dialogue, just return NONE, So the value of slot sdomsin-slot> is
An example of the Instruction-tunn -tuning Dutaset
Insteuction: Track the state of the shot <hotel-anes> in the inpul dialogue. Impet: [USER] I need book hotel in the cast that has 4 stars, ISY can help you with that, What your price range? [domain]] hotel, [slot] tren, it indieates area or place of the hotel. This slot is categorical and you can only choose from following available values: center, east, north, south, west. If the slot not mentioned in the dialogue, just return NONE. <hotel -aote---rrez---
Output [value]
1
Original Dialogue Dataset
Dialogue Context
Requested Slot: hayel-day
Value: Monday
Figure 4: Structure of the LDST model. the first step, we construct instruction dataset from original dataset using the Instruction Data Generation module. Next, we utilize the parameter-effiiien fine-tur fine-tuning technique to train the foundation model with the instruction dataset.
such the "[USER]" segment token used to in- dicate the start of a system utterance. And both the domain-slot description prompt and the PVL. prompt are supplementary descriptions of the re- quested slot, they all derive from the given schema in original dataset (PVL prompts are only available for categorical slots). Here, to simulate the situa- tion when the model may not get description of the requested slot it's possible values during the testing phase, we add these two prompt templates randomly with a 50% probability, respectively.
former architecture, greatly reducing the number of trainable parameters for downstream tasks. For
example, in our experiment with LLaMa 7B, the number of learnable parameters is 8.4M, which is only 0.12% of the total model parameters. De- note by the trainable parameters as a weight ma- trix Wo E Rdxc Unlike traditional methods that directly modify the values of Wo. LoRA in- troduces an additional set of trainable parame- ters, denoted AW, which are directly injected into the original Wo. We represent the update as W Wo + AW Wo + BA, where B Rdxr A E Rrxk The rank r < min(d, k).. During train- ing, Wo is frozen and does not receive any gradient updates, we only update the parameters in A and B. Note both Wo and AW = B 7 are multiplied with the same input, and their respective output vectors summed coordinate For the origi- nal output Woa, LoRA modified forward pass yields:
Ouput Prompt Finally, the output field consists of the corresponding value Vjj of the requested slot S.. By following the aforementioned process, we obtained newly and diverse instruction dataset for the next step of fine-tuning the model.
3.2 Parameter Efficient Tuning
In this part, we describe how to fine-tune the foun- dation model using a parameter efficient approach. LDST takes the instruction and input field from the dataset as inputs and retrieves the corresponding slot value V as output:
= Wox + AWr = Wox + BAx.
(2)
Finally, the learning objective of the generation process in LDST is to minimize the negative log- likelihood of VJ given the context X; and slot Sy:
VJ= = Decoder(2)
(1)
where Decoder indicates that the foundation model (e.g., LLaMa) uses the Transformer-decoder frame- work, and X' denotes the instruction data, i.e., the combination of instruction and input fields.
T
L
ΣΣ log (V3 S,).
As shown in Figure 4, to enhance the efficiency of the fine-tuning process and reduce memory requirements, we utilize Low-Rank Adaptation (LoRA) (Hu et al., 2021). LoRA freezes the pre- trained model weights and injects trainable rank de- composition matrices into each layer of the Trans-
Experiments
4.1 Datasets
We conducted experiments using the benchmark datasets for multi-domain task-oriented dialogue, and Table gives detailed statics on these datasets.

---

Characterisies
SGD
MuldWOZ2.3 2.2
MuliwwZ 2.4
Domain
SGDD-hseeiie
TransferQA
SDM-DST B3ST
LDST
domains
16
8
Messaging
10.22000
13.337.9
366666
896/90.4
No. of dialogues
16,142
8,438
8,438
Payment
11.5/34.8
24.7660.7
16,58200
92.39%.4
Total no, of turms
329,964
113,556
113,556
Truins
13.663.5
174/64.9
46.7/56.9
$1.994.0
Avg. per dialogue
20.44
13.46
Alam
57.2/18
58.38117
58.3187.5
94,4966.
Avg. tokens per turn
9:75
13.13
13.38
Average
20,534,2
25,96518
40,4/76.8
83.34
89.39444
215
Have schema description
Yes
Yos
Yos
Table 2: Zero-shot results (JGA(9)/AVG(()) on SGD.
Unseen domains in test set
Yes
No
No
Table 1: Statistics of the datasets used for training in our experiments.
Domain
TRADE:
SUMMT
SimpleT(M)
TSUST
Attraction
19.87
22.60
28.01
33.09
57.10
75.61
Hotel
13.70
19.50
17.69
21.21
63.32
Restiont
11.52
16.50
15.57
21.82
38.90
73.72
59.50
59.22
79.00
91.47
Guided Dialogue (SGD) SGD (Rastogi et al., 2020) is the most challenging dataset, consist- ing of over 16,000 conversations between human- user and a virtual assistant. It encompasses 26 services across 16 domains, such as events, restau- rants, and media. Notably, SGD introduces unseen domains in its test set, challenging the generaliza- tion ability of the model.
Trin
22.37
22.50
27.75
35.42
39.00
71,03
Avenuge
25.76
29.65
35.20
47.38
75.03
Table 3: Zero-shot results (IGA(S))AVG(()) on Multi-
WOZ 2.0.
Zero-shot Cross-domain Results Following pre- vious zero-shot settings (Wang et al., 2022c), all models are first trained on some domains and then evaluated on the test-set of the unseen domain. Here we compare with the baseline models that can predict dialogue state unseen domains: SGD- baseline (Rastogi al., 2020), TransferQA (Lin et al., 2021a), SDM-DST (Wang et al., 2022a), SUMBT (Lee et al., 2019), SimpleTOD (Hosseini- Asl al., 2020), TSDST (Lin et al., 2021 and D3ST method (Zhao et al., 2022).
MultiWOZ 2.2 and MultiWOZ 2.4 MultWWOZ 2.4 (Ye et al., 2022a) is updated version top of MultWWZZ 2.2 (Zang al., 2020) to improve DST evaluation, and the validation set and test set of MultiWOZ 2.4 have been carefully reannotated. We therefore treat it as clean dataset for testing. We also conduct experiments on MultiWOZ 2.2 which is known to contain annotation noise. We used this noisy dataset to test the robustness of the model and to analyse the ability of the model to detect annotation errors present in the test set.
Tables 2 and highlight the exceptional perfor- mance of our approach in zero-shot cross-domain DST. Specifically, achieves a remarkable 6.0% absolute increase in the JGA score when compared the larger T5-XXL (1 (1IB)-based D3ST model, elevating it from 83.3% to an impressive 89.3%. Additionally, the AGA score experiences substantial surge of 17.6%, es- calating from 76.8% to remarkable 94.4%.
the SGD dataset, LDST
4.2 Evaluation Metrics
We adopt the following evaluation metrics, consis- tent with previous works (Ye et al., 2022b): Joint Goal Accuracy (JGA) and Average Goal Accu- racy (AGA). JGA serves as the primary metric for DST evaluation and represents the ratio of dialogue turns for which the entire state is correctly pre- dicted. AGA is the average accuracy of the active slots in each turn. A slot becomes active if its value is mentioned in the current turn and is not inherited from previous turns.
On the MultiWOZ. 2.0 dataset, observe a sig- nificant advancement in the average JGA score, surging from 47.38% 75.03%. reflecting an ab- solute improvement of 27 27.65%. Notably, the Pay- ment domain in the SGD datasct displays the most prominent improvement, with the JGA metric soar- ing from 24.7% to an astounding 92.3%. This re- markable enhancement is attributed to the Payment domain's 's independence from the source domains. This significant boost only demonstrates the powerful transfer learning capabilities of the LDST model but also emphasizes its valuable implications for the DST research community.
4.3
Main Results
We conducted full-scale evaluations of the LLM- driven LDST model in three distinct experimental settings, where the model showed tremendous performance improvement in both zero-shot and few-shot settings. These findings can provide valu- able insights and contribute to the research com- munity through substantial advances in the field of DST. The detailed results are as follows:
Few-shot Results In the few-shot settings, we follow the multi multi-domain scenario from Wu et al.

---

MultiWO7. 2.4
Methods
Basci-maaal
MaleWOz
(4 Panmmeemo)
IGA
AGA
KGA
AGA
KGA
AGA
Models
1%
5%
10%
SCID-besaline
42.00
25.40
90.60
DS2-BART
30.35
42.53
41.73
TRADE
55.05
DST
BER (110M)
$1.20
DS2-T5
36.76
49.89
51.05
TipPy
BFRT suc! (10M)
53.
64.15
IC-DST GPT-Neo
17.36
29.62
34.38
Segt5ey DU
BERT (1100)
$4.40
90.00
67.10
30.10
91.00
MotuASSIST
75.57
SM2-11b
40.03
51.14
51.97
DiCKS-DST
(222(2
61.13
L.ST
46.77
56.48
62.45
TOATOU
63.79
SDP-DST
57.60
DIST
(118
58.60
75.90
86.40
Table 4: Results (in JGA(%)) of few-shot experiments MultWWOZ 2.4.
ChaGGPT
Tur-bo
61.52
91.85
83.16
99.27
81.81
98.46
(7B)
75.13
75.32
95.83
Table 5: Results of fine-tuning with full training data. represents the results are reported in the original paper. means that the exact number of parameters is uncertain but definitely exceeds 100 billion.
(2019), where randomly select 1%, 5%, and 10% of training dialogues train, and evaluation is conducted on the full test set of each domain. The evaluation results on MultiWOZ 2.4 are shown in Table 4, where we compare with SOTA few-shot DST methods: DS2-BART (Shin et al., 2022), DS2- T5 (Shin al., 2022), IC-DST GPT-Nco (Hu et al., 2022), and SM2- lb (Chen al., 2023).
LDST (LLaMa-7B)
Transfer
(T5xx--11B)
SGD MultiWOZ 2.4
28.9
31.6
MultiW 2.4 SGD
23.1
25.9
The results indicate a clear trend: as the amount of training data increases, the performance of all models consistently improves. Notably, our LDST model stands out in this setting. At the 10% data setting, it achieved significant performance gains by elevating the JGA metric from 51.97% to 62.45%, marking an impressive 10.48% absolute improvement. Even at the 5% data setting, our approach surpassed traditional methods that were using 10% of the data. This highlights LDST's remarkable capacity to excel in learning and cap- turing the core aspects of the task with smaller dataset.
Table 6: Results (in JGA(%)) of cross-dataset transfer between SGD and MultiWOZ 2.4.
SOTA performance the SGD dataset (with JGA score of 86.5%), surpassing LDST's 84.47%. However, it's important note that paDST re- lies on additional techniques, which contain back- translation between English and Chinese for data augmentation and special manual rules for model predictions. In contrast, LDST relies solely on the default SGD dataset without additional aids. Another SOTA method is D3ST, which uses T5- XXL backbone model with 11B parameters (our LDST utilizes 7B model, for outcomes based different foundational models and different model sizes, please consult Appendix B). D3ST surpasses LDST on the SGD dataset. However, it's notewor- thy that LDST outperforms D3ST on Multiwoz 2.2 and 2.4. Additionally, our model demonstrates im- proved effectiveness when compared to the I.LaMa backbone model, underscoring the ongoing impor- tance of fine-tuning LLMs in current research.
Results of Fine-tuning with Full Training Data We also evaluate the performance of LDST using the complete training data, and compare it with the following strong baselines, including SGD- baseline (Rastogi et al., 2020), TRADE (Wu et al., 2019), DS-DST (Zhang et al., 2019), TripPy (Heck et al., 2020), Seq2Seq-DU (Feng et al., 2020), MetaASSIST (Ye et al., 2022b), SDP-DST (Lee et al. 2021). TOATOD (Bang et al., 2023b), DiCoS- DST (Guo et al., 2022b), D3ST (Zhao et al., 2022), PaDST (Ma et al., 2019). And the results are shown Table 5.
Results of Cross-dataset Transfer We further performed experiments to assess cross-datases transfer capabilities, akin to the experiment out- lined in Table 4c by Zhao et al. (2022). The JGA sults are presented in Table 6. highlighting LDST's exceptional cross-dataset transfer abilities. When compared to the D3ST method, LDST showcases an average improvement of 2.7% in terms of JGA.
We initially note significant advancements in recent LLMs like ChatGPT and LLaMa. Notably, our model achieves competitive performance with ChatGPT and even surpasses it on the SGD dataset, particularly excelling in the AGA metric with score exceeding 98%.
The PaDST method has currently achieved

---

Models
JGA
AGA
1.0
I.aMM (backbone model)
68.61
96.37
LDST
Llabla
Baseline
+ Truditional Instruction Tunnig
72.87
97.98
0.8
+ Ours
75.02
99.04
0.8
Table 7: Ablation study. The mean JGA(%) and AGA(%) scores on Multiwoz 2.2, Multiwoz 2.4 and SGD reported.
1GA
0.6
0.5
0.4
0.3
0.2
LLaMM
LJaMM Traditioal Instraction Tonnig
LDST (Gura)
Turn ID
84
Figure 6: JGA score at each turn on MultiWOZ 2.2.
JGS
errors are related the values "dontcare" and "not mentioned". For example, in cases where the ground truth is "dontcare", the model predicts "not mentioned", and vice versa. Among all 37 slots, the top five with highest error rates are "hotel-type" (338 errors), "restaurant- area" (225 errors), "hotel name" (221 errors), and "attraction name" (205 errors), collectively account- ing for 45.11% of the total errors. Specifically, for the "hotel-type" slot, 78.10% of the errors are attributed the model frequently confusing "not mentioned" with the value "hotel". For instance, the correct value for "hotel-type" was "hotel", , but the model incorrectly predicted as "not mentioned".
78
MuldWWZ
MeliWOZ
SGD
Figure 5: Comparison of the sensitivity of the models to different prompts during the testing phase.
(290 errors), "hotel
4.4 Ablation Study
To validate the effectiveness of the assembled domain-slot instruction tuning, we conducted a comparison with traditional instruction tuning. which employs fixed prompt template contain- ing all the descriptions for the requested slot (see details in Appendix A.2). The results, as displayed in Table 7, clearly demonstrate that our method outperforms traditional instruction tuning. We ob- served a substantial 2.15% improvement in the JGA score and a 1.06% improvement in the AGA score. Next, to analyse the sensitivity of the model to different prompts during the testing phase. we de- signed six different prompts and evaluated their effects, the results are shown Figure The results clearly indicate that LDST demonstrates significantly higher accuracy and lower variance in test results compared to the other two bascline methods. The mean variance of our method is 0.04, in contrast to 0.78 for the LLaMa model, represent- ing a substantial decrease of 0.74. These findings highlight that the utilization of the assembled tech- nique for instruction tuning effectively reduces the model's sensitivity to prompts. This not only pro- vides more stable and efficient inference process but also enhances overall robustness.
4.6 Effectiveness of LDST in Addressing the Main Challenges of DST
For the co-reference challenge, we analyze the MultiWOZ. 2.3 dataset (Han al., 2021), which includes 253 test samples annotated with co- reference relationships. Our evaluation reveals that the best baseline method achieves an accuracy rate of 91.1%, whereas LDST model achieves an im- pressive accuracy rate of 96.4%, showcasing the significant improvement offered by our approach. Additionally, we visualize the JGA score for each dialogue turn in Figure effectiveness in addressing error propagation. The result clearly shows that as the number of dialogue turns increases, the performance of all methods ex- periences a decline. However, our LDST model demonstrates remarkable resilience to eтrоr prop- agation, showcasing a significantly slower decline compared to LLaMa and the best baseline method. These results emphasize the LDST model's capac- ity capture and comprehend complex linguistic patterns and dependencies in multi-round conver- sations, making it promising solution to mitigate the challenges associated with the DST task.
demonstrate the
4.5 Error Analysis
We analyze the types of incorrect predictions in LDST by using the 2835 incorrectly predicted sam- ples on MultiWOZ 2.4. Firstly, 45.72% of the

---

5 Related Work
we utilized the latest gpt- gpt-3.5-turbo API, highly capable GPT-3.5 model optimized for chat at a frac- tion of the cost. Consequently, we achieved new SOTA performance with ChatGPT, showcasing its significant strengths.
5.1 Multi-Domain Dialogue State Tracking
Recent studies in multi-domain DST have exten- sively utilized the me-trrined language models to achieve seg-pperformacce results (Ravuru al.. 2022; Yu et al., 2022; Sun et al., 2022; Feng et al., 2021; Wang et al., 2022b; Xu et al., 2023c). For ex- ample, Xie et al. (2022) proposed a multi-stage cor- rectable dialogue state tracking method to mitigate the error propagation phenomenon, while Wang and Xin (2022) proposed jointly decision making and dialogue update technique prevent error accumulation. In addition, Wang et al. (2022a) and Manotumruksa et al. (2022) solve the challenge of co-referencing by learning correlations between slots, for example, by using a combination of slot prompts or hard copy mechanism. However, these approaches still have limitations, such as the lack of robustness in handling complex dialogue con- texts and the difficulty in capturing fine ane-ggained semantic relationships between slots and values.
With the emergence of open-source large lan- guage models, such as LLaMa (Touvron al., 2023), it provides a series of higher-quality back- bone models with different parameters. Leveraging LLaMa and the technique of instruction tuning has proven to achieve better results (Taori et al., 2023), opening new avenues for our research.
Conclusion
In this study, we conduct an initial examination of ChatGPT's capabilities in multi-domain DST, showcasing its superiority over previous methods. This comprehensive evaluation provides useful di- rections for researchers design and enhance dia- logue systems. To solve the limitations ChatGPT, we present L.ST,, an MLM--rrven DST framework based on smaller, open-source foundation models. By utilizing novel assembled domain-slot instruc- tion tuning method, LDST achieves performance on par with ChatGPT. Comprchensive evaluations across three distinct experimental settings demon- strate the remarkable performance improvements achieved by LDST compared to previous methods.
5.2 LLMs for Dialogue State Tracking
While large language models such as GPT-3 (Brown et al., 2020) and T5 (Raffel et al., 2020) have gained significant popularity, the efficient uti- lization of these models remains a sigmificant chal- lenge. Recent advancements in parameter-efficient fine-tuning (PEFT) techniques have effeetively alle- viated this problem, such as LoRA(Hu al., 2021) and Prefix Tuning(Liu al., 2021). For instance. both Lee et al. (2021) and Yang et al. (2023b) proposed a prompt-tuning method that leverages domain-specific prompts and context information to improve the performance of DST task. Mean- while, Ma et al. (2023) and Chen et al. (2023) intro- duced the prefix tuning approach, which involves modifying the input prompt by adding specific to- kens the beginning of the dialogue, aiming to enhance the efficiency of model fine-tuning. How- ever, these methods still face challenges, where the effectiveness heavily depends on prompt design. Recently, Heck et al. (2023) exclusively tested ChatGPT's performance on the Multiwoz 2. dataset. In contrast, our evaluation covers the Mul- tiwoz 2.2, 2.4, and SGD datasets, providing more comprehensive assessment. While both Pan al. (2023) and Hudeček and Dušek (2023) included re- sults on the Multiwoz 2.2, Multiwoz. 2.4, and SGD datasets, their results were relatively lower due to their use of the text-davinci-003 API. In contrast,
Limitations
This work has two main limitations: (1) Subjec- tivity in prompt design: Prompt engincering has shown significant potential for the application of LLMs. However, the prompt designs used in our study are subjective and may not necessarily rep- resent the optimal choices. Consequently, the ef- fectiveness of using these prompts for model fine- tuning or testing may not always yield the best results. Exploring more systematic automated tech- niques for prompt design could enhance the overall performance of the model. (2) Input length con- straints: In our study, we set the input length of the model to 512, which was determined based on statistical analysis and already contains more than 90% of the samples. While it is possible to increase the input length, doing so may result in slower training and inference times. Additionally, when the dialogue or description content becomes too long, the challenge of effectively truncating or summarizing the input arises. Further investigation

---

into handling longer input sequences without com- promising model efficiency would be beneficial.
Yujie Feng, Jiangtao Wang. Yasha Wang. and Xu Chu. 2023. Towards sustainable compressive population health: A gan-based year-by-year imputation method. ACM Transactions on Computing for Healthcare,
Acknowledgments
4(1):1-18.
We thank the anonymous reviewers for their valu- able feedback and constructive comments, which greatly contributed to improve the quality of this work. This research was partially supported by the grant of HK ITF ITS/359/2IFP.
Yujie Feng. Jiangtao Wang, Yasha Wang, and Sumi Helal. 2021. Completing missing prevalence rates for multiple chronic diseases by jointly leveraging both intra-and inter-disease population health data correlations. In Proceedings of the Web Conference 2021, pages 183-193.
Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023. Exploring the feasibility of chatgpt for event extraction. arXiv preprini 233iv:2333.088333 033836
References
Namo Bang, Jeehyun Lee, and Myoung-Wai Wan Koo.
Jinyu Guo, Kai Shuang, Jijie Li, Zihan Wang. and Yixuan Liu. 2022a. Beyond the granularity: Multi- perspective dialogue collaborative selection for dia- logue state tracking. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 2320) 2332, Dublin, Ireland. Association for Computational Linguistics.
2023a. Task-optimized adapters for an end-to-end fask-oriented dialogue system.
Namo Bang, Jeehyun Lee, and Myoung-Wo Wan Koo. 2023b. Task-optmmized adapters for an end-to- end task-oriented dialogue system. arXiv preprint arXiv:2305002668
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Jinyu Guo, Kai Shuang, Jijic Li, Zihan Wang, and Yixuan Liu. 20226. Beyond the granular- ity: peste-perspective dialogue collaborative selec- tion for dialogue state tracking. arXiv preprint arXiv:2205.1000
Ting Han, Ximing Liu, Ryuichi Takanabu, Yixin Lian, Chongxuan Huang. Dazhen Wan. Wei Peng, and Min- lie Huang. 2021. Multiwoz 2.3: A multi-domain task- oriented dialogue dataset enhanced with annotation corrections and co-reference annotation. In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qing- dao, China, October 13-17, 2021, Proceedings, Part 11 10, pages 206-218. Springer.
Yihan Cao. Siyu Li. Yixin Liu. Zhiling Yan, Yutong Dai, Philip S Yu, and Lichao Sun. 2023. A comprehensive survey of ai-generated content (aige): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:2303.04226.
Derek Chen, Kun Qian, and Zhou Yu. 2023. Stabilized in-context learning with pre-trained language models for few shot dialogue state tracking. arXiv preprint arXiv:2302.05932.
Michael Heck, Nurul Lubis, Benjamin Ruppik, Re- nato Vukovic, Shutong Feng, Christian Geishauser, Hsien-Chin Lin, Carel van Nickerk, and Milica Gašié. 2023. Chatgpt for zero-shot dialogue state track- ing: A solution or an opportunity? arXiv preprint arXiv:2306.00 31366
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling mnstrucion-finetuneed language models. arXiv preprint arXiv:2210.11 14416.
Michael Heck, Carel van Niekerk, Nurul Lubis, Chris- tian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gašić. 2020. Trippy: A triple copy strategy for value independent neural dialog state tracking. arXiv preprint (=xXi::0000028777
Yue Feng, Aldo Lipani, Fanghua Ye, Qiang Zhang, and Emine Yilmaz. 2022a. Dynamic schema graph fusion network for multi-domain dialogue state tracking. arXiv preprint arXiv:2204.0667
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33:20179. 79- 20191.
Yue Feng, Yang Wang, and Hang Li. 2020. A sequence- to-sequence approach to dialogue state tracking. arXiv preprint 1oXi::2011.09533
Yujie Feng, Jiangtao Wang. Yasha Wang. and Xu Chu. 2022b. Spatial-attention and demographic- augmented generative adversarial imputation network for population health data reconstruction. IEFE Transactions on Big Data.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint 11X.v:2100.09685.

---

Yushi Hu,
Lee. Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022. In-context learning for few-shot dialogue state tracking, arXiv preprint arXiv:2233.0m 085668..
Jarana Manotumruksa, Jeffrey Dalton, Edgar Meij, and Emine Yilmaz. 2022. Similarity- based multi-domain dialogue state tracking with copy mechanisms for task-based virtual personal assistants. In Proceedings of the ACM Web Conference 2022, pages 2006-2014.
Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain di- alog systems. ACM Transactions on Information Systems (TOIS), 38(3):1-32.
Wenbo Pan, Qiguang Chen, Xiao Xu, Wanxiang Che, and Libo 2023. A preliminary evaluation of chatgpt for zero-shot dialogue understanding. arXiv preprint arXiv:230.004256
Vojtëch Hudcček and Ondiej Dušek. 2023, Are Iims all you need for task-oriented dialogue? arXiv preprint arXiv:2304006556
Gao Qixiang, Guanting Dong, Yutao Mou, Liwen Wang. Chen Zeng, Daichi Guo. Mingyang Sun, and Weiran Xu. 2022. Exploiting domain- related keywords description for few-sho cross- domain dialogue state tracking. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2460-2465.
WXJiao, WX Wang, JT Huang, Xing Wang, and ZP Tu. 2023. Is chatgpt good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301.088 108455.
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with a language model using schema-driven prompting. arXiv preprint arXiv:2109.07066.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5485-555
Hwaran Lee, Jinsik Lee, and Yoon Kim. 2019. Sumbt: Slot-utterance matching for universal and scalable belief tracking. arXiv preprint
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages
Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, et al. 202 la. Zero-shot dialogue state tracking via cross-task trans- fer. arXiv preprint arXiv: 10n.:0099905455
Lohith Ravuru, Sconghan Ryu, Hyungtak Choi, Hachun Yang, and Hyeonmok Ko. 2022. Multi-domain dia- logue state tracking by neural-retrieval augmentation. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 169- 175.
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang. Zhou Yu, Andrea Madotto, Funjoon Cho, and Rajen Subba. 2021b. Leveraging slot descriptions for zero-shot cross-domain dialogue state tracking. arXiv preprint arXiv:2105.04222
Jamin Shin, Hangyeol Yu, Hyeongdon Moon, Andrea Madotto, and Juneyoung Park. 2022. Dialogue summaries as dialogue states (ds2), template-guided summarization for few-shot dialogue state tracking. arXiv preprint arXiv:2203.015522
Bo Liu, Liming Zhan, Zexin Lu, Yujie Feng, Lei Xue, and Xiao-Ming Wu. 2023. How good are large lan- guage models at distr--iittiittton detection? arXiv preprint arxiv:2308. 10261.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. P- tuning v2: Prompt tuming can be comparable to fine- tuning universally across scales and tasks. arXiv preprint xrXiv22110.0760
Zhoujian Sun, Zhengxing Huang, and Nai Ding- 2022. On tracking dialogue state by inheriting slot values in mentioned slot pools. arXiv preprint arXiv:220..07156
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evalu- ation of chatgpt question answering system for answering complex questions. arXiv preprint arXiv:2303.07992
Mingyu Derek Ma, Jiun-Yu Yu Kao, Shuyang Gao, Arpit Gupta, Di Jin, Tagyoung Chung, and Nanyun Peng. 2023. Parameter-efficeat state tracking by prompt tuning. arXiv preprint arXiv:2301. 10915.
low-resource dialogue
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-followiin model. Stan- ford Center for Research Foundation Models. hupp://rrff. stanford. eduw/202/033/ 3/0/////appaa. himl, 3(6)77.
Yue Ma. Zengfeng Zeng Dawei Zhu, Xuan Li, Yiy- ing Yang. Xiaoyuan Yao, Kaijie Zhou, and Jian- ping Shen. 2019. An end-to-end dialogue state tracking system with machine reading comprehen- sion and wide& deep classification, arXiv preprint arXiv:1912.09297.

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 139777
external medical knowledge graph for diagnosis pre- diction in healthcare data. In Proceedings of the ACM Web Conference 2023, pages 28 2819-2830.
Yongxin Xu, Kai Yang. Chaohe Zhang. Peinie Zou, Zhiyuan Wang. Hongxin Ding, Junfeng Zhao, Yasha Wang, and Bing Xie. 2023c. Vecocare: Visit cquuences-clincca notes joint learning for diagno- sis prediction in healthcare data. Proceedings of the Thirly-Second International Joint Conference on Ar- rificial Intelligence.
Haoming Wang and Wang Xin. 2022. How stop an avalanche? jodem: Joint decision making through compare and contrast for dialog state tracking. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 7030-7041.
Puhai Yang. Heyan Huang, Wei Wei, and Xian-Ling Mao. 2022. Toward real-life dialogue state tracking involving negative feedback utterances. In Proceed- ings of the 28th ACM SIGKDD Conference Knowl- edge Discovery and Data Mining, pages 2222-2232.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023. Cross- lingual summarization via chatgpt. urXiv preprint 142249
Qingyue Wang. Yanan Cao, Piji Li, Yanhe Fu, Zheng Lin, and Li Guo. 2022a. Slot dependency modeling for zero-shot dooss- dialogue state tracking. In Proceedings of the 29th International Conference on Computational Linguistics, pages 510-520.
Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023a. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arxiv:2302.08001.
Yuting Yang. Wenqiang Lei, Pei Huang, Juan Cao, Jin- tao Li, and Tat-Seng Chua. 2023b. A dual prompt learning framework for few-shot dialogue state track- ing. In Proceedings of the ACM Web Conference 2023, pages 1468-1477.
Yifan Wang, Jing Zhao, Junwei Bao, Chaoqun Duan, Youzheng Wu, and Xiaodong He. 2022b. LUNA:
Learning slot-turn alignment for dialogue state track- ing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 3319-3328, Seattle, United States. Asso- ciation for Computational Linguistics.
Fanghua Ye, Jarana Manotumruksa, and Emine Yil- maz. 2022a. MultiWOZ. 2.4: A multi-domain task- oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. In Proceedings of the 23rd Annual Meeting of the Spe- cial Interest Group on Discourse and Dialogue, pages 351-360, Edinburgh, UK. Association for Computa- tional I Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoor- molabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, al. 2022c. Benchmarking generalization via in-context instructions on 1, 1,600+ language tasks, arXiv preprint arXiv:2204.07705.
Fanghua Ye, Xi Wang, Jie Huang, Shenghui Li, Samuel Stern, and Emine Yilmaz. 2022b. Metaassist: Robust dialogue state tracking with meta learning. arXiv preprint 2XXv:2200.12397
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini- Asl, Caiming Xiong, Richard Sacher, and Pascale Fung. 2019. Transferable multi-domain state genera- tor for orientr dialogue systems, arXiv preprint 1905.08735.
Dian Yu, Mingqiu Wang, Yuan Cao, Laurent El Shafey, Izhak Shafran, and Hagen Soltau. 2022. Knowledge- grounded dialog state tracking. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 3428-3433 3335,, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hongyan Xie, Haoxiang Su, Shuangyong Song, Hao Huang, Bo Zou, Kun Deng, Jianghua Lin, Zhihui Zhang, and Xiaodong He. 2022. Correctable-dss Mitigating historical context mismatch between train- ing and inference for improved dialogue state track- ing. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pages 876-889.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultWWOZ 2.2 A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online. Association for Computa- tional Linguistics.
Jing Xu, Dandan Song, Chong Liu, Siu Cheung Hui, Fei Li, Qiang Ju. Xiaonan He, and Jian Xie. 2023a. Dialogue state distillation network with inter-slot con- trastive learning for dialogue state tracking. arXiv preprint ar arXiv:2302.08222
Guo Zhang, Kazuma Hashimoto, Chien-Shena Sheng Wu. Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong, 2019. Find or classify? dual strategy for slot-value predictions on multi-domaie dialog state tracking. arXiv preprint arXiv: 191..03544.
Yongxin Xu, Xu Chu, Kai Yang, Zhiyuan Wang, Peinie Zou, Hongxin Ding, Junfeng Zhao, Yasha Wang, and Bing Xie. 2023b. Seqcare: Sequential training with

---

Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu. Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022. Description- driven task-oriented dialog modeling. arXiv preprint arXiv:2201.0840
on the content of the dialogue. No explanation!
"input": Input dialogue: [USER] I would like to find salon. [SYSTEM] In which city do you prefer the salon to be located? [USER] In SFO will be great. [domain] Hotels_2, it indicates popular service for searching and booking houses for short term stay [slot] number_of_adultss it indicates Number of people for the reservation. This slot is categorical and you can only choose from the following available values: 1, 2, 3, 4, 5. If the slot is not mentioned in the dialogue, just return NONE.
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang. Kai Zhang, Cheng Ji. Qiben Yan, Lifang He, et al. 2023. A comprehensive survey on pretrained foundation models: A history from hert to chatgpt. arXiv preprint Stti::2322004144
Yihao Zhou, Guoshuai Zhao, and Xueming Qian. 2022. Dialogue state tracking based hierarchical slot at- tention and contrastive learning. In Proceedings of the 31st ACM International Conference on Informa- tion & Knowledge Management, pages 4737-4741.
Description of Prompt Templates
So the value of slot 2f is
A.1 Prompt Templates for ChatGPT Request
Initially, we noticed that the results reported in the studies by Pan et al. (2023); Hudeček and Dušek (2023) were notably lower in comparison to our re- sults. We attribute this observation to two primary factors, as outlined below.
Prompt type "multi return" "no demo"
"instruction": Now you need to perform the task dialogue state tracking. And the slot schema is in this list [restaurant-area, hotel-name, attraction-name,
Mitigating the Generation of Excessively Lengthy Responses ChatGPT often generated answers with excessively detailed explanations, de- viating from the expected response format. For example, when asked about the "train-leaveat" slot, ChatGPT might respond with extensive informa- tion such as "Monday at 05:16 for the first train and Monday at 16:16 for the last train," whereas the cor- rect response should be simply "05:16." To address this issue, we introduced a prompt that includes "No explanation!" an instruction to ChatGPT not to provide detailed explanations. Experimen- tal results indicated significant improvement in answer accuracy through this approach.
(the remaining slots are omitted here)], which is in a domain-slot format. You need to return the slot and its value in dict format if the value is not none, and no explanation! "input": Input dialogue: [USER] I would like to find salon. [SYSTEM] In which city do you prefer the salon to be located? [USER] In SFO will be great. Please return the value of slot list [restaurant-area, attraction-name, (the remaining slots are omitted here)].
hotel-name,
API Version Differences Another factor is the utilization of different API versions. The prior works all relied on the text-davinci-003 API, while we utilized
powerful 5-t-3.5-turbo API, highly capable GPT-3.5 model optimized for chat a fraction of the cost.
Prompt type "single return" + "one demo"
"instruction": Now you need perform the task of multi-domain dialogue state tracking. And will show you an example and you need to return to the state of the slot I asked about.
Below we provide specific samples for the four different prompts in Figure 2.
Prompt type 1: "siinll return" 'no demo?
"instruction": Now you need to perform the task of multi-domain dialogue state tracking. You need to return the value of the slot I'm asking about simply based
"input": The example is: Input dialogue: [User]: need train reservations from norwich to cambridge [SYSTEM]: have 133 trains matching your request.

---

So the value of slot < <train-depar ture> is
plate, ie., "single return" "no demo."
A.2 Prompt Template for "Traditional" Instruction Tuning
And your result should be Norwich. The following is the dialogue you need to test:
Here, we present the template for traditional in- struction tuning, where "traditional" implies the ap- plication of instruction tuning directly the DST task with a fixed prompt template. It's important highlight that this fixed prompt template includes all slot descriptions, such as the domain-slot de- scription and the list of potential values. This fixed prompt is utilized during both the fine-tuning and testing phases.
Input dialogue: [USER] would like to find a salon. [SYSTEM] In which city do you prefer the salon to be located? [USER] In SFO will be great. [domain] Hotels_2, it indicates A popular service for searching and booking houses for short term stay [slot] jumber_of_adults it indicates Number reservation. This slot is categorical and you can only choose from the following available values: 1, 2, 3, 4, 5. If the slot is not mentioned in the dialogue, just return NONE. So the value of slot <Hotels of _duutsss is
people for the
"instruction": Track the state of the slot <restaurant-area> in the input dialogue. "input": [Dialogue context omitted... [Domain] restaurant, [Slot] area, it indicates the area or place of the restaurant. This slot is categorical, and you can only choose from the following available values: north, east, south, west. If the slot is not mentioned in the dialogue, just return NONE. So the value of slot <restaurant-area> is "output": north
Prompt type 4: "multi return" + "one demo"
"instruction": Now you need to perform the task of multi-domain dialogue state tracking. And I will show you an example and you need to return the answer strictly in the format of the example.
Additional Results
"input": The example is: Input dialogue: [User]: need train reservations from norwich to cambridge [SYSTEM]: have 133 trains matching your request.
B.1 Comparison of ChatGPT with Zero-shot Methods
Essentially, the evaluation of ChatGPT inherently belongs to the zero-shot setting. However, since found that ChatGPT's results have been compa- rable traditional fine-tuning methods, we put its results in Table in the paper. Additionally, we in- troduce ChatGPT's results from zero-shot settings and the results are shown in table and below.
Train-Dep: Norwich, Train-Arrival: Cambridge, ...(the remaining slots are omitted here)
Output result:
And need to test this example:
Input dialogue: [USER] would like to find salon. [SYSTEM] In which city do you prefer the salon to be located? [USER] In SFO will be great.
Domain
SDM-DST
LDST
ChatGPT
Messaging
36.6
89.6
92.8
Please return the value of slot list Trestaurant-area, attraction-name, ... (the remaining slots are omitted here)].
Payment
16.5
92.3
94.1
Trains
46.7
81.0
83.3
hotel-name,
Alarm
58.3
94.3
95.7
Average
40.4
89.3
91.5
Table 8: Zero-shot results (in JGA(%) of ChatGPT on SGD.
For practical reasons related API request costs, we conducted tests using these four prompt tem- plates exclusively on the MultiWOZ. 2.2 dataset. Subsequent evaluations on the MultiWOZ 2.4 and SGD datasets focused on the most effective tem-
The results clearly demonstrate that ChatGPT outperforms the two strong baselines, SDM-DST and TSDST, by a huge margin. This is primarily

---

Domain
TSDST
LDST
ChatGPT
B.3
Results with LLaMa of different sizes
Attraction
33.09
75.61
78.50
In order to investigate how model size influences performance, we have incorporated supplementary experimental findings involving the LLaMa-138 and -30B models on the SGD dataset. These results are presented in Table 11 below.
Hotel
21.21
63.32
66.75
Restaurant
21.82
73.72
77.49
Taxi
65.09
91.47
92.38
Train
35,42
71.03
72.81
Average
35.20
75.03
77.58
Table Zero-shot results (in JGA(%)) of ChatGPT on MultWWOZ 2.0.
Methods
Backbone
# Training Epochs
SGD
ChalGPT
GPT-3.5-Turbo
n/a
D3ST
T5 XXL (11B)
not provided
86.4
LDST
LLaMa (7B)
2
84.5
because the evaluation is conducted zero-shot environment, where ChatGPT inherently holds an advantage. It's important to note that as an API service, ChatGPT cannot be tuned offline and is exclusively used for testing purposes.
LDST
LLaMa (13B)
LDST
LLaMa (30B)
0.5
86.9
Table 11: Results (in JGA(()) with backbones of vary- ing sizes.
In the zero-shot setting, the performance of tradi- tional methods (e.g., SDM-DST and TSDST) is worse due to the lack of domain-specific train- ing data. ChatGPT, equipped with its extensive model size and rich pre-trainee pre-trained knowledge, dra- matically surpasses the performance of traditional methods and sets the upper bound of performance in the zero-shot setting. It's also worth mentioning that ChatGPT's performance approaches the results traditional methods fine-tuned on the complete training dataset, which is why we include in Table 5 for comparison
The results provide a clear indication that an increase in model size corresponds to an improve- ment in the JGA score. However, in practical appli- cations, a 7B model not only offers a more suitable fit for local deployment but also showcases impres- sive performance.
B.4 Inference Time Analysis
The table 12 below provides the results of infer- time. It's worth highlighting that employ 8-bit quantization for the LLMs, which leads slower inference times compared to standard 32-bit configurations.
In contrast, our LDST, utilizing a customized instruction tuning method, effectively approaches ChatGPT's performance in the zero-shot setting, with an average performance difference of 2.4% in the JGA score.
Methods
Inference Speed (Samples/Min)
T5 Jarge-770M
531
LDST LDST_LLaMa LL
90
LDST LDST_LLaMa2-78
84
LLST_LLaMa-111
64
B.2 Results with Different Foundation Models
LDST LLaMa-30B
35
We further performed evaluations using alterna- tive foundation model, Llama2-7B (Touvron et al., 2023), as depicted in Table below.
Table 12: Inference time for different models.
T5 large the backbone model of the SDP-DST baseline method. From the table, it's clear that the inference speed decreases as the model size in- creases. For example, LDST LDST_LLaMa-7B predicts an average of 90 samples per minute. When com- pared to the baseline method based on T5-large (770M), the speed of LDST is approximately one- sixth that of the baseline.
Methods
Backhone
Multiwoz 244
SGD
ChatGPT
GPT-3.5-Turbo
83.2
84.8
LLaMa-7B
LLaMa (7B)
75.1
75.3
LDST L.ST_L.LMMs LLaMa
LLaMa (7B)
79.9
84.5
LDST LLaMa2
LLaMa2 (7B)
81.9
85.1
Table 10: Results (in JGA(S)) with different backbones.
B.5 Effect of LoRA Configurations
The results show that LDST LLaMa2 performed the best SGD, attaining JGA of 85.1% and demonstrating a performance akin to that of Chat- GPT on MultiWOZ 2.4. It suggests that a stronger backbone can lead to better DST performance.
In our work, utilized common configurations: lora_r and lora target modules=[query key_proj, value_proj, output_proj] in each self- attention module that needs to be updated.

---

To further clarify the impact of LoRA configura- tions on the experimental results, we performed ad- ditional analysis on the Multiwoz 2.4 dataset using I % training set (To save training time, we set the epoch to 1). We varied the lora_r parameter with values of 1, 2, 4, 8, and 16. In addition, we experi- mented with two different tar__ttrrte_tttleeeete modules con- figurations: [q_proj. y_proj] and [q [q_proj, k p_ro, v_proj, o_proj]. This resulted in 10 distinct experi- mental setups.
consider the dialogue labeled "PMUL4398. in the Multiwoz 2.2 training dataset. comprises dialogue turns between the system and the user. With Multiwoz 2.2 featuring 49 unique slots, this dialogue yields 294 (6*49) training data samples. Here is specific example:
"dialogue": [SYSTEM] What can I help you with [USER] i need a place to dine in the center thats expensive [SYSTEM] I have several options for you; do you prefer African, Asian, or British food? [USER] Any sort of food would be fine, as long as it is a bit expensive. Could I get the phone number for your recomendation? [domain] restaurant find places to dine and whet your appetite [slot] area area or place of the restaurant [Possible Values] centre, east, north, south, west "state": centre
Jorn Jora_target_ modules Moraur
2
16
lg proj, proj]
29.75
31.38
33.11
39.07
40.40
1g proj. k proj. proj, o projl
36.09
Table 13: Effect of LoRA configurations. All results are reported in JG.(%).
In these results, smaller value of "lora_r" r" in- dicates fewer LoRA parameters, while lora target modules determines which modules receive LoRA update matrices. Generally, updating more atten- tion matrices yields better results, and performance improves as "lora_r" increases. However, it's es- sential to note that higher "lora r" values might extend the model's s training time. Hence, selecting appropriate hyperparameters is crucial.
In this example, the "dialogue" field includes the content of the dialogue (A1,U1) (42, U2), the tracked slot <restaurant-area>> and it's description. The "state" field is the value of the corresponding slot. For the slots that are not mentioned in the dialogue, the "state" field is assigned to NONE.
B.6 Results on MultiWOZ 2.1 Dataset
For comprehensive evaluation, refer Table 14, which presents the results on the MultiWOZ 2.1
Step Instruction Data Generation While the preprocessing in Step provided valuable data, it didn't 't entirely align with the required format for instruction tuning. As a result, it led to suboptimal experimental performance. To address this, we in- troduced additional preprocessing stage known as the "Instruction Data Generation Module," depicted in Figure This module is designed construct more suitable prompts.
dataset, comparing ChatGPT by Heek et al. (2023) with the D3ST method by Zhao et al. (2022).
Methods
Bused-model
MultWOZZ
CharGPT
56.44
D3ST
T5 XXL (11B)
$7.80
LDST (ours)
LLaMa (7B)
56.69
Table 14: Results (in JGA(%)) MultiWOZ.11. 2.1.
The aforementioned details the entirety of the preprocessing procedure, after which it be leveraged for both model training and testing.
The results reveal that LDST's performance is marginally below that of D3ST. This could be at- tributed to potential noise in the test set annotations, mirroring our observations regarding the Multi- WOZ 2.2 dataset.
Evaluation Regarding evaluation, we also uti- lized the code provided by Lee et al. (2021) to cal- culate JGA and AGA scores. A prediction was con- sidered correct when it exactly matches the ground truth. During the testing phase, used sistent prompt template that included domain-slot descriptions and lists of potential values. Experi- mental findings showed that this template slightly outperformed others because of its provision more comprehensive slot information.
Implementation Details
C.1 Data Preprocessing and Evaluation
Step Standard Preprocessing In line with the approach used by Lee al. (2021), this initial step involves the extraction of dialogue content and sloue pairs from the raw data. For instance,

---

C.2 Experimental Settings
proach did exhibit significant runtime improve- ments compared traditional methods in these settings. However, it does illustrate that our LLM- driven approach achieves the most substantial per- formance improvements while still maintaining ef- ficiency. These additional insights into the model's runtime in various experimental setups provide comprehensive understanding of the time required for training model and its comparison to other baseline methods.
During the training phase, we utilized batch size of 128 and set the learning rate to le-4. The num- ber of epochs varied depending on the specific ex- periment setup. For zero-shot experiments, we trained for 3 epochs. For few-shot experiments, we conducted experiments with different percentages of labeled data: 1% few-shot experiments were trained for 10 epochs, 5% few-shot experiments for epochs, and 10% few-shot experiments for epochs. For fine-tuning on the full dataset experi- ments, we used 2 epochs.
The model's cutoff length was set at 512, based data analysis. This length was determined be optimal as it covered more than 90% of the data. For samples with imput lengths exceed- ing 512 tokens, we truncated them to fit within the cutoff length. Additionally, the parameter lorin_on_iippne was set to false, indicating that the model solely computed the loss based on the final output.
Regarding the hyperparameters of the LORA module, we set the lora rank to 8, alpha to 16, dropout rate to 0.05, and selected "q_proj"" "kprooj", "v_proj" and "o_proj" as the LORA tar- get modules. Furthermore, in order to reduce the memory usage of the model, employed 8-bit quantization techniques to further optimize the fine- tuning process.
We would also like to offer further insights into the training time comparison of our model. In ex- periments involving fine-tuning the full dataset, model had average training time of 8 hours. contrast, powerful baseline methods, such as SDP-DST (Feng al., 2023) and DiCoS-DST (Xu et al., 2023b), required approximately 60 hours for fine-tuning the T5 model based on our testing. This substantial difference in training time under- scores the efficiency of our approach. And for the TOATOD (Bang et al., 2023b) method, which also utilizes the PEFT technique, the fine-tuning process only focuses soft prompts, reducing the overall runtime 12 hours. This runtime is comparable to our method, demonstrating the effectiveness of our approach compared to traditional methods.
In the case of few-shot experiments, the train- ing time for 1 1% labeled data 5 hours, 5% la- beled data required 8 hours, and 10% labeled data took approximately 10 hours to train. In contrast, the runtime for zero-shot experiments averaged around 12 hours. It's worth noting that our ap-


Reviews:
[{"review_id": "yiGcCLIdid", "reviewer": "Reviewer_1Gie", "summary": "", "questions": "1. The instruction directly asks the LM to \"perform the tasks of dialog state tracking\". I wonder whether ChatGPT or Llama has a good understanding of what \"dialog state tracking\" is. Therefore, it would be good to add some explanations to what DST is so that LM can more easily follow the instruction.\n\n2. In the design of input prompts (line 278), it seems only categorical slots are being considered (list of possible values). What about non-categorical slots like date, address?\n\n3. The paper argues that the proposed instruction tuning is better than \"traditional\" instruction tuning (Table 6). What precisely is the definition of \"traditional\" instruction tuning? Better give some examples, and share some insights into why different instruction tuning methods lead to different performance.\n\n4. What's the comparison between LoRA tuning full model tuning? Better add a comparison.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1. About whether LMs truly understand the task of \"dialog state tracking\" and suggests providing explanations to enhance comprehension.\n\n>We're happy to explain this concern. In preliminary stages of the experiment, both ChatGPT and LLaMa demonstrated a good grasp of DST-related questions and provided correct explanations. This confirms their understanding of the \"perform the task of multi-domain dialog state tracking\" instruction.\n\n>To refine the prompt, an inverse prompting approach was employed. It involved providing LLMs with DST task inputs and outputs, enabling them to generate suitable request prompts. This process contributed to the chosen instruction prompt.\n\n>Having ensured that the LLMs understand the essence of DST, a key factor affecting the model's performance is the correct understanding of the meaning of slots. This is because slots do not usually reflect standard written English, e.g., \" arriveby\" and \"ref\". These custom meaning representations are typically abbreviated and/or under-specified, which creates a barrier to the effective utilization of pre-trained LMs.\n\n>To address this issue, we incorporated natural language descriptions from schema files into DST prompts, e.g., the description of “ref” is expanded to “reference number of the hotel booking”. This step aims to bridge the gap between custom slot representations and language model understanding.\n\n\n### Q2. In the design of input prompts (line 278), it seems only categorical slots are being considered (list of possible values). What about non-categorical slots like date, address?\n\n>Non-categorical slots cannot provide a possible value list (PVL) because their values are context-dependent and can have an infinite range, making PVL prompts impractical for them.\n\n>Historically, categorical and non-categorical slots were treated differently. Categorical slots were treated as classification tasks with N classes based on PVL. For non-categorical slots, strategies such as the copy mechanism were employed to extract values from dialogues and pinpoint value positions.\n\n>In the era of pre-trained language models, both slot types are now unified as generation tasks. Additionally, to ensure accuracy in categorical slots, an extra PVL is provided.\n\n### Q3. About the definition of \"traditional\" instruction tuning and why different instruction tuning methods result in varying performance.\n\n>\"Traditional\" refers to applying instruction tuning directly to the DST task using a **fixed** prompt template. This template includes the standard slot tracking instruction from Fig. 4 and adds both Domain-Slot Description prompt and Possible Values List prompt to each example. This will be clarified in the final version.\n\n>In contrast, our proposed method, \"Assembled Domain-Slot Instruction Tuning,\" is designed for complex multi-domain DST task. It generates diverse and highly specific instruction samples by randomly assembling different instruction prompts and input prompts.\n\n>To illustrate the effectiveness of our method, we present three examples. These examples vary in the information provided in the input prompts, showcasing the versatility of our approach:\n\n>***Training Example A*** (a categorical slot, with slot description and PVL):\\\n[Dialogue context omitted...]\n[Domain] restaurant, [Slot] area, it indicates the area or place of the restaurant. This slot is categorical, and you can only choose from the following available values: north, east, south, west. If the slot is not mentioned in the dialogue, just return NONE. So the value of slot \\<restaurant-area\\> is\n\n>***Training Example B*** (a categorical slot, without slot description and PVL):\\\n[Dialogue context omitted...]\n[Domain] train, [Slot] has\\_internet. So the value of slot \\<restaurant-has\\_internet\\> is\n\n>***Testing Example C*** (a slot unseen during training, no slot description available):\\\n[Dialogue context omitted...]\n[Domain] hotel, [Slot] ref. So the value of slot \\<hotel-ref\\> is\n\n>**Explanation for the Superior Performance of Our Method:**\n>* **Enhancing model generalization ability to slots**\\\n>Traditional tuning (with a fixed prompt template) relies heavily on slot descriptions where the training samples are all of type A, making the model dependent on this information. Our method introduces diverse training samples, including those without any descriptions (Type B). This forces the model to learn slot meanings by analyzing correct outputs, reducing prediction bias when encountering new, undescribed slots during testing. The experiments in Table 6 and Figure 5 confirm the effectiveness of our approach.\n>* **Reducing model sensitivity to prompts**\\\nDifferent prompts can significantly impact model performance. Our approach exposes the model to various prompt variations during training, improving prediction stability. As illustrated in Figure 5, our LDST method yields more consistent results with smaller variances when tested with various templates.\n\n>We also experimented with different backbone models of various sizes ([Llama2-7B (Touvron et al., 2023)](https://arxiv.org/abs/2307.09288), [Llama-13B (Touvron et al., 2023)](https://[arxiv.org/abs/2307.09288](https://arxiv.org/abs/2302.13971)), and [Vicuna-7B (Chiang et al., 2023)](https://lmsys.org/blog/2023-03-30-vicuna)), confirming our LDST method's effectiveness across various settings. LDST_LLaMa-13B achieved the best result on the SGD dataset with a JGA score of 86.2%. We will incorporate the results into the appendix for reader reference.\n>\n\n\n| Methods                              | Based-model(# Parameters) | Multiwoz2.4 | SGD  |\n|--------------------------------------|---------------------------|-------------|------|\n| ChatGPT                              | GPT-3.5-Turbo             | 83.2        | 84.8 |\n| LDST | LLaMa (7B)                | 79.9        | 84.5 |\n|                                      | Vicuna (7B)               | 77.1        | 85.0 |\n|                                      | LLaMa2 (7B)               | 81.9        | 85.1 |\n|                                      | LLaMa (13B)               | 82.4        | 86.2 |\n\n\n\n### Q4. What's the comparison between LoRA tuning full model tuning? Better add a comparison.\n\n>Thanks for your suggestion. In our study, we did consider full model fine-tuning. However, due to the immense number of parameters in LLMs, executing full fine-tuning on lab-level GPUs (i.e., NVIDIA RTX 3090 we use) becomes quite challenging. Nevertheless, we still try our best to conduct a comprehensive ablation study on LoRA tuning, especially for rank r and target modules. We report results on MultiWOZ 2.4 with 1% training data, summarized in the table below:\n\n\n| lora_target_modules \\ lora_r     | 1     | 2     | 4     | 8    | 16   |\n|----------------------------------|-------|-------|-------|------|------|\n| [q proj, v proj]                 | 29.75 | 31.38 | 33.11 | 39.07 | 40.40 |\n| [q proj, k proj, v proj, o proj] | 31.59 | 40.11 | 36.09 | 40.19 | 42.02 |\n\n>In these results, a smaller \"lora_r\" indicates fewer LoRA parameters, while \"lora_target_modules\" determines which modules receive LoRA update matrices. Generally, updating more attention matrices yields better results, and performance improves as \"lora_r\" increases. However, increasing \"lora_r\" might prolong model training time, so choosing the appropriate hyperparameters is crucial.\n\n\n### Q5. About Missing References.\n\n>We appreciate the information about other relevant studies, and we'll incorporate the below discussion into the related work section. \n\n>[Heck et al. (2023)](https://arxiv.org/abs/2306.01386) exclusively tested ChatGPT's performance on the Multiwoz 2.1 dataset. In contrast, our evaluation covers the Multiwoz 2.2, 2.4, and SGD datasets, providing a more comprehensive assessment. While both [Pan et al. (2023)](https://arxiv.org/abs/2304.04256) and [Hudeček et al. (2023)](https://arxiv.org/abs/2304.06556) included results on the Multiwoz 2.2, Multiwoz 2.4, and SGD datasets, their results were relatively lower due to their use of the text-davinci-003 API. In contrast, we utilized the latest gpt-3.5-turbo API, a highly capable GPT-3.5 model optimized for chat at a fraction of the cost. Consequently, we achieved new SOTA performance with ChatGPT, showcasing its significant strengths.\n\n>Moreover, compared to these papers, our work's primary contribution lies in fine-tuning open-source LLMs and evaluating their performance across three distinct experimental settings, providing valuable insights for the DST community.\n\n\n### Q6. Other questions.\n\n>We will add ChatGPT's results to Tables 2 and 3 as follows:\n>\nTable2: Zero-shot results on SGD dataset (JGA score)\n| Domain    | SDM-DST | ChatGPT | LDST (ours) |\n|-----------|---------|---------|------|\n| Messaging | 36.6    | 92.8    | 89.6 |\n| Payment   | 16.5    | 94.1    | 92.3 |\n| Trains    | 46.7    | 83.3    | 81.0 |\n| Alarm     | 58.3    | 95.7    | 94.3 |\n| Average   | 40.4    | 91.5    | 89.3 |\n\nTable3：Zero-shot results on MultiWOZ 2.2 dataset (JGA score)\n| Domain     | T5DST | ChatGPT | LDST (ours)  |\n|------------|-------|---------|-------|\n| Attraction | 33.09 | 78.50   | 75.61 |\n| Hotel      | 21.21 | 66.75   | 63.32 |\n| Restaurant | 21.82 | 77.49   | 73.72 |\n| Taxi       | 65.09 | 92.38   | 91.47 |\n| Train      | 35.42 | 72.81   | 71.03 |\n| Average    | 35.20 | 77.58   | 75.03 |\n\n>From the results, it's evident that ChatGPT surpasses the two baselines, SDM-DST and T5DST, by a huge margin. This is mainly because the assessment is conducted in a zero-shot setting, where ChatGPT inherently has an advantage. As an API service, ChatGPT cannot be tuned offline and is exclusively used for testing purposes.\n\n>In the zero-shot setting, the performance of traditional methods (e.g., SDM-DST and T5DST) is worse due to the lack of domain-specific training data. ChatGPT, with its vast model size and rich pre-trained knowledge, dramatically surpasses the performance of traditional methods and sets the upper bound of performance in the zero-shot setting (Also note that ChatGPT's performance approaches the results of traditional methods fine-tuned on the full training dataset, hence we include it in Table 5 for comparison). \n\n>In comparison, our LDST, with a customized instruction tuning method, effectively approaches ChatGPT's performance in the zero-shot setting, with an average gap of 2.4% JGA score.\n\n>Additionally, the three datasets in Table 6 refer to Multiwoz 2.2, Multiwoz 2.4 and SGD, and we will update the table title accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper demonstrates an instruction-tuning method for LM on DST tasks. It also shows that such a tuned model can outperform ChatGPT on DST tasks.", "reasons_to_reject": "The proposal (instruction tuning) is a fairly standard and widely adopted approach, and the novelty of the paper is limited.\nComparison with other existing work on using ChatGPT for DST is missing.", "questions_to_authors": "1. The instruction directly asks the LM to \"perform the tasks of dialog state tracking\". I wonder whether ChatGPT or Llama has a good understanding of what \"dialog state tracking\" is. Therefore, it would be good to add some explanations to what DST is so that LM can more easily follow the instruction.\n\n2. In the design of input prompts (line 278), it seems only categorical slots are being considered (list of possible values). What about non-categorical slots like date, address?\n\n3. The paper argues that the proposed instruction tuning is better than \"traditional\" instruction tuning (Table 6). What precisely is the definition of \"traditional\" instruction tuning? Better give some examples, and share some insights into why different instruction tuning methods lead to different performance.\n\n4. What's the comparison between LoRA tuning full model tuning? Better add a comparison.", "paper_topic_and_main_contributions": "This paper proposes a way to instruction-finetune Llama model for dialog state tracking (DST) tasks. The paper shows that a properly finetuned Llama 7B model works on-par or better than ChatGPT on several DST benchmarks.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Better add ChatGPT results in Table 2 and 3 for comparison.\n2. Indicate which dataset is used for Table 6."}, "extra_text": {"Missing_References": "There are multiple papers that study the effectiveness of ChatGPT on task-oriented dialog modeling, including DST, e.g.\n[1] Michael Heck et al., ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?\n[2] Wenbo Pan et al., A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding\n[3] Vojtěch Hudeček, Are LLMs All You Need for Task-Oriented Dialogue?\nIt would be great to compare with these papers."}}, {"review_id": "xGrYsbgapk", "reviewer": "Reviewer_Qbz6", "summary": "", "questions": "Questions:\n- Results on these datasets are very, _very_, dependent on how the data are preprocessed; can you expand on how you preprocessed the labels and performed the evaluation (can you add more details in the appendix)? (this is the most important of my questions and I am happy to increase the soundness score after the authors' response)\n- How much of the results depend on the backbone model (i.e., Llama)? It would be great to add evaluation with other backbones\n- How does performance scale with the size of the Llama model?\n- How much longer does Llama + adapter take to perform inference? (useful for practitioners)\n- What's the effect of the LoRA configurations?\n\nSuggestions:\n- Many of the baselines feature much smaller models (<1B) vs Llama. You could include the T5-XXL (11B) results from [D3ST (Zhao et al., 2022)](https://arxiv.org/abs/2201.08904) which can help you to argue that your approach is competitive with models with similar sizes. Also, it's a good comparison to encoder-decoder models.\n- Remove the focus from ChatGPT. As the paper is currently written, there is a lot of emphasis on ChatGPT. Given the good results of your methodology, I think ChatGPT can be reprioritised to be a nice-to-have evaluation.\n- Abstract can be shortened\n- Highlight the efficiency gains (time and computation) in training vs baselines\n- More structured appendix featuring the data preprocessing and evaluation process (possibly with examples)\n- I think the appendix includes some interesting info that would benefit the community (e.g., training details, quantisation). Usually, these would not be interesting for typical backbones (e.g, T5), however, for these new LLMs I think this would provide good insights for the community.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1. Results on these datasets are very dependent on how the data are preprocessed; can you expand on how you preprocessed the labels and performed the evaluation (can you add more details in the appendix)?\n\n>Firstly, we greatly appreciate your question and the opportunity to clarify our data preprocessing and evaluation procedures. Please note that we did not make any additional changes to the labels, just followed the well-established procedure outlined by [Lee et al., (2021)](https://arxiv.org/abs/2109.07506). The steps include:\n\n\n>**Data Preprocessing**\n>* **Step 1 - Standard Preprocessing**\\\nSame as [Lee et al., (2021)](https://arxiv.org/abs/2109.07506), this step extracts dialogue content and slot-value pairs from the raw data. \\\n>For instance, consider the dialogue labeled \"PMUL4398.json\" in the Multiwoz 2.2 training dataset. It comprises 6 dialogue turns between the system and the user. With Multiwoz 2.2 featuring 49 unique slots, this dialogue yields 294 (6*49) training data samples.\\\n>Here is a specific example:\\\n\\{ \"**dialogue**\": \"[SYSTEM] What can I help you with [USER] i need a place to dine in the center thats expensive [SYSTEM] I have several options for you; do you prefer African, Asian, or British food? [USER] Any sort of food would be fine, as long as it is a bit expensive. Could I get the phone number for your recommendation? [domain] restaurant find places to dine and whet your appetite [slot] area area or place of the restaurant [Possible Values] centre, east, north, south, west\", \\\n\"**state**\": \"centre\" \\} \\\nIn this example, the \"dialogue\" field includes the content of the dialogue \\{(A1,U1), (A2,U2)\\}, the tracked slot \\<restaurant-area\\>, and it's description; the \"state\" field is the value of the corresponding slot. For the slots that are not mentioned in the dialogue, the \"state\" field is assigned to NONE.\n>* **Step 2 - Instruction Data Generation**\\\n>Although the preprocessing in Step 1 yielded valuable data, this data didn't completely conform to the required format for instruction tuning, resulting in subpar experimental performance. Hence, we introduced an additional preprocessing stage, referred to as the \"Instruction Data Generation Module\" in Figure 4, to construct more appropriate prompts.\n\n>The aforementioned details the entirety of the preprocessing procedure, after which it can be leveraged for both model training and testing.\n\n\n>**Evaluation**\\\nRegarding evaluation, we also utilized the code provided by [Lee et al., (2021)](https://arxiv.org/abs/2109.07506) to calculate JGA and AGA. A prediction is considered correct when it exactly matches the ground truth.\n\n>During the testing phase, we employed a fixed prompt template with domain-slot descriptions and possible value lists. Experimental results indicated that it outperformed other templates slightly, as it provided more comprehensive slot information.\n>\n>Additionally, in Figure 5, we compared the impact of different templates on model performance to better assess the model's sensitivity to prompts. The detailed steps and specific examples will be provided in the appendix for reference.\n\n### Q2. How much of the results depend on the backbone model (i.e., Llama)? It would be great to add evaluation with other backbones.\n>We appreciate your suggestion. In the paper, we initially showcased the performance of the original LLaMa in Table 5, which approached but did not match SOTA methods. However, the performance can be significantly enhanced with our fine-tuned LDST model.\n\n>Additionally, we conducted evaluations with the recently released open-source LLM [Llama2-7B (Touvron et al., 2023)](https://arxiv.org/abs/2307.09288), illustrated in the table below. (Due to time constraints, we only tested on the SGD and Multiwoz 2.4 datasets. We will include the results on the remaining Multiwoz 2.2 datasets in the final version.)\n\n| Methods                              | Based-model(# Parameters) | Multiwoz 2.4 | SGD  |\n|--------------------------------------|---------------------------|-------------|------|\n| ChatGPT                              | GPT-3.5-Turbo             | 83.2        | 84.8 |\n| LLaMa-7B                             | LLaMa (7B)                | 75.1        | 75.3 |\n| LDST_LLaMa  | LLaMa (7B)    | 79.9        | 84.5 |\n| LDST_LLaMa2                 | LLaMa2 (7B)               | 81.9        | 85.1 |\n\n>The results show that LDST_LLaMa2 performed the best on SGD, attaining a JGA of 85.1% and demonstrating a performance akin to that of ChatGPT on MultiWOZ 2.4. It suggests that a stronger backbone can lead to better DST performance.\n\n\n\n### Q3. How does performance scale with the size of the Llama model?\n>Thank you for your question. To explore the impact of model size on performance, we have included additional experimental results using the LLaMa-13B and -30B models on SGD dataset. Here are the results:\n>\n| Methods    | Based-model(# Parameters) | SGD  |\n|------------|---------------------------|------|\n| ChatGPT    | GPT-3.5-Turbo             | 84.8 |\n| LDST       | LLaMa (7B)                | 84.5 |\n| LDST       | LLaMa (13B)               | 86.2 |\n| LDST       | LLaMa (30B)               | 86.9 |\n\n>The results in the table clearly show that as the model size increases, the JGA score improves. Nevertheless, in practical terms, a 7B model not only fits better for local deployment but also demonstrates commendable performance.\n>\n\n\n### Q4. How much longer does Llama + adapter take to perform inference?\n\n>Thanks for your suggestion. The table below provides the results of inference time. It's important to note that we utilize 8-bit quantization for the LLMs, leading to slower inference compared to the standard 32-bit configuration. \n\n| Methods     | T5_large-770M | LDST_LLaMa-7B |  LDST_LLaMa2-7B | LDST_LLaMa-13B | LDST_LLaMa-30B |\n|-------------|-------------------------|----------------|----------------|----------------|----------------|\n| Samples/Min | 531                     | 90            |  84             | 64             | 35             |\n\n>(T5 large is the backbone model of the [SDP-DST](https://arxiv.org/abs/2109.07506) baseline method.)\n\n>From the table, it's clear that the inference speed decreases as the model size increases. For instance, LDST_LLaMa-7B predicts an average of 90 samples per minute. When compared to the baseline method based on T5-large (770M), the speed of LDST is roughly a sixth of that of the baseline.\n>\n\n\n\n### Q5. What's the effect of the LoRA configurations?\n>Thank you for your suggestion. In our work, we employed a commonly used configuration: lora\\_r = 8 and lora\\_target\\_modules=[query\\_projector, key\\_projector, value\\_projector, output\\_projector] in each self-attention module that needs to be updated.\n>\n>To further clarify the impact of LoRA configuration on the experimental results, we performed additional analysis on the Multiwoz 2.4 dataset using 1\\% training set (To save training time, we set the epoch to 1). We varied the lora\\_r parameter with values of 1, 2, 4, 8, and 16. In addition, we experimented with two different lora\\_target\\_modules configurations: [q\\_proj, v\\_proj] and [q\\_proj, k\\_proj, v\\_proj, o\\_proj]. This resulted in 10 different experimental setups.\n\n| lora_target_modules \\ lora_r     | 1     | 2     | 4     | 8    | 16   |\n|----------------------------------|-------|-------|-------|------|------|\n| [q proj, v proj]                 | 29.75 | 31.38 | 33.11 | 39.07 | 40.40 |\n| [q proj, k proj, v proj, o proj] | 31.59 | 40.11 | 36.09 | 40.19 | 42.02 |\n\n>In these results, a smaller \"lora_r\" indicates fewer LoRA parameters, while \"lora_target_modules\" determines which modules receive LoRA update matrices. Generally, updating more attention matrices yields better results, and performance improves as \"lora_r\" increases. However, increasing \"lora_r\" might prolong model training time, so choosing the appropriate hyperparameters is crucial.\n\n\n### Q6. Response to Suggestions \n>Thank you very much for your valuable suggestions. We will incorporate the supplementary content mentioned above into the appendix of the paper, presenting it with the help of figures and tables to enhance its comprehensiveness. Additionally, we will rectify any typos present in the manuscript. Once again, we appreciate your feedback, which has contributed to improving the quality and clarity of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Methodology is simple and effective (good performance wrt baselines). \n- Evaluation is thorough (zero-shot, few-shot, and full data settings) and performed on the 2 main benchmarks in this literature (SGD and MultiWoz)\n-  Interesting insights on the applications of LLM to DST (more in the Questions section below)", "reasons_to_reject": "Currently, the data preprocessing and evaluation are not explained in depth. Since results are very much dependent on these factors, the paper needs to add more details about it to improve its soundness (happy to raise the soundness score once addressed).", "questions_to_authors": "Questions:\n- Results on these datasets are very, _very_, dependent on how the data are preprocessed; can you expand on how you preprocessed the labels and performed the evaluation (can you add more details in the appendix)? (this is the most important of my questions and I am happy to increase the soundness score after the authors' response)\n- How much of the results depend on the backbone model (i.e., Llama)? It would be great to add evaluation with other backbones\n- How does performance scale with the size of the Llama model?\n- How much longer does Llama + adapter take to perform inference? (useful for practitioners)\n- What's the effect of the LoRA configurations?\n\nSuggestions:\n- Many of the baselines feature much smaller models (<1B) vs Llama. You could include the T5-XXL (11B) results from [D3ST (Zhao et al., 2022)](https://arxiv.org/abs/2201.08904) which can help you to argue that your approach is competitive with models with similar sizes. Also, it's a good comparison to encoder-decoder models.\n- Remove the focus from ChatGPT. As the paper is currently written, there is a lot of emphasis on ChatGPT. Given the good results of your methodology, I think ChatGPT can be reprioritised to be a nice-to-have evaluation.\n- Abstract can be shortened\n- Highlight the efficiency gains (time and computation) in training vs baselines\n- More structured appendix featuring the data preprocessing and evaluation process (possibly with examples)\n- I think the appendix includes some interesting info that would benefit the community (e.g., training details, quantisation). Usually, these would not be interesting for typical backbones (e.g, T5), however, for these new LLMs I think this would provide good insights for the community.", "paper_topic_and_main_contributions": "This paper proposes a new method to perform Dialogue State Tracking (DST). DST performance is measured by the Joint Goal Accuracy on the Schema-Guided Dialogues (SGD) and MultiWoz (versions 2.2, 2.3, and 2.4) datasets, as is typically done in the literature. They evaluate their approach in the zero-shot, few-shot, and full data settings.\n\nThey propose a method called LDST that consists in training a LoRA adapter for a large language model (LLaMa; Touvron et al., 2023) module on an instruction dataset build off of the original training set of the benchmark datasets. To build the instruction-tuning datasets, they propose the Assembled Domain-Slot Instruction Generation method. This approach generates diverse instruction samples by randomly combining different instruction and input templates, exposing the model to diverse instruction types during the fine-tuning process to reduce the model’s sensitivity to prompts. They compare their method with some strong baselines from the literature and their application of ChatGPT to the DST task.\n\nThe main contribution is proposing the Assembled Domain-Slot Instruction Generation method to create an instruction finetuning dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Some citations are concatenated with the text, e.g. \"some things to say(CITATION, 2023)\"\n- I would suggest removing hype-y adjectives (e.g., \"exceptional\", etc)\n- \"mains\" -> \"main\" in subsection title 4.6"}}, {"review_id": "FprVKmmfL1", "reviewer": "Reviewer_GUBv", "summary": "", "questions": "1. On what date was ChatGPT used here? ChatGPT is provided as an API service, and can go through changes over time. This should be included in the paper for reproducibility, or at least versioning.\n\n2. For completeness and ease of citation by future work, would it be possible to also report results on MultiWOZ 2.1? 2.1 is the canonical split for the dataset, and the training data between 2.1 and 2.4 should be the same.\n\n3. I’m a little confused by the results on MultiWOZ 2.2 cross-domain transfer (Table 3), which has higher JGA than the full-data results (Table 5): 60.65 vs. 75.03 JGA. Moreover, the JGA reported on each domain in zero-shot is higher than the average JGA in the full-data setting. Is this correct? Does this mean that additional training data is hurting the performance of LDST? If so, please include additional analysis.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1. Evaluation date and API version of ChatGPT\n\n>Thank you for your valuable suggestion.\n\n>For the MultiWOZ 2.2 dataset, evaluations were conducted between April 15th and 18th, 2023. The evaluations of MultiWOZ 2.4 occurred between June 10th and 12th, 2023. The SGD dataset was assessed between June 14th and 17th, 2023. All evaluations were performed using the gpt-3.5-turbo API service.\n\n>We will include the above information in the final version to ensure reproducibility.\n\n\n### Q2. For completeness and ease of citation by future work, would it be possible to also report results on MultiWOZ 2.1?\n\n>Thanks for your suggestion. The table below provides the results on the MultiWOZ 2.1 dataset, where we compared ChatGPT by [Heck et al. (2023)](https://arxiv.org/abs/2306.01386) and the D3ST method by [Zhao et al. (2022)](https://arxiv.org/abs/2201.08904). \n\n| Methods    | Based-model(# Parameters) | MultiWOZ 2.1 |\n|------------|---------------------------|-------------|\n| ChatGPT    | GPT-3.5-text-davinci-003  | 56.44       |\n| D3ST       | T5 XXL (11B)              | 57.80       |\n| LDST (ours)       | LLaMa (7B)                | 56.69       |\n\n>It can be observed that the performance of our LDST is slightly lower than that of D3ST. This may be attributed to noise in the annotation of the test set, a phenomenon similar to our observations on the MultiWOZ 2.2 dataset (lines 193-198). We'll include the results into the paper for completeness and reference by future researchers.\n\n\n### Q3. Response to confusion about the results on MultiWOZ 2.2 cross-domain transfer (Table 3), which has higher JGA than the full-data results (Table 5)\n>Thanks for your question. This phenomenon is completely normal and occurs due to the different evaluation settings between the zero-shot and full-data experiments.\n\n>In the zero-shot setting (lines 384-387), we focus on evaluating the accuracy of the model within a single domain. On the other hand, in the full-data setting, we calculate the multi-domain JGA score, which requires the model to accurately predict slots across all domains simultaneously.\"\n\n\n>Take the following ground truth state list as an example, which includes three slots from three different domains. In the zero-shot setting, for instance, when predicting the attraction domain, the model only needs to predict one slot \\<attraction-area\\>. However, in the full-data fine-tuning setting, the model has to predict three slots, making its task more challenging, thus resulting in lower JGA scores.\n>* Groundtruth state list with 3 domains:\\\n(attraction-area = east, hotel-name = wartworth, restaurant-bookday = sunday)\n\n>Additionally, we provide the JGA score **for each domain** in the MultiWOZ 2.2 dataset in the full-data setting:\n>* Attraction: 87.85;  Hotel: 82.40;  Restaurant: 88.75;  Taxi: 96.49;  Train: 88.52;  Average: 88.81.\n\n>All the single-domain JGA scores are much higher than the multi-domain result (60.65) in Table 5, and when compared to the zero-shot results in Table 3, they show an average improvement of 13.78\\%.\n\n\n### Q4. Response to missing references to previous SOTA results on the SGD dataset\n>Thank you for pointing out the missing references. We will include them in our paper to ensure a comprehensive comparison. Here is a comparison with the missing references:\n\n>Firstly, the [paDST method (Ma et al., 2020)](https://arxiv.org/abs/1912.09297) has currently achieved sota performance on the SGD dataset (with a JGA of 86.5), surpassing LDST's 84.47. However, it's important to note that paDST relies on additional techniques, which contain back-translation between English and Chinese for data augmentation and special manual rules for model predictions. In contrast, LDST relies solely on the default SGD dataset without additional aids.\n\n>Another SOTA method is D3ST [(Zhao et al., (2022)](https://arxiv.org/abs/2201.08904), which uses T5-XXL as backbone model with 11B parameters (our LDST utilizes a 7B model). D3ST surpasses LDST on the SGD dataset. However, it's noteworthy that LDST outperforms D3ST on Multiwoz 2.2 and 2.4.\n\n>In addition to adding D3ST (T5-XXL)'s zero-shot experimental results in Tables 2 and 3, we also conducted experiments to evaluate the cross-dataset transfer capabilities, similar to the experiment in Table 4c in the work of [Zhao et al. (2022)](https://arxiv.org/abs/2201.08904). The results in JGA are provided as follows:\n>\n| Transfer      | D3ST (T5-XXL 11B) | LDST (LLaMa-7B) |\n|---------------|-------------------|-----------------|\n| SGD&rarr;MultiWOZ 2.4 | 28.9              | 31.6            |\n| MultiWOZ 2.4&rarr;SGD | 23.1              | 25.9            |\n\n>These results demonstrate LDST's superior cross-dataset transfer capabilities. In comparison to the D3ST method, LDST exhibits an average improvement rate of 2.7\\% in terms of JGA score.\n\n\n\n### Q5. Citing criticisms of ChatGPT.\n>Thank you for your suggestion. We will reference these studies to substantiate the shortcomings of ChatGPT:\n>\n* [Zhou C et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt](https://https://arxiv.org/abs/2302.09419)\n* [Yang X et al. Exploring the limits of chatgpt for query or aspect-based text summarization](https://https://arxiv.org/abs/2302.08081)\n* [Cao Y et al. A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt](https://https://arxiv.org/abs/2303.04226)\n\n\n### Q6. Response to the evaluation details of ChatGPT\n>We appreciate your feedback and are committed to improving the clarity of our paper's expressions. In response, we have made the following modifications to Section 2 to enhance clarity:\n>* **Clarification on Single Return and Multi Return**\\\nIn Figure 2, \"single return\" and \"multi return\" refer to the number of slot values returned in each ChatGPT API request. \"Single return\" involves requesting and receiving values for one slot at a time, while \"multi return\" entails requesting and receiving values for all slots simultaneously. For instance, in the MultiWOZ 2.2 dataset which has 49 different slots, \"multi return\" retrieves values for all 49 slots in a single request. \nThis causes a significant increase API requests for \"single return\" but simplifies the model's task, resulting in improved performance. Conversely, \"multi return\" reduces API requests but increases token count per request.\n>* **Explanation of No/One Demo**\\\n\"No/one demo\" denotes whether an example is provided in the prompt as a demonstration to aid the model's comprehension of the task. Selecting \"one demo\" is similar to adopting the in-context learning concept.\n\n>We will also include details on all four prompt template variations in the Appendix section.\n \n>Below is an example illustrating the case of \"multi return\" + \"one demo\":\n>* Prompt type: \"multi return\" + \"one demo\"\\\n\\{ \"**instruction**\": \"Now you need to perform the task of multi-domain dialogue state tracking. The slot schema is in this list [restaurant-area, hotel-name, attraction-name, ...(the remaining slots are omitted here)], which is in a domain-slot format. I will show you an example and you need to return the answer strictly in the format of the example.\", \\\n\"**input**\": \"The example is: Input dialogue: [User]: I need train reservations from norwich to cambridge [SYSTEM]: I have 133 trains matching your request.  ...\nOutput result: Train-Departure: Norwich, Train-Arrival: Cambridge, Train-Departure-Time: Monday, Attraction-Name: cineworld cinema ...\nAnd you need to test the following example:\nInput dialogue:  [USER] I would like to find a salon. [SYSTEM] In which city do you prefer the salon to be located? [USER] In SFO will be great. ...\nPlease return the value of slot list [restaurant-area, hotel-type, attraction-name, ...(the remaining slots are omitted here)], where restaurant-area means area or place of the restaurant, hotel-type means what is the type of the hotel, ...  \" \\} \n\n>For practical reasons related to API request costs, we conducted tests using these four prompt templates exclusively on the MultiWOZ 2.2 dataset. Subsequent evaluations on the MultiWOZ 2.4 and SGD datasets focused on the most effective template, i.e., \"single return\" + \"no demo.\" \n\n\n### Q7. Response to comparison with evaluation results of ChatGPT in other papers\n>We appreciate your feedback and will integrate the discussion into the related work section. Below is a comparison of our results with those from other studies:\n\n>[Heck et al. (2023)](https://arxiv.org/abs/2306.01386) exclusively tested ChatGPT's performance on the MultiWOZ 2.1 dataset. In contrast, our evaluation covers the MultiWOZ 2.2, MultiWOZ 2.4, and SGD datasets, providing a more comprehensive assessment. While [Pan et al. (2023)](https://arxiv.org/abs/2304.04256), [Bang et al. (2023)](https://arxiv.org/abs/2302.04023) and [Hudeček et al. (2023)](https://arxiv.org/abs/2304.06556) presented results for the MultiWOZ 2.2, MultiWOZ 2.4, and SGD datasets, we observed that their reported results were comparatively lower. We attribute this observation to two main factors:\n>* **Mitigating Common ChatGPT Errors**\\\nChatGPT often provided answers with excessive explanations, diverging from expected formats. For instance, when prompted for the \"train-leaveat\" slot, ChatGPT might respond with overly detailed information, like \"Monday at 05:16 for the first train and Monday at 16:16 for the last train\", while the correct answer should be simply \"05:16.\" \nTo address this issue, we added \"No explanation!\" in the prompt. This prompt instructs ChatGPT not to explain its answers. Experimental results showed that this significantly enhances answer accuracy.\n>* **API Version Differences**\\\nAnother factor is the utilization of different API versions. The prior works all relied on the text-davinci-003 API, while we utilized a more powerful gpt-3.5-turbo API, a highly capable GPT-3.5 model optimized for chat at a fraction of the cost. \n\n>Consequently, we achieved new SOTA performance with ChatGPT, showcasing its significant strengths. Moreover, compared to these papers, our work's primary contribution lies in fine-tuning open-source LLMs and evaluating their performance across three distinct experimental settings, providing valuable insights for the DST community.\n\n\n### 8. Response to additional ablation experiments on the backbone model\n>We appreciate your suggestions and have conducted additional experiments using various backbone models, including [Llama2-7B (Touvron et al., 2023)](https://arxiv.org/abs/2307.09288) and [Llama-13B (Touvron et al., 2023)](https://[arxiv.org/abs/2307.09288](https://arxiv.org/abs/2302.13971)). We also included results for a T5-XXL-based method, D3ST  [(Zhao et al., 2022)](https://arxiv.org/abs/2201.08904), and the results are as follows:\n\n| Methods                              | Based-model(# Parameters) | Multiwoz2.4 | SGD  |\n|--------------------------------------|---------------------------|-------------|------|\n| ChatGPT                              | GPT-3.5-Turbo             | 83.2        | 84.8 |\n| D3ST                                 | T5-XXL (11B)              | 75.9        | 86.4 |\n| LDST (Ours)                           | LLaMa (7B)                | 79.9        | 84.5 |\n|                                      | LLaMa2 (7B)               | 81.9        | 85.1 |\n|                                      | LLaMa (13B)               | 82.4        | 86.2 |\n\n>LDST_LLaMa-13B exhibits performance comparable to ChatGPT on the MultiWOZ 2.4 dataset and  achieves similar performance on SGD compared to D3ST. Moreover, our LDST outperforms D3ST on both Multiwoz 2.2 (see Table 5 in the paper) and 2.4 datasets by using only a 7B backbone model. These results underscore the robustness of LDST across different backbone models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "LDST is a novel approach to TOD. While the actual method (LoRa tuning of LLaMa) is not novel and is currently being done for many other NLP tasks, to the best of my knowledge this is the first work to try it on the TOD setting. Moreover, it reports SOTA results on both SGD and MultiWOZ, the canonical TOD benchmarks.", "reasons_to_reject": "While the results demonstrated by the paper are very strong, I have some issues with the paper:\n\n1. This paper does not cite the correct previous SOTA results. There are much stronger results on SGD than the previous SOTA this paper cites (Feng et al., 2022 on L188): Ma et al. 2020 (https://arxiv.org/abs/1912.09297) and Zhao et al. 2022 (https://arxiv.org/abs/2201.08904) which have 86.5 and 86.4 JGA on SGD respectively. While this is still a few points lower than LDST, it is not as large of an improvement. Both papers also report strong results on MultiWOZ, including the cross-domain transfer on MultiWOZ 2.2 (Table 3), and the few-shot results on MultiWOZ (Table 4) — these should also be included in the comparison.\n\n2. There is not enough detail on how ChatGPT is prompted when described in Section 2. From what I can tell, the authors do not provide any detail on how the prompts are designed other than Figure 2, and there is not enough information to infer more details (e.g. what is “multiple return”? What is “no/one demo”?). This is detrimental to the reproducibility of the prompting results.\n\n    a. Moreover, prior work on prompting ChatGPT for MultiWOZ has had much weaker results (https://arxiv.org/abs/2304.06556, https://arxiv.org/abs/2306.01386, https://arxiv.org/abs/2302.04023). Please cite and compare against these additional results, and provide an analysis on how the results done in this paper differ.\n\n3. While the results demonstrated by LDST are strong, the specific approach it follows (LoRa tuning on LLaMa) is not very original or innovative, and not directly relevant to the TOD setting. Many other groups are working on following this exact approach for many other NLP tasks.\n\n4. It’s also not clear how much of the win that LDST demonstrates is from using a stronger pretrained LLM (LLaMa) over prior works (e.g. Zhao et al. 2022, https://arxiv.org/abs/2201.08904, which uses T5 XXL 11b). This paper would benefit from an additional ablation experiment in which the LLaMa LM is replaced with a different LM, and compared to prior results built off that LM (e.g. GPT-2,3 or T5).", "questions_to_authors": "1. On what date was ChatGPT used here? ChatGPT is provided as an API service, and can go through changes over time. This should be included in the paper for reproducibility, or at least versioning.\n\n2. For completeness and ease of citation by future work, would it be possible to also report results on MultiWOZ 2.1? 2.1 is the canonical split for the dataset, and the training data between 2.1 and 2.4 should be the same.\n\n3. I’m a little confused by the results on MultiWOZ 2.2 cross-domain transfer (Table 3), which has higher JGA than the full-data results (Table 5): 60.65 vs. 75.03 JGA. Moreover, the JGA reported on each domain in zero-shot is higher than the average JGA in the full-data setting. Is this correct? Does this mean that additional training data is hurting the performance of LDST? If so, please include additional analysis.", "paper_topic_and_main_contributions": "This paper presents a study on prompting ChatGPT for TOD benchmarks SGD and MultiWOZ, and demonstrate strong performance. However, they note that ChatGPT is not a reliable backbone to build a TOD system around (“closed source nature, request restrictions, raising data privacy concerns, and lacking local deployment capabilities“), and thus also propose LDST, a framework built around an LLM to perform DST through LoRA finetuning of a smaller open-source pretrained LLM (LLaMa). The authors study the performance of LDST on zero-shot, few-shot, and full-data regimes, and demonstrate SOTA results.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Mentioned in Reasons to Reject (1)\n\n2. On L87-L99 cite and attribute these criticisms of ChatGPT, I think sources for these shortcomings of ChatGPT should be cited, in order to provide context for readers.\n\n3. This is contemporary work and doesn’t need to be cited by the authors, but I want to note that LDST’s approach (parameter efficient tuning of LLM) was also explored by Mo et al. 2023 (“Parameter-Efficient Fine-Tuning Method for Task-Oriented Dialogue Systems”, https://doi.org/10.3390/math11143048, published July 2023)"}}]

Author rebuttal:

Meta-review: The paper presents an instruction-tuning method for applying LLMs to the DST task. To build the instruction-tuning datasets, it proposes the Assembled Domain-Slot Instruction Generation method. This method generates diverse instruction samples by randomly combining different instruction and input templates, which helps to reduce the sensitivity to prompts during the fine-tuning process. This paper performs evaluations in the zero-shot, few-shot, and full data settings and the results are very strong. 

Initially, the paper fell short in not comparing against other DST works using ChatGPT. This issue has been addressed in the rebuttal, together with further results on different backbone models. The authors are encouraged to add these new results and all implementation details in their final version.
Decision: Accept-Main