Scalable-DSC: A Structural Template Prompt Approach to Scalable Dialogue State Correction
Authors: Haoxiang Su
Hongyan Xie
Hao Huang
Shuangyong Song
Ruiyu Fang
Xiaomeng Huang
Sijie Feng
Venue: EMNLP 2023 Main (2023)
Keywords: Dialogue state tracking
Error propagation
Dialogue state correction

Abstract: Dialogue state error correction has recently been proposed to correct wrong slot values in predicted dialogue states, thereby mitigating the error propagation problem for dialogue state tracking (DST). These approaches, though effective, are heavily intertwined with specific DST models, limiting their applicability to other DST models. To solve this problem, we propose Scalable Dialogue State Correction (Scalable-DSC), which can correct wrong slot values in the dialogue state predicted by any DST model. Specifically, we propose a Structural Template Prompt (STP) that converts predicted dialogue state from any DST models into a standardized natural language sequence as a part of the historical context, associates them with dialogue history information, and generates a corrected dialogue state sequence based on predefined template options. We further enhance Scalable-DSC by introducing two training strategies. The first employs a predictive state simulator to simulate the predicted dialogue states as the training data to enhance the generalization ability of the model. The second involves using the dialogue state predicted by DST as the training data, aiming at mitigating the inconsistent error type distribution between the training and inference. Experiments confirm that our model achieves state-of-the-art results on MultiWOZ 2.0-2.4.

Scalable-DSC: A Structural Template Prompt Approach to Scalable Dialogue State Correction
Haoxiang Su Hongyan Xie Hao Huangl.4 Shuangyong Song?,
Ruiyu Fang", Xiaomeng Huang² and Sijie Feng' School of Computer Science and Technology. Xinjiang University, Urumqi, China
China Telecom Corporation Ltd. AI Technology Company 3JD AI Research, Beijing, China Xinjiang Provincial Key Laboratory of Multi-lingual Information Technology, Urumqi, China {haoxisu98, hwanghao, verdantpaul ine @@gmail com xiehongyan @ejd com (songshy, fangry,
cn
Abstract
St
may help you?
U;:
need book
house
Dialogue state correction has recently been proposed to correct wrong slot values predicted dialogue states, thereby mitigat- ing the error propagation problem for dialogue state tracking (DST). These approaches, though effective, are heavily intertwined with specific DST models, limiting their applicability to other DST models. To solve this problem, we propose Scalable Dialogue State Correc- tion (Scalable-DSCe which can correct wrong slot values the dialogue state predicted by any DST model. Specifically, Structural Template Prompt (STP) that converts predicted dialogue state from any DST models into standardized natural language sequence as part of the historical context, associates them with dialogue history information, and generates corrected dialogue state sequence based on predefined template options. We fur- ther enhance Scalable-DSC by introducing two training strategies. The first employs predic- tive state simulator to simulate the predicted di- alogue states the training data enhance the generalization ability of the model. The second involves using the dialogue state predicted by DST as the training data, aiming at mitigating the inconsistent eтror type distribution between the training and inference. Experiments con- firm that our model achieves state-of-the-art results on Multi WOz 2
B
(hotel-name, avalon)
(hotel-name, autumn house)
Syi
Definitely, for how many people
U2:
me. Can you give information vue cinema?
(hotel-name, avalon) (hotel-book stay, 1)
(attractionnamme, vue cinema)
Blabet
(hotel-namo, autumn house) (hotel-book people, 1)
(attraction-name, vue cinema)
Brrd
Bard
propose
C₂ B
U1
C2 Blred
U2
Figure 1:
An example of eтror propagation in DST task. "S," and "U," refer to system utterance turn and the user utterance of the 1-th turn, respectively. "Bpredu and "Blabet" represent the predicted results of the dialogue state the first turn model, the ground truth of the first turn dialogue state. "Ci" refers to the first turn of dialogue history. The other symbols can inferred in sequence. The red characters indicate errors.
1-th
represent the dialogue states as a set of (slot, value) pairs.
Currently, some DST approaches (Ye et al.. 2021b; Tian et al., 2021; Zhou et al., 2021; Xu et al., 2023) attempt to track the user's goal from the dialogue state of the previous turn, which as a compact representation of the dialogue history information (Kim et al., 2020), and the dialogue utterance for the current turn. These approaches have improved the efficiency of DST tasks. How- ever, the errors generated in the current turn of the model are likely to be carried over to the next turn. Figure/ illustrates an example of error propaga- tion: the predicted dialogue state Bpred in the first turn, which contains the wrong slot values, influ- ences the prediction B of the DST model in the second turn. Thus, recent studies (Tian et al., 2021;
1 Introduction
Pask-oriented dialogue systems are becoming increasingly important in facilitating the daily life of human beings. Dialogue State Tracking module plays an important role in dialogue systems (Wang and Lemon, 2013a), which aims to accurately track the user's goals based on the dialogue history and
Equal Contribution.
Corresponding author.
*Our code available at sjuspeech/Scalable--De
com/

---

Xie al., 2022) focus on correcting wrong slot val- ues in predicted dialogue states to mitigate the error propagation. The correction modules of these ap- proaches are deeply intertwined with specific DST models, limiting their applicability to other DST models. Moreover, to mitigate historical context mismatch between training and inference, by uti- lizing certain strategies, these approaches generate predicted dialogue states which may contain wrong slot values to train the model. However, the in- consistent error distribution between the predicted dialogue states generated by certain strategics dur- ing training and the dialogue states predicted by DST during inference is ignored, which limits the correction capability of the model.
model training, we propose DST model based on the STP approach (STP-DST), as shown in Figure 2.
We conducted extensive experiments on Mul- tiWOZ 2.0-2.4 (Budiiidiiin et al., 2018; Eric et al., 2020; Zang et al., 2020; Han et al., 2021; Ye et al., 2021a), and the results indicate that Scalable- DSC significantly improves the performance of STP-DST by correcting dialogue state errors and achieves new state-of-the-art. The contributions of the paper are as follows:
(1) We propose Scalable-DSC, a standalone dia- logue state correction model, that can universally correct the predicted results of different DST mod- els, adaptively infer errors in the dialogue state sequence and then generate a complete and correct sequence.
To address the above problem, we propose Scalable-DSC. This a standalone dialogue state correction model whose responsibility is to correct the wrong slot values in the dialogue state predicted by the DST model. Firstly, to enhance the scala- bility of Mtalable-DSC.. we propose the Struetural Templates Prompt (STP) approach, which consists of four components its input schema: (1) Di- alogue History provides Scalable-DSC with real dialogue information, serving as crucial evidence to correct the dialogue state; (2) By using heuristic scripts, the predicted dialogue state of any DST model is transformed into natural language se- quence (State Sequence), which is considered as part of the model's historical context, and then dialogue state sequence generated with corrected slot values. (3) Slot Options enable the model to associate predefined template content with rele- vant slot descriptions, making the model aware of which slot information needs to be selected; and (4) Template Options prompts the model to select relevant template content for performing controlled conditional generation. This generation approach forces Scalable-DSC firstly generate a sequence of erroneous states and then continue generating a sequence of corrected states. Secondly, we employ two training strategies to optimize the Scalable- DSC model in a staged manner. Strategy 1: During training, we use predictive state simulator (Xie et al., 2022) to dynamically generate predicted di- alogue states, which contain different wrong slot values in each epoch to enhance the model's gen- eralizability. Strategy 2: We use the DST model to predict the dialogue state as the training data for the Scalable-DSS model. In addition, to pro- vide predictive state data during the Scalable-DSC
(2) We introduce a Structural Template Prompt- ing (STP) approach. It controls Scalable-DSC through structured prompt texts to determine what needs to be corrected, what needs to be associated, and what needs to be generated through the tem- plate prompting mechanism.
(3) We use predicted dialogue states generated by a predictive state simulator and predicted dia- logue states from the DST model to train Scalable- DSC in stages. This approach not only enhances the model's generalization ability but also allevi- ates the issue of inconsistent distributions between training and inference error types.
2 Methodology
Figure 2 illustrates the overall framework of cor- recting erroneous slot values in dialogue states us- ing Scalable-DSC. In the preprocessing section, Strategy employs a predictive state simulator simulate predicted dialogue states in the training set. Strategy 2 utilizes a DST model to generate predicted dialogue states in the training set. In the training section, we utilize the predicted di- alogue states Bprredsee generated from the pre- processing section as part of the input schema for the STP method in Scalable-DSC, resulting in the generation of the corrected target sequence B Berror @ Beorrect_sog In the inference section, Scalable-DSC DDC corrects the prediction results of the DST model on the test set.
2.1 The Input Schema of STP Approach
Figure illustrates the input schema of the STP approach in Scalable-DSC. The schema includes

---

Stategy
Preprocessing
B
(nebbumet--ams, wee))
wsel)
trettsurantrooc britss)
Simulator
trestaurnnt food britsh)
The user is looking for restourent located the west with moderate prios, which serves brissh
Staingg
fesslaurnn-aree wasl) (resttuaatt price moderate) (restaurant food, britsh)
DST
Training
DDalogue Hileory: [TURN] [user] need dir be tish food west. [systern)
interneted moderala priced [OURR, [URe] de care
what
restuurant? ?
State Bequenca: The user looking for restuurant locked in the weat with
moderate price. which serves brilisn
The lowlowing dislogue
incomned..
restaurant area or place the restourant liauu 2]: restaurant-fe the cursine of the restn you are looking for
Slot Options: [sitin]]] (I] restxurani-gr budget for the restoumn muni [[atia_]]]]
Scalable-OSC
moderatt price. Action The oser looking for restiuran located the wini with any prico, which serves Brisah
The looking
Template Options: (The following cialoque status is incomnett Actuallyy [SUBJECTT]
The user ho [VERS) is looking for| searching for locks for| soarches for wwetss
[CEJECT] a restaurant located in the (rauu (I] with (value ol price, which
seroon (onkon 3
Inforance
mestaurant
frestsurantaare reeeauranttpio trettarrnn-kodd brilssh)
The looking for restourant located britsh
testaauant-pricc, modarate) (resttuantt-ood food. briish)
DST
which
1r a a D.
The for incommet. with
Dislogue History: H, State Sequence:
Blot Options: I 8
Scalable-DSC
cheap Actuslyy
looking for
Tomplate Options: Tesc
mests.mm located in the with moderate which servens Eniish
Dialogue History:
He State Sequence:
Siet Options:
STP-DST
The user is looking for restourant lccated the wost with moderate proce which serves britsh
Tamplate Options: Thar
Figure 2: The overview framework of the Scalable-DSC model for correcting dialogue state errors. The red font "moderate" indicates incorrect slot value in the dialogue state. There is correlation between the text segments with the same background color.
Dialogue History, State Sequence, Slot Options, and Template Options. Each element is described as follows:
terance.
State Sequence: We convert the dialogue states into natural language sequences by using the heuris- tic script by DS2 (Shin et al., 2022). Figure shows an illustrative example of using heuristic script to convert dialogue states into natural language se- quences. Given slot-value pair "(hotel-people, 6)", since the slot value is not "dontcare", the slot name "hotel-people" finds the corresponding frag- ment "for people" from the slot sequence fragment. By using the slot value "6" to fill in the missing part of the sequence, we obtain "for 6 people", then add the commmon sequence fragment "The user is looking for place to stay". The final se- quence form is obtained as "The user is looking for place to stay for 6 people". Specifically, when the slot value is "dontcare". the slot name needs to be searched for the corresponding frag- ment from the Dontcare slot sequence fragment. Moreover, we define the dialogue state at turn as B, = ((S,, Vÿ) | S j s J}. where V is the cor-
Due to the fact that the dialogue history provides real information for generating di- alogue states. We use Dialogue History aids for Scalable-DSC to correct dialogue states that con- tain erroneous slot values. For STP-DST, it relies on the dialogue history and the previous dialogue state to predict the current dialogue state. Dialogue History for the (-th turn is denoted as H, D @ e D @ [CU RR_TRRRUUUUE RN] @ [user]] U.. D. = [TUR]] @ [user] @ Ue @ [syslem] @ S, is the utterance in the t-th turn of the dialogue, where U, and S; respectively refer to the user's utterance and the system's utterance in the t-th turn. We use special tokens [user] and [sys] as prefixes for user utterance sequences and system utterance sequences, respectively. The prefix for the cur- rent turn utterance sequence is CUURRTUUU RNN], and [TURN] separates each turn of dialogue ut-
Dialogue History:

---

responding value of the slot Sj. and J represents the size of a set of predefined slots. Sy refers the domain and slot names connected by "-". Beared in Figure 2 represents the predicted dialogue states for the t-th turn, while Brred sey refers to the se- quence obtained by converting Byred the goal of Scalable-DSC is to correct the errors in State Se- quence Brreed while STP-DST is to generate the Bireed based on the previous dialogue state (State Sequence) Birrd and the dialogue history H, Template Options: We concatenates all the se- quence fragments in order (Common sequence fragment and Slot sequence fragment) obtain it. providing the model with more granular guid- ance. We replace the missing slot value portions in the fragments using special tokens "[value_a] "i ((i J).
the i-th masked slot value position in the tem- plate options by employing the same special to- ken [[value_]]"( = I,...,,), J). The advantage of this lies in that when the model needs to fill in the slot values masked by special tokens in the template, is able to associate with the correspond- ing slot's ss detailed description information through the special tokens. This helps the model compre- hend the semanties of the masked slot value po- sition and enables more accurate text completion. The slot options for all slots is represented as O desc.,
[value_l] desci D . D[ where desc; is the connection between the domain name of the i-th slot and the detailed description of the i-th slot. For example, the detailed description of the slot named "restaurant-fod"" is: "restaurant- the cuisine of the restaurant you are looking for".
J). It is worth noting that we have designed slot values containing "any" as replacement for the "dontcare" slot value !. The benefit of this is that template options do not require connecting Dontcare slot sequence fragment of each slot, re- ducing the input length of the template sequence. Figure 2. DSC and DST respectively represent the template options for Scalable-DSS and STP- DST. Especially, for Scalable-DSC, Dsc added a sequence at the begimning of TST: "(The follow- ing dialogue status is incorrect Aciually)". This sequence serves as guide instruct Scalable- DSC in inferring incorrect dialogue state sequence and then correcting them.
2.2 Sequence Generation in STP Approach
The model automatically selects relevant template fragments from the template options and replaces the special token "[value_i) _i" in the fragments with actual slot values to generate a corrected dialogue state natural language sequence. Especially, the generation process of Scalable-DSC can be divided into two steps: (1) Generating the phrase "The following dialogue status is incorrect" and then generate the incorrect parts of the state sequence; (2) Generating the phrase "Actually" and then gen- erating a complete and correct state sequence. As shown in Figure 2, we define the erroncous part of the state sequence as Berror_seg and the cor- rect state sequence as Brorrect_seg It is worth noting that when there are no errors in the state sequence. Scalable-DSC directly generate the se- quence B Beorrect_sey
Dialogue State
State Sequence
The user is looking for place to stay an any price paople
(hotal-people, 6) phatel-pricerangee any)
Sequence Fragment
[SUBJECT] The user he [VERB] looking searching for looks for seerches for wants a place to stay
Common sequence fragment
2.3 Training Strategy of Scalable-DSC
sequence fragment
(hotel-pricerange) : with
(hotel-people)
people price
We propose two-stage training of the Scalable-DSC model, with two training strategies for the stages, Strategy 1, we replicate the predictive state simula- tor algorithm for Correctable-DST (Xie et al., 2022) (Refer to Appendix A for more details). Using this algorithm, we randomly replace slot values in dia- logue state labels based on a certain probability 8 to simulate predicted dialogue states Bşimulate for the (-th turn, which may contain wrong slot values, as training data to train Scalable-DSC. Strategy 2. We use the DST model to predict the dialogue state of the training set as the training set for Scalable- DSC.
hotel-people does not about number people
Dontcare slot sequence fragment
Sottel-priceaanee : does not care about the price range
Figure 3: The process of converting dialogue states into sequence of dialogue state transitions. represents the position for filling slot values using the regulariza- tion approach.
Slot Options: We establish the connection be- tween the detailed description of the i-th slot and
'The specific replacement slot values for each slot are shown in Appendix C.

---

2.4 Training Objective
is defined as the proportion of examples (dialogues) where the predicted dialogue state of last turn ex- actly matches the ground-truth dialogue state of the last turn.
The model is trained with a maximum likeli- hood objective function. Given training sam- ple d (X, where the X is the encoder context of the model, and Y is the target output text of the model. The encoder context of STP- DST is defined as XSS Dialogue History @ &tate Sequence: # BB B SSot Options:
3.3 Settings
We initialize the Scalable-DSC and STP-DST model parameters with pre-trained T5- (Raffel et al., 2020) from Huggingface Transformers (Wolf et al., 2020). We set the learning rate to 5e-5, and the batch size 6. The probability /3 is set to 0.06. We optimize the model parameters by using the AdamW optimizer (Loshchilov and Hutter, 2019). The model training was performed on two NVidia RTX 3090 GPUs.
@0 @ Template options @TDSr and YOST Bered_seg is the target output text of STP-DST. In the Scalable-DSC model, the encoder con- text is defined as XDSC = Dialogue History @ DtaState Sequence: @ B @Sot Options: @O @ Template Options @TDCC and the target output text YDSC B Bervoor_eq @ Beorrect The objective of model Le is defined as
3,4 Baselines
Y
We use the following models as the baselines for comparisons: STAR (Ye et al., 2021b) captures slot correlations by self-attention mechanism; DSS-DST (Guo et al., 2021) proposes a slot sta- tus binary classifier to determine whether slot should update its slot value or inherit its slot value; FPDSC (Zhou et al., 2021) is multi-level fusion approach that integrates dialogue history and pre- vious dialogue state; LUNA (Wang et al., 2022) is DST approach for aligning slot and dialogue utterances; DSDN (Xu et al., 2023) dynamically utilizes the information of the previous dialogue state to track the user's goal; Tripy (Heck et al., 2020) proposes three copying approaches for fill- ing slot values; SOM-DST (Kim al., 2020) is a DST model that utilizes BERT encoding and em- ploys an RNN copying mechanism for decoding dialogue states; MinTL (Lin et al., 2020) a DST model that predicts only subset of dialogue states; TransformerDST (Zeng and Nie, 2020) proposes using only BERT as both the encoder and decoder; SimpleTOD (Hosseini-Asl et al., 2020) is an end-to- end dialogue system that formulates dialogue state tasks as sequential tasks; DS2 (Shin et al., 2022) converts dialogue states into sequence texts to track user's goals; AG-DST (Tian et al., 2021) proposes the use of secondary decoding to achieve the goal of correcting the dialogue state; D3ST (Zhao et al., 2022) proposes the approach of combining schema information and intent information to accomplish the dialogue state tracking; Correctable-DST (Xie et al., 2022) proposes to first obtain incorrect slot information the dialogue state and use these in- correct slot information to guide the correction the DST model in generating the correct dialogue
Le Σ log Xx)
(1)
where θ represents the parameters of the model.
3 Experiments
3.1
Dataset
We adopt MultiWOZ 22t.://ZZZ.2...ABBuuaaaooowssss et al., 2018; Eric et al., 2020; Zang et al., 2020; Han et al., 2021: Ye et al., 2021a) as the datasets in our ex- periments. MultWWOZ is multi-domain dialogue dataset that consists of dialogues between simu- lated users and dialogue systems. The training, validation and test sets in the MultiWOZ dataset contain 8420, 1000 and 1000 dialogues respec- tively. The model we follow the previous work (Wu et al., 2019; Lee et al., 2019; Kim et al., 2020; Le et al., 2020; Ye et al., 2021b; Wang et al., 2022) in training and only used data from five domains (restaurant, hotel, attraction, taxi, train).
3.2 Evaluation Metric
We borrow and modify the DS2 (Shin et al., 2022) heuristic script to convert the generated state se- quences of the model into dialogue states for evalu- ation. The model is automatically evaluated based on two metrics (Williams et al., 2013). Joint Goal Accuracy is the primary evaluation metric for cross-domain DST tasks. It is the percentage of turns for which all the slots are correctly identified. Slot Accuracy only considers individual slot-level accuracy, calculating the proportion of the correct
slot values filled in the slot from a macro perspec- tive. Final Joint Goal Accuracy Xie et al. (2022)

---

MultiWOZ %)
Model
2.0
2.1
2.4
Predefined ontology
DSS-DST
56,93
60.73
58.04
FPDSC
53.17
59.07
LUNA
55.31
57.62
56.13
DSDN
55.30
59.25
Open-vocabulary
TripPy
55.29
63.0*
59.62
MinTL
51.417/52.1
52.92"/53.62
/53.62
55.82"
56,95
67.56*
TransformerDST
/5.714
55.56' 155.35
55.64"
57.17'
69.74*
SimpleTOD
55.76
54.02
51.3*
DS2
54.78
52.32
D3ST
57.8
60.8
75.9
AG-DST
57.26
Correctale-DDS
67.51
68.24
70.30
71.38
81.27
SOM-DST
/21.1/
52.47' /52.57
53.27"
55.69* /55.5*
67.54' /66.78
+Scalable-DSCINI
65.56(0.06)
65.81(0.066
67.06(0.06)
67.1 ((0.06)
74.13(0.06)
+Scalable-DSC[R]
66.13
65.32
66.42
67.20
73.80
-Scalahle-DSC|R+NI
65.41
67.02
68.56
67.39
74.92
-Scalable-SCINN+RI
68.61
66.83
67.14
67.23
74.37
STAR
52.26'/54.53
52.26'
/54.53
54.08" /56.36
60.49*
65.87*
74.63 "/73.62"
+Scalable-DSCINI
(0.94)).06)
68.9 (8.994
71.47(0.06)
72.62(0.06)
79.60(0.06)
+Scalable-DSC(R]
64.54
68.44
71.07
72.45
78.60
-Scalable-DSC/RRNN
68.92
67.85
71.63
71.74
77.81
~Scalable-DSC/N+RR
68.51
69.02
71.33
72.50
78.91
STP-DST
55.51
55.85
57.08
63.79
76.31
+Scalable-DSCINI
65.73(0.06)
68. (8.04(0000)
71.11(0.06)
74.47(0.06)
81.17(0.06)
+Scalable-DSC(R]
65.97
67.76
71.31
74.56
81.40
-Scalable-DC[[R-N]
67.91
68.35
71.64
74.92
80.52
-Scalable-DSC/N+R1 66. 10
69.13
71.83
74.46
81.62
Table I: Joint goal accuracy (%) on the test of MultWWZz2.02244 the results borrowed from Xie al. (2022). : the results borrowed from Zang et al. (2020). the results borrowed from Tian et al. (2021). * results are cited from the websites http://gttuub.com//xmemnn com/lexmen W/ 9 the results borrowed from (Ye al., 2021a). "_" indicates public number is available. "(0.06)" refers the predictive state simulator's probability.
state. Upper part of Table shows the baseline results.
results have demonstrated that the joint target accu- racy of these two models is significantly improved after correction by Scalable-DSCC This shows that Scalable-DSS DSC is able to effectively correct erro- neous dialogue state predicted by any DST model, demonstrating its scalability. Furthermore, to in- vestigate the impact of different training strategies on the corrective performance of Scalable-DSC. we evaluate four training configurations: (1) [N]: Training Scalable-DSC using predicted dialogue states simulated by a predictive state simulator as the training data. (2) [R]: Training Scalable-DSC using predicted dialogue states from DST as the training data. (3) [N+R]: First using [N] strategy and then the [R] strategy train the Scalable-DSC model stages. (4) [R+N]: First using the [R] strat- egy and then the [N] strategy to train the Scalable- DSC model in stages. We can see from Table that
4
Experiment Results
4.1
Main Results
The lower part of Table shows the results when applying Scalable-DSS with different training con- figurations to correct the predicted dialogue states of three DST models(STP-DST,, SOM-DST(Kim et al., 2020) and STAR(Ye et al., 2021b)). As can be seen, Scalable-DSC correct the erroneous dia- logue states predicted by STP-DST and achieves state-of-the-art performance on MultiWOZ 2.0- 2.4 with joint goal accuracy of 67.91%. %,, 69.1399 13% 71.83% 74.92%, and 8..62%. Furthermore, the dialogue states containing erroneous slot values predicted by the SOM-DST and STAR models can also be corrected by Scalable-DSe Experimental

---

Model
Joint Goal Accuracy($5)
Slot Errori
MultiWOZ.
Model
Slot (cct(%)
81.17
Ol
Pt
El
-any
(9.91))
STP-DST
97.56
0.66
1.39 0.39
slot options
80.23(-0.94)
+Scalable-DSCIN
98.32
0.06
1.22 0.40
template options
79.20(-1.97)
2.0
+Scalable-DSC[R]
98.33
0.10
1.17 0.40
Scalable-DSCIR
81.40
+Scalabb-DSC[R--]
98.49
0.10
1.00 0.41
.any
80.49(-0.91)
+Scalable-DC[NN-R
98,36
0.10 1,14 0.40
slot options
81.17(-0.23)
STP-DST
97.58
1.26 0.38
template options
80.41(-D.99)
*ccalable-DSCNN
98,35
0.02 1.26 0.37
2.1
+Scalable-DSC([R]
98.44
0.02 1.17 0.37
Table 2: Ablation analysis of different prompts for STP on the MuluiWOZ2.4 test set.
+Scalable-DSCIR-NN
1.29 0.24
SccalabeeDSC[[N+R
98.54
0.04 1.04 0.38
STP-DST
97.94
0.67 0.87 0.52
Scalable-DSC[N]
98.68
0.01 0.82 0.49
the correction performance of training Scalable- DSC in stages, combining two strategies, exceeds the performance of training Scalable-DSC using single strategy only.
2.2
+Scalable-DS[[R]
98.67
0.02 0.81 0.50
Scabablce-DSCRR-N
98.69
0.01 0.81 0,49
+Scalable-SC([NeR
98.70
0.05 0.74 0.51
STP-DST
98.30
0.34 0.70 0.66
*Scalable-DSCNN
0.69 0.47
2.3
+Scalable-DS[[R]
98.83
0.01 0.67 0.49
DScalable-SC(RR-I
98,99
0.08 0.61 0.32
5 Analysis
Scalable-SCINNNR
98.84
0.64 0.49
STP-DST
98.85
0.27 0.63 0.25
SSaalable-DCCIN]
99.18
0.02 0.55 0.25
5.1 Ablation Analysis of STP Approach.
2.4
*Scalahle-DSCjR
99.25
0.01 0.50 0.24
Scalable-DSCIIR-V
99.20
0.03 0,52 0.25
+Scalable-DS([NHR
99.27
0.01 0.48 0.24
We conduct ablation studies to investigate into the impact of different prompts STP. Table shows the results on Scalable-DSC trained with different training strategies to correct erroneous dialogue states predicted by STP-DST. The results show that when the slot value "dontcare" is not replaced with "any", the joint goal accuracy of Scalable-DSC trained separately using two strategies is reduced by 1.26% and 0.91 % respectively after correction. We believe that the reason for the performance decline that the template options require connect- ing the Dontcare slot sequence fragment of each slot. This not only increases the input length of the model but also requires additional decoding of the dontcare sequence, adding pressure to the decod- ing process of model. The inputs of Scalable-DSC trained separately using two training strategies do not concatenate slot options. Their performance decreases by 0.94% and 0.23% respectively. This demonstrates that with the assistance of slot op- tions, the model is capable of gaining a better un- derstanding of the implied meanings of masked slot values within template options. Furthermore, when the model is trained without template options the prompt texts, the performance of the two training strategies decreased by 1.97% and 0.99% respectively. These results elucidate the effective- ness of template options in enabling the model to generate controlled corrective information, thereby enhancing its corrective performance.
Table 3: Slot Accuracy and Slot Error Rate on Multi
O refers to over prediction type, P refers to partial prediction type, and E refers erro- neous prediction type. t: higher is better and J: lower is better.
5.2 Effectiveness Analysis of Scalable-DSCC
Analysis Scalable-DSC is carried out obtain deeper insights into the reason for the effectiveness. We continue to employ STP-DST as the target cor- rection model, and utilize the Scalable-DSC model to correct its predicted dialogue states. Follow- ing (Quan and Xiong, 2020; Xie et al., 2022), we categorize the slot errors into three types: over prediction, partial prediction, and erroneous pre- diction. Table 3 presents the slot accuracy and the slot error rate of different model combinations on the MultiWOZ 2.0-2.4. As shown, the Scalable- DSC model improves the slot accuracy of STP- DST, demonstrating the benefits brought by correc- tion. It should be noted that the largest proportion of slot type errors in the STP-DST model is lack of predictions, followed by over predictions, and the smallest is updated predictions. After being cor- rected by Scalable-DSC, we can observe decrease in the proportions of all three types of slot errors in the STP-DST model. Specifically, we found that Scalable- -DC primarily corrects the error of over prediction in STP-DST. This verifies that the im-

---

STP-DST
Sodlabe-DSCIIN
Scalable-DSC(R)
Scalabic-SCIR--I
Dcalable-DS(NN+R]
MultiWOZ
PET
AEI
RE
CE1
AE.
RET
CE
AE.
RE
CE.
AE.
RET
CE
2.0
5578
153
1901
3830
243
2002
3819
250
2374
3454
199
2040
3737
2.1
5528
1803
3785
65
2001
3592
125
2090
3563
97
2261
3364
2.2
4698
1690
3044
42
1683
3057
30
1701
3027
123
1800
3021
2.3
3894
10
1190
2714
14
1203
2705
131
1315
2710
77
1211
2760
2,4
2542
70
766
1864
432
1319
1655
91
842
1791
422
1336
1628
Table 4: Error analysis of Scalable-DSS DSC on MuldWOZ 2.0-2.4. "PE": the number of incorrectly predicted slot
values by the DST model. "AE": the number of newly introduced error slots after heing corrected by the Scalable- DSC model. "RE": the reduced number of error slots after correction by Scalable-DSC. "CE": the number of error slots after Scalable-DSC model correction.
provements in the Scalable-DSC model primarily come from correcting over prediction errors. The relevant results of the SOM-DST model and the STAR model are shown in Appendix F. We have also observed, however, that for the other two types of errors (partial prediction and erroneous predic- tion), the Scalable-DSC model brings less benefit in terms of correction.
2013b) relied on pre-defined rules to recognize dialogue states, Although good prior knowledge can solve cold-start performance issues, it requires large amount of manual rule-making, is inflex- ible, lacks scalability, and cannot simultancously track multiple types of state information. Subse- quently, some classification DST models based on a predefined ontology were proposed (Henderson et al., 2014; Mrkšić et al., 2016: Zhong et al., 2018; Ye et al., 2021b), which significantly improved the model performance. However, it is difficult to design complete and robust pre-defined on- tology file (Xu and Hu, 2018). Therefore, most researchers focus on developing open-vocabulary DST models (Gao et al., 2019; Wu et al., 2019; Chen et al., 2020; Shin et al., 2022; Su et al., 2021; Zhao et al., 2022), which extract dialogue states di- rectly from the complete dialogue history, ignoring the issue of incomplete context information caused by truncation of dialogue history. To address this challenge, (Tian al., 2021; Zhao al., 2021: Zhou al., 2021; Xu al., 2023) attempts track users' goals from the context of the dialogue and the previous dialogue state. However, error prop- agation has become the reason for limiting model performance. To address this problem, the idea of correcting dialogue state is proposed (Tian et al., 2021; Xie et al., 2022), but the methods heavily rely specific DST models, i.e., lacking scalability.
5.3 Error Analysis of Scalable-DSC
The aforementioned experiments have shown the effectiveness of Scalable-DSC. Here we provide further analysis on the error correction quality to gain more insights into why it could be beneficial. We first investigate whether Scalable-DSC intro- duced new errors by correcting the dialogue states predicted by STP-DST (i.e., whether Scalable-DSC corrected correct slot values into incorrect ones). To achieve this, we calculate the number of slots correctly predicted by STP-DST but erroneously corrected by Scalable-DSC. The "AE" column in Table 4 displays this quantity. We can observe that Scalable-DSCC introduces new slot errors dur- ing the correction process, although the number is small. Next, we investigate whether the number of erroneous slot predictions from the DST model de- creased due the correction by the Scalable- DSC model. The results in the "RE" column of Table indicate that the Scalable-DSC model significantly reduces the number of incorrectly predicted slots by correcting erroneous slot values in the STP-DST model. It is worth noting that the number of newly introduced errors is proportional to the number of eπrors corrected by the model. In other words, the stronger the model's correction ability, the more new errors it will introduce.
Prompt learning is a widely used technique natural language processing (NLP), aiming at re- ducing the gap between the target before training and downstream tasks (Liu et al., 2023). Some recent prompt-based based DST work has been pro- posed(Lee and Jha, 2019; Gao et al., 2020; Lec et al., 2021). They use different text prompts to pro- vide task information to the model. For example, (Su et al., 2021) uses slot names for prompt infor- mation, (Rastogi et al., 2020) uses slot description,
6 Related Work
Early DST methods(Williams and Young, 2007; Thomson and Young, 2010; Wang and Lemon,
(Lin et al., 2021) prompt information is possible

---

values. Some other work (Jiang et al., 2020) has proposed the use of automatic prompls that do not require manually predefined prompts.
References
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gasic. 2018. Multiwoz-a large- scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026.
7 Conclusion and Future Work
We have proposed DSCla a new dialogue state correction model aiming at decoupling the error correction functionality from specific DST model, i.e., scalability, by introducing the STP strat- egy. The key idea of STP is to convert the dialogue state into a natural language sequence and gener- ates a corrected dialogue state natural language sequence based on the guidance of template op- tions, combined with real dialogue history infor- mation. Extensive experiments have been carried out analyze the scalability and correction perfor- mance of the model. Results have confirmed the applicability of the model other DST models and demonstrated a new state-of-the-ar performance on MultiW072.0-2.4 As discussed, limitation still exists. How to better correct all types of errors in dialogue states and avoid the generation of new errors remains interesting research topic for the future work.
Junfan Chen, Richong Zhang, Yongyi Mao, and Jie Xu. 2020. Parallel interactive networks for multi- domain dialogue state generation. arXiv preprint arXiv:2009.066/1
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani- Tur. 2020. Mul- tiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking base- lines. In Proceedings of the 12th Language Re- sources and Evaluation Conference, pages 422-428.
Shuyang Gao, Sanchit Agarwal, Tagyoung Chung, Di Jin, and Dilek Hakkani-Tur. 2020. From machine reading comprehension dialogue state tracking: Bridging the gap. arXiv preprint arXiv:2004.0580 0552777..
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagy- oung Chung, and Dilek Hakkani-Tur -urr 2019. Dialog state tracking: neural reading comprehension ap- proach. arXiv preprint
Limitations
Jinyu Guo, Kai Shuang, Jijie Li, and Zihan Wang. 2021.
This work has two main limitations:
Dual slot selector via local reliability verification for dialogue state tracking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 139-151.
(1) The Scalable-DSC cannot completely cor- rect all types of errors in predicting dialogue states. As mentioned above, the Scalable-DSC model pri- marily corrects over-prediction errors, while the correction proportion for the other two types of errors is relatively.
Ting Han, Ximing Liu, Ryuichi Takanabu, Yixin Lian, Chongxuan Huang, Dazhen Wan, Wei Peng, and Min- lie Huang. 2021. Multiwoz 2.3: A multi-domain taik-oriented dialogue dataset enhanced with anno- tation corrections and co-reference annotation. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 206-218. 218. Springer.
(2) The Scalable-DSC has a problem of incor- rectly modifying the correct conversation state. From the error analysis experiments, it can be observed that the Scalable-DSC model exhibits a small proportion of incorrect modifications in MultiWOZ2.0-2.4.
Michael Heck, Carel van Niekerk, Nurul Lubis, Chris- tian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. 2020. Trippy: A triple copy strategy for value independent neural dialog state tracking. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 35-44.
Acknowledgements
We would like to thank the anonymous review- ers for their useful feedback. This work was sup- ported by the Opening Project of Key Laboratory of Xinjiang, China (2020D04047), Natural Sci- ence Foundation of China (61663044), the National Key R&D Program of China (2020AAA0107902), and the Excellent Doctoral Student Research Innovation Project of Xinjiang University (No. XJU2023BS065).
Matthew Henderson. Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recur- rent neural networks. In Proceedings of the 15th annual meeting of the special interest group dis- course and dialogue (SIGDIAL), pages 292-299.
Ehsan Hosseini-Asl, Bryan McCann, Shien Wu, Semih Yavuz, and Richard Socher. 2020. A simple

---

language model for task-oriented dialogue, Advances in Neural Information Processing Systems, 33:20179- 20191.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 11:140:1-140:6
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig, 2020. How
know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 34, pages 8689-8696.
Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang- Woo Lee. 2020. Efficient dialogue state tracking by selectively overwriting memory. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 567-582.
Jamin Shin, Hangyeol Yu, Hyeongdon Moon, Andrea Madotto, and Juneyoung Park. 2022. Dialogue summaries as dialogue states (ds2), template-guided summarization for few-shot dialogue state tracking. arXiv preprint arXiv:2203.0
Hung Le, Richard Socher, and Steven CH Hoi. 2020. Non-autoreerressive dialog state tracking. arXiv preprint arXiv:2002.0800 080244.4..
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with a language model using schema-driven prompting. arXiv preprint arxiv:2109.07506.
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2021. Multi-task pre-training for plug- and play orientte dialogue system. arXiv preprint (idi:2200911733
Hwaran Lee. Jinsik Lee, and Tae- Yoon Kim. 2019, Sumbr: Slot-utterance matching for universal and scalable belief tracking. arXiv preprint arXiv:1907.0742.
Blaise Thomson and Steve Young. 2010. Bayesian update of dialogue state: A pomdp framework for spoken dialogue systems. Computer Speech & Lan- guage, 24(4):562-55
Sungjin Lee and Rahul Jha. 2019. Zero-shot adaptive transfer for conversational language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6642-6649.
Xin Tian, Liankai Huang, Yingzhan Lin, Siqi Bao, Huang He, Yunyi Yang, Hua Wu, Fan Wang. and Shuqi Sun. 2021. Amendable generation for dialogue state tracking. arXiv preprint arXiv: 10. 15659.
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto. Eunjoon Cho, and Rajen Subba. 2021. Leveraging slot descriptions for zero-shot cross-domain dialogue state tracking. arXiv preprint a:Xiv:2105.04222.
Yifan Wang, Jing Zhao, Junwei Bao, Chaoqun Duan, Youzheng Wu, and Xiaodong He. 2022. Luna: Learn- ing slot-turn alignment for dialogue state tracking. arXiv preprint arXiv:2205.025500
Zhuoran Wang and Oliver Lemon. 2013a. A simple and generic belief tracking mechanism for the dialog state tracking challenge: On the believability of observed information. In Proceedings of the SIGDIAL 2013 Conference, pages 423-432.
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020. Mintl: Minimalist trans- fer learning for task-oriented dialogue systems. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 3391 --405.
Zhuoran Wang and Oliver Lemon. 2013b. A simple and generic belief tracking mechanism for the dialog state tracking challenge: On the helievability of observed information. In Proceedings of the SIGDIAL 2013 Conference, pages 423 432.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Computing Surveys, 1-35..
Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Con- ference, pages 404 413, Metz, France. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proc. of ICLR.
Nikola Mrkšić, Diarmuid O Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2016. Neu- ral belief tracker: Data-driven dialogue state tracking. arXiv preprint 13X.vv6606.0777
Jason D Williams and Steve Young. 2007. Partially ob- servable markov decision processes for spoken dialog systems. Computer Speech Language, 21(2):393 422.
Jun Quan and Deyi Xiong. 2020. Modeling long con- text for task-oriented dialogue state generation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7119- 7 7124.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- Cistac, Tim Rault, Remi Louf, Morgan Funtowicz. Joe Davison, Sam Shleifer, Patrick von Platen, Clara

---

Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. Proc. of EMENP, pages 38-45.
Victor Zhong, Caiming Xiong, and Richard Socher, 2018. Global-locally self-attentive encoder for di- alogue state tracking. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1458- 1467.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini- Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state genera- tor for task-oriented dialogue systems, arXiv preprint arXiv:1905.08745.
Jingyao Zhou, Haipang Wu, Zehao Lin, Guodun Li, and Yin Zhang, 2021. Dialogue state tracking with multi-level fusion of predicted dialogue states and conversations. arXiv preprint arXiv:2107.05168
Hongyan Xie, Haoxiang Su, Shuangyong Song. Hao Huang, Bo Zou, Kun Deng, Jianghua Lin, Zhihui Zhang. and Xiaodong He. 2022. Correctable-dst Mitigating historical context mismatch between train- ing and inference for improved dialogue state track- ing. In Proceedings of the 2022 Conference Em- pirical Methods in Natural Language Processing, pages 876-889.
Jing Xu, Dandan Song, Chong Liu, Siu Cheung Hui, Li, Qiang Ju, Xiaonan He, and Jian Xie. 2023. Dialogue state distillation network with inter-slot con- trastive learning for dialogue state tracking.
Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. arXiv preprint arXiv:1805.013:
Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. 2021a. Multiwoz 2.4: multi-domain task -riietede dialogue dataset with essential annotation corrections to improve state tracking evaluation. arXiv preprint arXiv:2104.00773
Fanghua Ye, Jarana Manotumruksa, Qiang Zhang, Shenghui Li, and Emine Yilmaz. 2021b. Slot self- attentive dialogue state tracking. In Proceedings of the Web Conference 2021, pages 1598-1608.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. Multiwoz 2.2: A dialogue dataset with addi- tional annotation corrections and state tracking base- lines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117.
Yan Zeng and Jian-Yun Nie. 2020. Jointly optimiz- ing state operation prediction and value genera- tion for dialogue state tracking. arXiv preprint arXiv:2010.14061 14061.
Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu. Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022. Description- driven task-oriented dialog modeling, arXiv preprint arXiv:2201.08904,
Jeffrey Zhao, Mahdis Mahdieh, Ye Zhang, Yuan Cao, and Yonghui Wu. 2021. Effective sequence-to- sequence dialogue state tracking. arXiv preprint arXiv:2108.1300 13990.

---

A Details of the Predictive State Simulator Algorithm.
2.2, and 2.4, demonstrating the effectiveness of our method in mitigating the issue of error propagation in dialouee..
The details of simulating predicted dialogue states using the Predictive State Simulator algorithm are presented in Algorithm 1.
C
Slot Value Replacement in Slots
Algorithm 1: Predictive State Simulator
Slot
Replacement Value
Input Dialogue state label B, V;)
hotel-parking, hotel-intemet
has type
Noise probability 8;
sttraction-type, hotel-type, restaurant-food
any type
Create list of historical values for over prediction 7(V.:
restaurant-people hotel-people, hotel-stay, train-people
num
Create list of historical values for partial prediction and erroneous prediction 7/V for slot_ _idiiddj I..... J do
restaurant-lime, taxi-leave, taxi-arivve trsin-leave, train-arrive
any
V not nome then
HV.. appendd)))
restaurant-area, taxi-departure,
faxi-destination, traintdeparlure,
end end
any where
attraction-area, train-desiinn rain-desiination, hotel-area,
restaurant pricerange, holel-stars,
HVV.. +V.:: sppendeenen for slor_id I,.,, do Generating random number beta;
notel-pricerange restaurant-name. name, attracton-name
any
restaurant-day day, hatel-day, train-day
day
13
V3 is none then beta B then
14
15
Shuffle HV.:
Figure 4: Replace the "dontcare" in 30 slots with differ- ent "any" slot values.
Vi HV.[0]
17
end
18
else
Figure shows the replacement values for the "dontcare" slot values in different slots.
if beta B then
Shuftle HV..:
21
V [V,.(00]
22
end
D Experiment on Per Slot Error Rate
end
24
We further investigated the error rates for each slot. We used STP-DST as our primary models and used the Scalable-DSC [N+R] model to correct the pre- dictions of the primary models. The results on the MultiWOZ 2.4 test set shown in Table 6. It can be seen that the Scalable-DSC model effectively reduces the error rates for most slots. However, we found that slots associated with "name" and "type" have higher error rates the primary model itself. Although the error rates decrease after correction, these slots still present a challenge.
Output: Dialogue State Bnoise with Added Noise
B Final joint goal accuracy
MuldWOZ((S)
Model
2.0
2.1
2.2
2.3
2.4
SOM-DST"
39.34
36.04
38.64
38.44
55.00
MaatLe(aarLLarge)"
38.70
39.04
41.04
40.24
56.96
Transformer-DS
39.69
40.27
41.54
58.40
STAR"
38.93 40.94
48.14
66.46
Conretable-DSTT
60.36 57.86
61.66
58.26
74.17
STP-DST
39,20 40.30
41.10
52.30
66.00
+Scalable-DSCINI
53.20 54.20 59.70
61.30
71.70
E Experiment on the model's generation type.
+ Scalable- DSC[]]
53,70 55.90 58.70
61.34 75.10
+ Scalable-DSCIR-NI
56.30 56.90 58,90
62.70 72.60
+
54.10 58.32 59.10 61.23 75.50
We believe that when correcting dialogue state, identifying the wrong slot first before correction is more beneficial in reducing the inference burden of the model than direct correction. This idea has been validated shown in Table 7. On MultiWOZ 2.4, when correcting the predictions of STP-DST, using Scalable-DSCC only generate corrected dialogue state sequences achieved a 80.90% Joint Goal Ac- curacy (JGA), while using the method proposed
Table 5: Final joint goal accuracy(%) the MultiWOZ 2.0-2.4 test set. the results borrowed from Xie et al. (2022).
Table 5 presents the final joint goal accuracy results of our approach on MultiWOZ. 2.0-2.4. shown that our approach outperforms the previous best results of Correctable- on datasets 2.1,

---

STP-DST
-Scalabl-DDCINN-BB
taxi-departure
1.38
0.70
taxi laxi-destintioon
1.20
0.69
taxi-leave
0.42
0.37
taxi-urrive
0.55
0.32
resturant-namme
2.02
1.46
restaurant- anea
0.75
0.83
restaurant-pisee prsterant-priieraage
1.26
1.03
restaurant- Food
1.15
restaurant-day
0.24
0.21
Slot Error (%)
MulsWOZ
Model
Slot Acct (%)
restaurant-people
0.37
0.26
Overi lack, Update]
restaurant-tine
0.47
0.40
SOM-DST
97.27
0.99
1.22
0.52
attraction name
2.46
1.54
Scalable-DSCIN]
98.39
0.07
0.98
0.56
attraction-arca
1.13
1.13
Scalable DSCIR]
98.24
uttraction-type
1.83
1.73
Scalable-DSCIRNN
98.43
0.26
0.86
0.45
hotel-name
1.34
0.94
Scalabls-DSCIN-R
98.45
0.10
0.91
0.55
2.0
hotel-intereet
1.68
STAR
97.58
0.83
0.92
hotel-arca
1.43
Sculable DSC
98.38
0.37
0.99
0.26
botel-parking
L.28
1.12
Scalable DSC[R]
98.15
0.39
1.02
0.43
hotel-stars
1.00
0,86
Scalable- Dacilss-SSSRRRRRR [
98.49
hotel-type
3.73
2.12
Scalable-DSe [S--R))))RRRR
98. 48.48
0.24
0.98
0.30
hotel pricerrnie
1668
1.55
SOM-DST
97.37
0.90
1.14
0.59
hotel-day
0.17
0.01
Scalable-DSCIN]
98.10
0.25
1.09
0.56
hotel-poue people
0.37
0.08
Scalable-DSCIII
98.23
0.14
1.08
hotel-stay
0.17
98.33
0.05
1.02
0.59
train degurture
2.1
Scalable RSC[N-R]
98.32
0.19
0.94
0.51
truin-day
0.32
0,20
97.36
train-arrive
0.74
0.66
Scalable-DSCIN
98.72
0.11
0.81
0.35
dein-destinaaion
0.99
0.71
Scalable-DSCIR]
98 64
0:12
0.86
0,38
train leave
2.04
1.93
DSC[R-N]
train-poct people
1.39
0.28
Scalable DSC[N+R
98.75
0,02
0.84
0.39
SOM-DST DST
97.47
0.85
1.06
0.62
Table 6: The error rate of each slot on MultiWOZ 2.4.
Scalahle-DSC[N]
98.35
0.98
Scalahle-DSC(R)
98.29
0.15
0.96
0.60
Scalable Scalable-DSCTR-N
98.40
0.04
0.95
0.62
98.33
0.14
0.96
0.57
2.2
in this paper achieved an 81.62% JGA. The ex- perimental results show that our proposed method achieved a 0.72% improvement, proving that identi- fying the wrong slot first before correction is more beneficial in reducing the inference burden of the model than direct correction.
SIAR
97.93
0.69
0.98
Scalable DSC[N]
98.61
0.18
0.83
0.38
Scalahle DSC[R]
98.59
0.06
0.96
0.39
Scalable-DSCRR
98.74
0.03
0.83
Scalable DSCTN+R]
98.62
0.19
0.81
0.38
SOM-DST
97 64
0.61
0.83
0.93
Scalable DSCIN]
98.37
Scabable-DSCRI
98.38
0,01
0.84
0.77
Sealable-DSCCE DSCabbll-DCCRR-M
98.41
0.02
0.79
0.78
Scalable Sculable-DCIN--R DSC[N+R
98.38
Model
JGA(%) Generation Type
2.3
STAR
98.13
0.46
0.70
0.72
STP-DST
76.31
ScaIable-DS[IN]
98.67
0.01
0.74
0.58
Sealabbe-DSCIINHg
81.62
Ours
Scalable-DSC(R)
98.65
DScalable-DSC[N+R]
80,90
Generate corrected states
SSalable-DDCRR+. DSC[R+N]
98.48
0.25
0.64
0.63
Scalabls-DCCN--R
98.69
0.01
0.73
0.57
Table 7: Joint goal accuracy (%) ) on the test set of Multi- WOZ2.4.
SOM-DST
98.58
0.31
0.67
0.44
Scalable-USCINI
98.76
0.20
Scalable DSC[R]
98.86
0.11
0.61
0.43
Scalable DSC[R+N]
98.96
0.04
0.57
0.44
Scalable-DSQNN-K
98.90
0.04
0.63
2.4
STAR
98.96
0.20
0.62
0.21
F Experiment on Slot Error Rate
Scalable-DSC/N]
99.11
0.09
0.57
0.23
Scalable-DCIRI
99.10
We used SOM-DST and STAR as target correction models, and Table 8 presents the performance af- ter using the Scalable-DSC model to correct the predicted dialogue state of the two DST models on the MultiWOZ. 2.0-2.4 dataset. The results have demonstrated that the Scalable-DSC model primar- ily corrects the over prediction errors of the DST model, which has become the main reason for the strong correction capability of the Scalable-DSC model.
Scalable-DCCI-NN
99.10
0.11
0.57
0.22
Scalable DSC[N+R]
99.23
0.06
0.49
0.22
Table 8: Slot Accuracy and Slot Error Rate on
MultWWZ2..0-244 t: higher is better and 4: lower
better.


Reviews:
[{"review_id": "ZnbpcVKqwj", "reviewer": "Reviewer_kfrt", "summary": "", "questions": "- While the performance gains in this paper and Xie et al. 2022 is impressive, it is difficult for me to understand why DSC is a necessary step. Is there any explanation for why DST with an additional dialogue state correction step performs better than without it, even when both cases receive the full dialogue history? While I understand the issue with error propagation, it's difficult to understand why there will be an error that is correctible in a future step when it was predicted wrong in the previous step. Can it be explained by adding consistency through redundancy? If so, could we just generate multiple dialogue states and take a majority vote to improve consistency? Or is there something about the correction step that is easier than the initial prediction? Related to this, lines 40-46 is incorrect as there are still many recent works that use the entire dialogue history to predict each turn's dialogue state. Also, even Xie et al. 2022 provides the entire history and also the predicted dialogue state of the previous turn, so the information provided by the dialogue history and predicted dialogue state should, in theory, be repetitive.\n- Line 47: efficiency in terms of what? inference speed? If the full dialogue history is provided anyways along with the predicted dialogue state, which is the case for many of the baselines mentioned here, it cannot be speed.\n- What is the usefulness of lines 93-95 of generating erroneous states and then continuing to generate a sequence of corrected states? Is this the training data formulation? This seems the same as Xie et al., 2022.\n- What is meant by lines 106-108? How does STP-DST provide training data that is different from the strategies described above? I think the paragraph meant to say that the strategies are for getting incorrect predicted dialogue states, which is part of the input to the DSC model while STP-DST is a way to get data that is in the right format for the full input output of the training data for the DSC model. This is made clearer when reading the methodology and seeing Figure 2, but it's confusing when reading this part.\n- I wonder if this model can actually be used to correct wrong reference dialogue states in the datasets. What are your thoughts?\n- Does the proposed approach require using two models, one for the initial prediction and the correction? If so, this harms latency, ease of deployment, and adds overhead of maintaining two separate models.\n- Why do you separate [N -> R] and [R->N] and not do [R+N], as in training the simulator at the same time?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your informative and constructive review. The responses to the comments are listed as follows.\n\n### **Questions:**\n\n**Q1:**\n\n**(a) While the performance gains in this paper and Xie et al. 2022 is impressive, it is difficult for me to understand why DSC is a necessary step. Is there any explanation for why DST with an additional dialogue state correction step performs better than without it, even when both cases receive the full dialogue history? While I understand the issue with error propagation, it's difficult to understand why there will be an error that is correctible in a future step when it was predicted wrong in the previous step. Can it be explained by adding consistency through redundancy? If so, could we just generate multiple dialogue states and take a majority vote to improve consistency? Or is there something about the correction step that is easier than the initial prediction?**\n\n**(b) Related to this, lines 40-46 is incorrect as there are still many recent works that use the entire dialogue history to predict each turn's dialogue state. Also, even Xie et al. 2022 provides the entire history and also the predicted dialogue state of the previous turn, so the information provided by the dialogue history and predicted dialogue state should, in theory, be repetitive.**\n\n| Model            | Epoch | JGA(%) | Train Goal                |\n| ---------------- | :---: | :----: | :------------------------ |\n| STP-DST[N]       |  10   | 78.80  | Predicting and Correcting |\n| STP-DST          |   8   | 76.31  | Predicting                |\n| +Scalable-DSC[N] |   3   | 81.17  | Correcting                |\n\n**Table1.** Joint goal accuracy (%) on the test set of MultiWOZ2.4. [N]: The model is trained using predicted dialogue states simulated by a predictive state simulator as training data. （All parameter settings for this experiment are consistent with those mentioned in our paper. ）\n\n**Response1：**\n\n(a)Both the DST and Scalable-DSC models use complete dialogue history and dialogue state. However, their training objectives are different. DST's training objective is to track dialogue state based on the previous dialogue state and history, while Scalable-DSC's training objective is to rectify errors in the dialogue state based on the dialogue history, which empowers the Scalable-DSC model with correction capability. Furthermore, the Correctable Dialogue State Tracking (Correctable-DST) model's training objective is to predict the dialogue state of the current turn while correcting errors in the dialogue state from the previous turn. We attempted to utilize the approach of Correctable-DST, enabling the STP-DST model to simultaneously predict and correct dialogue states. The results are shown in Table 1. Under the same parameter settings, we found that Scalable-DSC achieves the best performance with just 3 epochs of training, while STP-DST requires only 8 epochs of training. Therefore, correcting errors is easier than making initial predictions, and making predictions alone is easier than simultaneously predicting and correcting dialogue states. The experimental results also indicate that the performance of DST with an additional dialogue state correction step is better than without it. These will be added in the improved manuscript along with some explanations and analysis.\n\nWhen errors occur in the previous dialogue state, it indicates a discrepancy between the information in the dialogue history and the information in the dialogue state. Therefore, the dialogue history can be used as evidence for detecting and correcting errors in the dialogue state. The approach of generating multiple dialogue states and conducting majority voting to enhance consistency is feasible, but this will significantly impact decoding speed[3].\n\n(b) During the training phase, these DST models take the actual previous dialogue state as input. At this stage, the information provided by the dialogue history and the predicted dialogue state is theoretically redundant. However, during the inference phase, the predicted previous dialogue state is used, which may introduce errors compared to the dialogue history. This is because we cannot guarantee the complete accuracy of the predicted dialogue state. Therefore, in the inference phase, the information from the dialogue history and the predicted dialogue state may not be entirely consistent.\n\nIn lines 40-46, the wording compromises readability. We appreciate your valuable feedback and will address this in the subsequent version of the paper. The revised version is as follows: \"Some DST methods choose the previous turn's dialogue state as part of the input to track the user's goal\".\n\n**Q2: Line 47: efficiency in terms of what? inference speed? If the full dialogue history is provided anyways along with the predicted dialogue state, which is the case for many of the baselines mentioned here, it cannot be speed.**\n\n**Response2:**\n\nWe are grateful to the reviewers for pointing out the errors in our paper. Our intended message was that using the dialogue state from the previous turn, in conjunction with the dialogue utterance from the current turn, leads to more efficient prediction of the current turn's dialogue state in the DST model, resulting in better performance[6]. We will carefully adjust this statement based on the context of the paper during the revision process.\n\n**Q3: What is the usefulness of lines 93-95 of generating erroneous states and then continuing to generate a sequence of corrected states? Is this the training data formulation? This seems the same as Xie et al., 2022.**\n\n| Model              | MultiWOZ 2.4 | Generation Type                                            |\n| ------------------ | :----------: | ---------------------------------------------------------- |\n| STP-DST            |    76.31     | -                                                          |\n| +Scalable-DSC[N+R] |    81.62     | generate erroneous states+generate corrected states (ours) |\n| +Scalable-DSC[N+R] |    80.90     | generate corrected states                                  |\n\n**Table 2.** Joint goal accuracy (%) on the test set of MultiWOZ2.4. \"[N+R]\": First using [N] strategy and then the [R] strategy to train the Scalable-DSC model in stages. （All parameter settings for this experiment are consistent with those mentioned in our paper. ）\n\n**Response3:**\n\nWe believe that when correcting dialogue state, identifying the wrong slot first before correction is more beneficial in reducing the inference burden of the model than direct correction. This idea has been validated as shown in Table 2. On the MultiWOZ 2.4 dataset, when correcting the predictions of the STP-DST model, using Scalable-DSC to only generate corrected dialogue state sequences achieved a 80.90% Joint Goal Accuracy (JGA), while using the method proposed in this paper achieved an 81.62% JGA. The experimental results show that our proposed method achieved a 0.72% improvement, proving that identifying the wrong slot first before correction is more beneficial in reducing the inference burden of the model than direct correction. Compared to Correctable-DST[2], our proposed method infers both the wrong slot and value, with a finer granularity. These will be added in the improved manuscript along with some explanation and analysis.\n\n**Q4: What is meant by lines 106-108? How does STP-DST provide training data that is different from the strategies described above? I think the paragraph meant to say that the strategies are for getting incorrect predicted dialogue states, which is part of the input to the DSC model while STP-DST is a way to get data that is in the right format for the full input output of the training data for the DSC model. This is made clearer when reading the methodology and seeing Figure 2, but it's confusing when reading this part.**\n\n**Response4:** The predicted state data provided by STP-DST in lines 106-108 serves as the training data required for Strategy 2. Your understanding of the meaning is accurate.\n\n**Q5: I wonder if this model can actually be used to correct wrong reference dialogue states in the datasets. What are your thoughts?**\n\n| Model    | MultiWOZ 2.4(%) |\n| -------- | :-------------: |\n| SOM-DST  |      66.78      |\n| SOM-DST* |      69.68      |\n\n**Table 3.** Joint goal accuracy (%) on the test set of MultiWOZ2.4. * refer to training the model by incorporating the pseudo-labels generated after correcting the training set using the Scalable-DSC  model along with the original training set. （All parameter settings for this experiment are consistent with those mentioned in our paper. ）\n\n**Response 5:**\n\nThank you very much for the reviewer's suggestions. They will be valuable for our future research. We have attempted to use Scalable-DSC to correct for erroneous dialogue states within the dataset, taking inspiration from the experimental setup of ASSIST[4]. They considered the validation set to be the cleanest approximation of noise-free data, and likewise we used the validation set to train the Scalable-DSC model. This model was then used to correct mislabelled data within the training set. The pseudo-labelled data generated by Scalable-DSC was then used to train the SOM-DST model. The experimental results are presented in Table 3. On the MultiWOZ 2.4 dataset, the Joint Goal Accuracy (JGA) improved to 69.68% compared to the baseline result of 66.78% achieved by training the SOM-DST model solely on the original training data, an improvement of 2.9%.  Although this performance falls short of the ASSIST performance, it demonstrates the preliminary potential of the Scalable-DSC model to correct erroneous mislabelled data within the dataset. All parameter settings of the current experiment are consistent with those of the paper. We find this a very intriguing concept and are actively engaged in further exploration.\n\n**Q6: Does the proposed approach require using two models, one for the initial prediction and the correction? If so, this harms latency, ease of deployment, and adds overhead of maintaining two separate models.**\n\n| model              | MultiWOZ 2.0  | MultiWOZ 2.1  | MultiWOZ 2.2  | MultiWOZ 2.3  | MultiWOZ 2.4 |\n| ------------------ | :-----------: | :-----------: | :-----------: | :-----------: | :----------: |\n| SOM-DST            |     51.38     |     52.57     |     53.27     |     55.50     |    66.78     |\n| +Scalable-DSC[N+R] | 68.61(+17.23) | 66.83(+14.26) | 67.14(+13.87) | 67.23(+11.73) | 74.37(+7.59) |\n| STAR               |     54.53     |     56.36     |     60.49     |     65.87     |    73.62     |\n| +Scalable-DSC[N+R] | 68.51(+13.98) | 69.02(+12.66) | 71.33(+10.84) | 72.50(+6.63)  | 78.91(+5.29) |\n| STP-DST            |     55.51     |     55.85     |     57.08     |     63.79     |    76.31     |\n| +Scalable-DSC[N+R] | 66.10(+10.59) | 69.13(+13.28) | 71.83(+14.75) | 74.46(+10.67) | 81.62(+5.31) |\n\n**Table4.**  Joint goal accuracy (%) on the test set of MultiWOZ2.4. \"[N+R]\": First using [N] strategy and then the [R] strategy to train the Scalable-DSC model in stages. (All parameter settings for this experiment are consistent with those mentioned in our paper. ）\n\n**Response6:**\n\nWe found it necessary to use two models: one for initial prediction and one for correction. This adds overhead, but the performance gains are significant, e.g. on the MultiWOZ 2.0 dataset Scalable-DSC[N+R]  improved the joint target accuracy of SOM-DST by 17.23%, as shown in Table 4.\n\n**Q7: Why do you separate [N -> R] and [R->N] and not do [R+N], as in training the simulator at the same time?**\n\n| Model               | MultiWOZ 2.4(%) |\n| ------------------- | :-------------: |\n| STP-DST             |      76.31      |\n| +Scalable-DSC[N->R] |      81.62      |\n| +Scalable-DSC[R->N] |      80.52      |\n| +Scalable-DSC[R+N]  |      80.10      |\n\n**Table5.** Joint goal accuracy (%) on the test set of MultiWOZ2.4. \"[N->R]\": First using [N] strategy and then the [R] strategy to train the Scalable-DSC model in stages. \"[R->N]\" : First using the [R] strategy and then the [N] strategy to train the Scalable-DSC model in stages. \"[R+N]\" : Mix the data generated by N strategy with that generated by R strategy for model training. (All parameter settings for this experiment are consistent with those mentioned in our paper. ）\n\n**Response7:**\n\nTraining the Scalable-DSC model separately using the predicted dialogue state from other DST models as training data can lead to a better adaptation of the Scalable-DSC model to the distribution of error types in the validation and test sets predicted by other DST models. The use of R+N leads to an inconsistent data distribution of error types between the training and test sets of the Scalable-DSC model. Therefore, we adopt a staged training approach. We conducted experiments to validate this idea, and the experimental results are shown in Table 5. It can be observed that when the data from the noise simulator is combined with the data predicted by the DST model to train the Scalable-DSC model, its correction performance is inferior to the phased training Scalable-DSC model. These will be added in the improved manuscript along with some explanation and analysis. And if the paper is accepted, we will make the code publicly available on GitHub for verification purposes.\n\n### **Reasons To Reject:**\n\n**Q1: While the DSC model itself is scalable in that it is agnostic to the DST model, the steps employed for creating the templates seem to require quite a bit of manual labor. Therefore, it is questionable who scalable this approach is to new datasets in that it will require manual labor for creating templates specific to each domain.**\n\n**Response1:**\n\nThe structural templates we constructed were generated using a heuristic converter that supports interconversion between dialogue states and natural language sequences, and the heuristic converter we used was extracted and refactored from the work of DS2 [5]. The Appendix A4 section in DS2 describes how to extend the template generation approach to new domain data, It explains that the developer only needs to rewrite the dialogue heuristic converter between states and summaries is sufficient, similarly extending our proposed method to new domain data requires only a slight modification of the code for the heuristic converter, which can be implemented by a Python developer in just a few hours, ensuring that our correction method is easily scalable to other datasets. If our paper is accepted, we will publish the code for the heuristic converter on github.\n\n**Q2：There are no limitations mentioned on practical aspects of using a DSC model, especially one proposed that requires two separate models: one for the initial prediction and one for the correction. This harms latency and ease of deployment, and adds additional overhead compared to a single model.**\n\n**Response2:**\n\nThank you very much for your suggestions. This helps prevent confusion for others reading the paper. We have addressed the relevant issues in Response 6 of the aforementioned Section Questions. We found it necessary to use two models: one for initial prediction and one for correction. This adds overhead, but the performance gains are significant, e.g. on the MultiWOZ 2.0 dataset Scalable-DSC[N+R]  improved the joint target accuracy of SOM-DST by 17.23%, as shown in Table 4.\n\n**Q3：Minor: the paper makes some incorrect claims in the introduction about the majority of DST approaches using the previous dialogue state as a compact representation of the dialogue history information when in fact most of the baselines mentioned, and even the proposed model, uses the full dialogue history \\*and\\* the previous dialogue state.**\n\n**Response3:**\n\nThank you very much for your suggestions. They will help us improve the quality of the manuscript. We have addressed the relevant issues in Response 1 of the aforementioned Section Questions. In lines 40-46, the wording compromises readability.  We will address this in the subsequent version of the paper. The revised version is as follows: \"Some DST methods choose the previous turn's dialogue state as part of the input to track the user's goal\".\n\n**We appreciate the comments from the reviewers that have contributed to improving the readability of the paper. We have considered all suggestions and corrected grammar errors. We will optimize the example of Figure 2 in the improved manuscript.**\n\n## **References**\n\n[1] Tian X, Huang L, Lin Y, et al. Amendable Generation for Dialogue State Tracking[C]//Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI. 2021: 80-92.\n\n[2] Xie H, Su H, Song S, et al. Correctable-DST: Mitigating Historical Context Mismatch between Training and Inference for Improved Dialogue State Tracking[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 876-889.\n\n[3]  Won S, Kwak H, Shin J, et al. BREAK: Breaking the Dialogue State Tracking Barrier with Beam Search and Re-ranking[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023: 2832-2846.\n\n[4] Ye F, Feng Y, Yilmaz E. ASSIST: Towards Label Noise-Robust Dialogue State Tracking[C]//Findings of the Association for Computational Linguistics: ACL 2022. 2022: 2719-2731.\n\n[5] Shin J, Yu H, Moon H, et al. Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking[C]//Findings of the Association for Computational Linguistics: ACL 2022. 2022: 3824-3846.\n\n[6] Liu, Hong et al. “Revisiting Markovian Generative Architectures for Efficient Task-Oriented Dialog Systems.” *ArXiv* abs/2204.06452 (2022): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors build on previous work on dialogue state correction by proposing a model that is agnostic to the DST model while still yielding significant performance improvements to the models that it is paired with.\n- The authors share a rigorous study with many baselines and analysis on the type of errors, clearly demonstrating the strengths and weaknesses of this approach.", "reasons_to_reject": "- While the DSC model itself is scalable in that it is agnostic to the DST model, the steps employed for creating the templates seem to require quite a bit of manual labor. Therefore, it is questionable who scalable this approach is to new datasets in that it will require manual labor for creating templates specific to each domain. \n- There are no limitations mentioned on practical aspects of using a DSC model, especially one proposed that requires two separate models: one for the initial prediction and one for the correction. This harms latency and ease of deployment, and adds additional overhead compared to a single model.\n- Minor: the paper makes some incorrect claims in the introduction about the majority of DST approaches using the previous dialogue state as a compact representation of the dialogue history information when in fact most of the baselines mentioned, and even the proposed model, uses the full dialogue history *and* the previous dialogue state.", "questions_to_authors": "- While the performance gains in this paper and Xie et al. 2022 is impressive, it is difficult for me to understand why DSC is a necessary step. Is there any explanation for why DST with an additional dialogue state correction step performs better than without it, even when both cases receive the full dialogue history? While I understand the issue with error propagation, it's difficult to understand why there will be an error that is correctible in a future step when it was predicted wrong in the previous step. Can it be explained by adding consistency through redundancy? If so, could we just generate multiple dialogue states and take a majority vote to improve consistency? Or is there something about the correction step that is easier than the initial prediction? Related to this, lines 40-46 is incorrect as there are still many recent works that use the entire dialogue history to predict each turn's dialogue state. Also, even Xie et al. 2022 provides the entire history and also the predicted dialogue state of the previous turn, so the information provided by the dialogue history and predicted dialogue state should, in theory, be repetitive.\n- Line 47: efficiency in terms of what? inference speed? If the full dialogue history is provided anyways along with the predicted dialogue state, which is the case for many of the baselines mentioned here, it cannot be speed.\n- What is the usefulness of lines 93-95 of generating erroneous states and then continuing to generate a sequence of corrected states? Is this the training data formulation? This seems the same as Xie et al., 2022.\n- What is meant by lines 106-108? How does STP-DST provide training data that is different from the strategies described above? I think the paragraph meant to say that the strategies are for getting incorrect predicted dialogue states, which is part of the input to the DSC model while STP-DST is a way to get data that is in the right format for the full input output of the training data for the DSC model. This is made clearer when reading the methodology and seeing Figure 2, but it's confusing when reading this part.\n- I wonder if this model can actually be used to correct wrong reference dialogue states in the datasets. What are your thoughts?\n- Does the proposed approach require using two models, one for the initial prediction and the correction? If so, this harms latency, ease of deployment, and adds overhead of maintaining two separate models.\n- Why do you separate [N -> R] and [R->N] and not do [R+N], as in training the simulator at the same time?", "paper_topic_and_main_contributions": "The authors present a scalable dialogue state correction model that is agnostic to the dialogue state tracking model and yields large performance improvements for all dialogue state tracking models that it is combined with. It shows strong improvements mostly for over-predictions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- remove Specifically in line 74\n- line 84: be -> is\n- line 98: during the training -> during training\n- line 102: generalization ability -> generalizability\n- The example in Figure 2 shows some area slots as east while some as west. I think the intended slot value was supposed to be west for all examples. It would be better to make it consistent as to reduce confusion. At least, the output of STP-DST should be east instead of west if it is to be consistent with the inference results.\n- The ethics statement is empty."}}, {"review_id": "5CKDlgnO6x", "reviewer": "Reviewer_5vDD", "summary": "", "questions": "- What is final joint goal accuracy? It seems no explanations about how to calculate it.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We feel great thanks for your professional review work on our article. As you are concerned, there are several problems that need to be addressed. The responses to the comments are listed as follows.\n\n### **Questions:**\n\n**Q1: What is final joint goal accuracy? It seems no explanations about how to calculate it.**\n\n**Response1:**\n\nThank you for pointing out the issues in the paper. It was indeed our oversight not to introduce and reference the concept of the final joint target accuracy. It is defined the proportion of examples (dialogues) where the predicted dialogue state of last turn exactly matches  the ground-truth dialogue state of last turn. This evaluation metric was proposed by Xie et al[1]. We will clarify this point in the revised manuscript.\n\n### **Reasons To Reject:**\n\n**Q1: I'd like to see what types of the errors are still challenging and I think this part is more important to have a deep understanding about this problem and the proposed approach. It is still not clear how the proposed approaches work to handle what types of errors, e.g., capturing ontologies like hotel names, or selecting options (day of week) from a limited set, or reference error.**\n\n**Response1:**\n\nIn our method, we have explained (1) using a predictive state simulator during the data processing stage to generate dialogue states containing three types of errors; (2) generating predicted dialogue states from other DST models, and during the training phase, enabling our model to learn how to assess and rectify errors in the dialogue states predicted by other models based on the dialogue history. The Scalable-DSC capturing the correct slot values from the dialogue history rather than selecting from a limited set.\n\n**Q2: The dataset is too single. The series of MultiWOZ have a very small set of domains (5 domains if the used subset is consistent to previous works) and it contain too many repeated dialogues. I think experiments on more datasets will be more helpful to appeal this work, e.g., Schema Guided Dialogue (SGD) dataset.**\n\n**Response2:**\n\nThe publicly available datasets for DST evaluation include DSTC2, Wizard-of-OZ, Schema-Guided Dialogue datasets and MultiWOZ 2.0-2.4. Thank you very much for your suggestions. We believe that our approach can also be applied to datasets other than MultiWOZ 2.0-2.4.  For a more comprehensive validation, we will release the code to replication or potentially supplementing experiments on GitHub in the near future.\n\n**Q3: Some details of experimental setup are missing. MultiWOZ 2.2 is a little different to other MultiWOZ series datasets. It add more additional slots. Please clarify if the used slots are same to other datasets or including these additional slots. This part is quite important but it is missing in the paper.**\n\n**Response3:**\n\nIn lines 306-310 of the paper, we mentioned the utilization of slot information from five domains (restaurant, hotel, attraction, taxi, train) in all versions of the MultiWOZ dataset. Actually, The slots for these five domains remain consistent across the MultiWOZ 2.2 dataset and other datasets.\n\n**Q4: Lacking analysis. Could you please provide some analysis about what type of errors are improved? At least quantitive analysis like error rate per domain/slot, case studies are helpful to have a better understanding.**\n\n| Slot                  | STP-DST | STP-DST + Scalable-DSC[N+R] | SOM-DST | SOM-DST + Scalable-DSC[N+R] | STAR | STAR + Scalable-DSC[N+R] |\n| --------------------- | :-----: | :-------------------------: | :-----: | :-------------------------: | :--: | :----------------------: |\n| taxi-departure        |  1.38   |            0.70             |  1.92   |            1.33             | 0.85 |           0.59           |\n| taxi-destination      |  1.20   |            0.69             |  1.80   |            1.26             | 1.01 |           0.82           |\n| taxi-leave            |  0.42   |            0.37             |  0.38   |            0.29             | 0.00 |           0.00           |\n| taxi-arrive           |  0.55   |            0.32             |  0.77   |            0.32             | 0.00 |           0.00           |\n| restaurant-name       |  2.02   |            1.46             |  3.33   |            2.83             | 1.77 |           1.46           |\n| restaurant-area       |  0.75   |            0.71             |  1.64   |            1.22             | 0.96 |           0.80           |\n| restaurant-pricerange |  1.26   |            1.03             |  1.60   |            1.37             | 1.34 |           1.01           |\n| restaurant-food       |  1.15   |            0.84             |  1.64   |            0.86             | 1.07 |           0.80           |\n| restaurant-day        |  0.24   |            0.21             |  0.35   |            0.40             | 0.33 |           0.28           |\n| restaurant-people     |  0.37   |            0.21             |  0.39   |            0.24             | 0.24 |           0.13           |\n| restaurant-time       |  0.47   |            0.40             |  1.00   |            1.01             | 0.57 |           0.58           |\n| attraction-name       |  2.46   |            1.54             |  3.52   |            2.64             | 2.21 |           1.71           |\n| attraction-area       |  1.13   |            1.13             |  1.41   |            1.37             | 1.79 |           1.79           |\n| attraction-type       |  1.83   |            1.73             |  1.68   |            1.39             | 1.60 |           1.56           |\n| hotel-name            |  1.34   |            0.94             |  3.51   |            2.23             | 1.47 |           0.93           |\n| hotel-internet        |  1.81   |            1.68             |  1.14   |            1.04             | 2.01 |           1.65           |\n| hotel-area            |  1.43   |            1.11             |  1.66   |            1.12             | 2.83 |           2.47           |\n| hotel-parking         |  1.28   |            1.12             |  1.27   |            0.86             | 2.22 |           1.99           |\n| hotel-stars           |  1.00   |            0.86             |  1.03   |            0.93             | 0.71 |           0.65           |\n| hotel-type            |  3.73   |            2.12             |  4.43   |            1.87             | 3.37 |           4.96           |\n| hotel-pricerange      |  1.68   |            1.55             |  1.79   |            1.56             | 1.58 |           1.42           |\n| hotel-day             |  0.17   |            0.01             |  0.33   |            0.33             | 0.10 |           0.06           |\n| hotel-people          |  0.37   |            0.08             |  0.42   |            0.44             | 0.20 |           0.20           |\n| hotel-stay            |  0.17   |            0.05             |  0.27   |            0.32             | 0.27 |           0.19           |\n| train-departure       |  0.63   |            0.36             |  1.11   |            0.70             | 0.66 |           0.12           |\n| train-day             |  0.32   |            0.20             |  0.86   |            0.70             | 0.25 |           0.23           |\n| train-arrive          |  0.74   |            0.66             |  1.09   |            1.01             | 0.0  |           0.00           |\n| train-destination     |  0.99   |            0.71             |  0.99   |            0.76             | 0.78 |           0.32           |\n| train-leave           |  2.04   |            1.93             |  2.23   |            2.14             | 0.0  |           0.0            |\n| train-people          |  1.39   |            0.28             |  1.07   |            0.67             | 0.73 |           0.42           |\n\n**Table1.** The error rate of each slot on MultiWOZ 2.4. (All parameter settings for this experiment are consistent with those mentioned in our paper. ）\n\n**Response4:**\n\nThank you very much for the reviewer's suggestions. They will be valuable for our future research. We further investigated the error rates for each slot. We used the STP-DST, SOM-DST and STAR models as our primary models and used the Scalable-DSC [N+R] model to correct the predictions of the primary models. The results on the MultiWOZ 2.4 test set are shown in Table 1. It can be seen that the Scalable-DSC model effectively reduces the error rates for most slots. However, we found that slots associated with 'name' and 'type' have higher error rates in the primary model itself. Although the error rates decrease after correction, these slots still present a challenge. These will be added in the improved manuscript along with some explanations and analysis.\n\n### **References**\n\n[1] Xie H, Su H, Song S, et al. Correctable-DST: Mitigating Historical Context Mismatch between Training and Inference for Improved Dialogue State Tracking[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 876-889.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation is clear. Error accumulation is an important topic to DST task.\n- The experimental results somehow show the effectiveness of the proposed approaches.", "reasons_to_reject": "- I'd like to see what types of the errors are still challenging and I think this part is more important to have a deep understanding about this problem and the proposed approach. It is still not clear how the proposed approaches work to handle what types of errors, e.g., capturing ontologies like hotel names, or selecting options (day of week) from a limited set, or reference error.\n- The dataset is too single. The series of MultiWOZ have a very small set of domains (5 domains if the used subset is consistent to previous works) and it contain too many repeated dialogues. I think experiments on more datasets will be more helpful to appeal this work, e.g., Schema Guided Dialogue (SGD) dataset.\n- Some details of experimental setup are missing. MultiWOZ 2.2 is a little different to other MultiWOZ series datasets. It add more additional slots. Please clarify if the used slots are same to other datasets or including these additional slots. This part is quite important but it is missing in the paper.\n- Lacking analysis. Could you please provide some analysis about what type of errors are improved? At least quantitive analysis like error rate per domain/slot, case studies are helpful to have a better understanding.", "questions_to_authors": "- What is final joint goal accuracy? It seems no explanations about how to calculate it.", "paper_topic_and_main_contributions": "This paper is to address the error accumulation problem in dialogue state tracking (DST) task. In this work,  a standalone dialogue state correction model is proposed to help several DST models to handle the errors in prediction. The experimental results show encouraging results.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1RsEHkkw6Y", "reviewer": "Reviewer_QXpz", "summary": "", "questions": "Why do you structure the Scalable-DSC output in the form of error + correction?  Is it not possible to train the model to simply output the corrected state description?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We feel great thanks for your professional review work on our article. As you are concerned, there are several problems that need to be addressed. The responses to the comments are listed as follows.\n\n### **Questions:**\n\n**Q1: Why do you structure the Scalable-DSC output in the form of error + correction? Is it not possible to train the model to simply output the corrected state description?**\n\n| Model              | MultiWOZ 2.4 | Generation Type                                     |\n| ------------------ | :----------: | --------------------------------------------------- |\n| STP-DST            |    76.31     | -                                                   |\n| +Scalable-DSC[N+R] |    81.62     | generate erroneous states+generate corrected states |\n| +Scalable-DSC[N+R] |    80.90     | generate corrected states                           |\n\n**Table 1.** Joint goal accuracy (%) on the test set of MultiWOZ2.4. \"[N+R]\": First using [N] strategy and then the [R] strategy to train the Scalable-DSC model in stages. (All parameter settings for this experiment are consistent with those mentioned in our paper. ）\n\n**Response1:**\n\nWe believe that when correcting dialogue state, identifying the wrong slot first before correction is more beneficial in reducing the inference burden of the model than direct correction. This idea has been validated as shown in Table 2. On the MultiWOZ 2.4 dataset, when correcting the predictions of the STP-DST model, using Scalable-DSC to only generate corrected dialogue state sequences achieved a 80.90% Joint Goal Accuracy (JGA), while using the method proposed in this paper achieved an 81.62% JGA. The experimental results show that our proposed method achieved a 0.72% improvement, proving that identifying the wrong slot first before correction is more beneficial in reducing the inference burden of the model than direct correction.  These will be added in the improved manuscript along with some explanation and analysis.\n\n### **Reasons To Reject:**\n\n**Q1:  The method requires slot options to be enumerated and templates written to describe DST outputs and corrections in natural language. The system is tested on just one data set so it is unproven as to the general usefulness of this approach. The overall results match the SOTA but do not substantially improve it.**\n\n| model              | MultiWOZ 2.0  | MultiWOZ 2.1  | MultiWOZ 2.2  | MultiWOZ 2.3  | MultiWOZ 2.4 |\n| ------------------ | :-----------: | :-----------: | :-----------: | :-----------: | :----------: |\n| SOM-DST            |     51.38     |     52.57     |     53.27     |     55.50     |    66.78     |\n| +Scalable-DSC[N+R] | 68.61(+17.23) | 66.83(+14.26) | 67.14(+13.87) | 67.23(+11.73) | 74.37(+7.59) |\n| STAR               |     54.53     |     56.36     |     60.49     |     65.87     |    73.62     |\n| +Scalable-DSC[N+R] | 68.51(+13.98) | 69.02(+12.66) | 71.33(+10.84) | 72.50(+6.63)  | 78.91(+5.29) |\n| STP-DST            |     55.51     |     55.85     |     57.08     |     63.79     |    76.31     |\n| +Scalable-DSC[N+R] | 66.10(+10.59) | 69.13(+13.28) | 71.83(+14.75) | 74.46(+10.67) | 81.62(+5.31) |\n\n**Table2.**  Joint goal accuracy (%) on the test set of MultiWOZ2.4. \"[N+R]\": First using [N] strategy and then the [R] strategy to train the Scalable-DSC model in stages. (All parameter settings for this experiment are consistent with those mentioned in our paper. ）\n\n**Response1:**\n\nThe publicly available datasets used for DST evaluation include DSTC2, Wizard-of-OZ, Schema-Guided Dialogue dataset, and MultiWOZ 2.0-2.4. We believe that this method can also be applied to other datasets beyond MultiWOZ 2.0-2.4.  The structural templates we constructed were generated using a heuristic converter that supports interconversion between dialogue states and natural language sequences, and the heuristic converter we used was extracted and refactored from the work of DS2 [1]. The Appendix A4 section in DS2 describes how to extend the template generation approach to new domain data, It explains that the developer only needs to rewrite the dialogue heuristic converter between states and summaries is sufficient, similarly extending our proposed method to new domain data requires only a slight modification of the code for the heuristic converter, which can be implemented by a Python developer in just a few hours, ensuring that our correction method is easily scalable to other datasets. For a more comprehensive validation, we will release the code to replication or potentially supplementing experiments on GitHub in the near future.\n\nWe believe that the main contribution of this paper is that the Scalable-DSC model can correct prediction errors made by other DST models. Additionally, the performance improvement achieved by the Scalable-DSC model surpasses that of the Correctable-DST model, as shown in Table 2. On the MultiWOZ 2.0-2.4 dataset, the performance enhancement brought by the Scalable-DSC model to three DST models ranges between 5.31% and 17.23%, demonstrating a significant improvement.\n\n### **Typos Grammar Style And Presentation Improvements:**\n\n**Q1: line 200 - is the reference to Fig 3 correct?**\n\n**Response1:**\n\nThank you for pointing out our writing error. It will help us improve the quality of the manuscript. This should be Figure 2, and we will correct this mistake.\n\nWe thank the reviewers for pointing out our problems and helping us to revise the manuscript. We have considered all suggestions and will correct errors in an improved manuscript.\n\n### **References**\n\n[1] Shin J, Yu H, Moon H, et al. Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking[C]//Findings of the Association for Computational Linguistics: ACL 2022. 2022: 3824-3846.\n\n[2] Xie H, Su H, Song S, et al. Correctable-DST: Mitigating Historical Context Mismatch between Training and Inference for Improved Dialogue State Tracking[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 876-889.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed system is shown to reduce slot error propagation and overall results in terms of joint accuracy match or exceed the SOTA.  Additionally, the method can be used with existing DSTs as a post-processor therefore making it more widely applicable.  It is tested using 3 different DSTs and it substantially improves each of them.\n", "reasons_to_reject": "The method requires slot options to be enumerated and templates written to describe DST outputs and corrections in natural language.  The system is tested on just one data set so it is unproven as to the general usefulness of this approach.  The overall results match the SOTA but do not substantially improve it.\n", "questions_to_authors": "Why do you structure the Scalable-DSC output in the form of error + correction?  Is it not possible to train the model to simply output the corrected state description?\n", "paper_topic_and_main_contributions": "State of the art methods for dialogue state tracking (DST) sequentially predict the state (ie a set of slots and their values) based on the dialogue history and the previous state.  One problem with this approach is that errors in state prediction tend to propagate forward compounding the error rate.  A recent paper by Xie et al (Correctable-DST) showed that incorporating an explicit slot error detector can reduce this problem and their results represent the current SOTA on the much used MultiWoz dataset.\n\nThis paper attempts to provide a similar slot error correction facility whilst treating the dialogue state tracker as a black box.  To do this they train a T5 based transformer to take as input the DST output and generate an output which explicitly identifies errors.  The input prompt is augmented with a set of manually crafted the slot options and templates describing the possible output forms.\n\nThe system is trained on MultiWoz augmented by induced slot errors generated by a either a so-called 'Predictive State Simulator' or by the DST itself.  The results are comparable or slightly better than the earlier Correctable-DST.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 200 - is the reference to Fig 3 correct?  \n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a scalable dialogue state error correction approach, and when paired with DST models (to which the DSC is agnostic), the system achieves state-of-the-art performance on the MultiWOZ datasets. The empirical studies presented in the paper is considered as rigorous, with room for improvement by expanding experiments towards non-MultiWOZ DST datasets. Also the practical limitations of employing a DSC component should be discussed more thoroughly in the paper.
Decision: Accept-Main