BREAK: Breaking the Dialogue State Tracking Barrier with Beam Search and Re-ranking
Year: 2023

Abstract:
Despite the recent advances in dialogue state tracking (DST), the joint goal accuracy (JGA) of the existing methods on MultiWOZ 2.1 still remains merely 60%. In our preliminary error analysis, we find that beam search produces a pool of candidates that is likely to include the correct dialogue state. Motivated by this observation, we introduce a novel framework, called BREAK (Beam search and RE-rAnKing), that achieves outstanding performance on DST. Our proposed method performs DST in two stages: (i) generating k-best dialogue state candidates with beam search and (ii) re-ranking the candidates to select the correct dialogue state. This simple yet powerful framework shows state-of-the-art performance on all versions of MultiWOZ and M2M datasets. Most notably, we push the joint goal accuracy to 80-90% on MultiWOZ 2.1-2.4, which is an improvement of 23.6%, 26.3%, 21.7%, and 10.8% over the previous best-performing models, respectively. The data and code will be available at https://github.com/tony-won/DST

heading: 1 Introduction
text: Dialogue state tracking (DST) is an essential component of task-oriented dialogue (TOD) systems to help users achieve their specific goals, such as booking restaurants or finding attractions (Budzianowski et al., 2018). The task of DST is to understand the meaning of user utterances and keep track of users’ intentions throughout the conversation. Since the results of DST affects the subsequent TOD tasks, i.e., dialogue policy and response generation, the accuracy of DST is crucial without a doubt (Kim et al., 2020; Lee et al., 2019). In DST, the dialogue state is typically represented by a set of (slot, value) pairs, e.g., (“hotel-area”, “centre”). Here, the list of slots is a pre-defined set, and the corresponding values are extracted from the dialogue context.
Thanks to large-scale pre-trained language models (PLMs) (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2020), generation-based approaches to DST have achieved remarkable progress in recent years (Hosseini-Asl et al., 2020; Feng et al., 2021; Lee et al., 2021b). Generationbased approaches sequentially generate values in the pre-defined sequence format, conditioned on the dialogue context. Most importantly, as they perform DST in an open-vocabulary setting rather than relying on a pre-defined ontology, this formulation has the potential to handle unseen values during training (Kim et al., 2020; Lee et al., 2021b). Due to this advantage, various techniques built on generative PLMs have been proposed to improve the performance of DST, but the joint goal accuracy on MultiWOZ 2.1 (Eric et al., 2020) still remains less than 60% 1.
1In general, performance is even worse when not using schema description, extra dialogue data, or large-scale models.
2832
To identify performance bottlenecks, we analyze the failure cases produced by generation-based DST models built upon PLMs (Radford et al., 2019; Raffel et al., 2020; Zhao et al., 2021). We find that most errors contain only one or two incorrect slot values. Furthermore, even at the decoding steps where the incorrect slot value has the highest output probability, the probability of the ground truth value still ranks very high, mostly in the top 4. The overall analysis motivates us to look into the beam search candidates rather than relying on decoding strategies that strictly select the sequence with the highest conditional probability. This is because beam search typically produces a set of candidates with high overlap (Meister et al., 2021), so it is useful in scenarios where only a few errors need to be corrected. Moreover, it allows tokens with a high output probability to be reconsidered as potential slot values.
Motivated by these observations, we propose a novel framework for generation-based DST, called BREAK (Beam search and RE-rAnKing). BREAK consists of two stages at the inference phase: (i) generating multiple dialogue state candidates using beam search and (ii) re-ranking the candidates to select the correct dialogue state. Unlike the existing methods that rely solely on the model’s generative power, our method effectively obtains the correct answer by re-examining the beam search candidates with a re-ranker. To the best of our knowledge, our work is the first to explore beam search and re-ranking in DST.
The contributions of our work are summarized as follows:
• Our analysis reveals that generation-based DST models still have a high output probability for ground truth values even when making wrong predictions, which provides a basis for re-considering beam search candidates rather than taking a single decoded sequence as the correct dialogue state.
• Motivated by our observation, we propose a simple yet powerful framework for generationbased DST that utilizes beam search and reranking.
• Our method achieves state-of-the-art performance by a significant margin on all versions of MultiWOZ and M2M datasets, breaking the existing performance barrier.

heading: 2 Preliminaries
text: In this section, we formally describe the problem and generation-based approach for DST. Then we report our in-depth analysis of the errors produced by generation-based DST models.

heading: 2.1 Problem Statement
text: We treat the DST task as a sequence-to-sequence problem, where the model processes the input sequence of utterances and generates a dialogue state tracked up to the current turn. More formally, let the input Ct = [(U1,M1), ..., (Ut,Mt)] be a sequence of utterances up to turn t, where each U and M represent the user utterance and system response, respectively. Given the dialogue context Ct, the model outputs a dialogue state Yt = {(sn, vn)|sn ∈ S}. Here, S = {s1, ..., sN} denotes the set of pre-defined slots that comprise N domain-slot pairs, and vn is the slot-specific value for slot sn. To sum up, we aim to learn a dialogue state tracker F : Ct 7→ Yt that takes the dialogue context Ct as input and keeps track of the dialogue states Yt accurately throughout the dialogue.

heading: 2.2 Generation-based Model for DST
text: In this work, we are particularly interested in generation-based models built upon Transformers (Vaswani et al., 2017). Our method can be applied to either encoder-decoder (Raffel et al., 2020; Lewis et al., 2020) or decoder-only (Radford et al., 2019) models, yet we formally describe our method with the encoder-decoder structure.
The input of the model consists of all turns of dialogue up to turn t. All sequences are concatenated with [USER] and [SYS], where [USER] and [SYS] are special tokens for indicating the speaker of each utterance.
Ct = [USER]⊕ U1 ⊕ [SYS]⊕M1⊕ · · · ⊕ [SYS]⊕Mt−1 ⊕ [USER]⊕ Ut. (1)
Given the dialogue context, the encoder maps an input sequence Ct to a sequence of continuous representations H(l)t as follows:
H(0)t = Emb(Ct), (2)
H(l)t = Encl(H (l−1) t ), (3)
where Emb(·) and Encl(·) represent the initial embedding layer and the l-th layer of the encoder, respectively.
The decoder then generates a dialogue state token-by-token in a pre-defined sequence format. In other words, it sequentially predicts the probability of the current token conditioned on the encoder output embeddings H(L)t and all the previously generated tokens. Here, L denotes the number of layers of the encoder. The output probability of the decoder at any decoding step j is given as:
Pθ(yj |y<j , Ct) = Dec(y<j ,H(L)t ), (4)
where θ represents the parameters of the encoderdecoder model.
The training objective of the auto-regressive process is to maximize the log-likelihood of the target sequence Yt = ⟨y1, y2, ...⟩ for the given input text Ct as follows:
L = − |Yt|∑
j=1
logPθ(yj |y<j , Ct). (5)
During inference, greedy search, which selects the token with the highest probability at each time step, is generally applied to produce the output sequence.

heading: 2.3 Preliminary Study on DST
text: To identify performance bottlenecks in generationbased DST, we analyze the failure cases predicted with T5 (Raffel et al., 2020) using greedy search 2. The error analysis for other models are provided in the Appendix A.
First, we investigate how many slot values are incorrectly predicted in each instance of MultiWOZ 2.4 (Ye et al., 2022b). Our experiment shows that 91.6% of the wrong predictions contain only one or two incorrect slot values, as shown in Figure 2-(a), which indicates that only a few slot-level errors contribute to the low JGA. This result is consistent with the fact that most of the existing DST models exhibit very high slot accuracy3 (97~99%) while having low JGA (Wu et al., 2019; Kim et al., 2020; Wang et al., 2022; Ye et al., 2022a,c).
To further examine the errors, we explore the output probability distribution over the vocabulary at decoding steps where slot values are incorrectly predicted. Specifically, we check the ranking of the probability of the ground truth value when sorted in descending order. To illustrate with an example, suppose that the predicted value is 13:15 and the ground-truth value is 13:45. The mis-predicted word is 15, and therefore we check the ranking of the correct word 45 at 15’s decoding step. As a result, we find that the probability of decoding the ground truth value generally ranks very high. As shown in Figure 2-(b), around 92% of the wrong predictions have ground truth values within the 4th place.
All of our findings naturally lead to the use of beam search. First, beam search can be useful in
2We fine-tune T5-small on MultiWOZ 2.4 and set the output format as clozse-style described in Section 4.1.
3Slot accuracy individually compares the predicted value of each slot to its ground-truth value at each turn.
scenarios where only one or two errors need to be corrected, as they generate a set of sequences with high overlap (Meister et al., 2021). More importantly, beam search candidates are likely to contain the high-ranking tokens investigated in our analysis. In fact, generated candidates exhibit only a few unique values for each slot and have a small number of slot-level errors, as reported in Table 1. These observations suggest that the k-best dialogue states generated by beam search can serve as a valuable candidate pool by combining highly probable slot values. This presents an opportunity to reconsider them as potential dialogue states.

heading: 3 BREAK: Beam Search and Re-Ranking
text: Based on the analysis in Section 2.3, we propose a novel framework for generation-based DST. Our approach, dubbed BREAK, utilizes Beam Search and RE-rAnKing at the inference phase. Specifically, given a trained DST model, the main idea is to generate dialogue state candidates using beam search and then find the correct dialogue state by re-ranking them.

heading: 3.1 Generating Candidates with Beam Search
text: The decoding process of dialogue state generation can be viewed as a problem of finding the optimal sequence Y ∗ = argmaxY log p(Y |X) given the input X . The current practice in generationbased DST is to use greedy search, the simplest
heuristic of finding Y ∗. However, as described in Section 2.3, greedy search often fails to generate the accurate slot values since it simply selects only one token with the highest conditional probabilities p(yj |y<j , X) at each decoder step j.
Instead of considering only the one best token, beam search keeps track of k most probable subsequences, allowing the exploration over a wider search space. Therefore, we adopt beam search to create valid candidates for dialogue states. The rationale behind using beam search is based on our analysis that the output probability of ground truth value is very high among all tokens. In the following sections, we denote the beam search candidates as Y .

heading: 3.2 Re-Ranking over Candidates
text: After generating candidates with beam search, we need to select the correct dialogue state among them. To this end, a re-ranker learns to rank candidates by computing the semantic alignment between the given dialogue context Ct and each candidate Y ′t ∈ Y .
For a re-ranker, we use a model with BERTbased architecture. The input sequence is the concatenation of the dialogue context and the dialogue state candidate, Ct⊕Y ′t . Then we take the final hidden state vector of the [CLS] token as the aggregate representation for input pair (Ct, Y ′t ). A simple softmax classifier is added on top of the aggre-
gate representation, which we denote by h(Ct, Y ′t ), to compute the probability of each label c ∈ {0, 1} as follows:
p(c|h(Ct, Y ′t )) = softmax(Wh(Ct, Y ′t )), (6)
where W is the weight matrix for the classification layer.
We train a re-ranker by minimizing crossentropy loss to achieve the goal of scoring the correct candidate higher than other candidates. To this end, we contruct a dataset consisting of the dialogue context (Ct), a pool of dialogue state candidates (Y), and the label indicating whether each input pair (Ct, Y ′t ∈ Y) is correct or not. A finetuned dialogue state tracker 4 is employed to construct this data. Using this model, we make inference on the DST training set with beam search to produce Y for each Ct. Then the ground truth is labeled as a positive sample, and all the wrong predictions are labeled as negative samples. The same process is applied to the validation set.
At test time, the candidate with the largest score, which is the probability of being the correct answer (c = 1), is selected as the correct dialogue state as follows:
Ŷt = argmax Y ′t ∈Y
p(c = 1|h(Ct, Y ′t )). (7)

heading: 4 Experimental Setup

heading: 4.1 Model Variations
text: Depending on the form of the output dialogue state YT , we consider three variants of the model:
(i) Sequential w/o none (SEQ): The decoder sequentially generates a set of slot-value pairs except when the value is none. The output sequence Yt has the following format: si = vi, sj = vj , · · · , where vi and vj are not none.
(ii) Sequential w/ none (SEQ-Full): In contrast to SEQ, the output sequence Yt includes none slot values. In other words, the decoder sequentially generates slot values for all pre-defined slots, with the format of s1 = v1, s2 = v2, · · · , sN = vN .
(iii) Cloze-Style (CS): In this case, we formalize the DST problem as the equivalent cloze-style QA task. Specifically, we design a task-specific prompt
4We use the model weights with the best validation performance when evaluated with greedy decoding.
P as a cloze question, which has the following format:
P = s1 ⊕ [SLOT_1]⊕ s2 ⊕ [SLOT_2] ⊕ · · · ⊕ sN ⊕ [SLOT_N] (8)
where sn indicates the slot name (e.g., train -day), and [SLOT_n] is a special token for a placeholder that fills in the corresponding slot value. The task-specific prompt P is concatenated with the dialogue context Ct:
Xt = P ⊕ Ct. (9)
Given this prompt-augmented input Xt, the model outputs the sequence Yt, which represents a cumulative dialogue state up to the current turn.
Yt = [SLOT_1]⊕ v1 ⊕ [SLOT_2]⊕ v2⊕ ...⊕ [SLOT_N]⊕ vN
(10)
where vk is the corresponding slot values for the specific slot [SLOT_k].

heading: 4.2 Datasets
text: MultiWOZ is the most extensively used benchmark for DST. It is a large-scale multi-domain dialogue dataset that contains about 10k multi-turn dialogues spanning over 8 domains. We conduct our experiments on MultiWOZ 2.1-2.4 (Eric et al., 2020; Zang et al., 2020; Han et al., 2021; Ye et al., 2022b), the improved versions made by continuously refining annotation errors from MultiWOZ 2.0 (Budzianowski et al., 2018). Following the previous works (Wu et al., 2019; Kim et al., 2020), we use only 5 domains {attraction, hotel, restaurant, taxi, train} with 30 domain-slot pairs, excluding {bus, hospital, police}.
Machines Talking To Machines (M2M) (Shah et al., 2018) is the simulation-based dataset that contains 3k dialogues from the restaurant (SimM) and movie (Sim-R) domains. To collect the conversations, the outlines of the dialogue are first generated using self-play between the user and system agencies. Then, the generated outlines are paraphrased by crowd workers to get more diverse utterances.

heading: 4.3 Evaluation Metric
text: Joint goal accuracy (JGA) is a widely used metric to evaluate the performance of DST models. By definition, JGA is True if and only if all predicted values for all slots exactly match the ground-truth labels, otherwise False.

heading: 4.4 Upper Bound of BREAK
text: Since BREAK eventually selects one of the beam search candidates as the correct answer, we also present the upper bound of JGA for the dialogue state tracker f . The upper bound fupp is calculated as follows:
fupper = M∑
i=1
1{Y (i) ∈ Y(i)f }/M, (11)
where M denotes the total number of samples in the test set. The ground truth and beam search candidates of the ith sample are represented as Y (i) and Y(i)f , respectively.

heading: 4.5 Implementation Details
text: For a fair comparison, we use the pre-processing script released by (Wu et al., 2019).

heading: 4.5.1 Training
text: Dialogue State Tracker. For our experiments, we employ T5-small (Raffel et al., 2020) and GPT2 (Radford et al., 2019) as a backbone using HuggingFace Transformers5. All the weights are initialized from the pre-trained checkpoint and then models are fine-tuned on MultiWOZ and M2M datasets. The detailed specification is as follows: (i) T5-small has 60M parameters containing 6 transformer blocks for both encoder and decoder, 8 attention heads, and 512 hidden units. (ii) GPT2 has 117M parameters containing 12 transformer blocks, 12 attention heads, and 768 hidden units. Both T5 and GPT2 are trained using AdamW (Loshchilov and Hutter, 2017) with a constant learning rate of 5e-5. Exceptionally, we use a learning rate of 1e-4 to train T5 on MultiWOZ datasets. During training, we set a batch size to 16 and a dropout rate to 0.1. The maximum sequence length of the encoder is set to the default value but set to 100 longer when using the cloze-style format.
Re-Ranker. We use the pre-trained RoBERTabase (Liu et al., 2019) for a re-ranker. RoBERTa-
5github.com/huggingface/transformers
base is built upon the BERT-based architecture with 12 transformer blocks, 12 attention heads, and 768 hidden units. The model is trained using AdamW (Loshchilov and Hutter, 2017) with a constant learning rate of 1e-5. During training, we set a batch size to 48 and a dropout rate to 0.1. The maximum sequence length is 512.

heading: 4.5.2 Inference
text: We run each evaluation three times with different seeds and report the average number for more reliable results.

heading: 5 Experimental Results
text: Unless otherwise noted, all T5-based results are obtained using the form of the cloze-style (CS). This is due to the computational efficiency, and more details are described in Section 5.4.

heading: 5.1 Overall Results
text: We present the evaluation results on MultiWOZ 2.1-2.4 in Table 2. In our experiments, we compare our method with the strong baselines: STAR (Ye et al., 2021), LUNA (Wang et al., 2022), MetaASSIST (STAR) (Ye et al., 2022c), SOM-DST (Kim et al., 2020), TripPy (Heck et al., 2020), SimpleTOD (Hosseini-Asl et al., 2020), Seq2SeqDU (Feng et al., 2021), SDP (Lee et al., 2021b), D3ST (XXL) (Zhao et al., 2022), ConvBERTDG + Multi (Mehri et al., 2020), and TripPy + SCORE (Yu et al., 2020).
To validate the efficacy of our method, we first measure the upper bound of JGA described in Section 4.4. With a beam size of 50, both T5 and GPT2 show nearly 90% upper bound JGA, particularly around 94-95% on MultiWOZ 2.4. These results demonstrate that k-best candidates produced by beam search are likely to contain the correct dialogue state that greedy search could not predict.
Combined with re-ranking, BREAK consistently outperforms the existing methods by significant margins on all versions of MultiWOZ dataset. Most remarkably, our method achieves 23.6%, 26.3%, 21.7%, and 10.8% absolute performance improvement on MultiWOZ 2.1-2.4, respectively. In consequence, we push the boundaries of the performance on MultiWOZ to 80-90%. Note that we obtain these results without using extra training data or increasing the model size.
Table 3 shows the evaluation results on M2M. BREAK achieves state-of-the-art performance on all three evaluated datasets. Notably, on Sim-R,
our method shows better performance than SMDDST which has a kind of oracle upper bound. A significant challenge faced by M2M appears to be the model’s ability to generalize in slots with high out-of-vocabulary rates 6. T5 exhibits relatively lower accuracy in those slots, whereas BREAK-T5 demonstrates comparable performance to the other slots 7.

heading: 5.2 Effect of the Beam Size
text: Figure 4 shows the performance of our method on MultiWOZ 2.1 and Sim-M with varying sizes of the beam search candidates. A larger beam size naturally leads to elevating the upper bound JGA of T5 since it can cover lower-ranking ground truth values. In our preliminary error analysis, most of the ground truth values are found to have very high-ranking output probabilities among the vocabulary. This finding is strongly supported by the dramatic increase in T5upper when the beam size increases from 1 to 2. Moreover, the performance of BREAK-T5 shows a similar trend to T5upper, indicating that a re-ranker finds the correct dialogue state well from the candidates with high overlap. However, a large beam size (>10) rather causes performance degradation on Sim-M. Since there are only five slots in Sim-M, a large number of similar candidates can act as noise to a re-ranker.
6For example, the ‘movie’ slot of Sim-M and the ‘restaurant-name’ slot of Sim-R have out-of-vocabulary rates of 100% and 39.1%, respectively.
7‘movie’: 91.50% → 95.95%, ‘restaurant-name’: 93.77% → 96.30%

heading: 5.3 Per-Turn Joint Goal Accuracy
text: In Figure 5, we compare the per-turn accuracy of our method with STAR and MetaASSIST (STAR) on MultiWOZ 2.1 and MultiWOZ 2.4. We also report the results of STAR-GT and MetaASSISTGT, which use the ground truth dialogue state of the previous turn as the input at every turn.
In general, the per-turn accuracy drastically decreases as the number of turns increases. This is because DST on longer dialogue contexts is more challenging, and JGA accumulates errors from the early turn until the end. Nevertheless, BREAK-T5 shows relatively stable performance regardless of the turn lengths. It even performs better than STARGT and MetaASSIST-GT for most turn lengths.
For one-turn dialogues, however, the performance is comparable to or even worse than the baseline T5. Since similar candidates are compared for such a short dialogue context, it is difficult for a re-ranker to distinguish the correct one. For longerturn dialogues, BREAK-T5 absolutely outperforms other baselines, whereas the performance of T5 and STAR is severely degraded.

heading: 5.4 Effect of the Dialogue State Form
text: Table 4 and Table 5 shows the performance and latency of our method for three different variations of the output sequence format. We measure the inference time per instance on RTX A5000 with a batch size of 1. In our experiments, GPT2/SEQ-Full 8 and T5/CS perform best overall. While GPT2/SEQFull exhibits comparable performance to T5/CS, it takes about 2.8 times longer inference time 9. Since beam search is computationally expensive, we mainly report the results of T5/CS in this paper for time efficiency. The SEQ format is faster than other formats due to its short output sequence length, but its performance is relatively poor. This suggests that it is advantageous for BREAK to express the output sequence with a fixed template containing the entire slot list. In conclusion, our proposed cloze-style (CS) format is the most efficient for our method in terms of both performance and computation.

heading: 6 Related Work

heading: 6.1 Generation-based DST
text: Recently, there have been promising results on the MultiWOZ datasets using generation-based ap-
8GPT2 is known to be sensitive to additional special tokens. For this reason, we do not consider GPT2/CS.
9This comes from the replacement of the slot name with one special token, e.g., taxi-leaveat→ [SLOT_0].
proaches. These models basically leverage the powerful generative capabilities of large-scale PLMs. On top of that, various techniques have been proposed to further improve the performance of DST: using schema descriptions (Feng et al., 2021; Lee et al., 2021b; Zhao et al., 2022), pre-training with multiple dialogue corpora or novel objectives (Peng et al., 2021; Su et al., 2022; Zhao et al., 2021), multi-task learning on different taskoriented tasks (Lin et al., 2020; Hosseini-Asl et al., 2020; Peng et al., 2021; Su et al., 2022), or increasing the size of PLMs (Zhao et al., 2022). On the other hand, our work does not require external dialogue data or additional information for the task.

heading: 6.2 Beam Search and Re-Ranking
text: Many recent studies in neural machine translation (NMT) and natural language generation (NLG), have proposed re-ranking over multiple candidates. These candidates are traditionally generated from a conditional language model with beam search decoding. This approach is particularly beneficial for auto-regressive models because the re-ranking model evaluates the candidate by attending over the entire sequence, which cannot be done in the decoding process. In NMT, re-ranker models are generally trained with the final evaluation metrics like BLEU (Lee et al., 2021a). In NLG, re-rankers are trained to realize all the attributes in the structured meaning representation (Dušek and Jurčíček, 2016; Juraska et al., 2018). However, stochastic decoding is also preferred over beam search to ensure diversity in the natural sentences (Kedzie and McKeown, 2019; Eikema and Aziz, 2020; Bhattacharyya et al., 2021; Fernandes et al., 2022). In contrast, DST aims to predict the accurate dialogue state, making the use of beam search even more appropriate.

heading: 7 Conclusion
text: We propose a simple yet effective framework for generation-based DST that breaks the performance barrier in DST. We design our framework based on our findings that the probability of ground truth value being generated by DST models is very high in most decoding steps. Our method effectively tracks the dialogue state by (i) generating beam search candidates and (ii) re-ranking them via assessing the semantic matching with the dialogue context. By exploring the highly probable dialogue state candidates discovered by beam search, our
method significantly reduces errors compared to the decoding process that generates a single definitive dialogue state. In our experiments, we achieve state-of-the-art performance on MultiWOZ and M2M datasets by a significant margin, regardless of the backbone PLMs. For future work, we plan to improve the computational efficiency of the current framework to apply in real-world settings.
Limitations
Our method shows impressive performance but relies entirely on beam search during inference. However, it is well known that beam search is a computationally expensive algorithm. With the beam size of 50, the latency increases from 3.6 times (T5/SEQ-FULL) to 7 times (T5/SEQ) compared to greedy decoding. In addition, the re-ranking process causes another latency (about 12ms in our experiments). Therefore, it may not be suitable for real-world DST scenarios. We leave this issue for future work. Potential directions may include reducing the current two-step pipeline to an efficient one-step process by employing a novel objective function, using data augmentation, or changing the sequential decoding process to a nonautoregressive approach that can be applied in a parallel manner.
Ethics Statement
All datasets and models used in the experiments are from the publicly available website or Github.

heading: Acknowledgements
text: This work was supported by LG AI Research. This work was partly supported by Institute of Information communications Technology Planning Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics]. This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (No. 2021R1A2C2008855).

heading: A Error Analysis of DST models
text: In addition to T5 (Raffel et al., 2020), we conduct error analysis for GPT2 (Radford et al., 2019) and STAR (Ye et al., 2021). T5 and GPT2 are the most commonly used backbone models for generationbased DST, which generate slot values sequentially. On the other hand, STAR performs pre-defined ontology-based DST by computing the distance between the dialogue context and each slot value.
Regarding the slot-level errors, all three models show similar tendencies. The majority of incorrect predictions (>90%) result from one or two slotlevel errors, as shown in Figure 6-(a). However, when it comes to the output probability, T5 and GPT2 follow similar patterns, while STAR shows distinct behavior.
As shown in Figure 6-(b), at the decoding steps where incorrect slot values are generated, we observe that STAR has a relatively low-ranking output probability for ground truth values. While T5 and GPT2 have a ground truth value in the top-4 in over 90% of cases, STAR has only about half of the cases in the top-6. Consequently, STAR is less likely to contain the correct answer among the beam search candidates, making it difficult to benefit from our proposed method. These results appear to be related to the characteristics of STAR, as highlighted in Table 6, where STAR tends to produce over-confident errors.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Yes, we provide the limitations of our work in Section 7 (conclusion) and Limitation Section.
7 A2. Did you discuss any potential risks of your work? There seem to be no potential risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Yes, we summarize our claims in the Abstract and Introduction sections.

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: We used Grammarly to correct some grammatical errors.
B 3 Did you use or create scientific artifacts? We used MultiWOZ and M2M datasets for our experiments. And to build our model, we use the HuggingFace Transformer library.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: For our used dataset and pre-trained language models, we cite the paper.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We specified the purpose for which the data and models are used.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We used only publicly widely used datasets
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, we described the datasets we used in Section 4.
C 3 Did you run computational experiments?

heading: C. Yes. Section 4. Experimental setup and Section 5. Experimental results.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Yes. We report all the details about the experimental setup in section 4 and the latency in section 5.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? In section 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In section 5. We run each evaluation three times with different seeds and report the average number for more reliable results.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We specified that we used the pre-trained language models using the Huggingface library in section 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For future work, we plan to improve the computational efficiency of the current framework to apply in real-world settings. Limitations
Our method shows impressive performance but relies entirely on beam search during inference. However, it is well known that beam search is a computationally expensive algorithm. With the beam size of 50, the latency increases from 3. 6 times (T5/SEQ-FULL) to 7 times (T5/SEQ) compared to greedy decoding. In addition, the re-ranking process causes another latency (about 12ms in our experiments). Therefore, it may not be suitable for real-world DST scenarios. We leave this issue for future work. Potential directions may include reducing the current two-step pipeline to an efficient one-step process by employing a novel objective function, using data augmentation, or changing the sequential decoding process to a nonautoregressive approach that can be applied in a parallel manner