Turn-Level Active Learning for Dialogue State Tracking
Authors: Zihan Zhang
Meng Fang
Fanghua Ye
Ling Chen
Mohammad Reza Namazi Rad
Venue: EMNLP 2023 Main (2023)
Keywords: dialogue state tracking
active learning
data annotation
TL;DR: A turn-level active learning method for dialogue state tracking.

Abstract: Dialogue state tracking (DST) plays an important role in task-oriented dialogue systems. However, collecting a large amount of turn-by-turn annotated dialogue data is costly and inefficient. In this paper, we propose a novel turn-level active learning framework for DST to actively select turns in dialogues to annotate. Given the limited labelling budget, experimental results demonstrate the effectiveness of selective annotation of dialogue turns. Additionally, our approach can effectively achieve comparable DST performance to traditional training approaches with significantly less annotated data, which provides a more efficient way to annotate new dialogue data.

Turn-Level Active Learning for Dialogue State Tracking
Zihan Zhang', Meng Fang", Fanghua Ye', Ling Chen', Mohammad- Namazi-Rad"
"University of Technology Sydney UUniversity of Liverpool UUnieesitty College London "University of Wollongong uts edu. au, Meng fanghua. ye. 19êucl. ac. uk, Ling. Chen@uts. edu. au, mrad@uow. edu. au
Zihan.
ac. uk
Abstract
quired. However, if it is possible to find the most informative and valuable turn in a dialogue la- bel, which enables the training achieve comparable performance, we could save the need to annotate the entire dialogue, and could efficiently utilize the large-scale dialogue data col- lected through API calls.
Dialogue state tracking (DST) plays im- portant role in task-oriented dialogue systems. However, collecting large amount of turn- by-turn annotated dialogue data is costly and inefficient. In this paper, we propose a novel turn-level active learning framework for DST to actively select turns in dialogues to annotate. Given the limited lahelling budget, experimen- tal results demonstrate the effectiveness of se- lective annotation of dialogue turns. Addition- ally. our approach can effectively achieve com- parable DST performance traditional train- ing approaches with significantly less annotated data, which provides more efficient way to annotate new dialogue data'.
DST model to
Active Learning (AL) aims reduce annotation costs by choosing the most important samples to label (Settles, 2009; Fang et al.. 2017; Zhang al., 2022). It iteratively uses an acquisition strategy find samples that benefit model performance the most. Thus, with fewer labelled data, it is possible to achieve the same or better performance. AL has been successfully applied to many fields in natural language processing and computer vision (Schumann and Rehbein, 2019; Casanova et al., 2020; Ein-Dor et al., 2020; Hu and Neubig, 2021). However, the adoption of AL in DST has been studied very rarely. Xie al. (2018) have studied to use AL to reduce the labelling cost DST, using dialogue-level strategy. They select a batch dialogues in each AL iteration and label the entire dialogues (e.g., every turn of each dialogue), which is inefficient to scale to tremendous unlabelled data. To our knowledge, turn-level AL remains unstudied for the task of DST.
1 Introduction
Dialogue state tracking (DST) constitutes an es- sential component of lask-oriented dialogue sys- tems. The task of DST is to extract and keep track of the user's intentions and goals as the dialogue progresses (Williams et al., 2013). Given the dia- logue context, DST needs predict all (domain- slot, value) at each turn. Since the subsequent system action and response rely on the predicted values of specified domain-slots, an accurate pre- diction the dialogue state is vital.
Despite the importance of DST, collecting anno- tated dialogue data for training is expensive and time-consuming, and how to efficiently annotate di- alogue is still challenging. It typically requires hu- man workers manually annotate dialogue states (Budzianowski et al., 2018) or uses Machines Talking To Machines (M2M) framework to simu- late user and system conversations (Shah et al., 2018). Either way, every turn in the conversa- tion needs to be annotated because existing DST approaches are generally trained in fully super- vised manner, where turn-level annotations are re-
Furthermore, existing DST approaches (Wu et al., 2019; Heck al., 2020; Tian et al., 2021; Zhu al., 2022) treat each dialogue turn single, independent training instance with no difference. In fact, in the real-world, utterances in a dialogue have different difficulty levels (Dai et al., 2021) and do not share equal importance in conversation. For example, in 1ig. turn-1 is simple and only contains a single domain-sla and value (i.c., hotel- name=Avalon), while turn-2 is more complex and generates three new domain-slots, i.e. hotel-book day, hotel-book people, hotel-book stay. Given the limited labelling budget, it is an obvious choice to label turn-2 instead of turn- since the former is
Code and data are available at https://githe:n http///ttttb.coooo/ con/h yintell/AL-DST.

---

Torn
User
System
Dialogue State
Domain
hotel
taxi
you tell
Avalan hotel?
Slot
name
book day
book people
book
arrivchy
departure
destination
Value
Avalon
None
Nome
None
Nome
Nome
Nome
The Avdion 4 star moderstely priced gursthouse in the norts with free interne.. Would you like to book there?
Domain
hoteel
day
book people
book stay
anriveby
departure
Yes. Can you book it for people on Saturday?
Value
Avalon
Securday
Nome
Nome
Nome
Wi need rooes rights
S3ot
name
book day
book people
book anay
ardiveby
departure
destination
reservation restaurant.
Value
Avalan
Securday
17:45
Avalon
Franãte and Bennys
Your taxi has been booked take you from Avalon to Prankie and Bennys 17-45, Your laxt will be black contact number 07715692347.
Dousain
hotel
text
people
book stay
ardiveby
destination
That aounds great. Thank
much.
Securday
Bennys
Will you any thing
Domatn
hotel
taxt
Slot
name
book day
book people
book
arrivchy
departure
destination
you that will me. goodbyn
Avalon
Sanurday
17:45
Avalon
Franilie Bennys
All tums are used in training
Only selected used training
Figure An example of DST from the MultiWOZ dataset (Budzianowski and the right sides are from user and system, respectively. Orange color denotes only the selected turn is used in the weakly- seakkly-upervised training setup. Only two domains (e.g hotel, taxi) are shown due to space limitation. (best viewed in color).
2018). Utterances at the left
more informative In addition, we observe that the complete states of the dialogue session are updated at turn-8, while turn-9 and turn- 10 simply show hu- mans' politeness and respect without introducing any domain-slots. Therefore, while the "last turn" has been studied before (Lin al., 2021a), it is often not the case that only the last turn of dialogue session generates summary states. Using redundant turns such as turn-9 and turn-10 10 for train- ing not only requires additional labelling but also possibly distracts the DST model since it introduces irrelevant context information, thus hindering the overall performance (Yang al., 2021).
significantly less annotated data, demonstrating both effectiveness and data efficiency. We summa- rize the main contributions of our work as follows:
We propose a novel model-agnostic turn-level Active Learning framework for dialogue state tracking, which provides a more efficient way annotate new dialogue data. To our best knowledge, this is the first attempt to apply turn-level AL to DST.
The superiority of our approach twofold: firstly, our approach strategically selects the most valuable turn from each dialogue to label, which largely saves annotation costs; secondly, using significantly reduced annota- tion data, method achieves the same better DST performance under the weakly- supervised setting.
Built on these motivations, we investigate a prac- tical yet rarely studied problem: given a large amount of unlabelled dialogue data with lim- ited labelling budget, how can we annotate the raw data more efficiently and achieve comparable DST performance? To this end, we propose a novel turn-level AL framework for DST that selects the most valuable turn from each dialogue for labelling and training. Experiments on MultiWOZ 2.0 and 2.1 show that our approach outperforms two strong DST baselines the weakly suserviied scenarios and achieves comparable DST performance with
We investigate how turn-level AL can boost the DST performance by analyzing the query sizes, base DST models, and turn selection strategies.
Related Work
2.1 Dialogue State Tracking
Dialogue state tracking an essential yet chal- lenging task in task-oriented dialogue systems (Williams et al., 2013). Recent sfate-of-the--e DST
Herr,, informative refers to the turn that has more valid
dialogue states.

---

models (Wu et al., 2019; Kim et al., 2020; Heck
selects contrastive samples with the maximum dis- agrecing predictive likelihood. Both are designed for classification tasks, so these strategies are not directly applicable. Therefore, studying an AL ac- quisition strategy that is suitable for DST is still an open question.
al., 2020; Ye al., 2021; Tian et al., 2021; Lee et al., 2021; Zhu et al., 2022: Hu et al., 2022) us- ing different architectures and mechanisms have achieved promising performance on complex multi- domain datasets (Budzianowski et al., 2018; Eric et al., 2020). However, they are generally trained with extensive annotated data, where each dialogue turn requires comprehensive labelling.
3 Preliminaries
We formalize the notations and terminologies used in the paper as follows.
To mitigate the cost of dialogue annotation, some works train DST models existing domains and perform few-shot learning to transfer prior knowl- edge new domains (Wu et al., 2019; Zhou and Small, 2019), while others further improve transfer learning by pre-training extensive heterogeneous dialogue corpora using constructed tasks (Wu et al., 2020; Peng et al., 2021; Lin et al., 2021b; Su et al., 2022). Recently, Liang et al. (2021); Lin et al. (2021a) propose weakly-supervised training setup, in which only the last turn of each dialogue is used. Despite the promising results, we have shown the potential drawbacks of using the last turns in Section 1. In contrast, in this work, we consider the differences between the turns and strategically select the turn that benefits the DST model the most from a dialogue for training.
Active Learning (AL) AL aims to strategically select informative unlabelled data to annotate while maximizing model's ' training performance (Set- tles, 2009). This paper focuses on pool-based active learning, where an unlabelled data pool is available. Suppose that we have model M, small seed set of labelled data L and large pool of unlabelled data U. typical iteration of AL contains three steps: (I) train the model M using L; (2) apply an acquisition function A to select k instances from U and ask an oracle to annotate them; and (3) add the newly labelled data into L.
Dialogue State Tracking (DST) Given a dia- logue D = ((X,, BI)...., (Xy, By-) that con- tains T turns, X, denotes the dialogue turn con- sisting of the user utterance and system response at turn t, while B₁ is the corresponding dialogue state. The dialogue state at turn is defined B. = {(d), Sj, 8,), "1), i J}. where dj and 8j denote domain (e.g. attraction) and slot (e.g. area) respectively, Vj is the corresponding value (c.g. south) of the domainn-soot and J is the total number of predefined domain-slot pairs. Given the dialogue context up to turn f, i.e. H, = (X Xt], the objective of DST is to predict the value for each domain-shee in dialogue state Bt.
2.2
Active Learning
Active Learning uses acquisition strategy to select data to minimize the labelling cost while maximizing the model performance (Settles, 2009). While AL has been successfully used in many fields, such as image segmentation (Casanova et al.,
2020), named entity recognition (Shen et al., 2017). text classification (Schumann and Rehbein, 2019). and machine translation (Zeng et al., 2019; Hu and Neubig, 2021), rare work has attempted to apply AL to DST. Moreover, recently proposed AL acqui- sition methods are, unfortunately, not applicable to turn-level DST since they are designed for specific tasks or models. For instance, BADGE (Ash et al., 2019) calculates gradient embeddings for each data point in the unlabelled pool and uses clustering to sample a batch, whereas we treat each turn within a dialogue as a minimum data unit and only select single turn from each dialogue; therefore, the diversity-based methods are not applicable to our scenario. ALPS (Yuan et al., 2020) uses the masked language model loss of BERT (Devlin et al., 2019) to measure uncertainty in the downstream text clas- sification task, while CAL (Margatina et al., 2021)
Labelling Suppose that we have selected a turn t from the dialogue D (1 < t < T) to label. An oracle (e.g. human annotator) reads the dialogue history from X1 to X, and labels the current di- alogue state B,. We use the gold training set simulate human annotator in our experiments.
Full vs. Weakly-supervisee Training Generally, the training dataset for DST is built in the way that each turn in a dialogue (concatenated with all pre- vious turns) forms individual training instance. That is, the input of a single training instance for turn t is defined as M, = X1 e X2 + ... @ X;- where @ denotes the concatenation of sequences,

---

and the output is the corresponding dialogue state
To avoid overfitting, re-initialize the base DST model and re-train on the current accumu- lated labelled data C. After iterations, we acquire the final training set C.
Br. By providing the entire dialogue utterances from the first turn to turn i to the model, the in- formation from the earlier turns is kept in the di- alogue history. Let Dp be the set of training in- stances created for the dialogue D and t is the selected turn. Given the example in Fig.1, for full supervision, all turns are used for training (i.e., PD ((M, ,.... (My. Br))), whereas in weakly-superviser training. only the selected turn is used (i.e., D = ((Mr B.))).
Algorithm 1 Turn-level AL for DST
Require: Initial DST model M, unlabelled dia- logue pool 21 labelled data pool L, number of queried dialogues per iteration k, acquisition function A, total iterations R
if C + then
2: Mo + Train M on L
Warm-start
4 Active Learning for Dialogue State Tracking
3: else
4: Mo + M
b Cold-start
5: end if
In this section, we first define our turn-level AL- based DST framework, followed by the turn selec- tion strategies.
for iterations r 1 R do
X, = 0
Up Random sample dialogues from U
for dialogue D u, do
4.1 Turn-Level AL for DST
10:
X + A(, x= X, U (X)
Select a turn
Framework. Our turn-level AL-based DST con- sists of two parts. First, we use AL to model the differences between turns in dialogue and find the turn that is the most beneficial to label. The pseudo- code of this step is shown Algo. 1. Second, after acquiring all labelled turns, we train DST model as normal and predict the dialogue states for all turns in the test set for evaluation, as described in Section 3. In this paper, we assume the training set is unlabelled and follow the cold-start setting (Algo. Line 4), where the initial labelled data pool L 0. We leave the warm-start study for future work.
11:
12:
end for
13:
f, + Oracle labels X
14:
C LUL,
15:
U = u Ur
16:
M, Re-initialize and re-train M L
17:
end for 18: return
> The final training set
4.2 Turn Selection Strategies
As mentioned in Section 2.2, recently proposed AL acquisition strategies are not applicable to DST, Therefore, we adapt the common uncertainty-basee acquisition strategies to select turn from dia- logue:
Active Learning Loop. In each iteration, we first randomly sample k dialogues from the unlabelled pool 4. Then, we apply turn acquisition function A and the intermediate DST model trained from the last iteration each dialogue D to select unlabelled turn (Algo. Line 10). It is notewor- thy that we consider each turn within a dialogue as a minimum data unit to perform query selec- tion. This is significantly different from Xie et al. (2018), where they select a few dialogues from the unlabelled pool and label all turns as the training instances. Orthogonal to Xie et al. (2018)'s work, it is possible to combine dialogue-level AL. However, leave it as future work because the AL strategies select dialogues and turns could be different to achieve the best per- formance, In this work, we focus on investigating the effectiveness of AL strategies for turn selection.
Random Sampling (RS) We randomly select a turn from a given dialogue. Despite its simplicity, RS acts as a strong baseline in literature (Settles, 2009; Xie et al., 2018; Ein-Dor et al., 2020).
X Random Mr)
(1)
where T is the total number of turns in the dialogue.
Maximum Entropy (ME) (Lewis and Gale, 1994) Entropy measures the prediction uncertainty of the dialogue state in dialogue turn. In partie- ular, we calculate the entropy of each turn in the dialogue and select the highest one. To do that, we use the base DST model to predict the value of the jth domain-slo at turn t, which gives us the value
turn-level strategy with

---

prediction distribution P/- We then calculate the entropy of the predicted value using P/ (Eq.2):
Base DST Model. We use KAGE-GPT2 (Lin et al., 2021a) as the basc DST model to imple- ment all experiments. KAGE-GPT2 a gen- erative model that incorporates a Graph Atten- tion Network to explicitly learn the relationships (2) between domain-slots before predicting slot val- ues. It shows strong performance in both full and weakly-supervised scenarios on MultiWOZ 2.0 (Budzianowski et al., 2018). To show that the effectiveness of our AL framework is not tied specific base models, we also experiment with an end-to end-to-end task-oriented dialogue model PPTOD (Su et al., 2022). PPTOD pre-trained on large dia- logue corpora gains competitive results on DST in the low-resource settings. The model training and implementation details are in Appendix B.
e/ =
Pf l log P1
1=1
e, =
X argmax(e1.......
(4)
where V is all possible tokens in the vocabulary. We then sum the entropy of all domain-slots as the turn-level entropy (Eq.3) and select the maximum dialogue turn 4)..4).
Least Confidence (LC) LC LC typically selects in- stances where the most likely label has the low- est predicted probability (Culotta and McCallum, 2005). In DST, we use the sum of the predic- tion scores for all J domain-slots to measure the model's confidence when evaluating dialogue turn, and select the turn with the minimum confi- dence:
5.2 Evaluation Metries
We use Joint Goal Accuracy (JGA) to evaluate DST performance, which the ratio of correct dia- logue turns. It is a striet metric since a turn con- sidered as correct if and only if all the slot values are correctly predicted. Following the community convention, although is not a distinguishable met- ric (Kim et al., 2022), we also report Slot Accuracy (SA), which compares the predicted value with the ground truth for each domain-slot at each dialogue define new evaluation metric, Reading Cost (RC), which measures the number of turns a human annotator needs to read to label a dialogue turn. As shown in Fig. I, to label the dialogue state B, at turn t, a human annotator needs to read through the dialogue conversations from X1 to X, to understand all the domain-slot values that are mentioned in the dialogue history:
C1 =
(5) turn. Additionally,
Argmin(C).......
(6)
where of = max(logits)) denotes the maximum prediction score of the jth domain-slot at turn and logits/ is the predictive distribution.
5 Experiments
Σ
C 11=1 TD.
5.1 Setup
RC
C
Datasets. We evaluate the weakly-superriseed DST performance on the MultWOZ 2.0 (Budzianowski et al., 2018) and MultiWOZ 2.1 (Eric et al., 2020) datasets? as they are widely adopted in DST. We use the same preprocessing Lin et al. (2021a) and Su et al. (2022), and focus on five domains (i.e. restaurant, train, hotel, taxi, attraction). The statistics of the datasets are summarized in Appendix A.
where |.C] denotes the total number of annotated dialogues and TD, is the number of turns of the dialogue D,. If all last turns are selected, then RC - 1, in which case the annotator reads all turns in all dialogues to label, resulting high cost. Note that we take JGA and RC as primary evaluation metrics.
5.3 Baselines
We also tried to use the SGD dataset (Rastogi et al., 2020). However, the PPTOD model is already pre-trained on this dataset, making it unsuitable for downstream evalua- tion. KAGE-GPT2 requires the predefined ontology to build a graph neural network, but SGD does not provide all possible values for non-categorical slots (See Scction 8).
Our main goal is to use AL actively select the most valuable turn from each dialogue for training, therefore reducing the cost of labelling the entire dialogues. We evaluate the effectiveness of our

---

MuldWOZ 2.0
MultWOZ 2.1
Training Data
Model
JGA
SA 7
RC
JGA
SA
RC
Without Active Learning
PPTOD KAGE-GPT2
53.3740.46
97.26
100
57. 37.10an.01
97.94=001
100
Full Data (100%)
54.8610.12
97.47.0.02
100
52.131n.0
97.1820.22
100
PPTOD Lus
43.83+1.55
96,87
100
45,94
97.11=001
100
Last Turn (14.4%)
KAGE-GPTTT
55.434025
97.14+0.01
100
49.1240.13
97.005-0:22
100
With Active Learning (k = 2000)
FPTODs.
43.06
#6.01
100
43.57
96.16=0.11
100
CUDS (~114))
KAGE-GPT2+CUDS
47.0611.45
16.14
100
47.56.1107
96.33
100
PPTOD
43.712..81
96.6410.08
58.73
46.9621011
96.5620.08
58.55
PFTOD
45.79
97.0610.01
47.37 85.21
96.97.0.05
81.951216
PPTODuue +ME
46.92:t0.79
97.12:0006
$7.37=32.9
18.20
97.33=0.12
67 67.684330.
Selected Turn (14.4%) (Ours)
KAGB-GPT2+RS
50.37+0.52
+0.06
58.58-28.7
46.98-+0.64
96.81-0.07
58.48--28.5
GAABBGGPT2LLC
50.56++0.77
97.10=0.01
70.51=3003
18.13:00.00
96, 96.94=0.11
79.41+24.0
KAGE-GPT2+ME
51.34
97.16
62.58428.5
50.02:1110
97.13=0011
71.02426.7
Table I: The mean and standard deviation of joint goal accuracy (%), slot accuracy (%) and reading cost (%) after the final AL iteration the test sets. *: re-implement using Xie et al. (2018)'s method. RS, LC and ME are active turn selection methods mentioned in Section 4.2. Note that we take JGA and RC as primary evaluation metrics since SA indistnguishable (Kim et al., 2022).
approach from two angles. First, we compare DST performance of two settings without involving AL to show the benefits that AL brings:
Results & Analysis
6.1 Main Results
Full Data (100%): all the turns are used for training, which shows the upper limit of the base DST model performance.
Due to space limitation, we report the final results after the four AL iterations in Table 1. We present the intermediate results Fig.2.
Last Turn (14.4%+) +)): following Liang al. (2021) and Lin et al. (2021a), for each dia- logue, only the last turn is used for training. Second, when using AL, compare turn-level framework with the dialogue-level approach:
Our turn-level AL strategy improves DST per- formance. From Table we first observe that, using the same amount of training data (14.4%), our proposed AL approach (i.e. PPTODpase +ME and KAGE-GPT2+ME) outperforms the non-AL set- tings, Last Turn, in terms of both joint goal ac- curacy and slot accuracy. Specifically, compared with PPTOD Turn, our PPTOD ME sig- nificantly boosts the JGA by 3.1 %% on MultiWOZ 2.0 and 2.3% MultiWOZ 2.1. KAGE-GPT2-ME also improves its baselines by around 0.9% on both datasets. Compared with the dialogue-level AL strategy CUDS, our turn-level methods improve the JGA by a large margin (2.3%~4.339 on both datasets). Considering that DST is difficult task (Budzianowski et al., 2018; Wu et al., 2019; Lee et al., 2021), such JGA improvements demonstrate the effectiveness of our turn-level AL framework, which can effectively find the turns that the basc DST model can learn the most from.
CUDS (~14%) (Xie et al., 2018): dialogue- level method that selects batch of dialogues in each AL iteration based on the combina- tion of labelling cost, uncertainty, and diver- sity, and uses all the turns for training. We carefully maintain the number of selected dia- logues in each iteration so that the total num- ber of training instances is roughly the same (i.e., A ~ 2000) for fair comparison.
Selected Turn (14.4%)
apply Algo. I and set 7888, f 0, A 2000 and use the turn selection methods mentioned in Section 4.2 to conduct experiments. As trade-off between computation time and DST perfor- mance, here we use h: = 2000; however, we find that a smaller 1: tends to have a better per- formance (Section 6.2). Given &o = 2000, we have selected 7,888 turns after four rounds, and use them to train a final model.
Our turn-level AL strategy reduces annotation cost. The reading costs (RC) of PPTODbase +ME and KAGE-GPT2+ME drop by a large margin (around 29%~44%) compared to the Last Turn and CUDS settings, indicating the benefits and necessity
$1 $14.4% 1%%=+
tumss ueed Dotc TumD
7888
F4945

---

KAGE-GPT2
PP
KAGE-GPT2-ME
PPTOD+ME
0.6
0.6
0.6
Joint Goal Accuracy
benefi- NJ DW00l100
Fut Datas 1004
J:l (41
ME
0.5
0.5
0.5 Lest Tumt (4.47)
0.5
JARE Tumb
0.4
0.4
0.4
0.4
0.3
0.3
0.3
2000
1000
6000
7008
2000
1000
6000
7880
2000
4000
6000
7889
2000
4000
5000
7885
labelled dialogue turns
of labelled dialogue turns
labelled dialogue turns
of labelled dialogue turns
Figure 2: Joint goal accuracy on test sets of AL over four iterations with 2000 dialogues queried per iteration.
on so-far selected turns after each iteration and gains a more robust perception of unseen data, thus tending to choose the turns that cial to the model. In contrast, KAGE-GPT2 un- derperforms the Last Turn setting in early itera- tions, achieving slightly higher accuracy in the fi- nal round. Despite this, the overall performance of KAGE-GPT2 is still better than PPTOD un- der the weakly-supervised settings. This is pos- sibly because the additional graph component in KAGE-GPT2 enhances the predictions at interme- diate turns and the correlated domain-slots (Lin et al., 2021a). However, when using CUDS, both DST models underperform lot on both datasets, especially during early iterations. This indicates that the dialogue-level strategy, which does not distinguish the importance of turns in a dialogue, might not be optimal for selecting training data. In Section 6.2, we show that smaller query size k achieve higher data efficiency.
Random Sampling (RS)
0.4
0.3
0.2
Joint Goal Accuracy
0.5
Confidence (LC)
0.4
0.3
0.2
Maximum Entrapy (ME)
0.5
0.3
0.2
500
1000
1500
2000
2500
3000
# of labelled dialogue turns
Figure 3: Joint goal accuracy on test sets of KAGE-
GPT2 on MultiWOZ 2.0 with 1: 500, 1000, 1500.
6.2 Ablation Studies
selecting dialogue turns. This significantly saves the annotation cost because human annotator does not need read the entire dialogue to label the last turn but only needs to read until the selected turn.
In this section, we further investigate the factors that impact turn-level AL framework.
Effect of Dialogue Query Size. Theoretically, the smaller size of queried data per AL iteration, the more intermediate models are trained, resulting the better model performance. Moreover, smaller query size is more realistic since the annotation budget generally limited and there lack enough annotators to label large amount of dialogues af- ter each iteration. To this end, we initialize the unlabelled pool by randomly sampling 3,000 di- alogues from the MultiWOZ 2.0 training set, and apply AL framework to KAGE-GPT2. using different query sizes, i.c., 500, 1000, 1500, which leads 3,3,2 rounds respectively. From Fig.3, we first observe that smaller & im-
Our approach uses less annotated data can achieve the same or better DST performance. To further explore the capability of our AL ap- proach, we plot the intermediate DST performance during the four iterations, as shown in Fig.2. No- tably, PPTOD with Least Confidence (L.C) and Maximum Entropy (ME) turn selection methods surpass the Last Turn baselines at just the second third iteration on MultiWOZ 2.0 and MultiWOZ 2.1 respectively, showing the large data efficiency our approach (only 7.3% 10.9% data are used). This can be explained that PPTOD is fine-tuned

---

proves the intermediate DST performance: when k 500, both LC and ME strategies boost the ac- curacy by a large margin at the second iteration than
RAGEHAS
PFT0O+RS
PPTOO
PPTOO-YE
0.3
PPIOP Lestium
k = 1000, and at the third iteration than l: 1500. This suggests that, with the same number train- ing data, the multiple-trained DST model gains the ability to have more accurate perception of the unseen data. By calculating the prediction uncer- tainty of the new data, the model tends to choose the turns that it can learn the most from. con- trast, RS chooses random turn regardless of how many AL rounds, therefore does not show the same pattern as LC and ME. Finally, we find a smaller tends to achieve higher data efficiency when using LC and ME strategies. It is clear from the figure that h = 500 uses the least data when reaching the same level of accuracy. However, the drawback of smaller query size is that it increases overall com- putation time as more intermediate models have to be trained. We provide computational cost analysis in Section 6.3.
Joint Goal Accuraccy
KAGE- Laetuur
0.2
0.15
0.1
0.05
500
labelled dialogue turns
Figure 4: Joint goal accuracy test sets of KAGE- GPT2 and PPTOD on MultiWOZ 2.0 with k: = 100. Results are averaged over three runs.
Method
KAGE-GPT2
PPTOD
LC
16.5
81.13
ME
68. 28.18
58.68
Table 2: Reading Cost (RC) (%) of different turn selec- tion methods. The lower the better.
Effect of Base DST Model. It is no doubt that the base DST model is critical to our turn-level AL framework it directly determines the upper and lower limit of the overall performance. How- ever, we are interested to see how our approach can further boost the performance of different DST models. We randomly sample 4 = 500 dialogues from the MultiWOZ 2.0 training set and set the query size k = 100 for both models. As shown in Fig.4, we also report the results of the two models using the non-AL strategy of Last Turn, which can be considered the lower performance baselines.
racy of PPTOD increases slowly, indicating the model gradually becomes insensitive to the newly labelled data.
Dacchng CuE
We first confirm that both PPTOD and KAGE-GPT2 outperform their Last Turn base- lines after applying our AL framework, demon- strating both data efficiency and effectiveness of approach. Secondly, notice that PPTOD achieves comparable accuracy in the first two rounds, while KAGE-GPT2 nearly stays at 0 re- gardless of the turn selection methods, showing the superiority of PPTOD under the extreme low-resource scenario. This is possibly because PPTODbasc is pre-trained on large dialogue cor- pora thus gains few-shot learning ability (Su et al., 2022), whereas only 200 training data not enough for KAGE-GPT2 to be fine-tuned. How- ever, in the later iterations, the performance of KAGE-GPT2 grows significantly, especially when using the ME strategy, eventually reaching the same level as PPTODruse- In contrast, the accu-
Figure 5: Visualization of the turns selected by PPTODaas at the final round (k- 100). ME reduces RC the most.
Effect of Turn Selection Strategy. From Fig.2, while both ME and LC improve over the RS base- line, ME does not consistently outperform LC dur- ing AL iterations in terms of the joint goal accuracy, and vice versa. However, as shown in Table 1, LC results in a higher Reading Cost (RC) than ME, which means LC tends to select latter half of turns in dialogues. Conversely, ME significantly reduces RC in the last iteration (Fig.5; more in Appendix C) and is consistently better than LC and RS for

---

Table 3: Example (MUL0295) the selected turn (marks by v/ by PPTOD using ME and LC.
Dialogue MUD0295
MI LC
[8]:
Turn
[U]: looking an expensive place dine
Stare: restaratereroosccorre
kymmayy urcascentre, restaurannt pniceronge--cppossiee
expersive.
Turn
[U]: want to book table for people at 14:00 on Sananday,
Stote: frestourent book desssnndd, restaurant book people:3, 3, restourant book Mo0=//000/
[S]: booking was suoersefll make will be reserved 15 minutes, reforence numbar viywaad)
Turn 3
[I] thank you much. 1 would also like to find train to take me kings lyrn 10:15.
(nam destinatonn-nnng yyu,, mann artt-arztheyyy 1arrivoby111155551
[5]: there are 35 deporiures with those criteria, what time de you want kave?
Turn
[t]]
should anive
pleuse sunday please.
State: fmaie (nainddyy d
[S] how lickets will you nced??
just ticket. will
id, cost
departure
Stare: ()
lhere train arriving kings sunsay depurts
7.84 id te61188.
Turn
[U]: great! that s all needed, thanks a lot for the help.
Stote:
Method
of Training data (%)
JGA
RC
Runtime (hour)
to show the potential cost reduction of our method. Suppose a dialogue D has T turns in total, and it takes minutes for human annotator to read each turn (i.e., reading time), y minutes anno- tate a single turn (i.e., annotating time), 2 dollars per minute to hire a human annotator. Assuming our proposed method selects the tth (1 S t T) turn to annotate. The total annotation cost, includ- ing the reading time and annotating time of three methods, are listed in Table 5. Since the Full Di- alogue baseline takes each accumulated turn as training instance (Section 3), it requires the high- annotation cost. Our method only annotates a single turn per dialogue, the same as the Last Turn baseline. Therefore, the annotation cost lies in the selected turn t, which measured by RC in our experiments. As shown in Table and discussed in Section 6.1, our method generally saves RC by large margin (around 29%~~33% across differ- ent models) compared the Last Turn baseline and saves more compared the Full data setting, Therefore, from a thcoretical cost estimation point of view, our proposed method can save annotation costs while maintaining DST performance.
Full 21072 (100R)
46.7
2.3
Last Turn
3000 (14.2%)
41.4
100
0.6
Table Computational cost comparison using KAGE-
GPT2 MultiWOZ 2.0 with 4 3000 and k 1000.
Method
Total Annotation Cost ($)
Full Dialogue
Last Turn
(T+x+1*g))
Selected Turn (Ours)
(++2+++ xy),, where S157
Table 5: Annotation cost estimation comparison of dif- ferent methods.
both DST models (Fig.4), which demonstrates the effectiveness ME under small query size k. We report their RC in Table 2, which also confirms that ME saves reading costs than LC. An example of the turns selected by ME and LC in a dialogue is shown in Table 3, more examples in Appendix D.
6.3 Cost Analysis
Our AL-based method saves anmotation costs and achieves comparable DST performance with tradi- tional methods at the expense of increased compu- tation time. In this section, we conduct cost anal- ysis, including computation and annotation costs. We initialize the unlabelled pool 2 by randomly sampling 3,000 dialogues from the MultiWOZ 2.0 training set, and apply our AL framework to KAGE- GPT2, and set the query size as = 1000. As shown in Table 4, our method improves JGA and RC than the Last Turn baseline, but with an in- creased runtime since our method requires three rounds of iteration.
Conclusion
This paper tackles the practical dialogue annota- tion problem by proposing a novel turn-level AL framework for DST, which strategically selects the most valuable turn from each dialogue for labelling and training. Experiments show that our approach outperforms strong DST basclines in the weakly- supervised scenarios and achieves the same or bet- ter joint goal and slot accuracy with significantly less annotated data, Further analysis are conducted to investigate the impact of AL query sizes, base DST models and turn selection methods.
Due to lack of budget, we are unable to employ human annotators to evaluate the actual annotation cost. Instead, we conduct a theoretical cost analysis

---

8 Limitations
batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learn- ing Representations.
We acknowledge the limitations of this paper as follows.
Pawet Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašié, 2018. MultiWOZ- a large- scale multi-domain Wizard-of-Oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
First, our AL approach adds extra computation time compared to directly training DST model us- ing only the last turns of dialogues. A smaller query size (e.g., k) may further increase the runtime as more intermediate models have to be trained. That is, we achieved similar or even better DST perfor- mance with significantly reduced annotation data at the cost of increased computation time. Therefore, the trade-off between computational cost, DST per- formance, and annotation cost needs to be well- determined.
Arantxa Casanova, Pedro O. Pinheiro, Negar Ros- tamzadeh, and Christopher J. Pal. 2020. Reinforced active learning for image segmentation. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Second, we are unable to employ human annota- tors to evaluate the actual cost due to a lack of bud- get. In practice, the number of annotators required depends on the financial budget, project timeline, and the proficiency of annotators. Estimating the exact number of annotators and the annotation cost is challenging. As mitigation,, we provide theo- retical cost analysis in Section 6.3. However, it is rough estimation and may not reflect the actual cost.
Aron Culotta and Andrew McCallum. 2005. Reducing labeling effort for structured prediction tasks. In AAAI, volume 5, pages 746-751.
Yinpei Dai. Hangyu Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, and Xiaodan Zhu. 2021. Preview, at- tend and review: Schema- aware curriculum learning for multi-domain dialogue state tracking. ings of the Annual Meeting of the Association for Computational Linguistics and the I Itth International Joint Conference on Natural Language Processing (Volume 2: Short Pupers), ), pages 879-885, Online. Association for Computational Linguisties.
Proceed-
Third, our experiments are limited to the Multi- WOZ 2.0 (Budzianowski et al., 2018) and Multi- WOZ 2.1 (Eric et al., 2020) datasets. We also tried to use the SGD dataset (Rastogi et al., 2020). How- ever, the PPTOD model is already pre-trained on downstream this dataset, making it unsuitable for
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minncapolis, Minnesota. Association for Computational Linguistics.
evaluation. KAGE-GPT2 requires the predefined ontology (i.e., the all possible domain-slot value pairs in the dataset) to build a graph neural net- work, but SGD does not provide all possible val- ues for non-categorical slots. For example, Mul- tWOZ has all possible values predefined for the non-categorical domain-slot train-arriveBy, while SGD does not have it since it is innumerable. Our AL framework is built upon the base DST model and thus suffers the same drawbacks; we may try other DST models and datasets in the future.
Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim, 2020. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7949-7962, Online. Association for Computa- tional Linguistics.
Acknowledgements
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. Mul- tiWOZ 1: A consolidated multi-domain dialogue dataset with state corrections and state tracking base- lines. In Proceedings of the 12th Language Re- sources and Evaluation Conference, pages 422-428, Marseille, France. European Language Resources Association.
This work is supported by TPG Telecom. We would like to thank anonymous reviewers for their valuable comments.
References
Jordan Ash. Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2019. Deep
Meng Fang. Yuan Li, and Trevor Cohn. 2017. Learning how to active learn: A deep reinforcement learning

---

approach. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 595-605, Copenhagen, Denmark, Association for Computational Linguistics.
Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul Crook. Zhiguang Wang. Zhou Yu, Eunjoon Cho, Rajen Subba, and Pascale Fung. 202 Zero-shot dialogue state tracking via cross-task transfer. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 7890-7900, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
Michael Heck, Carel van Niekerk, Nurul Lubis, Chris- tian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. 2020. TripPy: A triple copy strategy for value independent neural dialog state tracking. In Proceedings of the 21th Annual Meeting of the Special Interest Croup on Discourse and Dialogue, pages 35-44, 1st virtual meeting. Association for Computational Linguistics.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. 2021. Active learning by ac- quiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 650-663, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Junjie Hu and Graham Neubig. 2021. Phrase-level ac- tive learning for neural machine translation. In Pro- ceedings of the Sixth Conference Machine Trans- lation, pages 1087-1099, Online. Association for Computational Linguistics.
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In- lentext context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627-2602 1627-2643, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2021. Soloist: Building task bots at scale with transfer learning and machine teaching. Transactions of the Association for Computational Linguistics, 9:807-824.
Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang- Woo Lee. 2020. Efficient dialogue state tracking by selectively overwriting memory. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 567-582, Online. Association for Computational Linguistics.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Ad- vances in Neural Information Processing Systems, 34.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. To- wards scalable multi-domain conversational agents: The schema- guided dialogue dataset. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8689-88)
Takyoung Kim, Hoonsang Yoon, Yukyung Lee, Pilsung Kang, and Misuk Kim. 2022. Mismatch between multi-turn dialogue and its evaluation metric in dia- logue state tracking. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 297- 309, Dublin, Ireland. Association for Computational Linguistics.
Raphael Schumann and Ines Rehbein. 2019. Active learning via membership query synthesis for semi- supervised sentence classification. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 472-481, Hong Kong. China. Association for Computational Linguis- tics.
Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. Dialogue state tracking with a language model using schema-driven prompting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4937-4949, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Burr Settles. 2009. Active learning literature survey. Computer Sciences Technical Report 1648, Univer- sity of Wisconsin-Madiion.
David D Lewis and William A Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR '94, pages 3-12. Springer.
Pararth Shah, Dilek Hakkani-Tüi, Bing Liu, and Gokhan Tür. 2018 Bootstrapping neural conversational agent with dialogue self-play, crowdsourcing and oneline reinforcement learning. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Indus- try Papers), pages 41-51, New Orleans Louisiana. Association for Computational Linguisties.
Shuailong Liang, Lahari Poddar, and Gyuri Szarvas. 2021. Attention guided dialogue state track- ing with sparse supervision. arXiv preprint 1rX.v:2101.11958.
Weizhe Lin, Bo-Hsiang Tseng, and Bill Byrne 2021a. Knowledge-aware graph-enhanced GPT-2 for dia- logue state tracking. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 7871-7881 Online and
Yanyao Shen, Hyokun Yun, Zachary Lipton, Yakov Kro- nrod, and Animashree Anandkumar. 2017. Deep active learning for named entity recognition. In

---

Proceedings of the 2nd Workshop on Representa- tion Learning for NLP, pages 252-256, Vancouver, Canada, Association for Computational Linguisties.
Michelle Yuan, Alsuan-Tier Lin, and Jordan Boyd- Graber. 2020. Cold-start active learning through self- supervised language modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7935-7948, Online. Association for Computational Linguistics.
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Zhang. 2022. Multi-task pre-training for plug-and-pla -lay task-oriented dialogue system. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4661-4676, Dublin, Ireland. Association for Computational Linguistics.
Xiangkai Zeng. Sarthak Garg, Rajen Chatterjee, Ud- hyakumar Nallasamy. and Matthias Paulik. 2019. Empirical evaluation of active learning techniques for neural MT. In Proceedings of the 2nd Workshop Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 84-93, Hong Kong, China. Association for Computational Linguisties.
Xin Tian, Liankai Huang, Yingzhan Lin. Siqi Bao. Huang He, Yunyi Yang. Hua Wu, Fan Wang, and Shuqi Sun. 2021. Amendable generation for dialogue state tracking. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 80-92, Online. Association for Computational Linguistics.
Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2022. A survey of active learning for natural language pro- cessing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6166 6190, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Con- ference, pages 404-413, Metz, France. Association for Computational Linguistics.
Li Zhou and Kevin Small. 2019. Multi-domain dia- logue state tracking as dynamic knowledge graph enhanced question answering. arXiv preprint arXiv:1911.0611
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented di- alogue. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 917-929, Online. Association for Computational Linguistics.
Qi Zhu, Bing Li, Fei Mi, Xiaoyan Zhu, and Minlie Huang. 2022. Continual prompt tuning for dialog state tracking.
Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1124-1137, Dublin, Ireland. Association for Computational Lin- guistics.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Aso Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 808 8819, Florence, Italy. Association for Computational Linguistics.
Kaige Xie, Cheng Chang. Liliang Ren, Lu Chen, and Kai Yu. 2018. Cost-sensitive active learning for di- alogue state tracking. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue. pages 209-213, Melbourne, Australia, Association for Computational Linguistics.
Puhai Yang, Heyan Huang, and Xian-Ling Mao. 2021. Comprehensive study: How the context information of different granularity affects dialogue state track- ing? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 2481-2491, Online. Association for Computa- tional Linguistics.
Fanghua Ye, Jarana Manotumruksa, Qiang Zhang, Shenghui Li, and Emine Yilmaz. 2021. Slot self- attentive dialogue state tracking. In Proceedings of the Web Conference 2021, WWW '21, page 1598-1608, New York, NY, USA. Association for Computing Machinery.

---

A Datasets Statistics
MuliWWOZ200
MultiWOZ2.1
Dialogues
7888
7888
Domains
5
Slots
30
30
Total turns
54945
54961
Train
Last turns
7888
7888
Avg. turns per dialogue
6.97
6.97
Max turns per dialogue
22
22
Min turns per dialogue
Dialogues
1000
1000
Validation
Total turns
7374
7374
Dialogues
1000
999
Test
Total turns
7372
7368
Table 6: Statistics of the datasets in the experiments.
B Configuration Details
We use the official release of KAGE-GPT2 (Lin et al., 2021a) and PPTOD* (Su et al., 2022) to implement our turn-level AL framework.
KAGE-GPT2 We use the L4P4K2-DSGraph model setup and follow its sparse supervision (last turn) hyperparameter settings. Specifically, the loaded pre-trained GPT-2 model has 12 layers, 768 hidden size, 12 heads and 17M parameters, which is provided by HuggingFace". AdamW optimizer with linear decay rate x 10 12 is used when training. The GPT-2 component and the graph component are jointly trained, with the initial learning rates are 6.25 x 10-5 and 8 x 10-5 respectively. The training batch size used is 2, while the batch size for validation and evaluation is 16.
We use the released base checkpoint, which is initialized with T5-base model with around 220M parameters. PPTODrase is pre-trained on large dialogue corpora, for more details, we refer readers to the original paper. When training, Adafactor optimizer used and the learning rate 1 x 10-3, Both training, validation, and evaluation batch size used is 4.
PPTOD
Turn Selection During each AL iteration, we the trained model from the last iteration to evaluate all the turns within dialogue and then select turn based on the acquisition strategy.
pre-trained GPT-2 model for CAGE-GPT2 model from the released pre-trained base checkpoint for PPTOD, and then train the model usual with all current accumulated labelled turns. We train the DST model for 150 epochs using the current accumulated labelled pool L. and early stop when the performance not improved for 5 epochs on the validation set. Importantly, instead of using the full 7,374 validation set, we only use the last turn of each dialogue to simulate the real-world scenario, where a large amount annotated validation set is also difficult to obtain (Perez et al., 2021). However, we use the full test set when evaluating.
Training At the end of each iteration, re- re-initialize re-initialize
C Visualization of Selected Turns
To clearly compare the reading costs different turn selection methods, we visualize the distributions of the selected turns at the final round for the setting in Section 6.2, shown in Fig.5 and Fig.6. A dot means selected turn from dialogue, while the ends of the box represent the lower and upper quartiles, and the
-orprareedee----------alll-e- -frrr-Dilllgg--tttte--rrrak State-Track
ing
ponttps://gihhiin com/ coom/awslabs awslabs/pptod httit::://hhgggnggfae ca/ nomoollss

---

median (second quartile) is marked by a line inside the box. higher RC means the turn is selected from the second half of the conversation (RC means the last turn is selected); thus, human annotator needs to read most the conversation label its state, which is more costly. From the figures, overall, RS distributes randomly, while ME has a much lower reading cost than LC, especially for PPTOD
0.0
0.8
0.7
0.4
0.5
044
0.3
0.2
0.1
R5
LC
ME
Figure 6: Visualization of the turns selected by KAGE-GPT2 the final round (A 100).
D Example of Selected Turns
Table 3, Table 7 and Table 8 present the examples of selected turns by ME and LC using PPTOD from MultiWOZ. 2.0. [S] and [U] denote the system and user utterance respectively, while State represents the dialogue states that are mentioned at the current turn. / marks the selected turn by the strategy and is the only turn in the dialogue used for training. Although not always the case, we can see that both ME and LC can select the earliest turn that summarizes the entire dialogue, which not only saves the need to read through the whole conversation but also keeps the valuable context information intact much possible. However, still, a more suitable AL query strategy for DST is worthy of being studied.
Table Example (MUL1068) of the selected by PPTOD using ME and LC.
16666
[8]
[U]: looking gueshouse sty during my would like the section
Store: (hotel- arva-oust honel-ryp--gin oye-guecthouee
[S] there enines,
have any other preforences hotel choice?
Tum
[U]: would like star place that's the pricier side.
Stute: fenee--piiorongeeeeee mepprernnggeeeeiiiii.... hoted--ttraae
[S] thrre are no 4 star expensive goesthouses on side town. would you interested cheap or modkrate -ly priced insteud"
Tum
| would like that cheap then.
[S]: sutum house is cheap and has free internet and parking would you like me to book reservatio for you!
reservation saturday for guests night stay. Stute: (hotel-doa doymootr,,y hotei-hook people =5 hotel-book sty=2)
[U]: yes please, you
available your specification [U]: uy for night instend. if that woeks, can have the reference nimher?
sony,
you try differeent
stay?
Tum
fhond-hbook
[S: how abous the autumn house for night staraig saturday and your reference number is svciwiwa. find something called gonville and caius college. help? State: [atiraction-nome goni-lllleegellleee
Tum
[U]: also
canus collegr)
[5] pomvill and caios college located the center town trinity thing clsc for you?
[I]] get posteode
Stote: !!
problem, postal cb2lim. you need phone number as [U]: no thunks am all set. thank for your help today.
Tum

---

Table 8: Example (PMUL2281) of the selected turn by PPTOD using ME and LC.
Dialogue PX1U1,2281
I.
[S]:
[U]: can you help me place
State: f/
[S]: sure! what type food would you like to eat what area?
Tum
would like north indian food expensive, south.
State: (restamrant sostawrat-fodd-eeett mooth indian)
resiaurant though [U] indian food woold be fine then, you tell me the name?
any north
many indian restaurani is.
Tum
State: fresteurant pricennne--eppnii
restaurant food=indian]
[S]: there several indian restaurant may sugyest the golden curry the sounds like looking for.
Tum
[U] i was actually hoping for restmurant in the south, are there any aviilable??
frestawant
[S]: found called toj tandoori, more information?
Tum
[U]: more informatin please
State: ()
[5]: is expensive and in the south.
can get postcode
please?
State: f)
[S]: sure! the post code Pax.
Tum
thanks! also looking for nightclub. i'll need postcode, please?
State: cntnnccioon-yeenniheetee
[5]: ballare nightclub
town. [U]: i'll that. what the postcode please?
pounds.
Tum
Stote: ()
their postoode ch2Ana. help you with anything today?
Tum
[U]: no thanks, that was all needed today. goodhye.
(/


Reviews:
[{"review_id": "vPYeQDICY8", "reviewer": "Reviewer_fWKM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fWKM,\n\nWe appreciate your valuable time, review and suggestions. We want to address your concerns in detail as follows.\n\n> Q1. Code can be released for other users to work on top of the current work\n\n**A1:** Thanks for your interest. We will release all code and data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tResults are presented well with detailed ablation study, The ablation analysis and case studies provide additional insight into the effectiveness of the approach.\n•\tIt presents a novel approach with strong performance, and clear motivation.  Turn selection based on maximum entropy showed significant improvement across both Multi-Woz dataset. \n•\tThis is a very well written paper, coherent and easy to understand. \n•\tOverall, this is an effective paper with strong results.\n", "reasons_to_reject": "Code can be released for other users to work on top of the current work", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a turn level active learning framework for DST that can help train a DST with minimal amount of annotation.  Novelty of the paper is that it is looking into turn level annotation rather than conversation level which is a unstudied task for DST. Their results and ablation studies shows comparable performance to traditional training with a significantly less annotated data. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EgTDqPgS9c", "reviewer": "Reviewer_Es6J", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Es6J,\n\nWe appreciate your valuable time, review and suggestions. We want to address your concerns in detail as follows.\n\n> Q1. The paper lacks comparisons of the training computation cost between their proposed approach and the previous dialogue-level approach. It is unclear how much computation cost is added for the improved performance (or reduced annotation cost).\n\n**A1:**\nThanks for your suggestion. Using the PPTOD-base model as an example, we did a preliminary analysis of the computation cost (as shown in the table below). We will provide more analysis when we can modify our paper.  \n\nSpecifically, when training using the full dataset, it took about 4 hours and achieved 53.2% JGA score on MultiWOZ 2.0. When training using only the last turns, it took about 0.8 hours and achieved 43.5% JGA score. When using our proposed active learning framework and using four iterative loops, it took about 1.9 hours in total and achieved 46.8% JGA score. All experiments run on a single RTX 3090. As discussed at L585, our proposed method can achieve similar or even better DST performance with significantly reduced annotation data than the last turn baseline, with the cost of slightly increased computational cost.\n\n| **Method**                                | **% of Training data &#8595;** | **JGA &#8593;** | **RC &#8595;** | **Running time (hour) &#8595;** |\n|-------------------------------------------|------------------------|---------|--------|-------------------------|\n| PPTOD-base + Full data                     | 100%                   | 53.2    | 100    | 4                       |\n| PPTOD-base + Last Turn                     | 14.4%                  | 43.5    | 100    | 0.8                     |\n| **PPTOD-base + selected turn + ME (ours)** | 14.4%                  | 46.8    | 55.9   | 1.9                     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experimental results of the proposed turn-level approach on the MultiWOZ dataset are significanttly better than the previous dialogue-level active learning approach.\n2. The paper is well written and easy to follow.\n3. The authors did comprehensive analyses with different hyperparameters and different turn selection methods.\n", "reasons_to_reject": "1. The paper lacks comparisons of the training computation cost between their proposed approach and the previous dialogue-level approach. It is unclear how much computation cost is added for the improved performance (or reduced annotation cost).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a novel turn-level active learning framework for Dialogue State Tracking(DST) to reduce the human annotation cost of the DST dataset. The proposed approach achieves significantly better DST performance with redcued annotation cost.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zl9kI0jmEz", "reviewer": "Reviewer_ttCV", "summary": "", "questions": "One underlying motivation for using the last-turn annotation scheme is that such data could be easily accessed from commercial service data [1]. It can be easy to obtain the actions done by the customer service staff after a phone call. It’d be appreciated if this case can be discussed and compared to the proposed approach.\n\n[1] Weizhe Lin, Bo-Hsiang Tseng, and Bill Byrne. 2021. Knowledge-aware graph-enhanced GPT-2 for dialogue state tracking.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ttCV,\n\nWe appreciate your valuable time, review and suggestions. We want to address your concerns in detail as follows.\n\n> Q1. Some more discussions regarding the recent progress in DST will be appreciated. MultiWOZ is a traditional DST dataset, while recently the Schema-guided DST dataset is actively investigated. This new setting challenges the few-shot/zero-shot generalisability of DST systems on new services/domains. Though not required in this work, it would be interesting to see a discussion on how this approach could be generalised to/used in the SGDST setting.\n\n**A1:** Thanks for your suggestion. \nOur proposed method can be directly applied to the SGD dataset.\nThe SGD dataset captures real-world scenarios and contains unseen domains and services in the evaluation set for zero-shot performance evaluation. Using the SGD dataset, it might be interesting to see how our proposed framework can benefit unseen domains. We will add experiments on the SGD dataset when we are allowed to modify our paper.\n\n> Q2. It remains unclear how this approach can be used in practice. Imagine that we want to train a new DST system and we want to annotate domain-specific data for this purpose, what steps are required (in practice) to combine annotators with the on-the-fly training process? Does the model training run in the background while a batch of annotators do async-ed annotation?\n\n**A2:** As shown in Algo1 (L234), following the conventional active learning practices, this process runs iteratively. Firstly, we can start from a base DST model (such as PPTOD, warm start), or a pre-trained LM (such as GPT-2, cold start). Then, given a batch of unannotated dialogues, we use our method to select a turn for each dialogue. Thirdly, we ask annotators to read the dialogue until the selected turn to collect this batch of annotated dialogue data. Finally, we use the collected annotated data to fine-tune the DST model, conduct evaluation and record performance. In the next loop, we start with the latest DST model and annotate the next batch of raw dialogues. \n\n> Q3. The two datasets being evaluated are very similar (v2.0 and v2.1). It would be better if a third dataset can be evaluated.\n\n**A3:** Although similar, MultiWOZ2.1 differs from 2.0 in that it fixes annotation errors and serves as another challenging benchmark for the DST problem. The same DST model may perform differently on these two datasets [1]. \nHaving said that, as responded in A1, we will add experiments on the SGD dataset when we are allowed to modify our paper.\n\n> Q4. One underlying motivation for using the last-turn annotation scheme is that such data could be easily accessed from commercial service data [1]. It can be easy to obtain the actions done by the customer service staff after a phone call. It’d be appreciated if this case can be discussed and compared to the proposed approach.\n\n**A4:** We agree that obtaining the last-turns of conversations and their associated actions might be easy in practical scenarios. However, the next action is determined by the policy learning module in the pipeline-based task-oriented dialogue system [1], whereas we focused on the DST module. Hence, although actions can be easy to obtain, it is less helpful for training a DST model. \n\n\n[1] Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System, ACL’22", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Combining active learning to select samples for annotation with the aid of the model being trained is a novel idea, which is rarely explored in literature.\n2. The experiments show that with the proposed approach, several recent advanced DST systems can achieve performance comparable to training with full data and weakly supervised data. In particular, Maximum Entropy (ME) achieves better performance than training with last-turn annotations while reducing the RC (Reading Cost) by ~40%, which means annotators do not have to read the full dialogue. This can greatly reduce the workload of annotators.\n3. The writing is fluent and clear. The presentation is good. \n4. The experiments are complete. Two different models and two datasets were experimented, demonstrating that to some extent this approach is generalisable. I found the analyses interesting, especially those about Reading Cost.", "reasons_to_reject": "1. Some more discussions regarding the recent progress in DST will be appreciated. MultiWOZ is a traditional DST dataset, while recently the Schema-guided DST dataset is actively investigated. This new setting challenges the few-shot/zero-shot generalisability of DST systems on new services/domains. Though not required in this work, it would be interesting to see a discussion on how this approach could be generalised to/used in the SGDST setting. -- after discussion, the authors promised to add some experiments related to this question.\n2. It remains unclear how this approach can be used in practice. Imagine that we want to train a new DST system and we want to annotate domain-specific data for this purpose, what steps are required (in practice) to combine annotators with the on-the-fly training process? Does the model training run in the background while a batch of annotators do async-ed annotation? -- after discussion, the authors promised to add some discussion related to this question.\n3. The two datasets being evaluated are very similar (v2.0 and v2.1). It would be better if a third dataset can be evaluated.", "questions_to_authors": "One underlying motivation for using the last-turn annotation scheme is that such data could be easily accessed from commercial service data [1]. It can be easy to obtain the actions done by the customer service staff after a phone call. It’d be appreciated if this case can be discussed and compared to the proposed approach.\n\n[1] Weizhe Lin, Bo-Hsiang Tseng, and Bill Byrne. 2021. Knowledge-aware graph-enhanced GPT-2 for dialogue state tracking.", "paper_topic_and_main_contributions": "The paper attempts to tackle a very realistic problem in DST - annotating dialogue turns in turn level is highly expensive. Even with the last-turn annotation strategy proposed in literature (the annotator annotates the last turn instead of the full dialogue), the annotator must read through all turns in a dialogue in order to annotate all dialogue states correctly. This paper presents a turn-level selection strategy based on active learning to significantly reduce annotation costs while maintaining the dialogue state tracking performance. This topic is rarely investigated in current literature, and it shows great potential in reducing the required resources for annotating dialogue states for building advanced DST systems.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses an essential yet under-explored problem in Dialogue State Tracking: minimizing the amount of turn-level annotation. The proposed active learning approach at the turn level, which is based on maximum entropy, yields performance comparable to training with a complete (and considerably larger) training corpus. Both the ablation studies and case analyses offer valuable insights. A notable limitation of this research is the additional computational cost it introduces. While the authors have present the computation study in their rebuttal, I recommend incorporating the analysis into the camera-ready version of the paper.

All reviewers agree that the paper is solid and offers value to the dialogue field.
Decision: Accept-Main