THE FIFTH DIALOG STATE TRACKING CHALLENGE
Seokhwan Kim1 , Luis Fernando D’Haro1 , Rafael E. Banchs1 , Jason D. Williams2 ,Matthew Henderson3 , Koichiro Yoshino4
1
Institute for Infocomm Research, Singapore. 2 Microsoft Research, USA.
3
Google, USA. 4 Nara Institute of Science and Technology, Japan.
ABSTRACT

dialog turns in a given sub-dialog.
Different to the previous (fourth) challenge, in this edition we introduced a cross-language dialog state tracking task,
aiming at addressing the problem of adaptation to a new language. In this case, the goal is to build a tracker for the target language using existing resources in the source language
and the corresponding machine translated sentences in the target language. In addition to the main task, we also proposed
different pilot tracks for the core components in developing
end-to-end dialog systems following the same cross-language
setting. Our goal with this new focus is to contribute to theprogress on improving the language portability of state-ofthe-art monolingual technologies while reducing the costs ofdeveloping dialog systems for resource-poor target languages.

Dialog state tracking – the process of updating the dialog
state after each interaction with the user – is a key component of most dialog systems. Following a similar scheme tothe fourth dialog state tracking challenge, this edition again
focused on human-human dialogs, but introduced the task ofcross-lingual adaptation of trackers. The challenge received a
total of 32 entries from 9 research groups. In addition, several
pilot track evaluations were also proposed receiving a totalof 16 entries from 4 groups. In both cases, the results showthat most of the groups were able to outperform the provided
baselines for each task.
Index Terms— Dialog state tracking, cross-lingual adaptation, challenge report.

2. CHALLENGE OVERVIEW
1. INTRODUCTION

2.1. Challenge design

Dialog state tracking refers to the task of updating the dialog
state after each interaction between the system and the user,
where the dialog state represents the history of the conversation up to the current timestep. Since most dialog managers
choose their next action based wholly or at least in part on
the dialog state, dialog state tracking is one of the key problems in building a dialog system. Yet dialog state tracking
is difficult because errors in speech recognition and language
understanding render the true state of the dialog partially observable, and because natural language can be ambiguous.
The Dialog State Tracking Challenge was created to promote research in this area and to provide a common benchmark for evaluating this task. Four different challenges havebeen organized previously [1, 2, 3, 4]. Like in the fourth challenge, in this fifth edition we also focused on state tracking
for human-human dialogs that has the desirable property thatno pre-specified policy is followed, yielding dialogs that arevaried. This approach hedges against building a dialog state
tracking that over-fits to a particular fixed dialog manager, andtakes a step toward building dialog systems directly from a
corpus of human-human interactions. For this challenge, dialog states are defined at the sub-dialog segment level using
a frame structure that consists of slot-value pairs representing the main subject in each sub-dialog. Therefore, trackers
are required to fill out the frames by considering the previous

978-1-5090-4903-5/16/$31.00 ©2016 IEEE

This fifth challenge shares much in common with DSTC4 [4]in the definitions of the target tasks and the characteristics ofthe datasets. This section gives a summary of the challenge
and the newly introduced cross-language aspects of the tasks.
2.1.1. Main Task
The main task aims at tracking the dialog state defined as aframe structure filled with slot-value pairs representing the
subject of each sub-dialog in human-human dialogs. Subdialogs have been manually segmented from a full dialog session and annotated with the topic category. For each turn ina given sub-dialog, its topic-specific frame should be filled
out considering all the dialog history up to the turn. Fig. 1
shows examples of Chinese dialog segments annotated with
their dialog state frames.
Different from DSTC4, this challenge addresses a crosslanguage dialog state tracking problem. Firstly, a training setof labelled dialogs in English and a small development set in
Chinese were released to participants at the beginning of thechallenge. During the developing phase, the participants built
their systems for state tracking in Chinese dialogs using theEnglish training set. Finally, the performance of each tracker
was evaluated on the unlabelled test set in Chinese by comparing the system outputs with reference annotations.

511

GlobalSIP 2016

Speaker
Guide
Tourist
Guide
Tourist
Guide
TouristGuide
Guide
TouristGuide
Guide
Tourist
Guide

Tourist
Guide
TouristUtterance我介绍你这个甘榜格南。
(I recommend you this Kampong Glam.)
对。(Right.)
你看,它是个-它是马来村嘛(You see, it is a- it’s a Malay Village)对，甘榜- (Right, Kampong-)它就卖了很多马来食物。
(It sells a lot of Malay food.)比较有特色的食物,(It’s quite a unique food,)对，哦。(Right.)
马来食物，基本上，它是香。
(Malay food, basically, it smells very nice.)那我们住宿呢？(Then, where do we stay?)
我介绍一间呵，叫V Hotel的。
(Let me recommend to you, the V Hotel.)
这个酒店，价格这个不贵。
(This hotel, the price is not expensive.)
好的。 (Okay.)
如果要去，我建议的这个马来文化村，(If you want to go, I suggest this Malay cultural village,)马来村? (Malay village?)步行大概我看十五分钟吧。
(I think it would take fifteen minutes on foot.)
好。 (That’s good.)

Dialog State

Table 1: Overview of DSTC5 data

TOPIC: Attraction
TYPE OF PLACE:
Ethnic enclave
NBHD:
Kampong Glam

Set
Train
Dev
Test
Test
Test
Test

TOPIC: Food
CUISINE:
Malay cuisine
NEIGHBORHOOD:
Kampong GlamTOPIC:
Accommodation
INFO: Pricerange
NAME: V Hotel

Task
ALL
ALL
MAIN
SLU
SAP
SLG

Language
EnglishChineseChineseChineseChineseChinese# dialogs
35
2
10
8
8
8

# utterances
31,3043,130
14,87812,655
11,456
12,34621 hours per language, were manually transcribed and annotated with speech act and semantic labels at the turn level.
The whole English part of the dataset, which was usedpreviously in DSTC4, was released as training set for all thetasks in DSTC5. The Chinese dialog set was divided into
five subsets, including the development set (with just two sessions) and four test sets corresponding to the main task andthree pilot tasks (Table 1). Each dialog session in both thetraining and the test sets for the main task was manually segmented and annotated with one of the five major topic categories and its corresponding frame structure.
In addition to the original dialogs, 5-best translations were
provided for each utterance. These translations were generated by using an English-to-Chinese machine translation system for the training set and a Chinese-to-English system forthe development and test sets. All translated utterances were
given with the corresponding word alignment information.
Along with the dialog corpus, a ontology was also provided to describe the tagset definitions as well as the domain
knowledge regarding tourism in Singapore. The original
DSTC4 ontology, which was created based on the English
dialogs, have been expanded with additional contents occurring in the Chinese dialogs. It also included 5-best results ofEnglish-to-Chinese machine translation for each entry.

TOPIC:
Transportation
INFO: Duration
TYPE: Walking
FROM: V Hotel
TO: Kampong Glam

Fig. 1: Example human-human dialog in Chinese and dialog
state labels for the main task of DSTC5.

2.1.2. Pilot Tasks
As in DSTC4, this challenge also included these pilot tasks:• Spoken language understanding (SLU): Tagging agiven utterance with speech acts and semantic slots.
• Speech act prediction (SAP): Predicting the speech actof the next turn imitating the policy of one speaker(tourist or guide).
• Spoken language generation (SLG): Generating a response utterance for one speaker by using the corresponding speech act and semantic slot information.
• End-to-end system (EES): Developing an end-to-end
system by pipe- lining and/or combining different SLU,SAP and SLG systems1 .

3. EVALUATION
3.1. Main Task

Following the same cross-language scenario as in themain task, each system for the pilot tasks was to be trainedon English dialogs and then evaluated over Chinese dialogs.
Further information can be found in DSTC5 handbook [5].

3.1.1. Evaluation metrics
Main task systems were required to generate a tracking output
for every turn in a given log file. Trackers were allowed to use
all the transcriptions and sub-dialog details provided in thelog object from the beginning of the session, up to the current
turn. Trackers were prohibited from using any informationfrom the future turns, because in a practical dialog system,
this information would not yet be available.
To examine the capabilities of a tracker for both understanding the contents in a given sub-dialog and predicting its
dialog states, two different schedules were considered to select the utterances for the target of evaluation:

2.2. Data
The challenge uses the TourSG corpus [5], which consists of
dialog in both English and Chinese about tourist informationfor Singapore collected from Skype calls between tour guides
and tourists. All the recorded dialogs, with a total duration of
1 As in DSTC4, no teams opted to participate in the end-to-end system
track. We conjecture this is because of the substantial effort required to fieldan end-to-end system.

512

5500

• Schedule 1: all turns are included

5000• Schedule 2: only the turns at the end of sub-dialogs areincluded

4500

extraneous
missing
false

4000

• Accuracy: Fraction of sub-dialogs in which the tracker’s
output is equivalent to the gold standard frame structure• Precision: Fraction of slot-value pairs in the tracker’s
outputs correctly filled

Number of errors

The following metrics were used for evaluation:

3500
3000
2500
2000
1500
1000

• Recall: Fraction of slot-value pairs in the gold standard
labels correctly filled

5000• F-measure: The harmonic mean of precision and recall
3.1.2. Baseline tracker

baseline team1 team2 team3 team4 team5 team6 team7 team8 team9
entry4 entry3 entry3 entry2 entry0 entry1 entry1 entry0 entry0

Fig. 3: Number of errors made by type for the best entry fromeach team in the main task on the test set.

A simple baseline tracker was provided for the main task. It
is based on the rule-based system used in DSTC4, which determines the slot values by fuzzy string matching between theentries in the ontology and the transcriptions of the utterances
mentioned from the beginning of a given sub-dialog to thecurrent turn. To adapt it for the cross-language execution, thefollowing two different methods were implemented.

respectively, under both schedules. Fig. 2 reveals that thehighly-ranked trackers in the overall comparison tend to produce evenly good results across all topic categories. To investigate the reasons for the performance differences among the
trackers, the slot-level errors under Schedule 2 from the best
entry of each team were categorized into the three error types
following [6]:• Method 1: The translated utterances from Chinese-toEnglish are matched to the English entries in the original ontology (team 0 / entry 0).

• Missing attributes: if the reference contains values fora slot, but the tracker outputs no value for that slot.

• Method 2: The Chinese utterances are matched tothe translated entries in the ontology from English-toChinese (team 0 / entry 1).

• Extraneous attributes: if the reference contains no value
for a slot, but the tracker outputs values for that slot.

For both methods, only the top-1 hypothesis of the 5-best
translations was used for each matching. If a part of a given
utterance was matched with an entry for a slot in the ontology
with over a certain level of similarity, the entry was simply
assigned as a value for the particular slot in the tracker’s output.

• False attributes: if the reference contains values for aslot, but the tracker outputs wrong values for that slot.
The error distributions in Fig. 3 indicate that team2.entry3
achieved the higher performances by much lower numbers of
extraneous and false attributes than the others. On the other
hand, team4.entry2 yielded competitive results by reducing
the missing slot errors.

3.1.3. Results
In total 32 entries were submitted from 9 research teams. To
preserve anonymity, the teams were identified by numbers
from 1 to 9. The baseline system was marked as team 0.
Table 2 shows the averaged results over the whole test
set for each submitted entry. More specific scores by topic
and slot type and all the submitted entries are available on
the DSTC5 repository 2 . As seen from the table, most of
the trackers outperformed the baseline in all the combinationsof schedules and metrics. Especially, the best entries fromteam 2 achieved almost three times and more than twice ashigh performances as the baseline in accuracy and F-measure,

3.2. Pilot Tasks: SLU and SAP
3.2.1. Evaluation metrics
Each pilot task includes two subtasks, one each for modelling
the tour guide and the tourist. Both SLU and SAP tasks sharethe same utterance-level annotations for speech acts and semantic tags. In the SLU task, a system is required to produce
both semantic tags and speech acts for a given unlabelled utterance spoken by the target role speaker. For the SAP task, an
input included the utterance from the other speaker labelledwith both speech acts and semantic tags along with the resulting semantic tags for the next turn by the target speaker. Then,

2 https://github.com/seokhwankim/dstc5

513

Table 2: Main task results on the test set. Team 0 is the rule-based baseline. Bold denotes the best result in each column.

Team
0
0
1
1
1
1
1
2
2
2
2
2
3
3
3
3
4
4
4
4
4
5
5
5
5
5
6
6
6
7
7
8
8
9

Entry
0
1
0
1
2
3
4
0
1
2
3
4
0
1
2
3
0
1
2
3
4
0
1
2
3
4
0
1
2
0
1
0
1
0

Accuracy
0.02500.01610.03970.03860.03930.03870.04170.0736
0.05670.05290.07880.06990.03510.03030.02890.0341
0.05830.04070.05150.05520.04540.03300.01870.01830.03130.00930.03890.03400.04910.00920.00850.01920.00680.0231

Schedule 1
Precision Recall
0.11480.11020.17430.12790.33200.29340.36150.26100.36730.26390.40870.24360.36500.27950.46640.34490.37120.38180.36290.38920.51950.33150.4862
0.34320.32160.15150.26480.22350.31820.15380.29260.20950.40080.2776
0.35540.32670.36820.37350.37170.35830.34730.36770.33770.23180.14740.23250.19730.12360.15060.16480.42650.05310.44670.20920.38970.25330.46840.21930.42870.04310.58920.04100.31300.10480.09240.03950.11390.1090Accuracy
0.0321
0.02220.05510.05970.05510.0597
0.0612
0.09640.07120.0681
0.09560.08720.05050.03670.04060.04510.07650.04130.06350.06810.05590.05200.02300.01680.04130.01150.04820.03830.06430.01070.01150.02140.00690.03140.6

0.12

baseline1
team1.entry4
team2.entry0
team3.entry0
team4.entry00.10

team5.entry0
team6.entry2
team7.entry1
team8.entry0
team9.entry00.5

Schedule 2
Precision Recall
0.14250.15000.19790.17740.34290.37120.37850.33240.3794
0.33580.42600.30870.38110.35480.52170.38490.43400.4196
0.42160.43030.56430.37690.54270.38420.33500.20450.27880.28730.33770.20780.3076
0.27330.41270.32840.35690.35750.37680.41400.3806
0.4026
0.35100.40430.36370.30440.16110.25260.20030.10420.17280.20620.4286
0.05510.45090.2516
0.40630.31240.47580.26230.40000.04410.53690.04380.30210.1046
0.09480.04140.14120.1487

F-measure0.14620.18710.35650.35400.35630.35800.36750.44300.42670.42590.45190.44990.25390.28300.25730.28950.36580.3572
0.39450.39130.37580.33140.19670.13710.18800.09770.32300.3532
0.33810.07940.08090.15540.05770.1449baseline2
team1.entry4
team2.entry3
team3.entry3
team4.entry2

team5.entry0
team6.entry1
team7.entry1
team8.entry0
team9.entry00.4
F-measure

0.08
Accuracy

F-measure0.11240.14750.31150.30320.30710.30520.3166
0.3966
0.37640.37560.40470.40240.20600.24240.20740.24420.32800.34050.37080.36490.35720.27490.18040.15200.15740.09450.28490.30700.29880.07830.07670.15700.05540.1114

0.06

0.3

0.04

0.2

0.02

0.1

0.00

tio
attracn0.0

gnion food
ortatio shoppin
m od a ttranspa c co m

attractionotranspgntion
rtatio shoppin
m od aa c comf oodFig. 2: Comparisons of the main task performances per topic category among the best tracker from each team514

Table 3: Results of speech act identification in SLU on the test set.

Team
0
2
2
2
2
3
3
5
5
5
5
5
7

Entry
0
0
1
2
3
0
1
0
1
2
3
4
0

Precision
0.45880.54500.53050.55330.51270.42790.43400.40850.39050.46390.45400.44590.5007

Guide
Recall0.24800.39110.39690.38290.42510.35830.36350.33640.3216
0.38200.37390.36720.2976

F-measure
0.32190.45540.45400.4526
0.46480.39000.3956
0.36900.35270.41900.4101
0.40280.3733

Precision
0.36940.5001
0.53310.51070.56050.45910.44980.50260.45190.4916
0.48710.49840.5079TouristRecall0.18280.5501
0.52630.54250.49990.42410.41190.44840.40310.43850.43460.4446
0.4156

F-measure0.2446
0.52390.5297
0.5261
0.52850.44090.43000.47390.4261
0.46350.45940.47000.4571

Table 4: Results of semantic tagging in SLU on the test set.

Team
0
3
3
5
5
5
5
5
7

Entry
0
0
1
0
1
2
3
4
0

Precision
0.4666
0.46500.46500.5006
0.54690.35770.3486
0.33950.4400

Guide
Recall
0.31870.31820.31820.29230.18930.24760.25410.21110.3207

F-measure
0.37870.37790.37790.36910.28130.29260.29390.26030.3710the system is expected to generate the speech acts for the next
utterance. The following evaluation metrics were used:

Precision
0.52590.5331
0.53310.50830.51210.30310.29320.29470.4408

TouristRecall0.26590.26200.26200.31100.30810.22370.21490.20720.2926

F-measure
0.35320.35130.35130.38590.38470.25740.24800.24330.35173.2.2. Baseline system
A simple baseline system was also provided for the crosslanguage SLU pilot task. It used a pair of support vector machines (SVMs) and conditional random fields (CRFs) models trained with basic bag-of-words features for multi-label
speech act prediction and semantic tagging, respectively. The
models were trained with the English training dataset. Then,the trained models were used to analyze the English translations of each Chinese utterance in the test set. Finally, thepredicted labels were projected into the original utterancesin Chinese by means of the word alignment information obtained from the machine translation system.
The baseline for the SAP task used SVM models for
multi-label speech act prediction using the following features:the semantic tags in the current and the previous utterances,
the speech act tags of the recent utterance spoken by the otherspeaker, and the distance from the other speaker’s turn tothe current utterance. Since all these features are languageindependent, the model trained on the English training set
was directly applied on the Chinese test set.

• Speech acts for SLU and SAP– Precision: Fraction of speech act labels that are
correctly predicted.
– Recall: Fraction of speech act labels in the gold
standard that are correctly predicted.
– F-measure: The harmonic mean between precision and recall• Semantic tags for SLU– Precision: Fraction of correctly predicted semantic tags in the generated tag sequences encodedusing BIO scheme.
– Recall: Fraction of correctly predicted semantic
tags in the gold standard tag sequences encodedusing BIO scheme.
– F-measure: The harmonic mean between precision and recall

515

Table 5: SLG results on the test set.

3.2.3. Results
For the SLU task, 4 teams participated with a total of 12 entries submitted 3 . For the SAP task, no submissions were received. Table 3 and Table 4 present the summary of the SLU
results for speech act identification and semantic tagging, respectively. All the submitted entries achieved much better
performances than the baseline in speech act identification forboth speakers. Especially, all the entries from team2 outperformed not only the baseline, but also all the other teams.
However, most participants failed to show significant improvements in semantic tagging performances. Only two entries team5.entry0 and team5.entry1 produced better results
than the baseline and only for the case of the tourist turns.

Team
0
5
5
5
5

Entry
0
0
1
2
3

Guide
AM-FM BLEU
0.19810.38540.28180.32640.31800.33710.27370.28520.24050.2758

Tourist
AM-FM BLEU
0.26020.5921
0.32210.48500.36350.52490.31000.47410.42580.5302

Table 6: Example of generated sentences by the baseline andthe best entry from the participant
Input Info
Speech act: FOLLOW(INFO, WHERE)
Semantic tags: ATTRACTION(LOCATION: 圣淘沙 (Sentosa))
Reference现在你看到的这个就是圣淘沙，啊。
(what you see now, this, it is Sentosa.)
Baseline
就是那个位置在圣淘沙。 (it is that position on Sentosa)
BLEU: 0.280 AM: 0.626 FM: 0.783
AM-FM: 0.705
Best Entry
那是圣淘沙。 (that is Sentosa)
BLEU: 0.103 AM: 0.461 FM: 0.964
AM-FM: 0.713

3.3. Pilot Task: SLG
3.3.1. Evaluation metrics
In this task, the system was intended to receive as input thesemantic tags and speech acts from the target speaker only,
and it was expected to produce the final surface form of theutterance. Here, the goal was to maximize the syntactic andsemantic similarity between the gold standard and the generated sentence. The following similarity metrics were used:• BLEU: Geometric average of n-gram precision (for n= 1, 2, 3, 4) of the system generated utterance with respect to the reference utterance [7].

4. CONCLUSIONS• AM-FM: Linear interpolation of (1) the cosine similarity between the system generated utterance and the reference utterance and (2) the normalized n-gram probability of the system generated utterance [8]. In brief,
AM measures the semantic similarity between the reference and the generated sentence, while FM measuressimilarity of the n-gram language model probabilities.

We have presented the official evaluation results of the Fifth
Dialog State Tracking Challenge (DSTC5). This edition has
focused on dialog state tracking in human-human dialogs, but
introducing the problem of adaptation to a new language. A
total of 9 teams participated in the main task with an overall
number of 32 entries submitted. In addition, four pilot tasks
were proposed that received a total of 12 entries from 4 groups
for the Spoken Language Understanding (SLU) and 4 entries
from one group for the Spoken Language Generation (SLG)
tasks, respectively.
Following the same success of DSTC4, we have confirmed the feasibility of performing dialog state tracking
at the sub-dialogue level in the context of human-humandialogs, while simultaneously addressing the problem of handling a cross-lingual setting. Although the reported results
were in most cases better than the provided baselines, it isclear that more work is needed to produce high-accuracy
trackers in this setting. Additionally,there is the need forincreasing the volume of training data to test more complex
approaches, such as methods based on deep learning.
Overall, we feel that DSTC5 has been a meaningful step
toward the long-term goal of creating modular, data-driven
end-to-end dialog systems systems. Based on the sustainedlevel of participation over the DSTC series, and on continuing technical advances, we have begun planning for a sixth
edition of the challenge.

3.3.2. Baseline system
The SLG baseline was based on an example-based language
generation approach, which used k-nearest neighbors algorithm on the vector space with the speech act and semantic
tags features. For each input, the system searched for the mostsimilar item in the English training set, and then returned the
top-1 hypothesis of its Chinese translations as result.
3.3.3. Results
For the SLG task, only one team participated with a total of 4
entries. The results are depicted in Table 5. All the submitted
entries achieved better performance than the baseline, especially for generating the tourist sentences. Table 6 provides
an example of a sentence generated by the baseline and thecorresponding entry from the participant.
3 Team 2 participated only in speech act identification.

516

5. REFERENCES[1] Jason Williams, Antoine Raux, Deepak Ramachandran,and Alan Black, “The dialog state tracking challenge,” inProceedings of the SIGDIAL 2013 Conference, 2013, pp.
404–413.
[2] Matthew Henderson, Blaise Thomson, and Jason
Williams, “The second dialog state tracking challenge,”in 15th Annual Meeting of the Special Interest Group on
Discourse and Dialogue, 2014, p. 263.
[3] Matthew Henderson, Blaise Thomson, and Jason D
Williams, “The third dialog state tracking challenge,”in Spoken Language Technology Workshop (SLT), 2014
IEEE. IEEE, 2014, pp. 324–329.
[4] Seokhwan Kim, Luis Fernando D’Haro, Rafael E.
Banchs, Jason Williams, and Matthew Henderson, “The
Fourth Dialog State Tracking Challenge,” in Proceedingsof the 7th International Workshop on Spoken Dialogue
Systems (IWSDS), 2016.
[5] Seokhwan Kim, Luis Fernando D’Haro, Rafael E
Banchs, Jason Williams, Matthew Henderson,
and Koichiro Yoshino,“Dialog state tracking challenge 5 handbook,”
2016,
https://github.com/seokhwankim/dstc5/raw/
master/docs/handbook_DSTC5.pdf.
[6] Ronnie W Smith, “Comparative error analysis of dialog
state tracking,” in 15th Annual Meeting of the Special
Interest Group on Discourse and Dialogue, 2014, p. 300.
[7] Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu, “Bleu: a method for automatic evaluation ofmachine translation,” in Proceedings of the 40th annual
meeting on association for computational linguistics. Association for Computational Linguistics, 2002, pp. 311–
318.
[8] Rafael E Banchs, Luis F D’Haro, and Haizhou Li,“Adequacy–fluency metrics: Evaluating mt in the continuous space model framework,” Audio, Speech, and Language Processing, IEEE/ACM Transactions on, vol. 23,
no. 3, pp. 472–482, 2015.

517