Overview of the Sixth Dialog System Technology
Challenge: DSTC6
Chiori Horia , Julien Perezb , Ryuichi Higasinakac , Takaaki Horia , Y-Lan
Boureaud , Michimasa Inabae , Yuiko Tsunomorif , Tetsuro Takahashig ,Koichiro Yoshinoh , Seokhwan KimiaMitsubishi Electric Research Laboratories, Cambridge, MA, USAbNaver Labs Europe, Grenoble, Francec
NTT Corporation, Japand
Facebook AI Research, New York, USAe
Hiroshima City University, Japanf
NTT DOCOMO, Inc., Japang
Fujitsu Laboratories. LTD., JapanhNara Institute of Science and Technology, Ikoma, Nara, Japani
Adobe Research, San Jose, CA, USA

Abstract
This paper describes the experimental setups and the evaluation results ofthe sixth Dialog System Technology Challenges (DSTC6) aiming to develop
end-to-end dialogue systems. Neural network models have become a recent
focus of investigation in dialogue technologies. Previous models required
training data to be manually annotated with word meanings and dialogue
states, but end-to-end neural network dialogue systems learn to directly output natural-language system responses without needing training data to be
manually annotated. Thus, this approach allows us to scale up the size of
training data and cover more dialog domains. In addition, dialogue systems
require a meta-function to avoid deploying inappropriate responses generated by themselves. To challenge such issues, the DSTC6 consists of three
tracks, 1. End-to-End Goal Oriented dialogue Learning to select system responses, 2. End-to-End Conversation Modeling to generate system responses
using Natural Language Generation (NLG) and 3. Dialogue Breakdown Detection. Since each domain has different issues to be addressed to develop dialogue systems, we targeted restaurant retrieval dialogues to fill slot-value inTrack 1, customer services on Twitter by combining goal-oriented dialogues
and ChitChat in Track 2 and human-machine dialogue data for ChitChat
in Track 3.
DSTC6 had 141 people declaring their interests and 23 teams submit-

Preprint submitted to Computer Speech & Language

January 17, 2019ted their final results. 18 scientific papers were presented in the wrap-up
workshop. We find the blending end-to-end trainable models associated tomeaningful prior knowledge performs the best for the restaurant retrieval for
Track 1. Indeed, Hybrid Code Network and Memory Network have been thebest models for this task. In Track 2, 78.5% of the system responses automatically generated by the best system were rated better than acceptable byhumans and this achieves 89% of the number of the human responses rated
in the same class. In Track3, the dialogue breakdown detection technologies
performed as well as human agreements, in both data-sets of English andJapanese.
Keywords: DSTC, end-to-end dialogue system, conversation model,
sequence-to-sequence model, Natural Language Generation, dialogue
breakdown

2

1. Introduction
Recent advancements in artificial intelligence have contributed to closing
the gap between the technologies and their uses in our daily life. One ofthe practical successes is that natural language dialogues have been used
as a means of human machine interface implemented in many consumer
devices. However, the current dialogue systems still have limited capabilities
of conducting natural interactions which is generally taken for granted inhuman-human conversations.
As a collaborative effort towards further advancements in dialogue technologies, Dialog State Tracking Challenges (DSTCs) have provided common
test beds for various research problems focusing on, but not limited to, thetask of dialog state tracking. Given the complexity of the dialogue phenomenon and the interest of the research community in a wider variety of
dialogue related problems, the DSTC has rebranded itself as Dialog System
Technology Challenges for its sixth edition.
Starting as an initiative to provide a common testbed for the task of
dialogue state tracking, the first Dialog State Tracking Challenge (DSTC)
was organized in 2013, followed by Dialog State Tracking Challenges 2 & 3
in 2014. More recently, Dialog State Tracking Challenge 4 and Dialog State
Tracking Challenge 5 have been completed in 2015 and 2016. Since 2014,
the challenge as evolved in several ways. First, from human-computer interactions, the challenges started to investigate human-human interactions.
Then, the event started to offer pilot tasks on Spoken Language Understanding, Speech Act Prediction, Natural Language Generation and End-to-end
System Evaluation which increased the reach of the challenge into the research community of dialogue systems and AI.
Given the remarkable success of the first five editions of the DSTC, andunderstanding both, the complexity of the dialogue phenomenon and theinterest of the research community in a wider variety of dialogue related
problems, the DSTC rebrands itself as ”Dialog System Technology Challenges” for its sixth edition. In this sixth edition of the DSTC, the call fortask proposals has resulted into three tracks, 1. End-to-End Goal Oriented
Dialogue Learning, 2. End-to-End Conversation Modeling and 3. Dialogue
Breakdown Detection as shown in Table 1. The objective of the tracks is toinvite interested organizations conduct dialogue related challenges in specificareas of research and under the umbrella of the DSTC.
These three tasks are selected from the viewpoints of impact and difficulty for dialogue research community. The first track for End-to-End Goal
Oriented Dialogue Learning task inherits previous dialogue state tracking
3Table 1: Specification of 3 Tracks of DSTC6

TargetObjective
Dialogue type
#dialogues
Evaluation metrics
Languages
TargetObjective
Dialogue type

#dialogues

Evaluation metrics
Languages
TargetObjective
Dialogue type
#dialogues
Evaluation metrics
Languages

Track 1
Sentence selection
To select the next utterance in a list of candidates
in the context of goal-oriented dialogue management
Task-oriented dialogue between user and system
for restaurant retrieval
40,000 generated dialogues and 4 Knowledge Bases
Mean Reciprocal Rank
English only
Track 2
Sentence generation
System response generation of natural language using
models trained from text dialogue data without intention
annotation
Task-oriented dialogue between user and human operator
for customer service
We used 1,024 twitter accounts for training and
100 and 116 for test and validation including
the domains of Airline, Car, Retail, Fast food chains, etc.
The dataset contains 888,201, 107,506, 2,000 dialogues
for train, development, test, respectively.
BLEU, Meteor, ROUGE L, CIDEr, Skip Thought,
Embedding Agerage, Vector Exream, Greedy Matching,
Human rating using Likert scale for response quality
English only
Track 3
Dialogue Breakdown detection
To detect whether a system utterance causes a dialogue
breakdown in a given dialogue context
Non-task-oriented (Chat-oriented) dialogue between userand system
English: 615 dialogues, Japanese: 150 dialogues(NB. for Japanese, an additional 1,546 dialogues fromprevious series of DBDCs could be used)
Classification-related metrics: accuracy, precision, recall,
F-measure, and distribution-related metrics:
Jensen-Shannon Divergence and Mean Squared Error
English/Japanesechallenges, especially the second challenge, with modern approaches of endto-end learning that try the direct prediction of next system action to theuser utterance and its dialogue history. The second track for End-to-End
Conversation Modeling is standing on other recent trends of the dialoguesystem area, which tries to model a conversation as directly generating sentences given a user query in the open domain. Due to the rises of neuralconversation modeling, the task attracts much attentions from the researchcommunity of dialogue system. The third track for Dialogue Breakdown De4

tection stands on more practical viewpoint. Controlling statistical dialogue
models to suppress unexpected responses becomes an important task due tothe development of statistical dialogue models, especially if we want to use
dialogue systems on real products.
Since each domain has different issues to be addressed to develop dialogue systems, we targeted restaurant retrieval dialogues to fill slot-value inTrack 1, customer services on Twitter by combining goal-oriented dialogues
and ChitChat in Track 2 and human-machine dialogue data for ChitChat
in Track 3. It is noted that ChitChat doesn not have a specific goal toaccomplish such as a slot filling task that sets values in a table of backend
systems. Furthermore, the content structure of ChitChat is not as restricted
and most answers can be accepted by humans.
1.1. Workshop summary and future DSTCs
The workshop for the Dialog System Technology Challenge (DSTC) was
held on December 10th, 2017 at long beach, CA, USA. The organizers had
pre-survey to know interests of dialogue community people, and 141 people declared their interests to the proposed three tasks. Finally, 23 teams
submitted their final results for tasks and 18 scientific papers are presented
in the workshop. The workshop also had 53 participants including on-site
registrations. Detailed results are described in the sections below. The
workshop also had many supporting organizations including sponsors, andthe challenge data was created with their supports.

52. End-to-End Goal Oriented Dialogue Learning (Track 1)
2.1. Introduction
Goal-oriented dialogue requires reasoning competencies that go beyond
language modeling. For example, asking questions to clearly define a user
request, querying Knowledge Bases (KB’s), interpreting results from queries
to display options to users or completing a transaction are some of theimportant competencies a dialogue system has to master in order to be
useful. On the one hand, such difficulties make it hard to ascertain how well
end-to-end dialogue models would do, and whether they are in a position
to replace traditional dialogue methods in a goal-directed setting. On theother hand, because end-to-end dialogue systems make no assumption on thedomain or dialogue state structure, they are holding the promise of easily
scaling up to new domains. This challenge aims to make it easier to analyze
the performance of end-to-end systems in a goal directed setting, using an
expanded version of the Facebook AI Research open resource proposed in[4]. The goal of the challenge is to assess the capabilities of the proposedsystems to fulfill a set of four basic tasks related to transactional dialogues.
The capability of accomplishing all four tasks on a single dialogue corpus
has been tested as a final task.
2.2. The task - Restaurant Reservation
The transactional dialogue simulation system is based on an underlying
KB. The facts contain the restaurants that can be booked and their properties queried. Each restaurant is defined by a type of cuisine (10 choices, e.g.,French, Thai), a location (10 choices, e.g., London, Tokyo), a price range(cheap, moderate or expensive), a rating (from 1 to more than 200), andother characteristics like dietary restrictions and atmosphere. For simplicity, we assume that each restaurant only has availability for a single party
size (2, 4, 6 or 8 people). Each restaurant also has an address and a phone
number listed in the KB.
The KB can be queried using API calls, which return the list of facts
related to the corresponding restaurants. Each query must contain a certain
number of slots: a location, a type of cuisine, a price range, a party size, andpossibly other required slots like dietary restriction, depending on the set
used. Each data file has the same set of required slots for every dialogue. A
query can return facts concerning one, several or no restaurant (dependingon the party size). Using the KB, conversations are generated in the format
shown in Figure 1. Each example is a dialogue comprising utterances froma user and a bot, as well as API calls and the resulting facts. dialogues
6are generated after creating a user request by sampling an entry for each of
the required slots: e.g. the request in Figure 1 is [cuisine: British, location:
London, party size: six, price range: expensive]. We use natural language
patterns to create user and bot utterances. There are more patterns for theuser than for the bot. Indeed, the user can use several ways to say something,while the bot always uses the same way to make it deterministic. Thosepatterns are combined with the KB entities to form thousands of different
utterances. We split types of cuisine and locations in half, and create two
KB’s, one with all facts about restaurants within the first halves and one
with the rest. In [4], the two KB’s had 4,200 facts and 600 restaurants
each (5 types of cuisine × 5 locations × 3 price ranges × 8 ratings). The
data provided here has been expanded to comprise more slots and thus yieldmany more restaurants, but the two KB still have disjoint sets of restaurants,
locations, types of cuisine, phones and addresses, while sharing all other sets
of values. We use one of the KB’s to generate train and test dialogues, using
only one of the extra slots in the queries. There are 4 sets of test dialogues:(1) one that uses the same KB as for the train dialogues, and the same setof slots in the queries; (2) one that uses the second KB (with disjoint setsof restaurants, locations, cuisines, phones and addresses), termed Out-OfVocabulary (OOV), but the same set of slots in the queries; (3) one thatuses the same KB as for the train dialogues, but one additional slot for
the queries; and (4) one that uses the second KB (OOV) and an additional
required slot.
For training, systems have access to the training examples and both KBs.
Evaluation is conducted on all four test sets. Beyond the intrinsic difficultyof each task, the challenge on the OOV test sets is for models to generalize to
new entities (restaurants, locations and cuisine types) unseen in any training
dialogue – something natively impossible for embedding methods. Ideally,models could, for instance, leverage information coming from the entities ofthe same type seen during training.
We generate five datasets, one per task. Training sets are relatively small(10,000 examples) to create realistic learning conditions. The dialogues fromthe training and test sets are different, never being based on the same user
requests. Thus, we test if models can generalize to new combinations offields.
2.3. End-to-end dialogue learning as sentence-selection
The task of end-to-end dialogue learning has been recently formalized
as next-sentence prediction. One of the main motivation of such approachis the abundance of human-to-human dialogue in industrial systems which
7

contrasts with the lack of annotated data due to the cost and challenge ofsuch process. Formally, a transaction dialogue system based on a sentence
selection model needs to choose, among a potentially large number of available utterances extracted from a corpus of dialogues, the most adequateanswer with respect to a current dialogue. Several challenges can be identified (1) dialogue representation (2) Reasoning capabilities (3) Back-end
system handling. A series of models have been proposed.
First, Long Short Term Memory (LSTM) [17] is a recurrent neural network that has recently known important success in most of the classic Natural Language Processing task. LSTM has become a common model forsentence encoding. In the context of utterance selection, several models
have leverage its expressive capability of learn sentence ranking models [26].
Second, Memory Networks [39] are a recent class of models that havealso been applied to a range of natural language processing tasks, including
question answering [5], language modeling and non-goal-oriented dialogue[8]. By first writing and then iteratively reading from a memory component(using layers called hops) that can store historical dialogues and short-term
context to reason about the required response, they have been shown toperform well on those tasks and to outperform some other end-to-end architectures based on simpler Recurrent Neural Networks.
Then, Hybrid Code Networks [48] (HCNs) learns an recurrent neuralnetwork but also allow a developer to express domain knowledge via software and action templates. Indeed, simple operations like sorting a list of
database results or updating a dictionary of entities can expressed in a few
lines of software, yet may take thousands of dialogues to learn. In addition,
this neural network can be trained with supervised learning or reinforcement
learning, by changing the gradient update applied.
Regarding the learning strategies, the use of pairwaise ranking loss has
been proposed. As an alternative, reinforcement learning has been investigated in order to leverage non-diffentiable loss through policy gradient [47].
More recently, adversarial loss has been studied and compared to humanchoice [27].
2.4. Description of the Dialogue Dataset
We broke down a goal-directed objective into several sub-tasks to test
some crucial capabilities that dialogue systems should have (and hence provide error analysis by design). All the tasks involve a restaurant reservation
system, where the goal is to book a table at a restaurant. Solving our tasks
requires manipulating both natural language and symbols from a KB. The
tasks are generated by a simulation. Grounded with an underlying KB of
8

restaurants and their properties (location, type of cuisine, etc.), these tasks
cover several dialogue stages and test if candidate models can learn various
abilities such as performing dialogue management, querying KB’s, interpreting the output of such queries to continue the conversation or dealing withnew entities not appearing in dialogues from the training set.
Task 1: Issuing API calls. A user request implicitly defines a query that cancontain from 0 to 4 of the required fields (sampled uniformly; in Figure 1,
it contains 3). The bot must ask questions for filling the missing fields andeventually generate the correct corresponding API call. The bot asks for
information in a deterministic order, making prediction possible.
Task 2: Updating API calls. Starting by issuing an API call as in Task
1, users then ask to update their requests. The order in which fields are
updated is random. The bot must ask users if they are done with their
updates and issue the updated API call.
Task 3: Displaying options. Given a user request, the KB is queried using
the corresponding API call and the resulting facts are added to the dialogue
history (if too many facts satisfy the call, a random subset is returned to
avoid overly lengthy data). The bot must propose options to users by listing
the restaurant names sorted by their corresponding rating (from higher tolower) until users accept. For each option, users have a 25% chance of accepting. If they do, the bot must stop displaying options, otherwise propose
the next one. Users always accept the option if this is the last remaining
one. We only keep examples with API calls retrieving at least 3 options.
Task 4: Providing extra information. Given a user request, we sample arestaurant and start the dialogue as if users had agreed to book a table
there. We add all KB facts corresponding to it to the dialogue. Usersthen ask for the phone number of the restaurant, its address or both, withproportions 25%, 25% and 50%, respectively. The bot must learn to use the
KB facts correctly to answer.
Task 5: Conducting full dialogues. For Task 5, we combine Tasks 1-4 to generate full dialogues just as in Figure 1. Unlike in Task 3, we keep examples
if API calls return at least 1 option instead of 3.
The dataset is organized in 5 JSON files corresponding to each of thetasks previously mentioned. Basically, a dialogue piece is followed by alist of next-utterance candidates. In the training set, the answer (the single
correct candidate) is provided. The goal is to rank the candidates. Precisions@{1,2,5} will be used to evaluate the models. These measures correspondto the probability of the correct utterance to be the 1st best, part of the92-best and 5-best hypotheses output by each model, respectively. Rankof candidate utterances will be 1-indexed. Evaluation uses per-response
accuracies. Evaluation is conducted in a ranking, not a generation, setting:at each turn of the dialogue, the participants have to test whether they
can predict bot utterances and API calls by selecting a candidate, not bygenerating it.1 Candidates are ranked from a set of candidate utterancesand API calls.
Table 2: Provided data for Track 1. Tasks 1-5 were generated using our simulator andshare the same KB. Each task have two test sets, one using the vocabulary of the
training set and the other using out-of-vocabulary words.

Dialogues

Average statistics

Datasets

Tasks 1-5 share thesame data source

Tasks
Number of utterances:- user utterances- bot utterances- outputs from API calls
Vocabulary size
Candidate set size
Training dialogues
Validation dialogues
Test dialogues

T1
12
5
7
0

T2
17
7
10
0

T3
T4
43
15
7
4
10
4
23
7
3,747
4,212
1,000
1,000
1,000

T5
55
13
18
24

2.5. Results
Table 3 introduces methods proposed during the challenge. End-to-End
Memory Network [40], Dynamic Memory Network [20] and Hybrid Code
Networks [48] were the main trainable building blocks of the proposed systems. In addition contextual rules were proposed to improve performances.
Table 4 details the results obtained for the participating teams. The two
first teams managed to solve the task by obtaining 1.0 precision in the testset. KB-2 introduced a novel request table slot (ambiance) in the test-set.
This slot is available in the knowledge base in both the train and test set.
3. End-to-End Conversation Modeling using NLG (Track 2)
3.1. Introduction
End-to-end training of neural networks is a promising approach to automatic construction of dialogue systems using a human-to-human dialogue
corpus. Recently, Vinyals et al. tested neural conversation models using
1[28] termed this setting Next-Utterance Classification.

10

Table 3: Methods implemented in submitted systems for Track 1.

Teams
1/4
2
3
5
6
7
8
9

MethodExtended Hybrid Code Networks
A hierarchical Long Short-Term Memory (LSTM) based ranking
module, a Conditional Random Field (CRF)
End-to-End Slot-Value Independent Recurrent Entity Network
Memory Network with Negative Sample
Memory Network with an extra output memory representation
named D-Layer with Knowledge based enhancement
End-to-End Memory Networks with named entities abstraction
and contextual numbering
Embedding projection of the text and candidate with rankloss
optimization
Quantized language model

Table 4: Summaries of the team performances for Track 1 using Precision1, Precision2
and Precision5.
Teams
team01/04
team02
team03
team05
team06
team07
team08
team09team10KB-1
1.000/1.000/1.000
1.000/1.000/1.000
0.984/0.997/0.999
0.619/0.692/0.831
0.890/0.946/0.995
0.994/0.998/1.000
0.663/0.798/0.912
0.997/0.997/0.998
0.371/0.435/0.652

KB-1-OOV
1.000/1.000/1.000
1.000/1.000/1.000
0.990/0.998/1.000
0.590/0.668/0.797
0.890/0.946/0.994
0.994/0.998/1.000
0.622/0.762/0.923
0.996/0.996/0.998
0.361/0.445/0.644

11

KB-2
1.000/1.000/1.000
1.000/1.000/1.000
0.927/0.958/0.990
0.600/0.671/0.822
0.739/0.810/0.932
0.959/0.982/0.986
0.516/0.669/0.835
0.965/0.967/0.980
0.299/0.385/0.639

KB-2-OOV
1.000/1.000/1.000
1.000/1.000/1.000
0.930/0.962/0.991
0.573/0.645/0.782
0.751/0.821/0.914
0.962/0.986/0.990
0.488/0.637/0.833
0.967/0.970/0.978
0.314/0.401/0.648

Figure 1: Task design of goal-oriented dialogue for Track 1. A user (in green)chats with a bot (in blue) to book a table at a restaurant. Models must predict bot
utterances and API calls (in dark red). Task 1 tests the capacity of interpretinga request and asking the right questions to issue an API call. Task 2 checks theability to modify an API call. Task 3 and 4 test the capacity of using outputs
from an API call (in light red) to propose options (sorted by rating) and to provide
extra-information. Task 5 combines everything.

OpenSubtitles [43]. Lowe et al. released the Ubuntu Dialogue Corpus [26]for research in unstructured multi-turn dialogue systems. Furthermore, theapproach has been extended to accomplish task oriented dialogues to provide
information properly with natural conversation. For example, Ghazvininejad et al. proposed a knowledge grounded neural conversation model [10],where the research is aiming at combining conversational dialogues with12

task-oriented knowledge using unstructured data such as Twitter data for
conversation and Foursquare data for external knowledge. However, thetask is still limited to a restaurant information service, and has not yet beentested with a wide variety of dialogue tasks. In addition, it is still unclear
how to create intelligent dialogue systems that can respond like a humanagent.
In consideration of these problems, we proposed a challenge track to
the 6th dialog system technology challenges (DSTC6) 2 . The focus of thechallenge track is to train end-to-end conversation models from human-tohuman conversation in order to accomplish end-to-end dialogue tasks for acustomer service. The dialogue system plays the role of a human agent andgenerates natural and informative sentences in response to users questions
or comments given a dialogue context.
3.2. Tasks
In this challenge track, a system has to generate sentence(s) in response
to a user input in a given dialogue context, where it can use external knowledge from public data, e.g. web data. The quality of the automaticallygenerated sentences is evaluated with objective and subjective measures tojudge whether or not the generated sentences are natural and informative
for the user (see Fig. 2).

Figure 2: Sentence generation and evaluation in the end-to-end conversation modelingtrack.

This track aims to generate system responses for Customer service dialogue using Twitter data:
Task A: Full or part of the training data will be used to train conversation
models.
Task B: Any open data, e.g. from web, are available as external knowledge
to generate informative sentences. But they should not overlap withthe training, validation and test data provided by organizers.
2

http://workshop.colips.org/dstc6/

13

Challenge attendees can select either A or B, or both. The tools to download
Twitter data and extract the dialogue text were provided to all attendees
at the challenge track in DSTC6 [18]. The attendees needed to collect thedata by themselves. Data collected before Sep. 1st, 2017 was available astrial data, and the official training, development and test data were collectedfrom Sep. 7th to 18th, 2017. The dialogues were used for the test set was
not disclosed until Sep. 25th.
3.3. Data Collection
3.3.1. Twitter data
In the Twitter task, we used dialogue data collected from multiple Twitter accounts for customer service. Each dialogue consisted of real tweets
between a customer and an agent. A customer usually asked a question orcomplained something about a product or a service of the company, andan agent responded to the customer accordingly. In this challenge, eachparticipant is supposed to develop a dialogue system that mimics agents
behaviors. The system will be evaluated based on the quality of generated
sentences in response to customers tweets. For the challenge, we provided adata collection tool to all participants so that they could collect the data bythemselves because Twitter does not allow distribution of Twitter data bya third party In this task, it is assumed that each participant continued tocollect the data from specific accounts in the challenge period. To acquirea large amount of data, the data collection needed to be done repeatedly,
e.g. by running the script once a day, because the amount of data we candownload is limited and older tweets cannot be accessed after they expire.
At a certain point of time, we provided an additional tool to extract subsets
of collected data for training, development (validation), and evaluation sothat all the participants were able to use the same data for the challenge.
Until the official data sets were fixed, trial data sets were available to develop dialogue systems, which were selected from the data collected by eachparticipant. But once the official data sets were determined, the systemneeded to be trained from scratch only using the official data sets.
Challenge attendees need to use a common data collection tool includedin the provided package. The trial data sets can be extracted from downloaded twitter dialogues using a data extraction script in the package. The
official data are collected through the period of Sep. 7th to 18th in 2017, using the data collection tool. The official training, development and test sets
can also be extracted using another data extraction script. Finally, the participants are supposed to collect the data sets summarized in Table 5. More

14

Table 5: Twitter data used for Track 2.

#dialogue
#turn
#word

training
888,201
2,157,389
40,073,697

development
107,506
262,228
4,900,743

test
2,000
5,266
99,389information can be found in ”https://github.com/dialogtekgeek/DSTC6End-to-End-Conversation-Modeling”.
We trained the data from 1024 twitter accounts for training, 100 accounts
for test and 116 account for validation, respectively. The business domain
contains Airline, Car, Retail, Fast food chains, etc. The lists of accounts
used for the challenge can be found in the following link3 .
3.4. Text Preprocessing
Twitter dialogues contain a lot of noisy text with special expressions
and symbols. Therefore, text preprocessing is important to clean up andnormalize the text. Moreover, all the participants need to use the same preprocessing at least for target references to assure fair comparisons between
different systems in the Challenge.
3.4.1. Twitter data
Twitter data contains a lot of specific information such as Twitter account names, URLs, e-mail addresses, telephone/tracking numbers and hashtags. This kind of information is almost impossible to predict correctly
unless we use a lot of training data obtained from the same site. To alleviate this difficulty, we substitute those strings with abstract symbols suchas <URL>, <E-MAIL>, and <NUMBERS> using a set of regular expressions. In
addition, since each tweet usually starts with a Twitter account name of therecipient, we removed the account name. But if such names appear withina sentence, we leave them because those names are a part of sentence. We
leave hashtags as well for the same reason. We also substitute user nameswith <USER> e.g.
hi John, can you send me a dm ?
→hi <USER>, can you send me a dm ?

3https://github.com/dialogtekgeek/DSTC6-End-to-End-ConversationModeling/tree/master/tasks/twitter officiala ccountn ames {train|dev|test}.txt

15

Table 6: Submitted systems for Track2.
Team (Entry)

Model typeObjective function

baselineteam 1 (1)team 1 (2)team 2 (1)team 2 (2)team 2 (3)LSTM

Cross entropy

2LSTM+LSTM
LSTM
2LSTM+LSTM

team 2 (4)team 2 (5)team 3 (1)team 3 (2)2LSTM+LSTM
2LSTM+LSTM
+HRED
LSTM
2LDTM+Atten.

Adversarial
Cross entropy
Adversarial + Cosine
Similarity
Adversarial
Cross entropy

team 3 (3)2LSTM+Atten.

team 3 (4)team 3 (5)GWGM+Atten.
SEARG

Cross entropy
Cross entropy on diversified data
Cross entropy on trial
data+
Cross entropy
Cross entropy

team 4 (1)LSTM

Cross entropy

team 5 (1)team 5 (2)team 5 (3)

LSTM
LSTM
LSTM

team 5 (4)team 5 (5)

LSTM
LSTM

MMI maxdiv
MMI maxBLEU
MMI mixed (maxdiv +maxBLEU)
MMI uniform
Cross entropy

Additional techniques

Example method
Example method
Example method

Paper

[46]

MBR
Systemcombination[23]

Knowledge
enhanced model
Word embedding
initialization
MERT
MERT
MERT

[2]

[9]

Greedysearchfor decoding

team 6 (1)+ Trial data cannot be used for official evaluation. The results are not officially accepted.

Since the user’s name can be extracted from the attribute information ofeach tweet, we can replace it. Note that the text preprocessing is not perfect,
and therefore there may remain original phrases, which are not replaced or
removed successfully. The text also includes many abbreviations, e.g. “plshlp”, special symbols, e.g. “(-:” and wide characters “ c ♥♣” including
4-byte Emojis. These are left unaltered.
3.5. Submitted Systems
We received 19 sets of system outputs for the Twitter task, from six
teams, and four system description papers were accepted [46, 23, 2, 9]. In
this section, we summarize the techniques used in the systems, including thebaseline system for the challenge track.
The baseline system is an LSTM-based encoder decoder in [18], but thisis a simplified version of [43], in which back-propagation is performed only up

16

to the previous turn from the current turn, although the state information
is taken over throughout the dialogue.
Table 6 shows the baseline and submitted systems with their brief specifications including model type, objective function, and additional techniques.
An empty specification means that the team did not submit any systemdescription paper to the DSTC6 workshop.
Most systems employed a LSTM or BLSTM (2LSTM) encoder and a
LSTM decoder. Some systems used a hierarchical encoder decoder (team 2 (5)and team 3 (5)) and attention-based decoder (team 3 (2–4)). Several typesof objective functions were applied for training the models, where cross entropy, adversarial method, cosine similarity, and maximum mutual information (MMI) were used solely or combined. The objective functions except
cross entropy were designed to increase the diversity of responses. This
hopefully led to more realistic and informative responses.
Furthermore, several additional techniques are introduced to improve
the response quality. In [46], an example-based method is used to return
real human responses if a similar context exists in the training corpus, andminimum Bayes risk (MBR) decoding is used to improve objective scores.
The knowledge enhanced encoder decoder [23] searches for relevant documents in the web using the keywords in the dialogue context, and therelevant documents are used to enhance the decoder. In [2], different types
of word-embedding vectors are used for initialization of the models.
3.6. Evaluation
Challenge participants were allowed to submit up to 5 sets of system outputs. The outputs were evaluated with objective measures such as BLEU
and METEOR, and also evaluated by rating scores collected by humans using Amazon Mechanical Turk (AMT). The human evaluators rate the system
responses in terms of naturalness, informativeness, and appropriateness.
3.6.1. Objective evaluation
For the challenge track, we used nlg-eval4 for objective evaluation ofsystem outputs, which is a publicly available tool supporting various unsupervised automated metrics for natural language generation. The supported
metrics include word-overlap-based metrics such as BLEU, METEOR, ROUGE L,
and CIDEr, and embedding-based metrics such as SkipThoughts Cosine
Similarity, Embedding Average Cosine Similarity, Vector Extrema Cosine
4

https://github.com/Maluuba/nlg-eval

17

Similarity, and Greedy Matching Score. Details of these metrics are described in [37].
We prepared 10 more references for a ground truth of each response byhumans to operate reliable objective evaluation. The references included areal human response in the Twitter dialogue and 10 human-generated responses. We asked 10 different Amazon Mechanical Turkers for each dialogue
to compose a sentence for the final response given the dialogue context. We
provided the real human response as an example and asked them to maketheir responses to be different from the example while keeping to the dialogue topic. We also asked them not to copy and paste the example intheir response. When multiple references are available, nlg-eval computes
the similarity between the prediction and all the references one-by-one, andthen selects the maximum value.
3.6.2. Subjective evaluation
We collected human ratings for each system response using 5 point Likert
Scale, where 10 different Turkers rated system responses given a dialogue
context. We listed randomly 21 responses below the dialogue context, which
consists of 19 submitted outputs, a baseline output, and a human response
for each dialogue.
The Turkers rated each response by 5 level scores as
Level
Very good
Good
Acceptable
Poor
Very poor

Score
5
4
3
2
1

we instructed to the Turkers to consider naturalness, informativeness, andappropriateness of the response for the given context. If there were identicalresponses in the list, we reduced them into one response so that they wererated consistently. The average score was computed for each system andreported in Table 7.
3.6.3. Results
Tables 7 and 8 show evaluation results of 21 systems: the 19 submitted
systems, the baseline and the reference. Systems are listed as team M (N),where M is the team index and N is an identifier for a particular system
submitted by that team. “Ext. Data” in the table denotes whether or
not the system used external data for training and/or testing, where only
team 3 (5) used external data (web data) for response generation.
18

Table 7: Evaluation results with word-overlapping-based objective measures based on 11
references and a subjective measure based on 5-level ratings for Track 2.

Team (Entry)
baselineteam 1 (1)∗team 1 (2)∗team 2 (1)team 2 (2)team 2 (3)team 2 (4)team 2 (5)team 3 (1)team 3 (2)team 3 (3)∗∗team 3 (4)team 3 (5)team 4 (1)team 5 (1)team 5 (2)team 5 (3)team 5 (4)team 5 (5)team 6 (1)∗reference

Ext.
Data

X

BLEU4

METEOR

ROUGE L

CIDEr

0.16190.15980.16230.15040.21180.18510.15320.22050.16020.17790.17410.13420.10920.17160.14800.09910.14480.12610.15750.27620.20410.20200.20390.18260.21400.20400.18330.22100.20160.20850.20240.17620.17310.20710.18130.16870.18390.17540.19180.1656

0.35980.36080.35670.3446
0.39530.37480.34690.41020.3606
0.38290.37030.33660.32010.3671
0.33880.3146
0.33750.33100.36580.34820.08250.07800.08280.08030.10600.09650.08000.12790.0782
0.09780.09940.09470.07020.08980.10250.07080.09400.09450.11120.1235

Human
Rating3.36383.44153.42973.44533.38943.47773.43813.43323.45033.52393.50823.51073.39193.44313.52093.30533.53963.45453.50972.99063.7245∗ Results are not officially accepted since any system description paper has not been submitted.
∗∗ Results are not officially accepted since the system was tuned with the trial data [23].

Table 9 shows cross-validation results of 11 ground truths generated byhumans. Each manually generated sentence was evaluated by comparing
with other 10 ground truths using leave-one-out method.
In most objective measures, the system of team 2 (5) achieved highest
scores, where the system employed MBR decoding for system combination.
This result indicates that explicit maximization of objective measures andthe complementarity of multiple systems bring significant improvement forthe objective measures.
On the subjective measure with human rating, the system of team 5 (3)achieved the best score (3.5396). Although there were no big differences
between the human rating scores, we can see that techniques for improving
human rating actually contributed to increase the rating scores. For example, adversarial training (team 2 (3)), use of diversified data (team 3 (2)),

19Table 8: Evaluation results with embedding-based objective measures based on 11 references and a subjective measure based on 5-level ratings for Track 2.

Team (Entry)
baselineteam 1 (1)∗team 1 (2)∗team 2 (1)team 2 (2)team 2 (3)team 2 (4)team 2 (5)team 3 (1)team 3 (2)team 3 (3)∗∗team 3 (4)team 3 (5)team 4 (1)team 5 (1)team 5 (2)team 5 (3)team 5 (4)team 5 (5)team 6 (1)∗reference

Ext.
Data

X

Skip
Thought
0.63800.6451
0.63860.64510.70750.6706
0.64630.66360.64740.62590.63480.61270.61320.65290.61310.59520.60250.61510.64570.6989Embedding
Average
0.91320.90900.90260.90700.92710.91160.90770.92510.90740.92010.89850.88020.89770.91060.90870.89960.90830.89840.90760.8018VectorExtrema
0.60730.60390.60710.59900.63710.61550.59990.64490.60310.61060.60000.59130.58700.60790.59280.56750.59150.58140.60750.5854

GreedyMatching
0.75900.75720.75870.75340.77470.76130.75440.78020.75670.76830.75730.74120.74200.75960.74330.72570.74330.73300.75280.7316

Human
Rating3.36383.44153.42973.44533.38943.47773.43813.43323.45033.52393.50823.51073.39193.44313.52093.30533.53963.45453.50972.99063.7245∗ Results are not officially accepted since any system description paper has not been submitted.
∗∗ Results are not officially accepted since the system was tuned with the trial data [23].

Table 9: Cross validation results of 11 references for Track2.
Original
Ref (1)
Ref (2)
Ref (3)
Ref (4)
Ref (5)
Ref (6)
Ref (7)
Ref (8)Ref (9)Ref (10)
Average

BLEU4METEOR

ROUGE LCIDEr

0.52640.27580.26260.26510.26830.26820.2786
0.27290.25930.25290.2707
0.29100.38850.23570.23130.23350.23580.23900.23230.23820.2256
0.23480.23640.24830.65590.45250.45010.44990.45090.44640.44760.45230.44300.44360.45270.46770.7566
0.36150.34770.3571
0.36220.36110.35770.36780.34880.34400.37500.3945

Skip
Thought
0.71600.70910.71420.70640.70390.71040.70050.70490.70820.72020.71050.7095

Embedding
Average
0.94830.93080.93000.93280.93130.93010.92910.93190.9306
0.93250.93330.9328VectorExtrema
0.76250.66430.6646
0.6626
0.6637
0.6632
0.6642
0.66870.6604
0.66210.66790.6731and MMI-based objective function with maximum diversity (team 5 (1), (3))

20

GreedyMatching
0.86790.79580.79510.79580.7951
0.79250.79250.7971
0.79210.7944
0.79680.8014

5	
  
4.5	
  
4	
  

Human	
  Ra;ng	
  

3.5	
  
3	
  
2.5	
  
2	
  
1.5	
  

re
fe
re
nc
e
te
am 	
  
_5
(3
)
te
am 	
  
_3
(2
)
te
am 	
  
_5
(1
)
te
am 	
  
_3
(4
)
te
am 	
  
_5
(5
)
te
am 	
  
_3
(3
)
te
am 	
  
_2
(3
)
te
am 	
  
_5
(4
)
te
am 	
  
_3
(1
)
te
am 	
  
_2
(1
)
te
am 	
  
_4
(1
)
te
am 	
  
_1
(1
)
te
am 	
  
_2
(4
)
te
am 	
  
_2
(5
)
te
am 	
  
_1
(2
)
te
am 	
  
_3
(5
)
te
am 	
  
_2
(2
)	
  
ba
se
lin
e	
  
te
am
_5
(2
)
te
am 	
  
_6
(1
)	
  

1	
  

Systems	  Figure 3: Mean and standard deviation of human rating score for Track 2.

Figure 4: Distribution of human scores averaged sentence by sentence for Track 2.

5

Human Rating

4
3
2
1
Ref

T5(3)

T3(2)

T5(1)

T3(4)

T5(5)

T3(3)

T2(3)

T5(4)

T3(1)

T2(1) T4(1)
Systems

T1(1)

T2(4)

T2(5)

T1(2)

T3(5)

T2(2)

Base

T5(2)

T6(1)

Figure 5: Distribution of human rating score for each level of scores for Track2.

21

Table 10: Example of System output for Track 2.
Example 1: The worst system does not show sympathy to the user’s troubles.
Context
U: it ’s currently raining on me inside of a bus @njtransit
Best
4.0
S: we ’re sorry to hear this . please submit a report here so we may
further assist you . <URL>
Worst
1.0
S: good luck !
Reference
3.9
S: hi <USER> , would you please provide me with your origin ,destination , bus line , 4 digit vehicle # & time of travel ?
Example 2: The best system shows much deeper sympathy to show how they will solve the user’s
troubles than the reference. The worst system totally misunderstands the negative context for the
positive one.
Context
U: flight canceled . stuck . rebooked 8hrs later in a middle seatand then asked to pay more to change it back to originally booked
aisle
Best
4.5
S: sorry to hear about this , <USER> . could you dm us yourconfirmation code so we can take a closer look into this for you ?
<URL>
Worst
1.2
S: we ’re happy to hear that , <USER> . we hope to see you again soon!
Reference
3.3
S: sorry to hear about your flight woes . did you know you may be
eligible for compensation ? visit <URL>
Example 3: The best system shows much deeper sympathy using emotional expressions for the user’s
troubles than the reference. The worst system totally misunderstands the negative context for the
positive one.
Context
U: savannah was the worst stay over the weekend !
Best
4.8
S: oh no ! what happened ? we ’d like to look into this for you .
please send us a direct message with your reservation number .
Worst
1.5
S: that ’s what we like to hear , <USER> ! we ’ll be sure to passthis on to the team . have a great day !
Reference
4.0
S: we are sorry to hear of your experience . please email us at<E-MAIL> so we can assist you w / this matter .

had some improvement compared to the other systems. The automatic generation have not yet reached the score of the reference (3.7245), but someof them were better responses than the reference in terms of human rating.
The conversation modeling techniques applied to this challenge track for
Twitter dialogues are better than we expected.
Figures 3-5 show the human ratings for each system in several ways.
The systems are shown in the same order on the X axis for all three figures.
Figure 3 shows the mean and the standard deviation of the human ratingsfor each system (across all responses and all raters for that system). Figure
4 shows the distributions of the mean human rating score for each sentence
for each system. Figure 5 shows the distribution of all human rating scoresfor each system across all sentences. In this Figure, the area for each scoreof the violinplot shows a count of the number of scores of each level on
the Likert scale. The ”Reference” system (at the far left of each figure)22

Table 11: Example of System output for Track 2.
Example 4: It’s difficult to compare the best response sentences assuming the information from aback-end system with the reference waiting for a response from a database.
Context
U: when are the kaws tee shirts gonna restock again online ?
Best
4.7
S: hi <USER> , we don ’t have a restock date at this time . pleasekeep an eye on our social media pages for updates .
Worst
1.3
S: it does not .
Reference
3.7
S: we are checking on the restock date for the kaws tee ’s rn .
Example 5: The benefits for users and company agents who wants to get more information are
different.
Context
U: honestly can ’t wait to use my lush bath bomb ♥Best
4.4
S: hi <USER> , we ’re so pleased to hear that you ’re happy with yourpurchase . enjoy your new goodies !
Worst
1.4
S: we have great post-workout meals ! give us a try next ! get 40 %off : <URL> <URL>
Reference
3.6
S: nice which bath bomb ? :D
Example 6: More options are good for users.
Context
U: is queen sold out ? you website seems unable to tell me if it is .
desperate to know ...
S: hi <USER> , limited tickets still available . thanks
U: thanks for the response . so , how does one procure them ?
Best
4.4
S: hi , you can order them from our website <URL> or by calling<NUMBERS> . thanks ,
Worst
1.1
S: hi <USER> ,Reference
3.9
S: you can follow this link to purchase > > > <URL>

23

is ratings for the sentences extracted from the original (Twitter) dialogue
data. The Reference system had the best human ratings: it had the highest
mean rating in Fig. 3, the highest median sentence rating in Fig. 4 and themost sentences rated as level 5 (”Very good”) in Fig. 5. The worst system(at the right) had a much lower mean rating, and a long tail of poorly rated
sentences.
78.5% (1570/2000) of the system responses automatically generated by
the best system were rated better than acceptable and 88.3% (1765/2000) of
human’s responses were rated in the same class. The best system achieved
89% of the upper-bound given by he human responses.
Tables 10 and 11 show examples of the system outputs. The humans
prefer more supportive responses with sympathy. The system responses are
sometimes rated better than the original human responses.
3.7. Summary
This article described the end-to-end conversation modeling track of the
6th dialog system technology challenges (DSTC6). We received 19 system
outputs from six teams, and evaluated them based on several objective measures and a human-rating-based subjective measure. The final results are
summarized in the article. Objective and subjective measures were not always consistent across systems. Some mismatches between objective andsubjective measures have also been reported in [22]. Deeper analysis will benecessary on these results to enable us to design a better objective function
and perform practical automatic evaluation.

24

4. Track 3: Dialogue Breakdown Detection
4.1. Introduction
Although voice agent services and smart speakers are beginning to appear on the market, the limited capabilities of these systems mean thathumans and machines still cannot converse as naturally as two humans.
The main problem is that systems typically make inappropriate utterancesthat lead to dialogue breakdowns. By dialogue breakdown, we mean a situation in a dialogue where users cannot proceed with the conversation [29].
To avoid this situation, technology for dialogue breakdown detection is essential because such technology will enable systems to avoid the creationof inappropriate utterances and also to identify dialogue breakdowns when
they occur and perform the necessary recovery procedures.
The task of dialogue breakdown detection [13] is to detect whether asystem utterance causes a dialogue breakdown in a given dialogue context.
The participants of the dialogue breakdown detection track (Track 3) ofthe Dialog System Technology Challenges (DSTC) developed a dialogue
breakdown detector that outputs a dialogue breakdown label (B: breakdown,
PB: possible breakdown, or NB: not a breakdown) and a distribution of theselabels. The definitions of the labels are defined as follows.
NB: It is easy to continue the conversation.
PB: It is difficult to continue the conversation smoothly.
B: It is difficult to continue the conversation.
Similar tasks to detect problematic situations in dialogue have been tackled mainly in task-oriented dialogue systems [45, 21, 25, 30]. In these studies,features, such as speech recognition results, language understanding results,
and prosodic information, are extracted from user/system utterances andused to train a model that can detect problematic situations (also called
miscommunication or hotspot). On the contrary, there are only a few studies that work on chat-oriented dialogue systems [49, 14], and their reported
accuracies are not that high (about 60% accuracy). In addition, these studies use a dataset of a single dialogue system, which raises the generality
issue. This track uses the data of multiple dialogue systems in multiple languages in order to obtain more general conclusions. Recently, a study thatannotates chat-oriented dialogue systems with three kinds of labels (invalid,
acceptable, valid), has been underway in the WOCHAT initiative5 ; however,5

http://workshop.colips.org/wochat/25

the labels are currently used for annotation purposes and not for automatic
detection [6, 7].
Dialogue breakdown detection is closely related to user satisfaction estimation in dialogue [44, 31, 15, 36, 1]. The difference is that these approaches
aim to estimate user satisfaction after the dialogue so that the system canbe improved later by analyzing unsatisfactory dialogues, whereas dialogue
breakdown detection aims to detect a system’s possible error in real-time sothat the system can avoid dialogue breakdown. Since underlying techniques
can be similar, we believe methods for user satisfaction estimation can also
be useful for dialogue breakdown detection.
In this track, we prepared datasets in two languages: English and Japanese.
We used four systems in English (NB. one of the datasets comprises dialoguesof several dialogue systems) and three systems in Japanese. Overall, seventeams participated in this track, in which five teams worked on English6
and four teams on Japanese with two teams working on both. In this paper,
we describe the task setting, datasets, and evaluation metrics as well as theresults of the submitted runs of the participants.
4.2. Task setting
The task of dialogue breakdown detection is to detect whether a systemutterance causes a dialogue to break down in a given dialogue context [13].
For example, consider the following piece of dialogue.
S1: It’s nice to go shopping alone.
U1: I agree. That’s nice.
S2: Shopping takes time.
U2: Window shopping is also fun.
S3: It’s fun to go shopping with somebody.
Here, system utterances are prefixed with S and user utterances with U. The
dialogue context is from S1 to U2, and the target utterance for dialogue
breakdown detection is S3 (underlined). In this example, S3 is likely to
be causing a dialogue breakdown because S3 contradicts S1. Therefore, adetector that classifies this as a dialogue breakdown will be regarded as
accurate. The reason we say “likely” is that a human’s decision on dialogue
breakdown is highly subjective, and it is difficult to decide on a single gold
6

Although six teams submitted their runs in English, one team did not submit a technical paper; therefore, their run is not regarded as an official run and thereby is not includedin this paper.

26

label. For this reason, we use many annotators for dialogue breakdown
annotation and opt for majority voting and their probability distribution as
references.
Dialogue Context:
S1: It’s nice to go shopping alone.
U1: I agree. That’s nice.
S2: Shopping takes time.
U2: Window shopping is also fun.
Succeeding system utterance:
S3: It’s fun to go shopping with somebody.

B

NB

PB

Input

Dialogue breakdown detector

B

Single dialogue
Probability distribution
breakdown label of breakdown labels

OutputFigure 6: Illustration of task setting for Track 3.

Given pairs of dialogue context and a succeeding system utterance, theparticipants submit, for each pair, (1) a single dialogue breakdown label
and (2) the probability distribution of breakdown labels (see Fig. 6). Note
that, although some utterances may exist after the target utterance, theycannot be used for prediction because, for this track, we focus on avoiding
dialogue breakdown rather than recovery. Currently many dialogue systems
hold multiple utterance candidates for utterance generation; for example,
retrieval-based methods typically have top-N candidates retrieved from a
database, and generation-based methods have distributions over utterances
which can be considered as multiple utterance candidates. We believe, good
dialogue breakdown detection technology will make it possible to select appropriate system utterances from the candidates, which will realize smoothconversation with users. We are not making little of recovery, but, we consider that it is more important for the system not to cause severe problems
in dialogue (i.e., dialogue breakdown) as a first step.
In this track, each participant can submit up to three runs for eachlanguage, so several parameters for dialogue breakdown detection can betested.

27

Table 12: Statistics of English datasets for Track 3

No. of sessions
No. of annotators
NB
PB
B (Breakdown)
Fleiss’ κ (NB, PB, B)
Fleiss’ κ (NB, PB+B)

TKTK-100
100
30
35.1%
27.6%
37.3%
0.14
0.21

Development data
IRIS-100 CIC-115
100
115
30
30
32.9%
28.9%
27.8%
29.8%
39.4%
41.3%
0.11
0.054
0.15
0.084

YI-100
100
30
34.8%
36.1%
29.1%
0.011
0.020

TKTK-50
50
30
44.3%
29.2%
26.5%
0.13
0.19

Evaluation data
IRIS-50 CIC-50
50
50
30
30
34.5%
29.1%
29.3%
39.3%
36.2%
31.6%
0.090
0.00400.13
0.0072

YI-50
50
30
35.4%
40.3%
24.3%
−0.0060−0.0043

Table 13: Statistics of Japanese datasets for Track 3

No. of sessions
No. of annotators
NB
PB
B
Fleiss’ κ (NB, PB, B)
Fleiss’ κ (NB, PB+B)

Chat dialogue corpus
init100
rest1046
100
1046
24
2 or 3
59.2%
58.3%
22.2%
25.3%
18.6%
16.4%
0.28
0.28
0.40
0.40

DBDC1
DVL/EVL
20/80
30
37.1%
32.2%
30.6%
0.20
0.27

DBDC2 (DVL/EVL)
DCM
DIT
IRS
50/50 50/50 50/50
30
30
30
39.8% 33.0% 37.4%
30.2% 27.4% 24.3%
29.9% 39.5% 38.3%
0.31
0.24
0.36
0.44
0.38
0.48

DBDC3 (EVL)
DCM
DIT
IRS
50
50
50
30
30
30
34.9% 25.3% 29.3%
34.2% 28.3% 23.8%
30.9% 46.4% 46.9%
0.24
0.14
0.27
0.32
0.20
0.37

4.3. Datasets
We distributed two sets of data to participants: one consisting of development (training) data and the other of evaluation (test) data. Tables 12 and13 show the statistics of the datasets for English and Japanese, respectively
7 . In this section, we first describe the English datasets then the Japanese
ones. The datasets are publicly available at https://dbd-challenge.github.
io/dbdc3/data/ .
4.3.1. Datasets for English for Track3
Development data. We provided four datasets: TKTK-100, IRIS-100, CIC115, and YI-100. The dialogue data for TKTK-100 and IRIS-100 were takenfrom the WOCHAT TickTock and IRIS datasets8 . The source of CIC-115
is the human evaluation round of the Conversational Intelligence Challenge
7

We thank Rafael E. Banchs, Zhou Yu, Valentin Malykh, Idris Yusupov, and Yury
Kuratov for graciously providing us with datasets and chatbots to make this track possible.
We also thank our sponsors, Denso IT Laboratories, Nextremer, Honda Research InstituteJapan, and NTT DOCOMO, Inc., for supporting our data collection. We also thank theJapanese Society for Artificial Intelligence (JSAI) for supporting the event.
8
http://workshop.colips.org/wochat/data/index.html

28

(CIC)9 . For YI-100, we newly collected dialogue sessions by crowd-sourcing.
All dialogue sessions were 20 or 21 utterances long and included 10 system
responses. The four datasets are described below.
TKTK-100 We selected 100 out of 206 sessions in the original WOCHAT
dataset. TickTock sessions start from user utterances (see [51] forthe details of TickTock). Dialogue breakdown annotation was doneusing a crowd-sourcing service, CrowdFlower10 . For each utterance,
30 workers annotated one of the dialogue breakdown labels. Level-2
workers11 from Australia, Canada, New Zealand, UK, and USA wererecruited, and we requested non-native English speakers to refrainfrom participating in our annotation tasks.
IRIS-100 We selected 100 out of 163 sessions in the original WOCHAT
dataset. The dataset was processed in the same manner as TKTK100. Original IRIS sessions start from system utterances; however, we
cut the first system utterances to make the data format identical tothat of TKTK-100 for annotation purposes (see [3] for the details of
IRIS).
CIC-115 This dataset comes from the human evaluation round of the CIC
and DeepHack Turing school-hackathon12 . The dialogues are available
at the CIC site13 . From all 2, 778 dialogue sessions, we selected 115
dialogues performed between a human and a bot. Within the 115 dialogues, 85 dialogues start with a system utterance, and 30 dialogues
start with a user utterance. As per the convention of CIC, each dialogue comes with a short paragraph, which is used as the context ofthe dialogue. The paragraphs are from the SQuAD dataset14 . Dialogue breakdown annotation was done using a crowd-sourcing service,
Amazon Mechanical Turk (AMT)15 , with 30 workers. When recruiting
the workers, we specified that the task requires native English skills
for the task instructions. The workers saw a short paragraph from the
SQuAD before dialogue breakdown annotation.
9

http://convai.io
https://www.crowdflower.com/
11
Higher quality: smaller group of more experienced, higher accuracy contributors basedon the definition of CrowdFlower.
12
http://turing.tilda.ws
13
http://convai.io/data/
14
https://rajpurkar.github.io/SQuAD-explorer/
15
https://requester.mturk.com
1029

YI-100 We collected 100 dialogue sessions by using a chatbot developed
at the Moscow Institute of Physics and Technology16 by using AMT.
A worker was assigned to have a chat with the system that was morethan 10 utterances. Dialogue breakdown annotation was also doneusing AMT with 30 workers. YI sessions start from system utterances.
Evaluation data. In the formal run, we distributed the following evaluation
data.
TKTK-50 We collected 50 dialogue sessions of TickTock by using AMT
in the same way as YI-100. Dialogue breakdown annotation was doneusing CrowdFlower.
IRIS-50 We selected 50 dialogue sessions from the held-out IRIS data graciously provided by the IRIS team. Dialogue breakdown annotation
was done using CrowdFlower.
CIC-50 The data source of CIC-50 is held-out dialogue data collected by
CIC after the human evaluation round. Dialogue breakdown annotation was done using AMT.
YI-50 We collected 50 dialogue sessions of YI in the same way as YI-100.
Dialogue breakdown annotation was done using AMT.
4.3.2. Datasets for Japanese
For the Japanese datasets, we did not create new development data because we had already created several datasets in previous evaluation workshops (two series of dialogue breakdown detection challenges (DBDCs) heldin Japan; DBDC1 and DBDC2). We briefly describe the development data
and the newly created evaluation data.
Development data.
Chat dialogue corpus This dataset has 1,146 dialogue sessions. The dialogues were collected using NTT DOCOMO’s chat API (DCM) [32].
One hundred dialogues (called init100) were annotated by 24 annotators, and the rest of the dialogues (called rest1046) were annotatedby 2-3 annotators. Dialogue breakdown annotation was done by the
researchers working on chat-oriented dialogue systems in Japan.
16

https://www.slideshare.net/sld7700/skillbased-conversational-agent-80976302

30

Development data for DBDC1 This dataset has 20 dialogue sessions.
The dialogues were collected using DCM via a crowd-sourcing service,
CrowdWorks17 , and were annotated by 30 annotators by using another crowd-sourcing service, Yahoo! Crowd-sourcing18 . All datasets
in DBDC1 and DBDC2 were collected and annotated in the same way.
Evaluation data for DBDC1 This dataset contains 80 dialogue sessions.
The dialogues were collected using DCM.
Development data for DBDC2 This dataset has 150 dialogue sessions.
The dialogues were collected using DCM, DIT (Denso IT Laboratories’
system) [42], and IRS (IR-status-based system from [34]) systems.
Evaluation data for DBDC2 This dataset has 150 dialogue sessions, 50
dialogues each were collected from DCM, DIT, and IRS.
Evaluation data. The evaluation data for Japanese contained 150 dialogue
sessions. We used the same procedure we used to create the evaluation data
for DBDC2.
4.4. Evaluation metrics
For this track, we used two types of evaluation metrics: classificationrelated and distribution-related.
4.4.1. Classification-related metrics
Classification-related metrics were used to evaluate the accuracy in classifying breakdown labels. The accuracy is calculated by comparing the output of the detector and the gold label determined by majority voting. We
use a threshold t to obtain the gold label, that is, we first find the majority
label and check if the ratio of that label is above t. If so, the gold label
becomes that label and NB otherwise. We used the following metrics.
• Accuracy: the number of correctly classified labels divided by the total
number of labels to be classified.
• Precision, Recall, F-measure (B): the precision, recall, and F-measurefor the classification of B labels.
1718

http://crowdworks.jp
http://crowdsourcing.yahoo.co.jp31• Precision, Recall, F-measure (PB+B): The precision, recall, and Fmeasure for the classification of PB + B labels; that is, PB and B
labels are treated as a single label.
These metrics can provide intuitive results about the detection of dialogue breakdowns because they are used to directly evaluate whether dialogue breakdowns are correctly classified. However, the choice of an appropriate t remains an open issue. In this track, we used t = 0.0, which means
simple majority voting.
4.4.2. Distribution-related metrics
Distribution-related metrics were used to evaluate the similarity of the
distribution of breakdown labels, which is calculated by comparing the predicted distribution of the labels with that of the gold labels. We calculatethese values for each utterance and use the mean values for evaluation. We
used the following metrics.
• Jensen-Shannon Divergence (JSD) (NB,PB,B): distance between thepredicted distribution of the three labels and that of the gold labels
calculated by using JSD.
• JSD (NB,PB+B): JSD when PB and B are regarded as a single label.
• JSD (NB+PB,B): JSD when NB and PB are regarded as a single label.
• Mean Squared Error (MSE) (NB,PB,B): distance between the predicted distribution of the three labels and that of the gold labels calculated by using MSE.
• MSE (NB,PB+B): MSE when PB and B are regarded as a single label.
• MSE (NB+PB,B): MSE when NB and PB are regarded as a single
label.
These metrics are used to compare the distributions of the labels; thus,
enabling a direct comparison with the gold labels. However, the results may
not be as easily interpretable as the classification-related metrics because
they do not directly translate to detection performance.

32

Table 14: Submitted runs in English summarized by their key features. Bold font indicatesthe best result. An underline indicates the second best result. MemN2N and ETR denote
End-to-End Memory Network and Extra Trees Regressor, respectively.
Run

Model

KTH run1 [24]KTH run2
KTH run3
PLECO run1 [35]PLECO run2
RSL17BD run1 [19]RSL17BD run2
RSL17BD run3
NCDS run1 [33]NCDS run2
NCDS run3
SWPD run1 [50]CRF Baseline
Majority Baseline

SVM
LSTM
LSTM
MemN2NMemN2N
ETR
ETR
ETR
RNN
RNN
RNN
Bi-LSTM
CRF

Word/Sentence
embedding
X
X
X
X
X
X
X
X
X
X
X

Bag ofwords

Utterance
similarity
X

Turn
index

X
X
X

X
X
X

X

X
X

Acc

JSD

0.33750.44150.42200.29500.29000.42650.43100.42000.36050.36550.35650.42950.42850.37200.44450.0481
0.32680.07140.07740.04320.04120.0426
0.04120.0412
0.06680.08070.44090.0393

Table 15: Submitted runs in Japanese summarized by their key features for Track 3. Bold
font indicates the best result. An underline indicates the second best result. EoR denotes
Ensemble of Regressors.
Run

Model

PLECO run1 [35]PLECO run2
PLECO run3
RSL17BD run1 [19]RSL17BD run2
RSL17BD run3
OUARS run1 [41]OUARS run2
OUARS run3
NTTCS run1 [38]NTTCS run2
NTTCS run3
CRF Baseline
Majority Baseline

MemN2NMemN2NMemN2N
ETR
ETR
ETR
CNN
CNN, LSTM
CNN, LSTM
EoR
EoR
EoR
CRF

Word/Sentence
embedding
X
X
X
X
X
X
X
X
X
X
X
X

Bag ofwords

X

Utterance
similarity

Turn
index

X
X
X

X
X
X

X
X
X

X
X
X

Acc

JSD

0.50780.50120.53450.38900.39390.4024
0.55390.54300.55930.6036
0.59930.59270.52960.47210.11490.1011
0.09810.15640.15640.1582
0.09100.09890.09320.07140.07170.07410.38710.13114.5. Results
Overall, seven teams participated in this track: five teams worked on
English, four teams on Japanese, and two teams working on both. Eachteam could submit up to three runs for each language. We had 12 runs forEnglish and 12 for Japanese.
33

Table 16: Overall Results of Classification (English) for Track 3
Run
KTH run2
RSL17BD run2
SWPD run1
CRF Baseline
RSL17BD run1
KTH run3
RSL17BD run3
Majority Baseline
NCDS run2
NCDS run1
NCDS run3
KTH run1
PLECO run1
PLECO run2

Accuracy
0.44150.43100.42950.42850.42650.42200.42000.37200.36550.36050.35650.33750.29500.2900Run
PLECO run1
PLECO run2
CRF Baseline
KTH run1
KTH run3
Majority Baseline
SWPD run1
RSL17BD run2
NCDS run3
RSL17BD run1
RSL17BD run3
KTH run2
NCDS run2
NCDS run1

F1 (B)
0.3636
0.35650.35430.34870.33730.33430.32100.32010.31980.31260.30250.29490.20970.2076

Run
Majority Baseline
PLECO run1
PLECO run2
KTH run1
RSL17BD run2
RSL17BD run3
RSL17BD run1
NCDS run3
SWPD run1
CRF Baseline
KTH run3
KTH run2
NCDS run1
NCDS run2

F1 (PB+B)
0.89270.87440.87080.84230.84000.83570.83290.8046
0.76270.76220.75920.74400.34580.3397

Table 17: Overall Results of Jensen-Shannon Divergence (JSD) (English) for Track 3
Run
Majority Baseline
NCDS run1
RSL17BD run2
NCDS run2
RSL17BD run3
RSL17BD run1
KTH run2
NCDS run3
PLECO run1
PLECO run2
SWPD run1
KTH run3
CRF Baseline
KTH run1

JSD (NB,PB,B)
0.03930.04120.04120.04120.0426
0.04320.0481
0.06680.07140.07740.08070.32680.44090.4445Run
Majority Baseline
NCDS run1
NCDS run2
RSL17BD run2
RSL17BD run3
RSL17BD run1
KTH run2
PLECO run1
NCDS run3
SWPD run1
PLECO run2
KTH run3
KTH run1
CRF Baseline

JSD (NB,PB+B)
0.02370.02480.02480.02560.02580.02630.02670.04270.0436
0.04380.04820.1892
0.23430.2687Run
RSL17BD run2
RSL17BD run3
RSL17BD run1
NCDS run2
NCDS run1
Majority Baseline
KTH run2
SWPD run1
NCDS run3
PLECO run1
PLECO run2
KTH run1
KTH run3
CRF Baseline

JSD (NB+PB,B)
0.02250.02430.02470.02540.02540.02570.02620.04440.04880.05350.05650.20580.2166
0.2985

Tables 14 and 15 summarize the submitted runs of the participants in
English and Japanese, respectively, together with their accuracy and JSD
for reference. The tables indicate that many teams used neural networks(NNs) and word embeddings.
Tables 16, 17, and 18 show the results for English and Tables 19, 20,
and 21 for Japanese. The values in these tables are macro-averages over thesystems for each language.19
We also show the results of two baselines. One is a majority baseline that19
See [12] for the detailed results for each dataset in each language. Note that theresults of the Japanese runs are slightly different from [12] because, due to a technicaltrouble, several dialogues were not processed in our evaluation script. This problem wasfixed and the tables here show the updated results; there was no change in the order of
performance.

34

Table 18: Overall Results of Mean Squared Error (MSE) (English) for Track 3
Run
Majority Baseline
NCDS run1
NCDS run2
RSL17BD run2
RSL17BD run3
RSL17BD run1
KTH run2
PLECO run1
NCDS run3
PLECO run2
SWPD run1
KTH run3
CRF Baseline
KTH run1

MSE (NB,PB,B)
0.02240.02370.02370.02410.02500.02540.0281
0.04150.04370.04480.04710.16700.21850.2240Run
Majority Baseline
NCDS run1
NCDS run2
RSL17BD run2
RSL17BD run3
RSL17BD run1
KTH run2
PLECO run1
SWPD run1
PLECO run2
NCDS run3
KTH run3
KTH run1
CRF Baseline

MSE (NB,PB+B)
0.02780.02870.02880.03010.03010.03070.03150.04550.05010.05090.06770.16640.17520.2171MSE
RSL17BD run2
Majority Baseline
NCDS run2
NCDS run1
RSL17BD run3
RSL17BD run1
KTH run2
SWPD run1
NCDS run3
PLECO run1
PLECO run2
KTH run1
KTH run3
CRF Baseline

Table 19: Overall Results of Classification (Japanese) for Track 3
Run
Accuracy Run
F1 (B) Run
NTTCS run1
0.6036
NTTCS run1
0.6645 RSL17BD run2
NTTCS run2
0.5993
NTTCS run2
0.6612 OUARS run3
NTTCS run3
0.5927
NTTCS run3
0.6569 RSL17BD run1
OUARS run3
0.5593OUARS run3
0.6294 PLECO run2
OUARS run1
0.5539OUARS run1
0.6246 PLECO run1
OUARS run2
0.5430OUARS run2
0.6155 RSL17BD run3
PLECO run3
0.5345PLECO run3
0.6143 OUARS run2
CRF Baseline
0.5296PLECO run2
0.6027 OUARS run1
PLECO run1
0.5078PLECO run1
0.6003 NTTCS run3
PLECO run2
0.5012
CRF Baseline
0.5777 PLECO run3
Majority Baseline
0.4721
Majority Baseline 0.4448 NTTCS run1
RSL17BD run3
0.4024
RSL17BD run3
0.2822 NTTCS run2
RSL17BD run2
0.3939
RSL17BD run2
0.2762 CRF Baseline
RSL17BD run1
0.3890
RSL17BD run1
0.2598 Majority Baseline

MSE (NB+PB,B)
0.0246
0.02640.02700.02700.02710.02750.02860.04970.0572
0.06320.06730.14760.20440.2578F1 (PB+B)
0.82540.81950.81520.81170.80940.80660.80390.79890.7982
0.7941
0.7836
0.78290.77100.5549outputs the most frequent dialogue breakdown label in each system’s development data for English and development and evaluation data of DBDC2
for Japanese with averaged probability distributions. The other is a conditional random field (CRF)-based baseline that labels utterance sequenceswith the three breakdown labels by using CRFs. The features used werewords in a target utterance and the previous utterances. For the probability
distribution, the probability of 1.0 is given to the label determined by the
CRFs.
It can be seen from the tables that, for English, NCDS and RSL17BD
performed well in terms of JSD and MSE. In terms of classification-related
metrics, KTH, RSL17BD, and PLECO seem to have performed well. For
Japanese, except for F1(PB+B), NTTCS outperformed the other teams
35

Table 20: Overall Results of Jensen-Shannon Divergence (JSD) (Japanese) for Track 3
Run
NTTCS run1
NTTCS run2
NTTCS run3
OUARS run1
OUARS run3
PLECO run3
OUARS run2
PLECO run2
PLECO run1
Majority Baseline
RSL17BD run2
RSL17BD run1
RSL17BD run3
CRF Baseline

JSD (NB,PB,B)
0.07140.07170.0741
0.09100.0932
0.0981
0.09890.10110.11490.13110.15640.15640.15820.3871

Run
NTTCS run1
NTTCS run2
NTTCS run3
OUARS run1
OUARS run3
PLECO run3
OUARS run2
PLECO run2
RSL17BD run2
RSL17BD run3
RSL17BD run1
PLECO run1
Majority Baseline
CRF Baseline

JSD (NB,PB+B)
0.04900.04920.0511
0.06350.06570.07010.07040.07220.07490.07850.07860.08330.10680.2409

Run
NTTCS run1
NTTCS run2
NTTCS run3
OUARS run1
OUARS run3
OUARS run2
PLECO run3
PLECO run2
PLECO run1
Majority Baseline
RSL17BD run1
RSL17BD run2
RSL17BD run3
CRF Baseline

JSD (NB+PB,B)
0.04280.04290.04470.05550.05690.06050.0606
0.06270.07390.0752
0.09650.0971
0.09820.2798

Table 21: Overall Results of Mean Squared Error (MSE) (Japanese) for Track 3
Run
NTTCS run1
NTTCS run2
NTTCS run3
OUARS run1
OUARS run3
OUARS run2
PLECO run3
PLECO run2
PLECO run1
Majority Baseline
RSL17BD run1
RSL17BD run2
RSL17BD run3
CRF Baseline

MSE (NB,PB,B)
0.03850.03860.04000.04750.04800.05100.05290.05580.06440.06820.08960.08970.09070.2013

Run
NTTCS run1
NTTCS run2
NTTCS run3
OUARS run1
OUARS run3
OUARS run2
PLECO run3
PLECO run2
RSL17BD run2
RSL17BD run1
RSL17BD run3
PLECO run1
Majority Baseline
CRF Baseline

MSE (NB,PB+B)
0.04820.04840.05080.06040.06130.06590.06930.07440.07750.08150.08200.08610.10230.2117

Run
NTTCS run1
NTTCS run2
NTTCS run3
OUARS run1
OUARS run3
OUARS run2
PLECO run3
PLECO run2
Majority Baseline
PLECO run1
RSL17BD run1
RSL17BD run2
RSL17BD run3
CRF Baseline

MSE (NB+PB,B)
0.04730.04740.04900.05840.05880.06240.06600.06900.07500.08280.10320.10410.1046
0.2477

followed by OUARS.
We conducted a multiple comparison test (Steel-Dwass test) to examine whether the submitted runs were better than the baselines. For English, when we focused on accuracy, KTH run2 (NN-based method) significantly outperformed the majority baseline with p < 0.01, followed by
RSL17BD run1 RSL17BD run2, SWPD run1 with p < 0.05. No runs outperformed the CRF baseline. For F1(B) and F1(PB+B), no runs significantly outperformed the best-performing baseline, that is, the CRF-based
baseline for F1(B) and the majority baseline for F1(PB+B). With regards tothe distribution-related metrics, only RSL17BD run2 significantly outperformed the majority baseline (p < 0.05) for MSE(NB+PB, B). For Japanese,
only NTTCS run1 and run2 (methods based on the Ensemble of Regres-36

sors (EoR)) significantly outperformed the CRF-based baseline (p < 0.01).
For F1(B) and F1(PB+B), no runs significantly outperformed the bestperforming baseline. With regards to the distribution-related metrics, mostruns outperformed both the CRF-based and majority baselines. Overall,
we had better results for Japanese compared to those for English, probably
because of the low inter-annotator agreement in the English data.
We calculated the upper bound (see Table 22). For accuracy, we left-out
1 annotator from the 30 annotators and calculated whether that left-out
annotator can predict the majority label by the other 29 annotators. For
JSD and MSE, we split the 30 annotators into two groups and calculated
the JSD and MSE between those groups; this process was iterated 100 times
to obtain the average. The values were calculated using the datasets of thistrack.
Table 22: Upper bound (human-level) accuracy, JSD, and MSE. The values are comparedagainst the top run in each language for Track 3.

Accuracy
JSD(NB,PB,B)
MSE(NB,PB,B)

EnglishUpper bound Top run
0.429
0.442
0.03300.03930.02770.0224

Japanese
Upper bound Top run
0.625
0.60360.02890.07140.0206
0.0385In terms of accuracy, we are already at the upper bound in both languages (English: 0.442/0.429 = 1.03, Japanese: 0.6036/0.625 = 0.97). For
JSD and MSE in English, since the majority baseline was the top run, it isdifficult to discuss the results; we need to improve the quality of the data,such as inter-annotator agreement, so that the proposed method will at least
outperform the baselines. For Japanese, there still seems to be some gap
between the top run and upper bound.
This analysis indicates that, with current methods, we can detect dialogue breakdown with good accuracy, though there seems to be some room
for improvement for JSD and MSE. In other words, it is currently possible
to guess the majority decision but it is hard to predict the distribution of
dialogue breakdown labels.
4.6. Summary and Future work
We described the dialogue breakdown detection track in the Sixth Dialog
System Technology Challenge (DSTC6). We prepared both English andJapanese datasets, and seven teams competed using methods for detecting
37

dialogue breakdown. We obtained promising results and interesting methods
for dialogue breakdown detection. The NN-based method performed thebest in English and a method that used EoR was the best in Japanese. It
seems that NN-based approaches are struggling for lack of training data,
which is the general problem in dialogue processing with neural models. We
achieved a human-level accuracy for both languages although there is stillsome room for improvement in JSD and MSE.
In order to improve JSD and MSE, a more fine-grained understandingof breakdowns/errors in chat-oriented dialogue systems will be important.
There have been studies to create a taxonomy of errors in chat-oriented
dialogue systems [11, 16]. We want to incorporate such studies so as toimprove the prediction of distributions. Another interesting direction will
be to focus on a particular type of error, such as contradiction and common
sense violation, which may be rare in human-machine dialogue but is difficult
to handle by current natural language processing techniques.
5. Conclusion
We reported the results of the sixth Dialog System Technology Challenges (DSTC6). 23 teams challenged with one or two of the 3 tracks aiming
to select system responses for restaurant retrieval dialogues to fill slot-valuein Track 1, generate system responses using NLG for customer service on
twitter by combining goal-oriented dialogues and chitchat in Track 2 andhuman-machine dialogue data for ChitChat in Track 3. We find the blending end-to-end trainable models associated to meaningful prior knowledge
performs the best for the restaurant retrieval in Track 1. Indeed, Hybrid
Code Network and Memory Network have been the best models for thistask. Regarding customer service response generation using NLG in Track2, 78.5% of the system responses automatically generated by the best system were rated better than acceptable and this achieves 89% of the numberof the human responses rated in the same class. The responses rated aspoor and very poor don’t show sufficient sympathy to user’s troubles. The
worst responses generated by systems represented the opposite sentimentand was never observed in the human customer responses. Future works
contains customer sentiment understanding. In Track 3 for dialogue breakdown detection, the best system achieved a human-level accuracy for bothlanguages, English and Japanese.

38

References[1] Ando, A., Masumura, R., Kamiyama, H., Kobashikawa, S., Aono, Y.,
2017. Hierarchical LSTMs with joint learning for estimating customer
satisfaction from contact center calls. In: Proc. Interspeech. pp. 1716–
1720.
[2] Bairong, Z., Wenbo, W., Zhiyu, L., Chonghui, Z., Shinozaki, T., 2017.
Comparative analysis of word embedding methods for DSTC6 end-toend conversation modeling track. In: Proceedings of the 6th Dialog
System Technology Challenges (DSTC6) Workshop.
[3] Banchs, R. E., Li, H., 2012. IRIS: a chat-oriented dialogue system basedon the vector space model. In: Proc. ACL 2012 System Demonstrations.
pp. 37–42.
[4] Bordes, A., Boureau, Y.-L., Weston, J., 2017. earning end-to-end goaloriented dialog. arXiv preprint arXiv:1605.07683.
[5] Bordes, A., Usunier, N., Chopra, S., Weston, J., 2015. Large-scale
simple question answering with memory networks. arXiv preprint
arXiv:1506.02075.
[6] Charras, F., Duplessis, G. D., Letard, V., Ligozat, A.-L., Rosset, S.,
2016. Comparing system-response retrieval models for open-domain andcasual conversational agent. In: Proc. Workshop on Chatbots and Conversational Agent Technologies.
[7] Curry, A. C., Rieser, V., 2016. A subjective evaluation of chatbot engines.
[8] Dodge, J., Gane, A., Zhang, X., Bordes, A., Chopra, S., Miller, A.,
Szlam, A., Weston, J., 2016. Evaluating prerequisite qualities for learning end-to-end dialog systems. In: Proc. of ICLR.
[9] Galley, M., Brockett, C., 2017. The MSR-NLP system at dialog system technology challenges 6. In: Proceedings of the 6th Dialog System
Technology Challenges (DSTC6) Workshop.
[10] Ghazvininejad, M., Brockett, C., Chang, M.-W., Dolan, B., Gao, J.,
Yih, W.-t., Galley, M., 2017. A knowledge-grounded neural conversation model. arXiv preprint arXiv:1702.01932.

39

[11] Higashinaka, R., Funakoshi, K., Araki, M., Tsukahara, H., Kobayashi,Y., Mizukami, M., 2015. Towards taxonomy of errors in chat-oriented
dialogue systems. In: Proc. SIGDIAL. pp. 87–95.
[12] Higashinaka, R., Funakoshi, K., Inaba, M., Tsunomori, Y., Takahashi,T., Kaji, N., 2017. Overview of dialogue breakdown detection challenge
3. In: Proc. Dialog System Technology Challenges Workshop (DSTC6).
[13] Higashinaka, R., Funakoshi, K., Kobayashi, Y., Inaba, M., 2016. The
dialogue breakdown detection challenge: Task description, datasets,
and evaluation metrics. In: Proc. LREC. pp. 3146–3150.
[14] Higashinaka, R., Meguro, T., Imamura, K., Sugiyama, H., Makino, T.,
Matsuo, Y., 2014. Evaluating coherence in open domain conversational
systems. In: Proc. Interspeech. pp. 130–133.
[15] Higashinaka, R., Minami, Y., Dohsaka, K., Meguro, T., 2010. Modelinguser satisfaction transitions in dialogues from overall ratings. In: Proc.
SIGDIAL. pp. 18–27.
[16] Higashinaka, R., Mizukami, M., Funakoshi, K., Araki, M., Tsukahara,
H., Kobayashi, Y., 2015. Fatal or not? Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems. In: Proc. EMNLP.
pp. 2243–2248.
[17] Hochreiter, S., Schmidhuber, J., 1997. Long short-term memory. Neuralcomputation 9 (8), 1735–1780.
[18] Hori, T., 2017. DSTC6 end-to-end conversation modeling track:
tools and baseline system. https://github.com/dialogtekgeek/
DSTC6-End-to-End-Conversation-Modeling.
[19] Kato, S., Sakai, T., 2017. RSL17BD at DBDC3: Computing utterance
similarities based on term frequency and word embedding vectors. In:
Proc. Dialog System Technology Challenges Workshop (DSTC6).
[20] Kumar, A., Irsoy, O., Su, J., Bradbury, J., English, R., Pierce,
B., Ondruska, P., Gulrajani, I., Socher, R., 2015. Ask me anything:
Dynamic memory networks for natural language processing. CoRR
abs/1506.07285.
URL http://arxiv.org/abs/1506.07285

40

[21] Lendvai, P., Van Den Bosch, A., Krahmer, E., Swerts, M., 2002. Improving machine-learned detection of miscommunications in humanmachine dialogues through informed data splitting. In: Proc. the ESSLLI Workshop on Machine Learning Approaches in Computational Linguistics. pp. 1–15.
[22] Liu, C.-W., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L.,
Pineau, J., 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response
generation. arXiv preprint arXiv:1603.08023.
[23] Long, Y., Wang, J., Xu, Z., Wang, Z., Wang, B., 2017. A knowledge
enhanced generative conversational service agent. In: Proceedings of
the 6th Dialog System Technology Challenges (DSTC6) Workshop.
[24] Lopes, J., 2017. How generic can dialogue breakdown detection be? theKTH entry to DBDC3. In: Proc. Dialog System Technology Challenges
Workshop (DSTC6).
[25] Lopes, J., Chorianopoulou, A., Palogiannidi, E., Moniz, H., Abad,
A., Louka, K., Iosif, E., Potamianos, A., 2016. The SpeDial datasets:
datasets for spoken dialogue systems analytics. In: Proc. LREC.
[26] Lowe, R., Pow, N., Serban, I., Pineau, J., 2015. The ubuntu dialogue
corpus: A large dataset for research in unstructured multi-turn dialoguesystems. In: SIGDIAL Conference. The Association for Computer Linguistics, pp. 285–294.
URL http://aclweb.org/anthology/W/W15/W15-4640.pdf[27] Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., Pineau, J., 2016.
On the evaluation of dialogue systems with next utterance classification.
In: SIGDIAL Conference. The Association for Computer Linguistics,
pp. 264–269.
URL http://aclweb.org/anthology/W/W16/W16-3634.pdf[28] Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., Pineau, J., 2016.
On the evaluation of dialogue systems with next utterance classification.
arXiv preprint arXiv:1605.05414.
[29] Martinovsky, B., Traum, D., 2003. The error is the clue: Breakdown inhuman-machine interaction. In: Proc. ISCA Workshop on Error Handling in Spoken Dialogue Systems. pp. 11–16.

41

[30] Meena, R., Lopes, J., Skantze, G., Gustafson, J., 2015. Automatic detection of miscommunication in spoken dialogue systems. In: Proc.
SIGDIAL. pp. 354–363.
[31] Möller, S., Engelbrecht, K.-P., Schleicher, R., 2008. Predicting the quality and usability of spoken dialogue services. Speech Communication
50 (8-9), 730–744.
[32] Onishi, K., Yoshimura, T., 2014. Casual conversation technology
achieving natural dialog with computers. NTT DOCOMO Technical
Jouranl 15 (4), 16–21.
[33] Park, C., Kim, K., Kim, S., 2017. Attention-based dialog embedding
for dialog breakdown detection. In: Proc. Dialog System Technology
Challenges Workshop (DSTC6).
[34] Ritter, A., Cherry, C., Dolan, W. B., 2011. Data-driven response generation in social media. In: Proceedings of the Conference on Empirical
Methods in Natural Language Processing.
[35] Saito, A., Iki, T., 2017. End-to-end character-level dialogue breakdown
detection with external memory models. In: Proc. Dialog System Technology Challenges Workshop (DSTC6).
[36] Schmitt, A., Schatz, B., Minker, W., 2011. Modeling and predicting
quality in spoken human-computer interaction. In: Proc. SIGDIAL.
pp. 173–184.
[37] Sharma, S., El Asri, L., Schulz, H., Zumer, J., 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR abs/1706.09799.
URL http://arxiv.org/abs/1706.09799[38] Sugiyama, H., 2017. Dialogue breakdown detection based on estimating
appropriateness of topic transition. In: Proc. Dialog System Technology
Challenges Workshop (DSTC6).
[39] Sukhbaatar, S., Szlam, A., Weston, J., Fergus, R., 2015. End-to-end
memory networks. Proceedings of NIPS.
[40] Sukhbaatar, S., Szlam, A., Weston, J., Fergus, R., 2015. Weakly supervised memory networks. CoRR abs/1503.08895.
URL http://arxiv.org/abs/1503.08895
42

[41] Takayama, J., Nomoto, E., Arase, Y., 2017. Dialogue breakdown detection considering annotation biases. In: Proc. Dialog System Technology
Challenges Workshop (DSTC6).
[42] Tsukahara, H., Uchiumi, K., 2015. System utterance generation by label
propagation over association graph of words and utterance patterns foropen-domain dialogue systems. In: Proc. PACLIC. pp. 323–331.
[43] Vinyals, O., Le, Q., 2015. A neural conversational model. arXiv preprint
arXiv:1506.05869.
[44] Walker, M., Kamm, C., Litman, D., 2000. Towards developing generalmodels of usability with PARADISE. Natural Language Engineering
6 (3-4), 363–377.
[45] Walker, M., Langkilde, I., Wright, J., Gorin, A., Litman, D., 2000.
Learning to predict problematic situations in a spoken dialogue system:
Experiments with How May I Help You? In: Proc. NAACL. pp. 210–
217.
[46] Wang, W., Koji, Y., Harsham, B., Hori, T., Hershey, J. R., 2017. Sequence adversarial training and minimum Bayes risk decoding for endto-end neural conversation models. In: Proceedings of the 6th Dialog
System Technology Challenges (DSTC6) Workshop.
[47] Wen, T.-H., Miao, Y., Blunsom, P., Young, S. J., 2017. Latent intention
dialogue models. In: Precup, D., Teh, Y. W. (Eds.), Proceedings ofthe 34th International Conference on Machine Learning, ICML 2017,
Sydney, NSW, Australia, 6-11 August 2017. Vol. 70 of Proceedings of
Machine Learning Research. PMLR, pp. 3732–3741.
URL http://jmlr.org/proceedings/papers/v70/[48] Williams, J. D., Asadi, K., Zweig, G., 2017. Hybrid code networks:
practical and efficient end-to-end dialog control with supervised andreinforcement learning. CoRR abs/1702.03274.
URL http://arxiv.org/abs/1702.03274[49] Xiang, Y., Zhang, Y., Zhou, X., Wang, X., Qin, Y., 2014. Problematic
situation analysis and automatic recognition for chinese online conversational system. In: Proc. CLP. pp. 43–51.
[50] Xie, Z., Ling, G., 2017. Dialogue breakdown detection using hierarchicalbi-directional LSTMs. In: Proc. Dialog System Technology Challenges
Workshop (DSTC6).
43

[51] Yu, Z., Xu, Z., Black, A. W., Rudnicky, A. I., 2016. Strategy andpolicy learning for non-task-oriented conversational systems. In: Proc.
SIGDIAL. pp. 404–412.

44