IEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 2014

1497

PAPER

Special Section on Advances in Modeling for Real-world Speech Information Processing and its Application

Utilizing Human-to-Human Conversation Examples for a Multi
Domain Chat-Oriented Dialog System
Lasguido NIO†a) , Nonmember, Sakriani SAKTI†b) , Member, Graham NEUBIG†c) , Nonmember, Tomoki TODA†d) ,and Satoshi NAKAMURA†e) , Members

SUMMARY
This paper describes the design and evaluation of amethod for developing a chat-oriented dialog system by utilizing realhuman-to-human conversation examples from movie scripts and Twitterconversations. The aim of the proposed method is to build a conversational agent that can interact with users in as natural a fashion as possible,
while reducing the time requirement for database design and collection. A
number of the challenging design issues we faced are described, including(1) constructing an appropriate dialog corpora from raw movie scripts andTwitter data, and (2) developing an multi domain chat-oriented dialog management system which can retrieve a proper system response based on thecurrent user query. To build a dialog corpus, we propose a unit of conversation called a tri-turn (a trigram conversation turn), as well as extraction
and semantic similarity analysis techniques to help ensure that the content
extracted from raw movie/drama script files forms appropriate dialog-pair(query-response) examples. The constructed dialog corpora are then utilized in a data-driven dialog management system. Here, various approaches
are investigated including example-based (EBDM) and response generation
using phrase-based statistical machine translation (SMT). In particular, we
use two EBDM: syntactic-semantic similarity retrieval and TF-IDF based
cosine similarity retrieval. Experiments are conducted to compare and contrast EBDM and SMT approaches in building a chat-oriented dialog system, and we investigate a combined method that addresses the advantages
and disadvantages of both approaches. System performance was evaluatedbased on objective metrics (semantic similarity and cosine similarity) andhuman subjective evaluation from a small user study. Experimental results
show that the proposed filtering approach eﬀectively improve the performance. Furthermore, the results also show that by combing both EBDM
and SMT approaches, we could overcome the shortcomings of each.
key words: dialog corpora, response generation, example-based dialog
modeling, semantic similarity, cosine similarity, machine translation

1.

Introduction

The continuous growth of information technology is having an increasingly large impact on many aspects of ourdaily lives. The issue of communication via speech between human beings and information-processing machines
is also becoming more important [1]. A common dream isto realize a technology that allows humans to communicate
or have dialogs with machines through natural and spontaneous speech.
Manuscript received September 28, 2013.
Manuscript revised January 18, 2014.
†
The authors are with the Graduate School of Information Science, Nara Institute of Science and Technology, Ikoma-shi, 630–
0192 Japan.
a) E-mail: lasguido.kp9@is.naist.jpb) E-mail: ssakti@is.naist.jpc) E-mail: neubig@is.naist.jpd) E-mail: tomoki@is.naist.jpe) E-mail: s-nakamura@is.naist.jp
DOI: 10.1587/transinf.E97.D.1497

Natural language dialogue systems have so far mostly
focused on two main dialogue genres: goal-oriented dialog(such as ATIS flight reservation [2], DARPA Communicator dialog travel planning [3]) and non-goal-oriented dialog(such as chatterbot systems like Eliza [4] or Alice [5]). Dialog systems can also be described by the amount of humanintervention used in their construction, ranging from entirely
hand-made to completely data-driven. Seminal work often
limited interactions to a specific scenario (e.g. a Rogerian
psychotherapist [4]) or were based on complex, knowledgerich rule-based systems for generating responses, which required large amounts of human eﬀort to create or add newrules [5].
Example-based dialog modeling (EBDM) is datadriven approach for deploying dialog systems. It uses dialog examples that are semantically indexed to a database.
Proper responses for user input are generated based on these
dialog examples. Consequently, to achieve good coverageon various types of natural conversation, recording of a large
data set of real human-to-human conversation is necessary,
which is tedious and time consuming. Common solutions
use handmade scripted dialog scenarios which may resultin unnatural conversations. Some studies also propose constructing dialog examples from available log databases, such
conversation between human subjects and the Wizard ofOZ (WOZ) system [6], or human-to-human conversation in
Twitter [7].
However, covering all possible patterns that may exist
in real human-to-human conversation is still diﬃcult. Currently, most EBDM systems rely on either canned responses
by providing error messages [8] or templates for generation which may result a completely incomprehensible response [9]. On the other hand, SMT has been successfully
used to address various NLP tasks [10]–[12]. The investigation of SMT as an approach for response generation has also
been introduced by [13].
The goal of our work is to create a dialog agent that caninteract with the user in as natural a fashion as possible. In
this paper we focus on addressing the following two main
challenging issues:
Corpus construction: We propose a method to utilizing
human-to-human conversation examples from movies
and and Twitter data. The aim is to gain insights in
how to build a conversational agent that can interact
with users in as natural a way as possible, while reduc-c 2014 The Institute of Electronics, Information and Communication Engineers
Copyright IEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 20141498

ing the time requirement for database design and collection. Then, to help ensure that the content extracted
from raw movie/drama script files consists of appropriate dialog-pair (query-response) examples, we propose
using a unit called a tri-turn for extraction, as well as
semantic similarity analysis techniques.
Dialog management: We investigate various data-driven
approaches to dialog management, including two
EBDM techniques (syntactic-semantic similarity retrieval and TF-IDF based cosine similarity retrieval)and using phrase-based SMT to learn a conversational
mapping between user-input and system-output dialogpairs. We also propose a simple, but eﬀective way
to perform system combination of example-based andSMT-based techniques into one dialog management
framework. Experimental results demonstrate that our
combined system shows promise for overcoming theshortcomings of each approach.
In the next session, we first describe how to constructdialog corpora from raw movie scripts and Twitter data. We
then briefly describe the proposed dialog management systems based on EBDM and SMT technique in Sect. 3. Experimental set-up is briefly describe in Sects. 4. Then, a detailed evaluation of our proposed approaches is presented
in Sect. 5. Some related research works are discussed in
Sect. 6. Finally, conclusions are drawn in Sect. 7.
2.

Construction of Dialog Corpora

The dialogue corpora that we constructed in this study arebased on dialog-pair sentences. As the movie scripts andTwitter data used in this work contain very diﬀerent types oftext, we go through diﬀerent processes to construct them.
A movie scripts is a conversational manuscript that portray the conversations between and actions of between actors in a movie. Figure 1 (a) illustrates an example of one
movie scene with four actors talking to each other. The corresponding raw movie scripts that are available from the web
are usually written in HTML files shown in Fig. 1 (b). The
dialog between actors are arranged in chronological order.
Consequently, the conversation dialog contained in movie
scenes does not have a clear indication of which utterances
are responses to a particular utterance. Therefore, it is important to find a solution that is able to construct appropriate dialog-pair examples from raw movie scripts files. As
shown in Fig. 1 (c), we perform dialog tri-turn extraction tofind the candidate of dialog-pairs.
In contrast to the movie scripts data, the text on Twitter often represents real conversations between two or more
people. Therefore, we do not perform dialog tri-turn extraction in order to extract the related dialog-pair sentences.
Instead, the challenge with handling Twitter data is how toensure the integrity of the sentences. In this case, it is necessary to filter out sentence pairs that are not likely to be
useful for training the system.
Unifying both data sources into one dialog corpus, we

Fig. 1

Dialog corpora construction from movie script.

define two basic types of information about each dialog: actor and utterances. The utterances are the actual content ofeach dialog turn in the movie scripts or tweets. The actor
refers to the character name in the movies, or the name of
the Twitter user that posted each tweet. This actor and utterance information will be utilized to construct the dialog
corpus.
The details of dialog corpus construction, as illustrated
in Fig. 2, consists of three main steps: (1) preprocessing,which remove unnecessary information and normalize thetext, (2) a dialog-pairs extraction, which ensures that theconversation is between two people talking each other, andNIO et al.: UTILIZING HUMAN-TO-HUMAN CONVERSATION EXAMPLES FOR A MULTI DOMAIN CHAT-ORIENTED DIALOG SYSTEM

1499

Fig. 3Fig. 2

Dialog corpora construction from movie script.

(3) a semantic similarity calculation, which ensure that theeach query-response pair is semantically related. We describe the details of each step in the following sections.
2.1

Preprocessing

Preprocessing of the movie scripts is done by transforming
raw HTML files into easily readable text format. Since we
use a variety source of movie scripts that had various formats, we implemented several parsing algorithms to fetch
the information from the raw movie conversation. Furthermore, unnecessary explanatory information about the movie
scenes is also removed.
For the Twitter data, preprocessing removes information about person identity, hash tags, and URLs. Next, forboth data sets all the words in the sentences are labeled withparts of speech (POS) and named entities (NE). Finally, toensure the integrity of the Twitter data, English language filtering † and non-standard word (NSW) normalization [14] isalso performed.
2.2

Dialog Turn Extraction

To ensure that the dialog example database contains only
query-response pairs, we propose a simple and intuitive
method for selection of the dialog data: trigram turn sequences, or tri-turn. A tri-turn is defined as three turns in
a conversation between two actors X and Y that has the pattern X-Y-X. In other words, within a tri-turn the first and last
dialog turn are performed by the same actor and the second
dialog turn is performed by the other actor. Next the queryresponse pairs are made by separating the tri-turn pattern
X-Y-X into two pairs, X-Y and Y-X.
We found that when we observed this pattern, in thegreat majority of the cases this indicated that the first andsecond utterances (X-Y pair), as well as the second and thirdutterances (Y-X pair), formed a proper input-response pair†

http://search.cpan.org/˜ambs/Lingua-Identify-0.51/

Example of a tri-turn with two actors.

as shown in the c-a-c tri-turn in Fig. 3. However, noisy cases
which contain uncorrelated turns still exist (see the b-a-b triturn in Fig. 3), this happens because the speakers are not actually speaking to each-other. To address this problem, we
perform further filtering using the semantic similarity measure described in the following section.
2.3 Semantic Similarity
Semantic similarity (similar approach introduced in [15]) is
used to ensure a strong semantic relationship between eachdialog turn in the dialog-pair data. As shown in Eq. (1), it
computes the similarity between WordNet †† synsets in eachdialog turn. The dialog-pairs with high similarity are thenextracted and included into database. S syn1 and S syn2 respectively is a group of WordNet synsets for each word in thesentence S 1 and S 2 that are linked by a complex network oflexical relations. The similarity of sentence pair X-Y where
S 1 = X and S 2 = Y can be obtained by calculating the relations between S syn1 and S syn2 . Where |S syn1 ∩ S syn2 | is anumber of co-occurring WordNet synsets and |S syn1 |+|S syn2 |is a total number of eﬀective WordNet synsets.
sem sim (S 1 , S 2 ) =3.

2 × |S syn1 ∩ S syn2 ||S syn1 | + |S syn2 |(1)

Dialog Management Systems

The overview of our dialog management method system isshown in Fig. 4. It utilizes dialog templates within threestates: the greeting state, free-chat state, and farewell state.
For every user input query, dialog management will forward
to the response generators systems. The response generatorsthen provide various possible responses, which finally voted
by the system for the best responses and forwards it as ansystem responses output. In this study, various approachesof response generators are investigated including syntacticsemantic similarity and TF-IDF based cosine similarity retrievals of EBDM techniques, as well as phrase-based SMT
approaches which are discussed in the next section. Note
that, as current focus here is to investigate the optimal technique for retrieving a proper system response based on thecurrent user query, the utilizing of user history and dialog
context will not be discussed in this paper.
††

http://wordnet.princeton.edu/

IEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 2014

1500
Table 1

Total characters involved in one movie.
Characters

Percentage

1 – 10
11 – 20
21 – 30
31 – 40
41 – 50≥ 51

4.40%
28.62%
16.94%
18.02%
14.71%
17.30%

is used to retrieve a proper system response. To increase theemphasis on important words, an additional TF-IDF weighting (Eq. (4)) is performed to construct the term vector [16].
|T |T FIDF(t, T ) = Ft,T log(4)
DFt

Fig. 4

We define Ft,T as a term frequency ‘t’ in a sentence ‘T ’,
and DFt as a total number of sentence in the query-response
pairs that contain term ‘t’.

Dialog management systems.

3.2 SMT-Based
3.1

Example-Based Dialog Modeling

In EBDM, the system chooses a response from the examples
stored in the dialog database. In order to do so, it computes
a similarity measure between the user input and the querypart of the query-response pairs, and returns the associatedresponse for the query with the highest similarity. In thiswork, we examine syntactic-semantic similarity and TF-IDF
based cosine similarity as two similarity measures for use in
EBDM.
3.1.1

Syntactic-Semantic Similarity Retrievalsim(S 1 , S 2 ) = α[sem sim (S 1 , S 2 )] + (1 − α)[cos sim (S 1 , S 2 )]
(2)
where
S1 · S2.
 S1   S2 

Ŝ = arg maxP(S | T ).

(3)

In this work, we assumed that the semantic factor ismore important than syntactic factor, so we set the interpolation coeﬃcient α to be 0.7.

Experimental Set-Up

In this paper, the movie script dialog is collected throughFriends TV show scripts † , The Internet Movie Script
Database †† , and The Daily Script ††† . This resulted in a total of 1,786 movie scripts with 1,042,288 dialog-pairs. After performing dialog turn extraction and semantic similarity
filtering, the total number of dialog-pairs is 86,719. Table 1
shows the total diﬀerent characters involved in one movie.
Mostly around 28.62% collected movie scripts is played by
11 – 20 diﬀerent characters. Only 4.40% collected movie
scripts is played by 1 – 10 diﬀerent characters. Therefore
besides the main characters, the movie scripts usually composed by a cameo e.g. “a man in the radio”, “man 1”, “radio”, and so on. These cameo characters contribute to increase the characters variation on a single movie show.
For the Twitter data, Twitter tweets was collected
through the Twitter API †††† , resulting in a total of 1,076,447
dialog-pairs. After performing language filtering and semantic similarity filtering, the total number of dialog-pairs
†

3.1.2

TF-IDF Based Cosine Similarity Retrieval

Cosine similarity over the term vector as described in Eq. (3)

(5)

S

4.

A proper system response is retrieved by measuring bothsemantic and syntactic relations. These two measures are
combined using linear interpolation as shown in Eq. (2).
This value is calculated over the user input sentence (S 1 )and every input examples on database (S 2 ). These values arecalculated using semantic similarity in WordNet as a semantic factor and cosine similarity (Eq. (3)) over part-of-speech(POS) tag vectors as a syntactic factor.

cos sim (S 1 , S 2 ) =The second approach we tested was based on SMT [13].
With this approach, the dialog-pair data is treated as a parallel corpus for training an SMT system. Given the trained
SMT system, the user dialog is treated as an input and“translated” into the system response. The system response
is chosen to be system output S of maximal probability
given the user input T

http://ufwebsite.tripod.com/scripts/scripts.htm
http://imsdb.com/†††
http://dailyscript.com/††††
http://dev.twitter.com††

NIO et al.: UTILIZING HUMAN-TO-HUMAN CONVERSATION EXAMPLES FOR A MULTI DOMAIN CHAT-ORIENTED DIALOG SYSTEM

1501
Various approach on the system response generation.

Example of diﬀerent system responses.

Input

Shall we eat at my house?

Approach

Response 1
Response 2
Response 3
Response 4

Sorry, I ate already.
Yes, sure.
Of course, will you cook?
Great! But, where is your house?

EBDM- Syntactic-Semantic Similarity Retrieval- TF-IDF based Cosine Similarity Retrieval
SMT
Combination EBDM and SMT

was reduced to 67,500 and 7,048 respectively.
After extracting all the dialog-pairs, we randomly selected 500 and 1000 query-response pairs from Twitter andmovie conversation dialog, respectively, as a test set (thequery-response pairs are denoted as Qtest , Rtest ). Then,the remaining dialogs will be used as dialog examples for
EBDM, and training data for SMT (the dialog-pairs here aredenoted as Qtrain , Rtrain ).
The natural language processing tools and Wordnet
synsets used were provided by NLTK toolkit † , and theexample-based TF-IDF based cosine similarity retrieval was
performed using Apache Lucene †† . For the SMT approach,
Moses ††† was used to build the translation model and perform translation for the dialog system. Here, four-gram language models built with the Kneser-Ney smoothing and thelexicalized distortion model were used.
Given a query from the test set (Qtest ), the EBDM will
search the closest query examples using syntactic-semantic
similarity retrieval: sim(Qtest , Qtrain ) or TF-IDF based cosine similarity retrieval: cos(Qtest , Qtrain ), and output a response of Routput . However, as there may be a number ofvalid system responses for a single user query (see the example in Table 2), it is not trivial to evaluate the system
performance. Therefore, in this study, we attempt to assess each output response within semantic, syntactic, andhuman judge aspect. We investigate the performance of oursystem by utilizing TF-IDF based cosine similarity andsyntactic-semantic based similarity to represent the semantic and syntactic aspect, and subjective evaluation to represent the human judge aspect. For objective evaluation,
the Routput are evaluated by computing similarity with Rtest :sim(Routput , Rtest ) and cos(Routput , Rtest ). Also when performing subjective evaluation in user-system interaction, Qtest
are given, and the users are evaluate the naturalness of
Routput in comparison with Rtest .
5.

Evaluation

5.1

Evaluation of Dialog Corpora Construction

To demonstrate the eﬀect of semantic similarity and tri-turn
filtering in our data, we compare our system performancewith and without the tri-turn filtering. In this comparison,
the TF-IDF based cosine similarity and syntactic-semantic
similarity retrieval methods were used to retrieve responses†

Table 3

Table 2

http://nltk.org
http://lucene.apache.org/†††
http://statmt.org/moses/

Abbreviation

sssr
csm
smt
comb

Fig. 5Filtering eﬀect on the movie data.

Fig. 6Filtering eﬀect on the Twitter data.

(Table 3 shows the various approaches to retrieve the system
response). Semantic similarity filtering could improve theperformance significantly over the tri-turn filtering. However, the application of the tri-turn filtering has a role in reducing the amount of training example while maintaining
the evaluation score result. On the other hand, the diﬀerence in the amount of training examples resulting from thefiltering process also aﬀect the response retrieval time. Figure 5 and 6 depict average system evaluation score improvement and response time per input query for each applied
filter. The number in the horizontal axis shows the amountof dialog-pairs in each filtering steps.
5.2 Evaluation of the Dialog Management System††

Objective evaluation presented in Fig. 7 and Fig. 8 is per-

IEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 2014

1502

Fig. 7 Objective evaluation results on the movie data by various datadriven approaches.

formed using TF-IDF based cosine similarity and syntacticsemantic based similarity (with p < 0.05). The results
reveal that, within EBDM approach, TF-IDF based cosine similarity retrieval (denoted as “csm”) gives a better
response score than syntactic-semantic similarity retrieval(denoted as “sssr”). The csm approach exceeds the sssr approach because this approach utilizes cosine similarity over
the TF-IDF vector, while the sssr approaches compute cosine similarity over the POS tag vector. Furthermore, thetri-turn and semantic similarity filtering methods manage toincrease the response score.
Comparing the best EBDM approach “csm” againsts
the SMT approach “smt”, csm always give a better performance than smt. Analyzing the data in more detail, we
found that csm is better in handling when dialogs close to
Qtest exists in Qtrain , while smt can provide a better output
when there is no dialogs in Qtrain similar with Qtest . Combining both approaches (denoted as “comb”) in which thesystem uses EBDM if the similarity between user input anddialog examples exceeds given threshold, and responds with
SMT output otherwise, could overcome the shortcomingsof each approach. The objective evaluation results on system combination given various thresholds are presented inFig. 9. The results reveal that the best system is provided
by the combined system. The optimum score shown here isachieved by 0.4 and 0.6 for movie and Twitter data respectively.
A cross-domain evaluation between movie and Twitter data is also performed. In this experiment we use

Fig. 8 Objective evaluation results on the Twitter data by various datadriven approaches.

Fig. 9 Objective evaluation results of the combined system given various
threshold.

the “comb” retrieval approach to retrieve responses withinmovie and Twitter filtered data. Both the Twitter and movie
test data is tested in the movie, Twitter, and combined movie

NIO et al.: UTILIZING HUMAN-TO-HUMAN CONVERSATION EXAMPLES FOR A MULTI DOMAIN CHAT-ORIENTED DIALOG SYSTEM

1503

Fig. 10 Combination retrieval approach on cross-domain using
syntactic-semantic similarity as an evaluation metric.

and Twitter database. The results of the cross-domain evaluation can be seen in the Fig. 10.
In the subjective evaluation, 40 human annotators wereasked to give a naturalness score between 1-5 of the system
output, with higher scores indicating that the system is giving a natural and relevant system response to the user input.
Each person assesses 140 randomly selected query-response
pairs that were evenly distributed over all systems. The results of this evaluation are shown in Fig. 11. We also prepare
a dummy system as a baseline which outputs a response bysimply repeating the user input, i.e. user-input: “How areyou?”, then the system’s output is also: “How are you?”.
For greeting conversations, this simple approach may work.
But, for the other cases, the system may result in a completely incomprehensible response.
Along with objective evaluation, the results show thatthe csm approach significantly outperform the smt approach.
This may indicate that the smt responses consists of several matching phrases with the reference, but have not yet
reached the naturalness of real human responses. For instance, for a query input “I’ll call you back.”, the smt system will responded “I call me back.”. Because this sentence
is incomprehensible, many people will prefer the dummysystem response “I’ll call you back.” instead of the smt response. This factor seems to aﬀect the system combination
as well, where it reduced the score slightly compared withthe csm approach. Furthermore, the results of the subjective evaluation also demonstrate slightly higher scores onfiltered data. This shows that the tri-turn and semantic similarity filtering methods manage to increase the naturalnessof the response.
6.

Related Work

There have been a number of related works into EBDM
and SMT-based approaches for response generation in datadriven chat [17]–[19]. Work by [8] proposes a generic dialog
modeling framework for a multi-domain dialog systems tosimultaneously manage goal-oriented and chat dialogs for
information access and entertainment. However, the chatoriented dialog only includes small talk limited to 10 top-

Fig. 11 Subjective evaluation result on the movie and Twitter data by
various data-driven approaches.

ics of daily conversation. Furthermore, if the system cannot
find similar examples to determine the next system action,
it simply defines a “No Example” output error and provides
an in-coverage example of what the user could say at thecurrent dialog state. Finally, [20] introduce IRIS (Informal
Response Interactive System), a chat oriented dialog systemusing movie scripts that is based on a similar cosine similarity in vector space model. However, the system did not filter
any uncorrelated consecutive scripts in the movie data, and,as the authors state, this causes failures and diminishes theability to maintain a consistent conversation.
Despite the relatively large interest in data-driven approaches for chat or response generation, there is surprisingly little work comparing and contrasting approaches ordata sources. In this work, we attempt make an empiricalevaluation that will contrast these approaches and provide areference for future development in the area. In addition, tothe best of our knowledge, our method to combine examplebased and SMT-based response generation in dialog modeling, and utilizing tri-turn in dialog systems is also diﬀerent
from previous work.

IEICE TRANS. INF. & SYST., VOL.E97–D, NO.6 JUNE 2014

1504

7.

Conclusion

In this work, we investigated several approaches to build adata-driven chat-oriented dialog systems. The proposed triturn extraction and semantic similarity filtering are able toextract dialog-pair examples from multi-speaker dialog of
raw movie scripts and Twitter data. Experimental results
also reveal that that the tri-turn and semantic filtering improve the objective evaluation metrics score (TF-IDF based
cosine similarity and syntactic-semantic similarity evaluation metrics). Furthermore we also find out that this approach also helps to reduce the retrieval response time by
reducing dialog examples in training set. It relies on no explicit domain knowledge, and should therefore be applicableto other dialog applications with little or no modification.
Furthermore, we compared three diﬀerent approaches
to giving responses using an example database. We foundthat the example-based approach is very good in handling the queries which are similar to the examples inthe database, but achieves poor performance in handlingthe queries which are far diﬀerent from existing examples.
On the other hand, SMT-based systems showed the opposite tendency. We also introduced a system that combines
example-based and SMT-based approaches to take advantage of the characteristics of both approaches.
As future work, investigating ways to improve the naturalness and cohesion of responses generated by the SMT
approach may be necessary. Adding a learning process thatconsiders the context of the conversation could also leadto further improvements. This would allow the system toboth remember the context of the conversation and expand
the example database. Therefore, including the user history
and dialog context in the dialog management system is apromising future direction. Combining other approaches in
the chat-oriented dialog framework could also demonstrateinteresting results.
Acknowledgements
Part of this work was supported by the collaborative researchwith I2R † .

ACM, vol.9, no.1, pp.36–45, Jan. 1966.
[5] R. Wallace, Be Your Own Botmaster, A.L.I.C.E A.I. Foundation,
California, USA, 2003.
[6] H. Murao, N. Kawaguchi, S. Matsubara, Y. Yamaguchi, and Y.
Inagaki, “Example-based spoken dialogue system using WOZ system log,” Proc. SIGDIAL, pp.140–148, Saporo, Japan, 2003.
[7] F. Bessho, T. Harada, and Y. Kuniyoshi, “Dialog system using realtime crowdsourcing and twitter large-scale corpus,” Proc. SIGDIAL,
pp.227–231, Seoul, South Korea, 2012.
[8] C. Lee, S. Jung, S. Kim, and G.G. Lee, “Example-based dialog modeling for practical multi-domain dialog system,” Speech Commun.,vol.51, no.5, pp.466–484, May 2009.
[9] N. Chambers and J. Allen, “Stochastic language generation in a dialogue system: Toward a domain independent generator,” Proc. SIGDIAL, pp.9–18, Cambridge, Massachusetts, USA, 2004.
[10] A. Echihabi and D. Marcu, “A noisy-channel approach to question
answering,” Proc. ACL, Morristown, pp.16–23, NJ, USA, 2003.
[11] Y.W. Wong and R. Mooney, “Learning for semantic parsing with statistical machine translation,” Proc. HLT/NAACL, pp.439–446, NewYork City, NY, USA, 2006.
[12] Y.W. Wong and R. Mooney, “Generation by inverting a semantic
parser that uses statistical machine translation,” Proc. HLT/NAACL,
pp.172–179, Rochester, NY, USA, 2007.
[13] A. Ritter, C. Cherry, and W.B. Dolan, “Data-driven response generation in social media,” Proc. 2011 Conference on Empirical Methods
in Natural Language Processing, pp.583–593, Edinburgh, Scotland,
UK, July 2011.
[14] R. Sproat, A.W. Black, S.F. Chen, S. Kumar, M. Ostendorf, andC. Richards, “Normalization of non-standard words,” Computer
Speech and Language, vol.15, no.3, pp.287–333, 2001.
[15] D. Liu, Z. Liu, and Q. Dong, “A dependency grammar and wordnet
based sentence similarity measure,” J. Computational Information
Systems, vol.8, no.3, pp.1027–1035, 2012.
[16] G. Salton, A. Wong, and C.S. Yang, “A vector space model for automatic indexing,” Commun. ACM, vol.18, no.11, pp.613–620, Nov.
1975.
[17] S. Jung, C. Lee, and G. Lee, “Dialog studio: An example based
spoken dialog system development workbench,” Proc. Dialogs on
dialog: Multidisciplinary Evaluation of Advanced Speech-based Interactive Systems, Interspeech2006-ICSLP satellite workshop, Pittsburg, USA, 2006.
[18] C. Lee, S. Lee, S. Jung, K. Kim, D. Lee, and G. Lee, “Correlationbased query relaxation for example-based dialog modeling,” Proc.
ASRU, pp.474–478, Merano, Italy, 2009.
[19] K. Kim, C. Lee, D. Lee, J. Choi, S. Jung, and G. Lee, “Modelingconfirmations for example-based dialog management,” Proc. SLT,pp.324–329, Berkeley, California, USA, 2010.
[20] R.E. Banchs and H. Li, “IRIS: A chat-oriented dialogue system
based on the vector space model,” ACL (System Demonstrations),
pp.37–42, 2012.

References[1] J. Holmes and W. Holmes, Speech Synthesis and Recognition, Taylor & Francis, London, UK, 2001.
[2] E. Seneﬀ, L. Hirschman, and V. Zue, “Interactive problem solving
and dialogue in the ATIS domain,” Proc. Fourth DARPA Speech andNatural Language Workshop, 1991, pp.354–359, 1991.
[3] M. Walker, J. Aberdeen, J. Boland, E. Bratt, J. Garofolo, L.
Hirschman, A. Le, S. Lee, S. Narayanan, K. Papineni, B. Pellom,
J. Polifroni, A. Potamianos, P. Prabhu, A. Rudnicky, G. Sanders, S.
Seneﬀ, D. Stallard, and S. Whittaker, “DARPA communicator dialog travel planning systems: The June 2000 data collection,” Proc.
EUROSPEECH, 2000, pp.1371–1374, 2000.
[4] J. Weizenbaum, “Eliza – computer program for the study of natural language communication between man and machine,” Commun.
†

http://www.i2r.a-star.edu.sg/

Lasguido Nio
received his B.C.S. and his
M.C.S. degree from Universitas Indonesia, Indonesia in 2012 and 2013. He is currently a firstyear Doctor Student in Nara Institute of Scienceand Technology, Nara, Japan. His research interest include information retrieval and dialog
systems.

NIO et al.: UTILIZING HUMAN-TO-HUMAN CONVERSATION EXAMPLES FOR A MULTI DOMAIN CHAT-ORIENTED DIALOG SYSTEM

1505

Sakriani Sakti
received her B.E. degree
in Informatics (cum laude) from Bandung Institute of Technology, Indonesia, in 1999. In 2000,
she received “DAAD-Siemens Program Asia
21st Century” Award to study in Communication Technology, University of Ulm, Germany,and received her MSc degree in 2002. During
her thesis work, she worked with Speech Understanding Department, DaimlerChrysler Research Center, Ulm, Germany. Between 2003–
2009, she worked as a researcher at ATR SLC
Labs, Japan, and during 2006–2011, she worked as an expert researcher
at NICT SLC Groups, Japan. While working with ATR-NICT, Japan, she
continued her study (2005–2008) with Dialog Systems Group University
of Ulm, Germany, and received her PhD degree in 2008. She actively
involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003–2007), A-STAR and U-STAR (2006–2011). She also served
as a visiting professor of Computer Science Department, University of Indonesia (UI) in 2009–2011. Currently, she is an assistant professor of theAugmented Human Communication Lab, NAIST, Japan. She is a memberof JNS, SFN, ASJ, ISCA, IEICE amd IEEE. Her research interests include
statistical pattern recognition, speech recognition, spoken language translation, cognitive communication, and graphical modeling framework.

Graham Neubig
received his B.E.
from University of Illinois, Urbana-Champaign,
U.S.A, in 2005, and his M.E. and Ph.D. in informatics from Kyoto University, Kyoto, Japan in2010 and 2012 respectively. He is currently an
assistant professor at the Nara Institute of Science an Technology, Nara, Japan. His researchinterests include speech and natural language
processing, with a focus on machine learning
approaches for applications such as machinetranslation, speech recognition, and spoken di-

Tomoki Toda
was born in Aichi, Japan on
January 18, 1977. He earned his B.E. degreefrom Nagoya University, Aichi, Japan, in 1999
and his M.E. and D.E. degrees from the Graduate School of Information Science, NAIST,
Nara, Japan, in 2001 and 2003, respectively. He
was a Research Fellow of JSPS in the Graduate School of Engineering, Nagoya Institute ofTechnology, Aichi, Japan, from 2003 to 2005.
He was an Assistant Professor of the Graduate School of Information Science, NAIST from2005 to 2011, where he is currently an Associate Professor. He has also
been a Visting Researcher at the NICT, Kyoto, Japan, since May 2006.
From March 2001 to March 2003, he was an Intern Researcher at the ATR
Spoken Language Communication Research Laboratories, Kyoto, Japan,
and then he was a Visiting Researcher at the ATR until March 2006. He wasalso a Visiting Researcher at the Language Technologies Institute, CMU,
Pittsburgh, USA, from October 2003 to September 2004 and at the Department of Engineering, University of Cambridge, Cambridge, UK, fromMarch to August 2008. His research interests include statistical approaches
to speech processing such as voice transformation, speech synthesis, speech
analysis, speech production, and speech recognition. He received the 18th
TELECOM System Technology Award for Students and the 23rd TELECOM System Technology Award from the TAF, the 2007 ISS Best Paper
Award and the 2010 ISS Young Researcher’s Award in Speech Field fromthe IEICE, the 10th Ericsson Young Scientist Award from Nippon Ericsson K.K., the 4th Itakura Prize Innovative Young Researcher Award andthe 26th Awaya Prize Young Researcher Award from the ASJ, the 2009
Young Author Best Paper Award from the IEEE SPS, the Best Paper Award(Short Paper in Regular Session Category) from APSIPA ASC 2012, the2012 Kiyasu Special Industrial Achievement Award from the IPSJ, and the2013 Best Paper Award (Speech Communication Journal) from EURASIPISCA. He was a member of the Speech and Language Technical Committeeof the IEEE SPS from 2007 to 2009. He is a member of IEEE, ISCA, IPSJ,
and ASJ.

alog.
Satoshi Nakamura
received his B.S. from
Kyoto Institute of Technology in 1981 and Ph.D.
from Kyoto University in 1992. He was a director of ATR Spoken Language Communication Research Laboratories in 2000–2008, and avice president of ATR in 2007–2008. He wasa director general of Keihanna Research Laboratories, National Institute of Information andCommunications Technology, Japan in 2009–2010. He is currently a professor and a director of Augmented Human Communication laboratory, Graduate School of Information Science at Nara Institute of Science and Technology. He is interested in modeling and systems of spoken
dialog system, speech-to-speech translation. He is one of the leaders ofspeech-to-speech translation research projects including C-STAR, IWSLT
and A-STAR. He headed the world first network-based commercial speechto-speech translation service for 3-G mobile phones in 2007 and VoiceTra
project for iPhone in 2010. He received LREC Antonio Zampoli Award,
the Commendation for Science and Technology by the Ministry of Scienceand Technology in Japan. He is an elected board member of ISCA, International Speech Communication Association, and an elected member ofIEEE SPS, speech and language TC.