2-8-1
A Dialog System with Human-to-Human Conversation Example
Lasguido Nio, Sakriani Sakti, Graham Neubig, Tomoki Toda, Satoshi Nakamura (NAIST)
1Introduction

Example-based dialog modeling (EBDM) is datadriven approach for deploying dialog systems [1, 2].
Some studies propose constructing dialog examples
from available human-to human conversation log
databases [3, 4] or movie scripts [5]. However, theseworks did not ﬁlter any uncorrelated consecutive
lines in the movie data. As the authors state, thiscauses failures and diminishes the ability to maintain a consistent conversation.
In this paper, we summarize our work on a dialog agent that utilizes human-to-human conversation examples from movies and Twitter to reduce
the time requirement for database design and collection, and allow the agent to interact with theuser in as natural as fashion as possible. Next, weinvestigate various data-driven approaches to dialog management, including two EBDM techniques(syntactic-semantic similarity retrieval and TF-IDF
based cosine similarity retrieval) and using statistical machine translation (SMT) to learn a conversational mapping between user-input and systemoutput dialog pairs.

2

pattern X-Y-X into two pairs, X-Y and Y-X.
However we found that even after the tri-turn ﬁltering, noisy cases that contain uncorrelated turns
still exist. This happens because the speakers are
not actually speaking to each-other. To address
this problem, we perform further ﬁltering using thesemantic similarity (similar to the approach introduced in [6]). This method ensure a semantic relationship between each dialog turn in the dialog pairdata by computing the similarity between WordNet1
synsets in each dialog turn (see Eq. (1)).
semsim (S1 , S2 ) =We collect a dialog corpus of dialog pairs frommovie scripts and Twitter data.
2.1 Preprocessing
We remove unnecessary explanatory informationfrom the movie script, and information about personal identity, hash tags, and URLs from the Twitter data. Both data sets are labeled with parts ofspeech (POS) and named entities (NE). Unifying
both data sources into one dialog corpus, we deﬁne
two basic types of information about each dialog:
actor and utterances. The utterances are the actual
content of each dialog turn in the movie scripts ortweets. The actor refers to the character name in themovies, or the name of the Twitter user that posted
each tweet. This actor and utterance information
will be utilized to construct the dialog corpus.
2.2 Turn Extraction and Filtering
To ensure that the dialog example database contains only query-response pairs, we use a simple andintuitive method for selection of the dialog data: trigram turn sequences, or tri-turns. A tri-turn is deﬁned as three turns in a conversation between twoactors X and Y that has the pattern X-Y-X. Withina tri-turn, the ﬁrst and last dialog turn are performed by the same actor and the second dialog turnis performed by the other actor. Next the queryresponse pairs are made by separating the tri-turn(1)

Ssyn1 and Ssyn2 respectively are groups of WordNet synsets for each word in the sentences S1 and S2 ,linked by a complex network of lexical relations. The
similarity of sentence pair X-Y (S1 = X; S2 = Y)
can be obtained by calculating the relations between
Ssyn1 and Ssyn2 . |Ssyn1 ∩ Ssyn2 | is the number ofco-occurring WordNet synsets and |Ssyn1 | + |Ssyn2 |is a total number of eﬀective WordNet synsets. Dialog pairs with high similarity are then extracted andincluded into the database.

3Dialog Data Collection

2 × |Ssyn1 ∩ Ssyn2 ||Ssyn1 | + |Ssyn2 |Dialog Management System

3.1 Syntactic-Semantic Similarity Retrieval
In this approach (sssr), a proper system responseis retrieved by measuring both semantic and syntactic relations. These two measures are combinedusing linear interpolation as shown in Eq. (2). Thisvalue is calculated over the user input sentence (S1 )and every input examples on database (S2 ). These
values are calculated using Eq. (1) as a semantic
factor and cosine similarity (Eq. (3)) over part-ofspeech (POS) tag vectors as a syntactic factor.
sim(S1 , S2 ) = α[semsim (S1 , S2 )] + (1 − α)[cossim (S1 , S2 )](2)wherecossim (S1 , S2 ) =S1 · S2.
∥ S1 ∥ ∥ S2 ∥(3)

3.2

TF-IDF-based Cosine Similarity Retrieval
Cosine similarity over the term vector (csm) as
described in Eq. (3) is used to retrieve a proper
system response. To increase the emphasis on important words, additional TF-IDF weighting (Eq.
(4)) is performed to construct the term vector [7].
(T F IDF (t, T ) = Ft,T log|T |DFt)(4)

We deﬁne Ft,T as term frequency t in a sentence
T , and DFt as the total number of sentences in thequery-response pairs that contain term t.

1 http://wordnet.princeton.edu/

日本音響学会講演論文集- 41 -2014年9月

3.3 SMT-based Generation
The dialog-pair data is treated as a parallel corpus
for training an SMT system [8]. Given the trained
SMT system, the user dialog is treated as an input and “translated” into the system response. The
system response is chosen to be system output S ofmaximal probability given the user input TŜ = arg maxP (S | T ).

(5)

S

4

lyzing the data in more detail, we found that csm
is better in handling when dialogues close to Qtest
exists in Qtrain , while smt can provide a better output when there is no dialogs in Qtrain similar with
Qtest . Combining both approaches (comb) in which
the system uses EBDM if the similarity between user
input and dialog examples exceeds given threshold,and responds with SMT output otherwise, could
overcome the shortcomings of each approach.
sssr

csm

smt

comb

Movie
data

no-ﬁltertriturn+semantic

2.5
2.7

2.9
3.5

2.2
2.9

2.3
3.0

Twitterdata

no-ﬁltertriturn+semantic

2.7
2.8

2.4
3.1

2.1
2.3

2.1
2.7

Experimental Evaluation

After extracting all the dialog-pairs, we randomly
separate our query-response pairs from Twitter andmovie conversation dialog, as a test set (the queryresponse pairs are denoted as ⟨Qtest , Rtest ⟩), andas dialog examples for EBDM, or training datafor SMT (the dialog-pairs here are denoted as⟨Qtrain , Rtrain ⟩).
Given a query from the test set (Qtest ), EBDM
will search the closest query examples using
syntactic-semantic similarity retrieval: sim(Qtest ,Qtrain ) or TF-IDF based cosine similarity retrieval:cos(Qtest , Qtrain ), and output a response of Routput .
To evaluate a number of valid system response,
we assess each output response with semantic, syntactic, and manual evaluation criteria. We utilize
TF-IDF-based cosine similarity as a semantic criterion, syntactic-semantic similarity as a syntactic
criterion, and subjective evaluation as a manual
opinion score. For objective evaluation, the Routput
are evaluated by computing similarity with Rtest :sim(Routput , Rtest ) and cos(Routput , Rtest ). During
subjective evaluation, users evaluate the naturalnessof dialog-pair Qtest and Routput by giving them ascore between 1-5.
TF-IDF based Cosine Similarity
sssr

csm

smt

comb

no-ﬁltertriturn

55.86%
58.27%

52.20%
53.86%

38.29%
37.85%

60.53%
62.05%

semantic
triturn+semantic

62.33%
61.85%

65.58%
66.52%

48.43%
49.26%

69.64%
70.55%

Twitter

no-ﬁlter

51.73%

52.65%

30.44%

55.68%

data

semantic

55.74%

66.12%

49.95%

71.44%

Movie
data

Syntactic-Semantic base Similarity

Movie
dataTwitter
data

sssr

csm

smt

comb

no-ﬁltertriturnsemantic

55.93%
57.95%
72.18%

54.24%
55.44%
75.44%

43.11%
41.77%
62.35%

64.49%
64.80%
80.03%

triturn+semantic

71.36%

76.81%

62.58%

80.78%

no-ﬁltersemantic

51.52%
82.95%

52.47%
83.64%

36.06%
73.86%

58.18%
85.73%

Table 1

Objective evaluation result.

To demonstrate the eﬀect of semantic similarity
and tri-turn ﬁltering in our data, we compare oursystem performance with and without the tri-turnand semantic ﬁltering. Table 1 shows the resultof objective evaluation given various ﬁlter and response retrieval techniques. The tri-turn and semantic ﬁltering manage to increase the evaluation
score. Comparing the csm and smt approaches, csm
always give a better performance than smt. Ana-日本音響学会講演論文集

Table 2

Subjective evaluation result.

The subjective evaluation result (see Table 2) also
demonstrate slightly higher scores on ﬁltered data.
This shows that the tri-turn and semantic similarity ﬁltering methods manage to increase the naturalness of the response. Furthermore, because response
sentences from the smt system are sometimes incomprehensible, more people prefer the csm responses.
This also aﬀected the comb performance, where thecsm response was slightly better than the comb approach.

5Conclusion

We investigated several approaches to build adata-driven chat-oriented dialog systems. The proposed tri-turn extraction and semantic similarity ﬁltering are able to extract dialog pair examples frommulti-speaker dialog of raw movie scripts and Twitter data. Experimental results also reveal that thatthe tri-turn and semantic ﬁltering improve the objective evaluation metrics. We also introduced a system that combines example-based and SMT-based
approaches to take advantage of the characteristics
of both approaches.

参考文献[1] C. Lee, S. Lee, S. Jung, K. Kim, D. Lee, and G. Lee,
“Correlation-based query relaxation for example-based
dialog modeling,” in Proc. of ASRU, Merano, Italy, 2009,
pp. 474–478.
[2] K. Kim, C. Lee, D. Lee, J. Choi, S. Jung, and G. Lee,
“Modeling confirmations for example-based dialog management,” in Proc. of SLT, Berkeley, California, USA,2010, pp. 324–329.
[3] H. Murao, N. Kawaguchi., S. Matsubara, Y. Yamaguchi,
and Y. Inagaki, “Example-based spoken dialogue systemusing WOZ system log,” in Proc. of SIGDIAL, Sapporo,Japan, 2003, pp. 140–148.
[4] F. Bessho, T. Harada, and Y. Kuniyoshi, “Dialog systemusing real-time crowdsourcing and twitter large-scale corpus,” in Proc. of SIGDIAL, Seoul, South Korea, 2012,
pp. 227–231.
[5] R. E. Banchs and H. Li, “IRIS: a chat-oriented dialoguesystem based on the vector space model,” in ACL (System Demonstrations), 2012, pp. 37–42.
[6] D. Liu, Z. Liu, and Q. Dong, “A dependency grammar
and wordnet based sentence similarity measure,” Journalof Computational Information Systems, vol. 8, no. 3, pp.
1027–1035, 2012.
[7] G. Salton, A. Wong, and C. S. Yang, “A vector spacemodel for automatic indexing,” Commun. ACM, vol. 18,
no. 11, pp. 613–620, Nov. 1975.
[8] A. Ritter, C. Cherry, and W. B. Dolan, “Data-driven response generation in social media,” in Proc. of EMNLP,
Edinburgh, Scotland, UK, 2011, pp. 583–593.

- 42 -2014年9月