Conversational Response Re-ranking Based on Event Causalityand Role Factored Tensor Event Embedding
1121
1Shohei Tanaka , Koichiro Yoshino , Katsuhito Sudoh , Satoshi Nakamura
1Nara Institute of Science and Technology 2PRESTO, Japan Science and Technology Agency

Abstract | Method to select coherent and diverse responses based on event causality
Neural Conversational Model (NCM) struggles with maintaining response coherency and diversity to continue dialogues.

To solve the problem, we proposed a method to re-rank response candidates by using Event Causality and Event Embedding.
Experimental results (dist, human evaluation, etc.) show that our method improves response coherency and diversity to continue dialogues.

1 Introduction | Improving dialogue continuity of NCM

3 Experiments | Improvement of diversity and coherency

Dull Response Problem of NCM

Automatic Evaluation

NCM [Vinyals et al., 2015] often generates simple and dull responses due tothe limitation of its ability.
I see.
I don’t know.

Response Re-ranking Based on Event Causality**cause-effect relations between two events
I’m stressed out recently.

 I see.

You are better to relieve stress.

SystemUser
Response selection using a causality (“be stressed out” precedes “relieve stress”)

We expected that our method improves response coherency anddiversity to continue dialogues.

2 Method | Response Re-ranking Using Event Causality
Overview of Re-ranking Based on Event Causality…

1

I’m exhausted.

1. I see.
2. I don’t know.
3. You must relax.

NCM…

Dialogue
History

Response Candidates
2

1. You must relax.
2. I see.
3. I don’t know.

Re-rankingFunction…
Event Causality Pairs“be exhausted” -> “relax” is used.

(e.g. be stressed out -> relieve stress)

 NCM generates response candidates from a dialogue history.
 Re-ranking function selects response candidates with event
causality related to the dialogue history.
Proposed method uses event causality pairs [Shibata et al., 2014] to findcausalities between the dialogue history and response candidates.

Causality Matching Based on Event Embedding
Event causality pairs do not include all causalities in dialogue
because they are obtained from limited Web corpus.
relieve stress…
(e.g. be stressed out -> relieve stress)…

An event in the dialogue (“relax”) can not
be found if it is not included in the pairs.

Solve this problem by using event embedding [Weber et al., 2018].
Find a similar event causality pair on vector space.
…

be exhausted…

Dialogue History
Events (Embedded)…

Re-ranking
Re-ranked (%) dist-1 dist-2 PMI
1-best0.06
0.18 1.77
EncDec w/o embedding
12.72
0.06
0.19 1.78
w/ embedding
69.39
0.07
0.21 1.77
1-best0.07
0.20 1.84
HRED w/o embedding
12.25
0.06
0.20 1.84
w/ embedding
71.53
0.06
0.20 1.86• Re-ranked ratios were increased drastically by the embedding.
• Dist is a number of distinct words in responses.
-> Diversity was improved.
• PMI is a mutual information between a history and a response.
-> Coherency was improved.

Human Evaluation
Word coherency

28.62

Dialogue continuity

Event Causality Pairs (Embedded)
be stressed out

…

37.47

40.84

0%

20%

1-bestneither

33.91

20.63

40%

38.53

60%

80%

100%

Re-ranking w/o embedding

Preference Test; 1-best v.s. Re-ranking w/o embedding; # evaluators: 10; # dialogues: 100
Word coherency

30.10

44.50

35.50

0%

20%

1-bestneither

26.30

40%

25.40
38.20

60%

80%

100%

Re-ranking w/ embedding

Preference Test; 1-best v.s. Re-ranking w/ embedding; # evaluators: 10; # dialogues: 100• Coherency was improved by the model without the embedding
but worsened with the embedding.
-> Evaluators did not acknowledge causalities because the event
embedding over-generalized events.
• Continuity was improved by the model with the embedding.
-> Number of dull responses was reduced.
Proposed method improved coherency, diversity, and continuity.

4 Case Study | Re-ranking examples
Appropriate Re-ranking

Event Causality Pairs

relax

Response Events(not included)

NCM

Dialogue continuity

Re-ranked
Response Candidates

Response Events(included)

Comparison in Automatic Metrics

relieve stress…

…

relax

Response Events (Embedded)
A causality in the dialogue (“be exhausted” -> “relax”) is found
if a similar causality (“be stressed out” -> “relieve stress”) is included in the pairs.

User: Because of my fears, I have been stressed out.
System (1-best): Are you OK?
System (Re-ranked): Are you OK? Don’t work too hard.

(“work too hard” -> “be stressed out”) was used appropriately.
Examples such as e.g. 1 are rarely observed.

Over-generalization
User: Our president gives me beer, but he can’t drink.
System (1-best): I see.
System (Re-ranked): I drink.

(“drink” -> “can drink”) was obtained by generalizing(“enter restaurant” -> “order beer”), which is included in the pairs.
Coherency and naturalness were worsened by re-ranking.
Updating the event embedding and maintaining the response
naturalness are necessary.