SIGDIAL 2010, 24th Sept 2010

Modeling User Satisfaction
Transitions in Dialogues fromOverall Ratings
Ryuichiro Higashinaka, Yasuhiro Minami,Kohji Dohsaka, Toyomi Meguro
NTT Corporation1

Objective• Predict user satisfaction transitions during a dialogue– Useful for an analysis to improve dialogue systems

User Satisfaction

GOOD
7

Something must havegone wrong here654
3

Some useful clues toimprove user satisfaction

21BAD

1

2

3

4

5

6

7

8

Utterance number91011122

Related Work• Plenty of work on predictingoverall user satisfaction of a dialogue– PARADISE (Walker et al., Möller et al.)– N-gram based method (Hara et al.)• Little work on predicting user satisfaction
transitions during a dialogue– One exception: Engelbrecht et al., 2009
– Models user satisfaction transitions using
hidden Markov models (HMMs)
3

Engelbrecht et al., 2009
• Uses manually-labeled reference transitions
Dialogue acts (DAs): 1. Polite 2. Rude 3. Greeting

Dialoguedata

U1 Greeting
U2 Greeting
U3 Polite
U4 Polite
U5 RudeU6 RudeU7 Polite
U8 Greeting
U9 Greeting

212211221ModelDA & rating
sequence

HMM

Rating 1(bad)Rating 2(good)

Polite Rude Greeting

Polite Rude Greeting

Train

Estimate from which rating-related
state each dialogue act is most likely
to have been generated

Output probability distributions
U1 Greeting
U2 Greeting
U3 Polite
U4 RudePredict
Viterbi decoding

Problem: High cost for making reference data

21214

Approach• Train an HMM from dialogues with overall ratings– No utterance-level references are necessary– Requires only a single overall rating per dialogue
OverallRating 1Dialogue
dataOverall
Rating 2

U1 Greeting
U2 Greeting
U3 RudeU4 RudeU5 Greeting
U1 Greeting
U2 Greeting
U3 Polite
U4 RudeU5 Polite

ModelDA Seq

Connect
Rating 1(bad)ModelDA Seq

Rating 1(bad)HMM
Rating 2(good)

Rating 2(good)

How do we connect the states?
5How to connect the states• Connect with equal transition probabilities
 Common dialogue acts (e.g. Greeting) arealways predicted as the same rating• Add a state (common state) trained from the dataof all ratings to avoid evaluating rating-wide DAs
0.50.5Rating 1(bad)Polite Rude GreetingToo broad! may
never come herein decoding

0.5
0.5

Rating 2(good)

Rating 0(all ratings)

Polite Rude Greeting

Polite Rude Greeting

Need tosharpen thedistribution

6

Solution: Concatenated Training• We can sharpen the output distributions
Merge
Rating 1(bad)Rating 2(good)

Rating 0

Rating 0

RetrainDialogues(rating 1)

Rating 1(bad)Split into pairs

Rating 2(good)

Rating 0

Probabilities
are averaged

RetrainDialogues(rating 2)

We iterate this until all pairs
converge against training data7

Example
Rating 1(bad)Rating 2(good)
75%

75%
25%

Polite Rude Funny Greet

Rating 0

Rating 3(Excellent)

75%
25%

Polite Rude Funny Greet

25% 25% 25% 25%

Polite Rude Funny Greet

25%

Polite Rude Funny Greet

NOTE:We ignore transition
probabilities.
We assume the same
initial probabilities 8

75*0.75=56.25
25*0.5=12.5
 4.5:1

EM algorithm updates probabilities
based on the expected counts
75%
Rating 1(bad)

82%
25%
18%

Polite Rude Funny Greet
Rating 0

Polite Rude Funny Greet

75% of “Rude” output from Rating 1
25% of “Rude” output from Rating 0
50% of “Greet” output from Rating 1
50% of “Greet” output from Rating 0Retrain

Dialogues(rating 1)“Rude” 75 counts“Greet” 25 counts

75*0.25=18.75
25*0.5=12.5
 1.5:1
60%

40%

25% 25% 25% 25%

Polite Rude Funny Greet

Polite Rude Funny Greet

9Rating 1(bad)

82%

82%
18%

Polite Rude Funny Greet

Rating 060%

Rating 3(excellent)

18%

Polite Rude Funny Greet
Rating 060%

40%

Polite Rude Funny Greet

Rating 2(good)

82%

18%

Polite Rude Funny Greet
Rating 060%

40%

40%

Polite Rude Funny Greet Polite Rude Funny Greet
40%

Common utterances
more likely to be output
from common states

20%

Polite Rude Funny Greet

Rating-specificutterances more likely tobe output from the states
for each rating
10

100%

Polite Rude Funny Greet
Rating 1(bad)

100%

100%

Polite Rude Funny Greet
Rating 2(good)

Polite Rude Funny Greet
Rating 3(excellent)

Rating 0

We can predictthe rating for
dialogue acts
more
accurately

100%

Polite Rude Funny Greet

11

Experiment• Dialogue data (text chat) in two domains– Animal Discussion (human-system)• User and system discuss about animals– Attentive Listening (human-human)• Listener attentively listens to a speaker– Annotated with dialogue acts and US ratings (7-levels)
Animal Discussion

Attentive Listening# dialogues with overall ratings

180

1260

# dialogues with utterance-levelratings

90(subset)

100(subset)

29

40

Dialogue act types• Comparison with a random baseline and an upper bound(Engelbrecht et al.) trained from reference transitions

12

Evaluation Criterion• Match rate per rating (MR/r)– Equally treats easy and difficult-to-guess ratings• We want to predict rare but important ratings– Reliable than other metrics(Higashinaka et al., 2010, IWSDS)U1U2U3U4U5U6U7U8U9U10 U11 U12Ref

44

556

6

7

4

3

2

1

1

Hyp

400

6

6

70

50

3

10

Hyp’

44

4

66

7

7

5

53

1

1

Avg

Rating

12

34

56

7

Match rate

2/2

0/1

0/1

2/3

0/2

1/2

1/1

MR/r

0.45
13

Results
Animal Discussion domain
random simple
Smoothness
0.143
0.137
Closeness
0.143
0.156
Willingness
0.143
0.152

Simple connection of ratingrelated states with equaltransition probabilities

concat
0.177
0.164
0.181

Upper B
0.217
0.231
0.224

concat
0.167
0.159
0.224

Upper B
0.231
0.237
0.227

Attentive Listening domain
random simple
Smoothness
0.141
0.118
Closeness
0.143
0.090
Good Listener 0.143
0.121

Common states with concatenatedtraining improve prediction performance14

Summary and Future work
• A novel approach for predicting user
satisfaction transitions using HMMs
– Uses only the dialogues with overall user
satisfaction ratings– Reduces the cost for training prediction models– Can be a viable option for evaluation• Overall ratings can be obtained easily• Future work
– New emissions to improve prediction accuracy– Apply our HMMs to other tasks
15