IWSDS 2010, 2nd Oct 2010

Issues in Predicting User Satisfaction
Transitions in Dialogues: IndividualDifferences, Evaluation Criteria,and Prediction Models
Ryuichiro Higashinaka, Yasuhiro Minami,
Kohji Dohsaka, Toyomi Meguro
NTT Corporation1

Background• Emerging work on predicting user
satisfaction transitions during a dialogue– Useful for a turn-by-turn analysis of theperformance of a dialogue system– Useful for pinpointing situations where thedialogue quality begins to degrade or improve• Recent work
– Modeling transitions by HMMs
(Engelbrecht et al., 2009, Higashinaka et al., 2010)
2

User Satisfaction Transitions
GOOD 7User Satisfaction

6BAD

Something musthave gone wrong
here

54
3

Some useful clues toimprove usersatisfaction

2112

34

5

6

78

Utterance number

910 11 123

Open Issues• Individual differences– How user satisfaction transitions differ
among raters?

• Evaluation criteria– What evaluation criteria to use for
evaluating user satisfaction transitions?

• Prediction models– What model should we adopt forprediction?
4

(1) Individual Differences• Subjective nature of user satisfaction• Prediction model made from one rater’stransitions may not generalize• Need to investigate how raters agree inrating user satisfaction transitions•We check correlations and distributions of ratings
between different raters•We discuss the feasibility of creating a generalprediction model5

(2) Evaluation Criteria• In any engineering work, it is necessary toestablish an evaluation measure• No established measure• Mean squared error of rating probabilities– Used in Engelbrecht et al. 2009
– Limitation: dialogue has to follow a predefined
scenario too restrictive for common use•We propose several candidates for evaluation
metrics and experimentally decide the best one
6(3) Prediction Models• Hidden Markov models (HMMs)– Used in previous work
– Generative model• Conditional random fields (CRFs)– Recent trend in sequential labeling– Best performance in many NLP tasks– Discriminative model•We compare HMMs and CRFs to investigate
which model is more suitable for the task ofpredicting user satisfaction transitions
7

HMMs and CRFs
HMM

Emissions

Emissions•Speaker ID•Dialogue act•Speaker ID•Dialogue act

Rating X

Rating Y

CRF
Speaker ID

s-2

s-1

s0

s1

s2

Dialogue act

DA-2

DA-1

DA0

DA1

DA2Rating

k-2

k-1

k0

k1

k2

…

Prediction is done by finding the most likely rating sequence
for the sequence of speaker IDs and dialogue acts8

Data Collection• Dialogue data (text chat) in two domains– Animal Discussion (AD)• Discuss likes and dislikes about animals• Human-system dialogue• Useful for obtaining preferences of users– Attentive Listening (AL)• Listener attentively listens to the speaker tosatisfy the speaker’s desire to be heard• Human-human dialogue• Useful for counseling purposes9

Data Statistics

10Data Annotation• User satisfaction ratings by two raters– Raters rated each system (listener) utterance
as if they were the user (speaker)– 7-levels (1: bad  7: good)– Third-party ratings for consistency– User satisfaction ratings from three aspects• Smoothness of a dialogue• Closeness perceived by the user
• Willingness to talk or Good Listener• Dialogue acts for all utterances
11

Example: Animal Discussion

29 dialogue act types

12

Example: Attentive Listening

Listener self-discloses a lot to propel
the speaker to speak

40 dialogue act types

13

Individual Differences• Correlations between the two raters

When 7 ratings are converted into 2 ratings

Spearman’s rankcorrelation coefficients• Distributions of the ratings
AD

Rater-1

AL

Rater-2

Rater-1

Rater-2 14

Individual Differences (cont’d)• Very low correlation between raters– Even decisions about good/bad do not match• Distributions may vary greatly– Especially for human-human dialogues

Currently, it would be difficult to create ageneral prediction model We aim to create a rater-dependent
prediction model in this work15

Evaluation Criteria• Six possible metrics to calculate thesimilarity between reference transitions
and hypothesis transitions Ref: 4 4 3 2 2 1 2 3
Hyp: 4 5 6 5 2 1 4 5
1. Match Rate (MR)
2. Mean Absolute Error (MAE)
3. Spearman’s rank correlation coefficient (rho)
4. Kullback-Leibler Divergence (KL)
5. Match Rate per Rating (MR/r)
6. Mean Absolute Error per Rating (MAE/r)•Equally treat difficult and easy-to guess ratings16• R: reference transitions for a dialogue• H: hypothesis transitions• L: length of a dialogue (# utterances)
How exactly tworatings match
Distance between
the two transitions

Similarity ofrating orders17• R: reference transitions for all dialogues• H: hypothesis transitions• K: maximum user satisfaction level (=7)
Similarity of rating
distributions
Match Rate

MR and MAE pereach rating•Equally treatsdifficult and easy-toguess ratings•Important topredict rare but
important cases 18

Assumptions for choosing
the best metric• The suitable metric– should show the lowest performancefor “random choice” and “no choice”(e.g., majority baseline) they do not perform any prediction– should show similar performance values
for the data of different raters
 the difficulty of prediction should beindependent of the raters
19

Experiment• Trained HMMs and CRFs using thereference user satisfaction transitions ofeach rater for each domain• Random and majority baselines• Procedure– Choose the best metric according to ourassumptions– Analyze the performance of HMMs and CRFs
using the best metric
20

The best metric• Random and majority baselines beat
HMMs and CRFs in MR, MAE, and MAE/r• Spearman’s rank correlation (rho) and KL
greatly differ depending on the rater• MR/r beats random and majority baselines
and have similar values for different raters
MR/r becomes our recommended
evaluation metric21

Results (MR/r)
AD domain

Smoothness

Closeness

Willingness

HMM

CRF

HMM

CRF

HMM

CRF

Rater-1

0.217

0.172

0.231

0.162

0.224

0.208

Rater-2

0.210

0.177

0.232

0.176

0.234

0.238

AL domain

Rater-1
Rater-2

Smoothness
HMM
CRF
0.228 0.193
0.210 0.185

Closeness
HMM
CRF
0.231 0.190
0.195 0.168

Good Listener
HMM
CRF
0.222 0.202
0.208 0.185

HMMs outperform CRFs in most cases

22Summary and future work
• Three issues in predicting transitions– Individual differences• Large differences between raters• It is better to aim for rater-dependent model– Evaluation criteria• Match Rate per rating (MR/r)– Prediction models• HMMs outperform CRFs
• CRFs overtuned to output likely ratings• Future work
– other metrics, improving prediction performancewith other features
23