Evaluating Discourse Understanding in Spoken Dialogue Systems
Ryuichiro Higashinaka, Noboru Miyazaki, Mikio Nakano, and Kiyoaki Aikawa
NTT Communication Science Laboratories, NTT Corporation
Atsugi, Kanagawa 243-0198, JapanOverview• We propose a method for creating an evaluation measure for discourse understanding in spoken

Experiment: Data Collection

dialogue systems.
• We enumerated possible discourse-understanding-related metrics and used the metrics to create byregression methods a discourse understanding measure that correlates closely with systemperformance.
• The correctness of a dialogue state update is the most important factor in improving system
performance.

Problem# of incorrect slots
Slot Error Rate (SER) =# of slotsbelow) for each dialogue.
• Task completion times were normalized to make themcomparable among systems.

WI

MR-1

MR-2

Subjects

1228

28Collected
dialogues

192

224

224

Task
completion
rate

95.8%
(184/192)84.8%(190/224)

79.0%(177/224)• We used multiple linear regression and support vector regression as regression methods.

• Discourse understanding is a process of updating a dialogue state by a user utterance.
• There is no appropriate measure to evaluate dialogue state sequences.
• For example, which of the two metrics below better evaluates the discourse understanding?
MetricMetric11• Dialogue data collection in two domains.
• Weather information service domain (WI)
• Meeting room reservation domain (MR-1, MR-2)• We hand-annotated reference dialogue states.
• We calculated the values of 13 metrics (see the table

Metrics
Metrics
1. Slot accuracy

MetricMetric22# of correctly updated slots
Updated Precision (UP) =# of updated slots

2. Insertion error rate

3. Deletion error rate

4.Experiment:
Substitution error
rate Collection
5. Slot error rate
Data

6. Update precision

7. Update insertion error rate

8. Update deletion error rate

9. Update substitution error rate

10. Speech understanding rate

11. Slot accuracy for filled slots

12. Deletion error rate for filled slots

13. Substitution error rate for filled slots-- Problem in an example dialogue -Tell me theweather for
tomorrow

Whatarea?

Tokyo“Tomorrow”“Tomorrow”waswasmisrecognized
System’s Dialogue State Sequence
misrecognized
as“today”!
as “today” !
place-place
place-date
today
date
date-info-info
infoweather

Reference Dialogue State Sequence----No!
tomorrow

Experiment: Results
OK…

• In both domains, obtained regression models show relatively good correlation with system
performance.

UserSystem

place
date
info

Today’sweatherin Tokyo?

SER=1/3
SER=1/3
UP=1/2
UP=1/2place-date tomorrow
infoweather

Which metric is more suitablefor evaluation?
Or should we use both?

Tokyotodayweather

Misunderstood
Misunderstood
item
item“today”“today”
still
stillremains
remains! !

placeTokyo
date tomorrow
info
weather

SER=1/3
SER=1/3
UP=1/1UP=1/1placeTokyo
date tomorrow
info
weather

SER=0/3
SER=0/3
UP=1/1UP=1/1placeTokyo
date tomorrow
info
weather

Average
AverageSER
SER: :(1/3(1/3++1/31/3++0/3)
0/3)/ /33==0.22
0.22
Average
AverageUP
UP : :(1/2(1/2++1/11/1++1/1)
1/1)/ /33==0.83
0.83• Support vector regression performs better than multiple linear regression.
• The analysis of the obtained models revealed that update precision is the most important factor
in reducing the task completion time.
Distribution of actual and predicted taskcompletion times (SVR, WI+MR-1+MR-2)
Multiple LinearRegression

Support Vector
Regression

WI

0.488 (0.549)

0.471 (0.323)

MR-1

0.291 (0.649)

0.370 (0.367)

MR-2

0.478 (0.557)

0.494 (0.326)

MR-1+MR-2

0.432 (0.572)

0.442 (0.335)

WI+
MR-1+MR-2

0.415 (0.583)

0.456 (0.325)

R-square andRoot mean square error (in brackets)

Predicted task completion timeMayI helpyou?

1.510.5

0-0.5-1-1.5-1.5-1-0.500.5

11.5

Actual task completion time

Approach• Enumerate possible discourse-understanding-related metrics.
• Use the metrics to create by regression methods a discourse understanding measure that correlates
closely with system performance.

Conclusion• We proposed a method for creating an evaluation measure for discourse understanding inspoken dialogue systems.

Explaining variables

possible metrics about discourse understanding

Explained variables

Task completion time as system performance metric• Obtained measures show good correlation with system performance.
• The correctness of a dialogue state update is the most important factor for system improvement.