A METHOD FOR EVALUATING INCREMENTAL UTTERANCE UNDERSTANDING
IN SPOKEN DIALOGUE SYSTEMS
Ryuichiro HIGASHINAKA, Noboru MIYAZAKI, Mikio NAKANO, Kiyoaki AIKAWA
NTT Communication Science Laboratories
NTT Corporation
3-1 Morinosato-Wakamiya, Atsugi, Kanagawa 243-0198, Japan{rh, nmiya, nakano}@atom.brl.ntt.co.jp, aik@idea.brl.ntt.co.jp

1. Introduction
Evaluation measures for components in a spoken dialogue system :• Speech Recognizer → WER (word error rate)• Speech Recognizer and Language Understanding Component → CER (concept error rate)• Speech Recognizer, Language Understanding Component and Discourse Understanding Component → ??

How can we evaluate the three components(understanding components) as a whole ?

Language
UnderstandingComponent

SpeechRecognizer

Discourse
Understanding
Component
updateSuch an evaluation measure is especially needed,because we are promoting an understanding method
ISSS (Incremental Sentence Sequence Search) whichcauses the dialogue state to update frequently.
We propose to create an evaluation measure byfinding an equation that associates the behaviorof the understanding components with thesystem's performance.

SpeechSynthesizer

Dialogue
ManagerDialogue State

Fig1. Basic architecture of a spoken dialogue system

What’s ISSS : ISSS accepts both sentences and sentence
fragments ( i.e., words, phrases ) and incrementally updates
the dialogue state. If ambiguity is found in the understandingof the fragments, ISSS holds multiple dialogue states orderedby priority, so that the system can decide on a single dialogue
state after any speech interval.

2. Approach
Find the representation of the understanding
components’ behavior
We label the system’s dialogue state(dialogue state hypothesis) in two respects.
(1) the correctness of the dialogue state itself (Fig2)(2) the correctness of the update (Fig3)
We derive ten metrics to express the correctnessof the dialogue states in a dialogue. (see 2.1)

When a misrecognition happens, the error is normally inherited to thenext dialogue state, thus it is not appropriate to use only the resulting
dialogue state itself for its correctness. (Fig1)

How correct is this dialogue state?

System’s Dialogue States“from three”

Initial Dialogue State

DateRoomStart Time
End Time

Date-----

RoomRoom---Start Time

2 o’clock

Start Time

2 o’clock

End Time--

End Time

4 o’clockMisrecognition occurred and error inherited !

Correct Dialogue States

Find the representation of
the system’s performance

Initial Dialogue StateDate

We use task completion time. Task completiontime correlates closely with user satisfaction.

Room
Start Time
End Time“to four”

Date---“from three”

Date-----“to four”

Date

Room---

Room---Start Time

3 o’clock

Start Time

3 o’clock

End Time--

End Time

4 o’clockFig1. Example of dialogue state updates

Create an equation that can estimate thesystem’s performance from the understanding
components’ behavior.
We perform a multiple linear regression analysis.
Dialogue State Hypothesis
Date

Dialogue State Reference
correct

DatedeletionRoom

2 o’clockinsertion

3 o’clock

substitution

Room

5/20
--

Start Time
End Time

NOTE : We use a frame
representation for
dialogue states, each
attribute-value pair iscalled a slot.

Initial Dialogue State
Date

5/20
--

Start Time--

End Time--

Dialogue State Hypothesis
Date

Dialogue State Reference
correctly left

Dateupdate deletion

Room2 o’clock

update insertion

Start Time--

4 o’clockcorrect update

End Time

4 o’clock

5/20
Room 3

Room5/20--

Start Time--

Start Time

End Time

4 o’clock

End Time

Fig2. Labeling of a dialogue state

Room

5/20
Room 3Fig3. Labeling of a dialogue state update2.1 Ten Metrics

3. Experiment

5 metrics concerning the dialogue state itself

Domain# of correct slots# of insertion
insertion error rate =# of slots# of slots# of deletion# of substitution
deletion error rate =substitution error rate =# of slots# of slots
total # of insertion,deletion,and substitution=
slot error rate
# of slots
slot accuracy =4 metrics concerning the dialogue state update
update precision=# of correctly changed slots# of changed slots in hypothesis

update insertion error rate=

# of changed slots in hypothesis# of unchanged slots in reference

update deletion error rate=

# of unchanged slots in hypothesis# of changed slots in reference

update substitution error rate =1 metric concerning all dialogue states in a dialogue
speech understanding rate =# of dialogue states with 100% slot accuracy# of dialogue states in a dialogue

Speech recognizer

Julius 3.1p

Speech synthesizer

FinalFluet (NTT SP Lab)

Number of subjects

18 (male : 9, female : 9 )Number of collected dialogues

180 (3595 utterances)

Task completion rate

63.6 %

Number of dialogues used foranalysis

108

Number of dialogue strategies *2

Number of task patterns *5

Dialogue State Reference Tagging :References are semi-automatically createdusing a simulation system and then manuallycorrected.

# of incorrectl y changed slots in hypothesis# of changed slots in reference• We calculate above nine metrics for each dialogue state,and use their average values for their values in a dialogue.
• We use additional one metric shown below.

Meeting room reservation*Dialogue Strategy and Task Pattern :Task completion time is normalized using(1) Dialogue Strategy (dialogue manager’s behavior)(2) Task Pattern ( room reservation patterns )to focus only on dialogue states and the system’sperformance.

4. Results• We performed a multiple linear regression analysis using(1) the task completion time normalized by the task pattern and the dialogue strategy as the explained variable(2) the ten metrics (see 2.1) as explaining variables• By stepwise regression, seven metrics were incorporated as a result. Below is the equation :task completion time (actual)Y = - 4.19 - 12.49 x1 + 12.77 x2 - 0.03 x3- 17.74 x4 + 4.54 x5 + 2.11x6 + 2.98 x7
Where Y is the predicted task completion time, x1 the insertion error rate, x2 the substitution error rate, x3 the update precision,x4 the update insertion error rate, x the update deletion error rate, x6 the update substitution error rate, and x7 the speech understanding rate.
52

task completion
time

1.5
1
0.5
0-0.5
-1
-1.5
-2
-2

-1

0

1

2

-0.40

insertion error rate

-0.07

deletion error rate

0.29

substitution error rate

0.40

slot error rate

0.40

update precision-0.45

update insertion error rate

0.15

update deletion error rate

0.62

update substitution error rate

0.24

speech understanding rate

-0.42

3task completion time (predicted)
RSquare=0.57 RMSE=0.6324
Fig4. Distribution of actual andpredicted task completion times

slot accuracy

As Fig4 shows, the model fitscomparatively well with
RSquare 0.57, the RSquare
Adjusted 0.54, and RMSE
(Root Mean Square Error) 0.63.
The correlation coefficients ofthe ten metrics against taskcompletion time are shown inTable1.
The update deletion error rate
has a relatively highcorrelation with correlation
coefficient 0.62 followed by- 0.45 of update precision.

Table1. Correlation coefficients of the ten metrics
against the task completion time

5. Summary and Future Plans• We proposed a method for evaluating incremental utterance understanding, which involves speech recognition,
language understanding, and discourse processing in spoken dialogue systems, by performing a multiple linear
regression analysis using the task completion time as the explained variable and various metrics concerning dialogue
states as explaining variables.
• The resulting model shows a validity as an evaluation measure, and indicates that the use of both the dialogue states
and their way of update is effective.
Our future plans include: validation of our approach in other domains (e.g., more complex domains, more real-worldbased domains), use of user satisfaction metrics in addition to task completion time.