Speech Understanding in a Multiple Recognizer with an Anaphora
Resolution Process
Kazutaka Shimada

Akira Uzumaki
Mai Kitajima
Tsutomu Endo
Department of Artificial Intelligence,
Kyushu Institute of Technology
680-4 Iizuka Fukuoka 820-8502 Japan             
Abstract

In this paper, we propose a simple and effective method for speech understanding.
The method incorporates some speech recognizers. We use two types of recognizers;
a large vocabulary continuous speech recognizer and a domain-specific speech recognizer. The multiple recognizer is a robust and flexible method for speech understanding. In this paper, we focus on two
issues: (1) selection of outputs from themultiple recognizer and (2) anaphora resolution in the multiple recognizer. For the
output selection, we use a simple edit distance measure computed from output sentences from each recognizer. Our method
has high scalability and accuracy for the
output selection. Next, we describe ananaphora resolution process using outputs
from the multiple recognizer. The experimental results show the effectiveness ofthe proposed method.

1 Introduction
Speech understanding and dialogue systems havebeen developed for practical use recently. Thesesystems often recognize user utterances incorrectly. It is important to deal with speech recognition errors for speech understanding systems.
Extracting keywords and understanding an utterance using them reduce speech recognition errors(Bouwman et al., 1999; Komatani and Kawahara,
2000). Another approach is to use domain-specificgrammars and linguistic models. However thesemethods can not handle out of domain and spontaneous utterances. One approach for the improvement is to repair recognition errors by users. There
are many studies on detection of recognition errors in a speech output. Goto et al. (2005) haveproposed some systems with nonverbal speech

information, such as “SPEECH STARTER” and“SPEECH SPOTTER”. Ogata and Goto (2005)
have proposed a speech input interface with aspeech-repair function. Although repairing recognition errors by humans is effective in terms of development of a speech understanding system withhigh recognition accuracy, it is costly for users.
Combining some recognizers is one of the best approaches to improve the accuracy of speech understanding systems (Isobe et al., 2007; Utsuro et al.,2004). Utsuro et al. (2004) have obtained highaccuracy by using some speech recognizers’ outputs. However they dealt with word error reduction only. Although Isobe et al. (2007) have proposed a multi-domain speech recognition system
based on some domain-specific recognizers, theirsystem cannot treat out-of-domain utterances suchas a chat between users. However the chat utterances often include significant information as thecontext of the dialogue.
In this paper we propose a simple and effective
speech understanding method based on a large vocabulary continuous speech recognizer (LVCSR)
and some domain-specific speech recognizers(DSSR). We call it “One Commoner and Some
Specialists (OCSS) model”. Figure 1 shows theoutline of the model. In our system, the LVCSR
is the commoner, namely domain-independent,
and the DSSRs are specialists, namely domaindependent. We focus on the difference between
outputs generated from the commoner and specialists. By using this method, we can recognize
speech inputs for domain-dependent with highaccuracy and also handle context information indomain-independent speech inputs.
The task of this system is speech understanding
for a livelihood support robot. The DSSRs recognize particular utterances about orders; e.g., orderutterances from elders who need care and order utterances from nurses. We construct the grammarbased DSSR for order utterances with small vo-OCSS model (One Commoner and Some Specialists)
Input

Everyone listensLVCSR

DSSR with small vocabulary
Specialist
Order Specialist

LVCSR
Commoner

DSSR for Order
Non-Order Utterance

I dropped a remote controller under this bed....
*Select

Recognition

remote controller bed

The most similar : the best output? (misrecognize)

Order Utterance

Question Specialist

Please pick it up.
Explanation Specialist■Commoner■Specialist

There is no limitation for

There are many out-of -domain

recognition, but low accuracy

utterances, but high accuracy

Integration

Figure 1: The OCSS model
cabulary and high accuracy for each order type.
We use the LVCSR for recognition of utterancesthat the DSSR can not recognize, such as a chatbetween users. The information recognized by the
LVCSR is of assistance for context constructionof a dialogue. If we handle these different speech
recognizers selectively and integratively, we realize a flexible and robust speech understanding
method. Figure 2 shows the effectiveness of the
proposed multiple recognizer. The DSSR achieves
the order recognition with high accuracy and the
LVCSR supplies lack of information in the orderutterances.
In this paper we discuss two tasks for the OCSS
model based method; (1) selection of the outputsand (2) anaphora resolution in the method. The 1st
task is the selective usage of the multiple speech
recognizer. In other words, it is to select outputs from each recognizer. For example, with respect to the utterance “Please pick it up” in Figure 2, it is important which result to select. The
2nd task is an integration process in our method.
By using previous outputs from One Commoner(LVCSR) and Some Specialists (DSSRs), we resolve an anaphora in the current output. For example, with respect to the utterance “Please pick
it up” in Figure 2, the system identifies that theword “it” in the utterance is the words “remotecontroller” which were recognized by the LVCSR
in the previous utterance.

2 Output Selection in OCSS model
In this section, we explain the process of output
selection in the OCSS model. In this process, we
focus on a difference of outputs generated fromeach recognizer. Even human beings tend to misunderstand words which consist of similar pronun-* LVCSR often
makes a mistake

big it upContext
information*Select

Recognition

pick it up
remote“it” = controller

Results withhigh accuracy

Figure 2: The effectiveness of our method.
ciations (Komatani et al., 2005). Here we focus on
the output of the LVCSR. If an input is an order utterance, the DSSR and the LVCSR generate similar outputs on phoneme-level because the LVCSR
is domain independence. On the other hand, if the
input is not an order utterance, they often generate
different outputs even on the phoneme-level because the DSSR never generates the correct result
for non-order utterances.
Komatani et al. (2007) have reported an utterance verification method based on a difference of
acoustic likelihood values computed from two recognizers. Kumar et al. (2005) have utilized Bhattacharyya distance to measure an acoustic similarity of different languages for multilingual speech
recognition. In this paper, we use the edit distanceas the similarity measure. The correspondence
such as the edit distance is one of the most effective measures to identify high confidence words in
outputs (Utsuro et al., 2004) and to extract similar word pairs (Komatani et al., 2005). In our
method, if an input is an order utterance, the edit
distance between the outputs from the DSSRs andthe LVCSR becomes small. However if the input
is not an order utterance, that between the outputs from the DSSRs and the LVCSR becomeslarge. In our method, we compute the edit distance of utterance-level and word-level by using a
DP matching algorithm. In the process, we compute the edit distance between phonemes of words
for both levels.
The rules to judge an utterance are applied in
the following order:
1. Compute the edit distance of the utterancelevel (EDutter ) between the LVCSR and each
DSSR. For the outputs of which the edit distance is less than threshutter , we select theoutput of the DSSR which contains the minimum EDutter , as the final output.

*Utterance-level
Input: saifu wo mottekite(Bring my wallet)
DSSR:

2. Compute the edit distance of the wordlevel (EDword ) between the LVCSR and each
DSSR. For the output of which the edit distance is less than threshword , we select theoutput of the DSSR which contains the minimum EDword as the final output. Otherwise,
the LVCSR as the final output.

walletbrings a i f u o m o q t e k i t e
Insert, Replace or Delete

LVCSR: s a i g o o m o q t e
finallythink
ED utter = Distance / # of phonemes in two results= 6 / 26 = 0.23

*Word-level
Input: yuka no ue no taoru wo hirotte kureru?
(Could you pick up the towel on the floor?)
DSSR:

floor
on
towel
pick upyuka|noue|no| taoru |o| hiroqte |kureru
2

The EDutter is the edit distance value on theutterance-level. The EDword is the average ofthe edit distance value computed on word-level.
These values are normalized by the number ofphonemes in the outputs. The threshutt er andthreshword are threshold values for the judgment.
These values are decided experimentally.
In the computation of the word-level, we eliminate word pairs that are matched completely first.
Next, we compute all the combinations of theother. Finally, we employ the minimum combinations as the word-level edit distance. Figure 3
shows an example of the calculation of the EDutterand EDword . In the figure, the dotted line denotes
completely matched words. The numerals with arrows denote the original edit distance of the word
pair. In the alignment process of word pairs, weselect pairs which have the minimum value of theedit distance. In other words, we admit overlap ofword pairs. For example, noue vs. no and novs. no in Figure 3.

The Distance is 6.

0

3

0

LVCSR: y u k a | n o | n a k a | w a | r o | o | h i r o q | t e | k u r e r u
floor
in
road
pick upBest Combinations: noue vs. no, no vs. no, taoru vs. wa|ro, hiroqte vs. hiroq|teDistance/Phoneme: 2/6=0.33,
0/4=0,
3/9=0.33,
0/14=0EDword = Sum / # of words in the DSSR result = (0.33+0.33)/7 = 0.09

Figure 3: The edit distance calculation
For DSSR outputs
I want to dirnk the canned drink on the table.
A Grammar in DSSRs
S->Sub Verb to Drink_V Drink_N (in/on/...) Location[Drink(Want),[[`canned drink’ , obj], `on the table’ , loc]](a)
For LVCSR outputs
Mr. Tanaka is in the room 555.
Rules
Mr. *** -> agt
in *** -> loc[Tanaka, agt] [Room 555, loc](b)

Figure 4: Examples of the output analysis

In this section we explain an anaphora resolution
process in the OCSS model.

semantic constraints such as “Drink N” and “Location”. For outputs from a LVCSR, we extract
keywords by using some rules based on surface
expression. Figure 4 (b) shows examples of theprocess. In the figure, “obj”, “loc” and “agt” denote case markers.

3.1

3.2

3 Understanding and Anaphora
Resolution

Understanding of Outputs from OCSS
model

The output in the previous section, namely the output selection process, is an output of a speech recognizer. For the anaphora resolution process, weneed to analyze the output.
For outputs from DSSRs, we convert them intoa semantic frame. We utilize grammar informationof DSSRs for the process. Each DSSR consists of
100-200 words and approximately 100 grammar
patterns including approximately 50 categories.
Figure 4 (a) shows an example of the grammar patterns and categories. The categories often contain

Anaphora Resolution

If an utterance contains an anaphoric expression,
our system detects the antecedent from previous
utterances. In this paper we handle demonstrative
pronouns only. In this process, we focus on thefollowing points: (1) the distance from the current
utterance and (2) change of situation.
The anaphora resolution process is based on thefollowing score calculation:
1. extract all words compatible with the casemarker of anaphoric expressions1
1 Correctly, words with the same marker and words with-

2. compute a base score:



basei

N

disti

kinou reizouko ni ju-su wo simattahazu.
(Yesterday, I put a canned drink in the refrigerator.): Same situation: Different situation

N disti2

1st: ju-su <obj:0.95>, reizouko <loc:0.85>(canned drink) (refrigerator)
2nd: ju-tan <obj:0.05>, reizouko <loc:0.85>(carpet)(refrigerator)
3rd: .....

where N is the number of previous utterancesthat our system treats in this anaphora resolution process. In this paper, N 10. disti
is the distance between the current utterance
that contains the anaphoric expression andthe previous utterance that contains the antecedent. Here “change of situation” denotes“change of a speaker” or “change of the location of a robot”.

sore wo nomitaina(I want to drink it.)[dinrk(want) [it, obj]]
ju-su: base=10 (the nearest)- nBest: 10 * 0.95 = 9.5- Marker: 10 / 2 = 5 (Same: obj)
- Semantic: 10 / 2 = 5 (Same:drink)- Score = 19.5
ju-tan: base=10- nBest: 10 * 0.05= 0.5- Marker: 10 / 2 = 5 (Same: obj)
- Semantic: 0 (Different: furniture)- Score: 5.5

3. compute the following scores by using thebase score:n-best: The word accuracy of the LVCSR is
not often enough2 . As a result, the antecedent often does not exist in the outputs from the previous utterances. To
solve this problem, we use 10-best candidates of speech recognizers outputs.

nBestibasei ¢ CNi
basei

i

Markeri20sore (it) = ju-su (canned drink)

Figure 5: An example of the anaphora resolution
words that possess the same semantic label.

 base: LVCSR outputs: DSSRs outputs

where CNi denotes the confidence measure computed from the LVCSR foreach word.
Marker: We tag a case marker to each wordby using surface expression rules. However, all words are not always tagged by
the rules because of lack of rules. Therefore we distinguish words with the same
marker and words without a marker.

 basereizoko is not a candidate.
- the marker is not ``obj’’iSemantici

out a marker are extracted. In other words, words with markers that are different from the anaphoric expression’s marker
are not extracted for this anaphora resolution process.
2 For the 1st output, it was less than 40% in a preliminary
experiment.

0: Same Semantics
Otherwise

4. compute the score of antecedent candidates
in each utterance.
ScoreinBesti  Markeri  Semantici

5. compute the final score of antecedent candidates
FScorei ∑ Scorei
i¾W: Same marker: No marker

Semantics: We tag some semantic labels towords in DSSRs. For example, thesemantic label of “juice” and “tea” is“drink”. Here assume that the current
utterance is “ I want to drink it”. In
this situation, the antecedent of the word“it” has to contain the semantic label“drink”. Therefore we add a score to2

where W is all antecedent candidates.
6. extract the word that contains the maximum
score as the antecedent.
Figure 5 shows an example of the process.

4 Experiments
In this section, we evaluate (1) the output selection
based on the edit distance and (2) the anaphora resolution process described in the previous sections.
4.1

Speech recognizer in the experiment

We used Julius as the LVCSR and Julian as the
DSSR (Lee et al., 2001). The Julian consists of a

vocabulary and a grammar file. For the grammar
file we describe sentence structures in a BNF style,using word category names as terminate symbols.
The vocabulary file defines words with its pronunciations (i.e., phoneme sequences) for each category. Here we design grammar and vocabulary
files of the Julian which accepts only specific utterances from users. In this experiment, we used 4
DSSRs that we constructed by hand. The DSSRs
are as follows:¯ Order Utterances from patients (Order-P)
e.g., Please bring the remote controller on thetable.

¯ Order Utterances from nurses (Order-N)e.g., “Carry these meals to patient’s rooms”¯ System Commands
e.g., “Move to the right by 50cm”¯ Question Utterances
e.g., “Where is my cellphone?”
4.2

Result of Output Selection

The 1st task in the experiment was to detect aspeech recognizer which was suitable to generate
the output in the multiple recognizer. The dataset
consists of 20 utterances for each DSSR and 20
out-of-domain utterances such as greetings. The
number of test subjects was 10. In other words,
we evaluated our method with 1000 utterances: 5
categories (DSSRs and LVCSR) ¢ 20 utterances¢ 10 test subjects. The threshutter and threshword
were 0.26 and 0.08 respectively. These thresholds
were determined on a preliminary experiment withanother dataset.
Table 1 shows the experimental result. The Fvalue of the output selection was 0.916 on average3 . Besides, we verified that the change of theF-value was small even if we changed the thresholds within the compass of 0.20-0.264 . Therefore,
our method based on the edit distance is simple
and robust.
4.3

Result of Anaphora Resolution

Next, we evaluated the anaphora resolution process in the OCSS model. The dataset of this experiment consisted of 206 utterances that included
3 In addition, the word recognition accuracy of each DSSR
was 0.940 on average.
4 The best F-value on this experiment was 0.924 in the casethat threshutter =0.20.

Table 1: Output Selection Accuracy.
DomainOrder-P
Order-N
Commands
Questions
LVCSR
Average

Recall
0.965
0.965
0.930
0.975
0.765
0.920

Precision
0.873
1.000
1.000
0.878
0.827
0.916

F
0.917
0.982
0.964
0.924
0.795
0.916

Table 2: Accuracy of the Anaphora Resolution.
Methoddist
Methodno
10
Methoddist
10Methoddist
1st
Methoddist
10

Accuracy
32%
41%
48%
52%

50 anaphoric expressions. We compared the following combinations:¯ Methodno10 dist : In this method, basei was always 10.

¯ Methoddist
10 : In this method, basei was always
computed from N disti , i.e., it did not handle “Different situation” in Section 3.2 .

 : In this method, we used the 1st¯ Methoddist
1st
output only for the candidate extraction, i.e.,
it did not handle the nBest in Section 3.2 .

 : The proposed method.
¯ Methoddist
10
Table 2 shows the experimental result. The effectiveness of use of the distance from the current utterance was verified from the comparino dist and the Methoddist .
son between the Method1010Also the effectiveness of use of change of situation was shown from the comparison between thedist 
Methoddist
10 and the Method10 . The effectiveness of use of 10-best outputs was shown from and thethe comparison between the Methoddist
1st
dist 
dist 
Method10 likewise. The Method10 obtained
the best performance
However, the accuracy was insufficient (52%).
Most of the mistakes in the anaphora resolution
were due to misunderstanding of the speech recognizer (LVCSR). If the correct antecedent does notexist in the 10-best outputs from the LVCSR, ourmethod can not detect it essentially. We comparedresults from actual outputs and transcripts, namely
perfect outputs. Table 3 shows the difference between actual outputs and transcripts. This result

Table 3: Accuracy of the Anaphora Resolution.
Input
Accuracy

Actual Output
52%

Transcript
88%

shows that it is important to improve the word accuracy of the LVCSR.
Our method was based on a simple scoring process. Iida et al. (2005) have proposed a machinelearning-based approach to anaphora resolution.
Applying other methods to our system is one of
our future work.

5 Conclusions
In this paper, we described a speech understanding
method based on a multiple speech recognizer. We
called it “OCSS model”. The method was combination of one LVCSR and several DSSRs. By using this method, we realized a flexible and robust
speech understanding method.
In this paper, we evaluated two processes ofthe method: (1) output selection and (2) anaphoraresolution. For the output selection, the method
was based on the edit distance between each output. In the experiment, we obtained high F-value(more than 0.9). This result shows that our method
is simple and robust. For the anaphora resolution, the method was based on (1) the distance between an anaphora expression and an antecedent(2) change of situation such as speaker’s change.
Although the proposed method was effective ascompared with more simple methods, the accuracy
was insufficient. The reason why the accuracy of
the anaphora resolution was low was the accuracyof the LVCSR was low. To improve the accuracy,
we need a LVCSR with more high accuracy.
Our future work includes (1) a large-scale experiment especially the anaphora resolution and(2) evaluation of the proposed method for otherdomains.

Acknowledgment
This research was supported by NEDO, Intelligent
RT Software Project, 2009.

References
C. Bouwman, J. Sturm, and L. Boves. 1999. Incorporating confidence measures in the dutch train
timetable information system developed in thearice project. In Proceedings of ICASSP.
M. Goto, K. Itou, and T. Kobayashi. 2005. Speechinterface exploiting intentionally-controlled
nonverbal speech information. In Proceedingsof the 18th Annual ACM Symposium on User
Interface Software and Technology (UIST2005), pages 36–36.
R. Iida, K. Inui, and Y. Matsumoto. 2005. The
issue of combining anaphoricity determination
and antecedent identification in anaphora resolution. In International Conference on Natural Language Processing and Knowledge Engineering (IEEE NLP-KE), pages 244–249.
T. Isobe, K. Itou, and K. Takeda. 2007. A likelihood normalization method for the domain
selection in the multi-decoder speech recognition system. IEICE TRANSACTIONS on
Information and Systems (Japanese Edition),90(7):1773–1780.
K. Komatani, Y. Fukubayashi, T. Ogata, and H. G.
Okuno. 2007. Introducing utterance verificationin spoken dialogue system to improve dynamic
help generation for novice users. In Proceedings of the 8th SIGdial Workshop on Discourseand Dialogue, pages 202–205.
K. Komatani, R. Hamabe, T. Ogata, and H. G.
Okuno. 2005. Generating confirmation to distinguish phonologically confusing word pairs in
spoken dialogue systems. In Proceedings of 4th
IJCAI Workshop on Knowledge and Reasoningin Practical Dialogue Systems, pages 40–45.
K. Komatani and T. Kawahara. 2000. Flexible
mixed-initiative dialogue management using
concept-level confidence measures of speech
recognizer output. In Proceedings of COLING
2000, volume 1, pages 467–473.
S. C. Kumar, V. P. Mohandas, and H. Li. 2005.
Multilingual speech recognition: A unified approach. In Proceedings of InterSpeech 2005,
pages 3357–3360.
A. Lee, T. Kawahara, and K. Shikano. 2001.
Julius - an open source real-time large vocabulary recognition engine. In Proceedings of Eurospeech, pages 1691–1694.
J. Ogata and M. Goto. 2005. Speech repair: Quickerror correction just by using selection operation for speech input interfaces. In Proceedingsof Interspeech 2005, pages 133–136.
T. Utsuro, H. Nishizaki, Y. Kodama, and S. Nakagawa. 2004. Estimating highly confident portions based on agreement among outputs ofmultiple lvcsr models. Systems and Computers
in Japan, 35(7):33–40.