Recognition Confidence Measures: Detection of Misrecognitions and Out-Of-Vocabulary Words
Authors: Sheryl R. Young
Venue:  (1994)

This paper describes and evaluates a new technique for measuring confidence in word strings produced by speech recognition systems. It detects misrecognized and out-of-vocabulary words in spontaneous spoken dialogs. The system uses multiple, diverse knowledge sources including acoustics, semantics, pragmatics and discourse structure to determine if a word string is misrecognized. When likely misrecognitions are detected, a series of tests distinguishes out-of-vocabulary words from other error sources. The work is part of a larger effort to automatically recognize and understand new words when spoken in a spontaneous spoken dialog. We describe a system that combines newly developed acoustic confidence measures with the semantic, pragmatic and discourse structure knowledge enbodied in the MINDS-II system. The newly developed acoustic confidence metrics output independent probabilites that a word is recognized correctly along with a measure of how reliably we can estimate if a word is wrong. The acoustic confidence metrics are derived from normalized acoustic recognition scores. The acoustic scores are normalized by estimates of the denominator of Bayes equation. To evaluate the utility of using the acoustic techniques together with higher-level constraints, the preliminary system restricted component interaction. Words with normalized acoustic scores that had a 95% or greater probability of being incorrect were flagged prior to being input to the MINDS-II analysis module. For this study, MINDS-II independently used its higher-level knowledge to detect recognition errors that were semantically or contextually inappropriate. Misrecognized word strings were then re-recognized using an RTN-based speech decoder guided by a dynamically derived, highly constrained grammar that restricts the possible words that can be matched, biasing the recognizer against illogical and highly improbable content. Speaker goals and plans, contextual appropriateness, discourse and spontaneous speech structure are all considered in the derivation of grammars. A grammar is dynamically derived for each string of misrecognized words encountered within an utterance and essentially defines a set of content predictions for the word string. Although a rudimentary procedure was employed estimate the conjoined effect of merging these two knowledge sources, the results indicate that the conjoined usage of normalized acoustic confidence measures of accuracy and the higher-level, semantic, pragmatic and discourse level constraints embodied in the MINDS-II system enables the larger system to overcome the weaknesses of each individual technique. The techniques detect complementary phenomena. Significantly more recognition errors and out-of-vocabulary words are detected when both of these techniques are used together than when either is used alone. Alone, the acoustic methods can only detect between 2/3 and 3/4 of the recognition errors. Similarly, the higher-level constraint based methods The ACM Portal is published by the Association for Computing Machinery. Copyright © 2010 ACM, Inc. Terms of Usage Privacy Policy Code of Ethics Contact Us Useful downloads: Adobe Acrobat QuickTime Windows Media Player Real Player