Word and acoustic confidence annotation for large vocabulary speech recognition

We present improvements in confidence annotation of automatic speech recognizer output for large vocabulary, speak erindependent systems. Several strong additions to the set of predictor variablesused for this purpose are discussed. Extensions which allow prediction of separate types of errors , as opposed to the simple presence of an error, are presented. A new develop m nt, acoustic confidence annotation , is explored, in which a predictor is built that indicates the likely successes and failures of th e acoustic models alone. Four separate learning mechanisms are compared in terms of their ability to provide good confidence annotati ns from the same set of predictor variables. Performance figure s a reported on both read news (the North American Business news corpus) and conversational telephone speech (the Switchbo ard c rpus), both in American English. The Sphinx-II system [1] is u sed for the NAB tests. The Janus system [2] is used for the Switchb oard tests. 1. Annotation of Read Speech This section describes a confidence annotation system for th e Sphinx-II recognizer. The data used is the speaker-indepen dent portion of the NAB corpus. The development test set, used for training the confidence annotator, contains 1523 utterance s. The evaluation test set contains 1162 utterances. We annotate h ypothesized words with probabilities of membership in three class es: 1. correct ( P (C)): the correct word is guessed within 2 frames of its location in the forced alignment of the reference tran script, or 2. incorrect/oov ( P (OOV )): the word was guessed incorrectly because it “covered” a spoken word that was not in the recognizer’s dictionary, or 3. incorrect/other ( P (Other)): the wrong word was guessed or the right word was guessed with an incorrect segmentation. The predictor variables are constructed from: The contents of an N-best list that contains complete word and phone segmentation and score information for each of 150 elements, including the best hypothesis, Language model score and calculation source information from the same best-scoring hypothesis, Words pronunciations and their frequencies in acoustic tra ning materials, The results of a parallel “phone-only” decoding, in which th e recognizer is not constrained to either phone sequences fro m the dictionary or word sequences in the language model in its recognition of phone sequences, The results of a completely unconstrained frame-by-frame decoding in which the best possible acoustic score and basephone are determined for each frame of the utterance, Three distance metrics between basephones, including a simple match count at the frame level, a phonologicallybased similarity measure ( HWC ), and an empirically derived confusion-based distance measure, and After constructing predictor variables from these sources w compare various combinations in their ability to assign probab ilities of error class membership to words hypothesized by the recogni zer. Ability to successfully provide error class prediction is m easured with the reduction in cross-entropy measure described in [3 ]. Useful groups of predictor variables include: 1. percPhAll : The percentage of frames in the hypothesized word whose basephones match the basephones in the phoneonly decoding. 2. Combined+Duration: The combined acoustic and language model score together with the duration (in frames) of the word. (This group is an approximation to the information typically available to the recognizer during its normal dec oding passes.) 3. Nbest : The N-best homogeneity score, which is a measure of the weighted ratio of all paths which pass through the hypothesized word, as represented in the N-best list [4] [5] [6 ] [7]. 4. Nbest+AvgWF: The N-best homogeneity score together with the frame-averaged number of words present in the N-best lis t at the hypothesized location of the word in question. 5. LMscore: Language model score only. 6. LMscore+LMsource: Language model score together with the case/branch of its origin in the backoff algorithm which calculates the score. 7. avgHPAll: The average value of the HWC metric of the hypothesized word’s phone w.r. t. the phone-only decoding at the same point in the utterance. 8. percPhAll+avgHPAll : The combined effects of the simple match andHWC metrics. 9. ConfBest+avgSenR+avgHPBest+acNormS :A mix of several acoustic predictors that use the best senone at each frame as a reference or normalization. 10. TrainWC: Count of number of times the word was seen in the acoustic training data.