Abstract:
A system and a method to correct semantic interpretation recognition errors presented in this invention applies to Automatic Speech Recognition systems returning recognition results with semantic interpretations. The method finds the most likely intended semantic interpretation given the recognized sequence of words and the recognized semantic interpretation. The key point is the computation of the conditional probability of the recognized sequence of words given the recognized semantic interpretation and a particular intended semantic interpretation. It is done with the use of Conditional Language Models which are Statistical Language Models trained on a corpus of utterances collected under the condition of a particular recognized semantic interpretation and a particular intended semantic interpretation. Based on these conditional probabilities and the joint probabilities of the recognized and intended semantic interpretations, new semantic interpretation confidences are computed. The semantic interpretation with the maximum new confidence is declared the corrected semantic interpretation.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application claims priority to U.S. Provisional Patent Application No. 61/054,970 filed May 21, 2008 and U.S. Provisional Patent Application No. 61/130,120 filed May 27, 2008, which are hereby incorporated by reference in their entirety. 
    
    
     FIELD OF THE INVENTION 
     The present invention applies to post processing of speech recognition results. 
     STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
     Not Applicable. 
     BACKGROUND OF THE INVENTION 
     There exist many methods of post-processing of speech recognition results with the goal to improve word error rate, see (Ringer &amp; Allen, Error correction via a Post-Processor for continuous Speech recognition, 1996; Ringer &amp; Allen, Robust Error Correction of Continuous Speech Recognition, 1997; Jeong, et al., Speech Recognition Error Correction Using Maximum Entropy Language Model, 2004; U.S. Pat. No. 6,064,957). These post-processing methods are usually based on the paradigm of a channel with noise that takes as input a user utterance with a sequence of words contained in it, recognizes it and returns the recognized sequence of words as a distorted (noised) version. Usually these methods rely on a corpus of utterances that need to be transcribed in terms of words contained in them. Such transcribing is a time consuming, expensive and error-prone process. 
     The present invention can also be viewed through the same paradigm of the noisy channel, but with the following differences. The channel takes as input: a user utterance with (a) a sequence of words contained in it, and (b) a concept—a semantic tag representing the meaning of the uttered words. The channel outputs: (a) a recognized sequence of words and (b) a recognized concept—a recognized semantic tag. 
     Another difference consists in the channel quality criterion: instead of word error rate, we are interested in semantic tag error rate. So the post-processing aims at reducing the semantic tag recognition error rate using the recognized words just as a means, not as a goal. 
     BRIEF SUMMARY OF THE INVENTION 
     This invention presents a method and a system for post processing of speech recognition results. Recognition results are produced by an Automatic Speech Recognition (ASR) system using some mechanism to assign a semantic interpretation to a recognized sequence of words. One of the possible ways to do that is to use as single Statistical Language Model (SLM) describing the utterances for all possible semantic interpretations. In this case an ASR system would originally return just a sequence of recognized words from which the recognized semantic interpretation would be worked out. Another possible way would be to use a set of parallel SLMs each labeled with a particular semantic interpretation (semantic tag) so that a particular recognized sequence of words would be automatically assigned a recognized semantic interpretation defined by the particular SLM that accepted this recognized sequence of words. 
     The presented method finds such a potential intended semantic interpretation (which may be different from the recognized semantic interpretation) that minimizes the error rate of semantic interpretation recognition given the recognition result (recognized sequence of words and recognized semantic interpretation). Although the following explanation is based on the example with several parallel SLMs (called first level SLMs) each labeled with the appropriate semantic interpretation, the invention applies to ASR using any mechanism to assign recognized semantic interpretations. 
     The method works by maximizing the a-posteriori probability of the intended semantic concept (interpretation) which is computed with the use of Conditional Language Models that are created from a corpus of recognition results collected under the condition of a particular recognized semantic interpretation and a particular intended semantic interpretation. The method capitalizes on the correlation that exists between the intended semantic interpretation and the recognized sequence of words even if the recognized semantic interpretation is wrong, that is different from the intended one. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The accompanying drawings, wherein like referenced numerals are employed to designate like parts or steps, are included to provide further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. 
       In the drawings: 
         FIG. 1  shows the input to and the possible outputs of an Automatic Speech Recognition Engine (ASR). 
         FIG. 2  shows the principles behind the current invention where post-processing improves accuracy of the recognition of a semantic concept by compensating for error introduced in the channel. 
         FIG. 3  shows example of prior art that relies on the same basic principle of noisy channel, however applies to a different type of recognition results. 
         FIG. 4  shows how the use of multiple Statistical Language Models (SLMs) results in recognition of both the word sequence and semantic interpretation. 
         FIG. 5  shows how Conditional Language Models (CLMs) can be obtained for a setup from  FIG. 4 . 
         FIG. 6  shows how CLM obtained in  FIG. 5  can be used to improve recognition of semantic interpretations. 
         FIG. 7  shows the detailed steps of the process of training a CLM. 
         FIG. 8  shows the detailed steps of using the Conditional Language Models for post-processing. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       FIG. 1  shows a generic Automatic Speech Recognition (ASR) setup where an ASR engine  1  uses grammar  3  to recognize spoken utterance  2 . The result of the recognition may be either a sequence of words Wrec  4 , or a combination of a Wrec and a semantic interpretation Crec  5 . Generally, recognized Wrec  4  or Wrec &amp; Crec  5  will not always correspond to the content of the original utterance  2  due to recognition errors. 
       FIG. 2  shows the principles behind the idea of correcting the semantic interpretation returned by a speech recognition system. A speaker thinks  21  about some concept Cint  22  and then speaks  23  word or words Wint  24  that identify that concept (subscript “int” stands for “intended”). Words can be articulated in a variety of ways, with various prosody, and so on. These spoken words together with any background noise are then captured by a microphone. A microphone may have its own specific characteristics as far as the sound capture is concerned. The sound then is digitized and the digital representation enters the speech recognizer which performs the recognition. All of these steps  25  introduce distortions (noise). Speech recognizer will compensate for some of this noise, but it cannot compensate for all variations in the entirety of the channel  25 . This means that the result of recognition (semantic concept Crec and/or word sequence Wrec)  26  will be incorrect in some percentage of the recognition acts. Now, if some of the recognition errors result from a systematic noise introduced in the channel  25 , it is possible to apply post-processing  27  to compensate for these systematic errors, this way leading to, on average, better recognition Ĉpp  28 . 
     The method and system described in this patent may improve the accuracy of such semantic interpretation recognition using the recognized sequence of words and Conditional Language Models in the post-processing step  27 . 
     As a similar basic principle has been used in prior-art to improve recognition results, it is important to highlight how the present invention is different from that prior art. 
       FIG. 3  (based on: Ringer &amp; Allen, Robust Error Correction of Continuous Speech Recognition, 1997) showing prior art is similar to  FIG. 2  especially concerning the channel  13  as compared to channel  25 . The only difference between channels  25  and  13  is that the speech recognizer in  13  returns only word sequences as recognition results, as opposed to speech recognizer in channel  25  which additionally returns semantic interpretations (compare also recognition results  4  and  5  in  FIG. 1 ). 
     The steps of  FIG. 3  start with a speech act  11  where a user utters a sequence of words Wint  12 . In this case the intended semantic meaning of this word sequence is irrelevant. The aspects of articulation, prosody, audio environment, microphone, etc. in channel  13  are similar to those aspects in channel  25 . The speech recognizer returns a recognized sequence of words Wrec  14 . The task of error correcting post-processor  15  is to improve the average word error accuracy so that on average Ŵrec  16  better matches Wint. This goal is significantly different from the goal of the current invention. Also the data that the current invention operates on is significantly different from the data available and used in prior art depicted in  FIG. 3 . 
       FIG. 4  shows an example general speech recognition setup to which the current invention applies. Here the ASR  61  uses Statistical Language Models (SLMs)  62  for Speech Recognition. Such SLMs are usually trained on a transcribed corpus and they usually generate many more utterances than the corpus contained. In the following, to differentiate them better from the Conditional Language Models, we call them the First Level SLMs, and we proceed from the fact that they are created per each different semantic interpretation (concept) C[i] in a usual way, for instance, as n-gram SLMs. Using these First Level SLMs  62  ASR  61 , when presented with an utterance  60 , returns both semantic interpretation Crec and word sequence Wrec as recognition result  5 . 
     Usually, such SLMs over-generate a lot, for instance, they generate any sequence of words out of total lexicon contained in the corpus. Here is an extreme example. If a very frequent word “the” is in the corpus, a sequence “the the the the the” would be in the output set of phrases generated by such an SLM although it would have a very low probability. 
     Such kind of over-generation that creates word sequences irrelevant to a corresponding semantic interpretation (concept) and leads to recognition errors is called in the following a bad over-generation. The present invention aims at reducing the rate of errors related to bad over-generation. 
     Suppose, we have 3 concepts, and for all of them we created the First Level SLMs in some usual way from the corresponding corpora. Suppose a user uttered an utterance relevant to the second concept. The observations show that if the utterance is recognized as belonging to the right (second) concept, the sequence of recognized words is very close to what the user really said, so the bad over-generation was not used for covering this utterance. However, if the utterance was erroneously misrecognized as belonging to the wrong (say, first) concept, the recognized sequence of words usually shows bad over-generation examples: word combinations non-typical for the first concept utterances appear in this sequence to cover some critical words or phrases relevant to the second concept, but not relevant to the first one. 
     Now, suppose a user uttered an utterance relevant to the third concept, and it was misrecognized as belonging to the first concept due to the bad over-generation: this case of over-generation and the resulting recognized word sequence will be different, because the bad over-generation will try to cover some critical words or phrases from the third concept by the words from the first one. 
     Finally, suppose a user uttered an utterance relevant to the first concept, and it was correctly recognized as belonging to the first concept. Instances of bad over-generation would be less likely in this case, and if they happen, they would be statistically typical for the case when the first concept utterance was uttered and the first concept was recognized, because, the first-concept-specific words and phrases would be most likely recognized correctly and in the right order. 
     The conclusion is: given the recognized concept, the recognized sequence of words is statistically dependent on the intended concept, which is the concept that a user intended to convey when he produced the utterance. 
     This statistical dependence is the basis of the present invention and is represented by the Conditional Language Models depending on the recognized and intended concepts. 
     More precisely, a Conditional Language Model CLM(Crec, Cint) is a Statistical Language Model depending on a recognized semantic interpretation (concept) Crec and the intended semantic interpretation (concept) Cint. 
       FIG. 5  shows a general setup in which such Conditional Language Model CLM(Crec, Cint)  64  is trained  63 . CLMs are constructed from the corpus  68  containing utterances labeled  67  with (Wrec, Crec) which come from the recognition results  5  (produced by ASR  61  using the First Level SLMs  62 ). The intended semantic interpretation Cint of the uttered words  60  is detected and added manually offline 
     Once such Conditional Language Models are created, they can be used for getting more accurate semantic interpretation results.  FIG. 6  shows a modified setup of  FIG. 4  where now CLM  64  is used to post-process  65  the recognition result  5  to generate improved semantic interpretation C*int  66  that on average would have a lower error rate compared to Crec. 
     Suppose ASR  61  returned a recognized word sequence Wrec and semantic interpretation Crec ( 5 ). The method described in this invention will find such a potentially intended semantic interpretation C*int  66  that gives the best match between the recognized word sequence Wrec and the Conditional Language Model CLM(Crec, Cint). 
       FIG. 7  shows the steps that are done during training of the Conditional Language Model  64 . It starts with collection  41  of the corpus of utterances from the real traffic in the context of the same speech application. Every utterance is recognized against the First Level SLMs  44  to find for it (Crec, Wrec)  45 , where Crec—recognized concept and Wrec—recognized sequence of words. This is done automatically in online mode when the application is processing real incoming calls. The following steps are done offline. The utterances are labeled  42  with Cint—correct intended concept  43  which is done with the help of a human. After that, the entire corpus is split  46  into subsets subcorpus(Crec, Cint) of utterances everyone of which was recognized as concept Crec while the users expressed in them the concept Cint. Next, the a-priori probability of a pair (Crec, Cint) is evaluated  47  by dividing the size of subcorpus(Crec, Cint) by the size of the entire corpus: 
     
       
         
           
             
               P 
               ⁡ 
               
                 ( 
                 
                   
                     C 
                     rec 
                   
                   , 
                   
                     C 
                     int 
                   
                 
                 ) 
               
             
             = 
             
               
                 # 
                 ⁢ 
                 
                   subcorpus 
                   ⁡ 
                   
                     ( 
                     
                       
                         C 
                         rec 
                       
                       , 
                       
                         C 
                         int 
                       
                     
                     ) 
                   
                 
               
               
                 # 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 corpus 
               
             
           
         
       
     
     Finally, using subcorpus(Crec, Cint) Conditional Language Model CLM(Crec, Cint)  64  is trained  48  for some pairs (Crec, Cint), for instance, for pairs whose likelihood of occurrence P(Crec, Cint) is high enough. The recognized word sequences Wrec for all the utterances in subcorpus(Crec, Cint) constitute the corpus that is used to create CLM(Crec, Cint)—a statistical language model for the pair (Crec, Cint). These statistical language models can be, but are not limited by, n-gram Statistical Language Models. 
     Once statistical Conditional Language Models are created for some pairs (Crec, Cint), the conditional probability P(Wrec/Crec, Cint) for a particular word sequence Wrec can be computed from such models in standard ways. 
     One of the important benefits of this method is that it is not necessary to manually transcribe the corpus of spoken uttreances for CLMs in terms of words. It is sufficient just to label every utterance with the intended concept, while the transcription in terms of recognized words and recognized concept is given by the recognizer. 
       FIG. 8  shows steps executed during use of the CLM for post-processing recognition results like, e.g. in  FIG. 6 . It starts with recognition  44  of an input utterance  51  against the First Level SLMs finding recognized sequence of words Wrec and recognized semantic interpretation Crec  45 . 
     For every possible intended concept Cint, for which exists CLM(Crec, Cint)  64 , the method computes  55  from this CLM the conditional probability of Wrec given (Crec, Cint), which is P(Wrec/Crec, Cint). 
     Next, the method finds C*int  57  that maximizes P(Wrec/Crec, Cint)*P(Crec, Cint)  56 , which is proportional to the a-posteriori probability of an intended concept Cint given the recognized sequence of words Wrec and the recognized concept Crec. 
     More formally speaking, in order to minimize the semantic interpretation Cint recognition error rate, we maximize the a-posteriori probability of an intended concept Cint given the recognized concept Crec and recognized word sequence Wrec: 
     
       
         
           
             
               C 
               int 
               * 
             
             = 
             
               
                 arg 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   
                     max 
                     
                       C 
                       int 
                     
                   
                   ⁢ 
                   
                     P 
                     ⁡ 
                     
                       ( 
                       
                         
                           
                             C 
                             int 
                           
                           / 
                           
                             C 
                             rec 
                           
                         
                         , 
                         
                           W 
                           rec 
                         
                       
                       ) 
                     
                   
                 
               
               = 
               
                 
                   arg 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     
                       max 
                       
                         C 
                         int 
                       
                     
                     ⁢ 
                     
                       
                         P 
                         ⁡ 
                         
                           ( 
                           
                             
                               
                                 C 
                                 
                                   int 
                                   , 
                                 
                               
                               ⁢ 
                               
                                 C 
                                 rec 
                               
                             
                             , 
                             
                               W 
                               rec 
                             
                           
                           ) 
                         
                       
                       
                         P 
                         ⁡ 
                         
                           ( 
                           
                             
                               C 
                               rec 
                             
                             , 
                             
                               W 
                               rec 
                             
                           
                           ) 
                         
                       
                     
                   
                 
                 = 
                 
                   arg 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     
                       max 
                       
                         C 
                         int 
                       
                     
                     ⁢ 
                     
                       
                         
                           P 
                           ⁡ 
                           
                             ( 
                             
                               
                                 
                                   W 
                                   rec 
                                 
                                 / 
                                 
                                   C 
                                   rec 
                                 
                               
                               , 
                               
                                 C 
                                 int 
                               
                             
                             ) 
                           
                         
                         * 
                         
                           P 
                           ⁡ 
                           
                             ( 
                             
                               
                                 C 
                                 rec 
                               
                               , 
                               
                                 C 
                                 int 
                               
                             
                             ) 
                           
                         
                       
                       
                         P 
                         ⁡ 
                         
                           ( 
                           
                             
                               C 
                               rec 
                             
                             , 
                             
                               W 
                               rec 
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     Where: 
     P(C rec ,C int ) is the a-priori probability of the combination of the recognized concept Crec and of the intended concept Cint, and 
     P(W rec /C rec ,C int ) is the conditional probability of word sequence Wrec given the recognized concept Crec and the intended concept Cint, and is computed from the Conditional Language Model CLM(Crec, Cint). 
     The probability P(C rec ,W rec ) does not affect the choice of the intended concept Cint and can be omitted. 
     C*int is then declared as the post-processing result  57 . 
     Although the main focus of the present invention is to improve the recognition of semantic tags, it can also be used to improve the word error rate by simply re-recognizing the same utterance against the First Level SLM corresponding to the semantic tag C*int resulting from the post-processing. 
     In addition to finding a better hypothesis C*int for the intended concept (semantic tag) Cint, the presented invention can be used to improve the decision whether (a) to accept the final post-processing result without confirmation, (b) to confirm it or (c) to reject it. This can be done by using a new confidence computed from the a-posteriori probability of the best intended concept hypothesis (or components that allow to compute it as explained above) together with the original recognition confidence for Crec. 
     It will be appreciated by those skilled in the art that changes could be made to the embodiments described above without departing from the broad inventive concept thereof. It is understood, therefore, that this invention is not limited to the particular embodiments disclosed, but is intended to cover modifications within the spirit and scope of the present invention as defined in the appended claims.