Abstract:
A dynamic exponential, feature-based, language model is continually adjusted per utterance by a user, based on the user&#39;s usage history. This adjustment of the model is done incrementally per user, over a large number of users, each with a unique history. The user history can include previously recognized utterances, text queries, and other user inputs. The history data for a user is processed to derive features. These features are then added into the language model dynamically for that user.

Description:
BACKGROUND 
       [0001]    Speech recognition generally involves two kinds of pattern recognition. The first kind of pattern recognition using an acoustic model to identify sounds and sequences of sounds that may be words or parts of words. The second kind of pattern recognition uses a language model to identify sequences of words. The language model provides a linguistically based score representing the probability of a word given a word history. An n-gram model means the word history is n words long. Both models typically are probabilistic and are generated from a training set of valid utterances. Other than this similarity, however, both of these models typically are designed and implemented, and generally treated, as independent from each other, except they are used in an interleaved fashion to recognize words in an utterance. Such techniques are described generally in F. Jelinek,  Statistical Methods for Speech Recognition , MIT Press, 1997. 
         [0002]    There are several different classes of language models. One class of language models is exponential language models, such as “model M” described in “Performance Prediction for Exponential Language Models,” by Stanley Chen, in the proceedings of NAACL-HLT, 2009. In an exponential language model, word n-gram probabilities are modeled with a log-linear model, and word-class information is used in the definition of the features. Assuming an n-gram model on words w, and a lambda λ for each word history in the training data, subject to length and frequency restrictions, the form of the basic exponential language model is (Equation (0): 
         [0000]    
       
         
           
             
               P 
                
               
                 ( 
                 
                   
                     w 
                     i 
                   
                   | 
                   
                     
                       w 
                       
                         i 
                         - 
                         n 
                         + 
                         1 
                       
                     
                      
                     
                         
                     
                      
                     … 
                      
                     
                         
                     
                      
                     
                       w 
                       
                         i 
                         - 
                         1 
                       
                     
                   
                 
                 ) 
               
             
             = 
             
               
                 
                   exp 
                    
                   
                     ( 
                     
                       
                         λ 
                         
                           
                             w 
                             
                               
                                 i 
                                 - 
                                 n 
                                 + 
                                 1 
                               
                                
                               
                                   
                               
                             
                           
                            
                           … 
                            
                           
                               
                           
                            
                           
                             w 
                             
                               i 
                               - 
                               1 
                             
                           
                            
                           
                             w 
                             i 
                           
                         
                       
                       + 
                       … 
                       + 
                       
                         λ 
                         
                           
                             w 
                             
                               i 
                               - 
                               1 
                             
                           
                            
                           
                             w 
                             i 
                           
                         
                       
                       + 
                       
                         λ 
                         
                           w 
                           i 
                         
                       
                     
                     ) 
                   
                 
                 
                   
                     ∑ 
                     
                       w 
                       ′ 
                     
                   
                    
                   
                     exp 
                      
                     
                       ( 
                       
                         
                           λ 
                           
                             
                               w 
                               
                                 
                                   i 
                                   - 
                                   n 
                                   + 
                                   1 
                                 
                                  
                                 
                                     
                                 
                               
                             
                              
                             … 
                              
                             
                                 
                             
                              
                             
                               w 
                               
                                 i 
                                 - 
                                 1 
                               
                             
                              
                             
                               w 
                               ′ 
                             
                           
                         
                         + 
                         … 
                         + 
                         
                           λ 
                           
                             
                               w 
                               
                                 i 
                                 - 
                                 1 
                               
                             
                              
                             
                               w 
                               ′ 
                             
                           
                         
                         + 
                         
                           λ 
                           
                             w 
                             ′ 
                           
                         
                       
                       ) 
                     
                   
                 
               
               . 
             
           
         
       
     
         [0003]    In this model, the presence of an n-gram sequence is a feature, and there is a lambda for each feature. 
         [0004]    Typically, one or more fixed language models are built and used unchanged for an entire data set, for both training and classification. If a fixed set of language models is used, interpolation is performed on one or more of the large language models in the set. To extend such a model to accommodate information specific to a user, it is common to interpolate a user-specific n-gram language model with a generic n-gram language model. However, this solution is computationally inefficient for large numbers of users. 
       SUMMARY 
       [0005]    This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. 
         [0006]    A dynamic exponential, feature-based, language model is continually adjusted per utterance by a user, based on the user&#39;s usage history. This adjustment of the model is done incrementally per user, over a large number of users, each with a unique history. The user history can include previously recognized utterances, text queries, and other user inputs. The history data for a user is processed to derive features. These features are then added into the language model dynamically for that user. 
         [0007]    Such a system can be embodied, for example, as a computer system or computing machine, a computer implemented process, or an article of manufacture that includes storage and instructions on the storage that are for a computer. The system stores data defining an exponential language model and data defining user history. 
         [0008]    Accordingly, in one aspect, personalization features are defined, from the user history, having corresponding features in the language model. An input to the language model is processed by using the features in the language model and the personalization features. 
         [0009]    In various embodiments, the user history is continually updated according to user activity. Personalization features of a same length, such as all one word features, can have a same weight. In other words, the weights are tied together for each set of n-gram weights. The personalization features can include word based features including words from the user history. The personalization features can include class based features including classes of words from the user history. The personalization features can include features with both words and class information. The personalization features can include information derived from a locality of the user and words associated with the locality. 
         [0010]    In another aspect, training an exponential language model involves receiving a set of words for training the language model and sorting the n-grams of words. The words are applied to the model and parameters of the model are adjusted according to outputs of the model. Adjusting the parameters of the model includes computing a normalization constant, wherein the computation of the normalization constant includes a component for unigrams which is computed once per class, and a higher order component computed on demand for each input. 
         [0011]    In the following description, reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific example implementations of this technique. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the disclosure. 
     
    
     
       DESCRIPTION OF THE DRAWINGS 
         [0012]      FIG. 1  is a data flow diagram illustrating an example environment in which voice recognition can be used. 
           [0013]      FIG. 2  is a data flow diagram illustrating an example implementation of personalizing a language model using user history. 
           [0014]      FIG. 3  is a flowchart describing an example implementation of a training process for a language model. 
           [0015]      FIG. 4  is a flowchart describing an example implementation of sorting data for training. 
           [0016]      FIG. 5  is a flowchart describing an example implementation of a voice recognition process 
           [0017]      FIG. 6  is a flowchart describing an example implementation of a personalizing a language model using user history. 
           [0018]      FIG. 7  is a block diagram of an example computing device in which such a system can be implemented. 
       
    
    
     DETAILED DESCRIPTION 
       [0019]    The following section provides an example operating environment in which personalization of a language model can be implemented. 
         [0020]    Referring to  FIG. 1 , a search engine or information portal  100  provides a way for many users to access information and/or services from user devices  102  over a computer network  104  such as the internet. Example user devices include mobile device, mobile phones, handheld computers, laptop computers, notebook computers, desktop computers, and the like. The information portal  100  typically is supported by one or more server computers (not shown), with a primary address that is used to access the service over the network, but which redirects communication and traffic to a server computer that works with the user device  102  during a communication session. 
         [0021]    The interaction between the user device  102  and the information portal  100  involves the user device sending user input  106  to the information portal  100 . The user input can be any kind of data that is indicative of information requested by the user from the information portal  100 , such as a resource identifier (e.g., a uniform resource locator (URL)), query terms, form data and the like. The information portal  100  responds to the request with the requested information  108 , which can be search results, a requested document, or other information. 
         [0022]    The user input, in some cases, includes speech  110 . Speech is processed by a voice processing system  112 , to provide one or more recognized words  114 . The recognized words also can be used by the information portal  100  to provide information. For example, the recognized words can be search terms, a URL, form data and the like. 
         [0023]    In many information portals, information about the user input and information requested are stored as history data  120 . For each user, a history data file can be stored in storage  122 . The history data  120  typically is in a simple text format or in a markup language format, and includes several days, if not months or even years, of information. 
         [0024]    The voice processing system uses a language model  116  to help recognize speech, which provides information about what strings of words can be validly recognized based on a training set of phrases and sentences, as described in more detail below. Such a model typically is developed once by a training system  124  using a large training set, and the same model is used for many users. The language model  116  is dynamically updated on a per-user, per-utterance basis with information from the user&#39;s history data  120 , as will now be described in more detail. 
         [0025]    The purpose of dynamically updating the model on a per-user, per-utterance basis is to improve recognition. For example, assume the user has recently accessed a web page describing the “Triple Door Lounge.” If the user speaks the phrase “Call the Triple Door Lounge,” then this phrase in the user&#39;s history will increase the probability that the phrase “Triple Dour Lounge” is recognized. How this is achieved will be described below. 
         [0026]    Given this context, an example implementation of the language model-based voice recognition system will be described in more detail in connection with  FIGS. 2-5 . 
         [0027]    In  FIG. 2 , a class-based exponential language model  200  is used for voice recognition. While there are many types of language models, an exponential language model, and in particular, a class-based exponential language model, is suitable for extension using user history data to improve recognition on a dynamic, per-user, per-utterance basis. 
         [0028]    An exponential language model is generally described by equation (1) above. Such a language model can be factored and defined in another form, described below as equation (1)(b). In this model, each word is assigned to a single class, and n-gram probabilities are estimated as the product of a class part and a word part. In this case, the model is defined by Equation (1)(b): 
         [0000]        P ( w   i   |w   i 2   w   i 1 )= P ( c   i   |c   i 2   c   i 1   ,w   i 2   w   i 1 ) P ( w   i   |w   i 2   w   i 1   ,c   i ) 
         [0000]    where (the following being Equations (2) and (3)): 
         [0000]    
       
         
           
             
               P 
                
               
                 ( 
                 
                   
                     
                       w 
                       i 
                     
                     | 
                     
                       
                         w 
                         
                           i 
                           - 
                           2 
                         
                       
                        
                       
                         w 
                         
                           i 
                           - 
                           1 
                         
                       
                     
                   
                   , 
                   
                     c 
                     i 
                   
                 
                 ) 
               
             
             = 
             
               
                 exp 
                  
                 
                   ( 
                   
                     
                       λ 
                       
                         
                           w 
                           
                             i 
                             - 
                             2 
                           
                         
                          
                         
                           w 
                           
                             i 
                             - 
                             1 
                           
                         
                          
                         
                           w 
                           i 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         
                           w 
                           
                             i 
                             - 
                             1 
                           
                         
                          
                         
                           w 
                           i 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         w 
                         i 
                       
                     
                   
                   ) 
                 
               
               
                 
                   ∑ 
                   
                     
                       w 
                       ′ 
                     
                     ∈ 
                     
                       c 
                       i 
                     
                   
                   
                       
                   
                 
                  
                 
                   exp 
                    
                   
                     ( 
                     
                       
                         λ 
                         
                           
                             w 
                             
                               i 
                               - 
                               2 
                             
                           
                            
                           
                             w 
                             
                               i 
                               - 
                               1 
                             
                           
                            
                           
                             w 
                             ′ 
                           
                         
                       
                       + 
                       
                         λ 
                         
                           
                             w 
                             
                               i 
                               - 
                               1 
                             
                           
                            
                           
                             w 
                             ′ 
                           
                         
                       
                       + 
                       
                         λ 
                         
                           w 
                           ′ 
                         
                       
                     
                     ) 
                   
                 
               
             
           
         
       
       
         
           
             
               P 
                
               
                 ( 
                 
                   
                     
                       c 
                       i 
                     
                     | 
                     
                       
                         c 
                         
                           i 
                           - 
                           2 
                         
                       
                        
                       
                         c 
                         
                           i 
                           - 
                           1 
                         
                       
                     
                   
                   , 
                   
                     
                       w 
                       
                         i 
                         - 
                         2 
                       
                     
                      
                     
                       w 
                       
                         i 
                         - 
                         1 
                       
                     
                   
                 
                 ) 
               
             
             = 
             
               
                 exp 
                  
                 
                   ( 
                   
                     
                       
                         
                           
                             λ 
                             
                               
                                 c 
                                 
                                   i 
                                   - 
                                   2 
                                 
                               
                                
                               
                                 c 
                                 
                                   i 
                                   - 
                                   1 
                                 
                               
                                
                               
                                 c 
                                 i 
                               
                             
                           
                           + 
                           
                             λ 
                             
                               
                                 
                                   c 
                                   
                                     i 
                                     - 
                                     1 
                                   
                                 
                                  
                                 
                                   c 
                                   i 
                                 
                               
                               + 
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             λ 
                             
                               c 
                               i 
                             
                           
                           + 
                           
                             λ 
                             
                               
                                 w 
                                 
                                   i 
                                   - 
                                   2 
                                 
                               
                                
                               
                                 w 
                                 
                                   i 
                                   - 
                                   1 
                                 
                               
                                
                               
                                 c 
                                 i 
                               
                             
                           
                           + 
                           
                             λ 
                             
                               
                                 w 
                                 
                                   i 
                                   - 
                                   1 
                                 
                               
                                
                               
                                 c 
                                 i 
                               
                             
                           
                         
                       
                     
                   
                   ) 
                 
               
               
                 
                   ∑ 
                   
                     c 
                     ′ 
                   
                 
                  
                 
                   exp 
                    
                   
                     ( 
                     
                       
                         
                           
                             
                               λ 
                               
                                 
                                   c 
                                   
                                     i 
                                     - 
                                     2 
                                   
                                 
                                  
                                 
                                   c 
                                   
                                     i 
                                     - 
                                     1 
                                   
                                 
                                  
                                 
                                   c 
                                   ′ 
                                 
                               
                             
                             + 
                             
                               λ 
                               
                                 
                                   
                                     c 
                                     
                                       i 
                                       - 
                                       1 
                                     
                                   
                                    
                                   
                                     c 
                                     ′ 
                                   
                                 
                                 + 
                               
                             
                           
                         
                       
                       
                         
                           
                             
                               λ 
                               
                                 c 
                                 ′ 
                               
                             
                             + 
                             
                               λ 
                               
                                 
                                   w 
                                   
                                     i 
                                     - 
                                     2 
                                   
                                 
                                  
                                 
                                   w 
                                   
                                     i 
                                     - 
                                     1 
                                   
                                 
                                  
                                 
                                   c 
                                   ′ 
                                 
                               
                             
                             + 
                             
                               λ 
                               
                                 
                                   w 
                                   
                                     i 
                                     - 
                                     1 
                                   
                                 
                                  
                                 
                                   c 
                                   ′ 
                                 
                               
                             
                           
                         
                       
                     
                     ) 
                   
                 
               
             
           
         
       
     
         [0029]    where c i  represents the class of word w i  and w′εc i  ranges over the members of class c i . A λ parameter may be defined for each n-gram pattern in the training data, or restricted to commonly occurring patterns. Each word is mapped deterministically to a single class, ensuring the probabilistic validity of Eq&#39;n (1)(b). The word classing may be done with a variety of methods, such as described in P. Brown, V. D. Pietra, P. deSouza, J. Lai, and R. Mercer, “Class based n-gram models of natural language,” Computational Linguistics, vol. 18, no. 4, 1992. 
         [0030]    These expressions represent the probability associated with an n-gram (e.g., “The Triple Door”) that can be a combination of words or words and classes (e.g., “The Triple [noun]”). These probabilities are determined by training the model with a set of sentences and phrases from the language. The equations above assume that n is three, but the model can be extended to any number of n. Thus, given an n-gram as an input  202  in  FIG. 2 , a probability  204  is generated which represents a conditional probability of a word given preceding n−1 words (as the left-hand-side in of the equation (1)(b) above for n=3). 
         [0031]    These probabilities can be modified dynamically on a per-user, per-utterance basis by utilizing the user history  206 . In particular, given each feature in the language model  200 , a binary personalization feature is derived from the history, for which the value is 1 if a suffix of words occurs in some n-gram in the user history. A weight for the feature is determined by the feature&#39;s length. There is one weight for unigrams, one for bigrams and so on. Separate weights can be provided for the class-based and word-based part of the model. The result, therefore, is a set of weights that can be applied to the probability associated with an n-gram in response to an input to the language model. 
         [0032]    With a larger amount of data, and some additional data, more specific features could be added, such as “the user has uttered this word sequence in his current location” or “the user has requested The Triple Door” or “the user has requested Wal-Mart in Everett” as distinct features with different weights. Such additional data to define a more specific feature can include, but is not limited to, the user location related to the history data, the source of the history data (whether an utterance, a query, a click-through of a link, a data form, a contact in an address book, etc.), and other metadata stored with the history data. 
         [0033]    The generation of the language model through training will now be described in connection with  FIGS. 3 and 4 . 
         [0034]    Such a language model is trained using a large dataset of sentences and phrases, which can include over one billion words. Training generally involves organizing the training data ( 300  in  FIG. 3 ), so as to make computation more efficient. Next, the model is trained  302  by adjusting values of the model so as to minimize errors. Maximum entropy training is described, for example, in Berger et al., “A Maximum Entropy Approach to Natural Language Processing,” Computational Linguistics 1996. Normalizers used in the training process can be cached  304  upon completion of training. 
         [0035]    One implementation for organizing the data, referring now to  FIG. 4  is the following. First, every n-gram occurrence in the data set is written  400  to a data file or other structure allowing the n-grams to be sorted easily. The file is sorted  402 , for example, by the first word in each n-gram. Duplicates are eliminated  404 . Unknown words can be replaced  406  by a token or constant value, e.g. “&lt;unk&gt;”, so as to normalize the dataset. This data can be further sorted  408  by n−1 gram word history. For example, for a set of n-grams to be sorted, the first n−1 words in each n-gram are used as the key for sorting, so that n-grams that share the same first n−1 words are always grouped together. If there is a tie in the sorting, n-grams that share the same first n−1 words can be further sorted by the class of the n-th word, as an example way to break the tie. This implementation is merely one way of sorting the data and the invention is not limited to this implementation. A variety of sorting techniques can be implemented, examples of which are described in Wu and Khudanpur, “Efficient Training Methods for Maximum Entropy Language Modeling,” in ICSLP 2000. 
         [0036]    In the training process, word normalization involves substantial computation. Most word pairs never occur, and this fact can be used to reduce the computation to an amount proportional to the number of bigram successors of the second to last word. Thus, Recall that λ w     i-2     w     i-1     w′  and λ w     i-1     w′  only exist for n-grams that occur in the training data. Thus if succs(c i , w i 1 ) is defined to the words in class c i  that have been seen following we may write (with the three addends being Equations (4), (5) and (6), in order from left to right): 
         [0000]    
       
         
           
             
               
                 ∑ 
                 
                   
                     w 
                     ′ 
                   
                   ∈ 
                   
                     c 
                     i 
                   
                 
                 
                     
                 
               
                
               
                 exp 
                  
                 
                   ( 
                   
                     
                       λ 
                       
                         
                           w 
                           
                             i 
                             - 
                             2 
                           
                         
                          
                         
                           w 
                           
                             i 
                             - 
                             1 
                           
                         
                          
                         
                           w 
                           ′ 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         
                           w 
                           
                             i 
                             - 
                             1 
                           
                         
                          
                         
                           w 
                           ′ 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         w 
                         ′ 
                       
                     
                   
                   ) 
                 
               
             
             = 
             
               
                 
                   ∑ 
                   
                     
                       w 
                       ′ 
                     
                     ∈ 
                     
                       c 
                       i 
                     
                   
                   
                       
                   
                 
                  
                 
                   exp 
                    
                   
                     ( 
                     
                       λ 
                       
                         w 
                         ′ 
                       
                     
                     ) 
                   
                 
               
               + 
               
                 
                   ∑ 
                   
                     
                       w 
                       ′ 
                     
                     ∈ 
                     
                       succs 
                        
                       
                         ( 
                         
                           
                             c 
                             i 
                           
                           , 
                           
                             w 
                             
                               i 
                               - 
                               1 
                             
                           
                         
                         ) 
                       
                     
                   
                   
                       
                   
                 
                  
                 
                   exp 
                    
                   
                     ( 
                     
                       
                         λ 
                         
                           
                             w 
                             
                               i 
                               - 
                               2 
                             
                           
                            
                           
                             w 
                             
                               i 
                               - 
                               1 
                             
                           
                            
                           
                             w 
                             ′ 
                           
                         
                       
                       + 
                       
                         λ 
                         
                           
                             w 
                             
                               i 
                               - 
                               1 
                             
                           
                            
                           
                             w 
                             ′ 
                           
                         
                       
                       + 
                       
                         λ 
                         
                           w 
                           ′ 
                         
                       
                     
                     ) 
                   
                 
               
               - 
               
                 
                   ∑ 
                   
                     
                       w 
                       ′ 
                     
                     ∈ 
                     
                       succs 
                        
                       
                         ( 
                         
                           
                             c 
                             i 
                           
                           , 
                           
                             w 
                             
                               i 
                               - 
                               1 
                             
                           
                         
                         ) 
                       
                     
                   
                   
                       
                   
                 
                  
                 
                   exp 
                    
                   
                     ( 
                     
                       λ 
                       
                         w 
                         ′ 
                       
                     
                     ) 
                   
                 
               
             
           
         
       
     
         [0037]    Equation (4) is thus computed once per class, after each round of parameter re-estimation. Equations (5) and (6) are computed on demand on only involve considering the bigram successors of the second-to-last word. 
         [0038]    Additionally, training a class-based exponential language model such as Model M involves implementing L 1  and L 2  regularization. In this implementation, the R-prop gradient descent method, described in M. Reidmiller, “RProp—Description and Implementation Details,” University of Karlsruhe Tech. Report, 1994, works efficiently. 
         [0039]    Turning now to the recognition process using a trained model,  FIG. 5  is a flowchart describing recognition. First, the voice recognitions system (e.g.,  112  in  FIG. 1 ) receives  500  the utterance. The voice recognition system processes  502  the speech using an acoustic model for decoding and the language model modified by the user history. A word or set of words is then selected  506  as the recognized word or set of words. This information is stored  508  in the user history (e.g., history data  120  in  FIG. 1 ). 
         [0040]    The process of recognizing a word using a user history involves processing  600  the user history to extract a set of personalization features. For example, if a user&#39;s history includes “Triple Door Lounge” and “The Century Ballroom”, then word features include “Triple Door”, “Door”, “Century Ballroom”, “Century”, and class features include “Article Adjective Noun.” Each standard feature of the language model is associated with an additional feature of value “1” if the standard feature is in the set of personalization features. A weight is associated with each personalization feature. All n-grams of the same length have the same weight. Separate weights are used for word and class features, which allows the more specific word features to be weighted differently from less specific class features. More weights provide more degrees of freedom in the model, thus potentially increasing modeling power, but also increasing training complexity. Separating weights for words and classes is effective in increasing the modeling power while increasing the number of weights minimally. The values of the weights are typically between −10 and 10, although there are no restrictions in practice. Thus, the user history is transformed into a set of personalization features, each with a weight. A personalization feature can also be derived from words in the user history and other information. For example, a feature can be derived by combining a user&#39;s location with a user history and other words. The words associated with a user location can be used. For example, a feature can be a log of a ratio of a word&#39;s frequency in the user&#39;s locality to its overall frequency in a corpus of documents. 
         [0041]    Given an n-gram, each personalization feature of which it is a stem is then identified  602 . All of these personalization features that match are set  604  to a value of “1”. The weights corresponding to these features have been preset and are known. The weights for the personalization features set to “1” are combined  606  with the model to provide the final probability for the given n-gram. Equations 2 and 3 now become Equation 7: 
         [0000]    
       
         
           
             
                 
             
              
             
               
                 P 
                  
                 
                   ( 
                   
                     
                       
                         w 
                         3 
                       
                       | 
                       
                         
                           w 
                           1 
                         
                          
                         
                           w 
                           2 
                         
                       
                     
                     , 
                     h 
                   
                   ) 
                 
               
               = 
               
                 
                   P 
                    
                   
                     ( 
                     
                       
                         
                           c 
                           3 
                         
                         | 
                         
                           
                             c 
                             1 
                           
                            
                           
                             c 
                             2 
                           
                         
                       
                       , 
                       
                         
                           w 
                           1 
                         
                          
                         
                           w 
                           2 
                         
                       
                       , 
                       h 
                     
                     ) 
                   
                 
                  
                 
                   P 
                    
                   
                     ( 
                     
                       
                         
                           w 
                           3 
                         
                         | 
                         
                           
                             w 
                             1 
                           
                            
                           
                             w 
                             2 
                           
                         
                       
                       , 
                       
                         c 
                         3 
                       
                       , 
                       h 
                     
                     ) 
                   
                 
               
             
           
         
       
       
         
           
             
               P 
                
               
                 ( 
                 
                   
                     
                       c 
                       3 
                     
                     | 
                     
                       
                         c 
                         1 
                       
                        
                       
                         c 
                         2 
                       
                     
                   
                   , 
                   
                     
                       w 
                       1 
                     
                      
                     
                       w 
                       2 
                     
                   
                   , 
                   h 
                 
                 ) 
               
             
             = 
             
               
                 1 
                 Z 
               
                
               
                 exp 
                  
                 
                   ( 
                   
                     
                       λ 
                       
                         
                           c 
                           1 
                         
                          
                         
                           c 
                           2 
                         
                          
                         
                           c 
                           3 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         
                           c 
                           2 
                         
                          
                         
                           c 
                           3 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         c 
                         3 
                       
                     
                     + 
                     
                       λ 
                       
                         
                           w 
                           1 
                         
                          
                         
                           w 
                           2 
                         
                          
                         
                           c 
                           3 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         
                           w 
                           2 
                         
                          
                         
                           c 
                           3 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         
                           c 
                           1 
                         
                          
                         
                           c 
                           2 
                         
                          
                         
                           c 
                           3 
                         
                       
                       p 
                     
                     + 
                     
                       λ 
                       
                         
                           c 
                           2 
                         
                          
                         
                           c 
                           3 
                         
                       
                       p 
                     
                     + 
                     
                       λ 
                       
                         c 
                         3 
                       
                       p 
                     
                     + 
                     
                       λ 
                       
                         
                           w 
                           1 
                         
                          
                         
                           w 
                           2 
                         
                          
                         
                           c 
                           3 
                         
                       
                       p 
                     
                     + 
                     
                       λ 
                       
                         
                           w 
                           2 
                         
                          
                         
                           c 
                           3 
                         
                       
                       p 
                     
                   
                   ) 
                 
               
             
           
         
       
       
         
           
             
               P 
                
               
                 ( 
                 
                   
                     
                       w 
                       3 
                     
                     | 
                     
                       
                         w 
                         1 
                       
                        
                       
                         w 
                         2 
                       
                     
                   
                   , 
                   
                     c 
                     3 
                   
                 
                 ) 
               
             
             = 
             
               
                 1 
                 Z 
               
                
               
                 exp 
                  
                 
                   ( 
                   
                     
                       λ 
                       
                         
                           w 
                           1 
                         
                          
                         
                           w 
                           2 
                         
                          
                         
                           w 
                           3 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         
                           w 
                           2 
                         
                          
                         
                           w 
                           3 
                         
                       
                     
                     + 
                     
                       λ 
                       
                         w 
                         3 
                       
                     
                     + 
                     
                       λ 
                       
                         
                           w 
                           1 
                         
                          
                         
                           w 
                           2 
                         
                          
                         
                           w 
                           3 
                         
                       
                       p 
                     
                     + 
                     
                       λ 
                       
                         
                           w 
                           2 
                         
                          
                         
                           w 
                           3 
                         
                       
                       p 
                     
                     + 
                     
                       λ 
                       
                         w 
                         3 
                       
                       p 
                     
                   
                   ) 
                 
               
             
           
         
       
     
         [0000]    where h is the user&#39;s history, Z is the normalization constant (analogous to the denominators in Eq&#39;ns. 2 and 3), and λ p  is the weight associated with seeing the corresponding feature in the history. This weight is tied across features, meaning all features of the same length, e.g., all one-word features, have the same weight, as described above. 
         [0042]    Using these techniques, user history can be efficiently used to personalize a language model without retraining the model. The model can be updated dynamically during use, on a per-user, per-utterance basis. 
         [0043]    Having now described an example implementation, a computing environment in which such a system is designed to operate will now be described. The following description is intended to provide a brief, general description of a suitable computing environment in which this system can be implemented. The system can be implemented with numerous general purpose or special purpose computing hardware configurations. Examples of well known computing devices that may be suitable include, but are not limited to, personal computers, server computers, hand-held or laptop devices (for example, media players, notebook computers, cellular phones, personal data assistants, voice recorders), multiprocessor systems, microprocessor-based systems, set top boxes, game consoles, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. 
         [0044]      FIG. 7  illustrates an example of a suitable computing system environment. The computing system environment is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of such a computing environment. Neither should the computing environment be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the example operating environment. 
         [0045]    With reference to  FIG. 7 , an example computing environment includes a computing machine, such as computing machine  700 . In its most basic configuration, computing machine  700  typically includes at least one processing unit  702  and memory  704 . The computing device may include multiple processing units and/or additional co-processing units such as graphics processing unit  720 . Depending on the exact configuration and type of computing device, memory  704  may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. This most basic configuration is illustrated in  FIG. 7  by dashed line  706 . Additionally, computing machine  700  may also have additional features/functionality. For example, computing machine  700  may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in  FIG. 7  by removable storage  708  and non-removable storage  710 . Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer program instructions, data structures, program modules or other data. Memory  704 , removable storage  708  and non-removable storage  710  are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by computing machine  700 . Any such computer storage media may be part of computing machine  700 . 
         [0046]    Computing machine  700  may also contain communications connection(s)  712  that allow the device to communicate with other devices. Communications connection(s)  712  is an example of communication media. Communication media typically carries computer program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal, thereby changing the configuration or state of the receiving device of the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. 
         [0047]    Computing machine  700  may have various input device(s)  714  such as a display, a keyboard, mouse, pen, camera, touch input device, and so on. Output device(s)  716  such as speakers, a printer, and so on may also be included. All of these devices are well known in the art and need not be discussed at length here. 
         [0048]    The training of the language model, and its use in voice recognition, can be implemented in the general context of software, including computer-executable instructions and/or computer-interpreted instructions, such as program modules, being processed by a computing machine. Different computer systems can be used for training and recognition. Generally, program modules include routines, programs, objects, components, data structures, and so on, that, when processed by a processing unit, instruct the processing unit to perform particular tasks or implement particular abstract data types. This system may be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices. 
         [0049]    The terms “article of manufacture”, “process”, “machine” and “composition of matter” in the preambles of the appended claims are intended to limit the claims to subject matter deemed to fall within the scope of patentable subject matter defined by the use of these terms in 35 U.S.C. §101. 
         [0050]    Any or all of the aforementioned alternate embodiments described herein may be used in any combination desired to form additional hybrid embodiments. It should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific implementations described above. The specific implementations described above are disclosed as examples only.