Patent Publication Number: US-9842106-B2

Title: Method and system for role dependent context sensitive spoken and textual language understanding with neural networks

Description:
FIELD OF THE INVENTION 
     This invention relates generally to dialog processing, and more particularly to Natural Language Understanding (NLU) methods and systems for dialogs including spoken and textual utterances. 
     BACKGROUND OF THE INVENTION 
     Methods and systems of Natural Language Understanding (NLU), which can perform, for example, Spoken Language Understanding (SLU), are used in computerized dialog systems to estimate intentions of utterances. As broadly defined herein, the “spoken” utterances can be in the form of speech or text. If the utterances are spoken, then the utterances can be obtained from, for example, an automatic speech recognition (ASR) system. If the utterances are text, then the utterances can be obtained from, e.g., a text processing systems or keyboard input. 
     Conventional intention estimation methods can be based on phrase matching, or classification methods, such as boosting, support vector machines (SVM), and Logistic Regression (LR) using Bag of Word (BoW) features of each utterance as inputs. However, the BoW features do not have enough capability to indicate semantic information represented by word sequences due to, for example, missing order of words in the sequences. 
     To consider a history of a word sequence in each utterance, a Recurrent Neural Networks (RNNs) can be applied for utterance classification using 1-of-N coding instead of the BoW features. Additionally, Long Short-Term Memory (LSTM) RNNs are a form of RNNs designed to improve learning of long-range context, and can be effective for context dependent problems. Those of approaches classify utterances without considering context among utterances. Additionally, it is essential to consider a broader context of a sequence of utterances of an entire dialog to understand intention accurately. Some of the prior art models using RNNs and LSTMs use word sequence context within a single utterance and also consider a broader context of a sequence of utterances of an entire dialog. 
     Furthermore, each utterance has different expressions in terms of context of party-dependent features such as for task-oriented roles like agents and clients, business dependent terminolgies and expressions, gender dependent languages, relationships among participants in the dialogs. However, conventional methods do not consider such party-dependent features due to the different roles. 
     SUMMARY OF THE INVENTION 
     The embodiments of the invention provide a method and system for processing utterances. The utterances are acquired either from an automatic speech recognition (ASR) system or text. The utterances have associated identities of each party, such as role A utterances and role B utterances. The information corresponding to utterances, such as word sequence and identity, are converted to features. Each feature is received in an input layer of a neural network (NN). A dimensionality of each feature is reduced, in a projection layer of the NN, to produce a reduced dimensional feature. The reduced dimensional feature is processed, where the feature is propagated through hidden layers. In case of recurrent neural network (RNN), hidden layers have reccurent connections and long short-term memory (LSTM) can be applied to hidden layers of the RNN. Then, in an output layer of the NN, posterior probabilities of labels are determined. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1A  is a schematic of party-dependent neural networks with a shared context history of an entire dialog among two parties; 
         FIG. 1B  is a schematic of party-dependent expressions through different layers of a single neural network; 
         FIG. 1C  is a schematic of a context sensitive spoken language understanding (SLU) method and system in the form of a recurrent neural network with two parallel hidden, long short-term memory (LSTM) layers according to embodiments of the invention; 
         FIG. 2  is a schematic of set of LSTM cells in the hidden layers according to embodiments of the invention; 
         FIG. 3  is a schematic of a propagation process of the context-sensitive SLU according to embodiments of the invention; 
         FIG. 4  is a schematic of details of the two parallel LSTM layers according to embodiments of the invention; and 
         FIG. 5  is a schematic of temporal process of the role-dependent SLU according to embodiments of the invention. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     The embodiments of our invention provide a method and system for language understanding, e.g., a spoken language understanding (SLU). The method and can estimate intentions of utterances in a dialog. As broadly defined herein, the utterances can be in the form of speech or text. If the utterances are spoken, then the utterances can be obtained from, for example, an automatic speech recognition (ASR) system. If the utterances are text, then the utterances can be obtained from, e.g., a text processing systems or keyboard input. 
     Context-Sensitive SLU Using NNs 
       FIG. 1A  shows party-dependent neural networks  100  with a shared context history of an entire dialog among two parties for role A  101  on the left, and role B  102  on the right, respectively. This embodiment of the invention uses two neural networks (NNs), which considering party-dependent expressions with or without long-term context over a dialog. 
       FIG. 1B  shows party-dependent expressions through different layers of a single neural network  100 . The variables used in the figures are described detail below. 
       FIG. 1C  schematically shows the RNN  100  in an alternative form. Here, the input layer  110  receives input word vectors  111  from the ASR  105 . The word vectors correspond to utterances acquired from a client  101  and an agent  102 . Typically, the client and agent take turns speaking the utterances during their respective roles A and B. 
     The method and networks can be implemented in a processor connected to memory and input/output interfaces as known in the art. 
     By convention, each network is shown with the input layer  110  at the bottom, and the output layer  140  at the top. The input layer  110  receives input word vectors  111  corresponding to the utterances by multiple parties. The utterances have associated identities of each party. The identities relate to the roles performed by the parties. The word vectors correspond to utterances acquired for role A  101 , e.g., a client party, and role B  102 , e.g., an agent party. Typically, the parties take turns generating the utterances for each role during a dialog. 
     A projection layer  120  reduces the dimensionality of the word vector to produce a reduced dimensional word vector. A recurrent hidden layer  130  is constructed as long short-term memory (LSTM) with recurrent connections with party gates  131  that can retain and forget context information. The LSTM layers produce activation vectors for the utterances. The output layer  140  estimates posterior probabilities of output labels  141  based on the activation vectors. 
     To understand the intentions in a dialog of the multiple parties accurately, it is important to consider party-dependent expressions in each utterance and the function of each utterance in the context of a sequence of dialog turns as well. 
     To consider both context of an entire dialog and party-dependent expressions of each utterance, we provide an efficient NLU approach based on neural networks (NN) that model the context sensitive party-dependent expressions through either party-dependent neural and networks shared context history among parties shown in  FIG. 1A , or a single neural network facilitated with party-dependent different layers as shown in  FIG. 1B . 
     Each word is input sequentially into the NNs using either word vector representation such as BoW or 1-of-N coding with or without features of additional attributes such as sematic, syntactic, task-oriented information. The features of word sequence are propagated through one of the party-dependent hidden layers, and semantic information, such as concept tags, are output at the end of each utterance. Concept tags only represent symbols. Semantic information can be symbols, and/or structured information such as a graph. 
     In case of the RNN, to propagate contextual information through a dialog, the activation vector of the RNN for an utterance serves as input to the RNN for the next utterance to consider contex of an entire dialog. The embodiments train the RNN layers of a context sensitive model to predict sequences of semantic information from the word sequences with considering party-dependent expressions. 
     The utterances in the dialog corpus are characterized for each party in terms of roles, such as agent or client. In order to precisely model the party-dependent utterances, we provide multiple party-dependent neural networks which are shared context history of an entire dialog among parties as shown  FIG. 1A . In addition, we provide a single neural network facilitated with party-dependent different layers depicted in  FIG. 1B . 
     The different party-dependent features are modeled by switching between the party-dependent hidden layers. In these models, words of each utterance are input one at a time, and semantic information are output at the end of each utterance. In case of the RNN, the party-dependent hidden layers jointly represent both the context within each utterance, and the context within the dialog. In other NNs, the party-dependent hidden layers represent only the characteristics of each party&#39;s utterance. 
     As shown in  FIG. 2 , we use a set of LSTM cells  200  in the hidden layer  130 , instead of network units as in conventional RNNs. The LSTM cells can remember a value for an arbitrary length of time using gates. The LSTM cell contains input  210 , forget  220 , and output  230  gates, which respectively determine when the input is significant enough to remember, when to forget the input, and when the input contributes to the output. 
     A sequence of M utterances is u 1 , . . . , u τ , . . . , u M . Each utterance u m  includes a word sequence w τ,1 , . . . , w τ,t , . . . , w τ,T     τ    and a concept tag a τ . The input word vectors x τ,t    111  received from the ASR are
 
 x   τ,t =OneHot( w   τ,t ),  (1)
 
where the word w τ,t  in a vocabulary V is converted by 1-of-N coding using a function OneHot(w), i.e., x τ,t  ε {0,1} |v| .
 
     The input vector is projected by the projection layer  120  to a D dimensional vector
 
 x   τ,t   ′=W   pr   x   τ,t   +b   pr ,  (2)
 
which is then fed to the recurrent hidden layers  130 , where W pr  is a projection matrix, and b pr  is a bias vector.
 
     At the hidden layers  130 , activation vectors h τ,t  are determined using the LSTM cells according
 
 i   τ,t =σ( W   xi   x′   τ,t   +W   hi   h   τ,t-1   +W   ci   c   τ,t-1   +b   i ),  (3)
 
 f   τ,t =σ( W   xf   x′   τ,t   +W   hf   h   τ,t-1   +W   cf   c   τ,t-1   +b   f ),  (4)
 
 c   τ,t   =f   τ,t   c   τ,t-1   +i   τ,t  tan  h ( W   xc   x′   τ,t   +W   hc   h   τ,t-1   +b   c ),  (5)
 
 o   τ,t =σ( W   xo   x′   τ,t   +W   ho   h   τ,t-1   +W   co   c   τ,t   +b   o ), and  (6)
 
 h   τ,t   =o   τ,t  tan  h ( c   τ,t ),  (7)
 
where σ( ) is an element-wise sigmoid function, and i τ,t , f τ,t , o τ,t  and c 96 ,t  are the input gate  210 , forget gate  220 , output gate  230 , and cell activation vectors for the t th  input word in the x-th utterance, respectively.
 
     Weight matrices W zz  and bias vectors b z  are identified by a subscript z ε {x, h, i, f,o, c}. For example, W hi  is a hidden-input gate matrix, and W xo  is an input-output gate matrix. 
     The output vector  141  is determined at the end of each utterance as
 
 y   τ =softmax( W   HO   h   τ,T     τ     +b   O ),  (8)
 
where W HO  is a transformation matrix, and b O  is a bias vector to classify the input vector into different categories according to the hidden vector. Softmax( ) is an element-wise softmax function that converts the classification result into label probabilities, i.e., y τ  ε [0,1] |L|  for label set L
 
                         a   ^     τ     =         arg   ⁢           ⁢   max       a   ∈   ℒ       ⁢           ⁢       y   τ     ⁡     [   a   ]           ⁢     ,                     (   9   )               
where y τ [a] indicates the component of y τ  for label a, which corresponds to the probability P (a|h τ,T     τ   ) of the label.
 
     To inherit the context information from previous utterances, the hidden and cell activation vectors at the beginning of each utterance are
 
 h   τ,0   =h   τ-1,T     τ-1     (10)
 
 c   τ,0   =c   τ-1,T     τ-1′     (11)
 
where τ&gt;1 and h 1,0 =c 1,0 =0, as shown in  FIG. 2 .
 
       FIG. 3  shows a propagation process of our context-sensitive SLU. Words w i,j  are sequentially input to the LSTM layers  130 , and a label  141  that is output corresponds to the utterance concept at the end of the utterance, where symbol EOS represents “end of sentence.” 
     In contrast with the prior art, our model considers the entire context from the beginning to the end of the dialog. Accordingly, the label probabilities can be inferred using sentence-level intentions an dialog-level context. In contrast, the conventional models only considers each utterance independently. 
     Role-Dependent LSTM Layers 
     The LSTM layers can be trained using a human-to-human dialog corpus annotated with concept tags, which represent, e.g., client and agent intentions for a hotel reservation as shown in Table 1 below. The columns, left to right, indicate the speaker, e.g., agent and client, uterances, and concept tags. The uterrances are characterized by each role of agent and client. 
     
       
         
           
               
             
               
                 TABLE 1 
               
             
            
               
                   
               
               
                 An Example of Hotel Reservation Dialog 
               
            
           
           
               
               
               
            
               
                 Speaker 
                 Utterance 
                 Concept tags 
               
               
                   
               
               
                 Agent 
                 hello, 
                 greeting 
               
               
                 Agent 
                 new york city hotel, 
                 introduce-self 
               
               
                 Agent 
                 may i help you? 
                 offer + help 
               
               
                 Cleint 
                 i would like to make a 
                 request-action + reservation + hotel 
               
               
                   
                 reservation for a room, 
               
               
                 Agent 
                 very good, 
                 acknowledge 
               
               
                 Agent 
                 may i have the spelling 
                 request-information + name 
               
               
                   
                 of your name, please? 
               
               
                 Client 
                 it is mike, smith, 
                 give-information + name 
               
               
                 Agent 
                 uh when would you like 
                 request-information + temporal 
               
               
                   
                 to stay? 
               
               
                 Client 
                 from august tenth to 
                 give-information + temporal 
               
               
                   
                 august sixteenth, 
               
               
                   
                 seven days, 
               
               
                 Agent 
                 i see, 
                 acknowledge 
               
               
                 Agent 
                 you would be arriving on 
                 verify-give-information + temporal 
               
               
                   
                 the tenth? is the right? 
               
               
                 Client 
                 that is right, 
                 affirm 
               
               
                 Agent 
                 great, 
                 acknowledge 
               
               
                 Agent 
                 and, what sort of room 
                 request-information + room 
               
               
                   
                 would you like? 
               
               
                 Client 
                 well, it is just for myself, 
                 give-information + party 
               
               
                 Client 
                 so a single room would 
                 give-information + room 
               
               
                   
                 be good, 
               
               
                 Agent 
                 okay, 
                 acknowledge 
               
               
                 Agent 
                 a single, 
                 verify-give-information + room 
               
               
                 Agent 
                 starting on the tenth and 
                 verify-give-information + temporal 
               
               
                 Agent 
                 you would be checking 
                 verify-give-information + temporal 
               
               
                   
                 out on the sixteenth? is 
               
               
                   
                 that right? 
               
               
                 Client 
                 yes, 
                 affirm 
               
               
                 Client 
                 and i like the second of 
                 give-information + room 
               
               
                   
                 third floor, if possible, 
               
               
                 Agent 
                 i see, 
                 acknowledge 
               
               
                   
               
            
           
         
       
     
     As shown in  FIG. 4 , two parallel LSTM layers  310  and  320  representing client (A) and agent (B) utterances are incorporated in the model. Role gates  311  and  321  control which role, client or agent, is active. 
     The two LSTM layers have different parameters depending on the speaker roles. Thus, the input vector  111  for the client utterances is processed by the layer  310 , and by the layer  320  processes the agent utterances. The active role for a given utterance is controlled by a role variable R, which is used to gate the output to each LSTM layer. 
     Then, the gated outputs are passed from the recurrent LSTM layers to the output layer  140 . The recurrent LSTM inputs thus receive the output from the role-dependent layer active at the previous frame, allowing for transitions between roles. 
     Error signals in the training phase are back-propagated through the corresponding layers. The role of each speaker does not change during a dialog and the speaker of each utternace is known. However, the model structure leaves open the possibility of dynamically inferring the roles. Accordingly, we can determine the activation at the output layer as
 
 y   τ =softmax(δ R,R     τ   ( W   HO   h   τ,T     τ     (R)   +b   O )),  (12)
 
where h τ,T     τ     (R)  is the hidden activation vector given by the LSTM layer of role R, and δ R,R     τ    is Kronecker&#39;s delta, i.e., if R τ  is the role of the x-th utterance equals role R, R is 1, otherwise R is 0. At the beginning of each utterance, the hidden and cell activation vectors of the role-dependent layer are
 
 h   τ,0   (R     τ     )   =h   τ-1,T     τ-1     (R     τ-1     ) , and  (13)
 
 c   τ,0   (R     τ     )   =c   τ-1,T     τ-1     (R     τ-1     ) .  (14)
 
       FIG. 5  shows the temporal process of the role-dependent SLU. For each utterance in a given role, only the LSTM layer for that role is active, and the hidden activation and the cell memory are propagated over dialog turns. The figure shows the client utterances (Role A), and the agent utterances (Role B). With this architecture, both LSTM layers can be trained considering a long context of each dialog, and the model can predict role-dependent concept labels accurately. 
     Effect of the Invention 
     The invention provides an efficient context sensitive SLU using role-based LSTM layers. In order to determine long-term characteristics over an entire dialog, we implemented LSTMs representing intention using consequent word sequences of each concept tag. We have evaluated the performance of importing contextual information of an entire dialog for SLU and the effectiveness of the speaker role based LSTM layers. The context sensitive LSTMs with roll-dependent layers out-performs utterance-based approaches and improves the SLU baseline by 11.6% and 8.0% (absolute). 
     Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.