Abstract:
A spoken dialog system comprises a speech recognition unit for recognizing a user&#39;s input speech to generate a character sequence corresponding thereto; a sentence contents database for storing therein a plurality of sentence contents; a knowledge search unit for searching through the sentence contents to find a match for the character sequence, in the sentence contents database; a dialog model unit for delivering the character sequence to the knowledge search unit to receive the sentence contents therefrom, and setting a dialog model by using the sentence contents; a system response unit for generating an output sentence which harmonizes with the user&#39;s input speech or expresses a situation of the system; and a speech synthesis unit for converting the output sentence into the output speech.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates to a spoken dialog system and a method for generating a response in the system; and, more particularly, to a spoken dialog system for realizing a natural dialog between an user and the system and a response method therefor, by generating an output sentence which concords with user&#39;s intention and a situation of the system, in the spoken dialog system with a speech interface based on HCI (Human-Computer Interaction). 
       BACKGROUND OF THE INVENTION 
       [0002]    HCI is a relatively new field, and its main focus is generally on designing an easy to use computer system. The basic concepts of HCI get materialized during a developing process of a user-centered computer system, rather than that of developer-centered one. Further, it mainly deals with a designing-evaluating-completing process of a computer operating system for interaction with humans. 
         [0003]    On one hand, such a typical spoken dialog system based on HCI is applied to systems such as a brainy robot, a telematics system, a digital home, and the like, all aimed at performing, for example, a weather search, a schedule management, a news search, a TV program guide, an email management, etc. 
         [0004]    The spoken dialog system applied to these systems generates the output sentence by performing one of the followings: using an interactive information search service, wherein a large amount of dialog examples having sets, each set including a user&#39;s intention and a situation of the system responding to the user&#39;s intention, is employed; filling a sentence template stored in a pre-built sentence template database with sentence contents which may correspond to search results from a separate database; generating a literary sentence based on a system grammar via a natural language processing such as a construction generation, a morpheme generation, a text generation, and the like. 
         [0005]      FIG. 1  is a schematic view showing a conventional spoken dialog system. 
         [0006]    As shown in  FIG. 1 , such a conventional spoken dialog system based on the HCI includes, for example, a speech recognition unit  10 , a dialog model unit  12 , a knowledge search unit  14 , a sentence contents database  16 , and a speech synthesis unit  18 . 
         [0007]    The speech recognition unit  10  performs a speech recognition and delivers a character sequence corresponding to the recognized speech to the dialog model unit  12 . The speech recognition includes a process of detecting a user&#39;s input speech; a process of amplifying the speech detected to a specific level; a process of extracting feature parameters from the speech; and other processes necessary to perform the speech recognition. 
         [0008]    The dialog model unit  12  delivers the character sequence recognized by the speech recognition unit  10  to the knowledge search unit  14 . Further, the dialog model unit  12  generates an output sentence as a response to the user by using sentence contents received from the knowledge search unit  14 . 
         [0009]    The sentence contents database  16  stores therein a number of sentence contents to be used for a user response sentence, for examples including a weather search, a schedule management, a news search, a TV program guide, an email management, etc. 
         [0010]    The knowledge search unit  14 , in response to the character sequence from the dialog model unit  12 , searches through the sentence contents stored in the sentence contents database  16  to find a match for the character sequence. 
         [0011]    The speech synthesis unit  18  converts the output sentence generated by the dialog model unit  12  into an output speech before providing it to the user. 
         [0012]    Since the main object of the conventional spoken dialog system configured in the aforementioned manner is to deliver information, the system is configured to clearly deliver the information therefrom, i.e., the output sentence, to the user audibly. 
         [0013]    However, since such conventional spoken dialog system only uses a pattern matching, there may occasionally be discrepancies between the intention of the user and the output sentence generated. In order to attain a natural dialog between the user and the system as if it were made between persons, the output sentence is required to be corresponded with the intention of the user and reflect the situation of the system while delivering the information requested by the user. However, the conventional spoken dialog system has the following drawback: a natural dialog cannot be realized because the output sentence cannot be accurately corresponded with the intention of the user in detail and the situation of the system (e.g., a manner of the speaker to the dialog made) cannot be reflected in the system response. 
       SUMMARY OF THE INVENTION 
       [0014]    It is, therefore, an object of the present invention to provide a spoken dialog system for realizing an interactive speech interface as natural as a dialog between persons by generating an output sentence corresponding with the intention of the user and reflects the situation of the system, and a response method therein. 
         [0015]    In accordance with one aspect of the present invention, there is provided a spoken dialog system, the system including: 
         [0016]    a speech recognition unit for recognizing a user&#39;s input speech to generate a character sequence corresponding thereto; 
         [0017]    a sentence contents database for storing therein a plurality of sentence contents; 
         [0018]    a knowledge search unit for searching through the sentence contents stored to find a match for the character sequence, in the sentence contents database; 
         [0019]    a dialog model unit for delivering the character sequence to the knowledge search unit to receive the sentence contents therefrom, and setting a dialog model by using the sentence contents; 
         [0020]    a system response unit for generating an output sentence which harmonizes with the user&#39;s input speech or expresses a situation of the system; and 
         [0021]    a speech synthesis unit for converting the output sentence into an output speech. 
         [0022]    In accordance with another aspect of the present invention, there is provided a method for generating a response in a spoken dialog system, the method including the steps of: 
         [0023]    recognizing a user&#39;s input speech to generate a character sequence corresponding thereto; 
         [0024]    searching through sentence contents to fine a match for the character sequence; 
         [0025]    setting a dialog model by using the sentence contents searched; 
         [0026]    generating an output sentence which harmonizes with the user&#39;s input speech or expresses a situation of the system; and 
         [0027]    converting the output sentence into an output speech. 
     
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0028]    The above and other objects and features of the present invention will become apparent from the following description of preferred embodiments given in conjunction with the accompanying drawings, in which: 
           [0029]      FIG. 1  is a schematic view showing a conventional spoken dialog system; 
           [0030]      FIG. 2  provides a schematic view showing a spoken dialog system in accordance with the present invention; 
           [0031]      FIG. 3  describes a detail view showing a system response unit of the spoken dialog system in accordance with the present invention; and 
           [0032]      FIGS. 4A and 4B  depict a flow chart showing a system response method of the spoken dialog system in accordance with the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
       [0033]    Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. 
         [0034]      FIG. 2  provides a schematic view showing a spoken dialog system in accordance with the present invention. 
         [0035]    As shown in  FIG. 2 , the spoken dialog system in accordance with an embodiment of the present invention includes a speech recognition unit  100 , a dialog model unit  102 , a knowledge search unit  104 , a sentence contents database  106 , a system response unit  108 , and a speech synthesis unit  110 . 
         [0036]    The speech recognition unit  100  performs a speech recognition and delivers a character sequence corresponding to the recognized speech to the dialog model unit  102 . The speech recognition includes a process of detecting a user&#39;s input speech; a process of amplifying the speech detected to a specific level; a process of extracting feature parameters of the speech; and other processes necessary to perform the speech recognition. 
         [0037]    The dialog model unit  102  delivers the character sequence recognized by the speech recognition unit  100  to the knowledge search unit  104  to receive sentence contents searched therethrough and establishes a dialog model by using the sentence contents obtained by the knowledge search unit  104 . Also, the dialog model unit  102  attains a basic sentence by using the sentence contents received from the knowledge search unit  104 . 
         [0038]    The sentence contents database  106  stores therein the sentence contents to be used in a user response sentence, for example, a weather search, a schedule management, a news search, a TV program guide, an email management, etc. 
         [0039]    The knowledge search unit  104  searches the sentence contents stored in the sentence contents database  106  to be matched with the character sequence received from the dialog model unit  102 . 
         [0040]    The system response unit  108  generates an output sentence by initially generating a plurality of candidate sentences, then selecting one of the candidate sentences which is determined to be harmonized with the user&#39;s input speech or expressing the situation of the system, and finally assigning an ending form of the sentence and an intonation pattern to the selected sentence. Further, the output sentence is provided to the speech synthesis unit  110 . If the system response unit  108  does not select one of the candidate sentences harmonizing with the user&#39;s input speech or expressing the situation of the system, it delivers the basic sentence as the output sentence to the speech synthesis unit  110 . 
         [0041]    The speech synthesis unit  110  converts the output sentence generated by the system response unit  108  into an output speech to provide it to the user. Further, if the speech synthesis unit  110  receives the basic sentence from the system response unit  108 , it converts the received basic sentence into the output speech and produces the same as an output. 
         [0042]    A difference between the spoken dialog system in accordance with the present invention and the conventional spoken dialog system is that the former is provided with the system response unit  108  whereas the latter is not. Here, the system response unit  108  generates the output sentence which is, as mentioned above, determined to be harmonized with the user&#39;s input speech or expressing the situation of the system. In this manner, the spoken dialog system in accordance with the present invention realizes a natural dialog between the user and the system. 
         [0043]      FIG. 3  describes a detail view showing a system response unit  108  of the spoken dialog system in accordance with the present invention. 
         [0044]    As shown in  FIG. 3 , the system response unit  108  includes a candidate sentence generator  1080 ; a sentence template database  1081 ; a sentence selector  1082 ; a harmonizing rule database  1083 ; an expression rule database  1084 ; an ending form determiner  1085 ; an ending form rule database  1086 ; an intonation pattern determiner  1087 ; and an intonation pattern rule database  1088 . 
         [0045]    The candidate sentence generator  1080  generates the candidate sentences by using the dialog model and the sentence template database  1081 . 
         [0046]    The sentence template database  1081  stores therein the candidate sentences to be provided to the candidate sentence generator  1080 . 
         [0047]    The sentence selector  1082  selects one of the candidate sentences, the one selected being harmonized with the user&#39;s input speech or expressing the situation of the system, and delivers the selected sentence to the ending form determiner  1085 . 
         [0048]    The harmonizing rule database  1083  stores therein user speech harmonizing rules to be provided to the sentence selector  1082 . 
         [0049]    The expression rule database  1084  stores therein system situation expression rules to be provided to the sentence selector  1082 . The sentence selector  1082  uses the user system situation expression rules when it selects the sentence which expresses the situation of the system. 
         [0050]    The ending form determiner  1085  assigns a situation dependent ending form of the sentence to the sentence selected by the sentence selector  1082 , and delivers the selected sentence, to which the ending form of the sentence is assigned, to the intonation pattern determiner  1087 . 
         [0051]    The ending form rule database  1086  stores therein ending form changing rules to be provided to the ending form determiner  1085 . The ending form determiner  1085  uses the ending form changing rules when it assigns the ending form of the sentence to the selected sentence. 
         [0052]    The intonation pattern determiner  1087  assigns a situation dependent intonation pattern to the sentence received from the ending form determiner  1085 , and delivers the sentence as the output sentence to the speech synthesis unit  110 . 
         [0053]    The intonation pattern rule database  1088  stores therein intonation pattern changing rules to be provided to the intonation pattern determiner  1087 . The intonation pattern determiner  1087  uses the intonation pattern changing rules when it assigns the intonation pattern to the selected sentence. 
         [0054]    Accordingly, the system response unit  108  of the spoken dialog system in accordance with the present invention realizes a natural dialog between the user and the system by generating the output sentence in the following manner. First, the candidate sentence generator  1080  generates the plurality of candidate sentences, one of which will be output to the user. Thereafter, the sentence selector  1082  selects one of the candidate sentences which is determined to be harmonized with the user&#39;s input speech or expressing the situation of the system. Further, the ending form determiner  1085  and the intonation pattern determiner  1087  assign the situation dependent ending form of the sentence and the situation dependent intonation pattern, respectively, to the selected sentence. 
         [0055]      FIGS. 4A and 4B  depict a flow chart showing a system response method of the spoken dialog system in accordance with the present invention. 
         [0056]    With reference to  FIGS. 4A and 4B  along with  FIGS. 2 and 3 , a response method in the spoken dialog system according to another embodiment of the present invention will be described as follows. 
         [0057]    First, the speech recognition unit  100  performs a speech recognition and delivers a character sequence corresponding to a user&#39;s input speech to the dialog model unit  102  (S 100 ). The speech recognition includes a process of detecting the user&#39;s input speech; a process of amplifying the speech detected to a specific level; a process of extracting feature parameters from the speech; and other processes necessary to perform the speech recognition. 
         [0058]    The dialog model unit  102  delivers the character sequence recognized by the speech recognition unit  100  to the knowledge search unit  104  (S 102 ). Thereafter, the knowledge search unit  104  searches through the sentence contents stored in the sentence contents database  106  to find a match for the character sequence, and delivers the searched sentence contents to the dialog model unit  102  (S 104 ). 
         [0059]    Then, the dialog model unit  102  establishes a dialog model and a basic sentence by using the sentence contents searched by the knowledge search unit  104  (S 106 ). The sentence contents used in obtaining the dialog model includes, for example, service areas (a weather forecast, a schedule, news, a TV program guide, an email, etc.), speech acts/system actions, concept strings (a person, a place, a time, the number of times, a date, a genre, a program, etc.), and search results. 
         [0060]    In the system response unit  108 , the candidate sentence generator  1080  generates (extracts) the plurality of candidate sentences from the sentence template database  1081  by using the dialog model set by the dialog model unit  102  (S 108 ). 
         [0061]    The sentence selector  1082  extracts harmonizing features from the user&#39;s input speech by using the user speech harmonizing rules stored in database  1083  (S 110 ). The harmonizing rule database  1083  stores therein data of harmonizing features (i.e., harmonizing rules), e.g., such as a table for difficulty levels of words; a table for adverbs which expresses intensity of meaning; a table for emotional interjections, emotional adjectives, emotional nouns, and the like. 
         [0062]    After that, the sentence selector  1082  determines whether or not to apply the user speech harmonizing rules in sentence selection (S 112 ). 
         [0063]    If the sentence selector  1082  determines to apply the user speech harmonizing rules in a sentence selection, the sentence selector  1082  selects a sentence, which has the harmonizing features extracted (i.e., harmonizes with the user&#39;s input speech), from the candidate sentences (S 114 ). 
         [0064]    Table 1 shows examples for selecting an optimal sentence among the candidate sentences by using the user speech harmonizing rules (e.g., six rules as in Table 1). 
         [0000]    
       
         
               
               
               
             
               
               
               
             
           
               
                   
                 TABLE 1 
               
               
                   
                   
               
               
                   
                 Rules 
                 Examples 
               
               
                   
                   
               
             
             
               
                   
               
             
          
           
               
                 1 
                 Select a sentence which has the most 
                 User&gt; How is the weather today? 
               
               
                   
                 similar sentence pattern to the user&#39;s 
                 System&gt; Today&#39;s weather is nice. 
               
               
                   
                 input speech. 
               
               
                 2 
                 Select a sentence which uses words 
                 User&gt; I&#39;m not feeling well. What 
               
               
                   
                 with difficulty levels similar to or 
                 should I do? 
               
               
                   
                 easier than those used in the user&#39;s 
                 System&gt; Do not exercise and have 
               
               
                   
                 input speech. 
                 yourself a good sleep. 
               
               
                   
                   
                 User&gt; I am physically fragile. What 
               
               
                   
                   
                 should I do about it? 
               
               
                   
                   
                 System&gt; Try to avoid any exercise and 
               
               
                   
                   
                 have yourself a good sleep. 
               
               
                 3 
                 Select a sentence which harmonizes 
                 User&gt; Well done. 
               
               
                   
                 with the intensity of the user&#39;s input 
                 System&gt; Thank you. 
               
               
                   
                 speech. 
                 User&gt; You were excellent today. 
               
               
                   
                   
                 System&gt; Don&#39;t mention it. It was my 
               
               
                   
                   
                 pleasure having you. 
               
               
                 4 
                 Select a sentence in which response 
                 User&gt; I am planning to play a round of 
               
               
                   
                 words appropriate to the user&#39;s input 
                 golf tomorrow. What would be the 
               
               
                   
                 speech (e.g., ‘yes’, ‘oh, yes’, ‘no’, 
                 weather like tomorrow? 
               
               
                   
                 or the like) are inserted. 
                 System&gt; Oh yes, it is going to be 
               
               
                   
                   
                 raining tomorrow. 
               
               
                   
                   
                 User&gt; No appointment scheduled for 
               
               
                   
                   
                 tomorrow? 
               
               
                   
                   
                 System&gt; Yes, there is one in the 
               
               
                   
                   
                 afternoon tomorrow. 
               
               
                 5 
                 Select a sentence which has an 
                 User&gt; The room is warm. 
               
               
                   
                 appropriate level harmonizing with the 
                 System&gt; How&#39;s the temperature? 
               
               
                   
                 intensity of the meaning. 
                 User&gt; The room is too warm. 
               
               
                   
                   
                 System&gt; Would you like it to be 
               
               
                   
                   
                 cooler? 
               
               
                 6 
                 Select a sentence which is appropriate 
                 User&gt; Shoot. 
               
               
                   
                 to cases in which emotional 
                 System&gt; What&#39;s the matter? 
               
               
                   
                 interjections, emotional adjectives, 
                 User&gt; Oh no. 
               
               
                   
                 emotional nouns, or the like are used 
                 System&gt; Something wrong? 
               
               
                   
                 alone or with other sentence in the 
               
               
                   
                 user&#39;s input speech. 
               
               
                   
               
             
          
         
       
     
         [0065]    When the sentence selector  1082  determines not to apply the user speech harmonizing rules in the step S 112 , a control process proceeds to a step S 110  where it is determined whether or not to apply the system situation expression rules (S 116 ). 
         [0066]    If it is determined to apply the system situation expression rules stored in the expression rule database  1084 , in the sentence selection for the sentence selector  1082  selects a sentence which expresses the situation of the system (S 118 ). On the contrary, if it is determined to apply none of the user speech harmonizing rules and the system situation expression rules, the system response unit  108  delivers the basic sentence as the output sentence to the speech synthesis unit  110 . The speech synthesis unit  110  then converts the basic sentence into an output speech to provide to the user (S 120 ). 
         [0067]    Table 2 shows examples for selecting the sentence expressing the situation of the system by using the system situation expression rules (e.g., a situation where the system requests a confirmation of the user, a situation where the user requests a confirmative answer of the system, a situation where the system cannot answer, and the like). 
         [0000]    
       
         
               
               
               
             
               
               
               
             
           
               
                   
                 TABLE 2 
               
               
                   
                   
               
               
                   
                 Rules 
                 Examples 
               
               
                   
                   
               
             
             
               
                   
               
             
          
           
               
                 1 
                 Select a sentence expressing a 
                 User&gt; What would be 
               
               
                   
                 situation where the system requests 
                 the weather like tomorrow? 
               
               
                   
                 a confirmation of the user 
                 System&gt; The weather in 
               
               
                   
                   
                 Daejeon-city? 
               
               
                 2 
                 Select a sentence expressing a 
                 User&gt; Have you recorded 
               
               
                   
                 situation where the user requests a 
                 the program? 
               
               
                   
                 confirmative answer of the system. 
                 System&gt; Yes, I have 
               
               
                   
                   
                 recorded thebaseball game. 
               
               
                 3 
                 Select a sentence expressing a 
                 User&gt; Tell me the next 
               
               
                   
                 situation where the system cannot 
                 week&#39;s TV schedule. 
               
               
                   
                 answer (e.g., for a request 
                 System&gt; Nothing has 
               
               
                   
                 naturally impossible to carry out, 
                 been scheduled yet for 
               
               
                   
                 for a request which can be carried 
                 next week at the moment. 
               
               
                   
                 out but has no answer, or for a 
               
               
                   
                 request for which the system has no 
               
               
                   
                 answer now). 
               
               
                   
               
             
          
         
       
     
         [0068]    After the sentence selector  1082  selects the sentence which harmonizes with the user&#39;s input speech or expresses the situation of the system, the ending form determiner  1085  determines whether or not to apply the ending form changing rules stored in the ending form rule database  1086  to the selected sentence (S 122 ). 
         [0069]    If the ending form determiner  1085  determines to apply the ending form changing rules, it assigns a situation dependent ending form of the sentence to the sentence selected (S 124 ). 
         [0070]    Table 3 shows examples for changing the ending form of the sentence to make a natural dialog by using the ending form changing rules. In Table 3, situations of the system are classified into a reportive, an inferential, an assertive, and an exceptional situation; and the ending form of the sentence is changed according to the respective situations. 
         [0000]    
       
         
               
               
               
             
               
               
               
             
           
               
                   
                 TABLE 3 
               
               
                   
                   
               
               
                   
                 Conditions for Applying Rules 
                 Situation 
               
               
                   
                   
               
             
             
               
                   
               
             
          
           
               
                 1 
                 When the system outputs the output 
                 Reportive 
               
               
                   
                 sentence by referring to data 
               
               
                   
                 rather than those registered by 
               
               
                   
                 the user. 
               
               
                 2 
                 When the system outputs a result 
                 Inferential 
               
               
                   
                 of an inference as an output 
               
               
                   
                 sentence. 
               
               
                 3 
                 When the system delivers an 
               
               
                   
                 uncertain situation due to an 
               
               
                   
                 occurrence of a recognition error. 
               
               
                 4 
                 When the system answers or asks 
                 Assertive 
               
               
                   
                 repetitively. 
               
               
                 5 
                 When the system speeches a sure 
               
               
                   
                 answer. 
               
               
                 6 
                 When the system describes the 
               
               
                   
                 situation of the system. 
               
               
                 7 
                 When the system cannot find an 
                 Exceptional 
               
               
                   
                 answer. 
               
               
                 8 
                 When the system needs to deny the 
               
               
                   
                 user&#39;s speech. 
               
               
                   
               
             
          
         
       
     
         [0071]    After the ending form determiner  1085  changes the ending form of the sentence by applying the ending form changing rules in the step S 124  or determines not to apply the ending form changing rules to the sentence in the step S 122 , the intonation pattern determiner  1087  determines whether or not to apply the intonation pattern changing rules stored in the intonation pattern rule database  1088  to the sentence (S 126 ). If the intonation pattern determiner  1087  determines to apply the intonation pattern changing rules, it assigns a situation dependent intonation pattern to the sentence by using the intonation pattern changing rules (S 128 ). 
         [0072]    Table 4 shows examples for changing the intonation pattern of the sentence by using the intonation pattern changing rules. In Table 4, the situations of the system are classified into mutual confirmation, assertion, emphasis/persuasion, and assurance/request, and the intonation pattern of the sentence is changed according to the respective situations. To be specific, Pattern symbols H (High tone), L (Low tone), and M (Middle tone, i.e., approximately middle of the High tone and the Low tone) in Table 4 confirm the K-TOBI (Korean Tone Break Indices). 
         [0000]    
       
         
               
               
               
               
             
               
               
               
               
             
           
               
                   
                 TABLE 4 
               
               
                   
                   
               
               
                   
                 Conditions for Applying Rules 
                 Situation 
                 Intonation pattern 
               
               
                   
                   
               
             
             
               
                   
               
             
          
           
               
                 1 
                 When the system generates a 
                 Mutual 
                 HL (High-Low) 
               
               
                   
                 sentence (asks a question) about 
                 confirmation 
                 tone 
               
               
                   
                 the old information already 
               
               
                   
                 mentioned in the dialog. 
               
               
                 2 
                 When the system describes. 
                 Assertion 
                 ML (Middle-Low) 
               
               
                   
                   
                   
                 tone 
               
               
                 3 
                 When the system denies the 
                 Emphasis/ 
                 LML (Low-Middle- 
               
               
                   
                 user&#39;s speech. 
                 Persuasion 
                 Low) tone 
               
               
                 4 
                 When the system counsels. 
                 Assurance/ 
                 LM (Low-Middle) 
               
               
                   
                   
                 Request 
                 tone 
               
               
                   
               
             
          
         
       
     
         [0073]    After the intonation pattern determiner  1087  changes the intonation pattern of the sentence by applying the intonation pattern changing rules in the step S 128  or determines not to apply the intonation pattern changing rules to the sentence in the step S 126 , the speech synthesis unit  110  converts the selected sentence generated by the system response unit  108 , and outputs it (S 130 ). 
         [0074]    Therefore, the system response method in the spoken dialog system in accordance with the present invention realizes a natural dialog between the user and the system by generating the output sentence, which is determined to be harmonized with the user&#39;s input speech or expressing the situation of the system, and assigning the situation dependent ending form and/or the situation dependent intonation pattern to the output sentence. 
         [0075]    While the invention has been shown and described with respect to the embodiments, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the scope of the invention as defined in the following claims.