Patent Publication Number: US-10311869-B2

Title: Method and system for automation of response selection and composition in dialog systems

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a 35 U.S.C. § 371 National Stage Application of PCT/US2015/56687, entitled “METHOD AND SYSTEM FOR AUTOMATION OF RESPONSE SELECTION AND COMPOSITION IN DIALOG SYSTEMS” by Weng et al., filed Oct. 21, 2015, which claims the benefit of priority to U.S. Provisional Application Ser. No. 62/066,508 entitled “METHOD AND SYSTEM FOR AUTOMATION OF RESPONSE SELECTION AND COMPOSITION IN DIALOG SYSTEMS” by Weng et al., filed Oct. 21, 2014, the disclosures of which are hereby incorporated herein by reference in their entirety. 
    
    
     TECHNICAL FIELD 
     This disclosure relates generally to communication systems, more particularly, to voice communication systems. 
     BACKGROUND 
     An increasing number of devices are entering into people&#39;s life, for example, at home, on the go, or in the office. When people perform primary tasks, such as cooking, driving, or writing, they may want to work on other tasks, for example, adjusting room temperature, turning on/off indoor or outdoor lights, listening to the news, or checking who is at the door front, or controlling the garage door. Communication systems such as voice communication systems can be used in settings such as when human hands are not able to easily reach the buttons, knobs, or touch screens. Certain existing systems, such as Siri, are increasing the awareness of communication technologies to the general public. 
     However, such existing systems as those described above have several limitations. For example, the existing systems may not take into consideration rich expressive nature of human languages and the expressions used in these systems may not take into consideration how or which expressions are used for the different user groups in different contexts and with different use purposes. Thus, certain existing systems may use a one-size-fits all approach. As a result, a large percentage of users may not enjoy the use of such systems. There exists a need for a system that can address the shortcomings of the existing systems. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a functional block diagram of an exemplary embodiment of a system according to the disclosed subject matter. 
         FIG. 2  illustrates an exemplary method for building the knowledge bases for the database according to the disclosure in the system of  FIG. 1 . 
     
    
    
     DETAILED DESCRIPTION 
     For the purposes of promoting an understanding of the principles of the embodiments described herein, reference is now made to the drawings and descriptions in the following written specification. No limitation to the scope of the subject matter is intended by the references. This disclosure also includes any alterations and modifications to the illustrated embodiments and includes further applications of the principles of the described embodiments as would normally occur to a person of ordinary skill in the art to which this document pertains. 
     In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it may be evident that such aspects may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by multiple components. Similarly, for instance, a component may be configured to perform functionality that is described as being carried out by multiple components. 
       FIG. 1  illustrates a functional block diagram of an exemplary embodiment of a system  100 . The system  100  can be a dialog system with details concerning system responses under different conditions. Other components of a typical intelligent dialog system may be used, which are described, for example, in U.S. Pat. No. 7,716,056 and U.S. Published Patent Application No. 2014/0019522, the entire disclosures of which are incorporated by reference herein. In one exemplary embodiment, the system  100  includes a user category classification and detection module  101 , a user mood detection and tracking module  103 , a user physical and mind state and energy level detection module  105 , a user acquaintance module  107 , a user personality detection and tracking module  109 , a conversational context detection and management module  111 , and a dialog manager module  113 . The system  100  also includes a response generation module  115 . The system  100  can additionally include a processor  117  and a database  119 . In one embodiment, each of these modules  101 ,  103 ,  105 ,  107 ,  109 ,  111 ,  113 ,  115 , processor  117 , and database  119  in the system  100  can be configured to directly interface with each other. In another embodiment, a different combination of the modules  101 ,  103 ,  105 ,  107 ,  109 ,  111 ,  113 ,  115 , processor  117 , and database  119  in the system  100  can be configured to directly interface with each other. 
     In one embodiment, the processor  117  can include, but is not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) device, or a micro-controller. The system  100  can further include a memory. The memory may be incorporated in the database  119 . In another embodiment, the memory is a separate module from the database. The processor  117  is configured to execute or perform programmed instructions that are stored in the memory. The memory can be any suitable type of memory, including solid state memory, magnetic memory, or optical memory, just to name a few, and can be implemented in a single device or distributed across multiple devices. The programmed instructions stored in memory include instructions for implementing various functionality in the system  100 , including the various modules  101 ,  103 ,  105 ,  107 ,  109 ,  111 ,  113 ,  115  and the database  119  described herein. 
     In one example, the system  100  can incorporate rich expressive characteristics in the responses of the speech dialog systems. Specifically, the system  100  can use information such as, but not limited to, categories of age, gender and professional user groups, and the relationship within concerned user groups (such as family member relations, team organizational structures), categories of user moods, types of contextual information related to communication needs (such as conversational context, environmental context), ontological relations of these entities, groups, states and information, genres of language to be selected for different families and contexts, and categories of prosodic marks, or the like. In one example, the system  100  can be configured to select output sentences from the system  100  based on this information and annotate with corresponding prosodic and emotion marks. Additionally or alternatively, the system  100  can also include different processes configured to select the expressions depending upon the system&#39;s needs  100 . In one example, a number of example application domains can be covered, such as home device controls, sports news selection, information inquiry, mobility assistance, customer support, or the like. In another example, the system  100  can also use multiple aspects of the user, multiple aspects of the context, and multiple aspects of the language to construct or synthesize responses. In another example, the system  100  can identify and refine candidate responses with different conditions from various data sources for response construction. 
     In one exemplary embodiment, the user category classification and detection module  101  can be configured to classify and detect user categories. For example, users can be classified into different categories along different dimensions, such as age, gender, profession, and relationship. In one example, the system  100  can obtain information about the user through processes such as a user enrollment process at the time when the user starts to use the system  100 . In another example, the system  100  can obtain information about the user dynamically through explicit or implicit interaction with the user. For example, the system  100  may detect the user&#39;s age group via the voice of the user, or via video of the person&#39;s face. In another example, system  100  may also ask the user after it performs an action requested by the user. 
     With further reference to  FIG. 1 , the user category classification and detection module  101  can classify users based on the user&#39;s age. In one example, user category classification and detection module  101  can identify whether the user is a child, a teenager, an adult, or a senior. The information about the age of the user can be collected, for example, during a user enrollment process and a system acquaintance process from the user based on the user&#39;s speech and the system  100 . In another example, speech-based age detection can also be used to identify information about the age of the user. 
     With further reference to  FIG. 1 , the user category classification and detection module  101  can classify users based on the user&#39;s gender. The information about the gender of the user can be collected, for example, during a user enrollment process and a system acquaintance process from the user based on the user&#39;s speech and the system  100 . In another example, speech-based age detection can also be used to identify information about the age of the user. 
     With further reference to  FIG. 1 , the user category classification and detection module  101  can classify users based on the user&#39;s profession. Examples of classification of user&#39;s profession can include, but is not limited to, engineer, scientist, banker, teacher, factory worker, farmer, government employee, soldier, or the like. The information about the profession of the user can be collected, for example, during a user enrollment process and a system acquaintance process from the user based on the user&#39;s speech and the system  100 . 
     With further reference to  FIG. 1 , the user category classification and detection module  101  can classify users based on the user&#39;s relationship with other users of the system  100 , members of an organization where the system  100  is used, or the like. Examples of classification of user&#39;s relationship can include, but is not limited to, family, co-worker, manager vs. team members, or the like. The information about the relationship of the user can be collected, for example, during a user enrollment process and a system acquaintance process from the user based on the user&#39;s speech and the system  100 . 
     With further reference to  FIG. 1 , the user mood detection and tracking module  103  can be configured to detect and track the user&#39;s mood. In one example, the user&#39;s mood can be registered with the dialog manager module  113  in each dialog turn. Examples of user&#39;s mood can include, but is not limited to happy, angry, sad, neutral, or the like. The user&#39;s mood can be encoded as an emotion indicator. In one example, the user&#39;s mood can be detected via the characteristics of the user&#39;s voice or a video of the user&#39;s facial expression when the user communicates with the system  100 . The user&#39;s voice can be an acoustic signal and the video of the user can be a video stream. 
     With further reference to  FIG. 1 , the user physical and mind state and energy level detection module  105  can detect and identify the user&#39;s physical and mental state and energy level. In one example, the user&#39;s physical and mental state and energy level can include, but is not limited to, whether the user is sleepy, whether the user just woke up, whether the user is tired, whether the user is energetic, whether the user is performing other tasks at the same time, whether the user interacts the system  101  as his/her primary task or pays enough attention in talking to the system, or the like. In one example, the user&#39;s physical and mental state and energy level can be detected and identified from the energy level of user voice, the talking tones of the user, and sounds in the user&#39;s speech. This information can be detected and registered in the system  100 . In another example, the user&#39;s physical and mental state and energy level can also be detected and identified from the explicit statement from the user. For example, if the user says “I am tired” or “I need to go to bed,” the user physical and mind state and energy level detection module  105  can detect that the user is tired. In yet another example, the user physical and mental state and energy level can be detected by wearable devices on the user, either embedded in the body or attached on the person, and transmitted to the system  100 . Examples of such wearable devices include, but are not limited to, electromyography (EMG) for muscle activity detection and interpretation, electroencephalography (EEG), near-infrared spectroscopy (NIRS) for brain activity and interpretation, or the like. In another example, the user&#39;s physical and mental state and energy level can be stored, for example, in the database  119  of the system  100  and can be used by the system  100  to predict the user&#39;s physical and mental state and energy level, user&#39;s mood, or the like. 
     With further reference to  FIG. 1 , the user&#39;s acquaintance module  107  can be used to accumulate user&#39;s usage of the system  100 . In one example, the user&#39;s acquaintance module  107  can be used to estimate whether the user is familiar with the system  100 . In another example, the user&#39;s acquaintance module  107  can be used to identify the level at which the users know other users or speakers, such as whether the users are having a first time encounter, the users have had a short time acquaintance or a long time close friend relationship. In another example, the system  100  can remember if the user&#39;s has used the system  100  for the first time, the length of time the user has used the system  100 , the frequency with which the user uses the system  100 , and the features of the system  100  that the user uses. Statistics can be accumulated in the system  100  to compute a value of the acquaintance. Such statistics may also record inter-user interaction such as how often users, for example, two users, send messages to each other, over what period of times, and what messages they exchange. Such statistics may also record how often two specific users use the same system at the roughly the same time. These statistics are then used to estimate the familiarity and closeness of the users. The user&#39;s acquaintance module  107  can have a time factor that can reduce the user&#39;s familiarity score based on the duration that user does not use the system  100 . Furthermore, certain statistics can be disabled by the users either during the initial enrollment phase or during the interaction with the system  100 . 
     With further reference to  FIG. 1 , the user&#39;s personality detection and tracking module  109  can be used to detect the user&#39;s personality. In one example, this information can be detected based on information collected from the user&#39;s interaction with the system  100 . A user&#39;s personality can be classified as along several aspects, for example, humor vs. procedural, quiet vs. talkative, as well as a fast, medium, and slow pace, or the like. In another example, the user&#39;s personality detection and tracking module  109  can use values to calculate the scores about speaker&#39;s personality. Examples of values the user&#39;s personality detection and tracking module  109  can use include, but are not limited to, speaking rate, utterance length, number of utterances for a task request, fun words in the utterances, or the like. 
     With further reference to  FIG. 1 , the conversational context detection and management module  111  can be configured to compute and administer a number of values. In another example, the conversational context detection and management module  111  can be configured to collect statistics for these aspects. For example, the aspects can include, but are not limited to, interaction setting such as family setting, business setting, casual setting, or the like. In one example, the values can be computed during the user&#39;s initial enrollment period and during subsequent uses. In one example, information about the user&#39;s relationship can place a higher weight in identifying the value of the interaction setting. In one example, based on the values from the conversational context detection and management module  111 , the response generation module  115  may generate a response based on a genre of expressions that is, for example, “formal”, “informal”, “very casual,” or the like. 
     With further reference to  FIG. 1 , another example of a value the conversational context detection and management module  111  can be configured to compute the response urgency or pace, such as high, medium, low. In one example, this value can be computed using the pace of the user&#39;s speech or can be explicitly requested from the user during the interaction with the system  100 . In another example, the response urgency can be estimated or conveyed by the results obtained during the interaction with the system. If the next train departs in 20 minutes when the user queries about the train schedule for his/her trip, the system  111  would indicate high urgency so that the user can get ready quickly. If an immediate turn is needed within next 10 meters, the response needs to catch user&#39;s attention so that he/she can be ready for the move. The response of the urgency can be reflected by the selection of the lexicon or words, such as “soon” or “quickly”, and/or by a speaking tone or speaking rate, or even a specifically designated sound, or the like. In one example, based on the values from the conversational context detection and management module  11 , the response generation module  115  may generate short, medium, or elaborate expressions for communication with the user. In another example, the exact wording generated by the response generation module  115  can also depend on different information on the user from the different modules  101 ,  103 ,  105 ,  107 ,  109 ,  111 , and  113 . 
     With further reference to  FIG. 1 , another example of a value the conversational context detection and management module  111  can be configured to compute the operation time such as operating the system  100  in the morning, during day time, meal time, or right before bed, relative to the user&#39;s scheduled events. In one example, this value can be selected based on the clock in the system  100  or an adaptation from the use of the system  100 . In another example, this value may also be adjusted by the user explicitly via the dialog manager module  113 . In one example, different expressions can be generated by the response generation module  115  at a different time because the users can be in different attention modes during these time periods. 
     With further reference to  FIG. 1 , another example of a value the conversational context detection and management module  111  can be configured to compute the conversational context such as a few conversational stages including introduction, greeting, continuation, switching topics, conclusion, repair of a miscommunication due to speaker or hearer, facing failure in solving problems at hand, or the like. In one example, this information can be tracked and updated with input from dialog manager module  113 . 
     With further reference to  FIG. 1 , the dialog manager module  113  can be configured to communicate with the response generation module  115 . In one example, the dialog manager module  113  can provide the response generation module  115  information such as content information that the user requested and the context information. In one example, based on the information about the user obtained from the different modules  101 ,  103 ,  105 ,  107 ,  109 ,  111 , and  113  and the context information, the response generation module  115  decides what expressions to construct by selecting different wordings with different prosodic marks. 
     With further reference to  FIG. 1 , the response generation module  115  can be configured to use multiple aspects of human communication to design the responses of the system  100 . In one example, the multiple aspects can include system  100  personality, user (conversational partner), context, and language genre. In another example, in addition to the content selection as the input to the response generation module  115 , the responses from the system  100  can be generated using an exemplary method. The exemplary method for generating responses from the system  100  can include selecting proper wording. In one example, the words and expressions can be dynamically or statically selected conditioned on the user&#39;s state and context as described herein. Examples of lexical and expression variations can include, but are not limited to, number of syllables, length variations of different pronunciations with full or shortened forms, difficulty degree to pronounce by common people, confusability with other words, humorous or formal or the like. The exemplary method to generating a response can further include selecting a proper tone. In one example, the words and phrases can be marked, including stressing a phrase, lengthening or shortening certain vowel or consonant, rising or falling at the end of an utterance. The exemplary method to generating a response can further include selecting a volume level. In one example, the system  100  can be configured to control a proper speech output volume given the acoustic environment, such as loud noisy and quiet, and the speaker&#39;s or user&#39;s volume, such as weak, energetic, or normal. 
     With further reference to  FIG. 1 , the system  100  can further include a database  119  where the modules  101 ,  103 ,  105 ,  107 ,  109 ,  111 ,  113 ,  115  can construct and maintain knowledge bases. The knowledge bases can be used in the system  100  to support response generation by the response generation module  115 . The databases can be located locally in the system  100  or can engage with the system  100  from an external location. In one example, the knowledge bases can include a rich word dictionary with lexical information of duration, formality, and intelligibility. A rich word dictionary can include information such as number of syllables, length variations of different pronunciations with full or shortened forms, difficulty degree to pronounce by common people, confusability with other words. The knowledge bases can also include a rich expression dictionary. In one example, the rich expression dictionary can include expressions conditioned with the user and context information disclosed herein, expression frequencies occurred as the output from the system, indication of whether the expression is recognized by the speech recognizer, usage, and conditions including what situations and to whom the expression is used. Example of conditions as disclosed herein include, but are not limited to, age, gender, profession, relationship, acquaintance, emotion indicator, personality indicator, interaction setting, response urgency or pace, operation time, conversational context, or the like. Examples of conditions may further include, but are not limited to, the ontological relationship among all the previously mentioned aspects, which may be hierarchical, scalar, categorical, and/or binary, or the like. Multiple phrases or terms can be attached to these conditions, such as for family members, one may use parent, father, mother, boy, kid, etc. 
       FIG. 2  illustrates an exemplary method for building the knowledge bases for the database  119  in the system  100 . In one example, the method can be used to obtain conditions together with word level and expression level information. In another example, the method can use a semi-automatic approach to label data corpora with the conditions described herein. These corpora can include, but are not limited to, online tweet corpora or discussion forum in related domains, telephony conversational data corpora, such as Switchboard from LDC, and movie transcription corpora of different conversations or dialogs. In another example, speakers or users and their communication partners can be identified with a specific user id (such as specific user ids used in twitter or switchboard), or characters in movie. The specific user ids can be used for annotation of the corpora. 
     With further reference to  FIG. 2 , the exemplary method can include first identifying a criterion (Step  201 ). This can involve selecting one or more sets of criteria. Identifying the criterion can involve selecting one or more system personalities together with associated sets of criteria (Step  201 ). The criteria can be personality criteria. This identification or selection can be performed manually for a small set of the data and use one or more machine learning algorithms to train statistical models for classification candidates into speakers or non-speakers for selection or identification (Step  201 ). Examples of machine learning algorithms include, but not limited to, support vector machine (SVM), decision tree, maximum entropy (ME), neural networks, among others, or the like. The method can further include identifying a number of users, for example number of user ids or characters, in the corpora that meet the criteria (Step  203 ). The exemplary method can further include identifying the conversation episodes and conversation partners (Step  205 ). After identifying the conversation episodes and conversation partners, the method can involve or engage these identified people with the selected personality (reflecting the system personality) (Step  205 ). The exemplary method can further include labelling the users such as the conversational partners into the user groups such as middle age, female, adult, talking to another family member, or the like, as well as how familiar the conversation partners are (Step  207 ). This can further include involving their conversations with corresponding context (e.g., family setting, medium pace) (Step  207 ). The method can further include labelling the prosodic marks. In one example, the prosodic marks can be labelled with, among other things, a speech recognizer for phone duration, speaking rate, hesitation, repeats, hesitation, revision, or interrupting; emotion detector for happy, angry, sad, or neutral mood; speaker energy detection for a loud or soft speaker (Step  209 ). The method can further include extracting and storing the information in the database  119 , for example into a rich word dictionary and the rich expression dictionary in the database  119  (Step  211 ). 
     In one embodiment, the disclosed subject matter can enable the system  100  to use multiple aspects of human communication labeled in the database  119  to design the responses of the system  100 . In some embodiments other external sources and other conditions can be used in the system  100 . In other embodiments, these external sources and other conditions are be used together with word level and expression level information in the system  100 . 
     In another embodiment, the disclosed subject matter can enable to system  100  to generate responses using word choices, expression selection in real life, prosody or emotion in sentences, frequencies of such response in dialog systems, context-sensitive and personalized. In one exemplary embodiment databases  119  can also come from different corpora on different subjects such twitter, discussion forum, LDC conversational corpus, YouTube video, movies. In one exemplary embodiment, different user related aspects such as age, gender, profession, mood, relationship, familiarity, context related aspects such as conversational setting, urgency, time, state, and genre-related aspects can be integrated in the system  100 . The disclosed subject matter provides an example of modules that detect and track these states. The disclosed subject matter also provides the construction of knowledge bases that can be used by the system  100 . 
     In one exemplary embodiment, the disclosed subject matter can provide to the users a personalized context-sensitive response system  100  with different personalities for uses in home, office, enterprise, or mobile applications. 
     It will be appreciated that variants of the above-described and other features and functions, or alternatives thereof, may be desirably combined into many other different systems, applications or methods. Various presently unforeseen or unanticipated alternatives, modifications, variations or improvements may be subsequently made by those skilled in the art that are also intended to be encompassed by the foregoing disclosure.