Patent Publication Number: US-9431027-B2

Title: Synchronized gesture and speech production for humanoid robots using random numbers

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 61/436,546 entitled “Synthesized Gesture and Speech Production for Humanoid Robots,” filed on Jan. 26, 2011, which is incorporated by reference herein in its entirety. 
    
    
     FIELD OF THE INVENTION 
     The present invention is related to producing gestures in a robot by analyzing a text of speech that is to be generated by the robot. 
     BACKGROUND OF THE INVENTION 
     Various modes may be used for human-robot interaction (HRI). One mode of HRI is the use of a computer user interface. An example of this mode is the use of a computer to upload commands or instructions to the robot. Such interactions are generally reserved for a user that is familiar with computer programming and has in-depth knowledge of a robot&#39;s operation. Another mode of operation is using speech synthesis and speech recognition. For general public lacking technical knowledge of robots, a natural way of interacting with the robots is by speaking to the robots and listening to any speech generated by the robots. Such mode of operations is intuitive on the part of humans but may require hardware devices and computer programs to process and synthesize speeches. 
     One way of enhancing interactions between humans and robots is to use gestures. A gesture is a form of non-verbal communication made by bodily actions. By itself or in conjunction with speech, gestures allow individuals to communicate with others effectively. 
     Gestures can be categorized into different types. One category of gestures is emblems. Emblems refer to self-contained gestures whose meaning can be understood without spoken words. Emblems include, for example, waving a hand to say goodbye or hello. Iconics are a category of gestures used in conjunction with words to indicate concrete things. Iconics include, for example, tracing out a trajectory of a path. Metaphorics are a group of gestures that provide imagery of the abstract. Metaphorics include a gesture referring to the sides of an argument by appearing to be holding invisible items in left and right hands. Deictics are gestures that utilize parts of the body to point out both concrete and abstract things during a conversation. Deictics include, for example, using an arm with the index finger extended at a target of interest. Beats are a group of gestures that are expressed in rhythmic hand motions in synchrony with the cadences of speech. More than one type of gestures may be expressed during a course of speech. 
     Humanoid robots have appearance and features similar to humans. Hence, many people feel a natural affinity for these robots. However, the humanoid robots often remain stationary during interactions with humans. Hence, users often feel interactions with the robots unnatural and awkward compared to human. 
     SUMMARY OF THE INVENTION 
     Embodiments relate to generating gestures in a robot by analyzing a speech text using different sets of rules to identify one or more candidate gestures for different types of gestures. A gesture may be selected from one or more candidate gestures for execution. Actuators in a robot may be controlled by generating actuator signals corresponding to the selected gesture. 
     In one embodiment, a voice output generated by synthesizing the speech text is synchronized with the gesture generated by the robot. The synchronization may be performed by adjusting the selected gesture or the synthesized voice. 
     In one embodiment, the speech text is tagged with information by analyzing the speech text. The one or more candidate gestures are identified by further analyzing the tagged information in addition to the speech text. 
     In one embodiment, the tagged information indicates types of words of the speech elements. 
     In one embodiment, an expressivity parameter is received to indicate a degree of expressivity to be expressed by the robot. A higher expressivity parameter may increase the chance of a gesture viewed as more expressive to be selected while a lower expressivity parameter may increase the change of a gesture viewed as less expressive to be selected. The gesture is selected based further on the expressivity parameter. 
     In one embodiment, at least one of amplitude, frequency and speed of the selected gesture is modified based on a random number. In this way, randomness may be introduced into the gesture, rendering gestures non-repetitive and making the gesture look more natural. 
     In one embodiment, the selected gesture is planned by adding a preparatory motion before making a motion corresponding to the selected gesture. The preparatory motion may move an effector from an end position of a prior gesture or a starting gesture to an initial position of the selected gesture. 
     In one embodiment, a plurality of pattern modules are used to detect matching of patterns in the speech text. The plurality of pattern modules include a first pattern module configured to apply a first set of rules to detect emblems, a second pattern module configured to apply a second set of rules to detect iconics, a third pattern module configured to apply a third set of rules to detect metaphorics, a fourth pattern module configured to apply a fourth set of rules to detect deictics, and a fifth pattern module configured to apply a fifth set of rules to detect beats. One or more of the pattern modules may apply grammar rules to the speech text. 
     The features and advantages described in the specification are not all inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The teachings of the present disclosure can be readily understood by considering the following detailed description in conjunction with the accompanying drawings. 
         FIG. 1  is a schematic diagram of a robot and a remote computer communicating with the robot, according to one embodiment. 
         FIG. 2  is a block diagram of the robot, according to one embodiment. 
         FIG. 3A  is a block diagram of a gesture generator in the robot, according to one embodiment. 
         FIG. 3B  is a block diagram illustrating software components of the gesture generator of  FIG. 3A , according to one embodiment. 
         FIG. 4  is a diagram illustrating active grammar identifiers of different grammar modules, according to one embodiment. 
         FIG. 5  is a graph illustrating weights associated with selecting a gesture to be expressed by the robot, according to one embodiment. 
         FIG. 6  is a block diagram illustrating a motion generator, according to one embodiment. 
         FIGS. 7A and 7B  are flowcharts illustrating processes of generating a gesture, according to one embodiment. 
     
    
    
     DETAILED DESCRIPTION OF THE DISCLOSURE 
     A preferred embodiment is now described with reference to the figures where like reference numbers indicate identical or functionally similar elements. 
     Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment. 
     Some portions of the detailed description that follows are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times, to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices, without loss of generality. 
     However, all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or “determining” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices. 
     Certain aspects of the embodiments include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by a variety of operating systems. 
     Embodiments also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability. 
     The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. In addition, the embodiments are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings as described herein, and any references below to specific languages are provided for disclosure of enablement and best mode. 
     In addition, the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure is intended to be illustrative, but not limiting, of the scope, which is set forth in the following claims. 
     Embodiments relate to generating gestures in a robot in synchrony with a speech output by the robot. The gestures are generated by analyzing a speech text and selecting an appropriate gesture for a time frame from a plurality of candidate gestures. The speech text is analyzed and tagged with information relevant to generating of the gestures. Based on the speech text, the tagged information and other relevant information, a gesture identifier is selected. A gesture template corresponding to the gesture identifier is retrieved and then processed by adding relevant parameter to generate a gesture descriptor representing a gesture to be taken by the robot. A gesture motion is planned based on the gesture descriptor and analysis of timing associated with the speech. Actuator signals for controlling the actuators such as arms and hands are generated based on the planned gesture motion. 
     As used herein, a “speech text” indicates a text of speech described in a natural language or a high-level language that can be converted to the natural language. The speech text may include a string of words. 
     As used herein, a “gesture identifier” refers to information that identifies a gesture from other gestures at a conceptual level. The gesture identifier may indicate, for example, “waving a hand,” and “pointing to a target object.” 
     As used herein, a “gesture template” refers to mapping of a gesture identifier to a trajectory of an effector (e.g., hand) in a robot. The trajectory may be dependent on one or more parameters associated with corresponding sensors or information processed from the sensors. 
     As used herein, a “gesture descriptor” represents an initial version of trajectory of an effector for expressing a gesture. The gesture descriptor is self-contained information that can be interpreted to generate a gesture without further information. The gesture descriptor may be further refined in a gesture planning stage to prepare a current gesture or blend the current gesture with a previous gesture. 
     As used herein, an “actuator signal” represents a machine level signal indicating coordinates, velocity and/or acceleration of an actuator (e.g., a motor) in a robot. 
     Overview of Robot Configuration 
     Figure ( FIG. 1  is a schematic diagram of a robot  100  and a remote computer  150  communicating with the robot  100 , according to one embodiment. The robot  100  may include, among other components, a plurality of body parts, actuators for causing relative movements between the body parts, a local computer  140 , sensors and output devices (e.g., speaker). The plurality of body parts may include, for example, arms, hands, torso, head, legs and feet. The relative movements of these body parts are caused by actuators such as motors. The sensors may be attached to the body parts to sense the pose of the robot as well as to capture visual images or acoustic signals. 
     The local computer  140  is hardware, software, firmware or a combination thereof for processing sensor signals and other input commands, generating actuator signals, and communicating with other computing devices. In one embodiment, the local computer  140  communicates with a remote computer  150  via a channel  152  to perform distributed data processing or to load software or firmware from the remote computer. The channel  152  may be embodied using wired or wireless technology. 
     The body parts of the robot  100  may be moved relative to each other by actuators to generate gestures. In one embodiment, the gesture may be generated by defining a trajectory of an effector of the robot such as a hand  120 . In other embodiments, trajectory of other effectors such as a foot or head of the robot  100  may also be defined to generate gestures. 
     Although  FIG. 1  illustrates a humanoid form, embodiments may be used in robots of various other configurations to generate gestures using body parts available to the robots. 
       FIG. 2  is a block diagram of the robot  100 , according to one embodiment. The robot  100  may include, among other components, a gesture generator  210 , a motion generator  230 , actuators  250 , sensors  220 , a panoramic attention module  240 , a voice synthesizer  260 , and a speaker  266 . The gesture generator  210 , the motion generator  230 , the panoramic attention module  240  and the voice synthesizer  260  may be embodied as software, firmware, hardware or a combination thereof. Further, the gesture generator  210 , the motion generator  230 , the panoramic attention module  240  and the voice synthesizer may be embodied by a single computing device. 
     The gesture generator  210  is hardware, software, firmware or a combination thereof for generating gesture descriptors  214  by analyzing and processing speech text  204 . The speech text  204  may be generated by a computer program or received from a human operator. The gesture generate  210  is described below in detail with reference to  FIGS. 3A and 3B . 
     In one embodiment, the speech text  204  is represented in the form of a natural human language. The speech text  204  may include meaningful words of phrases for communicating with human. Additional information may be added in the speech text  204  to indicate, for example, changes in the context of conversation, intonation or speed of the speech. In another embodiment, the speech text  204  is represented in a compressed format that abbreviates certain words or phrases to facilitate processing and communication of the speech text. 
     The gesture descriptors  214  include information on joint angles or the trajectory of effectors (e.g., hands  120 ). The gesture descriptors  214  are self-contained instructions that allow the motion generator  230  to plan and generate actuator signals  234  for controlling the actuators  250 . In one embodiment, a single gesture descriptor  214  is active at a single time. However, the active gesture descriptor  214  may change during the course of the speech text and even during the course of a sentence. 
     The motion generator  230  receives the gesture descriptors  214  and processes the gesture descriptors  214  to generate the actuator signals  234 . Each of the actuator signals  234  may be associated with a single actuator (e.g., a motor) to control its operation. The actuator signals  234  may define the coordinate of effectors, angles of joints in the robot  100  or velocity or acceleration of associated actuators  250 . The motion generator  230  performs various functions associated with blending or stitching of gestures, controlling the expressivity and avoiding collision between body parts, as described below in detail with reference to  FIG. 6 . 
     The sensors  220  are hardware components for sensing various physical properties and converting these properties into electrical sensor signals  224 . The sensor signals  224  may include perceptive sensor signals as well as pose sensor signals. The perceptive sensor signals allow the robot to recognize and process various objects or events surrounding the robot. The perceptive sensor signals may be generated, for example, by cameras or microphones. The pose sensor signals indicate the relative positions and/or movements of the body parts of the robot  100 . The pose sensor signals enable detection of the actual pose of the robot  100 . 
     The panoramic attention module  240  processes the sensor signals  224  to map the locations of events or objects surrounding the robot  100  into a panoramic coordinate system. By processing and identifying the locations of events or objects, the panoramic attention module  240  allows the robot  100  to perform gestures that are consistent with the locations or events or objects. For example, the panoramic attention module  240  enables the robot  100  to point to an object or person (i.e., entities) during the speech based on entity information  244 . An example technique for embodying a panoramic attention module  240  is described, for example, in U.S. patent application Ser. No. 12/819,032, filed on Jun. 18, 2010, entitled “Panoramic Attention for Humanoid Robots,” which is incorporated by reference herein in its entirety. 
     The voice synthesizer  260  synthesizes electronic signals  264  to generate speech on a speaker  266  or other audio output devices, using a method well known in the art. The voice synthesizer  260  also provides an output  262  to the motion generator  230  to allow the motion generator  230  to check the progress of speech and make any adjustments to the speed of the gestures so that the speech and the gestures can be synchronized. 
     One or more of the components illustrated in  FIG. 2  may be embodied in the remote computer  150  to alleviate the volume of data to be processed at the local computer  140  or to simplify hardware components installed on the robot  100 . In one embodiment, separate sensors may be connected to the remote computer  150  to provide additional information to the robot  100  concerning the environment surrounding the robot  100 . 
     Example Gesture Generator 
       FIG. 3A  is a block diagram of the gesture generator  210  in the robot  100 , according to one embodiment. The gesture generator  210  receives the speech text  204  and generates the gesture descriptors  214  representing the gestures to be taken by the robot  100 . The gesture generator  210  may include, among other components, a processor  310 , an output interface  314 , an input interface  318  and memory  330 . These components are connected via a bus  322 . As described above with reference to  FIG. 2 , the gesture generator  210  may be combined with other components of the robot  100  such as the motion generator  230 , panoramic attention module  240  and the voice synthesizer  260 . 
     The processor  310  is a hardware component that reads and executes instructions, and outputs processed data as a result of the execution of the instructions. The processor  310  may include more than one processing core to increase the capacity and speed of data processing. 
     The output interface  314  is a component that allows the gesture generator  210  to output data to other components of the robot  100 . For example, the output interface  314  may send the gesture descriptors  214  to the motion generator  230  when the motion generator  230  is embodied on a computing device separate from the gesture generator  210 . The output interface  314  may be embodied using various protocols including among others, IEEE 1394 and universal serial bus (USB). 
     The input interface  318  is a component that allows the gesture generator  210  to receive data from other components from the robot  100 . For example, the input interface  318  may receive the speech text  204  from an external source. The input interface  318  may be embodied using various protocols including, among others, IEEE 1394 and USB. In one embodiment, the input interface  318  is combined with the output interface  314  to perform bi-directional communication with various components of the robot  100 . 
     The memory  330  stores instruction modules and/or data for performing data processing operations at the processor  310 . The details of instructions modules in the memory  330  are described below in detail with reference to  FIG. 3B . 
     Although  FIG. 3  illustrates the construction of the gesture generator  210 , each of the motion generator  230 , panoramic attention module  240  and the voice synthesizer  260  may include the same of similar components as the gesture generator  210  to perform their functions. 
       FIG. 3B  is a block diagram illustrating software components of the gesture generator  210  of  FIG. 3A , according to one embodiment. One or more software components illustrated in  FIG. 3B  may also be embodied as dedicated hardware components or firmware. The memory  330  may store, among other software components, an interface  334 , a speech content analyzer  338 , a plurality of grammar modules  340 A through  340 N (hereinafter collectively referred to as “the grammar modules  340 ”), a speech timing analyzer  348  and a gesture selection module  350 . 
     The interface  334  receives the speech text  204  from an external source (e.g., the remote computer  150 ) and stores the speech text  204  for processing or reference by other software modules. To enable communication with the external source, the interface  334  may comply with a communication protocol. 
     The speech content analyzer  338  receives the buffered speech text  336  and may tag the speech text  336  with additional information. The tagged information may indicate contextual hints such as a change in the topic and the degree of excitement associated with the speech. In one or more embodiment, the speech content analyzer  338  uses Stanford Log-linear Part-Of-Speech Tagger, as described in K. Toutanova and C. Manning, “Enriching the knowledge sources used in a maximum entropy part-of-speech tagger,” In Proceedings of the Workshop on Balanced Perception and Action in ECAs at AAMAS (April, 2004), which is incorporated by references herein in its entirety, to assign a word type (e.g., noun, verb, adjective) to each word in the speech text  336 . In addition to identifying the word types, the speech content analyzer may tag certain words or phrases indicating an emotional or psychological state (e.g., calm, excited, neutral, happy or sad). The speech content analyzer may also detect any changes in the topic or detect emphasis in the speech text  336  (e.g., indicated by any italicized words). 
     The original speech text  336  with (or without) the tagged information  339  is provided to a plurality of grammar modules  340 . Each grammar module is a pattern matching module that analyzes the speech text  336  and the tagged information  339  to detect certain patterns in the speech text  336 . Based on the detected patterns, applicability of a certain type or category of gestures can be determined. In one embodiment, five grammar modules  340  are provided in the memory, each identifying and activating the following types or categories of gestures: (i) emblems, (ii) iconic, (iii) metaphorics, (iv) deictics, and (v) beats. For a sequence of words or sentences, one or more of these types of gestures may become active or applicable during the course of the word sequence, as described below in detail with reference to  FIG. 4 . When categories or types of gestures become active or applicable, each of the corresponding grammar modules  340  generates gesture identifiers  342 A through  342 N (hereinafter collectively referred to as “the identifiers  342 ”). Alternatively, the gesture identifiers  342  are streamed from the grammar modules  340  to the gesture selection module  350  regardless of the activation or applicability of the types of gesture but flagged with information to indicate the activation or applicability of the types of gestures. 
     Specifically, each of the grammar modules  340  has a set of rule that analyzes the speech text  336  and the tagged information to determine if a certain type of gestures is applicable. Taking an example of emblems, a grammar module (e.g.,  340 A) dedicated to emblems analyzes if certain key words or phrases appear in the speech text  336 . If such key words or phrases appear, the grammar module outputs a gesture identifier (e.g.,  342 A) indicating that an emblem is active and what the gestures should represent. For example, if the speech text  336  indicates “Hello” or “Bye,” the grammar module (e.g.,  340 A) generates and outputs a gesture identifier (e.g.,  342 A) that an emblem is applicable or active and that the corresponding gesture should be a gesture of waving hands to a target person. Taking another example of beats, a grammar module (e.g.,  340 B) dedicated to beats may generate and output gesture identifiers (e.g.,  342 B) on a periodic basis depending on the cadences of the associated speech. 
     Some of the grammar modules  340  may also detect certain types of words (e.g., verbs) by using the speech content analyzer  338  and map these words to certain types of gestures (e.g., eruptive-type of gestures). Further, the grammar modules  340  may find higher level patterns such as certain types of phrases (e.g., “between . . . and . . . ”, which may cause the robot to take certain gestures between the words). Different grammar modules  340  may detect different key words or phrases, and hence, generate gesture identifiers at different parts of the word sequence. 
     The gesture selection module  350  receives the gesture identifiers  342  from a plurality of the grammar module  340  and selects which gesture type should be expressed by the robot  100 . In one embodiment, the gesture selection module  350  selects a gesture type based on expressivity parameter  362 . The expressivity parameter  362  may be provided by a human or a computer algorithm. Alternatively, the expressivity parameter  362  may be set algorithmically by a computer program or based on the analysis of the speech text. The gesture selection module  350  selects the type of gesture corresponding to a certain time frame of the speech and sends the gesture identifier  352  of the selected gesture type to the motion generator  230 . The selected gesture identifier  352  may change for a set of sentences, for individual sentences or for each portion of the sentence (e.g., a phrase or a word). Hence, the gesture expressed by the robot  100  may change after the robot  100  generates an output for a set of sentences, for an individual sentence or for a portion of the sentence. 
     The speech timing analyzer  348  also receives the speech text  336  and parses through the speech text  336  to determine the timing when each word or phrase is to be generated by the voice synthesizer  260 . In one embodiment, a text-to-speech engine is used to build a table indicating the amount of time needed to speak or generate each word or phrases. The table is then referenced to estimate the timing when each speech element (e.g., word or phrase) is spoken or generated. Based on the analysis, the speech timing analyzer  348  generates and outputs timing information  344  to the motion generator  230 . As described below in detail with reference to  FIG. 6 , the timing information  344  may be compared with the voice synthesizer output  262  to confirm the progress of speech and correct any timing discrepancy with the predicted reproduction time of the speech. 
     In one embodiment, the robot  100  may include other components that provide information other than the speech text  204 . Such information may also be taken into account by the speech content analyzer  338  and/or the gesture selection module to provide additional information to the grammar modules  340  or to select the gesture identifiers  342 . 
     Selection of Gesture Type 
       FIG. 4  is a diagram illustrating active gesture identifiers of different grammar modules  340 , according to one embodiment. As a word sequence (e.g., a sentence) is analyzed by a plurality of grammar modules, each of the grammar modules  340  may detect and generate the gesture identifiers  342  at different points of the word sequence. 
     As illustrated in  FIG. 4 , as a sliding line  410  moves from the left (i.e., starting point of the sentence) to the right (i.e., the ending point of the sentence), the gestures identifier  342  are activated at different points of the word sequence. For example, the grammar module  340 A remains inactive during period  404 A,  404 C and  404 E whereas the same module  340 A generates gesture identifiers  342 A during periods  404 B and  404 D. The grammar module  340 B remains inactive during periods  408 A and  408 B but generates a gesture identifier  342 B during period  408 B. The grammar module  340 N remains inactive during periods  412 B,  412 D,  412 F,  412 H and  412 J but generates a gesture identifier  342 B during period  412 A,  412 C,  412 E,  412 G,  4121  and  412 K. In the example of  FIG. 4 , the sliding line  410  is at a point in the word sequence where the gesture identifier  342 B is active but the gesture identifiers  342 A and  342 N are inactive. In other words, a type of gestures detected by the gesture module  340 B is active but the types of gestures detected by the gesture modules  340 A and  340 N are inactive. 
     After the gesture selection module  350  receives the active gesture identifiers  340 , the gesture selection module  350  determines the gesture to be expressed by the robot  100  based on the active gesture identifiers  342  and the expressivity parameter  362 . In one embodiment, different weights are assigned w i (x) to each type of gestures based on the expressivity parameter x. The expressivity parameter x indicates the strength or level of expressivity to be perceived when gestures are taken by the robot  100 . No gesture has the lowest expressivity, beats have relatively low expressivity, iconic gestures have medium expressivity, metaphoric gestures have higher expressivity than the iconic gestures, deictics have higher expressivity and the emblems have the highest expressivity. 
       FIG. 5  is a graph illustrating weights w a (x) through w f (x) for different types of gestures, according to one embodiment. The distribution of weight w i (x) for each type of gestures are centered at different values of an expressivity parameter x. Given the mean value μ i  and variance σ i   2  variance for a given gesture type i, the weight w i (x) for selecting gesture type i can be modeled as a Gaussian over the expressivity parameter x as follows: 
                       w   i     ⁡     (   x   )       =       1       σ   i     ⁢       2   ⁢   π           ⁢     ⅇ       -       (     x   -     μ   i       )     2       /     (     2   ⁢     σ   i   2       )                   Equation   ⁢           ⁢     (   1   )                 
where x takes a value not smaller than 0 and not larger than 1.
 
     In  FIG. 5 , lines  510 ,  514 ,  518 ,  522 ,  528  and  532  represent weights w a (x) for emblems, w b (x) for iconic gestures, w c (x) for metaphoric gestures, w d (x) for deictic gestures, w e (x) for no gesture and w f (x) for beat gestures, respectively. The emblem gestures are set with a high mean and relatively low variance while iconic, metaphoric and deictic gestures have their means centered at intermediate values of x. The distribution of ‘no gesture’ and beats are set wide over the entire range of expressivity parameter x. 
     Although only six types of gestures (including ‘no gesture’) were used in the example of  FIG. 5 , more or fewer types of gestures may be used by the robot  100 . 
     In one embodiment, the gesture type to be expressed by the robot  100  is selected probabilistically using the following equation: 
                       P   i     ⁡     (   x   )       =         w   i     ⁡     (   x   )           ∑     j   ∈   C       ⁢           ⁢       w   j     ⁡     (   x   )                   Equation   ⁢           ⁢     (   2   )                 
where all active candidate gesture types for a word in the word sequence are collected in the set C. As the expressivity parameter x increases, the likelihood of selecting more expressive types of gestures increases. Conversely, as the expressivity parameter x decreases, the likelihood of selecting less expressive types of gestures increases. It is advantageous to select the gesture type probabilistically, among other reasons, because randomness can be introduced to the selection of gestures expressed by the robot  100 . Due to the randomness, the robot  100  does not express the same gestures even when speaking the same text, causing the humans to perceive the gestures of the robot  100  more natural. In one embodiment, a selected gesture type is applicable to a part of a sentence such as a word or a phrase. Based on the selected gesture type, the corresponding gesture identifier  352  is sent to the motion generator  230 .
 
Example Motion Generator
 
       FIG. 6  is a block diagram illustrating the motion generator  230 , according to one embodiment. The motion generator  230  may include, among other components, a motion planner  610 , a motion template database  620 , a motion randomizer  630  and a motion controller  640 . These components may be embodied by the same hardware architecture as the gesture generator  210  illustrated in  FIG. 3 . 
     The motion planner  610  generates a gesture descriptor  612  indicating the gestures to be taken by the robot  100  based on the selected gesture identifier  352 , the timing information  344  and the entity information  244 . Specifically, the motion planner  610  retrieves a gesture template  626  corresponding to the selected gesture identifier  352  and, if needed, fills in the parameters based on the entity information  244  to generate the gesture descriptor  612 . 
     The motion template database  620  stores a plurality of gesture templates (templates A through Z). A gesture template describes a trajectory of one or more effectors in the robot  100 . Each gesture template may be associated with trajectories a different combination of effectors. Some gesture templates may describe a trajectory of a hand only whereas other gesture templates may describe trajectories of a hand and a head. A gesture template could operate to different parameters such as joint angle trajectories over time. Further, some gesture templates may need additional information or parameter to be added before the trajectory can be defined. For example, a gesture template associated with a gesture pointing to a listener may include a parameter indicating where the listener is located. 
     Specifically, a template stored in the motion template database  620  is mapped to a gesture identifier. After the motion planner  610  receives a selected gesture identifier  352 , the motion planner  610  retrieves the gesture template  626  mapped to the gesture identifier  352  from the motion template database  620 . For gesture templates that need additional information or parameter, the motion planner  610  requests and receives the entity information  244  indicating the coordinate of a human in the environment of the robot  100 . The motion planner  610  extracts the coordinate of a relevant entity or other information and adds the coordinate or other information to the gesture template to generate a gesture descriptor  612 . 
     In one embodiment, the gesture template  626  includes a set of key points that represent points to be taken by effectors. Kochanek-Bartels (TCB) cubic splines may then be used to define trajectory curves by interpolating over the set of key points. In one embodiment, tension-continuity-bias (TCB) splines are used to control how smoothly or tightly the trajectories follow the set of key points. 
     The motion planner  610  may modify the gesture as defined by the gesture template based on the timing information  344  to ensure that the gesture takes place in synchrony with the speech. For this purpose, the motion planner  610  may use the voice synthesizer output  262  to determine if the estimated timing of speech and the actual timing on the speech generated by the voice synthesizer  260  match. If the timing does not match, the motion planner  610  delays or advances the motions according to the gesture descriptor  612 . 
     Furthermore, the amplitude of the trajectory of the effectors may be reduced or increased if the trajectory as defined by the gesture descriptor  612  is too large or too small to be finished in synchrony with the word or phrase being spoken. Alternatively, the trajectory may be cut short or repeated so that the time for expressing the gesture is in line with the word or phrase being spoken. 
     The motion planner  610  may also adds a preparatory motion before the trajectory corresponding to the finalized gesture template so that the robot  100  may make smooth transition from a previous gesture or from a starting pose. The preparatory motion may include moving the effector to a location where the initial pose of the gesture that is to take place and moving other parts of the robot  100  (e.g., moving the torso of the robot  100  to face a target human). Such preparatory motion can be taken before a corresponding word or phrase is generated by the voice synthesizer  260 . The trajectory for such preparatory motion is included in the gesture descriptor  612 . 
     The motion planner  610  also modifies the gestures as defined by the gesture descriptor  612  to retract or blend the current motion with other motions of the robot  100 . For example, if a repetitive motion of reaching out an arm is to be taken repetitively, the motion planner  610  adds a retrieving motion to a neutral resting position before taking another reaching motion to make the motions of the robot  100  appear natural. 
     The motion controller  640  receives the gesture descriptor  612  and generates the actuator signals  250 . In order to generate the actuator signals  250 , the motion controller  640  receives a random number  622  from the motion randomizer  630  to afford randomness to the trajectory of the effectors. The random number  622  may cause the amplitude of the trajectory to be increased or decreased and/or change the speed of the effectors. In this way, the robot  100  does not repeat the same gestures evens when the same gesture descriptor  612  is received from the motion planner  610 , rendering the gestures appear more natural to human. 
     In one embodiment, the trajectory of the effector is defined using style parameters that depend on the random number  622 . For example, the style parameters may be defined according to the following equation:
 
 S={A,F,T   i   ,C   i   ,B   i   ,t   i   |i =0  . . . n}   Equation (3)
 
where n is the number of key frames in the current gesture description, A is amplitude of the gesture trajectory, F is the frequency of a gesture element (i.e., a unit of motion included in a gesture) that is repetitive, T i  is tension, C i  is continuity, B i  is bias, and t i  is time for keyframe i that is normalized from 0 to 1. A key frame is a sampled point on the trajectory at a given time. By describing a series of key frames, the trajectory is defined by interpolating these points to smoothly transition over different spatial points. Key frames are often used in splines to define is a smooth and continuous transition between spatial points. The tension refers to the “tightness” of the curve around the interpolated key frames in the trajectory. A high tension curve bends sharply at each key frame whereas a low tension curve bends less sharply at each key frame. In terms of a mathematical model, the tension corresponds to the length of the tangent vector at each key frame. Continuity refers to the mathematical continuity of a curve. C 0  means that the curves merely connect. C 1  means the curves connect at the point and also have the same 1 st  derivative at that point (i.e., same speed). C 2  means the curves connect at the same point and have matching 1 st  derivatives and 2 nd  derivatives (i.e., acceleration). Mathematically, continuity refers to the sharpness in change between incoming and outgoing tangent vectors at each point. Bias refers to the amount of overshoot of the curve. Mathematically, the bias refers to the direction of the tangent at each point. A bias value of −1 has the tangent vector “undershooting” or more in the direction of the previous point in the trajectory while +1 has the tangent vector more in the direction of the next point in the trajectory.
 
     The random number  622  may take different ranges of values based on the status of the robot  100  as determined by analyzing the speech text  336  or as indicated by information provided from an external source. For example, the random number  622  may take a value not smaller than 0.6 and not larger than 1 for an “excited” state, take a value not smaller than 0.3 and not larger than 0.7 for a “neutral” state, and take a value not smaller than 0 and not larger than 0.4 for a “calm” state. Further, the parameters A, F, B i  and T i  in equation (3) may be linear or non-linear transformations of the random number  622 . More complex functions of the random number  622  and style parameters may also be used. In one embodiment, the random number  622  is generated over different ranges depending on the given style tags (e.g., calm, excited or neutral). The random number  622  is then used in functions to set values for the style parameters in Equation 3. For example, in an excited state, a very high tension value and a high bias value may be set for fast and tight arm motions whereas, in a calm state, a high value may be set for continuity to create smooth trajectories. 
     The motion controller  640  may also make modifications to the trajectory as defined by the gesture descriptor  612  to avoid collision between the body parts of the robot  100  in a manner well known in the art. 
     Example Process of Generating Gesture 
       FIGS. 7A and 7B  are flowcharts illustrating the processes of generating a gesture, according to one embodiment. The gesture generator  210  receives  706  a speech text. The speech text may be received from a human operator or a computer algorithm. The received text is then analyzed to tag  710  the text with additional hints or other information. The gesture generator  210  also receives  714  an expressivity parameter x. 
     The gesture generator  210  analyzes  716  the speech text and the tagged information using a plurality of grammar modules  340  to generate gesture identifiers  342  for each type of gestures that are determined as being active or applicable for a certain speech element (e.g., a word or phrase) of the word sequence. A single gesture identifier is generated from a single grammar module for a speech element time although the gesture identifier may change for different speech elements within the same word sequence. 
     The gesture generator  210  then selects  718  a gesture identifier among active gesture identifiers generated by the grammar module  340  based on the expressivity parameter x. If the expressivity parameter x is high, a gesture with higher expressivity is likely to be selected. Conversely, if the expressivity parameter x is low, a gesture with lower expressivity is likely to be selected. In one embodiment, equation (3) is used to select a gesture identifier among the active gesture identifiers. 
     The motion generator  230  then retrieves  722  a motion template corresponding to the selected gesture identifier. The motion generator  230  then generates  724  a gesture descriptor by adding parameters or additional information to the gesture template. The added parameters or information may indicate, for example, the coordinate of a target human. 
     The speech text is also analyzed  726  to determine the timing of speech elements (e.g., words or phrases) in the speech. In one embodiment, the starting times of speech elements are determined so that the timing for starting gestures corresponding to the speech elements may take place at the time the speech element is generated by the voice synthesizer  260 . 
     The motion generator  230  plans  730  a gesture motion based on the analyzed timing of speech elements, the gesture descriptor and a previous gesture (if any). As part of the planning, the motion generator  230  also generates a preparation motion from a prior gesture motion or a starting pose to an initial position for the current gesture motion. Furthermore, as part of the planning, the motion generator  230  may modify the gesture as defined by the gesture descriptor to make motions appear more natural. 
     The motion generator  230  also receives  734  a randomizer parameter. The actuator signals are then generated  738  based on the planned motions and the randomizer parameter. For example, the amplitude, the speed or the frequency (for repetitive motions) may be modified based on the randomized parameter to afford randomness to the gesture. The randomized parameters may modify the style that the gesture is expressed by the robot  100 . In this way, the gestures of the robot  100  would not appear as being mechanically repetitive even if the robot  100  takes the same or similar gestures. The motion generator  230  may also consider possibility of collision between the body parts when generating the actuator signals and modifies the actuator signals so that any collision between the body parts can be avoided. 
     The generated actuator signals are then sent  742  to the actuators to cause relative movements of the body parts. 
     Embodiments as described with reference to  FIGS. 7A and 7B  are merely illustrative. Various modifications can be made to the processes as shown in  FIGS. 7A and 7B . For example, analyzing  726  timing of speech elements in parallel with other processes or occur before generating  724  the gesture descriptor. Also, one or more processes may be omitted. For example, receiving  714  expressivity parameters may be obviated and replaced with generation of a random value corresponding to the expressivity parameters. 
     Alternative Embodiments 
     In one or more embodiments, the processes of generating the gestures may be used in computer graphics or simulations as opposed to a real robot. Simulated gestures may be generated and presented to a user on a screen to facilitate generation of a video sequence including computerized virtual characters or evaluate the actual operations to be performed on a robot. 
     In one or more embodiments, the speech text is processed at a plurality of grammar modules in series as opposed to being processed in parallel. As the speech text is processed at each grammar module, a gesture identifier may be generated at each speech text. Each of the gesture identifiers corresponding to the same word or phrase is collected and then selected for further processing by the motion generator. 
     In one or more embodiments, the processing associated with the gesture generator is performed remotely at the remote computer  150 , and the processing associated with the motion generation is performed locally at the local computer  140  or vice versa. Further, parts of the processes in the gesture generator or the motion generator may be performed on the remote computer  150 . 
     Although several embodiments are described above, various modifications can be made within the scope of the present disclosure. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.