Patent Description:
A "virtual personal assistant" (VPA) is a type of computer program that interacts with a user to perform various operations on behalf of the user. In so doing, a VPA typically processes vocalizations received from the user and interprets those vocalizations as one or more commands. The VPA then maps those commands to one or more corresponding operations that can be executed on behalf of the user. Upon executing the operations, the VPA can engage with the user conversationally, via a synthesized vocalization, to report the results of the operation. For example, a VPA could process a vocalization received from a user and interpret that vocalization as the command, "check email. " The VPA could map that command to an operation for retrieving new emails and execute the operation to obtain new emails for the user. The VPA could then synthesize a vocalization indicating the number of new emails retrieved.

In some implementations, a VPA can be implemented within a vehicle to allow a user to interact with various features of the vehicle without diverting significant attention away from driving. For example, suppose the user needs to adjust the climate control settings of the vehicle to lower the interior temperature to a more comfortable level. The user could vocalize the command "activate air conditioner" in order to instruct the VPA to activate the air conditioner. Upon activating the air conditioner, the VPA could synthesize a vocalization that indicates to the user that the relevant operation has been executed. In this manner, VPAs implemented in vehicles can help users avoid having to divert attention away from driving to manually interact with various vehicle features, thereby increasing overall driving safety.

One drawback of the above techniques is that conventional VPAs can only interpret the semantic components of vocalizations and, therefore, cannot correctly interpret vocalizations that communicate information using emotional components. Consequently, a VPA implemented in a vehicle can sometimes fail to properly perform a given operation on behalf of the user and create a situation where the user has to divert attention away from driving in order to manually perform that operation. For example, suppose a user is listening to the radio and a very loud song suddenly begins to play. The user might quickly instruct the VPA, "decrease volume now!" However, the VPA could fail to interpret the urgency associated with this type of user command and decrease the volume by only one level. To rectify this miscommunication, the user would have to divert attention away from driving and manually decrease the volume of the radio to a more appropriate level, which would decrease overall driving safety.

Another drawback of the above techniques is that, because conventional VPAs cannot correctly interpret vocalizations that communicate information using emotional components, VPAs oftentimes cannot converse with users in a realistic manner. Consequently, a VPA implemented in a vehicle can cause the user to disengage with the VPA or turn the VPA off entirely, thereby creating situations where the user has to divert attention away from driving in order to manually interact with various vehicle features. For example, suppose a user is really excited about receiving a promotion at work and instructs the VPA to determine the fastest route home. If the VPA synthesizes dull, monotone vocalizations to recite the relevant navigation instructions, the user could find the interaction with the VPA depressing and end up turning off the VPA in order to preserve his/her level of excitement. Such outcomes decrease overall driving safety.

<CIT> describes an interface device and method of use, comprising audio and image inputs, a processor for determining topics of interest, and receiving information of interest to the user from a remote resource, an audiovisual output for presenting an anthropomorphic object conveying the received information, having a selectively defined and adaptively alterable mood, an external communication device adapted to remotely communicate at least a voice conversation with a human user of the personal interface device.

<CIT> describes techniques for selecting an emotion type code associated with semantic content in an interactive dialog system. Fact or profile inputs are provided to an emotion classification algorithm, which selects an emotion type based on the specific combination of fact or profile inputs. The emotion classification algorithm may be rules-based or derived from machine learning. A previous user input may be further specified as input to the emotion classification algorithm. The techniques are especially applicable in mobile communications devices such as smartphones, wherein the fact or profile inputs may be derived from usage of the diverse function set of the device, including online access, text or voice communications, scheduling functions, etc..

<CIT> describes methods, systems, and related products that provide emotion-sensitive responses to user's commands and other utterances received at an utterance-based user interface. Acknowledgements of user's utterances are adapted to the user and/or the user device, and emotions detected in the user's utterance that have been mapped from one or more emotion features extracted from the utterance. Extraction of a user's changing emotion during a sequence of interactions is used to generate a response to a user's uttered command. In some examples, emotion processing and command processing of natural utterances are performed asynchronously.

<CIT> describes an emotion recognition apparatus and controlling method thereof. The emotion recognition apparatus includes a communicator, a sensing part configured to collect a user's bio-signal using at least one sensor, a feedback device configured to adjust an feedback element, a storage configured to store correlation information between the user's bio-signal and an emotion factor and correlation information between the emotion factor and the feedback element, and a controller configured to acquire user's situation information through the communicator, acquire user's emotion information on the basis of the user's bio-signal, determine whether feedback information is allowed to be provided on the basis of at least one of the user's situation information and the user's emotion information, and control the feedback device to provide the feedback information when the feedback information is allowed to be provided, to make a user feel familiar with feedbacks of the apparatus.

As the foregoing illustrates, what is needed in the art are more effective ways for VPAs to interact with users when performing operations on behalf of users.

Various embodiments include a computer-implemented method for interacting with a user while assisting the user, including capturing a first input that indicates one or more behaviors associated with the user, determining a first emotional state of the user based on the first input, generating a first vocalization that incorporates a first emotional component based on the first emotional state, wherein the first vocalization relates to a first operation that is being performed to assist the user, and outputting the first vocalization to the user.

At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a VPA to more accurately determine one or more operations to perform on behalf of the user based on the emotional state of the user. Accordingly, when implemented within a vehicle, the disclosed VPA helps to prevent the user from diverting attention away from driving in order to interact with vehicle features, thereby increasing overall driving safety.

So that the manner in which the above recited features of the various embodiments can be understood in detail, a more particular description of the inventive concepts, briefly summarized above, may be had by reference to various embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of the inventive concepts and are therefore not to be considered limiting of scope in any way, and that there are other equally effective embodiments.

In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to one skilled in the art that the inventive concepts may be practiced without one or more of these specific details.

As noted above, conventional VPAs can only interpret the semantic components of vocalizations, and therefore cannot correctly interpret vocalizations that communicate information using emotional components. Consequently, a VPA implemented in a vehicle can sometimes fail to properly perform certain operations on behalf of the user and can therefore create situations where the user has to divert attention away from driving in order to personally perform those operations. In addition, because conventional VPAs cannot correctly interpret vocalizations that communicate information using emotional components, VPAs cannot engage with users conversationally in any sort of realistic manner. Consequently, a VPA implemented within a vehicle can cause the user to disengage with the VPA or turn the VPA off entirely, thereby creating a situation where the user has to divert attention away from driving in order to interact with various vehicle features.

To address these issues, various embodiments include a VPA that is configured to analyze various types of input that indicate one or more behaviors associated with a user. The input may include vocalizations that represent explicit commands for the VPA to execute as well as non-verbal cues associated with the user, such as facial expressions and/or changes in posture, among others. The VPA determines the emotional state of the user based on the input. The VPA also determines one or more operations to perform on behalf of the user based on the input and the determined emotional state. The VPA then executes the one or more operations and synthesizes an output based on the emotional state of the user and the one or more operations. The synthesized output includes one or more semantic components and one or more emotional components derived from the emotional state of the user. The emotional component(s) of the output can match the emotional state of the user or contrast with the emotional state of the user, among other possibilities. The VPA observes the behavior of the user in response to the synthesized output and then implements various modifications, based on the observed behavior, to improve the effectiveness of future interactions with the user.

At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques enable a VPA to more accurately determine one or more operations to perform on behalf of the user based on the emotional state of the user. Accordingly, when implemented within a vehicle, the disclosed VPA helps to prevent the user from diverting attention away from driving in order to interact with vehicle features, thereby increasing overall driving safety. Another technical advantage of the disclosed techniques is that the disclosed techniques enable a VPA to generate conversationally-realistic responses that attempt to reflect the emotional state of the user. Conversationally-realistic responses maintain user engagement with the VPA and, therefore, reduce the number of situations where a user turns off the VPA and interacts with vehicle features manually, which increases overall driving safety. These technical advantages represent one or more technological advancements over prior art approaches.

<FIG> illustrate a system configured to implement one or more aspects of the various embodiments. As shown in <FIG>, a system <NUM> includes a computing device <NUM> coupled to one or more input devices <NUM> and one or more output devices <NUM>.

Input devices <NUM> are configured to capture input <NUM> that reflects one or more behaviors associated with a user <NUM>. As referred to herein, a "behavior" includes any voluntary and/or involuntary actions performed by the user. For example, and without limitation, a "behavior" could include explicit commands issued by the user, facial expressions enacted by the user, changes in affect presented consciously or unconsciously by the user, as well as changes in user posture, heart rate, skin conductivity, pupil dilation, and so forth. Input devices <NUM> may include a wide variety of different types of sensors that are configured to capture different types of data that reflect behaviors associated with the user. For example, and without limitation, input devices <NUM> could include audio capture devices that record vocalizations issued by the user, optical capture devices that record images and/or video depicting the user, pupillometry sensors that measure the pupil dilation of the user, infrared sensors that measure blood flow in the face and/or body of the user, heart rate sensors that generate a beats-per-minute reading associated with the user, galvanic skin response sensors that measure changes in skin conductivity of the user, body temperature sensors that detect changes in the core and/or surface body temperature of the user, brainwave sensors that detect different brainwave patterns, and so forth. As described in greater detail below, computing device <NUM> processes input <NUM> to generate output <NUM>.

Output devices <NUM> are configured to transmit output <NUM> to user <NUM>. Output <NUM> can include any technically feasible type of data associated with any given sensory modality, although in practice output devices <NUM> generate and transmit audio output to user <NUM>. As such, output devices <NUM> generally include one or more audio output devices. For example, and without limitation, audio devices <NUM> could include one or more speakers, one or more acoustic transducers, a set of headphones, a beamforming array, an acoustic field generator, and/or a sound cone.

Computing device <NUM> may be any technically feasible type of computer system, including a desktop computer, a laptop computer, a mobile device, a virtualized instance of a computing device, a distributed and/or cloud-based computer system. Computing device <NUM> includes a processor <NUM>, input/output (I/O) devices <NUM>, and a memory <NUM>, coupled together. Processor <NUM> includes any technically feasible set of hardware units configured to process data and execute software applications. For example, and without limitation, processor <NUM> could include one or more central processing units (CPUs), one or more graphics processing units (GPUs), and/or one or more application-specific integrated circuits (ASICs). I/O devices <NUM> include any technically feasible set of devices configured to perform input and/or output operations. For example, and without limitation, I/O devices <NUM> could include a universal serial bus (USB) port, a serial port, and/or a firewire port. In one embodiment, I/O devices <NUM> may include input devices <NUM> and/or output devices <NUM>. Memory <NUM> includes any technically feasible storage media configured to store data and software applications. For example, and without limitation, memory <NUM> could include a hard disk, a random-access memory (RAM) module, and/or a read-only memory (ROM). Memory <NUM> includes a virtual private assistant (VPA) <NUM>. VPA <NUM> is a software application that, when executed by processor <NUM>, performs various operations based on input <NUM> to generate output <NUM>.

In operation, VPA <NUM> processes input <NUM> captured via input devices <NUM> and determines the emotional state of user <NUM> based on that input. VPA <NUM> also determines one or more operations to perform on behalf of user <NUM> based on input <NUM> and the determined emotional state. VPA <NUM> determines the emotional state of user <NUM> and the one or more operations to perform on behalf of user <NUM> using techniques described in greater detail below in conjunction with <FIG>. VPA <NUM> then executes the one or more operations or causes the one or more operations to be executed by another system. VPA <NUM> synthesizes output <NUM> based on the emotional state of user <NUM> and the one or more operations. The synthesized output includes semantic components related to the one or more operations and emotional components derived from and/or influenced by the emotional state of the user. VPA <NUM> transmits output <NUM> to user <NUM> via output devices <NUM>. VPA <NUM> observes the behavior of user <NUM> in response to the synthesized output and then implements various modifications that can increase usability of and/or user engagement with VPA <NUM>.

As a general matter, system <NUM> may be implemented as a stand-alone system or be integrated with and/or configured to interoperate with any other technically feasible system. For example, and without limitation, system <NUM> could be integrated with and/or configured to interoperate with a vehicle, a smart home, a smart headphone, a smart speaker, a smart television set, one or more Internet-of-Things (IoT) devices, or a wearable computing system, among others. <FIG> illustrates an exemplary implementation where system <NUM> is integrated with a vehicle.

As shown in <FIG>, system <NUM> is integrated with a vehicle <NUM> where user <NUM> resides. System <NUM> is coupled to one or more subsystems <NUM>(<NUM>) through <NUM>(N) within that vehicle. Each subsystem <NUM> provides access to one or more vehicle features. For example, and without limitation, a given subsystem <NUM> could be a climate control subsystem that provides access to climate control features of the vehicle, an infotainment subsystem that provides access to infotainment features of the vehicle, a navigation subsystem that provides access to navigation features of the vehicle, an autonomous driving subsystem that provides access to autonomous driving features of the vehicle, and so forth. VPA <NUM> within system <NUM> is configured to access one or more subsystems <NUM> in order to execute operations on behalf of user <NUM>.

Referring generally to <FIG>, persons skilled in the art will understand that system <NUM> in general and VPA <NUM> in particular can be implemented in any technically feasible environment in order to perform operations on behalf of user <NUM> and to synthesize vocalizations based on the emotional state of user <NUM>. Various examples of devices that can implement and/or include VPAs include mobile devices (e.g., cellphones, tablets, laptops, etc.), wearable devices (e.g., watches, rings, bracelets, headphones, AR/VR head mounted devices, etc.), consumer products (e.g., gaming, gambling, etc.), smart home devices (e.g., smart lighting systems, security systems, smart speakers, etc.), communications systems (e.g., conference call systems, video conferencing systems, etc.), and so forth. VPAs may be located in various environments including, without limitation, road vehicle environments (e.g., consumer car, commercial truck, ride hailing vehicles, snowmobiles, all-terrain vehicles (ATVs), semi and fully autonomous vehicles, etc.), aerospace and/or aeronautical environments (e.g., airplanes, helicopters, spaceships, electric vertical takeoff and landing aircraft (eVTOLs), etc.), nautical and submarine environments (e.g., boats, ships, jet skis), and so forth. Various modules that implement the overarching functionality of VPA <NUM> are described in greater detail below in conjunction with <FIG>.

<FIG> is a more detailed illustration of the VPA of <FIG>, according to various embodiments. As shown, VPA <NUM> includes a semantic analyzer <NUM>, an emotion analyzer <NUM>, a response generator <NUM>, an output synthesizer <NUM>, and a mapping modifier <NUM>. These various elements are implemented as software and/or hardware modules that interoperate to perform the functionality of VPA <NUM>.

In operation, semantic analyzer <NUM> receives input <NUM> from user <NUM> and performs a speech-to-text transcription operation on vocalizations included in that input to generate input transcription <NUM>. Input transcription <NUM> includes textual data that reflects commands, questions, statements, and other forms of linguistic communication that user <NUM> issues to VPA <NUM> in order to elicit a response from VPA <NUM>. For example, and without limitation, input transcription <NUM> could include a command indicating an operation that user <NUM> wants VPA <NUM> to perform. Input transcription <NUM> could also indicate a question that user <NUM> wants VPA <NUM> to answer or a statement that user <NUM> makes to VPA <NUM>. Semantic analyzer <NUM> transmits input transcription <NUM> to response generator <NUM>.

Emotion analyzer <NUM> also receives input <NUM> from user <NUM> and then performs an emotional analysis operation on input <NUM> in order to determine an emotional state <NUM> associated with user <NUM>. Emotion analyzer <NUM> may also determine emotional state <NUM> based on input transcription <NUM>. Emotion analyzer <NUM> can perform any technically feasible approach to characterizing the emotional state of a living entity when generating emotional state <NUM> and in doing so may process any technically feasible form of data included within input <NUM>. For example, and without limitation, emotion analyzer <NUM> could process a vocalization received from user to quantify the pitch, tone, timbre, volume, and/or other acoustic features of that vocalization. Emotion analyzer <NUM> could then map those features to a specific emotional state or emotional metric. In another example, without limitation, emotion analyzer <NUM> could process a video of a facial expression enacted by user <NUM> and then classify that facial expression as corresponding to a particular emotional state. In one embodiment, emotional state <NUM> may include a valence value indicating a particular type of emotion and an intensity value indicating an intensity with which that type of emotion is expressed and/or an arousal level corresponding to that type of emotion, as also described below in conjunction with <FIG>. Persons skilled in the art will understand that emotion analyzer <NUM> can implement any technically feasible approach for characterizing emotions and can generate emotional state <NUM> to include any technically feasible data that describes emotions. Emotion analyzer <NUM> transmits emotional state <NUM> to response generator <NUM>.

Response generator <NUM> is configured to process input transcription <NUM> and emotional state <NUM> in order to generate operations <NUM>. Each operation <NUM> may correspond to a command that is received from user <NUM> and included within input transcription <NUM>. VPA <NUM> can execute a given operation <NUM> in response to a given command on behalf of user <NUM> or offload that operation to another system to be executed on behalf of user <NUM>. For example, and without limitation, VPA <NUM> could offload a given operation <NUM> to one of the vehicle subsystems <NUM> shown in <FIG>. In one embodiment, response generator <NUM> may generate a set of eligible operations based on input transcription <NUM> and then select a subset of those operations to execute based on emotional state <NUM>.

Response generator <NUM> is further configured to process input transcription <NUM> and emotional state <NUM> in order to generate semantic components <NUM>. Semantic components <NUM> include textual data that is synthesized into output <NUM> and subsequently transmitted to user <NUM>, as further described below. Semantic components <NUM> can include words, phrases, and/or sentences that are contextually relevant to input transcription <NUM>. For example, and without limitation, semantic components <NUM> could include an acknowledgement that a command was received from user <NUM>. Semantic components <NUM> can also describe and/or reference operations <NUM> and/or the status of executing those operations. For example, and without limitation, semantic components <NUM> could include an indication that a specific operation <NUM> was initiated in response to a command received from user <NUM>.

Response generator <NUM> is also configured to process input transcription <NUM> and emotional state <NUM> in order to generate emotional components <NUM>. Emotional components <NUM> indicate specific emotional qualities and/or attributes that are derived from emotional state <NUM> and incorporated into output <NUM> during synthesis. For example, and without limitation, a given emotional component <NUM> could include a specific pitch, tone, timbre, volume, diction speed, and/or annunciation level with which the vocalization should be synthesized to reflect particular emotional qualities and/or attributes.

In various embodiments, response generator <NUM> is configured to generate voice responses that, although having the same or similar semantic content, can vary based on various speech levels, such as (i) overall tempo, loudness, and pitch of the synthesized voice, (ii) vocal affect parameters which will be explained in more detail below, (iii) non-verbal and non-language vocalizations, paralinguistic respiration (e.g., laugh, cough, whistles, etc.), and (iv) non-speech altering sounds (e.g., beeps, chirps, clicks, etc.). These voice responses vary in their perceived emotional effect, e.g., the same semantic content can be rendered with speech that appears soft and sweeping to the user, or rash and abrupt. These variations in perceived emotional effect may be generated by using words with soft vs. hard sounds, and polysyllabic vs. abrupt rhythms. For example, sounds such as "<NUM>," "m," and "n," and long vowels or diphthongs, reinforced by a gentle polysyllabic rhythm, are interpreted as "nicer" than words with hard sounds such as "g" and "k," short vowels, and an abrupt rhythm. The field of Sound Symbolism (as described in, for example, http://grammar. com/od/rs/g/soundsymbolismterm. htm) provides a variety of heuristics that attempt to instill affect by connecting particular sound sequences with particular meanings in speech. The vocal affect parameters as noted above generally include (i) pitch parameters (e.g., accent shape, average pitch, contour slope, final lowering, and pitch range), (ii) timing parameters (e.g., speech rate and stress frequency), (iii) voice quality parameters (e.g., breathiness, brilliance, laryngealization, loudness, pause discontinuity, and pitch continuity), and (iv) articulation parameters. The voice output may also include, other than audible speech, non-linguistic vocalizations such as laughter, breathing, hesitation (e.g., "uhm"), and/or non-verbal consent (e.g., "aha").

In some instances, a given emotional component <NUM> can be complementary to or aligned with emotional state <NUM>. For example, and without limitation, if emotional state <NUM> indicates that user <NUM> is currently "happy," then emotional components <NUM> could include a specific tone of voice commonly associated with "happiness. " Conversely, a given emotional component <NUM> can diverge from emotional state <NUM>. For example, and without limitation, if emotional state <NUM> indicates that user <NUM> is currently "angry," then emotional components <NUM> could include a specific tone of voice commonly associated with "calmness. " <FIG> set forth various examples of how response generator <NUM> generates emotional components <NUM> based on emotional state <NUM>.

In one embodiment, response generator <NUM> may implement a response mapping <NUM> that maps input transcription <NUM> and/or emotional state <NUM> to one or more operations <NUM>, one or more semantic components <NUM>, and/or one or more emotional components <NUM>. Response mapping <NUM> may be any technically feasible data structure based on which one or more inputs can be processed to generate one or more outputs. For example, and without limitation, response mapping <NUM> could include an artificial neural network, a machine learning model, a set of heuristics, a set of conditional statements, and/or one or more look-up tables, among others. In various embodiments, response mapping <NUM> may be obtained from a cloud-based repository of response mappings that are generated for different users by different instances of system <NUM>. Further, response mapping <NUM> may be modified using techniques described in greater detail below and then uploaded to the cloud-based repository for use in other instances of system <NUM>.

Response generator <NUM> transmits semantic components <NUM> and emotional components <NUM> to output synthesizer <NUM>. Output synthesizer <NUM> is configured to combine semantic components <NUM> and emotional components <NUM> to generate output <NUM>. Output <NUM> generally takes the form of a synthetic vocalization. Output synthesizer <NUM> transmits output <NUM> to user <NUM> via output devices <NUM>. With the above techniques, VPA <NUM> uses the emotional state of user <NUM> in order to more effectively interpret inputs received from user <NUM> and to more effectively generate vocalizations in response to user <NUM>. In addition, VPA <NUM> can adapt based on the response of user <NUM> to a given output <NUM> in order to improve usability and engagement with user <NUM>.

In particular, VPA <NUM> is configured to capture feedback <NUM> that reflects one or more behaviors user <NUM> performs in response to output <NUM>. VPA <NUM> then updates emotional state <NUM> to reflect any observed behavioral changes in user <NUM>. Mapping modifier <NUM> evaluates one or more objective functions <NUM> based on the updated emotional state <NUM> to quantify the effectiveness of output <NUM> in causing specific types of behavioral changes in user <NUM>. For example, and without limitation, a given objective function <NUM> could quantify the effectiveness of mapping a given input transcription <NUM> to a particular set of operations <NUM> based on whether emotional state <NUM> indicates that user <NUM> is pleased or displeased. In another example, without limitation, a given objective function could quantify the effectiveness of selecting specific semantic components <NUM> when generating output <NUM> based on whether emotional state <NUM> indicates interest or disinterest. In yet another example, without limitation, a given objective function <NUM> could quantify the effectiveness of incorporating "soothing" tones into output <NUM> to calm user <NUM> when user <NUM> occupies a "nervous" emotional state.

As a general matter, a given objective function <NUM> can represent a target behavior for user <NUM>, a target emotional state <NUM> for user <NUM>, a target state of being of user <NUM>, a target level of engagement with VPA <NUM>, or any other technically feasible objective that can be evaluated based on feedback <NUM>. In embodiments where response generator <NUM> includes response mapping <NUM>, mapping modifier <NUM> may update response mapping <NUM> in order to improve subsequent outputs <NUM>. In the manner described, VPA <NUM> can adapt to the specific personalities and idiosyncrasies of different users and therefore improve over time at interpreting and engaging with user <NUM>.

<FIG> set forth examples of how the VPA of <FIG> characterizes and translates the emotional state of a user, according to various embodiments. As shown in <FIG>, emotional state <NUM> is defined using a graph <NUM> that includes a valence axis <NUM>, an intensity axis <NUM>, and a location <NUM> plotted against those two axes. Valence axis <NUM> defines a spectrum of locations that may correspond to different types of emotions, such as "joy," "happiness," "anger," and "excitement," among others. Intensity axis <NUM> defines a range of intensities corresponding to each type of emotion. Location <NUM> corresponds to a particular type of emotion set forth via valence axis <NUM> and a particular intensity with which that emotion is expressed. In the example shown, location <NUM> corresponds to a high level of joy.

Emotion analyzer <NUM> generates emotional state <NUM> based on any of the different types of analyses described previously. Response generator <NUM> then generates emotional component <NUM> that similarly defines a graph <NUM> that includes a valence axis <NUM> and an intensity axis <NUM>. Graph <NUM> also includes location <NUM> that represents the emotional qualities to be included in output <NUM> during synthesis. In the example shown, location <NUM> corresponds to a high level of joy, similar to location <NUM>, and output <NUM> is therefore generated with emotional qualities that are meant to compliment emotional state <NUM> of user <NUM>. Response generator <NUM> can also generate emotional components <NUM> that differ from emotional state <NUM>, as shown in <FIG>.

Referring now to <FIG>, emotional state <NUM> includes location <NUM> that corresponds to an elevated level of frustration. Response generator <NUM> generates emotional component <NUM> to include location <NUM> that corresponds to a low level of cheer. Output <NUM> is therefore generated with emotional qualities that differ from emotional state <NUM> but can potentially modify that emotional state by reducing the frustration of user <NUM> via cheerful vocalizations.

Referring generally to <FIG>, in various embodiments, response generator <NUM> may implement response mapping <NUM> in order to map specific locations on graph <NUM> to other locations on graph <NUM>, thereby achieving the different types of translations between emotional states <NUM> and emotional components <NUM> described by way of example above. Persons skilled in the art will understand that VPA <NUM> can implement any technically feasible approach to characterizing emotional states and/or emotional components beyond the exemplary techniques described in conjunction with <FIG>.

As discussed, system <NUM> in general and VPA <NUM> in particular can be integrated into a wide variety of different types of systems, including vehicles. <FIG> set forth exemplary scenarios where system <NUM> is integrated into a vehicle and the disclosed techniques enable greater usability, thereby preventing user <NUM> from needing to divert attention away from driving.

<FIG> set forth an example of how the VPA of <FIG> responds to the emotional state of a user, according to various embodiments. As shown in <FIG>, system <NUM> is integrated within vehicle <NUM> where user <NUM> resides, as also shown in <FIG>. User <NUM> excitedly exclaims how they forgot about a dentist appointment and asks that VPA <NUM> (within system <NUM>) quickly provide a route to the dentist. VPA <NUM> analyzes input <NUM> and detects specific vocal characteristics commonly associated with feelings of excitement and/or anxiety. As shown in <FIG>, VPA <NUM> then generates an output <NUM> that matches the detected level of excitement. In particular, VPA <NUM> urgently states that the fastest route has been located and then reassures user <NUM> that they will arrive in a timely manner. Subsequently, as shown in <FIG>, user <NUM> expresses relief, which VPA <NUM> processes as feedback <NUM> in order to inform future interactions with user <NUM>. In this example, VPA <NUM> facilitates user engagement by appearing emotionally responsive to user <NUM>, thereby encouraging user <NUM> to continue using VPA <NUM> instead of personally performing various vehicle-oriented operations.

<FIG> set forth an example of how the VPA of <FIG> compensates for the emotional state of a user, according to various embodiments. In this example, VPA <NUM> implements emotional components that are different from the emotional state of user <NUM> in order to effect a change in the emotional state of user <NUM>. As shown in <FIG>, user <NUM> expresses annoyance at having to drive in traffic. VPA <NUM> analyzes input <NUM> and detects specific vocal characteristics commonly associated with feelings of annoyance and/or frustration. As shown in <FIG>, VPA <NUM> then generates an output <NUM> with emotional qualities markedly different from these particular feelings. Specifically, VPA <NUM> despondently apologies and admits that there are no other routes available. Subsequently, as shown in <FIG>, user <NUM> forgets the previous feelings of annoyance and provides consolation to VPA <NUM>, which VPA <NUM> processes as feedback <NUM> in order to inform future interactions with user <NUM>. In this example, VPA <NUM> facilitates user engagement by appearing emotionally sensitive to user <NUM>, thereby encouraging user <NUM> to continue using VPA <NUM> instead of personally performing various vehicle-oriented operations.

<FIG> is a flow diagram of method steps for synthesizing vocalizations that reflect the emotional state of a user, according to various embodiments. Although the method steps are described in conjunction with the systems of <FIG>, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present embodiments.

As shown, a method <NUM> begins at step <NUM>, where VPA <NUM> captures an input that indicates one or more behaviors associated with a user. VPA <NUM> interacts with input devices <NUM> shown in <FIG> to capture the input. Input devices <NUM> may include a wide variety of different types of sensors that are configured to capture different types of data associated with the user, including audio capture devices that record vocalizations issued by the user, optical capture devices that record images and/or video depicting the user, pupillometry sensors that measure the pupil dilation of the user, infrared sensors that measure blood flow in the face and/or body of the user, heart rate sensors that generate a beats-per-minute reading associated with the user, galvanic skin response sensors that measure changes in skin conductivity of the user, body temperature sensors that detect changes in the core and/or surface body temperature of the user, brainwave sensors that detect different brainwave patterns, and so forth. Any and all such data can be processed to determine the emotional state of the user, as described in greater detail below.

At step <NUM>, VPA <NUM> determines the emotional state of the user based on the input. In doing so, VPA <NUM> implements emotion analyzer <NUM> to process any of the above types of data in order to map that data and/or processed versions thereof to the emotional state of the user. Emotion analyzer <NUM> can define the emotional state of the user using any technically feasible approach. In one embodiment, emotion analyzer <NUM> may describe the emotional state of the user via a valence versus intensity dataset, such as that shown in <FIG>. In particular, emotion analyzer <NUM> analyzes specific qualities associated with the input, such as the pitch, timbre, tone, and/or volume associated with user vocalizations, among others things, and then maps those qualities to a particular location within a multidimensional valence versus intensity space. Persons skilled in the art will understand that VPA <NUM> can implement any technically feasible approach to generating data that indicates the emotional state of the user when performing step <NUM>.

At step <NUM>, VPA <NUM> determines one or more operations to perform on behalf of the user based on the input captured at step <NUM> and the emotional state determined at step <NUM>. VPA <NUM> implements response generator <NUM> in order to process a transcription of the input to determine one or more relevant operations to perform on behalf of the user. For example, if the input corresponds to a command to play music, then VPA <NUM> could process a transcription of the input and then activate the stereo system within a vehicle where the user resides. VPA <NUM> can implement any technically feasible approach to generating transcriptions of input, including speech-to-text, among other approaches. In some embodiments, response generator <NUM> can also select a relevant set of operations to perform based on the emotional state of the user determined at step <NUM>. For example, if user occupies a "sad" emotional state, then VPA <NUM> could select a particular radio station that plays somber music.

At step <NUM>, VPA <NUM> executes the one or more operations on behalf of the user in order to assist the user in performing those operations. In some embodiments, VPA <NUM> may execute the one or more operations by executing one or more corresponding subroutines and/or software functions. In other embodiments, VPA <NUM> may be integrated with another system, and VPA executes the one or more operations by causing that system to perform those operations. For example, with implementations where VPA <NUM> is integrated into a vehicle, as described above, VPA <NUM> could execute a given operation to cause one or more subsystems within the vehicle to perform the given operation. In such implementations, VPA <NUM> advantageously executes operations associated with the vehicle on behalf of the user and therefore prevents the user from having to divert attention away from driving to personally perform those operations.

At step <NUM>, VPA <NUM> synthesizes an output based on the emotional state determined at step <NUM> and the one or more operations determined at step <NUM>. VPA <NUM> implements response generator <NUM> in order to generate semantic components of the output as well as emotional components of that output. The semantic components of the output include one or more words, phrases, and/or sentences that relate in a meaningful way to the one or more operations. For example, if a given operation pertains to a set of navigation instructions for navigating a vehicle, then the semantic components of the output could include language associated with a first navigation instruction. In one embodiment, response generator <NUM> may generate the semantic components of the output to have emotive characteristics derived from the emotional state of the user. The emotional components of the output may indicate variations in pitch, tone, timbre, and/or volume that are derived from the emotional state of the user and that are meant to evoke a specific emotional response from the user. The emotional components of the output may also include other factors that influence how the semantic components of the output are conveyed to the user, such as the speed of delivery, timing, and so forth. Based on the semantic components and emotional components, output synthesizer <NUM> generates the output and transmits the output to the user via output devices <NUM>.

At step <NUM>, VPA <NUM> observes the behavior of the user in response to the output synthesized at step <NUM>. VPA <NUM> captures any of the above-mentioned types of data that describe various behaviors associated with the user in order to determine how the user responds to the output. In particular, emotion analyzer <NUM> can analyze any captured data to determine how the emotional state of the user changes in response to the output. For example, and without limitation, emotion analyzer <NUM> could determine that the user, who previously occupied a "frustrated" emotional state, shifted to a "relaxed" emotional state in response to an output that included "soothing" emotional components.

At step <NUM>, VPA <NUM> modifies response generator <NUM> and/or response mapping <NUM> included therein based on the observed behavior. In one embodiment, VPA <NUM> may implement mapping modifier <NUM> in order to evaluate one or more objective functions <NUM> and determine whether response generator <NUM> and/or response mapping <NUM> should be modified. Each objective function <NUM> may reflect a target set of behaviors for the user, a target emotional state, a target state of being, and so forth. For example, and without limitation, objective functions <NUM> could quantify the well-being, connectedness, productivity, and/or enjoyment of the user, and mapping modifier <NUM> could adjust response mapping <NUM> to maximize one or more of these objectives. Mapping modifier <NUM> can evaluate each objective function <NUM> in order to determine the degree to which the observed behavior corresponds to a given target behavior and then modify response mapping <NUM> to increase the degree to which the user expresses the target behavior. VPA <NUM> can implement any technically feasible approach to quantifying the expression of a given behavior.

VPA <NUM> implements the method <NUM> in order to perform some or all of the various features described herein. Although in some instances VPA <NUM> is described in relation to in-vehicle implementations, persons skilled in the art will understand how the disclosed techniques confer specific advantages relative to the prior art in a wide range of technically feasible implementations. As a general matter, the disclosed techniques enable VPA <NUM> to interpret vocalizations received from users more effectively and more accurately and enable VPA <NUM> to generate more conversationally realistic response to users, thereby achieving significantly better usability compared to prior art approaches.

In sum, a virtual private assistant (VPA) is configured to analyze various types of input that indicate one or more behaviors associated with a user. The input may include vocalizations that represent explicit commands for the VPA to execute, emotional state derived from these vocalizations, as well as implicit non-verbal cues associated with the user, such as facial expressions and/or changes in posture, among others. The VPA determines the emotional state of the user based on the input. The VPA also determines one or more operations to perform on behalf of the user based on the input and the determined emotional state. The VPA then executes the one or more operations and synthesizes an output based on the emotional state of the user and the one or more operations. The synthesized output includes one or more semantic components and one or more emotional components derived from the emotional state of the user. The emotional component(s) of the output can match the emotional state of the user or contrast with the emotional state of the user, among other possibilities. The VPA observes the behavior of the user in response to the synthesized output and then implements various modifications, based on the observed behavior, to improve the effectiveness of future interactions with the user.

Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the appended claims.

Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "module," a "system," or a "computer. " Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

Claim 1:
A computer-implemented method for interacting with a user while assisting the user, the method comprising:
capturing a first input that indicates one or more behaviors associated with the user; determining a first emotional state of the user based on the first input;
generating a first vocalization that incorporates a first emotional component based on the first emotional state, wherein the first vocalization relates to a first operation that is being performed to assist the user;
outputting the first vocalization to the user;
generating the first emotional component based on the first emotional state and a response mapping that translates emotional states to emotional components;
capturing a second input that indicates at least one behavior the user performs in response to the output; and
modifying the response mapping based on the second input and a first objective function that is evaluated to determine how closely the at least one behavior corresponds to a target behavior,
wherein the step of determining the first emotional state of the user further comprises:
determining a first valence value based on the first input that indicates a location within a spectrum of emotion types; and
determining a first intensity value based on the first input that indicates a location within a range of intensities corresponding to the location within the spectrum of emotion types, and
wherein the first emotional component corresponds to at least one of a second valence value or a second intensity value that differs from the first emotional state or wherein the first emotional component corresponds to the first valence value and the first intensity value.