Patent Description:
Artificial intelligence (Al) systems are systems capable of implementing human level intelligence and refer to systems in which a machine autonomously learns, determines, and becomes smarter unlike existing rule-based smart systems. A recognition rate may be increased and preferences of a user may be understood more accurately in proportion to iterations of the Al systems, and thus the existing rule-based smart systems are gradually being replaced by the Al systems based on deep learning.

Al technology includes machine learning (or deep learning) and element technologies using machine learning.

Machine learning refers to an algorithm for classifying/learning features of input data, and the element technology refers to a technology for mimicking functions of human brains, e.g., recognition and determination, by using the machine learning algorithm such as deep learning, and includes fields of technologies such as linguistic understanding, visual understanding, inference/prediction, knowledge representation, and operation control.

Examples of various fields to which the Al technology is applicable are as described below. Linguistic understanding refers to a technology for recognizing and applying/processing languages/characters of humans, and includes natural language processing, machine translation, dialog systems, queries and responses, voice recognition/synthesis, etc. Visual understanding refers to a technology for recognizing and processing an object like human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, space understanding, image enhancement, etc. Inference/prediction refers to a technology for determining information and logically performing inference and prediction, and includes knowledge/probability-based inference, optimization prediction, preference-based planning, recommendation, etc. Knowledge representation refers to a technology for automatically processing human experience information into knowledge data, and includes knowledge construction (data generation/classification), knowledge management (data utilization), etc. Operation control refers to a technology for controlling autonomous driving of vehicles and motion of robots, and includes motion control (e.g., navigation, collision avoidance, or driving control), manipulation control (action control), etc..

<CIT> and <CIT> describe digital assistants which detect user emotion and modify their behavior accordingly.

Provided is a method and system for providing emotional interaction to a user, based on emotion information of the user related to a user event.

Terminology used in this specification will now be briefly described before describing the disclosure in detail.

Although the terms used herein are selected, as much as possible, from general terms that are widely used at present while taking into consideration the functions obtained in accordance with the disclosure, these terms may be replaced by other terms based on intentions of one of ordinary skill in the art, customs, emergence of new technologies, or the like. In a particular case, terms that are arbitrarily selected by the applicant may be used and, in this case, the meanings of these terms may be described in corresponding parts of the detailed description. Therefore, it is noted that the terms used herein are construed based on practical meanings thereof and the whole content of this specification, rather than being simply construed based on names of the terms.

It will be understood that the terms "comprises", "comprising", "includes" and/or "including", when used herein, specify the presence of stated elements, but do not preclude the presence or addition of one or more other elements. The term ". unit" or ". module" is used to denote an entity for performing at least one function or operation, and may be implemented using hardware, software, or a combination of hardware and software.

Hereinafter, the disclosure will be described in detail by explaining embodiments of the disclosure with reference to the attached drawings. The disclosure may, however, be embodied in many different forms and is not construed as being limited to the embodiments of the disclosure set forth herein. In the drawings, parts not related to embodiments of the disclosure are not illustrated for clarity of explanation, and like reference numerals denote like elements throughout.

<FIG> is a schematic diagram of a system for providing event-emotion-based responses, according to an embodiment.

Referring to <FIG>, the system for providing event-emotion-based responses, according to an embodiment may include an electronic device <NUM>. According to an embodiment, in addition to the electronic device <NUM>, the system for providing event-emotion-based responses may further include a server (not shown), and a wearable device for measuring a biometric signal of a user. An embodiment in which the system for providing event-emotion-based responses includes the electronic device <NUM> and an external wearable device will be described in detail below with reference to <FIG>.

The electronic device <NUM> according to an embodiment may be implemented in various forms. For example, as used herein, the electronic device <NUM> may include a digital camera, a smartphone, a laptop computer, a tablet PC, an e-book reader, a digital broadcast receiver, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, or an MP3 player, but is not limited thereto. As used herein, the electronic device <NUM> may be a wearable device that may be worn by a user. The wearable device may include at least one of an accessory-type device (e.g., a watch, a ring, a wrist band, an ankle band, necklace, glasses, or contact lenses), a head-mounted device (HMD), a fabric- or clothes-integrated device (e.g., electronic clothing), a body-attachable device (e.g., a skin pad)), and a body-implantable device (e.g., an implantable circuit), but is not limited thereto. In the following description, for convenience of explanation, a case in which the electronic device <NUM> is a smartphone will be described as an example.

According to an embodiment, the electronic device <NUM> may be a device for providing an interactive interface. The interactive interface may be a user interface for receiving an input (e.g., a voice input or a text input) from a user and providing a response to the input from the user. According to an embodiment, the interactive interface may include, for example, a virtual assistant or an artificial intelligence (AI) assistant, but is not limited thereto. The virtual assistant or the Al assistant may be a software agent for processing a task required by a user and providing a service specialized for the user.

According to an embodiment, the electronic device <NUM> may obtain (or analyze or inter) an emotional state of a user related to a user event <NUM> by using an Al model, and adaptively provide a response through the interactive interface, based on the emotional state of the user. For example, when an emotional state of a first user related to the user event <NUM> is an afraid state <NUM>, the electronic device <NUM> may determine a first interaction type, and respond to an utterance of the first user, based on the first interaction type. When the emotional state of the first user related to the user event <NUM> is an angry state <NUM>, the electronic device <NUM> may determine a second interaction type, and respond to an utterance of the first user, based on the second interaction type. When the emotional state of the first user related to the user event <NUM> is a happy state <NUM>, the electronic device <NUM> may determine a third interaction type, and respond to an utterance of the first user, based on the third interaction type. Herein, the first to third interaction types may be different types and thus the electronic device <NUM> may provide event-based emotional interaction to the user.

As used herein, the user event <NUM> may be an event related to the user, e.g., an event detected by the electronic device <NUM> during a conversation with the user, or an event pre-registered by the user in, for example, a scheduling application, a calendar application, a contact list application, or a social networking service (SNS) application. According to an embodiment, the user event <NUM> may include, for example, an event related to work (e.g., a meeting, a presentation, or a business trip), an event related to an acquaintance (e.g., a wedding of a friend, a birthday party of a family member, or a wedding anniversary), an event related to hobbies (e.g., watching a sport game, watching a movie, reservation of a ticket for a musical performance, a photography club event, participation in a tennis game, or participation in a marathon race), an event related to a celebrity (e.g., a concert or a fan meeting), or an official event (e.g., a presidential election, the World Cup, or Christmas), but is not limited thereto.

Al-related functions according to the disclosure are performed using a processor and a memory. The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors such as a central processing unit (CPU), an application processor (AP), and a digital signal processor (DSP), dedicated graphics processors such as a graphics processing unit (GPU) and a vision processing unit (VPU), or dedicated Al processors such as a numeric processing unit (NPU). The one or more processors control processing of input data based on a predefined operation rule or Al model stored in the memory. Alternatively, when the one or more processors are dedicated Al processors, the dedicated Al processors may be designed in a hardware structure specialized in processing a specific Al model.

The predefined operation rule or Al model is made through training. Herein, being made through training means that a basic Al model is trained based on multiple pieces of training data by using a learning algorithm and thus a predefined operation rule or Al model configured to achieve desired characteristics (or purposes) is made. The training may be performed by a device having an Al function according to the disclosure, or by a separate server and/or system. The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited thereto.

The Al model may include a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values and performs neural network calculation through calculation between a calculation result of a previous layer and the plurality of weight values. The plurality of weight values of the plurality of neural network layers may be optimized by a result of training the Al model. For example, the plurality of weight values may be modified to reduce or minimize a loss value or a cost value obtained by the Al model during the training process. An artificial neural network may include, for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, but is not limited thereto.

Therefore, according to an embodiment, events meaningful to a user may be detected through a conversation, and proactive interaction or necessary information may be provided to the user, based on the detected events, thereby providing true emotional interaction. An operation, performed by the electronic device <NUM>, of providing emotional interaction, based on an emotional state of a user for the user event <NUM> will now be described in detail with reference to <FIG>.

<FIG> is a flowchart of a method, performed by the electronic device <NUM>, of providing an interactive interface, according to an embodiment.

In operation S210, the electronic device <NUM> may obtain emotion information of a user related to a user event registered in the electronic device <NUM>.

According to an embodiment, the electronic device <NUM> may check a user event registered in an event database (DB). The event DB may include or be connected to, for example, a scheduling application (e.g., a to-do list application), a calendar application, an SNS application, or a contact list application. The user event may be an event manually input by the user, or an event automatically input by the electronic device <NUM> or a server.

According to an embodiment, the electronic device <NUM> may obtain the emotion information of the user related to the user event by analyzing a response from the user and leaning the emotion information of the user related to the user event. For example, the electronic device <NUM> may query the user about schedule information by using a virtual assistant interface. In this case, the electronic device <NUM> may receive, from the user, a response including information about at least one event. The electronic device <NUM> may identify an emotional state of the user for the at least one event by analyzing the response received from the user. The electronic device <NUM> may learn emotion information of the user for the at least one event. Herein, the learning of the emotion information of the user for the at least one event may mean determining a weight of at least one emotion element connected to the at least one event.

According to an embodiment, the electronic device <NUM> may extract a user event from voice data of an utterance of the user, and register the extracted user event as a new event in the event DB. The operation, performed by the electronic device <NUM>, of extracting the user event from the voice data of the utterance of the user will be described in detail below with reference to <FIG>.

According to an embodiment, the emotion information of the user related to the user event may be information about an emotional state of the user for the user event. Herein, the emotion information of the user may be information indicating a single emotional state, or information indicating a complex emotional state including two or more emotions. For example, the emotion information may include at least one emotion from among joy, nervousness, afraidness, happiness, fear, displeasure, annoyance, embarrassment, sadness, depression, regret, jealousy, disappointment, confidence, love, hatred, love and hatred, interest, exciting, stability, relief, and horror, but is not limited thereto.

According to an embodiment, the electronic device <NUM> may obtain the emotion information of the user related to the user event at a time when an utterance of the user related to the user event is detected. According to another embodiment, the electronic device <NUM> may obtain the emotion information of the user for the user event to proactively provide interaction before an utterance of the user related to the user event. For example, the electronic device <NUM> may check a scheduled time of the user event, and obtain current emotion information of the user for the user event to provide a notification related to the user event <NUM> minutes before the user event.

According to an embodiment, the electronic device <NUM> is adapted to obtain the emotion information of the user related to the user event by using the Al model. The electronic device <NUM> is adapted to obtain the emotion information of the user corresponding to a specific user event from an event-emotion DB generated (or refined) by the Al model. The event-emotion DB may store average emotion information of ordinary people for a specific event, or emotion information personalized per user for a specific event, but is not limited thereto.

According to an embodiment, the Al model may be trained based on at least one of emotion information analyzed by an expert and previous emotion information of the user. The Al model may be stored in the electronic device <NUM> or in a server (e.g., a cloud server) connected to the electronic device <NUM>. The Al model will be described in detail below with reference to <FIG>.

According to an embodiment, the electronic device <NUM> may determine a progress stage of a user event by comparing a current time to a scheduled time of the user event. Herein, the current time may include a time when an utterance is detected from a user, but is not limited thereto. For example, the electronic device <NUM> may determine whether the current time is a time before the user event occurs or a time after the user event is finished. The electronic device <NUM> may obtain emotion information of the user related to the user event, considering the progress stage of the user event. For example, when the user event is participation in a marathon race, before the user event occurs, the electronic device <NUM> may determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>'. On the other hand, after the user event (e.g., participation in a marathon race) is finished, the electronic device <NUM> may determine the emotion information of the user as 'happiness weight: <NUM>, joy weight: <NUM>'.

According to an embodiment, the electronic device <NUM> may obtain the emotion information of the user, considering additional information related to the user event. For example, the electronic device <NUM> may subdivide the emotion information of the user, considering at least one of a person related to the user event, a scheduled time of the user event, and a place related to the user event. In this case, the electronic device <NUM> may obtain the additional information related to the user event by directly querying the user about the additional information or by parsing an email, a message, or the like. The operation, performed by the electronic device <NUM>, of subdividing the emotion information of the user, considering the additional information related to the user event will be described in detail below with reference to <FIG>.

According to an embodiment, the electronic device <NUM> may obtain default emotion information related to the user event from a table in which events are mapped to emotion information. The default emotion information may be previous emotion information of the user related to the user event or average emotion information of ordinary people for the user event. The table in which events are mapped to emotion information may be stored in the event-emotion DB connected to the Al model, but is not limited thereto.

According to an embodiment, the electronic device <NUM> may obtain actual emotion information of the user related to the user event by obtaining biometric information of the user (e.g., heart rate information, pulse information, breathing information, eye twitching information, facial expression change information, sweating information, body temperature change information, or voice input), and modifying the default emotion information related to the user event, based on the biometric information of the user. For example, the electronic device <NUM> may obtain the default emotion information related to at least one event from the table in which events are mapped to emotion information, identify an emotional state of the user for the at least one event, based on information obtained by analyzing voice data of an utterance of the user, and learn the emotion information of the user for the at least one event by modifying the default emotion information, based on the emotional state of the user. The modifying of the default emotion information may mean adjusting a weight corresponding to each of emotion elements included in the default emotion information. For example, when the default emotion information is 'nervousness weight: <NUM>, joy weight: <NUM>', the electronic device <NUM> may increase the nervousness weight from '<NUM>' to '<NUM>' and reduce the joy weight from '<NUM>' to '<NUM>', based on the biometric information of the user.

According to an embodiment, the electronic device <NUM> may obtain the biometric information of the user by using a sensor included therein, or obtain the biometric information of the user from an external device (e.g., an external wearable device). For example, the electronic device <NUM> may obtain a face image of the user by using at least one camera (e.g., a general image sensor, an infrared sensor, or a depth sensor), and detect a facial expression change or eye twitching of the user by analyzing the face image of the user. In addition, the electronic device <NUM> may detect a change in body temperature of the user by using the infrared sensor. In this case, the electronic device <NUM> may predict a current emotional state of the user, based on the facial expression change or the eye twitching of the user. Furthermore, the electronic device <NUM> may receive voice data through a microphone from the user, and predict an actual emotional state of the user by analyzing the voice data. The operation, performed by the electronic device <NUM>, of receiving the biometric information of the user from the external device will be described in detail below with reference to <FIG>.

In operation S220, the electronic device <NUM> may determine an interaction type corresponding to the user event, based on the emotion information of the user.

Herein, the interaction type refers to an interaction style that is taken to provide a response to the user, and a range of additional information to be provided in addition to information related to the user event, a tone for providing interaction, etc. may differ depending on the interaction type. For example, the interaction type may include an encourage type (e.g., change topics weight: <NUM>, cheer-up weight: <NUM>), an empathize type (e.g., empathize weight: <NUM>, cheer-up weight: <NUM>)), a console type (e.g., console weight: <NUM>, empathize weight: <NUM>), or a cheer-up type (e.g., cheer-up weight: <NUM>, provide additional information weight: <NUM>), but is not limited thereto.

According to an embodiment, when the emotion information of the user for a first event is 'nervousness weight: <NUM>' and 'joy weight: <NUM>', the electronic device <NUM> may determine the interaction type corresponding to the first event as the 'encourage type'. When the emotion information of the user for a second event is 'joy weight: <NUM>' and 'nervousness weight: <NUM>', the electronic device <NUM> may determine the interaction type corresponding to the second event as the 'empathize type'.

According to an embodiment, the electronic device <NUM> is adapted to determine the interaction type corresponding to the user event by using the Al model. The electronic device <NUM> is adapted to determine the interaction type mapped to the emotion information, based on an interaction method DB generated (or refined) by the Al model. The interaction method DB may store a table in which emotion information is mapped to interaction types, but is not limited thereto.

According to an embodiment, the electronic device <NUM> may determine the interaction type by reflecting preferences of the user. For example, the electronic device <NUM> may determine information indicating whether to use a simple image (e.g., an emoticon) to provide interaction, an interaction method (e.g., an interaction method using text, an interaction method using voice, or an interaction method using a notification window), or a voice color, by reflecting preferences of the user.

According to an embodiment, the electronic device <NUM> may determine a tone for providing interaction on an utterance of the user, considering the emotion information of the user. In addition, the electronic device <NUM> may determine a color for providing interaction on an utterance of the user, considering the emotion information of the user. The operation, performed by the electronic device <NUM>, of determining the tone or the color will be described in detail below with reference to <FIG> and <FIG>.

In operation S230, the electronic device <NUM> may provide a response to an utterance of the user related to the user event, based on the determined interaction type. Herein, the providing of the response to the utterance of the user may include proactive interaction for inducing an utterance of the user (e.g., providing a notification related to the user event).

According to an embodiment, the electronic device <NUM> may provide the response to the utterance of the user related to the user event by using the Al model. For example, the electronic device <NUM> may extract a response corresponding to the utterance of the user from a response DB generated (or refined) by the Al model. The response DB may store a table in which interaction types are mapped to phrases inputable from the user, and response phrases, but is not limited thereto.

A case in which a first phrase is input from the user will now be described as an example. The electronic device <NUM> may select a first response phrase when the determined interaction type is the encourage type, or select a second response phrase when the determined interaction type is the empathize type.

According to an embodiment, for different interaction types, the same response phrase may be selected with different pieces of additional information. The case in which the first phrase is input from the user will now be described as an example. When the determined interaction type is the encourage type, the electronic device <NUM> may select the first response phrase and select first text as the additional information. Otherwise, when the determined interaction type is the empathize type, the electronic device <NUM> may select the first response phrase and select second text as the additional information.

According to an embodiment, the electronic device <NUM> may provide the response in the form of text on a chat window, or in the form of voice. The electronic device <NUM> may provide the response in the form of text on a notification window (e.g., a pop-up window), but is not limited thereto.

According to an embodiment, the electronic device <NUM> may select a color corresponding to the emotion information of the user and display the response to the utterance of the user related to the user event by using the selected color. For example, when a first color is selected based on first emotion information of the user for the first event, the electronic device <NUM> may provide the response on a notification window displayed in the first color. Otherwise, when a second color is selected based on second emotion information of the user for the second event, the electronic device <NUM> may provide the response on a notification window displayed in the second color.

According to an embodiment, the electronic device <NUM> may provide the response by applying an expression style preferred by the user. For example, when 'You can do it' and 'Go for it' are providable as the response to the first phrase input from the user, the electronic device <NUM> may select 'You can do it' corresponding to the expression style preferred by the user.

The electronic device <NUM> may select a tone preferred by the user, voice preferred by the user, a notification method preferred by the user, or a color preferred by the user, but is not limited thereto.

According to an embodiment, the electronic device <NUM> may consider a situation of the user for proactive interaction. For example, the electronic device <NUM> may determine a notification providing method related to the user event, considering the situation of the user. The electronic device <NUM> may provide a notification related to the user event, based on the determined notification providing method. For example, the electronic device <NUM> may merely display a notification (or a reminder) related to an evening event on a screen without sound when the user is giving a presentation in the afternoon, and then provide a notification (or a reminder) again in the form of vibration or sound when a current time is close to a scheduled time of the evening event.

According to an embodiment, the electronic device <NUM> may provide a diary interface including the emotion information of the user related to the user event. For example, the electronic device <NUM> may provide a simple image or graph indicating the emotion information of the user per progress stage of the user event. The operation, performed by the electronic device <NUM>, of providing the diary interface will be described in detail below with reference to <FIG> and <FIG>.

Although a case in which the electronic device <NUM> provides a response to a voice input of the user is described above as an example in relation the <FIG>, a user input is not limited thereto. The electronic device <NUM> may also provide a response to a text input of the user, based on the emotion information of the user. An Al model will now be described in detail with reference to <FIG>.

<FIG> is a schematic diagram for describing an operation of generating an event-emotion-based response model (or an Al model) through deep learning, according to an embodiment.

Referring to <FIG>, according to an embodiment, an Al processor included in the electronic device <NUM> or a server may generate an event-emotion-based response model <NUM> for determining an interaction method, by training an artificial neural network. In the following description, for convenience of explanation, the event-emotion-based response model <NUM> may also be called an Al model or a response model. The 'training' of the artificial neural network may mean constructing a mathematical model in which connection of neurons in the artificial neural network may make an optimal decision, by appropriately changing weights based on data.

According to an embodiment, the Al processor may obtain data created by an expert, published data, or the like as training data <NUM>, and generate the event-emotion-based response model <NUM> by using the training data <NUM>. The data created by the expert may include emotion information analyzed or extracted by the expert from phrases predicted to be uttered by a user. The published data may include emotion information analyzed or extracted from text collected from the web or an SNS server. For example, the published data may include emotion information of happiness analyzed/extracted from the phrase 'Skipped school today skipping school tomorrow haha'. The data created by the expert or the published data may be converted to a preset format and thus be used as the training data <NUM> for generating the event-emotion-based response model <NUM>.

According to an embodiment, the Al processor may obtain emotion information (e.g., feedback information) input from the user, emotion information obtained from biometric information of the user, or the like as personalized training data <NUM>, and generate the event-emotion-based response model <NUM> by using the personalized training data <NUM>. The emotion information (e.g., the feedback information) input from the user or the emotion information obtained from the biometric information of the user may be converted to a preset format and thus be used as the personalized training data <NUM> for generating the event-emotion-based response model <NUM>.

According to an embodiment, the event-emotion-based response model <NUM> may be trained using the training data <NUM> to generate an event-emotion DB <NUM>, an interaction method DB <NUM>, and a response DB <NUM>. According to an embodiment, the event-emotion-based response model <NUM> may refine the event-emotion DB <NUM>, the interaction method DB <NUM>, and the response DB <NUM> by adding or refining the training data <NUM>.

According to an embodiment, the event-emotion DB <NUM> is a DB that defines correlations between events and emotions, and may include a table in which events are mapped to emotion information. For example, the event-emotion DB <NUM> may include information such as "first event: nervousness (<NUM>), joy (<NUM>)" and "second event: fear (<NUM>), nervousness (<NUM>)".

According to an embodiment, the event-emotion DB <NUM> may store a table in which emotion information is subdivided based on times, people, and places related to the events. For example, the event-emotion-based response model <NUM> may subdivide emotion information of the user, considering the times, the people, and the places related to the events. When detailed information about the times, the people, and the places related to the events is not stored in an event DB, the event-emotion-based response model <NUM> may obtain the information by querying the user about the times, the people, and the places related to the events. The table in which emotion information is subdivided will be described in detail below with reference to <FIG>.

According to an embodiment, the interaction method DB <NUM> is a DB that defines correlations between emotions and interaction methods, and may store a table in which emotion information is mapped to interaction types. For example, the interaction method DB <NUM> may include information such as "first emotion information (nervousness (<NUM>), joy (<NUM>))-first interaction type (change topics (<NUM>), encourage (<NUM>))" and "second emotion information (relief)-second interaction type (empathize (<NUM>))".

According to an embodiment, the response DB <NUM> is a DB that defines correlations between interaction methods (or interaction types) and response phrases, and may store a table in which interaction types are mapped to response phrases. For example, the response DB <NUM> may include information such as "first interaction type (e.g., encourage type: change topics (<NUM>), encourage (<NUM>))-response phrase (Go for it (<NUM>)/You can do it (<NUM>))".

According to an embodiment, the event-emotion-based response model <NUM> may obtain the personalized training data <NUM> and personalize the event-emotion DB <NUM>, the interaction method DB <NUM>, and the response DB <NUM> by using the personalized training data <NUM>.

According to an embodiment, the electronic device <NUM> may obtain feedback on a user event from the user after the user event is finished. The electronic device <NUM> may input the feedback on the user event as the personalized training data <NUM> to the event-emotion-based response model <NUM>. In this case, the event-emotion-based response model <NUM> may refine the table in which events are mapped to emotion information, which is included in the event-emotion DB <NUM>, based on the feedback on the user event. For example, when the electronic device <NUM> has determined the emotion information of the user for a first event as 'nervousness (<NUM>), joy (<NUM>)' but the feedback input from the user indicates that the emotion information of the user is 'nervousness (<NUM>), joy (<NUM>)', the electronic device <NUM> may modify the emotion information of the user for the first event from 'nervousness (<NUM>), joy (<NUM>)' to 'nervousness (<NUM>), joy (<NUM>)' in the event-emotion DB <NUM>.

According to an embodiment, the electronic device <NUM> may obtain feedback on the emotion information of the user before the user event occurs. In this case, the event-emotion-based response model <NUM> of the electronic device <NUM> may refine the emotion information of the user per progress stage of the user event, which is included in the event-emotion DB <NUM>.

According to an embodiment, the electronic device <NUM> may obtain biometric information of the user and personalize the event-emotion DB <NUM> by using actual emotion information of the user, which is predicted based on the biometric information.

The electronic device <NUM> may personalize the interaction method DB <NUM> and the response DB <NUM> by obtaining information about an interaction type preferred by the user or an expression style preferred by the user.

A method, performed by the electronic device <NUM>, of detecting a new event from voice data of an utterance of a user, and providing a response to the utterance of the user related to the new event by using the event-emotion-based response model <NUM> will now be described in detail with reference to <FIG> and <FIG>.

<FIG> is a flowchart of a method of registering a new event, based on voice data of an utterance of a user, according to an embodiment.

In operation S410, the electronic device <NUM> may extract information related to a user event from voice data of an utterance of a user.

According to an embodiment, the electronic device <NUM> may generate data as a result of analyzing the voice data, based on natural language processing. For example, the electronic device <NUM> may convert the voice data into text data, and detect the information related to the user event from the text data, based on natural language processing. For example, the electronic device <NUM> may detect a 'wedding' as a new user event by analyzing the voice data stating that 'My friend Jenna is having a wedding this Saturday', and obtain 'who gets married: Jenna, relationship: friend, wedding date: this Saturday (<NUM>/<NUM>)' as information related to the new user event (e.g., the wedding).

In operation S420, the electronic device <NUM> may register the user event as a new event by using the extracted information. For example, the electronic device <NUM> may register the wedding as a new event by storing, in an event DB connected to an Al model, the information (e.g., 'who gets married: Jenna, relationship: friend, wedding date: this Saturday (<NUM>/<NUM>)') related to the user event (e.g., the wedding) detected from the voice data.

According to an embodiment, the event DB may be connected to, for example, a scheduling application, a calendar application, a contact list application, or an SNS application, but is not limited thereto. The event DB is a DB that defines user events meaningful to the user, and may include information about events manually input by the user, and information about events automatically registered by a system.

In operation S430, the electronic device <NUM> may extract emotion information of the user from the voice data of the utterance of the user. For example, the electronic device <NUM> may convert the voice data into text data, and detect information related to an emotional state from the text data, based on natural language processing. For example, the electronic device <NUM> may detect 'nervousness' as the emotion information of the user by analyzing voice data stating that 'I am nervous to sing at my friend Jenna's wedding this Saturday'.

In operation S440, when the emotion information of the user is not extracted from the voice data, the electronic device <NUM> may obtain the emotion information of the user related to the user event from an event-emotion DB. Operation S440 may correspond to operation S220 of <FIG>, and thus a detailed description thereof is not provided herein.

In operation S450, the electronic device <NUM> may determine an interaction type corresponding to the user event, based on the emotion information of the user.

When the emotion information of the user is extracted from the voice data of the utterance of the user, the electronic device <NUM> may determine the interaction type corresponding to the user event, based on the extracted emotion information. When the emotion information of the user is not extracted from the voice data of the utterance of the user, the electronic device <NUM> may determine the interaction type corresponding to the user event, based on the emotion information of the user, which is obtained from the event-emotion DB.

According to an embodiment, the electronic device <NUM> is adapted to determine an interaction type mapped to the emotion information, based on an interaction method DB generated (or refined) by the Al model. Operation S450 corresponds to operation S220 of <FIG>, and thus a detailed description thereof is not provided herein.

In operation S460, the electronic device <NUM> may provide a response to an utterance of the user related to the user event, based on the determined interaction type.

According to an embodiment, the electronic device <NUM> is adapted to extract a response mapped to the interaction type, from a response DB generated (or refined) by the Al model, and provide the extracted response through an interactive interface to the user. Operation S460 corresponds to operation S230 of <FIG>, and thus a detailed description thereof is not provided herein.

<FIG> is a schematic diagram for describing an operation, performed by the electronic device <NUM>, of responding to an utterance of a user, according to an embodiment.

In operation S510, when a user utters, the electronic device <NUM> may obtain voice data of the user through a microphone, and understand language of the user by analyzing the voice data. That is, the electronic device <NUM> may extract a user intent, extract information related to a user event, and extract emotion information for the user event from the voice data (S510). For example, when the user utters that 'I have to give an important presentation in the afternoon', the electronic device <NUM> may extract 'register a new event' as the user intent, and extract 'presentation, afternoon' as information related to the new event. In this case, any emotion information may not be extracted from the utterance of the user.

In operation S520, when the language of the user is understood, the electronic device <NUM> may determine interaction. For example, the electronic device <NUM> may register 'presentation, afternoon' as a new event in an event DB <NUM> according to the user intent. In operation S521, the electronic device <NUM> may extract first emotion information (e.g., 'nervousness weight: <NUM>, joy weight: <NUM>') corresponding to the presentation from the event-emotion DB <NUM>. In operation S522, the electronic device <NUM> may extract a first interaction method (e.g., first interaction type: other information (<NUM>), cheer-up (<NUM>)) corresponding to the first emotion information (e.g., 'nervousness weight: <NUM>, joy weight: <NUM>') from the interaction method DB <NUM>.

In operation S530, the electronic device <NUM> may set a response. For example, the electronic device <NUM> may modify the response, based on the user event and the interaction method. In operation S531, the electronic device <NUM> may extract 'You are giving a presentation' from the response DB <NUM> as a default response to the utterance of the user, and extract 'Good luck' as an additional response. The electronic device <NUM> may output 'You are giving a presentation. Good luck!" through an interactive interface.

Because 'presentation, afternoon' is registered as the new event, the electronic device <NUM> may monitor a progress of the 'presentation'. For example, the electronic device <NUM> may obtain emotion information of the user before and after the presentation. The electronic device <NUM> may refine a personal emotion DB, based on the obtained emotion information of the user. For example, the electronic device <NUM> may personalize the event-emotion DB <NUM> by modifying default emotion information included in the event-emotion DB <NUM> (S540).

An operation, performed by the electronic device <NUM>, of responding to a user, based on emotion information of the user for a user event pre-registered in the event DB <NUM> will now be described.

<FIG> is a schematic diagram for describing an operation, performed by the electronic device <NUM>, of extracting a user event registered in the event DB <NUM>, according to an embodiment.

The electronic device <NUM> may receive first voice data <NUM> based on a first utterance of a user <NUM>. For example, the electronic device <NUM> may receive, from the user <NUM>, a query input asking 'What is my schedule today?'. In this case, the electronic device <NUM> may extract an intent of the user <NUM> (e.g., check today's schedule of the user <NUM>) by analyzing the first voice data <NUM> received from the user <NUM>.

The electronic device <NUM> may access the event DB <NUM> connected to the event-emotion-based response model <NUM>, to check today's schedule of the user <NUM> according to the intent of the user <NUM>. The electronic device <NUM> may check event information related to the user <NUM> (e.g., a presentation to the vice president) from the event DB <NUM>, and provide a first response <NUM>. For example, the electronic device <NUM> may provide the first response <NUM> stating that 'A presentation to the vice president is scheduled this afternoon'.

In this case, the electronic device <NUM> may receive second voice data <NUM> from the user <NUM> having heard the first response <NUM>. For example, the electronic device <NUM> may receive the second voice data <NUM> stating that 'Oh, I see! I have to do well. To provide a response to the second voice data <NUM>, the electronic device <NUM> may obtain, from an event-emotion DB, default emotion information (e.g., 'nervousness weight: <NUM>, joy weight: <NUM>') corresponding to a user event of 'a presentation to the vice president'. In addition, the electronic device <NUM> may extract emotion information from the second voice data <NUM>. For example, the electronic device <NUM> may extract 'afraidness' as the emotion information from the second voice data <NUM>. In this case, the electronic device <NUM> may modify the default emotion information (e.g., 'nervousness weight: <NUM>, joy weight: <NUM>'), based on the emotion information (e.g., afraidness) extracted from the second voice data <NUM>. For example, the electronic device <NUM> may modify the default emotion information (e.g., 'nervousness weight: <NUM>, joy weight: <NUM>') to 'nervousness weight: <NUM>, joy weight: <NUM>, afraidness weight: <NUM>'.

The electronic device <NUM> may determine an interaction type as an 'encourage type (provide necessary information + add encouraging phrases)', based on the modified emotion information (e.g., 'nervousness weight: <NUM>, joy weight: <NUM>, afraidness weight: <NUM>'). The electronic device <NUM> may provide a second response <NUM> in the encourage type. For example, the electronic device <NUM> may output the second response <NUM> stating that 'The presentation is at <NUM>:<NUM> PM (provide necessary information). Take your time and don't forget every material you need. (add encouraging phrases)'.

<FIG> is a flowchart of a method, performed by the electronic device <NUM>, of obtaining emotion information of a user, based on a progress stage of a user event, according to an embodiment.

In operation S710, the electronic device <NUM> may determine a progress stage of a user event by comparing a current time to a scheduled time of the user event. Herein, the progress stage of the user event may include, for example, a stage (e.g., a day, <NUM> hour, or <NUM> minutes) before an event occurs, a stage during the event, or a stage after the event is finished, but is not limited thereto.

For example, when the scheduled time of the user event is <NUM>:<NUM> PM and the current time is <NUM>:<NUM> PM, the electronic device <NUM> may determine the progress stage of the user event as '<NUM> hour before the user event occurs'.

According to an embodiment, the current time may be a time when an utterance is detected from a user. That is, the electronic device <NUM> may determine whether the utterance is detected from the user before the user event occurs or after the user event is finished.

In operation S720, the electronic device <NUM> may obtain emotion information of the user related to the user event, based on the progress stage of the user event.

Because an emotional state of the user for the user event may probably change before and after the user event, the electronic device <NUM> may obtain the emotion information of the user, considering the progress stage of the user event. For example, the electronic device <NUM> may extract a low 'nervousness weight' after the user event is finished compared to before the user event occurs.

In operation S730, the electronic device <NUM> may determine an interaction type corresponding to the user event, based on the emotion information of the user.

According to an embodiment, the electronic device <NUM> may determine an interaction type mapped to the emotion information, based on an interaction method DB generated (or refined) by an Al model. Operation S730 corresponds to operation S220 of <FIG>, and thus a detailed description thereof is not provided herein.

In operation S740, the electronic device <NUM> may provide a response to an utterance of the user related to the user event, based on the determined interaction type.

According to an embodiment, the electronic device <NUM> may extract a response mapped to the interaction type, from a response DB generated (or refined) by the Al model, and provide the extracted response through an interactive interface to the user. In this case, the response may include a proactive response (e.g., providing a notification for the user event). Operation S740 corresponds to operation S230 of <FIG>, and thus a detailed description thereof is not provided herein.

<FIG> is a schematic diagram for describing an operation, performed by the electronic device <NUM>, of providing a notification or a response, based on a progress stage of a user event, according to an embodiment.

To check any new event to be registered in an event DB, the electronic device <NUM> may start a conversation with a user <NUM> every morning <NUM> by stating that "Good morning. Do you have any schedule today?". In this case, when a first utterance stating that 'I have to give a presentation to the vice president at <NUM>:<NUM> PM today' is detected from the user <NUM>, the electronic device <NUM> may register '<NUM>:<NUM> PM, a presentation to the vice president' as a new event by analyzing voice data of the user <NUM>. The electronic device <NUM> may obtain emotion information (e.g., joy weight: <NUM>, nervousness weight: <NUM>) corresponding to 'a presentation to the vice president' by using an Al model, and determine an interaction type for the first utterance, based on the emotion information (e.g., joy weight: <NUM>, nervousness weight: <NUM>). For example, the electronic device <NUM> may determine a first encourage type (e.g., cheer-up weight: <NUM>, change topics weight: <NUM>) as the interaction type. The electronic device <NUM> may output a response to the first utterance (e.g., I see. I will support you. Don't worry. ), based on the first encourage type.

Before providing a notification related to a user event, to the user <NUM>, the electronic device <NUM> may compare a current time (e.g., <NUM>:<NUM> PM) to a scheduled time (e.g., <NUM>:<NUM> PM) of the user event (e.g., a presentation to the vice president). The electronic device <NUM> may determine that the current time corresponds to a time <NUM> before the user event (e.g., a presentation to the vice president) occurs, based on the comparison result.

The electronic device <NUM> may obtain emotion information (e.g., joy weight: <NUM>, afraidness weight: <NUM>, nervousness weight: <NUM>) of the user <NUM>, considering that the current time corresponds to the time <NUM> before the user event occurs. The electronic device <NUM> may determine a second encourage type (e.g., cheer-up weight: <NUM>, change topics weight: <NUM>) as an interaction type corresponding to the emotion information (e.g., joy weight: <NUM>, afraidness weight: <NUM>, nervousness weight: <NUM>) of the user <NUM>. The electronic device <NUM> may provide proactive interaction, based on the second encourage type. For example, the electronic device <NUM> may provide a notification stating that 'You must be nervous about the presentation. Don't forget every material you need!". When the user <NUM> utters that "Thank you. I'll do my best. ' in response to the notification of the electronic device <NUM>, the electronic device <NUM> may output a response stating that 'OK. Good luck!'.

At a time <NUM> after the user event is finished, the electronic device <NUM> may query the user <NUM> about emotion information related to the user event. For example, the electronic device <NUM> may output a query asking 'How was the presentation today?'. When the user <NUM> answers that 'Well, I have prepared a lot but I got told off a bit' in response to the query, the electronic device <NUM> may obtain emotion information of the user <NUM>, considering that the current time corresponds to the time <NUM> after the user event is finished. In this case, the electronic device <NUM> may extract emotion information from voice data of the user <NUM>. For example, the electronic device <NUM> may obtain 'regret weight: <NUM>, relief weight: <NUM>, joy weight: <NUM>' as the emotion information for the presentation.

The electronic device <NUM> may determine an empathize type (e.g., empathize weight: <NUM>, provide other information weight: <NUM>) as an interaction type corresponding to the emotion information (e.g., regret weight: <NUM>, relief weight: <NUM>, joy weight: <NUM>) of the user <NUM>. The electronic device <NUM> may provide a first response stating that 'You've worked hard on it. It's too bad', based on the determined interaction type (e.g., the empathize type). When the user <NUM> utters that 'Yeah. I should refresh myself!' in response to the first response, the electronic device <NUM> may output a second response stating that 'You can do better next time'.

The electronic device <NUM> may record the emotion information of the user <NUM> after the user event is finished, and re-use the emotion information later. For example, when a similar event occurs to the user <NUM> later, the electronic device <NUM> may remind the user <NUM> about information related to a previous user event, and induce the user <NUM> to recall a good memory.

<FIG> is a schematic diagram for describing an operation, performed by the electronic device <NUM>, of obtaining emotion information of a user, based on a person, a place, or a time related to a user event, according to an embodiment.

According to an embodiment, in the event-emotion DB <NUM> connected to the Al model <NUM>, emotion information based on an event <NUM> may be subdivided based on a person, a place, or a time related to the event <NUM>. In this case, the electronic device <NUM> may obtain detailed emotion information of a user, based on the person, the place, or the time related to the event <NUM>.

For example, when the event <NUM> is a presentation <NUM>, the electronic device <NUM> may determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>' by default. However, when information about a person to whom the presentation <NUM> is given is obtained, the electronic device <NUM> may determine the emotion information of the user, considering the information about the person to whom the presentation <NUM> is given. For example, when the person to whom the presentation <NUM> is given is the chief executive officer (CEO), the electronic device <NUM> may determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>'. Otherwise, when the person to whom the presentation <NUM> is given is a friend, the electronic device <NUM> may determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>'.

When information about a place where the presentation <NUM> is given is obtained, the electronic device <NUM> may determine the emotion information of the user, considering the place. For example, when the presentation <NUM> is given at a conference room, the electronic device <NUM> may determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>, afraidness weight: <NUM>'. Otherwise, when the presentation <NUM> is given at an outing, the electronic device <NUM> may determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>'.

According to another embodiment, when the event <NUM> is a friend's wedding <NUM>, the electronic device <NUM> may determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>' by default. However, when time information of the friend's wedding <NUM> is obtained, the electronic device <NUM> may determine the emotion information of the user, considering the time information of the friend's wedding <NUM>. For example, when the friend's wedding <NUM> is in the morning, the electronic device <NUM> may determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>, laziness weight: <NUM>'. Otherwise, the electronic device <NUM> may determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>' when the friend's wedding <NUM> is in the afternoon, or determine the emotion information of the user as 'nervousness weight: <NUM>, joy weight: <NUM>' when the friend's wedding <NUM> is in the evening.

When information about a place where the friend's wedding <NUM> is held is obtained, the electronic device <NUM> may determine the emotion information of the user, considering the place. For example, when the friend's wedding <NUM> is held outdoors, the electronic device <NUM> may determine the emotion information of the user as 'joy weight: <NUM>'.

According to an embodiment, the electronic device <NUM> may learn emotion information of a user for at least one event included in notification information, based on feedback information of the user on the at least one event. The feedback information of the user is information on an emotional state of the user for the at least one event, and may be obtained based on an input of the user. For example, the electronic device <NUM> may provide notification information for at least one event through a virtual assistant interface to a user, and obtain feedback information of the user on the at least one event included in the notification information. The electronic device <NUM> may identify an emotional state of the user for the at least one event by analyzing the feedback information of the user, and learn emotion information of the user for the at least one event, based on the emotional state of the user. The operation, performed by the electronic device <NUM>, of learning the emotion information of the user for the at least one event, based on the feedback information of the user will now be described with reference to <FIG> and <FIG>.

<FIG> and <FIG> are schematic diagrams for describing an operation of personalizing the event-emotion DB <NUM>, according to an embodiment.

Referring to <FIG>, before a user event occurs, the electronic device <NUM> may provide a notification related to the user event (e.g., You have <NUM> hour before the presentation) through a notification window <NUM> to a user.

In this case, the electronic device <NUM> may provide an emotion candidate list and obtain actual emotion information before the user event from the user. For example, the electronic device <NUM> may provide the emotion candidate list including pleasure, nervousness, sadness, thrill, neutral, regret, anger, annoyance, etc., and receive a user input for selecting 'nervousness' in the emotion candidate list.

The electronic device <NUM> may personalize default emotion information stored in the event-emotion DB <NUM>, based on the actual emotion information input by the user. For example, emotion information <NUM> of the user before the presentation, which is stored in the event-emotion DB <NUM>, may be 'joy weight: <NUM>, nervousness weight <NUM>'. However, because the user selects 'nervousness' in the emotion candidate list displayed on the notification window <NUM>, the electronic device <NUM> may personalize the emotion information of the user for the presentation by increasing the nervousness weight included in the default emotion information. For example, personalized emotion information <NUM> for the presentation may be 'joy weight: <NUM>, nervousness weight: <NUM>'.

Referring to <FIG>, the electronic device <NUM> may obtain feedback of the user on the user event after the user event is finished. For example, the electronic device <NUM> may provide a query window <NUM> for checking an emotion for the user event. According to an embodiment, the electronic device <NUM> may display, on the query window <NUM>, an emotion candidate list including a plurality of images indicating emotional states. In this case, the electronic device <NUM> may receive, from the user, an input for selecting an image <NUM> indicating joy in the emotion candidate list.

The electronic device <NUM> may personalize default emotion information for the presentation, which is stored in the event-emotion DB <NUM>, based on actual emotion information input by the user. For example, emotion information <NUM> of the user for the presentation, which is stored in the event-emotion DB <NUM>, may be 'joy weight: <NUM>, nervousness weight <NUM>'. However, because the user has selected the image <NUM> indicating joy in the emotion candidate list displayed on the query window <NUM>, the electronic device <NUM> may personalize the emotion information of the user for the presentation by increasing the joy weight included in the default emotion information. For example, personalized emotion information <NUM> for the presentation may be 'joy weight: <NUM>, nervousness weight: <NUM>'.

An operation, performed by the electronic device <NUM>, of monitoring biometric information of a user by using an external device, and modifying default emotion information, based on the biometric information of the user will now be described.

<FIG> is a schematic diagram for describing an operation, performed by the electronic device <NUM>, of modifying default emotion information, based on biometric information, according to an embodiment.

In operation S1210, the electronic device <NUM> may obtain default emotion information <NUM> related to a user event from the event-emotion DB <NUM>. The default emotion information <NUM> may be average emotion information of ordinary people for the user event, or previous emotion information of the user for the user event. For example, the default emotion information <NUM> for a presentation, which is stored in the event-emotion DB <NUM>, may be 'joy weight: <NUM>, nervousness weight: <NUM>'.

In operation S1220, to check actual emotion information of the user, the electronic device <NUM> may obtain biometric information of the user from an external device <NUM>. For example, the electronic device <NUM> may receive heart rate information or body temperature change information of the user from a smart watch worn on a wrist of the user. In addition, the electronic device <NUM> may receive voice data of the user, which is recorded by the smart watch. The electronic device <NUM> may receive the biometric information of the user from the external device <NUM> in a predetermined cycle or when a specific event occurs, but is not limited thereto.

According to an embodiment, the electronic device <NUM> may obtain actual emotion information of the user, based on the biometric information of the user. For example, the electronic device <NUM> may determine that nervousness of the user is increased, when a heart rate of the user is increased and a body temperature of the user is also increased, or determine that nervousness of the user is reduced, when the heart rate of the user is stabilized. In addition, when a result of analyzing the voice data of the user indicates that voice is shaking or words used when nervous (e.g., well. ) are used a lot, the electronic device <NUM> may determine that nervousness of the user is increased. Otherwise, when the result of analyzing the voice data of the user indicates that the user cheers or laughs a lot, the electronic device <NUM> may determine that joy of the user is increased.

According to an embodiment, the electronic device <NUM> may obtain the biometric information of the user before, during, and after a user event registered in the event DB <NUM>, but is not limited thereto. In this case, the electronic device <NUM> may monitor changes in the emotion of the user as the user event progresses.

In operation S1230, the electronic device <NUM> may modify the default emotion information <NUM> for the presentation, which is stored in the event-emotion DB <NUM>, based on the biometric information of the user. For example, when the heart rate of the user is stable, the electronic device <NUM> may reduce the nervousness weight included in the default emotion information <NUM>, from '<NUM>' to '<NUM>'. Therefore, modified emotion information <NUM> may be 'joy weight: <NUM>, nervousness weight: <NUM>'.

<FIG> is a flowchart of a method of determining a tone, based on emotion information of a user, according to an embodiment.

In operation S1310, the electronic device <NUM> may obtain emotion information of a user related to a user event.

According to an embodiment, the electronic device <NUM> may obtain the emotion information of the user related to the user event at a time when an utterance of the user related to the user event is detected. According to another embodiment, the electronic device <NUM> may obtain the emotion information of the user for the user event to proactively provide interaction before an utterance of the user related to the user event.

According to an embodiment, the electronic device <NUM> may obtain the emotion information of the user related to the user event by using an Al model. For example, the electronic device <NUM> may obtain the emotion information of the user corresponding to a specific user event from the event-emotion DB <NUM> generated (or refined) by the Al model. The event-emotion DB <NUM> may store average emotion information of ordinary people for a specific event, or emotion information personalized per user for a specific event, but is not limited thereto.

Operation S1310 may correspond to operation S210 of <FIG>, and thus a detailed description thereof is not provided herein.

In operation S1320, the electronic device <NUM> may determine a tone for providing a response to an utterance of the user, based on the emotion information of the user related to the user event. For example, the electronic device <NUM> may determine a speech rate, a pitch average, a pitch range, an intensity, voice quality, pitch changes, and articulation, based on the emotion information of the user related to the user event, but is not limited thereto.

For example, referring to an emotion-tone table <NUM> shown in <FIG>, when the emotion information of the user indicates fear, the electronic device <NUM> may determine the speech rate to be 'much faster', determine the pitch average to be 'very much higher', determine the pitch range to be 'much wider', determine the intensity to be 'normal', determine the voice quality to be 'irregular voicing', determine the pitch changes to be 'normal', and determine the articulation to be 'precise'.

Otherwise, when the emotion information of the user indicates anger, the electronic device <NUM> may determine the speech rate to be 'slightly faster', determine the pitch average to be 'very much higher', determine the pitch range to be 'much wider', determine the intensity to be 'higher', determine the voice quality to be 'breathy chest tone', determine the pitch changes to be "abrupt on stressed syllable', and determine the articulation to be 'tense'.

In operation S1330, the electronic device <NUM> may provide a response to an utterance of the user related to the user event in the determined tone. Therefore, according to an embodiment, the electronic device <NUM> may efficiently provide emotional interaction to the user by varying the tone of voice for providing the response, based on the emotion information of the user for the user event.

<FIG> is a schematic diagram for describing an operation, performed by the electronic device <NUM>, of providing a notification by using a color corresponding to emotion information of a user, according to an embodiment.

According to an embodiment, the electronic device <NUM> may determine a color corresponding to emotion information of a user. For example, referring to an emotion-color table <NUM>, the electronic device <NUM> may determine red for emotions such as excitement, energy, passion, action, and desire, or determine orange for emotions such as optimistic, uplifting, rejuvenating, friendliness, and fun, but is not limited thereto.

According to an embodiment, the electronic device <NUM> may adjust brightness or saturation. For example, a low brightness or saturation level may be determined for an emotion close to the center of the emotion-color table <NUM>.

According to an embodiment, the electronic device <NUM> may provide a notification by using the determined color. For example, when a user event is a 'trip' and an emotional state of the user indicates 'fun', the electronic device <NUM> may display, on an orange notification window <NUM>, text indicating <NUM> day before the trip (e.g., You have <NUM> day before the trip). Otherwise, when the user event is an 'opening' and the emotional state of the user indicates 'hope', the electronic device <NUM> may display, on a green notification window <NUM>, text indicating <NUM> day before the opening (e.g., You have <NUM> day before the opening).

<FIG> and <FIG> are schematic diagrams for describing an operation, performed by the electronic device <NUM>, of providing a diary interface, according to an embodiment.

Referring to a first screen <NUM> of <FIG>, the electronic device <NUM> may display a list of user events on a diary interface. In this case, the electronic device <NUM> may display representative emotions of a user for the user events in front of event names. For example, when a representative emotion of the user for son's birthday is 'pleasure', the electronic device <NUM> may display 'pleasure' next to the son's birthday. In addition, when a representative emotion of the user for a presentation is 'nervousness', the electronic device <NUM> may display 'nervousness' next to the presentation. According to an embodiment, the representative emotions may be displayed in the form of emoticons.

According to an embodiment, the electronic device <NUM> may sort the list of the user events in chronological order or by emotion, but is not limited thereto.

Referring to a second screen <NUM> of <FIG>, when the user selects the son's birthday in the list of the user events, the electronic device <NUM> may specifically provide emotion information related to the son's birthday to the user. For example, the electronic device <NUM> may provide changes in the emotion of the user from before to after the user event, in the form of a simple diary. In this case, the electronic device <NUM> may display the changes in the emotion of the user in the form of simple emoticons. In addition, when the electronic device <NUM> includes photographic images related to the user event, the electronic device <NUM> may add the photographic images to the diary viewed to the user. The electronic device <NUM> may receive, from the user, an additional comment related to the user event. For example, the user may add an interesting episode or a photographic image related to the son's birthday.

Referring to <FIG>, the electronic device <NUM> may provide an emotion graph <NUM> corresponding to each user event on a diary interface. An emotion graph may be a graph showing changed in the emotion of the user from before to after an event. According to an embodiment, the emotion graph may be a <NUM>-dimensional graph or a <NUM>-dimensional graph, but is not limited thereto.

<FIG> and <FIG> are block diagrams of the electronic device <NUM> according to an embodiment.

As illustrated in <FIG>, the electronic device <NUM> according to an embodiment may include an outputter <NUM>, a processor <NUM>, and a memory <NUM>. However, not all of the illustrated elements are essential elements. The electronic device <NUM> may include a larger or smaller number of elements compared to the illustrated elements.

For example, as illustrated in <FIG>, the electronic device <NUM> according to an embodiment may include a sensor <NUM>, a communicator <NUM>, an audio/video (A/V) inputter <NUM>, and a user inputter <NUM> in addition to the outputter <NUM>, the processor <NUM>, and the memory <NUM>.

The above-mentioned elements will now be described one by one.

The outputter <NUM> is used to output an audio signal, a video signal, or a vibration signal, and may include, for example, a display <NUM>, a sound outputter <NUM>, and a vibration motor <NUM>.

The sound outputter <NUM> outputs audio data received from the communicator <NUM> or stored in the memory <NUM>. In addition, the sound outputter <NUM> outputs a sound signal related to a function performed by the electronic device <NUM> (e.g., incoming call sound, message received sound, or notification sound). The sound outputter <NUM> may include, for example, a speaker or a buzzer. According to an embodiment, the sound outputter <NUM> may provide a voice response to an utterance of a user related to a user event.

The vibration motor <NUM> may output a vibration signal. For example, the vibration motor <NUM> may output a vibration signal corresponding to output of audio or video data (e.g., incoming call sound or message received sound). The vibration motor <NUM> may output a vibration signal when touch on a touchscreen is input.

The outputter <NUM> may provide an interactive interface. The interactive interface may be provided in the form of a message or chat window or in the form of a voice input/output interface, but is not limited thereto.

The outputter <NUM> may provide a diary interface including emotion information of the user related to user events. For example, the diary interface may list previous emotional states and episodes related to every event registered in a reminder application or a calendar application. In this case, the previous emotional states may be displayed as emoticons or a simple graph.

The sensor <NUM> may include at least one of a magnetic sensor <NUM>, an acceleration sensor <NUM>, a tilt sensor <NUM>, an infrared sensor <NUM>, a gyroscope sensor <NUM>, a location sensor (e.g., a global positioning system (GPS) sensor) <NUM>, a temperature/humidity sensor <NUM>, a proximity sensor <NUM>, and an illuminance sensor <NUM>, but is not limited thereto. Functions of the sensors may be intuitively inferred from their names by one of ordinary skill in the art, and thus a detailed description thereof is not provided herein.

The processor <NUM> generally controls overall operations of the electronic device <NUM>. For example, the processor <NUM> may control all of the outputter <NUM>, the sensor <NUM>, the communicator <NUM>, the A/V inputter <NUM>, the user inputter <NUM>, and the memory <NUM> by executing programs stored in the memory <NUM>.

According to an embodiment, the processor <NUM> may include an Al processor for generating a learning network model, but is not limited thereto. According to an embodiment, the Al processor may be implemented as a chip separate from the processor <NUM>.

The processor <NUM> may query the user about schedule information by using a virtual assistant interface, and receive a response including information about at least one event, from the user. The processor <NUM> may identify an emotional state of the user for the at least one event by analyzing the response to the query, and learn emotion information of the user for the at least one event, based on the emotional state of the user. The processor <NUM> may determine an interaction type for the at least one event, based on the emotion information of the user, and provide notification information for the at least one event though the virtual assistant interface, based on the interaction type.

The processor <NUM> may obtain the emotion information of the user related to the user event registered in the electronic device <NUM>. According to an embodiment, the electronic device <NUM> may obtain the emotion information of the user related to the user event from an Al model trained based on at least one of emotion information analyzed by an expert and previous emotion information of the user. According to an embodiment, the processor <NUM> may determine a progress stage of the user event by comparing a current time to a scheduled time of the user event, and obtain the emotion information of the user related to the user event, based on the progress stage of the user event.

The processor <NUM> may extract information related to the user event from voice data of an utterance of the user, and register the user event as a new event in the electronic device <NUM> by using the extracted information. The processor <NUM> may extract the emotion information of the user from the voice data of the user.

The processor <NUM> may obtain default emotion information related to the user event from a table in which events are mapped to emotion information, obtain biometric information of the user by using the sensor <NUM> or an external device, and obtain the emotion information of the user related to the user event by modifying the default emotion information, based on the biometric information of the user. The processor <NUM> may identify the emotional state of the user for the at least one event, based on information obtained by analyzing the voice data of the utterance of the user, and learn the emotion information of the user for the at least one event by modifying the default emotion information, based on the emotional state of the user.

The processor <NUM> may obtain feedback on the user event through the user inputter <NUM> from the user after the user event is finished, and refine the table in which events are mapped to emotion information, based on the obtained feedback. For example, the processor <NUM> may provide notification information for at least one event through a virtual assistant interface to a user, and obtain feedback information of the user on the at least one event included in the notification information. The processor <NUM> may identify an emotional state of the user for the at least one event by analyzing the feedback information of the user, and learn emotion information of the user for the at least one event, based on the emotional state of the user.

The processor <NUM> may obtain the emotion information of the user, considering at least one of a person related to the user event, a scheduled time of the user event, and a place related to the user event.

The processor <NUM> may determine an interaction type corresponding to the user event, based on the emotion information of the user. According to an embodiment, the processor <NUM> may determine a response to the utterance of the user or a tone for providing the notification information, based on the emotion information of the user related to the user event.

The processor <NUM> may control the outputter <NUM> to provide the response to the utterance of the user related to the user event though the interactive interface, based on the determined interaction type. For example, the processor <NUM> may select a color corresponding to the emotion information of the user, and display the response to the utterance of the user or the notification information related to the user event by using the selected color.

The processor <NUM> may provide the response by applying an expression style preferred by the user. The processor <NUM> may determine a notification providing method related to the user event, considering a situation of the user, and provide a notification related to the user event, based on the determined notification providing method. For example, the processor <NUM> may provide only a visual notification when the user is in an important meeting, and then provide a vibration or sound notification after the meeting is finished.

The communicator <NUM> may include one or more elements for enabling communication between the electronic device <NUM> and a wearable device or between the electronic device <NUM> and a server (not shown). For example, the communicator <NUM> may include a short-range wireless communicator <NUM>, a mobile communicator <NUM>, and a broadcast receiver <NUM>.

The short-range wireless communicator <NUM> may include, for example, a Bluetooth communicator, a Bluetooth low energy (BLE) communicator, a near field communicator (NFC), a wireless local area network (WLAN) (or Wi-Fi) communicator, a Zigbee communicator, an infrared data association (IrDA) communicator, a Wi-Fi direct (WFD) communicator, an ultra-wideband (UWB) communicator, and an Ant+ communicator, but is not limited thereto.

The mobile communicator <NUM> transmits and receives radio signals to and from at least one of a base station, an external terminal device, and a server in a mobile communication network. Herein, the radio signals may include various types of data based on transmission and reception of voice call signals, video call signals, or text/multimedia messages.

The broadcast receiver <NUM> receives broadcast signals and/or broadcast-related information through broadcast channels from outside. The broadcast channels may include satellite channels and terrestrial channels. According to another embodiment, the electronic device <NUM> may not include the broadcast receiver <NUM>.

According to an embodiment, the communicator <NUM> may obtain biometric information of a user of the external device. For example, the communicator <NUM> may collect heart rate information, breathing information, and body temperature information of the user from the wearable device connected to the electronic device <NUM>.

The A/V inputter <NUM> is used to input audio signals or video signals, and may include a camera <NUM> and a microphone <NUM>. The camera <NUM> may obtain image frames such as still images or moving images through an image sensor in a video call mode or an image capturing mode. The images captured through the image sensor may be processed through the processor <NUM> or a separate image processor (not shown). The image frames processed by the camera <NUM> may be stored in the memory <NUM> or be transmitted outside through the communicator <NUM>. According to an embodiment, the camera <NUM> may include at least one of a telescopic camera, a wide-angle camera, and a general camera, but is not limited thereto.

The microphone <NUM> receives an external sound signal and processes the same into electrical voice data. For example, the microphone <NUM> may receive a sound signal from the external device or the user. The microphone <NUM> may use various noise cancellation algorithms to cancel noise occurring when the external sound signal is received.

The user inputter <NUM> refers to a means used by the user to input data for controlling the electronic device <NUM>. For example, the user inputter <NUM> may include a keypad, a dome switch, a touchpad (e.g., a capacitive overlay, resistive overlay, infrared beam, surface acoustic wave, integral strain gauge, or piezoelectric touchpad), a jog wheel, or a jog switch, but is not limited thereto.

The memory <NUM> may store programs for processing and control operations of the processor <NUM>, and store input/output data (e.g., voice data, photographic images, metadata, personalized training data, and biometric information of the user).

The memory <NUM> may include at least one type of a storage medium from among flash memory, a hard disk, a multimedia card micro, a memory card (e.g., a secure digital (SD) or extreme digital (XD) card), random access memory (RAM), static RAM (SRAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), programmable ROM (PROM), magnetic memory, a magnetic disc, and an optical disc.

The programs stored in the memory <NUM> may be categorized into a plurality of modules, e.g., a response model <NUM>, based on functions thereof. The response model <NUM> corresponds to the event-emotion-based response model <NUM> of <FIG>, and thus a detailed description thereof is not provided herein.

The memory <NUM> may include an event DB <NUM>. The event DB <NUM> may correspond to the event DB <NUM> of <FIG>. In addition, the memory <NUM> may store an event-emotion DB <NUM>, an interaction method DB <NUM>, and a response DB <NUM> generated by the response model <NUM>. The event-emotion DB <NUM>, the interaction method DB <NUM>, and the response DB <NUM> respectively correspond to the event-emotion DB <NUM>, the interaction method DB <NUM>, and the response DB <NUM> of <FIG>, and thus a detailed description thereof is not provided herein.

The electronic device <NUM> may cooperate with a server (not shown) to provide a response message to the user.

In this case, the server (not shown) may generate an event-emotion-based response model by learning criteria for determining a response situation (e.g., a situation for providing an event-emotion-based response), and the electronic device <NUM> may determine a response situation by using the event-emotion-based response model generated based on the learning result of the server (not shown). In this case, the electronic device <NUM> may provide data for the event-emotion-based response model to the server (not shown), and the server (not shown) may determine a response situation and provide the determined response situation to the electronic device <NUM>. Alternatively, the electronic device <NUM> may receive data for the event-emotion-based response model from the server (not shown), and determine a response situation by using the received event-emotion-based response model.

A method according to an embodiment may be implemented in the form of program commands that can be executed through various computer means, and be recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, or combinations thereof. The program commands recorded on the computer-readable recording medium may be those specially designed and constructed for the purposes of the disclosure, or those well-known and available to one of ordinary skill in the art of computer software. Examples of the computer-readable recording medium include magnetic media (e.g., hard disks, floppy disks, and magnetic tapes), optical media (e.g., CD-ROMs or DVDs), magneto-optical media (e.g., floptical disks), and hardware devices (e.g., ROMs, RAMs, and flash memories) that are specially configured to store and execute program commands. Examples of the program commands include both machine code produced by a compiler, and high-level language code that may be executed by the computer using an interpreter.

Some embodiments may be implemented in the form of a computer-readable recording medium including instructions executable by a computer, e.g., a program module executed by a computer. The computer-readable recording medium may be an arbitrary available medium accessible by a computer, and examples thereof include all of volatile, non-volatile, detachable, and non-detachable media. The computer-readable recording medium may include a computer storage medium and a communication medium. Examples of the computer storage medium include all of volatile, non-volatile, detachable, and non-detachable media implemented using an arbitrary method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Examples of the communication medium typically include computer-readable instructions, data structures, program modules, other data of modulated data signals such as carrier waves, or other transmission mechanisms, and include arbitrary information transmission media. Some embodiments may be implemented as a computer program including instructions executable by a computer, e.g., a computer program executed by a computer, or a computer program product.

Claim 1:
A method, performed by an electronic device (<NUM>), of providing a virtual assistant interface, the method comprising:
querying a user about a schedule by using the virtual assistant interface;
receiving a response to the query from the user, the response comprising information about at least one event related to the schedule; wherein the receiving of the response to the query comprises receiving voice data of an utterance of the user as the response to the query,
identifying, from the voice data, the at least one event related to the schedule and an emotional state of the user for the at least one event related to the schedule by analyzing the response to the query;
when the emotion state of the user is not identified from the voice data, obtaining the emotion state of the user related to the at least one event from an event-emotion database where events are mapped to emotion states, wherein the event-emotion database is generated or refined by an Al model,
learning emotion information of the user for the at least one event related to the schedule, based on the emotional state of the user for the at least one event related to the schedule;
determining an interaction type for the at least one event related to the schedule by obtaining the interaction type corresponding to the emotion information of the user for the at least one event related to the schedule from an interaction method database generated or refined by the Al model; and
providing notification information for the at least one event related to the schedule through the virtual assistant interface, based on the interaction type;
obtaining feedback information on the at least one event from the user after the at least one event is finished; and
refining the emotion information of the user in the event emotion database for the at least one event, based on the obtained feedback information.