Patent Description:
An electronic device may output a response corresponding to an input of a user. In this regard, the electronic device may output a response including content that is more appropriate for the user, in further consideration of contextual information of the user.

However, contextual information of the user may continuously change in real time, and thus, even when a response is generated in consideration of the contextual information of the user, the response that is not appropriate for a current context of the user may be provided, due to that the contextual information changes in real time.

Therefore, there is a need to develop a method of providing a response appropriate to a user input based on contextual information of the user which changes in real time.

The <CIT> discloses a computer-implemented method for state-free generation of context-sensitive conversational responses based on the input message and conversation context data. The <CIT> discloses a method for virtual personal assistant in which a user state and multi-modal cues are used to provide a context-response to a user input. The <CIT> discloses a method of displaying a response in response to an inquiry of a user input together with an image of a subject providing the response. The <CIT> discloses a method and a computer device for suggesting emoji characters to be added to the inputted text by the user by using an aggregate emotional state.

Accordingly, an aspect of the disclosure is to provide an electronic device and an operating method thereof for outputting a response appropriate for a user input.

Another aspect of the disclosure is to provide a computer program product including a computer-readable recording medium having recorded thereon a program for executing the operating method in a computer. Technical problems to be solved are not limited to those mentioned above, and other technical problems may exist.

In accordance with the invention, a method of outputting a response to a user input in an electronic device is provided according to claim <NUM>.

In accordance with another aspect of the invention, an electronic device for outputting a response to a user input is provided according to claim <NUM>.

According to yet another aspect of the invention, a computer program product is provided according to claim <NUM>.

Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the invention as defined by the claims.

The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of various embodiments of the invention as defined by the claims. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the various embodiments described herein can be made without departing from the scope of the invention as defined by the claims.

Throughout the specification, it will also be understood that when an element is referred to as being "connected to" another element, it can be directly connected to the other element, or it can be electrically connected to the other element by having an intervening element interposed therebetween. Also, the term "include" an element does not preclude the other elements but further includes the element unless otherwise stated.

<FIG> is a diagram of a system that provides a response to a user input according to an embodiment of the disclosure.

Referring to <FIG>, according to an embodiment of the disclosure, a system for providing a response to a user input may include an electronic device <NUM>. The electronic device <NUM> may receive an input <NUM> from a user and output a response to the received input <NUM>.

The electronic device <NUM> according to an embodiment of the disclosure may be configured to receive the input <NUM> from the user and output a response to the received input <NUM>, and may be implemented in various forms. For example, the electronic device <NUM> described herein may be, but not limited to, a digital camera, a smart phone, a laptop computer, a tablet personal computer (PC), an e-book terminal, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, an moving picture experts group phase <NUM> or phase <NUM> (MPEG-<NUM> or MPEG-<NUM>) audio layer <NUM> (MP3) player, an artificial intelligence (Al) speaker, or the like.

The electronic device <NUM> may be a wearable device worn by the user. The wearable device may include, but not limited to, at least one of an accessory type device (e.g., a watch, a ring, a wrist band, an ankle band, a necklace, spectacles, and contact lenses), a head-mounted type device (HMD), a textile- or clothing-integrated type device (e.g., e-textiles), a body-attached type device (e.g., a skin pad), or a bio-implantable type device (e.g., implantable circuit). Hereinafter, a smart phone will be described as an example of the electronic device <NUM> for convenience.

According to an embodiment of the disclosure, an application may be used in the electronic device <NUM> to provide a response to the input. The application may provide an interactive interface to receive speech input from the user and output a response to the speech input of the user. The application according to an embodiment of the disclosure may include, but not limited to, a virtual assistant, an Al assistant, or the like. The virtual assistant or the Al assistant may be a software agent that processes a task required by the user and provides a personalized service for the user.

The electronic device <NUM> generates a response to the input <NUM> of the user and output the generated response.

The input <NUM> of the user according to an embodiment of the disclosure may include a query to obtain particular information. Also, the input <NUM> of the user according to an embodiment of the disclosure may include instructions for controlling the electronic device <NUM>. Without being limited to the above-described example, the input <NUM> according to an embodiment of the disclosure may include various inputs for various purposes and may be received by the electronic device <NUM> from the user.

The input <NUM> of the user according to an embodiment of the disclosure may be received by the electronic device <NUM> from the user, for example, received by the electronic device <NUM> as a speech input, a text input, a gesture input, or inputs via various types of sensors. Without being limited to the above-described example, the input <NUM> of the user may be input to the electronic device <NUM> via various input methods.

The response is generated by combining a first response including a first content with a second response including a second content, which is a different type from the first content. The first response includes the first content containing response information corresponding to the input of the user. In addition, the second response, as a response corresponding to the input of the user, includes the second content which is a different type from that of the first content and generated based on contextual information of the user.

The second response according to an embodiment of the disclosure may further include at least one content, which is the same type as that of the first content and generated based on contextual information, in addition to the second content of a different type from that of the first content.

Contextual information of the user according to an embodiment of the disclosure may include various types of information related the user such as behavioral traits of the user, individual traits of the user (e.g., age and profession), preferences of the user, interests of the user, aesthetic preferences of the user, surrounding environments of the user, a state of the user, social network service (SNS) accounts of the user, SNS activities of the user, operation states of the electronic device <NUM>, position information of the user, or position information of the electronic device <NUM>. Without being limited to the above-described example, the contextual information of the user may include various types of information indicating contexts of the user.

Thus, a response to the input of the user is provided based on the first response, which corresponds to the input of the user in which contextual information of the user is not considered, and the second response, which includes various types of information and is generated based on the contextual information of the user.

For example, the second response may include additional information generated based on current contextual information of the user and determined to be currently appropriate for the user. Also, the second response includes a type of content that is different from the content type of the first response, generated based on contextual information, and determined to be preferred by the user.

In addition, the type of the first content included in the first response includes at least one of text, a moving image, an image, or an audio, as a content type directly corresponding to the input of the user. Also, the type of the second content included in the second response includes, for example, at least one of text, a moving image, an image, an audio, a light-emitting diode (LED) output, a vibration output, a visual or audible effect, or a user interface as a content type additionally provided based on contextual information of the user. Without being limited to the above-described example, the first response and the second response may include various types of contents.

The above-described user interface, as a user interface additionally provided to the user in accordance with contextual information of the user, may include an interface for information retrieval, an interface for performing a route search function, an interface for translation, or the like. For example, the user interface may be provided by at least one application previously installed in the electronic device <NUM>.

Thus, the response includes the first response containing basic response information for the input in which contextual information is not considered and further includes the second response containing of information of various content types generated based on contextual information about the user which changes in real time.

The first response, as a basic response to the user input, is combined with the second response including additional contents generated based on contextual information, and a combined response is output, thereby providing better user experience. Also, by combining the first response with the second response including various types of contents and outputting the combined response, a richer response more appropriate for the user may be provided to the user, when compared to a case of providing the first response alone.

For example, an input <NUM> of "What is today's schedule?" is received by the electronic device <NUM>, the electronic device <NUM> may generate a first response including "conference at <NUM> p. " by text as a basic response corresponding to the input <NUM>.

The electronic device <NUM> also generates a second response by considering contextual information of the user. For example, when contextual information of the user indicates that the user is driving a car and cannot see a display of the electronic device <NUM>, the second response may include audio data converted from the text of the first response of "conference at <NUM> p. " based on the above-described contextual information of the user via a text to speech (TTS) technique.

The electronic device <NUM> according to an embodiment of the disclosure may generate a final response to be output by combining the first response with the second response and output the generated final response. For example, as a result of combining the first response with the second response, the electronic device <NUM> may display "conference at <NUM> p. " by text on the display simultaneously outputting the audio data of the second response via a speaker.

While a basic response to the user input is provided without any change according to a context of the user, an additional response according to contextual information of the user is further provided. Thus, according to an embodiment of the disclosure, because the basic response to the user input is not changed by contextual information, the user may easily predict a response for the input and an input a query satisfying a user's intent. In addition, because the response generated by considering contextual information is additionally provided, better user experience may be provided, as compared to when only the basic response to the input is provided.

Also, the context of the user continuously changes in real time, and thus contextual information of the user used while processing the user input may conflict with a current context of the user. For example, a response to the user input generated using contextual information different from a current contextual information of the user may be output due to delayed updating of the contextual information of the user. Thus, contextual information used to output the response may not match the context of the user after the response is output.

However, the basic response to the user input (e.g., first response) is not changed by the contextual information. Therefore, even when the contextual information is different from current contextual information of the user, the basic response to the user input is consistently provided.

The second response according to an embodiment of the disclosure may be additionally generated based on information related to the first response, as well as the contextual information. For example, the second response may be generated additionally based on feature information of the first content of the first response. Also, the second response may be generated additionally based on first data for generating the first response. The first data according to an embodiment of the disclosure is data used for generating the first response and may be generated based on analysis results of the input of the user.

The second response according to an embodiment of the disclosure may be additionally generated based on various information related to the first response without being limited to contextual information.

A method of outputting a response to a user input according to an embodiment of the disclosure may operate according to on-device AI technology only within the device. For example, the electronic device <NUM> may process the user input and output a response to the user input without exchanging data with a cloud server. In addition, the electronic device <NUM> may perform the operation according to an embodiment of the disclosure based on various types of information of the user, which are collected by the electronic device <NUM> in real time, without using data stored in the cloud server.

According to the on-device AI technology, the electronic device <NUM> may train an AI model based on data locally collected and may make determinations based on a trained Al model. Because the electronic device <NUM> does not transmit the collected data to the outside and utilizes the data according to the on-device AI technology, personal information of the user may be protected and data processing speed may be increased.

For example, the electronic device <NUM> may operate using the on-device AI technology without being connected to the cloud server depending on whether the AI model trained by the electronic device <NUM> is sufficient to perform the operation using only information collected by the electronic device <NUM>, without using big data and regardless of unstable network environments of the electronic device <NUM>.

However, the electronic device <NUM> may also perform the operation by exchanging data with the cloud server or the external device in conjunction with performing the operation according to the on-device AI technology. The electronic device <NUM> may also perform the operation according to an embodiment of the disclosure by combining the above-described on-device AI technology and data exchange with the cloud server.

For example, when the operation performed using the cloud server is more efficient than that performed using the on-device AI technology based on the network environments and computation capability of the electronic device <NUM>, for example, when a data processing speed of the operation using the cloud server is higher than that using the on-device AI technology or data not including personal information of the user is transmitted to the cloud server, the operation according to an embodiment of the disclosure may be performed using the cloud server.

<FIG> is a block diagram of an electronic device according to an embodiment of the disclosure.

<FIG> is a more detailed block diagram of an electronic device according to an embodiment of the disclosure.

Referring to <FIG>, the electronic device <NUM> includes a user input unit <NUM>, a processor <NUM>, and an output unit <NUM>. However, all of the components shown in <FIG> are not essential components of the electronic device <NUM>. The electronic device <NUM> may be implemented using more or less components than those shown in <FIG>.

For example, the electronic device <NUM> according to an embodiment of the disclosure may further include a sensor <NUM>, a communicator <NUM>, an audio/video (A/V) input unit <NUM>), and a memory <NUM> in addition to the user input unit <NUM>, the processor <NUM>, and the output unit <NUM> as shown in <FIG>.

The user input unit <NUM> refers to a device through which the user inputs data to control the electronic device <NUM>. For example, the user input unit <NUM> may be, but not limited to, a key pad, a dome switch, a touch pad (e.g., a touch capacitive type touch pad, a pressure resistive type touch pad, an infrared beam sensing type touch pad, a surface acoustic wave type touch pad, an integral strain gauge type touch pad, and a piezo electric type touch pad), a jog wheel, or a jog switch.

The user input unit <NUM> receives the input <NUM> of the user. For example, the user input unit <NUM> may receive the input <NUM> of the user via various input methods using a microphone, a touch pad, a switch, a gesture input device, or the like of the electronic device <NUM>.

The output unit <NUM> may output an audio signal, a video signal, or a vibration signal. The output unit <NUM> may include a display <NUM>, a sound output unit <NUM>, and a vibration motor <NUM>.

According to an embodiment of the disclosure, the output unit <NUM> may output a response to the input <NUM> of the user. According to an embodiment of the disclosure, the response may include various types of second contents. For example, the various types of contents such as a moving picture, an image, an audio, a light-emitting diode (LED) output, text, and a user interface may be output via the display <NUM> or the sound output unit <NUM>, and a vibration output type content may be output via the vibration motor <NUM>.

The display <NUM> may display and output information processed by the electronic device <NUM>. According to an embodiment of the disclosure, the display <NUM> may display information of a response corresponding to the input <NUM> of the user.

Meanwhile, when the display <NUM> and the touch pad constitute a touch screen in a layer structure, the display <NUM> may be used as an input device as well as an output device. The display <NUM> may include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional (3D) display, or an electrophoretic display. In addition, the electronic device <NUM> may include two or more displays <NUM> according to implementations of the electronic device <NUM>.

The sound output unit <NUM> outputs audio data received from the communicator <NUM> or stored in the memory <NUM>. The sound output unit <NUM> according to an embodiment of the disclosure may output audio data as a response corresponding to the input <NUM> of the user.

The vibration motor <NUM> may output a vibration signal. In addition, the vibration motor <NUM> may output a vibration signal when a touch is input onto the touch screen. The sound output unit <NUM> according to an embodiment of the disclosure may output a vibration signal as a response corresponding to the input <NUM> of the user.

The processor <NUM> is generally configured to control the overall operation of the electronic device <NUM>. For example, the processor <NUM> is configured to control the overall operation of the user input unit <NUM>, the output unit <NUM>, the sensor <NUM>, the communicator <NUM>, the A/V input unit <NUM>, and the like by executing programs stored in the memory <NUM>. The electronic device <NUM> may include at least one processor <NUM>.

The processor <NUM> may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output calculations. The commands may be provided to the processor <NUM> from the memory <NUM> or may be received by the communicator <NUM> and provided to the processor <NUM>. For example, the processor <NUM> may be configured to execute commands in accordance with program codes stored in a recording medium such as a memory.

The at least one processor <NUM> according to an embodiment of the disclosure may generate a response to the input <NUM> of the user. The response according to an embodiment of the disclosure may be generated by combining the first response including the first content with the second response including the second content of a different type from that of the first content.

The first response may include the first content containing response information matching the user input, and the second response as a response to the user input may include the second content that is different from the first content and is generated based on contextual information of the user.

The sensor <NUM> may detect a state of the electronic device <NUM> or an ambient state of the electronic device <NUM> and transmit obtained information to the processor <NUM>.

According to an embodiment of the disclosure, contextual information of the user may be obtained based on the information obtained by the sensor <NUM>. The second response may be generated based on the contextual information. For example, the sensor <NUM> may detect an emotion of the user from a facial expression of the user, and the electronic device <NUM> may obtain contextual information of the user based thereon.

Also, according to an embodiment of the disclosure, feedback information from the user on the response provided according to an embodiment of the disclosure may be obtained based on the information obtained by the sensor <NUM>. A trained model that is used to generate the second response may be modified and refined based on the feedback information.

The sensor <NUM> may include, but is not limited to, at least one of a geomagnetic sensor <NUM>, an acceleration sensor <NUM>, a temperature/humidity sensor <NUM>, an infrared ray sensor <NUM>, a gyroscope sensor <NUM>, a position sensor (e.g., global positioning system (GPS)) <NUM>, an atmospheric pressure sensor <NUM>, a proximity sensor <NUM>, or a red, green, and blue (RGB) sensor (illuminance sensor) <NUM>.

The communicator <NUM> (e.g., a transceiver) may include at least one component enabling communications between the electronic device <NUM> and a server (not shown) or an external device (not shown). For example, the communicator <NUM> may include a short-range wireless communicator <NUM>, a mobile communicator <NUM>, and a broadcasting receiver <NUM>.

According to an embodiment of the disclosure, the communicator <NUM> may receive data required to generate the response to the input <NUM> from the outside. For example, the communicator <NUM> may receive at least one content to be output as a response to the input <NUM> from the outside.

The short-range wireless communicator <NUM> may include, but not limited to, a Bluetooth communicator, a Bluetooth Low Energy (BLE) communicator, a near field communicator, a wireless local area network (WLAN) or Wi-Fi communicator, a Zigbee communicator, an infrared data association (IrDA) communicator, a Wi-Fi Direct (WFD) communicator, a ultra-wideband (UWB) communicator, and an Ant+ communicator.

The mobile communicator <NUM> may transmit and receive wireless signals to and from at least one of a base station, an external terminal, or a server on a mobile communication network. In this regard, the wireless signals may include various types of data for transmitting and receiving audio call signals, video communication call signals, or text/multimedia message.

The broadcasting receiver <NUM> may receive broadcasting signals and/or information related to broadcasting from the outside. The broadcasting channels may include satellite channels and terrestrial channels. According to an embodiment of the disclosure, the electronic device <NUM> may not include the broadcasting receiver <NUM>.

The A/V input unit <NUM> is configured to input audio signals or video signals and may include a camera <NUM> and a microphone <NUM>. The camera <NUM> may obtain image frames of a still image or a moving image via an image sensor in a video call mode or an image capturing mode. An image captured by the image sensor may be processed by the processor <NUM> or a separate image processor (not shown). The microphone <NUM> receives audio signals from the outside and processes the received signals into electrical sound data.

Information input via the A/V input unit <NUM> may be used to obtain contextual information of the user. For example, the state, shape, position, or the like of the user may be determined based on video or audio data generated by the A/V input unit <NUM>, and contextual information of the user may be obtained based thereon.

The memory <NUM> may store programs for processing and controlling of the processor <NUM> and store data input to or output from the electronic device <NUM>.

The memory <NUM> according to an embodiment of the disclosure may store one or more instructions and the at least one processor <NUM> of the electronic device <NUM> may perform the operation according to an embodiment of the disclosure by executing the one or more instructions.

In addition, the memory <NUM> according to an embodiment of the disclosure may store information required to generate a response to the input <NUM>. For example, the memory <NUM> may store a trained model, virtual assistant application, collected contextual information of the user, and the like, which may be used to generate the response to the input <NUM>. Without being limited to the above-described example, the memory <NUM> may store various types of information required to generate the response to the input <NUM>.

The memory <NUM> may include at least one type of storage medium selected from a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., secure digital (SD) or extreme digital (XD) memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), a magnetic memory, a magnetic disk, or an optical disk.

Programs stored in the memory <NUM> may be categorized into a plurality of modules according to functions thereof, e.g., a user interface (UI) module <NUM>, a touch screen module <NUM>, and a notification module <NUM>.

The UI module <NUM> may provide a specialized UI or a graphical user interface (GUI) interworking with the electronic device <NUM> according to applications. The touch screen module <NUM> may detect a touch gesture of the user on a touch screen and transmit information of the touch gesture to the processor <NUM>. The touch screen module <NUM> according to an embodiment of the disclosure may recognize and analyze a touch code. The touch screen module <NUM> may be configured as a separate hardware component including a controller.

Various sensors may be provided in or near the touch screen to detect a touch on the touch screen or a proximity touch. A tactile sensor may be an example of the sensor for detecting a touch on the touch screen. A tactile sensor is configured to detect a touch of an object with a human's tactile ability or more. The tactile sensor may obtain various information such as roughness of a contact surface, rigidity of a contact object, and temperature of a contact point.

The touch gesture of the user may include tap, touch-and-hold, double tap, drag, pan, flick, drag-and-drop, swipe, and the like.

The notification module <NUM> may generate a signal for notifying occurrence of an event of the electronic device <NUM>.

<FIG> is a diagram of a processor according to an embodiment of the disclosure.

Referring to <FIG>, the processor <NUM> may include a first response generator <NUM>, a second response generator <NUM>, a response renderer <NUM>, a prioritizer <NUM>, and a feedback unit <NUM> as components to perform the operation of generating a response to the input <NUM>.

However, the components shown in <FIG> are only examples, and the electronic device <NUM> may be implemented using more or less components than those shown in <FIG>.

The first response generator <NUM> according to an embodiment of the disclosure is configured to determine a user's intent by analyzing the input of the user received by the electronic device <NUM> and generate a first response corresponding to the user's intent.

For example, when the input received by the first response generator <NUM> is a speech input, text corresponding to the speech input may be obtained by performing speech recognition. Also, the first response generator <NUM> may determine the user's intention by performing natural language processing on the text corresponding to the input and generate a first response corresponding to the user's intent.

The second response generator <NUM> according to an embodiment of the disclosure is configured to obtain contextual information of the user and generate a second response to the user input based on the contextual information.

The second response generator <NUM> according to an embodiment of the disclosure may use a pre-trained model to generate the second response to the user input based on the contextual information. The trained model available for generation of the second response may be, for example, a generative model configured to generate a new second response from contextual information, such as a generative adversarial network (GAN).

According to an embodiment of the disclosure, a user's preference may be determined based on contextual information. According to an embodiment of the disclosure, the operation of determining the user's preference based on contextual information may be conducted by a convolution neural network (CNN) available for data recognition. For example, the user's preference may be determined based on contextual information by the CNN and the second response may be generated from the above-described generative model based on the determined preference.

The contextual information used to determine the user's preference may include various information on the user such as feedback information from the user on the provided response, information on contents used or purchased by the user, and information on traits of the user (e.g., age, residence, and profession).

Without being limited to the above-described example, various types of trained models may be used to generate the second response. According to an embodiment of the disclosure, at least one trained model used to generate the second response may be continuously modified to generate a second response appropriate for the user based on various data related to the user and collected by the electronic device <NUM>.

The response renderer <NUM> according to an embodiment of the disclosure is configured to combine the first response with the second response. For example, the response renderer <NUM> may combine the first response with the second response such that the first response is basically output and a visual or auditory response is additionally output as the second response. For example, a position or exterior of the first content of the first response may be appropriately modified according to a visual effect or a user interface included in the second response. In addition, the first response and the second response may be processed such that various types of the second contents of the second response may be output together with the first content of the first response. Without being limited to the above-described example, the response renderer <NUM> may combine the first response with the second response using various methods.

The prioritizer <NUM> according to an embodiment of the disclosure is configured to determine priorities of respective responses based on contextual information when a plurality of combined responses are obtained by the response renderer <NUM>. For example, when a second response including a plurality of second contents is generated, the electronic device <NUM> may generate a plurality of combined responses by combining the first content of the first response with the plurality of second contents. Also, when a plurality of first contents are generated, a plurality of combined responses may be generated by combining the plurality of first contents with at least one second content.

According to an embodiment of the disclosure, the prioritizer <NUM> may determine priorities of the respective combined responses. For example, because priorities are determined based on contextual information, the most appropriate response to the context of the user may be determined to have a highest priority. The electronic device <NUM> according to an embodiment of the disclosure may provide at least one of the plurality of combined responses to the user according to the determined priorities.

The feedback unit <NUM> according to an embodiment of the disclosure is configured to obtain information of feedback of the user on at least one response in which the first response is combined with the second response. For example, the feedback unit <NUM> may obtain information of an utterance, a face, a gesture, an emotion, or the like of the user and obtain feedback information based on the obtained information.

According to an embodiment of the disclosure, a generative model used to generate the second response may be modified and refined based on feedback information. For example, based on feedback information, the generative model may be modified and refined to generate the second response more preferred by the user by the generative mode.

<FIG> is a flowchart illustrating a method of outputting a response to a user input according to an embodiment of the disclosure.

Referring to <FIG>, in operation <NUM>, the electronic device <NUM> receives an input from a user. The user input according to an embodiment of the disclosure may be received by the electronic device <NUM> via various input methods such as a speech input, a button input, a touch input, a gesture input, and a sensor input. According to an embodiment of the disclosure, the electronic device <NUM> may receive the user input via a virtual assistant application providing an interactive interface.

In operation <NUM>, the electronic device <NUM> generates the first response including the first content as a response to the user input. The first content of the first response is obtained based on a user's intent by analyzing the input of the user without considering contextual information of the user. For example, when the analyzed user's intent is "information on today's weather", a first content indicating "sunny, bad fine dust level" may be generated as the first response corresponding to "information on today's weather".

In operation <NUM>, the electronic device <NUM> obtains contextual information of the user to generate the second response. The contextual information according to an embodiment of the disclosure may include traits, state, surrounding context, and the like of the user obtainable in real time. According to an embodiment of the disclosure, the contextual information used to generate the second response may be continuously updated based on various types of information obtained in real time.

In operation <NUM>, the electronic device <NUM> generates the second response including the second content, which is a different type from that of the first content and generated based on the contextual information obtained in operation <NUM>. For example, when the first response includes a text type first content, the second response may include a second content having one of various types different from the text type of the first content, such as a moving image, an image, a user interface, and an audio. Also, the second response may further include a content of the same type as that of the first content in addition to the content of a different type from that of the first content. The second response of the disclosure is generated based on the contextual information of the user obtained in operation <NUM>, in a different manner from that of the first response.

In operation <NUM>, the electronic device <NUM> combines the first response with the second response. For example, the electronic device <NUM> may process the contents of the first response and the second response to be appropriately arranged and displayed on a display or to be appropriately output via another output device, thereby generating a response in which the first response is combined with the second response.

In operation <NUM>, the electronic device <NUM> outputs the combined response obtained in operation <NUM> as a response to the user input of operation <NUM>.

<FIG> is a diagram illustrating an example of generating a second response according to an embodiment of the disclosure.

Referring to <FIG>, the electronic device <NUM> may obtain data including various types of information of the user in operation <NUM> and obtain contextual information indicating the context of the user from the data in operation <NUM>. For example, the above-described data may include various types of sensor data collected by at least one sensor included in the electronic device <NUM>. In addition, the contextual information may include various information indicating the context of the user and extracted based on the data, such as information on a surrounding environment of the user, information on a position of the user, information on an emotional state of the user.

In operation <NUM>, the electronic device <NUM> may generate the second response via the generative model <NUM> based on the contextual information obtained in operation <NUM>. For example, the electronic device <NUM> may generate the second response including the second content, as a response determined to be appropriate for a current context of the user based on the contextual information.

In addition, according to an embodiment of the disclosure, the electronic device <NUM> may output a response to the user input by appropriately combining the second response with the first response, which is generated without considering contextual information.

In operation <NUM>, the electronic device <NUM> may output the response in which the first response is combined with the second response as a response to the user input and modify and refine the generative model <NUM> based on feedback information from the user on the output response. For example, the electronic device <NUM> may obtain feedback information from the user on the output response based on various types of data (e.g., sensor data) about the user. The electronic device <NUM> may also modify and refine the generative model <NUM> based on the feedback information to generate the second response appropriate for the user.

According to an embodiment of the disclosure, the electronic device <NUM> may determine at least one feature of contextual information based on feedback information. Also, the electronic device <NUM> may obtain contextual information to generate the second response based on the determined at least one feature.

Contextual information according to an embodiment of the disclosure may include contextual information of at least one feature corresponding to each context of the user. For example, the contextual information may include contextual information with features respectively indicating different contexts of the user, such as emotional information of the user, noise information of a surrounding environment, and position information of the electronic device <NUM>.

According to an embodiment of the disclosure, the electronic device <NUM> may further obtain contextual information of a different feature from an existing feature from various types of sensor data collected by the at least one sensor based on feedback information.

For example, although emotional information of the user, which may be determined from audio data out of the sensor data collected by the electronic device <NUM>, may be obtained as the contextual information, noise information of a surrounding environment may be additionally obtained as the contextual information from audio data based on feedback information (e.g., the response output from the electronic device <NUM> is not audible due to noise of the surrounding environment).

Thus, because contextual information with a more appropriate feature is obtained from the same sensor data based on feedback information, and a more appropriate second response may be provided to the user.

<FIG> is a diagram illustrating an example of generating second data to generate a second response according to an embodiment of the disclosure.

Referring to <FIG>, the electronic device <NUM> may receive a user input <NUM> and obtain first data <NUM> to generate the first response to the received user input <NUM> in operation <NUM>. For example, the electronic device <NUM> may determine a user's intent by analyzing information of the user input <NUM> and obtain the first data <NUM> to generate the first response by searching via various methods based on a determined result.

The first data <NUM> according to an embodiment of the disclosure is data obtained without considering contextual information of the user, and the first response including the first content may be generated based on the first data <NUM>.

The electronic device <NUM> may obtain M1, M2, and Mn, corresponding to the user input <NUM>, as the first data <NUM> to generate the first response corresponding to the user input <NUM>. For example, when the user input <NUM> is "How to cook breaded chicken cutlets?," information of cooking ingredients for making a chicken cutlet, calories of respective ingredients, and nutrient profiles thereof may be obtained as the first data <NUM>. According to an embodiment of the disclosure, the first data <NUM> may include information required to generate the first response directly related to the user input <NUM> without considering contextual information <NUM>.

The electronic device <NUM> may obtain the contextual information <NUM> about the user. For example, the contextual information <NUM> about the user may include various information of the user such as diet information, physical condition information (e.g., medical information), and allergy information of the user.

According to an embodiment of the disclosure, the electronic device <NUM> may obtain second data <NUM> from the first data <NUM> based on the contextual information <NUM>. Also, the electronic device <NUM> may generate the second response based on the second data <NUM>. For example, the second data <NUM> may be obtained by modifying the first data <NUM> based on the contextual information <NUM>.

According to an embodiment of the disclosure, the second response may be generated by inputting the contextual information <NUM> and the first data <NUM> to the above-described generative model used to generate the second response. According to an embodiment of the disclosure, the generative model used to generate the second response may include at least one generative model, e.g., a first generative model for generating the second data <NUM> from the first data <NUM> based on the contextual information <NUM> and a second generative model for generating the second response from the second data <NUM>. Thus, according to an embodiment of the disclosure, the operation of generating the second response performed by the generative model may include an operation of generating the second data <NUM> from the first data <NUM> based on the contextual information <NUM> performed by the first generative model and an operation of generating the second response from the second data <NUM> performed by the second generative model.

Without being limited to the above-described example, the generative model may include a simple generative model of generating the second data <NUM> from the first data <NUM> based on the contextual information <NUM>, and then generating the second response from the second data <NUM>.

In addition, without being limited to the above-described example, the second response may be generated from the contextual information <NUM> and the first data <NUM> by various methods.

In operation <NUM>, the electronic device <NUM> may obtain the second data <NUM> from the first data <NUM> based on the contextual information <NUM> about the user. The electronic device <NUM> may obtain M'<NUM>, M'<NUM>, M'n, and the like from the second data <NUM> for generating the second response.

For example, the electronic device <NUM> may obtain a method of cooking a chicken cutlet reorganized in consideration of calories based on the diet information of the user or a method of cooking a chicken cutlet reorganized in consideration of allergy information as the second data <NUM>.

Thus, the second response according to an embodiment of the disclosure may be generated based on the second data <NUM> generated based on the contextual information <NUM> in a different manner from that of the first response generated based on the first data <NUM> without considering the contextual information <NUM>.

<FIG> is a diagram illustrating an example of obtaining feedback information according to an embodiment of the disclosure.

Referring to <FIG>, the electronic device <NUM> may obtain emotional information of the user <NUM> from sensor data <NUM> and obtain feedback information <NUM> from the emotional information of the user <NUM>.

According to an embodiment of the disclosure, emotional information of the user may be obtained based on respective sensor data <NUM>.

According to an embodiment of the disclosure, the electronic device <NUM> may obtain a speech signal from the sensor data <NUM> and obtain text corresponding to the speech signal in operation 804a. Also, the electronic device <NUM> may obtain feature information of the text based on the text corresponding to the speech signal in operation 806a. For example, the feature information of the text may include information indicating meaning of the text such as a topic, a keyword, or an intention of the text.

Also, according to an embodiment of the disclosure, the electronic device <NUM> may obtain an image signal from the sensor data <NUM> and obtain a facial image of the user included in the image signal in operation 804b. The electronic device <NUM> may also obtain information on a facial expression of the user by analyzing a shape of the face of the user included in the facial image of the user in operation 806b.

According to an embodiment of the disclosure, the electronic device <NUM> may also obtain gesture information from the sensor data <NUM> in operation 804c. For example, gesture information of the user collected by a motion sensor, a touch sensor, or the like may be obtained. The electronic device <NUM> may also obtain feature information of a gesture from the gesture information in operation 806c. For example, the electronic device <NUM> may determine an intention of the user making a gesture by analyzing the gesture information and obtain information on the determined user's intention as feature information of the gesture. For example, when a gesture of the user making X is detected, feature information of the gesture may include information indicating an opposite opinion.

According to an embodiment of the disclosure, the electronic device <NUM> may determine the emotional information of the user <NUM> based on the feature information of the text, information of the facial expression, and feature information of the gesture respectively obtained in operations 806a, 806b, and 806c. Also, the electronic device <NUM> may obtain the feedback information <NUM> from the emotional information of the user <NUM>. For example, the emotional information of the user may include information on emotions such as satisfaction, dissatisfaction, and anger, and the feedback information <NUM> including negative, positive, and neutral feedback may be obtained from the emotional information of the user.

According to an embodiment of the disclosure, the generative model used to generate the second response may be modified and refined based on the feedback information <NUM>. Thus, according to an embodiment of the disclosure, a more appropriate second response generated by the generative model modified and refined based on the feedback information <NUM> may be provided.

<FIG> is a flowchart illustrating a method of generating a second response including a second content according to an embodiment of the disclosure.

Referring to <FIG>, the electronic device <NUM> may generate the second response by obtaining feature information of the first content of the first response, and generating the second content based on the feature information of the first content and contextual information of the user. According to an embodiment of the disclosure, feature information of the first content may be obtained based on a trained model for recognition of certain data.

For example, the electronic device <NUM> may obtain image data, as the first content, and generate audio data by the generative model based on the feature information of the image data and contextual information of the user.

According to an example shown in <FIG>, although audio data, as the second content, may be obtained based on the feature information of the image data, as the first content, the disclosure is not limited thereto, and various types of second contents, such as audio data, image data, user interface, vibration signal, and LED effect, may be obtained based on various types of feature information of the first content.

Hereinafter, in <FIG>, a method of generating the second response including the second content when the first content is image data and the second content is audio data will be described.

Referring to <FIG>, in operation <NUM>, the electronic device <NUM> may obtain image data. For example, the image data may be image data obtained as the first response.

In operation <NUM>, the electronic device <NUM> may recognize an image from the image data obtained in operation <NUM> using a trained model for recognition of the image, and obtain feature information of the image data as a result thereof.

According to an embodiment of the disclosure, the trained model used to recognize the image data may be a CNN used to classify and detect objects in an image. Without being limited to the above-described example, various types of data recognition models operating based on a neural network may be used to obtain feature information of an image according to an embodiment of the disclosure.

Recognition results of the image data according to an embodiment of the disclosure may include information on an object recognized in the image, information on position of the object recognized in the image, information on motion of the object recognized in the image, and the like. For example, when an image input to the image recognition model is an image of a cat, the recognition result of the image obtained by the image recognition model may include "cat. " Thus, feature information of the image may include "cat" as a recognition result of the image.

In operation <NUM>, the electronic device <NUM> may obtain contextual information of the user. The contextual information obtained in operation <NUM> may be used to generate audio data of the second response.

In operation <NUM>, the electronic device <NUM> may obtain audio data on the image data based on the feature information of image data obtained in operation <NUM> and contextual information obtained in operation <NUM>.

For example, the electronic device <NUM> may generate audio data based on feature information of image data using the generative model (e.g., GAN) for generation of audio data form image data. Also, the generated audio data may be modified based on the contextual information.

For example, when it is determined that there is a high level of noise in the surrounding environment of the electronic device <NUM> based on contextual information, the electronic device <NUM> may modify the audio data based on contextual information to raise a volume of the audio data generated based on feature information of image data.

Without being limited to the above-described example, the electronic device <NUM> may obtain audio data on image data using various methods based on feature information of the image data obtained as a result of performing image recognition and based on contextual information.

In operation <NUM>, audio data generated according to an embodiment of the disclosure may be a second response combined with the first response to be output as a response to the user input.

<FIG> is a diagram illustrating a method of training a generative model to generate a second response, according to an embodiment of the disclosure.

Referring to <FIG>, a generative model <NUM> trained by a server <NUM> may be transmitted to the electronic device <NUM>, and the electronic device <NUM> may continuously modify a generative model <NUM> according to information collected by the electronic device <NUM> based on the generative model <NUM> received from the server <NUM>.

The generative models <NUM> and <NUM> according to an embodiment of the disclosure are trained models used to generate the second response including the second content, which is a different type from that of the first content of the first response, and generated based on contextual information.

The server <NUM> according to an embodiment of the disclosure may train the generative model <NUM> based on various training data <NUM> collected by the server <NUM>. For example, the training data <NUM> may include social network service (SNS) data <NUM> associated with SNS activities of the user.

For example, the server <NUM> may obtain various information of the user, such as information of user's preferences, information of a surrounding environment, and user profile information, based on the SNS data <NUM>, and train the generative model <NUM> based on the obtained information.

The server <NUM> according to an embodiment of the disclosure may transmit the generative model <NUM> trained by the server <NUM> to the electronic device <NUM>. The electronic device <NUM> according to an embodiment of the disclosure may further train the generative model <NUM> received from the server <NUM> based on training data <NUM> including various information collected by the electronic device <NUM>.

The electronic device <NUM> according to an embodiment of the disclosure may continuously train the generative model <NUM> received from the server <NUM> based on the training data <NUM> collected by the electronic device <NUM>, and generate the second response according to the generative model <NUM> trained by the electronic device <NUM>. According to an embodiment of the disclosure, the generative model <NUM> may be continuously trained and modified based on the training data <NUM> collected by the electronic device <NUM> according to the on-device method without being processed by the server <NUM>.

The training data <NUM> collected by the electronic device <NUM> may include various user data <NUM> collected in relation to the user (e.g., sensor data and input data) and feedback information from the user <NUM>. For example, the feedback information from the user <NUM> may include feedback information from the user on the response provided according to an embodiment of the disclosure. Without being limited to the above-described example, data collected by the electronic device <NUM> may include various types of data that may be collected in relation to the user by the electronic device <NUM>.

<FIG> is a diagram illustrating a method of training a generative model <NUM> according to an embodiment of the disclosure.

According to an embodiment of the disclosure, when the electronic device <NUM> generates a first response <NUM> and a second response <NUM>, the generative model <NUM> may be trained according to an example shown in <FIG>.

Referring to <FIG>, the electronic device <NUM> may train the generative model <NUM> based on training data <NUM> collected by the electronic device <NUM>. The training data <NUM> according to an embodiment of the disclosure may correspond to the training data <NUM> that may be collected by the electronic device <NUM> shown in <FIG> as information related to the user. For example, the training data <NUM> may include feedback information from the user, sensor data obtained about the user, various types of user data related to the user, and the like.

In addition, the training data <NUM> may include contextual information of the user according to an embodiment of the disclosure. Thus, according to an embodiment of the disclosure, the generative model <NUM> may be modified to provide a second response <NUM> appropriate for the user according to contextual information that continuously changes.

The electronic device <NUM>, according to an embodiment of the disclosure, may generate the first response <NUM> for an input of the user. Also, the electronic device <NUM> may generate the second response <NUM> for the input of the user including a content of a different type from the content type of the first response <NUM> based on contextual information <NUM> and generative model <NUM>.

According to an embodiment of the disclosure, the electronic device <NUM> may evaluate <NUM> whether the second response <NUM> generated by the generative model <NUM> is appropriate based on the training data <NUM> collected by the electronic device <NUM>. For example, as the degree of correlation between the training data <NUM> and the second response <NUM> increases, consistency of the second response <NUM> provided according to an embodiment of the disclosure is maintained with information related the user (e.g., contextual information), and thus the second response <NUM> may be evaluated as more appropriate.

Therefore, according to an embodiment of the disclosure, the second response <NUM> obtained by the generative model <NUM> may be evaluated based on the degree of correlation between the second response <NUM> and the training data <NUM>, and the generative model <NUM> may be trained according to the evaluation result.

According to an embodiment of the disclosure, the second response <NUM> may also be evaluated further based on the first response <NUM> in addition to the training data <NUM>. For example, the second response <NUM> may be evaluated based on a degree of correlation among the first response <NUM>, the training data <NUM>, and the second response <NUM>.

The electronic device <NUM> according to an embodiment of the disclosure may train the generative model <NUM> based on the evaluation result. For example, the electronic device <NUM> may modify the generative model <NUM> based on the evaluation result to obtain a better evaluation result for the second response <NUM> at <NUM>.

The electronic device <NUM> according to an embodiment of the disclosure may continuously train the generative model <NUM> based on the degree of correlation between the second response <NUM> and the training data <NUM> about the user collected by the electronic device <NUM>. Thus, the electronic device <NUM> may obtain a more personalized generative model <NUM> than the generative model <NUM> of the server <NUM> by training the generative model <NUM> according to the training data <NUM> about the user collected by the electronic device <NUM>.

As described above, the embodiment of the disclosure shown in <FIG> is not limited to the electronic device <NUM>, and the first response <NUM> and the second response <NUM> may be generated by the server <NUM> shown in <FIG> and the generative model <NUM> may be trained based on the training data <NUM> collected by the server <NUM>. For example, the first response <NUM> and the second response <NUM> may be generated by the server <NUM> for the user input at a request of the electronic device <NUM>. Also, the generative model <NUM> of the server <NUM> may be trained, in the same manner as the generative model <NUM> is trained by the electronic device <NUM> shown in <FIG>, according to an evaluation result, based on the training data <NUM> collected by the server <NUM> and the second response <NUM>.

<FIG> is a diagram illustrating an example of outputting a response to a user input by an electronic device according to an embodiment of the disclosure.

Referring to <FIG>, a first response <NUM> including image data may be generated for a user input. The first response <NUM> may be combined <NUM> with a second response <NUM> generated by the generative model <NUM> and a combined response <NUM> may be output.

According to an embodiment of the disclosure, at least one feature vector <NUM> for generation of the second response <NUM> may be extracted from the contextual information <NUM>. The at least one feature vector <NUM> may include information indicating at least one feature of the contextual information <NUM>.

For example, the contextual information <NUM> may include contextual information of at least one feature indicating each context and at least one feature vector corresponding to each contextual information of the at least one feature may be extracted.

According to an embodiment of the disclosure, the electronic device <NUM> may extract at least one feature vector <NUM> from the contextual information <NUM> and generate the second response <NUM> by inputting the extracted feature vector <NUM> to the generative model <NUM>.

According to an embodiment of the disclosure, the feature vector may be determined by relative position information determined according to at least one feature indicated by contextual information in a graph in which one or more cluster, which are classified according to preset criteria, are distributed. For example, a feature vector value may be determined such that a center of a cluster including a feature more similar to contextual information is located to be closer to a position of the feature vector. Without being limited to the above-described example, the feature vector for the contextual information <NUM> may include various forms of information indicating features of contextual information.

Also, according to an embodiment of the disclosure, the electronic device <NUM> may further extract feature information <NUM> of content included in the first response <NUM> and input the extracted result to the generative model <NUM>. Thus, the electronic device <NUM> may generate a second response <NUM> highly related to the first response <NUM> in addition to the contextual information <NUM>.

The second response <NUM> according to an embodiment of the disclosure may include audio data, which is a content of a different type from the content of the first response <NUM>.

According to an embodiment of the disclosure, the electronic device <NUM> may generate feedback information <NUM> based on data obtained by the sensor <NUM>, and update_the contextual information <NUM> based on feedback information. The data obtained by the sensor <NUM> may include sensor data for determining reaction to or feedback on the response <NUM> output according to an embodiment of the disclosure.

The image data and audio data shown in <FIG>, contents of the first response <NUM> and the second response <NUM> respectively, are only examples and are not limited thereto, and the first response <NUM> and the second response <NUM> according to an embodiment of the disclosure may include various types of contents.

<FIG> is a diagram illustrating an example of acquiring contextual information of a user based on user profile information according to an embodiment of the disclosure.

Referring to <FIG>, profile information <NUM> and <NUM> of Users A and B may be converted (e.g., encoded) into feature vectors <NUM> and <NUM>, respectively, by the processor <NUM>. According to an embodiment of the disclosure, the feature vector may be determined according to relative position information determined in accordance with characteristics of predetermined data. The feature vectors <NUM> and <NUM> of the Users A and B may be expressed as shown in a graph <NUM>.

According to an embodiment of the disclosure, the feature vector values converted based on profile information may be used as, contextual information, to generate a second response. According to an embodiment of the disclosure, the electronic device <NUM> may generate the second response <NUM> by inputting the feature vector values converted based on profile information to the generative model <NUM> shown in <FIG>.

Without being limited to the above-described example, user profile information may be converted into various forms of information indicating of contextual information of the user, and the second response <NUM> may be generated based on the converted information.

<FIG> is a diagram illustrating an example of generating a second response based on user profile information according to an embodiment of the disclosure.

Referring to <FIG>, second responses including different contents, which vary according to use profile information, may be provided for the same input, and responses respectively generated based on the second responses may be provided to respective users. The profile information, as contextual information according to an embodiment of the disclosure, may be used to generate the second response.

According to an embodiment of the disclosure, for a user input requesting for "Book a cab" <NUM>, first data <NUM> related to an available cab may indicate that Cab A is expected to take <NUM> minutes to reach a destination with a fare of <NUM> $. Also, Cab B is expected to take <NUM> minutes to reach the destination with a fare of <NUM> $. For example, because Cab A, unlike Cab B, provides a carpool-type cab service, a lower-fare service may be provided although it takes longer to arrive at the destination.

According to an embodiment of the disclosure, when the user input <NUM> requesting for a cab is received from User C, the electronic device <NUM> may generate a first response based on booking information of Cab A and Cab B included in the first data <NUM> (Metadata).

According to an embodiment of the disclosure, priorities <NUM> (Rank) of Cab A and Cab B included in the first data <NUM> (metadata) may be determined based on profile information <NUM> (Vi) about User C, which is used to provide guide information of Cab A <NUM> or guide information of Cab B. The priorities <NUM> (Rank) may be determined by a predetermined function (f). For example, the electronic device <NUM> may determine that the priority of Cab B is higher, due to a less time to arrive at the destination despite a higher fare, based on at least one of salary information, age information, or profession information included in profile information of User C. Thus, the electronic device <NUM> may generate a second response including information on how to book Cab B based on the profile information of User C.

Also, the second response according to an embodiment of the disclosure may further include a moving image, an audio, a visual effect, a user interface, or the like, as a second content of a different type from text, which is the content type of the first response. For example, the second response may further include a user interface of an application providing a cab booking serve for Cab B and installed in the electronic device <NUM>.

In addition, according to an embodiment of the disclosure, the electronic device <NUM> may combine the first response with the second response and provide a combined response to User C. For example, the electronic device <NUM> may combine the first response with the second response such that booking information on Cab A and Cab B is displayed based on the first response and the user interface providing a cab booking service for Cab B is simultaneously displayed based on the second response. The combined response may be provided to User C.

Meanwhile, according to an embodiment of the disclosure, when a user input <NUM> requesting for booking a cab is received from User D, the electronic device <NUM> may determine priorities <NUM> of Cab A and Cab B included in the first data <NUM> (metadata) based on profile information <NUM> (Vi) about User D. In the case of User D, in contrast to User C, the electronic device <NUM> may determine that the priority of Cab A is higher, due to a lower fare despite a longer time to arrive at the destination, based on at least one of salary information, age information, or profession information included in the profile information of User D. Thus, the electronic device <NUM> may generate a second response including information on how to book Cab A based on profile information of User D.

Also, according to an embodiment of the disclosure, the electronic device <NUM> may combine the first response with the second response such that booking information on Cab A and Cab B is displayed based on the first response and the user interface providing a cab booking service for Cab A is simultaneously displayed based on the second response. The combined response may be provided to User D.

<FIG> is a diagram illustrating an example of providing a response to a user input according to an embodiment of the disclosure.

Referring to <FIG>, second responses including different contents, which vary according to user profile information, may be provided for the same input, and responses respectively generated based on the second responses may be provided to respective users.

According to an embodiment of the disclosure, when a user input of "How much is a billion?" <NUM> and <NUM> is received from User E or User F, a first response may be generated based on first data <NUM> generated based on the user input. The first response according to an embodiment of the disclosure may be generated based on the first data <NUM> generated based on the user input without considering user profile information.

According to an embodiment of the disclosure, second data <NUM> and <NUM> may be generated from the first data <NUM> based on various types of contextual information of each user such as profile information, preference information, and SNS activity information. Thus, second responses respectively generated based on the second data <NUM> and <NUM> according to an embodiment of the disclosure may include personalized responses in accordance with the contexts of the users.

For example, when User E is a student, the second data <NUM> of User E may include data for generating a response to the user input <NUM> based on interests, knowledge levels, or the like of User E. Also, when User F is a kid, the second data <NUM> of User F may include data for generating a response to the user input <NUM> based on interests, knowledge levels, or the like of User F.

Thus, the second responses for inputs of Users E and F may be generated to suit the interests, knowledge levels, or the like of the users. For example, the second response to User E may include a response "A billion corresponds to <NUM><NUM> or ten million rupees". Also, the second response to User F may include a response "A billion is a tremendous number and a large pool may be filled with a billion balls".

Referring to <FIG>, second responses including different contents, which vary according to user profile information, may be provided for the same input of "Show me a sport content", and responses respectively generated based on the second responses may be provided to respective users.

Profile information of User G according to an embodiment of the disclosure may include news, as an interest, and home, as a current position. Also, profile information of User H may include soccer, particularly Player <NUM>, Team <NUM>, as an interest, and home, as a current position. In addition, profile information of User I may include travel and sport, as interests, and Hotel A in a city center, as a current position.

According to an embodiment of the disclosure, a first response may be generated based on first data <NUM> including information on sport contents that may be provided by the electronic device <NUM> in response to the user input. For example, the first data <NUM> may include information on a sport game that is currently relayed and may be provided by the electronic device <NUM>.

According to an embodiment of the disclosure, the electronic device <NUM> may also generate different second data <NUM>, <NUM>, and <NUM> from the first data <NUM> for respective users based on the above-described user profile information.

For example, the electronic device <NUM> may determine that User G is not interested in sports but only interested in sport news and generate audio data including a content, User G might be interested in, indicating that a soccer game between Team <NUM> and Team <NUM> is in progress, as second data <NUM> based on the first data <NUM>. The audio data of the second data <NUM> may include a voice reading the above content, in a news tone, by considering a taste of User G.

In addition, for example, the electronic device <NUM> may generate the second data <NUM> including audio data based on the first data <NUM> by considering features that User H is interested in soccer games, Player <NUM>, and Team <NUM>. The audio data of the second data <NUM> for User H may include a voice reading a content that a soccer game between Teams <NUM> and <NUM> where Player <NUM> is playing is in progress, in an excited tone. The second data <NUM> may include a clip image, a still image, or the like of Player <NUM>.

For example, the electronic device <NUM> may also generate the second data <NUM> including audio data based on the first data <NUM> by considering features that User I is interested in travel. The audio data of the second data <NUM> may include a voice reading a content that a soccer game between Teams <NUM> and <NUM> is in progress in Spanish stadium <NUM>. The second data <NUM> may further include information on a position, an image, or the like of Spanish stadium <NUM>.

According to an embodiment of the disclosure, second responses for the respective users may be generated respectively based on the second data <NUM>, <NUM>, and <NUM> which are generated for the users. Also, responses for the respective user inputs may be provided based on the respective second responses. Thus, according to an embodiment of the disclosure, different responses may be provided for the same user input, in accordance with user profile information.

Referring to <FIG>, second responses including different contents, which vary according to user profile information, may be provided for the same input of "What is an atom", and responses respectively generated based on the second responses may be provided to respective users.

According to an embodiment of the disclosure, profile information of User J may include information indicating that User J has a basic level of scientific knowledge and prefers a simple user interface. Also, profile information of User K may include information indicating that User K likes science and has a scientific knowledge level of middle school students. Also, profile information of User L may include information indicating that User L is a kid interested in cartoons and has a scientific knowledge level of elementary school students.

According to an embodiment of the disclosure, a first response may be generated based on first data <NUM> including basic information of atoms that can be provided by the electronic device <NUM> in response to the user input. For example, the first data <NUM> may include information of the meaning of atom that may be found in an encyclopedia.

Also, according to an embodiment of the disclosure, the electronic device <NUM> may generate different second data <NUM>, <NUM>, and <NUM> for respective users from the first data <NUM> based on the above-described user profile information.

For example, the electronic device <NUM> may include the second data <NUM> including image data and audio data briefly explaining about atoms by considering that User J has a basic level of scientific knowledge and prefers a simple user interface.

Also, the electronic device <NUM> may include the second data <NUM> including image data and audio data explaining atoms suitable for middle school students by considering that User K likes science and has a scientific knowledge level of middle school students.

Also, the electronic device <NUM> may include the second data <NUM> including image data and audio data explaining atoms using cartoon characters suitable for elementary school students by considering that User L is a kid interested in cartoons and has a scientific knowledge level of elementary school students.

According to an embodiment of the disclosure, second responses for respective users may be generated respectively based on the second data <NUM>, <NUM>, and <NUM> which are generated for the users. Also, responses for the respective user inputs may be provided based on the respective second responses. Thus, according to an embodiment of the disclosure, different responses may be provided for the same user input, in accordance with user profile information.

<FIG> is a diagram illustrating an example of providing a response to user input according to an embodiment of the disclosure.

Referring to <FIG>, different responses may be provided according to contextual information for an input of User M.

According to an embodiment of the disclosure, profile information of contextual information of User M may include information indicating that a preferred cab service provider is Cab A, information indicating that User M prefers a lower fare to arriving earlier, and information of a time that User M gets to work. Also, the contextual information of User M may include information on emergency.

According to an embodiment of the disclosure, the electronic device <NUM> may receive an input of "Book a cab for a hospital <NUM>" from User M. First data <NUM> according to an embodiment of the disclosure may be generated based on information input by User M. For example, the electronic device <NUM> may generate the first data <NUM> based on, "hospital" that is destination information obtained from the user input.

The first data <NUM> according to an embodiment of the disclosure may include booking information of Cab A and Cab B. However, when contextual information according to an embodiment of the disclosure includes information indicating an emergency situation, the electronic device <NUM> may generate second data <NUM> from the first data <NUM> based on the contextual information. For example, the electronic device <NUM> may generate the second data <NUM> including a user interface providing a booking service for Cab B which arrives earlier although User M does not prefer expensive Cab B. Also, the second data <NUM> may further include an audio guidance of "A cab toward the hospital is arriving shortly" related to the booking service for Cab B.

According to an embodiment of the disclosure, the first response generated based on the first data <NUM> is combined with a second response generated based on the second data <NUM>, and the combined response may be provided to User M. User M may input additional information to the electronic device <NUM> for booking a cab based on the combined response of the first response and the second response.

According to an embodiment of the disclosure, the electronic device <NUM> may receive an input of "Book a cab for a mall <NUM>" from User M. First data <NUM> according to an embodiment of the disclosure may be generated based on information input by User M. For example, the electronic device <NUM> may generate first data <NUM> based on "mall" that is destination information obtained from the user input.

The first data <NUM> according to an embodiment of the disclosure may include booking information of Cab A and Cab B. The electronic device <NUM> may generate second data <NUM> from the first data <NUM> based on profile information of User M according to an embodiment of the disclosure. For example, the electronic device <NUM> may generate second data <NUM> including a user interface providing a booking service for Cab A which provides a service with a lower fare and is preferred by User M. Also, the second data <NUM> may further include an audio guidance of "A cab is leaving for the mall as the destination" related to the booking service for Cab A.

According to an embodiment of the disclosure, the electronic device <NUM> may receive an input of "Book a cab for the office <NUM>" form User M. First data <NUM> according to an embodiment of the disclosure may be generated based on information input by User M. For example, the electronic device <NUM> may generate the first data <NUM> based on, "office" that is destination information obtained from the user input.

The first data <NUM> according to an embodiment of the disclosure may include booking information of Cab A and Cab B. The electronic device <NUM> may generate second data <NUM> from the first data <NUM> based on profile information of User M according to an embodiment of the disclosure. For example, the electronic device <NUM> may generate second data <NUM> including a user interface providing a booking service for Cab A which provides a service with a lower fare, arrives at the office before <NUM>:<NUM> a. , and is preferred by User M. Also, the second data <NUM> may further include an audio guidance of "A cab will arrive at the office before <NUM>:<NUM> a. " related to the booking service for Cab A.

According to an embodiment of the disclosure, better user experience may be provided by providing additional responses generated based on contextual information as well as a basic response to a user input.

According to an embodiment of the disclosure, more appropriate responses may be provided to users by providing additional responses generated based on contextual information as well as a basic response to a user input.

The embodiments of the disclosure may be embodied as a computer-readable recording medium, e.g., a program module to be executed in computers, which includes computer-readable instructions. The computer-readable recording medium may include any usable medium that may be accessed by computers, volatile and non-volatile medium, and detachable and non-detachable medium. Also, the computer-readable recording medium may include a computer storage medium and a communication medium. The computer storage medium includes all volatile and non-volatile media, and detachable and non-detachable media which are technically implemented to store information including computer-readable instructions, data structures, program modules or other data. The communication medium includes computer-readable instructions, a data structure, or a program module, and includes other information transmission media.

Also, throughout the specification, the "unit" may be a hardware component such as a processor or a circuit and/or a software component executed by the hardware such as the processor.

Claim 1:
A method performed by an electronic device, comprising:
receiving (<NUM>) a user input from a user; and
in response to receiving the user input, characterized in,
generating (<NUM>) a first response comprising first content of a first type based on the user input irrespective of contextual information of the user,
obtaining (<NUM>) contextual information of the user of a context which changes continuously in real time, and wherein there is delayed updating of the contextual information of the user,
generating (<NUM>) a second response comprising second content of a second type based on the contextual information, the second type of the second content being different from the first type of the first content, wherein the first type of the first content comprises at least one of text, a moving picture, an image, or audio content, and wherein the second type of the second content comprises at least one of text, a moving picture, an image, audio content, a light-emitting diode, LED, output, a vibration output, a visual effect, an audible effect, or a user interface,
generating a combined response (<NUM>) based on the first response and the second response, and
outputting (<NUM>) the combined response by outputting the first response together with the second response.