ELECTRONIC DEVICE AND METHOD TO CONTROL EXTERNAL APPARATUS

An electronic device is disclosed that, in response to at least one of an utterance intent and a control target device not being identified from utterance data, classifies a situation factor based on the utterance data, determines one or more external devices that match the classified situation factor, and generates and presents to the user terminal one or more action scenarios for one or more external devices determined for the classified situation factor.

BACKGROUND

The disclosure relates to an electronic device for controlling an external device.

2. Description of Related Art

Recently, many devices, such as air conditioners, TVs, and dryers, support actions through voice commands. For example, when a user utters a command such as “Lower the air conditioner temperature,” “Turn on the TV to channel 11,” and “Tell me when the drying is finished,” each device converts the command into text and performs an action according to the user intent in response to execution. When these devices operate on one platform, the user must clearly say which device to execute the command (perform a single command) or set commands (perform a plurality of commands) for various combinations of devices in advance in the form of a quick command or the like to control several devices.

SUMMARY

Embodiments of the disclosure may provide an electronic device configured to indirectly discern the intent of a user utterance through a classification of a situation factor even when the user utterance intent is not discerned.

According to an example embodiment, the electronic device may include: a communication module comprising communication circuitry configured to establish wireless communication with a user terminal, a memory configured to store computer-executable instructions, and a processor configured to execute the computer-executable instructions by accessing the memory, and the instructions, when executed, cause the processor to: receive utterance data from the user terminal using the communication module, classify a situation factor based on the utterance data, determine one or more external devices that match the classified situation factor, and generate and present one or more action scenarios for the one or more external devices determined for the classified situation factor to the user terminal.

According to an example embodiment, a method implemented by a processor may include: receiving utterance data from a user terminal using a communication module, classifying a situation factor based on the utterance data, determining one or more external devices that match the classified situation factor, and generating and presenting one or more action scenarios for the one or more external devices determined for the classified situation factor to the user terminal.

According to various example embodiments, the electronic device may quickly convert a user situation (e.g., a negative situation) to another situation (e.g., a positive situation) by discerning the intent of the indirect utterance of the user and controlling an external device more flexibly in response to various utterances of the user.

DETAILED DESCRIPTION

Hereinafter, various example embodiments will be described in greater detail with reference to the accompanying drawings. When describing the example embodiments with reference to the accompanying drawings, like reference numerals refer to like elements and a repeated description related thereto may not be provided.

FIG.1is a block diagram illustrating an example configuration of an integrated intelligence system according to various embodiments.

Referring toFIG.1, the integrated intelligence system10may include a user terminal100, an intelligent server200, and a service server300.

The user terminal100may be a terminal device (or an electronic device) capable of connecting to the Internet, and may be, for example, a mobile phone, a smartphone, a personal digital assistant (PDA), a notebook computer, a TV, a large home appliance, a wearable device, a head-mounted display (HMD), or a smart speaker.

According to an embodiment, the user terminal100may include a communication interface (e.g., including communication circuitry)110, a microphone120, a speaker130, a display140, a memory150, and/or a processor (e.g., including processing circuitry)160. The components listed above may be operationally or electrically connected to each other.

The communication interface110may include various communication circuitry and be connected to an external device to transmit and receive data. The microphone120may receive sound (e.g., a user utterance) and convert it into an electrical signal. The speaker130may output the electrical signal as sound (e.g., a voice). The display140may be configured to display an image or a video. The display140may also display a graphic user interface (GUI) of an app (or an application program) to be executed.

The memory150may store a client module151, a software development kit (SDK)153, and a plurality of apps155. The client module151and the SDK153may configure a framework (or a solution program) for performing general-purpose functions. In addition, the client module151or the SDK153may configure a framework for processing a voice input.

The plurality of apps155may be a program for performing a designated function. According to an embodiment, the plurality of apps155may include a first app155_1and a second app155_2. According to an embodiment, each of the plurality of apps155may include a plurality of actions for performing a designated function. For example, the apps may include an alarm app, a messaging app, and/or a scheduling app. According to an embodiment, the plurality of apps155may be executed by the processor160to sequentially execute at least some of the plurality of actions.

The processor160may include various processing circuitry and control the overall action of the user terminal100. For example, the processor160may be electrically connected to the communication interface110, the microphone120, the speaker130, and the display140to perform a designated action.

The processor160may also execute a program stored in the memory150to perform a designated function. For example, the processor160may execute at least one of the client module151and the SDK153to perform the following action for processing voice input. The processor160may control, for example, an action of the plurality of apps155through the SDK153. The following action described as an action of the client module151or the SDK153may be an action performed by execution of the processor160.

The client module151may receive a voice input. For example, the client module151may receive a voice signal corresponding to a user utterance received by the microphone120. The client module151may transmit the received voice input to the intelligent server200. The client module151may transmit state information of the user terminal100with the received voice input to the intelligent server200. The state information may be, for example, execution state information of an app.

The client module151may receive a result corresponding to the received voice input. For example, when the intelligent server200is capable of calculating a result corresponding to the received voice input, the client module151may receive the result corresponding to the received voice input. The client module151may display the received result on the display140.

The client module151may receive a plan corresponding to the received voice input. The client module151may display the result of executing a plurality of actions of an app on the display140according to the plan. The client module151may, for example, sequentially display the results of executing the plurality of actions on the display. The user terminal100may display, as another example, only some results (e.g., the result of the last action) of executing the plurality of actions on the display.

According to an embodiment, the client module151may receive a request to obtain information necessary for calculating a result corresponding to the voice input from the intelligent server200. According to an embodiment, the client module151may transmit the necessary information to the intelligent server200in response to the request.

The client module151may transmit information on the results of executing the plurality of actions according to the plan to the intelligent server200. The intelligent server200may confirm that the received voice input has been correctly processed using the information on the results.

The client module151may include a speech recognition module. According to an embodiment, the client module151may recognize a voice input for performing a limited function through the speech recognition module. For example, the client module151may execute an intelligent app for processing a voice input to perform an organic operation through a designated input (e.g., Wake up!).

The intelligent server200may receive information related to a user voice input from the user terminal100through a communication network. According to an embodiment, the intelligent server200may change data related to the received voice input into text data. According to an embodiment, the intelligent server200may generate a plan for performing a task corresponding to the user voice input based on the text data. The plan may also be referred to as a scenario herein.

According to an embodiment, the plan may be generated by an artificial intelligence (AI) system. The AI system may be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)). The AI system may be a combination thereof or other AI systems. According to an embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI system may select at least one plan among the predefined plans.

The intelligent server200may transmit a result according to a generated plan to the user terminal100or transmit the generated plan to the user terminal100. According to an embodiment, the user terminal100may display a result according to the plan on the display. According to an embodiment, the user terminal100may display the result of executing an action according to the plan on the display.

The intelligent server200may include a front end210, a natural language platform220, a capsule database (DB)230, an execution engine240, an end user interface250, a management platform260, a big data platform270, and/or an analytic platform280, each of which may include various processing circuitry and/or executable program instructions.

The front end210may receive a voice input received from the user terminal100. The front end210may transmit a response corresponding to the voice input. The front end210may include, for example, a communication module including various communication circuitry for establishing wireless communication with the user terminal.

The communication module may support establishing a direct (e.g., wired) communication channel or wireless communication channel between an electronic device and a user terminal, an external device, or a server, and performing communication through an established communication channel. The communication module may include one or more communication processors that operate independently from a processor (e.g., an application processor) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication (PLC) module). A corresponding communication module among these communication modules may communicate with an external user terminal and/or an external device through a first network (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, Internet, or a computer network (e.g., a local area network (LAN) or a wide area network (WAN))). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., a plurality of chips).

According to an embodiment, the natural language platform220may include an automatic speech recognition (ASR) module221, a natural language understanding (NLU) module223, a planner module225, a natural language generator (NLG) module227, and/or a text to speech (TTS) module229, each of which may include various processing circuitry and/or executable program instructions.

The ASR module221may convert a voice input received from the user terminal100into text data. The NLU module223may discern the intent of the user using the text data of the voice input. For example, the NLU module223may discern the intent of the user by performing syntactic analysis or semantic analysis. The NLU module223may discern the meaning of a word extracted from the voice input using a linguistic feature (e.g., a grammatical element) of a morpheme or phrase, and determine the intent of the user by matching the discerned meaning of the word to an intent.

The NLG module227may change designated information into text form. The information changed to text form may be in the form of a natural language utterance. The TTS module229may change the information in text form into information in speech form.

According to an embodiment, some functions or all functions of the natural language platform220may also be implemented in the user terminal100.

The capsule DB230may store information on the relationship between the plurality of concepts and actions corresponding to the plurality of domains. A capsule according to an embodiment may include a plurality of action objects (or action information) and concept objects (or concept information) included in the plan. According to an embodiment, the capsule DB230may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment, the plurality of capsules may be stored in a function registry included in the capsule DB230.

The capsule DB230may include a strategy registry that stores strategy information necessary for determining a plan corresponding to a voice input. The strategy information may include reference information for determining one plan when there are a plurality of plans corresponding to the voice input. According to an embodiment, the capsule DB230may include a follow-up registry that stores information on follow-up actions for suggesting a follow-up action to the user in a designated situation. The follow-up action may include, for example, a follow-up utterance. According to an embodiment, the capsule DB230may include a layout registry that stores layout information of information output through the user terminal100. According to an embodiment, the capsule DB230may include a vocabulary registry that stores vocabulary information included in capsule information. According to an embodiment, the capsule DB230may include a dialog registry that stores information on a dialog (or an interaction) with the user. The capsule DB230may update the stored objects through a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating the vocabulary. The developer tool may include a strategy editor for generating and registering a strategy for determining a plan. The developer tool may include a dialog editor for generating a dialog with the user. The developer tool may include a follow-up editor for activating a follow-up objective and editing a follow-up utterance that provides a hint. The follow-up objective may be determined based on a current set objective, a preference of the user, or an environmental condition. In an embodiment, the capsule DB230may also be implemented in the user terminal100.

The execution engine240may calculate a result using the generated plan. The end user interface250may transmit the calculated result to the user terminal100. Accordingly, the user terminal100may receive the result and provide the received result to the user. The management platform260may manage information used in the intelligent server200. The big data platform270may collect data of the user. The analytic platform280may manage the quality of service (QoS) of the intelligent server200. For example, the analytic platform280may manage the components and processing rate (or efficiency) of the intelligent server200.

The service server300may provide a service (e.g., food order or hotel reservation) designated to the user terminal100. According to an embodiment, the service server300may be a server operated by a third party. The service server300may provide information to be used for generating a plan corresponding to the received voice input to the intelligent server200. The provided information may be stored in the capsule DB230. In addition, the service server300may provide result information according to the plan to the intelligent server200.

In the integrated intelligence system10described above, the user terminal100may provide various intelligent services to the user in response to a user input. The user input may include, for example, an input through a physical button, a touch input, or a voice input.

In an embodiment, the user terminal100may provide a speech recognition service through an intelligent app (or a speech recognition app) stored therein. In this case, for example, the user terminal100may recognize a user utterance or a voice input received through the microphone and provide a service corresponding to the recognized voice input to the user.

In an embodiment, the user terminal100may perform a designated action alone or with the intelligent server and/or the service server based on the received voice input. For example, the user terminal100may execute an app corresponding to the received voice input and perform a designated action through the executed app.

In an embodiment, when the user terminal100provides a service with the intelligent server200and/or the service server, the user terminal may detect a user utterance using the microphone120and generate a signal (or voice data) corresponding to the detected user utterance. The user terminal may transmit the voice data to the intelligent server200using the communication interface110.

The intelligent server200according to an embodiment may generate a plan for performing a task corresponding to the voice input or a result of performing an action according to the plan in response to the voice input received from the user terminal100. The plan may include, for example, a plurality of actions for performing a task corresponding to a voice input of the user, and a plurality of concepts related to the plurality of actions. The concepts may define parameters input for the execution of the plurality of actions or result values output by the execution of the plurality of actions. The plan may include connection information between the plurality of actions and the plurality of concepts.

The user terminal100may receive the response using the communication interface110. The user terminal100may output a voice signal generated in the user terminal100to the outside using the speaker130or may output an image generated in the user terminal100to the outside using the display140.

FIG.2is a diagram illustrating a form in which relationship information between a concept and an action is stored in a database according to various embodiments.

A capsule DB (e.g., the capsule DB230) of the intelligent server200may store capsules in the form of a concept action network (CAN). The capsule DB may store an action for processing a task corresponding to a voice input of a user and a parameter necessary for the action in the form of a CAN.

The capsule DB may store a plurality of capsules (capsule A401, capsule B404) corresponding to each of a plurality of domains (e.g., an application) in the concept action network400. According to an embodiment, one capsule (e.g., the capsule A401) may correspond to one domain (e.g., a location (geo) or an application). In addition, one capsule may correspond to at least one service provider (e.g., CP1402or CP2403) for performing a function for a domain related to the capsule. According to an embodiment, one capsule may include at least one or more actions4100and at least one or more concepts4200for performing a designated function.

The natural language platform220may generate a plan for performing a task corresponding to the received voice input using the capsules stored in the capsule DB. For example, the planner module225of the natural language platform220may generate the plan using the capsules stored in the capsule DB. For example, a plan407may be generated using actions4011and4013and concepts4012and4014of a capsule A401and an action4041and a concept4042of a capsule B404. The natural language platform220may be included in a voice assistant.

FIG.3is a diagram illustrating a user terminal displaying a screen for processing voice input received through an intelligent app according to various embodiments.

The user terminal100may execute an intelligent app to process a user input through the intelligent server200.

According to an embodiment, when the user terminal100recognizes a designated voice input (e.g., Wake up!) or receives input through a hardware key (e.g., a dedicated hardware key), an intelligent app for processing the voice input may be executed on the screen310. For example, the user terminal100may execute the intelligent app in a state of executing a scheduling app. According to an embodiment, the user terminal100may display an object311(e.g., an icon) corresponding to the intelligent app on the display140. According to an embodiment, the user terminal100may receive a voice input by a user utterance. For example, the user terminal100may receive a voice input of “Tell me the schedule for this week.” According to an embodiment, the user terminal100may display a user interface (UI)313(e.g., an input window) of the intelligent app in which text data of the received voice input is displayed on the display.

According to an embodiment, the user terminal100may display on a screen320a result corresponding to the received voice input to the display. For example, the user terminal100may receive a plan corresponding to the received user input and display a “The schedule for this week is” on the display according to the plan.

The electronic devices according to various example embodiments may be various types of electronic devices. The electronic devices may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to an embodiment is not limited to the above-described devices, and the electronic device is mainly described as an intelligent server (e.g., the intelligent server200ofFIG.1).

FIG.4is a block diagram illustrating an example configuration of an electronic device according to various embodiments.

An electronic device410according to an embodiment may be implemented as an intelligent server (e.g., the intelligent server200ofFIG.1). For example, the electronic device410may include an utterance classifier411(e.g., the speech recognition module221and the natural language understanding module223ofFIG.1), a situation classifier412, a scenario generator413(e.g., the planner module225inFIG.1), an Internet of things (IoT) device cluster414, a user IoT device provider415, and a response generator416(e.g., the natural language generator module227inFIG.1), each of which may include various processing circuitry and/or executable program instructions. The utterance classifier411, the situation classifier412, the scenario generator413, the IoT device cluster414, the user IoT device provider415and the response generator416may be a module, but each action is not limited to being performed by the module. Each action may be performed by a processor of the electronic device410. The utterance classifier411and the situation classifier412may be individually implemented as a machine learning model but are not limited thereto.

The electronic device410may receive utterance data of a user490from a user terminal using the communication module. The utterance data is data representing an utterance of the user490and may be, for example, data including a voice signal corresponding to the utterance. However, the utterance data is not limited thereto and may include text data indicating the utterance content of the user490converted by the ASR module221described inFIG.2.

The utterance classifier411may classify the intent of the utterance of the user490from the utterance data. For example, a processor of the electronic device410may determine the intent of the utterance of the user490from the utterance data and identify the control target device intended from the utterance data. The control target device may represent a device that the user490intends to control through the utterance. The intent of the utterance may represent an action and/or a function that the user490intends to control through the utterance, and further, the intent of the utterance may also include indicating the control target device. As will be described in greater detail below, in response to a case in which the intent of the utterance of the user490and the control target device are identified, the electronic device410may generate a scenario (e.g., a plan) for executing the control target device in a plurality of actions determined by the intent of the user490through the scenario generator413as described inFIG.2. The electronic device410may control an external device440through a device dispatcher421of an IoT server420when a discerned utterance intent of the user490is a command for controlling a device. The device dispatcher421may manage and control the external device440. For example, the electronic device410may receive utterance data including “Turn on the air conditioner” from the user terminal, identify the air conditioner as a control target device from the utterance data, and identify activation of the air conditioner as the intent of the utterance. The electronic device410may perform a scenario of activating the identified air conditioner based on the utterance data. The external device440is a control target device according to an action scenario determined based on an analysis of the utterance, and may include, for example, a home appliance and an IoT device.

In the electronic device410, when the user490provides an utterance with unclear intent, the utterance classifier411may fail to classify intent from the utterance data. For another example, even though the intent of the utterance is clear, the electronic device410may fail to identify a control target device. The electronic device410according to an embodiment may attempt to interpret the utterance data in another aspect through the situation classifier412.

In addition, the electronic device410may attempt to interpret the utterance data in another aspect through the situation classifier412regardless of the classification of the intent. For example, when a control target device and the intent of the utterance to control the control target device are identified from the utterance data, the electronic device410may search for a scenario that matches the identified control target device and the intent of the utterance. The electronic device410may further provide an additional scenario (e.g., an action scenario) determined based on the situation factor classified as described later, while providing a scenario that matches the control target device and the intent of the utterance.

The situation classifier412may classify a situation indicated by utterance data even when the intent of the utterance of the user490and a control target device are not identified. For example, the electronic device410, in response to a case in which at least one of the intent of the utterance and a control target device is not identified from the utterance data, may classify a situation factor for the user490based on the utterance data. The situation factor may represent the factor that caused the situation of the user490who provided the utterance data. Herein, a negative situation factor is mainly described as an example of a situation factor. For example, the negative situation factor may represent a factor that caused the negative situation of the user490who provided the above-described utterance data. The electronic device410may indirectly estimate the intent of the utterance of the user490implied in the utterance data by inferring the situation factor from the utterance data. Table 1 below describes examples of situation factors.

The situation factor may represent a factor that causes a situation (e.g., a negative situation) of the user in terms of including the body, mind, and emotion of the user490. In Table 1, the situation factor (e.g., a negative situation factor) may include, for example, a weather factor, a mood factor, and a cleanliness factor. The weather factor is a weather condition that causes a situation of the user490and may include, for example, hot weather and cold weather. However, the weather factor is not limited thereto, and may include wet weather and dry weather. The mood factor is a mood state that causes a situation of the user490and may include, for example, bored and annoyed. However, the mood factor is not limited thereto, and may include an emotional state including anger, sadness, anxiety, hurt, and embarrassment. The cleanliness factor is a cleanliness state that causes a situation of the user490and may include a dirty state of the surrounding environment of the user490, a dirty state of the user490, and a dirty state of a person and an object (e.g., an animal). However, the situation factor is not limited to Table 1 above, and the negative situation factor may include a stress factor, and the situation factor may include not only negative situation factors but also various situation factors, and may vary depending on the design.

The electronic device410may classify the situation factor from the utterance data into at least one of a weather factor, a mood factor, and a cleanliness factor that may cause a situation of the user490. Herein, an example of classifying a single situation factor from one utterance data is mainly described, but the present disclosure is not limited thereto, and the electronic device410may classify a plurality of situation factors from the utterance data. The electronic device410may interpret a complex situation from the utterance data. The situation classifier412may analyze the utterance of the user490with a classification criterion different from that of the utterance classifier411. The electronic device410may discern a mood or feeling of the user490from an utterance text of the user490through a classification of the above-described situation factors. The electronic device410may classify a current state of the user490based on the mood implied in the utterance and/or the situation of the user490rather than a domain, an intent, and a parameter value of the user490. For example, the electronic device410may receive utterance data lacking a clear utterance intent, such as “I'm frustrated” from the user terminal. The electronic device410may classify an anger factor (e.g., frustrated) as a situation factor from utterance data of “I'm frustrated.”

The scenario generator413may generate a scenario based on the situation factor classified by the situation classifier412described above. For example, the scenario generator413may identify a candidate device group capable of supporting a currently classified situation factor through the IoT device cluster414. In addition, the scenario generator413may obtain a list of devices registered in the user terminal and a state of the registered device through the user IoT device provider415. The scenario generator413may identify the external device440available for the user terminal among the supportable candidate device groups.

For example, the electronic device410may determine one or more external devices440that match the classified situation factor. The electronic device410may identify a candidate device group capable of supporting the classified situation factor. The electronic device410may search for the candidate device group that matches the situation factor classified from the utterance data based on the IoT information422registered in the IoT server420. The candidate device group will be described in greater detail below with reference toFIG.5. The electronic device410may transmit the classified situation factor to the IoT server420and request a reply from the candidate device group capable of supporting the situation factor. However, it is not limited thereto, and the electronic device410may receive IoT information422from the IoT server420and search for a type of device belonging to the candidate device group capable of supporting the situation factor among the received IoT information422. The IoT information422may include information about a device (e.g., the IoT device) having an action and/or a function for assisting the user. The external device may be classified according to device types and the device type may be clustered into the candidate device group. The device type and the candidate device group is described in greater detail below with reference toFIG.5.

The electronic device410may determine an external device440available to the user490among candidate device groups. For example, the electronic device410may search for a device registered in the user terminal among devices belonging to the candidate device group. The electronic device410may determine a device capable of controlling an action and/or a function of the corresponding device among devices registered in the user terminal as an available external device440. For example, the available external device440may include a device having a current connection with the user terminal, a turned-on device, and a turn-onable device among devices registered in the user terminal. The electronic device410may attempt user authentication of the user terminal through a user authentication server430. When the electronic device410succeeds in the user authentication requested by the user terminal, a user token431(e.g., a unique key) corresponding to the user490may be obtained. The electronic device410may obtain information for a device registered in the user terminal using the unique key of the user490. The electronic device410may receive user IoT information423registered in a user account for which user authentication is successful from the IoT server420. The user IoT information423may include a list of devices registered in the user account and a state of the registered devices. The state of the device may include whether the device is powered and an action mode. For reference, the device registered in the user terminal may be, for example, a device registered to be accessible from a voice assistant of the user terminal.

The electronic device410may generate one or more action scenarios for one or more external devices440determined for the classified situation factor. The electronic device410may present one or more generated action scenarios to the user terminal. For example, the scenario generator413may determine an action scenario executable for a state of the external device440available in the user terminal among predetermined candidate scenarios for the candidate device group that matches the situation factor classified from the situation classifier412. The electronic device410may determine one of a plurality of candidate scenarios or a combined scenario of two or more as an action scenario. The scenario generator413may provide a generated action scenario to the response generator416. The response generator416may transmit response data according to the generated scenario to the user terminal. The response data may include content (e.g., text, images, and/or voice) that asks a user about the execution of the generated action scenario.

In the above-described example, when the situation factor is classified as an anger factor (e.g., “Frustrated”), the electronic device410may obtain a recommendable device candidate group for the situation caused by the anger factor from IoT device cluster414. The electronic device410may generate an action scenario in which an air conditioner is operated in a clean mode and classical music is played in an audio device based on a state of the external device440available to the user terminal among the device candidate groups. The electronic device410may transmit response data including content that inquires and suggests, “Would you like to change the air conditioner to clean mode and play classical music?” to the user terminal. The user terminal may visually, audibly, and/or tactilely output the response data received from the electronic device410.

As another example, the electronic device410may receive utterance data including “Oh, it's too hot,” “I'm sweating,” “I hate summer,” “It's sticky,” or “I hope winter comes soon” from the user terminal. As described above in Table 1, the electronic device410may classify the situation factor as a weather factor such as hot weather. The electronic device410may select a temperature control device group as a device candidate group capable of supporting hot weather. The electronic device410may determine the air conditioner as an external device440available to the user terminal among the temperature control device groups. The electronic device410may generate an action scenario for turning on the air conditioner. In order to recommend execution of the above-described action scenario, the electronic device410may transmit response data asking “The weather is too hot. Should I turn on the air conditioner?” to the user terminal.

Accordingly, the electronic device410according to an embodiment may indirectly discern the intent of the utterance of the user490through classification of the above-described situation factor even when the intent of the utterance of the user490is not discerned. For example, the utterance data of “Why is the house so messy?” may imply that the user490wants to clean the house and is in a bad mood because the current state of the house of the user490is not organized. The electronic device410may discern the current mood or feeling of the user490and recommend and/or suggest an action scenario of devices (e.g., IoT devices as external devices440) connected to a voice assistant executed on the user terminal, even when the user490does not make a clear utterance about device control. In the above-described example, “robot cleaner” and “air purifier” may be registered and connected to the voice assistant of the user490. The electronic device410may classify the situation factor as a cleanliness factor indicating dirty from the utterance data of “Why is the house so messy?” of the user490. The electronic device410may suggest response data to the user terminal of “Do you want to clean the surroundings by operating the robot cleaner and refresh your mood with the air purifier?” with respect to the cleanliness factor. When the suggested response data is approved by the user terminal (e.g., when the user490commands execution), the electronic device410may instruct an action of the robot cleaner and the air purifier via the IoT server420described above. Accordingly, the electronic device410may provide the user490with assistance from various devices through the voice assistant in various situations without certain device control utterance. For example, the electronic device410may recommend and present a predefined (e.g., specified) function to the user490based on the classification criterion according to the situation factor.

FIG.5is a diagram illustrating an example candidate device group capable of supporting a classified situation factor according to various embodiments.

The IoT information422may include a device type of a released device (e.g., a home appliance). For example, the IoT information422may include a device type designated and mapped to the device according to an action and/or a function of the device by each device and the service provider. The IoT information422may be continuously updated by the service provider. InFIG.5, a list520of device types in the IoT information422may include, for example, an air purifier type, a TV type, a fan type, a speaker device type, an air conditioner type, a lighting device type, a robot cleaner type, and an air dresser device type.

The IoT device cluster414may include information on a candidate device group580and classify and manage a situation factor590mapped to each device of the list520obtained from the IoT information422. The device type may be clustered into the candidate device group580according to a function that the device belonging to the device type has in common and a situation factor that may be removed by the function. The electronic device may manage the candidate device group580in which the device type is clustered for each situation factor. In the example shown inFIG.5, the candidate device group580may be categorized into a temperature control device group511, a sound and image device group512, and a laundry and cleaning device group513. The electronic device may map, in each candidate device group580, a situation factor (e.g., a negative situation factor that may be removed) that may be processed by a common action and/or function of devices of the device type belonging to the group. For example, the candidate device group580may be a candidate device group having an action and/or a function capable of converting a situation (e.g., a negative situation as a first situation) to another situation (e.g., a positive situation as a second situation) according to the situation factor mapped to the group. In the present disclosure, the candidate device is mainly described as a home appliance that assists the user life, but is not limited thereto. The candidate device may be a device having an action and/or a function capable of converting a situation of the user into another situation. The mapping between the candidate device group580and the situation factor, for example, may be generated and managed by a service provider. In the example shown inFIG.5, the IoT device cluster414may map the temperature control device group511including a fan type (e.g., FanDeviceType) and an air conditioner type (e.g., AirConditionerDeviceType) to a weather factor (e.g., “Weather-HOT” and “Weather-COLD”) among the situation factors.

For reference, the IoT server may update the IoT information422when a new product (e.g., a new home appliance) is released. The electronic device may receive an update on the new product from the IoT server. The electronic device may identify a device type of the updated new product and map the candidate device group580to which the device type belongs. Accordingly, even when a new product is released, the electronic device may identify the device type of the product, and immediately map and manage the candidate device group580to which the identified device type belongs.

According to an embodiment, the electronic device may determine one or more devices capable of converting a situation caused by the classified situation factor into another situation as one or more external devices that match the classified situation factor.

For example, the one or more external devices may include one or a combination of two or more of a temperature control device, an audio device, a video device, a lighting device, an air purifier, a home appliance for laundry, and a cleaner. The temperature control device is a device capable of controlling a temperature and may include an air conditioner and a heater. The audio device is a device that outputs audio and may include a CD player. The video device is a device that outputs video and may include a DVD player. The lighting device is a device that outputs light and may include a device having a light emitting diode (LED) module. The air purifier may be a device having a function of purifying air. The home appliance for laundry may include a washing machine (e.g., a washer), a drying machine (e.g., a dryer), and a clothing care machine (e.g., an air dresser). The cleaner may include a robot cleaner. In the above-described example, the temperature control device group511may include a temperature control device. The sound and image device group512may include an audio device and a video device. The laundry and cleaning device group513may include an air purifier, a home appliance for laundry, and a cleaner.

According to an embodiment, the electronic device may select the candidate device group580capable of supporting the situation factor classified from utterance data as described above. The electronic device may select one external device among the selected device groups.

For example, when the classified situation factor includes a weather factor, the electronic device may select the temperature control device group511as a candidate device group580capable of supporting the weather factor. The electronic device may determine the temperature control device belonging to the temperature control device group511capable of supporting the weather factor as one or more external devices. As another example, when the classified situation factor includes a mood factor, the electronic device may select the sound and image device group512as a candidate device group580capable of supporting the mood factor. The electronic device may determine one or a combination of two or more of an audio devices, a video device, and a lighting device as one or more external devices in the sound and image device group512. For another example, when the classified situation factor includes a cleanliness factor, the electronic device may select the laundry and cleaning device group513as a candidate device group580capable of supporting the cleanliness factor. The electronic device may determine one or a combination of two or more of an air purifier, a home appliance for laundry, and a cleaner as one or more external devices in the laundry and cleaning device group513.

FIG.6is a diagram illustrating generation of an action scenario based on a situation factor classified in the electronic device, a candidate device group, and a device state according to various embodiments.

The electronic device (e.g., the intelligent server200ofFIG.1) according to an embodiment may search to determine whether there is an available device610connected to the user terminal while belonging to the selected group. The available device610is a device capable of performing an action required to convert a situation (e.g., a negative situation) according to the situation factor to another situation (e.g., a positive situation), and may include, for example, a device registered in the user account, a device around the user, and a device around a location where the required action should be executed. For example, the situation factor classified in the example shown inFIG.6may be “dirty” as a cleanliness factor. The electronic device may select the laundry and cleaning device group513mapped to “dirty” as a device candidate group. The electronic device may determine whether a device belonging to the laundry and cleaning device group513is connected to the user terminal and whether the device is available. The electronic device may search for a device belonging to the identified device candidate group among devices registered in the user account. For example, the scenario generator413may recognize an air purifier type device and a robot cleaner type device belonging to the laundry and cleaning device group513among devices registered in the user account. In addition, the scenario generator413may collect a device in which a state of the available device610registered in the user account is in an inactive state (e.g., an Off state). When an available device610is found, the electronic device may determine the available device610as an external device. The scenario generator413may determine the air purifier and the robot cleaner as external devices to be included in the generation of the action scenario630.

The scenario generator413may collect a situation factor classified from utterance data of the user, a matched device candidate group, and an external device of the user registered in the voice assistant, and the state of the device. The scenario generator413may generate a device control scenario to be suggested to the user based on the collected information. InFIG.6, as described previously, the electronic device may classify the cleanliness factor as a situation factor from the utterance data expressing the intent of “dirt” (e.g., “How do I clean all this up?”), and determine an air purifier and a robot cleaner as external devices among device candidate groups mapped to the cleanliness factor.

According to an embodiment, the electronic device may present an action scenario630corresponding to a state of one or more external devices among predetermined candidate scenarios620for one or more external devices matched with the situation factor.

The electronic device may store a predefined action for each situation factor and device type classified by the scenario generator413. For example, when the external device matched to the classified situation factor and registered in the user terminal is turned off, the electronic device may generate an action scenario630for turning on the external device. However, the turn-on state and the turn-off state have been mainly described as states of the external device for convenience of description herein but are not limited thereto. Candidate scenarios may be prepared for a power state (e.g., a battery state), defective state (e.g., a normal operation or not) and various states of the external device.

For example, when the classified situation factor includes a cleanliness factor and the air purifier is available on the user terminal, the electronic device may instruct the air purifier to perform an air purifying action. In the example shown inFIG.6, the candidate scenario620predefined for “dirty” as a cleanliness factor among the situation factors may include an action of turning on the air purifier in a turned-off state.

Similarly, when the classified situation factor includes the cleanliness factor and the cleaning device is available on the user terminal, the electronic device may instruct the cleaning device to perform a cleaning action. The candidate scenario620predefined for the cleanliness factor may include an action of turning on the robot cleaner in a turned-off state.

The candidate scenario620predefined for a case in which the air purifier and the robot cleaner are already turned on may include not performing an action even though the cleanliness factor is identified as a situation factor.

The candidate scenario620predefined for a mood factor (e.g., bored) as a situation factor may define an action of changing a channel of the video device (e.g., a TV) in a turn-on state and an action of turning on the video device in a turn-off state, respectively. The electronic device may generate an action scenario630using one of a plurality of candidate scenarios620or a combined scenario of two or more thereof.

In the example shown inFIG.6, since the air purifier and the robot cleaner are turned off as available devices, the electronic device may generate an action scenario630including an action of turning on the air purifier and an action of turning on the robot cleaner.

However, the candidate scenario620and the action scenario630described inFIG.6are examples and are not limited thereto. For example, when the classified situation factor includes a mood factor and at least one of the audio and video devices is available in the user terminal, the electronic device may generate a scenario indicating a change (e.g., a change in a TV channel) of content played on at least one device. In addition to the description, the candidate scenario620may be prepared according to another situation factor, a type of available external device, and a state of available external devices, and the electronic device may select and combine the prepared candidate scenarios to generate an action scenario630. The presentation of the generated action scenario630is described inFIG.7in the following.

FIGS.7and8are diagrams illustrating example action scenarios generated according to various embodiments.

According to an embodiment, when a plurality of available external devices is matched with respect to the classified situation factor, the electronic device (e.g., the intelligent server200ofFIG.1) may present one or two more action scenarios of a scenario of operating all of a plurality of available external devices, a scenario of operating only some of the plurality of available external devices, and a scenario of not operating all external devices.

For example, inFIG.7, the electronic device may classify a mood factor as a situation factor from utterance data710of “I'm so frustrated.” The electronic device may determine an air conditioner and an audio device as available external devices that match the mood factor. The electronic device may present a scenario731of operating both the air conditioner and the audio device, a scenario732of operating only the air conditioner, a scenario733of operating only the audio device, and a scenario734of skipping an action of the external device to the user terminal. The user terminal may output an action scenario received from the electronic device with the available external device for the classified situation factor and response data720for guiding the performable action of the external device.

As another example, inFIG.8, the electronic device may classify a cleanliness factor as a situation factor from utterance data810of “The house is a mess.” The electronic device may determine the robot cleaner and the air purifier as available external devices that match the cleanliness factor. The electronic device may present a scenario831of operating both the robot cleaner and the air purifier, a scenario832of operating only the robot cleaner, a scenario833of operating only the air purifier, and a scenario834of skipping the action of external devices to the user terminal. The user terminal may output an action scenario received from the electronic device with the available external device for the classified situation factor and response data820for guiding the performable action of the external device.

According to an embodiment, the electronic device may transmit an instruction to perform an action according to a scenario selected on the user terminal among one or more generated action scenarios to the external device registered in the user terminal. For example, the electronic device may receive a selection of one action scenario among one or more action scenarios suggested from the user terminal. The electronic device may identify the external device corresponding to the selected action scenario and instruct each external device to perform an action included in the action scenario via the IoT server (e.g., the IoT server420ofFIG.4).

FIGS.9and10are flowcharts illustrating an example method of controlling an external device according to various embodiments.

In operation910, the electronic device (e.g., the intelligent server200ofFIG.1) may receive utterance data of the user from the user terminal using the communication module.

In operation920, the electronic device may classify a situation factor for the user based on the utterance data. For example, in response to a case in which at least one of the utterance intent the intent of the utterance and the control target device is not identified from the utterance data, the electronic device may classify the above-described situation factor from the utterance data. For example, in operation1021, the electronic device may determine whether the utterance data is an utterance for which the intent may not be discerned. In operation1022, when it is impossible to discern the intent, the electronic device may attempt to identify a situation factor that matches the utterance. In operation1080, when the identification of the situation factor fails, the electronic device may terminate the action for controlling the external device while outputting a voice guiding the action failure (e.g., “I am unable to support this action.”).

In operation930, the electronic device may determine one or more external devices that match the classified situation factor. For example, in operation1031, when a situation factor is identified from the utterance data, the electronic device may determine whether there is a candidate device group capable of supporting the identified situation factor. In operation1032, when there is a supportable candidate device group, the electronic device may determine whether there is an external device available in the user terminal. For example, the electronic device may determine whether there is a device registered in the user account of the user terminal and whether the device is accessible. In operation1080, when the search for a device candidate group capable of converting a first situation (e.g., a negative situation) into a second situation (e.g., a positive situation) different from the first situation fails or there is no available external device in the user account, the electronic device may terminate the action for controlling the external device.

In operation940, the electronic device may generate and present one or more action scenarios for one or more external devices determined for the classified situation factors to the user terminal. For example, in operation1041, when there is an external device available in the user terminal for the situation factor, the electronic device may generate an action scenario. As described above inFIG.6, the electronic device may generate an action scenario by selecting and/or combining a matching scenario among candidate scenarios prepared for each situation factor, device type, and device state. In operation1090, the electronic device may transmit response data to the user terminal asking whether to perform the generated action scenario. The user terminal may output the response data received from the electronic device.

The actions described inFIGS.9and10may be performed by a processor of the electronic device (e.g., the intelligent server400ofFIG.4) and are not limited to those performed in the above-described order. The actions described inFIGS.9and10may be performed in parallel or in time series with other actions and may be performed in an order different from the described order unless it is contrary to the description. In addition, the actions may be performed with the actions described inFIGS.1to8.