Electronic device performing operation based on user speech in multi device environment and operating method thereof

According to various embodiments, an electronic device performing an operation corresponding to an utterance of a user in a multi-device environment includes a communication module, a processor, and a memory. The processor operates a first electronic device for receiving and a second electronic device for executing in association with each other. The memory stores instructions that, when being executed, cause the processor to: determine whether a first utterance received at the first electronic device should be performed at the second electronic device; when the first utterance should be performed at the second electronic device, form a connection binding with the first electronic device and the second electronic device by using one or more channels through the communication module. When an operation corresponding to the first utterance is completed at the second electronic device, the processor terminates the connection binding with the first electronic device and the second electronic device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2019-0160994 filed on Dec. 5, 2019 and Korean Patent Application No. 10-2020-0084246 filed on Jul. 8, 2020 in the Korean Intellectual Property Office, the disclosures of which are herein incorporated by reference in their entirety.

BACKGROUND

Various embodiments of the disclosure relate to an electronic device which recognizes a user utterance in a multi-device environment and performs an operation corresponding to the user utterance, and an operating method thereof.

2. Description of Related Art

With the recent development of voice recognition technology, a voice recognition function may be implemented in various electronic devices provided with a voice input device (for example, a microphone). An intelligent assistance service for providing an intuitive interface between electronic devices is developing. The intelligent assistance service may infer an intent of a user by performing natural language processing with respect to an utterance of the user, and may process to control a control device based on the inferred intent of the user. For example, an electronic device may recognize an utterance of a user through a voice recognition function and may understand an intent of the user utterance. Thereafter, the electronic device may perform an operation corresponding to the user utterance. In particular, in a multi-device environment, a plurality of receivers and a plurality of executors should organically exchange information with each other, and operations corresponding to utterance should be performed without interruption even when a receiver and/or executor is changed.

The above information is presented as background information only to assist with an understanding of the disclosure.

SUMMARY

Various embodiments of the disclosure provide an electronic device which can process an utterance without interruption in a multi-device environment, and an operating method thereof.

Various embodiments of the disclosure provide an electronic device which can display a resulting response to an utterance according to a form of a user interface (UI) of a receiver and an executor, and corrects the response to the utterance and transmits the response to the receiver, and an operating method thereof.

Various embodiments of the disclosure provide an electronic device which can cause a fixed executor to process an operation corresponding to an utterance, while changing a receiver, and can process an operation corresponding to an utterance while changing an executor, and an operating method thereof.

Various embodiments of the disclosure provide an electronic device which can maintain and terminate a connection between a receiver and an executor, based on a connection maintaining time, a follow-up utterance waiting time, and an additional information input waiting time, and an operating method thereof.

According to various embodiments of the disclosure, an electronic device performing an operation corresponding to an utterance of a user in a multi-device environment may include a communication module, a processor, and a memory. The processor may be operatively connected with the communication module and may operate a first electronic device for receiving and a second electronic device for executing in association with each other. The memory may be operatively connected with the processor. The memory may store instructions that, when being executed, cause the processor to: determine whether a first utterance received at the first electronic device should be performed at the second electronic device; when the first utterance should be performed at the second electronic device, form a connection binding with the first electronic device and the second electronic device by using one or more channels through the communication module, and, when an operation corresponding to the first utterance is completed at the second electronic device, terminate the connection binding with the first electronic device and the second electronic device.

According to various embodiments of the disclosure, an operating method of an electronic device which operates a first electronic device for receiving and a second electronic device for executing in association with each other may include: determining whether a first utterance received at the first electronic device should be performed at the second electronic device; when the first utterance should be performed at the second electronic device, forming a connection binding with the first electronic device and the second electronic device by using one or more channels; and, when an operation corresponding to the first utterance is completed at the second electronic device, terminating the connection binding with the first electronic device and the second electronic device.

According to various embodiments of the disclosure, an electronic device performing an operation corresponding to an utterance of a user in a multi-device environment may include: a communication module; a processor operatively connected with the communication module and configured to operate a first electronic device for receiving and a second electronic device for executing in association with each other; and a memory operatively connected with the processor. The memory may store instructions that, when being executed, cause the processor to: generate a first channel with the first electronic device which receives the utterance; receive information of the utterance through the first channel; determine whether an action corresponding to the utterance should be executed at the second electronic device, based on the information of the utterance; generate a second channel with the second electronic device; request the second electronic device to execute the action corresponding to the utterance through the second channel; receive a result of executing the action corresponding to the utterance from the second electronic device; and transmit the result of executing the action corresponding to the utterance to the first electronic device.

According to various embodiments of the disclosure, an electronic device for receiving an utterance of a user in a multi-device environment may include: a communication module; a processor operatively connected with the communication module and configured to operate the first electronic device for receiving and a second electronic device for executing in association with each other; and a memory operatively connected with the processor. The memory may store instructions that, when being executed, cause the processor to: when the utterance is received, generate at least one channel and connect with the second electronic device; display the connection with the second electronic device through a UI; request the second electronic device to execute an action corresponding to the utterance through the at least one channel; receive a result of executing the action corresponding to the utterance from the second electronic device; and display the result of executing the action corresponding to the utterance through the UI.

With regard to the description of the drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

DETAILED DESCRIPTION

Hereinafter, various embodiments of the disclosure will be described with reference to the accompanying drawings. Components illustrated in the drawings may have their sizes exaggerated or reduced for convenience of explanation, and the disclosure is not necessarily limited to those illustrated in the drawings.

FIG.1illustrates a block diagram of an example electronic device101in a network environment100according to various embodiments;

The input device150may receive a command or data to be used by another component(s) (e.g., the processor120) of the electronic device101, from the outside (e.g., a user) of the electronic device101. The input device150may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

FIG.2illustrates a block diagram of an example integrated intelligence system according to an embodiment.

Referring toFIG.2, the integrated intelligence system according to an embodiment may include an intelligent server200, a service server300, and a user terminal400.

The user terminal400(for example, the electronic device101ofFIG.1) in an embodiment may be a terminal device (or an electronic device) connectable to the Internet, and for example, may be a mobile phone, a smartphone, a personal digital assistant (PDA), a notebook computer, a TV, a home appliance, a wearable device, a head mounted display (HMD), or a smart speaker.

According to the illustrated embodiment, the user terminal400may include a communication interface410, a microphone420, a speaker430, a display440, a memory450, or a processor460. The above-mentioned components may be operatively or electrically connected with one another.

The communication interface410in an embodiment may be connected with an external device and may be configured to transmit and receive data. The microphone420in an embodiment may receive a sound and may convert the sound into an electric signal. The speaker430in an embodiment may output an electronic signal as a sound (for example, a voice). The display440in an embodiment may be configured to display an image or a video. The display440in an embodiment may also display a graphic user interface (GUI) of an executed application (or an application program).

The memory450in an embodiment may store a client module451, a software development kit (SDK)453, and a plurality of applications455. The client module451and the SDK453may constitute a framework (or a solution program) to perform a generic-purpose function. In addition, the client module451or the SDK453may constitute a framework to process a voice input.

The plurality of applications455in an embodiment may be programs for performing designated functions. According to an embodiment, the plurality of applications455may include a first application455_1, a second application455_2. According to an embodiment, each of the plurality of applications455may include a plurality of operations to perform a designated function. For example, the applications may include an alarm application, a message application, and/or a schedule application. According to an embodiment, the plurality of applications455may be executed by the processor460to execute at least some of the plurality of operations in sequence.

The processor460in an embodiment may control overall operations of the user terminal400. For example, the processor460may be electrically connected with the communication interface410, the microphone420, the speaker430, and the display440to perform designated operations.

The processor460in an embodiment may execute a program stored in the memory450to perform a designated function. For example, the processor460may execute at least one of the client module451or the SDK453to perform the following operation for processing a voice input. The processor460may control, for example, operations of the plurality of applications455through the SDK453. The following operation described as the operation of the client module451or the SDK453may be an operation executed by the processor460.

The client module451in an embodiment may receive a voice input. For example, the client module451may receive a voice signal corresponding to a user utterance detected through the microphone420. The client module451may transmit the received voice input to the intelligent server200(for example, the server108ofFIG.1). The client module451may transmit state information of the user terminal400to the intelligent server200along with the received voice input. The state information may be, for example, execution state information of an application.

The client module451in an embodiment may receive a result corresponding to the received voice input. For example, when the intelligent server200is able to calculate the result corresponding to the received voice input, the client module451may receive the result corresponding to the received voice input. The client module451may display the received result on the display440.

The client module451in an embodiment may receive a plan corresponding to the received voice input. The client module451may display results of executing a plurality of operations of an application according to the plan on the display440. The client module451may display the results of executing the plurality of operations on the display in sequence. In another example, the user terminal400may display only some results (for example, the result of performing the final operation) of executing the plurality of operations on the display.

According to an embodiment, the client module451may receive, from the intelligent server200, a request for obtaining information necessary for calculating the result corresponding to the voice input. According to an embodiment, the client module451may transmit the necessary information to the intelligent server200in response to the request.

The client module451in an embodiment may transmit information on the results of executing the plurality of operations according to the plan to the intelligent server200. The intelligent server200may identify that the received voice input is correctly processed by using the information on the results.

The client module451in an embodiment may include a voice recognition module.

According to an embodiment, the client module451may recognize a voice input to perform a limited function through the voice recognition module. For example, the client module451may execute an intelligent application for processing a voice input for performing an organic operation through a designated input (for example, “Wake up!”).

The intelligent server200(for example, the server108ofFIG.1) in an embodiment may receive information related to a user voice input from the user terminal400(for example, the electronic device101ofFIG.1) through a communication network. According to an embodiment, the intelligent server200may convert data related to the received voice input into text data. According to an embodiment, the intelligent server200may generate a plan to perform a task corresponding to the user voice input, based on the text data.

According to an embodiment, the plan may be generated by an artificial intelligent (AI) system. The AI system may be a rule-based system, or may be a neural network-based system (for example, a feedforward neural network (FNN), a recurrent neural network (RNN)). Alternatively, the AI system may be a combination of the above-mentioned systems or a different AI system. According to an embodiment, the plan may be selected from a set of pre-defined plans, or may be generated in real time in response to a user request. For example, the AI system may select at least one plan from a plurality of pre-defined plans.

The intelligent server200in an embodiment may transmit a result corresponding to the generated plan to the user terminal400or may transmit the generated plan to the user terminal400. According to an embodiment, the user terminal400may display the result corresponding to the plan on the display. According to an embodiment, the user terminal400may display a result of executing an operation corresponding to the plan on the display.

The intelligent server200in an embodiment may include a front end210, a natural language platform220, a capsule database (DB)230, an execution engine240, an end user interface250, a management platform260, a big data platform270, or an analytic platform280.

The front end210in an embodiment may receive a received voice input from the user terminal400. The front end210may transmit a response corresponding to the voice input.

According to an embodiment, the natural language platform220may include an automatic speech recognition (ASR) module221, a natural language understanding (NLU) module223, a planner module225, a natural language generator (NLG) module227, or a text to speech (TTS) module229.

The ASR module221in an embodiment may convert the voice input received from the user terminal400into text data. The NLU module223in an embodiment may understand a user intent by using the text data of the voice input. For example, the NLU module223may understand the user intent by performing syntactic analysis or sematic analysis. The NLU module223in an embodiment may understand a meaning of a word extracted from the voice input by using linguistic features (for example, syntactic elements) of a morpheme or a phrase, and may determine the user intent by matching the understood meaning of the word with the intent.

The planner module225in an embodiment may generate a plan by using the intent determined at the NLU module223and a parameter. According to an embodiment, the planner module225may determine a plurality of domains necessary for performing a task based on the determined intent. The planner module225may determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to an embodiment, the planner module225may determine a parameter necessary for executing the determined plurality of operations, or a result value outputted by execution of the plurality of operations. The parameter and the result value may be defined as a concept of a designated format (or class). Accordingly, the plan may include the plurality of operations determined by the user intent, and the plurality of concepts. The planner module225may determine relationship between the plurality of operations and the plurality of concepts in a stepwise manner (or hierarchically). For example, the planner module225may determine an execution sequence of the plurality of operations determined based on the user intent, based on the plurality of concepts. In other words, the planner module225may determine an execution sequence of the plurality of operations, based on the parameter necessary for executing the plurality of operations and the result outputted by execution of the plurality of operations. Accordingly, the planner module225may generate a plan including connection information (for example, ontology) between the plurality of operations and the plurality of concepts. The planner module225may generate the plan by using information stored in the capsule database230in which a set of relationships between the concept and the operation is stored.

The NLG module227in an embodiment may change designated information to a text form. The information changed to the text form may be a form of a natural language utterance. The TTS module229may change information in a text form to information in a voice form.

According to an embodiment, some or all of the functions of the natural language platform220may be implemented in the user terminal400.

The capsule database230may store information on relationships between the plurality of concepts and operations corresponding to the plurality of domains. The capsule in an embodiment may include a plurality of operation (action) objects (or operation information) and concept objects (or concept information) included in the plan. According to an embodiment, the capsule database230may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment, the plurality of capsules may be stored in a function registry included in the capsule database230.

The capsule database230may include a strategy registry in which strategy information necessary in determining a plan corresponding to a voice input is stored. The strategy information may include reference information for determining one plan when there are a plurality of plans corresponding to a voice input. According to an embodiment, the capsule database230may include a follow-up registry in which information of a follow-up operation for suggesting a follow-up operation for a user in a designated context is stored. The follow-up operation may include, for example, a follow-up utterance. According to an embodiment, the capsule database230may include a layout registry that stores layout information of information outputted through the user terminal400. According to an embodiment, the capsule database230may include a vocabulary registry in which vocabulary information included in capsule information is stored. According to an embodiment, the capsule database230may include a dialog registry in which information of a dialog (or interaction) with a user is stored. The capsule database230may update a stored object through a developer tool. The developer tool may include, for example, a function editor for updating an operation object or a concept object. The developer tool may include a vocabulary editor for updating a vocabulary. The developer tool may include a strategy editor for generating and registering a strategy for determining a plan. The developer tool may include a dialog editor for generating a dialog with a user. The developer tool may include a follow-up editor for enabling a follow-up target and editing a follow-up utterance providing a hint. The follow-up target may be determined based on a currently set target, user's preference, or an environmental condition. The capsule database230in an embodiment may be implemented in the user terminal400.

The execution engine240in an embodiment may calculate a result by using the generated plan. The end user interface250may transmit the calculated result to the user terminal400. Accordingly, the user terminal400may receive the result and may provide the received result to the user. The management platform260in an embodiment may manage information used in the intelligent server200. The big data platform270in an embodiment may collect user data. The analytic platform280in an embodiment may manage quality of service (QoS) of the intelligent server200. For example, the analytic platform280may manage components and a processing rate (or efficiency) of the intelligent server200.

The service server300in an embodiment may provide a designated service (for example, an order of food or reservation of a hotel) to the user terminal400. According to an embodiment, the service server300may be a server that is managed by a third party. The service server300in an embodiment may provide information for generating a plan corresponding to a received voice input to the intelligent server200. The provided information may be stored in the capsule database230. In addition, the service server300may provide result information corresponding to the plan to the intelligent server200.

In the above-described integrated intelligence system, the user terminal400may provide various intelligent services to the user in response to a user input. The user input may include, for example, an input through a physical button, a touch input, or a voice input.

In an embodiment, the user terminal400may provide a voice recognition service through an intelligent application (or a voice recognition application) stored therein. In this case, for example, the user terminal400may recognize a user utterance or a voice input received through the microphone, and may provide a service corresponding to the recognized voice input to the user.

In an embodiment, the user terminal400may perform a designated operation solitarily or along with the intelligent server and/or the service server, based on the received voice input. For example, the user terminal400may execute an application corresponding to the received voice input, and may perform a designated operation through the executed application.

In an embodiment, when the user terminal400provides a service along with the intelligent server200and/or the service server, the user terminal400may detect a user utterance by using the microphone420, and may generate a signal (or voice data) corresponding to the detected user utterance. The user terminal400may transmit the voice data to the intelligent server200through the communication interface410.

The intelligent server200according to an embodiment may generate a plan for performing a task corresponding to the voice input, or a result of performing an operation according to the plan, in response to the voice input received from the user terminal400. The plan may include a plurality of operations for performing the task corresponding to the user voice input, and a plurality of concepts related to the plurality of operations. The concept may be a definition of a parameter inputted for execution of the plurality of operations, or a definition of a result value outputted by execution of the plurality of operations. The plan may include connection information between the plurality of operations and the plurality of concepts.

The user terminal400in an embodiment may receive the response by using the communication interface410. The user terminal400may output a voice signal generated inside the user terminal400to the outside by using the speaker430, or may output an image generated inside the user terminal400to the outside by using the display440.

FIG.3illustrates an example view showing a form of relationship information between concepts and operations that is stored in a database according to various embodiments.

Referring toFIGS.2and3, a capsule database (for example, the capsule database230) of the intelligent server200may store capsules in the form of a CAN. The capsule database may store an operation for processing a task corresponding to a user voice input, and a parameter necessary for the operation in the form of the CAN.

The capsule database may store a plurality of capsules (capsule A501, capsule B504) corresponding to a plurality of domains (for example, applications), respectively. According to an embodiment, one capsule (for example, capsule A501) may correspond to one domain (for example, geo, application). In addition, one capsule may correspond to at least one service provider (for example, CP1502, CP2503, or CP3506) for performing a function regarding the domain related to the capsule. According to an embodiment, one capsule may include at least one operation510and at least one concept520for performing a designated function.

The natural language platform220may generate a plan for performing a task corresponding to a received voice input by using a capsule stored in the capsule database. For example, the planner module225of the natural language platform may generate the plan by using the capsule stored in the capsule database. For example, a plan507may be generated by using operations5011,5013and concepts5012,5014of the capsule A501, and an operation5041and a concept5042of the capsule B504.

FIG.4illustrates an example view of a screen for processing a voice input received through an intelligent application in a user terminal according to various embodiments.

Referring toFIG.4, the user terminal400may execute an intelligent application for processing a user input through the intelligent server200.

According to an embodiment, when the user terminal400recognizes a designated voice input (for example, “Wake up!”) or receives an input through a hardware key (for example, a dedicated hardware key) on a screen310, the user terminal400may execute an intelligent application for processing the voice input. For example, the user terminal400may execute the intelligent application while executing a schedule application. According to an embodiment, the user terminal400may display an object (for example, an icon)311corresponding to the intelligent application on the display440. According to an embodiment, the user terminal400may receive a voice input by a user utterance. For example, the user terminal400may receive a voice input saying, “Let me know my schedule this week”. According to an embodiment, the user terminal400may display a UI313(for example, an input window) of the intelligent application displaying text data of the received voice input on the display.

According to an embodiment, on a screen320, the user terminal400may display a result corresponding to the received voice input. For example, the user terminal400may receive a plan corresponding to the received user input, and may display “schedule of this week” on the display according to the plan.

FIG.5illustrates a view of an example intelligent assistance system1000according to various embodiments.

Referring toFIG.5, the intelligent assistance system1000may include a first server1100(for example, the server108ofFIG.1or the intelligent server200ofFIG.2), a second server1200(for example, the server108ofFIG.1), a receiver1300(for example, an AI speaker), and a plurality of executors1400. InFIG.5, one receiver1300is illustrated, but a plurality of receivers1300may be disposed in a predetermined space. The first server1100and the receiver1300may be connected with each other by using a wired or wireless network. The first server1100and the second server1200may be connected with each other by using a wired or wireless network. The first server1100and the plurality of executors1400may be connected with each other in a wired or wireless network. The receiver1300and the plurality of executors1400may be connected with each other through the first server1100. However, this should not be considered as limiting, and the receiver1300and the plurality of executors1400may be connected with each other in a D2D method.

According to various embodiments, the receiver1300may include various devices including a configuration related to voice recognition and a voice input device (for example, a microphone). For example, the receiver1300may include the electronic device101ofFIG.1or the user terminal400ofFIG.2. The receiver1300may acquire an utterance from a user1010through the voice input device. The utterance may include a wake-up utterance instructing to enable and/or invoke an intelligent assistance service, and/or a control utterance instructing an operation (for example, power control, volume control) of a hardware/software configuration included in the plurality of executors1400.

The wake-up utterance may be a pre-set keyword, such as “Hi”, “Hello”, “Hi, ABC”. For example, “ABC” in the wake-up utterance may be a name (for example, “Bixby”) given to the receiver1300(or a voice recognition agent of the receiver1300(or AI)), like galaxy.

The control utterance may be acquired in the state where the intelligent assistance service is enabled or invoked by the wake-up utterance. However, this is merely an example, and embodiments of the present disclosure are not limited thereto. For example, the control utterance may be acquired along with the wake-up utterance.

According to various embodiments, the receiver1300may generate a control message (or a control instruction) based on at least a portion of the acquired utterance (or utterance data). The receiver1300may transmit the generated control message to a target executor to perform an operation corresponding to the utterance, from among the plurality of executors1410,1420,1430,1440, or1450, by using the first server1100. The control message may be generated based on a result of processing the utterance data.

The processing of the utterance data may be performed through natural language processing by the receiver1300and/or natural language processing by the first server1100. For example, the receiver1300may process the utterance data by itself by using a voice processing module included in the receiver1300, as will be described below with reference toFIG.6.

In addition, the receiver1300may transmit the utterance data to the first server1100, and may request a result of processing the utterance data, as will be described below with reference toFIG.7. For example, the receiver1300may include an utterance data processing capability of a first level, for example, a voice recognition module and an NLU module of the first level. In addition, the first server1100may include an utterance data processing capability of a second level higher than the first level, for example, a voice recognition module and an NLU module of the second level.

According to various embodiments, the plurality of executors1400may include a smartphone1410, a computer1420(for example, a personal computer, a notebook computer, or the like), a television1430, a refrigerator1440, and/or an illumination device1450. The executors1400according to various embodiments may further include an air conditioner, a thermostat, an alarm device, a gas valve control device, or a door lock device although they are not illustrated inFIG.5.

According to an embodiment, each of the plurality of executors1400may include a communication circuit to establish communication with the first server1100by using a designated protocol (for example, Bluetooth, Wi-Fi, or Zigbee), and to transmit and receive a variety of information. According to an embodiment, the plurality of executors1400may transmit information on their own operation states (for example, on/off information of a device) to the receiver1300or the first server1100. In addition, the plurality of executors1400may receive a control message (for example, an on/off control command of a device, or other operation control commands of a device) from the receiver1300or the first server1100. The plurality of executors1400may execute an operation corresponding to the control message. In addition, the plurality of executors1400may transmit a result of executing the operation corresponding to the control message to the receiver1300or the first server1100.

The first server1100may form a connection binding between the receiver1300and the plurality of executors1400to cause the operation corresponding to the utterance obtained at the receiver1300to be performed. The first server1100may form a first channel for communicating with the receiver1300, and may form a second channel for communicating with at least one of the plurality of executors1400. The first server1100may form the connection binding between the receiver1300and the executor1400, by using first device information of the receiver1300received through the first channel and second device information of the plurality of executors1400received through the second channel. In addition, the first server1100may control maintenance, termination, and/or reconnection of the connection binding between the receiver1300and the plurality of executors1400. In addition, the first server1100may control exchange of information and distribution of information between the receiver1300and the plurality of executors1400. The first server1100will be described in detail below with reference toFIGS.7and8.

According to various embodiments, the first server1100may connection-bind one receiver1300and one executor (for example,1430). However, this should not be considered as limiting, and the first server1100may connection-bind one receiver1300and a plurality of executors (for example,1430,1440). The receiver1300may receive an utterance to be performed at the plurality of executors1400. Herein, when the utterance of the user1010indicates a plurality of target executors, the first server1100may connection-bind the plurality of executors (for example, the TV1430and the refrigerator1440) to one receiver1300.

For example, when an utterance saying “Play the Infinite Challenge on the TV1430, and tell me the weather information on the refrigerator1440” is received at the receiver1300(for example, a speaker), the TV1430and the refrigerator1440may perform operations corresponding to the utterance of the user1010as a plurality of executors. That is, the first server1100may connection-bind one receiver1300and the plurality of executors (for example, the TV1430and the refrigerator1440).

To connection-bind the receiver1300and the plurality of executors1400, the receiver1300may transmit first device information thereof to the second server1200. Each of the plurality of executors1400may transmit their own second device information to the second server1200. For example, the first device information may include state information, necessary response information, or identification information of the receiver1300. The second device information may include state information, necessary response information, or identification information of the executor1400. The second server1200may store and manage the first device information and the second device information for connection-binding between the receiver1300and the plurality of executors1400. The second server1200may provide the first device information on the receiver1300and the second device information on each of the plurality of executors1400to the first server1100. The first server1100and the second server1200may be disposed with different configurations. However, this should not be considered as limiting, and the first server1100and the second server1200may be disposed with the same configuration. AlthoughFIG.5illustrates that the first server1100and the second server1200are configured separately by way of an example, the first server1100and the second server1200may be integrated into one server.

FIG.6illustrates a view of an example configuration of the receiver according to various embodiments.

Referring toFIGS.5and6, the receiver1300may include a processor1310, a memory1320, a communication module1330, and/or a voice processing module1340.

According to various embodiments, in response to an utterance being received, the processor1310may control the received utterance to be processed through the receiver1300and the first server1100. According to an embodiment, the processor1310may control the voice processing module1340to perform natural language processing with respect to utterance data received from the user1010. For example, the processor1310may control the voice processing module1340to acquire at least one of an intent of the utterance of the user1010, a domain for executing a task, or data (for example, a slot or a task parameter) necessary for understanding the intent of the user1010. In addition, the processor1310may control the communication module1330to provide the received utterance to the first server1100, thereby causing the received utterance to be processed through the first server1100.

According to various embodiments, the receiver1300may perform a function of a listener of receiving a user voice. The receiver1300may include a voice input device (not shown) (for example, a microphone) to receive a user voice. In addition, the receiver1300may provide a result of performing an operation corresponding to the utterance of the user. The receiver1300may include a sound output device (not shown) (for example, a speaker), a display (not shown), and/or one or more lamps (not shown) to provide the result of performing the operation corresponding to the utterance.

According to various embodiments, the processor1310may control to generate a control message (or a control instruction), based on at least one of a first processing result of the utterance data performed through the receiver1300and a second processing result of the utterance data performed through the first server1100. According to an embodiment, the processor1310may select a processing result to be used for generating the control message, based on pre-stored intent masking information. The intent masking information may be information in which an utterance processing object is designated for an intent. For example, the processor1310may identify an intent by processing a received utterance, and may determine whether the utterance related to the identified intent is defined to be processed through the receiver1300or is defined to be processed through the first server1100, based on the intent masking information.

According to another embodiment, the processor1310may process the intent masking information stored in the memory1320to be refined. According to an embodiment, the processor1310may control to provide the result of processing the received utterance to the first server1100. For example, the processor1310may transmit the result of processing the utterance data, performed by the receiver1300(for example, the voice processing module1340), to the first server1100, such that intent masking information corresponding to the result of processing can be received. In addition, the processor1310may process the intent masking information stored in the memory1320to be refined, based on at least a portion of the intent masking information received from the first server1100.

According to various embodiments, the voice processing module1340may understand an intent and/or a domain regarding the input of the user1010, by performing natural language processing with respect to the utterance acquired from the user1010. In addition, the voice processing module1340may generate a result of natural language processing by natural language understanding with respect to the input of the user1010. According to an embodiment, the voice processing module1340may include a voice recognition module (an ASR module)1340-1and/or an NLU module1340-2. In addition, the voice processing module1340may further include an NLG module and a TTS module, although they are not illustrated.

According to an embodiment, the voice recognition module1340-1may generate text data expressing the received utterance in a designated language. The voice recognition module1340-1may generate text data by using an acoustic model and/or a language model. The acoustic model may include information regarding vocalization, and the language model may include unit phoneme information and/or information of a combination of unit phoneme information. For example, the voice recognition module1340-1may convert the utterance of the user1010into text data by using the information on vocalization and/or the information related to unit phoneme information.

According to an embodiment, the NLU module1340-2may understand an intent regarding the input of the user1010or may understand a matching domain, by using a natural language processing model with respect to the text data generated by the voice recognition module1340-1. The NLU module1340-2may acquire an element (for example, a slot or a task parameter) necessary for expressing the intent of the user1010. For example, the NLU module1340-2may process utterance data, based on syntactic analysis and/or semantic analysis. Based on a result of processing, a domain or an intent corresponding to the utterance may be determined, and an element necessary for expressing the intent of the user1010may be acquired. According to an embodiment, the NLU module1340-2may include a plurality of NLU modules. The plurality of NLU modules may correspond to the plurality of executors1400, respectively. For example, each of the NLU modules may understand an intent regarding the input of the user1010or may understand a matching domain with reference to a natural language understanding database corresponding to each of the executors1410,1420,1440, or1450.

According to an embodiment, the voice processing module1340(for example, the NLG module) may generate data created in the middle of natural language processing in the form of a natural language. The data generated in the form of the natural language may be a result of natural language understanding. In addition, the NLG module may generate a result of executing indicating whether a control operation corresponding to a control utterance is performed by the plurality of executors1400, in the form of a natural language.

FIG.7illustrates a view of an example configuration of the first server1100(electronic device) according to various embodiments.FIG.8illustrates an example view to explain an intelligent assistance system including the first server and an operation of the first server according to an embodiment.

Referring toFIGS.5,7, and8, at least some components of the first server1100may correspond to at least some components of the receiver1300. For example, the first server1100may include a processor1110, a memory1120, a communication module1130, and/or a voice processing module1140, and additionally or selectively, may further include a matching information generation module1150. Accordingly, components of the first server1100corresponding to components of the receiver1300will not be described in detail. The intelligent assistance system1000may include a plurality of first servers1100according to a capacity for processing an utterance of the user1010.

According to various embodiments, the processor1110of the first server1100may control the voice processing module1140to process utterance data received from the receiver1300. In addition, the processor1110may provide a result of processing the utterance data to the receiver1300. For example, the result of processing may include at least one of an intent of the input of the user1010, a domain for executing a task, or data (for example, a slot or a task parameter) necessary for understanding the intent of the user1010.

According to various embodiments, the processor1110of the first server1100may control to provide intent masking information to the receiver1300as a result of processing. As described above, the intent masking information may be information in which an utterance processing object is designated for an intent. In addition, the intent masking information may be generated by the matching information generation module1150as will be described below.

According to various embodiments, the voice processing module1140of the first server1100may include a voice recognition module1140-1and an NLU module1140-2, similarly to the voice processing module1340of the receiver1300. According to an embodiment, the voice processing module1140of the first server1100may have a higher processing capability than the utterance data processing capability of the receiver1300. For example, the result of processing the utterance (or utterance data) by the voice processing module1140of the first server1100may have higher accuracy than that of the result of processing the utterance by the voice processing module1340of the receiver1300.

According to various embodiments, the matching information generation module1150of the first server1100may generate intent masking information based on the result of processing by the receiver1300(for example, the voice processing module1340). The intent masking information may be associated with a matching rate between a first processing result of utterance data performed by the receiver1300(for example, the voice processing module1340), and a second processing result of the utterance data performed by the first server1100(for example, the voice processing module1140). According to an embodiment, the first server1100may receive the first processing result from the receiver1300, and the matching information generation module1150may identify a matching rate on the first processing result by comparing the received first processing result and the second processing result performed by the first server1100. In addition, the matching information generation module1150may generate intent masking information indicating one of the receiver1300or the first server1100as an object to process a received utterance, based on the identified matching rate.

The receiver1300and the first server1100may transmit and/or receive data through a capsule execution service (CES)1500. In addition, the executor1400and the first server1100may transmit and/or receive data through the CES1500. The CES1500may support management of a first channel connecting the receiver1300and the first server1100, and management of a second channel connecting the executor1400and the first server1100. The CES1500may indicate, to the receiver1300, the first server1100to be connected from among a plurality of servers, and may indicate, to the executor1400, the first server1100to be connected from among the plurality of servers. That is, the CES1500may inform the receiver1300and the executor1400of which server should be connected from among the plurality of servers.

According to various embodiments, the processor1110of the first server1100may include a conversation manager1112, a classifier1114, an event manager1116, and/or a plurality of CANs1118.

According to various embodiments, the conversation manager1112may generate, maintain and/or manage a plurality of connection bindings1112a. The conversation manager1112may include a life-time module1112band/or a connection module1112c. The conversation manager1112may generate, maintain, and/or manage the connection binding1112abetween the receiver1300and the executor1400, such that communication can be established between the receiver1300and the executor1400after an utterance is acquired at the receiver1300.

According to various embodiments, the life-time module1112bmay set a connection maintaining time (for example, 3 seconds, 5 seconds, 8 seconds, 15 seconds, 30 seconds, 1 minute, 3 minutes, 5 minutes, 10 minutes, 30 minutes, or 60 minutes) for determining whether to sustain or terminate the connection binding1112a. The life-time module1112bmay count a time from a time that the first channel is generated, and may determine whether the connection maintaining time (for example, a receiver connection maintaining time) is exceeded after the first channel is generated. In addition, the life-time module1112bmay count a time from a time that the second channel is generated, and may determine whether the connection maintaining time (for example, an executor connection maintaining time) is exceeded after the second channel is generated. In addition, the life-time module1112bmay count a time from a time that an utterance is acquired at the receiver1300, and may determine whether the connection maintaining time (for example, an additional information input waiting time) is exceeded after the utterance is acquired. In addition, the life-time module1112bmay count a time from a time that the utterance is performed at the executor1400, and may determine whether the connection maintaining time (for example, a follow-up utterance waiting time) is exceed after the utterance is performed.

According to various embodiments, the connection module1112cmay maintain or terminate the connection binding between the receiver1300and the executor1400, based on determination of the connection maintaining time by the life-time module1112b.

Referring to table 1, a plurality of data types may be used when the receiver1300and the executor1400are connection-bound. The conversation manager1112may generate, maintain, and/or manage the connection binding between the receiver1300and the executor1400by using data types described in table 1.

For example, “conversationId” may refer to an ID value regarding each conversation. Herein, various continuous utterances may have the same conversationId value. “requestId” may refer to an ID value regarding each utterance request. “referenceRequestId” may refer to a value for referring to information regarding a previous utterance, and may be configured with requestId of the previous utterance. “Intent (Utterance)” may refer to utterance data. “installedCampanionApps” may refer to information of Apps installed in a device (or the receiver1300) sending a request. “Geo” may refer to position information of a device sending a request. “accountId (userId)” may refer to account user account information.

“viewportId” may refer to a conversation list ID value used at the first server1100(for example, the Bixby server). “deviceId” may refer to a device ID value used at the second server1200(for example, an IoT server). “svcld” may refer to a unique ID value of userId & deviceId join. “deviceModel” may refer to model information of a device sending a request. “canTypeId (deviceType)” may refer to a type of a device sending a request. “Language” may refer to setting of a language of a device sending a request.

Referring back toFIGS.7and8, the conversation manager1112may form the first channel (for example, a listener channel) for communicating between the receiver1300and the first server1100to form the connection binding, based on a request of the receiver1300. When the first channel is formed between the receiver1300and the first server1100, the receiver1300may transmit utterance information and the first device information of the receiver1300to the first server1100by using the first channel. Herein, the receiver1300may transmit the first device information to the second server1200. In this case, the receiver1300may transmit the first device information to the second server1200via the first server1100. However, this should not be considered as limiting. The receiver1300may directly transmit the first device information to the second server1200. For example, when an utterance is received from the receiver1300, the conversation manager1112may determine whether the corresponding utterance can be processed at the receiver1300or can be processed at the executor1400. When the utterance can be processed at the executor1400, the first server110may transmit the first device information to the second server1200.

In addition, the conversation manager1112may form the second channel (for example, an executor channel) for communicating between the executor1400and the first server1100. The executor1400may generate second device information (for example, state information of the executor1400, necessary response information, or identification information). When the second channel is formed between the executor1400and the first server1100, the executor1400may transmit the second device information to the first server1100through the second channel.

After the first channel and the second channel are formed, the first server1100may connection-bind the receiver1300and the executor1400, based on the first device information received from the receiver1300and the second device information received from the executor1400. In this case, the first server1100may maintain and terminate the connection binding between the receiver1300and the executor1400, based on a pre-set value.

In the above-described example, the first device information and the second device information may be transmitted and/or received through the first server1100and/or the second server1200. However, this should not be considered as limiting, and the first device information and the second device information may be transmitted and/or received between the receiver1300and the executor1400in a D2D method.

The classifier1114may classify to determine which executor1400performs the utterance requested by the receiver1300from among the plurality of executors1400. The classifier1114may cause utterance information to be transmitted to the executor1400to perform the utterance requested at the receiver1300, that is, to a target executor. In addition, the classifier1114may cause a result of performing the utterance to be transmitted to the corresponding receiver1300after an operation corresponding to the utterance is performed at the executor1400.

The event manager1116may cause the result of performing the utterance to be transmitted to the receiver1300and/or the executor1400after the operation corresponding to the utterance is performed. The event manager1116may determine through which channel the result of performing the utterance is transmitted to the receiver1300and/or the executor1400. The event manager1116may determine whether to transmit the result of performing the utterance without processing, or whether to correct the result of performing the utterance according to a UI form of the receiver1300and/or the executor1400, and to transmit the result. When the result of performing the utterance should be corrected, the event manager1116may correct contents of the utterance according to the UI form of the receiver1300and/or the executor1400, and may transmit the corrected result of performing the utterance to the receiver1300and/or the executor1400.

FIG.9illustrates a view of an example of connection-binding the receiver1300and the executor1400by the first server1100. In the following embodiment, respective operations may be performed in sequence, but may not necessarily be performed in sequence. For example, the order of the respective operations may be changed or at least two operations may be performed in parallel.

Referring toFIG.9, in operation S10, the receiver1300may transmit an operation request (for example, a Bixby request) corresponding to a received utterance to the first server1100. In this case, when the operation request corresponding to the utterance is made, data types described in table 1 may be used. The first server110may analyze the operation request (for example, the Bixby request) corresponding to the received utterance.

In operation S20, the first server1100may recognize which executor1400should perform the operation corresponding to the utterance, by analyzing the operation request corresponding to the utterance. The first server1100may transmit a device remote request to the second server1200to cause the operation corresponding to the utterance to be performed at the target executor1400. Herein, the second server1200may identify the target executor1400with reference to a userId value and a deviceId value The first server1100may include data for generating a connection binding between the receiver1300and the target executor1400in a payload, and may transmit the data to the second server1200.

In operation S30, the second server1200may transmit an executor device request to the target executor1400, based on the userId value and the deviceId value received from the first server1100. In this case, the second server1200may transmit the data for generating the connection binding, included in the payload, to the executor1400from the first server1100.

In operation S40, the executor1400may request the data for generating the connection binding, included in the received payload, from the first server1100(Bixby Request with Attach Info). In this case, requests for operations corresponding to different utterances may be made between the receiver1300and the plurality of first servers1100. In response to the request corresponding to each utterance, data should be transmitted to the first server1100. To achieve this, InstanceId may be used to request the same Instance of the first server110requested by the receiver1300.

In operation S50, the first server1100may generate a connection binding with the receiver1300through conversationId, viewportId, requestId received from the executor1400. That is, the first server1100may generate a connection binding with each of the executors1400according to each of the utterance requests made at the receiver1300.

In operation S60, the first server1100may transmit a response to generation of the connection binding to the executor1400, and may maintain the connection binding between the receiver1300and the executor1400for a predetermined time. The first server1100may terminate the connection binding between the receiver1300and the executor1400after the predetermined time is elapsed.

FIG.10illustrates an example flowchart S100for providing an intelligent assistance service at the receiver according to various embodiments.

Referring toFIGS.8and10, according to various embodiments, in operation S110, the receiver1300(for example, the processor1310ofFIG.6) may store intent masking information. The intent masking information may be information in which an utterance processing object is designated for an intent. For example, the intent masking information may be information that is defined to cause an utterance related to at least one intent to be processed through the receiver1300or the first server1100.

According to various embodiments, in operation S120, the receiver1300(for example, the processor1310ofFIG.6) may receive (or acquire) an utterance from the user1010through the input device (for example, a microphone). As described above, the utterance may include a wake-up utterance instructing to enable and/or invoke an intelligent assistance service, and/or a control utterance instructing an operation (for example, power control, volume control) of a hardware/software configuration included in the plurality of executors1400. According to an embodiment, the utterance may be received when the receiver1300operates in a low power mode. The low power mode may refer to a mode in which a first processor (for example, an auxiliary processor) for receiving a wake-up utterance is enabled and a second processor (for example, a main processor) for controlling the intelligent assistance service is disabled. However, this is merely an example, and the disclosure is not limited thereto. For example, the utterance may be received when both the first processor and the second processor are enabled.

According to various embodiments, in operation S130, the receiver1300(for example, the processor1310ofFIG.6) may determine one of the receiver1300or the first server1100as an object to process the utterance, based on the intent masking information. According to an embodiment, the processor1310may identify an intent regarding the received utterance. In addition, the processor1310may determine whether the object to process the identified intent is the receiver1300or the first server1100, based on the stored intent masking information. For example, the processor1310may process the received utterance by using the determined object to process. For example, the processor1310may provide a service based on a result of processing the utterance by the determined processing object.

FIG.11illustrates a flowchart S200of an example of distributing connection state information to the receiver and the executor by the first server.

Referring toFIGS.8and11, the event manager1116of the first server1100may cause a result of performing an utterance to be transmitted to the receiver1300and/or the executor1400after an operation corresponding to the utterance is performed. The event manager1116may determine through which channel the result of performing the utterance will be transmitted to the receiver1300and/or the executor1400. The event manager1116may determine whether to transmit the result of performing the utterance without processing, or may determine whether to correct the result of performing the utterance according to a UI form of the receiver1300and/or the executor1400, and to transmit the result. When the result of performing the utterance should be corrected, the event manager1116may correct the result of performing the utterance according to the UI form of the receiver1300and/or the executor1400, and may transmit the corrected result of performing the utterance to the receiver1300and/or the executor1400.

According to various embodiments, in operation S210, performance of the operation corresponding to the utterance may be completed at the executor1400.

In operation S215, the event manager1116of the first server1100may generate a response of the completion of the operation performance. The event manager1116may transmit the response of the operation performance completion to the receiver1300and/or the executor1400.

Herein, processing of the response performed at the first server1100may be appropriately distributed to the receiver1300and the executor1400. The response may be provided in a UI form that can be expressed by each receiver1300. In addition, the response may be provided in a UI form that can be expressed by each executor1400. In addition, the first server1100may transmit the response to the receiver1300and/or the executor1400as it is without correcting contents of the response. In addition, the first server110may correct the contents of the response and may transmit the response to the receiver1300and/or the executor1400.

In operation S220, the first server1100may determine whether the response should be transmitted to the receiver1300.

When the response should be transmitted to the receiver1300as a result of determining in operation S220(“Yes” in operation S220), the first server1100may determine whether it is necessary to correct contents of the response in operation S225.

When it is not necessary to correct the contents of the response as a result of determining in operation S225(“No” in operation S225), the first server1100may transmit the response to the receiver1300as it is without correcting the contents of the response in operation S230.

When it is necessary to correct the contents of the response as a result of determining in operation S225(“Yes” in operation S225), the first server1100may correct the contents of the response and may transmit the response to the receiver1300.

Through operations S220to S235, the first server1100may change the result of performing by the executor1400to an articulate phrase or graphic, and may provide the result to the user who uttered toward the receiver1300. For example, when the result of performing the utterance is displayed on a screen of a display device (for example, the receiver1300) in the form of a UI, there may be a difference in a screen size or a resolution according to the display device. Accordingly, the UI may be changed according to a screen size and a resolution of the display device, and may be displayed on the screen. For example, when a TV (the executor1400) is controlled through a speaker (for example, the receiver1300), a response regarding the result of performing by the TV may sound awkward when it is said by the speaker. In this case, contents of the response may be corrected like “The000command performed by the TV has been well carried out.” and may be provided through the speaker (for example, the receiver1300).

When the response is not to be transmitted to the receiver1300as a result of determining in operation S220(“No” in operation S220), the first server1100may determine whether the response should be transmitted to the executor1400in operation S240.

When it is not necessary to transmit the response to the executor1400as a result of determining in operation S240(“No” in operation S240), the first server1100may finish the operation.

When the response should be transmitted to the executor1400as a result of determining in operation S240(“Yes” in operation S240), the first server110may determine whether it is necessary to correct the contents of the response in operation S245.

When it is not necessary to correct the contents of the response as a result of determining in operation S245(“No” in operation S245), the first server1100may not correct the contents of the response and may transmit the response to the executor1400as it is in operation S250.

When it is necessary to correct the contents of the response as a result of determining in operation S245(“Yes” in operation S245), the first server1100may correct the contents of the response and may transmit the response to the executor1400in operation S255.

According to various embodiments, when the response is generated in response to the result of performing the utterance, the event manager1116of the first server1100may process the response in different conditions with respect to the receiver1300and the executor1400, respectively. For example, the receiver1300may transmit a request including an utterance to the first server1100. In this case, the receiver1300may declare what response is needed to the first server1100(for example, when a user utters “Tell me the weather on the TV” to the speaker, there is no need to give weather screen information to the speaker). The receiver1300may include, in the request, information on a response type, a device type, necessary according to a state of the receiver1300to suit to a format of the request containing the user utterance, and may transmit the request to the first server1100. When processing the response of the result of performing, the event manager1116may only select a necessary response according to the declared content, and may transmit the response to the receiver1300.

According to various embodiments, when a connection binding is formed between the receiver1300and the executor1400, the receiver1300may pre-determine what response is needed according to a device type (for example, according to presence/absence of a display, presence/absence of a speaker, or presence/absence of hardware), and may transmit the pre-determined response format to the first server1100. The first server1100may transmit a pre-determined first response format to the executor1400from the receiver1300. The first server1100may generate a response based on the first response format pre-determined at the receiver1300, and may transmit the response to the receiver1300.

According to various embodiments, when the connection binding is formed between the receiver1300and the executor1400, the executor1400may pre-determine what response is needed according to a device type (according to presence/absence of a display, presence/absence of a speaker, presence/absence of hardware), and may transmit the pre-determined response format to the first server1100. The first server1100may transmit a pre-determined second response format to the receiver1300from the executor1400. The first server1100may generate a response based on the second response format pre-determined at the executor1400, and may transmit the response to the executor1400.

According to various embodiments, when the connection binding with the executor1400is formed, the executor1400may declare a necessary response within the second device information and/or context information (for example, a response type or a device type), and may transmit the response to the first server1100.

When the executor1400declares the response and transmits the response to the first server1100, the refrigerator may declare like “Tell the weather at the air conditioner”. In this case, the refrigerator may explain through an NLG and a screen, and the air conditioner which is the executor may not show screen information. The information transmitted from the executor1400to the first server1100may include information on a response type or a type (for example, refer to the data type ofFIG.9) which is necessary according to a current state, and may be transmitted. When processing the response of the performed utterance, the event manager1116may select necessary responses declared at the executor1400, and may transmit the selected responses to the executor1400.

According to various embodiments, even when the receiver1300or the executor1400does not declare regarding the response, the event manager1116may correct the response according to a UI form of the receiver1300or the executor1400, and may transmit the response. The event manager1116may already know the first device information of the receiver1300currently connection-bound, and the second device information of the executor1400. In addition, the event manager1116may select a response of the result of performing the utterance according to a device type and a type (for example, presence/absence of a display, presence/absence of a microphone, or presence/absence of an additionally expressible function such as an LED) of the receiver1300and the executor1400, based on the previously recognized first device information and second device information. The event manager1116may transmit the selected response of the result to the receiver1300or the executor1400. For example, when an utterance saying “Tell me the weather today in Suwon on the TV (executor1400)” is received at the speaker (receiver1300), there may not be a need to transmit screen information of the weather to the speaker (receiver1300) without a display device. In this case, the event manager1116may select a response regarding the result of performing the utterance according to a UI form of the speaker (receiver1300), and may transmit the response to the speaker (receiver1300).

According to various embodiments, there may be a need to correct contents of the response of the natural language generator (generation?) (NLG) in common with respect to the receiver1300and the executor1400. For example, NLG or an additional operation (for example, an LED operation) occurring may be changed according to a device type and a type of the receiver1300and the executor1400, and may be transmitted to the receiver1300and the executor1400.

For example, an utterance saying, “Can you play the Infinite Challenge on the TV (executor1400)?” may be received at the speaker (receiver1300). In this case, when NLG of the result of performing the utterance by the TV (executor1400) is “I played it”, the event manager1116may correct the NLG to “I played the Infinite Challenge on the TV” and may transmit the corrected NLG to the speaker (receiver1300). In addition, the event manager1116may not generate any NLG for the TV (executor1400).

For example, when an utterance saying “Tell me how to save energy in the air conditioner (executor1400)” may be received at the refrigerator (receiver1300). In this case, the event manager1116may transmit NLG of the result of performing the utterance for the air conditioner (executor1400) only to the refrigerator (receiver1300), and may not cause the air conditioner (executor1400) to perform a separate follow-up operation.

According to various embodiments, the first server1100may maintain and/or terminate the connection binding between the receiver1300and the executor1400. For example, when the first channel with the receiver1300and the second channel with the executor1400are connection-bound, the first server1100may start counting the connection maintaining time of each of the first channel and the second channel. For example, while an utterance processing maintaining time of the receiver1300is being maintained, the first server1100may transmit a continuous and connective utterance received at the receiver1300to the executor1400. In this case, the connection binding between the receiver1300and the executor1400may be maintained for a predetermined connection maintaining time.

For example, after the utterance processing maintaining time of the receiver1300is ended, the first server1100may let a new receiver1300join the existing connection. The first server110may request the same executor1400to perform the operation corresponding to the utterance while changing the receiver1300. That is, the first server1100may cause the utterance to be processed while changing the receiver1300, without changing the executor1400. For example, the first server1100may cause the utterance to be processed while changing the executor1400, without changing the receiver1300.

FIG.12illustrates an example flowchart S300of an example of maintaining and terminating the connection binding of the receiver. Table 2 shows examples of a connection maintaining time of the receiver1300, a connection maintaining time of the executor1400, an additional information input waiting time, and a follow-up utterance waiting time.

TABLE 2Types of connectionmaintaining timeDescriptionConnection maintainingThis time is counted when a user utterance is received,time of the receiverand is extinguished after a given time (for example, 8(Conversation Life Time inseconds, 10 minutes) or when a user terminates.Listener)Connection maintainingThis time is counted when connection information istime of the executorreceived from the receiver and a connection is(Conversation Life Time inestablished, and is the same as time counted from theExecutor)time that a user utters to the device.Additional information inputThis time is a waiting time (for example, 8 seconds)waiting time (Promptuntil necessary information is received after theWaiting Time)necessary information is requested from a user becauseadditional information is required in processing anutterance. This time is extinguished when a user uttersadditional information, a time is elapsed, or the userterminates.Follow-up utterance waitingThis time is a waiting time to process an additionaltime (Continuous utterancecontinuous utterance after one utterance is processed.waiting time)This time is refreshed when the additional utterance isreceived and processed, and is extinguished when atime is elapsed or the user terminates.

Referring toFIGS.8and12and table 2, according to various embodiments, the receiver1300may receive an utterance in operation S310. When the utterance is received at the receiver1300, the first channel may be generated between the first server1100and the receiver1300.

In operation S320, the first server1100may count a time from a time that the receiver1300receives the utterance. The first server1100may determine whether the connection maintaining time with the receiver1300is ended.

As a result of determining in operation S320, the time may be elapsed after the time that the utterance is received at the receiver1300, and the connection maintaining time may be ended. When the connection maintaining time of the receiver1300is ended (“Yes” in operation S320), the first channel connecting the receiver1300and the first server110may be terminated in operation S330. Herein, the first channel may be terminated by the first server1100. However, this should not be considered as limiting, and the first channel may be terminated by the receiver1300.

As a result of determining in operation S320, when the connection maintaining time is not ended (“No” in operation S320), the first channel connecting the receiver1300and the first server110may be maintained in operation S340.

In operation S350, performance of an operation corresponding to the utterance may be completed at the executor1400. In this case, the first server1100may count a time from a time that the performance of the operation corresponding to the utterance is completed.

In operation S360, the first server1100may determine whether the follow-up utterance waiting time (continuous utterance waiting time) for a follow-up utterance is ended after the performance of the operation corresponding to the utterance is completed.

When the follow-up utterance waiting time is ended as a result of determining in operation S360(“Yes” in operation S360), the first server110may terminate the first channel between the receiver1300and the first server1100.

When the follow-up utterance waiting time for the follow-up utterance is not ended as a result of determining in operation S360(“No” in operation S360), the first server1100may maintain the first channel within the follow-up utterance waiting time, and may wait for the follow-up utterance.

FIG.13illustrates a flowchart S400illustrating of an example of maintaining and terminating the connection binding of the executor.

Referring toFIGS.8and13and table 2, according to various embodiments, in operation S410, an utterance processing request from the receiver1300may be received at the first server1100and the second channel may be generated between the first server1100and the executor1400.

In operation S420, the first server1100may count a time from a time that the second channel is generated between the first server1100and the executor1400. The first server1100may determine whether the connection maintaining time with the executor1400is ended from the time that the second channel is generated.

As a result of determining in operation S420, the time may be elapsed after the time that the second channel is generated, and the connection maintaining time may be ended. When the connection maintaining time of the second channel is ended (“Yes” in operation S420), the first server1100may terminate the second channel connecting the executor1400and the first server1100in operation S430.

When the connection maintaining time is not ended as a result of determining in operation S420(“No” in operation S420), the first server1100may maintain the second channel connecting the executor1400and the first server1100in operation S440.

In operation S450, performance of an operation corresponding to an utterance may be completed at the executor1400. In this case, the first server1100may count a time from a time that the performance of the operation corresponding to the utterance is completed.

In operation S460, the first server1100may determine whether the follow-up utterance waiting time for a follow-up utterance is ended after the time that the performance of the operation corresponding to the utterance is completed.

When the follow-up utterance waiting time for the follow-up utterance is ended as a result of determining in operation S460(“Yes” in operation S460), the first server1100may terminate the second channel between the executor1400and the first server1100. However, this should not be considered as limiting, and the second channel may be terminated by the executor1400.

When the follow-up utterance waiting time for the follow-up utterance is not ended as a result of determining in operation S460(“No” in operation S460), the first server110may maintain the second channel within the follow-up utterance waiting time and may wait for the follow-up utterance.

According to various embodiments, the first server1100may maintain the connection binding between the receiver1300and the executor1400for the additional information input waiting time (prompt waiting time).

Herein, the additional information input waiting time may be a waiting time (for example, 3 seconds, 5 seconds, 8 seconds, 15 seconds, 30 seconds, 1 minute, 3 minutes, 5 minutes, 10 minutes, 30 minutes, or 60 minutes) until necessary information is received from a user after the necessary information is requested from the user since additional information is required in processing an utterance.

For example, the first server1100may end the additional information input waiting time when the user utters additional information within the additional information input waiting time. For example, the first server110may terminate the connection binding between the receiver1300and the executor1400when a pre-set additional information input waiting time is exceeded.

Referring toFIG.8with table 2, connection maintaining times of the first channel and the second channel generated at the conversation manager1112may be different depending on a condition. For example, maintenance and termination of the first channel based on a first connection maintaining time between the receiver1300and the first server1100may be set variously. In addition, maintenance and termination of the second channel based on a second connection maintaining time between the executor1400and the first server1100may be set variously. For example, the receiver1300may be changed according to a movement of the user or an intensity of an utterance voice, and the same executor1400may be designated to process the utterance while the receiver1300is being changed. That is, when a first receiver is changed to a second receiver, a third channel may be generated between the first server110and the second receiver. The first server1100may transmit and receive device information of the second receiver and utterance information by using the third channel. The first server110may connection-bind the changed second receiver and the executor1400. Even when the receiver1300is changed, the first server1100may additionally form a connection binding with the same executor1400, and may cause an operation corresponding to the utterance to be performed. Herein, when the receiver1300is changed, the first server1100may terminate the first channel with the first receiver after the third channel is generated between the first server1100and the second receiver. However, this should not be considered as limiting, and the first server1100may maintain multi channels with the plurality of receivers1300with respect to the same executor1400. The first server1100may maintain the first channel with the first receiver for a pre-set connection maintaining time even after the third channel is generated between the first server1100and the second receiver.

For example, the executor1400may be changed while the receiver1300is being maintained, and an utterance may be processed while the executor1400is being changed. That is, an utterance may be received through the same receiver1300, and the executor1400performing an operation corresponding to the utterance may change from a first executor to a second executor. In this case, the third channel may be generated between the first server1100and the second executor. The first server1100may transmit and receive device information of the second executor and utterance information by using the third channel. The first server1100may connection-bind the changed second executor and the receiver1300. The first server1100may additionally form a connection binding with the same receiver1300even when the executor1400is changed, and may cause the operation corresponding to the utterance to be performed. Herein, when the executor1400is changed, the first server1100may terminate the second channel with the first executor after the third channel is generated between the first server1100and the second executor. However, this should not be considered as limiting, and the first server1100may maintain multi channels with the plurality of executors1400with respect to the same receiver1300. The first server1100may maintain the second channel with the first executor for a pre-set connection maintaining time even after the third channel is generated between the first server1100and the second executor.

For example, the receiver1300and the executor1400may be changed according to a movement of a user or an intensity of an utterance voice, and the utterance may be processed while the receiver1300and the executor1400are being changed. That is, when the receiver changes from the first receiver to the second receiver and the executor changes from the first executor to the second executor, the first server1100may generate the third channel between the first server1100and the second receiver, and may generate a fourth channel between the first server1100and the second executor. The first server1100may transmit and receive device information of the second receiver and the second executor and utterance information by using the third channel and the fourth channel. The first server1100may connection-bind the second receiver and the second executor. The first server1100may additionally form a connection binding even when the receiver1300and the executor1400are changed, and may cause an operation corresponding to the utterance to be performed. Herein, the first server1100may terminate the first channel with the first receiver after the third channel is generated between the first server1100and the second receiver. In addition, the first server1100may terminate the second channel with the first executor after the fourth channel is generated between the first server1100and the second executor. However, this should not be considered as limiting, and the first server1100may maintain multi channels with the plurality of receivers (for example, the first receiver and the second receiver) and the plurality of executors (for example, the first executor and the second executor). The first server1100may maintain the first channel with the first receiver for a pre-set connection maintaining time even after the third channel is generated between the first server1100and the second receiver. In addition, the first server1100may maintain the second channel with the first executor for a pre-set connection maintaining time even after the fourth channel is generated between the first server1100and the second executor.

According to various embodiments, in a state where the receiver1300and the executor1400are connection-bound to each other, the first server1100may process an utterance operation for waiting for an input requesting additional information. For example, an utterance saying “Change the channel on the TV (executor1400)” may be received at the refrigerator (receiver1300). In this case, in a state (prompt) where an additional input is requested like “What channel do you want to change to?”, when an additional utterance like “Number58” is received, the first server1100may transmit the additional utterance to the TV (executor1400) from the refrigerator (receiver1300), and may cause an operation corresponding to the additional utterance to be performed at the TV (executor1400). For example, the first server110may continuously process an additional utterance in the connection binding state of the receiver1300and the executor1400. An utterance saying “Change to channel58on the TV (executor1400)” may be received at the speaker (receiver1300). In this case, after the TV (executor1400) normally changes to channel58, a continuous utterance like “Change channel up” or “Change to channel52” may be additionally received. After the TV (executor1400) normally changes to channel58, when the continuous utterance like “Change channel up” or “Change to channel52” is additionally inputted, the first server1100may transmit the continuous utterance to the executor1400. By doing so, the continuous utterance like “Change channel up” or “Change to channel52” may be additionally processed.

According to various embodiments, in the connection binding state of the receiver1300and the executor1400, the first server1100may perform an utterance operation for controlling the same executor1400while changing the receiver1300.

For example, an utterance saying “Find the Infinite Challenge on the TV (executor1400)” may be received at the speaker (receiver1300) positioned in a living room. Concurrently, the user may move to a kitchen in the state in which a list of episodes of the Infinite Challenge is displayed on the TV (executor1400). A continuous utterance saying “Play the second episode” may continue at the refrigerator (receiver1300) positioned in the kitchen. In this case, the first server1100may transmit the continuous utterance to the TV (executor1400) such that the second episode of the Infinite Challenge can be played on the TV (executor1400). The continuous utterance may be processed within the follow-up utterance waiting time after the previous utterance is inputted. For example, when the follow-up utterance waiting time is exceeded, the utterance saying “Play the second episode on the TV (executor1400)” may be received at the refrigerator (receiver1300) positioned in the kitchen. In this case, when the connection maintaining time of the TV (executor1400) is maintained, the follow-up utterance may be processed. On the other hand, when the connection maintaining time of the TV (executor1400) is ended, the follow-up utterance may not be processed, and the follow-up utterance may be processed after a new connection is generated. For example, in the connection binding state of the receiver1300and the executor1400, the first server1100may process an utterance while changing the executor1400. In this case, an utterance saying “ . . . on the TV (executor1400)” may be received at the speaker (receiver1300) and then an utterance saying “ . . . on the refrigerator (executor1400)” may be continuously received at the speaker (receiver1300). Thereafter, an utterance “ . . . on the TV (executor1400)” may continuously occur at the speaker (receiver1300). In this case, the first server1100may continuously transmit a request for an utterance to the TV (executor1400). By doing so, the continuous utterance may be continuously performed at the TV (executor1400).

According to various embodiments, the first server110may control the operation of the same executor1400in a state where the connection with the receiver1300is terminated. The first server110may cause an utterance operation to be performed. In addition, the first server1100may control the operation of the same receiver1300in a state where the connection with the executor1400is terminated.

FIG.14Aillustrates an example flowchart S500for providing an intelligent assistance service at the first server1100according to various embodiments.

Referring toFIGS.5,8, and14A, in operation S510, a first electronic device (for example, the receiver1300) may receive a user utterance. The first server1100may generate a first channel with the first electronic device (for example, the receiver1300).

In operation S520, the first electronic device (for example, the receiver1300) may transmit a request corresponding to the utterance to the first server1100, and the first server1100may analyze the utterance.

In operation S530, the first server1100may determine which executor should perform an operation corresponding to the utterance from among a plurality of electronic devices. For example, the first server1100may determine whether the operation of the utterance should be performed at a second electronic device (for example, the executor1400) from among the plurality of electronic devices.

When the operation of the utterance is performed at the second electronic device (for example, the executor1400) (“Yes” in operation S530), the first server1100may generate a second channel with the second electronic device (for example, the executor1400) in operation S540. The first server1100may form a connection binding with the first electronic device and the second electronic device by using first device information of the first electronic device (for example, the receiver1300), which is received through the first channel, and second device information of the second electronic device (for example, the executor1400), which is received through the second channel.

In operation S550, the first server1100may determine whether performance of the operation corresponding to the utterance is completed at the second electronic device (for example, the executor1400).

When the performance of the operation corresponding to the utterance is completed at the second electronic device (for example, the executor1400) (“Yes” in operation S550), the first server1100may terminate the connection binding with the first electronic device (for example, the receiver1300) and the second electronic device (for example, the executor1400) in operation S560.

FIG.14Billustrates an example flowchart S600for providing an intelligent assistance service at the first server1100according to various embodiments.

Referring toFIGS.5,8, and14B, in operation S610, the first electronic device (for example, the receiver1300) may receive a user utterance. The first server1100may generate the first channel with the first electronic device.

In operation620, the first electronic device may transmit a request corresponding to the utterance to the first server1100, and the first server1100may analyze the utterance.

In operation630, the first server1100may determine which executor should perform an operation corresponding to the utterance from among the plurality of electronic devices. For example, the first server1100may determine whether the operation corresponding to the utterance should be performed at the second electronic device (for example, the executor1400) from among the plurality of electronic devices.

When the operation corresponding to the utterance is performed at the second electronic device (for example, the executor1400) as a result of determining in operation S630(“Yes” in operation S630), the first server1100may generate the second channel with the second electronic device (for example, the executor1400) in operation S640. The first server1100may form a connection binding with the first electronic device and the second electronic device by using the first device information of the first electronic device (for example, the receiver1300), which is received through the first channel, and the second device information of the second electronic device (for example, the executor1400), which is received through the second channel.

In operation650, the first server1100may count a connection maintaining time with the first electronic device (for example, the receiver1300) and the second electronic device (for example, the executor1400), from the time that the connection binding is formed based on information received from the second electronic device (for example, the executor1400).

In operation660, the first server1100may determine whether the connection maintaining time with the first electronic device (for example, the receiver1300) and the second electronic device (for example, the executor1400) is exceeded. Herein, the first server1100may count a first connection maintaining time with respect to the connection with the first electronic device (for example, the receiver1300), and may determine whether the first connection maintaining time is exceeded. In addition, the first server1100may count a second connection maintaining time with respect to the connection with the second electronic device (for example, the executor1400), and may determine whether the second connection maintaining time is exceeded.

When the connection maintaining time with the first electronic device (for example, the receiver1300) and the second electronic device (for example, the executor1400) is exceeded as a result of determining in operation S660(“Yes” in operation S660), the first server1100may terminate the connection with the first electronic device (for example, the receiver1300) and the second electronic device (for example, the executor1400) in operation S670.

When the connection maintaining time with the first electronic device (for example, the receiver1300) and the second electronic device (for example, the executor1400) is not exceeded as a result of determining in operation S660(“No” in operation S660), the first server1100may maintain the connection binding with the first electronic device (for example, the receiver1300) and the second electronic device (for example, the executor1400). Thereafter, the first server1100may determine whether performance of the operation corresponding to the utterance is completed at the second electronic device (for example, the executor1400) in operation S680.

When the performance of the operation corresponding to the utterance is completed at the second electronic device (for example, the executor1400) as a result of determining in operation S680(“Yes” in operation S680), the first server1100may terminate the connection binding with the first electronic device (for example, the receiver1300) and the second electronic device (for example, the executor1400).

When the performance of the operation corresponding to the utterance is not completed at the second electronic device (for example, the executor1400) as a result of determining in operation S680(“No” in operation S680), the first server1100may maintain the connection binding with the first electronic device (for example, the receiver1300) and the second electronic device (for example, the executor1400) in operation S690.

FIG.15illustrates an example view of an example of processing an utterance that does not clearly say a space where an executor is positioned.

Referring toFIGS.8and15, when the receiver1300and the executor1400are positioned in the same first space51, the first server1100may process an utterance that does not clearly say the position space of the receiver1300and the executor1400, such that an operation corresponding to the utterance is performed. Since the first server1100already recognizes where the receiver1300and the executor1400are positioned, the first server1100may process the utterance even when a user does not clearly say the position space Si of the receiver1300and the executor1400. For example, when the user utters “Hi Bixby, turn off the air conditioner” to the receiver1300positioned in the same space Si to control the executor1400, the first server1100may generate a first channel with the receiver1300based on the utterance acquired at the receiver1300. In addition, the first server1100may generate a second channel with the executor1400. The first server1100may connection-bind the receiver1300and the executor1400through the first channel and the second channel. The first server1100may determine the executor1400by using the utterance acquired at the receiver1300. The first server1100may transmit deep link information or information on the utterance to the executor1400, such that an operation corresponding to the utterance can be performed at the executor1400(for example, the air conditioner is turned off).

When performance of the operation corresponding to the utterance (for example, the air conditioner is turned off) is completed at the executor1400, the executor1400may transmit information regarding the completion of the performance of the operation corresponding to the utterance to the first server1100. The first server110may transmit the information regarding the completion of the performance of the operation corresponding to the utterance to the receiver1300. The receiver1300may provide a message saying “I turned off the air conditioner” in response to the completion of the performance of the operation corresponding to the utterance saying “Hi, Bixby, turn off the air conditioner”.

FIG.16illustrates a view of an example of processing an utterance that does not clearly say a space where an executor is positioned.

Referring toFIGS.8and16, when the receiver1300and the executor1400are positioned in different spaces, the first server1100may process an utterance that does not clearly say position spaces of the receiver1300(for example, a mobile phone) and the executor1400(for example, a TV), such that an operation corresponding to the utterance is performed. Since the first server1100already recognizes where the receiver1300and the executor1400are positioned, the first server1100may process the utterance even when the user does not clearly say the position spaces of the receiver1300and the executor1400. For example, the user may utter “Hi Bixby, play the music channel on the TV” to the receiver1300positioned in a first space (for example, a room), in order to control the TV (executor1400) positioned in a second space (for example, a living room). In this case, the first server1100may generate a first channel with the receiver1300based on the utterance acquired at the receiver1300. In addition, the first server1100may generate a second channel with the executor1400. Herein, the first server1100may determine the executor1400by using the utterance acquired at the receiver1300, and may generate the second channel with the executor1400. The first server1100may form a connection binding between the receiver1300(for example, the mobile phone) and the executor1400(for example, the TV) by using the first channel and the second channel. The first server1100may transmit an operation (action) regarding the utterance or deep link information to the executor1400based on context information received from the executor1400(for example, the TV). By doing so, the first server1100may cause the operation (for example, turning on the TV and changing to the music channel) corresponding to the utterance to be performed at the executor1400(for example, the TV).

When performance of the operation corresponding to the utterance is completed at the executor1400(for example, the TV), the executor1400(for example, the TV) may transmit information indicating that the performance of the operation corresponding to the utterance is completed to the first server1100. The first server1100may transmit the information indicating that the performance of the operation corresponding to the utterance is completed to the receiver1300(for example, the mobile phone). The receiver1300(for example, the mobile phone) may display a message saying “I turned on the music channel on the TV” in response to the performance of the operation corresponding to the utterance “Hi Bixby, play the music channel on the TV” being completed.

For example, the first server1100may maintain the connection binding between the receiver1300and the executor1400within an additional input waiting time after the operation corresponding to the first utterance of the user is performed. In this case, information regarding the existence of the executor1400connected may be provided through a UI (for example, a display, a speaker, or illumination) while the additional input waiting time is maintained.

For example, in a state where the first channel and the second channel are connected, a user's follow-up utterance may be received at the receiver1300(for example, the mobile phone). For example, a follow-up utterance saying “Hi Bixby, turn down the volume” may be received at the receiver1300(for example, the mobile phone). The first server1100may transmit the follow-up utterance acquired at the receiver1300(for example, the mobile phone) positioned in the first space to the executor1400(for example, the TV) positioned in the second space, such that an operation (for example, turning down the TV volume) corresponding to the follow-up utterance can be performed at the executor1400(for example, the TV). When performance of the operation corresponding to the follow-up utterance is completed at the executor1400(for example, the TV), the executor1400(for example, the TV) may transmit information indicating that the performance of the operation corresponding to the follow-up utterance is completed to the first server1100. The first server1100may transmit the information indicating that the performance of the operation corresponding to the follow-up utterance is completed to the receiver1300(for example, the mobile phone). The receiver1300(for example, the mobile phone) may display a message saying “I turned down the TV volume” in response to the performance of the operation corresponding to the utterance “Hi Bixby, turn down the volume” being completed. Herein, when the message saying “I turned down the TV volume” cannot be displayed through a sound, the message may be displayed by text through a display screen or a flashing lamp.

For example, a follow-up utterance that does not indicate the executor1400may be received at the receiver1300(for example, the mobile phone). Herein, the first server1100may receive state information of the receiver1300(for example, the mobile phone) through the first channel connected with the receiver1300(for example, the mobile phone). The first server1100may receive state information of the executor1400(for example, the TV) through the second channel with the executor1400(for example, the TV). The first server1100may recognize that the executor1400for the follow-up utterance is the TV, based on the state information of the receiver1300(for example, the mobile phone) and the executor1400(for example, the TV). The first server1100may transmit an operation (action) corresponding to the utterance or deep link information to the executor1400(for example, the TV), such that the operation corresponding to the utterance can be performed at the executor1400(for example, the TV).

FIG.17illustrates a view of an example of providing a response to an utterance at a receiver without a display.

Referring toFIGS.8and17, in the case where the receiver1300(for example, a speaker) and the executor1400(for example, a TV) are positioned in different spaces, the first server1100may process an utterance to cause an operation corresponding to the utterance to be performed when position spaces of the receiver1300(for example, the speaker) and the executor1400(for example, the TV) are specified. For example, the user may utter “Hi Bixby, turn on the music channel on the TV” to the receiver1300(for example, the speaker) positioned in a first space in order to control the TV (for example, the executor1400) positioned in a second space. In this case, the first server1100may generate a first channel with the receiver1300(for example, the speaker). The receiver1300(for example, the speaker) may transmit utterance information to the first server1100through the first channel. In addition, the first server1100may analyze the utterance and may determine the executor1400to perform the operation corresponding to the utterance, and may generate a second channel with the executor1400(for example, the TV). The executor1400(for example, the TV) may transmit current context information to the first server1100by using the second channel. The first server1100may form a connection binding between the receiver1300(for example, the speaker) and the executor1400(for example, the TV) through the first channel and the second channel. The first server1100may determine the operation (action) to be performed at the executor1400(for example, the TV) or deep link by using the utterance information received at the receiver1300(for example, the speaker) through the first channel. The first server1100may transmit the operation (action) to be performed or the deep link to the executor1400(for example, the TV) through the second channel, such the operation (action) (for example, tuning on the TV and changing to the music channel) corresponding to the utterance can be performed.

When performance of the operation corresponding to the utterance is completed at the executor1400(for example, the TV), the executor1400(for example, the TV) may transmit information indicating that the performance of the operation corresponding to the utterance is completed to the first server1100. The first server1100may transmit the information indicating that the performance of the operation corresponding to the utterance is completed to the receiver1300(for example, the speaker). The receiver1300(for example, the speaker) may provide a response saying “I turned on the music channel on the TV” in response to the performance of the operation corresponding to the utterance “Hi Bixby, turn on the music channel on the TV” being completed.

According to an embodiment, when the receiver1300(for example, the speaker) includes a display to display a UI screen, the message “I turned on the music channel on the TV” may be displayed on the UI screen. According to an embodiment, when the receiver1300(for example, the speaker) does not include a display to display a UI screen, a screen may not be displayed, and the message “I turned on the music channel on the TV” may be provided as a sound.

FIG.18illustrates an example view of an intelligent assistance system2000including a first server2100, and operations of a receiver2300, a plurality of executors2400, and the first server2100according to an embodiment.

Referring toFIG.18, the intelligent assistance system2000may include the first server2100(for example, the server108ofFIG.1, the intelligent server200ofFIG.2or the first server1100ofFIG.7), a second server2200(for example, the server108ofFIG.1or the second server1200ofFIG.8), the receiver2300(for example, the receiver1300ofFIG.8), and the plurality of executors2400(for example, the plurality of executors1400ofFIG.8). AlthoughFIG.18illustrates that one receiver2300is provided, a plurality of receivers2300may be disposed in a predetermined space.

The first server2100and the receiver2300may be connected with each other by using a wired or wireless network. The first server2100and the second server2200may be connected with each other by using a wired or wireless network. The first server2100and the plurality of executors2400may be connected with each other by using a wired or wireless network. The receiver2300and the plurality of executors2400may be connected with each other through the first server2100.

In an embodiment, a first channel (for example, a receiver channel) may be generated to transmit and receive data between the receiver2300and the first server2100. For example, generation of the first channel may be initiated by the receiver2300. For example, generation of the first channel may be initiated by the first server2100. For example, the receiver2300may request the first server2100to generate the first channel, based on response information received from the first server2100. The first server2100may generate the first channel for communicating with the receiver2300. Herein, the response information may be managed by a response manager2114of the first server2100.

In an embodiment, a second channel (for example, an executor channel) may be generated to transmit and receive data between at least one executor (for example, the first executor2410) (target executor) of the plurality of executors2400, and the first server2100. For example, generation of the second channel may be initiated by the first server2100. For example, the first executor2410may request the first server2100to generate the second channel, based on response information received from the first server2100. The first server2100may generate the second channel for communicating with the first executor2410. Herein, the response information may be managed by the response manager2114of the first server2100.

However, this should not be considered as limiting. The receiver2300and the plurality of executors2400may be connected with each other in a D2D method without passing through the first server2100and/or the second server2200.

According to various embodiments, the receiver2300may perform a function of a listener of receiving a user voice. The receiver2300may include various devices (for example, an AI speaker) including a configuration related to voice recognition and a voice input device (for example, a microphone). For example, the receiver2300may include the electronic device101ofFIG.1or the user terminal400ofFIG.2. The receiver2300may acquire an utterance from a user (for example, the user1010ofFIG.5) through a voice input device. The utterance may include a wake-up utterance instructing to enable and/or invoke an intelligent assistance service, and/or a control utterance instructing an operation (for example, power control or volume control) of a hardware and/or software configuration included in the plurality of executors2400. In an embodiment, the wake-up utterance and the control utterance may be acquired, respectively. In an embodiment, the control utterance may be acquired along with the wake-up utterance.

According to various embodiments, the receiver2300may generate a control message (or a control instruction), based on at least a portion of the acquired utterance (or utterance data). The receiver2300may transmit the generated control message to the first executor2410(for example, the target executor) which will perform the operation corresponding to the utterance from among the plurality of executors2400by using the first server2100. The control message may be generated based on a result of processing utterance data. The first executor2410to perform the operation corresponding to the utterance from among the plurality of executors2400may be referred to as a target executor.

In an embodiment, processing of the utterance data may be performed through natural language processing by the receiver2300and/or natural language processing by the first server2100. For example, the receiver2300may process the utterance data by itself by using a voice processing module (for example, the voice processing module1340ofFIG.6). In addition, the receiver2300may transmit the utterance data to the first server2100and may request a result of processing the utterance data. For example, the receiver2300may include an utterance data processing capability of a first level, for example, a voice recognition module (for example, the voice recognition module1340-1ofFIG.6), and an NLU module (for example, the NLU module1340-2ofFIG.6) of the first level. In addition, the first server2100may include an utterance data processing capability of a second level which is higher than the first level, for example, a voice recognition module (for example, the voice recognition module1140-1ofFIG.7) and an NLU module (for example, the NLU module1140-2ofFIG.7) of the second level. In addition, the receiver2300may provide a result of performing the operation corresponding to the user utterance. The receiver2300may include a sound output device (for example, a speaker), a display, and/or one or more lamps so as to provide the result of performing the operation corresponding to the utterance.

According to various embodiments, the plurality of executors2400may include a smartphone (for example, the smartphone1410ofFIG.5), a computer (for example, the computer1420ofFIG.5, a personal computer, or a notebook computer), a television (for example, the television1430ofFIG.5), a refrigerator (for example, the refrigerator1440ofFIG.5), and/or an illumination device (for example, the illumination device1450ofFIG.5). In addition, the executors2400according to various embodiments may further include an air conditioner, a thermostat, an alarm device, a gas valve control device, or a door lock device.

According to an embodiment, each of the plurality of executors2400may include a communication circuit to establish communication (for example, the second channel) with the first server2100by using a designated protocol (for example, Bluetooth, WiFi, or Zigbee), and to transmit and/or receive a variety of information. According to an embodiment, the plurality of executors2400may transmit information regarding their own operation states (for example, on/off information of a device) to the receiver2300or the first server2100. In addition, the plurality of executors2400may receive a control message (for example, an on/off control command of a device, or other operation control commands of a device) from the receiver2300or the first server2100, and may perform an operation corresponding to the control message. In addition, the plurality of executors2400may transmit a result of performing the operation corresponding to the control message to the receiver2300or the first server2100.

In an embodiment, at least some components of the first server2100may correspond to at least some components of the receiver2300(for example, the receiver1300ofFIG.6). For example, the first server2100may include a processor (for example, the processor1110ofFIG.7), a memory (for example, the memory1120ofFIG.7), a communication module (for example, the communication module1130ofFIG.7), a voice processing module (for example, the voice processing module1140ofFIG.7), and/or a matching information generation module (for example, the matching information generation module1150ofFIG.7). Herein, the matching information generation module may be selectively included in the first server2100.

According to various embodiments, the processor (for example, the processor1110ofFIG.7) of the first server2100may control the voice processing module (for example, the voice processing module1140ofFIG.7) to process utterance data received from the receiver2300. In addition, the processor (for example, the processor1110ofFIG.7) may provide the result of processing the utterance data to the receiver2300. For example, the result of processing may include at least one of an intent on the user input, a domain for executing a task, or data (for example, a slot or a task parameter) required to understand the user's intent.

According to various embodiments, the processor (for example, the processor1110ofFIG.7) of the first server2100may control to provide intent masking information to the receiver2300as a result of processing. As described above, the intent masking information may be information in which an utterance processing object is designated with respect to an intent. In addition, the intent masking information may be generated by the matching information generation module (for example, the matching information generation module1150ofFIG.7).

According to various embodiments, the voice processing module (for example, the voice processing module1140ofFIG.7) of the first server2100may include the voice recognition module (for example, the voice recognition module1140-1ofFIG.7) and the NLU module (for example, the NLU module1140-2ofFIG.7), similarly to the voice processing module (for example, the voice processing module1340ofFIG.6) of the receiver2300. According to an embodiment, the voice processing module (for example, the voice processing module1140ofFIG.7) of the first server2100may have a higher processing capability than the utterance data processing capability of the receiver2300. For example, a result of processing an utterance (or utterance data), acquired by the voice processing module (for example, the voice processing module1140ofFIG.7) of the first server2100, may have higher accuracy than a result of processing an utterance, acquired by the voice processing module (for example, the voice processing module1340ofFIG.6) of the receiver2300.

According to various embodiments, the matching information generation module (for example, the matching information generation module1150) of the first server2100may generate intent masking information based on the result of processing by the receiver2300(for example, the voice processing module1340ofFIG.6). The intent masking information may be associated with a matching rate between a first processing result on utterance data performed by the receiver2300(for example, the voice processing module1340ofFIG.6), and a second processing result on utterance data performed by the first server2100(for example, the voice processing module1140ofFIG.7). According to an embodiment, the first server2100may receive the first processing result from the receiver2300, and the matching information generation module (for example, the matching information generation module1150ofFIG.7) may identify a matching rate regarding the first processing result by comparing the received first processing result and the second processing result performed by the first server2100. In addition, the matching information generation module (for example, the matching information generation module1150) may generate intent masking information in which one of the receiver2300or the first server2100is designated as an object to process the received utterance, based on the identified matching rate.

In an embodiment, the first server2100may identify the first executor2410(for example, the target executor) as an executor to perform the operation corresponding to the utterance acquired at the receiver2300, and may connect the receiver2300and the executor2410. The first server2100may generate the first channel for communicating with the receiver2300. The first server2100may generate the second channel for communicating with the first executor2410which performs the operation corresponding to the utterance from among the plurality of executors2400.

For example, the receiver2300may generate first device information (for example, state information, necessary response information, or identification information of the receiver2300). According to an embodiment, the receiver2300may transmit the first device information to the first server2100by using the first channel.

In an embodiment, the first executor2410may generate second device information (for example, state information, necessary response information, or identification information of the first executor2410). According to an embodiment, the first executor2410may transmit the second device information to the first server2100by using the second channel. In an embodiment, the first server2100may receive the first device information of the receiver2300through the first channel, and may control maintenance, termination of a connection, and/or reconnection with the receiver2300. The first server2100may receive the second device information of the first executor2410through the second channel, and may control maintenance, termination of a connection, and/or reconnection with the first executor2410. In addition, the first server2100may control information exchange and information distribution between the receiver2300and the plurality of executors2400.

In an embodiment, for the connection between the receiver2300and the plurality of executors2400, the receiver2300may transmit the first device information thereof to the first server2100and/or the second server2200. The plurality of executors2400may transmit their respective second device information to the first server2100and/or the second server2200. The second server2200may store and manage the first device information and the second device information for the connection between the receiver2300and the plurality of executors2400. The second server2200may provide the first device information of the receiver2300and the second device information of each of the plurality of executors2400to the first server2100. The first server2100and the second server2200may be disposed with different configurations. However, this should not be considered as limiting, and the first server2100and the second server2200may be configured with the same configuration. AlthoughFIG.18illustrates that the first server2100and the second server2200are separately configured, the first server2100and the second server2200may be integrated into one server.

In an embodiment, the receiver2300and the first server2100may transmit and/or receive data through a capsule execution service (CES)2500. In addition, the plurality of executors2400and the second server2200may also transmit and/or receive data through the CES2500. The CES2500may support management of the first channel connecting the receiver2300and the first server2100. In addition, the CES2500may support management of the second channel connecting the plurality of executors2400and the first server2100. The CES2500may indicate, to the receiver2300, the first server2100to be connected from among the plurality of servers. In addition, the CES2500may indicate, to the executor2400, the first server2100to be connected from among the plurality of servers. That is, the CES2500may inform the receiver2300and the plurality of executors2400of which server should be connected from among the plurality of servers.

According to various embodiments, the processor (for example, the processor1110ofFIG.7) of the first server2100may include a conversation manager2110(for example, the conversation manager1112ofFIG.8), a device dispatch2120, a classifier3140(for example, the classifier1114ofFIG.8), an event manager2160(for example, the event manager1116ofFIG.8), and/or a plurality of CANs2170(for example, the CAN1118ofFIG.8). According to various embodiments, the conversation manager2110may include the response manager2114and a connection time module2116.

According to various embodiments, the conversation manager2110may generate, maintain, and manage a plurality of channels2112. In an embodiment, the conversation manager2110may generate, maintain, and manage the first channel (listener channel) for communicating with the receiver2300, and the second channel (executor channel) for communicating with the first executor2410(for example, the target executor). In an embodiment, the conversation manager2110may generate the first channel for communicating with the receiver2300through the response manager2114after an utterance is acquired at the receiver2300. The conversation manager2110may determine the first executor2410(for example, the target executor) to perform an operation (or an action) corresponding to the utterance from among the plurality of executors2400, based on the utterance received from the receiver2300, and may generate the second channel for communicating with the first executor2410through the response manager2114.

According to various embodiments, the response manager2114may distribute responses generated by executing the operation (or action) to the first channel (for example, the receiver channel) and the second channel (for example, the executor channel), after the operation corresponding to the utterance is performed at the first executor2410. The response manager2114may transmit the responses to the receiver2300and the first executor2410through the first channel (for example, the receiver channel) and the second channel (for example, the executor channel).

According to various embodiments, the connection time module2116may manage a connection maintaining time of the first channel and the second channel when the first channel for communicating with the receiver2300and the second channel for communicating with the first executor2410are formed. The connection time module2116may manage effectiveness (for example, maintaining a connection) for a predetermined connection maintaining time. According to an embodiment, the connection time module2116may include the configuration of the life-time module1112bofFIG.8and the configuration of the connection module1112cofFIG.8.

The connection time module2116may count a time from a time that the first channel is generated, and may determine whether a connection maintaining time (for example, a connection maintaining time of the receiver2300) is exceeded after the first channel is generated. In addition, the connection time module2116may count a time from a time that the second channel is generated, and may determine whether a connection maintaining time (for example, a connection maintaining time of the first executor2410) is exceeded after the second channel is generated. In addition, the connection time module2116may count a time from a time that an utterance is acquired at the receiver2300, and may determine whether a connection maintaining time (for example, an additional information input waiting time) is exceeded after the utterance is acquired. In addition, the connection time module2116may count a time from a time that the utterance is performed at the first executor2410, and may determine whether a connection maintaining time (for example, a follow-up utterance waiting time) is exceeded after the utterance is performed.

According to various embodiments, the receiver2300and the plurality of executors2400may have connection maintaining times different according to each device (for example, a connection maintaining time of the speaker is 8 seconds, a connection maintaining time of the air conditioner is 30 seconds, a connection maintaining time of the TV is 1 minute, or a connection maintaining time of the refrigerator is 2 minutes). In an embodiment, when the connection maintaining time of the receiver2300(for example, a first connection maintaining time) is ended, the receiver2300may inform a corresponding device, that is, the first executor2410, of a disconnection request via the first server2100. When the connection maintaining time of the first executor2410(for example, a second connection maintaining time) is ended, the first executor2410may inform a corresponding device, that is, the receiver2300, of a disconnection request via the first server2100. When the disconnection request is executed by any one of the receiver2300and the first executor2410, the first server2100may disconnect the receiver2300and the first executor2410.

In an embodiment, when a reconnection request is executed by any one of the receiver2300and the first executor2410, the first server2100may reconnect the receiver2300and the first executor2410.

In an embodiment, the connection time module2116may manage the first connection maintaining time of the receiver2300and the second connection maintaining time of the first executor2410. In a state where the first connection maintaining time of the receiver2300and the second connection maintaining time of the first executor2410are effective, an operation of continuing the connection maintenance at the receiver2300(for example, an operation indicating continuous use of the receiver2300) may be executed. In an embodiment, when a user's input operation (for example, touch, function selection, function execution, or utterance) for continuing the connection maintenance of the receiver2300is received, the receiver2300may perform an operation of maintaining the connection of the first channel. For example, the receiver2300may transmit a connection maintenance request of the first channel to the first server2100through the first channel. In this case, the connection time module2116may reset or refresh the first connection maintaining time. The connection time module2116may request the first executor2410to reset or refresh the second connection maintaining time.

In an embodiment, in the state where the first connection maintaining time of the receiver2300and the second connection maintaining time of the first executor2410are effective, an operation of continuing the connection maintenance at the first executor2410(for example, an operation indicating continuous use of the first executor2410) may be executed. In an embodiment, when a user's input operation (for example, touch, function selection, or function execution) for continuing (or extending) the connection maintenance of the first executor2410is received, the first executor2410may perform an operation of maintaining the connection of the second channel. For example, the first executor2410may transmit a connection maintenance request of the second channel to the first server2100through the second channel. In this case, the connection time module2116may reset or refresh the second connection maintaining time. The connection time module2116may request the receiver2300to reset or refresh the first connection maintaining time. That is, when the operation of continuing the connection maintenance at any one of the receiver2300and the first executor2410is executed, the connection time module2116may manage to reset or refresh the first connection maintaining time of the receiver2300and the second connection maintaining time of the executor2400.

In an embodiment, the receiver2300and the executors2400may recognize and manage their own channel maintaining time (for example, maintaining, resetting, or refreshing a connection maintaining time). That is, all of the electronic devices (for example, the receiver and/or the executors) may know and manage their own channel maintaining time. The electronic devices may readjust (for example, refresh) connection state information every predetermined time (for example, 5 seconds, 10 seconds, 30 seconds, 60 second, 5 minutes, 10 minutes or 30 minutes). Herein, when the connection of the electronic device (for example, the receiver and/or the executor) should be maintained due to control or additional operation of the user, the electronic device (for example, the receiver and/or the executor) may transmit a control value for resetting or refreshing its own channel maintaining time (for example, a value of a reset or refreshed connection maintaining time) to the first server2100. When the connection maintaining time is not reset or refreshed, the electronic device (for example, the receiver and/or the executor) may disconnect the channel and may transmit a disconnection request of the channel to the first server2100when the connection maintaining time is ended. When a disconnection is requested from any one of the receiver2300and the executor2410, the first server2100may terminate the connection between the receiver2300and the executor2410. Since the receiver2300and the executors2400can manage the connection and the disconnection of the channel through the first server2100, the receiver2300does not need to recognize the connection maintaining time of the executors2400connected thereto, and the executors2400do not need to recognize the connection maintaining time of the receiver2300connected thereto.

In an embodiment, the connection time module2116may identify whether the first connection maintaining time of the receiver2300and the second connection maintaining time of the first executor2410are reset or refreshed. The operation of identifying whether the first connection maintaining time and the second connection maintaining time are reset or refreshed may be performed at the connection time module2116on a predetermined interval basis. The connection time module2116may identify whether the connection maintaining time is reset or refreshed at the same intervals (for example, 5 seconds, 15 seconds, 30 seconds, 1 minute, 3 minutes, 5 minutes or 10 minutes) with respect to the receiver2300and the first executor2410. However, this should not be considered as limiting, and the connection time module2116may identify whether the connection maintaining time is reset or refreshed at different intervals with respect to the receiver2300and the first executor2410(for example, 30 seconds for the receiver2300or 1 minute for the first executor2410).

In an embodiment, the first connection maintaining time of the receiver2300(for example, the speaker) may be set to 1 minute, and the second connection maintaining time of the first executor2410(for example, the TV) may be set to 5 minutes. Herein, when 20 seconds are elapsed, the remaining first connection maintaining time of the receiver2300(for example, the speaker) may be 40 seconds, and the remaining second connection maintaining time of the first executor2410(for example, the TV) may be 4 minutes and 40 seconds. In this case, the operation of continuing the connection maintenance may be executed by the user operating the receiver2300(touching, selecting a function, executing a function, or uttering) or operating the first executor2410(touching, selecting a function, or executing a function). When the operation of continuing the connection maintenance is executed, the connection time module2116may reset the first connection maintaining time of the receiver2300to 1 minute again, and may reset the second connection maintaining time of the first executor2410to 5 minutes again.

In an embodiment, when the first channel is formed between the receiver2300and the first server2100, the receiver2300may transmit utterance information and first device information on the receiver2300to the first server2100by using the first channel. Herein, the receiver2300may also transmit the first device information to the second server2200. In this case, the receiver2300may transmit the first device information to the second server2200via the first server2100. However, this should not be considered as limiting, and the receiver2300may directly transmit the first device information to the second server2200. For example, when a user utterance is received from the receiver2300, the conversation manager2110may determine whether the corresponding utterance can be processed at the receiver2300, and/or which device of the plurality of executors can execute the utterance. When the user utterance can be processed at the first executor2410(for example, the target executor), the conversation manager2110may transmit the first device information to the second server2200.

In an embodiment, the first executor2410may generate second device information (for example, state information, necessary response information, or identification information of the first executor2410). When the second channel is formed between the first executor2410and the first server2100, the first executor2410may transmit the second device information to the first server2100through the second channel. After the first channel and the second channel are formed, the first server2100may connect the receiver2300and the first executor2410based on the first device information received from the receiver2300and the second device information received from the first executor2410.

In the above-described example, the first device information and the second device information are transmitted and received through the first server2100and/or the second server2200. However, this should not be considered as limiting, and the first device information and the second device information may be transmitted and received between the receiver2300and the plurality of executors2400in a D2D method.

In an embodiment, the device dispatch2120may include a rule NLU module2122. When a user utterance is inputted to the first server2100through the receiver2300, the rule NLU module2122may determine whether the utterance is to control another device (for example, at least one executor of the plurality of executors2400). The rule NLU module2122may provide information of the executor2400which executes an operation (or action) corresponding to a result of determining about the utterance to the device dispatch2120.

In an embodiment, the device dispatch2120may receive the information of the executor (for example, the first executor2410) which executes the operation (or action) corresponding to the utterance from the rule NLU module2122, and then may determine whether there is the executor (for example, the first executor2410) to execute the operation (or action). Herein, when there are a plurality of executors to execute the operation (or action) corresponding to the utterance, the device dispatch2120may ask the user which device will execute the operation (or action) corresponding to the utterance again, and may select the executor to execute the operation (or action). In an embodiment, when a connection between the receiver2300and the executor (for example, the first executor2410) to execute the operation (or action) corresponding to the utterance is determined to be effective by the connection time module2116, the device dispatch2120may omit the operation of selecting the executor (for example, the first executor2410) to execute the operation (action) corresponding to the utterance.

In an embodiment, the classifier3140may classify which executor2400will perform the operation corresponding to the utterance requested at the receiver2300from among the plurality of executors2400. The classifier3140may cause utterance information to be transmitted to the executor2400to perform the operation corresponding to the requested utterance, that is, to the target executor. In addition, when the executor to execute the operation (or action) corresponding to the utterance is the first executor2410, the classifier3140may cause a result of performing the utterance to be transmitted to the corresponding receiver2300after the operation corresponding to the utterance is performed at the first executor2410.

In an embodiment, the event manager2160may cause the result of performing the utterance to be transmitted to the receiver2300and/or the first executor2410after the operation corresponding to the utterance is performed. The event manager2160may determine through which channel the result of performing the utterance is transmitted to the receiver2300and/or the first executor2410. The event manager2160may determine whether to transmit the result of performing the utterance without processing, or may determine whether to correct the result of performing the utterance according to a UI form of the receiver2300and/or the first executor2410, and to transmit the corrected result. When the result of performing the utterance should be corrected, the event manager2160may correct the result of performing the utterance according to the UI form of the receiver2300and/or the first executor2410, and may transmit the corrected result of performing the utterance to the receiver2300and/or the first executor2410.

FIG.19illustrates an example flowchart1900for providing an intelligent assistance service at the receiver according to various embodiments.FIG.20illustrates a view of an example of a user interface1970informing that the receiver (for example, a mobile phone) and the executor (for example, a refrigerator) are connected.

Referring toFIGS.19and20, in operation1905, the receiver2300(for example, the mobile phone) may receive a user utterance.

In operation1910, the receiver2300(for example, the mobile phone) may generate a first channel between the receiver2300(for example, the mobile phone) and the first server2100, based on the received user utterance. The user utterance and context information of the receiver2300(for example, the mobile phone) may be transmitted to the first server2100through the first channel.

In operation1915, the first server2100(for example, the classifier3140) may determine which executor (for example, the first executor2410) of the plurality of executors2400will execute an operation (or action) corresponding to the user utterance.

When the first executor2410is determined to execute the operation (or action) corresponding to the user utterance, the first server2100may transmit a connection generation request to the first executor2410in operation1920. When the connection generation request is received, the first executor2410may generate a second channel with the first server2100, and the receiver2300(for example, the mobile phone) and the first executor2410may be connected with each other.

In an embodiment, the first executor2410may transmit its own context information (for example, a response type or a device type) to the first server2100through the second channel. The first server2100may transmit the context information of the first executor2410to execute the operation (or action) corresponding to the utterance to the classifier, by using the context information of the corresponding receiver2300and the first executor2410.

In operation1925, the first server2100may inform the receiver2300that the first executor2140to execute the operation (or action) corresponding to the utterance is connected.

In an embodiment, when execution of the operation (or action) corresponding to the utterance is completed at the first executor2410, information of the result (or response) of executing the operation (or action) may be provided to the event manager2160. The event manager2160may transmit the result (for example, a response) of executing the operation (or action) to the receiver2300and the first executor2410, respectively, according to an information setting value (for example, a response type) required by the receiver2300and the first executor2410. In an embodiment, the event manager2160may transmit information of a result (for example, a response) that is not set by the receiver2300and other executors to the receiver2300and other executors.

In an embodiment, when the user utters “Find the recipe of pizza on the first executor2410(for example, the refrigerator)” through the receiver2300(for example, the mobile phone), the receiver2300(for example, the mobile phone) and the first server2100may generate the first channel. The receiver2300(for example, the mobile phone) and the first server2100may transmit and/or receive the user utterance and context information of the receiver2300(for example, the mobile phone) through the first channel.

In an embodiment, the first server2100may determine whether the utterance “Find the recipe of pizza on the refrigerator”, received through the receiver2300(for example, the mobile phone), should be executed at the receiver2300(for example, the mobile phone) or the executor2400, through the rule NLU module2122of the device dispatch2120. In an embodiment, the first server2100may determine whether an operation (or action) corresponding to the utterance should be executed at the first executor2410(for example, the refrigerator) through the dispatch2120, and may determine whether the first executor2410(for example, the refrigerator) is an effective device.

In an embodiment, when the first executor2410(for example, the refrigerator) is determined to execute the operation (or action) corresponding to the utterance, the first server2100may transmit a connection generation request to the first executor2410(for example, the refrigerator) to form the second channel between the first server2100and the first executor2410(for example, the refrigerator). The first executor2410(for example, the refrigerator) may form the second channel between the first server2100and the first executor2410(for example, the refrigerator), based on the received connection generation request. The first executor2410(for example, the refrigerator) may transmit its own context information to the first server2100through the second channel.

In an embodiment, the first server2100may determine a CAN to perform the operation (or action) from among the plurality of CANs2170through the classifier3140, by using the context information of the receiver2100received through the first channel and the context information of the first executor2140received through the second channel.

In operation1930, the receiver2300may display connection information1972informing that the first executor2410(for example, the refrigerator) is connected, for the user through a UI1970. An example of a content informing the connection information1972may be “Please check your kitchen fridge”, or the content of the connection information1972may be displayed through a display (for example, the display440ofFIG.4) in various other forms. The UI1970informing the connection information may be displayed through the receiver2300until the connection is lost after the operation (or action) corresponding to the utterance is executed at the first executor2140(for example, the refrigerator). In an embodiment, a result (for example, a response) of executing the operation (or action) corresponding to the utterance may be displayed not only through the receiver2300, but also through a display of the first executor2140(for example, the refrigerator).

In operation1935, the first server2100may request the first executor2140(for example, the refrigerator) to execute the operation (or action) corresponding to the utterance. In an embodiment, the first server2100may request the first executor2140(for example, the refrigerator) to execute the operation (or action) corresponding to the utterance through the second channel.

In operation1940, the first executor2140(for example, the refrigerator) may execute the operation (or action) corresponding to the utterance. In an embodiment, in the state where the receiver2300(for example, the mobile phone) and the first executor2410(for example, the refrigerator) are connected through the utterance of the user “Find the recipe of pizza on the refrigerator”, when the user makes the next utterance “Tell me the weather” to the receiver2300(for example, the mobile phone), an operation (or action) for the corresponding utterance may be executed at the first executor2140(for example, the refrigerator).

In operation1945, the first executor2140(for example, the refrigerator) may notify the first server2100of the result of executing the operation (or action) corresponding to the utterance.

In operation1950, the first server2100may notify the receiver2300of the result of executing the operation (or action) at the first executor2140(for example, the refrigerator).

In operation1955, the receiver2300may receive a new user utterance. The receiver2300may display connection termination (or uplink) on the UI1970. When the user wishes to execute the utterance “Tell me the weather” through the receiver2300(for example, the mobile phone), rather than through the first executor2140(for example, the refrigerator), the user may touch (or click) the connection termination (or unlink) menu1974(for example, Unlink fridge) displayed on the UI1970.

When the connection termination (or unlink) menu1974displayed on the UI1970is selected on the receiver2300(for example, the mobile phone), the receiver2300(for example, the mobile phone) may transmit the connection termination (or unlink) to the first server2100in operation1960.

In operation1965, the first server2100may transmit the connection termination (or unlink) to the first executor2410(for example, the refrigerator), and may disable the second channel, thereby disconnecting the first executor2410(for example, the refrigerator).

In an embodiment, the user may return back to a home screen by touching (or clicking) a home key of the receiver2300(for example, the mobile phone), and then may input the utterance “Tell me the weather” to the receiver2300(for example, the mobile phone). In this case, the operation (or action) corresponding to the utterance may be executed at the receiver2300(for example, the mobile phone) rather than at the first executor2410(for example, the refrigerator), and the result of executing the operation corresponding to the utterance may be provided to the user.

FIG.21illustrates an example view of a receiver and an executor which are connected in a D2D method according to an embodiment.

Referring toFIG.21, a first electronic device3000may operate as a receiver (for example, the receiver2300ofFIG.18), and a second electronic device4000may operate as an executor (for example, the executor2400ofFIG.18). However, this should not be considered as limiting, and the second electronic device4000may operate as a receiver and the first electronic device3000may operate as an executor. The first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may be connected with each other in a D2D method by using a wired or wireless network.FIG.21illustrates that one first electronic device3000(for example, the receiver) and one second electronic device4000(for example, the executor) are connected in the D2D method, but this should not be considered as limiting, and one or a plurality of first electronic devices3000(for example, the receivers) and a plurality of second electronic devices4000(for example, the executors) may be connected in the D2D method.

According to various embodiments, the first electronic device3000(for example, the receiver) may perform a function of a listener of receiving a user voice. The first electronic device3000(for example, the receiver) may include various devices (for example, an AI speaker) including a configuration related to voice recognition and a voice input device (for example, a microphone). For example, the first electronic device3000(for example, the receiver) may include the electronic device101ofFIG.1or the user terminal400ofFIG.2. The first electronic device3000(for example, the receiver) may acquire an utterance from a user (for example, the user1010ofFIG.5) through the voice input device (for example, the microphone420ofFIG.2). The utterance may include a wake-up utterance instructing to enable and/or invoke an intelligent assistance service, and/or a control utterance instructing an operation (for example, power control, volume control) of a hardware and/or software configuration included in the second electronic device4000(for example, the executor). In an embodiment, the wake-up utterance and the control utterance may be acquired, respectively. In an embodiment, the control utterance may be acquired along with the wake-up utterance.

According to various embodiments, the first electronic device3000(for example, the receiver) may generate a control message (or a control instruction), based on at least a portion of the acquired utterance (or utterance data). The first electronic device3000(for example, the receiver) may transmit the generated control message to the second electronic device4000(for example, the executor) which will perform an operation corresponding to the utterance. The control message may be generated based on a result of processing the utterance data.

In an embodiment, the processing of the utterance data may be performed through natural language processing by the first electronic device3000(for example, the receiver). For example, the first electronic device3000(for example, the receiver) may process the utterance data by itself by using a voice processing module (for example, the voice processing module1340ofFIG.6).

For example, the first electronic device3000(for example, the receiver) may include an utterance data processing capability of a first level, for example, a voice recognition module (for example, the voice recognition module1340-1ofFIG.6) of the first level, and an NLU module (for example, the NLU module1340-2ofFIG.6) of the first level. In addition, the first electronic device3000(for example, the receiver) may include an utterance data processing capability of a second level which is higher than the first level, for example, a voice recognition module (for example, the voice recognition module1140-1ofFIG.7) of the second level, and an NLU module (for example, the NLU module1140-2ofFIG.6) of the second level. In addition, the first electronic device3000(for example, the receiver) may provide the result of performing the operation corresponding to the user utterance. The first electronic device3000(for example, the receiver) may include a sound output device (for example, a speaker), a display, and/or one or more lamps to provide the result of performing the operation corresponding to the utterance.

According to various embodiments, the second electronic device4000(for example, the executor) may include a smartphone (for example, the smartphone1410ofFIG.5), a computer (for example, the computer1420ofFIG.5, a personal computer, or a notebook computer), a television (for example, the television1430ofFIG.5), a refrigerator (for example, the refrigerator1440ofFIG.5), and/or an illumination device (for example, the illumination device1450ofFIG.5). In addition, the second electronic device4000(for example, the executor) according to various embodiments may further include an air conditioner, a thermostat, an alarm device, a gas valve control device, or a door lock device.

According to an embodiment, the second electronic device4000(for example, the executor) may include a communication circuit to form a channel with the first electronic device3000(for example, the receiver) by using a designated protocol (for example, Bluetooth, WiFi, or Zigbee), and to transmit and/or receive a variety of information. According to an embodiment, the second electronic device4000(for example, the executor) may transmit information on its own operation state (for example, on/off information of a device) to the first electronic device3000(for example, the receiver). In addition, the second electronic device4000(for example, the executor) may receive a control message (for example, an on/off control command of a device, or other operation control commands of a device) from the first electronic device3000(for example, the receiver), and may execute an operation corresponding to the control message. In addition, the second electronic device4000(for example, the executor) may transmit a result of executing the operation corresponding to the control message to the first electronic device3000(for example, the receiver).

In an embodiment, at least some components of the first electronic device3000(for example, the receiver) may correspond to at least some components of the second electronic device4000(for example, the executor). For example, the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may include a processor (for example, the processor1110ofFIG.7), a memory (for example, the memory1120ofFIG.7), a communication module (for example, the communication module1130ofFIG.7), a voice processing module (for example, the voice processing module1140ofFIG.7) and/or a matching information generation module (for example, the matching information generation module1150ofFIG.7).

According to various embodiments, the processor3100(for example, the processor1110ofFIG.7) of the first electronic device3000(for example, the receiver) may include a conversation manager3110(for example, the conversation manager1112ofFIG.8or the conversation manager2110ofFIG.18), a device dispatch3150(for example, the device dispatch2120ofFIG.18), a classifier3140(for example, the classifier1114ofFIG.8or the classifier3140ofFIG.18), an event manager3160(for example, the event manager1116ofFIG.8or the event manager2160ofFIG.18), and/or a CAN3170(for example, the CAN1118ofFIG.8or the CAN2170ofFIG.18). According to various embodiments, the conversation manager3110may include a response manager3112and/or a connection time module3114.

According to various embodiments, the processor4100(for example, the processor1110ofFIG.7) of the second electronic device4000(for example, the executor) may include a conversation manager4110(for example, the conversation manager1112ofFIG.8or the conversation manager2110ofFIG.18), a device dispatch4150(for example, the device dispatch2120ofFIG.18), a classifier4140(for example, the classifier1114ofFIG.8or the classifier3140ofFIG.18), an event manager4160(for example, the event manager1116ofFIG.8or the event manager2160ofFIG.18), and/or a CAN4170(for example, the CAN1118ofFIG.8or the CAN2170ofFIG.18). According to various embodiments, the conversation manager4110may include a response manager4112and/or a connection time module4114.

In the following description, it is assumed that the first electronic device3000operates as a receiver and the second electronic device4000operates as an executor. Accordingly, the configuration of the first electronic device3000(for example, the receiver) and an operating method thereof will be mainly described, and description of the configuration of the second electronic device4000(for example, the executor) which is the same as the first electronic device3000(for example, the receiver) will be omitted.

In an embodiment, a channel3130may be generated to transmit and receive data between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor). For example, generation of the channel3130may be initiated by the first electronic device3000(for example, the receiver). For example, generation of the channel3130may be initiated by the second electronic device4000(for example, the executor). The channel3130may be generated by transmitting and receiving a request for generating the channel3130between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor), and response information. The request and the response information which are transmitted and received between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may be managed by the response manager3112of the first electronic device3000(for example, the receiver) and/or the response manager4112of the second electronic device4000(for example, the executor).

According to various embodiments, the conversation manager3110of the first electronic device3000(for example, the receiver) and/or the conversation manager4110of the second electronic device4000(for example, the executor) may generate, maintain, and manage the channel3130. In an embodiment, after a user utterance is acquired, the conversation manager3110of the first electronic device3000(for example, the receiver) may determine the second electronic device4000(for example, the executor) to perform an operation (or action) corresponding to the utterance from among the plurality of executors, based on the received utterance. Thereafter, the channel3130for communicating with the second electronic device4000(for example, the executor) may be generated through the response manager3112.

In an embodiment, the device dispatch3150may include a rule NLU module3152. The rule NLU module3152may determine whether an utterance is to control another device (for example, at least one executor of the plurality of executors) when the user utterance is inputted to the first electronic device3000(for example, the receiver). The rule NLU module3152may provide information on an executor which executes an operation corresponding to a result of determining regarding the utterance to the device dispatch3150.

In an embodiment, after receiving the information of the second electronic device4000(for example, the executor) which executes the operation (or action) corresponding to the utterance from the rule NLU module3152, the device dispatch3150may determine whether there is the second electronic device4000(for example, the executor) to execute the operation (or action). In an embodiment, when a connection between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) to execute the operation (or action) corresponding to the utterance is determined to be effective by the connection time module3114,4114, the device dispatch3150may omit the operation of selecting the executor to execute the operation (or action) corresponding to the utterance.

According to various embodiments, the response manager4112of the second electronic device4000(for example, the executor) may distribute responses which are generated after the operation (or action) corresponding to the utterance is executed at the second electronic device4000(for example, the executor), through the channel3130. The response manager4112of the second electronic device4000(for example, the executor) may transmit the responses to the first electronic device3000(for example, the receiver) through the channel3130.

According to various embodiments, when the channel3130is formed, the connection time module3114of the first electronic device3000(for example, the receiver) and the connection time module4114of the second electronic device4000(for example, the executor) may manage a connection maintaining time of the channel3130. Each of the connection time modules3114,4114may manage effectiveness for a predetermined connection maintaining time.

According to various embodiments, the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may have connection maintaining times different according to each device (for example, a connection maintaining time of the speaker is 8 seconds, a connection maintaining time of an air conditioner is 30 seconds, a connection maintaining time of the TV is 1 minute, or a connection maintaining time of the refrigerator is 2 minutes). In an embodiment, when the connection maintaining time of the first electronic device3000(for example, the receiver) (for example, a first connection maintaining time) is ended, the first electronic device3000may inform the second electronic device4000(for example, the executor) of a disconnection request. Likewise, when the connection maintaining time of the second electronic device4000(for example, the executor) (for example, a second connection maintaining time) is ended, the second electronic device4000may inform the first electronic device3000(for example, the receiver) of a disconnection request. When the disconnection request is executed at any one of the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor), the channel3130between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may be disabled. That is, the connection may be disabled.

In an embodiment, when a reconnection request is executed at any one of the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor), the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may re-generate the channel3130and may be reconnected.

In an embodiment, the connection time module3114of the first electronic device3000(for example, the receiver) may manage the first connection maintaining time of the first electronic device3000(for example, the receiver). The connection time module4114of the second electronic device4000(for example, the executor) may manage the second connection maintaining time of the second electronic device4000(for example, the executor).

In an embodiment, in the state where the first connection maintaining time and the second connection maintaining time are effective, an operation of continuing connection maintenance at the first electronic device3000(for example, the receiver) (for example, an operation indicating continue use of the receiver) may be executed. In this case, the first electronic device3000(for example, the receiver) may reset or refresh the first connection maintaining time, and the connection time module3114may request the second electronic device4000(for example, the executor) to reset or refresh the second connection maintaining time.

In an embodiment, in the state where the first connection maintaining time and the second connection maintaining time are effective, an operation of continuing connection maintenance at the second electronic device4000(for example, the executor) (for example, an operation indicating continue use of the executor) may be executed. In this case, the second electronic device4000(for example, the executor) may reset or refresh the second connection maintaining time, and the connection time module4114may request the first electronic device3000(for example, the receiver) to reset or refresh the first connection maintaining time.

In an embodiment, in the state where the first connection maintaining time and the second connection maintaining time are effective, when the operation of continuing connection maintenance is executed at any one device of the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor), each of the connection time modules3114,4114may manage to reset or refresh the first connection maintaining time and the second connection maintaining time.

In an embodiment, an operation of identifying whether the first connection maintaining time and the second connection maintaining time are reset or refreshed may be performed at each of the connection time modules3114,4114on a predetermined interval basis. Each of the connection time modules3114,4114may identify whether the connection maintaining time is reset or refreshed at the same intervals (for example, 5 seconds, 15 seconds, 30 seconds, 1 minute, 3 minutes, 5 minutes or 10 minutes) with respect to the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor). However, this should not be considered as limiting, and each of the connection time modules3114,4114may identify whether the connection maintaining time is reset or refreshed at different intervals with respect to the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) (for example, 30 seconds for the receiver or 1 minute for the executor).

In an embodiment, when a user utters “Find the recipe of pizza on the refrigerator” through the first electronic device3000(for example, the receiver), the rule NLU module3152of the first electronic device3000(for example, the receiver) may determine whether the corresponding utterance should be executed at the first electronic device3000(for example, the receiver). When the corresponding utterance is not executed at the first electronic device3000(for example, the receiver) as a result of determining by the rule NLU module3152, the device dispatch3150may determine whether the refrigerator is an effective device. In the disclosure, it is assumed that the second electronic device4000(for example, the executor) is the refrigerator.

In an embodiment, when it is determined that the second electronic device4000(for example, the executor) is an effective device as an executor, context information and utterance information to be transmitted from the first electronic device3000(for example, the receiver) to the second electronic device4000(for example, the executor) may be determined through the classifier3140. The classifier3140may classify to determine which executor will perform the utterance requested at the first electronic device3000(for example, the receiver) from among the plurality of executors. The classifier3140may cause utterance information to be transmitted to the executor to perform the utterance requested at the first electronic device3000(for example, the receiver), that is, to the second electronic device4000(for example, the executor). In addition, the classifier3140may cause a result of performing the utterance to be transmitted to the first electronic device3000(for example, the receiver) after an operation corresponding to the utterance is performed at the second electronic device4000(for example, the executor).

In an embodiment, the first electronic device3000(for example, the receiver) may generate the channel3130and may connect to the second electronic device4000(for example, the executor) for a predetermined connection maintaining time. Thereafter, first device information of the first electronic device3000(for example, the receiver) and second device information of the second electronic device4000(for example, the executor) may be transmitted and/or received. Thereafter, the first electronic device3000(for example, the receiver) may transmit context information and utterance information to the second electronic device4000(for example, the executor).

In an embodiment, the second electronic device4000(for example, the executor) may determine a capsule (domain) (for example, the CAN4170) to execute the operation (or action) corresponding to the utterance, based on the context information and the utterance information received from the first electronic device3000(for example, the receiver). The second electronic device4000(for example, the executor) may transmit the context information and the utterance information received from the first electronic device3000(for example, the receiver) to the capsule (domain) to execute the operation (or action) corresponding to the utterance. After the operation corresponding to the utterance is performed, the event manager4160of the second electronic device4000(for example, the executor) may transmit the result of performing the utterance to the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor).

In an embodiment, the event manager3160,4160may determine whether to transmit the result of performing the utterance without processing, or may determine whether to correct the result of performing the utterance according to a UI form of the first electronic device3000(for example, the receiver) and/or the second electronic device4000(for example, the executor), and to transmit the corrected result. When the result of performing the utterance should be corrected, the event manager3160,4160may correct the result of performing the utterance according to the UI form of the first electronic device3000(for example, the receiver) and/or the second electronic device4000(for example, the executor), and may transmit the corrected result of performing the utterance to the first electronic device3000(for example, the receiver) and/or the second electronic device4000(for example, the executor).

In an embodiment, the response manager3112of the first electronic device3000(for example, the receiver) may provide the result of performing the utterance, received from the second electronic device4000(for example, the executor), to the user through a UI (for example, the UI1970ofFIG.20).

In an embodiment, the result of performing at the CAN4170of the second electronic device4000(for example, the executor) may be processed by a response manager of another executor. However, this should not be considered as limiting, and the executor may transmit the result of executing the operation (or action) to the receiver and the result of performing at the CAN4170may be processed through the response manager3112of the receiver.

FIG.22illustrates a flowchart52200for providing an intelligent assistance service through the receiver and the executor which are connected in the D2D method according to various embodiments.

Referring toFIGS.21and22, the first electronic device3000(for example, the receiver) may receive a user utterance in operation2205.

In operation2210, the first electronic device3000(for example, the receiver) may determine a target executor to perform an operation (or action) corresponding to the utterance from among the plurality of executors, based on the received user utterance. In the following description, it is assumed that the second electronic device4000(for example, the executor) is the target executor to perform the operation (or action) corresponding to the utterance.

In operation2215, the first electronic device3000(for example, the receiver) may request the second electronic device4000(for example, the executor) to generate the channel3130.

In operation2220, the second electronic device4000(for example, the executor) may receive a request for generation of the channel3130from the first electronic device3000(for example, the receiver), and may generate the channel3130between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor).

In operation2225, the first electronic device3000(for example, the receiver) may display a notification of connection with the second electronic device4000(for example, the executor) through a UI (for example, the UI1970ofFIG.20).

In operation2230, the first electronic device3000(for example, the receiver) may transmit the user utterance and context information to the second electronic device4000(for example, the executor), and may request the second electronic device4000to execute the operation (or action) corresponding to the utterance.

In operation2235, the second electronic device4000(for example, the executor) may execute the operation (or action), based on the user utterance and the context information which are received from the first electronic device3000(for example, the receiver).

In operation2240, the second electronic device4000(for example, the executor) may notify the first electronic device3000(for example, the receiver) of a result of executing the operation (or action) corresponding to the utterance.

In operation2245, the second electronic device4000(for example, the executor) may identify a predetermined second connection maintaining time, and may wait for reception of a follow-up operation (or action) execution request for the second connection maintaining time.

In operation2250, the first electronic device3000(for example, the receiver) may display the result of executing the operation (or action) corresponding to the utterance, received from the second electronic device4000(for example, the executor), through the UI.

In operation2255, the first electronic device3000(for example, the receiver) may identify a predetermined first connection maintaining time, and may wait for reception of a follow-up utterance for the first connection maintaining time.

In operations2245and2255, when the connection maintaining time is ended at any one of the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor), the channel3130between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may be disabled. That is, the connection between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may be disabled. However, this should not be considered as limiting, and, when resetting or refreshing of the connection maintaining time is requested at any one of the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor), the connection between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may be maintained.

In an embodiment, in the state where the first connection maintaining time and the second connection maintaining time are effective, an operation of continuing connection maintenance may be executed at any one of the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor). In this case, each of the connection time modules3114,4114of the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor) may reset or refresh the first connection maintaining time and the second connection maintaining time. However, this should not be considered as limiting. The connection termination (or unlink) menu1974may be displayed on a UI (for example, the UI1970ofFIG.20) of the first electronic device3000(for example, the receiver). When the user selects the connection termination (or unlink) menu1974displayed on the UI of the first electronic device3000(for example, the receiver), the first electronic device3000(for example, the receiver) may transmit connection termination (or unlink) to the second electronic device4000(for example, the executor). When the connection termination (or unlink) is received from the first electronic device3000(for example, the receiver), the second electronic device4000(for example, the executor) may disable the channel3130between the first electronic device3000(for example, the receiver) and the second electronic device4000(for example, the executor).

According to various embodiments of the disclosure, an electronic device101,1100,2100performing an operation corresponding to an utterance of a user in a multi-device environment may include a communication module190, a processor120, and a memory130. The processor120may be operatively connected with the communication module190and may operate a first electronic device for receiving (for example, the receiver1300,2300) and a second electronic device for executing (for example, the executor1400,2400) in association with each other. The memory130may be operatively connected with the processor120. The memory130may store instructions that, when being executed, cause the processor120to: determine whether a first utterance received at the first electronic device (for example, the receiver1300,2300) should be performed at the second electronic device (for example, the executor1400,2400); when the first utterance should be performed at the second electronic device (for example, the executor1400,2400), form a connection binding with the first electronic device (for example, the receiver1300,2300) and the second electronic device (for example, the executor1400,2400) by using one or more channels through the communication module190; and, when an operation corresponding to the first utterance is completed at the second electronic device (for example, the executor1400,2400), terminate the connection binding with the first electronic device (for example, the receiver1300,2300) and the second electronic device (for example, the executor1400,2400).

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to: receive first device information on the first electronic device (for example, the receiver1300,2300) and second device information on the second electronic device (for example, the executor1400,2400); and based on the first device information and the second device information, connect the first electronic device and the second electronic device (for example, the receiver1300and the executor1400,2400).

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to, after the operation corresponding to the first utterance is completed, transmit a response resulting from processing of the first utterance to the first electronic device (for example, the receiver1300,2300) in the form of a UI expressible by the first electronic device (for example, the receiver1300,2300), based on the first device information.

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to abstain from transmitting a response which is not a UI expressible by the first electronic device (for example, the receiver1300,2300) to the first electronic device (for example, the receiver1300,2300).

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to, after the operation corresponding to the first utterance is completed, transmit a response resulting from processing of the first utterance to the second electronic device (for example, the executor1400,2400) in the form of a UI expressible by the second electronic device (for example, the executor1400,2400), based on the second device information.

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to abstain from transmitting a response which is not a UI expressible by the second electronic device (for example, the executor1400,2400) to the second electronic device (for example, the executor1400,2400).

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to, after the operation corresponding to the first utterance is completed, correct a response content of the second electronic device (for example, the executor1400,2400) resulting from processing of the first utterance, and to transmit the corrected response content to the first electronic device (for example, the receiver1300,2300).

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to: when the first electronic device and the second electronic device (for example, the executor1400,2400) are connected, receive a first response format from the first electronic device (for example, the receiver1300,2300); and transmit a response resulting from processing of the first utterance to the first electronic device (for example, the receiver1300,2300), based on the first response format.

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to: when the first electronic device and the second electronic device (for example, the executor1400,2400) are connected, receive a second response format from the second electronic device (for example, the executor1400,2400); and transmit a response resulting from processing of the first utterance to the second electronic device (for example, the executor1400,2400), based on the second response format.

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to: count a first connection maintaining time from a time that a first channel for connecting with the first electronic device (for example, the receiver1300,2300) is generated, and to maintain or terminate the connection with the first electronic device (for example, the receiver1300,2300) based on the first connection maintaining time; and count a second connection maintaining time from a time that a second channel for connecting with the second electronic device (for example, the executor1400,2400) is generated, and to maintain or terminate the connection with the second electronic device (for example, the executor1400,2400) based on the second connection maintaining time.

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to: count a follow-up utterance waiting time from a time that the first utterance is received from the first electronic device (for example, the receiver1300,2300); and maintain or terminate the connection with the first electronic device (for example, the receiver1300,2300), based on the follow-up utterance waiting time.

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to: set an additional information input waiting time for waiting until additional information is received from the user after the additional information is requested from the user; and when the additional information is received within the additional information input waiting time or the additional information is not received for the additional information input waiting time, terminate the connection with the first electronic device and the second electronic device (for example, the executor1400,2400).

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to, when a new second utterance that should be performed at the second electronic device (for example, the executor1400,2400) is received at the first electronic device (for example, the receiver1300,2300) after the connection with the first electronic device (for example, the receiver1300,2300) and the second electronic device (for example, the executor1400,2400) is terminated, reconnect the first electronic device (for example, the receiver1300,2300) and the second electronic device (for example, the executor1400,2400).

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to: in a state in which the connection with the second electronic device (for example, the executor1400,2400) based on the first utterance is maintained, when a new second utterance is received at a third electronic device for receiving, generate a connection with the third electronic device for receiving; and connect the second electronic device (for example, the executor1400,2400) and the third electronic device.

According to various embodiments, the instructions of the electronic device101,1100,2100may cause the processor120to: in a state in which the connection with the first electronic device (for example, the receiver1300,2300) based on the first utterance is maintained, when a new second utterance that should be performed at a fourth electronic device for executing is received at the first electronic device (for example, the receiver1300,2300), generate a connection with the fourth electronic device for executing; and connect the first electronic device (for example, the receiver1300,2300) and the fourth electronic device.

According to various embodiments, an operating method of an electronic device101,1100,2100which operates a first electronic device for receiving (for example, the receiver1300,2300) and a second electronic device for executing (for example, the executor1400,2400) in association with each other, may include: determining whether a first utterance received at the first electronic device (for example, the receiver1300,2300) should be performed at the second electronic device (for example, the executor1400,2400); when the first utterance should be performed at the second electronic device (for example, the executor1400,2400), forming a connection binding with the first electronic device (for example, the receiver1300,2300) and the second electronic device (for example, the executor1400,2400) by using one or more channels; and, when an operation corresponding to the first utterance is completed at the second electronic device (for example, the executor1400,2400), terminating the connection binding with the first electronic device (for example, the receiver1300,2300) and the second electronic device (for example, the executor1400,2400).

According to various embodiments, the operating method of the electronic device101,1100,2100may further include: receiving first device information on the first electronic device (for example, the receiver1300,2300) and second device information on the second electronic device (for example, the executor1400,2400); and, based on the first device information and the second device information, connecting the first electronic device and the second electronic device (for example, the executor1400,2400).

According to various embodiments, the operating method of the electronic device101,1100,2100may include: after the operation corresponding to the first utterance is completed, transmitting a response resulting from processing of the first utterance to the first electronic device (for example, the receiver1300,2300) in the form of a UI expressible by the first electronic device (for example, the receiver1300,2300), based on the first device information; and transmitting a response resulting from processing of the first utterance to the second electronic device (for example, the executor1400,2400) in the form of a UI expressible by the second electronic device (for example, the executor1400,2400), based on the second device information.

According to various embodiments, the operating method of the electronic device101,1100,2100may not transmit a response which is not a UI expressible by the first electronic device (for example, the receiver1300,2300) to the first electronic device (for example, the receiver1300,2300), and may not transmit a response which is not a UI expressible by the second electronic device (for example, the executor1400,2400) to the second electronic device (for example, the executor1400,2400).

According to various embodiments, after the operation corresponding to the first utterance is completed, the operating method of the electronic device101,1100,2100may correct a response content of the second electronic device (for example, the executor1400,2400) resulting from processing of the first utterance, and may transmit the corrected response content to the first electronic device (for example, the receiver1300,2300).

According to various embodiments, an electronic device101,1100,2100performing an operation corresponding to an utterance of a user in a multi-device environment may include: a communication module; a processor120operatively connected with the communication module190and configured to operate a first electronic device for receiving (for example, the receiver1300,2300) and a second electronic device for executing (for example, the executor1400,2400) in association with each other; and a memory130operatively connected with the processor120. The memory130may store instructions that, when being executed, cause the processor120to: generate a first channel with the first electronic device (for example, the receiver1300,2300) which receives the utterance; receive information of the utterance through the first channel; determine whether an action corresponding to the utterance should be executed at the second electronic device (for example, the executor1400,2400), based on the information of the utterance; generate a second channel with the second electronic device (for example, the executor1400,2400); request the second electronic device (for example, the executor1400,2400) to execute the action corresponding to the utterance through the second channel; receive a result of executing the action corresponding to the utterance from the second electronic device (for example, the executor1400,2400); and transmit the result of executing the action corresponding to the utterance to the first electronic device (for example, the receiver1300,2300).

According to various embodiments, the processor120of the electronic device101,1100,2100may maintain the first channel for a predetermined first connection maintaining time, and may maintain the second channel for a predetermined second connection maintaining time.

According to various embodiments, when any one of the first connection maintaining time and the second connection maintaining time is ended, the processor120of the electronic device101,1100,2100may disable the first channel and the second channel.

According to various embodiments, in a state in which the first connection maintaining time and the second connection maintaining time are effective, when an operation of continuing connection maintenance is executed at any one of the first electronic device (for example, the receiver1300,2300) and the second electronic device (for example, the executor1400,2400), the processor120of the electronic device101,1100,2100may reset the first connection maintaining time and the second connection maintaining time.

According to various embodiments, in a state in which the first connection maintaining time and the second connection maintaining time are effective, when a connection termination (unlink) request is received from the first electronic device (for example, the receiver1300,2300), the processor120of the electronic device101,1100,2100may terminate the connection with the second electronic device (for example, the executor1400,2400).

According to various embodiments, an electronic device101,1100,2100for receiving an utterance of a user in a multi-device environment may include: a communication module190; a processor120operatively connected with the communication module190and configured to operate the first electronic device for receiving (for example, the receiver1300,2300) and a second electronic device for executing (for example, the executor1400,2400) in association with each other; a memory130operatively connected with the processor120. The memory130may store instructions that, when being executed, cause the processor120to: when the utterance is received, generate at least one channel and connect with the second electronic device (for example, the executor1400,2400); display the connection with the second electronic device (for example, the executor1400,2400) through a UI; request the second electronic device (for example, the executor1400,2400) to execute an action corresponding to the utterance through the at least one channel; receive a result of executing the action corresponding to the utterance from the second electronic device (for example, the executor1400,2400); and display the result of executing the action corresponding to the utterance through the UI.

According to various embodiments, the processor120of the electronic device101,1100,2100may maintain the at least one channel for a predetermined connection maintaining time, and may disable the at least one channel when the connection maintaining time is ended.

According to various embodiments, in a state in which the connection maintaining time is effective, when an operation of continuing connection maintenance is executed at the first electronic device (for example, the receiver1300,2300), the processor120of the electronic device101,1100,2100may reset the connection maintaining time.

According to various embodiments, when a new utterance is received in a state in which the connection maintaining time is effective, the processor120of the electronic device101,1100,2100may determine a device to execute an action corresponding to the new utterance; when the new utterance should be executed not at the second electronic device but at another device (for example, the executor1400,2400), the processor120may display a connection termination (unlink) menu through the UI, and may terminate the connection with the second electronic device (for example, the executor1400,2400) when the connection termination is selected by a user.

According to various embodiments, the electronic device101,1100,2100may connect the another device and the first electronic device (for example, the receiver1300,2300), and may request the another device to execute an action corresponding to the new utterance.

According to various embodiments, when a new utterance is received in a state in which the connection maintaining time is effective, the processor120of the electronic device101,1100,2100may determine a device to execute the new utterance, when the new utterance should be executed at the first electronic device (for example, the receiver1300,2300), the processor120may return to a home screen of the first electronic device (for example, the receiver1300,2300), and may execute an action corresponding to the new utterance.

According to various embodiments, an operation corresponding to an utterance can be performed in a multi-device environment without interruption.

According to various embodiments, a resulting response corresponding to an utterance can be displayed according to a UI form of a receiver and an executor.

According to various embodiments, a response corresponding to an utterance can be corrected and transmitted to a receiver.

According to various embodiments, an operation corresponding to an utterance can be processed at a fixed executor while a receiver is changed.

According to various embodiments, an operation corresponding to an utterance can be processed while an executor is changed.

According to various embodiments, a connection with a receiver and an executor can be maintained and terminated based on a connection maintaining time.

According to various embodiments, a connection with a receiver and an executor can be maintained and terminated based on a follow-up utterance waiting time.

According to various embodiments, a connection with a receiver and an executor can be maintained and terminated based on an additional information input waiting time.

Various effects directly or indirectly grasped through the present document may be provided.