Patent Description:
Artificial intelligence systems are computer systems capable of implementing human-like intelligence, which allow machines self-learning and decisions and provide a better recognition as they are used more.

Artificial intelligence technology may include element techniques, such as machine learning (deep learning) which utilizes algorithms capable of classifying and learning the features of obtained data on their own and copying the perception or determination by the human brain using machine learning algorithms.

Such element techniques may include linguistic understanding which recognizes human languages/words, visual understanding which recognizes things as if humans visually do, inference/prediction which determines information and perform logical inference and prediction, knowledge expression which processes human experience information as knowledge data, and motion control which controls robot motions and driver-less vehicles.

Linguistic understanding may refer, for example, to technology for recognizing and applying/processing a human being's language or text, and this encompasses natural language processing, machine translation, dialog system, answering inquiries, and speech recognition/synthesis.

Inference prediction may refer, for example, to a technique of determining and logically inferring and predicting information, encompassing knowledge/ probability-based inference, optimization prediction, preference-based planning, and recommendation.

Knowledge expression may refer, for example, to a technique of automatically processing human experience information, covering knowledge buildup (data production/classification) or knowledge management (data utilization).

<CIT> discloses an electronic device (telephone) that, during an ongoing call, may activate a voice service application in response to a wake-up word. It then sends speech to a speech interpretation server that returns instructions to be carried out by the electronic device.

<CIT> discloses techniques for establishing video telephony and screen sharing between two communication devices.

An electronic device does not support technology for activating an intelligent system when, for example, a telephone call is connected between the electronic device and another electronic device. The user may encounter such an occasion that he needs to obtain necessary information or share information with the other party using the intelligent system while being on the phone with the other party.

According to various example embodiments of the disclosure, an electronic device and a method of executing the functions of the electronic device may activate an intelligent system during a call, for example, a call with another electronic device.

According to various example embodiments of the disclosure, an electronic device and a method of executing the functions of the electronic device may identify an utterance obtained while being on the phone with another electronic device and provide services related to the utterance.

According to an example embodiment of the disclosure, an electronic device comprises at least one communication circuit, a speaker, a microphone, at least one processor operatively connected with the communication circuit, the speaker, and the microphone, and a memory operatively connected with the at least one processor, wherein the memory stores instructions which, when executed by the processor, cause the electronic device to: perform a call with a first external electronic device through the communication circuit, activate a voice-based intelligent service while the call is performed, identify a user's voice input through the microphone while the call is performed, and transmit data associated with the voice input to an external server through the communication circuit at least in part for automatic speech recognition (ASR) and/or natural language understanding (NLU). According to the invention, the device is further defined as specified by claim <NUM>.

According to an example embodiment of the disclosure, a method of executing a function of an electronic device comprises performing a call with a first external electronic device, activating a voice-based intelligent service while the call is performed, identifying a user's voice input through a microphone of the electronic device while the call is performed, and transmitting data associated with the voice input to an external server at least in part for ASR and/or NLU. According to the invention, the method is further defined as specified by claim <NUM>.

Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses exemplary embodiments of the disclosure.

Hereinafter, various example embodiments of the disclosure are described with reference to the accompanying drawings. However, it should be appreciated that the disclosure is not limited to the embodiments and the terminology used herein.

The same or similar reference denotations may be used to refer to the same or similar elements throughout the disclosure and the drawings. As used herein, the terms "A or B" or "at least one of A and/or B" may include all possible combinations of A and B.

As used herein, the terms "configured to" may be interchangeably used with other terms, such as "suitable for," "capable of," "modified to," "made to," "adapted to," "able to," or "designed to" in hardware or software or any combination thereof based on the context. The term "configured to" may refer, for example, to a situation in which a device can perform an operation together with another device or parts. For example, the term "processor configured (or set) to perform A, B, and C" may refer, for example, and without limitation, to a generic-purpose processor (e.g., a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device, a dedicated processor (e.g., an embedded processor), or the like, for performing the operations.

Examples of the electronic device according to embodiments of the disclosure may include, for example, and without limitation, at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), a MP3 player, a medical device, a camera, a wearable device, or the like. The wearable device may include, for example, and without limitation, at least one of an accessory-type device (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD)), a fabric- or clothes-integrated device (e.g., electronic clothes), a body attaching-type device (e.g., a skin pad or tattoo), a body implantable device, or the like.

In some embodiments, examples of the smart home appliance may include, for example, and without limitation, at least one of a television, a digital video disk (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washer, a drier, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box (e.g., Samsung HomeSync™ , Apple TV™ , or Google TV™) , a gaming console (Xbox™, PlayStation™), an electronic dictionary, an electronic key, a camcorder, an electronic picture frame, or the like.

According to an embodiment of the disclosure, the electronic device may include, for example, and without limitation, at least one of various medical devices (e.g., diverse portable medical measuring devices (a blood sugar measuring device, a heartbeat measuring device, a body temperature measuring device, etc.), a magnetic resource angiography (MRA) device, a magnetic resource imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global navigation satellite system (GNSS) receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, an sailing electronic device (e.g., a sailing navigation device or a gyro compass), avionics, security devices, vehicular head units, industrial or home robots, drones, automatic teller's machines (ATMs), point of sales (POS) devices, internet of things (IoT) devices (e.g., a bulb, various sensors, a sprinkler, a fire alarm, a thermostat, a street light, a toaster, fitness equipment, a hot water tank, a heater, or a boiler), or the like.

According to various embodiments of the disclosure, examples of the electronic device may include, for example, and without limitation, at least one of part of a piece of furniture, building/structure or vehicle, an electronic board, an electronic signature receiving device, a projector, various measurement devices (e.g., devices for measuring water, electricity, gas, or electromagnetic waves), or the like. According to embodiments of the disclosure, the electronic device may be flexible or may be a combination of the above-enumerated electronic devices.

As used herein, the term "user" may denote a human or another device (e.g., an artificial intelligent electronic device) using the electronic device.

<FIG> is a block diagram illustrating an example electronic device <NUM> in a network environment that may be used when implementing various embodiments.

Referring to <FIG>, the electronic device <NUM> in the network environment <NUM> may communicate with an electronic device <NUM> (or a second external electronic device) via a first network <NUM> (e.g., a short-range wireless communication network), or an electronic device <NUM> (or an external electronic device or a first external electronic device) or a server <NUM> via a second network <NUM> (e.g., a long-range wireless communication network).

A corresponding one of these communication modules may communicate with the external electronic device via the first network <NUM> (e.g., a short-range communication network, such as Bluetooth, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network <NUM> (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)).

<FIG> is a block diagram illustrating an example configuration of a program <NUM>.

Referring to <FIG> the program <NUM> may include an operating system (OS) <NUM> to control one or more resources of the electronic device <NUM>, middleware <NUM>, or an application <NUM> executable in the OS <NUM>. The OS <NUM> may include, for example, Android™, iOS™, Windows™, Symbian™, Tizen™, or Bada™. At least part of the program <NUM>, for example, may be pre-loaded on the electronic device <NUM> during manufacture, or may be downloaded from or updated by an external electronic device (e.g., the electronic device <NUM> or <NUM>, or the server <NUM>) during use by a user.

The OS <NUM> may control management (e.g., allocating or deallocation) of one or more system resources (e.g., process, memory, or power source) of the electronic device <NUM>. The OS <NUM>, additionally or alternatively, may include one or more driver programs to drive other hardware devices of the electronic device <NUM>, for example, the input device <NUM>, the sound output device <NUM>, the display device <NUM>, the audio module <NUM>, the sensor module <NUM>, the interface <NUM>, the haptic module <NUM>, the camera module <NUM>, the power management module <NUM>, the battery <NUM>, the communication module <NUM>, the subscriber identification module <NUM>, or the antenna module <NUM>.

The middleware <NUM> may provide various functions to the application <NUM> such that a function or information provided from one or more resources of the electronic device <NUM> may be used by the application <NUM>. The middleware <NUM> may include, for example, an application manager <NUM>, a window manager <NUM>, a multimedia manager <NUM>, a resource manager <NUM>, a power manager <NUM>, a database manager <NUM>, a package manager <NUM>, a connectivity manager <NUM>, a notification manager <NUM>, a location manager <NUM>, a graphic manager <NUM>, a security manager <NUM>, a telephony manager <NUM>, and/or a voice recognition manager <NUM>.

The application manager <NUM>, for example, may manage the life cycle of the application <NUM>. The window manager <NUM>, for example, may manage one or more graphical user interface (GUI) resources that are used on a screen. The multimedia manager <NUM>, for example, may identify one or more formats to be used to play media files, and may encode or decode a corresponding one of the media files using a codec appropriate for a corresponding format selected from the one or more formats. The resource manager <NUM>, for example, may manage the source code of the application <NUM> or a memory space of the memory <NUM>. The power manager <NUM>, for example, may manage the capacity, temperature, or power of the battery <NUM>, and determine or provide related information to be used for the operation of the electronic device <NUM> based at least in part on corresponding information of the capacity, temperature, or power of the battery <NUM>. According to an embodiment, the power manager <NUM> may interwork with a basic input/output system (BIOS) (not shown) of the electronic device <NUM>.

The database manager <NUM>, for example, may generate, search, or change a database to be used by the application <NUM>. The package manager <NUM>, for example, may manage installation or update of an application that is distributed in the form of a package file. The connectivity manager <NUM>, for example, may manage a wireless connection or a direct connection between the electronic device <NUM> and the external electronic device. The notification manager <NUM>, for example, may provide a function to notify a user of an occurrence of a specified event (e.g., an incoming call, message, or alert). The location manager <NUM>, for example, may manage locational information on the electronic device <NUM>. The graphic manager <NUM>, for example, may manage one or more graphic effects to be offered to a user or a user interface related to the one or more graphic effects.

The security manager <NUM>, for example, may provide system security or user authentication. The telephony manager <NUM>, for example, may manage a voice call function or a video call function provided by the electronic device <NUM>. The voice recognition manager <NUM>, for example, may transmit a user's voice data to the server <NUM>, and receive, from the server <NUM>, a command corresponding to a function to be executed on the electronic device <NUM> based at least in part on the voice data, or text data converted based at least in part on the voice data. According to an embodiment, the middleware <NUM> may dynamically delete some existing components or add new components. According to an embodiment, at least part of the middleware <NUM> may be included as part of the OS <NUM> or may be implemented as another software separate from the OS <NUM>.

The application <NUM> may include, for example, a home <NUM>, dialer <NUM>, short message service (SMS)/multimedia messaging service (MMS) <NUM>, instant message (IM) <NUM>, browser <NUM>, camera <NUM>, alarm <NUM>, contact <NUM>, voice dialer <NUM>, email <NUM>, calendar <NUM>, media player <NUM>, album <NUM>, watch <NUM>, health <NUM> (e.g., for measuring the degree of workout or biometric information, such as blood sugar), and/ or environmental information <NUM> (e.g., for measuring air pressure, humidity, or temperature information) application.

According to an embodiment, the application <NUM> may further include an information exchanging application (not shown) that is capable of supporting information exchange between the electronic device <NUM> and the external electronic device. The information exchange application, for example, may include a notification relay application adapted to transfer designated information (e.g., a call, message, or alert) to the external electronic device or a device management application adapted to manage the external electronic device. The notification relay application may transfer notification information corresponding to an occurrence of a specified event (e.g., receipt of an email) at another application (e.g., the email application <NUM>) of the electronic device <NUM> to the external electronic device. Additionally or alternatively, the notification relay application may receive notification information from the external electronic device and provide the notification information to a user of the electronic device <NUM>.

The device management application may control the power (e.g., turn-on or turn-off) or the function (e.g., adjustment of brightness, resolution, or focus) of the external electronic device or some component thereof (e.g., a display device or a camera module of the external electronic device). The device management application, additionally or alternatively, may support installation, delete, or update of an application running on the external electronic device.

Before describing various example embodiments of the disclosure, an integrated intelligent system to which an embodiment of the disclosure may apply is described.

<FIG> is a diagram illustrating an example integrated intelligent system that may be used to implement an embodiment.

Referring to <FIG>, an integrated intelligent system <NUM> may include at least two electronic devices (e.g., the electronic device <NUM> (e.g., a user terminal) and the external electronic device <NUM>), an intelligent server <NUM>, a personal information server <NUM>, and/or a proposing server <NUM>. For example, the two electronic devices may perform a call via a communication network.

The at least two electronic devices (e.g., the electronic device <NUM> and the external electronic device <NUM>) may provide services necessary for the user through apps (or application programs) (e.g., an alarm app, message app, photo (gallery) app, etc.) stored in the at least two electronic devices. For example, the electronic device (e.g., <NUM> or <NUM>) may execute and operate another app through an intelligent app (or speech recognition app) stored in the electronic device (e.g., <NUM> or <NUM>). The intelligent app of the electronic device (e.g., <NUM> or <NUM>) may receive user inputs to execute and operate the other app through the intelligent app. The user inputs may be received through, e.g., a physical button, touchpad, voice input, or remote input. According to an embodiment of the disclosure, the electronic device (e.g., <NUM> or <NUM>) may be various terminal devices (or electronic devices) connectable to the internet, such as a cellular phone, smartphone, personal digital assistant (PDA), or laptop computer.

According to an embodiment of the disclosure, the electronic device (e.g., <NUM> or <NUM>) may receive a user utterance as a user input. The electronic device (e.g., <NUM> or <NUM>) may receive the user utterance and generate a command to operate the app based on the user utterance. Accordingly, the electronic device (e.g., <NUM> or <NUM>) may operate the app using the command.

The intelligent server <NUM> may receive the user's voice input from the electronic device (e.g., <NUM> or <NUM>) through a communication network and convert the voice input into text data. According to an embodiment of the disclosure, the intelligent server <NUM> may generate (or select) a path rule based on the text data. The path rule may include information about actions (or operations) to perform the functions of the app or information about parameters necessary to execute the operations. Further, the path rule may include the order of the operations of the app. The electronic device (e.g., <NUM> or <NUM>) may receive the path rule, select an app according to the path rule, and execute the operations included in the path rule on the selected app.

As used herein, the term "path rule" may generally refer, for example, to a sequence of states for the electronic device to perform a task requested by the user, but not limited thereto. In other words, the path rule may contain information about a sequence. The task may be a certain action that, e.g., an intelligence app may provide. The task may include, for example, and without limitation, producing a schedule, transmitting a photo to a desired party, providing weather information, or the like. The electronic device (e.g., <NUM> or <NUM>) may perform the task by sequentially having at least one or more states (e.g., operation states of the electronic device (e.g., <NUM> or <NUM>)).

According to an embodiment, the path rule may be provided or created by an artificial intelligence (AI) system. The AI system may refer, for example, to a rule-based system and/or a neural network-based system (e.g., feedforward neural network (FNN)) or recurrent neutral network (RNN)). The AI system may be a combination thereof or a system different therefrom. According to an embodiment, the path rule may be selected from a set of pre-defined path rules or be created in real-time in response to a user request. For example, the AI system may select at least one among a plurality of pre-defined path rules or may dynamically (or real-time) create a path rule. The electronic device (e.g., <NUM> or <NUM>) may use a hybrid system to provide the path rule.

According to an embodiment, the electronic device (e.g., <NUM> or <NUM>) may execute the operation and display, on the display, a screen corresponding to the state of the electronic device (e.g., <NUM> or <NUM>) which executes the operation. As another example, the electronic device (e.g., <NUM> or <NUM>) may execute the operation and refrain from displaying a result of the operation on the display. The electronic device (e.g., <NUM> or <NUM>) may execute, e.g., a plurality of operations and display, on the display, at least in part of results of the plurality of operations. The electronic device (e.g., <NUM> or <NUM>) may display, on the display, e.g., the results of executing only the last operation in order. As another example, the electronic device (e.g., <NUM> or <NUM>) may display, on the display, a result of executing the operation according to receiving a user input.

The personal information server <NUM> may include a database storing user information. For example, the personal information server <NUM> may receive user information (e.g., context information or app execution) from the electronic device (e.g., <NUM> or <NUM>) and store the user information in the database. The intelligent server <NUM> may receive the user information from the personal information server <NUM> through the communication network and use the same in creating a path rule for user inputs. According to an embodiment of the disclosure, the electronic device (e.g., <NUM> or <NUM>) may receive user information from the personal information server <NUM> through the communication network and use the received user information as information for managing the database.

The proposing server <NUM> may include a database that stores information about functions to be provided and/or introductions of applications and/or functions in the terminal. For example, the proposing server <NUM> may receive user information of the user terminal (e.g., the electronic device <NUM>) from the personal information server <NUM> and include a database for functions that the user may use. The electronic device (e.g., <NUM> or <NUM>) may receive the information about functions to be provided from the proposing server <NUM> through the communication network and provide the information to the user.

<FIG> is a block diagram illustrating an example configuration of an intelligent server of an intelligent server that does not form part of the invention.

Referring to <FIG>, an intelligent server <NUM> may include an automatic speech recognition (ASR) module (e.g., including processing circuitry and/or executable program elements) <NUM>, a natural language understanding (NLU) module (e.g., including processing circuitry and/or executable program elements) <NUM>, a path planner module (e.g., including processing circuitry and/or executable program elements) <NUM>, a dialogue manager (DM) module (e.g., including processing circuitry and/or executable program elements) <NUM>, a natural language generator (NLG) module (e.g., including processing circuitry and/or executable program elements) <NUM>, and/or a text-to-speech (TTS) module (e.g., including processing circuitry and/or executable program elements) <NUM>. The intelligent server <NUM> may include a communication circuit, a memory, and a processor. The processor may execute commands stored in the memory to drive the ASR module <NUM>, the NLU module <NUM>, the path planner module <NUM>, the DM module <NUM>, the NLG module <NUM>, and the TTS module <NUM>. The intelligent server <NUM> may transmit or receive data (or information) to/from an external electronic device (e.g., the electronic device (e.g., <NUM> or <NUM>)) through the communication circuit.

The NLU module <NUM> or the path planner module <NUM> of the intelligent server <NUM> may generate a path rule.

The ASR module <NUM> may include various processing circuitry and/or executable program elements and convert user inputs received from the electronic device (e.g., <NUM> or <NUM>) into text data.

The ASR module <NUM> may convert user inputs received from the electronic device (e.g., <NUM> or <NUM>) into text data. For example, the ASR module <NUM> may include a speech recognition module. The speech recognition module may include an acoustic model and a language model. For example, the acoustic modem may include vocalization-related information, and the language model may include unit phonemic information and combinations of pieces of unit phonemic information. The speech recognition module may convert user utterances into text data using the vocalization-related information and unit phonemic information. Information about the acoustic model and the language model may be stored in, e.g., an automatic speech recognition (ASR) database (DB) <NUM>.

The NLU module <NUM> may include various processing circuitry and/or executable program elements and perform syntactic analysis and/or semantic analysis to understand the user's intent. Based on the syntactic analysis, the user input may be divided into syntactic units (e.g., words, phrases, or morphemes) and what syntactic elements the syntactic units have may be understood. The semantic analysis may be performed using, e.g., semantic matching, rule matching, or formula matching. Thus, the NLU module <NUM> may obtain a domain, intent, or parameters (or slots) necessary to represent the intent for the user input.

The NLU module <NUM> may determine the user's intent and parameters using the matching rule which has been divided into the domain, intent, and parameters (or slots) necessary to understand the intent. For example, one domain (e.g., an alarm) may include a plurality of intents (e.g., alarm settings or releasing alarm), and one intent may include a plurality of parameters (e.g., time, repetition count, or alarm sound). The plurality of rules may include, e.g., one or more essential element parameters. The matching rule may be stored in a natural language understanding (NLU) database (DB) <NUM>.

The NLU module <NUM> may identify the meaning of a word extracted from the user input using linguistic features (e.g., syntactic elements) such as morphemes and/or phrases, match the identified meaning of the word to the domain and intent, and determine the user's intent. For example, the NLU module <NUM> may calculate how many words extracted from the user input are included in each domain and intent to thereby determine the user's intent. The NLU module <NUM> may determine the parameters of the user input using the word which is a basis for determining the intent.

The NLU module <NUM> may determine the user's intent using the NLU DB <NUM> storing the linguistic features for determining the intent of the user input. According to an embodiment of the disclosure, the NLU module <NUM> may determine the user's intent using a personal language model (PLM). For example, the NLU module <NUM> may determine the user's intent using personal information (e.g., contacts list or music list). The PLM may be stored in, e.g., the NLU DB <NUM>. The ASR module <NUM>, but not the NLU module <NUM> alone, may recognize the user's voice by referring to the PLM stored in the NLU DB <NUM>.

The NLU module <NUM> may generate a path rule based on the intent of the user input and parameters. For example, the NLU module <NUM> may select an app to be executed based on the intent of the user input and determine operations to be performed on the selected app. The NLU module <NUM> may determine parameters corresponding to the determined operations to generate a path rule. The path rule generated by the NLU module <NUM> may include information about the app to be executed, operations (e.g., at least one or more states) to be executed on the app, and the parameters necessary to execute the operations.

The NLU module <NUM> may generate one or more path rules based on the parameters and intent of the user input. For example, the NLU module <NUM> may receive a path rule set corresponding to the electronic device (e.g., <NUM> or <NUM>) from the path planner module <NUM>, map the parameters and intent of the user input to the received path rule set, and determine the path rule.

The NLU module <NUM> may determine the app to be executed, operations to be executed on the app, and parameters necessary to execute the operations based on the parameters and intent of the user input, thereby generating one or more path rules. For example, the NLU module <NUM> may generate a path rule by arranging the app to be executed and the operations to be executed on the app in the form of ontology or a graph model according to the user input using the information of the electronic device (e.g., <NUM> or <NUM>). The generated path rule may be stored through, e.g., the path planner module <NUM> in a path rule database (PR DB) <NUM>. The generated path rule may be added to the path rule set of the database <NUM>.

The NLU module <NUM> may select at least one of a plurality of path rules generated. For example, the NLU module <NUM> may select the optimal one of the plurality of path rules. As another example, the NLU module <NUM> may select a plurality of path rules when at least in part of operations are specified based on the user utterance. The NLU module <NUM> may determine one of the plurality of path rules by the user's additional input.

The NLU module <NUM> may send the path rule to the electronic device (e.g., <NUM> or <NUM>) at a request for the user input. For example, the NLU module <NUM> may send one path rule corresponding to the user input to the electronic device (e.g., <NUM> or <NUM>). As another example, the NLU module <NUM> may send a plurality of path rules corresponding to the user input to the electronic device (e.g., <NUM> or <NUM>). For example, when at least in part of operations are specified based on the user utterance, the plurality of path rules may be generated by the NLU module <NUM>.

The path planner module <NUM> may include various processing circuitry and/or executable program elements and select at least one of the plurality of path rules.

The path planner module <NUM> may transfer a path rule set including the plurality of path rules to the NLU module <NUM>. The plurality of path rules in the path rule set may be stored in the form of, for example, a table in the path rule database <NUM> connected with the path planner module <NUM>. For example, the path planner module <NUM> may transfer a path rule set corresponding to information (e.g., OS information or app information) of the electronic device (e.g., <NUM> or <NUM>) which is received from the intelligent agent <NUM> (see, e.g., <FIG>) to the NLU module <NUM>. The table stored in the path rule database <NUM> may be stored, e.g., per domain or per domain version.

The path planner module <NUM> may select one or more path rules from the path rule set and transfer the selected one or more path rules to the NLU module <NUM>. For example, the path planner module <NUM> may match the user's intent and parameters to the path rule set corresponding to the electronic device (e.g., <NUM> or <NUM>) to select one or more path rules and transfer the selected one or more path rules to the NLU module <NUM>.

The path planner module <NUM> may generate one or more path rules using the user's intent and parameters. For example, the path planner module <NUM> may determine an app to be executed and operations to be executed on the app based on the user's intent and parameters to generate one or more path rules. The path planner module <NUM> may store the generated path rule in the path rule database <NUM>.

The path planner module <NUM> may store the path rule generated by the NLU module <NUM> in the path rule database <NUM>. The generated path rule may be added to the path rule set stored in the path rule database <NUM>.

The table stored in the path rule database <NUM> may include a plurality of path rules and/or a plurality of path rule sets. The plurality of path rule or the plurality of path rule sets may reflect the kind, version, type, or nature of the device performing each path rule.

The DM module <NUM> may include various processing circuitry and/or executable program elements and determine whether the user's intent determined by the path planner module <NUM> is clear. For example, the DM module <NUM> may determine whether the user's intent is clear based on whether parameter information is sufficient. The DM module <NUM> may determine whether the parameters determined by the NLU module <NUM> are sufficient to perform a task. When the user's intent is unclear, the DM module <NUM> may perform feedback to send a request for necessary information to the user. For example, the DM module <NUM> may perform feedback to send a request for parameter information to determine the user's intent.

The DM module <NUM> may include a content provider module. When the operation can be performed based on the intent and parameters determined by the NLU module <NUM>, the content provider module may generate the results of performing the task corresponding to the user input. The DM module <NUM> may send the results generated by the content provider module to the electronic device (e.g., <NUM> or <NUM>) in response to the user input.

The NLG module <NUM> may include various processing circuitry and/or executable program elements and convert designated information into text. The text information may be in the form of a natural language utterance. The designated information may be, e.g., information about an additional input, information indicating that the operation corresponding to the user input is complete, or information indicating the user's additional input (e.g., feedback information for the user input). The text information may be sent to the electronic device (e.g., <NUM> or <NUM>) and displayed on the display device <NUM>, or the text information may be sent to the TTS module <NUM> and converted into a voice.

The TTS module <NUM> may include various processing circuitry and/or executable program elements and convert text information into voice information. The TTS module <NUM> may receive the text information from the NLG module <NUM>, convert the text information into voice information, and send the voice information to the electronic device (e.g., <NUM> or <NUM>). The electronic device (e.g., <NUM> or <NUM>) may output the voice information through the sound output device <NUM>.

The NLU module <NUM>, the path planner module <NUM>, and the DM module <NUM> may be implemented in a single module. For example, the NLU module <NUM>, the path planner module <NUM>, and the DM module <NUM> may be implemented in a single module to determine the user's intent and parameter and to generate a response (e.g., a path rule) corresponding to the user's intent and parameters determined. Accordingly, the generated response may be transmitted to the electronic device (e.g., <NUM> or <NUM>).

A server (e.g., the intelligent server <NUM>) comprises a communication circuit, a processor operatively connected with the communication circuit, and a memory operatively connected with the processor, wherein the memory stores instructions which, when executed by the processor, cause the server to receive a voice input related to a first external electronic device (e.g., the external electronic device <NUM>) from an electronic device (e.g., the electronic device <NUM>) performing a call with the first external electronic device (e.g., the external electronic device <NUM>) through the communication circuit while the call is performed between the electronic device (e.g., the electronic device <NUM>) and the first external electronic device (e.g., the external electronic device <NUM>), perform at least in part ASR and/or NLU on the voice input, and transmit a result of performing the ASR and/or NLU to the electronic device (e.g., the electronic device <NUM>) and/or the first external electronic device (e.g., the external electronic device <NUM>).

According to an embodiment, the instructions when executed by the processor <NUM> cause the server to control the communication circuit to transmit information indicating that the intelligent service is activated while the electronic device (e.g., the electronic device <NUM>) performs the call with the first external electronic device (e.g., the external electronic device <NUM>) to the first external electronic device (e.g., the external electronic device <NUM>).

According to an embodiment, the instructions when executed by the processor <NUM> cause the server to, in response to the voice input including information related to inviting the first external electronic device (e.g., the external electronic device <NUM>) to a predetermined network, control the communication circuit to transmit information for requesting to acknowledge the invitation to the network to the first external electronic device (e.g., the external electronic device <NUM>).

<FIG> is a block diagram illustrating an example configuration of an electronic device of an intelligent server that may form part of an embodiment of the disclosure.

Referring to <FIG>, an electronic device <NUM> (or the external electronic device <NUM>) includes an input module (e.g., including input circuitry) <NUM> (e.g., the input device <NUM> of <FIG>), optionally a display device <NUM>, a sound output device (e.g., including sound output circuitry) <NUM>, a memory <NUM>, and a processor (e.g., including processing circuitry) <NUM>. The electronic device <NUM> may further include a housing. The components of the electronic device <NUM> may be disposed in or on the housing. The electronic device <NUM> further includes a communication circuit (e.g., the communication module <NUM> of <FIG>). The electronic device <NUM> is configured to transmit and receive data (or information) to/from an external server (e.g., the intelligent server <NUM>) through the communication circuit.

According to an embodiment of the disclosure, the input device <NUM> may include various input circuitry and receive a user input from the user. For example, the input device <NUM> may receive a user input from an external device (e.g., a keyboard or headset) connected thereto. As another example, the input device <NUM> may include, for example, and without limitation, a touchscreen combined with the display device <NUM> (e.g., a touchscreen display). As another example, the input device <NUM> may include, for example, and without limitation, a hardware key (or a physical key) positioned in the electronic device <NUM> (or the housing of the electronic device <NUM>).

According to the invention, the input device <NUM> includes a microphone capable of receiving user utterances as voice signals. For example, the input device <NUM> may include, for example, and without limitation, a speech input system and receive user utterances as voice signals through the speech input system. The microphone may be exposed through, e.g., a portion (e.g., a first portion) of the housing.

According to an embodiment of the disclosure, the display device <NUM> may display, for example, and without limitation, images, videos, and/or application execution screens. For example, the display device <NUM> may display a graphic user interface (GUI) of an app. According to an embodiment, the display device <NUM> may be exposed through, e.g., a portion (e.g., a second portion) of the housing.

According to an embodiment of the disclosure, the sound output device <NUM> may include various sound output circuitry and output voice signals. For example, the sound output device <NUM> may output voice signals generated from inside the electronic device <NUM> to the outside. According to an embodiment, the sound output device <NUM> may be exposed through, e.g., a portion (e.g., a third portion) of the housing.

According to an embodiment of the disclosure, the memory <NUM> may store a plurality of apps (or application programs) <NUM> and <NUM>. The plurality of apps <NUM> and <NUM> may be, e.g., programs for performing a function corresponding to a user input. According to an embodiment of the disclosure, the memory <NUM> may include the intelligent agent (e.g., including various executable program elements) <NUM>, the execution manager module (e.g., including various executable program elements) <NUM>, and/or the intelligent service module (e.g., including various executable program elements) <NUM>. The intelligent agent <NUM>, the execution manager module <NUM>, and the intelligent service module <NUM> may be frameworks (or application frameworks) to process received user inputs (e.g., user utterances).

According to an embodiment of the disclosure, the memory <NUM> may include a database that may store information necessary to recognize user inputs. For example, the memory <NUM> may include a log database capable of storing log information. As another example, the memory <NUM> may include a persona database capable of storing user information.

According to an embodiment of the disclosure, the memory <NUM> may store the plurality of apps <NUM> and <NUM>. The plurality of apps <NUM> and <NUM> may be loaded and operated. For example, the plurality of apps <NUM> and <NUM> stored in the memory <NUM> may be loaded and operated by the execution manager module <NUM>. The plurality of apps <NUM> and <NUM> may include execution service modules 141a and 143a including, for example, various executable program elements configured to perform functions. According to an embodiment, the plurality of apps <NUM> and <NUM> may execute a plurality of operations (e.g., a sequence of states) 141b and 143b through the execution service modules 141a and 143a to perform functions. In other words, the execution service modules 141a and 143a may be activated by the execution manager module <NUM> and execute the plurality of operations 141b and 143b.

According to an embodiment of the disclosure, when the operations 141b and 143b of the apps <NUM> and <NUM> are executed, the execution state screens according to the execution of the operations 141b and 143b may be displayed on the display device <NUM>. The execution state screens may be screens, e.g., in the state of the operations 141b and 143b having been completed. The execution state screens may be screens, e.g., in the state of the execution of the operations 141b and 143b having been stopped (partial landing) (e.g., when parameters required for the operations 141b and 143b are not input).

According to an embodiment of the disclosure, the execution service modules 141a and 143a may execute the operations 141b and 143b based on a path rule. For example, the execution service modules 141a and 143a may be activated by the execution manager module <NUM>, receive an execution request based on the path rule from the execution manager module <NUM>, and execute the operations 141b and 143b according to the execution request, thereby executing the functions of the apps <NUM> and <NUM>. The execution service modules 141a and 143a, when the execution of the operations 141b and 143b is complete, may send completion information to the execution manager module <NUM>.

According to an embodiment of the disclosure, when the plurality of operations 141b and 143b are executed on the apps <NUM> and <NUM>, the plurality of operations 141b and 143b may sequentially be executed. When the execution of one operation (e.g., operation <NUM> of the first app <NUM> or operation <NUM> of the second app <NUM>) is complete, the execution service modules 141a and 143a may open the next operation (e.g., operation <NUM> of the first app <NUM> or operation <NUM> of the second app <NUM>) and send completion information to the execution manager module <NUM>. For example, 'open an operation' may refer, for example, to transitioning the operation into an executable state or preparing for the execution of the operation. For example, unless the operation is open, the operation cannot be executed. Upon receiving the completion information, the execution manager module <NUM> may transfer an execution request for the next operation (e.g., operation <NUM> of the first app <NUM> or operation <NUM> of the second app <NUM>) to the execution service module. According to an embodiment of the disclosure, when the plurality of apps <NUM> and <NUM> are executed, the plurality of apps <NUM> and <NUM> may sequentially be executed. For example, when the execution of the last operation of the first app <NUM> (e.g., operation <NUM> of the first app <NUM>) is complete, and completion information is thus received, the execution manager module <NUM> may send an execution request for the first operation of the second app <NUM> (e.g., operation <NUM> of the second app <NUM>) to the execution service module 143a.

According to an embodiment of the disclosure, when the plurality of operations 141b and 143b are executed on the apps <NUM> and <NUM>, the resultant screens of execution of the plurality of operations 141b and 143b may be displayed on the display device <NUM>. According to an embodiment of the disclosure, at least in part of the plurality of resultant screens according to execution of the plurality of operations 141b and 143b may be displayed on the display device <NUM>.

According to an embodiment of the disclosure, the memory <NUM> may store an intelligent app (e.g., a speech recognition app) interworking with the intelligent agent <NUM>. The app interworking with the intelligent agent <NUM> may receive and process a user utterance as a voice signal. According to an embodiment of the disclosure, the app interworking with the intelligent agent <NUM> may be operated by particular inputs (e.g., inputs through the hardware key or touchscreen, or particular voice inputs) obtained through the input device <NUM>.

According to an embodiment of the disclosure, the intelligent agent <NUM>, the execution manager module <NUM>, and/or the intelligent service module <NUM> stored in the memory <NUM> may be executed by the processor <NUM>. The functions of the intelligent agent <NUM>, the execution manager module <NUM>, and/or the intelligent service module <NUM> may be implemented by the processor <NUM>. The functions of the intelligent agent <NUM>, the execution manager module <NUM>, and the intelligent service module <NUM> are described in connection with operations of the processor <NUM>. According to an embodiment of the disclosure, the intelligent agent <NUM>, the execution manager module <NUM>, and/or the intelligent service module <NUM> stored in the memory <NUM> may be implemented, for example, and without limitation, in software or hardware or any combination thereof.

According to an embodiment of the disclosure, the processor <NUM> may include various processing circuitry and control the overall operation of the electronic device <NUM>. For example, the processor <NUM> may control the input device <NUM> to receive user inputs. The processor <NUM> may control the display device <NUM> to display images. The processor <NUM> may control the sound output device <NUM> to output voice signals. The processor <NUM> may control the memory <NUM> to execute programs and fetch or store necessary information.

According to an embodiment of the disclosure, the processor <NUM> may execute the intelligent agent <NUM>, the execution manager module <NUM>, and/or the intelligent service module <NUM> stored in the memory <NUM>. Thus, the processor <NUM> may implement the function of the intelligent agent <NUM>, the execution manager module <NUM>, and/or the intelligent service module <NUM>.

According to an embodiment of the disclosure, the processor <NUM> may execute the intelligent agent <NUM> to generate commands to operate apps based on voice signals received as user inputs. According to an embodiment of the disclosure, the processor <NUM> may execute the execution manager module <NUM> to execute the apps <NUM> and <NUM> stored in the memory <NUM> based on the commands generated. According to an embodiment, the processor <NUM> may execute the intelligent service module <NUM> to manage the user information and process user inputs using the user information.

The processor <NUM> may execute the intelligent agent <NUM> to send user inputs received through the input device <NUM> to the intelligent server <NUM> and process the user inputs through the intelligent server <NUM>.

According to an embodiment of the disclosure, the processor <NUM> may execute the intelligent agent <NUM> to pre-process the user inputs before sending the user inputs to the intelligent server <NUM>. According to an embodiment of the disclosure, the intelligent agent <NUM> may include an adaptive echo canceller (AEC) module, a noise suppression (NS) module, an end-point detection (EPD) module, and/or an automatic gain control (AGC) module to pre-process the user inputs. The AEC module may include various executable program elements and remove echoes mixed in the user inputs. The NS module may include various executable program elements and suppress background noise mixed in the user inputs. The EPD module may include various executable program elements and detect end points of user voices contained in the user inputs to find when the user voices are present using the detected end points. The AGC module may include various executable program elements and recognize the user inputs and adjust the volume of the user inputs for a proper processing of the recognized user inputs. According to an embodiment of the disclosure, although able to execute all of the pre-processing components described above to provide a better performance, the processor <NUM> may execute some of the pre-processing components to be operated at reduced power.

According to an embodiment of the disclosure, the intelligent agent <NUM> may include a wake-up recognition module which may, for example, include various executable program elements stored in the memory <NUM> to recognize the user's invocation. Thus, the processor <NUM> may recognize the user's wake-up command through the wake-up recognition module, and upon receiving the wake-up command, the processor <NUM> may execute the intelligent agent <NUM> to receive user inputs. The wake-up recognition module may be implemented in a low-power processor (e.g., a processor included in an audio codec). According to an embodiment, the processor <NUM> may execute the intelligent agent <NUM> upon receiving a user input through the hardware key. When the intelligent agent <NUM> is executed, an intelligent app (e.g., a speech recognition app) interworking with the intelligent agent <NUM> may be executed.

According to an embodiment of the disclosure, the intelligent agent <NUM> may include a speech recognition module which may, for example, include various executable program elements, to execute user inputs. The processor <NUM> may receive user inputs to execute operations on the app through the speech recognition module. For example, the processor <NUM> may recognize, through the speech recognition module, limited user (voice) inputs (e.g., the "Click" sound made when the capturing operation is executed on the camera app) for executing operations, such as the wake-up command on the apps <NUM> and <NUM>. The processor <NUM> may assist the intelligent server <NUM> in recognize and quickly process user commands, which are processable in the electronic device <NUM>, through the speech recognition module. According to an embodiment, the speech recognition module of the intelligent agent <NUM> to execute user inputs may be implemented in an app processor.

According to an embodiment of the disclosure, the speech recognition module (including the speech recognition module of the wake-up recognition module) of the intelligent agent <NUM> may recognize user inputs using an algorithm for recognizing voice. The algorithm used to recognize voice may be, for example, and without limitation, at least one of, e.g., a hidden markov model (HMM) algorithm, an artificial neural network (ANN) algorithm, a dynamic time warping (DTW) algorithm, or the like.

According to an embodiment of the disclosure, the processor <NUM> may execute the intelligent agent <NUM> to convert the user's voice inputs into text data. For example, the processor <NUM> may send a user voice through the intelligent agent <NUM> to the intelligent server <NUM> and receive text data corresponding to the user voice from the intelligent server <NUM>. Thus, the processor <NUM> may display the converted text data on the display device <NUM>.

According to an embodiment of the disclosure, the processor <NUM> may execute the intelligent agent <NUM> to receive a path rule from the intelligent server <NUM>. According to an embodiment, the processor <NUM> may transfer the path rule to the execution manager module <NUM> through the intelligent agent <NUM>.

According to an embodiment of the disclosure, the processor <NUM> may execute the intelligent agent <NUM> to transfer an execution result log based on the path rule received from the intelligent server <NUM> to the intelligent service module <NUM>. The execution result log transferred may be accrued and managed in user preference information of a persona manager 149b.

According to an embodiment of the disclosure, the processor <NUM> may execute the execution manager module <NUM> to receive the path rule from the intelligent agent <NUM>, execute the apps <NUM> and <NUM>, and allow the apps <NUM> and <NUM> to execute the operations 141b and 143b contained in the path rule. For example, the processor <NUM> may send command information (e.g., path rule information) to execute the operations 141b and 143b to the apps <NUM> and <NUM> through the execution manager module <NUM> and receive completion information about the operations 141b and 143b from the apps <NUM> and <NUM>.

According to an embodiment of the disclosure, the processor <NUM> may execute the execution manager module <NUM> to transfer command information (e.g., path rule information) to execute the operations 141b and 143b of the apps <NUM> and <NUM> between the intelligent agent <NUM> and the apps <NUM> and <NUM>. The processor <NUM> may bind the apps <NUM> and <NUM> to be executed based on the path rule through the execution manager module <NUM> and transfer the command information (e.g., path rule information) about the operations 141b and 143b contained in the path rule to the apps <NUM> and <NUM>. For example, the processor <NUM> may sequentially transfer the operations 141b and 143b contained in the path rule to the apps <NUM> and <NUM> through the execution manager module <NUM>, sequentially executing the operations 141b and 143b of the apps <NUM> and <NUM> based on the path rule.

According to an embodiment, the processor <NUM> may execute the execution manager module <NUM> to manage the execution states of the operations 141b and 143b of the apps <NUM> and <NUM>. For example, the processor <NUM> may receive information about the execution states of the operations 141b and 143b from the apps <NUM> and <NUM> through the execution manager module <NUM>. When the execution states of the operations 141b and 143b are, e.g., partial landing states (e.g., when no parameters required for the operations 141b and 143b are input), the processor <NUM> may transfer information about the partial landing states to the intelligent agent <NUM> through the execution manager module <NUM>. The processor <NUM> may request the user to obtain necessary information (e.g., parameter information) using the information transferred through the intelligent agent <NUM>. When the execution states of the operations 141b and 143b are other states, e.g., operation states, the processor <NUM> may receive an utterance from the user through the intelligent agent <NUM>. The processor <NUM> may transfer information about the apps <NUM> and <NUM> being executed through the execution manager module <NUM> and the execution states of the apps <NUM> and <NUM> to the intelligent agent <NUM>. The processor <NUM> may send the user utterance through the intelligent agent <NUM> to the intelligent server <NUM>. The processor <NUM> may receive parameter information about the user utterance from the intelligent server <NUM> through the intelligent agent <NUM>. The processor <NUM> may transfer the parameter information received through the intelligent agent <NUM> to the execution manager module <NUM>. The execution manager module <NUM> may change the parameters of the operations 141b and 143b into new parameters using the received parameter information.

According to an embodiment, the processor <NUM> may execute the execution manager module <NUM> to transfer the parameter information contained in the path rule to the apps <NUM> and <NUM>. When the plurality of apps <NUM> and <NUM> are sequentially executed based on the path rule, the execution manager module <NUM> may transfer the parameter information contained in the path rule from one app to the other.

According to an embodiment of the disclosure, the processor <NUM> may execute the execution manager module <NUM> to receive a plurality of path rules. The processor <NUM> may select a plurality of path rules based on the user utterance through the execution manager module <NUM>. For example, when a user utterance specifies a certain app <NUM> to execute some operation 141a but does not specify another app <NUM> to execute the other operation 143b, the processor <NUM> may, through the execution manager module <NUM>, receive a plurality of different path rules by which the same app <NUM> (e.g., gallery app) to execute the operation 141a is executed and a different app <NUM> (e.g., message app or telegram app) to execute the other operation 143b is executed. The processor <NUM> may execute the same operations 141b and 143b (e.g., the same continuous operations 141b and 143b) of the plurality of path rules through the execution manager module <NUM>. When the same operations have been executed, the processor <NUM> may, through the execution manager module <NUM>, display, on the display device <NUM>, the state screen for selecting the different apps <NUM> and <NUM> each contained in a respective one of the plurality of path rules.

According to an embodiment of the disclosure, the intelligent service module <NUM> may include a context module (e.g., including various executable program elements) 149a, a persona module (e.g., including various executable program elements) 149b, and/or a proposing module (e.g., including various executable program elements) 149c.

The processor <NUM> may execute the context module 149a to identify current states of the apps <NUM> and <NUM> from the apps <NUM> and <NUM>. For example, the processor <NUM> may execute the context module 149a to receive context information indicating the current states of the apps <NUM> and <NUM> to identify the current states of the apps <NUM> and <NUM> through the received context information.

The processor <NUM> may execute the persona module 149b to manage the personal information about the user using of the electronic device <NUM>. For example, the processor <NUM> may execute the persona module 149b to obtain use information and execution results of the electronic device <NUM> and manage the user's personal information using the obtained use information and execution results of the electronic device <NUM>.

The processor <NUM> may execute the proposing module 149c to predict the user's intention and recommend a command for the user based on the user's intention. For example, the processor <NUM> may execute the proposing module 149c to recommend a command for the user based on the user's current state (e.g., time, place, context, or app).

According to an embodiment of the disclosure, an electronic device (e.g., the electronic device <NUM>) comprises at least one communication circuit (e.g., the communication module <NUM>), a speaker (e.g., the sound output device <NUM>), a microphone <NUM>, at least one processor (e.g., a processor <NUM>) operatively connected with the communication circuit, the speaker, and the microphone, and a memory <NUM> operatively connected with the processor <NUM>, wherein the memory <NUM> stores instructions which, when executed by the processor <NUM>, cause the electronic device to perform a call with a first external electronic device (e.g., the electronic device <NUM>) through the communication circuit, activate a voice-based intelligent service while the call is performed, identify a user's voice input through the microphone while the call is performed, and transmit data associated with the voice input to an external server (e.g., the intelligent server <NUM>) through the communication circuit at least partially for automatic speech recognition (ASR) and/or natural language understanding (NLU).

According to an embodiment, the instructions when executed by the processor <NUM> cause the electronic device to activate the intelligent service in response to receiving a voice input including a predetermined word or a predetermined input while the call is performed.

According to an embodiment, the instructions when executed by the processor <NUM> cause the electronic device to control a display to display information indicating that the intelligent service is activated while the call is performed and control the communication circuit to transmit the information to the first external electronic device (e.g., the electronic device <NUM>).

According to an embodiment of the disclosure, the instructions when executed by the processor <NUM> cause the electronic device to, when the voice input includes information related to inviting the first external electronic device (e.g., the electronic device <NUM>) to a predetermined network, control the communication circuit to transmit information for requesting to acknowledge the invitation to the network to the first external electronic device (e.g., the external electronic device <NUM>).

According to an embodiment of the disclosure, the instructions when executed by the processor <NUM> cause the electronic device to, when the voice input includes information related to controlling at least one electronic device (e.g., home appliances or IoT devices) controlled by the first external electronic device (e.g., the electronic device <NUM>) connected through a network, control the communication circuit (e.g., the communication module <NUM>) to transmit a request to authorize the control of the at least one electronic device to the first external electronic device (e.g., the electronic device <NUM>) while the call is performed.

According to an embodiment of the disclosure, the instructions when executed by the processor <NUM> cause the electronic device to identify, based on a user input, password information or biometric information and determine whether the password information or the biometric information corresponds to pre-stored information to authenticate the user. The biometric information may include, for example, and without limitation, iris information, voice pattern information, fingerprint information, or the like.

<FIG> is a flowchart illustrating example operations of an electronic device providing an intelligent service based on an utterance between users on a call that may form part of an embodiment.

Referring to <FIG>, in operation <NUM>, an electronic device (e.g., the electronic device <NUM>) may be on call with an external electronic device (e.g., the electronic device <NUM>).

In operation <NUM>, the electronic device may determine whether a voice-based intelligent service is activated. For example, in response to obtaining a voice input containing a predetermined word or a predetermined input (e.g., a key input) while the call is being performed, the electronic device may control to activate the intelligent service.

In operation <NUM>, when the intelligent service is activated, the electronic device may identify the user's voice input obtained through a microphone of the electronic device while performing the call.

According to an embodiment, the electronic device may display information indicating that the intelligent service is activated while performing the call and transmit the information to the external electronic device. For example, upon determining that the external electronic device may support the intelligent service, the electronic device may transmit the information to the external electronic device.

In operation <NUM>, the electronic device may transmit data related to utterance for natural language understanding and/or speech recognition to a server. For example, the electronic device may transmit the voice input-related data to an external server (e.g., the intelligent server <NUM>), at least partially, for automatic speech recognition (ASR) and/or natural language understanding (NLU).

According to an embodiment, when the voice input contains information related to inviting a first external electronic device (e.g., the electronic device <NUM>) to a predetermined network, the electronic device <NUM> may transmit information to request to identify the invitation to the network to the first external electronic device.

According to an embodiment of the disclosure, when the voice input includes information related to controlling at least one second external electronic device (e.g., the electronic device <NUM> of <FIG>) controlled by the first external electronic device <NUM> connected through a network, the electronic device <NUM> may transmit a request to authorize control of the at least one second external electronic device <NUM> to the first external electronic device <NUM> while the call is performed.

According to an embodiment of the disclosure, the electronic device <NUM> may request the user of the electronic device <NUM> to obtain biometric information and/or password information and determine whether the biometric information and/or password information obtained in response to the request corresponds to pre-stored information to authenticate the user. The biometric information may include, for example, and without limitation, iris information, voice pattern information, fingerprint information, or the like.

<FIG> is a diagram illustrating an example of executing an intelligence app on a user terminal.

<FIG> illustrates an example in which the electronic device <NUM> receives user inputs and executes an intelligent app (e.g., a speech recognition app) interworking with the intelligent agent <NUM>.

The electronic device <NUM> may execute an intelligent app to recognize voice through, for example, the hardware key <NUM>. For example, when the electronic device <NUM> receives user inputs through the hardware key <NUM>, the electronic device <NUM> may display a user interface (UI) <NUM> for the intelligent app on the display device <NUM>. The user may touch a speech recognition button 121a in the UI <NUM> of the intelligent app for inputting voice 120b with the intelligent app UI <NUM> displayed on the display device <NUM>. As another example, the user may continuously press the hardware key <NUM> for inputting voice 120b.

The electronic device <NUM> may execute an intelligent app to recognize voice through the microphone <NUM>. For example, when a predetermined voice (e.g., "wake up!" or "Bixby!") is input (120a) through the microphone <NUM>, the electronic device <NUM> may display the intelligent app UI <NUM> on the display device <NUM>.

<FIG> is a diagram illustrating an example of performing an intelligent service by an utterance received during a call between users in an intelligent system.

Referring to <FIG>, an intelligent system may include an electronic device <NUM> of a first user (e.g., 'Russ') <NUM> and an external electronic device <NUM> of a second user (e.g., 'Saskia') <NUM>.

When a predetermined input is received through at least one of the electronic devices <NUM> and <NUM> while a call is being performed between the electronic devices <NUM> and <NUM>, an intelligent service may be activated on the electronic devices <NUM> and <NUM>. For example, the predetermined input may include a predetermined voice (e.g., "Wake up!"), a voice (e.g., "Bixby!") to call the name of the intelligent service, a designated hardware input (e.g., selecting the button <NUM>), reception of designated data (e.g., a sensor value), or a designated action (e.g., a gesture). The intelligent service may be activated based on other various types of inputs determinable by the electronic device <NUM> or <NUM>.

As the intelligent service is activated on the electronic devices <NUM> and <NUM>, the electronic devices <NUM> and <NUM> may be connected to the intelligent server <NUM>, and each electronic device may output a notification to indicate that the intelligent service is activated. For example, after the intelligent service is activated, an utterance received to each electronic device may be transmitted to the intelligent server <NUM>, and the intelligent server <NUM> may identify a function related to the received utterance.

When the utterance contains information indicating a function (e.g., a call recording or memo) related to the call, the intelligent server <NUM> may control the electronic devices <NUM> and <NUM>, which are performing the call, to perform the function related to the utterance and display a result of performing the function.

The intelligent server <NUM> may determine whether to provide the function (e.g., keyword search) related to the utterance received to the electronic devices <NUM> and <NUM> performing the call to both or a particular one of the electronic devices <NUM> and <NUM>. For example, when the utterance does not contain information indicating the other party of the call, the intelligent server <NUM> may perform control so that the function related to the utterance is provided only to the electronic device of the user who has input the utterance.

In response to inputting a voice or receiving an input to activate an intelligent service while the call is being performed, the electronic device may provide the intelligent service based on the received voice.

<FIG> is a diagram illustrating an example configuration of an integrated intelligent system according to an embodiment of the disclosure.

Referring to <FIG>, an intelligent system may include at least two electronic devices (e.g., <NUM> and <NUM>) performing a call and an intelligent server <NUM>, and the intelligent system may further include, for example, and without limitation, at least one of an account server <NUM>, a service provider server <NUM> (e.g., a bank service server), an Internet-of-things (IoT) server <NUM>, and/or an application server <NUM>.

According to an embodiment, when an intelligent service is activated while the electronic devices (e.g., <NUM> and <NUM>) are performing a call, the intelligent server <NUM> may identify an utterance received during the call is being performed between the electronic devices (e.g., <NUM> and <NUM>). For example, the intelligent server <NUM> may transmit the utterance to at least one of the account server <NUM>, the service provider server <NUM>, the IoT server <NUM>, and/or the application server <NUM> to identify a function or service providable to the electronic devices (e.g., <NUM> and <NUM>) in relation to the utterance.

According to an embodiment of the disclosure, although the servers (e.g., the account server <NUM>, the service provider server <NUM>, the IoT server <NUM>, and/or the application server <NUM>) are depicted as a separate component from the intelligent server <NUM>, at least some of the servers may be omitted or modified, and the intelligent server <NUM> may be configured to perform some of the functions of the servers.

The account server <NUM> may manage and store account information about electronic devices accessing various networks (e.g., Samsung Connect™, SmartThings™, Samsung Smart Home™, etc.). For example, the account server <NUM> may identify information received from a particular electronic device (e.g., <NUM> or <NUM>), determine whether the identified information corresponds to designated information, and control to allow the electronic device to access a particular network.

According to an embodiment, an intelligent service may be activated while the electronic device <NUM> is performing a call with the external electronic device <NUM>, and such a voice as "Invite Saskia to Samsung Connect!") may be input. By identifying the voice (e.g., "Samsung Connect") indicating the particular network and the voice (e.g., "Invite Saskia") indicating the function related to the service of the network, the intelligent server <NUM> may request the account server <NUM> to identify whether the electronic devices of the users related to the utterance may access the particular network.

The IoT server <NUM> may control connections among home appliances constituting a home network and enable a particular electronic device (e.g., <NUM> or <NUM>) to control the functions of the home appliances.

The application server <NUM> may control applications installed on the electronic device (e.g., <NUM> or <NUM>).

<FIG> is a diagram illustrating an example of performing an intelligent service by an utterance received during a call between users in an intelligent system according to an embodiment of the disclosure.

Referring to <FIG>, an intelligent system may include at least two electronic devices (e.g., <NUM> and <NUM>) performing a call, an intelligent server <NUM>, an account server <NUM>, or an IoT server <NUM>.

According to an embodiment, the user (e.g., "Russ") of the electronic device <NUM> and the user (e.g., "Saskia") of the external electronic device <NUM> may be performing a call and, as a particular one of the electronic devices identifies a predesignated utterance or a selection of a designated button <NUM>, an intelligent service may be activated on the electronic devices.

According to an embodiment, the utterance <NUM> received through at least one of the electronic devices may be transmitted to the intelligent server <NUM> and, to provide a service corresponding to the utterance <NUM>, the intelligent server <NUM> may send a request for a service related to the utterance <NUM> to at least one server (e.g., the account server <NUM> or the IoT server <NUM>). For example, the utterance <NUM> may be to make a request for control to allow the other party (e.g., the external electronic device <NUM>) to approach a particular network and may include "Invite Saskia (e.g., the user of the external electronic device <NUM>) to my house.

According to an embodiment of the disclosure, the intelligent server <NUM> may provide a service for the other party's connection to the account server <NUM>, which controls connection to the network related to the utterance <NUM>, or the IoT server <NUM>, which controls at least one electronic device (e.g., home applications) included in the network, and for controlling the electronic devices in the network.

According to an embodiment of the disclosure, in response to the utterance <NUM>, the intelligent server <NUM> may transmit information indicating that an invitation to the network has been made by the user (e.g., Russ) of the electronic device <NUM> to the external electronic device <NUM>, and the external electronic device <NUM> may display a message <NUM> including the information and at least one item (e.g., accept or decline) to allow the second user to select a response to the information.

<FIG> is a flowchart illustrating example operations of an electronic device and server providing an intelligent service based on an utterance between users who perform a call according to an embodiment of the disclosure.

Referring to <FIG>, an intelligent system may include at least two electronic devices (e.g., <NUM> and <NUM>) performing a call and/or an intelligent server <NUM>.

In operation <NUM>, the electronic device <NUM> and the external electronic device <NUM> may perform a call.

In operation <NUM>, the electronic device <NUM> may detect a reception of an invocation to activate the intelligent system. For example, the invocation may include a voice including designated words or selection of a designated button.

According to an embodiment, as the intelligent system is activated on the electronic device <NUM>, the external electronic device <NUM> which is performing the call with the electronic device <NUM>, may output a notification to indicate that the intelligent service is provided or a message to inquire whether the intelligent service is activated based on a later utterance received.

In operation <NUM>, the electronic device <NUM> may copy microphone path data. For example, the microphone path data may include voice data obtained through the microphone <NUM> of the electronic device <NUM>.

In operation <NUM>, the electronic device <NUM> may transfer intelligent service data to the intelligent server <NUM>. For example, the intelligent service data may include at least part of voice data obtained during the electronic device <NUM> is performing the call with the external electronic device <NUM> after the intelligent service is activated among the microphone path data.

In operation <NUM>, the electronic device <NUM> may transmit (e.g., upload) intelligent service data (e.g., voice data) to the intelligent server <NUM>.

According to an embodiment, the transmitted data may include the voice data "Invite Saskia (the user of the external electronic device <NUM>) to Bixby home!" to allow the electronic device <NUM> to invite the external electronic device <NUM> to a particular network.

In operation <NUM>, the intelligent server <NUM> may analyze the intelligent service data transferred from the electronic device <NUM>.

In operation <NUM>, the intelligent server <NUM> may search for a function requested by the electronic device <NUM> based on the analyzed data.

In operation <NUM>, the intelligent server <NUM> may generate a path rule to perform the requested function. For example, the path rule may include the function of transmitting a message to request an invitation to access a network to the external electronic device <NUM> to configure the network with the external electronic device <NUM> and to allow the external electronic device <NUM> to control at least one device (e.g., home appliances) in the network.

In operation <NUM>, the intelligent server <NUM> may transmit the generated path rule to the electronic device <NUM> and the external electronic device <NUM>.

In operation <NUM>, the electronic device <NUM> may analyze the path rule.

In operation <NUM>, the external electronic device <NUM> may analyze the path rule.

In operation <NUM>, the electronic device <NUM> may identify an application appropriate for performing operations included in the path rule among applications installed on the electronic device <NUM> based on the information contained in the analyzed path rule.

In operation <NUM>, the external electronic device <NUM> may identify an application appropriate for performing operations included in the path rule among applications installed on the external electronic device <NUM> based on the information contained in the analyzed path rule.

In operation <NUM>, the electronic device <NUM> may transmit an invitation request to the external electronic device <NUM> via the identified application. For example, the invitation request may be transmitted in the form of a message (e.g., a short message service (SMS) or push message) via the IoT server <NUM> or the application server <NUM>.

In operation <NUM>, the external electronic device <NUM> may identify the invitation request received from the electronic device <NUM> through the identified application and identify whether the user of the external electronic device <NUM> accepts the invitation request.

In operation <NUM>, the external electronic device <NUM> may identify that the user has accepted the invitation request. For example, the external electronic device <NUM> may identify whether the user has accepted the invitation request based on the user's voice input or the user's input responsive to the message indicating the invitation request which is displayed in the form of a message.

In operation <NUM>, the external electronic device <NUM> may transfer a message indicating that the invitation request has been accepted to the electronic device <NUM>. For example, the external electronic device <NUM> may transmit a message to acknowledge the acceptance of the invitation request to the electronic device <NUM> directly or via a server (e.g., the IoT server <NUM> or the application server <NUM>).

In operation <NUM>, the electronic device <NUM> may display a result of the invitation request. For example, the result may be output through a speaker (e.g., the sound output device <NUM>) or a display (e.g., the display device <NUM>) in various forms (e.g., a voice, vibration, or message).

<FIG> is a diagram illustrating an example method for generating a path rule by a path natural language understanding module (NLU) that may be implemented as part of embodiment.

Referring to <FIG>, according to an embodiment, the NLU module <NUM> may functions of an app into any one operation (e.g., state A to state F) and store in the path rule database <NUM>. For example, the NLU module <NUM> may store a path rule set including a plurality of path rules A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-E-F and A-B2 divided into any one operation in the path rule database <NUM>.

According to an embodiment of the disclosure, the path rule database <NUM> of the path planner module <NUM> may store the path rule set to perform the functions of the app. The path rule set may include a plurality of path rules including the plurality of operations (e.g., a sequence of states). In the plurality of path rules, the operations executed based on the parameters each input to a respective one of the plurality of operations may sequentially be arranged. According to an embodiment of the disclosure, the plurality of path rules may be configured in the form of ontology or a graph model and stored in the path rule database <NUM>.

According to an embodiment of the disclosure, the NLU module <NUM> may select the optimal one A-B1-C3-D-F of the plurality of path rules A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-E-F and A-B2 corresponding to the parameters and the intent of the user input.

According to an embodiment of the disclosure, the NLU module <NUM> may transfer the plurality of path rules to the electronic device <NUM> unless there is a path rule perfectly matching the user input. For example, the NLU module <NUM> may select the path rule (e.g., A-B1) partially corresponding to the user input. The NLU module <NUM> may select one or more path rules (e.g., A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-E-F, A-B2) including the path rule (e.g., A-B1) partially corresponding to the user input and transfer the selected one or more path rules to the electronic device <NUM>.

According to an embodiment of the disclosure, the NLU module <NUM> may select one of the plurality of path rules based on an additional input of the electronic device <NUM> and transfer the selected path rule to the electronic device <NUM>. For example, the NLU module <NUM> may select one (e.g., A-B1-C3-D-F) among the plurality of path rules (e.g., A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-E-F, A-B2) based on an additional user input (e.g., an input to select C3) of the electronic device <NUM> and send the selected path rule to the electronic device <NUM>.

According to an embodiment of the disclosure, the NLU module <NUM> may determine the user's intent and parameters corresponding to the additional user input (e.g., an input to select C3) to the electronic device <NUM> through the NLU module <NUM> and send the user's intent or parameters determined to the electronic device <NUM>. The electronic device <NUM> may select one (e.g., A-B1-C3-D-F) among the plurality of path rules (e.g., A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-E-F, A-B2) based on the parameters or intent sent.

Accordingly, the electronic device <NUM> may complete the operations of the apps <NUM> and <NUM> by the selected path rule.

According to an embodiment of the disclosure, when a user input having insufficient information is received by the intelligent server <NUM>, the NLU module <NUM> may generate a path rule partially corresponding to the received user input. For example, the NLU module <NUM> may send the partially corresponding path rule to the intelligent agent <NUM>. The processor <NUM> may execute the intelligent agent <NUM> to receive the path rule and transfer the partially corresponding path rule to the execution manager module <NUM>. The processor <NUM> may execute a first app <NUM> based on the path rule through the execution manager module <NUM>. The processor <NUM> may, through the execution manager module <NUM>, send information about the insufficient parameters to the intelligent agent <NUM> while executing the first app <NUM>. The processor <NUM> may, through the intelligent agent <NUM>, send a request for additional input to the user using the information about the insufficient parameters. Upon receiving an additional input from the user, the processor <NUM> may, through the intelligent agent <NUM>, send the path rule to the intelligent server <NUM> for processing. The NLU module <NUM> may generate an added path rule based on the parameter information and intent of the additional user input and send the path rule to the intelligent agent <NUM>. The processor <NUM> may, through the intelligent agent <NUM>, send the path rule to the execution manager module <NUM> to execute a second app <NUM>.

According to an embodiment of the disclosure, when a user input having some missing information is received by the intelligent server <NUM>, the NLU module <NUM> may send a request for user information to the personal information server <NUM>. The personal information server <NUM> may send, to the NLU module <NUM>, information about the user who has obtained the user input stored in the persona database. The NLU module <NUM> may select a path rule corresponding to the user input having some missing operations using the user information. Accordingly, although a user input having some missing information is received by the intelligent server <NUM>, the NLU module <NUM> may send a request for the missing information and receive an additional input, or the NLU module <NUM> may use the user information, determining a path rule corresponding to the user input.

Table <NUM> below may represent an example path rule related to tasks requested by the user according to an embodiment.

Referring to Table <NUM>, a path rule generated or selected by an intelligent server (e.g., the intelligent server <NUM> of <FIG>) according to a user utterance (e.g., "Share photos") may include at least one state <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>. For example, the at least one state (e.g., any one operation state of the terminal) may correspond to at least one of executing a photo application (PictureView) <NUM>, executing a photo search function (SearchView) <NUM>, outputting a search result display screen (SearchViewResult) <NUM>, outputting a search result display screen with no photo selected (SearchEmptySelectedView) <NUM>, outputting a search result display screen with at least one photo selected (SearchSelectedView) <NUM>, and/or outputting a shared application selection screen (CrossShare) <NUM>. According to an embodiment, the path rule parameter information may correspond to at least one state. For example, it may be included in the state <NUM> of outputting a search result display screen with at least one photo selected.

As a result of performing the path rule including the sequence of states <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM>, the task (e.g., "Share photos!") requested by the user may be performed.

The electronic devices may include, for example, and without limitation, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, a home appliance, or the like.

As used herein, each of such phrases as "A or B," "A and/or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C," may include all possible combinations of the items enumerated together in a corresponding one of the phrases. It is to be understood that if an element (e.g., a first element) is referred to, with or without the term "operatively" or "communicatively", as "coupled with," "coupled to," "connected with," or "connected to" another element (e.g., a second element), the element may be coupled with the other element directly (e.g., wiredly), wirelessly, or via a third element.

As used herein, the term "module" may include a unit implemented in hardware, software, firmware, or any combinations thereof, and may interchangeably be used with other terms, for example, "logic," "logic block," "part," or "circuitry". For example, according to an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Wherein, the term "non-transitory" simply means that the storage medium is a tangible device, but this term does not differentiate between when data is semi-permanently stored in the storage medium and when the data is temporarily stored in the storage medium.

According to various embodiments, one or more other components may be added. According to various embodiments, operations performed by the module, the program, or another component may be carried out sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order or one or more other operations may be added.

As is apparent from the foregoing description, according to various embodiments of the disclosure, an electronic device and a method of executing a function of the electronic device may activate an intelligent system to process the user's utterance during a call with another electronic device. Thus, the user may receive various services from the intelligent system according to utterances received during the call.

Claim 1:
An electronic device (<NUM>), comprising:
at least one communication circuit (<NUM>);
a speaker (<NUM>);
a microphone (<NUM>);
at least one processor (<NUM>) operatively connected with the communication circuit (<NUM>), the speaker (<NUM>), and the microphone (<NUM>); and
a memory (<NUM>) operatively connected with the at least one processor (<NUM>),
wherein the memory (<NUM>) stores instructions which, when executed by the processor (<NUM>), cause the electronic device (<NUM>) to:
perform a call with a first external electronic device (<NUM>) through the communication circuit (<NUM>);
activate an intelligent service on the electronic device (<NUM>) while the call is performed;
when the intelligent service is activated, identify a voice input through the microphone (<NUM>) while the call is performed and transmit data associated with the voice input to an external server (<NUM>) through the communication circuit (<NUM>) at least in part for automatic speech recognition, 'ASR', and/or natural language understanding, 'NLU', the voice input including information related to inviting the first external electronic device (<NUM>) to a designated network;
receive a path rule from the external server (<NUM>) for performing a function of transmitting an invitation to access the designated network to the first external electronic device (<NUM>) and to allow the first external electronic device (<NUM>) to control at least one device in the network;
analyze the path rule to identify an application appropriate for performing operations included in the path rule among applications installed on the electronic device (<NUM>); and
control the at least one communication circuit (<NUM>) to transmit the invitation to the first external electronic device (<NUM>) via the identified application.