Patent Description:
Today, various types of electronic devices such as smart phones, tablet personal computers, or wearable devices are widely used. In order to increase the usability of such electronic devices, not only hardware components but also software components have been continuously improved. For example, the electronic device may provide an intelligent agent service for performing various functions in response to a user voice input.

Electronic devices and methods for recognizing user's speech, based on voice recognition and/or lip reading, are described in <CIT>, <CIT>, <CIT>, <CIT>, and <CIT>.

An intelligent agent service may recognize a voice, analyze the recognized voice, and provide a service, which a user desires to perform, to the user. The intelligent agent service operates on the basis of a user's voice and thus the voice recognition performance thereof may be influenced by noise (for example, surrounding noise) other than the user's voice. For example, recognizing a user's voice in the state in which surroundings of the electronic device is silent (for example, low noise) and recognizing a user's voice in the state in which surroundings of the electronic device is noisy (for example, high noise) may be significantly different.

In accordance with an aspect of the disclosure, an electronic device is provided according to claim <NUM>.

In accordance with another aspect of the disclosure, a method by an electronic device is provided according to claim <NUM>. Further preferable aspects are set forth in the dependent claims.

An electronic device according to various embodiments of the disclosure uses image information to improve the accuracy of an intelligent agent service. The electronic device analyzes a function, which the user desires to perform, by applying a lip reading technology based on an acquired image. The electronic device can determine a function which the user desires on the basis of at least one of a user's voice and the shape of a user's lips and provide the determined function to the user. The user can detect a time point at which a user speech starts and a time point at which the user speech ends on the basis of the shape of the user's lips and correct the received unclear user voice through the lip shape, thereby provide a more accurate intelligent agent service to the user. Further the invention enables informing the user about a failure to recognize lip movements when the brightness condition is low.

It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document: the terms "include" and "comprise," as well as derivatives thereof, mean inclusion without limitation; the term "or," is inclusive, meaning and/or; the phrases "associated with" and "associated therewith," as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term "controller" means any device, system or part thereof that controls at least one operation, such a device may be implemented in hardware, firmware or software, or some combination of at least two of the same.

Definitions for certain words and phrases are provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

According to an embodiment, the electronic device <NUM> may include a processor <NUM>, memory <NUM>, an input device <NUM>, an audio output device <NUM>, a display device <NUM>, an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a haptic module <NUM>, a camera module <NUM>, a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module(SIM) <NUM>, or an antenna module <NUM>.

The audio output device <NUM> may output sound signals to the outside of the electronic device <NUM>. The audio output device <NUM> may include, for example, a speaker or a receiver.

According to an embodiment, the audio module <NUM> may obtain the sound via the input device <NUM>, or output the sound via the audio output device <NUM> or a headphone of an external electronic device (e.g., an electronic device <NUM>) directly (e.g., wiredly) or wirelessly coupled with the electronic device <NUM>.

<FIG> is a block diagram illustrating an integrated intelligence system according to an embodiment.

Referring to <FIG>, the integrated intelligence system according to an embodiment may include a user terminal <NUM>, an intelligent server <NUM>, and a service server <NUM>.

The user terminal <NUM> according to an embodiment may be a terminal device (or an electronic device) capable of being connected to the Internet, and may include, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a notebook computer, a TV, white goods, a wearable device, an HMD, or a smart speaker.

According to the embodiment, the user terminal <NUM> may include a communication interface <NUM>, a microphone <NUM>, a speaker <NUM>, a display <NUM>, a memory <NUM>, or a processor <NUM>. The listed elements may be operatively or electrically connected to each other.

The communication interface <NUM> according to an embodiment may be connected to an external device and configured to transmit and receive data. The microphone <NUM> according to an embodiment may receive a sound (for example, a user speech) and convert the same to an electrical signal. The speaker <NUM> according to an embodiment may output the electrical signal in the form of a sound (for example, voice). The display <NUM> according to an embodiment may be configured to display an image or a video. The display <NUM> according to an embodiment may display a graphic user interface (GUI) of an executed app (or application).

The memory <NUM> according to an embodiment store a client module <NUM>, a software development kit (SDK) <NUM>, and a plurality of apps <NUM>. The client module <NUM> and the SDK <NUM> may configure framework (or a solution program) for performing a universal function. Further, the client module <NUM> or the SDK <NUM> may configure framework for processing a voice input.

The plurality of apps <NUM> may be programs for performing a predetermined function. According to an embodiment, the plurality of apps <NUM> may include a first app 229_1 and a second app 229_2. According to an embodiment, each of the plurality of apps <NUM> may include a plurality of operations for performing predetermined functions. For example, the apps may include an alarm app, a message app, and/or a schedule app. According to an embodiment, the plurality of apps <NUM> may be executed by the processor <NUM> so as to sequentially perform at least some of the plurality of operations.

The processor <NUM> according to an embodiment may control the overall operation of the user terminal <NUM>. For example, the processor <NUM> may be electrically connected to the communication interface <NUM>, the microphone <NUM>, the speaker <NUM>, and the display <NUM> to perform predetermined operations.

The processor <NUM> according to an embodiment may perform a predetermined function by executing a program stored in the memory <NUM>. For example, the processor <NUM> may perform the following operation for processing a voice input by executing at least one of the client module <NUM> or the SDK <NUM>. The processor <NUM> may control, for example, the operation of the plurality of apps <NUM> through the SDK <NUM>. The following operation which is the operation of the client module <NUM> or the SDK <NUM> may be performed by the processor <NUM>.

The client module <NUM> according to an embodiment may receive a voice input. For example, the client module <NUM> may receive a voice signal corresponding to a user speech detected through the microphone <NUM>. The client module <NUM> may transmit the received voice input to the intelligent server <NUM>. The client module <NUM> may transmit state information of the user terminal <NUM> along with the received voice input to the intelligent server <NUM>. The state information may be, for example, execution state information of the app.

The client module <NUM> according to an embodiment may receive the result corresponding to the received voice input. For example, if the intelligent module <NUM> obtains the result corresponding to the received voice input, the client module <NUM> may receive the result corresponding to the received voice input. The client module <NUM> may display the received result on the display <NUM>.

The client module <NUM> according to an embodiment may receive a plan corresponding to the received voice input. The client module <NUM> may display the result obtained by performing the plurality of operations of the app on the display <NUM> according to the plan. The client module <NUM> may sequentially display, for example, the execution result of the plurality of operations on the display. In another example, the user terminal <NUM> may display, only some result of the plurality of operations (only the result of the last operation) on the display.

According to an embodiment, the client module <NUM> may receive a request for acquiring information used for obtaining the result corresponding to the voice input from the intelligent server <NUM>. According to an embodiment, the client module <NUM> may transmit the information to the intelligent server <NUM> in response to the request.

The client module <NUM> according to an embodiment may transmit result information of the execution of the plurality of operations to the intelligent server <NUM> according to the plan. The intelligent server <NUM> may identify that the received voice input is correctly processed using the result information.

The client module <NUM> according to an embodiment may include a voice recognition module. According to an embodiment, the client module <NUM> may recognize a voice input for performing a limited function through the voice recognition module. For example, the client module <NUM> may execute an intelligent app for processing a voice input to perform an organic operation through a predetermined input (for example, Wake up!).

The intelligent server <NUM> according to an embodiment may receive information related to a user voice input from the user terminal <NUM> through a communication network. According to an embodiment, the intelligent server <NUM> may change data related to the received voice input into text data. According to an embodiment, the intelligent server <NUM> may generate a plan for performing a task corresponding to the user voice input on the basis of the text data.

According to an embodiment, the plan may be generated by an artificial intelligent (AI) system. The intelligent system may be a rule-based system, a neural network-based system (for example, a feedforward neural network (FNN)), or a recurrent neural network (RNN)). Alternatively, the intelligent system may be a combination thereof or an intelligent system different therefrom. According to an embodiment, the plan may be selected from a combination of predefined plans or generated in real time in response to a user request. For example, the intelligent system may select at least one plan from among a plurality of predefined plans.

The intelligent server <NUM> according to an embodiment may transmit the result of the generated plan to the user terminal <NUM> or transmit the generated plan to the user terminal <NUM>. According to an embodiment, the user terminal <NUM> may display the result of the plan on the display. According to an embodiment, the user terminal <NUM> may display the result of the operation according to the plan on the display.

The intelligent server <NUM> according to an embodiment may include a front end <NUM>, a natural language platform <NUM>, a capsule DB <NUM>, an execution engine <NUM>, and end user interface <NUM>, a management platform <NUM>, a big data platform <NUM>, or an analytic platform <NUM>.

According to an embodiment, the front end <NUM> may receive the received voice input from the user terminal <NUM>. The front end <NUM> may transmit a response to the voice input.

According to an embodiment, the natural language platform <NUM> may include an automatic speech recognition module (ASR module) <NUM>, a natural language understanding (NLU) module <NUM>, a planner module <NUM>, natural language generator (NLG) module <NUM>, or a text to speech (TTS) module <NUM>.

The automatic speech recognition module <NUM> according to an embodiment may convert the voice input received from the user terminal <NUM> into text data. The natural language understanding module <NUM> according to an embodiment may detect a user's intention on the basis of text data of the voice input. For example, the natural language understanding module <NUM> may detect a user's intention by performing syntactic analysis or semantic analysis. The natural language understanding module <NUM> according to an embodiment may detect a meaning of a word extracted from the voice input on the basis of a linguistic characteristic of a morpheme or a phrase (for example, grammatical element) and match the detected meaning of the word and the intention so as to determine the user's intention.

The planner module <NUM> according to an embodiment may generate a plan on the basis of the intention determined by the natural language understanding module <NUM> and a parameter. According to an embodiment, the planner module <NUM> may determine a plurality of domains used for performing a task on the basis of the determined intention. The planner module <NUM> may determine a plurality of operations included in the plurality of domains determined on the basis of the intention. According to an embodiment, the planner module <NUM> may determine a parameter used for performing the plurality of determined operations or a result value output by the execution of the plurality of operations. The parameter and the result value may be defined by a concept of a predetermined type (or class). According to an embodiment, the plan may include a plurality of operations determined by the user's intention and a plurality of concepts. The planner module <NUM> may gradually (or hierarchically) determine the relationship between the plurality of operations and the plurality of concepts. For example, the planner module <NUM> may determine the execution order of the plurality of operations determined on the basis of the user's intention based on the plurality of concepts. In other words, the planner module <NUM> may determine the execution order of the plurality of operations on the basis of the parameter used for performing the plurality of operations and the result output by the execution of the plurality of operations. Accordingly, the planner module <NUM> may generate a plan including information on the relationship (ontology) between the plurality of operations and the plurality of concepts. The planner module <NUM> may generate a plan on the basis of information stored in the capsule database <NUM> that stores a set of relationships between concepts and operations.

The natural language generator module <NUM> according to an embodiment may convert predetermined information in the form of text. The information converted into the form of text may be the form of a natural language speech. The text to speech module <NUM> may convert information in the form of text into information in the form of voice.

According to an embodiment, some or all of the functions of the natural language platform <NUM> may also be implemented by the user terminal <NUM>.

The capsule database <NUM> may store information on the relationship between a plurality of concepts and operations corresponding to a plurality of domains. The capsule according to an embodiment may include a plurality of operation objects (action objects or action information) and concept objects (or concept information). According to an embodiment, the capsule database <NUM> may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment, the plurality of capsules may be stored in a function registry included in the capsule database <NUM>.

The capsule database <NUM> may include a strategy registry storing strategy information used when a plan corresponding to a voice input is determined. When there are a plurality of plans corresponding to the voice input, the strategy information may include reference information for determining one plan. According to an embodiment, the capsule database <NUM> may include a follow-up registry storing the following operation to suggest a follow-up operation to the user in a predetermined situation. The follow-up operation may include, for example, the following speech. According to an embodiment, the capsule database <NUM> may include a layout registry storing layout information corresponding to information output through the user terminal <NUM>. According to an embodiment, the capsule database <NUM> may include a vocabulary registry storing vocabulary information included in the capsule information. According to an embodiment, the capsule database <NUM> may include a dialog registry storing dialog (or interaction) information with the user. The capsule database <NUM> may update the stored object through a developer tool. The developer tool may include a function editor for updating, for example, the operation object or the concept object. The developer tool may include a vocabulary editor for updating a vocabulary. The developer tool may include a strategy editor for generating and registering strategy to determine a plan. The developer tool may include a dialog editor for generating dialog with the user. The developer tool may include a follow-up editor for activating a follow-up goal and editing the following speech that provides a hint. The follow-up goal may be determined on the basis of the current goal, a user's preference, or an environment condition. According to an embodiment, the capsule database <NUM> may be implemented inside the user terminal <NUM>.

The execution engine <NUM> according to an embodiment may obtain the result on the basis of the generated plan. The end user interface <NUM> may transmit the obtained result to the user terminal <NUM>. Accordingly, the user terminal <NUM> may receive the result and provide the received result to the user. The management platform <NUM> according to an embodiment may manage information used by the intelligent server <NUM>. The big data platform <NUM> according to an embodiment may collect user data. The analytic platform <NUM> according to an embodiment may manage quality of service (QoS) of the intelligent server <NUM>. For example, the analytic platform <NUM> may manage elements and a processing speed (or efficiency) of the intelligent server <NUM>.

The service server <NUM> according to an embodiment may provide a predetermined service (for example, food ordering or hotel booking) to the user terminal <NUM>. According to an embodiment, the service server <NUM> may be a server operated by a third party. The service server <NUM> according to an embodiment may provide information for generating a plan corresponding to the received voice input to the intelligent server <NUM>. The provided information may be stored in the capsule database <NUM>. Further, the service server <NUM> may provide result information of the plan to the intelligent server <NUM>.

In the integrated intelligence system described above, the user terminal <NUM> may provide various intelligent services to the user in response to the user input. The user input may include, for example, an input through a physical button, a touch input, or a voice input.

According to an embodiment, the user terminal <NUM> may provide a voice recognition service through an intelligent app (or a voice recognition app) stored in the user terminal <NUM>. In this case, for example, the user terminal <NUM> may recognize a user speech (utterance) or a voice input received through the microphone and provide a service corresponding to the recognized voice input to the user.

According to an embodiment, the user terminal <NUM> may perform a predetermined operation on the basis of the received voice input along or together with the intelligent server and/or the service server. For example, the user terminal <NUM> may execute an app corresponding to the received voice input and perform a predetermined operation through the executed app.

According to an embodiment, when the user terminal <NUM> provides the service together with the intelligent server <NUM> and/or the service server, the user terminal may detect a user speech through the microphone <NUM> and generate a signal (or voice data) corresponding to the detected user speech. The user terminal may transmit the voice data to the intelligent server <NUM> through the communication interface <NUM>.

The intelligent server <NUM> according to an embodiment may generate a plan for performing a task corresponding to the voice input or the result of the operation according to the plan in response to the voice input received from the user terminal <NUM>. The plan may include, for example, a plurality of operations for performing a task corresponding to the voice input of the user and a plurality of concepts related to the plurality of operations. The concepts may be parameters input to execution of the plurality of operations or may be defined for result values output by the execution of the plurality of operations. The plan may include the relationship between the plurality of operations and the plurality of concepts.

The user terminal <NUM> according to an embodiment may receive the response through the communication interface <NUM>. The user terminal <NUM> may output a voice signal generated by the user terminal <NUM> to the outside through the speaker <NUM> or output an image generated by the user terminal <NUM> to the outside through the display <NUM>.

<FIG> illustrates the form in which information on the relationship between concepts and operations is stored in a database according to various embodiments.

A capsule database (for example, the capsule database <NUM>) of the intelligent server <NUM> may store capsules in the form of a concept action network (CAN). The capsule database may store an operation for processing a task corresponding to a user voice input and a parameter used for the operation in the form of a concept action network (CAN).

The capsule database may store a plurality of capsules (capsule A <NUM> and capsule B <NUM>) corresponding to a plurality of domains (for example, applications). According to an embodiment, one capsule (for example, capsule A <NUM>) may correspond to one domain (for example, location (geo) or application). Further, one capsule may correspond to at least one service provider (for example, CP #<NUM><NUM>, CP #<NUM><NUM>, CP#<NUM><NUM>, and CP #<NUM><NUM>) for performing a function of a domain related to the capsule. According to an embodiment, one capsule may include one or more operations for performing a predetermined function and one or more concepts.

The natural language platform <NUM> may generate a plan for performing a task corresponding to the received voice input through the capsules stored in the capsule database. For example, the planner module <NUM> of the natural language platform may generate a plan through capsules stored in the capsule database. For example, a plan <NUM> may be generated using operations 241_1 and 241_3 and concepts 241_2 and 241_4 of capsule A <NUM> and an operation 244_1 and a concept 244_2 of capsule B <NUM>.

<FIG> illustrates screens on which the user terminal processes a received voice input through an intelligent app according to various embodiments.

The user terminal <NUM> may execute an intelligent app in order to process a user input through the intelligent server <NUM>.

According to an embodiment, in a screen <NUM>, when recognizing a predetermined voice input (for example, wake up!) or receiving an input through a hardware key (for example, a dedicated hardware key), the user terminal <NUM> may execute an intelligent app for processing the voice input. The user terminal <NUM> may execute the intelligent app in the state in which, for example, a schedule app is executed. According to an embodiment, the user terminal <NUM> may display an object <NUM> (for example, icon) corresponding to the intelligent app on the display <NUM>. According to an embodiment, the user terminal <NUM> may receive the voice input by a user speech. For example, the user terminal <NUM> may receive the voice input "Let me know my schedule this week!" According to an embodiment, the user terminal <NUM> may display a user interface (UI) <NUM> (for example, an input window) of the intelligent app displaying text data of the received voice input on the display.

According to an embodiment, in a screen <NUM>, the user terminal <NUM> may display the result corresponding to the received voice input on the display. For example, the user terminal <NUM> may receive a plan corresponding to the received user input and display the "schedule this week" according to the plan on the display.

An electronic device according to various embodiments disclosed herein may be various types of devices. According to an embodiment, the electronic devices are not limited to those described above.

It is to be understood that if an element (e.g., a first element) is referred to, with or without the term "operatively" or "communicatively", as "coupled with," or "connected with,", it means that the element may be coupled with the other element directly (e.g., wiredly), wirelessly, or via a third element.

For example, a processor (e.g., the processor <NUM>) of the machine (e.g., the electronic device <NUM>) may invoke at least one of the one or more instructions stored in the storage medium, and execute it. This allows the machine to be operated to perform at least one function according to the invoked at least one instruction. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded) online via an application store (e.g., Play Store<IMG>), or between two user devices (e.g., smart phones) directly.

According to various embodiments, one or more components of the above-described components or operations may be omitted, or one or more other components or operations may be added.

<FIG> schematically illustrates an intelligent agent system according to various embodiments of the disclosure.

According to various embodiments, the electronic device <NUM> performs a function according to a user's command through an intelligent agent system <NUM>. For example, the intelligent agent system <NUM> may include a context information identification module <NUM>, a voice information recognition module <NUM>, an image information recognition module <NUM>, and a function execution module <NUM>. The intelligent agent system <NUM> of <FIG> may include at least some elements or modules of the integrated intelligent system in <FIG> or may correspond to a combination of at least some elements or modules.

According to various embodiments, the context information identification module <NUM> identifies at least one piece of context information related to the electronic device <NUM>. For example, the context information may include a level of noise around the electronic device <NUM>. If the user of the electronic device <NUM> attends a concert, the context information identification module <NUM> may acquire and store information indicating that it is very noisy around the electronic device <NUM> as the context information. In another example, the context information may include information on an application being executed by the electronic device <NUM>. The context information identification module <NUM> may acquire the information on the application being executed, such as a video or music being reproduced by the electronic device <NUM>, as the context information. With respect to a specific application, the context information identification module <NUM> may acquire information on an operation state of the specific application as the context information. For example, if the specific application is a video application, the context information identification module <NUM> may identify whether the video is currently reproduced or stopped. In another example, the context information identification module <NUM> may identify whether the specific application is being executed in a foreground or a background. The electronic device <NUM> may use at least one piece of context information acquired by the context information identification module <NUM> to provide an intelligent agent function by the electronic device <NUM>.

According to various embodiments, the voice information recognition module <NUM> may receive a user's voice and determine what a user's command is on the basis of the received voice. For example, the voice information recognition module <NUM> may convert a voice into text using automatic speech recognition (ASR) and perform a function based on the converted text.

According to various embodiments, if it is very noisy around the electronic device <NUM>, the voice cannot be smoothly received from the user. In another example, if a place in which the user is located is a library where silence is required, the user may user may have difficulty inputting with a loud voice. In the above case, the electronic device <NUM> may have a difficulty in performing the intelligent agent service through the voice input.

According to various embodiments, the image information recognition module <NUM> receives information on a user image through the camera module <NUM> of the electronic device <NUM> may perform lip reading on the basis of the received image information. For example, the lip reading may be a technology for analyzing motion of a user's lips and recognizing a word or a sentence. The image information recognition module <NUM> may use a method of finding the most suitable word or sentence on the basis of the received lip image through a trained model. Accordingly, unlike the voice information recognition module <NUM>, the image information recognition module may not generate text.

As described above, if it is difficult to identify a function which the user desires to perform only through the user's voice, the lip reading may be performed on the basis of the received image information and thus the accuracy of the intelligent agent service can be improved.

According to various embodiments, the image information recognition module <NUM> may compensate for voice information to be processed by the voice information recognition module <NUM> through lip reading. The electronic device <NUM> may identify a user's command by combining voice information and lip reading information and perform a function corresponding to the identified user's command.

According to various embodiments, the image information recognition module <NUM> may identify the number of lips included in the image, a user who is making a voice command, and information on a start point or an end point of the speech.

According to various embodiments, the image information recognition module <NUM> may provide at least one piece of information related to lip reading to the user interface. For example, the image information recognition module <NUM> may provide the user interface with information indicating whether lip reading is being performed, information on lips included in the image, information on a lip performing lip reading, or information on a lip reading error.

According to various embodiments, the image information recognition module <NUM> may provide a guide about lip reading in order to facilitate lip reading. For example, the image information recognition module <NUM> may provide a close up of a user's lips and provide the same through a separate preview screen. In another example, if it is difficult to detect a user's lips in the received image, the image information recognition module <NUM> may guide the electronic device <NUM> to control orientation.

According to various embodiments, if the number of lips included in the image is plural, the image information recognition module <NUM> may display the lips such that the lip recognized to be currently making a speech is distinguished from other lips simply included in the image. For example, the lip determined to be actually making the voice command may be displayed with borders around a peripheral region thereof, and the lips simply included in the image may be displayed without special indication. In another example, the lips determined to be actually inputting the voice command may be displayed with a first border therearound, and lips simply included in the image may be displayed with a second border therearound. A shape (for example, a solid line or a dotted line), a color, and a thickness of the first border may be different from those of the second border.

According to various embodiments, the function execution module <NUM> may determine what a user's command is on the basis of at least one piece of context information of the electronic device <NUM>, voice information, or image information and perform a function corresponding to the determined command.

<FIG> is a block diagram illustrating an electronic device for performing lip reading according to various embodiments of the disclosure.

According to various embodiments, an electronic device <NUM> (for example, the electronic device <NUM> of <FIG> or the user terminal <NUM> of <FIG>) may be divided into a plurality of modules for performing lip reading.

According to various embodiments, a client module <NUM> may serve to access an intelligent agent service and output the result of the intelligent agent service. For example, the client <NUM> may perform a lip reading function according to information on an application currently being executed and context of the electronic device <NUM>.

According to various embodiments, an automatic speech recognition (ARS) module <NUM> may convert an input voice signal into text.

According to various embodiments, a lip verification module <NUM> may identify whether lips are detected in image information acquired through the camera module <NUM> and identify whether movement of the lip can be recognized. For example, the lip identification module <NUM> may determine whether the image information acquired through the camera module corresponds to an image in which the lips can be recognized and determine whether to continue lip recognition or perform only voice recognition.

According to various embodiments, a lip recognizer module <NUM> may detect movement of the lips on the basis of a general lip reading model <NUM> and a personal lip reading model <NUM> in a recognized lip area and recognize or estimate a user speech.

According to various embodiments, the general lip reading model <NUM> is a model learned on the basis of a plurality of unspecified lip movements and speech information corresponding thereto and may secure the recognition performance of general lip reading.

According to various embodiments, the personal lip reading model <NUM> may secure the recognition performance optimized for the user of the electronic device <NUM> by learning movement of the user's lips to generate a model. For example, since every person has different a speech habit and lip movement, performance deterioration may individually occur, and thus the recognition performance of lip reading may be improved through the personal lip reading model <NUM>.

According to various embodiments, the following operation may be performed through electronic device <NUM>. For example, if the intelligent agent service is executed by a user input (for example, press of a physical key or a speech of wake-up), context information of an application <NUM> or <NUM> being executed is received. For example, if a video or music is being reproduced, a camera module may be activated to recognize lips. If it is determined that lips can be recognized on the basis of image information collected from the camera module, it is determined what a user speech is through the received user speech and lip movement.

<FIG> is a flowchart illustrating a method <NUM> of performing a function corresponding to a user command through an intelligent agent service of an electronic device according to various embodiments of the disclosure.

According to various embodiments, the electronic device <NUM> may execute an intelligent agent service in operation <NUM>. The electronic device <NUM> executes the intelligent agent service by receiving a user input (for example, press of a physical key or a speech of wake-up).

The electronic device <NUM> identifies context information of the electronic device <NUM> in operation <NUM>. For example, the context information may include a level of noise around the electronic device <NUM>. In another example, the context information may include information on an application being executed by the electronic device <NUM>.

The electronic device <NUM> determines whether to acquire image information on the basis of the identified context information in operation <NUM>. For example, if noise around the electronic device <NUM> is serious, a voice cannot be smoothly received from the user, and thus the electronic device <NUM> may perform control to acquire image information. In another example, if the user is located in a place where silence is required, the user has a difficulty in inputting an input with a loud voice. In the above case, through additional acquisition of the image information, the accuracy of the intelligent agent service can be improved.

The electronic device <NUM> acquires information on an image of the user by activating the camera module in operation <NUM>.

The electronic device <NUM> recognizes a user speech on the basis of the acquired image information in operation <NUM>.

According to various embodiments, if it is determined not to acquire image information, the electronic device <NUM> may acquire voice information from the user in operation <NUM>.

The electronic device <NUM> may recognize a user speech on the basis of the voice information in operation <NUM>.

The electronic device <NUM> may perform a function corresponding to generated text in operation <NUM>.

According to various embodiments, referring to <FIG>, the electronic device <NUM> may selectively acquire image information or voice information and execute a function corresponding to a user speech. However, the disclosure is not limited to the embodiments but includes an embodiment in which the electronic device acquires both the image information and the voice information and performs a function corresponding to a user speech based on a combination thereof (for example, a combination speech).

<FIG> illustrates a method by which an electronic device recognizes a user speech on the basis of a user voice and a user's lip shape according to various embodiments of the disclosure.

According to various embodiments, the electronic device <NUM> determines whether to perform lip reading on the basis of context information in operation <NUM>. If it is determined to perform lip reading, the electronic device acquires information on an image of the user in operation <NUM>. If it is determined not to perform lip reading, the electronic device may acquire voice information in operation <NUM>. In another example, even when it is determined to perform lip reading, the electronic device <NUM> additionally acquires voice information in operation <NUM>.

If image information is acquired through lip reading, the electronic device <NUM> identifies information <NUM> on the lips. For example, a method of identifying the lip information <NUM> may include lip detection, lip activation detection, and lip tracking. For example, the lip tracking may be detection of a user's lips in the image information. For example, the lip activation detection may be detection of a start of movement of the lips. For example, the lip tracking may be continuous tracking of a shape change of the lips starting movement.

According to various embodiments, the electronic device <NUM> may recognize a user speech on the basis of the lip information <NUM> (for example, recognize a user speech only through lip movement).

According to various embodiments, the electronic device <NUM> may convert a user's voice into text through the automatic speech recognition module <NUM> in step <NUM> (for example, convert a user speech into text only through the received voice).

According to various embodiments, the electronic device <NUM> may recognize a user speech by combining the lip information <NUM> and the information acquired from the automatic speech recognition module <NUM> in step <NUM> (for example, recognize a user speech by combining the received voice and the lip movement).

According to various embodiments, if the electronic device <NUM> acquires image information through lip reading, the electronic device <NUM> may refer to a database <NUM> in order to recognize a user speech. The corresponding database <NUM> may be stored in a server or the memory <NUM> in the electronic device <NUM>.

According to various embodiments, the electronic device <NUM> may recognize a user speech corresponding to the converted text in operation <NUM>.

<FIG> illustrates a user interface when an electronic device performs a function through an intelligent agent service according to various embodiments.

According to various embodiments, referring to reference numeral <NUM>, the electronic device <NUM> may execute a first application (for example, a music reproduction application). The electronic device <NUM> may receive an input for activating an intelligent agent service from the user while the first application is executed.

According to various embodiments, referring to reference numeral <NUM>, if the first application (for example, the music reproduction application) is continuously executed in a background seamlessly (for example, music is being reproduced) in the state in which the intelligent agent service is activated, the electronic device <NUM> may receive at least one piece of a user's voice and information on the lips through the intelligent agent service. For example, the electronic device <NUM> may display information indicating that the intelligent agent service is activated and thus at least one piece of the voice and the lip information is received through a separate object or an icon <NUM>.

According to various embodiments, referring to reference numeral <NUM>, if the camera module <NUM> is activated to perform lip reading, the electronic device <NUM> may display a separate object or an icon <NUM> in order to inform the user of the same. For example, the lip reading object or the icon <NUM> may be displayed with a change in the state depending on the state of lip reading. For example, if movement of the user's lips is accurately read, the object or the icon <NUM> may be clearly displayed. If it is difficult to detect movement of the user's lips (for example, if detection of the lip location fails or if it is dark around the user), the object or the icon <NUM> may be blurredly displayed or it may flicker. In another example, if movement of the user's lip is accurately read, the object or the icon <NUM> may be displayed with a first color. If it is difficult to detect movement of the user's lips (for example, if detection of the lip location fails or if it is dark around the user), the object or the icon <NUM> may be displayed with a second color different from the first color.

According to various embodiments, referring to reference numeral <NUM>, the electronic device <NUM> may display the result of conversion from the user speech to text in real time. For example, the electronic device <NUM> may provide the user with an intermediate result <NUM> while the user speaks, and if the user finishes the speech, may provide the user with a final result <NUM>.

According to various embodiments, referring to reference numeral <NUM>, the electronic device <NUM> may temporarily stop music currently reproduced through the music reproduction application in order to receive a command from the user. In this case, since music which is currently reproduced is stopped, the user's voice may be more smoothly received. However, the user cannot listen to music seamlessly, it may inconvenience the user.

According to various embodiments, referring to reference numeral <NUM>, if the user speech corresponds to execution of the second application (for example, a gallery), the electronic device <NUM> may switch the screen to the second application. Thereafter, if a back input is received from the user, the electronic device <NUM> may return to an execution screen of the first application.

<FIG> illustrate a user interface when an electronic device performs a function through an intelligent agent service according to various embodiments of the disclosure.

According to various embodiments, referring to <FIG>, the electronic device <NUM> may reproduce a movie through a video reproduction application in a screen <NUM>. In such a situation, the electronic device <NUM> may receive an input for activating the intelligent agent service from the user. The electronic device <NUM> may receive the input from the user while the reproduced video is not paused or stopped. For example, the electronic device <NUM> may identify context information indicating that the currently executed application is a video reproduction application and that a movie is currently reproduced through the video reproduction application. In order not to interrupt the user watching a movie, the electronic device <NUM> may activate a lip reading mode while not stopping reproduction of the video and detect movement of the user's lips. Of course, in this case, a user's voice may also be received, and the electronic device <NUM> may recognize a command which the user desires to execute on the basis of the user's voice and lip movement. Meanwhile, the electronic device <NUM> may display an object or an icon <NUM> informing of activation of the lip reading mode through a user interface.

According to various embodiments, referring to <FIG>, the electronic device <NUM> may receive an input inquiring about weather on Friday from the user in a screen <NUM>. In response thereto, the electronic device <NUM> may provide the response "Snowflakes will fall on Friday". Since the user is currently watching a movie, if the electronic device <NUM> responds through a separate voice or hide the currently executed application and switches the screen to the intelligent agent service screen, smooth watching of the movie may be interrupted. Accordingly, the electronic device <NUM> may display a response <NUM> corresponding to a user command in the form similar to subtitles without a separate voice response or screen switching.

According to various embodiments, referring to <FIG>, the electronic device <NUM> may reproduce music through a music reproduction application in a screen <NUM>. In such a situation, the electronic device <NUM> may receive an input for activating the intelligent agent service from the user. The electronic device <NUM> may receive the input from the user while the reproduced music is not paused or stopped. For example, the electronic device <NUM> may identify context information indicating that the currently executed application is a music reproduction application and that music is currently reproduced through the music reproduction application. In order not to interrupt the user listening music, the electronic device <NUM> may activate a lip reading mode while not stopping reproduction of the music and detect movement of the user's lips. Of course, in this case, a user's voice may also be received, and the electronic device <NUM> may recognize a command which the user desires to execute on the basis of the user's voice and lip movement.

According to various embodiments, referring to <FIG>, the electronic device <NUM> may recognize a request for transmitting a message from the user in a screen <NUM>. In response thereto, the electronic device <NUM> may provide a response <NUM> "To whom ? <NUM>" means that a question asking to whom the message transmitted. In addition, the electronic device <NUM> may display an indicator <NUM> indicating that user feedback is currently received and display an object or an icon <NUM> indicating that lip reading is being executed. The indicator, the object, or the icon is a graphic user interface used to refer to something and may be used as a similar meaning but can be replaced with another expression. Similar to <FIG>, since the user is currently listening to music, if the electronic device <NUM> responds through a separate voice or hides the currently executed application and switches the screen to the intelligent agent service screen, uninterrupted listening of the music may be interrupted. Accordingly, the electronic device <NUM> may display a response <NUM> corresponding to a user command in a partial area in which the currently executed application is displayed in the form similar to a message without a separate voice response.

<FIG> is a flowchart illustrating a method <NUM> of performing a function corresponding to a user command on the basis of whether the electronic device is capable of lip reading according to various embodiments of the disclosure.

The electronic device <NUM> acquires information on an image of the user in operation <NUM>.

The electronic device <NUM> determines whether lips can be recognized (for example, whether lip reading can be performed) in the acquired image information in operation <NUM>.

According to various embodiments, if the lips can be recognized, the electronic device <NUM> may display a user interface (UI) for lip reading in operation <NUM>.

According to various embodiments, the electronic device <NUM> may perform lip reading and output an intermediate result of the lip reading in operation <NUM>.

According to various embodiments, the electronic device <NUM> may determine whether the input of the user command is completed in operation <NUM>.

According to various embodiments, if the input of the user command is completed, the electronic device <NUM> may output the final result of the lip reading in operation <NUM>.

According to various embodiments, if the input of the user command is not completed, the electronic device <NUM> may continue the lip reading and output the intermediate result of the lip reading in operation <NUM>.

According to various embodiments, if the lip cannot be recognized, the electronic device <NUM> may display a user interface (UI) for voice recognition in operation <NUM>.

According to various embodiments, the electronic device <NUM> may perform voice recognition and output the intermediate result for voice recognition in operation <NUM>.

According to various embodiments, if the input of the user command is completed, the electronic device <NUM> may output the final result of voice recognition in operation <NUM>.

According to various embodiments, if the input of the user command is not completed, the electronic device <NUM> may continue the voice recognition and output the intermediate result for the voice recognition in operation <NUM>.

The electronic device <NUM> performs an input command on the basis of at least one of the final result of lip reading and the final result of voice recognition in operation <NUM>.

<FIG> schematically illustrate a method of providing a user interface for a lip reading operation of an electronic device according to various embodiments of the disclosure.

According to various embodiments, referring to <FIG>, the electronic device <NUM> may perform lip reading if noise around the electronic device is higher than or equal to a predetermined level or voices of people are simultaneously input. The electronic device <NUM> may recognize what a user speech, received through a lip recognizer module and an automatic speech recognition module, is about. For example, the electronic device <NUM> may detect movement of lips included in image information as indicated by reference numerals <NUM> to <NUM> and recognize a user speech on the basis thereof.

According to various embodiments, referring to <FIG>, if image information for performing lip reading has preset brightness or lower, the accuracy of lip reading is low, so that the electronic device <NUM> may make a request for the user to move a brighter place. For example, if an image <NUM> of the user is acquired in a dark place, a mark <NUM> may be displayed around the user's lips. For example, the mark <NUM> may be indicated by a solid line if the lip reading is normally performed and may be indicated by dotted lines if it is difficult to perform the lip reading. In another example, the mark <NUM> may be expressed with a green color if the lip reading is normally performed and may be expressed with a red color if it is difficult to perform the lip reading.

According to various embodiments, referring to <FIG>, the electronic device <NUM> may detect a plurality of lips in the image information <NUM>. The electronic device <NUM> may determine which lip is currently inputting a voice command among the detected lips. For example, the electronic device <NUM> may compare movement of the shape of the user's lips with the input voice and determine the lips actually inputting the voice command among the plurality of lips on the basis thereof.

According to various embodiments, referring to <FIG>, the electronic device <NUM> may display the lips such that lips <NUM> currently inputting a voice command are distinguished from other lips <NUM> in the image information <NUM>. For example, the lips <NUM> currently inputting the voice command may be displayed with a thick solid line, and the other lips <NUM> may be displayed with light dotted lines.

According to various embodiments, the electronic device <NUM> may control weighted values of voice recognition and lip reading according to a state of the received image information. For example, if brightness of the received image is equal to or lower than preset brightness, it may be determined that the accuracy of lip reading is low. Accordingly, if the user command is analyzed through a combination of voice recognition and lip reading, a high weighted value may be allocated to the result acquired through the voice recognition and a relatively low weighted value may be allocated to the result acquired through lip reading.

According to various embodiments, if the received voice information does not correspond to the user who is actually inputting the voice command, the electronic device <NUM> may control the weighted values of the voice recognition and the lip reading. For example, if a voice of the user actually inputting the voice command and voices of other people are simultaneously received, it may be determined that the accuracy of the voice recognition is low. Accordingly, if the user command is analyzed through a combination of voice recognition and lip reading, a low weighted value may be allocated to the result acquired through the voice recognition and a relatively high weighted value may be allocated to the result acquired through lip reading.

<FIG> is a flowchart illustrating a method <NUM> of detecting movement of lips to recognize a user speech and performing a function corresponding to the recognized speech while an application is executed according to various embodiments of the disclosure.

The electronic device <NUM> receives an input for activating an intelligent agent service while an application is executed in operation <NUM>.

The electronic device <NUM> identifies context information of the electronic device <NUM> in operation <NUM>.

The electronic device <NUM> acquires information on an image of the user through a camera on the basis of the identified context information in operation <NUM>.

The electronic device <NUM> detects movement of lips included in the acquired image information and recognizes a user speech in operation <NUM>.

The electronic device <NUM> performs a function corresponding to the recognized speech in operation <NUM>.

An electronic device according to various embodiments includes a camera, a microphone, a display, a memory, and a processor, wherein the processor is configured to receive an input for activating an intelligent agent service from a user while at least one application is executed, identify context information of the electronic device, control to acquire image information of the user through the camera, based on the identified context information, detect movement of a user's lips included in the acquired image information to recognize a speech of the user, and perform a function corresponding to the recognized speech.

The processor may be configured to acquire noise around the electronic device through the microphone and store information on the acquired noise around the electronic device as the context information.

The processor may be configured to activate the camera if the noise around the electronic device is higher than or equal to a preset value.

The processor may be configured to store at least one of a type or an execution state of the at least one application being executed as the context information and activate the camera if the at least one application being executed is reproducing music or a video.

The processor is configured to control to receive a voice signal through the microphone, recognize a combination speech corresponding to a combination of the voice signal and the movement of the lips, and perform a function corresponding to the recognized combination speech.

The processor is configured to, if brightness of the acquired image information is lower than a preset value, display a user interface indicating failure of recognition of the movement of the lips to the user through the display.

The processor may be configured to, if a plurality of lips are detected based on the acquired image information, identify the voice signal and movements of the plurality of lips and display a user interface for distinguishing lips corresponding to the user from other lips through the display.

The processor may be configured to set a weighted value of a first speech recognized based on movement of the lip corresponding to the user to be higher than a weighted value of a second speech recognized based on the voice signal acquired through the microphone and recognize the combination speech.

The processor may be configured to, even when the intelligent agent service is activated, seamlessly provide at least one piece of information on the at least one application.

A method by an electronic device according to various embodiments includes an operation of receiving an input for activating an intelligent agent service from a user while at least one application is executed, an operation of identifying context information of the electronic device, an operation of acquiring image information of the user through the camera, based on the identified context information, an operation of detecting movement of a user's lips included in the acquired image information to recognize a speech of the user, and an operation of performing a function corresponding to the recognized speech.

The method may further include an operation of acquiring noise around the electronic device through the microphone and an operation of storing information on the acquired noise around the electronic device as the context information.

The method may further include an operation of, if the noise around the electronic device is higher than or equal to a preset value, activating the camera.

The method may further include an operation of storing at least one of a type or an execution state of the at least one application being executed as the context information and an operation of activating the camera if the at least one application being executed is reproducing music or a video.

The method further includes an operation of receiving a voice signal through the microphone, an operation of recognizing a combination speech corresponding to a combination of the voice signal and the movement of the lips, and an operation of performing a function corresponding to the recognized combination speech.

The method further includes an operation of, if brightness of the acquired image information is lower than a preset value, displaying a user interface indicating failure of recognition of the movement of the lips to the user through the display.

The method may further include an operation of, if a plurality of lips are detected based on the acquired image information, identifying the voice signal and movements of the plurality of lips and an operation of displaying a user interface for distinguishing lips corresponding to the user from other lips through the display.

The method may further include an operation of configuring a weighted value of a first speech recognized based on movement of the lip corresponding to the user to be higher than a weighted value of a second speech recognized based on the voice signal acquired through the microphone and recognizing the combination speech.

The method may further include an operation of, even when the intelligent agent service is activated, seamlessly providing at least one piece of information on the at least one application.

Claim 1:
An electronic device (<NUM>) comprising:
a camera (<NUM>);
a microphone (<NUM>);
a display (<NUM>);
a memory (<NUM>); and
a processor (<NUM>; <NUM>) configured to:
receive an input for activating an intelligent agent service from a user while at least one application is executed,
identify context information of the electronic device,
control to acquire image information of the user through the camera based on the identified context information,
detect movement of a user's lips included in the acquired image information to recognize a speech of the user, and
perform a function corresponding to the recognized speech,
wherein the processor is further configured to:
control to receive a voice signal through the microphone;
recognize a combination speech corresponding to a combination of the voice signal and the movement of lips; and
perform a function corresponding to the recognized combination speech, and
characterised in that
the processor is further configured to, based on brightness of the acquired image information being lower than a preset value, display a user interface indicating failure of recognition of the movement of the lips to the user through the display.