Electronic apparatus and control method thereof

An electronic apparatus is provided. The electronic apparatus includes a microphone, a memory configured to store a plurality of keyword recognition models, and a processor, which is coupled with the microphone and the memory, configured to control the electronic apparatus, wherein the processor is configured to selectively execute at least one keyword recognition model among the plurality of keyword recognition models based on operating state information of the electronic apparatus, based on a first user voice being input through the microphone, identify whether at least one keyword corresponding to the executed keyword recognition model is included in the first user voice by using the executed keyword recognition model, and based on at least one keyword identified as being included in the first user voice, perform an operation of the electronic apparatus corresponding to the at least one keyword.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119(a) of a Korean patent application number 10-2019-0156146, filed on Nov. 28, 2019 in the Korean Intellectual Property Office, the disclosure of which is incorporated by reference herein in its entirety.

BACKGROUND

The disclosure relates to an electronic apparatus and a control method thereof. More particularly, the disclosure relates to an electronic device for recognizing keywords included in a user voice and performing an operation based on the recognized keywords, and a method of controlling thereof.

2. Description of the Related Art

A voice recognition system using an artificial intelligence system has been recently employed in various fields. The artificial intelligence system is a system that machines get smarter by learning and determining itself, unlike the existing rule-based smart systems. As the artificial intelligence system is used, its recognition rate is improved and the user's taste can be understood more accurately, and the existing rule-based smart system is gradually being replaced by a deep learning-based artificial intelligence system.

The artificial intelligence techniques are formed of element technologies utilizing machine learning (e.g., deep learning) and machine learning.

The machine learning is an algorithm technology for classifying and training features of input data, the element technology is technology that simulates functions such as cognition, determination, etc. of human brains by utilizing the algorithm of the machine learning such deep learning, or the like, and is composed of verbal comprehension, visual comprehension, inference/prediction, knowledge expression, motion control, or the like. In particular, the verbal comprehension is a technology of recognizing human language/letters and applying/processing it, and includes natural language processing, machine translation, dialogue system, question and answer, voice recognition/synthesis or the like.

In the case of the existing voice recognition system, there has been a limitation that the user cannot stop an operation of the speech recognition system through user's voice requesting to stop the operation while the system provides an incorrect response without accurately identifying the intention of the user's voice.

SUMMARY

Aspects of the disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of the disclosure is to provide an electronic apparatus configured to recognize a keyword through at least one keyword recognition mode executed based on operating state information of the electronic apparatus and a method for controlling thereof

In accordance with an aspect of disclosure, an electronic apparatus is provided. The electronic apparatus includes a microphone, a memory configured to store a plurality of keyword recognition models, and a processor, which is coupled with the microphone and the memory, configured to control the electronic apparatus, wherein the processor is configured to selectively execute at least one keyword recognition model among the plurality of keyword recognition models based on operating state information of the electronic apparatus, based on a first user voice being input through the microphone, identify whether at least one keyword corresponding to the executed keyword recognition model is included in the first user voice by using the executed keyword recognition model, and based on at least one keyword identified as being included in the first user voice, perform an operation of the electronic apparatus corresponding to the at least one keyword.

In accordance with another aspect of disclosure, a control method is provided. The method of controlling an electronic apparatus includes selectively executing at least one keyword recognition model among the plurality of keyword recognition models based on operating state information of the electronic apparatus, based on a first user voice being input through a microphone, identifying whether at least one keyword corresponding to the executed keyword recognition model is included in the first user voice by using the executed keyword recognition model, and based on at least one keyword identified as being included in the first user voice, performing an operation of the electronic apparatus corresponding to the at least one keyword.

DETAILED DESCRIPTION

The disclosure is addressed to provide an electronic apparatus configured to recognize a keyword through at least one keyword recognition mode executed based on operating state information of the electronic apparatus and a method for controlling thereof

FIG.1Ais a view illustrating a configuration and operations of an electronic apparatus according to an embodiment of the disclosure. Referring toFIG.1A, an electronic apparatus100may include a microphone110, a memory120and a processor130. However, the configuration illustrated inFIG.1Ais a view for implementing embodiments of the disclosure, and appropriate hardware and software configurations apparent to those skilled in the art may be additionally included in the electronic apparatus100.

The microphone110may receive user voice. In particular, the microphone110may receive user voice including a keyword requesting various operations. In addition, the microphone110may receive a user query requesting specific information.

The microphone110may be provided inside the electronic apparatus100, but may be provided outside and electrically connected to the electronic apparatus100. In addition, when the microphone110is provided outside, the microphone110may transmit a user voice signal generated through a wired/wireless interface (e.g., Wi-Fi, Bluetooth) to the processor130.

The memory120may store instructions or data related to at least one other component of the electronic apparatus100. In particular, the memory120may be implemented as non-volatile memory (e.g., dynamic random-access memory (RAM), static RAM), volatile memory (flash memory), programmable read-only memory (PROM), magneto resistive random-access memory (RRAM) and resistive RAM (RRAM), a hard disk drive (HDD), a solid state drive (SSD), and the like.

Volatile memory refers to a memory that requires constant power supply to maintain stored information. The volatile memory may be implemented as a separate component from the processor130that can be linked to the processor130, but this is only an embodiment, and the volatile memory may be implemented as form included in the processor130as a component of a processor130. In addition, the nonvolatile memory refers to a memory capable of maintaining stored information even when power supply is ceased.

The memory120may be accessed by the processor, and perform readout, recording, correction, deletion, update, and the like, on data by the processor130. According to an embodiment of the disclosure, the term of the memory may include the memory120, read-only memory (ROM) (not illustrated) and random access memory (RAM) (not illustrated) within the processor130, and a memory card (not illustrated) attached to the electronic apparatus100(e.g., micro secure digital (SD) card or memory stick). Further, the memory120may store programs, data, and so on to constitute various screens to be displayed on the display area of the display210.

In particular, the memory120may store the keyword recognition model10and a conversation system20. In the following description of the disclosure, the keyword recognition model10and the conversation system20will be described on the premise that they are separate structures, but this is only for clearly describing the operation of the electronic apparatus100according to the disclosure. The keyword recognition model10may be implemented in a form included as one component of the conversation system20.

The memory120may store a program for executing the keyword recognition model10and the conversation system20. In addition, the memory120may store a program for acquiring conversation history information related to the conversation system20. In addition, the memory120may store a conversation history while the conversation system20is running, and may delete previously stored conversation history information when the execution of the conversation system20is ceased.

The keyword recognition model10may be stored in a non-volatile memory and loaded into the volatile memory under the control of the processor130. The process of loading the keyword recognition model10from the non-volatile memory to the volatile memory by the processor130will be described below.

The keyword recognition model10determines whether a signal pattern of user voice input through the microphone110is similar to or matches the signal pattern corresponding to a specific keyword, and determines whether the specific keyword is included in the input user voice. When the keyword recognition model10recognizes that the specific keyword is included in the input user voice, the electronic apparatus100may be controlled to perform an operation corresponding to the specific keyword. The keyword recognition model10may include a plurality of keyword recognition models corresponding to each of a plurality of keywords. One keyword recognition model may be configured to recognize the plurality of keywords. In other words, as illustrated inFIG.1A, the keyword recognition model10may include first and second keyword recognition models10-1and10-2. However, this is only an example, and the keyword recognition model10may include one or more than three keyword recognition models, and may be added, deleted, or changed by a user command. The configuration and operation of the keyword recognition model will be described in detail with reference toFIG.1B.

A keyword is at least one word for executing a specific function, keywords may be expressed as a trigger (trigger) word, a wake-up (wake-up) or a function word. Also, the keyword may be pre-registered when the electronic apparatus100is manufactured, and may be added/modified/deleted by the user. An embodiment in which keywords are added/modified/deleted by the user will be described in detail with reference toFIG.5.

The conversation system20may include an artificial intelligence model that provides a response by performing voice recognition and language analysis with respect to user voice input through the microphone110. The conversation system20may include various software modules, and each software module may be controlled by the processor130. Description of each software module will be described with reference toFIG.6.

The memory120may store an operating state database (not shown). The operating state database may include information on the keyword recognition model to be executed according to the operating state of each electronic apparatus. For example, when a state of the electronic apparatus is a first operating state, the operating state database may include information that the first keyword recognition model among the plurality of keyword recognition models is executed by the processor130. The operating state database may be constructed in a table form of a relational database, as shown in Table 1 below, but this is only an example and may be constructed in various forms. However, states described in the Table 1 and models corresponding to the states are only one embodiment for describing the operating state database and may be implemented in various ways. In addition, the operating state database may be modified/added/deleted by a user command.

TABLE 1Type of operating stateKeyword recognition model to be executedFirst operating stateFirst keyword recognition modelSecond operating stateSecond keyword recognition modelThird operating stateKeyword recognition model is not executed

The processor130may be electrically connected to the memory120to control an overall operation of the electronic apparatus100. In particular, the processor130may selectively execute at least one keyword recognition model among a plurality of keyword recognition models based on operating state information of the electronic apparatus100. Meanwhile, the operation of the processor130executed by at least one keyword recognition model may include an operation that the keyword recognition model stored in the non-volatile memory is loaded as volatile memory (e.g., dynamic random access memory (DRAM), static RAM (SRAM), and the like under the control of the processor130.

Specifically, the processor130may identify an operating state of the electronic apparatus100according to whether the conversation system20stored in the memory120is deactivated. For example, when the conversation system20is deactivated, the processor130may identify that the electronic apparatus100is operating in the first operating state. As another example, when the conversation system20is activated to process text corresponding to the user voice, the processor130may identify that the electronic apparatus100is operating in the second operating state. As another example, when the conversation system20is activated and recognizes the user voice through the ASR module of the conversation system, the processor130may identify that the electronic apparatus100operates in the third operating state.

The processor130may identify the keyword recognition model to be executed according to the operating state of the electronic apparatus100identified through the operating state database stored in the memory120, and load the identified keyword recognition model as a volatile memory in a nonvolatile memory. In other words, the processor130may execute a keyword recognition model corresponding to the identified operating state of the electronic apparatus100. For example, when the electronic apparatus100is identified to be operated in the first operating state, the processor130may identify that the keyword recognition model corresponding to the first operating state among the plurality of keyword recognition models as a first keyword recognition model10-1through the operating state database stored in the memory120. The processor130may execute the identified first keyword recognition model10-1. As another example, if the electronic apparatus100is identified to be operated in the second operating state, the processor130may identify that the keyword recognition model corresponding to the second operating state among the plurality of keyword recognition models as the second keyword recognition model10-2through the operating state database. The processor130may execute the identified second keyword recognition model10-2. As another example, when the electronic apparatus100is identified to be operated in the third operating state, the processor130may identify that there is no keyword recognition model to be executed through the operating state database. Thus, the processor130may not load the keyword recognition model stored in the non-volatile memory, and obtain text corresponding to the user voice through an ASR module20-1.

When the conversation system20is deactivated, a case in which a plurality of software modules included in the conversation system20are not operated may be included. In addition, when the activated conversation system20processes the text corresponding to the user voice, a case in which all modules except for the ASR module among a plurality of software modules included in the conversation system20are operated may be included.

When a first user voice is input through the microphone110, the processor130may identify whether at least one keyword corresponding to the keyword recognition model executed using the executed keyword recognition model is included in the first user voice. The processor130may perform an operation of the electronic apparatus100corresponding to the at least one keyword based on at least one keyword identified as being included in the first user voice.

Specifically, if it is identified that at least one keyword including the meaning of requesting a predefined operation is included in the first user voice by using the first keyword recognition model executed while the electronic apparatus100is operating in the first operating state, the processor130may perform the predefined operation. The predefined operation may be added or deleted by the user, which will be described in detail with reference toFIG.5.

If it is identified that the keyword including the meaning of requesting a next or previous operation is included in the first user voice by using the first keyword recognition model10-1executed while the electronic apparatus100is operating in the first operating state, the processor130may identify the operation performed by the electronic apparatus by the conversation system20before identifying whether the first keyword recognition model10-1is included in the first user voice based on the conversation history information stored in the conversation history storage. The processor130may control to perform an operation corresponding to the next or previous operation of the operation performed by the identified electronic apparatus100. For example, when it is identified that the keyword including the meaning of requesting the next operation is included in the first user voice by using the first keyword recognition model10-1, the processor130may identify that the electronic apparatus100is operating a music application to listen to music through the conversation system20before identifying the first keyword recognition model10-1is included in the first user voice. In addition, the processor130may control to play the next song on the currently playing song as an operation corresponding to the next operation of the identified operation.

If it is identified that the keyword including the meaning of stopping the operation is included in the first user voice by using the second keyword recognition model10-2executed while the electronic apparatus100is operating in the second operating state, the processor130may control to stop processing of the text corresponding to the second user voice of the conversation system20. For example, while a response sentence for the second user voice is output in a form of voice by the conversation system20, the keyword including the meaning of stopping the operation using the second keyword recognition model10-2is identified to be included in the first user voice, the processor130may control a TTS module of the conversation system20to stop the operation of outputting the response sentence in the form of voice.

If it is identified that the keyword including a meaning of repeating the operation using the second keyword recognition model10-2executed while the electronic apparatus100is operating in the second operating state is included in the first user voice, the processor130may stop processing the text corresponding to the second user voice performed by the conversation system20, and control to repeat the processing of the text of the conversation system20based on the conversation history information. For example, while the conversation system20is outputting a response sentence for user voice requesting information about today's weather in the form of voice, if it is identified that the keyword (e.g., “again”) including the meaning of repeating the operation is included in the first user voice by using the second keyword recognition model10-2, the processor130may stop the TTS module of the conversation system20from outputting the response sentence in the form of voice. The processor130may identify information on the response sentence for today's weather through the conversation history information, and control the module to output the response sentence for today's weather again through the conversation system20by using the identified information.

If it is not identified that the keyword is included in the first user voice by using the second keyword recognition model10-2executed while the electronic apparatus100is operating in the second operating state, the processor130may input the first user voice including the keyword into the ASR module20-1of the conversation system20to obtain text corresponding to the first user voice. In other words, if it is determined that the keyword is not included in the first user voice using the keyword recognition model, the processor130may input the first user voice to the conversation system20to perform natural language processing on the user voice. In other words, when the keyword recognition model does not recognize the keyword included in the user voice, the processor130may enter the user voice into the conversation system to obtain a response to the user voice, thereby allowing a more natural conversation with the user.

AI-related functions, according to the disclosure, operate through a processor and a memory. The processor may be composed of one or a plurality of processors. In this case, the one or the plurality of processors may be a general-purpose processor such as a CPU, an AP, or a digital signal processor (DSP), a graphic processor such as a GPU or a vision processing unit (VPU), or an AI-only processor such as an NPU. The one or the plurality of processors may control to process input data according to predefined operation rules or artificial intelligence models stored in the memory. Alternatively, when one or more processors are AI-only processors, the AI-only processors may be designed with a hardware structure specialized for processing a specific AI model.

The predefined operation rules or artificial intelligence models are characterized by being generated through learning. By being generated through learning, the basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, thereby generating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). Such learning may be performed on the apparatus on which artificial intelligence according to the disclosure is performed, or may be performed through a separate server and/or system. There are examples of the learning algorithm such as a supervised learning, unsupervised learning semi-supervised learning, or reinforcement learning, but not limited thereto.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values and performs a neural network calculation through calculation between a result of calculation of a previous layer and a plurality of weight values. The plurality of weight values of the plurality of neural network layers may be optimized by a learning result of the artificial intelligence model. For example, a plurality of weight values may be updated such that a loss value or a cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include deep neural network (DNN), for example, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN) or Deep Q-Networks, but are not limited thereto.

FIG.1Bis a flowchart illustrating a configuration operations of a keyword recognition model according to an embodiment of the disclosure.

Referring toFIG.1B, each of at least one keyword recognition model may include a keyword database50for recognizing keywords. However, this is only an embodiment, and the keyword recognition model10may include the keyword database50common to all of at least one keyword recognition model.

The keyword database50may store signal patterns corresponding to keywords and information on operations corresponding to each keyword (e.g., commands for performing operations corresponding to keywords). Information on the operations corresponding to the keywords may be constructed by storing in a table form of a relational database, as shown in Table 2 below. However, this is only an example, and information on operations corresponding to keywords may be constructed in various forms.

TABLE 2Type ofCommands for performing operationskeywordcorresponding to keywordsCancelStop_Processing or Stop_SpeakingStopStop_SpeakingNextNext_ActionPreviousPrevious_Action

The types of keywords and commands for performing operations corresponding to the keywords shown in Table 2 are only an embodiment for specifically describing the disclosure, and may be variously implemented by the type of the electronic apparatus100, and may be modified/added/deleted by the user. Meanwhile, information on signal patterns corresponding to keywords stored in the keyword database50and information on operations corresponding to each keyword may be added or deleted by the user. An embodiment in which keywords are added/deleted by the user will be described in detail with reference toFIG.5.

When user voice40is input, the keyword recognition model30may identify whether a keyword corresponding to the keyword recognition model30is included in the user voice40. As an embodiment, the keyword recognition model30may identify whether a keyword is included in the user voice40by utilizing a keyword spotting technology. Specifically, when the user voice40is input, the keyword recognition model30may extract a signal pattern of the user voice40. The keyword recognition model30may output a similarity value between signal patterns by comparing and matching the signal pattern of the extracted user voice40and the signal pattern corresponding to the keyword. In addition, if there is a portion (or a section) in which the similarity value between signal patterns corresponding to keywords in a signal pattern section of the user voice40exceeds a threshold value, the processor130may identify that the keyword corresponding to the keyword recognition model30is included in the user voice40. If there is no portion (or section) in which the similarity value between the signal patterns corresponding to the keywords in the signal pattern section of the user voice40exceeds the threshold value, the processor130may identify that the keyword corresponding to the keyword recognition model30is not included in the keyword recognition model30. Meanwhile, the threshold value may be a preset value through experiment results, etc., and may be changed by a user command

As an embodiment, if it is identified that the keyword corresponding to the keyword recognition model30is not included in the user voice40, the processor130may input the user voice40into the conversation system20. As another embodiment, the processor130may control the display140to display a message that the keyword is not included in the user voice40or control a speaker150to output the message in the form of voice.

In addition, the keyword recognition model30may identify information on the operation corresponding to the keyword included in the user voice40through the keyword database50and output an action signal60based on the identified information. Accordingly, the processor130may control to perform the operation of the electronic apparatus100based on the action signal60. For example, the second keyword recognition model executed while the electronic apparatus100is operating in the second operating state may recognize a keyword “stop”. In addition, the second keyword recognition model may identify that a command for performing an operation corresponding to the keyword “stop” is “Stop_Speaking” through the keyword database50. Also, the second keyword recognition model may output an action signal including the identified command. Accordingly, the processor130may control to stop processing of text corresponding to the user voice executed through the conversation system based on the action signal obtained through the second keyword recognition model.

As another example, the first keyword recognition model executed while the electronic apparatus100is operating in the first operating state may recognize a keyword “next”. Then, the first keyword recognition model may identify that a command for performing an operation corresponding to the keyword “next” is “Next_Action” through the keyword database50, and output the action signal60including the identified command. Accordingly, based on the obtained action signal60and conversation history information, the processor130may control to perform an operation corresponding to a next operation of the operation performed by the electronic apparatus by the conversation system20before the first keyword recognition model recognizes the keyword.

If the electronic apparatus100does not recognize the keyword using the second keyword recognition model executed while the electronic apparatus100is operation in the second operating state, the processor130may input the first user voice including the keyword into the ASR module of the conversation system20to obtain the text corresponding to the first user voice, and obtain a response70to the user voice based on the text. In other words, when the keyword is not recognized using the keyword recognition model, the processor130may input the user voice including the keyword into the conversation system20to obtain the response to the user voice, thereby providing a more natural response to the user.

FIG.2is a flowchart illustrating a method of controlling an electronic apparatus according to an embodiment of the disclosure.

Referring toFIG.2, the electronic apparatus100may selectively execute at least one keyword recognition model for recognizing user voice among the plurality of keyword recognition models based on operating state information of the electronic apparatus100(S210). For example, when the electronic apparatus100operates in the first operating state in which the conversation system is deactivated, the electronic apparatus100may execute the first keyword recognition model corresponding to the first state among the plurality of keyword recognition models. As another example, when the electronic apparatus100operates in the second operating state in which the conversation system is activated and processes text corresponding to the inputted second user voice, the electronic apparatus100may execute the second keyword recognition model corresponding to the second state among the plurality of keyword recognition models. As another example, when the electronic apparatus100operates in a third operating state that recognizes the second user voice inputted by the activated conversation system through the ASR module of the conversation system, the electronic apparatus100may stop the operation of the plurality of keyword recognition models and obtain texts corresponding to the second user voice through the ASR module.

When the first user voice is input through the microphone110, the electronic apparatus100may identify whether at least one keyword corresponding to the keyword recognition model executed by using the executed keyword recognition model is included in the first user voice (S220). The electronic apparatus100may perform an operation corresponding to the at least one keyword based on at least one keyword identified as being included in the first user voice (S230).

In an embodiment, when it is identified that a keyword including a meaning of requesting a predefined operation is included in the first user voice by using the first keyword recognition model executed while the electronic apparatus is operating in the first operating state, the electronic apparatus100may perform the predefined operation, which will be described in detail with reference toFIG.3A.

In another embodiment, when it is identified that a keyword including a meaning of requesting a next or previous operation by using the first keyword recognition model executed while the electronic apparatus is operating in the first operating state, the electronic apparatus100may identify an operation performed by the electronic apparatus by the conversation system before the first keyword recognition model recognizes a keyword included in the first user voice based on conversation history information. The electronic apparatus100may perform an operation corresponding to the next or previous operation of the operation performed by the identified conversation system. The corresponding embodiment will be described in detail with reference toFIG.3B.

In another embodiment, when it is identified that a keyword including a meaning of stopping or repeating the operation is included in the first user voice using the second keyword recognition model executed while the electronic apparatus is operating in the second operating state, the electronic apparatus100may stop processing of text corresponding to the second user voice by the conversation system or perform the process again from the beginning. This embodiment will be described in detail with reference toFIGS.4A and4C.

In another embodiment, if it is not identified that the keyword is not included in the first user voice using the keyword recognition model executed while operating in the second operating state, the electronic apparatus100may input the user voice into the conversation system to perform natural language processing for the user voice. This embodiment will be described in detail with reference toFIG.4B.

FIGS.3A and3Bare views illustrating an operation of an electronic apparatus when a first keyword recognition model is executed according to various embodiments of the disclosure.

FIGS.3A and3Bare views illustrating a process of identifying that the electronic apparatus100uses the first keyword recognition model to identify a keyword corresponding to the keyword recognition model is included in the first user voice, and performing an operation based on the identified keyword.

When the electronic apparatus100operates in the first operating state in which the conversation system is deactivated, the electronic apparatus100may execute the first keyword recognition model among the plurality of keyword recognition models. If it is identified that the first user voice includes the keyword including the meaning of requesting the predefined operation using the first keyword recognition model, the electronic apparatus100may perform the predefined operation.

For example, a signal pattern matching the keyword “weather” and information in which an operation corresponding to the keyword “weather” executes a weather application and outputs information on today's weather in the form of voice may be pre stored in the keyword database included in the first keyword recognition model. The operation corresponding to the keyword “weather” pre-stored in the electronic apparatus100may be changed by the user.

Referring toFIG.3A, when user voice300including the keyword “weather”, which is one of pre-stored keywords, is input, the electronic apparatus100may compare and match the signal pattern of the user voice300and the signal pattern corresponding to the pre-stored keyword by using the executed first keyword recognition model to obtain a similarity value between the signal patterns. If it is identified that there is a section in which the similarity value to the signal pattern matching the “weather” among the signal pattern sections corresponding to the user voice300exceeds a threshold value, the electronic apparatus100may identify the keyword “weather” is included in the user voice300.

In addition, the first recognition keyword model may output an action signal including a command to execute a weather application based on the keyword database and perform an operation of outputting information about today's weather in the form of voice. Accordingly, referring toFIG.3B, the electronic apparatus100may display a screen310-1for executing the weather application based on the action signal output by the first recognition keyword model, and output the information about today's weather in the form of voice310-2.

In one embodiment of the disclosure, when it is identified that the user voice includes a keyword including the meaning of requesting the next or previous operation using the first keyword recognition model executed while operating in the first operating state, the electronic apparatus100may identify an operation performed by the conversation system before the first keyword recognition model recognizes the keyword based on the conversation history information.

Referring toFIG.3B, when user voice320requesting playing music is input from the user, the electronic apparatus100may obtain a response to the user voice320through a conversation system. The electronic apparatus100may execute an application that plays music in response to the user voice320obtained through the conversation system, and play music while displaying an application execution screen330. Also, since the conversation system stored in the electronic apparatus100is inactive, the electronic apparatus100operates in the first operating state, such that the electronic apparatus100may execute the first keyword recognition model. At this time, the conversation history information stored in the electronic apparatus100may include a type of an application executed in response to the user voice320and information on an operation performed through the application.

Then, when a keyword340(e.g., “next”) including a meaning of requesting a next operation is input from the user, the electronic apparatus100may identify the user voice input through the executed first keyword recognition model includes the keyword “next”. Then, the electronic apparatus100may identify that the electronic apparatus100is currently playing music by executing a music application by the conversation system before the keyword340is input through the stored conversation history information. In addition, while identifying an operation performed by the electronic apparatus based on the conversation history information, the electronic apparatus100may obtain an action signal including a command to perform the next operation through the first keyword recognition model. In addition, the electronic apparatus100may play the next song in the application currently being executed as an operation corresponding to the next operation in accordance with the identified operation based on the obtained action signal, and display a play screen350of the next song.

FIG.4Ais a view illustrating that the electronic apparatus identifies that user voice includes a keyword through the second keyword recognition model, and perform an operation based on the identified keyword according to an embodiment of the disclosure.

FIG.4Aillustrates a process in which the electronic apparatus100identifies that a keyword is included in a user's voice through a second keyword recognition model and performs an operation based on the identified keyword, according to an embodiment of the disclosure It is a drawing for doing.

Referring toFIG.4A, the electronic apparatus100may execute the second keyword recognition model among a plurality of keyword recognition models in a second operating state that processes text corresponding to the user voice input by an activation of a conversation system. For example, when user voice requesting information about weather is input, the electronic apparatus100may execute a weather application in response to the user voice obtained through the conversation system, and output information about today's weather400in the form of voice. Since a response sentence for the user voice requesting weather information is output in the form of voice through the TTS module of the conversation system, the electronic apparatus100may execute the second keyword recognition model corresponding to the second operating state among the plurality of keyword models.

In addition, if it is identified that a keyword (e.g., “stop”) including a meaning of stopping the operation is included in user voice410through the second keyword recognition model executed while operating in the second operating state, the electronic apparatus100may stop processing the text corresponding to the second user voice by the conversation system. For example, while outputting a response sentence for the user voice requesting weather information in the form of voice, the electronic apparatus100may identify that the user voice includes a keyword including a meaning to stop the operation through the second keyword recognition model. Then, the electronic apparatus100may obtain an action signal including a command to stop the operation of the conversation system through the second keyword recognition model. The electronic apparatus100may stop process of text performed by the conversation system (e.g., an operation in which the conversation system outputs a response sentence for a user voice requesting weather through the TTS module in the form of voice form) based on the obtained action signal. Accordingly, as illustrated inFIG.4A, when the user voice including the keyword to stop an operation is input while outputting the response sentence400for the user voice in the form of voice through the conversation system, the electronic apparatus100may stop the operation of the conversation system.

If it is not identified that the keyword including the meaning of stopping the operation is included in the user voice410through the second keyword recognition model while outputting the response sentence to the user voice in the form of voice through the conversation system, the electronic apparatus100may input the user voice410into the conversation system to obtain text corresponding to the user voice410. In one embodiment, due to noise in the surrounding environment, temporary change in user voice, and the like, the electronic apparatus100may use the second keyword recognition model to obtain similarity values between signal patterns corresponding to signal patterns corresponding to the user voice410below a threshold value. As another example, if user voice including a keyword different from the pre-stored keyword is input to the electronic apparatus100due to the user's error, the electronic apparatus100may obtain a similarity value below a threshold value through the second keyword recognition model. In this case, the electronic apparatus100may obtain a text corresponding to the user voice by inputting the voice input from the user into the conversation system. Also, the electronic apparatus100may obtain a response to the user voice based on the obtained text.

FIG.4Bis a view illustrating an operation of an electronic apparatus when a second keyword recognition model is executed according to an various embodiments of the disclosure.

Referring toFIG.4B, if it is not identified that the user voice410includes the keyword including the meaning of stopping the operation while the response sentence for the user voice requesting the weather is output in the form of voice, the electronic apparatus100may input the user voice410into the conversation system. The electronic apparatus100may output voice “Do you want me to stop talking?”420in response to the user voice410. As another example, the electronic apparatus100may stop an operation (e.g., execution of the weather application) performed by the conversation system based on the conversation history information. In other words, even if the keyword is not recognized through the second keyword recognition model, the electronic apparatus100may perform a natural conversation with the user by obtaining a response by inputting the user voice including the keyword into the conversation system.

When it is identified that the user voice includes a keyword including the meaning of repeating the operation by using the second keyword recognition model executed while operating in the second operating state, the electronic apparatus100may stop processing text corresponding to the user voice that is performed by the conversation system, and perform processing text based on the conversation history information again.

FIG.4Cis a view illustrating an operation of an electronic apparatus when a second keyword recognition model is executed according to an various embodiments of the disclosure.

Referring toFIG.4C, when a response sentence430for the user voice requesting exchange rate information is output in the form of voice, the electronic apparatus100may perform the second keyword recognition model corresponding to the second operating state among the plurality of keyword recognition models. In addition, if it is identified that the user voice includes the keyword including the meaning of repeating the operation (e.g., “again”)440, the electronic apparatus100may stop processing text corresponding to the user voice performed by the conversation system. (e.g., the operation of outputting the response sentence430for the user voice in which the conversation system requests exchange rate information through the TTS module in the form of voice).

The electronic apparatus100may perform text processing corresponding to the user voice again based on the conversation history information. For example, the electronic apparatus100may identify a response sentence for the user voice requesting exchange rate information through the conversation history information, and output the response sentence450identified through the conversation system in the form of voice.

FIG.5is a view illustrating a process that an electronic apparatus registers keywords according to an embodiment of the disclosure. In one embodiment, when a user command for registering a keyword is input, the electronic apparatus100may display a UI500capable of registering the keyword.

Referring toFIG.5, section (a), the UI500capable of registering keywords may include a message for the user to utter a keyword that the user wants to register, but this is only an embodiment, and the UI500may be variously implemented according to a type of the electronic apparatus100or a user command

When voice corresponding to the keyword (e.g., “keyword”)510is input from the user, the electronic apparatus100may obtain and store a signal pattern corresponding to the input voice. In one embodiment, the electronic apparatus100may store a signal pattern corresponding to voice on the keyword database. Meanwhile, the electronic apparatus100may display a message requesting utterance of the keyword a predetermined number of times for an accuracy of the signal pattern corresponding to the keyword.

In addition, referring toFIG.5, section (b), when a signal pattern corresponding to the keyword is obtained from the user, the electronic apparatus100may display a UI520requesting information about an operation to be performed when the keyword is spoken. When user voice (e.g., “Run exchange rate application”)510including information on an operation to be performed when a keyword is spoken is input from the user, the electronic apparatus100may obtain a response (e.g., an operation of executing the exchange rate application) to the user voice510through the conversation system, and identify the obtained information as the operation corresponding to the keyword. Also, the electronic apparatus100may match and store information about a signal pattern corresponding to the keyword and an operation corresponding to the identified keyword. As described with reference toFIG.1B, the electronic apparatus100may construct and store information and keywords with respect to the operation corresponding to the identified keyword in the form of a table in a relational database, but is only embodiment. The electronic apparatus100may receive information about the application to be executed when speaking the keyword by the user, and information about the content to be played through not only voice but also a variety of input means. For example, when a user touch with respect the content to be executed or an icon of the application when uttering a keyword is input, the electronic apparatus100may match and store an operation of executing the input content or application with a signal pattern corresponding to the keyword.

FIG.6is a block diagram illustrating a configuration of an electronic apparatus in detail according to an embodiment of the disclosure. Referring toFIG.6, the electronic apparatus100may include a microphone110, a memory120, a processor130, a display140, a speaker150, an inputter160, and a communicator170. However, since the microphone110, the memory120, and the processor130have been described in detail with reference toFIG.1, redundant description will be omitted.

The conversation system20may be stored in the memory120. In addition, the conversation system20may include a plurality of software modules that can be controlled by the processor130as shown inFIG.6. When a trigger voice for activating the conversation system20is input by the user through the microphone110or a user command is input through an inputter (e.g., an AI agent button, etc.) capable of activating the conversation system, the ASR module20-1of the conversation system20may be activated.

The automatic speech recognition (ASR) module20-1may perform voice recognition for the user voice input through the microphone110and output text corresponding to the recognized voice. For example, if the user voice “How is the weather today” is input, the ASR module20-1may recognize the user voice and output text “How is the weather today” corresponding to the recognized voice.

The natural language understanding (NLU) module20-2may determine user's intention an parameters by using a matching rule divided in to a parameter (or slot) required to identify a domain of the user voice and intention based on a result of voice recognition obtained through the ASR module20-1. Specifically, one domain (e.g., an alarm) may include a plurality of intentions (e.g., alarm setting, alarm cancellation), and one intention may include a plurality of parameters (e.g., time, number of repetitions, notification sound, etc.). The matching rule may be stored in the NLU database (not shown). In addition, the NLU module20-2may identify the meaning of words extracted from the user input using linguistic features such as morphemes and phrases (e.g., grammatical elements), and match the meaning of the identified word to the domain and intention to determine the user's intention.

For example, when the text corresponding to the user voice obtained through the ASR module20-1is “How is the weather today”, the NLU module20-2may identify the meaning of “weather today”, “how”, or the like, and classify the domain of the text as “weather” to obtain user's intention requesting information about todays' weather.

A dialogue management (DM) module20-3may identify whether the user's intention identified by the NLU module20-2is clear. For example, the DM module20-3may identify whether the user's intention is clear based on whether the parameter information is sufficient.

Also, the DM module20-3may generate a result of performing a task corresponding to a user input based on the intention and parameters identified in the NLU module20-2. In other words, the DM module20-3may obtain information for generating response sentences based on the intention of the user voice, data stored in a knowledge base (KB)20-6, or information received by an external server identified through the NLU module20-2. Information for generating a response sentence may be obtained based on the obtained information. For example, in the NLU module20-2, when the user obtains the intention to request information about “weather today”, the DM module20-3may obtain response information including information related to the weather today based on the data stored in the knowledge base20-6.

A natural language generator (NLG) module20-4outputs a response sentence to the user voice in a form of a text, which is the form of natural language utterance, based on response information obtained through the DM module20-3and the conversation history information stored in a history storage20-7.

A text to speech (TTS) module20-5may convert the generated response sentence into voice. In particular, the TTS module20-5may convert text obtained through the NLG module20-4into voice. Thereby, the conversation system20may provide a response sentence for the user voice as voice.

The knowledge base20-6may store information for personalized responses. In particular, information stored in the knowledge base20-6may vary. For example, the knowledge base20-6may store information related to a user command input to the electronic apparatus100, information about an application or content included in the electronic apparatus100, and the like.

The conversation history storage20-7may store conversation history information. The conversation history information is related to the user voice input through the microphone110, and may include information about a voice recognition result obtained before the user voice is input, a language analysis result, and a response output by the conversation system20. Can. In addition, the conversation history information may store information on a task performed by the electronic apparatus100before the user's voice is input.

The conversation history storage20-7may store conversation history information. The conversation history information may be related to user voice input through the microphone110, and may include information on a result of voice recognition obtained before the user voice is input, information on a result of language analysis, and information on responses outputted by the conversation system20. Also, the conversation history information may store information on the work that has been performed by the electronic apparatus100before the user voice is input.

The display140may display various information under the control of the processor130. In particular, the display140may display a UI registering a keyword or a UI requesting information about an operation to be performed when the keyword is spoken. In addition, the display140may display a message that each keyword recognition model does not recognize the keyword.

In addition, the display140may be implemented as a touch screen together with a touch panel, but this is only an example and may be variously implemented according to the type of the electronic apparatus100.

The speaker150may include various audio output circuitry and is configured to output various kinds of alarm sounds or voice messages in addition to various audio data on which various processing operations such as decoding, amplification, and noise filtering are performed by an audio processor. In particular, the speaker150may output a response to the user voice obtained through the conversation system as a voice message in the form of natural language. Specifically, a configuration for outputting audio may be implemented as a speaker, but is only an embodiment, and may be implemented as an output terminal capable of outputting audio data.

The inputter160may receive a user input for controlling the electronic apparatus100. In particular, the inputter160may include a touch panel for receiving a user touch using a user's hand or a stylus pen, etc., and a button for receiving a user manipulation, or the like. In addition, the inputter160may be implemented with other input devices (e.g., keyboard, mouse, motion inputter, etc.). In particular, the inputter160may receive information on an operation corresponding to a keyword from a user.

The communicator170may communicate with an external device. In this case, the communicator170may include communicating through a third device (e.g., a repeater, a hub, an access point, a server or a gateway, etc.) which is communicatively connected to the external device. For example, the communicator170may receive information related to contents or applications from the external server.

The communicator170may include various communication modules to perform communication with the external device. For example, the communicator170may include a wireless communication module such as a cellular communication module using at least one among long-term evolution (LTE), LTE Advance (LTE-A), 5th Generation (5G), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), and global system for mobile communications (GSM). According to an another embodiment, wireless communication may include at least one of, for example, wireless fidelity (Wi-Fi), Bluetooth, Bluetooth low energy (BLE), Zigbee, near field communication (NFC), magnetic secure transmission, radio frequency (RF), or body area network BAN.

As described above, according to various embodiments of the disclosure, the electronic apparatus may recognize a keyword through at least one keyword recognition model executed based on state information of the electronic apparatus to determine an operation, thereby enabling the user to more efficiently use the conversation system.

However, it should be understood that the disclosure is not limited to the specific embodiments described hereinafter, but includes various modifications, equivalents, and/or alternatives of the embodiments of the disclosure. In relation to explanation of the drawings, similar drawing reference numerals may be used for similar constituent elements.

In the disclosure, the terms “include”, “may include”, “comprise”, and “may comprise” designate the presence of features (e.g., elements, such as numbers, functions, operations, or components) that are written in the specification, but do not exclude the presence or possibility of addition of one or more other features.

In the description, the term “A or B”, “at least one of A or/and B”, or “one or more of A or/and B” may include all possible combinations of the items that are enumerated together. For example, the term “A or B” or “at least one of A or/and B” may designate (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

The expression “1”, “2”, “first”, or “second” as used herein may modify a variety of elements, irrespective of order and/or importance thereof, and only to distinguish one element from another. Accordingly, without limiting the corresponding elements.

When an element (e.g., a first element) is “operatively or communicatively coupled with/to” or “connected to” another element (e.g., a second element), an element may be directly coupled with another element or may be coupled through the other element (e.g., a third element). On the other hand, when an element (e.g., a first element) is “directly coupled with/to” or “directly connected to” another element (e.g., a second element), an element may not be existed between the other element.

In the description, the term “configured to” may be changed to, for example, “suitable for”, “having the capacity to”, “designed to”, “adapted to”, “made to”, or “capable of” under certain circumstances. The term “configured to (set to)” does not necessarily mean “specifically designed to” in a hardware level. Under certain circumstances, the term “device configured to” may refer to “device capable of” doing something together with another device or components. For example, “a sub-processor configured (or configured to) perform A, B, and C” may refer to a generic-purpose processor (e.g., central processing unit (CPU) or an application processor) capable of performing corresponding operations by executing a dedicated processor (e.g., an embedded processor) or one or more software programs stored in a memory device to perform the operations.

An electronic apparatus according to various embodiments may include at least one of, for example, smartphone, tablet personal computer (PC), mobile phone, video phone, e-book reader, desktop PC, laptop PC, netbook computer, server, personal digital assistant (PDA), medical device, or a wearable device. In some embodiments, the electronic apparatus may include at least one of a television, a refrigerator, an air conditioner, an air purifier, a set top box, a media box (e.g., Samsung HomeSync™, Apple TV™, or Google TV™).

In an embodiment of the disclosure, the term “a user” may indicate a person using an electronic apparatus or an apparatus which uses an electronic apparatus (e.g., artificial intelligent electronic apparatus). Hereinafter, this will be greater described with reference to the drawings.

According to an embodiment of the disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage media which is readable by a machine (e.g., a computer). The device may include the electronic device according to the disclosed embodiments, as a device which calls the stored instructions from the storage media and which is operable according to the called instructions. When the instructions are executed by a processor, the processor may directory perform functions corresponding to the instructions using other components or the functions may be performed under a control of the processor. The instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage media may be provided in a form of a non-transitory storage media. The ‘non-transitory’ means that the storage media does not include a signal and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily in the storage media. For example, the ‘non-transitory’ may include a buffer that data is temporarily stored.

According to an embodiment of the disclosure, the methods according to various embodiments described above may be provided as a part of a computer program product. The computer program product may be traded between a seller and a buyer. The computer program product may be distributed in a form of the machine-readable storage media (e.g., compact disc read only memory (CD-ROM) or distributed online through an application store (e.g., PlayStore™). In a case of the online distribution, at least a portion of the computer program product may be at least temporarily stored or provisionally generated on the storage media, such as a manufacturer's server, the application store's server, or a memory in a relay server.

Further, each of the components (e.g., modules or programs) according to the various embodiments described above may include a single entity or a plurality of entities, and some subcomponents of the above-mentioned subcomponents may be omitted or the other subcomponents may be further included to the various embodiments. Generally, or additionally, some components (e.g., modules or programs) may be integrated into a single entity to perform the same or similar functions performed by each respective component prior to integration. Operations performed by a module, a program module, or other component, according to various embodiments, may be sequential, parallel, or both, executed iteratively or heuristically, or at least some operations may be performed in a different order, omitted, or other operations may be added.