Patent Description:
An artificial intelligence system is a computer system that implements human-level intelligence, and improves its awareness as a machine performs self-learning, determination, and uses itself.

Artificial intelligence technology may include machine learning (deep learning) technology which uses an algorithm that classifies/learns the features of input data by itself, and element technologies which mimic functions of a human brain, such as cognition, determination, and the like, using the machine learning technology.

The element technologies may include, for example, at least one from among linguistic understanding technology that recognizes human languages/characters, visual understanding technology that recognizes objects in the manner of human vision, inferring/predicting technology that performs logical inferring and prediction by estimating information, knowledge expression technology that processes human experience information to be knowledge data, and operation control technology that controls autonomous vehicle driving and robot movement.

The linguistic understanding technology is a technology that recognizes human languages/characters and applies/processes the same, and may include natural language processing, machine translation, conversation system, question and answer, speech recognition/synthesis and the like.

As an example of the linguistic understanding technology, as disclosed in <CIT>, <CIT> and <CIT>, respectively, an ASR module recognizes a user utterance using an ASR model, produced by training based on a machine learning technology, and may produce text data corresponding to the utterance.

In association with text data that an ASR module produces via speech recognition, a user can directly correct an erroneously recognized part. For example, an electronic device may display text data, produced via speech recognition, via a user interface (UI), and may receive a reaction thereto from a user. If the user reaction means that an erroneous recognition is present, the electronic device may display a UI via which the user is capable of inputting text for replacing the erroneous part. As described above, a method which receives feedback associated with an erroneous part from a user may require an additional operation from the user, and if the additional operation is not performed, a user utterance may be repeatedly erroneously recognized. The repetitive ASR error may cause an inconvenience for users when the users use voice commands.

According to various embodiments of the disclosure, the electronic device may reduce errors in recognition of utterance, so as to increase accuracy of voice-based electronic device control, and to reduce inconvenience for users.

In accordance with an aspect of the disclosure, an electronic device as claimed in claim <NUM> is provided.

In accordance with an aspect of the disclosure, a method of operating an electronic device as claimed in claim <NUM> is provided. Advantageous Effects of Invention.

According to various embodiments of the disclosure, an electronic device produces a personalized ASR model for a user by analyzing a user utterance, and performs speech recognition using the same, and thus, may reduce errors in utterance recognition, and may decrease an inconvenience for users.

The description of the embodiments illustrated in <FIG> and <FIG> is according to the claimed invention, whereas other parts of the description are for illustrative purposes only.

The input device <NUM> may receive commands or data to be used in an element (e.g., the processor <NUM>) of the electronic device <NUM> from the outside of the electronic device <NUM> (e.g., a user).

The antenna module <NUM> may transmit signals or power to the outside (e.g., an external electronic device), or may receive the same from the outside. According to an embodiment, the antenna module may be implemented by an electric conductor or a conductive pattern. According to an embodiment, the antenna module may further include another component (e.g., an RFIC) in addition to an electric conductor or a conductive pattern. According to an embodiment, the antenna module <NUM> may include one or more antennas. Among them, at least one antenna appropriate for a communication scheme that is used in a communication network, such as the first network <NUM> or the second network <NUM>, may be selected by, for example, the communication module <NUM>. Transmission or reception of signals or power may be performed between the communication module <NUM> and an external electronic device via the at least one selected antenna.

<FIG> is a diagram illustrating an integrated intelligent system according to various embodiments of the disclosure. Referring to <FIG>, an integrated intelligent system <NUM> may include a user equipment (UE) <NUM> (e.g., the electronic device <NUM> of <FIG>), an intelligent server <NUM> (e.g., the server <NUM> of <FIG>), a cloud server <NUM> or a third party server <NUM>.

According to various embodiments of the disclosure, the UE <NUM> may provide services that users need, via apps (or application programs) (e.g., an alarm app, a message app, a picture (gallery) app, and the like) stored in the UE <NUM>. For example, the UE <NUM> may perform another app via an intelligent app (or a speech recognition app) stored in the UE <NUM>. The UE <NUM> may receive a user input for executing the other app using the intelligent app. The user input may be received via, for example, a physical button, a touch pad, a voice input, a remote input, and the like. According to an embodiment, the UE <NUM> may correspond to a terminal device (or an electronic device) capable of connecting to the Internet, such as a portable phone, a smart phone, a personal digital assistant (PDA), a notebook computer, or the like.

According to various embodiments of the disclosure, the UE <NUM> may receive a user utterance as a user input. The UE <NUM> may receive a user utterance, and may produce a command to execute an app based on the user utterance. Accordingly, the UE <NUM> may execute the app using the command.

According to various embodiments of the disclosure, the UE <NUM> may produce speech data corresponding to the user utterance, and may transmit the speech data to the intelligent server <NUM>.

According to various embodiments of the disclosure, the UE <NUM> may pre-process the speech data before transmitting the speech data to the intelligent server <NUM>. For example, the UE <NUM> may perform a speech pre-processing operation, including an operation of removing echos included in the speech data, an operation of removing background noise included in the speech data, and an operation of adjusting the volume of speech included in the speech data, and may transmit the pre-processed speech data to the intelligent server <NUM>.

According to various embodiments of the disclosure, the intelligent server <NUM> may perform an operation of converting speech data received from the UE <NUM> via a communication network <NUM> (e.g., the network <NUM> of <FIG>) into text data. For example, the intelligent server <NUM> may perform a conversion-to-text operation using an acoustic model (AM) and a language model (LM) learned by the machine learning technology and stored in the intelligent server <NUM>.

According to various embodiments of the disclosure, the intelligent server <NUM> may transmit the converted text data to the UE <NUM> and/or cloud server <NUM> via the communication network <NUM>. The UE <NUM> may output the received text data on a display.

According to various embodiments of the disclosure, the cloud server <NUM> may perform natural language understanding (NLU) processing on the text data received from the intelligent server <NUM>. The NLU processing may be an operation of recognizing the intention of user utterance based on the text data. For example, the cloud server <NUM> may perform syntactic analysis (syntactic analyze) or semantic analysis (semantic analyze) on the text data, so as to recognize a user intention.

According to various embodiments of the disclosure, the cloud server <NUM> may identify a user intention determined via NLU, may identify an operation appropriate for the user intention, and may identify information associated with parameters needed for performing the operation corresponding to the user intention. If a parameter needed for performing the operation corresponding to the user intention is insufficient, the cloud server <NUM> may request a parameter from the UE <NUM>.

According to various embodiments of the disclosure, the cloud server <NUM> may transmit information associated with the operation, the parameter, and an application that is to perform the operation, to the UE <NUM>. The UE <NUM> may select an application based on the information, and may run the selected application so as to perform an operation according to the user input (command).

According to various embodiments of the disclosure, the cloud server <NUM> may perform the operation corresponding to the user intention using the parameter, and may transmit the result thereof to the UE <NUM>.

According to various embodiments of the disclosure, the cloud server <NUM> may transmit, to the third party server <NUM>, a signal that requests execution of the operation corresponding to the user intention. The third party server <NUM> may be a server that is operated by a subject different from a subject that operates the intelligent server <NUM> and the cloud server <NUM>. The third party server <NUM> performs the operation corresponding to the user intention (e.g., ordering a pizza), and may transmit the result thereof to the UE <NUM>.

According to various embodiments of the disclosure, at least one (recognizing a user intention) of the operations performed by the cloud server <NUM> may be performed by the intelligent server <NUM>.

According to various embodiments, at least one (e.g., conversion to text) of the operations performed by the intelligent server <NUM> and/or at least one of the operations performed by the cloud server <NUM> may be performed by the UE <NUM>.

<FIG> is a diagram illustrating a system, configured to operate personalized automatic speech recognition (ASR), according to various embodiments of the disclosure.

Referring to <FIG>, a first electronic device <NUM> (e.g., the server <NUM> of <FIG>) may include a server (e.g., at least one of the intelligent server <NUM> of <FIG> or the cloud server <NUM>) to implement an integrated intelligent system (e.g., the integrated intelligent system <NUM> of <FIG>), and may perform data communication with a second electronic device <NUM> (e.g., the UE <NUM>) connectable to the integrated intelligent system, may perform management associated with a service and authority providable to the second electronic device <NUM>, and may manage a repository, and the like.

According to various embodiments of the disclosure, the first electronic device <NUM> may include a first processor <NUM>, an automatic speech recognition (ASR) model repository 318a, an NLU repository 318b, an action planning repository 318c, an ASR utterance data repository 318d, and an ASR error data repository 318e. The first processor <NUM> may include an ASR module <NUM>, an analysis module <NUM>, an NLU module <NUM>, a user interface (UI) module <NUM>, a personalized ASR model production module <NUM>, and an action planner module <NUM>.

According to various embodiments of the disclosure, at least one of the modules may be separately prepared hardware that is different from the first processor <NUM> and may be configured in the first electronic device <NUM>.

According to various embodiments of the disclosure, the modules may be software stored in a memory (not illustrated) of the first electronic device <NUM>. Accordingly, the first processor <NUM> may execute the modules so as to perform operations thereof.

According to various embodiments of the disclosure, the ASR module <NUM> may perform speech recognition on user speech data received from the second electronic device <NUM>, and may produce text data. For example, the ASR module <NUM> may convert speech data into text data using an acoustic model and/or a language model. The acoustic model may include information related to an utterance. The language model may include unit phoneme information and information associated with a combination of pieces of unit phoneme information. The acoustic model and the language model may be stored in, for example, the ASR model repository 318a.

According to various embodiments of the disclosure, the ASR utterance data repository 318d may store user utterance data. For example, when utterance data is stored in the ASR utterance data repository 318d by the ASR module <NUM> as shown in Table <NUM>, identification (ID) may be assigned in order to identify utterance data. Speech data received from the second electronic device <NUM>, text data corresponding to the speech data, and attribute information (i.e., metadata) that describes the speech data may be included. Metadata may include, for example, user information of a corresponding utterance and time information indicating the time at which the speech data is converted into the text data (or the time at which the corresponding utterance is made or the time at which the corresponding utterance data is stored in the ASR utterance data repository 318d). According to an embodiment, the metadata may further include information indicating a location where the corresponding utterance is made. For example, the location information is at least one piece of information among information (e.g., GNSS information, internet protocol (IP), address information, specific place name information, and the like) that the second electronic device <NUM> collects using, for example, the communication module <NUM> when the corresponding speech data is produced in the second electronic device <NUM>, and may be received together with the speech data by the first electronic device <NUM> from the second electronic device <NUM>.

According to various embodiments of the disclosure, the NLU module <NUM> may perform syntactic analysis (syntactic analyze) or semantic analysis (semantic analyze) on a user input (e.g., text data received from the ASR module <NUM>), so as to understand a user intention. The syntactic analysis is an analysis method that divides a user input into syntactic units (e.g., words, phrases, morphemes, and the like), and understands the syntactic elements of the units obtained by division. The semantic analysis may be performed using semantic matching, rule matching, formula matching, and the like.

According to various embodiments of the disclosure, the NLU module <NUM> may obtain a domain, an intention (intent), or a parameter (or a slot) needed for expressing the intention, from a user input.

According to an embodiment, the NLU module <NUM> may determine a user intention and a parameter using a matching rule that is classified based on a domain, an intention (intent), and a parameter (or a slot) needed to recognize the intention. For example, a single domain (e.g., an alarm) may include a plurality of intentions (e.g., setting an alarm, cancelling an alarm, and the like), and a single intention may include a plurality of parameters (e.g., a time, the number of repetitions, an alarm sound, and the like). The plurality of parameters may include, for example, one or more essential element parameters. The matching rule may be stored in a natural language understanding (NLU) repository 318b.

According to an embodiment, the NLU module <NUM> may understand the meaning of a word extracted from a user input using the linguistic characteristic (e.g., syntactic element) of a morpheme, a phrase, and the like, and may match the meaning of the word with a domain and an intention so as to determine a user intention. For example, the NLU module <NUM> may calculate how many words, extracted from the user input, are included in each domain and intention, and may determine a user intention. According to an embodiment, the NLU module <NUM> may determine the parameter of a user input using a word, based on which a user intention is recognized. According to an embodiment, the NLU module <NUM> may determine a user intention using the NLU repository 318b in which linguistic characteristics used for understanding the intention of a user input are stored. According to another embodiment, the NLU module <NUM> may determine a user intention using a personal language model (PLM). For example, the NLU module <NUM> may determine a user intention using personalized information (e.g., a contact information list or a music list). The PLM may be stored in, for example, the ASR model repository 318a. According to an embodiment, as well as the NLU module <NUM>, the ASR module <NUM> may also recognize a user speech with reference to a PLM stored in the NLU repository 318b.

According to various embodiments of the disclosure, the action planer module <NUM> may produce action planning which is used for performing a user intention based on the user intention and a parameter understood from a user input. For example, the action planner module <NUM> may select at least one application to be executed based on a user intention, and may determine at least one action (or operation) to be performed by the at least one selected application. The action planer module <NUM> may determine a parameter corresponding to the determined operation, and may produce action planning. According to an embodiment, the produced action planning may include information associated with an application, information associated with an operation to be performed in the application, and information associated with a parameter needed for the application to perform the operation. According to an embodiment, if multiple operations need to be performed, the produced action planning may include information associated with an application to perform each operation and information indicating the order of the operations.

According to various embodiments of the disclosure, the action planning may be produced by the NLU module <NUM>.

According to various embodiments of the disclosure, the action planner module <NUM> may store the produced action planning in the action planning repository 318c.

According to various embodiments of the disclosure, the action planner module <NUM> may select action planning corresponding to a user input from the action planning repository 318c. For example, the action planner module <NUM> may select one or more pieces of action planning corresponding to a user intention and a parameter from among pieces of action planning stored in the action planning repository 318c.

According to various embodiments of the disclosure, the data analysis module <NUM> may determine a text (hereinafter, an ASR error part) estimated to include an error among a plurality of pieces of text data, produced as the result of speech recognition by the ASR module <NUM>, and may determine a text (hereinafter, a substitute candidate) which is to replace the error part and is estimated to correspond to a user intention. According to an embodiment, the data analysis module <NUM> may extract similar text data from among text data of the same user (e.g., having the same user information) stored in the ASR utterance data repository 318d. The data analysis module <NUM> may compare extracted text data so as to extract different parts. The data analysis module <NUM> may determine one of the different parts as an ASR error part and another one of the different part as a substitute candidate, based on attribute information of each piece of the extracted text data. For example, the ASR module <NUM> may perform speech recognition and may recognize a first utterance of a user, "what's up", as "what's app". A second utterance of the user may be also, "what's up", and the ASR module <NUM> may perform speech recognition and may recognize the same as "what's up". The data analysis module <NUM> may determine "what's app", which is recognized via speech recognition performed earlier in time, to be an ASR error part, and may determine "what's up", which is recognized via speech recognition performed later in time, to be a substitute candidate.

According to various embodiments of the disclosure, the ASR error data repository 318e may store an error part (e.g., an ASR error part determined by the data analysis module <NUM>) that the ASR module <NUM> produces via erroneous speech recognition and a substitute candidate (e.g., a substitute candidate determined by the data analysis module <NUM>) estimated to be an actual utterance of a user, as shown in Table <NUM>. In Table <NUM>, a number contained in a round bracket refers to the number of times that corresponding text data is produced.

According to various embodiments of the disclosure, the data analysis module <NUM> may compare text data, produced when the ASR module <NUM> performs speech recognition on user speech data, with data in the ASR error data repository 318e, and may identify an ASR error part in the text data. The data analysis module <NUM> may identify a candidate to replace the ASR error part from the ASR error data repository 318e.

According to various embodiments of the disclosure, the data analysis module <NUM> may determine an ASR error part and/or substitute candidate, based on a user reaction (feedback) to an operation performed according to the result of speech recognition by the ASR module <NUM>. According to an embodiment, the NLU module <NUM> may understand a user intention using a substitute candidate determined by the data analysis module <NUM>. The action planer module <NUM> may produce action planning based on the user intention. The action planning may be transferred to the second electronic device <NUM> and an operation corresponding thereto may be performed in the second electronic device <NUM>. The user reaction may be negative to the performed operation (e.g., if the user input requires cancellation of the performed operation), the negative user reaction may be transferred to the data analysis module <NUM>, and the data analysis module <NUM> may exclude the substitute candidate from a group of candidates for replacing an ASR error part. For example, the data analysis module <NUM> may compare utterance ID #<NUM> and utterance ID #<NUM> in Table <NUM>, and may determine that "log" is an ASR error part and "lock" is a substitute candidate. However, if a user reaction to the utterance ID #<NUM> is negative, the utterance ID #<NUM> may be excluded from a substitute candidate group.

According to various embodiments of the disclosure, the data analysis module <NUM> may determine an ASR error part and/or substitute candidate, based on a user reaction (feedback) to the result of speech recognition by the ASR module <NUM>. According to an embodiment, as the result of speech recognition by the ASR module <NUM>, text data may be transferred to the second electronic device <NUM>, and may be displayed on the display <NUM>. The second electronic device <NUM> may perform an operation according to a user input received via an input device <NUM>, and may transfer the information associated with the performed operation to the data analysis module <NUM>. The information transferred to the data analysis module <NUM> may indicate that the user reaction to the speech recognition result is negative. For example, text data, "go to main screen and lock my phone", corresponding to the utterance ID #<NUM> is transferred to the second electronic device <NUM>, and the operation performed according to the user input may be at least partially irrelevant to the transferred text data. For example, the operation actually performed in the second electronic device <NUM> according to the user input may be "go to milk music (music reproduction application) and lock the phone". The data analysis module <NUM> may compare the speech recognition result transferred to the second electronic device <NUM> and the operation performed in the second electronic device <NUM>, and may determine an ASR error part and/or substitute candidate. For example, "main screen" is determined to be an ASR error part, and "milk music" is determined to be a substitute candidate.

According to various embodiments of the disclosure, the personalized ASR model production module <NUM> may produce a personalized ASR model of a user by performing machine learning (deep learning) on an ASR model based on the result (e.g., a substitute candidate) of analysis by the data analysis module <NUM> and user speech data corresponding thereto. The produced personalized ASR model may be stored in the ASR model repository 318a and the ASR module <NUM> may perform speech recognition by utilizing the stored personalized ASR model.

According to various embodiments of the disclosure, the personalized ASR model production module <NUM> may personalize (update) an ASR model, based on the result of analysis by the data analysis module <NUM> and user speech data corresponding thereto. At least one of a hidden Markov model (HMM)-based method, an artificial neural network (ANN)-based method, a support vector machine (SVM)-based method, or a dynamic time warping (DTW)-based method may be used as an ASR method. However, the disclosure is not limited thereto. An ASR model that is referred to may be different depending on an ASR method used. According to an embodiment, deep neural network-hidden Markov model (DNN-HMM) may be used as a recognition method [reference <NUM>: <NPL>]. The DNN-HMM-based ASR method performs modeling of an acoustic model (AM) using DNN, and performs modeling of a language model (LM) using HMM.

According to various embodiments of the disclosure, if DNN-HMM is used as a speech recognition algorithm, an AM or an LM may be corrected in order to personalize (update) an ASR model. The method that corrects the AM may be implemented by adding an adaptation layer after the last layer of a neural network used for the AM model [reference <NUM>: <NPL>)].

According to various embodiments of the disclosure, there may be the method that corrects the LM in order to personalize an ASR model. Even in the LM, an AR model may be personalized by correcting a pronunciation model (PM) that expresses words in a phoneme sequence (or a senone sequence). As described above, a linguistic expression in DNN-HMM may be implemented as wFST. FST (or finite state acceptor (FSA)) information that expresses pronunciation is updated among FSTs included in the whole LM, and thus, the pronunciation of a predetermined word may be personalized. For example, the FST of a predetermined word in the PM may be corrected or a weight may be applied to the arc of the FST [reference <NUM>: <NPL>].

According to various embodiments of the disclosure, the user interface (UI) module <NUM> may transmit action planning, produced by the action planner module <NUM> or selected from the action planning repository 318c, to the second electronic device <NUM> in response to a user input. The second electronic device <NUM> may select an application based on the received action planning, and may perform the selected application so as to perform an operation according to the user input.

According to various embodiments of the disclosure, the UI module <NUM> may determine whether a parameter determined by the NLU module <NUM> is sufficient for performing an operation associated with the user input. If the parameter is insufficient, the UI module <NUM> may transfer feedback that requests needed information from a user, to the second electronic device <NUM>.

According to various embodiments of the disclosure, the UI module <NUM> may control the ASR model production module <NUM> based on a user reaction to the result of analysis by the data analysis module <NUM>. For example, the UI module <NUM> may transmit information associated with an ASR error part determined by the data analysis module <NUM> to the second electronic device <NUM>. The second electronic device <NUM> may display, on the display, a user interface (UI) for receiving a user reaction to the received information. For example, the UI may include an ASR error part (or utterance including the same) and a substitute candidate (or utterance including the same). In addition, the UI may further include at least one of time information or place information related to an utterance corresponding to the ASR error part, as information for helping a user's understanding. Alternatively, the UI may further include a link for reproducing speech data corresponding to the ASR error part. The second electronic device <NUM> may reproduce speech data in response to user selection of the link, and may output the same via a speaker. The second electronic device <NUM> may transmit a user reaction received via the UI to the UI module <NUM>. If the received user reaction means error correction, the UI module <NUM> may control the personalized ASR model production module <NUM> so as to produce a personalized ASR model.

According to various embodiments of the disclosure, the UI module <NUM> may control the ASR model <NUM> based on a user reaction to the result of analysis by the data analysis module <NUM>. For example, if a received user reaction means error correction, the UI module <NUM> may control the ASR module <NUM> so as to perform speech recognition using a personalized ASR model produced by the personalized ASR model production module <NUM>.

According to various embodiments of the disclosure, the UI module <NUM> may determine whether to store a personalized ASR model in the ASR model repository 318a, based on a user reaction to the result of speech recognition by the ASR module <NUM> to which the personalized ASR model is applied. For example, the UI module <NUM> may transmit the result to the second electronic device <NUM>. The UI module <NUM> may receive a user reaction to the result from the second electronic device <NUM>. If the received user reaction means result acceptance (e.g., if a first speech recognition result to which a personalized ASR model is applied and a second speech recognition result to which an existing ASR model, as opposed to the personalized ASR model, is applied are displayed on a touch screen, and a user gesture of a touch to the first speech recognition result is received), the UI module <NUM> may store the personalized ASR model in the ASR model repository 318a, and thus, the ASR module <NUM> is capable of performing speech recognition using the stored personalized ASR model. If the received user reaction means result non-acceptance (e.g., if the second speech recognition result is selected), the UI module <NUM> may not store the produced personalized ASR model in the ASR model repository 318a, and discards the same. Accordingly, the ASR module <NUM> to which the personalized ASR model is not applied may perform speech recognition.

According to various embodiments of the disclosure, the second electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>) may include a second processor <NUM>, a display <NUM>, a memory <NUM>, an input device <NUM>, and a speaker <NUM>.

According to various embodiments of the disclosure, the input device <NUM> (e.g., the input device <NUM> of <FIG>) may receive a user input from a user. For example, the input device <NUM> may include a sensor circuit (e.g., a pressure sensor and a fingerprint sensor) and a touch circuit coupled to the display <NUM>, a microphone, and a hardware key.

According to various embodiments of the disclosure, the input device <NUM> may receive a user input (e.g., speech data via a headset) via an external device electrically connected to a connection terminal (not illustrated) (e.g., the connection terminal <NUM> of <FIG>).

According to various embodiments of the disclosure, the display <NUM> (e.g., a display included in the display device <NUM> of <FIG>) may display an image, a video, and/or an execution screen of an application. For example, the display <NUM> may display a UI for receiving a user reaction to information received from the first electronic device <NUM>. Alternatively, the display <NUM> may display an execution screen of an application, as the result of an operation performed under control of an execution engine module <NUM>.

According to various embodiments of the disclosure, the speaker <NUM> (e.g., a speaker included in the acoustic output device <NUM> of <FIG>) may output a sound signal. For example, the speaker <NUM> may convert speech data produced in the first electronic device <NUM> into a sound signal, and may output the sound signal. Alternatively, the speaker <NUM> may output a sound signal as the result of an operation performed under control of the execution engine module <NUM>.

According to various embodiments of the disclosure, the memory <NUM> (e.g., the memory <NUM> of <FIG>) may store a plurality of applications. For example, the stored applications may be selected and executed by the execution engine module <NUM>.

According to various embodiments of the disclosure, the second processor <NUM> may include an intelligent agent <NUM>, the execution engine module <NUM>, and a hint provision module <NUM>.

According to various embodiments of the disclosure, at least one of the intelligent agent <NUM>, the execution engine module <NUM>, and the hint provision module <NUM> may be configured in the second electronic device <NUM>, as separately prepared hardware, different from the second processor <NUM>.

According to various embodiments of the disclosure, the intelligent agent <NUM>, the execution engine module <NUM>, and the hint provision module <NUM> may be software stored in the memory <NUM>. Accordingly, the second processor <NUM> may execute the intelligent agent <NUM>, the execution engine module <NUM>, and the hint provision module <NUM> so as to perform operations thereof.

According to various embodiments of the disclosure, the intelligent agent <NUM> may include a wake-up recognition module that recognizes a call from a user. The wake-up recognition module may recognize a wake-up command from a user via a speech recognition module, and if the wake-up command is recognized, the intelligent agent <NUM> may be activated in order to receive a user input. According to an embodiment, the wake-up recognition module may be implemented in a low-power processor (e.g., a processor included in an audio codec). According to an embodiment, the intelligent agent <NUM> may be activated according to a user input provided via a hardware key. If the intelligent agent <NUM> is activated, an intelligent app (e.g., a speech recognition app) interoperating with the intelligent agent <NUM> may be executed.

According to various embodiments of the disclosure, the intelligent agent <NUM> may include a speech recognition module for recognizing a user input. The speech recognition module recognizes a user input for enabling an app to perform an operation. For example, the speech recognition module may recognize a limited user input (e.g., utterance, such as "snap", that executes photographing while a camera app is running) that executes an operation, such as the wake-up command, in an app.

According to various embodiments of the disclosure, the speech recognition module of the intelligent agent <NUM> may recognize a user input, by assisting the ASR module <NUM> of the first electronic device <NUM>. For example, the speech recognition module may recognize a user command, which may be processible by the second electronic device <NUM>, and quickly process the same.

According to various embodiments of the disclosure, the speech recognition module and the wake-up recognition module of the intelligent agent <NUM> may recognize a user input using a speech recognition algorithm. The algorithm used for recognizing a speech may be, for example, at least one of a hidden Markov model (HMM), an artificial neural network (ANN), or dynamic time warping (DTW).

According to various embodiments of the disclosure, the intelligent agent <NUM> may convert user speech data into text data, and may display the text data on the display <NUM>.

According to various embodiments of the disclosure, the intelligent agent <NUM> may transmit speech data received via the input device <NUM> to the first electronic device <NUM>. The intelligent agent <NUM> may receive text data from the first electronic device <NUM> as the result of processing the speech data, and may display the same on the display <NUM>.

According to various embodiments of the disclosure, the intelligent agent <NUM> may receive action planning from the first electronic device <NUM>, as the result of processing the speech data. The intelligent agent <NUM> may transfer the received action planning to the execution engine module <NUM>, and thus, the execution engine module <NUM> is capable of performing corresponding operations.

According to various embodiments of the disclosure, the intelligent agent <NUM> may perform speech pre-processing (e.g., speech pre-processing by the UE <NUM> of <FIG>), before transmitting speech data to the first electronic device <NUM>. For example, the intelligent agent <NUM> may include an adaptive echo canceller (AEC) module, a noise suppression (NS) module, an endpoint detection (EPD) module, or an automatic gain control (AGC) module. The AEC module may cancel echos included in the user input. The NS module may suppress background noise included in the user input. The EPD module may detect the end point of user speech included in the user input, so as to detect a part where the user speech is present. The AGC module recognizes the user input and may control the volume of the user input to be appropriate for processing the user input.

According to various embodiments of the disclosure, the execution engine module <NUM> may receive action planning from the first electronic device <NUM> via the intelligent agent <NUM>, and may execute a corresponding application so that the application performs corresponding operations.

According to various embodiments of the disclosure, the hint provision module <NUM> may provide (propose) a hint related to ASR error correction to a user. For example, the hint provision module <NUM> may receive information related to ASR error part from the first electronic device <NUM>, and may display a UI included in the received information on the display <NUM>.

According to various embodiments of the disclosure, the hint provision module <NUM> may receive a user reaction to the provided hint from a user via the input device <NUM>, and may transmit the user reaction to the first electronic device <NUM>.

Although not illustrated, the second electronic device <NUM> may include a communication circuit (e.g., the communication circuit <NUM> of <FIG>) for data communication with the first electronic device <NUM>, and the first electronic device <NUM> may also include a communication circuit (e.g., a communication module that is the same as the communication module <NUM> of <FIG>, and performs the same function) for data communication with the second electronic device <NUM>.

<FIG> is a flowchart illustrating operations for operating a personalized ASR, according to various embodiments.

Referring to <FIG>, the operations of <FIG> according to the claimed invention are performed by a processor (e.g., the first processor <NUM> of <FIG>) of an electronic device.

According to the claimed invention, in operation <NUM>, the processor (e.g., the ASR module <NUM> of <FIG>) stores a plurality of pieces of text data, produced by performing speech recognition on user speech data using a predetermined ASR model, together with the speech data in a repository (e.g., the ASR utterance data repository 318d of <FIG>).

According to the claimed invention, in operation <NUM>, the processor obtains an ASR error part and a substitute candidate from the plurality of pieces of stored text data. For example, the processor (e.g., the data analysis module <NUM> of <FIG>) may execute an operation of extracting an ASR error part and a substitute candidate from among pieces of text data of the same user, which are stored in the ASR utterance data repository 318d.

According to the claimed invention, in operation S430, the processor (e.g., the personalized ASR model production module <NUM> of <FIG>) produces a personalized ASR model of the user by performing deep learning on the ASR model based on the substitute candidate and user speech data corresponding thereto.

According to the claimed invention, in operation <NUM>, the processor (e.g., the UI module <NUM> of <FIG>) receives a user reaction to the ASR error part and the substitute candidate. For example, a display (e.g., the display <NUM> of <FIG>) functionally connected to the processor may display a UI including the ASR analysis result (e.g., an ASR error part and a substitute candidate). A user reaction to the displayed ASR analysis result may be transferred to the processor via an input device (e.g., the input device <NUM> of <FIG>) functionally connected to the processor.

According to various embodiments of the disclosure, in operation <NUM>, the processor (e.g., the UI module <NUM>) may apply, based on the user reaction, the produced personalized ASR model to ASR. For example, if the user reaction is positive (e.g., if a user input indicating that an ASR analysis result is accepted is transferred to the processor), the processor stores the personalized ASR model in a repository (e.g., the ASR model repository 318a of <FIG>) so that the ASR module (e.g., the ASR module <NUM> of <FIG>) is capable of performing speech recognition using the personalized ASR model stored in the repository.

<FIG> is a flowchart illustrating operations for analyzing a speech recognition result according to various embodiments.

Referring to <FIG>, the operations (e.g., operation <NUM> of <FIG>) of <FIG> according to the claimed invention are performed by a processor (e.g., the first processor <NUM> or the data analysis module <NUM> of <FIG>) of an electronic device.

According to the claimed invention, in operation <NUM>, the processor groups pieces of utterance data, which are produced consecutively during a predetermined period of time, among the plurality of pieces of utterance data stored in the utterance data repository, into a single utterance group. Moreover, for example, the method that performs grouping may be performed based on an utterance that is made in a predetermined place, an utterance that is made in a predetermined context, an utterance that is made in a predetermined location, and the like. For example, with reference to Table <NUM>, utterance IDs #<NUM>, #<NUM>, and #<NUM>, which are produced consecutively within one minute, among utterance data of speaker A, may be grouped into a single utterance group.

According to various embodiments of the disclosure, in operation <NUM>, the processor may exclude (filter out) one or more pieces of utterance data that satisfy a predetermined exceptional condition from the utterance group. For example, text data which are linguistically the same may be excluded from the utterance group. As another example, text data having a difference in only the locations of words is classified as the same text data and may be excluded from the utterance group.

As another example, text data having a difference in only contraction, as shown below, is classified as the same text data and may be excluded from the utterance group.

As another example, text data having a difference in only a word that is frequently used and means the opposite, such as on/off and the like, as shown below, is classified as an exceptional case, and may be excluded from the utterance group.

As another example, text data having a difference in only numbers used, as shown below, is classified as an exceptional case and may be excluded from the utterance group.

As another example, text data to which some words are added, as shown below, is classified as an exceptional case and may be excluded from the utterance group.

According to the claimed invention, in operation <NUM>, the processor selects a pair of pieces of similar utterance data from the utterance group. According to an embodiment, the processor may calculate similarity among pieces of utterance data using an algorithm such as soundex, double metaphone, and the like. According to the claimed invention, the processor selects a pair of pieces of utterance data having a similarity greater than a predetermined threshold as similar utterance data. For example, in the utterance group including utterance IDs #<NUM>, <NUM>, and <NUM>, the processor may select ID#<NUM> and ID#<NUM>, which have a difference in only a single word, as a pair of similar utterance data.

According to various embodiments of the disclosure, the soundex algorithm may assign the same code value to English names having similar pronunciation. For example, in the case of "smith" and "smyth", they have a difference in spelling but have the same pronunciation, and thus, the same code value (e.g., S530) may be assigned. For example, in the case of English characters, the first character may be included in a result value unconditionally. If a, e, h, i, o, u, w, or y comes as the second character, the algorithm disregards the same, and proceeds with a subsequent character. Subsequently, a character may be replaced with a code value, with reference to a table such as Table <NUM> as shown below. The corresponding table may be obtained based on the shape of a mouth during pronouncing. In the case in which the same value appears successively at least two times among the replaced values, if a previous character is a letter included in the table, it may be disregarded. If they are not consecutive letters, they may be included in a result value. A result value corresponding to a total of <NUM> letters may be obtained. If letters are insufficient, "<NUM>" may be filled in the result value.

According to various embodiments of the disclosure, the double metaphone algorithm may use more various speech codes based on a method of converting spelling into a speech code, such as the soundex algorithm, and may return a maximum of two candidates for vague pronunciation.

According to the claimed invention, in operation <NUM>, the processor extracts a difference between the pair of pieces of similar utterance data, wherein the difference comprises a pair of words. For example, a pair of words "log" and "lock" may be extracted as the difference between utterance ID #<NUM> and utterance ID #<NUM>.

According to the claimed invention, in operation <NUM>, the processor determines a substitute candidate based on the extracted difference. For example, the processor may determine that "lock" is a word (substitute candidate) that a user utters and "log" is an ASR error part in the pair of words.

<FIG> is a diagram illustrating a user interface (UI) for receiving a user reaction to an ASR error analysis result according to various embodiments.

Referring to <FIG>, a processor (e.g., the first processor <NUM> of <FIG>) according to various embodiments of the disclosure may provide a UI <NUM> to a user via a display (e.g., the display <NUM> of <FIG>) functionally connected to the processor. For example, the processor may display first text data <NUM> in the upper part of the UI <NUM>, as the result of ASR, and may display second text data <NUM>, which is a hint (a substitute candidate) to replace the first text data <NUM>, in the lower part of the UI <NUM>, as the result of ASR error analysis. A user reaction (e.g., a touch input of a user to data displayed on a touch screen) to the first text data <NUM> or the second text data <NUM> may be transferred to the processor via an input device (e.g., the input device <NUM> of <FIG>) functionally connected to the processor. If the user reaction selects the first text data <NUM>, the processor may produce action planning corresponding to the first text data <NUM>. If the user reaction selects the second text data <NUM>, the processor may produce action planning corresponding to the second text data <NUM>, and may operate a personalized ASR using the second text data <NUM>.

Referring to <FIG>, a processor (e.g., the first processor <NUM> of <FIG>) according to various embodiments of the disclosure may display first text data <NUM> via a UI <NUM> as the result of ASR, and may display second text data <NUM> via the UI <NUM> as the result of ASR error analysis. For example, the processor may provide the result of ASR, "open my files and show my listen <NUM>" among voice assistant history, to a user via the UI <NUM>. The processor may provide text data, "open my files and show my recent file", which is corrected via ASR error analysis, to the user via the UI <NUM>.

According to various embodiments of the disclosure, the processor may display, via the UI <NUM>, a reproduction icon <NUM> for reproducing user speech data to which ASR is to be performed. If the reproduction icon <NUM> is selected by the user via an input device, the processor may output a corresponding sound signal via a speaker.

According to various embodiments of the disclosure, the processor may display, via the UI <NUM>, a personalization application icon <NUM> (e.g., an icon that reads, "apply this to me") for receiving a user reaction to the result of ASR error analysis. If the personalization application icon <NUM> is selected by a user via the input device, the user reaction meaning that the ASR error analysis is accepted is transferred to the processor, and the processor may operate a personalized ASR.

Referring to <FIG>, a processor (e.g., the first processor <NUM> of <FIG>) according to various embodiments of the disclosure may display, via a UI <NUM>, text data <NUM> corrected via ASR error analysis among voice assistant history of a user, together with an icon <NUM> for reproducing a corresponding speech, and an icon <NUM> for enabling the user to select whether to apply a personalized ASR. For example, the processor may display a corrected word among words in the user utterance. Alternatively, the processor may display the entirety of a corresponding sentence including a corrected word.

According to various embodiments of the disclosure, user utterance-based ASR personalization may be performed with respect to a predetermined group of persons, as opposed to a predetermined person. For example, the processor (e.g., the first processor <NUM> of <FIG>) may collect utterance data from all users who use a predetermined speech service, and may perform ASR error analysis on the collected utterance data, so as to estimate an utterance that is converted into text, different from a user intention, and how frequently the erroneous conversion occurs. Table <NUM> below shows an example of collected words of users which are estimated to be ASR errors, together with the frequency of occurrence. As shown in Table <NUM>, the processor may extract a predetermined word that is erroneously recognized as a different word with a high probability. The processor may recognize that an ASR error occurs in association with the predetermined word, as opposed to an individual. Therefore, by improving ASR in association with a word for which an ASR error frequently occurs, the corresponding word is recognized well. As a group from which a corresponding utterance is collected, all users may be considered. Further, various types of groups, such as a region-based group, a time-based group, an age-based group, and the like, may be also considered. Accordingly, by changing a word used in a predetermined region into predetermined text, or the like, the ASR operation may be improved.

Claim 1:
An electronic device (<NUM>), comprising:
a processor (<NUM>); and
a memory (<NUM>) electrically connected to the processor,
wherein the memory stores instructions which, when being executed by the processor, cause the processor to:
store (<NUM>) a plurality of pieces of text data, produced by recognizing user speech data using a predetermined automatic speech recognition, ASR, model, together with the speech data as a piece of utterance data in an utterance data repository functionally connected to the processor;
obtain (<NUM>) an ASR error part and a substitute candidate to replace the ASR error part from the plurality of pieces of text data stored in the utterance data repository, wherein the instructions, when being executed by the processor, further cause the processor to:
group (<NUM>) pieces of utterance data, which are produced consecutively during a predetermined period of time, among the plurality of pieces of utterance data stored in the utterance data repository, into a single utterance group;
select (<NUM>) a pair of pieces of similar utterance data in the utterance group;
extract (<NUM>) a difference between the pair of the pieces of similar utterance data, wherein the difference comprises a pair of words; and
obtain (<NUM>) a word, produced later in time than the other in the pair of the words to be the substitute candidate based on the difference;
perform (<NUM>), based on the substitute candidate and user speech data corresponding to the substitute candidate, deep learning on the ASR model so as to produce a personalized ASR model;
receive (<NUM>) a user reaction with respect to the ASR error part and the substitute candidate via an input device functionally connected to the processor; and
update (<NUM>), based on the user reaction, the ASR model to the personalized ASR model.