Patent Description:
Automatic speech recognition (ASR) is a technology for receiving a human voice utterance and converting the human voice utterance into text. Speech recognition is used in various electronic devices" such as a smartphone, an air conditioner, a refrigerator, and an artificial intelligence (Al) home assistant. For example, an apparatus detects a human voice utterance as an input, recognizes the received utterance by using a speech recognition model trained to recognize speech, and converts the recognized utterance into text. The text may be a final output of the apparatus.

Recently, deep neural network (DNN) algorithms have been used in various machine learning fields and performance of speech recognition has been improved. For example, in the field of speech recognition, performance has been greatly improved by using a neural network, and a speech recognition model (e.g., an ASR model) for speech recognition has been developed. As Al systems have been improved, recognition rates have been increased and user preferences have been more accurately understood, and thus existing rule-based smart systems have been gradually replaced with deep learning-based Al systems. In similar regards, publication <CIT> relates to systems and methods for user recognition by analyzing audio data received from a speech-controlled device, publication <CIT> relates to methods for identifying and correcting errors in text recognized by an automatic speech recognition module, and publication <CIT> relates to systems and methods for analyzing audio data from a voice-controlled device to interpret therefrom a command related to an accessory device and a command related to the voice-controlled device.

Provided are a method according to claim <NUM> and a server according to claim <NUM> that provide an output value of an automatic speech recognition (ASR) model of a device to a server and modify the output value of the ASR model by using an artificial intelligence (Al) model of the server.

The embodiments falling under <FIG>, <FIG> and <FIG> do not form part of the claimed invention but are useful for its understanding.

Hereinafter, embodiments of the disclosure will be described in detail in order to enable one of ordinary skill in the art to easily embody and practice the disclosure with reference to the attached drawings. The disclosure may, however, be implemented in many different forms and should not be construed as being limited to the embodiments of the disclosure set forth herein. In the drawings, parts unnecessary to the description in the drawings are omitted for clarity of the disclosure, and like reference numbers denote like elements throughout.

Throughout the specification, it will be understood that when a part is referred to as being "connected" to another part, it may be "directly connected" to the other part or "electrically connected" to the other part with intervening elements therebetween. It will be further understood that when a part "includes" or "comprises" an element, unless otherwise defined, the part may further include other elements, not excluding the other elements.

The disclosure will now be described more fully with reference to the accompanying drawings, in which embodiments of the disclosure are shown.

<FIG> is a block diagram illustrating a speech recognition system according to an embodiment of the disclosure.

Referring to <FIG>, a speech recognition system according to an embodiment of the disclosure includes a device <NUM> and a server <NUM>.

The device <NUM> may include an automatic speech recognition (ASR) model, and the server <NUM> may include a text modification model. The device <NUM> may recognize a voice input of a user by using the ASR model and may output text. The server <NUM> may modify the text generated by the device <NUM>.

An ASR model is a speech recognition model for recognizing a voice by using an integrated neural network, and may output text from a voice input of a user. The ASR model may be an artificial intelligence (Al) model including, for example, an acoustic model, a pronunciation dictionary, and a language model. Alternatively, the ASR model may be an end-to-end speech recognition model having a structure including an integrated neural network without separately including, for example, an acoustic model, a pronunciation dictionary, and a language model.

Because an end-to-end ASR model uses an integrated neural network, the end-to-end ASR model may convert a voice into text without a process of recognizing a phoneme from the voice and then converting the phoneme into text. Text may include at least one character. A character refers to a symbol used to describe a human language in a visible form. Examples of characters may include Korean letters (Hangul), alphabets, Chinese characters, numbers, phonetic marks, punctuation marks, and other symbols.

Also, for example, text may include a character string. A character string refers to a sequence of characters. For example, text may include at least one grapheme. A grapheme is a smallest unit including at least one character and representing sound. For example, in an alphabetic writing system, one character may become a grapheme, and a character string may refer to a sequence of graphemes. For example, text may include a morpheme or a word. A morpheme is a smallest meaningful unit including at least one grapheme. A word is an independent linguistic unit having a grammatical function and including at least one morpheme a grammatical function.

The device <NUM> may receive a voice input of a user, may recognize the received voice input by using the ASR model, and may provide text that is an output value of the ASR model to the server <NUM>. Also, the server <NUM> may receive the text that is the output value of the ASR model from the device <NUM> and may modify the received text. The server <NUM> may identify a degree to which the text that is the output value of the ASR model is related to a specific domain registered in the server <NUM>. The server may modify the text by using a text modification model of the identified domain. Also, the server <NUM> may provide modified text to the device <NUM>.

A text modification model that is an Al model trained to modify at least a part of text that is a speech recognition result may include, for example, a sequence-to-sequence mapper. A text modification model may be an AI model trained by using text output from an ASR model and preset ground truth text. A text modification model may be an AI model trained for each domain. For example, a text modification model of a first domain may be trained by using text that is an output value of an ASR model and ground truth text specialized for the first domain. Also, for example, a text modification model of a second domain may be trained by using the text that is the output value of the ASR model and ground truth text specialized for the second domain.

A text modification model may be an AI model trained by using pieces of text output from multiple types of ASR models and preset pieces of ground truth text. In this case, because the text modification model is trained by using the pieces of text output from the multiple types of ASR models, an accurate output value (e.g., modified text) may be provided regardless of which type of ASR model text input to the text modification model is output from.

Modified text includes at least one of a modified character, and may further include at least one of a modified grapheme, a modified morpheme, or a modified word. For example, when text output from an ASR model includes a mistake, modified text may include a character corrected from the mistake. Also, for example, when text output from an ASR model includes a word with an incorrect meaning inappropriate for the context, modified text may include a word with a correct meaning that replaces the word with incorrect meaning. Also, for example, modified text may be generated by replacing a specific word in text output from an ASR model with a similar word.

Examples of the device <NUM> may include, but are not limited to, a smartphone, a tablet personal computer (PC), a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a micro-server, a global positioning system (GPS) device, an e-book terminal, a digital broadcasting terminal, a navigation, a kiosk, an MP3 player, a digital camera, a home appliance, and other mobile or non-mobile computing devices. Also, the device <NUM> may be a wearable device, such as a watch, glasses, a hair band, or a ring having a communication function and a data processing function. However, the disclosure is not limited thereto, and the device <NUM> may include any type of device capable of transmitting data and receiving data through a network <NUM> to the server <NUM> and from the server <NUM> for performing speech recognition.

Examples of the network <NUM> include a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and a combination thereof. The network <NUM> is a data communication network in a comprehensive sense for enabling network constituent elements of <FIG> to smoothly communicate with each other and includes a wired Internet, a wireless Internet, and a mobile wireless communication network.

<FIG> is a block diagram illustrating a speech recognition system including text modification models related to a plurality of domains, according to an embodiment of the disclosure.

Referring to <FIG>, the device <NUM> may include an ASR model. The server <NUM> may include text modification models for modifying text output from the ASR model. For example, the server <NUM> may include a first text modification model corresponding to a first domain and a second text modification model corresponding to a second domain.

The device <NUM> may obtain a feature vector by extracting a feature from a voice input, and may provide the obtained feature vector as input to the ASR model. The device <NUM> may provide an output value output from the ASR model to the server <NUM>. The output value output from the ASR model may include text in various forms. For example, the device <NUM> may provide sentence-unit text to the server <NUM> or may provide a text stream to the server <NUM>, but the system is not limited thereto.

The server <NUM> may receive the text from the device <NUM>, and may select a (text) domain related to the received text. A domain indicates a field related to subject matter of an input voice and may be pre-set according to, for example, the meaning of the input voice or the attribute of the input voice. A domain may be classified according to, for example, a service related to an input voice when the subject matter of the input device is a service. Also, a text modification model may be trained for each domain, and in this case, the text modification model trained for each domain may be a model trained by using input text related to the domain and ground truth text corresponding to the input text. The server <NUM> may select at least one of a plurality of preset domains, and may select at least one of text modification models corresponding to the selected domain. Also, the server <NUM> may obtain modified text by inputting the text received from the device <NUM> to the text modification model of the selected domain. The server <NUM> may provide the modified text to the device <NUM>.

The server <NUM> may provide various types of voice assistant services to the device <NUM> by using the modified text. A voice assistant service may be a service of providing conversation with a user who provides commands or questions to the voice assistant service. In a voice assistant service, a response message may be provided to a user as if a person directly talks to the user in consideration of the user's situation, a condition of a device, etc. In addition, in a voice assistant service, information required by a user may be appropriately generated and provided to the user as if a personal assistant of the user provides the information. A voice assistant service may provide to the user information or a function requested by the user, in conjunction with various services, such as a broadcasting service, a content sharing service, a content providing service, a power management service, a game service, a chat service, a document creating service, a search service, a call service, a photographing service, a transportation recommendation service, and a video playback service.

<FIG> is a flowchart illustrating a method of a device and a server in a speech recognition system recognizing a voice input and obtaining modified text, according to an embodiment of the disclosure.

The device <NUM> may perform an operation in <FIG> by executing instructions stored in a memory of the device <NUM>. For example, the device <NUM> may perform an operation in <FIG> by executing at least one of a speech recognition evaluation module <NUM>, a domain identification module <NUM>, a natural language understanding (NLU) determination module <NUM>, a domain registration module <NUM>, an ASR model <NUM>, or an NLU model <NUM> of <FIG>, which will be described below. However, the disclosure is not limited thereto, and the device <NUM> may execute other programs stored in the memory to perform other operations associated with the programs stored in the memory.

Also, the server <NUM> may perform an operation in <FIG> by executing instructions stored in a memory of the server <NUM>. For example, the server <NUM> may perform an operation by executing at least one of a domain management module <NUM>, a voice analysis management module <NUM>, a text modification module <NUM>, or an NLU module <NUM> of <FIG>, which will be described below. However, the disclosure is not limited thereto, and the server <NUM> may execute other programs stored in the storage to perform a certain operation of the server <NUM>.

In operation S300, the device <NUM> may obtain a feature vector from a voice signal. The device <NUM> may receive a voice input (e.g., utterance) of a user through a microphone, and may generate the feature vector indicating a feature of the voice signal by using the voice signal obtained through the microphone. When noise is included in the voice signal, the device <NUM> may remove the noise in the voice signal, and may obtain the feature vector from the voice signal from which the noise is removed. Also, for example, the device <NUM> may extract the feature vector indicating the feature of the voice signal from the voice signal. For example, the device <NUM> may receive data indicating the feature vector of the voice signal from an external device.

In operation S305, the device <NUM> may obtain text from the feature vector by using an ASR model. The device <NUM> may provide the feature vector as input to the ASR model in the device <NUM> to recognize a voice of the user. When a plurality of ASR models are included in the device <NUM>, the device <NUM> may select one of the plurality of ASR models, and may convert the feature vector into a format suitable for the selected ASR model. The ASR model of the device <NUM> may be an AI model including, for example, an acoustic model, a pronunciation dictionary, and a language model. Alternatively, the ASR model of the device <NUM> may be an end-to-end speech recognition model having a structure including an integrated neural network without separately including, for example, an acoustic model, a pronunciation dictionary, and a language model.

In operation S310, the device <NUM> may obtain a reliability of the text. A reliability of text may be a value indicating a degree to which the text output from an ASR model is matched to an input voice, and may include, for example, but not limited to, a confidence score. Also, a reliability of text may be related to a probability that the text will match an input voice. For example, the reliability of the text may be calculated based on at least one of likelihoods of a plurality of pieces of estimated text output from the ASR model of the device <NUM> or posterior probabilities that at least one character in the text will be replaced by another character. For example, the device <NUM> may calculate the reliability based on a likelihood output as a result of Viterbi decoding. Alternatively, for example, the device <NUM> may calculate the reliability based on posterior probabilities output from a softmax layer in the end-to-end ASR model. For example, the device <NUM> may determine a plurality of pieces of estimated text estimated in a speech recognition process of the ASR model of the device <NUM>, and may calculate the reliability of the text based on a correlation of characters in the plurality of pieces of estimated text. Also, for example, the device <NUM> may obtain the reliability of the text by using the speech recognition evaluation module <NUM> of <FIG>.

In operation S315, the device <NUM> may determine whether to transmit the text to the server <NUM>. The device <NUM> may determine whether to transmit the text to the server <NUM>, by comparing the reliability of the text with a preset threshold value. When the reliability of the text is equal to or greater than the preset threshold value, the device <NUM> may determine not to transmit the text to the server <NUM>. Also, when the reliability of the text is less than the preset threshold value, the device <NUM> may determine to transmit the text to the server <NUM>.

Also, the device <NUM> may determine whether to transmit the text to the server <NUM>, based on at least one text having a high reliability from among the plurality of pieces of estimated text in the speech recognition process of the ASR model. For example, the plurality of pieces of estimated text estimated in the speech recognition process of the ASR model include first estimated text having a high reliability and second estimated text having a high reliability. If a difference between the reliability of the first estimated text and the reliability of the second estimated text is equal to or less than a certain threshold value, the device <NUM> may determine to transmit the text to the server <NUM>. Also, for example, the plurality of pieces of estimated text estimated in the speech recognition process of the ASR model include first estimated text having a high reliability and second estimated text having a high reliability. If a difference between the reliability of the first estimated text and the reliability of the second estimated text is greater than the certain threshold value, the device <NUM> may determine not to transmit the text to the server <NUM>.

When it is determined in operation S315 that the text is to be transmitted to the server <NUM>, in operation S320, the device <NUM> may request the server <NUM> to modify the text.

The device <NUM> may transmit the text to the server <NUM> and may request the server <NUM> for modified text. In this case, for example, the device <NUM> may transmit a type of the ASR model and an identification value of the ASR model in the device <NUM> to the server <NUM> while requesting the server <NUM> for the modified text, but the disclosure is not limited thereto.

Also, for example, the device <NUM> may provide domain information related to the text output from the ASR model of the device <NUM> to the server <NUM> while requesting the server <NUM> for the modified text. Domain information that is information for identifying a domain may include, but is not limited to, a name of the domain and an identifier of the domain. The device <NUM> may identify the domain related to the text by using the domain identification module <NUM> in the device <NUM>. For example, the device <NUM> may identify the domain related to the text, based on a domain reliability of the text output from the ASR model of the device <NUM>. A domain reliability may be a value indicating a degree to which at least part of text is related to a specific domain. For example, the device <NUM> may calculate a confidence score indicating a degree to which the text output from the ASR model is related to a domain pre-registered in the device <NUM>. Also, the device <NUM> may identify the domain related to the text, based on the calculated domain reliability. The device <NUM> may identify the domain related to the text based on rules or may obtain the domain reliability related to the text by using an AI model trained for domain identification. Also, for example, the AI model for domain identification may be a part of an NLU model, or a model separate from the NLU model.

When it is determined in operation S315 that the text is not to be transmitted to the server <NUM>, the device <NUM> may provide a voice assistant service by using the text output from the ASR model. For example, when the reliability of the text output from the ASR model is equal to or greater than a preset threshold value, the device <NUM> may perform operations for the voice assistant service by using the text output from the ASR model. Also, for example, the plurality of pieces of estimated text estimated in the speech recognition process of the ASR model include first estimated text having a highest reliability and second estimated text having a second highest reliability. If a difference between the reliability of the first estimated text and the reliability of the second estimated text is greater than a certain threshold value, the device <NUM> may provide the voice assistant service by using the first estimated text having the highest reliability.

For example, the device <NUM> may display the text output from the ASR model on a screen. For example, the device <NUM> may perform an operation for conversation with the user based on the text output from the ASR model. Also, for example, the device <NUM> may provide, through conversation with the user based on the text output from the ASR model, various services, such as a broadcasting service, a content sharing service, a content providing service, a power management service, a game service, a chat service, a document creating service, a search service, a call service, a photographing service, a transportation recommendation service, and a video playback service.

In operation S325, the server <NUM> may identify a domain for text modification. When the server <NUM> receives the domain information from the device <NUM>, the server <NUM> may identify the domain for text modification from the domain information. Alternatively, when the server <NUM> does not receive the domain information from the device <NUM>, the server <NUM> may identify a domain related to the text received from the device <NUM> by using a domain identification module <NUM> in the server <NUM>. For example, in this case, the server <NUM> may identify the domain related to the text, based on the domain reliability of the text received from the device <NUM>. For example, the server <NUM> may calculate a confidence score indicating a degree to which the text received from the device <NUM> is related to a domain that is pre-registered for text modification. Also, the server <NUM> may identify the domain related to the text received from the device <NUM>, based on the domain reliability calculated for the pre-registered domain. The server <NUM> may identify the domain related to the text based on rules or may obtain the domain reliability related to the text by using an AI model trained for domain identification. Also, for example, the AI model for domain identification may be a part of an NLU model or a model separate from the NLU model.

In operation S330, the server <NUM> may modify the text by using a text modification model corresponding to the determined domain. The server <NUM> may include a plurality of text modification models corresponding to a plurality of domains, and may select a text modification model corresponding to the domain identified in operation S325 from among the plurality of text modification models.

The server <NUM> may select a domain corresponding to the domain identified in operation S325 form among domains registered in the server <NUM>, and may select a text modification model of the selected domain. The domain corresponding to the domain identified in operation S325 from among the domains registered in the server <NUM> may be a domain that is the same as or similar to the identified domain. For example, when the plurality of domains registered in the server <NUM> are "movie," "location," and "region name," and the domain identified in operation S325 is 'movie,' the server <NUM> may select "movie. " For example, when the plurality of domains in the server <NUM> are "video content," "location," and "region name" and the domain identified in operation S325 is 'movie,' the server <NUM> may select "video content. " In this case, information about an identification value similar to an identification value of each domain of the server <NUM> may be stored in the server <NUM>.

Also, the server <NUM> may generate modified text by using the selected text modification model. The server <NUM> may input the text to the selected text modification model, and may obtain the modified text output from the text modification model. In this case, the server <NUM> may preprocess a format of the text received from the device <NUM> to be suitable for the text modification model, and may input a processed value to the text modification model.

When the text received from the device <NUM> is related to a plurality of domains, the server <NUM> may select a plurality of text modification models corresponding to the plurality of domains for text modification. In this case, the server <NUM> may obtain the modified text to be provided to the device <NUM>, from pieces of modified text output from the plurality of text modification models. For example, when the server <NUM> generates a plurality of pieces of modified text by using the plurality of text modification models, the server <NUM> may compare reliabilities of the plurality of pieces of modified text and may determine modified text having a high or highest reliability as the modified text to be provided to the device <NUM>. A reliability of the modified text may be a value indicating a degree to which the modified text matches an input voice, and may include, for example, but is not limited to, a confidence score.

Also, for example, the server <NUM> may obtain the modified text to be provided to the device <NUM>, by extracting some pieces of text from pieces of modified text output from the plurality of text modification models and combining the extracted some pieces of text. For example, when the server <NUM> generates first modified text and second modified text by using the plurality of text modification models and a reliability of part of the first modified text and a reliability of part of the second modified text are high, the server <NUM> may obtain the modified text to be provided to the device <NUM> by combining the part of the first modified text and the part of the second modified text to generate text of higher reliability than either the first modified text or the second modified text.

In operation S335, the server <NUM> may provide the modified text to the device <NUM>.

Although the device <NUM> requests the server <NUM> for the modified text and the server <NUM> provides the modified text to the device <NUM> in <FIG>, the disclosure is not limited thereto. The server <NUM> may provide various types of voice assistant services to the device <NUM> by using the modified text. A voice assistant service may be a service for providing conversation with a user. In a voice assistant service, a response message may be provided to a user by the voice assistant as if a person directly talks to the user in consideration of the user's situation and a condition of a device. In addition, in a voice assistant service, information required by a user may be appropriately generated and provided to the user as if a personal assistant of the user provides the information. A voice assistant service may provide information or a function required by a user to the user, in conjunction with various services, such as a broadcasting service, a content sharing service, a content providing service, a power management service, a game service, a chat service, a document creating service, a search service, a call service, a photographing service, a transportation recommendation service, and a video playback service.

In this case, the server <NUM> may provide information for performing conversation with the user to the device <NUM>, by using a natural language understanding (NLU) model, a dialog manager (DM) model, a natural language generating (NLG) model, etc. in the server <NUM>. Thereby, the server <NUM> may provide the voice assistant service based on the text. Also, the server <NUM> may directly control another device based on a result obtained by interpreting the text. Also, the server <NUM> may generate control information for enabling the device <NUM> to control another device based on a result obtained by interpreting the modified text, and may provide the generated control information to the device <NUM>.

<FIG> is a diagram illustrating a server identifying a domain related to text and selecting a text modification model of the domain related to the text, according to an embodiment of the disclosure.

Referring to <FIG>, text output from an ASR model <NUM> of the device <NUM> may be provided to the domain identification module <NUM> in the server <NUM>. The server <NUM> may identify a domain related to the text received from the device <NUM>, by using the domain identification module <NUM> in the server <NUM>. In this case, the server <NUM> may identify the domain related to the text, based on a domain reliability of the text received from the device <NUM>. For example, the server <NUM> may calculate a confidence score indicating a degree to which the text received from the device <NUM> is related to a domain that is pre-registered for text modification. The domain identification module <NUM> that is an AI model trained for domain identification may output the domain reliability by using the text as an input value. Also, for example, the domain identification module <NUM> may be a part of an NLU model or a model separate from the NLU model. Alternatively, the domain identification module <NUM> may identify the domain related to the text based on rules.

In <FIG>, the domain identification module <NUM> may obtain, for example, a first domain reliability of a first domain, a second domain reliability of a second domain, and a third domain reliability of a third domain.

Also, a model selection module <NUM> of the server <NUM> may select a text modification model for text modification. For example, the model selection module <NUM> may compare the first domain reliability of the first domain, the second domain reliability of the second domain, and the third domain reliability of the third domain, and may determine that the first domain reliability of the first domain is a highest domain reliability. Also, the model selection module <NUM> may select a text modification model <NUM> of the first domain from among a plurality of text modification models <NUM>, <NUM>, and <NUM> in the server <NUM>.

The server <NUM> may input the text received from the device <NUM> to the text modification model <NUM> of the first domain, and may obtain modified text output from the text modification model <NUM>. Next, the server <NUM> may provide the modified text to the device <NUM>.

<FIG> is a diagram illustrating a device identifying a domain related to text and a server selecting a text modification model of the domain related to the text, according to an embodiment of the disclosure.

Referring to <FIG>, text output from an ASR model <NUM> of the device <NUM> may be provided to the domain identification module <NUM> in the device <NUM>. The device <NUM> may identify a domain related to the text output from the ASR model <NUM>, by using the domain identification module <NUM> in the device <NUM>. In this case, the device <NUM> may identify the domain related to the text, based on a domain reliability of the text output from the ASR model <NUM>. For example, the device <NUM> may calculate a confidence score indicating a degree to which the text output from the ASR model <NUM> is related to a pre-registered domain. The domain identification module <NUM> that is an AI model trained for domain identification may output the domain reliability by using the text as an input value. Also, for example, the domain identification module <NUM> may be a part of an NLU model or a model separate from the NLU model. Alternatively, the domain identification module <NUM> may identify the domain related to the text based on rules.

Also, the device <NUM> may provide the text output from the ASR model <NUM> to the server <NUM>. Also, the device <NUM> may provide the domain reliability obtained by the domain identification module <NUM> to the server <NUM>. Alternatively, the device <NUM> may identify the domain related to the text based on the domain reliability obtained by the domain identification module <NUM>, and may provide identification information of the identified domain to the server <NUM>.

The model selection module <NUM> of the server <NUM> may select a text modification model for text modification. When the device <NUM> provides the domain reliability to the server <NUM>, the model selection module <NUM> may compare, for example, the first domain reliability of the first domain, the second domain reliability of the second domain, and the third domain reliability of the third domain, and may determine that the domain reliability of the first domain is a highest domain reliability. Also, the model selection module <NUM> may select a text modification model <NUM> of the first domain from among a plurality of text modification models <NUM>, <NUM>, and <NUM> in the server <NUM> corresponding to the domain having the highest domain reliability.

Alternatively, when the device <NUM> provides an identification value of the domain related to the text to the server <NUM>, the model selection module <NUM> may select the text modification model <NUM> of the first domain according to a domain identification value received from the device <NUM>.

<FIG> is a diagram illustrating a server and a device identifying a domain related to text and the server selecting a text modification model of the domain related to the text, according to an embodiment of the disclosure.

Referring to <FIG>, text output from an ASR model <NUM> of the device <NUM> may be provided to a first domain identification module <NUM> in the device <NUM>. The first domain identification module <NUM> may be an identification module <NUM>. The device <NUM> may obtain a first domain reliability of the text output from the ASR model <NUM> by using the first domain identification module <NUM> in the device <NUM>. Also, the device <NUM> may provide the text output from the ASR model <NUM> and the first domain reliability obtained from the first domain identification module <NUM> to the server <NUM>.

The server <NUM> may receive the text from the device <NUM>, and may provide the received text to a second domain identification module <NUM> in the server <NUM>. The second domain identification module <NUM> may be the domain identification module <NUM>. The server <NUM> may obtain a second domain reliability of the text received from the device <NUM>, by using the second domain identification module <NUM> in the server <NUM>.

Next, the model selection module <NUM> of the server <NUM> may select a text modification model for text modification, based on the first domain reliability and the second domain reliability. For example, the model selection module <NUM> may select a first domain related to the text from among domains registered in the server <NUM> based on a weighted sum of the first domain reliability and the second domain reliability, and may select a text modification module <NUM> of the selected domain. Also, for example, in this case, because the first domain reliability and the second domain reliability are normalized, weight values may be respectively reflected in the first domain reliability and the second domain reliability, but the disclosure is not limited thereto. A method, performed by the server <NUM>, of selecting a domain related to the text based on the first domain reliability and the second domain reliability will be described in more detail with reference to <FIG>.

<FIG> is a flowchart illustrating a method, performed by a server, of selecting a domain related to text by using a domain reliability obtained by a device and a domain reliability obtained by the server, according to an embodiment of the disclosure.

In operation S700, the server <NUM> may receive, from the device <NUM>, a first domain reliability of text calculated by the first domain identification module <NUM> of the device <NUM>. The first domain identification module <NUM> of the device <NUM> may calculate a confidence score indicating a degree to which the text output from the ASR model <NUM> is related to a pre-registered domain. In this case, for example, the domain identification module <NUM> that is an AI model trained for domain identification may output the first domain reliability by using the text as an input value. When a plurality of domains are registered in the device <NUM>, the first domain identification module <NUM> may obtain a first plurality of domain reliabilities each indicating a degree to which the text is related to each of the plurality of domains.

In operation S710, the server <NUM> may calculate a second domain reliability of the text received from the device <NUM> by using the second domain identification module <NUM>. The second domain identification module <NUM> of the server <NUM> may calculate a confidence score indicating a degree to which the text received from the device <NUM> is related to a pre-registered domain. In this case, for example, the domain identification module <NUM> that is an AI model trained for domain identification may output the second domain reliability by using the text as an input value. When a plurality of domains are registered in the server <NUM>, the second domain identification module <NUM> may obtain a second plurality of domain reliabilities each indicating a degree to which the text is related to each of the plurality of domains.

In operation S720, the server <NUM> may select a domain related to the text, based on the first domain reliability and the second domain reliability. The server <NUM> may select the domain related to the text from among the plurality of domains that are pre-registered based on a weighted sum of the first domain reliability and the second domain reliability. For example, when a domain having a high first domain reliability and a domain having a high second domain reliability are different from each other, the server <NUM> may assign a preset first weight value to the first domain reliability and may assign a preset second weight value to the second domain reliability. Also, the server <NUM> may select the domain related to the text from among the plurality of pre-registered domains, based on the first domain reliability to which the first weight value is assigned and the second domain reliability to which the second weight value is assigned. In this case, for example, because the first domain reliability and the second domain reliability are normalized, weight values may be respectively reflected in the first domain reliability and the second domain reliability, but the disclosure is not limited thereto. For example, when the first domain reliability is a reliability of a domain of a higher layer and the second domain reliability is a reliability of a domain of a lower layer, a low weight value may be assigned to the first domain reliability and a high weight value may be assigned to the second domain.

According to an embodiment of the disclosure, the server <NUM> may consider a reliability of the text output from the ASR model of the device, to select a domain. In this case, the reliability of the text output from the ASR model may be obtained by the device <NUM> and may be provided to the server <NUM>, but the disclosure is not limited thereto. Also, for example, the server <NUM> may assign a preset third weight value to the reliability of the text output from the device <NUM>, and may select a domain related to the text from among the plurality of pre-registered domains, based on the first domain reliability to which the first weight value is assigned, the second domain reliability to which the second weight value is assigned, and the reliability of the text to which the third weight value is assigned.

For example, when a domain having a high first domain reliability and a domain having a high second domain reliability are the same, the server <NUM> may select the domain having the high first domain reliability, from among the plurality of domains, without considering weight values.

<FIG> is a diagram illustrating a server selecting a text modification model by using a domain identification module selected from among a plurality of domain identification modules in the server, according to an embodiment of the disclosure.

Referring to <FIG>, text output from an ASR model <NUM> of the device <NUM> may be provided to a first domain identification module <NUM> in the device <NUM>. The first domain identification module <NUM> may be the domain identification module <NUM>. The device <NUM> may obtain a first domain reliability of the text output from the ASR model <NUM>, by using the first domain identification module <NUM> in the device <NUM>. Also, the device <NUM> may provide the text output from the ASR model <NUM> and the first domain reliability obtained from the first domain identification module <NUM> to the server <NUM>.

The server <NUM> may select one of a plurality of second domain identification modules <NUM> in the server <NUM>, based on the first domain reliability received from the device <NUM>. Domains for speech recognition in a speech recognition system may be hierarchically set. The domains for speech recognition may include, for example, first layer domains, second layer domains that are sub-domains of the first layer domains, third layer domains that are sub-domains of the second layer domains, and fourth layer domains that are sub-domains of the third layer domains. Also, for example, the second domain identification modules <NUM> may include, for example, at least one second layer domain identification module <NUM>-<NUM>, at least one third layer domain identification module <NUM>-<NUM>, and at least one fourth layer domain identification module <NUM>-<NUM>. Also, for example, a first layer domain may correspond to the first domain identification module <NUM>, a second layer domain may correspond to the second layer domain identification module <NUM>-<NUM>, a third layer domain may correspond to the third layer domain identification module <NUM>-<NUM>, a fourth layer domain may correspond to the fourth layer domain identification module <NUM>-<NUM>, and a fifth layer domain may correspond to a text modification model. In this case, the server <NUM> may identify a second layer domain having a high reliability from among a plurality of second layer domains according to the first domain reliability calculated from the first domain identification module <NUM>, by using a domain identification module selection module <NUM>. Also, the domain identification module selection module <NUM> of the server <NUM> may select the second layer domain identification module <NUM>-<NUM> corresponding to the identified second layer domain.

Also, for example, the server <NUM> may obtain a second domain reliability of the text by using the selected second layer domain identification module <NUM>-<NUM>. For example, the domain identification module selection module <NUM> of the server <NUM> may identify a third layer domain having a high reliability from among a plurality of third layer domains based on the second domain reliability, and may select the third layer domain identification module <NUM>-<NUM> corresponding to the identified third layer domain.

Also, for example, the server <NUM> may obtain a third domain reliability of the text by using the selected third layer domain identification module <NUM>-<NUM>. For example, the domain identification module selection module <NUM> of the server <NUM> may identify a fourth layer domain having a high reliability from among a plurality of fourth layer domains based on the third domain reliability, and may select the fourth layer domain identification module <NUM>-<NUM> corresponding to the identified fourth layer domain.

Also, for example, the server <NUM> may obtain a fourth domain reliability of the text by using the selected fourth layer domain identification module <NUM>-<NUM>. The model selection module <NUM> of the server <NUM> may select a text modification model <NUM> corresponding to a third domain from among a plurality of text modification models based on the fourth domain reliability.

Next, the server <NUM> may input the text received from the device <NUM> to the text modification model <NUM> of the third domain, and may obtain modified text output from the text modification model <NUM>. The server <NUM> may provide the modified text to the device <NUM>.

Although layers of the second domain identification modules <NUM> in the server <NUM> include a second layer, a third layer, and a fourth layer, and the server <NUM> sequentially selects the second layer domain identification module <NUM>-<NUM>, the third layer domain identification module <NUM>-<NUM>, and the fourth layer domain identification module <NUM>-<NUM> in <FIG>, the disclosure is not limited thereto.

The server <NUM> may select a domain for text modification, in consideration of the second domain reliability calculated from the second layer domain identification module <NUM>-<NUM>, the third domain reliability calculated from the third layer domain identification module <NUM>-<NUM>, and the fourth domain reliability calculated from the fourth layer domain identification module <NUM>-<NUM>. In this case, the server <NUM> may normalize the second domain reliability calculated from the second layer domain identification module <NUM>-<NUM>, the third domain reliability calculated from the third layer domain identification module <NUM>-<NUM>, and the fourth domain reliability calculated from the fourth layer domain identification module <NUM>-<NUM>, and may select the domain for text modification by comparing the normalized values of the domain reliability scores.

For example, the first domain identification module <NUM> may calculate domain reliabilities of first layer domains, the second layer domain identification module <NUM>-<NUM> may calculate domain reliabilities of second layer domains, the third layer domain identification module <NUM>-<NUM> may calculate domain reliabilities of third layer domains, and the fourth layer domain identification module <NUM>-<NUM> may calculate domain reliabilities of fourth layer domains.

Layers of the second domain identification modules <NUM> in the server <NUM> may include only a second layer. Alternatively, layers of the second domain identification modules <NUM> in the server <NUM> may include other layers in addition to second through fourth layers. In this case, the server <NUM> may include a domain identification module corresponding to each layer, according to the layers of the second domain identification modules <NUM> in the server <NUM>.

For example, the first domain identification module <NUM> may calculate a domain reliability of "location" that is a first layer domain as <NUM>%, and may calculate a reliability of "weather" that is a first layer domain as <NUM>%. The second layer domain identification module <NUM>-<NUM> may calculate a reliability of "Canada" that is a second layer domain as <NUM>%, may calculate a reliability of "USA" that is a second layer domain as <NUM>%, and may calculate a reliability of "Rain" that is a second layer domain as <NUM>%. The third layer domain identification module <NUM>-<NUM> may calculate a reliability of "Brish Columbia" that is a third layer domain as <NUM>%, may calculate a reliability of "Ontario" that is a third layer domain as <NUM>%, may calculate a reliability of "New York" that is a third layer domain as <NUM>%, and may calculate a reliability of "Precipitation" that is a third layer domain as <NUM>%.

Also, for example, a domain selection module <NUM> may assign a first weight value to the domain reliability of "location" that is the first layer domain and the reliability of "weather" that is the first layer domain. The domain selection module <NUM> may assign a second weight value to the reliability of "Canada" that is the second layer domain, the reliability of "USA" that is the second layer domain, and the reliability of "Rain" that is the second layer domain. The domain selection module <NUM> may assign a third weight value to the reliability of "Brish Columbia" that is the third layer domain, the reliability of "Ontario" that is the third layer domain, the reliability of "New York" that is the third layer domain, and the reliability of "Precipitation" that is the third layer domain. In this case, the second weight value may be greater than the first weight value and may be less than the third weight value. Also, the domain selection module <NUM> may select the domain for text modification in consideration of reliabilities to which the first weight value is assigned, reliabilities to which the second weight value is assigned, and reliabilities to which the third weight value is assigned.

Also, for example, the domain selection module <NUM> may calculate a first weighted sum of the reliability of "location," the reliability of "Canada," and the reliability of "Columbia. " The domain selection module <NUM> may calculate a second weighted sum of the reliability of "location," the reliability of "Canada," and the reliability of "Ontario. " The domain selection module <NUM> may calculate a third weighted sum of the reliability of "location," the reliability of "USA," and the reliability of "New York. " Also, for example, the domain selection module <NUM> may calculate a fourth weighted sum of the reliability of "Weather," the reliability of "Rain," and the reliability of "Precipitation.

For example, the domain selection module <NUM> may determine that the first weighted sum is the highest by comparing calculated weighted sums, and may determine "Brish Columbia" as the domain for text modification.

Also, for example, the domain selection module <NUM> may select a second layer domain, based on a domain reliability of first layer domains and a reliability of second layer domains. The domain selection module <NUM> may select sub-domains related to the selected second layer domain, and may modify the text by using text modification models corresponding to the selected sub-domains.

Also, although the first domain identification module <NUM> in the device <NUM> corresponds to a first layer and the second domain identification modules <NUM> in the server <NUM> correspond to second through fourth layers in <FIG>, the disclosure is not limited thereto. For example, the first domain identification module <NUM> in the device <NUM> may correspond to a first layer, and the second domain identification modules <NUM> in the server <NUM> may correspond to first through third layers.

Weight values of domain reliabilities may be assigned based on context information related to the device <NUM>. The context information may include at least one of, but is not limited to, ambient environment information of the device <NUM>, state information of the device <NUM>, state information of a user, device usage history information of the user, or schedule information of the user. The ambient environment information of the device <NUM> that is environment information within a certain radius from the device <NUM> may include, for example, but is not limited to, weather information, temperature information, humidity information, illuminance information, noise information, and sound information. The state information of the device <NUM> may include, but is not limited to, information about a mode of the device <NUM> (e.g., a sound mode, a vibration mode, a silent mode, a power saving mode, a cutoff mode, a multi-window mode, and an automatic rotation mode), position information of the device <NUM>, time information, activation information of a communication module (e.g., Wi-Fi ON, Bluetooth OFF, GPS ON, or NFC ON), network connection state information of the device <NUM>, and information about an application executed in the device <NUM> (e.g., application identification information, an application type, an application usage time, or an application usage cycle). The state information of the user that is information about the user's movement and life pattern may include, but is not limited to, information about the user's walking state, exercising state, driving state, sleep state, and mood state. The device usage history information of the user that is information about events in which the user uses the device <NUM> may include, but is not limited to, information about execution of applications, functions executed by the applications, phone conversations of the user, and text messages of the user.

For example, weight values of domain reliabilities may be determined based on context information about an application being executed on the device <NUM>. For example, when a voice input of the user is an input for the application being executed on the device <NUM>, a high weight value may be assigned to a domain reliability of a domain related to the application. Alternatively, the domain related to the application may be directly determined as the domain for text modification. For example, when a voice input saying "Acrovista" is input while a map application is being executed on the device <NUM>, a high weight value may be assigned to a map domain or the map domain may be directly determined as the domain for text modification.

For example, weight values of domain reliabilities may be determined based on a conversation history of the user through a voice assistant service provided by the device <NUM>. For example, when a voice input saying "Search for IU" is input to the device <NUM> while the user is talking about music to the device <NUM> through the voice assistant service, a high weight value may be assigned to a music domain or the music domain may be directly determined as the domain for text modification.

For example, weight values of domain reliabilities may be determined based on sensing information collected by the device <NUM>. A weight value may be assigned to a domain based on location information (e.g., GPS information) obtained by the device <NUM>. For example, when a location of the device <NUM> is near a movie theater, a high weight value may be assigned to a movie domain. For example, when a voice input of the user is input to the device <NUM> while a restaurant is being searched in the device <NUM>, a high weight value may be assigned to a domain related to a location where the device <NUM> is positioned.

Weight values of domain reliabilities may be assigned based on trend information. For example, a high weight value may be assigned to a domain of main news or a domain of a real-time search word through a portal site.

<FIG> is a flowchart illustrating a method of a server selecting a domain for text modification by using a domain identification module selected from among a plurality of domain identification modules, according to an embodiment of the disclosure.

<FIG> is a diagram illustrating a first domain identification module, a second domain identification module, and a text modification module related to domains that are hierarchically classified according to an embodiment of the disclosure.

In <FIG> and <FIG>, for example, domains for speech recognition may be classified into a first layer, a second layer, and a third layer.

In operation S900, the server <NUM> may receive, from the device <NUM>, a first domain reliability of text calculated by the first domain identification module <NUM> of the device <NUM>. The server <NUM> may receive, from the device <NUM>, the first domain reliability of the text output from an ASR model of the device <NUM>. Referring to <FIG>, for example, a first domain identification module <NUM> of the device <NUM> may correspond to "all" that is a first layer domain related to "location," and a first domain reliability calculated from the first domain identification module <NUM> may be a domain reliability of a second layer domain related to "country. " For example, the first domain reliability may include a domain reliability of a domain "Canada" and a domain reliability of a domain "USA. " Also, for example, a second domain identification module <NUM> may correspond to the domain "Canada," and the second domain identification module <NUM> may correspond to the domain "USA.

In operation S910, the server <NUM> may select at least one of a plurality of second domain identification modules <NUM>, based on the first domain reliability. The domain identification module selection module <NUM> of the server <NUM> may select the second layer domain identification module <NUM>-<NUM> from among the plurality of second domain identification modules <NUM> based on the first domain reliability. Referring to <FIG>, for example, the server <NUM> may compare the domain reliability of "Canada" and the domain reliability of domain "USA," and may determine that the domain reliability of "Canada" is higher than a certain threshold value. Also, the server <NUM> may select the second domain identification module <NUM> corresponding to "Canada" from among the second domain identification module <NUM> corresponding to the domain "Canada" and the second domain identification module <NUM> corresponding to the domain "USA. " In this case, the second domain identification module <NUM> and the second domain identification module <NUM> may be second domain identification modules corresponding to the second layer.

In operation S920, the server <NUM> may calculate a second domain reliability of the text by using the selected second layer domain identification module <NUM>-<NUM>. The second layer domain identification module <NUM>-<NUM> may calculate the second domain reliability by using the text as an input. Referring to <FIG>, the second domain reliability calculated from the second domain identification module <NUM> may be a domain reliability of a third layer domain related to "province or state. " For example, the second domain reliability may include a domain reliability of a domain "British Columbia," a domain reliability of a domain "Ontario," a domain reliability of a domain "New York," and a domain reliability of a domain "Illinois. " Also, for example, a text modification model <NUM> may correspond to the domain "British Columbia," a text modification model <NUM> may correspond to the domain "Ontario," a text modification model <NUM> may correspond to the domain "New York," and a text modification model <NUM> may correspond to the domain "Illinois.

In operation S930, the server <NUM> may select a domain related to text modification, based on the second domain reliability. The model selection module <NUM> of the server <NUM> may select one of a plurality of text modification models <NUM>, <NUM>, and <NUM> based on the second domain reliability. Referring to <FIG>, for example, the server <NUM> may compare the domain reliability of the domain "British Columbia," the domain reliability of the domain "Ontario," the domain reliability of the domain "New York," and the domain reliability of the domain "Illinois," and may determine that the domain reliability of the domain "British Columbia" is higher than a certain threshold value. Also, the server <NUM> may select the domain "British Columbia" as a domain for text modification. Accordingly, the text may be modified by the text modification model <NUM> corresponding to the domain "British Columbia.

Although the first domain identification module <NUM> corresponds to the first layer, the second domain identification modules <NUM> correspond to the second layer, and the text modification models <NUM>, <NUM>, and <NUM> correspond to the third layer in <FIG> and <FIG>, the disclosure is not limited thereto. For example, the second domain identification modules <NUM> may correspond to more layers. For example, the first domain identification module <NUM> may correspond to the first layer, the second domain identification modules <NUM> may correspond to the second layer, the third layer, and a fourth layer, and the text modification models <NUM>, <NUM>, and <NUM> may correspond to a fifth layer, but the disclosure is not limited thereto.

In <FIG>, the server <NUM> may normalize a domain reliability calculated from the first domain identification module <NUM>, a domain reliability calculated from the second domain identification module <NUM>, and a domain reliability calculated from the second domain identification module <NUM>, and may select a text modification model for text modification by comparing normalized values.

<FIG> is a diagram illustrating a server modifying text by using a plurality of text modification models, according to an embodiment of the disclosure.

Referring to <FIG>, text output from an ASR model <NUM> of the device <NUM> may be provided to the domain identification module <NUM> in the server <NUM>. The server <NUM> may identify a domain related to the text received from the device <NUM>, by using the domain identification module <NUM> in the server <NUM>. In this case, the server <NUM> may identify the domain related to the text based on a domain reliability of the text received from the device <NUM>.

The domain identification module <NUM> may obtain, for example, a domain reliability of a first domain, a domain reliability of a second domain, and a domain reliability of a third domain. For example, the domain identification module <NUM> may divide the text into a plurality of sections, and may obtain the domain reliability of the first domain, the domain reliability of the second domain, and the domain reliability of the third domain for each section. For example, the domain identification module <NUM> may classify the text into a first section, a second section, and a third section, and may obtain the domain reliability of the first domain, the domain reliability of the second domain, and the domain reliability of the third domain for each of the first section, the second section, and the third section. For example, when the server <NUM> receives a text stream, the server <NUM> may obtain first domain reliabilities of text of the first section, may obtain second domain reliabilities of text of the second section, and may obtain third domain reliabilities of text of the third section. In this case, while receiving a text stream, the server <NUM> may modify text that is already received in real time, without waiting for text that is to be received later, and may identify a domain related to the text at a higher speed.

Alternatively, for example, when the server <NUM> receives a text stream, the server <NUM> may accumulate and calculate domain reliabilities of text of a plurality of sections. For example, while receiving a text stream, the server <NUM> may divide text of one sentence into a plurality of sections, and may obtain domain reliabilities of text of a first section, may obtain domain reliabilities of text of first and second sections, and may obtain domain reliabilities of text of first through third sections. In this case, because the server <NUM> calculates domain reliabilities in units of sentences by accumulating a plurality of sections, the server <NUM> may more efficiently identify a domain related to text.

Alternatively, for example, for text of a sentence unit, the domain identification module <NUM> may obtain a first domain reliability of a first domain, a second domain reliability of a second domain, and a third domain reliability of a third domain.

According to an embodiment of the disclosure, the model selection module <NUM> of the server <NUM> may select a text modification model for text modification. For example, when the text is divided into a plurality of sections, the server <NUM> may select different text modification models according to the sections of the text. For example, when the text is divided into a first section, a second section, and a third section, the server <NUM> may select a text modification model <NUM> to modify the first section of the text, may select a text modification model <NUM> to modify the second section of the text, and may select a text modification model <NUM> to modify the third section of the text.

Alternatively, for example, the server <NUM> may select a plurality of text modification models to modify text of a sentence unit. For example, the server <NUM> may select the text modification model <NUM>, the text modification model <NUM>, and the text modification model <NUM> to modify the text of the sentence unit.

Next, the server <NUM> may obtain modified text for the text received from the device <NUM>, by using first modified text output from the text modification model <NUM>, second modified text output from the text modification model <NUM>, and third modified text output from the text modification model <NUM>.

For example, the server <NUM> may select at least one of the first modified text, the second modified text, or the third modified text, and may obtain the modified text for the text received from the device <NUM> by using any portions of the selected modified text. Alternatively, for example, the server <NUM> may obtain the modified text for the text received from the device <NUM>, by selecting one of the first modified text, the second modified text, and the third modified text. Alternatively, for example, the server <NUM> may obtain the modified text for the text received from the device <NUM>, by combining at least part of the first modified text, at least part of the second modified text, and at least part of the third modified text.

Next, the server <NUM> may provide the modified text to the device <NUM>.

<FIG> is a flowchart illustrating a method of a server accumulating and calculating domain reliabilities of text of a plurality of sections, according to an embodiment of the disclosure.

In operation S1200, the server <NUM> may obtain a first section of text. The text may be divided into a plurality of sections, and the sections of the text may be divided, for example, in units of syntactic words, words, or phrases. The server <NUM> may receive the text from the device <NUM> as a text stream. In this case, the server <NUM> may obtain the first section of the text while receiving the text stream in real time. Alternatively, the server <NUM> may receive the text as a sentence from the device <NUM>, and may extract text of the first section from the received text.

In operation S1210, the server <NUM> may calculate a domain reliability of the text of the first section. The server <NUM> may calculate the domain reliability of the text of the first section, for domains registered in the server <NUM>.

In operation S1220, the server <NUM> may obtain a second section of the text. When the server <NUM> receives the text as a text stream from the device <NUM>, the server <NUM> may obtain the second section of the text while receiving the text stream in real time. Alternatively, the server <NUM> may receive the text as a sentence from the device <NUM>, and may extract text of the second section from the received text.

In operation S1230, the server <NUM> may calculate domain reliabilities of the text of the first and second sections. The server <NUM> may accumulate the text of the first section and the text of the second section, and may calculate domain reliabilities of the text of the first section and the text of the second section.

In operation S1240, the server <NUM> may obtain an nth section of the text. When the server <NUM> receives the text as a text stream from the device <NUM>, the server <NUM> may obtain the nth section of the text while receiving the text stream in real time. Alternatively, the server <NUM> may receive the text as a sentence from the device <NUM>, and may extract text of the nth section from the received text.

In operation S1250, the server <NUM> may calculate domain reliabilities of text of first through nth sections. The server <NUM> may accumulate the text of the first section through the text of the nth section, and may calculate domain reliabilities of the text of the first section through the text of the nth section.

In operation S1260, the server <NUM> may determine a domain for modifying the text received from the device <NUM>, based on the domain reliabilities of the text of the first through nth sections.

<FIG> is a diagram illustrating a server obtaining domain reliabilities of a text stream accumulated in units of syntactic words, according to an embodiment of the disclosure.

Referring to <FIG>, the domain identification module <NUM> of the server <NUM> may divide text in units of syntactic words, may accumulate the divided text, and may obtain domain reliabilities of the accumulated text.

For example, when "new" that is text of a first section is input to the domain identification module <NUM>, the domain identification module <NUM> may output "Rejected" as a domain identification value because a domain reliability related to "new" is "<NUM>" that is a low value.

Next, when "TWICE" that is text of a second section is input to the domain identification module <NUM>, the domain identification module <NUM> may accumulate "new" that is the text of the first section and "TWICE" that is the text of the second section, and may output "Music" as a domain identification value related to "new TWICE" that is accumulated text and "<NUM>" as a domain reliability.

Next, when "Yes or No" that is text of a third section is input to the domain identification module <NUM>, the domain identification module <NUM> may accumulate "new" that is the text of the first section, "TWICE" that is the text of the second section, and "Yes or No" that is the text of the third section, and may output "Music" as a domain identification value related to "new TWICE Yes or No" that is accumulated text and "<NUM>" as a domain reliability.

Next, when "Play" that is text of a fourth section is input to the domain identification module <NUM>, the domain identification module <NUM> may accumulate "new" that is the text of the first section, "TWICE" that is the text of the second section, "Yes or No" that is the text of the third section, and "Play" that is the text of the fourth section, and may output "Music" as a domain identification value related to "Play new TWICE Yes or No" that is accumulated text and "<NUM>" as a domain reliability.

Accordingly, the server <NUM> may select a domain modification model of a domain "Music," as a domain modification model for modifying "Play new TWICE Yes or No.".

Although the domain identification module <NUM> outputs a domain and a domain reliability having a highest value in <FIG>, the disclosure is not limited thereto. The domain identification module <NUM> may output a domain reliability of each of a plurality of domains registered in the server <NUM>.

Although domain reliabilities of text of a plurality of sections are accumulated and calculated in <FIG>, the disclosure is not limited thereto. For example, the server <NUM> may sequentially select text modification models for modifying text of a plurality of sections. For example, when the server <NUM> receives a text stream, the server <NUM> may select a text modification model for modifying text of a first section by calculating a domain reliability of the text of the first section, may select a text modification model for modifying text of a second section by calculating a domain reliability of the text of the second section, and may select a text modification model for modifying text of an nth section by calculating a domain reliability of the text of the nth section, while receiving the text stream.

<FIG> is a flowchart illustrating a method of a server dividing text into a plurality of sections and selecting a domain of text of each of the plurality of sections, according to an embodiment of the disclosure.

In operation S1400, the server <NUM> may calculate a domain reliability of text for each of a plurality of domains. The server <NUM> may calculate domain reliabilities of a plurality of domains registered in the server <NUM> for the text received from the device <NUM>.

In operation S1410, the server <NUM> may divide the text into a plurality of sections, by comparing the calculated domain reliabilities. The server <NUM> may divide the text into the plurality of sections, by identifying a text section having a high domain reliability for each domain.

In operation S1420, the server <NUM> may select a domain for text modification for text of each section. The server <NUM> may select a domain having a highest domain reliability for text of each section, as a domain corresponding to the text of each section.

<FIG> is a diagram illustrating an example of a server comparing domain reliabilities of text according to a plurality of domains and selecting and modifying a text modification model of text of each section, according to an embodiment of the disclosure.

Referring to <FIG>, the server <NUM> may divide text into a plurality of sections while receiving a text stream, and may calculate a domain reliability of text of each section in real time. For example, the server <NUM> may receive a text stream saying "I will meet Gil Hong today near Yeomtong Station to see the A vengers. " The server <NUM> may identify "today near Yeomtong Station" as a first section, may identify "I will meet Gil Hong" as a second section, and may identify "to see the A vengers" as a third section, while receiving the text stream. Also, while receiving the text stream, the server <NUM> may sequentially calculate domain reliabilities related to "today near Yeomtong Station," domain reliabilities related to "I will meet Gil Hong," and domain reliabilities related to "to see the A vengers.

For example, for each section of the text saying "I will meet Gil Hong today near Yeomtong Station to see the A vengers," the domain identification module <NUM> of the server <NUM> may calculate a domain reliability of a domain "Movie," a domain reliability of a domain "location," and a domain reliability of a domain "contacts.

For example, the domain selection module <NUM> of the server <NUM> may compare the domain reliability of the domain "Movie," the domain reliability of the domain "location," and the domain reliability of the domain "contacts. " The domain selection module <NUM> may determine that the domain reliability of the domain "location" is high for "today near Yeomtong Station. " The domain selection module <NUM> may determine that the domain reliability of the domain "contacts" is high for "I will meet Gil Hong. " The domain selection module <NUM> may determine that the domain reliability of the domain "Movie" is high for "to see the A vengers. " Accordingly, the domain selection module <NUM> may sequentially identify "today near Yeomtong Station" as a first section, may identify "I will meet Gil Hong" as a second section, and may identify "to see the A vengers" as a third section, from the text stream saying "I will meet Gil Hong today near Yeomtong Station to see the A vengers.

Also, for example, the domain selection module <NUM> may select a domain related to "today near Yeomtong Station" as the domain "location," may select a domain related to "I will meet Gil Hong" as the domain "contacts," and may select a domain related to "to see the A vengers" as the domain "Movie.

A text modification model of the domain "location" may modify "today near Yeomtong Station" to "today Yeongtong Station," a text modification model of the domain "contacts" may modify "I will meet Gil Hong" to "I will meet Gil Dong," and a text modification model of the domain "Movie" may modify "to see the A vengers" to "to see the Avengers. " At least one of a text modification operation of the text modification model of the domain "location," a text modification operation of the text modification model of the domain "contacts," or a text modification operation of the text modification model of the domain "Movie" may be sequentially performed while the text stream is received.

<FIG> is a diagram illustrating a server modifying text received from a device by using modified text output from a plurality of text modification models, according to an embodiment of the disclosure.

Referring to <FIG>, for example, the server <NUM> may provide text <NUM> saying "I will meet Gil Hong today near Yeomtong Station to see the A vengers" to a text modification model of a domain "location," a text modification model of a domain "contacts," and a text modification model of a domain "Movie.

Accordingly, the text modification model of the domain "location" may output modified text <NUM> saying "I will meet Gil Hong today near Yeongtong Station to see the A vengers," the text modification model of the domain "movie" may output modified text <NUM> saying "I will meet Gil Hong today near Yeomtong Station to see the Avengers," and the text modification model of the domain "contacts" may output modified text <NUM> saying "I will meet Gil Dong today near Yeomtong Station to see the A vengers.

Next, the server <NUM> may identify "Yeongtong" that is a modified word in the modified text <NUM>, "the Avengers" that is a modified word in the modified text <NUM>, and "Gil Dong" that is a modified word in the modified text <NUM>, and may generate "I will meet Gil Dong today near Yeongtong Station to see the Avengers" that is modified text <NUM> to be provided to the device <NUM>.

<FIG> is a block diagram illustrating a server according to an embodiment of the disclosure.

Referring to <FIG>, the server <NUM> according to an embodiment of the disclosure may include a communication interface <NUM>, a processor <NUM>, and a storage <NUM>, and the storage <NUM> may include the domain management module <NUM>, the text modification module <NUM>, the NLU module <NUM>, and the voice analysis management module <NUM>.

The communication interface <NUM> may include at least one element for communication with the device <NUM> and another server. The communication interface <NUM> may transmit/receive information for speech recognition and a voice assistant service to/from the device <NUM> and another server. The communication interface <NUM> may perform communication via, for example, but not limited to, a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof.

The processor <NUM> controls an overall operation of the server <NUM>. The processor <NUM> may generally control operations of the server <NUM> described herein, by executing programs stored in the storage <NUM>.

The storage <NUM> may store a program used by the processor <NUM> to perform processing and control, and may store data input to the server <NUM> or output from the server <NUM>. The storage <NUM> may include at least one type of storage medium of, but not limited to, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., a secure digital (SD) or extreme digital (XD) memory), a random-access memory (RAM), a static random-access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, or an optical disk.

The programs stored in the storage <NUM> may be classified into a plurality of modules according to their functions, for example, the domain management module <NUM>, the text modification module <NUM>, the NLU module <NUM>, and the voice analysis management module <NUM>.

The domain management module <NUM> provides text received from the device <NUM> to the text modification module <NUM>. The domain management module <NUM> may include the domain identification module selection module <NUM>, at least one domain identification module <NUM>, and the domain selection module <NUM>.

The domain identification module selection module <NUM> may select the domain identification module <NUM>. When there are a plurality of domain identification modules <NUM>, the domain identification module selection module <NUM> may select at least some of the plurality of domain identification modules <NUM>.

The domain identification module selection module <NUM> may select one of the plurality of domain identification modules <NUM> in the server <NUM>, based on a first domain reliability received from the device <NUM>. Domains for speech recognition in a speech recognition system may be hierarchically set. The domains for speech recognition may include, for example, first layer domains, second layer domains that are sub-domains of the first layer domains, and third layer domains that are sub-domains of the second layer domains. Also, for example, a first layer domain may correspond to the domain identification module <NUM> of the device <NUM>, a second layer domain may correspond to the domain identification module <NUM> of the server <NUM>, and a third layer domain may correspond to the text modification module <NUM>. In this case, the domain identification module selection module <NUM> may identify a second layer domain having a high reliability from among a plurality of second layer domains according to the first domain reliability calculated from the domain identification module <NUM> of the device <NUM>. Also, the domain identification module selection module <NUM> may select the domain identification module <NUM> corresponding to the identified second layer domain.

The domain identification module <NUM> may identify a domain for text modification. When the server <NUM> receives domain information from the device <NUM>, the domain identification module <NUM> may identify the domain for text modification from the domain information. Alternatively, when the server <NUM> does not receive the domain information from the device <NUM>, the domain identification module <NUM> may identify a domain related to the text, based on a domain reliability of the text received from the device <NUM>. For example, the domain identification module <NUM> may calculate a confidence score indicating a degree to which the text received from the device <NUM> is related to a domain that is pre-registered for text modification. Also, the domain identification module <NUM> may identify the domain related to the text received from the device, based on a domain reliability calculated for the domain that is pre-registered. The domain identification module <NUM> may identify the domain related to the text based on rules or may obtain the domain reliability related to the text by using an AI model trained for domain identification. Also, for example, the AI model for domain identification may be a part of an NLU model or a model separate from the NLU model.

The domain identification module <NUM> may identify the domain related to the text, by accumulating text of a plurality of sections and calculating domain reliabilities of the accumulated text. Alternatively, the domain identification module <NUM> may divide the text into a plurality of sections and may identify a domain related to text of each section.

The domain selection module <NUM> may select a text modification model corresponding to the domain identified by the domain identification module <NUM>, from among a plurality of text modification models <NUM>, <NUM>, and <NUM>.

The domain selection module <NUM> may select a domain corresponding to the domain identified by the domain identification module <NUM> from among domains registered in the server <NUM>, and may select a text modification model of the selected domain.

When the domain identification module <NUM> divides the text into a plurality of sections and identifies a domain related to text of each section, the domain selection module <NUM> may select a domain for text of each section.

The text modification module <NUM> modifies the text received from the device <NUM>. The text modification module <NUM> may modify the text by using a text modification model corresponding to the determined domain. The text modification module <NUM> may include the text modification model <NUM> of a first domain, the text modification model <NUM> of a second domain, and the text modification model <NUM> of a third domain.

The text modification module <NUM> may generate modified text by using the selected text modification model. The text modification module <NUM> may input the text to the selected text modification model, and may obtain the modified text output from the text modification model. In this case, the text modification module <NUM> may preprocess a format of the text received from the device <NUM> to be suitable for processing the text modification model, and may input a preprocessed value to the text modification model.

When the text received from the device <NUM> is related to a plurality of domains, the text modification module <NUM> may select a plurality of text modification models corresponding to the plurality of domains for text modification. In this case, the text modification module <NUM> may obtain the modified text to be provided to the device <NUM>, from pieces of modified text output from the plurality of text modification models. For example, when the text modification module <NUM> generates a plurality of pieces of modified text by using the plurality of text modification models, the text modification module <NUM> may compare reliabilities of the plurality of pieces of modified text and may determine modified text having a high or highest reliability as the modified text to be provided to the device <NUM>. A reliability of the modified text may be a value indicating a degree to which the modified text matches an input voice, and may include, for example, but is not limited to, a confidence score.

Also, for example, when the text modification module <NUM> generates a plurality of pieces of modified text by using the plurality of text modification models, the text modification module <NUM> may extract modified parts from the plurality of pieces of modified text and may obtain the modified text to be provided to the device <NUM> by using the extracted modified parts.

Also, for example, the text modification module <NUM> may obtain the modified text to be provided to the device <NUM>, by extracting some pieces of text from pieces of modified text output from the plurality of text modification models and combining the extracted some pieces of text. For example, when the text modification module <NUM> generates first modified text and second modified text by using the plurality of text modification models and a reliability of part of the first modified text and a reliability of part of the second modified text are high, the text modification module <NUM> may obtain the modified text to be provided to the device <NUM> by combining the part of the first modified text and the part of the second modified text to generate modified text having a reliability higher than the first modified text and the second modified text.

Also, for example, when a domain related to text of each section is selected, the text modification module <NUM> may provide the text of each section to a corresponding domain modification model. In this case, the text modification module <NUM> may obtain the modified text to be provided to the device <NUM> by combining pieces of modified text of sections output from respective domain modification models.

The NLU module <NUM> may interpret the modified text output from the text modification module <NUM>. The NLU module <NUM> may include a plurality of NLU models of a plurality of domains, such as a first NLU model <NUM> and a second NLU model <NUM>. A result value generated when the NLU module <NUM> interprets the text may include, for example, intent and parameters. The intent that is information determined by interpreting the text by using an NLU model may indicate, for example, a user's utterance intention. The intent may include information indicating the user's utterance intention (hereinafter, referred to as intention information) and also a numerical value corresponding to the information indicating the user's intention. The numerical value may indicate a probability that the text is related to information indicating a specific intention. When a plurality of pieces of intention information indicating the user's intention are obtained as a result obtained by interpreting the text by using the NLU model, intention information having a largest numerical value from among the plurality of intention information may be determined as the intent. Also, the parameters may indicate detailed information related to the intent. The parameters may be information related to the intent, and a plurality of types of parameters may correspond to one intent.

Also, a result value generated when the NLU module <NUM> interprets the text may be used to provide a certain voice assistant service to the device <NUM>.

The voice analysis management module <NUM> may evaluate the text modified by the text modification module <NUM>, and may determine whether to perform NLU processing on the modified text. The voice analysis management module <NUM> may include a speech recognition evaluation module <NUM> and an NLU determination module <NUM>.

The speech recognition evaluation module <NUM> may calculate a reliability of the text modified by the text modification module <NUM>. The reliability of the modified text may be a value indicating a probability that the modified text matches an input voice, and may include, for example, but is not limited to, a confidence score. Also, the speech recognition evaluation module <NUM> may calculate a domain reliability of the modified text. The speech recognition evaluation module <NUM> may calculate the domain reliability indicating a degree to which the modified text is related to a domain that is pre-registered in the server <NUM> for NLU processing.

The NLU determination module <NUM> may determine whether to perform NLU processing on the modified text in the server <NUM>. The NLU determination module <NUM> may determine whether to perform NLU processing in the server <NUM>, based on the reliability of the modified text and the domain reliability of the modified text. The NLU determination module <NUM> may determine whether a domain related to the modified text is a domain for performing NLU processing in the device <NUM> or a domain for performing NLU processing in the server <NUM>.

<FIG> is a block diagram illustrating a device according to an embodiment of the disclosure.

Referring to <FIG>, the device <NUM> according to an embodiment of the disclosure may include a communication interface <NUM>, an input/output interface <NUM>, a processor <NUM>, and a memory <NUM>, and the memory <NUM> may include at least one ASR model <NUM>, at least one NLU model <NUM>, the speech recognition evaluation module <NUM>, the domain identification module <NUM>, and the NLU determination module <NUM>.

The communication interface <NUM> may include at least one component for communication with the server <NUM> and an external device. The communication interface <NUM> may transmit/receive information for speech recognition and a voice assistant service to/from the server <NUM> and the external device. The communication interface <NUM> may perform communication via, for example, but not limited to, a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof.

The input/output interface <NUM> may receive data input to the device <NUM>, and may output data from the device <NUM>. The input/output interface <NUM> may include a user input interface, a camera, a microphone, a display, and an audio output interface. The user input interface may include, but is not limited to, a key pad, a dome switch, a touchpad (e.g., a capacitive overlay type, a resistive overlay type, an infrared beam type, an integral strain gauge type, a surface acoustic wave type, a piezoelectric type, or the like), a jog wheel, or a jog switch.

The display may display and output information processed by the device <NUM>. For example, the display may display a graphical user interface (GUI) for a voice assistant service. When the display forms a layer structure together with a touchpad to construct a touchscreen, the display may be implemented as an input device as well as an output device. The display may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT-LCD), an organic light-emitting diode (OLED), a flexible display, a three-dimensional (3D) display, or an electrophoretic display.

The audio output interface may output audio data and may include, for example, a speaker and a buzzer.

The camera may obtain an image frame,, such as a still image or a moving picture, by using an image sensor in a video call mode or a photographing mode. An image captured by the image sensor may be processed by the processor <NUM> or a separate image processor.

The microphone may receive a user's utterance and may process the user's utterance into electrical voice data.

The processor <NUM> controls an overall operation of the device <NUM>. The processor <NUM> may control the overall operation of the device <NUM> described herein, by executing programs stored in the memory <NUM>.

The memory <NUM> may store a program used by the processor <NUM> to perform processing and control, and may store data input to or output from the device <NUM>. The memory <NUM> may include at least one type of storage medium of, but not limited to, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory), a random-access memory (RAM), a static random-access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, or an optical disk.

The programs stored in the memory <NUM> may be classified into a plurality of modules according to their functions, for example, the ASR model <NUM>, the NLU model <NUM>, the speech recognition evaluation module <NUM>, the domain identification module <NUM>, and the NLU determination module <NUM>.

The ASR model <NUM> may obtain text from a feature vector generated from a voice input of a user. The processor <NUM> of the device <NUM> may input the feature vector to the ASR model <NUM> to recognize the user's voice. When a plurality of ASR models <NUM> are included in the device <NUM>, the processor <NUM> of the device <NUM> may select one of the plurality of ASR models <NUM> and may convert the feature vector into a format suitable for the selected ASR model <NUM>. The ASR model <NUM> may be an AI model including, for example, an acoustic model, a pronunciation dictionary, and a language model. Alternatively, the ASR model <NUM> may be an end-to-end speech recognition model having a structure including an integrated neural network without separately including, for example, an acoustic model, a pronunciation dictionary, and a language model.

The NLU model <NUM> may interpret the text output from the ASR model <NUM>. Alternatively, the NLU model <NUM> may interpret modified text provided from the server <NUM>. A result value generated when the NLU model <NUM> interprets the text or the modified text may be used to provide a certain voice assistant service to the user.

The speech recognition evaluation module <NUM> may obtain a reliability of the text output from the ASR model <NUM>. The reliability of the text may be a value indicting a degree to which the text output from the ASR model <NUM> is related to an input voice, and may include, for example, but not limited to, a confidence score. Also, the reliability of the text may be related to a probability that the text will match the input voice. For example, the reliability of the text may be calculated based on at least one of likelihoods of a plurality of pieces of estimated text output from the ASR model <NUM> of the device <NUM> or posterior probabilities that at least one character in the text will be replaced by another character. For example, the speech recognition evaluation module <NUM> may calculate the reliability based on a likelihood output as a result of Viterbi decoding. Alternatively, for example, the speech recognition evaluation module <NUM> may calculate the reliability based on posterior probabilities output from a softmax layer in the end-to-end ASR model. Alternatively, for example, the speech recognition evaluation module <NUM> may determine a plurality of pieces of estimated text estimated in a speech recognition process of the ASR model <NUM> of the device <NUM>, and may calculate the reliability of the text based on a correlation of characters in the plurality of pieces of estimated text.

Also, the speech recognition evaluation module <NUM> may determine whether to transmit the text output from the ASR model <NUM> to the server <NUM>. The speech recognition evaluation module <NUM> may determine whether to transmit the text to the server <NUM>, by comparing the reliability of the text with a preset threshold value. When the reliability of the text is equal to or greater than the preset threshold value, the speech recognition evaluation module <NUM> may determine not to transmit the text to the server <NUM>. Also, when the reliability of the text is less than the preset threshold value, the speech recognition evaluation module <NUM> may determine to transmit the text to the server <NUM>.

Also, the speech recognition evaluation module <NUM> may determine whether to transmit the text to the server <NUM>, based on at least one text having a high reliability from among the plurality of pieces of estimated text estimated in the speech recognition process of the ASR model <NUM>. For example, when the plurality of pieces of estimated text estimated in the speech recognition process of the ASR model <NUM> include first estimated text having a high reliability and second estimated text having a high reliability, and a difference between the reliability of the first estimated text and the reliability of the second estimated text is equal to or less than a certain threshold value, the speech recognition evaluation module <NUM> may determine to transmit the text to the server <NUM>.

The domain identification module <NUM> may identify a domain related to the text output from the ASR model <NUM>. The domain identification module <NUM> may identify the domain related to the text, based on a domain reliability of the text output from the ASR model <NUM>. For example, the domain identification module <NUM> may calculate a confidence score indicating a degree to which the text output from the ASR model <NUM> is related to a pre-registered domain. The domain identification module <NUM> that is an AI model trained for domain identification may output the domain reliability by using the text as an input value. Also, for example, the domain identification module <NUM> may be a part of an NLU model, or a model separate from the NLU model. Alternatively, the domain identification module <NUM> may identify the domain related to the text based on rules.

Domains for speech recognition in a speech recognition system may be hierarchically set, and the domain identified by the domain identification module <NUM> of the device <NUM> may be a domain of a higher layer than a domain identified by the domain identification module <NUM> of the server <NUM>.

The NLU determination module <NUM> may determine whether to perform NLU processing on the text output from the ASR model <NUM> in the device <NUM> or in the server <NUM>. The NLU determination module <NUM> may determine whether the domain related to the text output from the ASR model <NUM> is a domain for performing NLU processing in the device <NUM>. When the domain related to the text output from the ASR model <NUM> is a domain pre-registered in the device <NUM>, the NLU determination module <NUM> may determine that the device <NUM> will perform NLU processing. Also, when the domain related to the text output from the ASR model <NUM> is not a domain pre-registered in the device <NUM>, the NLU determination module <NUM> may determine that the device <NUM> will not perform NLU processing.

An Al-related function according to the disclosure is performed through a processor and a memory. The processor may include at least one processor. In this case, the at least one processor may include a general-purpose processor, such as a central processing unit (CPU), an access point (AP), or a digital signal processor (DSP), a graphics processor, such as a graphics processing unit (GPU) or a vision processing unit (VPU), or an AI processor, such as a neural processing unit (NPU). The at least one processor controls input data to be processed according to a pre-defined operation rule or AI model stored in the memory. Alternatively, when the at least one processor is an AI processor, the AI processor may be designed to have a hardware structure specialized to process a specific AI model.

The pre-defined operation rule or AI model is created through learning and training. When the pre-defined operation rule or AI model is created through learning, it means that the pre-defined operation rule or AI model set to achieve desired characteristics (or purposes) is created when a basic AI model is trained by using a plurality of training data by using a learning algorithm. The learning may be performed by a device itself in which Al according to the disclosure is used or may be performed through a separate server and/or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

The AI model may include a plurality of neural network layers. The plurality of neural network layers may respectively have a plurality of weight values, and each perform a neural network operation through computation on the plurality of weight values and a computation result of a previous layer. The plurality of weight values of the plurality of neural network layers may be optimized by a training result of the Al model. For example, during a learning process, the plurality of weight values may be refined to reduce or minimize a loss value or a cost value obtained by the Al model. An artificial neural network may include a deep neural network (DNN), and examples of the artificial neural network may include, but are not limited to, a convolutional neural network (CNN), a DNN, a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and a deep Q-network.

Some embodiments of the disclosure may be implemented as a recording medium including computer-executable instructions, such as a computer-executable program module. A computer-readable medium may be an arbitrary available medium accessible by a computer, and examples thereof include all volatile and non-volatile media and separable and non-separable media. Further, examples of the computer-readable medium may include a computer storage medium and a communication medium. Examples of the computer storage medium include all volatile and non-volatile media and separable and non-separable media, which are implemented by an arbitrary method or technology, for storing information, such as computer-readable instructions, data structures, program modules, or other data. The communication medium typically includes computer-readable instructions, data structures, program modules, or other data of a modulated data signal.

Also, the term "unit" used herein may be a hardware component, such as a processor or a circuit and/or a software component executed in a hardware component, such as a processor.

Claim 1:
A method of a server modifying a speech recognition result provided from a device, the method comprising:
receiving, from the device, output text from an automatic speech recognition, ASR, model of the device;
classifying the output text into a plurality of sections;
identifying a plurality of domains related to subject matter of the output text, the plurality of domains being respectively related to the plurality of sections;
selecting, a plurality of text modification models corresponding to the plurality of domains, wherein each of the plurality of text modification models is an artificial intelligence, Al, model trained to analyze text related to the subject matter;
modifying the output text to generate modified text using the plurality of text modification models; and
providing the modified text to the device,
wherein the modifying of the output text comprises respectively modifying pieces of text of the plurality of sections, by using the plurality of text modification models, wherein the modified text comprises at least one modified character compared to the text received from the device,
the method being characterized in that:
the identifying of the plurality of domains comprises:
receiving, from the device, a first domain reliability of the received text calculated by the device;
calculating a second domain reliability of the received text; and
identifying the plurality of domains related to the subject matter of the received text, based on the first domain reliability and the second domain reliability.