Patent Description:
Speech recognition is a process for converting an audio signal of speech uttered by a user into text information. An electronic apparatus for speech recognition transduces or otherwise converts an audio signal into a digital signal, and inputs the digital signal to a speech recognition model. The apparatus then obtains text information corresponding to the utterance of the user from the speech recognition model.

In order to train a speech recognition model to convert the digital signal to text information, a user's voice may be analyzed on the basis of prior recorded utterances, typically on the order of <NUM> to <NUM> words. The speech recognition model may develop and use a plurality of weights or parameters to analyze the utterances of the user.

If a space for storage of a preset word, weight, or parameter used in the speech recognition model is unavailable, the speech recognition model must instead be kept in a local or short-term memory. If there is limited memory for the speech recognition model, the processing speed may slow, possibly to impractical levels.

For example, when a speech recognition model is implemented as an on-device type stored in a terminal device (e.g., a smartphone) of a user, there may be a problem in that limitation of memory usage and storage space occurs, resulting in an inconvenient or even ineffectual speed of operation. Prior art includes a paper titled "<NPL>.

Provided are an electronic apparatus in which different weight information used in a speech recognition model partially shares data, and a control method thereof.

According to the present invention, there is provided an electronic apparatus according to claim <NUM>.

Optional features of the electronic apparatus are provided in dependent claims <NUM> to <NUM>.

According to the present invention, there is provided a method according to claim <NUM> of controlling an electronic apparatus storing a speech recognition model and first recognition information corresponding to a first user voice obtained through the speech recognition model, the speech recognition model including a first network, a second network, and a third network.

Optional features of the method are provided in dependent claims <NUM> to <NUM>.

The disclosure will be described in greater detail with reference to the attached drawings.

The embodiments subsequently described are for illustrating - but not in accordance with - the present invention defined by the appended claims.

Expressions such as "have," "may have," "include," "may include" or the like represent presence of corresponding numbers, functions, operations, or parts, and do not exclude the presence of additional features.

Expressions such as "at least one of A or B" and "at least one of A and B" should be understood to represent "A," "B" or "A and B.

As used herein, terms such as "first," and "second," may identify corresponding components, regardless of order and/or importance, and are used to distinguish a component from another without limiting the components.

In addition, a description that one element (e.g., a first element) is operatively or communicatively coupled with/to" or "connected to" another element (e.g., a second element) should be interpreted to include both the first element being directly coupled to the second element, and the first element being coupled to the second element through a third element.

A singular expression includes a plural expression, unless otherwise specified. It is to be understood that terms such as "comprise" or "consist of" are used herein to designate a presence of a characteristic, number, step, operation, element, component, or a combination thereof, and not to preclude a presence or a possibility of adding one or more of other characteristics, numbers, steps, operations, elements, components or a combination thereof.

A term such as "module," "unit," and "part," is used to refer to an element that performs at least one function or operation and that may be implemented as hardware or software, or a combination of hardware and software. Except when each of a plurality of "modules," "units," "parts," and the like must be realized in an individual hardware, the components may be integrated in at least one module or chip and be realized in at least one processor (not shown).

In the following description, a "user" may refer to a person using an electronic apparatus or an apparatus using an electronic apparatus (e.g., artificial intelligence electronic apparatus).

Various example embodiments will be described in greater detail below with reference to the accompanying drawings.

<FIG> is a block diagram illustrating an electronic apparatus <NUM> according to an embodiment of the disclosure.

Referring to <FIG>, the electronic apparatus <NUM> may include a memory <NUM> and a processor <NUM>.

The electronic apparatus <NUM> according to various embodiments may include, for example, at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a desktop PC, a laptop PC, a personal digital assistant (PDA), or a portable multimedia player (PMP). In some embodiments, the electronic apparatus <NUM> may include at least one of, for example, a television, a digital video disk (DVD) player, a media box (for example, SAMSUNG HOMESYNCTM, APPLE TV TM, or GOOGLE TVTM).

The memory <NUM> may be implemented as an internal memory such as a read-only memory (ROM) (for example, electrically erasable programmable read-only memory (EEPROM)) and a random-access memory (RAM) or a memory separate from the processor <NUM>. In this case, the memory <NUM> may be implemented as at least one of a memory embedded within the electronic apparatus <NUM> or a memory detachable from the electronic apparatus <NUM> according to the usage of data storage. For example, the data for driving the electronic apparatus <NUM> may be stored in the memory embedded within the electronic apparatus <NUM>, and the data for upscaling of the electronic apparatus <NUM> may be stored in the memory detachable from the electronic apparatus <NUM>.

A memory embedded in the electronic apparatus <NUM> may be implemented as at least one of a volatile memory such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM), or a non-volatile memory (for example, one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, a flash memory (for example, NAND flash or NOR flash), a hard disk drive (HDD) or a solid state drive (SSD). In the case of a memory detachably mounted to the electronic apparatus <NUM>, the memory may be implemented as a memory card (for example, a compact flash (CF), secure digital (SD), micro secure digital (micro-SD), mini secure digital (mini-SD), extreme digital (xD), multi-media card (MMC), etc.), an external memory (for example, a universal serial bus (USB) memory) connectable to the USB port, or the like.

The processor <NUM> may perform overall control operations of the electronic apparatus <NUM>. To be specific, the processor <NUM> plays a role to control overall operations of the electronic apparatus <NUM>.

The processor <NUM> according to an embodiment may be implemented with at least one of a digital signal processor (DSP), a microprocessor, and a time controller (TCON). The embodiment is not limited thereto and may include at least one of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), and an advanced reduced instruction set computing (RISC) machine (ARM) processor or may be defined as a corresponding term. The processor <NUM> may be implemented in a system on chip (SoC) type or a large scale integration (LSI) type in which a processing algorithm is built therein or in a field programmable gate array (FPGA) type. The processor <NUM> may perform various functions by executing computer executable instructions stored in the memory <NUM>.

The memory <NUM> may store a speech recognition model <NUM>, such as illustrated in <FIG> according to an embodiment of the disclosure. The speech recognition model <NUM> includes a plurality of networks, such as neural networks. The speech recognition model <NUM> may include a first network <NUM>, a second network <NUM>, and a third network <NUM>. The speech recognition model <NUM> may receive audio signals or other voice data corresponding to a user voice as input data, and may generate recognition information (or text information) corresponding to the user voice as output data. The recognition information may refer to text information corresponding to a user voice.

A detailed description of operation of the speech recognition model <NUM> will now be described.

The processor <NUM> may obtain one or more first audio signals corresponding to a first user voice. The processor <NUM> may obtain first recognition information corresponding to the first user voice through the speech recognition model <NUM>. The processor <NUM> may input the first audio signals in digital format, or other first voice data corresponding to the first user voice, into the speech recognition model <NUM> as input data, and obtain first recognition information corresponding to the first user voice from the speech recognition model <NUM> as output data. The processor <NUM> may store the first recognition information in the memory <NUM>. Accordingly, the memory <NUM> may store the first recognition information corresponding to the first user voice.

The processor <NUM> may obtain the first vector by inputting the second user voice to the first network <NUM> among the plurality of networks, obtain the second vector by inputting the first recognition information to the second network <NUM> including the first weight information among the plurality of networks, and input the first vector and the second vector to the third network <NUM> including the second weight information of the plurality of networks to obtain second recognition information corresponding to the second user voice. At least some of the second weight information may be the same information as the first weight information.

The processor <NUM> may obtain a second user voice different from the first user voice. The processor <NUM> may obtain second recognition information corresponding to the second user voice through the speech recognition model <NUM>. The processor <NUM> may input the second user voice into the speech recognition model <NUM> as input data, and obtain second recognition information corresponding to the second user voice from the speech recognition model <NUM> as output data.

The processor <NUM> may obtain the first vector through the first network <NUM>. The processor <NUM> may input the second user voice to the first network <NUM> to obtain a first vector. The first vector may mean a hidden vector obtained based on the user voice (second user voice). A detailed description related to the first vector will be described by equation <NUM>-<NUM> of <FIG>.

The processor <NUM> may obtain the second vector through the second network <NUM>. The processor <NUM> may input first recognition information corresponding to the first user voice to the second network <NUM> to obtain a second vector. The second network <NUM> may include (or store) first weight information. The processor <NUM> may obtain a second vector based on the first recognition information and the first weight information. The second vector may mean a hidden vector obtained based on a previous output result (first recognition information). A detailed description related to the second vector will be described through Equation <NUM>-<NUM> and Equation <NUM>-<NUM> of <FIG>.

The processor <NUM> may obtain second recognition information corresponding to the second user voice through the third network <NUM>. The processor <NUM> may input the first vector and the second vector to the third network <NUM> to obtain second recognition information. The processor <NUM> may obtain a third vector based on the first vector and the second vector through the third network <NUM>. The third vector may mean a vector that combines the first vector and the second vector. The third network <NUM> may include (or store) second weight information. The processor <NUM> may obtain second recognition information based on the second weight information and the third vector. The operation of obtaining the second recognition information will be described through Equation <NUM> and Equation <NUM> of <FIG>.

The first weight information and the second weight information may include the same information. Information included in the first weight information may be included in the second weight information. The second weight information may include information included in the first weight information. For example, the weight included in the first weight information may be included in the second weight information. The second weight information may include a weight included in the first weight information.

The second weight information may further include additional weight information other than the first weight information. For example, the second weight information may include a weight included in the first weight information and an additional weight.

The processor <NUM> may store the weight included in the first weight information in the first area of the memory <NUM> and store the additional weight in the second area of the memory <NUM>. The processor <NUM> may use the weight stored in the first area of the memory <NUM> as the first weight information. The processor <NUM> may use the weight stored in the first area of the memory <NUM> and the additional weight stored in the second area of the memory <NUM> as second weight information. The storage space may be efficiently used by storing a weight overlapping the first weight information and the second weight information in one area. The description related to the area of memory will be described in <FIG>.

The weight information may be written as parameter information or embedding. For example, the first weight information may be written as first parameter information or first embedding.

The speech recognition model <NUM> may be a recurrent neural network transducer (RNN-T).

The RNN-T model may be a real-time speech recognition model performing a prediction operation in an intermediate process in which a user voice is continuously inputted. The RNN-T model may include a transcription network, a prediction network, and a joint network.

The transcription network may be a network that obtains a vector corresponding to real-time user voice. The prediction network may be a network that obtains a vector corresponding to a previous user voice. The joint network may be a network that combines the vector output from the transfer network and the vector output from the prediction network.

The first network <NUM> may be a transcription network, the second network <NUM> may be a prediction network, and the third network <NUM> may be a joint network.

When the second user voice is received, the processor <NUM> may obtain a feature vector corresponding to the second user voice, and obtain a first vector based on the feature vector corresponding to the second user voice and the first sub-network included in the first network <NUM>.

Thee processor <NUM> may vectorize the second user voice to obtain a feature vector. The processor <NUM> may obtain a feature vector corresponding to a user voice by using a Mel-filter bank or Mel-Frequency Cepstral Coefficients (MFCC), a Spectrogram, and the like.

The processor <NUM> may input a feature vector corresponding to the second user voice to the first sub-network to obtain a first vector. The first sub-network may mean a network that converts the feature vector to a hidden vector. The first vector may mean a hidden vector.

A detailed description related to the first vector will be described through Equation (<NUM>-<NUM>) of <FIG>. The feature vector may correspond to "X_t" of <FIG>. The first sub-network may correspond to "f_trans" of <FIG>. The first vector may also correspond to the "h_ trans,t" of <FIG>.

The processor <NUM> may obtain a one-hot vector corresponding to the first recognition information, and obtain the second vector based on the one-hot vector corresponding to the first recognition information, the first weight information, and a second sub-network included in the second network <NUM>.

The processor <NUM> may obtain first recognition information corresponding to the first user voice (previous user voice) and obtain a one-hot vector corresponding to the first recognition information. The one-hot vector may mean a vector consisting of <NUM> and <NUM>. In the one-hot vector, a sum of the vectors may be <NUM>. Thus, the one-hot vector may include a plurality of vectors having a value of "<NUM>" and one vector having a value of "<NUM>".

The second network <NUM> may include first weight information. The first weight information may be input embedding. The second network <NUM> may include a second sub-network. The second sub-network may mean a network that converts an intermediate vector (or embedding vector) corresponding to the first recognition information into a hidden vector. The processor <NUM> may obtain a second vector based on the one-hot vector, the first weight information, and the second sub-network through the second network <NUM>.

The detailed description related to the second vector will be described through Equation (<NUM>-<NUM>) and Equation (<NUM>-<NUM>) of <FIG>. The one-hot vector corresponding to the first recognition information may correspond to "y_u-<NUM>" of <FIG>. The first weight information may correspond to the "W_pred" of <FIG>. The intermediate vector may also correspond to the "e_u-<NUM>" of <FIG>. The second sub-network may also correspond to the "f_pred" of <FIG>. The second vector may also correspond to the "h_pred,u" of <FIG>.

The processor <NUM> may obtain a third vector based on the first vector, the second vector, and a third sub-network included in the third network <NUM>, and obtain the second recognition information based on a third vector and the second weight information.

The third network <NUM> may include a third sub-network. The third sub-network may be a network that obtains a third vector by combining the first vector obtained from the first network <NUM> and the second vector obtained from the second network <NUM>. The third vector may be a hidden vector.

The third network <NUM> may include second weight information. The second weight information may be output embedding. The processor <NUM> may obtain second recognition information based on the second weight information and the third vector. The processor <NUM> may multiply the second weight information and the third vector and normalize the multiplied value using a softmax function. The processor <NUM> may obtain second recognition information corresponding to the second user voice based on the normalized value.

The operation of obtaining the second recognition information is described through Equation <NUM> and Equation <NUM> of <FIG>. The third sub-network may correspond to the "f_joint" of <FIG>. The second weight information may correspond to the "W_joint" of <FIG>. The third vector may also correspond to the "h_joint" of <FIG>.

The first weight information may include a weight corresponding to a preset number of subwords, and the second weight information may include the weight included in the first weight information and an additional weight.

The subword may refer to a preset word assumed by a voice uttered by a user. The subword may be different according to a speech recognition model.

The first weight information may include V weights corresponding to a predetermined number of subwords. Here, the V weights may be determined by a learning operation. The second weight information may further include V weights and additional weights included in the first weight information.

The weight included in the first weight information may correspond to the weight W_p1, W_p <NUM>, W_p3,. , W_pV of <FIG>. The additional weight included in the second weight information may correspond to the additional weight W_null of <FIG>.

The processor <NUM> may store the weight included in the first weight information in the first area of the memory <NUM> and store the additional weight in the second area of the memory <NUM>. The processor <NUM> may use the weight stored in the first area of the memory <NUM> as the first weight information. The processor <NUM> may use the weight stored in the first area of the memory <NUM> and the additional weight stored in the second area of the memory <NUM> as second weight information. A specific operation of storing a weight in the memory <NUM> is shown in <FIG>.

The additional weight may be a weight used when there is no subword corresponding to the second user voice, and a dimension of the preset number of weights may be identical with a dimension of the additional weight.

The additional weight (W_null) may mean a weight which will be applied when the user voice does not correspond to any of V subwords.

When a user voice is received using the speech recognition model <NUM>, the processor <NUM> may determine how much the user voice is similar to each of the preset V subwords. For example, the processor <NUM> may determine that the probability that the user voice corresponds to the first subword is p1, the probability of corresponding to the second subword is p2,. , and the probability of corresponding to the Vth subword is pV. The processor <NUM> may determine the subword having the highest probability value among p1 to pV as recognition information corresponding to the user voice.

The processor <NUM> may further check whether the highest probability value among p1 to pV is greater than or equal to a threshold value. If the highest probability value is greater than or equal to the threshold, the processor <NUM> may determine the subword having the highest probability value as recognition information corresponding to the user voice.

If the highest probability value is less than the threshold value, the processor <NUM> may determine that there is no subword corresponding to the user voice. If there is no subword corresponding to the user voice, the processor <NUM> may obtain recognition information corresponding to the user voice by using the additional weight (W_null).

The dimension of the weight included in the first weight information and the second weight information may be the same. The representation associated with the dimension of the weight is described in <FIG> and <FIG>.

The first weight information may be trained based on a first gradient indicating a change amount of a loss value according to the first weight information, a second gradient indicating a change amount of a loss value according to the second weight information, and a learning rate, and the second weight information may be determined based on the trained first weight information.

The processor <NUM> may obtain a first gradient indicating the amount of change in the loss value according to the first weight information and a second gradient indicating the amount of change in the loss value according to the second weight information. In addition, the processor <NUM> may learn the first weight information based on the first gradient, the second gradient, and a learning rate. The processor <NUM> may determine the second weight information based on the trained first weight information.

Specific operations related to the foregoing are described in <FIG> and <FIG>. The first gradient may correspond to the "∇W_predL" of <FIG>. The second gradient may correspond to the "∇W_jointL" of <FIG>. The learning rate may correspond to η of <FIG>. The first weight information before the learning operation is performed may correspond to the "W_pred-old" of <FIG>. The first weight information after the learning operation has been performed may correspond to "W_pred-new" of <FIG>. The second weight information before the learning operation is performed may correspond to the "W_joint-old" of <FIG>. The second weight information after the learning operation has been performed may correspond to "W_joint-new" of <FIG>.

The first weight information and the second weight information may be trained based on an average value of first sub-weight information and second sub-weight information, where a first sub-weight may be calculated based on the first gradient indicating a change amount of a loss value according to the first weight information and a learning rate, and a second sub-weight may be calculated based on the second gradient indicating a change amount of a loss value according to the second weight information and the learning rate.

The processor <NUM> may obtain a first gradient indicating a variation of a loss value according to the first weight information and a second gradient indicating a variation of a loss value according to the second weight information. The processor <NUM> may obtain first sub-weight information based on the first gradient and the learning rate, and obtain second sub-weight information based on the second gradient and the learning rate. The processor <NUM> may learn the first weight information and the second weight information based on the average value of the first sub-weight information and the second sub-weight information.

Specific operations related thereto are described in <FIG> and <FIG>. The first gradient may correspond to the "∇W_predL" of <FIG>. The second gradient may correspond to "∇W_jointL" of <FIG>. The learning rate may correspond to η of <FIG>. The first sub-weight information may correspond to "W_pred-sub" of <FIG>. The second sub-weight information may correspond to "W_joint-sub" of <FIG>. The first weight information before the learning operation is performed may correspond to the "W_pred-old" of <FIG>. The first weight information after the learning operation has been performed may correspond to "W_pred-new" of <FIG>. The second weight information before the learning operation is performed may correspond to the "W_joint-old" of <FIG>. The second weight information after the learning operation has been performed may correspond to "W_joint-new" of <FIG>.

The electronic apparatus <NUM> may further include a microphone.

The microphone (not shown) is an element to receive a user voice or other sound and convert to audio data. The microphone (not shown) may receive the user voice in an active state. For example, the microphone may be integrally formed as an integral unit on at least one of an upper side, a front side direction, a side direction, or the like of the electronic apparatus <NUM>. The microphone may include various configurations such as a microphone for collecting user voice in an analog format, an amplifier circuit for amplifying the collected user voice, an audio-to-digital (A/D) conversion circuit for sampling the amplified user voice to convert into a digital signal, a filter circuitry for removing a noise element from the converted digital signal, or the like.

The processor <NUM> may obtain user voice through a microphone (not shown). The processor <NUM> may obtain recognition information corresponding to the user voice from the speech recognition model <NUM>.

The electronic apparatus <NUM> may obtain recognition information corresponding to the user voice by using the first weight information included in the second network <NUM> and the second weight information included in the third network <NUM>. The weight included in the first weight information may be included in the second weight information. Since the second weight information may use the weight included in the first weight information as it is, some weights may be shared. Therefore, the electronic apparatus <NUM> may reduce the size of the model without the degradation of the speech recognition performance.

Although only a simple configuration of the electronic apparatus <NUM> is shown above, various configurations may be additionally provided during implementation.

<FIG> is a diagram illustrating the speech recognition model <NUM> composed of a plurality of networks.

Referring to <FIG>, the speech recognition model <NUM> may include a first network <NUM>, a second network <NUM>, and a third network <NUM>.

The speech recognition model <NUM> may be an artificial intelligence model. For example, the speech recognition model <NUM> may be an RNN-T model.

The first network <NUM> may refer to a transcription network. The first network <NUM> may receive a user voice and obtain a first vector corresponding to the user voice.

The second network <NUM> may mean a prediction network. The second network <NUM> may receive a previous output result to obtain a second vector. The previous output result may refer to recognition information corresponding to a previous user voice. The second network <NUM> may obtain a second vector corresponding to a previous output result using the first weight information.

The third network <NUM> may refer to a joint network. The third network <NUM> may receive a first vector obtained from the first network <NUM> and a second vector obtained from the second network <NUM> to obtain an output result corresponding to the user voice. The output result may mean a target word corresponding to the user voice.

<FIG> is a diagram illustrating the speech recognition model <NUM> for obtaining recognition information based on a previous output value.

The first network <NUM> may obtain the user voice X_t. The first network <NUM> may obtain the first vector h_trans,t using Equation <NUM>-<NUM>. The first network <NUM> may input the received user voice X_t to the first sub-network f_trans to obtain the first vector h_trans,t. The first sub-network f_trans may be a network included in the first network <NUM>.

The second network <NUM> may obtain recognition information y_u-<NUM> corresponding to the previous user voice. The second network <NUM> may obtain the second vector h_pred,u using Equation <NUM>-<NUM> and Equation <NUM>-<NUM>. Through Equation <NUM>-<NUM>, the second network <NUM> may multiply the first weight information W_pred and the recognition information y_u-<NUM> corresponding to the previous user voice to obtain the intermediate vector e_u-<NUM>. The intermediate vector e_u-<NUM> may mean an embedding vector. Through Equation <NUM>-<NUM>, the second network <NUM> may input the intermediate vector e_u-<NUM> to the second sub-network f_pred to obtain the second vector h_pred,u.

The third network <NUM> may obtain recognition information y_u corresponding to the user voice X_t. The recognition information y_u corresponding to the user voice X_t may refer to recognition information (or target word) corresponding to the user voice. The third network <NUM> may obtain recognition information corresponding to the user voice by using Equation <NUM> and Equation <NUM>. Through Equation <NUM>, the third network <NUM> may input the first vector h_trans,t and the second vector h_pred,u to the third sub-network f_joint to obtain the third vector h_joint. Through Equation <NUM>, the third network <NUM> may input the second weight information W_joint and the third vector h_joint to the Softmax to obtain recognition information y_u corresponding to the user voice. In Equation <NUM>, p(y_u|X_t,y_u-<NUM>) may mean a probability value for recognition information y_u determined based on user voice (X_t) and recognition information y_u-<NUM> corresponding to the previous user voice.

In <FIG>, it is assumed that the recognition information corresponding to the previous user voice is <NUM>. However, the recognition information corresponding to the previous user voice may be plural, and the description related thereto is illustrated in <FIG>.

<FIG> is a diagram illustrating the speech recognition model <NUM> for obtaining recognition information based on a plurality of previous output values.

The first network <NUM> may obtain user voice (X_1:t). The first network <NUM> may obtain the first vector h_trans,t using Equation <NUM>-<NUM>. The first network <NUM> may input the received user voice (X_1:t) to the first sub-network f_trans to obtain the first vector h_trans,t. The first sub-network f_trans may be a network included in the first network <NUM>.

The second network <NUM> may obtain recognition information y_1:u-<NUM> corresponding to the previous user voice. The second network <NUM> may obtain the second vector h_pred, u using Equation <NUM>-<NUM> and Equation <NUM>-<NUM>. Through Equation <NUM>-<NUM>, the second network <NUM> may multiply the first weight information W_ pred and the recognition information y_1:u-<NUM> corresponding to the previous user voice to obtain the intermediate vector e_1:u-<NUM>. The intermediate vector e_1:u-<NUM> may mean an embedding vector. Through Equation <NUM>-<NUM>, the second network <NUM> may input the intermediate vector e_1:u-<NUM> to the second sub-network f_pred to obtain the second vector h_pred,u.

The third network <NUM> may obtain recognition information y_u corresponding to the user voice X_1:t. The recognition information y_u corresponding to the user voice X_1:t may refer to recognition information (or target word) corresponding to the user voice. The third network <NUM> may obtain recognition information corresponding to the user voice by using Equation <NUM> and Equation <NUM>. Through Equation <NUM>, the third network <NUM> may input the first vector h_trans,t and the second vector h_pred,u to the third sub-network f_joint to obtain the third vector h_joint. Through Equation <NUM>, the third network <NUM> may input the second weight information W_joint and the third vector h_joint to the Softmax to obtain recognition information y_u corresponding to the user voice. In Equation <NUM>, p(y_u|X_1:t,y_1:u-<NUM>) may mean a probability value for recognition information y_u determined based on the user speech X_1:t and recognition information y_1:u-<NUM> corresponding to the previous user voice.

<FIG> is a diagram illustrating a weight information storage method of an embodiment in which the first weight information and the second weight information are not identical.

Referring to <FIG>, the speech recognition model <NUM> may include a first network <NUM>, a second network <NUM>, and a third network <NUM>. The second network <NUM> may be a network using the first weight information W_pred. The third network <NUM> may be a network using the second weight information W_joint. The first weight information W_pred and the second weight information W_joint may be stored in the memory <NUM> of the electronic apparatus <NUM>.

The first weight information W_pred and the second weight information W_joint may include different weights. The electronic apparatus <NUM> may store the first weight information W_pred in the first area <NUM> of the memory <NUM> and may store the second weight information W_joint in the second area <NUM> of the memory <NUM>.

<FIG> is a diagram illustrating a weight information configuration of an embodiment in which the first weight information and the second weight information are not identical.

Referring to <FIG>, the first weight information W_pred <NUM> and the second weight information W_joint <NUM> may include different weights.

The first weight information W_pred <NUM> may include V weight values of D dimension. Here, V may mean a predetermined number of subwords. Here, D may mean a dimension of a predetermined subword weight. The first weight information W_pred <NUM> may include a first subword weight W_p1, a second subword weight W_p2, a third subword weight W_p3 to Vth subword weight W_pV.

The second weight information W_joint <NUM> may include V +<NUM> weights of D dimension. Here, V may mean a predetermined number of subwords. Here, D may mean a dimension of a predetermined subword weight. The weight included in the second weight information W_joint <NUM> may include a first subword weight W_j1, a second subword weight W_j2, a third subword weight W_j3 to Vth subword weight W_jV, and an additional weight W_null. The additional weight W_null may mean a weight applied when the user's voice does not correspond to any of the V subwords. Accordingly, the second weight information W_joint <NUM> may include weights W_j1, W_j2, W_j3,. , W_jV corresponding to V subwords, and an additional weight W_null. The second weight information W_joint <NUM> may include a total of V +<NUM> weights.

<FIG> is a diagram illustrating a weight information storage method of an embodiment in which the first weight information and the second weight information are the same.

Referring to <FIG>, the speech recognition model <NUM> may include a first network <NUM>, a second network <NUM>, and a third network <NUM>. The second network <NUM> may be a network using the first weight information W_pred. The third network <NUM> may be a network using the second weight information (W_joint). The first weight information W_pred and the second weight information W_joint may be stored in the memory <NUM> of the electronic apparatus <NUM>.

The first weight information W_pred and the second weight information W_joint may include the same weight. The weight included in the first weight information W_pred may also be included in the second network <NUM>. The first weight information W_pred and the second weight information W_joint may share a common weight. Therefore, the electronic apparatus <NUM> does not need to separately store the first weight information W_pred and the second weight information W_joint. The electronic apparatus <NUM> may use the weight included in the first weight information W_pred as the second weight information W_joint as it is.

The electronic apparatus <NUM> may store a weight included in the first weight information W_pred in the first area <NUM> of the memory <NUM>. The electronic apparatus <NUM> may store an additional weight W_null in the second area <NUM> of the memory <NUM>.

The electronic apparatus <NUM> may use the weight stored in the first region <NUM> of the memory <NUM> as the first weight information W_pred. The electronic apparatus <NUM> may use the weight stored in the first area <NUM> of the memory <NUM> and the weight W_null stored in the second area <NUM> of the memory <NUM> as the second weight information W_joint. As a result, the embodiment of <FIG> may reduce the storage space of the memory <NUM> than the embodiment of <FIG>.

<FIG> is a diagram illustrating a weight information configuration of an embodiment in which the first weight information and the second weight information are the same.

Referring to <FIG>, the first weight information W_pred <NUM> and the second weight information W_joint-new <NUM> may include the same weight.

The first weight information W_pred, <NUM> may include V weight values of D dimension. Here, V may mean a predetermined number of subwords. D may mean a dimension of a predetermined subword weight. The first weight information W_pred, <NUM> may include a first subword weight W_p1, a second subword weight W_p2, a third subword weight W_p3 to a Vth subword weight W_pV.

The second weight information W_joint-new, <NUM> may include V+<NUM> weights of D dimension. Here, V may mean a predetermined number of subwords. D may mean a dimension of a predetermined subword weight.

The second weight information W_joint-new, <NUM> may include a weight included in the first weight information W_pred, <NUM> and an additional weight W_null. The weight included in the second weight information W_joint-new, <NUM> may include a first subword weight W_p1, a second subword weight W_p2, a third subword weight W_p3 to a Vth subword weight W_pV and an additional weight W_null. The additional weight W_null may mean a weight applied when the user's voice does not correspond to any of the V subwords. Accordingly, the second weight information W_joint-new <NUM> may include weights W_p1, W_p2, W_p3,. , W_pV corresponding to V subwords, and an additional weight W_null. The second weight information W_joint-new <NUM> may include a total of V +<NUM> weights.

The electronic apparatus <NUM> may obtain the second weight information W_joint-new <NUM> based on Equation <NUM>. The electronic apparatus <NUM> may obtain a transposed matrix of the first weight information W_pred <NUM>. The electronic apparatus <NUM> may obtain the second weight information W_joint-new <NUM> by adding an additional weight W_null to a transposed matrix of the first weight information W_pred, <NUM>.

<FIG> is a flowchart illustrating an operation of obtaining recognition information corresponding to a user voice using the speech recognition model <NUM>.

Referring to <FIG>, the electronic apparatus <NUM> may receive a user voice in operation S905. The electronic apparatus <NUM> may input a user voice into a speech recognition model <NUM> including a plurality of networks in operation S910. The electronic apparatus <NUM> may obtain recognition information corresponding to the user voice from the speech recognition model <NUM> in operation S915. The recognition information corresponding to the user voice may be output from the speech recognition model <NUM>. The user voice is input data input to the speech recognition model <NUM>, and recognition information corresponding to the user voice may be output data output from the speech recognition model <NUM>.

<FIG> is a flowchart illustrating an operation of obtaining recognition information based on a first user voice and a second user voice.

Referring to <FIG>, the electronic apparatus <NUM> may store first recognition information corresponding to the first user voice in operation S1010. The electronic apparatus <NUM> may input the first user voice to the speech recognition model <NUM> to obtain first recognition information corresponding to the first user voice as output data. The electronic apparatus <NUM> may store the first recognition information in the memory <NUM>.

The electronic apparatus <NUM> may receive the second user voice in operation S1020. The electronic apparatus <NUM> may input the second user voice to the first network <NUM> to obtain the first vector in operation S1030. The operation of obtaining the first vector may be performed in the first network <NUM>.

The electronic apparatus <NUM> may input the first recognition information to the second network <NUM> to obtain a second vector in operation S1040. The operation of obtaining the second vector may be performed in the second network <NUM>.

In operation S1050, the electronic apparatus <NUM> may input the first vector and the second vector to the third network <NUM> to obtain second recognition information corresponding to the second user voice in operation S1050. The operation of obtaining the second recognition information may be performed in the third network <NUM>.

<FIG> is a flowchart illustrating a detailed operation of obtaining a first vector.

Referring to <FIG>, the operations S1110, S1120, S1140, and S1150 may correspond to S1010, S1020, S1040, and S1050 of <FIG> and will not be described further.

After operation S1120 of receiving the second user voice, the electronic apparatus <NUM> may obtain a feature vector corresponding to the second user voice in operation S1131. The feature vector may mean a vector obtained based on user voice. The electronic apparatus <NUM> may obtain a first vector h_trans,t based on a feature vector corresponding to a second user voice and a first sub-network f_trans included in the first network <NUM> in operation S1132.

<FIG> is a flowchart illustrating a detailed operation of obtaining a second vector.

Referring to <FIG>, the operations S1210, S1220, S1230, and S1250 may correspond to S1010, S1020, S1030, and S1050 of <FIG> and will not be described further.

In operation S1241, the electronic apparatus <NUM> may obtain a one-hot vector corresponding to the first recognition information after obtaining the first vector in operation S1230. The one-hot vector may mean a vector consisting of <NUM> and <NUM>. In the one-hot vector, a sum of the vectors may be <NUM>. The one-hot vector may include a plurality of vectors having a value of "<NUM>" and one vector having a value of "<NUM>".

The electronic apparatus <NUM> may obtain the second vector h_pred,u based on a one-hot vector corresponding to the first recognition information, the first weight information W_pred, and the second sub-network f_pred included in the second network <NUM> in operation S1242.

<FIG> is a flowchart illustrating a detailed operation of obtaining a third vector.

Referring to <FIG>, the operations S1310, S1320, S1330, and S1340 may correspond to S1010, S1020, S1030, and S1040 of <FIG> and will not be described further.

In operation S1352, the electronic apparatus <NUM> may obtain the third vector h_joint based on the third sub-network (f_joint) included in the first vector, second vector, and the third network <NUM> after the obtaining the second vector in operation S1340.

<FIG> is a diagram illustrating an operation of learning first weight information and second weight information based on a learning method according to an embodiment.

Referring to <FIG>, according to an embodiment, the electronic apparatus <NUM> may learn the first weight information W_pred and the second weight information W_joint.

The electronic apparatus <NUM> may obtain a first gradient ∇W_predL indicating a change amount of a loss value according to the first weight information W_pred in operation S1410-<NUM>. The electronic apparatus <NUM> may obtain a second gradient ∇W_jointL indicating the amount of change in the loss value according to the second weight information W_joint in operation S1410-<NUM>.

Here, L may mean a loss value obtained based on a loss function.

The gradient may mean a gradient vector. The first gradient VW_predL may mean a gradient vector indicating how much a loss value is changed as the first weight information W_pred changes. The second gradient ∇W_jointL may mean a gradient vector indicating how much a loss value is changed as the second weight information W_joint changes.

The electronic apparatus <NUM> may obtain the updated first weight information W_pred-new in operation S1420. The electronic apparatus <NUM> may obtain the updated first weight information W_pred-new based on the first weight information W_pred-old, learning rate η, the first gradient ∇W_predL obtained in operation S1410-<NUM>, and the second gradient ∇W_jointL obtained in operation S1410-<NUM>. The electronic apparatus <NUM> may obtain a sum value (∇W_predL+∇W_jointL) of the first gradient (∇W_predL) and the second gradient (∇W_jointL), and multiply the obtained sum value by a learning rate η to obtain an intermediate value ((η(∇W_predL+VW_jointL)). The electronic apparatus <NUM> may deduct intermediate value (η(∇W_predL+∇W_jointL)) from the first weight information W_pred-old to obtain the updated first weight information W_pred-new.

The electronic apparatus <NUM> may obtain the updated second weight information W_joint-new in operation S1430. The electronic apparatus <NUM> may obtain the updated second weight information W_joint-new by substituting the updated first weight information W_pred-new to the second weight information W_joint-old. Here, operation S1430 may correspond to Equation <NUM> of <FIG>.

The updated first weight information W_pred-new and the updated second weight information W_joint-new may include the same weight. The updated second weight information W_joint-new may further include an additional weight W_null than the first weight information W_pred-new.

<FIG> is a flowchart illustrating an operation of learning first weight information and second weight information based on a learning method according to an embodiment.

Referring to <FIG>, the electronic apparatus <NUM> may obtain a first gradient ∇W_predL indicating a change amount of a loss value according to the first weight information W_pred, and obtain a second gradient ∇W_jointL indicating a change amount of a loss value according to the second weight information W_joint in operation S1510. Here, the step S <NUM> may correspond to steps S <NUM>-<NUM> and S <NUM>- <NUM> of <FIG>.

The electronic apparatus <NUM> may obtain the value∇W_predL+∇W_jointL in which the first gradient ∇W_predL and the second gradient ∇W_jointL are added in operation S1521.

In operation S1522, the electronic apparatus <NUM> may obtain a value η(∇W_predL+∇W_jointL) obtained by multiplying the learning rate η by the added value ∇W_predL+∇W_jointL in operation S1521. In operation S1523, the electronic apparatus <NUM> may obtain the updated first weight information W_pred-new based on the first weight information W_pred-old and the value η(∇W_predL+∇W_jointL) obtained in operation S1522. Here, operations S1521, S1522, and S1523 may correspond to operation S1420 of <FIG>.

The electronic apparatus <NUM> may obtain updated second weight information W_joint-new based on the updated first weight information W_pred-new in operation S1530. The updated second weight information W_joint-new may include a weight included in the updated first weight information W_pred-new and an additional weight W_null. Here, operation S1530 may correspond to S1430 of <FIG>.

<FIG> is a diagram illustrating an operation of learning first weight information and second weight information based on a learning method according to another embodiment.

Referring to <FIG>, according to another embodiment, the electronic apparatus <NUM> may learn the first weight information W_pred and the second weight information W_joint.

The electronic apparatus <NUM> may obtain a first gradient ∇W_predL indicating a change amount of a loss value according to the first weight information W_pred in operation S1610-<NUM>. The electronic apparatus <NUM> may obtain a second gradient ∇W_jointL indicating the amount of change in the loss value according to the second weight information W_joint-old in operation S1610-<NUM>.

The electronic apparatus <NUM> may obtain the first sub-weight information W_pred-sub in operation S1620-<NUM>. The electronic apparatus <NUM> may obtain a value η∇W_predL obtained by multiplying the learning rate η by the first gradient ∇W_predL. The electronic apparatus <NUM> may obtain the first sub-weight information W_pred-sub by subtracting the value n∇W_pred from the first weight information W_pred-old.

The electronic apparatus <NUM> may obtain the second sub-weight information W_joint-sub in operation S1620-<NUM>. Specifically, the electronic apparatus <NUM> may obtain a value (η∇W_jointL) obtained by multiplying a learning rate η by a second gradient ∇W_jointL. The electronic apparatus <NUM> may obtain the second sub-weight information W_joint-sub by subtracting the value n∇W_jointL from the second weight information W_joint-old.

The electronic apparatus <NUM> may obtain the updated first weight information W_pred-new based on the first sub-weight information W_pred-sub and the second sub-weight information W_joint-sub in operation S1630-<NUM>. The electronic apparatus <NUM> may obtain an average value of the first sub-weight information W_pred-sub and the second sub-weight information W_joint-sub as the updated first weight information W_pred-new.

The electronic apparatus <NUM> may obtain updated second weight information W_joint-new based on the first sub-weight information W_pred-sub and the second sub-weight information W_joint-sub in operation S1630-<NUM>. The electronic apparatus <NUM> may obtain an average value of the first sub-weight information W_pred-sub and the second sub-weight information W_joint-sub as the updated second weight information W_joint-new.

<FIG> is a flowchart illustrating an operation of learning first weight information and second weight information based on a learning method according to another embodiment.

Referring to <FIG>, the electronic apparatus <NUM> may obtain a first gradient ∇W_predL indicating a change amount of a loss value according to the first weight information W_pred, and obtain a second gradient ∇W_jointL indicating a change amount of a loss value according to the second weight information W_joint in operation S1710. The operation S1710 may correspond to operations S1610-<NUM> and S <NUM>-<NUM> of <FIG>.

The electronic apparatus <NUM> may obtain a value η∇W_predL obtained by multiplying the learning rate η by the first gradient ∇W_predL and obtain a value η∇W_jointL obtained by multiplying the learning rate η by the second gradient ∇W_jointL in operation S1721.

The electronic apparatus <NUM> may obtain the first sub-weight information W_pred-sub based on the first weight information W_pred-old and the value η∇W_predL, and may obtain the second sub-weight information W_joint-sub based on the second weight information W_joint-old and the value η∇W_jointL in operation S1722. Here, operations S1721 and S1722 may correspond to operations S1620-<NUM> and S1620-<NUM> of <FIG>.

The electronic apparatus <NUM> may obtain the updated first weight information W_predn-new based on the average value of the first sub-weight information W_pred-sub and the second sub-weight information W_joint-sub in operation S1730-<NUM>.

The electronic apparatus <NUM> may obtain updated second weight information W_joint-new based on the average value of the first sub-weight information W_pred-sub and the second sub-weight information W_joint-sub in operation S1730-<NUM>. Here, the operations S1730-<NUM> and S1730-<NUM> may obtain the operations S1630-<NUM> and S1630-<NUM> of <FIG>.

<FIG> is a flowchart illustrating a method of controlling the electronic apparatus <NUM> according to an embodiment of the disclosure.

Referring to <FIG>, a method of controlling the electronic apparatus <NUM> storing a speech recognition model composed of a plurality of networks and first recognition information corresponding to a first user voice obtained through the speech recognition model may include obtaining a first vector by inputting a second user voice to a first network among the plurality of networks in operation S1805; obtaining a second vector by inputting the first recognition information to a second network including first weight information among the plurality of networks in operation S1810; and obtaining second recognition information corresponding to the second user voice by inputting the first vector and the second vector to a third network including second weight information among the plurality of networks in operation S1815, and at least a part of the second weight information may be information identical with the first weight information.

The speech recognition model may be a recurrent neural network transducer (RNN-T) model.

The first network may be a transcription network, the second network may be a prediction network, and the third network may be a joint network.

The obtaining the first vector in operation S1805 may include, based on receiving the second user voice, obtaining a feature vector corresponding to the second user voice, and obtaining the first vector based on the feature vector corresponding to the second user voice and a first sub-network included in the first network.

The obtaining the second vector in operation S1810 may include obtaining a one-hot vector corresponding to the first recognition information, and obtaining the second vector based on the one-hot vector corresponding to the first recognition information, the first weight information, and a second sub-network included in the second network.

The obtaining the second recognition information in operation S1815 may include obtaining a third vector based on the first vector, the second vector, and a third sub-network included in the third network, and obtaining the second recognition information based on a third vector and the second weight information.

The first weight information and the second weight information may be trained based on an average value of the first sub-weight information and the second sub-weight information, the first sub-weight may be calculated based on the first gradient indicating a change amount of a loss value according to the first weight information and a learning rate, and the second sub-weight may be calculated based on the second gradient indicating a change amount of a loss value according to the second weight information and the learning rate.

The method for controlling an electronic apparatus as shown in <FIG> may be performed on an electronic apparatus having the configuration of <FIG>, and may be executed on an electronic apparatus having other configurations.

The methods according to the various embodiments as described above may be implemented as an application format installable in an existing electronic apparatus.

The methods according to the various embodiments as described above may be implemented as software upgrade or hardware upgrade for an existing electronic apparatus.

The various embodiments described above may be performed through an embedded server provided in an electronic apparatus, or an external server of at least one electronic apparatus and a display device.

Meanwhile, various embodiments may be implemented in software, including instructions stored on machine-readable storage media readable by a machine (e.g., a computer). An apparatus may call instructions from the storage medium, and execute the called instruction, including an electronic apparatus according to the disclosed embodiments. When the instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. The instructions may include a code generated by a compiler or a code executable by an interpreter. Herein, the term "non-transitory" only denotes that a storage medium is tangible, and does not distinguish the case in which a data is semi-permanently stored in a storage medium from the case in which a data is temporarily stored in a storage medium.

According to an embodiment, the method according to the above-described embodiments may be included in a computer program product. The computer program product may be traded as a product between a seller and a consumer. The computer program product may be distributed online in the form of machine-readable storage media (e.g., compact disc read only memory (CD-ROM)) or through an application store (e.g., PLAYSTORE™) or distributed online directly. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a server of the manufacturer, a server of the application store, or a machine-readable storage medium such as memory of a relay server.

According to embodiments, the respective elements (e.g., module or program) mentioned above may include a single entity or a plurality of entities. According to embodiments, at least one element or operation from among the corresponding elements mentioned above may be omitted, or at least one other element or operation may be added. Alternatively or additionally, a plurality of components (e.g., module or program) may be combined to form a single entity. In this case, the integrated entity may perform functions of at least one function of an element of each of the plurality of elements in the same manner as or in a similar manner to that performed by the corresponding element from among the plurality of elements before integration. The module, a program module, or operations executed by other elements according to variety of embodiments may be executed consecutively, in parallel, repeatedly, or heuristically, or at least some operations may be executed according to a different order, may be omitted, or the other operation may be added thereto.

Claim 1:
An electronic apparatus comprising:
a memory storing a speech recognition model and first recognition information corresponding to a first user voice obtained through the speech recognition model, the speech recognition model including a first network, a second network, and a third network; and
a processor configured to:
obtain a first vector by inputting second voice data corresponding to a second user voice to the first network,
obtain a second vector by inputting the first recognition information to the second network of the speech recognition model which generates the second vector based on first weight information, and
obtain second recognition information corresponding to the second user voice by inputting the first vector and the second vector to the third network which generates the second recognition information based on second weight information,
wherein at least a part of the second weight information is the same as the first weight information,
wherein the first weight information includes at least one first weight corresponding to a preset number of subwords,
wherein the second weight information includes the at least one first weight and at least one additional weight,
wherein the at least one first weight is stored in a first area of the memory, and the at least one additional weight is stored in a second area of the memory, and
wherein the processor is further configured to use the at least one first weight stored in the first area and the at least one additional weight stored in the second area as the second weight information,
wherein the electronic apparatus is characterized in that:
the at least one additional weight is a weight used when no subword of the preset number of subwords corresponds to the second user voice, and
wherein a dimension of the at least one first weight corresponds to a dimension of the at least one additional weight.