Patent Description:
Speech recognition may refer to technology for recognizing or understanding an acoustic speech signal such as a speech sound uttered by a human being by analyzing the acoustic speech signal with a computing device. Speech recognition may include, for example, recognizing a speech by analyzing a pronunciation using a hidden Markov model (HMM) that processes a frequency feature extracted from speech data using an acoustic model, or directly recognizing text such as a word or a sentence from speech data using an end-to-end type model constructed as an artificial neural network (ANN), without a separate acoustic model.

In arxiv:<NUM>. 10022v2, a sequence-to-sequence (S2S) model is disclosed which combines a modified beam search identifying partial, stable hypotheses and ways of working with bidirectional long-short-time-memory (BLTSM) in the encoder and the use of chunked BLTSM, wherein the model performs run-on speech recognition with low word-based latency without sacrificing in word error rate performance.

In one general aspect, a decoding method includes: receiving an input sequence corresponding to an input speech at a current time; and in a neural network (NN) for speech recognition, generating an encoded vector sequence by encoding the input sequence, determining reuse tokens from candidate beams of two or more previous times by comparing the candidate beams of the previous times, and decoding one or more tokens subsequent to the reuse tokens based on the reuse tokens and the encoded vector sequence.

The determining of the reuse tokens includes:
determining a reuse time of tokens at a current time n, being the current time, subsequent to a previous time n-<NUM> subsequent to a previous time n-<NUM> based on a comparison result between a candidate beam of the previous time n-<NUM> and a candidate beam of the previous time n-<NUM>, wherein n is a natural number greater than or equal to "<NUM>"; and determining candidate beams accumulated up to the reuse time to be the reuse tokens.

The determining of the reuse time includes determining a time in which a largest number of substrings match in the candidate beam of the previous time n-<NUM> and the candidate beam of the previous time n-<NUM>, as the reuse time of the tokens at the current time n.

The method may include storing either one or both of: a candidate beam having a highest probability among probabilities of candidate beams up to the reuse time; and a beam state corresponding to the candidate beam having the highest probability.

Thedecoding of the one or more tokens may include: determining candidate beams that are to be used for decoding of a next time, based on a probability of a combination of tokens at previous times of the decoding among the two or more previous times; and decoding the one or more tokens using one or more candidate beams corresponding to a reuse time of tokens among the candidate beams.

The decoding of the one or more tokens may include: inputting the one or more candidate beams corresponding to the reuse time of the tokens among the candidate beams to an auto-regressive decoder layer included in the NN; and decoding the one or more tokens.

The decoding of the one or more tokens may include, in response to the input speech not being ended, decoding the one or more tokens a preset number of times.

The decoding of the one or more tokens may include: predicting probabilities of token candidates subsequent to the reuse tokens based on the reuse tokens and the encoded vector sequence; and determining the one or more tokens based on the probabilities of the token candidates.

The generating of the encoded vector sequence may include generating the encoded vector sequence by encoding the input sequence using an encoder layer included in the NN.

The method may include, in the NN, generating a cumulative sequence by accumulating the input sequence corresponding to the input speech at the current time to input sequences of the previous times, wherein the generating of the encoded vector sequence may include generating the encoded vector sequence by encoding the cumulative sequence.

The NN may include an attention-based encoder-decoder model including an encoder layer and an auto-regressive decoder layer.

The method may include generating a speech recognition result of the input speech based on the decoded one or more tokens subsequent to the reuse tokens.

In another general aspect, one or more embodiments include a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform any one of the foregoing methods.

In another general aspect, a decoding apparatus with a neural network (NN) for speech recognition includes: a communication interface configured to receive an input sequence corresponding to an input speech at a current time; and a processor configured to use the NN to:.

generate an encoded vector sequence by encoding the input sequence; determine reuse tokens from candidate beams of two or more previous times by comparing the candidate beams of the previous times; and decode one or more tokens subsequent to the reuse tokens based on the reuse tokens and the encoded vector sequence.

For the determining of the reuse tokens, the processor is configured to determine a reuse time of tokens at a current time n, being the current time, subsequent to a previous time n-<NUM> subsequent to a previous time n-<NUM> based on a comparison result between a candidate beam of the previous time n-<NUM> and a candidate beam of the previous time n-<NUM>, wherein n is a natural number greater than or equal to "<NUM>", and determine candidate beams accumulated up to the reuse time to be the reuse tokens.

For the determining of the reuse tokens, the processor is configured to determine a time in which a largest number of substrings match in the candidate beam of the previous time n-<NUM> and the candidate beam of the previous time n-<NUM>, as the reuse time of the tokens at the current time n.

The decoding apparatus may include a memory configured to store candidate beams that are to be used for decoding of a next time, wherein, for the decoding of the one or more tokens, the processor may be configured to determine the candidate beams that are to be used for decoding of the next time, based on a probability of a combination of tokens at previous times of the decoding among the two or more previous times, and decode the one or more tokens using one or more candidate beams corresponding to a reuse time of tokens among the candidate beams.

For the decoding of the one or more tokens, the processor may be configured to input the one or more candidate beams corresponding to the reuse time of the tokens among the candidate beams to an auto-regressive decoder layer included in the NN, and decode the one or more tokens.

For the decoding of the one or more tokens, the processor may be configured to predict probabilities of token candidates subsequent to the reuse tokens based on the reuse tokens and the encoded vector sequence, and determine the one or more tokens based on the probabilities of the token candidates.

For the generating of the encoded vector sequence, the processor may be configured to generate the encoded vector sequence by encoding the input sequence using an encoder layer included in the NN.

The processor may be configured to use the NN to generate a cumulative sequence by accumulating the input sequence corresponding to the input speech at the current time to input sequences of the previous times, and for the generating of the encoded vector sequence, generate the encoded vector sequence by encoding the cumulative sequence.

In another general aspect not forming part of the claimed invention, a decoding method includes: in a neural network (NN) for speech recognition, generating an encoded vector sequence by encoding an input sequence corresponding to an input speech at a current decoding time step, determining reuse tokens based on a largest sequence of tokens matching between candidate beams of previous time steps, and decoding one or more tokens subsequent to the reuse tokens based on the reuse tokens and the encoded vector sequence.

The determining of the reuse tokens may include determining, as the reuse tokens, portions of candidate beams of one of the previous time steps preceding the current time step up to a time corresponding to the largest sequence of tokens matching between the candidate beams of the previous time steps.

The largest sequence of tokens matching between candidate beams of previous decoding time steps is from an initial time step up to a time step previous to the current time step.

Also, descriptions of features that are known in the art, after an understanding of the disclosure of this application, may be omitted for increased clarity and conciseness.

The terminology used herein is for the purpose of describing particular examples only, and is not to be used to limit the disclosure. As used herein, the terms "include," "comprise," and "have" specify the presence of stated features, numbers, operations, elements, components, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, elements, components, and/or combinations thereof. The use of the term "may" herein with respect to an example or embodiment (for example, as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

Although terms of "first" or "second" are used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains consistent with and after an understanding of the present disclosure. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

<FIG> illustrates an example of a process in which partial decoding for speech recognition is performed. <FIG> illustrates a process in which partial decoding is performed on an input speech, for example, "Hi Bixby, what is the weather like tomorrow?".

A speech recognition apparatus with an attention-based encoder-decoder structure may perform processing by applying a high weight directly to a predetermined portion of a sound source without information about an alignment between a wave file and a token, and may be quickly trained based on a large quantity of data due to easy parallelization. In an encoder-decoder structure, an encoder may receive a sound source and may convert the sound source to a sequence of an encoded vector that may be processed by a decoder. Also, the decoder may receive a token output decoded by the decoder again, together with the vector encoded by the encoder, based on an auto-regressive scheme of receiving tokens decoded up to previous times again, and may predict a probability of a token that is to be output in a next time.

The speech recognition apparatus may repeat a decoding step of receiving a start token as an input, receiving an output decoded by the decoder again, and predicting a next token, may output an end token when an entire sentence ends, and may terminate the decoding step.

As described above, to achieve responsiveness showing that a user's speech command is immediately processed in speech recognition, a speech recognition apparatus of one or more embodiments may perform streaming to output an intermediate result while a speech is being input. In contrast, when all sound sources to be decoded in an attention-based encoder-decoder speech recognition model are input and started to be decoded at once by a typical speech recognition apparatus, the typical speech recognition apparatus may not perform streaming to output the intermediate result while the speech is being input, and thus a user may feel dissatisfaction with the resulting poor responsiveness. Accordingly, since responsiveness showing that a user's speech command is immediately processed improves speech recognition, speech recognition apparatus of one or more embodiments may perform streaming to output a recognition result while a speech is being input.

For streaming of outputting a decoding result while a speech is being input, a partial decoding scheme of decoding sound sources accumulated at regular intervals while continuously input sound sources are continuing to be accumulated in a buffer may be used.

For example, partial decoding may be performed on sound sources accumulated at extremely short intervals (for example, <NUM> milliseconds (msec)) for natural decoding. In this example, partial decoding may be performed to be "hi" in a step <NUM><NUM>, "hi bixby hu" in a step <NUM><NUM>, "hi bixby what esh" in a step <NUM><NUM>, and "hi bixby what is thea" in a step <NUM><NUM>.

When the partial decoding is performed as shown in <FIG>, a length of a sound source to be added for each step decreases when a number of decoding steps increases. However, since an overall operation needs to be reperformed on all inputs accumulated each time, a relatively large amount of processing time may be consumed. Also, since an end portion of a speech being input is suddenly cut (for example, "hu" in the step <NUM><NUM>, "esh" in the step <NUM><NUM>, and "thea" in the step <NUM><NUM>) every <NUM> msec, an inaccurate intermediate decoding result may be output.

In addition, since an amount of processing time used is in proportion to a number of repetitions of the decoding step, as the length of the sound source increases, a large amount of processing time may be used and thus partial decoding may not be terminated before <NUM> msec at which a next decoding step starts. In particular, partial decoding of a last portion of a sound source may be started when the sound source is completely ended. In this example, when a previous time in which a sound source is accumulated is not completely used even though the partial decoding is used, a final latency between a sound source end time and a time at which a final speech decoding result is received may increase.

<FIG> illustrates an example of an operation of a decoding apparatus. <FIG> illustrates an auto-regressive decoding process of an input speech, for example, "hi, bixby".

The decoding apparatus may focus on an input speech of a speaker by performing auto-regressive decoding in which a previous output of an artificial neural network (ANN) is used as an input for each token and a next output continues to be output, to calculate an output with an unspecified length. The term "token" used herein may indicate a unit forming one sequence, and the unit may include, for example, a word, a subword, a substring, a character, or a unit forming a single character (for example, an initial consonant and a vowel or a consonant placed under a vowel in the Korean alphabet).

For example, an input speech "hi, bixby" may be sequentially input. In this example, the decoding apparatus input may repeat a decoding step of an input speech through auto-regressive decoding as shown in <FIG>, to receive an output of a previous step every step and find a token of a next step. In the auto-regressive decoding, an output token of a previous step may have an influence on determining of an output token of a next step.

In an example, an ANN of an encoding apparatus may include, for example, an attention-based encoder-decoder speech recognition model. The encoding apparatus may perform decoding using a partial decoding scheme of repeatedly decoding sound sources accumulated at regular intervals while continuously input sound sources are continuing to be accumulated in a buffer.

The decoding apparatus may terminate a previous decoding step before partial decoding of a next step is started, and may immediately perform decoding while a user is speaking a speech command using the attention-based encoder-decoder speech recognition model without a special training scheme or a structural change for streaming.

In a step <NUM><NUM>, the decoding apparatus may perform partial decoding on "hi" together with a start token  that indicates a start of an input speech. The decoding apparatus may focus on "hi" by assigning a high weight to a portion corresponding to "hi" in a sound source and may decode the sound source.

In a step <NUM><NUM>, the decoding apparatus may perform partial decoding on "bix" subsequent to "hi". The decoding apparatus may focus on a portion corresponding to "bix" and may decode the sound source.

Similarly, in a step <NUM><NUM>, the decoding apparatus may perform partial decoding by focusing on "by" subsequent to "bix".

In a step <NUM><NUM>, the decoding apparatus may decode an end token <e> that is subsequent to "by" and that indicates a last portion of the input speech.

When partial decoding is performed, the decoding apparatus of one or more embodiments may immediately find a last token with the end token <e> from an input speech shifted rightwards by a start token , instead of needing to sequentially perform an operation during training, which may be advantageous in parallelization. When an output up to a predetermined step is exactly known during inference, the decoding apparatus may predict a token of a step immediately subsequent to the step as if continuing to perform decoding.

A decoding result may change due to an incorrect prediction or a new token included in a newly accumulated sound source in the vicinity of the sound source in a partial decoding result. However, as shown in <FIG>, a decoding result of a front portion in which a sufficiently large number of sound sources are accumulated may hardly change. Also, a decoder capable of performing training in parallel may not need to start auto-regressive decoding at a start token only, and may predict a probability of a token to be output in a next step when a token sequence is received as an input.

Based on the above description, the decoding apparatus of one or more embodiments may reuse a token corresponding to a stable portion of a previous partial decoding result in each partial decoding, and thus the decoding apparatus of one or more embodiments may reduce a processing load used to process a front portion repeated in an auto-regressive decoding step. The above scheme may be called "relay decoding", in that a result of a previous decoding step may be reused in a next decoding step similarly to a relay running in which a previous runner passes a baton to a next runner.

Also, in an example, a number of repetitions of a decoding step in auto-regressive decoding may be limited to a preset number of times, and thus the decoding apparatus of one or more embodiments may reduce a load used to process a back portion of the decoding step in each partial decoding. By reducing a total number of decoding steps of each partial decoding, the decoding apparatus of one or more embodiments may reduce a total number of operations and a time used in each decoding step. The above scheme may be called "limited relay decoding", in that a number of repetitions of partial decoding may be limited in addition to the above-described relay decoding.

<FIG> illustrates an example of a decoding method. <FIG> illustrates a process in which a decoding apparatus outputs at least one token for an input speech at a current time by an ANN for speech recognition through operations <NUM> to <NUM>.

The ANN may include, for example, an attention-based encoder-decoder model that includes an encoder layer and an auto-regressive decoder layer.

For example, the ANN may include a self-attention neural network (SANN), a recurrent neural network (RNN), a convolutional neural network (CNN), a bi-directional RNN (BRNN), a long short-term memory (LSTM), a bi-directional LSTM (BLSTM), a gated recurrent unit (GRU), and/or a bi-directional GRU (BGRU), but is not limited thereto.

The ANN may receive an input feature vector corresponding to an input speech and may output a recognition result corresponding to the input feature vector. Nodes of layers in the ANN may affect each other in a nonlinear relationship, and parameters of the ANN (for example, values output from each of the nodes or relationships between the nodes) may be optimized through training.

The ANN may receive an input at one end, may generate an output at another end, and may optimize a network weight based on the input and the output. For example, the ANN may include an encoder layer and an auto-regressive decoder layer. The encoder layer and the decoder layer may be referred to as an "encoder" and a "decoder", respectively. In the following description, the terms "encoder layer" and "encoder" may be interchangeably used with respect to each other, and the terms "decoder layer" and "decoder" may be interchangeably used with respect to each other.

The ANN may be, for example, an ANN with an encoder-decoder structure. In the ANN, all parameters of a neural network model may be simultaneously or separately trained for a single loss function. When all the parameters of the neural network model are simultaneously trained for a single loss function, the encoder and decoder may be trained at the same time.

The encoder and the decoder may be pre-trained to generate a sequence of a recognition result from a received feature vector. For example, the encoder and the decoder may be pre-trained to generate a sequence of a recognition result from a sequence of a correct answer text pair corresponding to an input speech.

In operation <NUM>, the decoding apparatus may receive an input sequence corresponding to an input speech at a current time. The input speech may include, for example, a wave file that represents an input speech signal in a form of a wave, a spectrogram that represents a wave file in a form of a frequency, or a mel-frequency cepstral coefficient (MFCC). The input sequence may include an input speech, and an input speech feature vector extracted from the input speech. The input sequence may correspond to a signal cut to be processed by an ANN.

In operation <NUM>, the decoding apparatus generates an encoded vector sequence by encoding the input sequence received in operation <NUM>. The decoding apparatus may encode the input sequence using an encoding layer included in the ANN, to generate the encoded vector sequence. The encoder layer may generate an encoded feature (for example, the encoded vector sequence) by encoding the received input sequence. For example, the encoder layer may also generate encoded information by transforming a dimension of a feature vector. The encoded feature (for example, the encoded vector sequence) generated by the encoder layer may be applied to the decoder. The encoder layer may sequentially receive and encode the input sequence and transmit the input sequence to the decoder layer, and the decoder layer may receive an encoding result (e.g., a result of the encoding) and sequentially output speech-recognized words one by one based on the encoding result.

In operation <NUM>, the decoding apparatus may generate a cumulative sequence in which the input sequence corresponding to the input speech at the current time is accumulated to input sequences of previous times. The decoding apparatus may generate the encoded vector sequence by encoding the cumulative sequence.

In operation <NUM>, the decoding apparatus determines reuse tokens from candidate beams of at least two previous times by comparing the candidate beams of the previous times. The decoding apparatus determines a reuse time of tokens at a current time n subsequent to a previous time n-<NUM> subsequent to the previous time n-<NUM> based on a comparison result between a candidate beam of the previous time n-<NUM> and a candidate beam of the previous time n-<NUM>, where n is a natural number greater than or equal to "<NUM>". For example, the decoding apparatus determines a time in which a large number of substrings are matched between the candidate beam of the previous time n-<NUM> and the candidate beam of the previous time n-<NUM> as the reuse time of the tokens at the current time n. The reuse time may also be referred to as a "stable time", in that a decoding result may remain unchanged and stable. In the following description, the terms "reuse time" and "stable time" may be interchangeably used with respect to each other. In an example, the previous time n-<NUM> may be a time corresponding to an (n-<NUM>)-th decoding step, the previous time n-<NUM> may be a time corresponding to an (n-<NUM>)-th decoding step, and the current time n may be a time corresponding to an n-th decoding step. In the following description, the previous time n-<NUM> and the (n-<NUM>)-th decoding step may be interchangeably used, and the previous time n-<NUM> and the (n-<NUM>)-th decoding step may be interchangeably used, and the current time n and the n-th decoding step may be interchangeably used.

The decoding apparatus may determine candidate beams accumulated up to the reuse time to be the reuse tokens. When an encoded feature is received from an auto-regressive encoder layer, the decoder layer may output a recognition result in units of tokens. The auto-regressive decoder layer may generate an output token depending on input tokens selected up to a previous step. The decoding apparatus may determine a token selected as an output token to be a next input token. The decoding apparatus may use a reuse token when outputting a token of a next step in each decoding step using the auto-regressive decoder layer. For example, the auto-regressive decoder layer may use an output "weather" when recognizing "How" in a sentence "How is the weather?". A non-limiting example in which the decoding apparatus determines reuse tokens will be further described below with reference to <FIG> and <FIG>.

In operation <NUM>, the decoding apparatus decodes at least one token subsequent to the reuse tokens determined in operation <NUM> based on the reuse tokens and the vector sequence generated in operation <NUM>. For example, the decoding apparatus may determine candidate beams that are to be used for decoding of a next time based on a probability of a combination of tokens at previous times of decoding. The decoding apparatus may decode at least one token using at least one candidate beam corresponding to a reuse time of tokens among the candidate beams. In this example, the candidate beams may correspond to candidate beams that are to perform a beam search at a next time. The decoding apparatus may decode at least one token using the at least one candidate beam corresponding to the reuse time of the tokens among the candidate beams. The decoding apparatus may determine at least one candidate beam corresponding to the reuse time of the tokens by a beam search algorithm that reuses candidate beams having high probabilities among the candidate beams. The decoding apparatus may store candidate beams that are to be used for decoding of a next time for each previous time of decoding. For example, the decoding apparatus may store at least one of a candidate beam having a highest probability among probabilities of candidate beams up to the reuse time, and a beam state corresponding to the candidate beam having the highest probability. A non-limiting example of the beam state will be described below.

The decoding apparatus may decode at least one token by inputting the at least one candidate beam corresponding to the reuse time of the tokens among the candidate beams to the auto-regressive decoder layer included in the ANN. The auto-regressive decoder layer may perform auto-regressive decoding to determine an output token based on tokens that are recognized up to a previous step, for each token. The decoding apparatus may calculate an output token based on information calculated from the encoder layer for each step. The output token may be dependent on input tokens selected up to previous steps.

In operation <NUM>, the decoding apparatus may predict token candidates subsequent to the reuse tokens based on the reuse tokens and the vector sequence. The decoding apparatus may determine at least one token based on probabilities of the token candidates. The decoding apparatus may determine a token candidate having a highest probability among the token candidates as a finally decoded token.

For example, when the input speech is not ended, the decoding apparatus may decode at least one token a preset number of times. The preset number of times may be, for example, three times or five times. Since all auto-regressive decoding steps do not need to be completely performed when a speech is not ended, the decoding apparatus may reduce a decoding time by performing decoding steps a limited number of times. In this example, the decoding apparatus may store a candidate beam for each decoding step and/or a beam state of the candidate beam. A non-limiting example in which the decoding apparatus decodes at least one token will be further described below with reference to <FIG> and <FIG>.

<FIG> illustrates an example of a decoding method in an ANN for speech recognition. <FIG> illustrates an example in which a decoding apparatus decodes an input speech "Hi Bixby, what is the weather like tomorrow?" by reuse tokens.

For example, "hi" may be determined to be received in a partial decoding step <NUM><NUM>. Since two partial decoding steps immediately preceding the partial decoding step <NUM><NUM> are absent, the decoding apparatus may perform auto-regressive decoding on "hi" without a candidate beam.

It may be determined that "hi bixby hu" is received in a partial decoding step <NUM><NUM>. Similarly, the decoding apparatus may perform auto-regressive decoding on "hi bixby hu" without a candidate beam, because two partial decoding steps immediately preceding the partial decoding step <NUM><NUM> are absent.

It may be determined that "hi bixby what eshh" is received in a partial decoding step <NUM><NUM>. The decoding apparatus may generate candidate beams corresponding to "hi bixby what eshh" of the partial decoding step <NUM><NUM> (for example, candidate beams of an (n-<NUM>)-th partial decoding step <NUM> of <FIG>).

In addition, it may be determined that "hi bixby what is thea" is received in a partial decoding step <NUM><NUM>. The decoding apparatus may generate candidate beams corresponding to "hi bixby what is thea" of the partial decoding step <NUM><NUM> (for example, candidate beams of an (n-<NUM>)-th partial decoding step <NUM> of <FIG>).

The decoding apparatus may determine, as a reuse time, a time (for example, <NUM> msec) in which a largest number of substrings match in the candidate beams corresponding to "hi bixby what eshh" and the candidate beams corresponding to "hi bixby what is thea". The decoding apparatus may determine candidate beams ("hi bixby what") accumulated up to the reuse time to be a reuse token.

The decoding apparatus may determine a reuse token corresponding to a stable time (that is, a reuse time) by comparing partial decoding results of previous times, as described above. Also, the decoding apparatus may regard a candidate beam of the reuse time as a stable candidate beam and may use the candidate beam as an input of a decoder instead of a start token. The decoding apparatus may start decoding using reuse tokens to predict a token of a next step as if auto-regressive decoding continues to be performed from a start token.

The decoding apparatus may recalculate a probability of a candidate beam that is to be used when a beam search is started from a stable candidate beam and a newly accumulated sound source in a next partial decoding step. For example, a hidden state used for an attention calculation may be calculated at once without a need to repeat multiple decoding steps through a matrix operation, similar to training of the ANN.

<FIG> illustrates an example of determining reuse tokens. <FIG> illustrates a process of determining reuse tokens instead of using a beam search algorithm.

When a decoding apparatus receives its output again by an auto-regressive scheme and predicts a probability of a next token for all reuse tokens, all possible combinations may be compared.

For example, a current time may be assumed to correspond to an n-th partial decoding step <NUM> among partial decoding steps. In this example, the decoding apparatus may determine a candidate beam (for example,  hi bixby) up to a third token that corresponds to a time in which a largest number of substrings match in output results (for example, candidate beams) of two steps (for example, the (n-<NUM>)-th partial decoding step <NUM> and the (n-<NUM>)-th partial decoding step <NUM>) immediately preceding the n-th partial decoding step <NUM>, as a stable candidate beam.

The decoding apparatus may determine the stable candidate beam " hi bixby" <NUM>, instead of a start token , as a reuse token when the n-th partial decoding step <NUM> is started by the above-described relay decoding, and may use the stable candidate beam " hi bixby" <NUM> as an input <NUM> of a decoder. The decoding apparatus may copy the candidate beam " hi bixby" <NUM> to the input <NUM> of the decoder.

In the n-th partial decoding step <NUM>, the decoding apparatus may recalculate a probability of a candidate beam that is to be used for a beam search from the candidate beam " hi bixby" <NUM> and a newly accumulated sound source (for example, "what").

However, when the same portion (for example, up to the third token ( hi bixby)) is determined as the candidate beam <NUM> and is completely copied to the input <NUM> of the decoder during auto-regressive decoding, similar to the n-th partial decoding step <NUM>, a size of a beam may be reduced to "<NUM>" for each partial decoding step, which may lead to a decrease in performance.

<FIG> illustrates another example of determining reuse tokens. <FIG> illustrates a process of determining a reuse token using a beam search algorithm.

In an example, probabilities of combinations of a maximum number of tokens may be calculated using a beam search algorithm that uses top several token candidates having high probabilities among tokens as inputs again, and a correct answer similar to an optimum may be found among the probabilities. Thus, the decoding apparatus of one or more embodiments may prevent performance from decreasing due to a reduction in a size of a beam to "<NUM>" for each partial decoding step.

For example, when a decoding step is unconditionally repeated in a state in which candidate beams having a highest probability are maintained, a predicted probability of an output token may gradually decrease. When a token candidate or a candidate beam having a higher probability than a candidate beam currently having a highest probability fails to be found, the decoding apparatus may terminate the beam search algorithm and may output a candidate beam having a highest probability among found candidate beams as a final decoding result.

For example, a current time may be assumed to correspond to an n-th step <NUM> among partial decoding steps. The decoding apparatus may regard a candidate beam (for example,  hi bixby) up to a third token that corresponds to a time in which a largest number of substrings match in output results (for example, candidate beams) of two partial decoding steps (for example, an (n-<NUM>)-th step <NUM> and an (n-<NUM>)-th step <NUM>) immediately preceding the n-th step <NUM>, as a stable candidate beam. In this example, the decoding apparatus may store a beam state corresponding to a result obtained by performing the beam search algorithm for each immediately preceding partial decoding step. The beam state may indicate sequences (hereinafter, referred to as "beams") input to a decoder layer in auto-regressive decoding. The decoder layer may output probabilities of all tokens (for example, the same number of tokens as a product of beams and a total number of tokens) obtained in a current step. When a probability of an original beam is multiplied by a probability of a token output at the current step, a probability of a sentence that is output up to the current step may be obtained. The decoding apparatus may extract the same number of tokens as a size of a beam in a descending order of probabilities among sentences that are increased by one token, and may use the extracted tokens as a beam to be input to the decoder layer in a next auto-regressive decoding step. Since the decoding apparatus selects a fixed number every time, the expression "beam search" may be used due to a constant width.

The beam state may correspond to a result of extracting a preset number of tokens in a descending order of probabilities among sentences that are increased by one token for each decoding step, as described above. Generally, a token finally applied to the decoder layer in a next decoding step may also be called a beam. However, in an example, a preset number of tokens may be stored in a descending order of probabilities in each step and the stored tokens may be called a "beam state".

The decoding apparatus may store a beam state of the (n-<NUM>)-th step <NUM> and a beam state <NUM> of the (n-<NUM>)-th step <NUM>.

When a position of a stable time is determined as, for example, a position up to the third token by comparing outputs of two immediately preceding partial decoding steps, the decoding apparatus may directly use the beam state <NUM> of the (n-<NUM>)-th step <NUM> corresponding to the position of the stable time, for relay decoding.

The decoding apparatus may use the beam state <NUM> that is a result obtained by performing the beam search algorithm up to the position of the stable time (for example, the third token) as an input <NUM> of a decoder, instead of a start token , when the n-th step <NUM> is started.

In the n-th step <NUM>, the decoding apparatus may recalculate a probability of a candidate beam that is to be used for a beam search from the input <NUM> and a newly accumulated sound source (for example, "what").

<FIG> illustrates an example of a decoding method. <FIG> illustrates an example <NUM> in which a number of times a partial decoding step is performed is reduced through the above-described relay decoding.

In an example, through the above-described relay decoding, tokens corresponding to a stable time may be reused, instead of a start token, in a next step, as indicated by an arrow <NUM> in the example <NUM>, thereby reducing execution of preceding steps <NUM> corresponding to the stable time during partial decoding. A decoding apparatus may perform auto-regressive steps <NUM> other than the preceding steps <NUM> in each partial decoding.

<FIG> illustrates another example of a decoding method. <FIG> illustrates an example <NUM> in which a number of times a partial decoding step is performed is reduced through the above-described limited relay decoding.

Similar to the description of <FIG>, a decoding apparatus may reuse tokens corresponding to a stable time, instead of a start token, in a next step, as indicated by arrows <NUM> and <NUM> in the example <NUM>, thereby reducing execution of preceding steps <NUM> corresponding to the stable time during partial decoding.

Also, the decoding apparatus may limit a number of repetitions of auto-regressive steps other than the preceding steps <NUM> in each partial decoding to, for example, three times or five times, and may not perform decoding in a following step <NUM> that exceeds the limited number.

For example, during training, a training sample may temporally include a silence back and forth due to a recording environment. In a sound source that is suddenly cut, the silence may be determined as a stable time, and accordingly a decoding result of a portion neighboring a cut-off portion may not be accurate. Also, it may be difficult to speak at least two or three tokens within <NUM> milliseconds (ms) that is one unit in which partial decoding is performed, at a normal speech rate of a person. Thus, a number of reuse tokens that may be additionally obtained in one partial decoding may not exceed a predetermined number of tokens.

In an example, when a sound source input is not ended and is accumulated, a number of repetitions of auto-regressive steps may be limited to three, the auto-regressive steps may be repeated three times after the preceding steps <NUM> and the following step <NUM> that exceeds the limited number of repetitions may be performed, which may lead to a reduction in a memory and a processing load.

In another example, when a sound source input is ended, the decoding apparatus may perform decoding up to an end of the sound source based on an end condition of an original beam search algorithm in a last partial decoding step <NUM> after the sound source input is ended.

<FIG> illustrates an example of an operating process of a decoding apparatus. Referring to <FIG>, a decoding apparatus <NUM> may include an encoder module <NUM> including an encoder layer, a limited auto-regressive decoding module <NUM> (hereinafter, referred to as a "decoding module <NUM>"), a stabilized beam decision module <NUM>, and a probability recalculation module <NUM>.

The decoding module <NUM> may include a decoder module <NUM>, and a beam search module <NUM>. The beam search module <NUM> may determine at least one candidate beam corresponding to a reuse time of tokens by a beam search algorithm that reuses candidate beams having high probabilities among candidate beams. The decoder module <NUM> may decode at least one token by inputting the at least one candidate beam corresponding to the reuse time of the tokens determined by the beam search module <NUM> among the candidate beams to the beam search module <NUM>.

For example, in an (n-<NUM>)-th partial decoding step, an input sequence <NUM> in a form of a partial wave corresponding to an input speech of a speaker may be assumed to be received.

The decoding apparatus <NUM> may generate an encoded vector sequence by encoding the input sequence <NUM> using an encoder module <NUM>-<NUM>. The encoded vector sequence may be transferred to a decoding module <NUM>-<NUM>. The encoded vector sequence may be applied to a decoder module <NUM>-<NUM> and may be decoded. A decoding result of the decoder module <NUM>-<NUM> may be transferred to a beam search module <NUM>-<NUM>. The beam search module <NUM>-<NUM> may determine a candidate beam having a highest probability among candidate beams corresponding to the decoding result of the decoder module <NUM>-<NUM> by the beam search algorithm. The candidate beam having the highest probability may be output as a decoding result <NUM> of the (n-<NUM>)-th partial decoding step, and may be reinput to the decoder module <NUM>-<NUM> for auto-regressive decoding.

In an (n-<NUM>)-th partial decoding step after <NUM> msec elapses, an input sequence <NUM> in a form of a partial wave corresponding to an input speech of a speaker may be assumed to be received. The input sequence <NUM> may be obtained by additionally accumulating a subsequent input speech corresponding to <NUM> msec to the input sequence <NUM>.

Since there are no two partial decoding steps immediately preceding the (n-<NUM>)-th partial decoding step and the (n-<NUM>)-th partial decoding step, the decoding apparatus <NUM> may execute general auto-regressive decoding a limited number of times (for example, three or five times) without a candidate beam.

The decoding apparatus <NUM> may store all candidate beams that are to be reinput to the decoder module <NUM> for auto-regressive decoding when performing the beam search algorithm in each partial decoding step. Also, the decoding apparatus <NUM> may store all beam states corresponding to results obtained by performing the beam search algorithm for each decoding step.

For example, the stabilized beam decision module <NUM> may compare decoding results of two immediately preceding steps, for example, the decoding result <NUM> of the (n-<NUM>)-th partial decoding step and the decoding result <NUM> of the (n-<NUM>)-th partial decoding step, and may determine a time in which a largest number of substrings match as a stable decoding step.

The stabilized beam decision module <NUM> may call a candidate beam corresponding to the stable decoding step among stored candidate beams, or a beam state corresponding to results obtained by performing the beam search algorithm in the two immediately preceding steps, according to execution of the two immediately preceding steps, and may provide the called candidate beam or the called beam state to a decoder module <NUM>-<NUM>. The decoding apparatus <NUM> may start an n-th partial decoding step from the called candidate beam or the called beam state.

In the n-th partial decoding step, the decoding apparatus <NUM> may generate an encoded vector sequence by encoding an input sequence <NUM> using an encoder module <NUM>-<NUM>. The encoder module <NUM>-<NUM> may transfer the encoded vector sequence generated by encoding the input sequence <NUM> to the probability recalculation module <NUM>.

Since the input sequence <NUM> is increased by <NUM> milliseconds (ms) compared to the (n-<NUM>)-th partial decoding step, the decoding apparatus <NUM> may recalculate a probability of a candidate beam associated with an increased amount of the sound source using the probability recalculation module <NUM>. Since tokens of the candidate beam are known already, the probability recalculation module <NUM> may obtain probabilities all at once by calculating probabilities for each of the tokens in parallel and summing the probabilities, instead of needing to perform auto-regressive decoding that sequentially repeats decoding steps. Thus, a time used to repeat auto-regressive decoding up to a corresponding step may be reduced despite the same number of operations.

The decoding apparatus <NUM> may prepare a candidate beam to be input to the decoder module <NUM>-<NUM> and an initial probability value to be input to a beam search module <NUM>-<NUM>, for the n-th partial decoding step, so that a decoding module <NUM>-<NUM> may perform auto-regressive decoding a limited number of times.

The decoder module <NUM>-<NUM> may decode the encoded vector sequence generated by encoding the input sequence <NUM>, and the candidate beam corresponding to the stable decoding step, or the beam state corresponding to results obtained by performing the beam search algorithm in the two immediately preceding steps. The candidate beam and the beam state may be received from the stabilized beam decision module <NUM>. An output result of the decoder module <NUM>-<NUM> may correspond to, for example, "<NUM> × <NUM>" candidate beams.

The beam search module <NUM>-<NUM> may output a candidate beam having a highest probability among the "<NUM> × <NUM>" candidate beams as a decoding result <NUM> of the n-th partial decoding step.

In an example, by forcing only a beginning portion having a high possibility of being stabilized to be decoded, a probability of an unstable sound source may decrease, and thus the decoding apparatus <NUM> of one or more embodiments may prevent a decoding step from being frequently repeated. Also, by performing auto-regressive decoding a set number of times, the decoding apparatus <NUM> of one or more embodiments may maintain a constant amount of time used for each partial decoding step even though a length of a sound source increases and the sound source is delayed.

In an example, when a high probability is maintained because a speech input has a clear pronunciation, the stabilized beam decision module <NUM> may determine that most of additional decoding results continue to be stable in each partial decoding step, and a number of stable (sound source) regions may quickly increase for each partial decoding step. In another example, when a decoding result is frequently changed, the stabilized beam decision module <NUM> may discard an added result in each partial decoding step and may repeatedly perform decoding until waves are sufficiently accumulated so that the decoding result is stabilized.

In a last partial decoding step in which no sound source is received, the decoding apparatus <NUM> may perform decoding until the beam search module <NUM> satisfies an end condition, without a limitation on a number of times for decoding, and may output a result.

<FIG> illustrates an example of a decoding apparatus. Referring to <FIG>, a decoding apparatus <NUM> may include a communication interface <NUM>, a processor <NUM> (e.g., one or more processors), and a memory <NUM> (e.g., one or more memories). The communication interface <NUM>, the processor <NUM>, and the memory <NUM> may communicate with each other via a communication bus <NUM>.

The communication interface <NUM> may receive an input sequence corresponding to an input speech at a current time.

The processor <NUM> may generate an encoded vector sequence by encoding the input sequence. The processor <NUM> may determine reuse tokens from candidate beams of at least two previous times by comparing the candidate beams of the previous times. The processor <NUM> may decode at least one token subsequent to the reuse tokens based on the reuse tokens and the encoded vector sequence. The processor <NUM> may generate a speech recognition result of the input speech based on the decoded at least one token subsequent to the reuse tokens. The communication interface <NUM> may output the speech recognition result (e.g., through a display and/or speaker of the communication interface <NUM>).

Also, the processor <NUM> may perform any or all of the methods described with reference to <FIG>, and/or one or more algorithms corresponding to any or all of the methods. The processor <NUM> may be a hardware-implemented data processing apparatus having a circuit that is physically structured to execute desired operations. For example, the desired operations may include code or instructions included in a program. The hardware-implemented data processing apparatus may include, for example, a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

The processor <NUM> may execute a program and may control the decoding apparatus <NUM>. Code of the program executed by the processor <NUM> may be stored in the memory <NUM>.

The memory <NUM> may store candidate beams that are to be used for decoding of a next time for each of previous times of decoding. The memory <NUM> may store a variety of information generated in a processing process of the above-described processor <NUM>. Also, the memory <NUM> may store a variety of data and programs. The memory <NUM> may include, for example, a volatile memory or a non-volatile memory. The memory <NUM> may include a high-capacity storage medium such as a hard disk to store a variety of data.

The decoding apparatuses, encoder modules, decoding modules, decoder modules, beam search modules, stabilized beam decision modules, probability recalculation modules, communication interfaces, processors, memories, communication buses, decoding apparatus <NUM>, encoder module <NUM>, encoder module <NUM>-<NUM>, encoder module <NUM>-<NUM>, encoder module <NUM>-<NUM>, decoding module <NUM>, decoding module <NUM>-<NUM>, decoding module <NUM>-<NUM>, decoding module <NUM>-<NUM>, decoder module <NUM>, decoder module <NUM>-<NUM>, decoder module <NUM>-<NUM>, decoder module <NUM>-<NUM>, beam search module <NUM>, beam search module <NUM>-<NUM>, beam search module <NUM>-<NUM>, beam search module <NUM>-<NUM>, stabilized beam decision module <NUM>, probability recalculation module <NUM>, decoding apparatus <NUM>, communication interface <NUM>, processor <NUM>, memory <NUM>, communication bus <NUM>, and other apparatuses, units, modules, devices, and components described herein with respect to <FIG> are implemented by or representative of hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the scope of the claims.

Claim 1:
A decoding method, the method comprising:
in a neural network 'NN' for speech recognition,
generating (<NUM>) an encoded vector sequence by encoding an input sequence corresponding to an input speech at a current time,
determining (<NUM>) a reuse time of tokens at a current time n, being the current time, subsequent to a previous time n-<NUM> subsequent to a previous time n-<NUM> based on a comparison result between a candidate beam of the previous time n-<NUM> and a candidate beam of the previous time n-<NUM>, wherein n is a natural number greater than or equal to "<NUM>";
determining candidate beams accumulated up to the reuse time to be the reuse tokens; and
decoding (<NUM>) one or more tokens subsequent to the reuse tokens based on the reuse tokens and the encoded vector sequence;
wherein the determining of the reuse time comprises determining a time in which a largest number of substrings match in the candidate beam of the previous time n-<NUM> and the candidate beam of the previous time n-<NUM>, as the reuse time of the tokens at the current time n.