Fast emit low-latency streaming ASR with sequence-level emission regularization utilizing forward and backward probabilities between nodes of an alignment lattice

A computer-implemented method of training a streaming speech recognition model that includes receiving, as input to the streaming speech recognition model, a sequence of acoustic frames. The streaming speech recognition model is configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of vocabulary tokens. The vocabulary tokens include a plurality of label tokens and a blank token. At each output step, the method includes determining a first probability of emitting one of the label tokens and determining a second probability of emitting the blank token. The method also includes generating the alignment probability at a sequence level based on the first probability and the second probability. The method also includes applying a tuning parameter to the alignment probability at the sequence level to maximize the first probability of emitting one of the label tokens.

TECHNICAL FIELD

This disclosure relates to using fast emit low-latency streaming ASR with sequence-level emission regularization.

BACKGROUND

Automatic speech recognition (ASR) attempts to provide accurate transcriptions of what a person has said by taking an audio input and transcribing the audio input into text. Streaming ASR models aim to achieve transcribing each word in the audio input as quickly and accurately as possible. End-to-end (E2E) recurrent neural network transducer (RNN-T) models have gained enormous popularity for streaming ASR models. These streaming ASR models learn to predict best by using future context of the audio input, which causes a significant delay between the user speaking and transcription generation. Some approaches, manipulate probabilities of the transcription in order to reduce the amount of delay. However, while manipulating probabilities of the transcription provides some success in reducing latency of streaming ASR models, the success comes at the cost of suffering from severe accuracy regression.

SUMMARY

One aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations for training a streaming speech recognition model. The operations include receiving, as input to the speech recognition model, a sequence of acoustic frames. The streaming speech recognition model is configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of vocabulary tokens. The vocabulary tokens include a plurality of label tokens and a blank token. At each step of a plurality of output steps, the operations include determining a first probability of emitting one of the label tokens and determining a second probability of emitting the blank token. The operations also include generating the alignment probability at a sequence level based on the first probability of emitting one of the label tokens and the second probability of emitting the blank token at each output step. The operations also include applying a tuning parameter to the alignment probability at the sequence level to maximize the first probability of emitting one of the label tokens.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the first probability of emitting one of the label tokens at the respective step corresponds to a probability of emitting one of the label tokens after previously emitting a respective label token. The second probability of emitting the blank token at the respective step may correspond to a probability of emitting the blank label after emitting one of the blank label or a label token at a step immediately preceding the respective step. Optionally, the first probability and the second probability may define a forward variable of a forward-backward propagation algorithm.

In some examples, the operations further include generating an alignment lattice that includes a plurality of nodes, the alignment lattice is defined as a matrix with T columns of nodes and U rows of nodes. Here, each column of the T columns corresponds to a corresponding step of the plurality of output steps and each row of the U rows corresponds to a label that textually represents the sequence of acoustic frames. In these examples, at each node location in the matrix of the alignment lattice, the operations may further include: determining a forward probability for predicting a subsequent node adjacent to the respective node, the forward probability includes the first probability and the second probability; and determining, from the subsequent node adjacent to the respective node, a backward probability of including the respective node in an output sequence of vocabulary tokens. Generating the alignment probability at the sequence level may include aggregating the forward probability and the backward probability for all nodes at each respective step of the alignment lattice.

In some implementations, applying the tuning parameter to the alignment probability at the sequence level balances a loss at the streaming speech recognition model and a regularization loss when training the streaming speech recognition model. The tuning parameter may be applied independent of any speech-word alignment information. In some examples, emission of the blank token at one of the output steps is not penalized. Optionally, the streaming speech recognition model may include at least one of a recurrent neural-transducer (RNN-T) model, a Transformer-Transducer model, a Convolutional Network-Transducer (ConvNet-Transducer) model, or a Conformer-Transducer model. The streaming speech recognition model may include a recurrent neural-transducer (RNN-T) model. The streaming speech recognition model may include a Conformer-Transducer model. In some implementations, after training the streaming speech recognition model, the trained streaming speech recognition model executes on a user device to transcribe speech in a streaming fashion. In other implementations, after training the streaming speech recognition model, the trained streaming speech recognition model executes on a server.

Another aspect of the disclosure provides a system of training a streaming speech recognition model. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving, as input to the speech recognition model, a sequence of acoustic frames. The streaming speech recognition model is configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of vocabulary tokens. The vocabulary tokens include a plurality of label tokens and a blank token. At each step of a plurality of output steps, the operations include determining a first probability of emitting one of the label tokens and determining a second probability of emitting the blank token. The operations also include generating the alignment probability at a sequence level based on the first probability of emitting one of the label tokens and the second probability of emitting the blank token at each output step. The operations also include applying a tuning parameter to the alignment probability at the sequence level to maximize the first probability of emitting one of the label tokens.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the first probability of emitting one of the label tokens at the respective step corresponds to a probability of emitting one of the label tokens after previously emitting a respective label token. The second probability of emitting the blank token at the respective step may correspond to a probability of emitting the blank label after emitting one of the blank label or a label token at a step immediately preceding the respective step. Optionally, the first probability and the second probability may define a forward variable of a forward-backward propagation algorithm.

In some examples, the operations further include generating an alignment lattice that includes a plurality of nodes, the alignment lattice is defined as a matrix with T columns of nodes and U rows of nodes. Here, each column of the T columns corresponds to a corresponding step of the plurality of output steps and each row of the U rows corresponds to a label that textually represents the sequence of acoustic frames. In these examples, at each node location in the matrix of the alignment lattice, the operations may further include: determining a forward probability for predicting a subsequent node adjacent to the respective node, the forward probability includes the first probability and the second probability; and determining, from the subsequent node adjacent to the respective node, a backward probability of including the respective node in an output sequence of vocabulary tokens. Generating the alignment probability at the sequence level may include aggregating the forward probability and the backward probability for all nodes at each respective step of the alignment lattice.

In some implementations, applying the tuning parameter to the alignment probability at the sequence level balances a loss at the streaming speech recognition model and a regularization loss when training the streaming speech recognition model. The tuning parameter may be applied independent of any speech-word alignment information. In some examples, emission of the blank token at one of the output steps is not penalized. Optionally, the streaming speech recognition model may include at least one of a recurrent neural-transducer (RNN-T) model, a Transformer-Transducer model, a Convolutional Network-Transducer (ConvNet-Transducer) model, or a Conformer-Transducer model. The streaming speech recognition model may include a recurrent neural-transducer (RNN-T) model. The streaming speech recognition model may include a Conformer-Transducer model. In some implementations, after training the streaming speech recognition model, the trained streaming speech recognition model executes on a user device to transcribe speech in a streaming fashion. In other implementations, after training the streaming speech recognition model, the trained streaming speech recognition model executes on a server.

DETAILED DESCRIPTION

Streaming automated speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible. However, reducing emission delay (i.e., the delay between a user speaking and text appearing) of each hypothesized word while retaining accuracy is challenging. Some approaches regularize or penalize emission delay by manipulating per-token or per-frame probability predictions in transducer models. While penalizing emission delay by manipulating per-token or per-frame probabilities successfully reduces emission delay, these approaches suffer from significant accuracy regressions. To increase accuracy of streaming speech recognition results, implementations herein are directed toward a method of training a sequence-level streaming speech recognition model. In particular, training the transducer model aims to reduce the emission latency while without suffering from accuracy regression. The emission latency represents the time period between when the user finishes speaking and when a transcription for the last word spoken by the user appears.

Referring now toFIG.1, an example speech environment100includes an automated speech recognition (ASR) system130that resides on a user device102of a user10and/or on a remote computing device160(e.g., one or more servers of a distributed system executing in a cloud-computing environment) in communication with the user device102via a network150. Although the user device102is depicted as a mobile computing device (e.g., a smart phone), the user device102may correspond to any type of computing device such as, without limitation, a tablet device, a laptop/desktop computer, a wearable device, a digital assistant device, a smart speaker/display, a smart appliance, an automotive infotainment system, or an Internet-of-Things (IoT) device. The user device102includes data processing hardware104and memory hardware106in communication with the data processing hardware104and stores instructions, that when executed by the data processing hardware104, cause the data processing hardware104to perform one or more operations.

The user device102further includes an audio system116with an audio capture device (e.g., microphone)116,116afor capturing and converting spoken utterances12within the speech environment100into electrical signals and a speech output device (e.g., a speaker)116,116bfor communicating an audible audio signal (e.g., as output audio data from the user device102). While the user device102implements a single audio capture device116ain the example shown, the user device102may implement an array of audio capture devices116awithout departing from the scope of the present disclosure, whereby one or more capture devices116ain the array may not physically reside on the user device102, but be in communication with the audio system116.

The user device102includes an audio subsystem120configured to receive an utterance12(e.g., captured by the one or more microphones116a) spoken by the user10and converts the utterance12into a corresponding digital format associated with input acoustic frames122capable of being processed by the ASR system130. In the example shown, the user10speaks a respective utterance12in a natural language of English for the phrase “What song is playing now?” and the audio subsystem120converts the utterance12into a corresponding sequence of acoustic frames122for input to the ASR system130. Thereafter, the ASR system130receives, as input, the acoustic frames122corresponding to the utterance12, and generates/predicts, as output, a corresponding transcription (e.g., recognition result/hypothesis)132of the utterance12. The time period between when the user10stops talking124(e.g., end of speech (EOS)124) and when the last token of the transcription (e.g. end of transcription134) is transcribed represents the emission latency136.

In the example shown, the user device102and/or the remote computing device160also executes a user interface generator140configured to present a representation of the transcription132of the utterance12to the user10of the user device102. In some configurations, the transcription132output from the ASR system130is processed, e.g., by a natural language understanding (NLU) module executing on the user device102or the remote computing device160, to execute a user command. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device102or the remote computing device160) may convert the transcription132into synthesized speech for audible output by another device. For instance, the original utterance12may correspond to a message the user10is sending to a friend in which the transcription132is converted to synthesized speech for audible output to the friend to listen to the message conveyed in the original utterance12. As shown in FIG.1A, an example speech environment100,100agenerates the transcription132with the ASR system130.

The ASR system130includes a streaming ASR model200that is configured to reduce the emission latency136between the EOS124and the end of transcription134. A training process201(FIG.2B) trains the ASR model200to encourage the ASR model200to emit characters of the transcription132rather than emitting blanks without penalizing the emission of blanks. In some examples, the ASR model200maximizes the probability of emitting a character transcriptions at a sequence level rather than at a per-frame or per-token level.

FIG.2Aillustrates an example ASR model200that includes a Recurrent Neural Network-Transducer (RNN-T) model architecture which adheres to latency constrains associated with interactive applications. The RNN-T model200provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device102(e.g., no communication with a remote server is required). WhileFIG.2Adepicts the ASR model200including the RNN-T model architecture, the ASR model200may also include other types of transducer models such as a Transformer-Transducer model architecture, a Convolutional Neural Network-Transducer (CNN-Transducer) model architecture, a Convolutional Network-Transducer (ConvNet-Transducer) model, or a Conformer-Transducer model architecture without departing from the scope of the present disclosure. An example Transformer-Transducer model architecture is described in detail in “Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss,” https://arxiv.org/pdf/2002.02562.pdf, the contents of which are incorporated by reference in their entirety. An example CNN-Transducer model architecture is described in detail in “Contextnet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context,” https://arxiv.org/abs/2005.03191, the contents of which are incorporated by reference in their entirety. An example Conformer-Transducer model architecture is described in detail in “Conformer: Convolution-augmented transformer for speech recognition,” https://arxiv.org/abs/2005.08100, the contents of which are incorporated by reference in their entirety.

The RNN-T model200ofFIG.2Aincludes an encoder network210, a prediction network220, and a joint network230. The encoder network210, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a recurrent network of stacked Long Short-Term Memory (LSTM) layers. For instance the encoder reads a sequence of d-dimensional feature vectors (e.g., acoustic frames122(FIG.1)) x=(x1, x2, . . . , xT), where xt∈d, and produces at each time step a higher-order feature representation. This higher-order feature representation is denoted as h1enc, . . . , hTenc.

Similarly, the prediction network220is also an LSTM network, which, like a language model (LM), processes the sequence of non-blank tokens output by a final Softmax layer240so far, y0, . . . , yui-1, into a dense representation pui. Finally, with the RNN-T model architecture, the representations produced by the encoder and prediction networks210,220are combined by the joint network230. The joint network then predicts P(yi|x1, . . . , xti, y0, . . . , yui-1), which is a distribution over the next output token. Stated differently, the joint network230generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses. Here, the “possible speech recognition hypotheses” correspond to a set of label tokens204each representing a symbol/character in a specified natural language. For example, when the natural language is English, the set of label tokens may include twenty-seven (27) symbols, e.g., one label token for each of the 26-letters in the English alphabet and one label designating a space. Accordingly, the joint network230may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of label token. This set of values can be a vector and can indicate a probability distribution over the set of label tokens. In some cases, the label tokens are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of label tokens is not so limited. For example, the set of label tokens can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution of the joint network230can include a posterior probability value for each of the different label tokens. Thus, if there are 100 different label tokens representing different graphemes or other symbols, the output yiof the joint network230can include 100 different probability values, one for each label token. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer240) for determining the transcription132.

The Softmax layer240may employ any technique to select the label token with the highest probability in the distribution as the next output symbol predicted by the transducer model200at the corresponding output step. In this manner, the RNN-T model200does not make a conditional independence assumption, rather the prediction of each label token is conditioned not only on the acoustics but also on the sequence of label tokens emitted so far. The RNN-T model200does assume a label token is independent of future acoustic frames110, which allows the RNN-T model to be employed in a streaming fashion.

In some examples, the encoder network210of the RNN-T model200is made up of eight 2,048-dimensional LSTM layers, each followed by a 540-dimensional projection layer. The prediction network220may have two 2,048-dimensional LSTM layers, each of which is also followed by 540-dimensional projection layer. Finally, the joint network230may also have 540 hidden units. The softmax layer240may be composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes in a plurality of training data sets.

FIG.2Billustrates a training process201for training the ASR model200. Training the transducer-based streaming ASR model200aims to minimize the emission latency136without suffering from accuracy regression. The transducer-based streaming ASR model200may interchangeably referred to as a ‘transducer model200’. The transducer model200receives a sequence of acoustic frames122from the audio subsystem120and is configured to learn an alignment probability206between the sequence of acoustic frames122(e.g., x=(x1, x2, . . . , xT) and an output sequence of vocabulary tokens204(e.g., y=(y1, y2, . . . , yU). The vocabulary tokens204are output elements of the transducer model200that include a plurality of label tokens204,204a(FIG.3A) and a blank token204,204b(FIG.3A). The label tokens204aare textual representations of the utterance12that may include graphemes (e.g., individual characters, and potentially punctuation and other symbols), wordpieces, and/or entire words. The blank token204bis a textual representation of the utterance12for a blank/empty output. Accordingly, the output sequence of vocabulary tokens includes a sequence of both the label tokens204aand the blank token204bto represent a transcription of the utterance12. In some examples, the vocabulary tokens204represent a character vocabulary with each label token204arepresenting an alphabetic character (i.e., A-Z) and the blank token204brepresenting a blank space. In other examples, the vocabulary tokens204represent a wordpiece vocabulary with each label token204arepresenting one or more alphabetic characters and the blank token204brepresenting a blank space. The vocabulary tokens204may also represent punctuation and other symbols. The vocabulary tokens204may include any combination of character vocabulary, wordpiece vocabulary, and/or punctuation and other symbols.

The alignment probability206refers to the likelihood of the transducer model200emitting a respective output sequence of vocabulary tokens204from all possible output sequences of vocabulary tokens204. To learn the alignment probability206the transducer model200extends the output sequence with blank tokens204b. Training the transducer model200aims to maximize the log-probability of a conditional distribution represented by:

In Equation (1),represents the transducer loss, a represents an alignment lattice, ŷ represents the ground truth output sequence of label tokens204a, x represents the sequence of acoustic frames122, and B represents a function that removes the blank tokens204bfrom the alignment lattice a. In some implementations, the training process201applies a lattice generator250, a forward probability layer260, a backward probability layer270, an alignment model280, and a loss model290. The lattice generator250is configured to generate an alignment lattice300for the sequence of acoustic frames122input to the ASR model200. The lattice generator250may be composed of the encoder210, the prediction network220, and the joint network230of the transducer model200. The alignment lattice300includes a matrix of vocabulary tokens204(e.g., label tokens204aand blank tokens204b) that correspond the sequence of acoustic frames122for the utterance12. At each output step, the transducer model200emits one of the label tokens204aor the blank token204bfrom the alignment lattice300.

Referring now toFIGS.3A-3Cthe alignment lattice300includes a matrix having a plurality of nodes254,254a-n. Each node254in the plurality of nodes254represents one of the vocabulary tokens204. That is, the alignment lattice300includes U rows of nodes254, each row corresponding to a label token204athat textually represents a portion of the sequence of acoustic frames122. Additionally, the alignment lattice300includes T columns of nodes254, each column corresponding to an output step from the plurality output steps. The transducer model200emits one of the label tokens204aor the blank token204bat each output step. The number of T columns of nodes254depends on the amount of output steps required to emit all of the label tokens204afor the corresponding sequence of acoustic frames122. The lattice generator250generates the alignment lattice300based on the sequence of acoustic frames122for the utterance12. Referring now toFIGS.2B and3A, the alignment lattice300includes five (5) rows of nodes254and seven (7) columns of nodes254for the utterance12“HELLO.” The label token204aof each row of the alignment lattice300represents an alphabetic character of the word “HELLO.” Here, the alignment lattice300includes seven columns because the transducer model200requires seven (7) output steps to emit the utterance12“HELLO.” The alignment lattice300may include any number of T columns and U rows required to represent the corresponding sequence of acoustic frames122.

The alignment lattice300provides the transducer model200with a matrix of vocabulary tokens204to generate the output sequence of vocabulary tokens204. That is, the transducer model200determines, at each node254in the alignment lattice300, whether to emit one of the label tokens204aor the blank token204b. Accordingly, at each output step the transducer model200either emits one of the label tokens204a(e.g., up arrow) or emits the blank token204b(e.g., right arrow). The transducer model200continues outputting vocabulary tokens204until the last label token204aemits. Once the transducer model200emits the last label token204athe output sequence of vocabulary tokens204is complete. The lattice generator250sends the alignment lattice300and the plurality of nodes254to the forward probability layer260and the backward probability layer270.

The forward probability layer260and backward probability layer270are configured to determine a likelihood of emitting one of the label tokens204aor the blank token204b. The forward probability layer260determines the likelihood of emitting vocabulary tokens204based on a per-token and/or per-frame probability. That is, the forward probability layer260determines the likelihood of emitting vocabulary tokens204based on only on the vocabulary token204or frame. The backward probability layer270determines the likelihood of emitting vocabulary tokens204based on a per-sequence probability. Accordingly, the backward probability layer270takes into account the previously emitted vocabulary tokens204when determining which vocabulary token204to emit next. Taken together, the forward probability layer260and the backward probability layer270determine which vocabulary token204to emit based on a per-token/per-frame and per-sequence probability.

In particular, the forward probability layer260determines a likelihood of emitting one of the label tokens204aor the blank token204bat the at a subsequent node254,254S of the alignment lattice300. That is, the forward probability layer260determines, from a respective node254of the alignment lattice300, the likelihood of emitting one of the label tokens204aor the blank token204bat the subsequent node254S. Here, the subsequent node254S is adjacent to the respective node254. The subsequent node254S may be to the right of the respective node254(e.g., node (T+1, U)) that indicates emitting the blank token204bor above the respective node254(e.g., node (T, U+1) that indicates emitting one of the label tokens204a.

The forward probability layer260determines the likelihood of emitting vocabulary tokens204based on a forward probability262. The forward probability262is represented by:
α(t,u)=ŷ(t,u−1)α(t,u−1)+b(t−1,u)α(t−1,u)  (2)

In Equation 2, α(t, u) represents the forward probability262, ŷ(t, u) represents the label token204a, b(t, u) represents the blank token204b, trepresents the column of the respective node254, and u represents the row of the respective node254.

For example, referring now toFIG.3B, the forward probability layer260determines from a respective node254(e.g., node (T, U)), a forward probability262that includes a first probability264of emitting one of the label tokens204aand a second probability266of emitting the blank token204b. Here, the first probability264represents the likelihood of progressing from the respective node254to a subsequent node254S (e.g., node (T, U+1) to emit one of the label tokens204a. In particular, the first probability264represents the likelihood of emitting the label token204a‘L’ at the next output step. Continuing with the example, the second probability266represents the likelihood of progressing from the respective node254to a subsequent node254S (e.g., node (T+1, U)) to emit the blank token204b. That is, the second probability266represents the likelihood of emitting a blank at the next output step. In some examples, the second probability266of emitting the blank token204bat the respective step corresponds to a probability of emitting the blank token204bafter emitting one of the blank token204bor a label token at a step immediately preceding the respective step.

The backward probability layer270is configured to determine a likelihood of the output sequence including the respective subsequent node254S. The backward probability layer270determines the likelihood of the output sequence including the respective subsequent node254S based on a backward probability272. The backward probability layer270determines the backward probability272based on all possible output sequences202and the proportion of all possible output sequences202that include the respective subsequent node254S represented by:
β(t,u)=ŷ(t,u)β(t,u+1)+b(t,u)β(t+1,u)  (3)

β(t, u) represents the backward probability, ŷ(t, u) represents the label token204a, b(t, u) the blank token204b, t represents the column of the respective node254, and u represents the row of the respective node254. Referring now toFIG.3C, from a subsequent node244S, the backward probability layer270determines the backward probability272of including the respective subsequent node254S in the output sequence202. In the example shown, the backward probability layer270determines three output sequences202, however, it is understood that the backward probability layer270may determine any number of output sequences202. In this example, a first output sequence202,202aand a second output sequence202,202binclude the respective subsequent node254S while a third output sequence202,202cdoes not include the respective subsequent node254S. The backward probability layer270determines the backward probability272based on the number of output sequences202that include the subsequent node254S from all possible output sequences202. In this example, the backward probability272of the output sequence202including the respective subsequent node254S is one-in-three.

In some implementations, the backward probability layer270determines the backward probability272for multiple subsequent nodes254S. That is, from a respective node254there are two possible subsequent nodes254S, either a subsequent node254S that represents the label token204aor a subsequent node254S that represents the blank token204b. In some examples, the backward probability layer270determines the backward probability272for both subsequent nodes254S. In other examples, the backward probability layer270determines a backward probability272only for subsequent nodes254S that satisfy a threshold. That is, the backward probability layer270only determines a backward probability272for a subsequent node254S that represents the label token204awhen the first probability264satisfies a threshold, and for a subsequent node254S that represents the blank token204bwhen the second probability266satisfies a threshold.

In some implementations, a node254of the alignment lattice300may represent both one of the label tokens204aand the blank token204b. Depending on how the output sequence202traverses through the node254will determine if the node254is one of the label tokens204aor the blank token204b. As shown inFIG.3C, two output sequences of vocabulary tokens204A,204B include the subsequent node254C. The first output sequence of vocabulary tokens204A progresses to the subsequent node254S by emitting the blank token204b(e.g., right arrow). In this instance, the subsequent node254S represents the blank token204b. The second output sequence of vocabulary tokens204B progresses to the subsequent node254S by emitting one of the label tokens204a(e.g., up arrow). Here, the subsequent node254S represents the one of the label tokens204a. Thus, whether a node254in the alignment lattice300represents the one of the label tokens204aor the blank token204bdepends on the output sequence of vocabulary tokens204. The forward probability layer260and the backward probability layer270send the forward probability262and the backward probability272, respectively, to the alignment model280. The transducer model200may emit one or more vocabulary tokens204at an output step. For example, at output step T=6 for a third output sequence of vocabulary tokens202C, the transducer model200progresses through three (3) label tokens204acorresponding to the letters “L”, “L”, and “O”. Here, the transducer model200at output step T=6, the transducer model200emits all three (3) of the label tokens204a.

Referring back toFIG.2B, the forward probability layer260and backward probability layer270send the forward probability262and backward probability272to the alignment model280. The alignment model280is configured to determine the alignment probability206based on the forward probability262and backward probability272. That is, the alignment model280generates the alignment probability206at the sequence level by aggregating the forward probability262and the backward probability272for all nodes at each respective output step of the alignment lattice300. The alignment model280determines the alignment probability206based on the following equations:

In Equations 4 and 5, At,urepresents the coordinates of the respective node254in the alignment lattice300. Accordingly, P(At,u|x) represents the probability of all complete output sequences of vocabulary tokens204up through a respective node254and P(ŷ|x) represents the probability of all output sequences of vocabulary tokens204in the alignment lattice300. In Equation 4, α(t, u)b(t, u)β(t+1, u) represents the probability of predicting the blank token204band α(t, u)ŷ(t, u)β(t, u+1) represents the probability of predicting the one of the label tokens204a. Thus, the alignment model280may determine the alignment probability206based on a per-token probability (e.g., forward probability262) and a per-sequence probability (e.g., backward probability272). In some examples, the alignment model280sends the alignment probability206to the ASR system130to determine whether to emit the one of the label tokens204aor the blank token204bto generate the transcription132that corresponds to the utterance19.

In some implementations, the training process201applies a loss model290configured to determine a transducer loss292for any node254of the alignment lattice300. The alignment model280may send the alignment probability206to the loss model290to determine the transducer loss292at each output step. The loss functions may be represented by:

The transducer model200maximizes the log-probability of all possible output sequences of vocabulary tokens204regardless of the emission latency. That is, the transducer model200treats emitting one of the label tokens204aand emitting the blank token the equally because the log-probability (e.g., Equation 1) is maximized. Accordingly, treating the emission of label tokens204aand blank tokens204bequally inevitably leads to emission latency136because transducer models200learn to predict better by using more future context, causing significant emission latency136. That is, the loss model290provides the transducer loss292as feedback to the alignment model280. The alignment model280uses the transducer loss292to minimize errors in vocabulary token204emissions. Thus, because emitting label tokens204aand blank tokens204bare treated equal, the alignment model280inevitably introduces emission latency136. Implementations herein are directed toward training the transducer model200to encourage predicting one of the label tokens204aover the blank token204bby maximizing the probability of the label token204arepresented by:

In Equations 8 and 9, {tilde over (P)}(At,u|x) represents the maximized probability of emitting one of the label tokens204a,represents the updated transducer loss, and A represents a tuning parameter. Thus, the first probability264and the second probability266define a forward variable of a forward-backward propagation algorithm. In particular, the alignment model280uses first probability264and second probability266determine the alignment probability206of emitting the output sequence of vocabulary tokens204and the loss model190uses the first probability264and second probability266to determine the transducer loss292to train the alignment model280. The alignment model280may be incorporated into the joint network230and/or Softmax layer240of the transducer model200ofFIG.2A.

The loss function for the updated transducer loss (e.g., Equation 8) applies a tuning parameter282to maximize the probability of emitting one of the label tokens204aat the sequence level. The tuning parameter282is configured to balance the transducer loss292and a regularization loss294. That is, the tuning parameter282balances the accuracy of vocabulary token204emission (e.g., transducer loss292) and penalizing emission latency136(e.g., regularization loss294). Accordingly, the transducer model200maximizes the probability of emitting one of the label tokens204awithout penalizing the probability of emitting one of the blank tokens204b. The tuning parameter282may be applied to the alignment model280independent of any speech-word alignment information. In some implementations, the tuning parameter282can be implemented based on an existing transducer model200because the new regularized transducer loss can be represented by:

In some examples, after training the transducer model200using the tuning parameter282, the trained transducer model200is configured to execute on the remote computing device160during inference. In other examples, the trained transducer model200execute on the user device102during inference. Executing the trained transducer model200reduces the emission latency136of generating the transcription132of the utterance12by maximizing the probability of emitting one of the label tokens204aat a sequence level without penalizing the probability of emitting the blank token204b.

FIG.4is a flowchart of an exemplary arrangement of operations for a method400of training a streaming speech recognition model. The method400, at step402, includes receiving, as input to the speech recognition model, a sequence of acoustic frames122. The streaming speech recognition model is configured to learn an alignment probability206between the sequence of acoustic frames122and an output sequence of vocabulary tokens204. The vocabulary tokens204include a plurality of label tokens204aand a blank token204b. At each step of a plurality of output steps, the method400, at step404, includes determining a first probability264of emitting one of the label tokens204a. At step406, the method400, includes determining a second probability266of emitting the blank token204b.

The method400, at step408, includes generating the alignment probability206at a sequence level based on the first probability264of emitting one of the label tokens204aand the second probability266of emitting the blank token204bat each output step. The method400, at step410, include applying a tuning parameter282to the alignment probability206at the sequence level to maximize the first probability264of emitting one of the label tokens204a.

The computing device500includes a processor510, memory520, a storage device530, a high-speed interface/controller540connecting to the memory520and high-speed expansion ports550, and a low speed interface/controller560connecting to a low speed bus570and a storage device530. Each of the components510,520,530,540,550, and560, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor510can process instructions for execution within the computing device500, including instructions stored in the memory520or on the storage device530to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display580coupled to high speed interface540. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices500may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The storage device530is capable of providing mass storage for the computing device500. In some implementations, the storage device530is a computer-readable medium. In various different implementations, the storage device530may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory520, the storage device530, or memory on processor510.

The high speed controller540manages bandwidth-intensive operations for the computing device500, while the low speed controller560manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller540is coupled to the memory520, the display580(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports550, which may accept various expansion cards (not shown). In some implementations, the low-speed controller560is coupled to the storage device530and a low-speed expansion port590. The low-speed expansion port590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device500may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server500aor multiple times in a group of such servers500a, as a laptop computer500b, or as part of a rack server system500c.