Patent Description:
Automatic speech recognition (ASR) systems have increased in popularity in recent years for assistant enabled devices. Improving the recognition of words infrequently spoken is an ongoing problem for ASR systems. Words that are infrequently spoken are rarely included in acoustic training data and, therefore, are difficult for ASR systems to accurately recognize in speech. In some instances, ASR systems include language models that train on text-only data to improve recognition of infrequently spoken words. However, these language models often include large memory and computational requirements that lead to a decrease in efficiency of ASR systems.

One aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations for performing speech recognition using a lookup-table recurrent language model, as set forth by independent computer-implemented method claim <NUM> and corresponding independent computer-readable medium claim <NUM>. The operations include receiving audio data that corresponds to an utterance spoken by a user and captured by a user device. The operations also include processing, using a speech recognizer, the audio data to determine a candidate transcription that includes a sequence of tokens for the spoken utterance. For each token in the sequence of tokens, the operations include determining a token embedding for the corresponding token using a first embedding table, determining a n-gram token embedding for a previous sequence of n-gram tokens using a second embedding table, and concatenating the token embedding and the n-gram token embedding to generate a concatenated output for the corresponding token. The operations also include rescoring, using an external language model, the candidate transcription for the spoken utterance by processing the concatenated output generated for each corresponding token in the sequence of tokens.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the external language model includes a recurrent neural network language model, as is e.g. known from the US patent document <CIT>. The external language model may integrate with the speech recognizer by Hybrid Autoregressive Transducer (HAT) factorizing the speech recognizer, as disclosed in the conference papaer "<NPL>. In some examples, the speech recognizer includes a conformer audio encoder and a recurrent neural network-transducer decoder. In other examples, the speech recognizer includes a transformer audio encoder and a recurrent neural network-transducer decoder. Optionally, each token in the sequence of tokens of the candidate transcription may represent a word in the candidate transcription. Each token in the sequence of tokens of the candidate transcription may represent a wordpiece in the candidate transcription.

In some implementations, each token in the sequence of tokens of the candidate transcription represents an n-gram, phoneme, or grapheme in the candidate transcription. In some examples, the first and second embedding tables are stored sparsely on memory hardware in communication with the data processing hardware. Determining the token embedding for the corresponding token may include retrieving the token embedding from the first embedding table via a look-up without requiring access to any graphics processing units and/or tensor processing units. Optionally, the data processing hardware may reside on user device.

Another aspect of the disclosure provides a system that includes data processing hardware and memory hardware storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations, as set forth by independent claim <NUM>. The operations include receiving audio data that corresponds to an utterance spoken by a user and captured by a user device. The operations also include processing, using a speech recognizer, the audio data to determine a candidate transcription that includes a sequence of tokens for the spoken utterance. For each token in the sequence of tokens, the operations include determining a token embedding for the corresponding token using a first embedding table, determining a n-gram token embedding for a previous sequence of n-gram tokens using a second embedding table, and concatenating the token embedding and the n-gram token embedding to generate a concatenated output for the corresponding token. The operations also include rescoring, using an external language model, the candidate transcription for the spoken utterance by processing the concatenated output generated for each corresponding token in the sequence of tokens.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the external language model includes a recurrent neural network language model. The external language model may integrate with the speech recognizer by Hybrid Autoregressive Transducer (HAT) factorizing the speech recognizer. In some examples, the speech recognizer includes a conformer audio encoder and a recurrent neural network-transducer decoder. In other examples, the speech recognizer includes a transformer audio encoder and a recurrent neural network-transducer decoder. Optionally, each token in the sequence of tokens of the candidate transcription may represent a word in the candidate transcription. Each token in the sequence of tokens of the candidate transcription may represent a wordpiece in the candidate transcription.

In some implementations, each token in the sequence of tokens of the candidate transcription represents an n-gram, phoneme, or grapheme in the candidate transcription. In some examples, the first and second embedding tables are stored sparsely on memory hardware in communication with the data processing hardware. Determining the token embedding for the corresponding token may include retrieving the token embedding from the first embedding table via a look-up without requiring access to any graphics processing units and/or tensor processing units. Optionally, the data processing hardware may reside on the user device.

Improving the recognition of rare word or sequences is an ongoing problem in speech recognition systems that misrecognize many input text utterances that appear with zero or low frequency in acoustic data. In particular, proper nouns such as street names, cities, etc. are rarely spoken (i.e., long tail content) and often not included in acoustic training data making the long tail content difficult to recognize for speech recognition systems. In some implementations, speech recognition systems integrate language models that train with text-only data that includes rarely spoken long tail content. That is, language models may be trained on a corpus of text-only data that includes long tail content that is absent in acoustic data and can bias speech recognition systems toward correctly decoding the long tail content.

In order to accurately model the vast amounts of long tail content, language models need to increase the size of an embedding vocabulary The embedding vocabulary represents an embedding identification associated with each token of a token vocabulary (i.e., words, wordpieces, n-grams, etc.) of the language model. In most instances, increasing the embedding vocabulary includes increasing the token vocabulary which requires large amounts of memory and computational resources that burden the speech recognition systems. Further, downstream tasks of the speech recognition system are also effected by an increased token vocabulary.

Implementations herein are directed towards a recurrent language model that increases the embedding vocabulary while keeping the size of the token vocabulary constant. That is, the embedding vocabulary increases independently of the token vocabulary allowing for more accurate modeling without burdening computational resources of the speech recognition system. In particular, the recurrent language model includes a first embedding table that generates a token embedding for a current token (e.g., word, wordpiece, n-gram, etc.) of a token sequence and a second embedding table that generates a sequence embedding for the previous tokens of the token sequence (e.g., n-gram sequence). Here, the token embedding provides the speech recognition system a likelihood of transcribing a particular word and the sequence embedding provides a likelihood of transcribing a particular word based on the particular sequence of previously transcribed words. Thus, the second embedding table increases the embedding vocabulary of the language model while the token vocabulary remains constant.

Scaling up the embedding vocabulary of the language model with the second embedding table requires no additional operations (e.g., computational resources) of the speech recognition system because the size of the embedding vocabulary has no effect on the number of embedding lookups or operations at each output step. Therefore, the only practical constraint for the size of the embedding vocabulary is the memory capacity. In some examples, the embedding tables are accessed sparsely and are not required to be stored on graphics processing units (GPU) and/or tensor processing units (TPU). Rather, the embedding tables may be stored on computer processing units (CPU) memory, disc, or other storage that includes capacities far greater than GPU and/or TPU memory.

Referring now to <FIG>. in some implementations, an example speech recognition system <NUM> includes a user device <NUM> associated with a respective user <NUM>. The user device <NUM> may be in communication with a remote system <NUM> via a network <NUM>. The user device <NUM> may correspond to a computing device, such as a mobile phone, computer, wearable device, smart appliance, audio infotainment system, smart speaker, etc., and is equipped with data processing hardware <NUM> and memory hardware <NUM>. The remote system <NUM> may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g., memory hardware).

The user device <NUM> may receive streaming audio <NUM> captured by the one or more microphones <NUM> of the user device <NUM> that corresponds to an utterance <NUM> spoken by the user <NUM> and extract acoustic features from the streaming audio <NUM> to generate audio data <NUM> corresponding to the utterance <NUM>. The acoustic features may include Mel-frequency cepstral coefficients (MFCCs) or filter bank energies computed over windows of the audio data <NUM> corresponding to the utterance <NUM>. The user device <NUM> communicates the audio data <NUM> corresponding to the utterance <NUM> to a speech recognizer <NUM> (also referred to herein as automated speech recognizer (ASR) model <NUM>). The ASR model <NUM> may reside and execute on the user device <NUM>. In other implementations, the ASR model <NUM> resides and executes on the remote system <NUM>.

With reference to <FIG>, the ASR model <NUM> may provide end-to-end (E2E) speech recognition by integrating acoustic, pronunciation, and language models into a single neural network, and does not require a lexicon or a separate text normalization component. Various structures and optimization mechanisms can provide increased accuracy and reduced model training time. The ASR model <NUM> may include a Conformer-Transducer model architecture, which adheres to latency constraints associated with interactive applications. The ASR model <NUM> provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the ASR model architecture suitable for performing speech recognition entirely on the user device <NUM> (e.g., no communication with a remote system <NUM> is required). The ASR model <NUM> includes an audio encoder <NUM>, a label encoder <NUM>, and a joint network <NUM>. The audio encoder <NUM>, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a neural network having a plurality of conformer layers. For instance, the audio encoder <NUM> reads a sequence of d-dimensional feature vectors (e.g., acoustic frames in streaming audio <NUM> (<FIG>)) x = (x<NUM>, x<NUM>, · · · , xT), where <MAT>, and produces at each time step a higher-order feature representation. This higher-order feature representation is denoted as ah<NUM>,. Optionally, the audio encoder <NUM> may include transformer layers in lieu of conformer layers. Similarly, the label encoder <NUM> may also include a neural network of transformer layers or a look-up table embedding model, which, like a language model (LM), processes the sequence of non-blank symbols output by a final Softmax layer <NUM> so far, y<NUM>,. , yui-<NUM>, into a dense representation Ihu that encodes predicted label history.

Finally, the representations produced by the audio and label encoders <NUM>, <NUM> are combined by the joint network <NUM> using a dense layer Ju,t. The joint network <NUM> then predicts P(zu,t |x, t, y<NUM>,. , yu-<NUM>), which is a distribution over the next output symbol. Stated differently, the joint network <NUM> generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses. Here, the "possible speech recognition hypotheses" correspond to a set of output labels (also referred to as "speech units") each representing a grapheme (e.g., symbol/character) or a wordpiece or word in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (<NUM>) symbols, e.g., one label for each of the <NUM>-letters in the English alphabet and one label designating a space. Accordingly, the joint network <NUM> may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution of the joint network <NUM> can include a posterior probability value for each of the different output labels. Thus, if there are <NUM> different output labels representing different graphemes or other symbols, the output zu,t of the joint network <NUM> can include <NUM> different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer <NUM>) for determining the transcription.

The Softmax layer <NUM> may employ any technique to select the output label/symbol with the highest probability in the distribution as the next output symbol predicted by the ASR model <NUM> at the corresponding output step. In this manner, the ASR model <NUM> does not make a conditional independence assumption, rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far.

Referring back to <FIG>, the ASR model <NUM> is configured to process the audio data <NUM> to determine a candidate transcription <NUM> for the spoken utterance <NUM>. Here, the candidate transcription includes a sequence of tokens <NUM>, 133a-n with each token <NUM> representing a portion of the candidate transcription <NUM> for the utterance <NUM>. That is, each token <NUM> in the sequence of tokens <NUM> may represent a potential word, wordpiece, n-gram, phoneme, and/or grapheme in the candidate transcription <NUM>. For example, the ASR model <NUM> generates a candidate transcription <NUM> of "driving directions to bourbon" for the spoken utterance <NUM> of "driving directions to Beaubien. " In this example, each token <NUM> in the sequence of tokens <NUM> for the candidate transcription <NUM> may represent a single word. Accordingly, the sequence of tokens <NUM> includes four (<NUM>) tokens <NUM> each representing a single word in the candidate transcription <NUM> (e.g., "driving directions to bourbon"). Notably, the fourth token <NUM> representing the term "bourbon" is misrecognized by the ASR model <NUM> for the correct term "Beaubien".

Moreover, the ASR model <NUM> may generate each token <NUM> in the sequence of tokens <NUM> as a corresponding probability distribution over possible candidate tokens. For example, for the fourth token <NUM> in the sequence of tokens <NUM>, the ASR model <NUM> may generate the tokens "bourbon" and "Beaubien" as corresponding possible tokens each having a corresponding probability or likelihood indicating a confidence that ASR model <NUM> recognized the possible token for the respective fourth token <NUM>. Here, the first and second candidate token are phonetically similar, however, the ASR model may favor the first candidate token associated with the term "bourbon" over the second candidate token associated with the term "Beaubien" since "Beaubien" is a proper noun (e.g., rare word or long tail content) likely not included in the acoustic training data of the ASR model <NUM>. That is, because "Beaubien" was not included in the acoustic training data, or only included in a few instances of the acoustic training data, the ASR model <NUM> may not correctly recognize this particular term in the utterance <NUM>. Stated differently, the ASR model <NUM> may be configured to output a higher probability/likelihood score for the first candidate token (e.g., bourbon) than for the second candidate token (e.g., Beaubien). Thus, the ASR model outputs fourth token <NUM> in the sequence of tokens <NUM> as "bourbon" because of the higher probability/likelihood score. While the example describes only two candidate tokens in the probability distribution over possible candidate tokens for simplicity, the number of possible candidate tokens in the probability distribution can be any number greater than two. In some examples, each token <NUM> in the sequence of tokens <NUM> is represented by an n-best list of possible candidate tokens <NUM> associated with a ranked list of the n possible candidate tokens <NUM> having the highest probability/likelihood scores Each candidate token <NUM> may be referred to as a speech recognition hypothesis. In additional examples, each token <NUM> in the sequence of tokens <NUM> is represented by the possible candidate token <NUM> having the highest probability/likelihood score in the probability distribution over possible candidate tokens <NUM>.

The ASR model <NUM> communicates the candidate transcription <NUM> that includes the sequence of tokens <NUM> to a language model (i.e., recurrent language model) <NUM>. The language model <NUM> may reside on the memory hardware <NUM> of the user device <NUM>, or optionally, the storage resources <NUM> of the remote system <NUM> or some combination thereof. The language model <NUM> is configured to determine the likelihood of outputting each token <NUM> in the sequence of candidate tokens <NUM>. That is, the language model <NUM> rescores the candidate transcription <NUM> output by the ASR model <NUM> by determining which candidate token <NUM> among possible candidate tokens <NUM> for each token <NUM> in the sequence of tokens <NUM> is most likely to correctly represent the corresponding token <NUM> in the candidate transcription <NUM> of the spoken utterance <NUM>. The language model <NUM> may assist in biasing speech recognition hypotheses output by the ASR model <NUM> toward rare words, such as proper nouns or long tail content, which were rarely included, or not included, in training audio data used to train the ASR model <NUM>. In the example above, the language model <NUM> may bias speech recognition by boosting the probability/likelihood score for the candidate token <NUM> associated with the word "Beaubien" in the probability distribution over possible candidate tokens <NUM> for the fourth token <NUM> in the sequence of tokens <NUM>. Here, the boosted probability/likelihood score for the candidate token <NUM> associated with the word "Beaubien" may now be higher than the probability/likelihood score for the candidate token "bourbon", thereby resulting in a rescored transcription <NUM> to now correctly recognize the word "Beaubien" from the spoken utterance <NUM>.

The language model <NUM> may include a recurrent neural network (RNN) language model. More specifically, the language model <NUM> may include a lookup-table language model configured to scale up the size of the RNN language model by increasing a number of rows in an embedding table, but with only a minimal increase in floating point operations. That is, the lookup-table allows embeddings to be sparsely stored on the memory hardware (e.g., CPU memory or disc) and retrieved from the table via a look up such that the size of the table adds no additional operations to each forward pass alleviating the need for storage on limited/constrained GPU/TPU memory.

The language model <NUM> includes a first embedding table <NUM>, a second embedding table <NUM>, a concatenator <NUM>, and a recurrent neural network (RNN) <NUM>. In some examples, the language model <NUM> is trained on text-only data that includes rarely spoken words (i.e., long tail content) to bias candidate tokens of the ASR model <NUM> to correctly transcribe long tail content.

The first embedding table <NUM> is configured to generate a token embedding <NUM> for a current token <NUM>, ti in the sequence of tokens <NUM>. Here, ti denotes the current token <NUM> in the sequence of tokens <NUM>. The first embedding table <NUM> determines a respective token embedding <NUM> for each token <NUM> independent from the rest of the tokens <NUM> in the sequence of tokens <NUM>. On the other hand, the second embedding table <NUM> includes an n-gram embedding table that receives a previous n-gram token sequence <NUM>, t<NUM>,. , tn-<NUM> at the current output step (e.g. time step) and generates a respective n-gram token embedding <NUM>. The previous n-gram token sequence (e.g., t<NUM>,. , tn-<NUM>) provides context information about the previously generated tokens <NUM> at each time step. The previous n-gram token sequence t<NUM>,. , tn-<NUM> grows exponentially with n at each time step. Thus, the n-gram token embedding <NUM> at each output steps assists with short-range dependencies such as spelling out rare words, and thereby improves the modeling of long tail tokens (e.g., words) in subword models.

At each time step, the concatenator <NUM> concatenates the token embedding <NUM> and the n-gram token embedding <NUM> into a concatenated output <NUM>. The RNN <NUM> receives the concatenated output <NUM> and ultimately rescores the candidate transcription <NUM> output by the ASR model <NUM> using the concatenated output <NUM> to generate the rescored transcription (e.g., final transcription) <NUM>. Moreover, the language model <NUM> may provide the rescored transcription <NUM> to the user device <NUM> via the network <NUM>. The user device <NUM> may visually display the rescored transcription <NUM> on a display of the user device <NUM> or audibly present the rescored transcription <NUM> by one or more speakers of the user device <NUM>. In other examples, the rescored transcription <NUM> may be a query instructing the user device <NUM> to perform an action.

At a single output step in the example shown, the second embedding table <NUM> may generate the n-gram token embedding <NUM> to represent the entire sequence of three previous n-gram tokens (e.g., "driving directions to") from the current token <NUM> (e.g., "bourbon) represented by the respective token embedding <NUM> looked-up by the first embedding table <NUM>. Thereafter, the concatenator <NUM> may concatenate the n-gram token embedding <NUM> and the respective token embedding <NUM> to generate the concatenated output <NUM> (e.g., "driving directions to bourbon"). The RNN <NUM> rescores the candidate transcription <NUM> by processing the concatenated output <NUM>. For example, the RNN <NUM> may rescore the candidate transcription <NUM> (e.g., driving directions to bourbon) such that the RNN <NUM> boosts the likelihood/probability score of "Bemibien" to now have a higher likelihood/probability score for the fourth token <NUM> than "bourbon. " The RNN <NUM> boosts the likelihood/probability score of "Beaubien" based, in part, on the determination that "bourbon" is not likely the correct token <NUM> with the current context information (e.g., previous n-gram token sequence t<NUM>,. , tn-<NUM>). Accordingly, the RNN <NUM> generates the rescored transcription <NUM> (e.g., driving directions to Beaubien). Notably, the fourth token <NUM> is now correctly recognized in the rescored transcription <NUM> even though the fourth token <NUM> was misrecognized in the candidate transcription <NUM>.

<FIG> illustrates a schematic view of the recurrent language model <NUM> of <FIG>. The first embedding table <NUM> includes a plurality of token embeddings <NUM>, 312a-n and the second embedding table <NUM> includes a plurality of n-gram token embeddings <NUM>, 322a-n. The first embedding table <NUM> is represented by a U × E matrix where E represents embedding dimension (i. e, embedding length) and U represents the number of token embeddings <NUM> in the first embedding table <NUM>. Generally, the number of embeddings is equal to the number V of unique tokens (e.g., number of word, wordpieces, n-grams, etc.). That is, the first embedding table <NUM> stores a respective token embedding <NUM> for each possible token <NUM> (e.g., word, wordpiece, etc.). For instance, when the tokens <NUM> include wordpieces, the recurrent language model <NUM> may include a <NUM>,<NUM>-wordpiece model having two LSTM layers with <NUM>-width and the dimensionality E of the token embedding <NUM> may be equal to <NUM>. The n-gram token embeddings <NUM> may each include a dimensionality of <NUM>,<NUM> when n is set to four (<NUM>). The second embedding table (e g. , n-gram embedding table) <NUM> may assign each previous n-gram token sequence (e.g., t<NUM>,. , tn-<NUM>) an embedding n-gram token embedding (e.g., embedding identifier) <NUM> via a modular hash as follows. <MAT> Notably, modular hashing necessitates collisions such that arbitrarily different n-gram embeddings will be hashed to the same n-gram token embedding <NUM>. However, collisions reduce and performance improves by increasing the number of unique tokens V.

In the example shown, the first and second embedding tables <NUM>, <NUM> receive a sequence of tokens <NUM> for a candidate transcription <NUM> corresponding to a spoken utterance <NUM> "Hello Pam. " Notably, the candidate transcription <NUM> misrecognizes the correct term "Pam" with the term "pan. " Here, the first embedding table <NUM> receives a current token <NUM> in the candidate transcription <NUM> corresponding to a third token 133c in the sequence of tokens <NUM> (e.g., t<NUM>). The first embedding table <NUM> determines a token embedding <NUM> from the plurality of token embeddings <NUM> for the third token 133c independent from the rest of the sequence of tokens <NUM>. In this example, the first embedding table <NUM> determines "pan" as the third token 133c (denoted by the black box) because "pan" may have a <NUM> likelihood/probability score while "Pam" has a <NUM> likelihood/probability score.

The second embedding table <NUM> receives the previous n-gram token sequence <NUM> in the candidate transcription <NUM> Here, the previous n-gram token sequence <NUM> includes " Hello" (e.g., t<NUM>, t<NUM>) where  denotes a token <NUM> for the start of a sentence. The second embedding table <NUM> determines a n-gram token embedding <NUM> of " Hello" (denoted by the black box) based on the previous n-gram token sequence <NUM>.

The concatenator <NUM> concatenates the token embedding <NUM>, output from the first embedding table <NUM> for the current token "pan", and the n-gram token embedding <NUM> for " Hello", and provides the concatenated output <NUM> (e.g., " Hello pan") to an RNN cell of the RNN <NUM>. Notably, the concatenated output <NUM> causes the RNN <NUM> to increase dense parameters in order to process the concatenated output <NUM>, however the scaling is linear rather than quadratic since the RNN output dimensionality remains fixed. The contextual information from the n-gram token embedding <NUM> is not only useful as an input to the RNN <NUM>, but is also useful at intermediate layers such that intermediate layers/cells receive context information via hidden state specific to that layer. Accordingly, the RNN <NUM> may inject the concatenated output <NUM> to the input activations of every layer in the RNN, drawing each from an embedding table specific to that layer.

Continuing with the above example, the RNN <NUM> rescores the candidate transcription <NUM> by processing the concatenated output <NUM>. That is, the RNN <NUM> may boost the probability of the term "Pam" based on the context information (e.g., previous n-gram token sequence <NUM>) included in the concatenated output <NUM>. Accordingly, the RNN <NUM> may adjust the likelihood/probability score of "pan" to <NUM> and the likelihood/probability score of "Pam" to <NUM>. As such, the RNN generates the rescored transcription to include "Pam" as the third token 133c and correctly recognize the spoken utterance <NUM>.

The n-gram embedding table <NUM> may effectively scale up to nearly a billion parameters. The recurrent language model <NUM> may be trained on a <NUM>-billion text-only sentence corpus. The language model <NUM> may be further preprocessed with misspelling removal against a whitelist of one million common words, and with log(n) scaling of sentence frequencies to ensure representation on the tail.

The recurrent language model <NUM> may integrate with the end-to-end ASR model <NUM> by first obtaining an effective likelihood by separating out log-posterior from an internal language model score of the ASR model <NUM> via Hybrid Autoregressive Transducer (HAT) factorization as follows. <MAT> Thereafter, the language model log-posterior score is added as follows. <MAT> The RNN-T decoder (e.g., prediction and joint networks <NUM>, <NUM> (<FIG>)) may be HAT factorized during training.

<FIG> is a flowchart of an exemplary arrangement of operation for a method <NUM> of performing speech recognition using a lookup-table recurrent language model. At operation <NUM>, the method <NUM> includes receiving audio data <NUM> that corresponds to an utterance <NUM> spoken by a user <NUM> and captured by user device <NUM>. At operation <NUM>, the method <NUM> includes processing the audio data <NUM> to determine a candidate transcription <NUM> for the utterance <NUM> using a speech recognizer <NUM>. Here, the candidate transcription <NUM> includes a sequence of tokens <NUM>, 133a-n. For each token <NUM> in the sequence of tokens <NUM>, the method <NUM> performs operations <NUM>-<NUM>. At operation <NUM>, the method <NUM> includes determining a token embedding <NUM> for the corresponding token <NUM> using a first embedding table <NUM>. At operation <NUM>, the method <NUM> includes determining a n-gram token embedding <NUM> for a previous n-gram token sequence <NUM>, t<NUM>,. , tn-<NUM> using a second embedding table <NUM>. At operation <NUM>, the method <NUM> includes concatenating the token embedding <NUM> and the n-gram token embedding <NUM> to generate a concatenated output <NUM> for the corresponding token <NUM>. At operation <NUM>, the method <NUM> includes rescoring, using an external language model (e.g., RNN) <NUM>, the candidate transcription <NUM> for the spoken utterance <NUM> by processing the concatenated output <NUM> generated for each corresponding token <NUM> in the sequence of tokens <NUM>.

The memory <NUM> stores information non-transitorily within the computing device <NUM> The memory <NUM> may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e. g , typically used for firmware, such as boot programs) Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto optical disks; and CD ROM and DVD-ROM disks.

To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e. g , a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e. g, a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input.

Claim 1:
A computer-implemented method (<NUM>) when executed on data processing hardware (<NUM>) causes the data processing hardware (<NUM>) to perform operations, the operations comprising:
receiving audio data (<NUM>) corresponding to an utterance (<NUM>) spoken by a user (<NUM>) and captured by a user device (<NUM>);
processing, using a speech recognizer (<NUM>), the audio data (<NUM>) to determine a candidate transcription (<NUM>) for the spoken utterance (<NUM>), the candidate transcription (<NUM>) comprises a sequence of tokens (<NUM>);
for each token (<NUM>) in the sequence of tokens (<NUM>):
determining, using a first embedding table (<NUM>), a token embedding (<NUM>) for the corresponding token (<NUM>);
determining, using a second embedding table (<NUM>), a n-gram token embedding (<NUM>) for a previous sequence of n-gram tokens (<NUM>); and
concatenating the token embedding (<NUM>) and the n-gram token embedding (<NUM>) to generate a concatenated output (<NUM>) for the corresponding token (<NUM>); and
rescoring, using an external language model (<NUM>), the candidate transcription (<NUM>) for the spoken utterance (<NUM>) by processing the concatenated output (<NUM>) generated for each corresponding token (<NUM>) in the sequence of tokens (<NUM>).