Patent Description:
Automatic speech recognition (ASR) systems provide a technology that is typically used in mobile devices and other devices. In general, ASR systems attempt to provide accurate transcriptions of what a user speaks to the mobile device. More specifically, ASR systems generate multiple candidate transcriptions and output the candidate transcription that is most likely to match the speech input. In some instances, ASR systems output inaccurate transcriptions that do not match what the user actually spoke. In these instances, the ASR system may rescore the multiple candidate transcriptions and output an accurate transcription that matches the speech input. One challenge of rescoring, however, is that ASR systems rely on language information of the speech input to accurately rescore the multiple candidate transcriptions. As such, it is often cumbersome for ASR systems to perform rescoring in multilingual speech environments. <NPL>, discloses a neural search algorithm that selects the most likely hypothesis using a sequence of acoustic representations and multiple hypotheses as input.

According to the invention, there are provided a computer-implemented method as set forth in claim <NUM>, a system as set forth in claim <NUM> and a computer-readable medium as set forth in claim <NUM>. One aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations of using multilingual re-scoring models for automatic speech recognition. The operations include receiving a sequence of acoustic frames extracted from audio data that corresponds to an utterance. During a first pass, the operations include processing the sequence of acoustic frames using a multilingual speech recognition model to generate N candidate hypotheses for the utterance. During a second pass, for each candidate hypothesis of the N candidate hypotheses, the method includes: generating, using a neural oracle search (NOS) model, a respective un-normalized likelihood score based on the sequence of acoustic frames and the corresponding candidate hypothesis; generating a respective external language model score using a language model; generating a standalone score that models prior statistics of the corresponding candidate hypothesis generated during the first pass; and generating a respective overall score for the candidate hypotheses based on the un-normalized score, the external language model score, and the standalone score. The operations also include selecting the candidate hypothesis having the highest respective overall score from among the N candidate hypotheses as a final transcription of the utterance.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, each candidate hypothesis of the N candidate hypotheses includes a respective sequence of word or sub-word labels. Here, each word or sub-word label is represented by a respective embedding vector. The external language model may be trained on text-only data. In some examples, the NOS model includes a language-specific NOS model. In these examples, the operations further include receiving a language identifier that indicates a language of the utterance and selecting the language-specific NOS model from among a plurality of language-specific NOS models that are each trained on a different respective language.

Optionally, the NOS model may include a multilingual NOS model. In some implementations the external language model includes a language-specific external language model. In these implementations, the operations further include receiving a language identifier that indicates a language of the utterance and selecting the language-specific external language model from among a plurality of language-specific external language models that are each trained on a different respective language. The NOS model may include two unidirectional long short-term memory (LSTM) layers. In some examples, the speech recognition model includes an encoder-decoder architecture that includes a conformer encoder having a plurality of conformer layers and a LSTM decoder having two LSTM layers.

Another aspect of the disclosure provides a system that includes data processing hardware and memory hardware storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations. The operations include receiving a sequence of acoustic frames extracted from audio data that corresponds to an utterance. During a first pass, the operations include processing the sequence of acoustic frames using a multilingual speech recognition model to generate N candidate hypotheses for the utterance. During a second pass, for each candidate hypothesis of the N candidate hypotheses, the method includes: generating, using a neural oracle search (NOS) model, a respective un-normalized likelihood score based on the sequence of acoustic frames and the corresponding candidate hypothesis; generating a respective external language model score using a language model; generating a standalone score that models prior statistics of the corresponding candidate hypothesis generated during the first pass; and generating a respective overall score for the candidate hypotheses based on the un-normalized score, the external language model score, and the standalone score. The operations also include selecting the candidate hypothesis having the highest respective overall score from among the N candidate hypotheses as a final transcription of the utterance.

Automatic speech recognition (ASR) systems are becoming increasingly popular in user devices as the ASR systems continue to provide more accrue transcriptions of what users speak. Still, in some instances, ASR systems generate inaccurate transcriptions that misrecognize what the user actually spoke. In some configurations, ASR systems generate N best candidate hypotheses for a spoken utterance and output the best candidate hypothesis as the final transcription. The N best candidate hypotheses configuration, however, has almost a <NUM>% lower word error rate (WER) compared to a one-best hypothesis configuration. Thus, in some implementations, the ASR systems will rescore the N best candidate hypotheses by integrating additional information to increase the WER. These rescoring implementations rely on language information (i.e., language identifier spoken by a user) in multilingual speech environments and only provide marginal WER improvements. The challenges discussed above identify a WER performance gap between ASR systems using N best candidate hypotheses configuration in comparison to the one-best candidate configuration.

Accordingly, implementations herein are directed towards methods and systems of executing a rescoring process that generates N candidate hypotheses for a corresponding utterance and selecting the most likely candidate hypothesis to output as a final transcription. In particular, during a first pass, the rescoring process generates N candidate hypotheses using a multilingual speech recognition model. Thereafter, during a second pass and for each candidate hypothesis, the rescoring process generate a respective un-normalized likelihood score using a neural oracle search (NOS) model, generates an external language model score, and generates a standalone score that models prior statistics of the candidate hypothesis. As will become apparent below, the NOS model may be a language-specific NOS model or a multilingual NOS model. Moreover, during the second pass, the rescoring process generates an overall score for each candidate hypothesis based on the un-normalized likelihood score, the external language model score, and the standalone score. The rescoring process selects the candidate hypothesis with the highest overall score as the final transcription for the utterance.

<FIG> is an example of a speech environment <NUM>. In the speech environment <NUM>, a user's <NUM> manner of interacting with a computing device, such as a user device <NUM>, may be through voice input. The user device <NUM> is configured to capture sounds (e.g., streaming audio data) from one or more users <NUM> within the speech environment <NUM>. Here, the streaming audio data may refer to a spoken utterance <NUM> by the user <NUM> that functions as an audible query, a command for the user device <NUM>, or an audible communication captured by the user device <NUM>. Speech-enabled systems of the user device <NUM> may field the query or the command by answering the query and/or causing the command to be performed/fulfilled by one or more downstream applications.

The user device <NUM> may correspond to any computing device associated with a user <NUM> and capable of receiving audio data. Some examples of user devices <NUM> include, but are not limited to, mobile device (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and stores instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations. The user device <NUM> further includes an audio system <NUM> with an audio capture device (e.g., microphone) <NUM>, 16a for capturing and converting spoken utterances <NUM> within the speech environment <NUM> into electrical signals and a speech output device (e.g., a speaker) <NUM>, 16b for communicating an audible audio signal (e.g., as output audio data from the user device <NUM>). While the user device <NUM> implements a single audio capture device 16a in the example shown, the user device <NUM> may implement an array of audio capture devices 16a without departing from the scope of the present disclosure, whereby one or more capture devices 16a in the array may not physically reside on the user device <NUM>, but be in communication with the audio system <NUM>.

In the speech environment <NUM>, an automated speech recognition (ASR) system <NUM> implementing a speech recognition model (i.e., ASR model) <NUM> resides on the user device <NUM> of the user <NUM> and/or on a remote computing device <NUM> (e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device <NUM> via a network <NUM>. The ASR system <NUM> may also implement one or more external language models <NUM> and a neural oracle search (NOS) model <NUM>. The user device <NUM> and/or the remote computing device (i.e., remote server) <NUM> also includes an audio subsystem <NUM> configured to receive the utterance <NUM> spoken by the user <NUM> and captured by the audio capture device 16a, and convert the utterances <NUM> into a corresponding digital format associated with input acoustic frames <NUM> capable of being processed by the ASR system <NUM>. In the example shown, the user speaks a respective utterance <NUM> and the audio subsystem <NUM> converts the utterance <NUM> into corresponding audio data (e.g., acoustic frames) <NUM> for input to the ASR system <NUM>. Thereafter, the speech recognition model <NUM> receives, as input, the audio data <NUM> corresponding to the utterance <NUM>, and generates/predicts, as output, a corresponding transcription <NUM> (e.g., speech recognition result/hypothesis) of the utterance <NUM>. As described in greater detail below, the speech recognition model <NUM> may include an end-to-end speech recognition model <NUM> trained with variable look ahead audio context to allow the model <NUM> to set, during inference, different durations of look ahead audio context when performing speech recognition depending on how sensitive a query specified by the utterance <NUM> is to latency and/or how much tolerance the user <NUM> has for latency. For instance, a digital assistant application <NUM> executing on the user device <NUM> may require the speech recognition depending on how sensitive a query specified by the utterance <NUM> is to latency and/or how much tolerance the user <NUM> has for latency.

In some implementations, the speech recognition model <NUM> performs streaming speech recognition on the audio data <NUM> during a first pass to generate N candidate hypotheses <NUM> (<FIG>), and the NOS and language models <NUM>, <NUM> rescore the N candidate hypotheses <NUM> during a second pass to generate a final transcription <NUM>. For instance, in the example shown, the speech recognition model <NUM> performs streaming speech recognition on the audio data <NUM> to produce partial speech recognition results (i.e., partial transcription) <NUM>, 120a (based on the N candidate hypotheses <NUM>), and the language and NOS models <NUM>, <NUM> rescore the N candidate hypotheses <NUM> to produce a final speech recognition result (i.e., final transcription) <NUM>, 120b. Notably, the speech recognition model <NUM> may use a first look ahead audio context that may be set to zero (or about <NUM> milliseconds) to produce the partial speech recognition results 120a Thus, the final speech recognition result 120b for the input utterance <NUM> may be delayed from the partial speech recognition results 120a for the input utterance.

The user device <NUM> and/or the remote computing device <NUM> also executes a user interface generator <NUM> configured to present a representation of the transcription <NUM> of the utterance <NUM> to the user <NUM> of the user device <NUM>. As described in greater detail below, the user device generator <NUM> may display the partial speech recognition results 120a in a streaming fashion during time <NUM> and subsequently display the final speech recognition result 120b during time <NUM>. In some configurations, the transcription <NUM> output from the ASR system <NUM> is processed, e.g., by a natural language understanding (NLU) module executing on the user device <NUM> or the remote computing device <NUM>, to execute a user command/query specified by the utterance <NUM>. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device <NUM> or the remote computing device <NUM>) may convert the transcription into synthesized speech for audible output by the user device <NUM> and/or another device.

In the example shown, the user <NUM> communicating with the digital assistant application <NUM> and the digital assistant application <NUM> displaying a digital assistant interface <NUM> on a screen of the user device <NUM> to depict a conversation between the user <NUM> and the digital assistant application <NUM>. In this example, the user <NUM> asks the digital assistant application <NUM>, "What time is the concert tonight?" This question from the user <NUM> is a spoken utterance <NUM> captured by the audio capture device 16a and processed by audio systems <NUM> of the user device <NUM>. In this example, the audio system <NUM> receives the spoken utterance <NUM> and converts it into acoustic frames <NUM> for input to the ASR system <NUM>.

Continuing with the example, the speech recognition model <NUM>, while receiving the acoustic frames (i.e., audio data) <NUM> corresponding to the utterance <NUM> as the user <NUM> speaks, encodes the acoustic frames <NUM> and then decodes the encoded acoustic frames <NUM> into the partial speech recognition results 120a. During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the partial speech recognition results 120a of the utterance <NUM> to the user <NUM> of the user device <NUM> in a streaming fashion that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken.

During the second pass, and after all of the acoustic frames <NUM> corresponding to the utterance <NUM> are received, the ASR system <NUM> rescores each candidate hypothesis <NUM> of the N candidate hypotheses <NUM> using the language and NOS models <NUM>, <NUM> and selects the candidate hypothesis <NUM> from among the N candidate hypotheses <NUM> that is the most likely the accurate transcription <NUM> of the utterance <NUM>. During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the final speech recognition results 120b of the utterance <NUM> to the user <NUM> of the user device <NUM>. In some implementations, the user interface generator <NUM> replaces the representation of the partial speech recognition results 120a with the representation of the final speech recognition result 120b. For instance, as the final speech recognition result 120b is presumed to be more accurate than the partial speech recognition results 120a produced without leveraging look ahead audio context, the final speech recognition result 120b ultimately displayed as the transcription <NUM> may fix any terms that may have been misrecognized in the partial speech recognition results 120a. In this example, the streaming partial speech recognition results 120a output by the speech recognition model <NUM> and displayed on the screen of the user device <NUM> at time <NUM> are associated with low latency and provide responsiveness to the user <NUM> that his/her query is being processed, while the final speech recognition result 120b displayed on the screen at time <NUM> improves the speech recognition quality in terms of accuracy, but at increased latency. However, since the partial speech recognition results 120a are displayed as the user speaks the utterance <NUM>, the higher latency associated with producing, and ultimately displaying the final recognition result is not noticeable to the user <NUM>.

In the example shown in <FIG>, the digital assistant application <NUM> may respond to the question posed by the user <NUM> using natural language processing. Natural language processing generally refers to a process of interpreting written language (e.g., the partial speech recognition results 120a and/or the final speech recognition result 120b) and determining whether the written language prompts any action. In this example, the digital assistant application <NUM> uses natural language processing to recognize that the question from the user <NUM> regards the user's schedule and more particularly a concert on the user's schedule. By recognizing these details with nature language processing, the automated assistant returns a response <NUM> to the user's query where the response <NUM> states, "Venue doors open at <NUM>:<NUM> PM and concert starts at 8PM. " In some configurations, natural language processing occurs on a remote server <NUM> in communication with the data processing hardware <NUM> of the user device <NUM>.

Referring to <FIG>, an example frame alignment-based transducer model 200a includes a Recurrent Neural Network-Transducer (RNN-T) model architecture which adheres to latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the frame alignment-based transducer model <NUM> may include other architectures such as transformer-transducer and conformer-transducer model architectures among others. The RNN-T model <NUM> provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device <NUM> (e.g., no communication with a remote server is required). The RNN-T model <NUM> incudes an encoder network <NUM>, a prediction network <NUM>, and a joint network <NUM>. The encoder network <NUM>, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, may include a recurrent network of stacked Long Short-Term (LSTM) layers. For instance, the encoder reads a sequence of d-dimensional feature vectors (e.g., acoustic frames <NUM> (<FIG>)) x = (X<NUM>, X<NUM>,. , XT), where <MAT>, and produces at each output step a higher-order feature representation. This higher-order feature representation is denoted as <MAT>.

Similarly, the prediction network <NUM> is also an LSTM network (i.e., LSTM decoder), which, like a language model (LM), processes the sequence of non-blank symbols (i.e., label history) <NUM> output by a final Softmax layer <NUM> so far, y<NUM>,. , yui-<NUM>, into a dense representation pui. Finally, with the RNN-T model architecture, the representations produced by the encoder and prediction/decoder networks <NUM>, <NUM> are combined by the joint network <NUM>. The prediction network <NUM> may be replaced by an embedding look-up table to improve latency by outputting looked-up sparse embeddings in lieu of processing dense representations. The joint network then predicts P(yi |Xti, y<NUM>,. , yui-<NUM>), which is a distribution over the next output symbol. Stated differently, the joint network <NUM> generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses. Here, the "possible speech recognition hypotheses" correspond to a set of output labels each representing a symbol/character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (<NUM>) symbols, e.g., one label for each of the <NUM>-letters in the English alphabet and one label designating a space. Accordingly, the joint network <NUM> may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution of the joint network <NUM> can include a posterior probability value for each of the different output labels. Thus, if there are <NUM> different output labels representing different graphemes or other symbols, the output yi of the joint network <NUM> can include <NUM> different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer <NUM>) for determining the transcription <NUM>.

The Softmax layer <NUM> may employ any technique to select the output label/symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model <NUM> at the corresponding output step. In this manner, the RNN-T model <NUM> does not make a conditional independence assumption, rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model <NUM> does assume an output symbol is independent of future acoustic frames <NUM>, which allows the RNN-T model to be employed in a streaming fashion.

In some examples, the encoder network (i.e., audio encoder) <NUM> of the RNN-T model <NUM> includes is an encoder-decoder architecture having a Conformer-based encoder that includes a stack of conformer layers. Here, each conformer layer includes a series of multi-headed self-attention, depth wise convolution, and feed-forward layers. In some example, the Conformer-based encoder may include a stack of <NUM> conformer layers. The encoder network <NUM> may include other types of encoders having multi-headed self-attention mechanisms. For instance, the encoder network <NUM> may be a Transformer-based encoder or a lightweight convolutional (LConv) based encoder. The encoder network <NUM> may also be RNN-based including a series of LSTM layers. The prediction network <NUM> may be a LSTM decoder having two <NUM>,<NUM>-dimensional LSTM layers, each of which is also followed by <NUM>-dimensional projection layer. Alternatively, the prediction network <NUM> may include a stack of transformer or conformer blocks, or an embedding look-up table in lieu of LSTM layers. Finally, the joint network <NUM> may also have <NUM> hidden units. The Softmax layer <NUM> may be composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes in a plurality of training data sets.

Referring now to <FIG> and 3B, in some implementations, the remote server <NUM> (<FIG>) executes an example rescoring process <NUM> for rescoring N candidate hypotheses <NUM> generated by the ASR model <NUM> during a first pass <NUM>. Alternatively, the user device <NUM> (<FIG>) may execute the example rescoring process <NUM> in addition to, or in lieu of, the remote server <NUM> (<FIG>). The rescoring process <NUM> includes the first pass <NUM> that generates N candidate hypotheses <NUM>, 204a-n (H<NUM>, H<NUM>,. , HN) for a sequence of acoustic frames <NUM> (X<NUM>, X<NUM>,. , XT) corresponding to an utterance <NUM>. Moreover, the rescoring process <NUM> includes a second pass <NUM> that rescores each candidate hypothesis <NUM> of the N candidate hypotheses <NUM> by integrating additional information sources, discussed in greater detail below. As such, the second pass <NUM> includes a sequence classification objective configured to select the candidate hypothesis <NUM> from among the N candidate hypotheses <NUM> that is the most likely accurate transcription of the utterance <NUM>.

In particular, the ASR model <NUM> receives the sequence of acoustic frames <NUM> extracted from audio data that corresponds to the utterance <NUM>. During the first pass <NUM>, the ASR model <NUM> processes the sequence of acoustic frames <NUM> to generate N candidate hypotheses <NUM> for the utterance <NUM>. Here, each candidate hypothesis <NUM> corresponds to a candidate transcription <NUM> for the utterance <NUM> and is represented by a respective sequence of word, sub-word, and/or grapheme labels that are represented by a respective embedding vector. Moreover, each candidate hypothesis <NUM> includes a standalone score <NUM> that models prior statistics of the corresponding candidate hypothesis <NUM>. That is, the standalone score <NUM> may indicate a confidence that the corresponding candidate hypothesis <NUM> is an accurate transcription for the utterance <NUM>. The confidence of the standalone score <NUM> may also indicate a frequency of previously realized utterances <NUM> (e.g., a number of times that the candidate hypothesis <NUM> was previously spoken).

The ASR model <NUM> may generate any number of candidate hypotheses <NUM> (e.g., N may be any integer value). In some examples, the ASR model <NUM> outputs a specified number of candidate hypotheses <NUM> based on a predefined parameter. For instance, the ASR model <NUM> outputs five (<NUM>) candidate hypotheses <NUM> (i.e., N = <NUM>) for every spoken utterance <NUM>. For instance, the N candidate hypotheses <NUM> may correspond to an N-best list of candidate hypotheses associated with the N candidate hypotheses having the highest standalone scores <NUM>. In other examples, the ASR model <NUM> outputs all candidate hypotheses <NUM> having a standalone score <NUM> that satisfies a threshold value.

In the example shown, the ASR model <NUM> processes the sequence of acoustic frames <NUM> that corresponds to the utterance <NUM> "play next song" spoken by the user <NUM> and generates three candidate hypotheses <NUM> (i.e., N = <NUM>). Namely, the candidate hypotheses <NUM> include "play next song" having a standalone score <NUM> of <NUM>, "hey next long" having a standalone score <NUM> of <NUM>, and "play next pong" having a standalone score <NUM> of <NUM>. Here, the rescoring process <NUM> may output the candidate hypothesis <NUM> "play next pong" as the partial transcription 120a (<FIG>) because it has the highest standalone score <NUM>. Alternatively, the rescoring process <NUM> may refrain from outputting the partial transcription until the rescoring process generates the final transcription. Notably, in this example, the candidate hypothesis <NUM> having the highest standalone score <NUM> is an inaccurate transcription of the utterance <NUM> spoken by the user <NUM>.

The ASR model <NUM> may be a multilingual ASR model configured to recognize utterances <NUM> spoken in multiple languages. That is, the single ASR model <NUM> may receive an utterance <NUM> in a first language and generate N candidate hypotheses <NUM> in the first language and receive another utterance <NUM> in a different second language and generate N candidate hypotheses <NUM> in the second language. Moreover, the single ASR model may receive an utterance <NUM> including code-mixed speech that includes terms in both the first and second languages. Thus, the rescoring process <NUM> may implement the single multilingual ASR model <NUM> in a multilingual speech environment.

In some implementations, the second pass <NUM> may receive the N candidate hypotheses <NUM> from the first pass <NUM>, and generate a corresponding overall score <NUM> by integrating additional information for each candidate hypothesis <NUM>. The overall score <NUM> may indicate a more accurate confidence level than the standalone score <NUM> from the first pass 301of whether each candidate hypothesis <NUM> is an accurate transcription. Thereafter, the second pass <NUM> may select the candidate hypothesis <NUM> having the highest overall score <NUM> as the transcription <NUM> (i.e., final transcription 120b (<FIG>)).

More specifically, during the second pass <NUM>, an external language model (LM) <NUM> receives the N candidate hypotheses <NUM> and generates a respective external language model score <NUM> for each candidate hypothesis <NUM>. In some implementations, the external LM <NUM> includes a RNN LM. Here, the external LM <NUM> may include a plurality of language-specific external LMs <NUM>, 310a-n each trained on text-only data (i.e., unpaired data) for a particular language. As such, external LM <NUM> and language-specific external LM <NUM> may be used interchangeably herein. Thus, each language-specific external LM <NUM> is configured to generate an external language model score (i.e., language model score) <NUM> for utterances <NUM> in a respective language. For example, a first language-specific external LM <NUM>, 310a trained on English text-only data generates language model scores <NUM> for utterances <NUM> spoken in English, and a second language-specific external LM <NUM>, 310b trained on Spanish text-only data generates language model scores <NUM> for utterances <NUM> spoken in Spanish. The plurality of external LMs <NUM> may by trained on any number of languages where each external LM <NUM> is trained with text-only data of a different respective language.

Accordingly, the external LM <NUM> may receive a language identifier <NUM> that indicates a language of the utterance <NUM> to select the language-specific external LM <NUM> from among the plurality of language-specific external LMs <NUM> that corresponds to the language of the utterance <NUM>. Put another way, the rescoring process <NUM> may select the language-specific external LM <NUM> based on the language identifier <NUM>. In some examples, the ASR model <NUM> determines the language identifier <NUM> based on processing the sequence of acoustic frames <NUM> of the utterance <NUM>. In other examples, the ASR model <NUM> obtains the language identifier <NUM> from an external source. For instance, a user may configure the ASR model for a particular language. In other instances, the ASR model <NUM> may determine an identity of the user <NUM> that spoke the utterance <NUM> and identify the language identifier <NUM> based on a language associated with the identified user <NUM>.

Accordingly, during the second pass <NUM>, the rescoring process <NUM> selects the external LM <NUM> that corresponds to the language of the utterance <NUM> based on the language identifier <NUM>, and generates the language model score <NUM> for each candidate hypothesis <NUM>. The language model score <NUM> indicates a likelihood that the sequence of hypothesized terms in the candidate hypothesis <NUM> are spoken by the user <NUM>. For example, the LM <NUM> will generate a higher language model score <NUM> for candidate hypothesis <NUM> "What is the weather today?" as opposed to the candidate hypothesis <NUM> "What is the weather hooray?" In particular, the LM <NUM> generates the higher language model score <NUM> for "What is the weather today?" because this sequence of hypothesized terms may have been included in the text-only training data more frequently than "What is the weather hooray?".

The example rescoring process <NUM> also includes the neural oracle search (NOS) model <NUM> that receives the N candidate hypotheses <NUM>, the sequence of acoustic frames <NUM>, and the label history <NUM> (e.g., previously output words, word-pieces, and/or graphemes). The label history <NUM> (y<NUM>:i-<NUM>) may be output by the ASR model <NUM>, the second pass <NUM> of the rescoring process <NUM> (e.g., via a rescorer <NUM>), or some combination thereof. In some examples, the label history <NUM> includes a transcription for a previous utterance <NUM> spoken by the user <NUM>. For instance, the user <NUM> may have previously spoke a previous utterance <NUM> of "do I have any meetings today?" that represents the label history <NUM> for a current utterance <NUM> of "what about tomorrow?" In other examples, the label history <NUM> includes all terms preceding a current label of the utterance. For instance, for the utterance <NUM> "play my playlist" the label history <NUM> may correspond to the terms "play my," where the current term (e.g., next hypothesized term) in the utterance <NUM> is "playlist. " Optionally, the NOS model <NUM> may receive the language identifier <NUM> indicating the language of the utterance <NUM> spoken by the user <NUM>.

<FIG> illustrates an example of the rescoring process <NUM>, 300a that includes a plurality of language-specific NOS models <NUM>, 320Sa-n. Here, each language-specific NOS model <NUM> is trained on pairwise data (i.e., transcribed audio training data) of a particular language. Accordingly, during the second pass <NUM>, the rescoring process <NUM> selects a language-specific NOS model <NUM> from among the plurality of language-specific NOS models <NUM> that corresponds to the language of the utterance <NUM> based on the language identifier <NUM>. As such, the example rescoring process 300a assumes that the language identifier <NUM> is available to select the correct language-specific NOS model <NUM>.

Alternatively, FIG. 3B illustrates an example rescoring process <NUM>, 300b that includes a multilingual NOS model <NUM>, <NUM>. In this example, the multilingual NOS model <NUM> is trained on pairwise data (i.e., transcribed audio training data) for any number of languages. Thus, the example rescoring process 300b may implement a single multilingual NOS model <NUM> in a multilingual speech environment. Notably, the example rescoring process 300b does not require the use of any language identifier <NUM> since selection of a language-specific NOS model <NUM> (as described with reference to <FIG>) associated with the language of the utterance <NUM> is not required. Thus, the utterance <NUM> may include a multilingual utterance that includes codemixing of speech across two or more languages. As used herein, the NOS model <NUM> may include either a language-specific NOS model <NUM> (<FIG>) that the rescoring process 300a selects based on the language identifier <NUM> or the multilingual NOS model (FIG.

With continued reference to <FIG> and 3B, the NOS model <NUM> includes a prior model that predicts the next label Yi given the label history <NUM>. That is, the prior model predicts a prior score for the next label based on previously recognized word, word-pieces, and/or graphemes. The prior model of the NOS model <NUM> may include a two-layer, <NUM> units per layer unidirectional LSTM. The prior model trains using labeled audio training data and a cross-entropy loss. Moreover, the NOS model <NUM> includes a posterior model that predicts a posterior score by combining the label history <NUM> with the sequence of acoustic frames <NUM> from the first pass <NUM> in a label-synchronous fashion. The posterior model of the NOS model <NUM> may include a two-layer, <NUM> units per layer unidirectional LSTM, with a two-layer, <NUM> units per layer label synchronous attention mechanism. The posterior model trains with labeled audio training data and a cross-entropy loss to predict the next label Yi given the label history <NUM> and the sequence of acoustic frames <NUM>. The NOS model <NUM> sums the token level prior score and the token level posterior score to generate the un-normalized likelihood score <NUM>. As such, the un-normalized likelihood score <NUM> is a sequence level score represented by the summation as follows: <MAT>.

In Equation <NUM>, Sθ<NUM> represents the un-normalized likelihood score <NUM>,.

The rescorer <NUM> receives the standalone score <NUM>, the language model score <NUM>, and the un-normalized likelihood score <NUM> for each candidate hypothesis <NUM> of the N candidate hypotheses <NUM> and generates the respective overall score <NUM>. In particular, the rescorer <NUM> generates the overall score <NUM> for each candidate hypothesis <NUM> based on any combination of the standalone score <NUM>, the language model score <NUM>, and the un-normalized likelihood score <NUM>. In some examples, the rescorer <NUM> sums the standalone score <NUM>, the language model score <NUM>, and the un-normalized likelihood score <NUM> linearly to determine a sequence-level overall score <NUM> represented by: <MAT> <MAT>.

In Equation <NUM>, Sθ<NUM>, represents the un-normalized likelihood score <NUM>, Sθ<NUM> represents the external language model score <NUM>, and Sθ<NUM> represents the standalone score <NUM>. To optimize model parameters of the rescorer <NUM> during training, the rescoring process <NUM> uses a cross-entropy objective between the posterior score and a sequence-level ground truth distribution. In some examples, the training process assigns the total ground truth distribution to the ground-truth transcription and assigns all other candidate hypotheses to zero. In other examples, the training process assigns the total ground truth distribution uniformly across all candidate hypotheses having a word error rate (WER) below the best candidate hypothesis (i.e., ground truth transcription). In yet other examples, the training process applies a Softmax function to a negative edit-distance between each candidate hypothesis and the ground truth transcription.

Thereafter, the rescorer <NUM> selects the candidate hypothesis <NUM> from among the N candidate hypotheses <NUM> having the highest overall score <NUM> as a final transcription <NUM> of the utterance <NUM>. In the example shown, the candidate hypotheses <NUM> include "play next song" having an overall score <NUM> of <NUM>, "hey next long" having an overall score <NUM> of <NUM>, and "play next pong" having an overall score <NUM> of <NUM>. Continuing with the example, the rescorer <NUM> selects the candidate hypothesis <NUM> of "play next song" (denoted by the solid line box) having the highest overall score <NUM> of <NUM> as the transcription <NUM> (e.g., final transcription 120b (<FIG>)). Notably, the candidate hypothesis <NUM> with the highest standalone score <NUM> (i.e., likelihood of being the correct transcription) is not the correct candidate hypothesis <NUM>, but the candidate hypothesis with the highest overall score <NUM> is the correct transcription from the second pass <NUM>.

<FIG> is a flowchart of an exemplary arrangement of operations for a computer-implemented method <NUM> of using multi-lingual re-scoring models for automatic speech recognition. At operation <NUM>, the method <NUM> includes receiving a sequence of acoustic frames <NUM> extracted from audio data that corresponds to an utterance <NUM>. At operation <NUM>, during a first pass <NUM>, the method <NUM> includes processing the sequence of acoustic frames <NUM> to generate N candidate hypotheses <NUM>, 204a-n for the utterance <NUM> using a multilingual speech recognition model (i.e., ASR model) <NUM>. During a second pass <NUM>, for each candidate hypothesis <NUM> of the N candidate hypotheses <NUM>, the method <NUM> performs operations <NUM>-<NUM>. At operation <NUM>, the method <NUM> includes generating a respective un-normalized likelihood score 325using a NOS model <NUM>. Here, the NOS model <NUM> generates the un-normalized likelihood score <NUM> based on the sequence of acoustic frames <NUM> and the corresponding candidate hypothesis <NUM>. At operation <NUM>, the method <NUM> includes generating a respective external language model score <NUM> using a language model <NUM>. At operation <NUM>, the method <NUM> includes generating a standalone score <NUM> that models prior statistics of the corresponding candidate hypothesis <NUM> generated during the first pass <NUM>. At operation <NUM>, the method <NUM> includes generating a respective overall score <NUM> for the candidate hypothesis <NUM> based on the un-normalized likelihood score <NUM>, the external language model score <NUM>, and the standalone score <NUM>. At operation <NUM>, the method <NUM> includes selecting the candidate hypothesis <NUM> having the highest respective overall score <NUM> from among the N candidate hypotheses <NUM> as a final transcription <NUM> of the utterance <NUM>.

Claim 1:
A computer-implemented method (<NUM>) when executed on data processing hardware (<NUM>) causes the data processing hardware (<NUM>) to perform operations comprising:
receiving a sequence of acoustic frames (<NUM>) extracted from audio data corresponding to an utterance (<NUM>);
during a first pass (<NUM>), processing, using a multilingual speech recognition model (<NUM>), the sequence of acoustic frames (<NUM>) to generate N candidate hypotheses (<NUM>) for the utterance (<NUM>);
during a second pass (<NUM>), for each candidate hypothesis (<NUM>) of the N candidate hypotheses (<NUM>):
generating, using a neural oracle search (NOS) model (<NUM>), a respective un-normalized likelihood score (<NUM>) based on the sequence of acoustic frames (<NUM>) and the corresponding candidate hypothesis (<NUM>);
generating, using an external language model (<NUM>), a respective external language model score (<NUM>);
generating a standalone score (<NUM>) that models prior statistics of the corresponding candidate hypothesis (<NUM>) generated during the first pass (<NUM>); and
generating a respective overall score (<NUM>) for the candidate hypothesis (<NUM>) based on the un-normalized likelihood score (<NUM>), the external language model score (<NUM>), and the standalone score (<NUM>); and
selecting the candidate hypothesis (<NUM>) having the highest respective overall score (<NUM>) from among the N candidate hypotheses (<NUM>) as a final transcription (<NUM>) of the utterance (<NUM>).