Patent ID: 12254875

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Automatic speech recognition (ASR) systems are becoming increasingly popular in user devices as the ASR systems continue to provide more accrue transcriptions of what users speak. Still, in some instances, ASR systems generate inaccurate transcriptions that misrecognize what the user actually spoke. In some configurations, ASR systems generate N best candidate hypotheses for a spoken utterance and output the best candidate hypothesis as the final transcription. The N best candidate hypotheses configuration, however, has almost a 50% lower word error rate (WER) compared to a one-best hypothesis configuration. Thus, in some implementations, the ASR systems will rescore the N best candidate hypotheses by integrating additional information to increase the WER. These rescoring implementations rely on language information (i.e., language identifier spoken by a user) in multilingual speech environments and only provide marginal WER improvements. The challenges discussed above identify a WER performance gap between ASR systems using N best candidate hypotheses configuration in comparison to the one-best candidate configuration.

Accordingly, implementations herein are directed towards methods and systems of executing a rescoring process that generates N candidate hypotheses for a corresponding utterance and selecting the most likely candidate hypothesis to output as a final transcription. In particular, during a first pass, the rescoring process generates N candidate hypotheses using a multilingual speech recognition model. Thereafter, during a second pass and for each candidate hypothesis, the rescoring process generate a respective un-normalized likelihood score using a neural oracle search (NOS) model, generates an external language model score, and generates a standalone score that models prior statistics of the candidate hypothesis. As will become apparent below, the NOS model may be a language-specific NOS model or a multilingual NOS model. Moreover, during the second pass, the rescoring process generates an overall score for each candidate hypothesis based on the un-normalized likelihood score, the external language model score, and the standalone score. The rescoring process selects the candidate hypothesis with the highest overall score as the final transcription for the utterance.

FIG.1is an example of a speech environment100. In the speech environment100, a user's104manner of interacting with a computing device, such as a user device10, may be through voice input. The user device10is configured to capture sounds (e.g., streaming audio data) from one or more users104within the speech environment100. Here, the streaming audio data may refer to a spoken utterance106by the user104that functions as an audible query, a command for the user device10, or an audible communication captured by the user device10. Speech-enabled systems of the user device10may field the query or the command by answering the query and/or causing the command to be performed/fulfilled by one or more downstream applications.

The user device10may correspond to any computing device associated with a user104and capable of receiving audio data. Some examples of user devices10include, but are not limited to, mobile device (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device10includes data processing hardware12and memory hardware14in communication with the data processing hardware12and stores instructions, that when executed by the data processing hardware12, cause the data processing hardware12to perform one or more operations. The user device10further includes an audio system16with an audio capture device (e.g., microphone)16,16afor capturing and converting spoken utterances106within the speech environment100into electrical signals and a speech output device (e.g., a speaker)16,16bfor communicating an audible audio signal (e.g., as output audio data from the user device10). While the user device10implements a single audio capture device16ain the example shown, the user device10may implement an array of audio capture devices16awithout departing from the scope of the present disclosure, whereby one or more capture devices16ain the array may not physically reside on the user device10, but be in communication with the audio system16.

In the speech environment100, an automated speech recognition (ASR) system118implementing a speech recognition model (i.e., ASR model)200resides on the user device10of the user104and/or on a remote computing device60(e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device10via a network40. The ASR system118may also implement one or more external language models310and a neural oracle search (NOS) model320. The user device10and/or the remote computing device (i.e., remote server)60also includes an audio subsystem108configured to receive the utterance106spoken by the user104and captured by the audio capture device16a, and convert the utterances106into a corresponding digital format associated with input acoustic frames110capable of being processed by the ASR system118. In the example shown, the user speaks a respective utterance106and the audio subsystem108converts the utterance106into corresponding audio data (e.g., acoustic frames)110for input to the ASR system118. Thereafter, the speech recognition model200receives, as input, the audio data110corresponding to the utterance106, and generates/predicts, as output, a corresponding transcription120(e.g., speech recognition result/hypothesis) of the utterance106. As described in greater detail below, the speech recognition model200may include an end-to-end speech recognition model200trained with variable look ahead audio context to allow the model200to set, during inference, different durations of look ahead audio context when performing speech recognition depending on how sensitive a query specified by the utterance106is to latency and/or how much tolerance the user106has for latency. For instance, a digital assistant application50executing on the user device10may require the speech recognition depending on how sensitive a query specified by the utterance106is to latency and/or how much tolerance the user106has for latency.

In some implementations, the speech recognition model200performs streaming speech recognition on the audio data110during a first pass to generate N candidate hypotheses204(FIG.3), and the NOS and language models310,320rescore the N candidate hypotheses204during a second pass to generate a final transcription120. For instance, in the example shown, the speech recognition model200performs streaming speech recognition on the audio data110to produce partial speech recognition results (i.e., partial transcription)120,120a(based on the N candidate hypotheses204), and the language and NOS models310,320rescore the N candidate hypotheses204to produce a final speech recognition result (i.e., final transcription)120,120b. Notably, the speech recognition model200may use a first look ahead audio context that may be set to zero (or about 240 milliseconds) to produce the partial speech recognition results120aThus, the final speech recognition result120bfor the input utterance106may be delayed from the partial speech recognition results120afor the input utterance.

The user device10and/or the remote computing device60also executes a user interface generator109configured to present a representation of the transcription120of the utterance106to the user104of the user device10. As described in greater detail below, the user device generator109may display the partial speech recognition results120ain a streaming fashion during time1and subsequently display the final speech recognition result120bduring time2. In some configurations, the transcription120output from the ASR system118is processed, e.g., by a natural language understanding (NLU) module executing on the user device10or the remote computing device60, to execute a user command/query specified by the utterance106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device10or the remote computing device60) may convert the transcription into synthesized speech for audible output by the user device10and/or another device.

In the example shown, the user104communicating with the digital assistant application50and the digital assistant application50displaying a digital assistant interface18on a screen of the user device10to depict a conversation between the user104and the digital assistant application50. In this example, the user104asks the digital assistant application50, “What time is the concert tonight?” This question from the user104is a spoken utterance106captured by the audio capture device16aand processed by audio systems16of the user device10. In this example, the audio system16receives the spoken utterance106and converts it into acoustic frames110for input to the ASR system118.

Continuing with the example, the speech recognition model200, while receiving the acoustic frames (i.e., audio data)110corresponding to the utterance106as the user104speaks, encodes the acoustic frames110and then decodes the encoded acoustic frames110into the partial speech recognition results120a. During time1, the user interface generator109presents, via the digital assistant interface18, a representation of the partial speech recognition results120aof the utterance106to the user104of the user device10in a streaming fashion that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken.

During the second pass, and after all of the acoustic frames110corresponding to the utterance106are received, the ASR system118rescores each candidate hypothesis204of the N candidate hypotheses204using the language and NOS models310,320and selects the candidate hypothesis204from among the N candidate hypotheses204that is the most likely the accurate transcription120of the utterance106. During time2, the user interface generator109presents, via the digital assistant interface18, a representation of the final speech recognition results120bof the utterance106to the user105of the user device10. In some implementations, the user interface generator109replaces the representation of the partial speech recognition results120awith the representation of the final speech recognition result120b. For instance, as the final speech recognition result120bis presumed to be more accurate than the partial speech recognition results120aproduced without leveraging look ahead audio context, the final speech recognition result120bultimately displayed as the transcription120may fix any terms that may have been misrecognized in the partial speech recognition results120a. In this example, the streaming partial speech recognition results120aoutput by the speech recognition model200and displayed on the screen of the user device10at time1are associated with low latency and provide responsiveness to the user104that his/her query is being processed, while the final speech recognition result120bdisplayed on the screen at time2improves the speech recognition quality in terms of accuracy, but at increased latency. However, since the partial speech recognition results120aare displayed as the user speaks the utterance106, the higher latency associated with producing, and ultimately displaying the final recognition result is not noticeable to the user104.

In the example shown inFIG.1, the digital assistant application50may respond to the question posed by the user104using natural language processing. Natural language processing generally refers to a process of interpreting written language (e.g., the partial speech recognition results120aand/or the final speech recognition result120b) and determining whether the written language prompts any action. In this example, the digital assistant application50uses natural language processing to recognize that the question from the user104regards the user's schedule and more particularly a concert on the user's schedule. By recognizing these details with nature language processing, the automated assistant returns a response 19 to the user's query where the response 19 states, “Venue doors open at 6:30 PM and concert starts at 8 PM.” In some configurations, natural language processing occurs on a remote server60in communication with the data processing hardware12of the user device10.

Referring toFIG.2, an example frame alignment-based transducer model200aincludes a Recurrent Neural Network-Transducer (RNN-T) model architecture which adheres to latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the frame alignment-based transducer model200may include other architectures such as transformer-transducer and conformer-transducer model architectures among others. The RNN-T model200provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device102(e.g., no communication with a remote server is required). The RNN-T model200incudes an encoder network210, a prediction network220, and a joint network230. The encoder network210, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, may include a recurrent network of stacked Long Short-Term (LSTM) layers. For instance, the encoder reads a sequence of d-dimensional feature vectors (e.g., acoustic frames110(FIG.1)) x=(X1, X2, . . . , XT), where Xt∈d, and produces at each output step a higher-order feature representation. This higher-order feature representation is denoted as h1enc, . . . , hTenc.

Similarly, the prediction network220is also an LSTM network (i.e., LSTM decoder), which, like a language model (LM), processes the sequence of non-blank symbols (i.e., label history)245output by a final Softmax layer240so far, y0, . . . , yu-1, into a dense representation pui. Finally, with the RNN-T model architecture, the representations produced by the encoder and prediction/decoder networks210,220are combined by the joint network230. The prediction network220may be replaced by an embedding look-up table to improve latency by outputting looked-up sparse embeddings in lieu of processing dense representations. The joint network then predicts P(yi|Xti, Y0, . . . , Yui-1), which is a distribution over the next output symbol. Stated differently, the joint network230generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses. Here, the “possible speech recognition hypotheses” correspond to a set of output labels each representing a symbol/character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, e.g., one label for each of the 26-letters in the English alphabet and one label designating a space. Accordingly, the joint network230may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution of the joint network230can include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output yiof the joint network230can include 100 different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer240) for determining the transcription120.

The Softmax layer240may employ any technique to select the output label/symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model200at the corresponding output step. In this manner, the RNN-T model200does not make a conditional independence assumption, rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model200does assume an output symbol is independent of future acoustic frames110, which allows the RNN-T model to be employed in a streaming fashion.

In some examples, the encoder network (i.e., audio encoder)210of the RNN-T model200includes is an encoder-decoder architecture having a Conformer-based encoder that includes a stack of conformer layers. Here, each conformer layer includes a series of multi-headed self-attention, depth wise convolution, and feed-forward layers. In some example, the Conformer-based encoder may include a stack of 17 conformer layers. The encoder network210may include other types of encoders having multi-headed self-attention mechanisms. For instance, the encoder network210may be a Transformer-based encoder or a lightweight convolutional (LConv) based encoder. The encoder network210may also be RNN-based including a series of LSTM layers. The prediction network220may be a LSTM decoder having two 2,048-dimensional LSTM layers, each of which is also followed by 640-dimensional projection layer. Alternatively, the prediction network220may include a stack of transformer or conformer blocks, or an embedding look-up table in lieu of LSTM layers. Finally, the joint network230may also have 640 hidden units. The Softmax layer240may be composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes in a plurality of training data sets.

Referring now toFIGS.3A and3B, in some implementations, the remote server60(FIG.1) executes an example rescoring process300for rescoring N candidate hypotheses204generated by the ASR model200during a first pass301. Alternatively, the user device10(FIG.1) may execute the example rescoring process300in addition to, or in lieu of, the remote server60(FIG.1). The rescoring process300includes the first pass301that generates N candidate hypotheses204,204a-n(H1, H2, . . . , HN) for a sequence of acoustic frames110(X1, X2, . . . , XT) corresponding to an utterance106. Moreover, the rescoring process300includes a second pass302that rescores each candidate hypothesis204of the N candidate hypotheses204by integrating additional information sources, discussed in greater detail below. As such, the second pass302includes a sequence classification objective configured to select the candidate hypothesis204from among the N candidate hypotheses204that is the most likely accurate transcription of the utterance106.

In particular, the ASR model200receives the sequence of acoustic frames110extracted from audio data that corresponds to the utterance106. During the first pass301, the ASR model200processes the sequence of acoustic frames110to generate N candidate hypotheses204for the utterance106. Here, each candidate hypothesis204corresponds to a candidate transcription120for the utterance106and is represented by a respective sequence of word, sub-word, and/or grapheme labels that are represented by a respective embedding vector. Moreover, each candidate hypothesis204includes a standalone score205that models prior statistics of the corresponding candidate hypothesis204. That is, the standalone score205may indicate a confidence that the corresponding candidate hypothesis204is an accurate transcription for the utterance106. The confidence of the standalone score205may also indicate a frequency of previously realized utterances106(e.g., a number of times that the candidate hypothesis204was previously spoken).

The ASR model200may generate any number of candidate hypotheses204(e.g., N may be any integer value). In some examples, the ASR model200outputs a specified number of candidate hypotheses204based on a predefined parameter. For instance, the ASR model200outputs five (5) candidate hypotheses204(i.e., N=5) for every spoken utterance106. For instance, the N candidate hypotheses204may correspond to an N-best list of candidate hypotheses associated with the N candidate hypotheses having the highest standalone scores205. In other examples, the ASR model200outputs all candidate hypotheses204having a standalone score205that satisfies a threshold value.

In the example shown, the ASR model200processes the sequence of acoustic frames110that corresponds to the utterance106“play next song” spoken by the user104and generates three candidate hypotheses204(i.e., N=3). Namely, the candidate hypotheses204include “play next song” having a standalone score205of 0.6, “hey next long” having a standalone score205of 0.3, and “play next pong” having a standalone score205of 0.8. Here, the rescoring process300may output the candidate hypothesis204“play next pong” as the partial transcription120a(FIG.1) because it has the highest standalone score205. Alternatively, the rescoring process300may refrain from outputting the partial transcription until the rescoring process generates the final transcription. Notably, in this example, the candidate hypothesis204having the highest standalone score205is an inaccurate transcription of the utterance106spoken by the user104.

The ASR model200may be a multilingual ASR model configured to recognize utterances106spoken in multiple languages. That is, the single ASR model200may receive an utterance106in a first language and generate N candidate hypotheses204in the first language and receive another utterance106in a different second language and generate N candidate hypotheses204in the second language. Moreover, the single ASR model may receive an utterance106including code-mixed speech that includes terms in both the first and second languages. Thus, the rescoring process300may implement the single multilingual ASR model200in a multilingual speech environment.

In some implementations, the second pass302may receive the N candidate hypotheses204from the first pass301, and generate a corresponding overall score355by integrating additional information for each candidate hypothesis204. The overall score355may indicate a more accurate confidence level than the standalone score205from the first pass301of whether each candidate hypothesis204is an accurate transcription. Thereafter, the second pass302may select the candidate hypothesis204having the highest overall score355as the transcription120(i.e., final transcription120b(FIG.1)).

More specifically, during the second pass302, an external language model (LM)310receives the N candidate hypotheses204and generates a respective external language model score315for each candidate hypothesis204. In some implementations, the external LM310includes a RNN LM. Here, the external LM310may include a plurality of language-specific external LMs310,310a-neach trained on text-only data (i.e., unpaired data) for a particular language. As such, external LM310and language-specific external LM310may be used interchangeably herein. Thus, each language-specific external LM310is configured to generate an external language model score (i.e., language model score)315for utterances106in a respective language. For example, a first language-specific external LM310,310atrained on English text-only data generates language model scores315for utterances106spoken in English, and a second language-specific external LM310,310btrained on Spanish text-only data generates language model scores315for utterances106spoken in Spanish. The plurality of external LMs310may by trained on any number of languages where each external LM310is trained with text-only data of a different respective language.

Accordingly, the external LM310may receive a language identifier107that indicates a language of the utterance106to select the language-specific external LM310from among the plurality of language-specific external LMs310that corresponds to the language of the utterance106. Put another way, the rescoring process300may select the language-specific external LM310based on the language identifier107. In some examples, the ASR model200determines the language identifier107based on processing the sequence of acoustic frames110of the utterance106. In other examples, the ASR model200obtains the language identifier107from an external source. For instance, a user may configure the ASR model for a particular language. In other instances, the ASR model200may determine an identity of the user104that spoke the utterance106and identify the language identifier107based on a language associated with the identified user104.

Accordingly, during the second pass302, the rescoring process300selects the external LM310that corresponds to the language of the utterance106based on the language identifier107, and generates the language model score315for each candidate hypothesis204. The language model score315indicates a likelihood that the sequence of hypothesized terms in the candidate hypothesis204are spoken by the user104. For example, the LM310will generate a higher language model score315for candidate hypothesis204“What is the weather today?” as opposed to the candidate hypothesis204“What is the weather hooray?” In particular, the LM310generates the higher language model score315for “What is the weather today?” because this sequence of hypothesized terms may have been included in the text-only training data more frequently than “What is the weather hooray?”

The example rescoring process300also includes the neural oracle search (NOS) model320that receives the N candidate hypotheses204, the sequence of acoustic frames110, and the label history245(e.g., previously output words, word-pieces, and/or graphemes). The label history245(y0:1-1) may be output by the ASR model200, the second pass302of the rescoring process300(e.g., via a rescorer350), or some combination thereof. In some examples, the label history245includes a transcription for a previous utterance106spoken by the user104. For instance, the user104may have previously spoke a previous utterance106of “do I have any meetings today?” that represents the label history245for a current utterance106of “what about tomorrow?” In other examples, the label history245includes all terms preceding a current label of the utterance. For instance, for the utterance106“play my playlist” the label history245may correspond to the terms “play my,” where the current term (e.g., next hypothesized term) in the utterance106is “playlist.” Optionally, the NOS model320may receive the language identifier107indicating the language of the utterance106spoken by the user104.

FIG.3Aillustrates an example of the rescoring process300,300athat includes a plurality of language-specific NOS models320S,320Sa-n. Here, each language-specific NOS model320S is trained on pairwise data (i.e., transcribed audio training data) of a particular language. Accordingly, during the second pass302, the rescoring process300selects a language-specific NOS model320S from among the plurality of language-specific NOS models320S that corresponds to the language of the utterance106based on the language identifier107. As such, the example rescoring process300aassumes that the language identifier107is available to select the correct language-specific NOS model320S.

Alternatively,FIG.3Billustrates an example rescoring process300,300bthat includes a multilingual NOS model320,320M. In this example, the multilingual NOS model320M is trained on pairwise data (i.e., transcribed audio training data) for any number of languages. Thus, the example rescoring process300bmay implement a single multilingual NOS model320M in a multilingual speech environment. Notably, the example rescoring process300bdoes not require the use of any language identifier107since selection of a language-specific NOS model320S (as described with reference toFIG.3A) associated with the language of the utterance106is not required. Thus, the utterance106may include a multilingual utterance that includes codemixing of speech across two or more languages. As used herein, the NOS model320may include either a language-specific NOS model320S (FIG.3A) that the rescoring process300aselects based on the language identifier107or the multilingual NOS model (FIG.3B).

With continued reference toFIGS.3A and3B, the NOS model320includes a prior model that predicts the next label Yigiven the label history245. That is, the prior model predicts a prior score for the next label based on previously recognized word, word-pieces, and/or graphemes. The prior model of the NOS model320may include a two-layer, 512 units per layer unidirectional LSTM. The prior model trains using labeled audio training data and a cross-entropy loss. Moreover, the NOS model320includes a posterior model that predicts a posterior score by combining the label history245with the sequence of acoustic frames110from the first pass301in a label-synchronous fashion. The posterior model of the NOS model320may include a two-layer, 512 units per layer unidirectional LSTM, with a two-layer, 128 units per layer label synchronous attention mechanism. The posterior model trains with labeled audio training data and a cross-entropy loss to predict the next label Yigiven the label history245and the sequence of acoustic frames110. The NOS model320sums the token level prior score and the token level posterior score to generate the un-normalized likelihood score325. As such, the un-normalized likelihood score325is a sequence level score represented by the summation as follows:

Sθ1(X⁢❘"\[LeftBracketingBar]"Y=y0:U)=∑i=0U⁢ϕ⁡(X,y0:i-1⁢❘"\[LeftBracketingBar]"Yi=y)∝∑i=0U⁢log⁢P⁡(Yi=y⁢❘"\[LeftBracketingBar]"X,y0:i-1)-∑i=0U⁢log⁢P⁡(Yi=y⁢❘"\[LeftBracketingBar]"y0:i-1)=log⁢P⁡(Y⁢❘"\[LeftBracketingBar]"X)-log⁢P⁡(Y)∝log⁢P⁡(X⁢❘"\[LeftBracketingBar]"Y)(1)

In Equation 1, Sθ1represents the un-normalized likelihood score325,

The rescorer350receives the standalone score205, the language model score315, and the un-normalized likelihood score325for each candidate hypothesis204of the N candidate hypotheses204and generates the respective overall score355. In particular, the rescorer350generates the overall score355for each candidate hypothesis204based on any combination of the standalone score205, the language model score315, and the un-normalized likelihood score325. In some examples, the rescorer350sums the standalone score205, the language model score315, and the un-normalized likelihood score325linearly to determine a sequence-level overall score355represented by:

P˜(Oracle=i⁢❘"\[LeftBracketingBar]"X,H1:N)=exp⁢(Score⁢(Hi,X))∑jexp⁢(Score⁢(Hj,X))(2)Score⁢(Hi,X)=λ1⁢Sθ1(X⁢❘"\[LeftBracketingBar]"Hi)+λ2⁢Sθ2(Hi)+Sθ3(i)(3)

In Equation 3, Sθ1represents the un-normalized likelihood score325, Sθ2represents the external language model score315, and Sθ3represents the standalone score205. To optimize model parameters of the rescorer350during training, the rescoring process300uses a cross-entropy objective between the posterior score and a sequence-level ground truth distribution. In some examples, the training process assigns the total ground truth distribution to the ground-truth transcription and assigns all other candidate hypotheses to zero. In other examples, the training process assigns the total ground truth distribution uniformly across all candidate hypotheses having a word error rate (WER) below the best candidate hypothesis (i.e., ground truth transcription). In yet other examples, the training process applies a Softmax function to a negative edit-distance between each candidate hypothesis and the ground truth transcription.

Thereafter, the rescorer350selects the candidate hypothesis204from among the N candidate hypotheses204having the highest overall score355as a final transcription120of the utterance106. In the example shown, the candidate hypotheses204include “play next song” having an overall score205of 0.9, “hey next long” having an overall score205of 0.3, and “play next pong” having an overall score205of 0.5. Continuing with the example, the rescorer250selects the candidate hypothesis204of “play next song” (denoted by the solid line box) having the highest overall score205of 0.9 as the transcription120(e.g., final transcription120b(FIG.1)). Notably, the candidate hypothesis204with the highest standalone score205(i.e., likelihood of being the correct transcription) is not the correct candidate hypothesis204, but the candidate hypothesis with the highest overall score355is the correct transcription from the second pass302.

FIG.4is a flowchart of an exemplary arrangement of operations for a computer-implemented method400of using multi-lingual re-scoring models for automatic speech recognition. At operation402, the method400includes receiving a sequence of acoustic frames110extracted from audio data that corresponds to an utterance106. At operation404, during a first pass301, the method400includes processing the sequence of acoustic frames110to generate N candidate hypotheses204,204a-nfor the utterance106using a multilingual speech recognition model (i.e., ASR model)200. During a second pass302, for each candidate hypothesis204of the N candidate hypotheses204, the method400performs operations406-412. At operation406, the method400includes generating a respective un-normalized likelihood score325using a NOS model320. Here, the NOS model320generates the un-normalized likelihood score325based on the sequence of acoustic frames110and the corresponding candidate hypothesis204. At operation408, the method400includes generating a respective external language model score315using a language model310. At operation410, the method400includes generating a standalone score205that models prior statistics of the corresponding candidate hypothesis204generated during the first pass301. At operation412, the method400includes generating a respective overall score355for the candidate hypothesis255based on the un-normalized likelihood score325, the external language model score315, and the standalone score205. At operation414, the method400includes selecting the candidate hypothesis204having the highest respective overall score355from among the N candidate hypotheses204as a final transcription120of the utterance106.

FIG.5is schematic view of an example computing device500that may be used to implement the systems and methods described in this document. The computing device500is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

The computing device500includes a processor510, memory520, a storage device530, a high-speed interface/controller540connecting to the memory520and high-speed expansion ports550, and a low speed interface/controller560connecting to a low speed bus570and a storage device530. Each of the components510,520,530,540,550, and560, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor510can process instructions for execution within the computing device500, including instructions stored in the memory520or on the storage device530to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display580coupled to high speed interface540. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices500may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory520stores information non-transitorily within the computing device500. The memory520may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory520may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

The storage device530is capable of providing mass storage for the computing device500. In some implementations, the storage device530is a computer-readable medium. In various different implementations, the storage device530may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory520, the storage device530, or memory on processor510.

The high speed controller540manages bandwidth-intensive operations for the computing device500, while the low speed controller560manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller540is coupled to the memory520, the display580(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports550, which may accept various expansion cards (not shown). In some implementations, the low-speed controller560is coupled to the storage device530and a low-speed expansion port590. The low-speed expansion port590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device500may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server500aor multiple times in a group of such servers500a, as a laptop computer500b, or as part of a rack server system500c.

Various implementations of the systems and techniques described herein can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

The processes and logic flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.