Patent Description:
Speech recognition systems are used to transcribe speech to text in many daily applications today. These speech recognition systems may be embedded on user devices such as smart home devices or smartphones, or used in cloud-related services. Typically, speech recognition systems are designed to be either streaming or non-streaming systems. Non-streaming speech recognition systems are afforded the opportunity to take advantage of the full sequence of audio when transcribing speech, but to account for the full sequence of audio, a non-streaming speech recognition system requires the receipt of the entire speech sequence prior to transcript generation. In contrast, streaming speech recognition systems have been developed for real-time speech recognition tasks, such as user assistants and real-time captioning. Yet due to their streaming constraints, these streaming speech recognition systems cannot utilize the full context of audio sequence and tend to perform poorer than their non-streaming counterparts. Due to the performance disparity between streaming and non-streaming speech recognition systems, there is a need to improve the performance of streaming speech recognition systems.

It is known according to the publication<NPL>), a learning method for ASR streaming models leveraging a non-streaming ASR model as teacher to generate transcripts on an arbitrarily large data set, which is then used to distill knowledge into ASR streaming models.

It is further known from the publication <NPL>, a technique teaching how to use knowledge distillation to combine acoustic models. A teacher ensemble is used to provide soft labels for a student model.

One aspect of the invention provides a computer-implemented method according to claim <NUM> for training a streaming automatic speech recognition student model. The method, when executed by data processing hardware causes the data processing hardware to perform operations. The operations include receiving a plurality of unlabeled student training utterances. The operations also include, for each unlabeled student training utterance, generating a transcription corresponding to the respective unlabeled student training utterance using a plurality of non-streaming automated speech recognition (ASR) teacher models. The operations further include distilling a streaming ASR student model from the plurality of non-streaming ASR teacher models by training the streaming ASR student model using the plurality of unlabeled student training utterances paired with the corresponding transcriptions generated by the plurality of non-streaming ASR teacher models.

Another aspect of the invention provides a system according to claim <NUM> for training a streaming automatic speech recognition student model. The system includes data processing hardware and memory hardware in communication with data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware causes the data processing hardware to perform operations. The operations include receiving a plurality of unlabeled student training utterances. The operations also include, for each unlabeled student training utterance, generating a transcription corresponding to the respective unlabeled student training utterance using a plurality of non-streaming automated speech recognition (ASR) teacher models. The operations further include distilling a streaming ASR student model from the plurality of non-streaming ASR teacher models by training the streaming ASR student model using the plurality of unlabeled student training utterances paired with the corresponding transcriptions generated by the plurality of non-streaming ASR teacher models.

Another aspect of the invention provides a computer-readable medium according to claim <NUM>. Implementations of the computer-implemented method or the system of the disclosure may include one or more of the following optional features. In some implementations, generating the transcription corresponding to the respective unlabeled student training utterance includes receiving, as input at the plurality of non-streaming ASR teacher models, the respective unlabeled student training utterance, predicting, at each non-streaming ASR teacher model, an initial transcription for the respective unlabeled student training utterance, and generating the transcription for the respective unlabeled student training utterance to be output by the plurality of non-streaming ASR teacher models based on the initial transcriptions of each non-streaming ASR teacher model predicted for the respective unlabeled student training utterance. In these implementations, generating the transcription for the respective unlabeled student training utterance to be output by the plurality of non-streaming ASR teacher models based on the initial transcriptions of each non-streaming ASR teacher model predicted for the respective unlabeled student training utterance includes constructing the transcription using output voting. Constructing the transcription using output voting includes aligning the initial transcriptions from each non-streaming ASR teacher model to define a sequence of frames, dividing each initial transcription into transcription segments, each transcription segment corresponding to a respective frame, selecting, for each respective frame, a most repeated transcription segment across all initial transcriptions, and concatenating the most repeated transcription segment of each respective frame to form the transcription.

In these implementations of either the method or the system, the streaming ASR student model may include a recurrent neural network transducer (RNN-T) architecture. The streaming ASR student model may include a conformer-based encoder. Each non-streaming ASR teacher model may include a connectionist temporal classification (CTC) architecture. Here, the CTC architecture includes a language model configured to capture contextual information for a respective utterance. Each non-streaming ASR teacher model may include a conformer-based encoder. In some examples, the plurality of non-streaming ASR teacher models includes at least two different recurrent neural network architectures. In these examples, a first non-streaming ASR teacher model includes a recurrent neural network architecture and a second non-streaming ASR teacher model includes a connectionist temporal classification (CTC) architecture.

Automatic speech recognition (ASR) systems continue to constantly develop to support the demands of speech-enabled devices. As speech-enabled devices incorporate ASR systems for various functionality including on-device applications, the ASR systems are relied on to meet on-device expectations. These on-device expectations include speech transcription with minimal latency to facilitate human-computer interactions that do not negatively affect a user experience. With the desire for minimal latency, these user devices or speech-enabled devices are generally unable to leverage a remote ASR system (e.g., an ASR system located on a remote computing system) since the remote communication would inherently introduce some degree of latency. In addition to demanding minimal latency, speech-enabled devices are often constrained with respect to the amount of resources that are available locally on the device. For these reasons, ASR systems have evolved to use end-to-end streaming models for on-device speech recognition tasks.

A streaming ASR model refers to a speech recognition model deployed in an ASR system that may be used to transcribe speech in real-time (or near real-time). To perform transcription in real-time, a streaming ASR model produces and updates transcription results (i.e., hypotheses or predictions) on a frame-by-frame basis. With a frame-by-frame transcription approach, a streaming ASR model accounts for little to no future context for a given speech input. That is, if the streaming ASR model receives two sentences of speech as input, the streaming ASR model cannot utilize the full context of the input, the two sentences, to produce its transcription, but rather produces transcription results as the streaming ASR model receives pieces of the two sentences (e.g., frames of word pieces) In contrast, a non-streaming ASR model leverages the full context of a given speech input to produce its transcription results. Therefore, if a non-streaming ASR model receives two sentences of speech as input, it would consider the entirety of the two sentences in order to produce its transcription. One reason on-device speech recognition tasks have come to use streaming ASR models is that non-streaming ASR models may be rather large models in order have the proper memory to consider the entirety of the speech input to produce transcription results. For this reason, non-streaming ASR models are often deployed remotely rather than on-device because these non-streaming ASR models may be resource intensive. Therefore, streaming ASR models may offer a more compact memory footprint when compared to non-streaming ASR models which makes streaming ASR models typically more suitable to be used on-device.

Unfortunately, as streaming ASR models are becoming increasingly common on-device, streaming ASR models are not without their setbacks. For example, by operating on a frame-by-frame basis, the streaming ASR model is not afforded the contextual benefits of a non-streaming ASR model. In other words, when a model is constrained in terms of the context that it may utilize to generate its transcription results, the overall accuracy of the transcription may suffer. Yet often, a user of the speech recognition device may prefer the speech recognition system (i.e., the ASR system, to occur in real-time, or a streaming manner, even though the results may not always be the most accurate. This tradeoff has been widely accepted, but also leaves room for improvement with respect to transcription accuracy, especially given the fact that ASR models, such as non-streaming ASR models, exist and tend to have better transcription performance than their streaming model counterparts.

To address the transcription performance for streaming ASR models, implementations described herein (not encompassed by the claimed invention) are directed toward leveraging a non-streaming ASR model as a teacher to generate transcripts for a streaming ASR student model. The transcripts generated by the non-streaming teacher model may then be used to distill knowledge into the streaming ASR model. In this respect, the non-streaming ASR model functions as a teacher model while the streaming ASR model that is being taught by the distillation process is a student model. With this distillation approach to training a streaming ASR model, the streaming ASR student model learns to generate a transcript for a given speech input that is imbued with some of the increased transcription accuracy of the non-streaming ASR teacher model. That is, since full-context ASR models (i.e., non-streaming ASR models) perform better than streaming ASR models, a non-streaming ASR model may function as a strong teacher that fosters a more robust streaming ASR student model. For example, experiments have shown that the word error rate (i.e., a common speech recognition performance metric) for the streaming ASR student model is reduced when compared to a streaming ASR model taught by conventional means. Moreover, this approach is relatively efficient in that, by generating transcripts with the non-streaming ASR teacher model that will be used directly for training the streaming ASR student model, the approach avoids additional distillation pre-training to correct issues such as alignment mismatch.

The distillation approach also has some benefits that translate to ASR models for different languages (e.g., a French streaming ASR model or a Spanish streaming ASR model). One common issue is that a particular language may have data scarcity. Meaning that, there does not currently exist a corpus of adequate training data for that particular language. This may occur for several reasons, but one such reason is that audio samples used to train ASR models generally need to be paired with a corresponding label or transcription of the audio sample. Unfortunately, in some languages that suffer from audio sample scarcity, it is even less common for their audio samples to have a corresponding label. This fact alone makes training a streaming ASR model difficult not only because labels inform the training process whether a model is reaching a particular level of prediction accuracy, but also because end-to-end (E2E) models are notoriously data-hungry. Generally speaking, speech recognition models have evolved from multiple models where each model had a dedicated purpose to integrated models where a single neural network is used to directly map an audio waveform (i.e., input sequence) to an output sentence (i.e., output sequence). This integration has resulted in a sequence-to-sequence approach, which generates a sequence of words (or graphemes) when given a sequence of audio features. With an integrated structure, all components of a model may be trained jointly as a single E2E neural network. Here, an E2E model refers to a model whose architecture is constructed entirely of a neural network. A fully neural network functions without external and/or manually designed components (e. g, finite state transducers, a lexicon, or text normalization modules). Additionally, when training E2E models, these models generally do not require bootstrapping from decision trees or time alignments from a separate system, but often demand a large corpus of training data. Due to this demand, E2E streaming ASR models may not be feasible for particular languages. Yet with a teacher-student distillation approach, a streaming ASR student model may be trained in a particular language using unlabeled audio samples. For example, the non-streaming ASR teacher model, which is already trained to predict a label (i.e., generate a transcription for an given audio samples), generates labels for unlabeled audio samples so that the unlabeled audio samples may be paired with their corresponding labels generated by the non-streaming ASR teacher model to form a training data set for the streaming ASR student model. This therefore allows a streaming ASR student model to be distilled from unlabeled audio samples.

According to the invention, a streaming ASR student model learns from distilling knowledge from multiple teacher models. That is, the transcriptions from multiple non-streaming ASR teacher models are used and may be combined to form a final transcription which will then be used in the distillation process to train the streaming ASR student model. In this approach, the final transcription functions to reduce errors that may occur in any single teacher model. In some implementations, when the streaming ASR model learns from multiple teacher models, the teacher models include different types of neural networks to diversely contribute to the final transcription. For instance, the teacher models may include Connectionist Temporal Classification (CTC) models, recurrent neural network-transducer (RNN-T) models, conformer models, etc., or any combination thereof. With a diverse ensemble of teacher models, the final transcription can be more accurate than any single model (e.g., due to inherent limitations of any particular model type).

Referring to <FIG>, in some implementations, a system <NUM> of a speech environment includes a user <NUM> providing a user interaction <NUM> to interact with a voice-enabled device <NUM> (also referred to as a device <NUM> or a user device <NUM>). Here, the user interaction <NUM> is a spoken utterance <NUM>, 12U corresponding to a query or a command to solicit a response from the device <NUM> or to have the device <NUM> execute a task specified by the query. In this sense, the user <NUM> may have conversation interactions with the voice-enabled device <NUM> to perform computing activities or find answers to questions.

The device <NUM> is configured to capture user interactions <NUM>, such as speech, from one or more users <NUM> within the speech environment. An utterance 12U spoken by the user <NUM> may be captured by the device <NUM> and may correspond to a query or a command for a digital assistant interface <NUM> executing on the device <NUM> to perform an operation/task. The device <NUM> may correspond to any computing device associated with the user <NUM> and capable of receiving audio signals. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.). computers, wearable devices (e.g., smart watches), music player, casting devices, smart appliances (e.g., smart televisions) and internet of things (IoT) devices, remote controls, smart speakers, etc. The device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations related to speech processing.

The device <NUM> further includes an audio subsystem with an audio capturing device (e g. , an array of one or more microphones) <NUM> for capturing and converting audio data within the speech environment into electrical signals. While the device <NUM> implements the audio capturing device <NUM> (also referred to generally as a microphone <NUM>) in the example shown, the audio capturing device <NUM> may not physically reside on the device <NUM>, but be in communication with the audio subsystem (e.g., peripherals of the device <NUM>). For example, the device <NUM> may correspond to a vehicle infotainment system that leverages an array of microphones positioned throughout the vehicle.

A speech-enabled interface (e.g., a digital assistant interface) <NUM> may field the query or the command conveyed in the spoken utterance 12U captured by the device <NUM>. The speech-enabled interface <NUM> (also referred to as interface <NUM> or an assistant interface <NUM>) generally facilitates receiving audio data <NUM> corresponding to an utterance 12U and coordinating speech processing on the audio data <NUM> or other activities stemming from the utterance 12U to generate a response <NUM>. The interface <NUM> may execute on the data processing hardware <NUM> of the device <NUM>. The interface <NUM> may channel audio data <NUM> that includes an utterance 12U to various systems related to speech processing. For instance, <FIG> illustrates that the interface <NUM> communicates with a speech recognition system <NUM> (e.g., an automatic speech recognition (ASR) system). Here, the interface <NUM> receives audio data <NUM> corresponding to an utterance 12U and provides the audio data <NUM> to the speech recognition system <NUM> In some configurations, the interface <NUM> serves as an open communication channel between the microphone <NUM> of the device <NUM> and the speech recognition system <NUM> In other words, the microphone <NUM> captures an utterance <NUM> in an audio stream <NUM> and the interface <NUM> communicates audio data <NUM> corresponding to the utterance 12U converted from the audio stream <NUM> to the speech recognition system <NUM> for processing. More specifically, the speech recognition system <NUM> includes a speech recognition model <NUM> that processes the audio data <NUM> to generate a transcription <NUM> for the utterance 12U and may perform semantic interpretation on the transcription <NUM> to identify an appropriate action to perform. The interface <NUM> used to interact with the user <NUM> at the device <NUM> may be any type of program or application configured to execute the functionality of the interface <NUM>. For example, the interface <NUM> is an application programming interface (API) that interfaces with other programs hosted on the device <NUM> or in communication with the device <NUM>.

Referring specifically to the example of <FIG>, an utterance 12U by the user <NUM> states, "Who taught Alexander the Great?" Here, receipt of the utterance 12U by the interface <NUM> causes the interface <NUM> to relay captured audio data corresponding to the query "who taught Alexander the Great?" to the speech recognition system <NUM> for processing. In some examples, the utterance 12U may include a hotword as a preceding invocation phrase that triggers the device <NUM> to wake-up (from a sleep or hibernation state) and initiates speech recognition on the hotword and/or one or more terms following the hotword. For example, detection of the hotword by a hotword detection system (e.g., a neural network-based model configured to detect acoustic features indicative of the hotword without performing speech recognition or semantic analysis) triggers the interface <NUM> to open the microphone <NUM> and relay subsequently captured audio data corresponding to the query "who taught Alexander the Great?" to the speech recognition system <NUM> for processing.

In response to receiving an utterance 12U in the audio stream <NUM>, the interface <NUM> relays the audio data <NUM> corresponding to this utterance 12U to the speech recognition system <NUM> and the speech recognition system <NUM> performs speech recognition on the audio data <NUM> to generate a speech recognition result (e.g., transcription) <NUM> for the utterance 12U. The speech recognition system <NUM> and/or the interface <NUM> performs semantic interpretation on the speech recognition result <NUM> to determine that the utterance 12U corresponds to a search query for the identity of the person who educated Alexander the Great. Here, the interface <NUM> may submit the transcription <NUM> to a search engine <NUM> that searches for, and returns a search result <NUM> of "Aristotle" for the query of "Who taught Alexander the Great?" The interface <NUM> receives this search result <NUM> of "Aristotle" from the search engine <NUM> and, in turn, communicates "Aristotle" to the user <NUM> as a response <NUM> to the query of the utterance 12U. In some examples, the response <NUM> includes synthesized speech audibly output from the device <NUM>.

In some implementations, the device <NUM> communicates via a network <NUM> with a remote system <NUM>. The remote system <NUM> may include remote resources <NUM>, such as remote data processing hardware <NUM> (e.g., remote servers or CPUs) and/or remote memory hardware <NUM> (e.g., remote databases or other storage hardware). The device <NUM> may utilize the remote resources <NUM> to perform various functionality related to speech processing. For instance, the search engine <NUM> may reside on the remote system <NUM>. In one example, the speech recognition system <NUM> may reside on the device <NUM> for performing on-device automated speech recognition (ASR) Here, <FIG> depicts the search engine <NUM> in a dotted box to indicate that these components may reside on-device <NUM> or server-side (i.e., at the remote system <NUM>).

Here, the ASR model <NUM> is a streaming ASR student model <NUM> that is deployed on-device rather than in the remote system <NUM>. This means that the model <NUM> utilizes the local resources <NUM>, <NUM> of the device <NUM> to perform speech recognition. <FIG> illustrates the speech recognition system <NUM> connected to the network <NUM> with a dotted line to indicate that speech recognition system <NUM> may leverage remote resources <NUM> of the remote system <NUM> in a particular circumstance. That is, prior to inference or deployment of the model <NUM> (i.e., prior to implementation), the streaming ASR model <NUM> learns how to generate speech recognition results (i. e, transcripts <NUM> or labels) from a non-streaming ASR model (e.g., shown as teacher model <NUM> in <FIG>) located in the remote system <NUM>. In this sense, the model <NUM> receives distilled knowledge from a non-streaming ASR model that is able to be larger (i.e., utilize a greater amount of remote resources <NUM> when compared to the local resources <NUM>, <NUM>) and generally more accurate at speech recognition performance than conventional streaming ASR models.

The student model <NUM> may have different types of neural network architecture. In some examples, the student model <NUM> is an RNN-T model with an encoder-decoder architecture. When the student model <NUM> is an RNN-T model, the student model <NUM> may include an encoder network with multiple layers of unidirectional Long Short Term Memory (LSTM) cells (e.g., <NUM> layers of LSTMs with <NUM> cells). Here, each LSTM may have a projection layer of, for example, <NUM> outputs For this RNN-T model, the decoder of the student model <NUM> may be two unidirectional LSTMs (e.g., with <NUM> units and <NUM> projections similar to the encoder). The encoder-decoder architecture of the RNN-T may also include a joint network. The joint network may be a fully connected layer with <NUM> units. In some configurations, the encoder of the student model <NUM> (e.g., when the student model <NUM> is a RNN-T model) uses a conformer encoder, but, as a streaming ASR model, any attention layers and/or convolutions are not fully contextual to ensure streaming capability.

<FIG> depict a training process <NUM> for the streaming ASR model <NUM> (not encompassed by the claimed invention) while <FIG> and <FIG> illustrate a variation (encompassed by the claimed invention) of this training process <NUM> that uses an ensemble of teacher models <NUM> to train the streaming ASR model <NUM>, which is referred to as an ensemble training process <NUM>. Both the training process <NUM> and the ensemble training process <NUM> may be relatively similar except to the extent that more than one teacher model <NUM>, 210a-n is used to generate a final transcription <NUM> in the ensemble training process <NUM>. This means that, although not depicted, the ensemble training process <NUM> may also include additional components, such as the segmenter <NUM> and the augmenter <NUM>, in its process flow.

Referring to <FIG>, the training process <NUM> is a distillation process where a non-streaming ASR model <NUM> (referred to as a no-streaming ASR teacher model <NUM> or simply a teacher model <NUM>) distills its knowledge to the streaming ASR model <NUM> (also referred to as the streaming ASR student model <NUM> or simply the student model <NUM>). Here, the teacher model <NUM> distills its knowledge to the student model <NUM> by training the student model <NUM> with a plurality of student training samples <NUM> that include, at least in part, labels or transcriptions <NUM> generated by the teacher model <NUM>. In this sense, during the training process <NUM>, the student model <NUM> learns to predict its own transcriptions <NUM> from the transcriptions <NUM> produced by the teacher model <NUM>. Since the student model <NUM> is a streaming model that produces and updates speech recognition results that form the transcription <NUM> on a frame-by-frame basis, the training process <NUM> therefore leverages more accurate non-streaming model transcripts <NUM> to improve the accuracy of the student model <NUM> deployed for real-time speech recognition tasks.

In some implementations, during the training process <NUM>, the teacher model <NUM> receives unlabeled training samples <NUM> from a corpus <NUM> of unlabeled training samples <NUM>. The corpus <NUM> generally refers to any collection of unlabeled audio data (e.g., a database or a data store for audio data samples). An unlabeled training sample <NUM> refers to a sample of audio data that is without an accompanying label. That is, an unlabeled training sample <NUM> is unsupervised data without a corresponding transcript providing the label for the sample of audio data When the teacher model <NUM> receives an unlabeled training sample <NUM>, as input, the teacher model <NUM> is configured to generate, as output, a transcription <NUM> that predicts a label for the unlabeled training sample <NUM> This prediction function by the teacher model <NUM> therefore labels the unlabeled training sample <NUM> to form a student model training sample <NUM> that includes both the audio data of the unlabeled training sample <NUM> along with its corresponding transcription <NUM> generated by the teacher model <NUM>. By using the teacher model <NUM>, a plurality or corpus <NUM> of unlabeled training samples <NUM> are converted to a corpus <NUM> of student training model samples <NUM>, 232a-n. The training process <NUM> then feeds the student training model samples <NUM> into the student model <NUM> to enable the student model <NUM> to learn to predict a transcription <NUM> based on the audio data of the previously unlabeled sample <NUM> along with its predicted transcription <NUM> generated by the teacher model <NUM>. In this manner, the training process <NUM> distills a streaming ASR student model <NUM> using unlabeled training samples <NUM> and transcriptions <NUM> from the non-streaming ASR teacher model <NUM> Moreover, this training process <NUM> has the benefit that it uses unlabeled audio samples. This may be advantageous because it can often be difficult to obtain or to generate accurate labels for samples of audio data (e.g., especially for particular languages). For instance, some training processes have to manual label unlabeled audio data or to train a model with a smaller body of training samples due to a lack of labeled audio data (i.e., a lack of transcripts for the audio data).

Referring to <FIG>, the training process <NUM> may include a segmenter 240that is configured to receive an unlabeled training sample <NUM> and generate an unlabeled segment <NUM> from the unlabeled training sample <NUM>. The unlabeled segment <NUM> refers to some length or portion of the unlabeled training sample <NUM>. For instance, unlabeled training samples <NUM> may be of various length (e.g., time duration) and the segmenter <NUM> may divide an unlabeled training sample <NUM> into a segments <NUM> of a lessor length or a finite length. For example, if the audio data of the unlabeled training sample <NUM> is two minutes long, the segmenter <NUM> may divide into smaller portions, such as <NUM> or <NUM> second segments <NUM>. In this respect, the segmenter <NUM> may function to generate a greater number of unlabeled audio data samples (i.e., segments <NUM>), to generate uniform length samples that will be fed to the teacher model <NUM> during the training process <NUM>, and/or to prevent the teacher model <NUM> from receiving samples that are difficult for the teacher model <NUM> to handle (e g. , of a long duration that requires a significant amount of memory to generate a transcript <NUM> with the full context of the sample). In some examples, the segmenter <NUM> converts the unlabeled training sample <NUM> into segments <NUM> of random length. Here, the random length may be constrained to be, for example, less than a particular duration, but otherwise seek to represent how a speech recognition model, during implementation, may receive a speech input of varying lengths and need to be able to produce accurate recognition results for that speech input. With an unlabeled training sample <NUM> segmented by the segmenter <NUM>, the teacher model <NUM> then receives unlabeled segments <NUM> of the unlabeled training sample <NUM> as input and predicts a transcript <NUM> for these unlabeled segments <NUM> similar to when the teacher model <NUM> receives unlabeled training samples <NUM> (e.g., as shown in <FIG>). As stated previously, although the segmenter <NUM> is not shown in the ensemble training process <NUM> of <FIG> and <FIG>, a segmenter with similar functionality may also be used in the same manner during the ensemble training process <NUM>.

Referring to <FIG>, the training process <NUM> may also include an augmenter <NUM> (e.g., in addition to the segmenter <NUM>). An augmenter <NUM> functions to alter the audio data corresponding to the unlabeled training samples <NUM> after the teacher model <NUM> has generated a transcript <NUM> for the unlabeled training samples <NUM> to form a student training samples <NUM>. Because the augmenter <NUM> is downstream from the teacher model <NUM> and the teacher model <NUM> has already produced a transcript <NUM> for the student training sample <NUM>, the augmenter <NUM> augments or alters the audio data of the unlabeled training samples <NUM> in a manner that would not have an impact on the predicted transcript <NUM>. This augmentation process by the augmenter forms an augmented audio sample <NUM> that will be used to train the student model <NUM>. For instance, the augmenter <NUM> may be used to add noise or perform some other type of spectral augmentation. That is, the augmenter <NUM> may modify the waveform of the audio data to form the augmented audio sample <NUM> for the student training sample <NUM>. In this respect, the student model <NUM> learns from a noisy student learning framework where the student training samples <NUM> include the augmented audio sample <NUM> along with its predicted transcription <NUM> from the teacher model <NUM>. A noisy learning framework may allow the student model <NUM> to learn that slight differences in audio samples may result in the same transcript <NUM> and thus provide an additional degree of robustness for the training process <NUM>. Much like the segmenter <NUM>, although the augmenter <NUM> is not shown in the ensemble training process <NUM> of <FIG> and <FIG>, an augmenter with similar functionality may also be used in the same manner during the ensemble training process <NUM>.

Referring to <FIG> and <FIG>, the ensemble training process <NUM> (encompassed by the claimed invention) differs from the training process <NUM> in that the ensemble training process <NUM> trains the student model <NUM> using more than one teacher model <NUM>. That is, the student model <NUM> is distilled from a plurality of teacher models <NUM>, 210a-n by training the student model <NUM> using the unlabeled training utterances <NUM> paired with their corresponding transcriptions <NUM> generated by the teacher models <NUM>. By using more than one teacher model <NUM>, the student model <NUM> learns from (i) a more diverse source (i.e., multiple teacher models <NUM>, 210a-n) and (ii) a final transcription <NUM> generated by the ensemble of teacher models <NUM> with improved transcription quality (e. g , when compared to a transcription from a single teacher model <NUM>).

In some examples, each teacher model <NUM> of the teacher ensemble receives the same unlabeled training sample <NUM> (or unlabeled segment <NUM>) and generates a transcription <NUM> for the received unlabeled training sample <NUM>. For example, <FIG> depicts three teacher models <NUM>, 210a-c where the first teacher model 210a generates a first transcript <NUM>, 212a for the unlabeled training sample <NUM>, the second teacher model 210b generates a second transcript <NUM>, 212b for the unlabeled training sample <NUM>, and the third teacher model 210c generates a third transcript <NUM>, 212c for the unlabeled training sample <NUM>. Because each teacher model <NUM> generates its own transcript <NUM> independently, the transcript <NUM> of one teacher model <NUM> may have differences when compared to the transcript <NUM> for another teacher model <NUM> even though the transcript <NUM> is intended to be a label for the same unlabeled training sample <NUM>.

With each teacher model <NUM> generating its own transcript <NUM>, these transcripts <NUM> are then merged or otherwise converted into a final transcript <NUM> to form the student training sample <NUM> where the final transcript <NUM> is the label for the unlabeled training sample <NUM>. In this sense, the transcripts <NUM> from each teacher model <NUM> may be referred to as initial transcripts <NUM> while the final transcript <NUM> is the label that is applied to the unlabeled training sample <NUM> to form the student training sample <NUM>. The techniques to merge these independently created transcripts <NUM> may vary, but generally the merging technique is a transcription error-reducing technique. For instance, during the ensemble training process <NUM>, all of the transcripts <NUM> generated for an unlabeled training sample <NUM> (or unlabeled segment <NUM>) may be compared to one another in order to determine the final transcript <NUM>.

One approach to merge these transcripts <NUM> is to use a voting process. For example, the voting process may be a Recognizer Output Voting Error Reduction (ROVER) technique. In a voting process, the initial transcripts <NUM> generated by the teacher models <NUM> are aligned and divided into segments S where each segment corresponds to an alignment frame F. Here, a segment S of a transcript <NUM> may be a wordpiece, a word, a phrase, etc. With all of the transcripts <NUM> aligned, the most repeated segment S across all transcripts <NUM> in a particular frame F is voted as the segment S to be included in the final transcript <NUM>. In this respect, each occurrence of a segment S in a frame receives a vote and the segment S with the greatest votes (i.e., a majority of votes) is included in the final transcript <NUM>. The ensemble training process <NUM> then concatenates all of the most repeated segments S for each frame F to generate the final transcript <NUM>.

<FIG> illustrates an example of an output voting technique. Here, each transcript <NUM> from the three teacher models 210a-c is divided into five segments S, S<NUM>-<NUM> where each segment S corresponds to one of five frames F, F<NUM>-<NUM>. In this example, each segment S corresponds to a single word (also known as a word transition network (WTN)). For the first frame F<NUM>, each of the three teacher models <NUM> have identical segments S<NUM> that include the word "I. " Because "I" is the most repeated segment (e.g., receiving three votes) in the first frame F<NUM>, the first frame F<NUM> of the final transcription <NUM> will include the word "I. " For the second frame F<NUM>, each of the three teacher models <NUM> have identical segments S<NUM> that include the word "like. " Because "like" is the most repeated segment (e.g., receiving three votes) in the second frame F<NUM>, the second frame F<NUM> of the final transcription <NUM> will include the word "like " For the third frame F<NUM>, two of the three teacher models <NUM> have identical segments S<NUM> that include the word "apples" while one of the three teacher models <NUM> includes the word "staples. " Because "apples" is the most repeated segment (e.g., receiving two of three votes) in the third frame F<NUM>, the third frame F<NUM> of the final transcription <NUM> will include the word "apples. " For the fourth frame F<NUM>, two of the three teacher models <NUM> have identical segments S<NUM> that include the word "and" while one of the three teacher models <NUM> produced a blank. Because "and" is the most repeated segment (e.g., receiving two of three votes) in the fourth frame F<NUM>, the fourth frame F<NUM> of the final transcription <NUM> will include the word "and " For the fifth frame F<NUM>, two of the three teacher models <NUM> have identical segments S<NUM> that include the word "pears" while one of the three teacher models <NUM> includes the word "bears. " Because "pears" is the most repeated segment (e.g., receiving two of three votes) in the fifth frame F<NUM>, the fifth frame F<NUM> of the final transcription <NUM> will include the word "pears. " Based on this process, each winning vote for a frame F is combined together to form the final transcription <NUM>, "I like apples and pears. " This process also shows that an initial transcript <NUM> for a particular teacher model <NUM> may have an error, but that error, when evaluated over all initial transcripts <NUM> is recognizable and able to be prevented from impacting (or less likely to impact) the final transcription <NUM>. For instance, the second teacher model 210b thought the third frame F<NUM> should be "staples" when it is more likely that the word should be "apples" in the third frame F<NUM>.

Non-streaming ASR models, such as the teacher model(s) <NUM>, may use different types of neural network architectures. A non-streaming ASR model may use bi-directional encoding and/or an attention mechanism Some examples of non-streaming ASR models include RNN-T models, conformer models, time-delay neural network (TDNN) models, Connectionist Temporal Classification (CTC) models For example, a conformer transducer model utilizes a combination of a convolution neural network (CNN) and a transformer model's encoder architecture to capture local and global context (i.e., contextual information) for input audio data. In some implementations, the teacher model <NUM> is a conformer model with several conformer blocks (e.g., sixteen blocks) in the encoder and a single Long-Short Term Memory (LSTM) decoder layer (e g. , with <NUM> cells) with a projection layer (e.g., of <NUM> outputs). Here, in this teacher model <NUM>, the attention layer encodes all frames for an input audio sample simultaneously making it a non-streaming model. As an example of a TDNN model, the teacher model <NUM> may include an encoder with a stack of macro layers (e.g., three macro layers) and a decoder network with a single direction LSTM (e.g., with <NUM> hidden units). Each macro layer may constructed from a <NUM>-D convolution, a <NUM>-D max pooling, and three bi-directional LSTM layers (e. g , with <NUM> hidden units in each direction with a <NUM>-dimensional projection). Here, the TDNN model architecture also includes a joint network (e.g., with <NUM> hidden units).

When the teacher model <NUM> has an RNN-T architecture, the exact architecture of the RNN-T may vary. For example, in one configuration of the RNN-T architecture, the encoder includes several macro layers (e.g., seventeen macro layers) where each macro layer has a plurality of attention heads (e.g., eight attention heads), <NUM>-D convolutions (e g. , with a kernel size of fifteen), and a relative positional embedding (e.g., of <NUM> dimensions). In this configuration, the decoder network of the RNN-T may be a single direction LSTM with <NUM> hidden units and a joint network with <NUM> hidden units. Here, the RNN-T architecture may have a final output that uses a <NUM> (four thousand) word piece model. In another RNN-T configuration, the encoder includes two layers of <NUM>×<NUM>2D convolution layers with a 4x time reduction and a channel size of <NUM>. It also includes several conformer blocks (e.g., sixteen blocks) with convolutions. These convolutions may have a kernel size of thirty-two and a positional embedding size of <NUM>. In this RNN-T configuration, the decoder layer has a single LSTM (e.g., with <NUM> cells and a projection layer of <NUM> outputs).

In some examples, the teacher model <NUM> has a CTC architecture. The CTC architecture for the teacher model <NUM> may include an additional language model to capture contextual information. In some implementations, a CTC version of the teacher model <NUM> has an encoder similar to an RNN-T such that the encoder includes two layers of <NUM>×<NUM>2D convolution layers with a 4x time reduction and a channel size of <NUM>. The CTC encoder may also include several conformer blocks (e.g., sixteen blocks) with convolutions. These convolutions may have a kernel size of thirty-two and a positional embedding size of <NUM>. For the CTC architecture, the decoder may differ from that of an RNN-T architecture in that the decoder is a simplified version with a single layer of <NUM> hidden units with a projection layer of <NUM> units.

When using the ensemble training process <NUM>, multiple teacher models <NUM> may all have the sample type of neural network architecture or different types of neural network architecture. That is, one teacher model <NUM> may have a different neural network architecture than a different teacher model <NUM>. For example, to generate diverse initial transcriptions <NUM>, the multiple teacher models <NUM> may have different neural network architectures. For instance, a first teacher model <NUM> may have a CTC architecture while a second teacher model <NUM> may have an RNN-T architecture. In some configurations, though the architecture of the neural network may differ between teacher models <NUM>, the encoder of the neural network may stay the same or relatively the same. For instance, all teacher models <NUM> have a conformer-based encoder. A conformer-based encoder may be beneficial to a non-streaming ASR model, such as the teacher model <NUM>, in that the conformer encoder is capable of accounting for both local and global context for a particular input of audio data.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of training a streaming student model <NUM> (encompassed by the claimed invention). At operation <NUM>, the method <NUM> receives a plurality of unlabeled student training utterances <NUM> For each unlabeled student training utterance <NUM>, at operation <NUM>, the method <NUM> generates a transcription <NUM> corresponding to the respective unlabeled student training utterance <NUM> using a plurality of non-streaming ASR teacher models <NUM>. At operation <NUM>, the method <NUM> distills a streaming ASR student model <NUM> using the plurality of unlabeled student training utterances <NUM> paired with the corresponding transcriptions <NUM> generated by the plurality of non-streaming ASR teacher models <NUM>.

<FIG> is a schematic view of an example computing device <NUM> that may be used to implement the systems (e.g., the speech recognition system <NUM>) and methods (e.g., the training process <NUM>, the ensemble training process <NUM>, and/or the method <NUM>) described in this document.

The computing device <NUM> includes a processor <NUM> (e.g., data processing hardware <NUM>, <NUM>), memory <NUM> (e.g., memory hardware <NUM>, <NUM>), a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>.

The high speed controller <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low speed controller <NUM> manages lower bandwidth-intensive operations Such allocation of duties is exemplary only. In some implementations, the low-speed controller <NUM> is coupled to the storage device <NUM> and a low-speed expansion port <NUM> The low-speed expansion port <NUM>, which may include various communication ports (e.g., USB, Bluetooth. Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g.. through a network adapter.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks. e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback: and input from the user can be received in any form, including acoustic, speech, or tactile input In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

Claim 1:
A computer-implemented method (<NUM>) when executed by data processing hardware (<NUM>) causes the data processing hardware (<NUM>) to perform operations comprising:
receiving a plurality of unlabeled student training utterances (<NUM>);
for each unlabeled student training utterance (<NUM>), generating a transcription (<NUM>) corresponding to the respective unlabeled student training utterance (<NUM>) using a plurality of non-streaming automated speech recognition (ASR) teacher models (<NUM>); and
distilling a streaming ASR student model (<NUM>) from the plurality of non-streaming ASR teacher models (<NUM>) by training the streaming ASR student model (<NUM>) using the plurality of unlabeled student training utterances (<NUM>) paired with the corresponding transcriptions (<NUM>) generated by the plurality of non-streaming ASR teacher models (<NUM>).