Patent Description:
Automatic speech recognition (ASR) systems attempt to provide accurate transcriptions of what a person has said by taking an audio input and transcribing the audio input into text. In many instances, supervised learning is used to train ASR systems with large quantities of labeled training data that includes audio data and a corresponding transcription. Obtaining the large quantity of labeled training data required to train the ASR systems, however, is often difficult because of the amount of time required, costs, and/or privacy concerns associated with collecting the large labeled training datasets. Training ASR systems using unlabeled training data that includes only audio data can alleviate some of the difficulties with collecting large quantities of labeled training data.

In the prior art, it is known from the publication <NPL>, techniques applied in the field of image processing for data augmentation when training a siamese neural network. It is further known according to the publication <NPL>), techniques for training a single supervised embedding model on labelled data from multiple well-resourced languages and then applying it to unseen zero-resource languages. It is further known from the publication <NPL>, techniques using side information in the form of known word pairs to train a siamese convolutional neural network.

One aspect of the disclosure provides a contrastive Siamese network for training a speech recognition model. The contrastive Siamese network includes an unsupervised subnetwork trained on a plurality of unlabeled audio samples that correspond to spoken utterances not paired with any corresponding transcriptions. The unsupervised subnetwork includes a target branch configured to receive a sequence of acoustic frames extracted from the unlabeled audio samples as input to an audio encoder of the speech recognition model and generate, at each of a plurality of time steps, a target branch output for a corresponding acoustic frame in the sequence of acoustic frames input to the audio encoder at the corresponding time step. The unsupervised subnetwork also includes a an augmented branch configured to: perform augmentation on the sequence of acoustic frames extracted from the unlabeled audio samples to generate a sequence of augmented acoustic frames; generate, at each of the plurality of time steps, a higher order feature representation for a corresponding augmented acoustic frame in the sequence of augmented acoustic frames as output from the audio encoder; and generate, at each of the plurality of time steps, using the higher order feature representation output from the audio encoder at the corresponding time step, a prediction of the target branch output generated by the target branch at the corresponding time step. The unsupervised subnetwork is configured to determine, at each of the plurality of times steps, an unsupervised loss term based on the target branch output generated by the target branch at the corresponding time step and the prediction of the target branch output generated by the augmented branch at the corresponding time step. Here, the unsupervised subnetwork is also configured to update parameters of the audio encoder based on the unsupervised loss term determined at each of the plurality of time steps.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the unsupervised loss term includes a contrastive loss term. The augmentation performed on the sequence of acoustic frames may include time modification and masking. In some examples, the target branch is further configured to generate, at each of the plurality of time steps, a higher order feature representation for the corresponding acoustic frame in the sequence of acoustic frames input to the audio encoder at the corresponding time step as output from the audio encoder. Here, the target branch is configured to generate the target branch output for the corresponding acoustic frame by modifying time characteristics of the higher order feature representation. In these examples, modifying the time characteristics of the higher order feature representation may include modifying, at each of the plurality of time steps, the time characteristics of the higher order feature representation generated as output from the audio encoder for the corresponding acoustic frame to match time characteristics associated with the higher order feature representation generated as output from the audio encoder for the corresponding augmented acoustic frame at the corresponding time step.

In some implementations, the augmented branch includes a prediction network of transformer layers configured to, at each of the plurality of time steps, receive the higher order feature representation output from the audio encoder at the corresponding time step as input and generate the prediction of the target branch output generated by the target branch at the corresponding time step as output. In some examples, the contrastive Siamese network includes a supervised subnetwork trained on a plurality of labeled audio samples that correspond to spoken utterances paired with corresponding transcriptions. In these examples, at each of a plurality of output steps for each labeled audio sample, the supervised subnetwork is configured to generate a corresponding speech recognition result for the labeled audio sample using the speech recognition model and determine a supervised loss term based on the corresponding speech recognition result for the labeled audio sample and the corresponding transcription of the labeled audio sample. Here, the supervised subnetwork updates parameters of the speech recognition model based on the supervised loss term determined at each of the plurality of output steps for each labeled audio sample in the plurality of labeled audio samples.

The corresponding speech recognition result generated for the labeled audio sample using the speech recognition model may include a probability distribution over possible speech recognition hypotheses for the labeled audio sample at the corresponding output step. In some examples, the supervised subnetwork is configured to update the parameters of the speech recognition model based on the supervised loss term independently of the unsupervised network updating the parameters of the audio encoder of the speech recognition model. In other examples, the supervised subnetwork is further configured to apply data augmentation to at least one of the labeled audio samples in the plurality of labeled audio samples input to the speech recognition model. In these other examples, the applied data augmentation includes at least one of adding noise, adding reverberation, or manipulating timing.

In some implementations, the trained speech recognition model includes a Transformer-Transducer (T-T) model that includes the audio encoder configured to receive a sequence of acoustic frames extracted from audio data characterizing a spoken utterance as input and generate a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of time steps. In these implementations, the T-T model also includes: a label encoder configured to receive a sequence of non-blank symbols output by a final softmax layer as input and generate a dense representation at each of the plurality of time steps; and a joint network configured to receive, as input, the higher order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the label encoder at each of the plurality of time steps and generate, at each of the plurality of time steps, a probability distribution over possible speech recognition hypothesis at the corresponding time step. Here, the audio encoder includes a neural network having a stack of strided convolutional layers and transformer layers.

Another aspect of the disclosure provides a a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations for training a speech recognition model using a contrastive Siamese network. The operations include receiving a plurality of unlabeled audio samples corresponding to spoken utterances not paired with corresponding transcriptions. At a target branch of a the Contrastive Siamese network, the operations include generating a sequence of encoder outputs for the plurality of unlabeled audio samples using an audio encoder of the speech recognition model and modifying time characteristics of the encoder outputs to generate a sequence of target branch outputs. At an augmentation branch of the contrastive Siamese network, the operations include performing augmentation on the unlabeled audio samples, generating a sequence of augmented encoder outputs for the augmented unlabeled audio samples using the audio encoder of the speech recognition model, and generating predictions of the sequence of target branch outputs generated at the target branch using a prediction network configured to receive the sequence of augmented encoder outputs. The operations also include determining an unsupervised loss term based on the target branch outputs generated at the target branch and the predictions of the target sequence of branch outputs generated at the augmentation branch. The operations also include updating parameters of the audio encoder based on the unsupervised loss.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the unsupervised loss term includes a contrastive loss term. Performing augmentation on the unlabeled audio samples may includes performing time modification and masking on the unlabeled audio samples. In some examples, the operations further include generating a higher order feature representation for the plurality of unlabeled audio samples as output from the audio encoder. In these examples, modifying the time characteristics of the encoder outputs to generate the sequence of target branch outputs includes modifying the time characteristics of the encoder outputs generated as output from the audio encoder to match time characteristics associated with the sequence of augmented encoder outputs from the audio encoder. In some implementations, the operations further include receiving the sequence of augmented encoder outputs as input to a prediction network of transformer layers of the augmented branch and generate, as output from the prediction network of transformer layers of the augmented branch, the predictions of the sequence of target branch outputs generated at the target branch.

In some examples, the operations further include receiving a plurality of labeled audio samples corresponding to spoken utterances paired with corresponding transcriptions, generating speech recognition results for the labeled audio samples using the speech recognition model, determining a supervised loss term based on the speech results for the labeled audio samples and the corresponding transcriptions of the labeled audio samples, and updating parameters of the speech recognition model based on the supervised loss term. In these examples, the operations may further include updating parameters of the speech recognition model based on the supervised loss term independently of updating parameters of the audio encoder based on the unsupervised loss term. Optionally, the operation further include applying data augmentation to at least one of the labeled audio samples. Here, applying data augmentations may include at least one of adding noise, adding reverberation, or manipulating timing.

In some implementations, the speech recognition model includes a Transformer-Transducer (T-T) model and the operations further include: receiving a plurality of unlabeled audio samples corresponding to spoken utterances not paired with corresponding transcriptions as input to the audio encoder of the T-T model; generating, by the audio encoder, a sequence of acoustic frames extracted from audio data characterizing a spoken utterance at each of a plurality of time steps; receiving a sequence of non-blank symbols output by a final softmax layer as input to a label encoder of the T-T model; and generating, by the label encoder, a dense representation at each of the plurality of time steps. In these implementations, the operations also include receiving, as input to a joint network of the T-T model, the higher order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the label encoder at each of the plurality of time steps and generating, by the joint network, at each of the plurality of time steps, a probability distribution over possible speech recognition hypothesis at the corresponding time step. Here, the audio encoder includes a neural network having a stack of strided convolutional layers and transformer layers.

Automatic speech recognition (ASR) systems are often trained using a supervised training technique that leverages labeled training data. The labeled training data includes speech audio data and corresponding transcriptions of the speech. Collecting large quantities of labeled training data is often difficult because of the associated costs, time required to collect the training data, and privacy concerns of users. In some instances, ASR systems train using unlabeled training data that includes only the speech audio data without any corresponding transcriptions. In these instances, the ASR may utilize only the unlabeled training data to train speech recognition systems (i.e., selfsupervised training) or the unlabeled training data may be used in addition to the labeled training data to train the speech recognition systems (i.e., semi-supervised training).

Implementations herein are directed towards a contrastive Siamese network that uses a semi-supervised training technique for training a speech recognition model. The Siamese network includes a supervised subnetwork that trains the speech recognition model with labeled audio samples that include utterances and corresponding transcriptions. That is, the supervised subnetwork receives acoustic frames extracted from the labeled audio samples and predicts speech recognition results. Thereafter, the supervised subnetwork determines a loss by comparing the predicted speech recognition result and the corresponding transcription and updates the speech recognition model based on the loss.

The Siamese network also includes an unsupervised subnetwork that trains the speech recognition with unlabeled audio samples. The unsupervised subnetwork may train the speech recognition model additionally or alternatively to the supervised subnetwork. The unsupervised subnetwork includes a target branch that receives acoustic frames from the unlabeled audio samples and generates a target branch output for each acoustic frame. The unsupervised subnetwork also includes an augmented branch that performs augmentation on the acoustic frames from the unlabeled audio samples and generates a higher order feature representation (i.e., an "encoder output") using the augmented acoustic frames. Accordingly, the augmented branch uses the higher order feature representation to predict the target branch output generated by the target branch. Using the target branch output from the target branch and the prediction of the target branch output by the augmented branch, the unsupervised subnetwork determines an unsupervised loss term and updates parameters of an audio encoder of the speech recognition model based on the unsupervised loss term.

<FIG> is an example of a speech environment <NUM>. In the speech environment <NUM>, a user's <NUM> manner of interacting with a computing device, such as a user device <NUM>, may be through voice input. The user device <NUM> is configured to capture sounds (e.g., streaming audio data) from one or more users <NUM> within the speech environment <NUM>. Here, the streaming audio data may refer to a spoken utterance <NUM> by the user <NUM> that functions as an audible query, a command for the user device <NUM>, or an audible communication captured by the user device <NUM>. Speech-enabled systems of the user device <NUM> may field the query or the command by answering the query and/or causing the command to be performed/fulfilled by one or more downstream applications.

The user device <NUM> may correspond to any computing device associated with a user <NUM> and capable of receiving audio data. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and stores instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations. The user device <NUM> further includes an audio system <NUM> with an audio capture device (e.g., microphone) <NUM>, 16a for capturing and converting spoken utterances <NUM> within the speech environment <NUM> into electrical signals and a speech output device (e.g., a speaker) <NUM>, 16b for communicating an audible audio signal (e.g., as output audio data from the user device <NUM>). While the user device <NUM> implements a single audio capture device 16a in the example shown, the user device <NUM> may implement an array of audio capture devices 16a without departing from the scope of the present disclosure, whereby one or more capture devices 16a in the array may not physically reside on the user device <NUM>, but be in communication with the audio system <NUM>.

In the speech environment <NUM>, an automated speech recognition (ASR) system <NUM> implementing a speech recognition model <NUM> resides on the user device <NUM> of the user <NUM> and/or on a remote computing device <NUM> (e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device <NUM> via a network <NUM>. The user device <NUM> and/or the remote computing device (i.e., remote server) <NUM> also includes an audio subsystem <NUM> configured to receive the utterance <NUM> spoken by the user <NUM> and captured by the audio capture device 16a, and convert the utterance <NUM> into a corresponding digital format associated with input acoustic frames <NUM> capable of being processed by the ASR system <NUM>. In the example shown, the user speaks a respective utterance <NUM> and the audio subsystem <NUM> converts the utterance <NUM> into corresponding audio data (e.g., acoustic frames) <NUM> for input to the ASR system <NUM>. Thereafter, the speech recognition model <NUM> receives, as input, the audio data <NUM> corresponding to the utterance <NUM>, and generates/predicts, as output, a corresponding transcription <NUM> (e.g., speech recognition result/hypothesis) of the utterance <NUM>. As described in greater detail below, the speech recognition model <NUM> may include a Transformer-Transducer (T-T) model <NUM> trained with variable look ahead audio context to allow the T-T model <NUM> to set, during inference, different durations of look ahead audio context when performing speech recognition depending on how sensitive a query specified by the utterance <NUM> is to latency and/or how much tolerance the user <NUM> has for latency. For instance, a digital assistant application <NUM> executing on the user device <NUM> may require the speech recognition to be streaming such that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken. Additionally, it is also likely that the user <NUM> of the user device <NUM> has a low tolerance for latency when issuing queries for the digital assistant application <NUM> to perform. In such scenarios when minimizing speech recognition latency is preferred, the speech recognition model <NUM> may apply zero or minimal look ahead audio context (also referred to as "right context") to provide streaming transcription capabilities in real-time as the user <NUM> is speaking the utterance <NUM>. On the other hand, when the user has a higher tolerance for speech recognition latency and/or the utterance <NUM> to be recognized is associated with long-form speech, the same speech recognition model <NUM> may apply a duration of look ahead audio context sufficient to provide an accurate transcription <NUM>, but incur increased latency based on the duration of look ahead audio context. Accordingly, the ASR system <NUM> may implement only a single speech recognition model <NUM> for a multitude of different speech recognition tasks to provide both streaming and non-streaming transcription capabilities without having to leverage separate ASR models on a task-by-task basis.

In some implementations, the speech recognition model <NUM> performs both streaming speech recognition and non-streaming speech recognition on the audio data <NUM> in parallel. For instance, in the example shown, the speech recognition model <NUM> performs, in parallel, streaming speech recognition on the audio data <NUM> to produce partial speech recognition results <NUM>, 120a, and non-streaming speech recognition on the same audio data <NUM> to produce a final speech recognition result <NUM>, 120b. Notably, the speech recognition model <NUM> may use a first look ahead audio context that may be set to zero (or about <NUM> milliseconds) to produce the partial speech recognition results 120a and use a second look ahead audio context of a longer duration than the first look ahead audio context to produce the final speech recognition result 120b. Thus, the final speech recognition result 120b for the input utterance <NUM> may be delayed from the partial speech recognition results 120a for the input utterance by a duration based on a difference between the second look ahead audio context and the first look ahead audio context.

The user device <NUM> and/or the remote computing device <NUM> also executes a user interface generator <NUM> configured to present a representation of the transcription <NUM> of the utterance <NUM> to the user <NUM> of the user device <NUM>. As described in greater detail below, the user interface generator <NUM> may display the partial speech recognition results 120a in a streaming fashion during time <NUM> and subsequently display the final speech recognition result 120b during time <NUM>. In some configurations, the transcription <NUM> output from the ASR system <NUM> is processed, e.g., by a natural language understanding (NLU) module executing on the user device <NUM> or the remote computing device <NUM>, to execute a user command/query specified by the utterance <NUM>. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device <NUM> or the remote computing device <NUM>) may convert the transcription into synthesized speech for audible output by the user device <NUM> and/or another device.

In the example shown, the user <NUM> interacts with a program or application <NUM> (e.g., the digital assistant application <NUM>) of the user device <NUM> that uses the ASR system <NUM>. For instance, <FIG> depicts the user <NUM> communicating with the digital assistant application <NUM> and the digital assistant application <NUM> displaying a digital assistant interface <NUM> on a screen of the user device <NUM> to depict a conversation between the user <NUM> and the digital assistant application <NUM>. In this example, the user <NUM> asks the digital assistant application <NUM>, "What time is the concert tonight?" This question from the user <NUM> is a spoken utterance <NUM> captured by the audio capture device 16a and processed by audio systems <NUM> of the user device <NUM>. In this example, the audio system <NUM> receives the spoken utterance <NUM> and converts it into acoustic frames <NUM> for input to the ASR system <NUM>.

Continuing with the example, the speech recognition model <NUM>, while receiving the acoustic frames (i.e., audio data) <NUM> corresponding to the utterance <NUM> as the user <NUM> speaks, encodes the acoustic frames <NUM> using a first look ahead audio context and then decodes the encoded acoustic frames <NUM> using the first look ahead audio context into the partial speech recognition results 120a. During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the partial speech recognition results 120a of the utterance <NUM> to the user <NUM> of the user device <NUM> in a streaming fashion such that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken. In some examples, the first look ahead audio context is equal to zero.

In parallel, and after all of the acoustic frames <NUM> corresponding to the utterance <NUM> are received, the speech recognition model <NUM> encodes all of the acoustic frames <NUM> corresponding to the utterance <NUM> using a second look ahead audio context and then decodes the acoustic frames <NUM> using the second look ahead audio context into a final speech recognition result 120b. The duration of the second look ahead audio context may be <NUM> seconds, <NUM> seconds, or any other duration. In some examples, an indication such as an endpoint indicating that the user <NUM> has finished speaking the utterance <NUM> triggers the speech recognition model <NUM> to encode all the acoustic frames <NUM> using the second look ahead audio context. During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the final speech recognition result 120b of the utterance <NUM> to the user <NUM> of the user device <NUM>. In some implementations, the user interface generator <NUM> replaces the representation of the partial speech recognition results 120a with the representation of the final speech recognition result 120b. For instance, as the final speech recognition result 120b is presumed to be more accurate than the partial speech recognition results 120a produced without leveraging look ahead audio context, the final speech recognition result 120b ultimately displayed as the transcription <NUM> may fix any terms that may have been misrecognized in the partial speech recognition results 120a. In this example, the streaming partial speech recognition results 120a output by the speech recognition model <NUM> and displayed on the screen of the user device <NUM> at time <NUM> are associated with low latency and provide responsiveness to the user <NUM> that his/her query is being processed, while the final speech recognition result 120b output by the speech recognition model <NUM> and displayed on the screen at time <NUM> leverages look ahead audio context to improve the speech recognition quality in terms of accuracy, but at increased latency. However, since the partial speech recognition results 120a are displayed as the user speaks the utterance <NUM>, the higher latency associated with producing, and ultimately displaying the final recognition result is not noticeable to the user <NUM>.

In the example shown in <FIG>, the digital assistant application <NUM> may respond to the question posed by the user <NUM> using natural language processing. Natural language processing generally refers to a process of interpreting written language (e.g., the partial speech recognition results 120a and/or the final speech recognition result 120b) and determining whether the written language prompts any action. In this example, the digital assistant application <NUM> uses natural language processing to recognize that the question from the user <NUM> regards the user's schedule and more particularly a concert on the user's schedule. By recognizing these details with natural language processing, the automated assistant returns a response <NUM> to the user's query where the response <NUM> states, "Venue doors open at <NUM>:<NUM> PM and concert starts at 8pm. " In some configurations, natural language processing occurs on a remote server <NUM> in communication with the data processing hardware <NUM> of the user device <NUM>.

With reference to <FIG>, the speech recognition model <NUM> may provide an end-to-end (E2E) speech recognition by integrating acoustic, pronunciation, and language models into a single neural network, and does not require a lexicon or a separate text normalization component. Various structures and optimization mechanisms can provide increased accuracy and reduced model training time. In some implementations, the speech recognition model <NUM> includes a Transformer-Transducer (T-T) model architecture, which adheres to latency constraints associated with interactive applications. The T-T model <NUM> may include the T-T model <NUM> described in <CIT>. The T-T model <NUM> provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the T-T model architecture suitable for performing speech recognition entirely on the user device <NUM> (e.g., no communication with a remote server <NUM> is required). The T-T model <NUM> includes an audio encoder <NUM>, a label encoder <NUM>, and a joint network <NUM>. The audio encoder <NUM>, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a neural network having a stack of strided convolutional layers <NUM> (<FIG> and <FIG>) and transformer layers <NUM> (<FIG> and <FIG>). For instance, the audio encoder <NUM> reads a sequence of d-dimensional feature vectors (e.g., acoustic frames <NUM> (<FIG>)) x = (x<NUM>, x<NUM>, · · ·, xT), where <MAT>, and produces at each time step a higher-order feature representation (also referred to as an "encoder output"). This higher-order feature representation is denoted as ah<NUM>,. Each transformer layer <NUM> of the audio encoder <NUM> may include a normalization layer, a masked multi-head attention layer with relative position encoding, residual connections, a stacking/unstacking layer, and a feedforward layer. Similarly, the label encoder <NUM> may also include a neural network of transformer layers or a look-up table embedding model, which, like a language model (LM), processes the sequence of non-blank symbols output by a final Softmax layer <NUM> so far, y<NUM>,. , yui-<NUM>, into a dense representation Ihu that encodes predicted label history. In implementations when the label encoder <NUM> includes the neural network of transformer layers, each transformer layer may include a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a feed forward layer, and a dropout layer. In these implementations, the label encoder <NUM> may include two transformer layers. In implementations when the label encoder <NUM> includes the look-up table embedding model with a bi-gram label context, the embedding model is configured to learn a weight vector of the d-dimension for each possible bigram label context, where d is the dimension of the outputs of the audio and label encoders <NUM>, <NUM>. In some examples, the total number of parameters in the embedding model is N<NUM> × d where N is the vocabulary size for the labels. Here, the learned weight vector is then used as the embedding of the bigram label context in the T-T model <NUM> to produce fast label encoder <NUM> runtimes.

Finally, with the T-T model architecture, the representations produced by the audio and label encoders <NUM>, <NUM> are combined by the joint network <NUM> using a dense layer Ju,t. The joint network <NUM> then predicts P(zu,t |x, t, y<NUM>,. , yu-<NUM>), which is a distribution over the next output symbol. Stated differently, the joint network <NUM> generates, at each output step (e.g., time step), a probability distribution <NUM> over possible speech recognition hypotheses. Here, the "possible speech recognition hypotheses" correspond to a set of output labels (also referred to as "speech units") each representing a grapheme (e.g., symbol/character) or a word piece in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (<NUM>) symbols, e.g., one label for each of the <NUM>-letters in the English alphabet and one label designating a space. Accordingly, the joint network <NUM> may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector (e.g., a one-hot vector) and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution of the joint network <NUM> can include a posterior probability value for each of the different output labels. Thus, if there are <NUM> different output labels representing different graphemes or other symbols, the output zu,t of the joint network <NUM> can include <NUM> different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer <NUM>) for determining the transcription <NUM>.

The Softmax layer <NUM> may employ any technique to select the output label/symbol with the highest probability in the distribution as the next output symbol predicted by the T-T model <NUM> at the corresponding output step. In this manner, the T-T model <NUM> does not make a conditional independence assumption, rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. While the speech recognition model <NUM> is described as having the T-T model architecture, the speech recognition model <NUM> may include other types of transducer-based architectures, such as a Conformer-Transducer (C-T) model architecture or a Recurrent Neural Network-Transducer (RNN-T) model architecture.

<FIG> and <FIG> illustrate a schematic view of a contrastive Siamese network <NUM> executing a semi-supervised training process for training the speech recognition model <NUM> (<FIG>). The contrastive Siamese network includes a supervised subnetwork training process <NUM> (<FIG>) and an unsupervised subnetwork training process <NUM> (<FIG>). The supervised subnetwork training process (i.e., supervised subnetwork) <NUM> trains the speech recognition model <NUM> using a plurality of labeled audio samples <NUM> that includes a sequence of acoustic frames <NUM> extracted from spoken utterances <NUM> paired with corresponding transcriptions (i.e., labels) <NUM>. The unsupervised subnetwork training process (i.e., unsupervised subnetwork) <NUM> trains the speech recognition model <NUM> using a plurality of unlabeled audio samples <NUM> that includes a sequence of acoustic frames <NUM> extracted from spoken utterances <NUM> without any paired transcriptions.

In some examples, the acoustic frames <NUM> used by the supervised subnetwork (i.e., supervised part) <NUM> are the same as the acoustic frames <NUM> used by the unsupervised subnetwork (i.e., unsupervised part) <NUM>. That is, the supervised part <NUM> and the unsupervised part <NUM> may train the speech recognition model <NUM> using the same acoustic frames <NUM>, <NUM> concurrently. In other examples, the acoustic frames <NUM> used to train the supervised part <NUM> are different from the acoustic frames <NUM> used to train the unsupervised part <NUM>. This scenario is especially beneficial since the unlabeled audio samples <NUM> without any corresponding transcriptions are easy to obtain and can be leveraged to train the speech recognition model <NUM>. As such, the speech recognition model <NUM> may be trained on any combination of labeled audio samples <NUM> and/or unlabeled audio samples <NUM>. In some examples, the sequence of acoustic frames <NUM>, <NUM> extracted from the unlabeled audio samples <NUM> and labeled audio samples <NUM> include log Mel-filterbank energies. A greater number acoustic frames <NUM> may be used to train the unsupervised part <NUM> than the number of acoustic frames <NUM> used to train the supervised part <NUM>. Optionally, a greater number of acoustic frames <NUM> may be used to train the supervised part <NUM> than the number acoustic frames <NUM> used to train the unsupervised part <NUM>. In some examples, the number of acoustic frames <NUM> used to train the supervised part <NUM> and the number of acoustic frames <NUM> used to train the unsupervised part <NUM> are the same.

The supervised part <NUM> and the unsupervised part <NUM> share the same audio encoder <NUM> that includes a stack of strided convolutional layers <NUM> and transformer layers <NUM> that are trained together using a same Adam optimizer and a same learning rate. When the speech recognition model <NUM> corresponds to a Conformer-Transducer model architecture, the audio encoder <NUM> may include conformer layers in lieu of transformer layers.

Referring now to <FIG>, the supervised part <NUM> of the contrastive siamese network <NUM> trains the speech recognition model <NUM> using the plurality of labeled audio samples <NUM>. The plurality of labeled audio samples <NUM> include the sequence of acoustic frames <NUM> extracted from the labeled audio samples <NUM> and the corresponding labels/transcriptions <NUM>. The supervised part <NUM> shares the same audio encoder <NUM> from the speech recognition model <NUM> as the unsupervised part <NUM> in addition to the label encoder <NUM> and joint network <NUM> (not shown in <FIG>).

In some implementations, the supervised part <NUM> includes a data augmentation module <NUM> that applies data augmentation to at least one acoustic frame <NUM> extracted from the labeled audio samples <NUM> to generate a sequence of augmented acoustic frames <NUM>, 306A. The data augmentation module <NUM> of the supervised part <NUM> may be the same (or different) data augmentation module <NUM> (<FIG>) of the unsupervised part <NUM>. In some examples, the data augmentation module <NUM> of the supervised part <NUM> applies different data augmentation techniques than the data augmentation module of the unsupervised part. Applying data augmentation to the acoustic frames <NUM> furthers the acoustic diversity of the audio frames used to train the speech recognition model <NUM>. In some examples, the data augmentation module <NUM> includes a time modifying component that manipulates timing of the sequence of acoustic frames <NUM>. The data augmentation module <NUM> may additionally or alternatively include a time masking component that masks portions of the acoustic frames <NUM>. Other techniques applied by the data augmentation module <NUM> may include adding/injecting noise and/or adding reverberation of the labeled audio samples <NUM>. One data augmentation technique includes using multistyle training (MTR) to inject a variety of environmental noises to the labeled audio samples <NUM>. Another data augmentation technique that the data augmentation module <NUM> may apply in addition to, or in lieu of, MTR, includes using spectrum augmentation (SpecAugment) to make the acoustics of the labeled audio samples <NUM> closer to the adverse acoustics of other labeled audio samples <NUM>. In combination, MTR and SpecAugment may inject noises into the labeled audio samples <NUM>, tile random external noise sources along time and inserted before and overlapped onto the representation, and filtering the noise-injective labeled audio samples prior to training the speech recognition model <NUM>.

The audio encoder <NUM> of the supervised part <NUM> receives the augmented sequence of acoustic frames 306A and generates an encoder output <NUM> for each augmented acoustic frame 306A. The encoder output <NUM> may include a probability distribution of possible speech recognition hypotheses. In particular, the strided convolutional layers <NUM> receive an augmented acoustic frame 306A and generate a corresponding output <NUM>. The transformer layers <NUM> receive the output <NUM> from the strided convolutional layers <NUM> and generate the encoder output <NUM>.

The label encoder <NUM> is a streaming transformer that does not attend to future labels <NUM>. Accordingly, the label encoder <NUM> receives a label <NUM> corresponding to the augmented acoustic frame 306A received by the audio encoder <NUM> and generates a linguistic embedding <NUM> (i.e., dense representation Ihu(<FIG>)). The supervised part <NUM> includes dense layers <NUM> that process the linguistic embedding <NUM> from the label encoder <NUM> and the encoder output <NUM> (i.e., acoustic embedding) from the audio encoder <NUM> to produce a corresponding speech recognition result <NUM> for each acoustic frame <NUM> (e.g., augmented acoustic frame 306A) input to the speech recognition model <NUM> at the corresponding time step. The dense layers <NUM> include a trainable bias vector <NUM> that performs a linear operation on the encoder output <NUM> and the linguistic embedding <NUM> to generate the speech recognition result <NUM>. A loss module <NUM> of the supervised part <NUM> determines a supervised loss term <NUM> based on the outputs from the dense layers <NUM> for the resulting speech recognition result <NUM>. That is, the loss module <NUM> compares the speech recognition result <NUM> to the label (e.g., ground truth transcription) <NUM> to generate the supervised loss <NUM>. The supervised loss term (e.g., RNN-T loss) <NUM> may be represented by: <MAT>.

In Equation <NUM>, rt represents a logit vector that specifies the probability of graphemes including the blank symbol, at represents the encoder output <NUM> from the audio encoder <NUM>, lt represents linguistic embeddings <NUM> from the label encoder <NUM>, and linear represents the conventional dense layers <NUM> with the trainable bias vector <NUM>.

The supervised part <NUM> updates parameters of the speech recognition model <NUM> based on the supervised loss term <NUM> determined at each of the plurality of output steps for each labeled audio sample <NUM> in the plurality of labeled audio samples <NUM>. In some implementations, the supervised part <NUM> is configured to update the parameters of the speech recognition model <NUM> based on the supervised loss term <NUM> independently of the unsupervised part <NUM> updating the parameters of the audio encoder <NUM> of the speech recognition model <NUM>.

Referring now to <FIG>, the unsupervised part <NUM> trains the speech recognition model <NUM> using a plurality of unlabeled audio samples <NUM> that includes a sequence of acoustic frames <NUM> extracted from spoken utterances <NUM> that are not paired with any transcriptions. As shown in the examples, the unsupervised part <NUM> of the contrastive Siamese network <NUM> includes a target branch <NUM> and an augmented branch <NUM> that share the same audio encoder <NUM> of the speech recognition model <NUM> (<FIG>) that includes the stack of strided convolutional layers <NUM> and the transformer layer <NUM>. The unsupervised part <NUM> is configured to extract linguistic information by matching output sequences (i.e., sequences of encoder outputs <NUM>, <NUM>) of the transformer audio encoders <NUM> from the target branch <NUM> and the augmented branch <NUM>.

The target branch <NUM> is configured to generate a target branch output <NUM> based on the sequence of acoustic frames <NUM> extracted from the unlabeled audio samples <NUM>. The audio encoder <NUM> of the target branch <NUM> receives the sequence of acoustic frames <NUM> and generates, at each time step, a higher order feature representation <NUM>. In particular, the strided convolutional layers <NUM> receive an acoustic frame <NUM> from the sequence of acoustic frames <NUM> and generate an output <NUM>. The transformer layers <NUM> receive the output <NUM> from the strided convolutional layers <NUM> and generate the higher order feature representation (i.e., encoder output) <NUM> for the corresponding acoustic frame <NUM>.

The transformer audio encoders <NUM> included in the target branch <NUM> and augmented branch <NUM> benefit from positional embeddings to capture temporal dynamics of the sequence of acoustic frames <NUM>. Accordingly, it is necessary to modify time characteristics of the higher order feature representation <NUM> output by the audio encoder <NUM> at the target branch <NUM> to avoid the audio encoder <NUM> at the unsupervised part <NUM> from generating encoder outputs <NUM>, <NUM> with a low contrastive loss just based on the positional embeddings. Put another way, without modifying time characteristics of the higher order feature representation <NUM> output by the audio encoder <NUM> at the target branch <NUM>, all outputs of the unsupervised part <NUM> of the contrastive siamese network <NUM> will "collapse" to a constant value. Accordingly, the target branch <NUM> applies a stop gradient operation <NUM> that modifies time characteristics of the higher order feature representation <NUM> to generate the target branch output <NUM> for the corresponding acoustic frame <NUM>. In some implementations, the stop gradient operation <NUM> modifies the time characteristics of the higher order feature representation <NUM> output by the audio encoder <NUM> by modifying the time characteristics of the higher order feature representation <NUM> to match time characteristics associated with a corresponding augmented acoustic frame <NUM>, 304A input to the audio encoder <NUM> at the augmented branch <NUM> to generate a corresponding higher order feature representation <NUM>. As will become apparent, the higher order feature representation <NUM> generated by the audio encoder <NUM> at the augmented branch <NUM> corresponds to an augmented higher order feature representation <NUM> (or augmented encoder output) having modified time characteristics based on the augmented acoustic frame 304A input to the audio encoder <NUM>.

The augmentation branch <NUM> of the unsupervised part <NUM> includes a data augmentation module <NUM> that applies data augmentation to each acoustic frame <NUM> extracted from an unlabeled audio sample <NUM>. The augmentation module <NUM> receives the sequence of acoustic frames <NUM> and generate a sequence of augmented acoustic frames 304A. Applying data augmentation to the acoustic frames <NUM> furthers the acoustic diversity of the audio frames used to train the speech recognition model <NUM>. In some examples, the data augmentation module <NUM> includes a time modifying component <NUM> that manipulates timing of the sequence of acoustic frames <NUM>. The data augmentation module <NUM> may additionally or alternatively include a time masking component <NUM> that masks portions of the acoustic frames <NUM>. Other techniques applied by the data augmentation module <NUM> may include adding/injecting noise and/or adding reverberation of the labeled audio samples. One data augmentation technique includes using multistyle training (MTR) to inject a variety of environmental noises to the unlabeled audio samples <NUM>. Another data augmentation technique that the data augmentation module <NUM> may apply in addition to, or in lieu of, MTR, includes using spectrum augmentation (SpecAugment) to make the acoustics of the augmented acoustic frames <NUM> closer to the adverse acoustics of other unlabeled audio samples <NUM>. In combination, MTR and SpecAugment may inject noises into the labeled audio samples <NUM>, tile random external noise sources along time and inserted before and overlapped onto the representation, and filtering the noise-injective unlabeled audio samples prior <NUM> to training the speech recognition model <NUM>.

The audio encoder <NUM> of the augmented branch <NUM> receives the augmented sequence of acoustic frames 304A from the data augmentation module <NUM> and generates the higher order feature representation <NUM> for the corresponding augmented acoustic frame 304A. In particular, the strided convolutional layers <NUM> receive an augmented acoustic frame from the sequence of augmented acoustic frames 304A and generate an output <NUM>. The transformer layers <NUM> receive the output <NUM> from the strided convolutional layers <NUM> and generate the higher order feature representation (i.e., augmented encoder output) <NUM> for the corresponding augmented acoustic frame 304A. Subsequently, a predication transformer network <NUM> receives the higher order feature representation <NUM> and generates a prediction <NUM> of the target branch output <NUM> generated by the target branch <NUM> at the corresponding time step. That is, the prediction transformer <NUM> may use contrastive loss <NUM> to learn to generate the prediction <NUM> as output from the augmented branch <NUM> that matches the target branch output <NUM> at each corresponding time step.

The unsupervised part <NUM> determines an unsupervised loss term <NUM> based on the target branch output <NUM> generated by the target branch <NUM> and the prediction <NUM> of the target branch output <NUM> generated by the augmented branch <NUM>. In some examples, the unsupervised loss term <NUM> includes a contrastive loss term represented by: <MAT>.

In Equation <NUM>, M includes a set of masked frame indices, K includes a set of distractor indices, ht is an encoder output, and ct is a convolutional neural network output. In other examples, the supervised loss term includes a reconstruction loss term L1 or cosine distance loss term. The unsupervised part <NUM> updates parameters of the audio encoder <NUM> based on the unsupervised loss term <NUM> determined at each of the plurality of time steps. Notably, using the audio frames <NUM> the target branch <NUM> generates an expected representation (i.e., target branch output <NUM>) based on the current state of the audio encoder <NUM> and the augmented branch <NUM> aims to match the expectation representation using the augmented audio frames 304A.

<FIG> is a flowchart of an exemplary arrangement of operations for a computer-implemented method <NUM> of training a speech recognition model using a contrastive Siamese network. At operation <NUM>, the method <NUM> includes receiving a plurality of unlabeled audio samples <NUM> corresponding to spoken utterances <NUM> not paired with any corresponding transcriptions (i.e., labels <NUM>). A target branch <NUM> of the contrastive Siamese network <NUM> performs operations <NUM> and <NUM>. At operation <NUM>, the method <NUM> includes generating a sequence of encoder outputs <NUM> for the plurality of unlabeled audio samples <NUM> using an audio encoder <NUM> of a speech recognition model <NUM>. At operation <NUM>, the method <NUM> includes modifying time characteristics of the encoder outputs <NUM> using the stop gradient operation <NUM> to generate a sequence of target branch outputs <NUM>.

An augmentation branch <NUM>, of the contrastive Siamese network <NUM> performs operations <NUM>-<NUM>. At operation <NUM>, the method <NUM> includes performing augmentation on the unlabeled audio samples <NUM> using an augmentation module <NUM>. At operation <NUM>, the method <NUM> includes generating a sequence of augmented encoder outputs <NUM> for the augmented unlabeled audio samples 304A using the audio encoder <NUM> of the speech recognition model <NUM>. At operation <NUM>, the method includes generating predictions <NUM> of the sequence of target branch output <NUM> generated at the target branch <NUM>.

At operation <NUM>, the method includes determining an unsupervised loss term <NUM> based on the target branch outputs <NUM> generated at the target branch <NUM> and the predictions <NUM> of the sequence of target branch outputs <NUM> generated at the augmentation branch <NUM>. At operation <NUM>, the method <NUM> includes updating parameters of the audio encoder <NUM> based on the unsupervised loss <NUM>.

Claim 1:
A contrastive Siamese network (<NUM>) for training a speech recognition model (<NUM>), the contrastive Siamese network comprising an unsupervised subnetwork (<NUM>) trained on a plurality of unlabeled audio samples (<NUM>) corresponding to spoken utterances (<NUM>) not paired with corresponding transcriptions (<NUM>), the unsupervised subnetwork (<NUM>) comprising:
a target branch (<NUM>) configured to:
receive, as input to an audio encoder (<NUM>) of the speech recognition model (<NUM>), a sequence of acoustic frames (<NUM>) extracted from the unlabeled audio samples (<NUM>); and
at each of a plurality of time steps, generate a target branch output (<NUM>) for a corresponding acoustic frame (<NUM>) in the sequence of acoustic frames (<NUM>) input to the audio encoder (<NUM>) at the corresponding time step; and
an augmented branch (<NUM>) configured to:
perform augmentation on the sequence of acoustic frames (<NUM>) extracted from the unlabeled audio samples (<NUM>) to generate a sequence of augmented acoustic frames (304A);
at each of the plurality of time steps, generate, as output from the audio encoder (<NUM>), a higher order feature representation (<NUM>) for a corresponding augmented acoustic frame (304A) in the sequence of augmented acoustic frames (304A); and
at each of the plurality of time steps, generate, using the higher order feature representation (<NUM>) output from the audio encoder (<NUM>) at the corresponding time step, a prediction (<NUM>) of the target branch output (<NUM>) generated by the target branch (<NUM>) at the corresponding time step,
wherein the unsupervised subnetwork (<NUM>) is configured to:
at each of the plurality of time steps, determine an unsupervised loss term (<NUM>) based on the target branch output (<NUM>) generated by the target branch (<NUM>) at the corresponding time step and the prediction (<NUM>) of the target branch (<NUM>) output generated by the augmented branch (<NUM>) at the corresponding time step; and
update parameters of the audio encoder (<NUM>) based on the unsupervised loss term (<NUM>) determined at each of the plurality of time steps.