Patent Description:
Automated speech recognition (ASR) systems have evolved from multiple models where each model had a dedicated purpose to integrated models where a single neural network is used to directly map an audio waveform (i.e., input sequence) to an output sentence (i.e., output sequence). This integration has resulted in a sequence-to-sequence approach, which generates a sequence of words (or graphemes) when given a sequence of audio features. With an integrated structure, all components of a model may be trained jointly as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is constructed entirely of a neural network. A fully neural network functions without external and/or manually designed components (e.g., finite state transducers, a lexicon, or text normalization modules). Additionally, when training E2E models, these models generally do not require bootstrapping from decision trees or time alignments from a separate system. These E2E automatic speech recognition (ASR) systems have made tremendous progress, surpassing conventional ASR systems in several common benchmarks including word error rates (WER). The architecture of E2E ASR models are largely application dependent. For instance, a number of applications that involve user interaction, such as voice-search or on-device dictation, require the model to perform recognition in a streaming fashion. Other applications, like offline video captioning, do not require the model to be streaming and can make use of future context to improve performance. Existing E2E approaches typically include separate ASR models for streaming and non-streaming applications. Managing multiple models for different applications provides unique challenges and can lead to slower execution times of the ASR systems.

Exemplary prior art systems are provided in e.g. <CIT>) and <NPL>.

One aspect of the present invention provides an automatic speech recognition (ASR) model in accordance with claim <NUM>.

A computer-implemented method in accordance with the invention is provided in claim <NUM>.

Further preferred embodiments are provided in the dependent claims.

End-to-end (E2E) automatic speech recognition (ASR) models are traditionally structured to operate in either a streaming mode or a non-streaming mode. Conventionally, an E2E ASR model includes an encoder and a decoder as the main components. Applications that involve end-user interaction, like voice-search or on-device dictation, may require the model to perform recognition in a streaming fashion, where the words are expected to be output as they are spoken with as little latency as possible. This prevents the use of models that use future context to improve accuracy, such as bi-directional LSTMs. By contrast, applications such as offline video captioning do not require streaming recognition and may make full use of any available future context to improve performance.

Implementations herein are directed toward a single E2E ASR model that uses cascaded encoders that can operate in both streaming and non-streaming modes. The cascaded encoders include a streaming encoder and a non-streaming encoder, and a single decoder of the ASR model is configured to learn to decode either the output from the streaming encoder or the output from the non-streaming encoder. In addition to ASR models, the architecture can apply to other models such as machine translation that implement both streaming and non-streaming modes.

<FIG> and <FIG> are examples of a speech environment <NUM>, 100a-b. In the speech environment <NUM>, a user's <NUM> manner of interacting with a computing device, such as a user device <NUM>, may be through voice input. The user device <NUM> (also referred to generally as a device <NUM>) is configured to capture sounds (e.g., streaming audio data) from one or more users <NUM> within the speech environment <NUM>. Here, the streaming audio data may refer to a spoken utterance <NUM> by the user <NUM> that functions as an audible query, a command for the device <NUM>, or an audible communication captured by the device <NUM>. Speech-enabled systems of the device <NUM> may field the query or the command by answering the query and/or causing the command to be performed/fulfilled by one or more downstream applications.

The user device <NUM> may correspond to any computing device associated with a user <NUM> and capable of receiving audio data. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and stores instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations. The user device <NUM> further includes an audio system <NUM> with an audio capture device (e.g., microphone) <NUM>, 16a for capturing and converting spoken utterances <NUM> within the speech environment <NUM> into electrical signals and a speech output device (e.g., a speaker) <NUM>, 16b for communicating an audible audio signal (e.g., as output audio data from the device <NUM>). While the user device <NUM> implements a single audio capture device 16a in the example shown, the user device <NUM> may implement an array of audio capture devices 16a without departing from the scope of the present disclosure, whereby one or more capture devices 16a in the array may not physically reside on the user device <NUM>, but be in communication with the audio system <NUM>.

In the speech environment <NUM>, an automated speech recognition (ASR) system <NUM> implementing an ASR model <NUM> (also referred to as the model <NUM>) resides on the user device <NUM> of the user <NUM> and/or on a remote computing device <NUM> (e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device <NUM> via a network <NUM>. The user device <NUM> and/or the remote computing device <NUM> also includes an audio subsystem <NUM> configured to receive the utterance <NUM> spoken by the user <NUM> and captured by the audio capture device 16a, and to convert the utterance <NUM> into a corresponding digital format associated with input acoustic frames <NUM> capable of being processed by the ASR system <NUM>. In the example shown in <FIG>, the user <NUM> speaks a respective utterance <NUM> and the audio subsystem <NUM> converts the utterance <NUM> into corresponding audio data (e.g., acoustic frames) <NUM> for input to the ASR system <NUM>. Thereafter, the model <NUM> receives, as input, the audio data <NUM> corresponding to the utterance <NUM>, and generates/predicts, as output, a corresponding transcription <NUM> (also referred to as a recognition result/hypothesis <NUM>) of the utterance <NUM>. As described in greater detail below (e.g., <FIG>), the model <NUM> may be trained in a single training stage to simplify the process of training the model <NUM> to operate in a streaming and a non-streaming mode. The model <NUM> also includes a decoder <NUM> (also referred to as a shared decoder <NUM>) shared between its encoders which enables the model <NUM> to be a single model that can operate in streaming and non-streaming mode (e.g., in contrast with two separate models where each model is dedicated to either a streaming mode or non-streaming mode). For instance, as shown in <FIG>, a digital assistant application <NUM> executing on the user device <NUM> may require the speech recognition to be streaming such that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken. Additionally, it is also likely that the user <NUM> of the user device <NUM> has a low tolerance for latency when issuing queries for the digital assistant application <NUM> to perform. In these scenarios where the application demands minimal latency, the model <NUM> operates in a streaming mode where the model <NUM> may provide streaming transcription capabilities in real-time as the user <NUM> is speaking the utterance <NUM>. On the other hand, when the user <NUM> has a higher tolerance for speech recognition latency and/or the utterance <NUM> to be recognized is associated with long-form speech (i.e., referring to speech consisting of full paragraphs or multiple sentences), the same model <NUM> may operate in a non-streaming mode and may leverage a prediction network to provide an accurate transcription <NUM>, but incur increased latency. Accordingly, the ASR system <NUM> may implement only a single ASR model <NUM> for a multitude of different speech recognition tasks to provide both streaming and non-streaming transcription capabilities without having to leverage separately trained ASR models on a task-by-task basis.

In some implementations, the model <NUM> performs streaming speech recognition on the audio data <NUM> first and then performs non-streaming speech recognition on the output of the streaming encoder. For instance, in the example shown, the model <NUM> performs streaming speech recognition on the audio data <NUM> using a first encoder (i.e., a low latency encoder (<FIG>)) to produce partial speech recognition results <NUM>, 120a (e.g., second probability distribution of speech recognition results 120a), and non-streaming speech recognition on the encoded audio data <NUM> using a second encoder (i.e., a high latency encoder (<FIG>)) to produce a final speech recognition result <NUM>, 120b (e.g., a first probability distribution of speech recognition results 120b). Notably, the model <NUM> outputs the partial speech recognition results 120a as the audio data <NUM> is received and subsequently outputs the final speech recognition result 120b once all the audio data <NUM> is received after the user <NUM> finishes speaking the utterance <NUM>. Thus, output of the final speech recognition result 120b for the input utterance <NUM> may be delayed from the partial speech recognition results 120a.

The user device <NUM> and/or the remote computing device <NUM> also executes a user interface generator <NUM> configured to present a representation of the transcription <NUM> of the utterance <NUM> to the user <NUM> of the user device <NUM>. As described in greater detail below, the user interface generator <NUM> may display the partial speech recognition results 120a in a streaming fashion during time <NUM> and subsequently display the final speech recognition result 120b during time <NUM>. In some configurations, the transcription <NUM> output from the ASR system <NUM> is processed, e.g., by a natural language understanding (NLU) module executing on the user device <NUM> or the remote computing device <NUM>, to execute a user command/query specified by the utterance <NUM>. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device <NUM> or the remote computing device <NUM>) may convert the transcription <NUM> into synthesized speech for audible output by the user device <NUM> and/or another device.

In the example of <FIG>, the user <NUM> in the speech environment 100a interacts with a program or application <NUM> (e.g., the digital assistant application 50a) of the user device <NUM> that uses the ASR system <NUM>. For instance, <FIG> depicts the user <NUM> communicating with the digital assistant application 50a and the digital assistant application 50a displaying a digital assistant interface <NUM> on a screen of the user device <NUM> to depict a conversation between the user <NUM> and a digital assistant of the digital assistant application 50a. In this example, the user <NUM> asks the digital assistant application 50a, "What song is playing right now?" This question from the user <NUM> is a spoken utterance <NUM> captured by the audio capture device 16a and processed by audio systems <NUM> of the user device <NUM>. In this example, the audio system <NUM> receives the spoken utterance <NUM> and converts it into acoustic frames <NUM> for input to the ASR system <NUM>.

Continuing with the example, the model <NUM>, while receiving the acoustic frames <NUM> corresponding to the utterance <NUM> as the user <NUM> speaks, encodes the acoustic frames <NUM> using a first encoder <NUM> (i.e., <FIG>) and then decodes an encoded representation of the acoustic frames <NUM> using a decoder <NUM> (<FIG>) into the partial speech recognition results 120a. During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the partial speech recognition results 120a of the utterance <NUM> to the user <NUM> of the user device <NUM> in a streaming fashion such that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken.

After all (or some amount) of the acoustic frames <NUM> corresponding to the utterance <NUM> are received, and the first encoder <NUM> has encoded these acoustic frames <NUM>, the second encoder <NUM> (i.e., <FIG>) encodes the encoding output from the first encoder <NUM> to generate an encoding for the set of acoustic frames <NUM> corresponding to the utterance <NUM> already encoded by the first encoder <NUM>. The decoder <NUM> then decodes the acoustic frames <NUM> that have been encoded by the second encoder <NUM> into a final speech recognition result 120b. For example, when the first encoder <NUM> encodes all of the acoustic frames <NUM> corresponding to the utterance <NUM> (e.g., as the acoustic frames <NUM> are received), the second encoder <NUM> encodes all of the acoustic frames <NUM> that have been encoded by the first encoder <NUM>. In this respect, by encoding over multiple encoded acoustic frames <NUM>, the second encoder <NUM> is able to provide greater contextual awareness (e.g., by receiving representations of all of the acoustic frames <NUM> for the utterance <NUM>) in a non-streaming fashion which may potentially reconcile or correct aspect(s) of the utterance <NUM> missed or misinterpreted by the streaming nature of the first encoder <NUM>. In some examples, an indication, such as an endpoint, that identifies that the user <NUM> has finished speaking the utterance <NUM> triggers the second encoder <NUM> of the model <NUM> to encode all the acoustic frames <NUM>. During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the final speech recognition result 120b of the utterance <NUM> to the user <NUM> of the user device <NUM>. In some implementations, the user interface generator <NUM> replaces (or modifies) the representation of the partial speech recognition results 120a with the representation of the final speech recognition result 120b. For instance, the final speech recognition result 120b is presumed to be more accurate than the partial speech recognition results 120a and the final speech recognition result 120b is ultimately displayed as the transcription <NUM> in order to fix any terms that may have been misrecognized in the partial speech recognition results 120a. In this example, the streaming partial speech recognition results 120a output by the model <NUM> and displayed on the screen of the user device <NUM> at time <NUM> are associated with low latency and provide responsiveness to the user <NUM> that his/her query is being processed. Accordingly, the partial speech recognition results 120a may incorrectly predict that the utterance <NUM> of the user <NUM> is "What song is play right now?" The final speech recognition result 120b output by the model <NUM> and displayed on the screen at time <NUM> at increased latency improves the speech recognition quality in terms of accuracy by identifying that the user <NUM> said "playing. " However, since the user interface generator <NUM> displays the partial speech recognition results as the user speaks the utterance <NUM>, the higher latency associated with producing, and ultimately displaying the final recognition result 120b is less noticeable to the user <NUM>.

In the example shown in <FIG>, the digital assistant application 50a may respond to the question posed by the user <NUM> using natural language processing. Natural language processing generally refers to a process of interpreting written language (e.g., the partial speech recognition results 120a and/or the final speech recognition result 120b) and determining whether the written language prompts any action. In this example, the digital assistant application 50a uses natural language processing to recognize that the question from the user <NUM> regards the user's environment and more particularly a song playing in the user's vicinity. By recognizing these details with natural language processing, the automated assistant returns a response <NUM> to the user's query where the response <NUM> states, "Tweezer is playing right now. " In some configurations, natural language processing occurs on the remote computing device <NUM> in communication with the data processing hardware <NUM> of the user device <NUM>.

<FIG> is another example of speech recognition with the ASR system <NUM> of the speech environment 100b. As shown in the example, the user <NUM> interacts with a voicemail application <NUM>, 50b displaying a voicemail application interface <NUM>, 18b on the screen of the user device <NUM> to transcribe a voicemail that was left for the user <NUM> by Jane Doe. In this example, latency is not important. Without a concern for latency, the model <NUM> of the ASR system <NUM> is able to take advantage of the full context of the audio by waiting until all of the acoustic frames <NUM> corresponding to the voicemail are generated. This voicemail scenario also illustrates how the model <NUM> is capable of handling a long-form of speech because a voicemail is often multiple sentences or even several paragraphs. The ability to handle long-form speech is particularly advantageous over other ASR models, such as two-pass models with LAS decoders, because these two pass-models often incur reduced performance (e.g., a higher word deletion rate on long-form speech) when recognizing long-form speech. For instance, by using an RNN-T decoder as the decoder <NUM> in combination with cascading encoders <NUM> (e.g., the first encoder <NUM> and the second encoder <NUM>), the model <NUM> is able to consume the full context of the input audio when performing speech recognition on long-form speech without incurring a reduction performance as in two-pass ASR models.

With continued reference to <FIG>, as discussed with respect to <FIG>, the model <NUM> encodes the acoustic frames <NUM> using the first encoder <NUM> while receiving the acoustic frames <NUM>. After the model <NUM> receives all of the acoustic frames <NUM> and encodes them with the first encoder <NUM>, the model <NUM> provides the first encoder output as input to the second encoder <NUM>. The second encoder <NUM> encodes the first encoder output before the decoder <NUM> generates the final speech recognition result 120b. Thereafter, the user interface generator <NUM> presents, via the digital assistant interface 18b, a representation of the final speech recognition result 120b without first displaying the partial speech recognition results 120a. For example, the final speech recognition result 120b is a transcript of the long-form voicemail from Jane Doe that states "Give me a call back when you get this. Just trying to figure out plans for the New Year.

<FIG> include example models 200a-c operating in various combinations of streaming and non-streaming modes. Specifically, each of the models 200a-c include a cascading encoder <NUM> and a decoder <NUM>. The cascading encoder <NUM> refers to a model structure where the encoding pathway includes two encoders <NUM>, <NUM> that cascade such that the output of one encoder <NUM> feeds the input of the other encoder <NUM> prior to decoding. Here, the encoders <NUM>, <NUM> can be cascaded irrespective of the underlying architecture for each encoder. In some examples, the encoders <NUM>, <NUM> include a stack of <NUM>-dimension conformer layers. Causal convolution and left-context attention layers may be used for each conformer layer to strictly restrict the model to use no future inputs. A multi-headed (e.g., <NUM> heads) attention mechanism may be used in a self-attention layer. The cascaded encoders, <NUM>, <NUM> may include <NUM> conformer layers. Here, the first encoder <NUM> may include <NUM> conformer layers while the second encoder <NUM> may include two conformer layers that take in additional right context (e.g., <NUM> seconds). Optionally, other types of layers incorporating self-attention mechanisms, such as transformer layers, may be used in lieu of conformer layers. The first encoder <NUM> may be referred to as a causal encoder and the second encoder <NUM> may be referred to as a non-causal encoder.

In other implementations, one encoder is constructed with an LSTM structure while the other encoder is constructed using bi-directional LSTM layers or conformer layers (e.g., a conformer-transducer). In other words, the encoders <NUM>, <NUM> may have different architectures or similar architectures. For instance, the cascading encoder <NUM> may be roughly analogous to an acoustic model (AM) in a traditional ASR system, and may include a recurrent network of stacked Long Short-Term Memory (LSTM) layers. Here, the first encoder <NUM> is a streaming encoder that includes unidirectional Long Short Term Memory (LSTM) layers while the second encoder <NUM> is a non-streaming encoder that includes bidirectional LSTM layers or conformer layers. In a cascading encoder <NUM>, where both encoders <NUM>, <NUM> include LSTM layers, the second encoder <NUM> that receives the output of the first encoder <NUM> may take advantage of the LSTM layers of the first encoder <NUM> such that the second encoder <NUM> includes fewer LSTM layers than the first encoder <NUM> (and fewer LSTM layers than a fully non-streaming model). By having fewer LSTM layers, the cascading encoder <NUM> may reduce the number of more computationally expensive bidirectional layers making the model <NUM> more streamlined than simply combining a traditional streaming model with a traditional non-streaming model.

Referring to <FIG>, the first encoder <NUM> reads a sequence of d-dimensional feature vectors (e.g., acoustic frames <NUM> shown in <FIG> and <FIG>) x = (x<NUM>, x<NUM>, · · · , xT), where <MAT>, and produces, at each time step, a first higher-order feature representation <NUM>. This first higher-order feature representation is denoted as es. Similarly, the second encoder <NUM> is connected in cascade to the first encoder <NUM>, and is trained to receive the first higher order feature es as input, and output a second higher order feature representation <NUM>. This second higher order feature representation is denoted as ea. Both the first encoder <NUM> and the second encoder <NUM> are directly connected to, and shared by the decoder <NUM>. Accordingly, the decoder <NUM> receives both the first higher order feature representation es and the second higher order feature representation ea as inputs.

The decoder <NUM> may include a recurrent neural network-transducer (RNN-T) architecture having a joint layer <NUM> and a prediction network <NUM>. The decoder <NUM> uses the joint layer <NUM> to combine (i.e., when the model <NUM> operates in non-streaming mode) the first and second higher order feature representations es, ea, output by the cascading encoder <NUM>, as well as an embedding output from the prediction network <NUM> for the previous prediction yr-<NUM>), in order to produce a decoder output. When the model <NUM> operates in the streaming mode, the joint layer <NUM> receives the output of the prediction network <NUM> and only the first higher order feature representation es output from the first encoder <NUM>. The decoder output can be a probability distribution, P (yi|yi-<NUM>,. , y<NUM>, x), over the current sub-word unit, yi, given the sequence of the N previous non-blank symbols previous units, {yi-<NUM>,. , yi-N}, and input, x. Although not illustrated, the model <NUM> may include a Softmax layer that receives output of the decoder <NUM>. In some implementations, the Softmax layer is separate from the decoder <NUM> and processes the output, yr, from the decoder <NUM>. The output of the Softmax layer is then used in a beam search process to select orthographic elements. In some implementations, the Softmax layer is integrated with the decoder <NUM>, such that the output yr of the decoder <NUM> represents the output of the Softmax layer.

The decoder <NUM> is configured to generate, at each output step, a probability distribution over possible speech recognition hypotheses. Stated differently, the joint network <NUM> generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses. Here, the "possible speech recognition hypotheses" correspond to a set of output labels/symbols (also referred to as "speech units") each representing a grapheme (e.g., symbol/character) or a word piece in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (<NUM>) symbols, e.g., one label for each of the <NUM>-letters in the English alphabet and one label designating a space. Accordingly, the joint network <NUM> may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector (e.g., a one-hot vector) and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output labels could also be other types of speech units, such as phonemes or sub-phonemes. The output distribution of the joint network <NUM> can include a posterior probability value for each of the different output labels. Thus, if there are <NUM> different output labels representing different graphemes or other symbols, the output of the joint network <NUM> can include <NUM> different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer) for determining the transcription <NUM>. In some examples, the first encoder <NUM> of the cascaded encoders model <NUM> is made up of eight <NUM>,<NUM>-dimensional LSTM layers, each followed by a <NUM>-dimensional projection layer. In these examples, the second encoder <NUM> of the model <NUM> may be made up of a two-layer bidirectional LSTM with around <NUM> million parameters.

Within the decoder <NUM>, the prediction network <NUM> may have two <NUM>,<NUM>-dimensional LSTM layers, each of which is also followed by <NUM>-dimensional projection layer, such that the LSTM-based prediction network may have about <NUM> million parameters. In other configurations, the prediction network <NUM> may instead include conformer or transformer layers in lieu of LSTM layers. In yet other configurations, the prediction network <NUM> includes a V2 embedding look up table that includes an embedding prediction network. At each time step, the V2 embedding lookup table may receive, as input, the previous two predictions (e.g., <NUM>-hot vectors) output by the joint network <NUM>, compute a respective embedding d<NUM>, d<NUM> for each of the previous two predictions, and provide a concatenated output [d<NUM>, d<NUM>] to the joint layer <NUM>. Comparatively, the V2 embedding lookup table may have only about two (<NUM>) million parameters, whereas an LSTM-based prediction network may include about <NUM> million parameters. Finally, the joint network <NUM> may also be a one-layer neural network with <NUM> hidden units. The Softmax layer may be composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes in a plurality of training data sets <NUM>, 132a-n (<FIG>). In some implementations, in order to limit the amount of future context that the cascaded encoders model <NUM> sees, the second encoder <NUM> uses some number of conformer layers (e.g., two layers) with a particular amount of right context (e.g., five seconds of right context), while the first encoder <NUM> continues to use LSTM layers. For these implementations, each conformer layer in the second encoder <NUM> may have <NUM> units to match the LSTM layers and adds around <NUM> million additional parameters.

Continuing with the example in <FIG>, in some implementations, the model 200a operates in both the streaming and non-streaming modes in parallel. When operating in both streaming and non-streaming mode at the same time, the model 200a first performs streaming speech recognition on the audio data <NUM> using the first encoder <NUM> to generate the first higher order representation es for both the second encoder <NUM> and the decoder <NUM>. The decoder <NUM> then produces the partial speech recognition results <NUM>, 120a based on the first higher order representation es. The model 200b also performs non-streaming speech recognition on the encoded audio data <NUM> where the second encoder <NUM> uses the first higher order representation es received from the first encoder <NUM> to generate the second higher order representation ea. The decoder <NUM> then produces the final speech recognition result <NUM>, 120b based on the second higher order representation ea. As noted by the time, the decoder <NUM> outputs the partial speech recognition results 120a in a streaming fashion using the output from the first encoder <NUM>, and then waits until the utterance is complete before operating in the non-streaming mode that uses the output from the second encoder <NUM> to produce the final speech recognition result 120b b. Thus, the final speech recognition result 120b for the input utterance <NUM> may be delayed from the partial speech recognition results 120a.

Referring to <FIG>, in some implementations, the model 200b operates only in the streaming mode. This may occur, for instance, when the user <NUM> is using applications such as voice-search or on-device dictation, which require as little latency as possible. Here, the model 200b performs streaming speech recognition on the audio data <NUM> using only the first encoder <NUM> to generate the first higher order representation <NUM> es for the decoder <NUM>. The decoder <NUM> then produces the partial speech recognition results <NUM>, 120a. Because the streaming mode of the model 200b produces the partial speech recognition results <NUM>, 120a quickly, the inaccuracy of the term "play" is generally acceptable to users <NUM>.

Referring to <FIG>, in some implementations the model 200c operates only in the non-streaming mode. The non-streaming mode may occur, for instance, in non-latency intensive applications such as when the user <NUM> is viewing a transcription of a voicemail left on his/her phone (e.g., <FIG>). As discussed above, this type of application benefits from using future context to improve performance in exchange for increased processing times. Here, the model 200c first uses the first encoder <NUM> to generate the first higher order representation <NUM> es at each time step for input to the second encoder <NUM>, but the decoder <NUM> does not decode any of the first higher order representations es. The model 200c then performs non-streaming speech recognition on all of the audio data <NUM> where the second encoder <NUM> uses the first higher order representation es received from the first encoder <NUM> to generate the second higher order representation <NUM> ea. The decoder <NUM> then produces the final speech recognition result <NUM>, 120b. Since producing streaming speech recognition in real-time has little value to the user and latency is not a factor, the model 200c may simply operate in only the non-streaming mode to produce the final speech recognition result <NUM>, 120b.

<FIG> shows an example of a training process <NUM> for training the model <NUM> to be operable for both streaming and/or non-streaming modes. In some configurations, the training process <NUM> executes on the remote computing device <NUM> of <FIG> and <FIG>. The training process <NUM> obtains a plurality of training utterances <NUM>, 132a-n stored in a sample database <NUM> and trains the model <NUM> on the training utterances <NUM>. The sample database <NUM> may reside on the memory hardware of the remote computing device <NUM>. As discussed above with respect to <FIG>, the first encoder <NUM> and the second encoder <NUM> share the same decoder <NUM>, and can be trained in a single stage, simplifying the training process <NUM>. This means that the non-streaming encoder <NUM> may be trained directly on the output of the streaming encoder <NUM> (e.g., the first higher order representation es) instead of on input acoustic features (e.g., input acoustic frames <NUM>).

As shown in <FIG>, there are two processing paths for the model <NUM>, one for the streaming mode of the model 200b (shown in <FIG>) and one for the non-streaming mode of the model 200c (shown in <FIG>). Because there are two input processing paths within training process <NUM>, the model's loss includes two loss functions. Specifically, the loss for the streaming mode of the model 200b is generally defined as a summation of the negative log probabilities corresponding to the probability distribution over possible speech recognition hypotheses given the input training utterances <NUM>. That is, the model loss from the first encoder <NUM> connection to the decoder <NUM> is defined as, <IMG> = - Σ{(x→es,y)}log P(y|es). The model loss for the non-streaming mode is also generally defined as a summation of the negative log probabilities corresponding to the probability distribution over possible speech recognition hypotheses given the input training utterances <NUM>. Therefore, the model loss from the second encoder <NUM> connection to the decoder <NUM> is defined as <IMG> = - Σ{(x→ea,y)} log P(y|ea). Based on these representations, the total loss between the two input paths is computed as a weighted sum of each input path loss <IMG> = λ<IMG> + (<NUM> - λ)<IMG> where λ is the weighting term. In the training process <NUM>, jointly training the cascaded encoders <NUM>, <NUM> includes minimizing the weighted sum of the loss between both input processing paths.

At each step-time during training process <NUM>, for each training utterance <NUM>, training can occur in either streaming or non-streaming. In other words, the input processing path is stochastically chosen as either training the model 200b, or the model 200c. By sampling the training utterances <NUM>, the training process only needs to calculate the loss once for each training utterance <NUM> at each training step, which greatly speeds up the training process <NUM>. In some implementations, where a longer training time is tolerated, an alternative training process is employed to train each input processing path with each training utterance and compute both the loss of the model 200b and the model 200c for each training utterance <NUM> at each training step.

In the example shown, training utterances 132b, 132c are chosen to train the first processing path represented by the cascaded encoders model 200b. The cascaded encoders model 200b receives the training utterances 132b, 132c, and the first encoder <NUM> converts the training utterances 132b, 132c into the first higher order feature representations (e.g., audio embeddings) as output. The decoder <NUM> then receives the first higher order feature representations of training utterances 132b, 132c as input and generates an output which is tested for its accuracy. Similarly, training utterances 132a, 132d are chosen to train the second processing path represented by the cascaded encoders model 200c. The cascaded encoders model 200c receives the training utterances 132a, 132d, and the first encoder converts the training utterances 132a, 132d into the first higher order feature representations (e.g., audio embeddings) as output. The second encoder <NUM> receives the first higher order feature representations of training utterances 132a, 132d as input and generates second higher order feature representations of the training utterances 132a, 132d as output. The decoder <NUM> then receives the second higher order feature representations of training utterances 132a, 132d as input and generates an output which is tested for its accuracy. This ensures that that the model <NUM> learns to operate in either streaming or non-streaming modes during inference.

<FIG> includes a flowchart of an example arrangement of operations for a method <NUM> of performing streaming and non-streaming speech recognition using a cascaded encoders model <NUM>. At operation <NUM>, the method <NUM> includes receiving, as input to a cascaded encoders model <NUM>, a sequence of acoustic frames <NUM>. At operation <NUM>, the method <NUM> further includes performing, using the cascaded encoders model, streaming speech recognition and non-streaming speech recognition on the sequence of acoustic frames <NUM>.

At operation <NUM>, the method <NUM> includes generating, by a first encoder <NUM>, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame <NUM> in the sequence of acoustic frames <NUM>. The method <NUM> further includes, at operation <NUM>, receiving, as input to a second encoder <NUM>, the first higher order feature representation generated by the first encoder <NUM> at each of the plurality of output steps. At operation <NUM>, the method <NUM> also includes generating, by the second encoder <NUM>, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature frame. The method <NUM> also includes, at operation <NUM>, receiving, as input to a decoder <NUM>, the second higher order feature representation generated by the second encoder <NUM> at each of the plurality of output steps. At operation <NUM>, the method <NUM> further includes generating, at each of the plurality of time steps, a first probability distribution over possible speech recognition hypotheses.

<FIG> is schematic view of an example computing device <NUM> that may be used to implement the systems (e.g., the audio subsystem <NUM>, the ASR system <NUM>, the user interface generator <NUM>, and/or the model <NUM>) and methods (e.g., the method <NUM>) described in this document.

The computing device <NUM> includes a processor <NUM> (e.g., data processing hardware), memory <NUM> (e.g., memory hardware), a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

Claim 1:
An automated speech recognition (ASR) model (<NUM>) comprising:
a first encoder (<NUM>) configured to:
receive, as input, a sequence of acoustic frames (<NUM>); and
generate, at each of a plurality of output steps, a first higher order feature representation (<NUM>) for a corresponding acoustic frame (<NUM>) in the sequence of acoustic frames (<NUM>);
a second encoder (<NUM>) configured to:
receive, as input, the first higher order feature representation (<NUM>) generated by the first encoder (<NUM>) at each of the plurality of output steps; and
generate, at each of the plurality of output steps, a second higher order feature representation (<NUM>) for a corresponding first higher order feature frame; and
a decoder (<NUM>) configured to:
receive, as input, the second higher order feature representation (<NUM>) generated by the second encoder (<NUM>) at each of the plurality of output steps;
generate, at each of the plurality of time steps, a first probability distribution over possible speech recognition hypotheses;
receive, as input, the first higher order feature representation (<NUM>) generated by the first encoder (<NUM>) at each of the plurality of output steps;
generate, at each of the plurality of time steps, a second probability distribution over possible speech recognition hypotheses;
wherein the decoder (<NUM>) comprises:
a prediction network (<NUM>) configured to:
receive, as input, a sequence of non-blank symbols output by a final Softmax layer; and
generate, at each of the plurality of output steps, a dense representation; and
a joint network (<NUM>) configured to:
receive, as input, the dense representation generated by the prediction network (<NUM>) at each of the plurality of output steps and one of:
when the ASR model (<NUM>) is operating in a streaming mode, the first higher order feature representation (<NUM>) generated by the first encoder (<NUM>) at each of the plurality of output steps; or
when the ASR model (<NUM>) is operating in a non-streaming mode, the second higher order feature representation (<NUM>) generated by the second encoder (<NUM>) at each of the plurality of output steps; and
generate, at each of the plurality of output steps, one of:
when the ASR model (<NUM>) is operating in the streaming mode, the second probability distribution over possible speech recognition hypotheses; or
when the ASR model (<NUM>) is operating in the non-streaming mode, the first probability distribution over possible speech recognition hypotheses.