Patent Description:
Modern automatic speech recognition (ASR) systems focus on providing not only quality/accuracy (e.g., low word error rates (WERs)), but also low latency (e.g., a short delay between the user speaking and a transcription appearing). Moreover, when using an ASR system today there is a demand that the ASR system decode utterances in a streaming fashion that corresponds to displaying a transcription of an utterance in real-time, or even faster than real-time, as a user speaks. To illustrate, when an ASR system is deployed on a mobile phone that experiences direct user interactivity, an application on the mobile phone using the ASR system may require the speech recognition to be streaming such that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken. Here, it is also likely that the user of the mobile phone has a low tolerance for latency. Due to this low tolerance, the speech recognition strives to run on the mobile device in a manner that minimizes an impact from latency and inaccuracy that may detrimentally affect the user's experience.

Prio Art Document <NPL>" discloses confidence estimation for speech recognition using attention in an encoder/decoder ASR framework.

One aspect of the disclosure provides a computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations that include receiving, from a speech recognizer, a speech recognition result for an utterance spoken by a user. The speech recognition result includes a sequence of hypothesized sub-word units that form one or more words of the utterance. Each sub-word unit is output from the speech recognizer at a corresponding output step. Using a confidence estimation module, for each sub-word unit in the sequence of hypothesized sub-word units, the operations also include: obtaining a respective confidence embedding that represents a set of confidence features associated with the corresponding output step when the corresponding sub-word unit was output from the speech recognizer; generating, using a first attention mechanism that self-attends to the respective confidence embedding for the corresponding sub-word unit and the confidence embeddings obtained for any other sub-word units in the sequence of hypothesized sub-word units that proceed the corresponding sub-word unit, a confidence feature vector; generating, using a second attention mechanism that cross-attends to a sequence of acoustic encodings each associated with a corresponding acoustic frame segmented from audio data that corresponds to the utterance, an acoustic context vector; and generating, as output from an output layer of the confidence estimation module, a respective confidence output score for the corresponding sub-word unit based on the confidence feature vector and the acoustic feature vector received as input by the output layer of the confidence estimation module. For each of the one or more words formed by the sequence of hypothesized sub-word units, the operations also include determining a respective word-level confidence score for the word and determining an utterance-level confidence score for the speech recognition result by aggregating the respective word-level confidence scores determined for the one or more words of the utterance. The respective word-level confidence score is equal to the respective confidence output score generated for the final sub-word unit in the word.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the set of confidence features represented by the respective confidence embedding include a softmax posteriors feature of the speech recognizer at the corresponding output step, and a sub-word embedding feature for the corresponding sub-word unit. In additional implementations, the set of confidence feature represented by the respective confidence embedding further include a log posterior log feature indicating a probability value associated a probability/likelihood of the corresponding sub-word unit output from the speech recognizer at the corresponding output step, and a top-K feature indicating a K largest log probabilities at the corresponding output step for a top-K candidate hypotheses rescored by the speech recognizer. Here, the top-K candidate hypotheses are each represented by a respective sequence of hypothesized sub-word units that form one or more words of the utterance.

In some examples, the sub-word units include wordpieces, while in other examples, the sub-word units include graphemes. The speech recognizer may include a transducer decoder model and a rescorer decoder model. The transducer decoder model is configured to generate multiple candidate hypothesis during a first pas and the rescorer decoder model is configured to rescore, during a second pass, a top-K candidate hypotheses from the multiple candidate hypotheses generated by the transducer decoder model during the first pass. Each candidate hypothesis corresponds to a candidate transcription for the utterance and represented by a respective sequence of hypothesized sub-word units. Moreover, the candidate hypothesis in the top-K candidate hypotheses rescored by the rescorer decoder model that is represented by the respective sequence of hypothesized sub-word units associated with a highest second pass log probability is output from the rescorer decoder model as the speech recognition result for the utterance spoken by the user. The transducer decoder model may include a Recurrent Neural Network-Transducer (RNN-T) model architecture and the rescorer decoder model may include a Listen, Attend, and Spell (LAS) model architecture.

In some implementations, the operations further include generating, using a linguistic encoder of the speech recognizer during the second pass, a multiple hypotheses encoding by encoding each of the multiple candidate hypotheses generated by the transducer decoder model during the first pass; and using the confidence estimation module, for each sub-word unit in the sequence of hypothesized sub-word units, generating, using a third attention mechanism that cross-attends to the multiple hypotheses encoding, a linguistic context vector. In these implementations, generating the respective confidence output score for the corresponding sub-word unit is further based on the linguistic context vector received as input by the output layer of the confidence estimation module. Encoding each of the multiple candidate hypothesis may include bi-directionally encoding each candidate hypothesis into a corresponding hypothesis encoding and generating the multiple candidate hypothesis may include generating the multiple hypothesis encoding by concatenating each corresponding hypothesis encoding.

The speech recognizer and the confidence estimation module may be trained jointly or separately. The confidence estimation model may be trained using a binary cross-entropy loss based on features associated with the speech recognizer. In some examples, the operations further include determining whether the utterance-level confidence score for the speech recognition result satisfies a confidence threshold, and when the utterance-level confidence score for the speech recognition result fails to satisfy the confidence threshold, transmitting audio data corresponding to the utterance to another speech recognizer. Here, the other speech recognizer is configured to process to the audio data to generate a transcription of the utterance. In these examples, the speech recognizer and the confidence estimation module may execute on a user computing device and the other speech recognizer may execute on a remote server in communication with the user computing device via a network.

Another aspect of the disclosure provides a system that includes data processing hardware and memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations that include receiving, from a speech recognizer, a speech recognition result for an utterance spoken by a user. The speech recognition result includes a sequence of hypothesized sub-word units that form one or more words of the utterance. Each sub-word unit is output from the speech recognizer at a corresponding output step. Using a confidence estimation module, for each sub-word unit in the sequence of hypothesized sub-word units, the operations also include: obtaining a respective confidence embedding that represents a set of confidence features associated with the corresponding output step when the corresponding sub-word unit was output from the speech recognizer; generating, using a first attention mechanism that self-attends to the respective confidence embedding for the corresponding sub-word unit and the confidence embeddings obtained for any other sub-word units in the sequence of hypothesized sub-word units that proceed the corresponding sub-word unit, a confidence feature vector; generating, using a second attention mechanism that cross-attends to a sequence of acoustic encodings each associated with a corresponding acoustic frame segmented from audio data that corresponds to the utterance, an acoustic context vector; and generating, as output from an output layer of the confidence estimation module, a respective confidence output score for the corresponding sub-word unit based on the confidence feature vector and the acoustic feature vector received as input by the output layer of the confidence estimation module. For each of the one or more words formed by the sequence of hypothesized sub-word units, the operations also include determining a respective word-level confidence score for the word and determining an utterance-level confidence score for the speech recognition result by aggregating the respective word-level confidence scores determined for the one or more words of the utterance. The respective word-level confidence score is equal to the respective confidence output score generated for the final sub-word unit in the word.

This aspect may include one or more of the following optional features. In some implementations, the set of confidence features represented by the respective confidence embedding include a softmax posteriors feature of the speech recognizer at the corresponding output step, and a sub-word embedding feature for the corresponding sub-word unit. In additional implementations, the set of confidence feature represented by the respective confidence embedding further include a log posterior log feature indicating a probability value associated a probability/likelihood of the corresponding sub-word unit output from the speech recognizer at the corresponding output step, and a top-K feature indicating a K largest log probabilities at the corresponding output step for a top-K candidate hypotheses rescored by the speech recognizer. Here, the top-K candidate hypotheses are each represented by a respective sequence of hypothesized sub-word units that form one or more words of the utterance.

Automated speech recognition (ASR) systems focus on providing not only quality/accuracy (e.g., low word error rates (WERs)), but also low latency (e.g., a short delay between the user speaking and a transcription appearing). Recently, end-to-end (E2E) ASR models, such as the Recurrent Neural Network-Transducer (RNN-T), the transformer or conformer transducer, and attention-based encoder-decoder models, have gained popularity in achieving state-of-the-art performance in accuracy and latency. In contrast to conventional hybrid ASR systems that include separate acoustic, pronunciation, and language models, E2E models apply a sequence-to-sequence approach to jointly learn acoustic and language modeling in a single neural network that is trained end to end from training data, e.g., utterance-transcription pairs.

Examples of sequence-to-sequence models include "attention-based" models and "listen-attend-spell" (LAS) models. A LAS model transcribes speech utterances into characters using a listener component, an attender component, and a speller component. Here, the listener is a recurrent neural network (RNN) encoder that receives an audio input (e.g., a time-frequency representation of speech input) and maps the audio input to a higher-level feature representation. The attender attends to the higher-level feature to learn an alignment between input features and predicted subword units (e.g., a grapheme or a wordpiece). The speller is an attention-based RNN decoder that generates character sequences from the input by producing a probability distribution over a set of hypothesized words. With an integrated structure, all components of a model may be trained jointly as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is constructed entirely of a neural network. A fully neural network functions without external and/or manually designed components (e.g., finite state transducers, a lexicon, or text normalization modules). Additionally, when training E2E models, these models generally do not require bootstrapping from decision trees or time alignments from a separate system.

Moreover, when using an ASR system today there is a demand that the ASR system decode utterances in a streaming fashion that corresponds to displaying a transcription of an utterance in real-time, or even faster than real-time, as a user speaks. To illustrate, when an ASR system is deployed on a mobile phone that experiences direct user interactivity, an application on the mobile phone using the ASR system may require the speech recognition to be streaming such that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken. Here, it is also likely that the user of the mobile phone has a low tolerance for latency. Due to this low tolerance, the speech recognition strives to run on the mobile device in a manner that minimizes an impact from latency and inaccuracy that may detrimentally affect the user's experience. However, sequence-to-sequence models such as the LAS model that function by reviewing an entire input sequence of audio before generating output text, do not allow for streaming outputs as inputs are received. Due to this deficiency, deploying the LAS model for speech applications that are latency sensitive and/or require real-time voice transcription may pose issues. This makes an LAS model alone not an ideal model for mobile technology (e.g., mobile phones) that often relies on real-time applications (e.g., real-time communication applications).

Another form of a sequence-to-sequence model known as a recurrent neural network transducer (RNN-T) does not employ an attention mechanism and, unlike other sequence-to-sequence models that generally need to process an entire sequence (e.g., audio waveform) to produce an output (e.g., a sentence), the RNN-T continuously processes input samples and streams output symbols, a feature that is particularly attractive for real-time communication. For instance, speech recognition with an RNN-T may output characters one-by-one as spoken. Here, an RNN-T uses a feedback loop that feeds symbols predicted by the model back into itself to predict the next symbols. Because decoding the RNN-T includes a beam search through a single neural network instead of a large decoder graph, an RNN-T may scale to a fraction of the size of a server-based speech recognition model. With the size reduction, the RNN-T may be deployed entirely on-device and be able to run offline (i.e., without a network connection); therefore, avoiding unreliability issues with communication networks.

The RNN-T model alone, however, still lags behind a large state-of-the-art conventional model (e.g., a server-based model with separate AM, PM, and LMs) in terms of quality (e.g., speech recognition accuracy). Yet a non-streaming E2E, LAS model has speech recognition quality that is comparable to large state-of-the-art conventional models. To capitalize on the quality of a non-steaming E2E LAS model, a two-pass speech recognition system (e.g., shown in <FIG>) was developed that includes a first-pass component of an RNN-T network followed by a second-pass component of a LAS network. With this design, the two-pass model benefits from the streaming nature of an RNN-T model with low latency while improving the accuracy of the RNN-T model through the second-pass incorporating the LAS network. Although the LAS network increases the latency when compared to only a RNN-T model, the increase in latency is reasonably slight and complies with latency constraints for on-device operation. With respect to accuracy, a two-pass model achieves a <NUM>-<NUM>% WER reduction when compared to a RNN-T alone and has a similar WER when compared to a large conventional model.

Confidence scores are an important feature of ASR systems that support many downstream applications to mitigate speech recognition errors. For example, unlabeled utterances with recognition results output from an ASR model that achieve high confidence may be used for semi-supervised training of the ASR model which may reduce the expense of using only transcribed utterances for training. On the other hand, in applications such as spoken dialog systems in which a user interacts with a digital assistant executing on a computing device, utterances with recognition results that achieve low word-level confidence may prompt the user to correct any mis-transcribed words. Additionally, recognition results with low confidence may result in passing audio for the corresponding utterance to a different ASR model (e.g., server-side) for improving recognition on the utterance.

While conventional hybrid ASR systems can easily estimate word-level confidence scores from word posterior probabilities computed from lattices or confusion networks and then aggregated to provide an utterance-level confidence, the deep neural networks employed by E2E ASR models tend to exhibit overconfidence when predicting words. As many E2E ASR models are configured to output recognition results at the sub-word level, simply learning confidence scores for each sub-word recognized by the ASR model using a corresponding fixed sub-word tokenization for the word as a reference sequence can lead to incorrect ground truth labels used for training confidence estimation models since recognition results may contain multiple valid tokenizations. For instance, a reference fixed sub-word sequence for the utterance "Good morning" may be "go, od, morn, ing" while a hypothesized sub-word sequence recognized by the ASR model may be "go, od, mor, ning, mom". Here, even though the word "morning" is correctly recognized by the ASR model, the sub-word labels for the corresponding hypothesized sub-words "mor" and "ning" recognized by the ASR model would be labeled incorrect because they do not match the corresponding reference fixed sub-words "morn" and "ing" for the word "morning".

To alleviate the drawbacks associated with estimating sub-word confidence scores for hypothesized sub-word sequences recognized by ASR models due to mismatches between reference fixed sub-word sequences, implementations herein are directed toward a confidence estimation module that applies self-attention in order to estimate word-level confidence for each recognized word using only the confidence of the final hypothesized sub-word unit recognized by the ASR model that makes up the corresponding word. Additional implementations of the present disclosure are further directed toward the confidence estimation module additionally leveraging cross-attention to attend to acoustic context for an utterance being recognized as well as linguistic context for multiple hypotheses (e.g., N-best list) recognized by the ASR model during a first pass. As will become apparent, the leveraging of both acoustic and linguistic cross-attention leads to increased accuracy in confidence scores estimated by the confidence estimation module.

<FIG> is an example speech environment <NUM> in which a user <NUM> interacts with a user device <NUM> through voice input. The user device <NUM> (also referred to generally as device <NUM>) includes a computing device that is configured to capture sounds (e.g., streaming audio data) from one or more users <NUM> within the speech-enabled environment <NUM>. Here, the streaming audio data <NUM> may refer to a spoken utterance by the user <NUM> that functions as an audible query, a command for the device <NUM>, or an audible communication captured by the device <NUM>. Speech-enabled systems of the device <NUM> may field the query or the command by answering the query and/or causing the command to be performed.

The user device <NUM> may correspond to any computing device capable of receiving audio data <NUM>. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, internet of things (IoT) devices, smart speakers/displays, vehicle infotainment systems, etc. The user device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations. The user device <NUM> further includes an audio subsystem <NUM> with an audio capture device (e.g., microphone) <NUM>, 116a for capturing and converting spoken utterances <NUM> within the speech-enabled system <NUM> into electrical signals and a speech output device (e.g., a speaker) <NUM>, 116b for communicating an audible audio signal (e.g., as output audio data from the device <NUM>). While the user device <NUM> implements a single audio capture device 116a in the example shown, the user device <NUM> may implement an array of audio capture devices 116a without departing from the scope of the present disclosure, whereby one or more capture devices 116a in the array may not physically reside on the user device <NUM>, but be in communication with the audio subsystem <NUM>. In the example shown, the user device <NUM> (e.g., using the hardware <NUM>, <NUM>) implements a speech recognizer <NUM> that is configured to perform speech recognition on audio data <NUM> corresponding to an utterance <NUM> spoken by the user <NUM>. Here, the audio capture device 116a is configured to capture acoustic sounds representing the utterance <NUM> and convert the acoustic sounds into the audio data <NUM> associated with a digital format compatible with the speech recognizer <NUM>. The digital format associated with the audio data <NUM> may correspond to acoustic frames (e.g., parameterized acoustic frames), such as mel frames. For instance, the parameterized acoustic frames correspond to log-mel filterbank energies.

While <FIG> shows the user device <NUM> implementing the speech recognizer <NUM> for performing speech recognition on-device, other implementations include a remote server <NUM> (<FIG>) implementing the speech recognizer <NUM> by processing the audio data <NUM> transmitted by the user device <NUM> via a network and providing a transcription <NUM> of the audio data <NUM> back to the user device <NUM>. In some additional implementations, the user device <NUM> utilizes both a local speech recognizer <NUM> residing on the user device <NUM> and a server-side speech recognizer <NUM> (<FIG>) that executes on the remote server <NUM>. Here, user device <NUM> may use the local speech recognizer <NUM> when a network connection is not available or for speech applications that are latency sensitive and/or require streaming transcription, while the server-side speech recognizer <NUM> may be leveraged when additional resources are required to improve speech recognition accuracy as described in greater detail below with reference to <FIG>.

In some examples, the user <NUM> interacts with a program or application <NUM> executing on the user device <NUM> that uses the speech recognizer <NUM>. For instance, <FIG> depicts the user <NUM> communicating with an automated assistant application <NUM>. In this example, the user (e.g., Bill) <NUM> greets the automated assistant application <NUM> by speaking an utterance <NUM>, "Good morning", that is captured by the audio capture device 116a and converted into corresponding audio data <NUM> (e.g. as acoustic frames) for processing by the speech recognizer <NUM>. In this example, the speech recognizer <NUM> transcribes the audio data <NUM> representing the utterance <NUM> into a transcription <NUM> (e.g., a text representation of "Good morning"). Here, the automated assistant application <NUM> may apply natural language processing on the transcription <NUM> to generate a response <NUM> for output to the user <NUM> that conveys the message, "Good Morning Bill, the first meeting today on your calendar is at <NUM>:<NUM> AM. " Natural language processing generally refers to a process of interpreting written language (e.g., the transcription <NUM>) and determining whether the written language prompts any action. In this example, the assistant application <NUM> uses natural language processing to recognize that the utterance <NUM> spoken by the user <NUM> is intended to invoke the assistant application <NUM> to accesses a calendar application of the user <NUM> and provide the response <NUM> indicating what time the user's <NUM> first meeting is today. That is, by recognizing these details with natural language processing, the assistant application <NUM> returns the response <NUM> to the user <NUM> as a synthesized speech representation for audible output through the audio output device 116a and/or as text for display on a screen in communication with the user device <NUM>. In some examples, the user device <NUM> displays transcriptions <NUM> of utterances <NUM> spoken by the user <NUM> and corresponding responses <NUM> from the assistant application <NUM> as a conversation on the screen. In some configurations, natural language processing may occur on a remote system in communication with the data processing hardware <NUM> of the user device <NUM>.

In some examples, the speech recognizer <NUM> processes incoming audio data <NUM> in real-time to provide a streaming transcriptions <NUM>. Here, the speech recognizer <NUM> is configured to produce a sequence of hypothesized sub-word units that make up the words of the utterance <NUM> spoken by the user <NUM>. The hypothesized sub-word units may include word pieces or individual characters (e.g., graphemes). In the example shown, the sequence of hypothesized sub-word units recognized by the speech recognizer include "SOS_go od_mor ning" in which the 'SOS' indicates a start of speech tag and each word boundary indicator ('_') indicates a beginning/starting sub-word unit for each word.

Referring to <FIG>, in some implementations, a speech recognizer <NUM>, 200a is configured in a two-pass decoding architecture and implements a confidence estimation module (CEM) <NUM> for estimating a confidence <NUM> of a final recognition result <NUM> predicted by the speech recognizer <NUM>. Notably, the speech recognizer 200a utilizes a transformer rescorer architecture to perform second pass <NUM> decoding/rescoring. Here, the final recognition result <NUM> corresponds to a sequence of sub-word units, such as word pieces or graphemes, that when aggregated together form a transcription <NUM> for an utterance. Generally speaking, the two-pass architecture of the speech recognizer <NUM> includes at least one shared encoder <NUM>, an RNN-T decoder <NUM>, and a rescorer decoder <NUM> enhanced by an acoustic encoder <NUM>. In two-pass decoding, a second pass <NUM> (e.g., shown as the rescorer decoder <NUM>) may improve initial outputs <NUM> from the first pass <NUM> (e.g., shown as the RNN-T decoder <NUM>) with techniques such as lattice rescoring or top-K re-ranking. In other words, the RNN-T decoder <NUM> produces multiple candidate hypotheses H as output <NUM> and the rescorer decoder <NUM> rescores/re-ranks the top-K candidate hypotheses H to identify a highest scoring candidate hypothesis as the final recognition result <NUM> corresponding to the transcription <NUM> (<FIG>). Although it is generally discussed that the rescorer decoder <NUM> functions in a rescoring mode that rescores streamed hypotheses H <NUM> from the RNN-T decoder <NUM>, the rescorer decoder <NUM> is also capable of operating in different modes, such as a beam search mode, depending on design or other factors (e.g., utterance length). Moreover, while examples herein depict the RNN-T decoder <NUM> performing decoding during the first utterance, the decoder <NUM> may similarly include other types of transducer model architectures without departing from the scope of the present disclosure. For instance, the decoder <NUM> may include one of Transformer-Transducer, Convolutional Neural Network-Transducer (ConvNet-Transducer), or Conformer-Transducer model architectures in lieu of the RNN-T model architecture.

The at least one shared encoder <NUM> is configured to receive, as input, the audio data <NUM> corresponding to the utterance <NUM> as a sequence of acoustic frames. The acoustic frames may be previously processed by the audio subsystem <NUM> into parameterized acoustic frames (e.g., mel frames and/or spectral frames). In some implementations, the parameterized acoustic frames correspond to log-mel filterbank energies with log-mel features. For instance, the parameterized input acoustic frames representing the audio data <NUM> input into the encoder <NUM> may be represented as x = (x<NUM>,. , xT), where <MAT> are log-mel filterbank energies, T denotes the number of frames in x, and d represents the number of log-Mel features. In some examples, each parameterized acoustic frame includes <NUM>-dimensional log-mel features computed within a short shifting window (e.g., <NUM> milliseconds and shifted every <NUM> milliseconds). Each feature may be stacked with previous frames (e.g., three previous frames) to form a higher-dimensional vector (e.g., a <NUM>-dimensional vector using the three previous frames). The features forming the vector may then be downsampled (e.g., to a <NUM> millisecond frame rate). For each acoustic frame x<NUM>:T of the audio data <NUM> input to the encoder <NUM>, the encoder <NUM> is configured to generate, as output <NUM>, a corresponding shared encoding es<NUM>:T.

Although the structure of the encoder <NUM> may be implemented in different ways, in some implementations, the encoder <NUM> includes a long-short term memory (LSTM) neural network. For instance, the LSTM neural network may include eight (<NUM>) LSTM layers. Here, each layer may have <NUM>,<NUM> hidden units followed by a <NUM>-dimensional projection layer. In some examples, a time-reduction layer is inserted with the reduction factor N = <NUM> after the second LSTM layer of the encoder <NUM>.

In some configurations, the encoder <NUM> is a shared encoder network. In other words, instead of each pass network <NUM>, <NUM> having its own separate encoder, each pass <NUM>, <NUM> shares a single encoder <NUM>. The sharing of the encoder <NUM> may reduce a model size and/or reduce a computational cost of the speech recognizer <NUM> utilizing the two-pass architecture. Here, a reduction in model size may help enable the speech recognizer <NUM> to run entirely on-device.

In some examples, the speech recognizer <NUM> of <FIG> also includes an additional encoder, such as the acoustic encoder <NUM>, to adapt the shared encoding es generated as output <NUM> from the shared encoder <NUM> for each acoustic frame x to be suitable for the second pass <NUM> of the rescorer decoder <NUM> as well as confidence <NUM> estimation by the CEM <NUM>. Here, the acoustic encoder <NUM> further encodes, during each time step, each shared encoding es generated as output <NUM> from the shared encoder <NUM> into a corresponding acoustic encoding e<NUM>:T <NUM>. In some implementations, the acoustic encoder <NUM> includes a LSTM encoder (e.g., a two-layer LSTM encoder) that further encodes each output <NUM> from the shared encoder <NUM> into the corresponding acoustic encoding e<NUM>:T <NUM>. Each of the number of frames in x denoted by T corresponds to a respective time step. By including the acoustic encoder <NUM>, the shared encoder <NUM> may still be preserved as a shared encoder between the first and second passes <NUM>, <NUM>.

In some implementations, the RNN-T decoder <NUM> includes a prediction network and a joint network. Here, the prediction network may have two LSTM layers of <NUM>,<NUM> hidden units and a <NUM>-dimensional projection per layer as well as an embedding layer of <NUM> units. The outputs <NUM> of the shared encoder <NUM> and the prediction network may be fed into the joint network that includes a softmax predicting layer. In some examples, the joint network of the RNN-T decoder <NUM> includes <NUM> hidden units followed by a softmax layer that predicts <NUM>,<NUM> mixed-case word pieces.

During the first pass <NUM>, the RNN-T decoder <NUM> receives, as input, the shared encoding es generated as output <NUM> from the shared encoder <NUM> for each acoustic frame x and generates, as output <NUM>, multiple candidate hypotheses H each represented by a respective sequence of hypothesized sub-word units y<NUM>, y<NUM>, y<NUM>,. For instance, in the example where the user <NUM> utters "Good morning", one candidate hypothesis H may include a first sequence of hypothesized sub-word units [_go, od, _mor, ning] and another candidate hypothesis H may include a second sequence of hypothesized sub-word units [_go, od, _morn, ing]. There may be a multitude of other candidate hypothesis H as well. Here, the respective sequence of hypothesized sub-word units y<NUM>, y<NUM>, y<NUM>,. , yM representing each candidate hypothesis H corresponds to a candidate transcription for the utterance <NUM>. Each sub-word unit yi in each respective sequence of hypothesized sub-word units y<NUM>, y<NUM>, y<NUM>,. , yM denotes a probability distribution over possible sub-units. The sub-word unit with the highest probability in the probability distribution may be selected as the hypothesized sub-word in the respective sequence of hypothesized sub-word units.

With continued reference to <FIG>, during the second pass <NUM>, the rescorer decoder <NUM> receives, as input, the sequence of acoustic encodings e, e<NUM>:T <NUM> output from the acoustic encoder <NUM> and the top-K candidate hypotheses H generated as output <NUM> from the RNN-T decoder <NUM>. In one example, K is equal to four (<NUM>) so that the top four candidate hypotheses H are provided as input to the rescorer decoder <NUM> for rescoring during the second-pass <NUM>. For each of the top-K candidate hypotheses H, the rescorer decoder <NUM> is configured to rescore each sub-word unit, and more particularly rescore the probability distribution for each sub-word unit, in the respective sequence of hypothesized sub-word units y<NUM>, y<NUM>, y<NUM>,. , yM using the following equation. <MAT> where ϕ denotes penultimate layer activations for the rescorer decoder <NUM>. The candidate hypothesis H represented by the respective sequence of hypothesized sub-word units y<NUM>, y<NUM>, y<NUM>,. , yM associated with a highest second pass log probability <MAT> may be output as the final recognition result <NUM> corresponding to the transcription <NUM>. The rescorer decoder <NUM> may implement a LAS architecture having four self-attention layers, two of which contain the cross-attention over the encoder.

To decrease the size of the softmax layer, the sub-word unit vocabulary of possible sub-word units is typically smaller compared to a word vocabulary. The sub-word unit vocabulary may include graphemes or wordpieces (WP). An example WP vocabulary may include <NUM>,<NUM> WPs. While examples of the present disclosure use WPs as the sub-word units generated as output from the speech recognizer, graphemes can be similarly utilized as the sub-word units output from the speech recognizer without departing from the scope of the present disclosure. Accordingly, to compute a word error rate (WER) for a candidate hypothesis H, the respective sequence of hypothesized sub-word units (e.g., WPs) needs to be converted into its corresponding word sequence w<NUM>, w<NUM>,. This procedure for converting a sub-word sequence into a word sequence is uniquely determined since the first sub-word unit (e.g., WP) of each word begins with the word boundary indicator ('_'). Similarly, for a word wj including Qj WPs, where yj, q denotes the q-th WP of the j-th word, a simple technique for computing word confidence can be expressed as follows. <MAT> wherein agg can be arithmetic mean, minimum, product, or a neural network. However, since each word wj can be divided into multiple different valid WP combinations due to a mismatch between WP correctness and word correctness, using Equation <NUM> to estimate word-level confidence creates an undesirable computational burden during training since a search over all possible reference tokenizations for the one having a fewest WP edits is required. As used herein, a WP edit includes a correct (cor) label when a hypothesized WP matches a reference WP, a substitution (sub) label when a valid hypothesized WP does not match a reference WP, and an insertion (ins) when a hypothesized WP is misrecognized. Table <NUM> below shows an example where the word "morning" is correctly transcribed, but results in two substitutions in the WP edit distance output.

Referring to <FIG>, in some additional implementations, a speech recognizer <NUM>, 200b is configured in the two-pass decoding architecture utilizes a deliberation rescorer architecture to perform the second pass <NUM> decoding/rescoring in place of the transformer rescorer architecture of <FIG>. A deliberation decoder <NUM> represents the deliberation rescorer architecture and includes a linguistic encoder <NUM>, a hypothesis attention mechanism <NUM>, an acoustic attention mechanism <NUM>, and the rescorer decoder <NUM>. The speech recognizer 200b includes the same shared encoder <NUM>, first-pass <NUM> RNN-T decoder <NUM>, and acoustic encoder <NUM> as the speech recognizer 200a of <FIG> described above. Compared to the transformer rescorer architecture utilized by the speech recognizer 200a of <FIG>, the speech recognizer 200b of <FIG> attends to both acoustics, by attending to the output <NUM> of the shared encoder <NUM> at the acoustic attention mechanism <NUM>, and one or more of the first-pass candidate hypotheses H each represented by the respective sequence of hypothesized sub-word units y<NUM>, y<NUM>, y<NUM>,. , yM, by attending to the outputs <NUM> from the RNN-T decoder <NUM> at the hypothesis attention mechanism <NUM>. In contrast, the speech recognizer <NUM> of <FIG> only attends to the acoustics by attending to the output <NUM> of the encoder <NUM> at the rescorer decoder <NUM> itself. By attending to both acoustics (e.g., the output <NUM> represented as shared encoding es) and the first-pass hypotheses, the deliberation decoder <NUM> generates, as output, the final recognition result <NUM> represented by the respective sequence of hypothesized sub-word units y<NUM>, y<NUM>, y<NUM>,. , yM associated with a highest second pass log probability rescored by the rescorer decoder <NUM>. Notably, each attention mechanism <NUM>, <NUM> forms a context vector <NUM>, <NUM> (e.g., an acoustic context vector <NUM> and a hypothesis context vector <NUM>) that is input into the rescorer decoder <NUM>. A concatenation of these context vectors <NUM>, <NUM> may be input to the rescorer decoder <NUM>. The attention mechanisms <NUM>, <NUM> may each include multi-headed attention (e.g., four heads).

With continued reference to <FIG>, during the second pass <NUM>, the linguistic encoder <NUM> further encodes each candidate hypothesis H generated as output <NUM> from the RNN-T decoder <NUM> into a corresponding hypothesis encoding h <NUM>. Accordingly, the linguistic encoder <NUM> may encode multiple candidate hypotheses into a multiple hypotheses encoding h <NUM> and provide the multiple hypotheses encoding h <NUM> as input to a linguistic cross-attention mechanism <NUM> (<FIG>) of the CEM <NUM> for use in estimating confidence of sub-word units. In this scenario, the corresponding hypothesis encodings h encoded from the multiple candidate hypotheses H may be concatenated into the multiple hypothesis encoding h <NUM> without providing any positional information to allow the CEM <NUM> to use consensus among the multiple hypothesis when scoring a current word. Further, the linguistic encoder <NUM> may also encode the output <NUM> for useful context information to include in the encoded hypotheses <NUM>. For example, the linguistic encoder <NUM> may include a bidirectional encoder capable of including the context information. Structurally, the linguistic encoder <NUM> may be a bidirectional LSTM (BLSTM) encoder (e.g., a <NUM>-layer BLSTM encoder). As a BLSTM encoder, each layer of the linguistic encoder <NUM> may include <NUM>,<NUM> hidden units followed by a <NUM>-dimensional projection.

The rescorer decoder <NUM> receives, as input, the acoustic context vector <NUM> attending to the sequence of acoustic encodings e, e<NUM>:T <NUM> output from the acoustic encoder <NUM> and a hypothesis context vector <NUM> attending to the encoded hypotheses <NUM> for the top-K candidate hypotheses H generated as output <NUM> from the RNN-T decoder <NUM>. For each of the top-K candidate hypotheses H, the rescorer decoder <NUM> uses the context vectors <NUM>, <NUM> to rescore each sub-word unit, and more particularly rescore the probability distribution for each sub-word unit, in the respective sequence of hypothesized sub-word units y<NUM>, y<NUM>, y<NUM>,.

To cure the inherent mismatch between WP correctness and word correctness resulting from speech recognizers <NUM> that output at the WP level as depicted in Table <NUM>, implementations herein are directed toward a transformer/deliberation-based CEM <NUM> that leverages a confidence output at the final WP of every word as a word-level confidence while ignoring the confidence of all other preceding WPs of every word. <FIG> shows an example of the transformer/deliberation-based CEM <NUM> overlain on top of the speech recognizer <NUM> utilizing the two-pass architecture. For clarity, <FIG> depicts only the actions of the CEM <NUM> predicting a confidence output c(yi) <NUM> for the i-th WP in a respective sequence of hypothesized WPs y<NUM>, y<NUM>, y<NUM>,. , yM representing the final recognition result <NUM> rescored by the rescorer decoder <NUM> to. Specifically, <FIG> depicts the CEM <NUM> predicting the confidence output c("ning") for the "ning" WP in the respective sequence of hypothesized WPs [_go, od, _mor, ning] that converts into the corresponding word sequence "good morning". As described in greater detail below, all dashed connections and the dashed linguistic cross-attention block <NUM> are only used for the deliberation CEM <NUM> but not the transformer CEM <NUM>.

During the first pass <NUM>, the RNN-T decoder <NUM> generates, as output <NUM>, multiple candidate hypotheses H each represented by a respective sequence of hypothesized sub-word units y<NUM>, y<NUM>, y<NUM>,. For instance, in the example for the utterance <NUM> "Good morning", one candidate hypothesis H generated as output <NUM> from the RNN-T decoder <NUM> may include a first sequence of hypothesized sub-word units [_go, od, _mor, ning] and another candidate hypothesis H may include a second sequence of hypothesized sub-word units [_go, od, _morn, ing]. At the same time, the acoustic encoder <NUM> generates the sequence of acoustic encodings e<NUM>:T <NUM> where T corresponds to a number of acoustic frames x segmented from the utterance <NUM>.

During the second pass <NUM>, the rescorer decoder <NUM> receives, as input, the sequence of acoustic encodings e, e<NUM>:T <NUM> output from the acoustic encoder <NUM> and the top-K candidate hypotheses H generated as output <NUM> from the RNN-T decoder <NUM>. For each of the top-K candidate hypotheses H, the rescorer decoder <NUM> is configured to rescore each sub-word unit, and more particularly rescore the probability distribution for each sub-word unit, in the respective sequence of hypothesized sub-word units p(y<NUM>), p(y<NUM>), p(y<NUM>),. , p(yM) using Equation (<NUM>), whereby a linear + softmax <NUM> may output the candidate hypothesis H associated with a highest second pass log probability <MAT> as the final recognition result <NUM>. Here, the final recognition result <NUM> includes the sequence of hypothesized sub-word (e.g., WP) units [sos_go, od, _mor, ning eos] with start of speech (sos) and end of speech (eos) tags.

For each sub-word unit (yi) in the sequence of hypothesized sub-word (e.g., WP) units [sos_go, od, _mor, ning eos] representing the final recognition result <NUM>, a confidence embedding b(yi) <NUM> representing a set of confidence features obtained from the speech recognizer <NUM> is provided as input to the CEM <NUM> for determining a respective confidence output c(yi) <NUM>. In the example shown, the i-th sub-word unit corresponds to the WP "ning". Here, the confidence embedding b(yi) <NUM> conveys one or more of a softmax posteriors feature ϕ(i|e, y<NUM>:i-<NUM>) of the rescorer decoder <NUM> using Equation (<NUM>), an input subword + positional embedding Emb(yi) feature (e.g., Emb(ning)), a log posterior log (p(yi)) feature, and a top-K(i) feature. The softmax posteriors feature indicates internal features for the WP "ning", internal features for the acoustic encoding e, e<NUM>:T, and the penultimate layer activations ϕ for the rescorer decoder <NUM>. The log posterior log (p(y<NUM>)) feature indicating a probability value associated with the probability/likelihood that sub-word unit yi includes the WP "ning", and the top-K(i) feature indicates the K largest log probabilities at decoder index (e.g., time step) i. Stated differently, the top-K(i) feature provides probability values for each candidate hypothesis H in the top-K at decoder index (e.g., time step) i. Since both the speech recognizer <NUM> and the CEM <NUM> are configured to generate an output for each time step at the sub-word (e.g., WP) level, implementing the CEM <NUM> as a transformer permits: (<NUM>) the use of word edit distance output as ground truth training labels by leveraging the confidence output c(yj, Qj) at the final WP of every word cword(wj) as a dedicated word-level confidence <NUM>; and (<NUM>) the incorporation of information/features from every WP that makes up the word. In the example shown, a self-attention mechanism <NUM> of the transformer-based CEM <NUM> applies self-attention to a confidence feature vector b, <NUM> based on the confidence embedding b(yi) <NUM> for the i-th sub-word unit corresponding to the WP "ning" as well as confidence embeddings for earlier sub-word units in the same word. The confidence feature vector b may be expressed by the following equations. <MAT> <MAT>.

Additionally, an acoustic cross-attention mechanism <NUM> of the transformer-based CEM <NUM> applies acoustic cross-attention (CA(e)) to the sequence of acoustic encodings e, e<NUM>:T <NUM> output from the acoustic encoder <NUM> to generate an acoustic context vector <NUM> for improving the accuracy in estimating the respective confidence output c(yi) <NUM> for the i-th sub-word unit corresponding the WP "ning". Finally, a linear + sigmoid block <NUM> uses the self-attention confidence feature vector SA(b) and the cross-attention CA(e) acoustic context vector <NUM> to permit the transformer-based CEM <NUM> to produce the dedicated confidence <NUM> for each word cword(wj) using a confidence output c(yj, Qj) at the final WP as follows. <MAT> <MAT> where the confidence <NUM> for the word cword(morning) in the example shown corresponds to the confidence output c(ning) of the final WP that makes up the word.

In some implementations, the speech recognizer <NUM> utilizes the deliberation decoder <NUM> of <FIG> for second pass rescoring to implement the deliberation-based CEM <NUM> that incorporates linguistic information from the multiple candidate hypotheses H generated as output <NUM> from the RNN-T decoder <NUM> during the first pass <NUM> to further improve confidence accuracy. In general, words tend to have higher confidence the greater the number of candidate hypotheses H the words are shared across. For instance, in the example from Table <NUM> for the utterance "Good morning", having the first sequence of hypothesized sub-word units [_go, od, _mor, ning] for the first candidate hypothesis H and the second sequence of hypothesized sub-word units [_go, od, _morn, ing] for the second candidate hypothesis H attend to each other will inform the deliberation-based CEM that they concatenate to the same word sequence, and should therefore map to similar confidence scores. Notably, since CEM <NUM> is tasked with estimating the confidence <NUM> for each word in a known hypothesis, the deliberation-based CEM can utilize future context of the hypothesis when scoring/estimating the current word. In addition to the acoustic cross-attention mechanism <NUM> that applies acoustic cross-attention CA(e) to the sequence of acoustic encodings e, e<NUM>:T <NUM> output from the acoustic encoder <NUM> to generate the acoustic context vector <NUM>, the deliberation-based CEM <NUM> also includes a linguistic cross-attention mechanism <NUM> that applies linguistic cross-attention (CA(h)) to the multiple candidate hypotheses encoding h <NUM> output from the linguistic encoder <NUM> (<FIG>) to generate a linguistic context vector <NUM> for use in estimating confidence of the sub-word units. The multiple hypotheses encoding h <NUM> output from the linguistic encoder <NUM> of <FIG> may be represented as follows. <MAT> where H is the number of candidate hypotheses attended to and MH is the number of WPs in the H-th hypothesis. In one example, H is equal to eight (<NUM>). In one example, H is equal to one (<NUM>) whereby only the hypothesis encoding h <NUM> associated with the final recognition result <NUM> is attended to for use in estimating confidence of the sub-word units.

Accordingly, when estimating the respective confidence output c(yi) <NUM> for the i-th sub-word unit corresponding the WP "ning", the deliberation-based CEM <NUM> incorporates both acoustic context, by attending to the sequence of acoustic encodings e, e<NUM>:T <NUM> via the acoustic cross-attention mechanism <NUM>, and linguistic context, by attending to the multiple hypotheses encoding h <NUM> via the linguistic cross-attention mechanism <NUM>. Finally, an output layer <NUM> corresponding to the linear + sigmoid block <NUM> of the CEM <NUM> uses the self-attention confidence feature vector SA(b), the cross-attention CA(e) acoustic context vector <NUM>, and the cross-attention CA(h) linguistic context vector <NUM> to permit the deliberation-based CEM <NUM> to produce the confidence output c(yi) for each sub-word WP as follows.

As with the transformer-based CEM <NUM>, the deliberation-based CEM <NUM> may use Equation (<NUM>) to determine the dedicated confidence <NUM> for each word cword(wj) by using the confidence output c(yj, Qj) at the final WP. In both the transformer- and deliberation-based CEMs <NUM>, an utterance-level confidence score <NUM> (<FIG>) may be determined by aggregating the dedicated confidence of <NUM> for each word cword(wj) in the corresponding word sequence w<NUM>:L. In some examples, the aggregation includes an arithmetic mean aggregator.

The CEM <NUM> may be trained jointly with the speech recognizer <NUM>, or the CEM <NUM> and the speech recognizer may be trained separately from one another. In some examples, the CEM <NUM> is trained using a binary cross-entropy word-level loss as follows. <MAT> where Table <NUM> shows that d(wj) is equal to one when a Levenshtein word-level edit distance for the word wj outputs the "correct" (cor) label when the hypothesized word matches the reference word, and d(wj) is equal to zero when the Levenshtein word-level edit distance for the word wj outputs the "insertion" (ins) or "substitution" (sub) labels when the hypothesized word does not match the reference word. Notably, since the speech recognizer <NUM> and the CEM <NUM> output at the sub-word level (e.g., output every WP), Table <NUM> also shows the CEM <NUM> applying an end-of word mask loss m to focus only on the final WP making up the word and ignore WP losses associated with earlier WPs that make up the same word.

<FIG> shows a schematic view <NUM> of an example confidence-based routine for selecting an appropriate speech recognizer to transcribe an utterance <NUM>. In the example shown, a first speech recognizer <NUM> configured in to the two-pass decoding architecture (e.g., including either the transformer rescorer architecture of <FIG> or the deliberation rescorer architecture of <FIG>) operates as a default speech recognizer for generating a transcription <NUM> by processing incoming audio data <NUM> corresponding to an utterance <NUM> spoken by a user <NUM>. The first speech recognizer <NUM> may correspond to a local speech recognizer that executes on a user device <NUM> associated with the user <NUM>. The first speech recognizer <NUM> also implements the CEM <NUM> for determining an utterance-level confidence score <NUM> for a speech recognition result <NUM> output by the first speech recognizer <NUM> that corresponds to the transcription <NUM>.

In some implementations, the confidence-based routine determines whether the utterance-level confidence score <NUM> for the utterance <NUM> transcribed by the first speech recognizer <NUM> satisfies a confidence threshold. In the example shown, utterance-level confidence scores <NUM> greater than the confidence threshold satisfy the confidence threshold while utterance-level confidence scores <NUM> less than or equal to the confidence threshold fail to satisfy the confidence threshold. When the utterance-level confidence score <NUM> satisfies (e.g., is greater than) the confidence threshold (e.g., decision block <NUM> is "Yes"), then the transcription <NUM> generated by the first speech recognizer <NUM> is accepted to achieve on-device gains in quality, latency, and reliability. Here, the accepted transcription <NUM> may display, or continue to display, on the user device <NUM> and/or be passed to a downstream natural language understanding (NLU) module for interpreting the transcription <NUM> and performing a related action/operation if necessary.

When the utterance-level confidence score <NUM> fails to satisfy (e.g., is less than) the confidence threshold (e.g., decision block <NUM> is "No"), then the confidence-based routine rejects the transcription <NUM> generated by the first speech recognizer <NUM> and passes the audio data <NUM> to the second speech recognizer <NUM> for processing to re-transcribe the utterance <NUM>. The transcription <NUM> generated by the second speech recognizer <NUM> may be passed back to the user device <NUM> and/or to the downstream NLU module for interpretation. In examples where the first speech recognizer <NUM> is local and executing on-device <NUM> and the second speech recognizer <NUM> is server-side and executing on a remote server <NUM>, the confidence-based routine causes the user device <NUM> to transmit the audio data <NUM> to the remote server <NUM> via a network (not shown) so that the second speech recognizer <NUM> executing thereon can transcribe the utterance <NUM>. The second speech recognizer <NUM> may leverage a large language model trained on large-scale language model training data making the second speech recognizer <NUM> more suitable for recognizing proper nouns or less-common words not present in the training data used to train the first speech recognizer <NUM>.

In some examples, the first speech recognizer <NUM> is generally more accurate (e.g., achieves lower word error rates) for recognizing short-form utterances than the second speech recognizer <NUM> implementing the larger language model and lexicon, the first speech recognizer <NUM> may ultimately be less accurate at recognizing long-tail utterances than the second speech recognizer <NUM>. As thus, the confidence-based routine may send all utterances with confidence scores <NUM> less than the confidence threshold to the second speech recognizer <NUM> for generating the transcription <NUM>, and transcribe a majority of utterances on-device <NUM> using the first speech recognizer <NUM> to gain quality, latency, and reliability.

While the second speech recognizer <NUM> is shown as being server-side, the second speech recognizer <NUM> could also execute on-device. For instance, the second speech recognizer <NUM> may be associated with a more computationally-intensive speech recognizer that may generate more accurate speech recognition results on certain utterances than the first speech recognizer <NUM>, but at the cost of reduced latency and increased power consumption. As such, the confidence-based routine may leverage the second speech recognizer <NUM> to transcribe utterances <NUM> when utterance-level confidence scores associated with recognition results generated by the first speech recognizer <NUM> are less than the confidence threshold.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of estimating word-level confidence for a word recognized by a speech recognizer using only a confidence of the final hypothesized sub-word unit for that word. The data processing hardware <NUM> (<FIG>) may execute instructions stored on the memory hardware <NUM> (<FIG>) to perform example arrangement of operations for the method <NUM>. At operation <NUM>, the method <NUM> includes receiving, from a speech recognizer <NUM>, a speech recognition result <NUM> for an utterance <NUM> spoken by a user <NUM>. The speech recognizer <NUM> may be configured in a two-pass decoding architecture as discussed above with reference to <FIG> and <FIG>. Here, the speech recognition result <NUM> is a highest-scoring candidate hypothesis re-scored by a rescoring decoder of the speech recognizer <NUM> and includes a sequence of hypothesized sub-word units that form one or more words of the utterance <NUM>, each sub-word unit output from the speech recognizer <NUM> at a corresponding output step.

Using a confidence estimation module (CEM) <NUM>, for each sub-word unit in the sequence of hypothesized sub-word units, the method <NUM> performs operations <NUM>, <NUM>, <NUM>, <NUM>. At operation <NUM>, the method <NUM> includes obtaining a respective confidence embedding <NUM> that represents a set of confidence features associated with the corresponding output step when the corresponding sub-word unit was output from the speech recognizer <NUM>. At operation <NUM>, the method <NUM> includes generating, using a first attention mechanism <NUM> that self-attends to the respective confidence embedding b(y) <NUM> for the corresponding sub-word unit and the confidence embeddings b(y<NUM>)-b(y<NUM>-i) obtained for any other sub-word units in the sequence of hypothesized sub-word units that proceed the corresponding sub-word unit, a confidence feature vector SA(b), <NUM>. At operation <NUM>, the method <NUM> includes generating, using a second attention mechanism <NUM> that cross-attends to a sequence of acoustic encodings e, e<NUM>:T <NUM> each associated with a corresponding acoustic frame xT segmented from audio data <NUM> that corresponds to the utterance <NUM>, an acoustic context vector CA(e) <NUM>. At operation <NUM>, the method <NUM> includes generating, as output from an output layer <NUM> of the CEM <NUM>, a respective confidence output score <NUM> for the corresponding sub-word unit based on the confidence feature vector SA(b) and the acoustic feature vector CA(e) <NUM> received as input by the output layer of the CEM <NUM>.

At operation <NUM>, for each of the one or more words formed by the sequence of hypothesized sub-word units, the method <NUM> includes determining a respective word-level confidence score for the word. Here, the respective word-level confidence score is equal to the respective confidence output score <NUM> generated for the final sub-word unit in the word. At operation <NUM>, the method <NUM> includes determining an utterance-level confidence score <NUM> for the speech recognition result <NUM> by aggregating the respective word-level confidence scores determined for the one or more words of the utterance <NUM>.

Claim 1:
A computer-implemented method (<NUM>) when executed on data processing hardware (<NUM>) causes the data processing hardware (<NUM>) to perform operations comprising:
receiving, from a speech recognizer (<NUM>), a speech recognition result (<NUM>) for an utterance (<NUM>) spoken by a user (<NUM>), the speech recognition result (<NUM>) comprising a sequence of hypothesized sub-word units that form one or more words of the utterance (<NUM>), each sub-word unit output from the speech recognizer (<NUM>) at a corresponding output step;
using a confidence estimation module (<NUM>), for each sub-word unit in the sequence of hypothesized sub-word units:
obtaining a respective confidence embedding (<NUM>) that represents a set of confidence features associated with the corresponding output step when the corresponding sub-word unit was output from the speech recognizer (<NUM>);
generating, using a first attention mechanism (<NUM>) that self-attends to the respective confidence embedding (<NUM>) for the corresponding sub-word unit and the confidence embeddings (<NUM>) obtained for any other sub-word units in the sequence of hypothesized sub-word units that proceed the corresponding sub-word unit, a confidence feature vector (<NUM>);
generating, using a second attention mechanism (<NUM>) that cross-attends to a sequence of acoustic encodings (<NUM>) each associated with a corresponding acoustic frame segmented from audio data (<NUM>) that corresponds to the utterance (<NUM>), an acoustic context vector (<NUM>); and
generating, as output from an output layer (<NUM>) of the confidence estimation module (<NUM>), a respective confidence output score (<NUM>) for the corresponding sub-word unit based on the confidence feature vector (<NUM>) and an acoustic feature vector (<NUM>) received as input by the output layer (<NUM>) of the confidence estimation module (<NUM>);
for each of the one or more words formed by the sequence of hypothesized sub-word units, determining a respective word-level confidence score for the word, the respective word-level confidence score equal to the respective confidence output score (<NUM>) generated for the final sub-word unit in the word; and
determining an utterance-level confidence score (<NUM>) for the speech recognition result (<NUM>) by aggregating the respective word-level confidence scores determined for the one or more words of the utterance (<NUM>).