Patent Description:
Modern automatic speech recognition (ASR) systems focus on providing not only high quality (e.g., a low word error rate), but also low latency (e.g., a short delay between the user speaking and a transcription appearing). For example, when using a device that implements an ASR system today, there is often an expectation that the ASR system decodes utterances in a streaming fashion that corresponds to real-time or even faster than real-time. <CIT> patent document discloses a method for providing joint end pointing and automatic speech recognition for natural language understanding.

One aspect of the disclosure provides a computer-implemented method as defined by independent claim <NUM>. The method comprising that, when executed on data processing hardware, causes the data processing hardware to perform operations including receiving a sequence of acoustic frames characterizing one or more utterances. At each of a plurality of output time steps, the operations further include: generating, by an encoder network of a speech recognition model, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; generating, by a prediction network of the speech recognition model, a hidden representation for a corresponding sequence of non-blank symbols output by a final softmax layer of the speech recognition model; and generating, by a first joint network of the speech recognition model that receives the higher order feature representation generated by the encoder network and the hidden representation generated by the prediction network, a probability distribution that the corresponding time step corresponds to a pause and an end of speech.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the operations further include determining that a probability that the corresponding time step corresponds to the end of speech satisfies an end of speech threshold; and in response to determining that the probability that the corresponding time step corresponds to the end of speech satisfies the threshold, triggering a microphone closing event.

In some implementations, the operations include determining that a probability that the corresponding time step corresponds to the pause satisfies a pause threshold; and emitting a pause token at the corresponding time step based on the determining that the probability of the corresponding time step corresponds to the pause satisfies the pause threshold.

In some examples, the operations include, at each of the plurality of output steps, generating, by a second joint network of the speech recognition model, a probability distribution over possible speech recognition hypotheses. In some implementations, the speech recognition model is trained by a two-stage training process. The two-stage training process may include: a first stage that trains the encoder network, the prediction network, and the second joint network on a speech recognition task; and a second stage that initializes and fine-tunes the first joint network to learn how to predict pause and end of speech locations in utterances. In some examples, parameters of the encoder network, the prediction network, and the second joint network are frozen during the second stage of the two-stage training process. In some implementations, the two-stage training process trains the speech recognition model on a plurality of transcribed training utterances having labels indicating pause and end of speech locations.

In some implementations, the encoder network includes a stack of self-attention blocks. The stack of self-attention blocks may include a stack of conformer blocks or a stack of transformer blocks.

In some examples, generating the hidden representation for the corresponding sequence of non-blank symbols includes for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step: generating, by the prediction network, using a shared embedding matrix, an embedding of the corresponding non-blank symbol; assigning, by the prediction network, a respective position vector to the corresponding non-blank symbol; and weighting, by the prediction network, the embedding proportional to a similarity between the embedding and the respective position vector. Generating the hidden representation further includes generating, as output from the prediction network, a single embedding vector at the corresponding time step, the single embedding vector based on a weighted average of the weighted embeddings, the single embedding vector including the hidden representation.

In some implementations, the prediction network includes a multi-headed attention mechanism, the multi-headed attention mechanism sharing the shared embedding matrix across each head of the multi-headed attention mechanism.

Another aspect of the disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware as defined by independent system claim <NUM>. The memory hardware storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations. The operations include receiving a sequence of acoustic frames characterizing one or more utterances. The operations further include, at each of a plurality of output steps: generating, by an encoder network of a speech recognition model, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; generating, by a prediction network of the speech recognition model, a hidden representation for a corresponding sequence of non-blank symbols output by a final softmax layer of the speech recognition model; and generating, by a first joint network of the speech recognition model that receives the higher order feature representation generated by the encoder network and the hidden representation generated by the prediction network, a probability distribution that the corresponding time step corresponds to a pause and an end of speech.

Yet another aspect of the disclosure provides a natural conversation automated speech recognition (ASR) model including an encoder, a prediction network, and a first joint network as defined by independent claim <NUM>. The encoder configured to receive, as input, a sequence of acoustic frames characterizing one or more utterances; and generate, at each of a plurality of time steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The prediction network configured to receive, as input, a sequence of non-blank symbols output by a final softmax layer; and generate, at each of the plurality of time steps, a hidden representation. The first joint network configured to receive, as input, the hidden representation generated by the prediction network at each of the plurality of time steps and the higher order feature representation generated by the encoder at each of the plurality of time steps; and generate, at each of the plurality of time steps, a probability distribution of whether the corresponding time step corresponds to a pause and an end of speech.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the ASR model triggers a microphone closing event based on a probability of the corresponding time step corresponding to end of speech satisfying a threshold.

In some examples, the ASR model also includes a second joint network configured to: receive, as input, the hidden representation generated by the prediction network at each of the plurality of time steps and the higher order feature representation generated by the encoder at each of the plurality of time steps; and generate, at each of the plurality of time steps, a probability distribution over possible speech recognition hypotheses.

In some implementations, the encoder network, the prediction network, and the second joint network are trained on a speech recognition task during a first training stage; and after the first training stage, the first joint network is initialized and fine-tuned to learn how to predict pause and end of speech locations in utterances while parameters of the encoder network, the prediction network, and the second joint network are frozen.

Still another aspect of the disclosure provides a natural conversational automated speech recognition (ASR) system including an ASR model having an encoder, a prediction network, and a first joint network; and a turn taking detector model. The encoder configured to receive, as input, a sequence of acoustic frames characterizing one or more utterances; and generate, at each of a plurality of time steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The prediction network configured to receive, as input, a sequence of non-blank symbols output by a final softmax layer; and generate, at each of the plurality of time steps, a hidden representation. The joint network configured to: receive, as input, the hidden representation generated by the prediction network at each of the plurality of time steps and the higher order feature representation generated by the encoder at each of the plurality of time steps; and generate, at each of the plurality of time steps, a probability distribution of whether the corresponding time step corresponds to a pause and an end of speech. The turn taking detector model configured to receive, as input, the higher order feature representation generated by the encoder at each of the plurality of time steps; and, for each higher order feature representation, generate a corresponding probability distribution of whether the higher order feature representation corresponds to talking, pause, and end of speech.

A still further aspect of the disclosure provides a natural conversational automated speech recognition (ASR) system including an ASR model having an encoder, a prediction network, and a joint network; and a turn taking detector model. The ASR model including an encoder configured to receive, as input, a sequence of acoustic frames characterizing one or more utterances; and generate, at each of a plurality of time steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The prediction network configured to receive, as input, a sequence of non-blank symbols output by a final softmax layer; and generate, at each of the plurality of time steps, a hidden representation. The joint network configured to receive, as input, the hidden representation generated by the prediction network at each of the plurality of time steps and the higher order feature representation generated by the encoder at each of the plurality of time steps; and generate, at each of the plurality of time steps, a probability distribution over possible speech recognition hypotheses. The turn-taking detector model configured to receive, as input, the hidden representation generated by the prediction network at each of the plurality of time steps; and generate a corresponding probability distribution of whether a next sub-word unit corresponds to a pause and an end of speech.

Many voice interaction applications, such as voice-activated digital assistants and dialog systems, use streaming automatic speech recognition (ASR) systems. However, to provide a human-like, natural conversational experience, ASR systems need to accurately recognize speech and interaction patterns that resemble human conversational speech, interactions, turn taking, etc. Problems involved in recognizing natural conversational speech include, but are not limited to, recognizing pauses and determining when a person has finished speaking. Even though a lot of natural conversational speech includes disfluencies, most conventional ASR systems assume no disfluencies are present. For example, most conventional ASR systems assume fluent, one-shot utterances for which a person knows what they want to say beforehand, and then speaks without disfluencies. Example disfluencies include, but are not limited to, pauses, pauses to think, random pauses, hesitations, word lengthening (e.g., "onnn. "), filler pauses or words (e.g., "uh", "um"), repeated phrases, and changing of actions. Disfluencies may introduce short or long pauses in an utterance, which may cause ambiguity during ASR, such that an ASR system may prematurely endpoint an utterance and interrupt a person before they have finished speaking. For example, a person may start an utterance by speaking "where can I" followed by a pause. During natural conversational speech, the pause may indicate that the person has not finished speaking. However, conventional ASR systems often respond with "sorry, I didn't get that" before the person has a chance to speak the rest of what they intended to speak, e.g., continue by speaking "order Thai food. " For natural conversational interactions, it is preferable that the ASR system either respond with an acknowledgement phrase, such as "mmh," during the pause to indicate that the ASR system is waiting for the person to finish speaking, or simply wait for the person to continue and finish speaking. Thus, it is important for natural conversational interactions that ASR systems accurately recognize and handle disfluencies and end-of-speaking events to allow a person to use disfluencies to "hold the floor" until they are done speaking while responding as quickly as possible once the person finishes speaking.

Implementations herein are directed toward integrating an ASR system with a disfluency detection model that is configured and trained to detect disfluencies and end-of-speaking events that occur naturally in spoken utterances of natural conversational interactions. Example disfluency detection models are built on top of, or integrated with, an end-to-end (E2E) ASR model, such as a recurrent neural network - transducer (RNN-T) model. In an example method, an ASR system receives a sequence of acoustic frames characterizing one or more utterances. For each of a plurality of output steps: an encoder network of the ASR model generates a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; a prediction network of the ASR model generates a hidden representation for a corresponding sequence of non-blank symbols output by a final softmax layer of the ASR model; and a first joint network of the ASR model (i.e., a disfluency detection joint network) receives the higher order feature representation and the hidden representation, and generates a probability distribution that the corresponding time step corresponds to a disfluency (e.g., a pause) and an end of speech event. A second joint network (e.g., a word piece joint network) of the ASR model receives the higher order feature representation and the hidden representation at the corresponding time step, and generates a probability distribution over possible speech recognition hypotheses at the corresponding time step.

<FIG> is a schematic diagram of an example speech environment <NUM>. In the speech environment <NUM>, a user's <NUM> manner of interacting with a computing device, such as a user device <NUM>, may be through voice input. The user device <NUM> (also referred to generally as a device <NUM>) is configured to capture sounds (e.g., streaming audio data) from one or more users <NUM> within the speech environment <NUM>. Here, the streaming audio data may refer to a spoken utterance <NUM> by the user <NUM> that functions as an audible query, a command for the device <NUM>, or an audible communication captured by the device <NUM>. Speech-enabled systems of the device <NUM> may field the query or the command by answering the query and/or causing the command to be performed/fulfilled by one or more downstream applications.

The user device <NUM> may correspond to any computing device associated with a user <NUM> and capable of receiving audio data. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, vehicle infotainment systems, internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM>. The memory hardware <NUM> stores instructions that, when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations. The user device <NUM> further includes an audio system <NUM> with an audio capture device (e.g., microphone) <NUM>, 16a for capturing and converting spoken utterances <NUM> within the speech environment <NUM> into electrical signals, and a speech output device (e.g., a speaker) <NUM>, 16b for communicating an audible audio signal (e.g., as output audio data from the device <NUM>). While the user device <NUM> implements a single audio capture device 16a in the example shown, the user device <NUM> may implement an array of audio capture devices 16a without departing from the scope of the present disclosure, whereby one or more capture devices 16a in the array may not physically reside on the user device <NUM>, but be in communication with the audio system <NUM>.

In the speech environment <NUM>, an automated speech recognition (ASR) system <NUM> implementing a recurrent neural network-transducer (RNN-T) model <NUM> and a disfluency detection model <NUM> resides on the user device <NUM> of the user <NUM> and/or on a remote computing device <NUM> (e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device <NUM> via a network <NUM>. The user device <NUM> and/or the remote computing device <NUM> also includes an audio subsystem <NUM> configured to receive the utterance <NUM> spoken by the user <NUM> and captured by the audio capture device 16a, and convert the utterance <NUM> into a corresponding digital format associated with input acoustic frames <NUM> capable of being processed by the ASR system <NUM>. In the example shown, the user speaks a respective utterance <NUM> and the audio subsystem <NUM> converts the utterance <NUM> into corresponding audio data (e.g., acoustic frames) <NUM> for input to the ASR system <NUM>. Thereafter, the RNN-T model <NUM> receives, as input, the acoustic frames <NUM> corresponding to the utterance <NUM>, and generates/predicts, as output, a corresponding transcription <NUM> of the utterance <NUM>. In the example shown, the RNN-T model <NUM> may perform streaming speech recognition to produce partial transcriptions (e.g., streaming speech recognition results) 120a, 120b as the user speaks.

The user device <NUM> and/or the remote computing device <NUM> also executes a user interface generator <NUM> configured to present representations of the transcriptions <NUM>, 120a-n of the utterance <NUM> to the user <NUM> of the user device <NUM>, and to present one or more responses <NUM>, 121a-n to queries and/or commands of the utterance <NUM>. As described in greater detail below, the user interface generator <NUM> may display the speech recognition results <NUM>, 120a-n and the responses <NUM>, 121a-n during or at different times. As shown, the user interface generator <NUM> may present the speech recognition results <NUM>, 120a-n (e.g., transcriptions) and the responses <NUM>, 121a-n to represent an interaction/conversation between the user <NUM> and an interactive program or application (e.g., a digital assistant application <NUM>).

In some configurations, the transcription <NUM> output from the ASR system <NUM> are processed, e.g., by a natural language processing/understanding (NLP/NLU) module executing on the user device <NUM> or the remote computing device <NUM>, to execute a user command/query specified by the utterance <NUM>. The digital assistant application <NUM> may provide an appropriate response <NUM> subsequent to executing the user command/query specified by the utterance <NUM>. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device <NUM> or the remote computing device <NUM>) may convert the transcription <NUM> and/or the response <NUM> into synthesized speech for audible output by the user device <NUM> and/or another device.

In the example shown, the user <NUM> interacts with a program or application (e.g., the digital assistant application <NUM>) of the user device <NUM> that uses the ASR system <NUM>. For instance, <FIG> depicts the user <NUM> communicating with the digital assistant application <NUM>, and the digital assistant application <NUM> displaying an interactive digital assistant interface <NUM> on a screen <NUM> of the user device <NUM> to depict a natural conversational interaction between the user <NUM> and the digital assistant application <NUM>. In this example, the user <NUM>, during time <NUM>, speaks a first portion 106a ("Where can I") of the utterance <NUM>, and then pauses (as represented by ellipsis ". ") during a second portion 106b of the utterance.

Continuing with this example, the RNN-T model <NUM>, while receiving the acoustic frames <NUM> corresponding to the utterance <NUM> as the user <NUM> speaks, performs speech recognition on the acoustic frames <NUM> to produce first speech recognition results 120a (i.e., "where can I") corresponding to the first portion 106a of the utterance <NUM>. In the example shown, the disfluency detection model <NUM> detects that the acoustic frames <NUM> corresponding to the second portion 106b of the utterance <NUM> are indicative of a pause in the user's speech rather than an end of speech event which would prematurely endpoint the utterance. During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the first speech recognition results 120a of the utterance <NUM> to the user <NUM> of the user device <NUM> in a streaming fashion such that words, word pieces, and/or individual characters appear on the screen <NUM> of the user device <NUM> as soon as they are spoken. Based on the disfluency detection model <NUM> detecting the presence of the pause during the second portion 106b, at time <NUM>, the digital assistant <NUM> responds with an acknowledgement response 121a ("Mhm") to indicate to the user <NUM> that the ASR system <NUM> and the digital assistant <NUM> are waiting for the user <NUM> to continue speaking.

Continuing with this example, the user <NUM> commences speaking a third remaining portion 106c ("order Thai food") of the utterance <NUM> after pausing and the RNN-T model <NUM> performs speech recognition on the audio frames <NUM> corresponding to the remaining portion 106c of the utterance <NUM> to produce second speech recognition results 120b (i.e., "order Thai food"). When the user <NUM> has finished speaking the remaining portion 106c of the utterance <NUM>, the disfluency detection model <NUM> detects an end of speech event to indicate that the utterance <NUM> is complete. During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the second speech recognition results 120b of the utterance <NUM> to the user <NUM> of the user device <NUM> in a streaming fashion such that words, word pieces, and/or individual characters appear on the screen <NUM> of the user device <NUM> as soon as they are spoken. Notably, the ASR system <NUM> may trigger a microphone closing event and process the transcription <NUM> (e.g., by the NLP/NLU module) to execute a user command/query (e.g., perform a search for Thai restaurants) specified by the utterance <NUM> responsive to the disfluency detection model <NUM> detecting the end of speech event. At time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a response 121b (i.e., "These Thai restaurants are nearby. ") to the query.

<FIG> is a schematic view of an example RNN-T model <NUM>, 200a integrating the disfluency detection model <NUM> as a first joint network (i.e., disfluency detection joint network) <NUM> for detecting disfluencies in spoken utterances that are indicative of speech and interaction patterns that resemble human conversational speech, interactions, and/or turn taking with a digital assistant. Problems involved in recognizing natural conversational speech include, but are not limited to, recognizing pauses and determining when a person has finished speaking. Example disfluencies include, but are not limited to, pauses, pauses to think, random pauses, hesitations, word lengthening (e.g., "onnn. "), filler pauses or words (e.g., "uh", "um"), repeated phrases, and changing of actions. These disfluencies, characterized by no voice activity detected for a threshold duration, can trigger conventional ASR systems to prematurely endpoint the utterance before the user has completed speaking the utterance. Such premature endpointing may result in misinterpreted queries that the digital assistant may not be able to process where the digital assistant may interrupt to prompt the user to repeat the query, thereby leading to user frustration.

As shown, the RNN-T model 200a includes an encoder network <NUM>, a prediction/decoder network <NUM>, a second joint network <NUM> (i.e., a word piece joint network <NUM>), and a final softmax output layer <NUM>. The encoder network <NUM> (e.g., an audio encoder), which is roughly analogous to an acoustic model (AM) in a traditional ASR system, receives a sequence of feature vectors x = (x<NUM>, x<NUM>,··· , xt) <NUM>, where <MAT> (e.g., the acoustic frames <NUM> of <FIG>), and produces at each time step a higher-order feature representation <NUM> (also generally referred to as an acoustic representation) denoted as <MAT>.

In the example shown, the prediction/decoder network <NUM> includes an LSTM-based prediction network that, like a language model (LM), processes a sequence of non-blank symbols y<NUM>,. , yu-<NUM> <NUM> output so far by the softmax layer <NUM> into a hidden representation <MAT> <NUM> (also generally referred to as a dense or linguistic representation) representing a probability distribution of whether a current time step corresponds to a pause and an end of speech, where y<NUM> represents a special start of sequence symbol.

<FIG> shows the prediction network <NUM> for the RNN-T model <NUM> that receives, as input, a sequence of non-blank symbols yu-n,. , yu-<NUM> that is limited to the N previous non-blank symbols 252a-n output by the final softmax layer <NUM>. In some examples, N is equal to two. In other examples, N is equal to five, however, the disclosure is non-limiting and N may equal any integer. The sequence of non-blank symbols 252a-n may indicate a partial speech recognition result 120a, 120b (<FIG>). In some implementations, the prediction network <NUM> includes a multi-headed attention mechanism <NUM> that shares a shared embedding matrix <NUM> across each head 302A-<NUM> of the multi-headed attention mechanism. In one example, the multi-headed attention mechanism <NUM> includes four heads. However, any number of heads may be employed by the multi-headed attention mechanism <NUM>. Notably, the multi-headed attention mechanism <NUM> improves performance significantly with minimal increase to model size. As described in greater detail below, each head 302A-H includes its own row of position vectors <NUM>, and rather than incurring an increase in model size by concatenating outputs 318A-H from all the heads, the outputs 318A-H are instead averaged by a head average module <NUM>.

Referring to the first head 302A of the multi-headed attention mechanism <NUM>, the head 302A generates, using the shared embedding matrix <NUM>, a corresponding embedding <NUM>, 306a-n (e.g., <MAT>) for each non-blank symbol <NUM> among the sequence of non-blank symbols yui-n,. , yui-<NUM> received as input at the corresponding time step from the plurality of time steps. Notably, since the shared embedding matrix <NUM> is shared across all heads of the multi-headed attention mechanism <NUM>, the other heads 302B-H all generate the same corresponding embeddings <NUM> for each non-blank symbol. The head 302A also assigns a respective position vector PVAa-An<NUM>, 308Aa-An (e.g., <MAT>) to each corresponding non-blank symbol in the sequence of non-blank symbols yu-n,. , yu-<NUM>. The respective position vector PV <NUM> assigned to each non-blank symbol indicates a position in the history of the sequence of non-blank symbols (e.g., the N previous non-blank symbols output by the final softmax layer <NUM>). For instance, the first position vector PVAa is assigned to a most recent position in the history, while the last position vector PVAn is assigned to a last position in the history of the N previous non-blank symbols output by the final softmax layer <NUM>. Notably, each of the embeddings <NUM> may include a same dimensionality (i.e., dimension size) as each of the position vectors PV <NUM>.

While the corresponding embedding generated by shared embedding matrix <NUM> for each for each non-blank symbol <NUM> among the sequence of non-blank symbols 252a-n, yu-n,. , yu-<NUM>, is the same at all of the heads 302A-H of the multi-headed attention mechanism <NUM>, each head 302A-H defines a different set/row of position vectors <NUM>. For instance, the first head 302A defines the row of position vectors PVAa-An308Aa-An, the second head 302B defines a different row of position vectors PVBa-Bn <NUM>Ba-Bn,. , and the Hth head <NUM> defines another different row of position vectors PVHa-Hn <NUM>Ha-Hn.

For each non-blank symbol in the sequence of non-blank symbols 252a-n received, the first head 302A also weights, via a weight layer <NUM>, the corresponding embedding <NUM> proportional to a similarity between the corresponding embedding and the respective position vector PV <NUM> assigned thereto. In some examples, the similarity may include a cosine similarity (e.g., cosine distance). In the example shown, the weight layer <NUM> outputs a sequence of weighted embeddings <NUM>, 312Aa-An each associated the corresponding embedding <NUM> weighted proportional to the respective position vector PV <NUM> assigned thereto. Stated differently, the weighted embeddings <NUM> output by the weight layer <NUM> for each embedding <NUM> may correspond to a dot product between the embedding <NUM> and the respective position vector PV <NUM>. The weighted embeddings <NUM> may be interpreted as attending over the embeddings in proportion to how similar they are to the positioned associated with their respective position vectors PV <NUM>. To increase computational speed, the prediction network <NUM> includes non-recurrent layers, and therefore, the sequence of weighted embeddings 312Aa-An are not concatenated, but instead, averaged by a weighted average module <NUM> to generate, as output from the first head 302A, a weighted average 318A of the weighted embeddings 312Aa-An represented by: <MAT>.

In Equation (<NUM>), h represents the index of the heads <NUM>, n represents position in context, and e represents the embedding dimension. Additionally, in Equation (<NUM>), H, N, and de include the sizes of the corresponding dimensions. The position vector PV <NUM> does not have to be trainable and may include random values. Notably, even though the weighted embeddings <NUM> are averaged, the position vectors PV <NUM> can potentially save position history information, alleviating the need to provide recurrent connections at each layer of the prediction network <NUM>.

The operations described above with respect to the first head 302A, are similarly performed by each other head 302B-H of the multi-headed attention mechanism <NUM>. Due to the different set of positioned vectors PV <NUM> defined by each head <NUM>, the weight layer <NUM> outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn at each other head 302B-H that is different than the sequence of weighted embeddings 312Aa-Aa at the first head 302A. Thereafter, the weighted average module <NUM> generates, as output from each other corresponding head 302B-H, a respective weighted average 318B-H of the corresponding weighted embeddings <NUM> of the sequence of non-blank symbols.

In the example shown, the prediction network <NUM> includes a head average module <NUM> that averages the weighted averages 318A-H output from the corresponding heads 302A-H. A projection layer <NUM> with SWISH may receive, as input, an output <NUM> from the head average module <NUM> that corresponds to the average of the weighted averages 318A-H, and generate, as output, a projected output <NUM>. A final layer normalization <NUM> may normalize the projected output <NUM> to provide the single embedding vector Pu <NUM> (i.e., hidden representation) at the corresponding time step from the plurality of time steps. The prediction network <NUM> generates only a single embedding vector Pu <NUM> at each of the plurality of time steps subsequent to an initial time step.

In some configurations, the prediction network <NUM> does not implement the multi-headed attention mechanism <NUM> and only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer <NUM> and layer normalization <NUM> to provide the single embedding vector Pu <NUM>.

Referring back to <FIG>, the word piece joint network <NUM> receives the single embedding vector Pu <NUM> from the prediction network <NUM>, and the higher-order feature representation <MAT> from the encoder <NUM>. The word piece joint network <NUM> generates a probability distribution <MAT> <NUM> over possible speech recognition hypotheses at the corresponding time step. For example, when the natural language is English, the set of output labels may include twenty-seven (<NUM>) symbols, e.g., one label for each of the <NUM>-letters in the English alphabet and one label designating a space. Accordingly, the word piece joint network <NUM> may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution of the word piece joint network <NUM> can include a posterior probability value for each of the different output labels. Thus, when there are <NUM> different output labels representing different graphemes or other symbols, the output <MAT> <NUM> of the word piece joint network <NUM> can include <NUM> different probability values, one for each output label. The probability distribution <MAT> over the possible speech recognition hypotheses indicates a probability for a speech recognition result <NUM> (<FIG>). That is, the joint network <NUM> determines the probability distribution for speech recognition results <NUM> using the single embedding vector <NUM> that is based on the sequence of non-blank symbols <NUM>. Stated differently, the word piece joint network <NUM> generates, at each output step (e.g., time step), a probability distribution <NUM> over possible speech recognition hypotheses. The probability distribution <MAT> <NUM> can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the softmax layer <NUM>) for determining the transcriptions <NUM>.

The softmax layer <NUM> may employ any technique to select the output label/symbol with the highest probability in the distribution <MAT> <NUM> as the next output symbol yu <NUM> predicted by the RNN-T model <NUM> at the corresponding output step. In this manner, the RNN-T model <NUM> does not make a conditional independence assumption. Instead, the RNN-T model <NUM> predicts each symbol conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model <NUM> does assume an output symbol is independent of future acoustic frames <NUM>, which allows the RNN-T model <NUM> to be employed in a streaming fashion. In some examples, the softmax layer <NUM> is composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes in a plurality of training data sets.

The first joint network (i.e., the disfluency detection joint network) <NUM> generates, based on the higher order feature representation <MAT> <NUM> output by the encoder network <NUM> and the single embedding vector Pu <NUM> output by the prediction network <NUM>, a probability distribution that the corresponding time step corresponds to a disfluency (e.g., a pause) or an end of speech event. Stated differently, the first joint network can output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels including <pause> for a pause, and <eos> for an end of speech event. This set of values can be a vector and can indicate a probability distribution over the set of output labels. The probability distribution providing the conditional probability of a pause and the conditional probability of an end of speech event can be determined using the following mathematical expressions: <MAT> <MAT> where yu is the output word piece hypothesis with the highest probability <MAT> <NUM>. The disfluency detection joint network <NUM> emits tokens <NUM> (e.g., <pause> and <eos> tokens) when the corresponding probability satisfies (e.g., exceeds) a predefined threshold. For example, when <MAT> satisfies (e.g., exceeds) an end of speech threshold an <eos> token <NUM> is emitted and when <MAT> satisfies (e.g., exceeds) a pause threshold a <pause> token <NUM> is emitted. The end of speech threshold and the pause threshold need not have the same value. In some examples, detection of an end of speech event (e.g., a probability that a corresponding time step corresponds to an end of speech satisfies an end of speech threshold) triggers a microphone closing event by the user device <NUM> by, for example, emitting an end of speech token <NUM> that causes the triggering of the microphone closing event.

With reference to <FIG> and <FIG>, in order to ensure the RNN-T 200a has the same speech recognition quality as conventional RNN-T, a training system <NUM> (<FIG>) trains the RNN-T <NUM> using a two-stage training processing. During a first stage, the training system <NUM> trainsthe encoder network <NUM>, the prediction network <NUM>, and the word piece joint network <NUM> on a speech recognition task to perform speech recognition. During a second stage, the training system <NUM> holds the parameters of the encoder network <NUM>, the prediction network <NUM>, and the word piece joint network <NUM> fixed, while initializing and fine-tuning (e.g., training) the disfluency detection joint network <NUM> to learn how to detect the presence of pauses and end of speech events. That is, the training system <NUM> trains the disfluency detection joint network <NUM> to learn how to predict pause and end of speech locations in utterances. The training system <NUM> trains the disfluency detection joint network <NUM> using a plurality of transcribed training utterances that has be annotated by, for example, a training data generator <NUM> (<FIG>) to include labels (e.g., <pause> and <eos>) indicating pause and end of speech locations. For example, the training data generator <NUM> can insert <pause> tokens for pauses, and insert <eos> tokens at the end of utterances.

For short-form utterances that contain a single voice query, the training data generator <NUM> appends <eos> tokens to the end of each utterance, and inserts <pause> tokens for silence segments determined, for example, using forced alignment. While short-form utterances can be used to model end of speech events and regular short pauses, they may not cover a broader range of possible disfluencies. Moreover, because only a single <eos> token is appended to the end of each utterance, the disfluency detection joint network <NUM> may learn to stop emitting any additional <eos> tokens after determining a first end of speech event in an utterance, which may cause disfluency detection problems for longer utterances or utterances with multiple disfluencies.

Referring back to <FIG>, the feature vectors x <NUM> input to the encoder network <NUM> may include <NUM>-dimensional log-Mel filter bank features formed by stacking three <NUM> millisecond (ms) acoustic frames with a <NUM> shift, and downsampling to a <NUM> frame rate. In some examples, the encoder network <NUM> includes twelve <NUM>-dimensional conformer layers. The conformer layers include causal convolution with a kernel size of <NUM>, and a stack of left-context attention layers with <NUM>-head self-attention. While the encoder network <NUM> described has a stack of multi-head attention layers/blocks with self-attention that include conformer layers/blocks (e.g., twelve conformer blocks), the present disclosure is not so limited. For instance, the encoder network <NUM> may include a stack of transformer layers/ or a stack of any other type of multi-head attention layers/bocks. The encoder network <NUM> may include a series of multi-headed self-attention, depth-wise convolutional and feed-forward layers. Alternatively, the encoder network <NUM> may include a plurality of long-short term memory (LSTM) layers in lieu of multi-head attention layers/blocks.

The prediction network <NUM> may include an LTSM-based network having an embedding dimension of <NUM>. The dimension Djoint of the fused representation <MAT> <NUM> may be set to <NUM>. In some examples, the word piece joint network <NUM> includes hidden units. Additionally or alternatively, the word piece joint network <NUM> does not include a fully connected (FC) layer. Alternatively, the prediction network <NUM> may include a stack of transformer or conformer blocks (or other type of multi-head attention blocks). The prediction network <NUM> may also be an embedding look-up table (e.g., a V2 embedding look-up table) to improve latency by outputting looked-up sparse embeddings in lieu of generating hidden representations. In some implementations, the prediction network <NUM> is a stateless prediction network.

The word piece joint network <NUM> and the prediction network <NUM> may collective form an RNN-T decoder of the RNN-T model <NUM>. In some implementations, to further reduce the size of the RNN-T decoder, i.e., the prediction network <NUM> and the word piece joint network <NUM>, parameter tying between the prediction network <NUM> and the word piece joint network <NUM> is applied. Specifically, for a vocabulary size |V| and an embedding dimension de, the shared embedding matrix <NUM> at the prediction network is <MAT>. Meanwhile, a last hidden layer includes a dimension size dh at the joint network <NUM>, feed-forward projection weights from the hidden layer to the output logits will be <MAT>, with an extra blank token in the vocabulary. Accordingly, the feed-forward layer corresponding to the last layer of the word piece joint network <NUM> includes a weight matrix [dh, |V]|. By having the prediction network <NUM> to tie the size of the embedding dimension de to the dimensionality dh of the last hidden layer of the word piece joint network <NUM>, the feed-forward projection weights of the word piece joint network <NUM> and the shared embedding matrix <NUM> of the prediction network <NUM> can share their weights for all non-blank symbols via a simple transpose transformation. Since the two matrices share all their values, the RNN-T decoder only needs to store the values once in memory, instead of storing two individual matrices. By setting the size of the embedding dimension de equal to the size of the hidden layer dimension dh, the RNN-T decoder reduces a number of parameters equal to the product of the embedding dimension de and the vocabulary size |V|. This weight tying corresponds to a regularization technique.

<FIG> is a schematic view of an example RNN-T model <NUM>, 200b integrating the disfluency detection model <NUM> as an acoustic-based turn taking detector <NUM> to compute, at each corresponding time step (i.e., for each acoustic frame <NUM> x(t), a probability distribution that the corresponding time step corresponds to the disfluency (e.g., pause) and the end of speech event. As shown, the RNN-T model 200b includes the encoder network <NUM>, the prediction/decoder network <NUM>, the word piece joint network <NUM>, and the final softmax output layer <NUM> of the RNN-T model 200a of <FIG>, but replaces the disfluency detection joint network <NUM> with the acoustic-based turn taking detection network <NUM>. Details of the encoder network <NUM>, the prediction/decoder network <NUM>, the word piece joint network <NUM>, and the final softmax output layer <NUM> are described above with reference to <FIG> and <FIG>.

The acoustic-based turn taking detection network <NUM> generates, for each input feature vector xt at time step t and based on the higher order feature representation <MAT> <NUM> produced by the encoder network <NUM> for input feature vectors xt, xt-<NUM>,. , xt-k <NUM>, the probability distribution that the corresponding time step corresponds to the pause and the end of speech event. Stated differently, the acoustic-based turn taking detection network <NUM> can output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels including <pause> for a pause, and <eos> for an end of speech event. This set of values can be a vector and can indicate a probability distribution over the set of output labels. The probability distribution providing the conditional probability of a pause and the conditional probability of an end of speech event can be determined using the following mathematical expressions: <MAT> <MAT>.

The turn taking detection network <NUM> emits tokens <NUM> (e.g., <pause> and <eos> tokens) when the corresponding probability satisfies (e.g., exceeds) a predefined threshold. For example, when <MAT> satisfies (e.g., exceeds) an end of speech threshold an <eos> token <NUM> is emitted and when <MAT> satisfies (e.g., exceeds) a pause threshold a <pause> token <NUM> is emitted. The end of speech threshold and the pause threshold need not have the same value. In some examples, detection of an end of speech event (e.g., a probability that a corresponding time step corresponds to an end of speech satisfies an end of speech threshold) triggers a microphone closing event by the user device <NUM> by, for example, emitting an end of speech token <NUM> that causes the triggering of the microphone closing event.

By sharing the encoder network <NUM>, the RNN-T 200a synchronizes speech recognition by the RNN-T decoder, i.e., the prediction network <NUM> and the word piece joint network <NUM>, with turn taking detection by the acoustic-based turn taking detection network <NUM>, which helps ensure correct interactions for natural conversational inputs. Moreover, because the turn taking detection network <NUM> reuses the encoder network <NUM>, it is not necessary to configure or include another separate encoder, which reduces computational complexity. Because delayed or late disfluency detection may introduce slow responses to queries and commands, some examples apply an emission regularization method (e.g., FastEmit) to the turn taking detection network <NUM> to reduce disfluency detection delays. To ensure the RNN-T 200b has the same speech recognition quality as conventional RNN-T, the training system <NUM> (<FIG>) trains the RNN-T 200b in two stages in a similar manner as described above with reference to the RNN-T 200a of <FIG>.

<FIG> is a schematic view of an example RNN-T model <NUM>, 200c integrating the disfluency detection model <NUM> as a semantic-based turn taking detection network <NUM> to compute, at each corresponding time step, a probability distribution that the corresponding time step corresponds to the disfluency (e.g., pause) and the end of speech event. As shown, the RNN-T model 200b includes the encoder network <NUM>, the prediction/decoder network <NUM>, the word piece joint network <NUM>, and the final softmax output layer <NUM> of the RNN-T model 200a of <FIG>, but replaces the disfluency detection joint network <NUM> with the acoustic-based turn taking detection network <NUM>. Details of the encoder network <NUM>, the prediction/decoder network <NUM>, the word piece joint network <NUM>, and the final softmax output layer <NUM> are described above with reference to <FIG> and <FIG>.

The semantic-based turn taking detection network <NUM> generates, for each time step and based on a past sequence of output symbols y<NUM>, y<NUM>,. , yu <NUM> output by the softmax layer <NUM>, the probability distribution that the corresponding time step corresponds to the disfluency (e.g., a pause) and the end of speech event. Stated differently, the semantic-based turn taking detection network <NUM> can output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels including <pause> for a pause, and <eos> for an end of speech event. This set of values can be a vector and can indicate a probability distribution over the set of output labels. The probability distribution providing the conditional probability of a pause and the conditional probability of an end of speech event can be determined using the following mathematical expressions: <MAT> <MAT>.

In some examples, the semantic-based turn taking detection network <NUM> includes a language model (LM) that determines probabilities that a next output symbol (e.g., sub-word unit) corresponds to a <pause> and an <eos>. The semantic-based turn taking detection network <NUM> emits tokens <NUM> (e.g., <pause> and <eos> tokens) when the corresponding probability satisfies (e.g., exceeds) a predefined threshold. For example, when <MAT> satisfies (e.g., exceeds) an end of speech threshold an <eos> token <NUM> is emitted and when <MAT> satisfies (e.g., exceeds) a pause threshold a <pause> token <NUM> is emitted. The end of speech threshold and the pause threshold need not have the same value. In some examples, detection of an end of speech event (e.g., a probability that a corresponding time step corresponds to an end of speech satisfies an end of speech threshold) triggers a microphone closing event by the user device <NUM> by, for example, emitting an end of speech token <NUM> that causes the triggering of the microphone closing event. To ensure the RNN-T 200c has the same speech recognition quality as conventional RNN-T, the training system <NUM> (<FIG>) trains the RNN-T 200c in two stages in a similar manner as described above with reference to the RNN-T 200a of <FIG>.

<FIG> depicts an example transcribed long-form training utterance <NUM>. <FIG> depicts an example annotated transcribed training utterance <NUM> generated by the training data generator <NUM> for the transcribed long-form training utterance <NUM> of <FIG>. For long-form utterances (e.g., tens of seconds long and/or containing multiple speech segments including more natural conversational voice inputs and interactions), the training data generator <NUM> determines silence segments <NUM>, 402a-n using, for example, forced alignment, and determines sentence boundaries based on silence durations. For example, the training data generator <NUM> labels short silent pauses (e.g., silent pauses 402a and 402b) with respective <pause> tokens (e.g., tokens 452a and 452b), at least initially labels long silent pauses (e.g., silent pauses 402c and 402d) with respective <eos> tokens, and labels a final silent pause (e.g., silent pause 402e) with an <eos> token (e.g., token 452e). In some examples, a silent pause is determined to be short or long by comparing the duration of the silent pause to a pre-determined threshold. For example, the training data generator <NUM> classifies silent pause durations less than the pre-determined threshold as short, and classifies silent pause durations greater than the pre-determined threshold as long. However, labeling long silent pauses in this way may cause the training data generator <NUM> to incorrectly label some long silent pauses (e.g., the silent pauses 402c and 402d) with <eos> tokens. Accordingly, the training data generator <NUM> re-labels silent pauses (e.g., the silent pause 402c) of any length that follow a hesitation word <NUM> (a filler, repeated phrase, etc.) with <pause> tokens (e.g., token 452c). Moreover, the training data generator <NUM> re-labels silent pauses (e.g., the silent pause 402d) of any length that follow word lengthening <NUM> with <pause> tokens (e.g., token 452d). In some examples, the training data generator <NUM> determines word lengthening when a last phoneme of a word or word piece has a duration that satisfies a criteria (e.g., exceeds <NUM> standard deviations as pre-computed for the phoneme).

<FIG> is a flowchart of an exemplary arrangement of operations for a computer-implemented method <NUM> for detecting disfluencies and performing speech recognition. At operation <NUM>, the method <NUM> includes receiving a sequence of acoustic frames (e.g., the feature vectors x = (x<NUM>, x<NUM>,· · · , xt) <NUM>, <NUM>) characterizing one or more utterances (e.g., the utterance <NUM>).

At each of a plurality of time steps, the method <NUM> performs operations <NUM>, <NUM>, <NUM> and <NUM>. At operation <NUM>, the method <NUM> includes generating, by an encoder network <NUM> of a speech recognition model (e.g., the RNN-T <NUM>), a higher order feature representation <MAT> <NUM> for a corresponding acoustic frame in the sequence of acoustic frames.

At operation <NUM>, the method <NUM> includes generating, by a prediction network <NUM> of the speech recognition model, a hidden representation (e.g., the single embedding vector Pu <NUM>) for a corresponding sequence of non-blank symbols y<NUM>, y<NUM>,. , yu <NUM> output by a final softmax layer <NUM> of the speech recognition model.

At operation <NUM>, the method <NUM> includes generating, by a first joint network (e.g., the disfluency detection joint network) <NUM> of the speech recognition model that receives the higher order feature representation generated by the encoder network and the dense representation generated by the prediction network, a probability distribution that the corresponding time step corresponds to a pause and an end of speech.

At operation <NUM>, the method <NUM> includes generating, by a second joint network (e.g., the word piece joint network) <NUM> of the speech recognition model, a probability distribution <MAT> <NUM> over possible speech recognition hypotheses.

<FIG> is schematic view of an example computing device <NUM> that can be used to implement the systems and methods described in this document. The computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computer devices.

The computing device <NUM> includes a processor <NUM> (i.e., data processing hardware) that can be used to implement the data processing hardware <NUM> and/or <NUM>, memory <NUM> (i.e., memory hardware) that can be used to implement the memory hardware <NUM> and/or <NUM>, a storage device <NUM> (i.e., memory hardware) that can be used to implement the memory hardware <NUM> and/or <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>.

Claim 1:
A computer-implemented method (<NUM>) when executed on data processing hardware (<NUM>) causes the data processing hardware (<NUM>) to perform operations comprising:
receiving a sequence of acoustic frames (<NUM>) characterizing one or more utterances (<NUM>); and
at each of a plurality of time steps:
generating, by an encoder network (<NUM>) of a speech recognition model (<NUM>), a higher order feature representation (<NUM>) for a corresponding acoustic frame in the sequence of acoustic frames (<NUM>);
generating, by a prediction network (<NUM>) of the speech recognition model (<NUM>), a hidden representation (<NUM>) for a corresponding sequence of non-blank symbols (<NUM>) output by a final softmax layer (<NUM>) of the speech recognition model (<NUM>); and
generating, by a first joint network (<NUM>) of the speech recognition model (<NUM>) that receives the higher order feature representation (<NUM>) generated by the encoder network (<NUM>) and the hidden representation (<NUM>) generated by the prediction network (<NUM>), a probability distribution that the corresponding time step corresponds to a pause and an end of speech.