Semantic Segmentation With Language Models For Long-Form Automatic Speech Recognition

A joint segmenting and ASR model includes an encoder to receive a sequence of acoustic frames and generate, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame. The model also includes a decoder to generate based on the higher order feature representation at each of the plurality of output steps a probability distribution over possible speech recognition hypotheses, and an indication of whether the corresponding output step corresponds to an end of segment (EOS). The model is trained on a set of training samples, each training sample including audio data characterizing multiple segments of long-form speech; and a corresponding transcription of the long-form speech, the corresponding transcription annotated with ground-truth EOS labels obtained via distillation from a language model teacher that receives the corresponding transcription as input and injects the ground-truth EOS labels into the corresponding transcription between semantically complete segments.

TECHNICAL FIELD

This disclosure relates to semantic segmentation with language models for long-form automated speech recognition (ASR).

BACKGROUND

Automatic speech recognition (ASR) is the process of transcribing input audio into text. ASR is an increasingly important technology that may be used to enable a user to interact with mobile or other devices using spoken (i.e., speech-based) interactions.

SUMMARY

One aspect of the disclosure provides a joint segmenting and automated speech recognition (ASR) model that includes an encoder and a decoder. The encoder is configured to receive, as input, a sequence of acoustic frames characterizing one or more spoken utterances, and generate, at each output step of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The decoder is configured to receive, as input, the higher order feature representation generated by the encoder at each output step of the plurality of output steps. The decoder is configured to generate, at each output step of the plurality of output steps, a probability distribution over possible speech recognition hypotheses, and an indication of whether the output step corresponds to an end of segment. The joint segmenting and ASR model is trained on a set of training samples, each training sample in the set of training samples including audio data characterizing multiple segments of long-form speech, and a corresponding transcription of the long-form speech. The corresponding transcription is annotated with ground-truth end of segment labels obtained via distillation from a language model teacher that receives the corresponding transcription as input and injects the ground-truth end of segment labels into the corresponding transcription between semantically complete segments.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the language model teacher is trained on a corpus of written text containing punctuation to teach the language model teacher to learn how to semantically predict ground-truth end of segment labels based on positions of punctuation in the written text. In some examples, the language model teacher includes a bi-directional recurrent neural network architecture.

In some examples, the decoder includes a prediction network configured to, at each output step of the plurality of output steps, receive, as input, a sequence of non-blank symbols output by a final Softmax layer, and generate a hidden representation In these examples, the decoder also includes a first joint network configured to: receive, as input, the hidden representation generated by the prediction network at each output step of the plurality of output steps and the higher order feature representation generated by the encoder at each output step of the plurality of output steps; and generate, at each output step of the plurality of output steps, the probability distribution over possible speech recognition hypotheses. The decoder further includes a second joint network configured to: receive, as input, the hidden representation generated by the prediction network at each output step of the plurality of output steps and the higher order feature representation generated by the encoder at each output step of the plurality of output steps; and generate, at each of output step the plurality of output steps, the indication of whether the output step corresponds to an end of segment.

In some implementations, at each output step of the plurality of output steps: the sequence of previous non-blank symbols received as input at the prediction network includes a sequence of N previous non-blank symbols output by the final Softmax layer; and the prediction network is configured to generate the hidden representation by: for each non-blank symbol of the sequence of N previous non-blank symbols, generating a respective embedding; and generating an average embedding by averaging the respective embeddings, the average embedding including the hidden representation. In some examples, the prediction network includes a V2embedding look-up table. In some implementations, a training process trains the joint segmenting and ASR model on the set of training samples by: initially training the first joint network to learn how to predict the corresponding transcription of the spoken utterance characterized by the audio data of each training sample; and after training the first joint network, initializing the second joint network with the same parameters as the trained first joint network and using the ground-truth end of segment label inserted into the corresponding transcription of the spoken utterance characterized by the audio data of each training sample.

In some examples, the encoder includes a causal encoder including a stack of conformer layers or transformer layers. In some implementations, the ground-truth end of segment labels are inserted into the corresponding transcription automatically without any human annotation. In some examples, the joint segmenting and ASR model is trained to maximize a probability of emitting the ground-truth end of segment label.

Another aspect of the disclosure provides a computer-implemented method executed on data processing hardware that causes the data processing hardware to implement a joint segmenting and automated speech recognition (ASR) model, the joint segmenting and ASR model including an encoder and a decoder. The encoder is configured to receive, as input, a sequence of acoustic frames characterizing one or more spoken utterances, and generate, at each output step of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The decoder is configured to receive, as input, the higher order feature representation generated by the encoder at each output step of the plurality of output steps. The decoder is configured to generate, at each output step of the plurality of output steps a probability distribution over possible speech recognition hypotheses, and an indication of whether the output step corresponds to an end of segment. The joint segmenting and ASR model is trained on a set of training samples, each training sample in the set of training samples including audio data characterizing multiple segments of long-form speech, and a corresponding transcription of the long-form speech. The corresponding transcription is annotated with ground-truth end of segment labels obtained via distillation from a language model teacher that receives the corresponding transcription as input and injects the ground-truth end of segment labels into the corresponding transcription between semantically complete segments.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the language model teacher is trained on a corpus of written text containing punctuation to teach the language model teacher to learn how to semantically predict ground-truth end of segment labels based on positions of punctuation in the written text. In some examples, the language model teacher includes a bi-directional recurrent neural network architecture.

In some examples, the decoder includes a prediction network configured to, at each output step of the plurality of output steps, receive, as input, a sequence of non-blank symbols output by a final Softmax layer, and generate a hidden representation In these examples, the decoder also includes a first joint network configured to: receive, as input, the hidden representation generated by the prediction network at each output step of the plurality of output steps and the higher order feature representation generated by the encoder at each output step of the plurality of output steps; and generate, at each output step of the plurality of output steps, the probability distribution over possible speech recognition hypotheses. The decoder further includes a second joint network configured to: receive, as input, the hidden representation generated by the prediction network at each output step of the plurality of output steps and the higher order feature representation generated by the encoder at each output step of the plurality of output steps; and generate, at each of output step the plurality of output steps, the indication of whether the output step corresponds to an end of segment.

In some implementations, at each output step of the plurality of output steps: the sequence of previous non-blank symbols received as input at the prediction network includes a sequence of N previous non-blank symbols output by the final Softmax layer; and the prediction network is configured to generate the hidden representation by: for each non-blank symbol of the sequence of N previous non-blank symbols, generating a respective embedding; and generating an average embedding by averaging the respective embeddings, the average embedding including the hidden representation. In some examples, the prediction network includes a V2embedding look-up table. In some implementations, a training process trains the joint segmenting and ASR model on the set of training samples by: initially training the first joint network to learn how to predict the corresponding transcription of the spoken utterance characterized by the audio data of each training sample; and after training the first joint network, initializing the second joint network with the same parameters as the trained first joint network and using the ground-truth end of segment label inserted into the corresponding transcription of the spoken utterance characterized by the audio data of each training sample.

In some examples, the encoder includes a causal encoder including a stack of conformer layers or transformer layers. In some implementations, the ground-truth end of segment labels are inserted into the corresponding transcription automatically without any human annotation. In some examples, the joint segmenting and ASR model is trained to maximize a probability of emitting the ground-truth end of segment label.

DETAILED DESCRIPTION

Automatic speech recognition (ASR) is the process of transcribing input audio into text. ASR is an increasingly important technology that may be used to enable a user to interact with mobile or other devices using spoken (i.e., speech-based) interactions. Recognizing long-form speech (e.g., minutes long) in short segments of a few or several seconds is a common practice for improving ASR accuracy and user-perceived latency. Model state may be wholly or partially discarded across segment boundaries, which may help to prevent a speech recognizer from entering strange states unseen during short-form training and make room for more diversity in beam search hypotheses. Conventional segment boundary classifiers rely on characteristics of input audio (e.g., periods of silence) to delineate segments of long-form speech. However, silence does not always accurately demarcate complete thoughts, as speakers may hesitate before finishing a sentence in real-world speech. Accordingly, there is a need for improved segmentation of long-form speech.

In disclosed implementations, an ASR model includes a semantic segment boundary classifier that is trained to predict semantic segment boundaries during speech recognition for long-form speech. The ASR model then uses the predicted semantic segment boundaries to segment the long-form speech into segments for speech recognition purposes. Here, semantic segmentation may refer to the use of punctuation to logically understand the meaning of long-form speech such that the long-form speech can be segmented into segments that contain complete thoughts for speech recognition purposes. Because ground-truth transcriptions used to train an ASR model rarely contain punctuation, the semantic segment boundary classifier is trained, using a bidirectional language model, to predict segment boundaries (e.g., complete thought boundaries) in long-form speech. Here, the bidirectional language model may be trained on a large corpus of written text to learn to predict the punctuation contained in the corpus of written text. The bidirectional language model is then used as a teacher model to predict semantic segment boundaries in ground-truth training transcriptions based on the predicted punctuation. End of segment (EOS) labels corresponding to segment boundaries predicted by the bidirectional language model are then inserted into the ground-truth training transcriptions. The ground-truth training transcriptions and corresponding training utterances are then used to train the semantic segment boundary classifier as a student model to predict the segment boundaries in the ground-truth training transcriptions.

FIG.1is an example system100that includes one or more users104interacting with a user device10through voice input. The user device10(also referred to generally as a user device10) is configured to capture sounds (e.g., streaming audio data110) from the one or more users104within the system100. Here, the streaming audio data110may refer to an utterance106spoken by the user104that functions as an audible query, a command for the user device10, or an audible communication captured by the user device10. Speech-enabled systems of the user device10may field the query or the command by answering the query and/or causing the command to be performed/fulfilled by one or more downstream applications.

The user device10may correspond to any computing device associated with the user104and capable of receiving audio data. Some examples of user devices10include, but are not limited to, mobile devices (e.g., smart watches), smart appliances, internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device10includes data processing hardware12and memory hardware14in communication with the data processing hardware12and stores instructions that, when executed by the data processing hardware12, cause the data processing hardware12to perform one or more operations. The user device10further includes an audio system16with an audio capture device16a(e.g., a microphone) for capturing and converting the utterances106into electrical signals and a speech output device16b(e.g., a speaker) for communicating with an audible audio signal (e.g., as output data from the user device10). The user device10may implement an array of audio capture devices16awithout departing from the scope of the present disclosure, whereby one or more capture devices16ain the array may not physically reside on the user device10, but be in communication with the audio system16.

The system100includes an automated speech recognition (ASR) system118that implements a joint segmenting and ASR model200(also referred to herein as ASR model200) and resides on the user device10of the user104and/or on a remote computing system60(e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device10via a network40. As described below in connection withFIG.2, the ASR model includes a semantic segment boundary classifier230to semantically identify segments of long-form speech for ASR processing. The remote computing system60may include physical and/or virtual (e.g., cloud based) resources, such as data processing hardware62(e.g., remote servers or CPUs) and/or memory hardware64(e.g., remote databases or other storage hardware). The memory hardware64is in communication with the data processing hardware62and stores instructions that, when executed by the data processing hardware62, cause the data processing hardware62to perform one or more operations.

The user device10and/or the remote computing system60also includes an audio subsystem108configured to receive the utterance106spoken by the user104and captured by the audio capture device16a, and convert the utterance106into a corresponding digital format associated with input acoustic frames110capable of being processed by the ASR system118. In the example shown, the user speaks a respective utterance106and the audio subsystem108converts the utterance106into a corresponding sequence of acoustic frames110for input to the ASR system118. Thereafter, the ASR model200receives, as input, the sequence of acoustic frames110corresponding to the utterance106, and generates/predicts, at each output step, a corresponding transcription120(e.g., speech recognition result/hypothesis) of the utterance106as the ASR model200receives (e.g., processes) each acoustic frame110in the sequence of acoustic frames110.

In the example shown, the ASR model200may perform streaming speech recognition to produce an initial speech recognition result120,120aand generate a final speech recognition result120,120bby improving the initial speech recognition result120a. The speech recognition results120may either correspond to a partial speech recognition result or an entire speech recognition result. Stated differently, the speech recognition result120may either correspond to a portion of an utterance106or an entire utterance106. For example, the partial speech recognition result may correspond to a portion of a spoken utterance or even a portion of a spoken term. However, as will become apparent, the ASR model200may perform additional processing on the final speech recognition result120bwhereby the final speech recognition result120bmay be delayed from the initial speech recognition result120a.

The user device10and/or the remote computing system60also executes a user interface generator107configured to present a representation of the transcription120of the utterance106to the user104of the user device10. As described in greater detail below, the user interface generator107may display the initial speech recognition results120ain a streaming fashion during time1and subsequently display the final speech recognition results120bin a streaming fashion during time2. In some configurations, the transcription120output from the ASR system118is processed, e.g., by a natural language understanding (NLU) or natural language processing (NLP) module executing on the user device10or the remote computing system60, to execute a user command/query specified by the utterance106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device10or the remote computing system60) may convert the transcription120into synthesized speech for audible output by the user device10and/or another device.

In the example shown, the user104interacts with a digital assistant application50or other program of the user device10that uses the ASR system118. For instance,FIG.1depicts the user104communicating with the digital assistant application50and the digital assistant application50displaying a digital assistant interface17on a screen18of the user device10to depict a conversation between the user104and the digital assistant application50. In this example, the user104asks the digital assistant application50, “What time is the concert tonight?” This question from the user104is a spoken utterance106captured by the audio capture device16aand processed by audio subsystem108of the user device10. In this example, the audio subsystem108receives the spoken utterance106and converts it into a sequence of acoustic frames110for input to the ASR system118.

Continuing with the example, the ASR model200, while receiving the sequence of acoustic frames110corresponding to the utterance106as the user104speaks, encodes the sequence of acoustic frames110and then decodes the encoded sequence of acoustic frames110into the initial speech recognition results120a. During time1, the user interface generator107presents, via the digital assistant interface17, a representation of the initial speech recognition results120aof the utterance106to the user104of the user device10in a streaming fashion such that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken. In some examples, the first look ahead audio context is equal to zero.

During time2, the user interface generator107presents, via the digital assistant interface17, a representation of the final speech recognition results120bof the utterance106to the user104of the user device10in a streaming fashion such that words, word pieces, and/or individual characters appear on the screen as soon as they are generated by the ASR model200. In some implementations, the user interface generator107replaces the representation of the initial speech recognition results120apresented at time1with the representation of the final speech recognition results120bpresented at time2. Here, time1and time2may include timestamps corresponding to when the user interface generator107presents the respective speech recognition result120. In this example, the timestamp of time1indicates that the user interface generator107presents the initial speech recognition results120aat an earlier time than the final speech recognition results120b. For instance, as the final speech recognition result120bis presumed to be more accurate than the initial speech recognition result120a, the final speech recognition result120bultimately displayed as the transcription120may fix any terms that may have been misrecognized in the initial speech recognition results120a. In this example, the streaming initial speech recognition results120aoutput by the ASR model200are displayed on the screen of the user device10at time1are associated with low latency and provide responsiveness to the user104that his/her query is being processed, while the final speech recognition result120boutput by the ASR model200and displayed on the screen at time2leverages an additional speech recognition model and/or a language model to improve the speech recognition quality in terms of accuracy, but at increased latency. However, since the initial speech recognition results120aare displayed as the user speaks the utterance106, the higher latency associated with producing, and ultimately displaying the final speech recognition results120bis not noticeable to the user104.

The final speech recognition result120bis presumed to be more accurate than the initial speech recognition result120abecause the ASR model200determines the initial speech recognition results120ain a streaming fashion and the final speech recognition results120busing the prior non-blank symbols from the initial speech recognition result120a. That is, the final speech recognition results120btake into account the prior non-blank symbols and, thus, are presumed more accurate because the initial speech recognition results120ado not take into account any prior non-blank symbols. Moreover, a rescorer (not shown for clarity of illustration) may update the initial speech recognition result120awith the final speech recognition result120bto provide the transcription via the user interface generator107to the user104.

In the example shown inFIG.1, the digital assistant application50may respond to the question posed by the user104using NLP or NLU. NLP/NLU generally refer to a process of interpreting written language (e.g., the initial speech recognition result120aand/or the final speech recognition result120b) and determining whether the written language prompts any action. In this example, the digital assistant application50uses NLP/NLU to recognize that the question106from the user104regards the user's schedule and more particularly a concert on the user's schedule. By recognizing these details with NLP/NLU, the automated assistant returns a response19to the user's query where the response19states, “Venue doors open at 6:30 PM and concert starts at 8 pm.” In some configurations, NLP/NLU occurs on the remote computing system60in communication with the data processing hardware12of the user device10.

FIG.2depicts an example ASR model200that includes a Recurrent Neural Network-Transducer (RNN-T) model architecture. The use of the RNN-T model architecture is exemplary only, and the ASR model200may include other architectures such as transformer-transducer and conformer-transducer model architectures, among others. The RNN-T model architecture provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device10(e.g., no communication with a remote computing system or server is required).

As shown, the ASR model200includes a shared encoder network210, a first decoder220a, a semantic segment boundary classifier230that includes a second decoder220b, and a final Softmax layer240. Here, the encoder network210and the first decoder220aform a first RNN-T model, and the encoder network210and the second decoder220bform a second RNN-T model. The first decoder220agenerates, at each of a plurality of output steps, a probability distribution224aover possible speech recognition hypotheses. The second decoder220bgenerates, at each of the plurality of output steps, an EOS indication232of whether the corresponding output step corresponds to an EOS. In some examples, the decoders220together form a decoder that generates, at each of a plurality of output steps, a probability distribution over possible speech recognition hypotheses, and an indication of whether the corresponding output step corresponds to an EOS.

In the illustrated example, the encoder network210includes a cascading encoder network that includes two encoders212a,212bthat cascade such that the output214aof the first encoder212afeeds the input of the second encoder212bprior to decoding. However, other encoder networks210may be used. Here, the first encoder212aand the second encoder212bmay be cascaded irrespective of the underlying architecture of each encoder. The encoders212may each include a stack of multi-head self-attention layers.

In some examples, the first encoder212aincludes a causal encoder having one of a plurality of unidirectional (LSTM) layers, a plurality of conformer layers, a plurality of transformer layers. For example, the first encoder212amay include nine (9) conformer layers each having a multi-headed (e.g., eight (8) heads) self-attention mechanism and a convolutional kernel size of fifteen (15). Moreover, the first encoder212amay perform a concatenation operation after a third conformer layer to achieve a time reduction rate of two whereby the resulting 1024-dimensional vectors are transformed by a fourth conformer layer and then projected back to a 512-dimensional vector using another linear transformation. Thereafter, another eight (5) conformer layers are followed by a final normalization layer. Thus, the first encoder212amay include 57 million parameters. Each layer of the first encoder212areceives zero right-context (e.g., receives zero future acoustic frames). The first encoder212amay include a plurality of multi-head attention layers other than conformer or transformer layers in other examples.

In some examples, the second encoder212bincludes a non-causal encoder having one of one or more bi-directional LSTM layers, a plurality of conformer layers, or a plurality of transformer layers. For instance, the second encoder212bmay include six (6) conformer layers of 640-dimensions and a final linear normalization layer thereby resulting in 117 million parameters. The second encoder212bmay receive additional right-context, for example a total of 15-right context frames across all layers to provide 900 milliseconds of additional right context. The second encoder212bmay include a plurality of multi-head attention layers other than conformer or transformer layers in other examples.

The first encoder212areceives a sequence of d-dimensional feature vectors (e.g., sequence of acoustic frames110) x=(x1, x2, . . . , XT), where xt∈ Rd. Here, each sequence of acoustic frames110characterizes a spoken utterance106. The first encoder212agenerates, at each output step of a plurality of output steps, a first higher order feature representation214afor a corresponding acoustic frame110in the sequence of acoustic frames110. Similarly, the second encoder212bis connected in cascade to the first encoder212aand receives, as input, the first higher order feature representation214aand generates, at each output step, a second higher order feature representation214bfor a corresponding first higher order feature representation214a. In some instances, the second encoder212bgenerates a second higher order feature representation214bfrom the first higher order feature representation214awithout receiving any of the acoustic frames110as input. In these instances, the second encoder212bgenerates the second higher order feature representations214busing only the first higher order feature representation214aas input. That is, the first higher order feature representations214areceived from the first encoder212aserves as additional right-context. The first encoder212aoutputs the first higher order feature representations214ato the second encoder212band the first decoder220awhile the second encoder212boutputs the second higher order feature representations214bto the second decoder220b.

In the illustrated example, the first decoder220aincludes a prediction network300and a joint network222a, and the second decoder220bincludes the prediction network300and a joint network222b. While the first and second decoders220a,220bshare a common prediction network300, the first decoder220aand the second decoder220bmay each include a separate respective prediction network300. In some implementations, the decoders220are trained separately. The decoder220acan be trained using, for example, any suitable RNN-T training process for training an ASR model. An example process for training the semantic segment boundary classier230is described below in connection withFIG.4.

The prediction network300may include a LSTM network and, like a language model (LM), receive, as input, a respective sequence of non-blank symbols242output by a final Softmax layer240and generate, at each output step, a dense representation350. In the example shown, the joint network222ais not conditioned on the outputs224bof the other joint network222b, and the joint network222bis not conditioned the outputs224aof the other joint network222a. As described in greater detail below, the representations350may include a single embedding vector. Notably, the sequence of non-blank symbols242received at a prediction network300captures linguistic dependencies between non-blank symbols242predicted during the previous output steps so far to assist a corresponding joint network222in predicting the probability of a next output symbol or blank symbol during the current output step. As described in greater detail below, to contribute to techniques for reducing the size of the prediction network300without sacrificing accuracy/performance of the ASR model200, a prediction network300may receive a limited-history sequence of non-blank symbols242yui-n, . . . , yui-1that is limited to the N previous non-blank symbols242output by the final Softmax layer240.

Each joint network222combines a respective higher-order feature representation214produced by the encoder network210and the representation350(i.e., single embedding vector350) produced by the prediction network300. Each joint network222predicts a distribution Zi=P(yi|xti, y0, . . . , Yut−1)224over the next output symbol. Stated differently, each joint network222generates, at each output step, a respective probability distribution224over possible speech recognition hypotheses. Here, the “possible speech recognition hypotheses” correspond to a set of output labels each representing a symbol/character in a specified natural language. The joint network222also generates, at each output step, a respective EOS indication232of whether the corresponding output step corresponds to an EOS. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, e.g., one label for each of the 26-letters in the English alphabet and one label designating a space. Accordingly, the joint networks220may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution224of a joint network222can include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output224of a joint network222can include 100 different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer240) for determining the transcription120.

The semantic segment boundary classifier230receives the second higher order feature representation214bgenerated by the second encoder212bat each of a plurality of output steps, and generates, at each output step, an EOS indication232of whether the current output step corresponds to an EOS. In some implementations, the semantic segment boundary classifier230outputs an EOS indication232when the posterior probability associated with predicting an EOS satisfies (e.g., falls below) a preset or predetermined threshold. In some examples, the semantic segment boundary classifier230is trained to directly predict EOS tokens. Additionally or alternatively, the semantic segment boundary classifier230may be trained to predict punctuation for a predicted transcription, and then to predict end of segments based on the predicted punctuation. Notably, the semantic segment boundary classifier230is trained to make both semantic segment boundary predictions and to predict a distribution224bover possible speech recognition hypotheses for a next output symbol.

The final Softmax layer240receives the probability distribution224afor the final speech recognition result120band selects the output label/symbol with the highest probability to produce the transcription120. For long-form speech, when an EOS indication232corresponding to a predicted semantic segment boundary is output by the semantic segment boundary classifier230, the Softmax layer240selects the output label/symbol with the highest probability to produce the transcription120. In some implementations, the states of the encoder network210and the decoders220are then reset, the beam search is then reset, and all hypotheses are discarded. Alternatively, the state of the encoder network210and the state of the decoder220for the top hypothesis224aselected by the Softmax layer240are retained. The final Softmax layer240may employ any technique to select the output label/symbol with the highest probability in the distribution224a. In this manner, the first decoder220adoes not make a conditional independence assumption, rather the prediction of each symbol yu242is conditioned not only on the acoustics but also on the sequence of labels242yu-n, . . . , yui−1output so far. The first decoder220adoes assume an output symbol242is independent of future acoustic frames110, which allows the ASR model200to be employed in a streaming fashion.

FIG.3is a schematic view of an example prediction network300for the ASR model200. The prediction network300receives, as input, a sequence of non-blank symbols242a-nyui-n, . . . , yui−1that is limited to the N previous non-blank symbols242a-noutput by the final Softmax layer240. In some examples, N is equal to two. In other examples, N is equal to five, however, the disclosure is non-limiting and N may equal any integer. The sequence of non-blank symbols242a-nindicates an initial speech recognition result120a(FIG.1). In some implementations, the prediction network300includes a multi-headed attention mechanism302that shares a shared embedding matrix304across each head302A-302H of the multi-headed attention mechanism. In one example, the multi-headed attention mechanism302includes four heads. However, any number of heads may be employed by the multi-headed attention mechanism302. Notably, the multi-headed attention mechanism improves performance significantly with minimal increase to model size. As described in greater detail below, each head302A-H includes its own row of position vectors308, and rather than incurring an increase in model size by concatenating outputs318A-H from all the heads, the outputs318A-H are instead averaged by a head average module322.

Referring to the first head302A of the multi-headed attention mechanism302, the head302A generates, using the shared embedding matrix304, a corresponding embedding306,306a-n(e.g., X ∈N×de) for each non-blank symbol among the sequence of non-blank symbols242a-nyui-n, . . . , yui−1received as input at the corresponding output step from the plurality of output steps. Notably, since the shared embedding matrix304is shared across all heads of the multi-headed attention mechanism302, the other heads302B-H all generate the same corresponding embeddings306for each non-blank symbol. The head302A also assigns a respective position vector PVAa-An308,308Aa-An (e.g., P ∈H×N×de) to each corresponding non-blank symbol in the sequence of non-blank symbols242a-nyui-n, . . . , yui−1. The respective position vector PV308assigned to each non-blank symbol indicates a position in the history of the sequence of non-blank symbols (e.g., the N previous non-blank symbols242a-noutput by the final Softmax layer230). For instance, the first position vector PVAais assigned to a most recent position in the history, while the last position vector PVAnis assigned to a last position in the history of the N previous non-blank symbols output by the final Softmax layer240. Notably, each of the embeddings306may include a same dimensionality (i.e., dimension size) as each of the position vectors PV308.

While the corresponding embedding generated by shared embedding matrix304for each for each non-blank symbol among the sequence of non-blank symbols242a-nyui-n, . . . , yui−1, is the same at all of the heads302A-H of the multi-headed attention mechanism302, each head302A-H defines a different set/row of position vectors308. For instance, the first head302A defines the row of position vectors PVAa-An308Aa-An, the second head302B defines a different row of position vectors PVBa-Bn308Ba-Bn, . . . , and the Hthhead302H defines another different row of position vectors PVHa-Hn308Ha-Hn.

For each non-blank symbol in the sequence of non-blank symbols242a-nreceived, the first head302A also weights, via a weight layer310, the corresponding embedding306proportional to a similarity between the corresponding embedding and the respective position vector PV308assigned thereto. In some examples, the similarity may include a cosine similarity (e.g., cosine distance). In the example shown, the weight layer310outputs a sequence of weighted embeddings312,312Aa-An each associated the corresponding embedding306weighted proportional to the respective position vector PV308assigned thereto. Stated differently, the weighted embeddings312output by the weight layer310for each embedding306may correspond to a dot product between the embedding306and the respective position vector PV308. The weighted embeddings312may be interpreted as attending over the embeddings in proportion to how similar they are to the positioned associated with their respective position vectors PV308. To increase computational speed, the prediction network300includes non-recurrent layers, and therefore, the sequence of weighted embeddings312Aa-An are not concatenated, but instead, averaged by a weighted average module316to generate, as output from the first head302A, a weighted average318A of the weighted embeddings312Aa-An represented by:

In Equation (1), h represents the index of the heads302, n represents position in context, and e represents the embedding dimension. Additionally, in Equation (1), H, N, and deinclude the sizes of the corresponding dimensions. The position vector PV308does not have to be trainable and may include random values. Notably, even though the weighted embeddings312are averaged, the position vectors PV308can potentially save position history information, alleviating the need to provide recurrent connections at each layer of the prediction network300.

The operations described above with respect to the first head302A are similarly performed by each other head302B-H of the multi-headed attention mechanism302. Due to the different set of positioned vectors PV308defined by each head302, the weight layer310outputs a sequence of weighted embeddings312Ba-Bn,312Ha-Hn at each other head302B-H that is different than the sequence of weighted embeddings312Aa-Aa at the first head302A. Thereafter, the weighted average module316generates, as output from each other corresponding head302B-H, a respective weighted average318B-H of the corresponding weighted embeddings312of the sequence of non-blank symbols.

In the example shown, the prediction network300includes a head average module322that averages the weighted averages318A-H output from the corresponding heads302A-H. A projection layer326with SWISH may receive, as input, an output324from the head average module322that corresponds to the average of the weighted averages318A-H, and generate, as output, a projected output328. A final layer normalization330may normalize the projected output328to provide the single embedding vector pui350at the corresponding output step from the plurality of output steps. The prediction network300generates only a single embedding vector pui350at each of the plurality of output steps subsequent to an initial output step.

In some configurations, the prediction network300does not implement the multi-headed attention mechanism302and only performs the operations described above with respect to the first head302A. In these configurations, the weighted average318A of the weighted embeddings312Aa-An is simply passed through the projection layer326and layer normalization330to provide the single embedding vector pui350.

In some implementations, to further reduce the size of the RNN-T decoder, i.e., the prediction network300and the joint network222, parameter tying between the prediction network300and the joint network222is applied. Specifically, for a vocabulary size V′ and an embedding dimension de, the shared embedding matrix304at the prediction network is E ∈V|×de. Meanwhile, a last hidden layer includes a dimension size dhat the joint network222, feed-forward projection weights from the hidden layer to the output logits will be W ∈dh×|V+1|, with an extra blank token in the vocabulary. Accordingly, the feed-forward layer corresponding to the last layer of the joint network222includes a weight matrix [dh, |V]|. By having the prediction network300to tie the size of the embedding dimension deto the dimensionality dhof the last hidden layer of the joint network222, the feed-forward projection weights of the joint network222and the shared embedding matrix304of the prediction network300can share their weights for all non-blank symbols via a simple transpose transformation. Since the two matrices share all their values, the RNN-T decoder only needs to store the values once on memory, instead of storing two individual matrices. By setting the size of the embedding dimension deequal to the size of the hidden layer dimension dh, the RNN-T decoder reduces a number of parameters equal to the product of the embedding dimension deand the vocabulary size [V]. This weight tying corresponds to a regularization technique.

FIG.4is a schematic view of an example training process400for training the semantic segment boundary classifier230to learn to predict the ends of segments for long-form training utterances. In this example, the semantic segment boundary classifier230is part of an RNN-T model410that includes the shared encoder network210and the semantic segment boundary classifier230. Here, the semantic segment boundary classifier230includes the second decoder220b, which includes the prediction network300band the joint network222b. The training process400may execute on the remote computing system60(i.e., on the data processing hardware62) or on the user device10(i.e., on the data processing hardware12). In the example shown, the training process400trains the semantic segment boundary classifier230using a set of training samples415. Here, each particular training sample420of the set of training samples415includes corresponding audio data422characterizing multiple segments of long-form speech, and a corresponding ground-truth transcription424of the long-form speech. In some examples, a ground-truth transcription424includes ground-truth EOS labels inserted by, for example, the automated two-stage training process ofFIGS.5A and5B. However, the ground-truth EOS labels may be inserted using other methods, such as manually during manual transcription.

For each particular training sample420in the set of training samples415, the training process400processes, using the RNN-T model410, the corresponding audio data422to obtain a corresponding predicted speech recognition hypothesis224band corresponding predicted EOS labels232. Thereafter, for each particular training sample420, a loss term module430receives the corresponding speech recognition hypothesis224band the corresponding predicted EOS labels232output by the RNN-T model410for the particular training sample420. The loss term module430then determines a loss432for the particular training sample420based on differences between the corresponding recognition hypothesis224band the corresponding predicted EOS labels232relative to the corresponding ground-truth transcription424. In some implementations, the loss432is an RNN-T loss. Notably, each ground-truth transcription424includes ground-truth EOS labels obtained, for example, via distillation from a language model teacher510(seeFIGS.5A and5B) that receives the corresponding transcription424as input and injects the ground-truth EOS labels into the corresponding transcription424between semantically complete segments.

Based on the loss432output by the loss term module430for each training sample420, the training process400trains the semantic segment boundary classifier230to minimize the loss432or maximize a probability of emitting the ground-truth EOS labels. Notably, the semantic segment boundary classifier230is also trained to learn to predict wordpieces to regularize timing of the predicted EOS labels with the predicted wordpieces in the speech recognition hypothesis224b. In some examples, the training process400trains the semantic segment boundary classifier230by adjusting, adapting, updating, fine-tuning, etc. one or more parameters of the second decoder220b, while parameters of the first decoder220aand the shared encoder network210are held fixed or frozen. In some implementations, the training process400sets the initial parameters of the second decoder220bto be equal to previously trained parameters of the first decoder220a. That is, the training process400may train the ASR model200by initially training the first joint network222ato learn how to predict transcriptions of spoken utterances, and then initializing the parameters of the second joint network222bto be equal to the parameters of the trained first joint network222a. In some examples, the training process400trains the semantic segment boundary classifier230using all or some of the training samples420originally used to train the encoder network210and the first decoder220aafter ground-truth EOS labels are added to the ground-truth transcriptions424for the training samples420(seeFIGS.5A and5B).

FIGS.5A and5Bdepict an example two-stage training process500for augmenting ground-truth training transcriptions with ground-truth EOS labels. Notably, the two-stage training500automatically inserts ground-truth EOS labels in ground-truth training transcriptions without any human annotation. In a first-stage training process500ashown inFIG.5A, a teacher model510is trained on a corpus of written-domain training samples512containing punctuation to teach the teacher model510to learn to predict the punctuation in the written text of the corpus of written-domain training samples512. For each particular written-domain training sample514of the corpus of written-domain training samples512, the training process500aprocesses, using the teacher model510, text516without punctuation of the particular written-domain training sample514to generate corresponding predicted punctuation518for the particular written-domain training sample514. A loss term module520thereafter receives the text516with punctuation of the particular written-domain training sample514and the corresponding predicted punctuation518, and determines a loss522based on punctuation differences there between. The training process500athen trains the teacher model510(e.g., by adjusting, adapting, updating, etc. one or more parameters of the teacher model510) to minimize the loss522. In some examples, the teacher model510is a language model. Additionally or alternatively, the teacher model510includes a bi-directional recurrent neural network architecture.

In a second-stage training process500bshown inFIG.5B, the teacher model510is used to augment the ground-truth transcriptions424of the set of training samples415to include ground-truth EOS labels. For each particular ground-truth transcription424of the set of training samples415(e.g., “Hi Ivy Bye Joe”), the training process500bprocesses, using the teacher model510, the particular ground-truth transcription424to generate corresponding predicted punctuation514for the particular ground-truth transcription424. Thereafter, an augmentor530augments the particular ground-truth transcription424by inserting ground-truth EOS labels, for example, <eos>labels into the ground-truth transcription424. In the example ofFIG.5B, the augmentor530inserts an<eos>label after both Ivy and Joe, which results in an augmented ground-truth transcription424of “Hi Ivy<eos>Bye Joe<eos>”. In some implementations, the augmentor530inserts a ground-truth EOS label for each comma, period, question mark, and exclamation point predicted by the teacher model510for the ground-truth transcription424.

FIG.6is a flowchart of an exemplary arrangement of operations for a computer-implemented method600for training a joint segmenting and ASR model (e.g., the ASR model200). The operations may be performed by data processing hardware610(e.g., the data processing hardware12of the user device10or the data processing hardware62of the remote computing system60) based on executing instructions stored on memory hardware620(FIG.6) (e.g., the memory hardware14of the user device10or the memory hardware64of the remote computing system60).

At operation602, the method600includes receiving a sequence of acoustic frames characterizing110one or more spoken utterances106. The method600includes at operation604generating, at each of a plurality of output steps, a higher order feature representation214for a corresponding acoustic frame110in the sequence of acoustic frames110. At operation606, the method600includes generating, at each of the plurality of output steps, a probability distribution224over possible speech recognition hypotheses. The method600at operation608includes generating, at each of the plurality of output steps, an indication232of whether the corresponding output step corresponds to an EOS. Here, the joint segmenting and ASR model200is trained on a set of training samples415, each training sample420in the set of training samples415including audio data422characterizing multiple segments of long-form speech; and a corresponding transcription424of the long-form speech, the corresponding transcription424annotated with EOS labels obtained via distillation from a language model teacher510that receives the corresponding transcription424as input and injects the EOS labels into the corresponding transcription424between semantically complete segments.

The computing device700includes a processor710(i.e., data processing hardware) that can be used to implement the data processing hardware12and/or62, memory720(i.e., memory hardware) that can be used to implement the memory hardware14and/or64, a storage device730(i.e., memory hardware) that can be used to implement the memory hardware14and/or64, a high-speed interface/controller740connecting to the memory720and high-speed expansion ports750, and a low speed interface/controller760connecting to a low speed bus770and a storage device730. Each of the components710,720,730,740,750, and760, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor710can process instructions for execution within the computing device700, including instructions stored in the memory720or on the storage device730to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display780coupled to high speed interface740. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices700may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The storage device730is capable of providing mass storage for the computing device700. In some implementations, the storage device730is a computer-readable medium. In various different implementations, the storage device730may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-or machine-readable medium, such as the memory720, the storage device730, or memory on processor710.

The high speed controller740manages bandwidth-intensive operations for the computing device700, while the low speed controller760manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller740is coupled to the memory720, the display780(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports750, which may accept various expansion cards (not shown). In some implementations, the low-speed controller760is coupled to the storage device730and a low-speed expansion port790. The low-speed expansion port790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device700may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server700aor multiple times in a group of such servers700a, as a laptop computer700b, or as part of a rack server system700c.

Unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, “A, B, or C” refers to any combination or subset of A, B, C such as: (1) A alone; (2) B alone; (3) C alone; (4) A with B; (5) A with C; (6) B with C; and (7) A with B and with C. Similarly, the phrase “at least one of A or B” is intended to refer to any combination or subset of A and B such as: (1) at least one A; (2) at least one B; and (3) at least one A and at least one B. Moreover, the phrase “at least one of A and B” is intended to refer to any combination or subset of A and B such as: (1) at least one A; (2) at least one B; and (3) at least one A and at least one B.