Patent Description:
An ongoing challenge in automated speech recognition (ASR) systems is to model transcriptions that do not exactly reflect the words spoken in an utterance. Particularly, numeric utterances, such as addresses, phone numbers, and postal codes are particularly hard members for modeling transcriptions due to the inherent out-of-vocabulary issues of long written-domain numeric sequences. This is a result of data sparsity since long numeric sequences are unlikely to be present in training data.

Sufficient amounts of training data are difficult to obtain for long numeric sequences because, for example, a number with ten digits as <NUM><NUM> possible instances. For a ten digit phone number, for example, it is unlikely that any particular phone number is seen at all, or in sufficient quantities, in training.

Recently, streaming, recurrent neural network transducer (RNN-T), end-to-end (E2E) models have shown great promise for on-device speech recognition, exhibiting improved word error rate (WER) and latency metrics as compared to conventional on-device models. These models, which fold the AM, PM, and LM into a single network, have shown competitive results compared to conventional ASR systems which have a separate AM, PM, and LMs. RNN-T, E2E models are particularly attractive for on-device ASR, as they can outperform on-device conventional models of comparable size.

"STREAMING END-TO-END SPEECH RECOGNITION FOR MOBILE DEVICES", Yanzhang He*, Tara N. Sainath*, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, Qiao Liang, Deepti Bhatia, Yuan Shangguan, Bo Li, Golan Pundak, Khe Chai Sim, Tom Bagby, Shuo-yiin Chang, Kanishka Rao, Alexander Gruenstein; concerns building an E2E speech recognizer using a recurrent neural network transducer. In experimental evaluations, it is found that the proposed approach can outperform a conventional CTC-based model in terms of both latency and accuracy in a number of evaluation categories.

When a user speaks the sequence of words "My phone number is <NUM>-<NUM>-<NUM>," an ASR system can output a transcription of that spoken, numeric sequence in the so-called "spoken domain," i.e., as those words are actually spoken ("My Phone Number is Six Five Oh Five Five Five One Two One Two"), or in the so-called "written domain," i.e., as those words would likely be reproduced in writing ("My phone number is <NUM>-<NUM>-<NUM>"). Sometimes, the ASR system can output the transcription of the numeric sequence in the written domain, but the transcription can represent the numeric sequence as numerical words (e.g., "Six Five Zero Five Five Five One Two One Two") rather than as a corresponding numerical representation (e.g., "<NUM>-<NUM>-<NUM>") as intended. Other numeric sequences, such as those that are included in addresses, phone numbers and postal codes, often have different spoken domain and written domain transcriptions.

It can be challenging for ASR systems to appropriately transcribe utterances with numeric sequences, particularly when the same numeric sequences are not seen during training, due to the inherent out-of-vocabulary ("OOV") issues of long written domain numeric sequences. This challenge arises because of data sparsity issues, namely the lack of sufficient long numeric sequences in training data.

The OOV issue is addressed in conventional ASR systems by training its acoustic model (AM) and/or its pronunciation model (PM) on spoken domain utterances for which numeric sequences are composed of in-vocabulary, spelled-out numbers (e.g., numerical words), then by inserting a weighted finite state transducer (WFST) verbalizer before a class-based language model (LM) to correct (or "denormalize" or "denorm") the spoken domain result into the written domain. Unfortunately, conventional ASR models and WFST verbalizers are not suitable for the low memory constraints of on-device ASR. Furthermore, due to the fact that the verbalizer and class-based LM use a predefined set of rules, these components do not scale well to changes in training data.

Referring to <FIG>, in some implementations, an automated speech recognition (ASR) system <NUM> is enhanced to recognize numeric sequences. In the example shown, the ASR system <NUM> resides on a user device <NUM> of a user <NUM> and/or on a remote computing device <NUM> (e.g., one or more serves of a distributed system executing in a cloud-computing environment) in communication with the user device. Although the user device <NUM> is depicted as a mobile computing device (e.g., a smart phone), the user device <NUM> may correspond to any type of computing device such as, without limitation, a tablet device, a laptop/desktop computer, a wearable device, a digital assistant device, a smart speaker/display, a smart appliance, an automotive infotainment system, or an Internet-of-Things (IoT) device.

The user device <NUM> includes an audio subsystem <NUM> configured to receive an utterance <NUM> spoken by the user <NUM> (e.g., the user device <NUM> may include one or more microphones for recording the spoken utterance <NUM>) and convert the utterance <NUM> into a corresponding digital format associated with parameterized input acoustic frames <NUM> capable of being processed by the ASR system <NUM>. In the example shown, the user speaks a respective utterance <NUM> for the phrase that would be transcribed in a spoken domain as "My Phone Number is Six Five Oh Five Five Five One Two One Two" and the audio subsystem <NUM> converts the utterance <NUM> into corresponding acoustic frames <NUM> for input to the ASR system <NUM>. For instance, the acoustic frames <NUM> may be a series of parameterized input acoustic frames that each include <NUM>-dimensional log-Mel features, computed with a short, e.g., <NUM>, window and shifted every few, e.g., <NUM>, milliseconds.

Thereafter, the ASR system <NUM> receives, as input, the acoustic frames <NUM> corresponding to the utterance <NUM>, and generates/predicts, as output, a corresponding transcription (e.g., recognition result/hypothesis) <NUM> for the utterance <NUM> in the written domain, i.e., the text "My phone number is (<NUM>) <NUM>-<NUM>". In the example shown, the user device <NUM> and/or the remote computing device <NUM> also executes a user interface generator <NUM> configured to present a representation of the transcription <NUM> of the utterance <NUM> to the user <NUM> of the user device <NUM>.

In some configurations, the transcription <NUM> output from the ASR system <NUM> is processed, e.g., by a natural language understanding (NLU) module executing on the user device <NUM> or the remote computing device <NUM>, to execute a user command. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device <NUM> or the remote computing device <NUM>) may convert the transcription into synthesized speech for audible output by another device. For instance, the original utterance <NUM> may correspond to a message the user <NUM> is sending to a friend in which the transcription <NUM> is converted to synthesized speech for audible output to the friend to listen to the message conveyed in the original utterance <NUM>.

Notably, the enhanced ASR system <NUM> includes a decoder <NUM> and a neural corrector/denormer <NUM>. The decoder <NUM> is configured to receive, as input, the parameterized input acoustic frames <NUM>, and generate, as output, an intermediate transcription <NUM> that represents the utterance <NUM> in the spoken domain or the written domain. The neural corrector/denormer <NUM> is configured to receive the intermediate transcription <NUM> output from the decoder <NUM> and perform one of written domain-to-written domain correction or spoken domain-to-written domain denorming. For instance, when the intermediate transcription <NUM> is in the spoken domain and the neural corrector/denormer <NUM> is configured as a neural denormer, the neural corrector/denoffiler <NUM> is configured to receive, as input, the intermediate transcription <NUM> in the spoken domain, and generate, as output, the transcription <NUM> for the utterance <NUM> in the written domain, i.e., the text "My phone number is (<NUM>) <NUM>-<NUM>". On the other hand, when the intermediate transcription <NUM> is in the written domain and the neural corrector/denormer <NUM> is configured as a neural corrector, the neural corrector/denormer <NUM> is configured to receive, as input, the intermediate transcription <NUM> in the written domain, and generate, as output, the transcription <NUM> as a "corrected" written domain transcription for the utterance <NUM>. Here, the intermediate transcription <NUM> in the written domain may represent the numeric sequence as numerical words such that the numeric sequence is spelled out as "Six Five Zero Five Five Five One Two One Two", whereby the neural corrector <NUM> generates the "corrected" transcription <NUM> in the written domain such that a corresponding numerical representation of "<NUM>-<NUM>-<NUM>" replaces the spelled out, in vocabulary numerical word representation from the intermediate transcription <NUM> output from the decoder <NUM>.

In some implementations, the enhanced ASR system <NUM> is trained on additional numeric training data <NUM> (<FIG>) generated by a text-to-speech (TTS) system <NUM> (<FIG>), and uses the neural corrector/denormer <NUM> to in lieu of a FST-based verbalizer to improve E2E model performance on training utterances <NUM> (<FIG>) having numeric sequences. Such an enhanced approach yields measurable improvement in several categories of numeric sequences, and exhibits significant improvement in WER for longer numeric sequences. The neural corrector/denormer <NUM> includes a small-footprint neural network to enable the neural corrector/denormer <NUM> suitable for use in an on-device environment (e.g., locally on the user device <NUM>) to perform correction or denorming on intermediate transcriptions <NUM> output from the decoder <NUM>. The large footprint associated with FST-based verbalizers prohibits their use in the on-device ASR environment due to the low memory constrains. Additionally, FST-based verbalizers use a predefined set of rules that do not scale well to changes in training data.

With reference to <FIG>, the decoder <NUM> may include an E2E, RNN-T model <NUM> which adheres to latency constrains associated with interactive applications. The RNN-T model <NUM> provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device <NUM> (e.g., no communication with a remote server is required). The RNN-T model <NUM> includes an encoder network <NUM>, a prediction network <NUM>, and a joint network <NUM>. The encoder network <NUM>, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a recurrent network of stacked Long Short-Term Memory (LSTM) layers. For instance the encoder reads a sequence ofd-dimensional feature vectors (e.g., acoustic frames <NUM> (<FIG>)) x = (x<NUM>, x<NUM>,. , xT), where <MAT>, and produces at each time step a higher-order feature representation. This higher-order feature representation is denoted as
<MAT>.

Similarly, the prediction network <NUM> is also an LSTM network, which, like a language model (LM), processes the sequence of non-blank symbols output by a final
<MAT>
Softmax layer <NUM> so far, into a dense representation
<MAT>
Finally, with the RNN-T model architecture, the representations produced by the encoder and prediction networks <NUM>, <NUM> are combined by the joint network <NUM>. The joint network then predicts
<MAT>
which is a distribution over the next output symbol. Stated differently, the joint network <NUM> generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses. Here, the "possible speech recognition hypotheses" correspond to a set of output labels each representing a symbol/character in a specified natural language. Accordingly, the joint network <NUM> may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. The output distribution of the joint network <NUM> can include a posterior probability value for each of the different output labels. Thus, if there are <NUM> different output labels representing different grapheines or other symbols, the output yi of the joint network <NUM> can include <NUM> different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthgraphic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer <NUM>) for determining the intermediate transcription115.

The Softmax layer <NUM> may employ any technique to select the output label/symbol with the highest probability in the distribution as the next output symbol predicted by the model <NUM> at the corresponding output step. In this manner, the RNN-T model <NUM> does not make a conditional independence assumption, rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model <NUM> does assume an output symbol is independent of future acoustic frames <NUM>, which allows the RNN-T model to be employed in a streaming fashion.

In some examples, the encoder network <NUM> of the RNN-T model <NUM> is made up of eight <NUM>,<NUM>-<NUM>-dimensional LSTM layers, each followed by a <NUM>-dimensional projection layer. A time-reduction layer with the reduction factor of two may be inserted after the second LSTM layer of the encoder to reduce model latency. The prediction network <NUM> may have two <NUM>,<NUM>-dimensional LSTM layers, each of which is also followed by <NUM>-dimensional projection layer. Finally, the joint network <NUM> may also have <NUM> hidden units, followed by a <NUM>,<NUM> wordpiece softmax output.

<FIG> shows an example architecture for the neural corrector/denormer <NUM> of the enhanced ASR system <NUM> of <FIG> for performing neural correction or neural denorming as a post-processing operation on outputs received from the RNN-T, E2E decoder <NUM> of the enhanced ASR system <NUM>. The architecture for the neural corrector/denormer <NUM> includes an encoder portion <NUM>, a tagger portion <NUM>, and an attention/decoder portion <NUM>. In some examples, the encoder portion <NUM><NUM> is a bidirectional RNN (BiRNN) that includes a bidirectional single-layer Gated Recurrent Unit (GRU) encoder <NUM> having <NUM> units that emits a <NUM>-dimensional hidden state. The tagger portion <NUM> may be a RNN including a single-layer GRU with <NUM> units and the attention/decoder portion <NUM> may be a bidirectional single-layer GRU with <NUM> units. Although <FIG> describes the neural corrector/denormer <NUM> primary in terms of neural correction, the neural corrector/denormer <NUM> may be additionally or alternatively used for neural denorming. In the example shown, "T" stands for trivial, "N" stands for non-trivial, "S" stands for start, and "C" stands for continuation.

When implemented as the neural corrector, the neural corrector/denormer <NUM> corresponds a written domain correction model <NUM> that receives, as input, a written-domain, intermediate transcription <NUM> generated by the RNN-T, E2E decoder <NUM>, and generates, as output, a corrected, written domain transcription <NUM> (e.g., a final transcription <NUM>). When implemented as a neural denormer, the neural corrector/denormer <NUM> corresponds to a written domain denorming model <NUM> that receives, as input, a spoken domain, intermediate transcription <NUM> generated by the RNN-T, E2E decoder <NUM>, and generates, as output, a written domain transcription <NUM> (e.g., a final transcription <NUM>). Thus, whether implemented as the neural corrector or denormer, the architecture of the neural corrector/denormer <NUM> provides an attention-based, sequence-to-sequence model that receives the output from another sequence-to-sequence model, i.e., the RNN-T, E2E decoder <NUM>.

In the example shown, the architecture of the neural corrector/denormer <NUM> is further adapted to the correction context by accounting for the fact that many of words in an input <NUM>, written domain phrase, e.g., "wake me up at", are simply copied into the written domain output <NUM>, e.g., "wake me up at," during correction. Specifically, the tagger RNN <NUM> is trained to run on the input sequence before the attention/decoder portion <NUM>, by tagging the words in the input sequence of the intermediate transcription <NUM> as either "trivial" (e.g., not requiring correction), in which case the word can simply be copied to the output, written domain sequence of the written domain transcription <NUM>, or "non-trivial" (e.g., requiring correction), in which case the word is passed into attention/decoder portion <NUM>. In addition to performing correction, the written domain correction model <NUM> could also be used to re-rank an n-best list in a second-pass setting.

In some configurations, the encoder/tagger portion <NUM>, <NUM> of the model <NUM>, which runs for all input, contains about four (<NUM>) million parameters, while the attention/decoder portion <NUM> of the model <NUM>, which runs only for text spans marked for correction, contains about six (<NUM>) million parameters. The small footprint of the neural correction model makes it attractive for the on-device context. The model is implemented in Tensorflow and is trained asynchronously on twelve graphics processing units (GPUs), with a batch size of sixteen. By contrast, the E2E, RNN-T decoder <NUM> of the ASR system includes about one-hundred fourteen (<NUM>) million parameters.

In one example, an input sequence x = {x<NUM>,. xI} is mapped to an output, written domain sequence y = {y<NUM>,. ,yT}, where the sequence vocabulary is composed of words. For instance, output labels associated numbers "<NUM>" and "<NUM>" in the written domain sequence are represented as numerical words such that each number is spelled out as "four" and "thirty". The BiRNN encoder <NUM> is defined as follows:
<MAT>
where
<MAT>
are hidden encoder states.

The tagger RNN <NUM> is defined as si = RNNtag(si-<NUM>, ti-<NUM>, hi), where s = si,. ,sI are hidden tagger states, with corresponding observations, i.e., tag sequence t = ti,. Each tag ti is a joined tag in the cross-product set of {trivial, non-trivial } × {start, continuation} to model whether a word is the beginning of a new segment needed to be corrected or a continuation of the previous segment. This refinement allows for the modeling of consecutive non-trivial segments. The learning objective of the tagger RNN <NUM> may be expressed by the following equation. <MAT>
Where P is defined as a linear projection of s followed by a softmax layer. Alignments for training the tagger RNN <NUM> may be obtained using a heuristic alignment technique that determines subsequences common to the input and output transcripts <NUM>, <NUM>. These common subsequences are marked as "trivial" (e.g., not requiring correction). For instance, common subsequences may generally include non-numerical words. In the example shown, the common subsequences marked/tagged as "trivial" by the tagger RNN <NUM> include the word sequence "Wake me up at". Here, since the phrase "Wake me up at" is in the written domain and does not require correction, the phrase can be copied over to the output to form part of the corrected, written domain transcription <NUM>.

The written domain correction model <NUM> uses the results of the tagger RNN <NUM> to extract text snippets to be corrected, such as the numerical words "four" and "thirty" in the intermediate transcription <NUM> marked as non-trivial. For example, if a text snippet spans from time s to e, the input spans {xs,. , xe} along with the context hidden states hs a and he become the input to the next stage attention model. The BiRNN encoder <NUM> is defined RNNmid over {xs,. Finally, the attention/decoder portion (RNNdec) <NUM> is defined as di,t = RNNdec((di, t-<NUM>, yi, t-<NUM>, ci,t), where ci,t is the result of the attention function of di,t-<NUM>, hs, he,and RNNmid({xs,. The two-dimensional indices (i, t) indicate t is relative to a given position i (s, e) in the input sequence. The learning objective of the attention/decoder portion <NUM> may be expressed by the following equation.

Accordingly, the results of the tagger RNN <NUM> enable the attention/decoder portion <NUM> of the neural corrector/denormer <NUM> to be applied to less than all of the text in the intermediate transcription <NUM>, e.g. only to relevant spans of text, thereby improving accuracy and decreasing latency, cost, and computation expense. For instance, in the example shown, the relevant spans of text to be corrected (e.g., snippets to be corrected) include the numerical words "four" and "thirty", whereby the attention/decoder <NUM> corrects "four" and "thirty" to be "<NUM>:<NUM>" such that the numerical representation "<NUM>:<NUM>" now represents the numeric sequence in the final transcription <NUM>. Notably, the model <NUM> is trained to also inject proper symbols/characters associated with the numeric sequence, such as currency symbols, a colon for time, etc. Here, the corrected portion "<NUM>:<NUM>" is appended to the copied over portion "Wake me up at" to provide the complete corrected, written domain final transcription <NUM>.

During training of the neural corrector/denormer <NUM>, the two learning objectives expressed in Equation (<NUM>) and Equation (<NUM>) translate to two cross-entropy losses that can be linearly combined. During inference (i.e., decoding time) of the neural corrector/denormer <NUM>, the attention/decoder portion <NUM> and the tagger RNN <NUM> work as a pipeline such that the attention/decoder portion <NUM> is only used as required by tagger RNN <NUM>, i.e., when the tagger RNN <NUM> tags words in the intermediate transcription <NUM> as "non-trivial" (e.g., not requiring correction).

With reference to <FIG> and <FIG>, the enhanced ASR system <NUM> can improve E2E, speech recognition performance on numeric sequences by applying one or more of several enhancements. For example, synthetic training data <NUM> can be used to successfully address data sparsity issues, by using a TTS system <NUM> to synthesize additional training data <NUM> for written domain numeric sequences. In doing so, synthesized speech training data <NUM> can be generated for challenging categories of numeric sequences in order to improve model coverage of those categories.

The data sparsit), or OOV issues can additionally or alternatively be addressed through the use of a neural correction network (e.g., neural corrector/denormer <NUM>), which is trained on written-domain ground truth transcription/RNN-T written-domain hypothesis pairs <NUM>, <NUM>, and which can learns to correct mistakes. In other examples, the RNN-T, E2E decoder <NUM> is trained to output numeric sequences in the spoken domain, and to denorm back to the written domain. In various implementations, such an approach can be implemented using an FST-based denormer or a neural denormer that is based on a written-domain correction model.

As noted above, in some examples, the ASR system <NUM> benefits by augmenting the training data <NUM> using the TTS system <NUM> to synthesize additional training data <NUM>, <NUM> for written domain numeric sequences, thereby addressing a "long tail" data sparsity issue of the RNN-T, E2E decoder <NUM>. To address this "long tail," data sparsity issue, additional training data <NUM>, <NUM> that represents challenging yet realistic numeric sequences can be generated. To this end, categories of numeric sequences that are frequently seen in logs, e.g., of digital assistant requests or of search engines, can be identified.

<FIG> shows an example process <NUM> for obtaining additional training data <NUM>, <NUM> for training the neural decoder/encoder <NUM> to learn to represent various categories of numeric sequences in the written domain. The remote computing device <NUM> (e.g., server) may execute the process <NUM> and train the models <NUM>, <NUM> accordingly. At stage <NUM>, the process <NUM> obtains a plurality of training utterance sets <NUM>, 402A-N each associated with a different respective numeric category A-N and including a plurality of respective transcript templates <NUM>, 404a-n. For instance, different respective numeric categories of numeric sequences may include, without limitation, a "Day" category, a "Percent" category, a "Postal Code" category, a "Time" category, or a "Year" category. Table <NUM> illustrates several such categories of numeric sequences including, specifically, categories that represent a variety of numeric ranges or sizes.

Stage <NUM> shows each transcript template <NUM> including a respective default phrase <NUM> in the written domain and a numeric slot <NUM>. The respective default phrase <NUM> in each unique transcript template <NUM> may represent spoken voice samples, e.g., "(spoken) "set second alarm for. ," from anonymized non-synthetic utterances. In some examples, one or more transcript templates <NUM>. may include a corresponding written default phrase <NUM> that is provided as input to the TTS system <NUM> for generating a synthetic speech representation of the default phrase <NUM>. At stage <NUM>, for each transcript template <NUM>, the process <NUM> generates one or more synthetic speech representations <NUM> of numeric values (e.g., (spoken) "four thirty," (spoken) "four thirty one," (spoken) "four thirty two," etc.), and for each of the one or more synthetic speech representations <NUM> of numeric values generated, injects the corresponding synthetic speech representation <NUM> of numeric values into the slot <NUM> of the corresponding transcript template <NUM> to generate a unique training utterance <NUM>. Here, the unique training utterance <NUM> includes an audible representation of the default phrase <NUM>, e.g., (spoken) "set second alarm for. ", for the corresponding transcript template <NUM> and the corresponding synthetic speech representation <NUM>, e.g., (spoken) "four thirty", of numerical values.

At stage <NUM>, the process <NUM> may implement the TTS system <NUM> to generate each synthetic speech representation <NUM> of numeric values based on corresponding training numerical value input text <NUM>. The TTS system <NUM> may include a multi-speaker TTS system that generates a mel-spectrogram that is conditioned on phonemes and an n-dimensional speaker embedding that is learned for each speaker during training. In some examples, n is equal to <NUM>. Here, the predicted mel-spectrogram may then be inverted to a time-domain waveform with a WaveRNN neural vocoder. Multi-style training (MTR) may be used to add artificial noise to synthesized audio. TTS training data may include audio data derived from speakers of a single language or from multiple languages, or of speakers that share a single accent or have multiple different accents. During inference, input, textual numeric sequences are mapped to phonemes, and a speaker is randomly selected.

Moreover, at stage <NUM>, the process <NUM> may implement an injector <NUM> that is configured to receive, as input, each synthetic speech representation <NUM> of numeric values output from the TTS system <NUM>, and provide, as output, the unique training utterance <NUM> by injecting the synthetic speech representation <NUM> into the slot <NUM> associated with the default phrase <NUM>. Such injection can occur by performing weighted sampling from a numeric WFST grammar which is weighted on the spoken domain. While the slot <NUM> is shown as following the default phrase <NUM> in the template <NUM>, the slot <NUM> may precede the default phrase <NUM> or may be interspersed anywhere between the start and end of the default phrase <NUM>.

In some examples, the audible representation of the default phrase <NUM> corresponding to at least one of the transcript templates <NUM> includes anonymized non-synthetic speech. Here, the utterances used to generate templates may be anonymized in one or more ways before they are generated, stored or used, so that personally identifiable information is removed. For example, a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained, e.g., such as to a city, ZIP code, or state level, so that a particular location of a user cannot be determined. The user may have control over how information is collected about him or her and used by the ASR system <NUM>. In additional examples, the audible representation of the default phrase <NUM> corresponding to at least one of the transcript templates <NUM> includes a synthetized speech representation of the default phrase <NUM>. For instance, the TTS system <NUM> may synthesize the respective default phrase <NUM> in one or more of the transcript templates <NUM>.

At stage <NUM>, the process <NUM> updates each respective transcript template <NUM> of the plurality of transcript templates <NUM> in each of the plurality of training utterance sets <NUM> representing a respective different category with the one or more respective unique training utterances <NUM> generated that each include the same default phrase <NUM> and a different synthetic speech representation <NUM> of numeric values. Furthermore, the process <NUM> may also generate, for each unique training utterance <NUM>, parallel textual transcriptions <NUM>, <NUM>, one in the spoken domain, e.g. (text) "set second alarm for four thirty", and the other in the written domain, e.g., (text) "set second alarm for <NUM>:<NUM>". This process <NUM> is repeated numerous times for each template <NUM>, each time synthesizing a unique training utterance for that category of numeric sequence. In some examples, the written domain transcription <NUM> of the unique training utterance <NUM> includes a ground-truth transcription that is used in conjunction with recognition hypothesis <NUM> output from the decoder <NUM> for training neural corrector/denormer <NUM> of <FIG>.

Referring back to <FIG>, some other examples, the ASR system <NUM> benefits from spoken domain training and FST denorming. With this enhancement, the RNN-T, E2E decoder <NUM> is trained on a spoken domain version of the training set, and the translation back to written domain is left to an FST denormer derived from a legacy, production grammar. In order to train the RNN-T, E2E decoder, transcriptions of utterances in both the spoken domain and the written domain are used as training data. These examples can be obtained by passing written domain transcripts from a training set through an FST verbalizer, then choosing a single, spoken domain verbalization by passing each candidate hypothesis through a lexicon, and force-aligning the resulting phone sequences against the phones in the utterance. The spoken domain transcription that was obtained using the verbalization grammar is used as TTS training data.

In additional examples, the ASR system <NUM> benefits from the use of neural denorming. Specifically, since an FST-based denorming approach can be challenging to place on a device with limited resources, a neural denormer could also be used in place of an FST. For instance, the written domain neural correction model may be adapted to the spoken domain by rephrasing or recharacterizing it as a neural denormer which consumes spoken domain training data and emits written domain output. The architecture of the neural denorming model is identical to the written correction model.

In testing, the ASR system <NUM> benefits over legacy systems by the introduction of synthesized training data, particularly for shorter numeric sequences, and that errors that plague rule-based FST denormers can be almost entirely mitigated. The avoidance of OOV issues obtained by training in the spoken domain appears to largely solve formatting problems experienced by written domain models, while using a neural denormer, which learns how to denorm from training data, sidesteps the denorming errors seen in the FST-based spoken domain model. Finally, the spoken domain denorming approach does not result in a significant degradation on the real-audio data sets. When multiple enhancements were used together, the improvements for E2E performance were greatest, particularly for longer utterances.

<FIG> provides a flowchart of an example arrangement of operations for a method <NUM> of generating final transcriptions <NUM> representing numerical sequences of utterances <NUM> in a written domain. Data processing hardware residing on the user device <NUM> or the remote server <NUM> may execute the operations for the method <NUM>. At operation <NUM>, the method <NUM> includes receiving, at the data processing hardware, audio data <NUM> for an utterance <NUM> containing a numeric sequence. At operation <NUM>, the method <NUM> includes decoding, by the data processing hardware, using a sequence-to-sequence speech recognition model <NUM>, the audio data <NUM> for the utterance <NUM> to generate, as output from the sequence-to-sequence speech recognition model <NUM>, an intermediate transcription <NUM> of the utterance <NUM>.

At operation <NUM>, the method <NUM> also includes processing, by the data processing hardware, using a neural corrector <NUM>, the intermediate transcription <NUM> output from the sequence-to-sequence speech recognition model <NUM> to generate a final transcription <NUM> that represents the numeric sequence of the utterance <NUM> in a written domain. The neural corrector is trained on a set of training samples <NUM>, where each training sample <NUM> includes a speech recognition hypothesis <NUM> for a corresponding training utterance and a ground-truth transcription <NUM> of the corresponding training utterance. The ground-truth transcription <NUM> of the corresponding training utterance is in the written domain. At operation <NUM>, the method <NUM> also includes providing, by the data processing hardware, the final transcription <NUM> representing the numeric sequence of the utterance <NUM> in the written domain for output.

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>), audio data (<NUM>) for an utterance (<NUM>) containing a numeric sequence;
decoding, by the data processing hardware (<NUM>), using a recurrent neural network-transducer (RNN-T), end-to-end decoder model (<NUM>), the audio data (<NUM>) for the utterance (<NUM>) to generate, as output from the recurrent neural network-transducer (RNN-T), end-to-end decoder model (<NUM>), an intermediate transcription (<NUM>) of the utterance (<NUM>), wherein the intermediate transcription is a written transcription of the utterance comprising numerical and non-numerical words, the numerical words representing the sequence of numbers;
processing, by the data processing hardware (<NUM>), using a neural corrector (<NUM>), the intermediate transcription (<NUM>) output from the recurrent neural network-transducer (RNN-T), end-to-end decoder model (<NUM>) to generate a final transcription (<NUM>) that represents the numeric sequence of the utterance (<NUM>), wherein the final transcription replaces the numerical words in the intermediate transcription with a corresponding numerical representation of the sequence of numbers, wherein processing using a neural corrector comprises:
processing only a first portion of the intermediate transcription (<NUM>) that includes the numerical words without processing a remaining second portion of the intermediate transcription (<NUM>) that includes non-numerical words,
wherein the neural corrector (<NUM>) was trained on a set of training samples, each training sample comprising a speech recognition hypothesis (<NUM>) for a corresponding training utterance (<NUM>) and a ground-truth transcription (<NUM>) of the corresponding training utterance (<NUM>); and
providing, by the data processing hardware (<NUM>), the final transcription (<NUM>) representing the numeric sequence of the utterance (<NUM>) for output.