USING TEXT-INJECTION TO RECOGNIZE SPEECH WITHOUT TRANSCRIPTION

A method includes receiving training data including transcribed speech utterances spoken in a general domain, modified speech utterances in a target domain, and unspoken textual utterances corresponding to the transcriptions of the modified speech utterances in the target domain. The modified speech utterances include utterances spoken in the target domain that have been modified to obfuscate one or more classes of sensitive information recited in the utterances. The method also includes generating a corresponding alignment output for each unspoken textual utterance of the received training data using an alignment model. The method also includes training a speech recognition model on the alignment outputs generated for the corresponding to the unspoken textual utterances, the un-transcribed speech utterances, and the transcribed speech utterances to teach the speech recognition model to learn to recognize speech in the target domain and phrases within the one or more classes of sensitive information.

TECHNICAL FIELD

This disclosure relates to using text-injection to recognize speech without transcriptions.

BACKGROUND

Automatic speech recognition (ASR), the process of taking an audio input and transcribing it into text, has greatly been an important technology that is used in mobile devices and other devices. In general, automatic speech recognition attempts to provide accurate transcriptions of what a person has said by taking an audio input (e.g., speech utterance) and transcribing the audio input into text. Modern ASR models continue to improve in both accuracy (e.g. a low word error rate (WER)) and latency (e.g., delay between the user speaking and the transcription) based on the ongoing development of deep neural networks. However, one challenge in developing deep learning-based ASR models is that parameters of the ASR models tend to over fit the training data, thereby resulting in the ASR models having difficulties generalizing unseen data when the training data is not extensive enough. Yet, this challenge is further complicated when training ASR models on low-resource speech domains that include an insufficient amount (or even zero) of transcribed speech training data.

SUMMARY

One aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations for training a speech recognition model using text-injection. The operations include receiving training data including transcribed speech utterances spoken in a general domain, modified speech utterances in a target domain that have been modified to obfuscate one or more classes of sensitive information recited in the utterances, and unspoken textual utterances corresponding to the transcriptions of the modified speech utterances in the target domain. Each transcribed utterance is paired with a corresponding transcription. Each modified speech utterance is paired with a corresponding transcription that redacts the sensitive information obfuscated from the modified speech utterance. The unspoken textual utterances include fake random data inserted into redacted portions of the transcriptions of the modified speech utterances where the sensitive information recited in the modified speech utterance has been redacted. The operations also include generating a corresponding alignment output for each unspoken textual utterance of the received training data using an alignment model. The operations also include training a speech recognition model on the alignment outputs generated for the unspoken textual utterances, the modified speech utterances, and the transcribed speech utterances to teach the speech recognition model to learn to recognize speech in the target domain and phrases within the one or more classes of sensitive information.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the one or more classes of sensitive information includes at least one of personable identifiable information, protected health information, or dates. In some examples, the redacted portions of the transcriptions of the modified speech utterances are tagged with a class identifier identifying the class of sensitive information that has been redacted. In these examples, the fake random data inserted into each redacted portion of the transcriptions of the modified speech utterances may be associated with the class of sensitive information identified by the class identifier at the redacted portion. The transcribed speech utterances in the general domain may include a greater number of hours of speech than the modified speech utterances.

In some implementations, the speech recognition model includes an audio encoder and a decoder. The audio encoder includes a stack of self-attention layers each including a multi-headed self-attention mechanism. Here, the training data may further include un-transcribed speech utterances spoken in the general domain. Each un-transcribed speech utterance not paired with any corresponding transcription. In these implementations, training the speech recognition model includes: for each un-transcribed speech utterance, generating a corresponding encoded representation of the un-transcribed speech utterance and training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the un-transcribed speech utterance; for each alignment output, generating a corresponding encoded representation of the alignment output and training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the alignment output; and, for each transcribed speech utterance, generating a corresponding encoded representation of the transcribed speech utterance and training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the transcribed speech utterance. The decoder may include one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or Recurrent Neural Network-Transducer (RNN-T) decoder. In some examples, generating the corresponding alignment output for each unspoken textual utterance of the received training data includes extracting an initial textual representation from the unspoken textual utterance, predicting a text chunk duration for each text chunk in the unspoken textual utterance, and upsampling the initial textual representation using the predicted text chunk duration for each text chunk in the unspoken textual utterance.

Another aspect of the disclosure provides a system that includes data processing hardware and memory hardware storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations. The operations include receiving training data including transcribed speech utterances spoken in a general domain, modified speech utterances in a target domain that have been modified to obfuscate one or more classes of sensitive information recited in the utterances, and unspoken textual utterances corresponding to the transcriptions of the modified speech utterances in the target domain. Each transcribed utterance is paired with a corresponding transcription. Each modified speech utterance is paired with a corresponding transcription that redacts the sensitive information obfuscated from the modified speech utterance. The unspoken textual utterances include fake random data inserted into redacted portions of the transcriptions of the modified speech utterances where the sensitive information recited in the modified speech utterance has been redacted. The operations also include generating a corresponding alignment output for each unspoken textual utterance of the received training data using an alignment model. The operations also include training a speech recognition model on the alignment outputs generated for the unspoken textual utterances, the modified speech utterances, and the transcribed speech utterances to teach the speech recognition model to learn to recognize speech in the target domain and phrases within the one or more classes of sensitive information.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the one or more classes of sensitive information includes at least one of personable identifiable information, protected health information, or dates. In some examples, the redacted portions of the transcriptions of the modified speech utterances are tagged with a class identifier identifying the class of sensitive information that has been redacted. In these examples, the fake random data inserted into each redacted portion of the transcriptions of the modified speech utterances may be associated with the class of sensitive information identified by the class identifier at the redacted portion. The transcribed speech utterances in the general domain may include a greater number of hours of speech than the modified speech utterances.

In some implementations, the speech recognition model includes an audio encoder and a decoder. The audio encoder includes a stack of self-attention layers each including a multi-headed self-attention mechanism. Here, the training data may further include un-transcribed speech utterances spoken in the general domain. Each un-transcribed speech utterance not paired with any corresponding transcription. In these implementations, training the speech recognition model includes: for each un-transcribed speech utterance, generating a corresponding encoded representation of the un-transcribed speech utterance and training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the un-transcribed speech utterance; for each alignment output, generating a corresponding encoded representation of the alignment output and training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the alignment output; and, for each transcribed speech utterance, generating a corresponding encoded representation of the transcribed speech utterance and training the audio encoder on a contrastive loss applied on the corresponding encoded representation of the transcribed speech utterance. The decoder may include one of a Connection Temporal Classification (CTC) decoder, a Listen Attend Spell (LAS) decoder, or Recurrent Neural Network-Transducer (RNN-T) decoder. In some examples, generating the corresponding alignment output for each unspoken textual utterance of the received training data includes extracting an initial textual representation from the unspoken textual utterance, predicting a text chunk duration for each text chunk in the unspoken textual utterance, and upsampling the initial textual representation using the predicted text chunk duration for each text chunk in the unspoken textual utterance.

DETAILED DESCRIPTION

One challenge in developing deep learning-based ASR models is that parameters of the ASR models tend to over fit the training data, thereby resulting in the ASR models having difficulties generalizing unseen data when the training data is not extensive enough. Thus, training the ASR models on larger training datasets improves the accuracy of the ASR model. For instance, the use of machine learning or other statistical methods can train ASR models on training data sets that include upwards of 10,000 hours of transcribed speech. Yet, performance of ASR models suffers when a domain associated with the training data is distinct from a domain that the ASR model receives during inference. For example, training an ASR model on transcribed speech on a domain associated with video meetings would be less effective in recognizing speech related to speech in a domain associated with healthcare.

In some scenarios, speech related to certain domains includes private and/or sensitive information such as names of people, dates, or other personal identifiers. For various ethical and legal reasons, this sensitive information may not be directly used to train ASR models. However, simply not training ASR models on this sensitive information results in ASR models having lower recognition accuracy of sensitive information during inference. One current approach includes de-identification, a process that redacts or eliminates personally identifiable information (PII) and other sensitive information from training data. For instance, the process may replace an audio segment that includes sensitive information with silence audio data and/or remove text representing sensitive information. In some instances, speech utterances are shorter (e.g., call-center applications), and thus, the de-identification process may eliminate training utterances entirely. Consequently, simply eliminating or redacting training data that includes sensitive information causes ASR models to have poor performance on similar sensitive information received during inference. Another approach includes replacing portions of training data that include sensitive information with synthesized speech. The synthesized speech may include the sensitive information or include information similar to the sensitive information. Yet, using synthetic speech in place of training data that includes sensitive information has several drawbacks. Namely, splicing synthetic speech into speech recordings that include sensitive information to get natural sounding training utterances is challenging and error prone. Moreover, even though state-of-the-art text-to-speech (TTS) models produce realistic sounding speech, training ASR models on synthetic speech is not as beneficial as training ASR models on human speech.

Accordingly, implementations herein are directed towards methods and systems of a training process that trains a speech recognition model using text-injection with unspoken textual utterances that include redacted information. The training process includes receiving training data that includes transcribed speech utterances spoken in a general domain, modified speech utterances in a target domain, and unspoken textual utterances. Each transcribed speech utterance is paired with a corresponding transcription. The modified speech utterances include utterances spoken in the target domain that have been modified to obfuscate one or more classes of sensitive information recited in the utterances. Moreover, each modified speech utterance is paired with a corresponding transcription that redacts the sensitive information obfuscated from the modified speech utterance. The unspoken textual utterances correspond to the transcriptions of the modified speech utterances in the target domain. The unspoken textual utterances include fake random data inserted into redacted portions of the transcriptions of the modified speech utterances where the sensitive information recited in the modified speech utterance has been redacted. The training process uses an alignment model to generate a corresponding alignment output for each unspoken textual utterance of the received training data. The training process also trains a speech recognition model on the alignment outputs, the modified speech utterances, and the transcribed speech utterances to teach the speech recognition model to learn to recognize speech in the target domain and phrases within the one or more classes of sensitive information. Notably, the training process trains the speech recognition model to recognize phrases within the one or more classes of sensitive information without directly using any of the sensitive information redacted from the modified speech utterances to train the speech recognition model.

FIG.1illustrates an automated speech recognition (ASR) system100implementing an ASR model200that resides on a user device102of a user104and/or on a remote computing device201(e.g., one or more servers of a distributed system executing in a cloud-computing environment) in communication with the user device102. Although the user device102is depicted as a mobile computing device (e.g., a smart phone), the user device102may correspond to any type of computing device such as, without limitation, a tablet device, a laptop/desktop computer, a wearable device, a digital assistant device, a smart speaker/display, a smart appliance, an automotive infotainment system, or an Internet-of-Things (IoT) device, and is equipped with data processing hardware111and memory hardware113.

The user device102includes an audio subsystem108configured to receive an utterance106spoken by the user104(e.g., the user device102may include one or more microphones for recording the spoken utterance106) and convert the utterance106into a corresponding digital format associated with input acoustic frames110capable of being processed by the ASR system100. In the example shown, the user speaks a respective utterance106in a natural language of English for the phrase “What is the weather in New York City?” and the audio subsystem108converts the utterance106into corresponding acoustic frames110for input to the ASR system100. Thereafter, the ASR model200receives, as input, the acoustic frames110corresponding to the utterance106, and generates/predicts, as output, a corresponding transcription120(e.g., recognition result/hypothesis) of the utterance106. In the example shown, the user device102and/or the remote computing device201also executes a user interface generator107configured to present a representation of the transcription120of the utterance106to the user104of the user device102. In some configurations, the transcription120output from the ASR system100is processed, e.g., by a natural language understanding (NLU) module executing on the user device102or the remote computing device201, to execute a user command. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device102or the remote computing device201) may convert the transcription into synthesized speech for audible output by another device. For instance, the original utterance106may correspond to a message the user104is sending to a friend in which the transcription120is converted to synthesized speech for audible output to the friend to listen to the message conveyed in the original utterance106.

Referring toFIG.2, an example ASR model200may include a Recurrent Neural Network-Transducer (RNN-T) model architecture which adheres to latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the frame alignment-based transducer model200may include other architectures such as transformer-transducer and conformer-transducer model architectures among others. The RNN-T model200provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device102(e.g., no communication with a remote server is required). The RNN-T model200includes an encoder network210, a prediction network220, and a joint network230. The encoder network210, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a stack of self-attention layers (e.g., Conformer or Transformer layers) or a recurrent network of stacked Long Short-Term Memory (LSTM) layers. For instance, the encoder reads a sequence of d-dimensional feature vectors (e.g., acoustic frames110(FIG.1)) x=(x1, x2, . . . , xT), where xt∈d, and produces at each output step a higher-order feature representation. This higher-order feature representation is denoted as h1enc, . . . , hTenc. In some examples, the encoder network210includes a dual encoder framework that has a text encoder202and a speech encoder204(FIGS.3B and3C).

Similarly, the prediction network220is also an LSTM network, which, like a language model (LM), processes the sequence of non-blank symbols output by a final Softmax layer240so far, y0, . . . , yui-1, into a dense representation pui. Finally, with the RNN-T model architecture, the representations produced by the encoder and prediction/decoder networks210,220are combined by the joint network230. The prediction network220may be replaced by an embedding look-up table to improve latency by outputting looked-up sparse embeddings in lieu of processing dense representations. The joint network then predicts P(yi|xti, y0, . . . , yui-1), which is a distribution over the next output symbol. Stated differently, the joint network230generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses. Here, the “possible speech recognition hypotheses” correspond to a set of output labels each representing a symbol/character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, e.g., one label for each of the 26-letters in the English alphabet and one label designating a space. Accordingly, the joint network230may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces, phonemes, and/or entire words, in addition to or instead of graphemes. The output distribution of the joint network230can include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output yiof the joint network230can include 100 different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer240) for determining the transcription120.

The Softmax layer240may employ any technique to select the output label/symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model200at the corresponding output step. In this manner, the RNN-T model200does not make a conditional independence assumption, rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model200does assume an output symbol is independent of future acoustic frames110, which allows the RNN-T model to be employed in a streaming fashion, the non-streaming fashion, or some combination thereof.

In some examples, the encoder network (i.e., audio encoder)210of the RNN-T model200includes a stack of self-attention layers/blocks, such as conformer blocks. Here, each conformer block includes a series of multi-headed self attention, depth wise convolution and feed-forward layers. The prediction network220may have two 2,048-dimensional LSTM layers, each of which is also followed by a 640-dimensional projection layer. Alternatively, the prediction network220may include a stack of transformer or conformer blocks, or an embedding look-up table in lieu of LSTM layers. Finally, the joint network230may also have 640 hidden units. The Softmax layer240may be composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes in a plurality of training data sets.

FIGS.3A-3Cillustrate an example training process300for training the ASR model200(FIG.2). The training process300described herein describes training the audio encoder210of the ASR model200, however, it is understood that the training process300may also include pre-training and/or fine-tuning of the audio encoder210. Moreover, implementations described herein contemplate the training process300training the audio encoder210of the ASR model200without training the decoder (e.g., prediction network220and the joint network230) of the ASR model200. Yet, it is understood that the training process300may additionally or alternatively train other components of the ASR model200(e.g., prediction network220and/or joint network230) jointly with the audio encoder210.

The training process300may train the audio encoder210using available training data that includes a set of unspoken textual utterances (Xtext)320, a set of labeled speech utterances (Xsup)307, and/or un-transcribed non-synthetic speech utterances (Xunsup)306. The labeled speech utterances307may include transcribed speech utterances304and/or modified speech utterances305. Each labeled speech utterance307is paired with a corresponding transcription (e.g., ground-truth label)302,303. In particular, each transcribed speech utterance304is paired with a first transcription302, and each modified speech utterance305is paired with a second transcription (e.g., modified transcription)303. The transcribed speech utterances304include utterances spoken in a general domain. In contrast, the modified speech utterances305include utterances spoken in a target domain different from the general domain. For instance, the general domain may include speech from a video library and the corresponding captions and the target domain may be speech associated with a healthcare domain, finance domain, or any other speech-related domain that includes personal and/or sensitive information. Here, the speech from the video library may only include some (or zero) speech associated with the healthcare domain. Thus, the transcribed speech utterances304in the general domain may include a greater number of hours of speech than the modified speech utterances305. The modified speech utterances305are modified to obfuscate one or more classes of sensitive information recited in the utterances. As such, the second transcriptions303that are paired with the modified speech utterances305redact the sensitive information obfuscated from the modified speech utterances305. In some examples, the training process300uses the transcribed speech utterances304and the modified speech utterances305from the labeled speech utterances307to train the audio encoder210. In other examples, the training process300only uses the transcribed speech utterances304(e.g., and not the modified speech utterances305) from the labeled speech utterances307to train the audio encoder210.

Each unspoken textual utterance320includes text-only data (i.e., unpaired data) such that each unspoken textual utterance320is not paired with any corresponding spoken audio representation (i.e., speech) of the utterance. The unspoken textual utterance320may include any sequence of text chunks including words, word-pieces, phonemes, and/or graphemes. The unspoken textual utterances320may correspond to the modified speech utterances305. Since the unspoken textual utterances320is not paired with any corresponding transcriptions, the training process300does not use the modified speech utterances305and the unspoken textual utterances320together when training the audio encoder210. Simply put, the training process uses the text-only data of the unspoken textual utterances320when using the unspoken textual utterances320to train the audio encoder210without using the corresponding audio data of the modified speech utterances305. Each un-transcribed non-synthetic speech utterance306(also referred to as simply “un-transcribed speech utterance306”) is spoken in the general domain and includes audio-only data (i.e., unpaired data) such that the un-transcribed speech utterance306is not paired with any corresponding transcription.

For simplicity, the training process300includes a contrastive self-supervised loss part300a(FIG.3A), a supervised loss part300b(FIG.3B), and a consistency regularization part300c(FIG.3C). The training process300pre-trains the audio encoder210on a total loss (Ltts4pretrain2) based on: contrastive losses (Lw2x)316derived using the contrastive self-supervised loss part300afrom the unspoken training text utterances (Xtext)320, the corpus of labeled speech utterances (Xsup)307, and the un-transcribed non-synthetic speech utterances (Xunsup)306; supervised losses (Laux)342,344derived using the supervised loss part300bfrom the unspoken training text utterances (Xtext)320and the labeled speech utterances (Xsup)307; and consistency losses (cons(θ))352derived using the consistency regularization part300c.

Referring toFIG.3A, the contrastive self-supervised loss part300aof the training process300may employ an alignment model400that is configured to generate, at each of a plurality of output steps, alignment outputs (i.e., textual representation)402for each of a plurality of unspoken training text utterances320. The unspoken textual utterances320includes unspoken text that is text-only data, i.e., unpaired data, such that each unspoken textual utterance (Xtext)320is not paired with any synthesized or non-synthesized speech. Accordingly, the alignment model400generates a corresponding alignment output402for each of the unspoken textual utterances320.

Referring now toFIG.4, in some examples, the alignment model400includes an embedding extractor410, duration predictor420, and an upsampler430. The embedding extractor410receives the unspoken textual utterance320that includes a sequence of text chunks including words, word-pieces, phonemes, and/or graphemes and extracts a corresponding initial textual representation (et)412. The initial textual representation412includes embedding lexical information from the unspoken textual utterance320. Additionally or alternatively, the embedding extractor410may receive a transcription302corresponding to a transcribed non-synthetic speech utterance304(FIG.3C). The duration predictor420receives the initial textual representation412from the embedding extractor410and predicts a corresponding text chunk duration (i.e., word, word-piece, phoneme, and/or grapheme duration)422. The text chunk duration422indicates a duration the corresponding text chunk would be spoken if a human (or text-to-speech system) spoke the unspoken textual utterance320. For example, the unspoken textual utterance320may include a sequence of phonemes and the duration predictor420predicts a phoneme duration422for each phoneme in the sequence of phonemes. In this example, the duration predictor420predicts the phoneme duration422by predicting a probability of non-zero duration for each phoneme and predicting a probability of continuous phoneme duration for each phoneme. As the sequence of phonemes includes regular phonemes, silences between word boundaries, and punctuation marks, only the regular phonemes are associated with non-zero duration while the silences and punctuation marks are generally associated with the continuous phoneme duration. Accordingly, the duration predictor420may use a sigmoid activation following a first one of two independent activations to predict the probability of non-zero duration and use a soft plus activation following a second one of the two independent projections to predict the continuous text chunk duration422for each text chunk. The duration predictor420determines, for each text chunk, whether the probability of non-zero duration is less than a threshold value, and when the probability of non-zero duration is less than the threshold value, a multiplier may zero-out the continuous text chunk duration422predicted by the softplus activation for the corresponding text chunk. Otherwise, when the probability of non-zero duration is not less than the threshold value, the predicted text chunk duration422may be set equal to the continuous phoneme duration predicted by the softplus activation.

The upsampler430receives, for each unspoken textual utterance320, the corresponding initial textual representation412and the predicted text chunk duration422, and generates an alignment output (êt)402having a number of frames by upsampling the initial textual representation412using the corresponding predicted text chunk duration422. Here, the alignment output402represents an aligned speech-text representation. In some examples, the alignment model400sends the alignment output402to a text encoder202of the audio encoder210(FIGS.3B and3C). In other examples (not shown), the alignment model400sends the alignment output402to a shared encoder250(e.g., bypassing the text encoder202) of the audio encoder210(FIGS.3B and3C). In these other examples, the alignment output402serves as the encoded textual representation312such that the shared encoder250may receive the alignment output402directly from the alignment model400. In yet other examples, paired training data is available and the upsampler430generates the alignment output402as follows:

Here, the upsampler430includes resampler and refiner layers that align the initial textual embedding412to align with a corresponding encoded audio representation314(FIGS.3B and3C) directly. However, when paired training data is not available, the upsampler430generates the alignment output402as follows:

In particular, the number of frames of the alignment output402indicates a predicted speech duration of the unspoken textual utterance320. Stated differently, the number of frames of the alignment output402maps (i.e., aligns) the sequence of text chunks of the unspoken textual utterance320to speech frames. Here, the upsampler430includes resampler and refiner layers that replicate the initial textual embedding412to match the predicted text chunk duration422(i.e., speech duration). As such, the alignment output402includes a textual representation of the unspoken textual utterance320having a timing component that aligns with how a human would speak the unspoken textual utterance320.

Notably, in most instances, a text-to-speech (TTS) system generates an audible output to give the unspoken textual utterance320the timing component of human speech such that a training process may use the audible output from the TTS system (i.e., synthetic speech) to train the audio encoder210. However, the alignment model400advantageously generates the alignment output402thereby mapping the sequence of text chunks to speech frames directly. As such, the training process300does not require any TTS system to generate synthetic speech from the unspoken textual utterances320to train the audio encoder210. That is, neither the training process300nor the alignment model400converts the unspoken textual utterance320into synthetic speech, but rather generates alignment outputs402(i.e., text alignments).

Referring back toFIG.3A, in some implementations, the audio encoder210includes a speech encoder204and a text encoder202, described in more detail with reference toFIGS.3B and3C. In the example shown, the audio encoder210(alternatively the speech encoder204or the text encoder202(FIGS.3B and3C)) includes a Conformer encoder including a stack of Conformer blocks each of which includes a stack of multi-headed self-attention, depth wise convolution, and feed-forward layers. Alternatively, the audio encoder210may include another type of encoder having a stack of multi-head self-attention layers/blocks, such as a transformer or performer encoder. The Conformer encoder210can naturally be split into a feature encoder, including a convolution subsampling block212, and a context network, including a linear layer214and a stack of Conformer blocks216. In some implementations, the convolution subsampling block212has two two-dimensional-convolution layers, both with strides (2, 2), resulting in a 4× reduction in the feature sequence length. The convolution subsampling block212receives, as input, a sequence of input features/vectors (e.g., mel-frequency spectrograms such as the acoustic frames110ofFIG.1) associated with each labeled speech utterance307and each un-transcribed non-synthetic speech utterance306, and generates, as output, for each of a plurality of output steps, an encoded audio feature211that corresponds to a respective one of labeled speech utterances307or a respective one of the un-transcribed non-synthetic speech utterances306. The convolution subsampling block212may receive, as input, each alignment output402generated by the alignment model400from the unspoken textual utterances320and generate, as output, for each of the plurality of output steps, an encoded textual feature213that corresponds to a respective one of the alignment outputs402.

The encoded audio and textual features211,213(i.e., interchangeably referred to as “encoded features211,213”) output from the convolution subsampling block212may be fed to a masking module218where some of the encoded features211,213are randomly chosen and replaced with a trained feature vector shared between all masked time steps to provide corresponding masked encoded audio features211,211mand masked encoded textual features213,213m. In some examples, the masking module218masks the randomly chosen encoded features211,213for masking by randomly sampling without replacement a certain proportion p of all time steps to be start indices and then masks the subsequent M consecutive time steps from every sample index, whereby some spans may overlap. After masking is applied, the linear layer214and the Conformer blocks216of the context network receive the masked encoded features211m,213m(or encoded features211,213not chosen by the masking module218) and outputs corresponding contrastive context vectors (i.e., encoded representation)215from masked encoded features211m,213m. Moreover, a quantizer217receives the encoded features211,213as input, and generates quantized vectors (i.e., target context vectors)219as output. Thereafter, a contrastive loss module315derives a contrastive loss (Lw2v)316between the contrastive context vectors215at the masked positions and the target context vectors219as follows.

where ctis contrastive context vector215centered over a masked output step (i.e., time step) t and qtrepresents a target context vector219at the output step t in a set of K+1 candidate target context vectors219which includes qtand K distractors. Distractors may be uniformly sampled from other masked output steps of the same utterance.

The contrastive loss316is optimized between the contrastive context vectors215at the masked positions and the target context vectors219. After the audio encoder210converges on the un-transcribed non-synthetic speech utterances306, the training procedure is repeated on both the alignment outputs402corresponding to the unspoken textual utterance320and the labeled speech utterance307. Thus, the contrastive loss316(Lw2v) is optimized for both real/human (non-synthetic) and unspoken textual utterances320represented by alignment outputs402, with additional auxiliary losses derived from the labeled speech utterances307and the alignment outputs402as described in greater detail below with reference toFIG.3B. Accordingly, the contrastive self-supervised loss part300aof the training process300trains the audio encoder210using the contrastive loss316derived from the corresponding encoded features211,213associated with each alignment output402, each labeled speech utterances307, and each un-transcribed non-synthetic speech utterance306provided as input to the audio encoder210. Training the audio encoder210may include updating parameters of the audio encoder210based on the contrastive losses316.

Referring toFIG.3B, the supervised loss part300bof the training process300is configured to inject lexical information into the audio encoder210during training based on supervised loss terms342,344derived from the transcribed non-synthetic speech utterances and the alignment outputs402corresponding to unspoken textual utterances320output by the alignment model400. Notably, the supervised loss part300bleverages one or more auxiliary decoders390for generating the supervised loss terms342,344. The auxiliary decoders390may include Connectionist Temporal Classification (CTC) decoders, Listen Attend Spell (LAS) decoders, or RNN-T decoders. These auxiliary decoders390may include at least one of a phoneme decoder configured to decode a sequence of phonemes or a wordpiece decoder configured to decode a sequence of word pieces. The auxiliary decoders390could also include a grapheme decoder configured to decode a sequence of graphemes.

During the supervised loss part300b, the text encoder202of the audio encoder is configured to receive alignment outputs402(i.e., text embeddings) from the alignment model400and the speech encoder204is configured to receive the labeled speech utterances307. That is, the text encoder202of the audio encoder210generates encoded textual representations312for alignment outputs402(e.g., corresponding to an unspoken textual utterance320) and the speech encoder204of the audio encoder210generates encoded audio representations314for speech inputs (i.e., labeled speech utterances307). Here, the encoded textual representations312and the encoded audio representations314may not both be compatible with the auxiliary decoders390. Thus, the audio encoder210may also include a shared encoder250that receives the encoded textual representations312as input, and generates a first encoded shared representation322(etext) as output. Moreover, the shared encoder250receives the encoded audio representations314as input, and generates a second encoded shared representation (esup)324as output. Accordingly, the shared encoder250generates the first and second encoded shared representations322,324into a shared latent representation space compatible with the auxiliary decoder390.

In particular, the shared encoder250receives, as input, each encoded textual representation312that corresponds to the alignment output402generated from the unspoken textual utterance320and generates, as output, for each of the plurality of output steps, the first encoded shared representation (etext)322that corresponds to the alignment output402at the corresponding output step. The auxiliary decoder390including the phoneme decoder, wordpiece decoder, or the byte decoder receives, as input, each first encoded shared representation322output from the shared encoder250and generates, as output, a first probability distribution392over possible speech recognition hypotheses for the corresponding alignment output402at the corresponding time step. In some examples, the first probability distribution392over possible speech recognition hypotheses includes one of possible phoneme labels, possible word piece labels, or possible grapheme labels. Thereafter, a supervised loss module340may determine an alignment output loss term342based on the first probability distribution392over possible speech recognition hypotheses for the alignment output402corresponding to the unspoken textual utterance320. Here, the corresponding unspoken textual utterance320in which the alignment output402is generated from, also serves as a ground-truth transcription. The supervised loss part300bmay train the audio encoder210on the alignment output loss term342by updating parameters of the audio encoder210based on the alignment output loss term342.

Similarly, during the supervised loss part300b, the shared encoder250receives, as input, each transcribed encoded audio representation314that corresponds to the labeled speech utterance307and generates, as output, for each of the plurality of output steps, a second encoded shared representation (esup)324that corresponds to the labeled speech utterance307at the corresponding output step. The auxiliary decoder390including the phoneme decoder, the wordpiece decoder, or the byte decoder receives, as input, each second encoded shared representation324output from the shared encoder250and generates, as output, a second probability distribution394over possible non-synthetic speech recognition hypotheses for the corresponding labeled speech utterance307at the corresponding output step. In some examples, the second probability distribution394over possible non-synthetic speech recognition hypotheses includes the one of possible phoneme labels, the possible word piece labels, or the possible grapheme labels. Thereafter, the supervised loss module340may determine a non-synthetic speech loss term344based on the second probability distribution394over possible non-synthetic speech recognition hypotheses and the corresponding transcription302,303paired with the labeled speech utterances307. Here, the corresponding transcription302,303serves as a ground-truth transcription and may include a sequence of target phonemes, target word pieces, and/or target graphemes. The supervised loss part300bmay train the audio encoder210on the non-synthetic speech loss term344by updating parameters of the audio encoder210based on the non-synthetic speech loss term344.

In some implementations, the supervised loss part300bof the training process300uses another auxiliary decoder390to generate a third probability distribution393over possible speech recognition hypotheses based on the first encoded shared representation (etext)322for the alignment output402at the corresponding output step, whereby the supervised loss module340may determine another alignment output loss term342based on the third probability distribution393and the unspoken textual utterance320corresponding to the alignment output402. Here, the other auxiliary decoder390includes the other one of the phoneme decoder, word piece decoder, or the grapheme decoder and the third probability distribution393over possible speech recognition hypotheses includes the other one of the possible phoneme labels, the possible word piece labels, or the possible grapheme labels. In these implementations, the other auxiliary decoder390also generates a fourth probability distribution395over possible non-synthetic speech recognition hypotheses for the corresponding second encoded shared representation324at the corresponding output step, whereby the supervised loss module340may determine another non-synthetic speech loss term344based on the fourth probability distribution395and the corresponding transcription302that is paired with the transcribed non-synthetic speech representation304. Here, the fourth probability distribution395over possible non-synthetic speech recognition hypotheses includes the other one of the possible phoneme labels, the possible word piece labels, or the possible grapheme labels. The supervised loss part300bof the training process300may similarly the audio encoder210on the other alignment output loss term342and the other non-synthetic speech loss term344.

The un-transcribed non-synthetic speech utterances306and the unspoken textual utterances320each correspond to “unpaired” training data whereby the contrastive loss (Lw2v)316derived from the unspoken textual utterances (Xtext)320may be combined with the supervised lossauxassociated with the alignment output loss term342to obtain an unspoken textual loss function,text, as follows.

Likewise, the contrastive loss (Lw2v)316derived from the un-transcribed non-synthetic speech utterances (Xunsup)306may be used to express an unsupervised speech loss function,unsup_speech, as follows.

During training of the audio encoder210, the alignment outputs402and the un-transcribed non-synthetic utterances306may be separated or mixed within each batch. In order to force the audio encoder210to learn representations that are effective for both alignment outputs402corresponding to unspoken textual utterances320and non-synthetic (human/real) speech, the loss mask σ is applied when combining the loss functionstextto obtain an unpaired data loss function,unpaired, as follows:

The labeled speech utterances307corresponds to “paired” and “supervised” training data whereby the derived contrastive loss Lw2vand the derived supervised lossauxassociated with the non-synthetic speech loss term344may be combined to obtain a paired data loss function,paired, as follows:

Referring toFIG.3C, the consistency regularization part (i.e., modality matching part)300cof the training process300is configured to promote the audio encoder210to learn consistent predictions between non-synthetic speech (e.g., real/human speech) and alignment outputs402corresponding to unspoken textual utterances320by generating a consistent loss term (cons(θ))352between training utterance pairs301that each include a corresponding one of the labeled speech utterances (Xsup)307and a paired alignment output404of the same utterance as the corresponding labeled speech utterance307. As such, the labeled speech utterance307and the paired alignment output404of each training utterance pair301is associated with a same ground-truth transcription. In short, the consistent loss term352between the transcribed non-synthetic speech utterance304and paired alignment output404of the same training utterance provides an unsupervised training aspect by encouraging the audio encoder210to behave consistently regardless of whether the training utterance belongs to non-synthetic speech (i.e., speech training data) or the alignment output (i.e., text training data) and independent of supervised loss terms between the ground-truth transcription302,303and each of: non-synthetic speech recognition hypotheses output by the auxiliary decoder390; and speech recognition hypotheses output by the auxiliary decoder390.

Similar to the alignment outputs402generated from the unspoken textual utterances320inFIG.3B, the alignment model400may generate each paired alignment output404using the corresponding transcription302,303that is paired with the labeled speech utterance307. Here, the non-synthetic speech representation304is associated with paired alignment output404generated by the alignment model400mapping the unspoken textual utterance320into speech frames. During the consistency regularization part300c, the text encoder202receives, as input, each paired alignment output404and generates, as output, for each of the plurality of output steps, an encoded textual representation313that corresponds to the paired alignment output404at the corresponding output step. The shared encoder250receives, as input, the encoded textual representation313and generates, as output, a first encoded shared representation (e*sup)323. The auxiliary decoder390including the phoneme decoder or the wordpiece decoder receives, as input, each first encoded shared representation323output from the shared encoder250and generates, as output, a first probability distribution311over possible speech recognition hypotheses for the corresponding paired alignment output404at the corresponding output step. In some examples, the first probability distribution311over possible speech recognition hypotheses includes one of possible phoneme labels or possible word piece labels.

Similarly, the speech encoder204receives, as input, each labeled speech utterance307as a sequence of features/vectors (e.g., mel-frequency spectrograms such as the acoustic frames110ofFIG.1) and generates, as output, for each of a plurality of output steps, a encoded audio representation314that corresponds to the labeled speech utterance307at the corresponding output step. The shared encoder250receives, as input, the encoded audio representation314and generates, as output, a second encoded shared representation (esup)324. The auxiliary decoder390including the phoneme decoder or the wordpiece decoder receives, as input, each second encoded shared representation324output from the shared encoder250and generates, as output, a second probability distribution394over possible non-synthetic speech recognition hypotheses for the corresponding labeled speech utterance at the corresponding time step. In some examples, the second probability distribution394over possible non-synthetic speech recognition hypotheses includes the one of the possible phoneme labels or the possible word piece labels.

With continued reference toFIG.3C, the consistency regularization part300cof the training process300further determines, at each of the plurality of time steps for each training utterance pair301, the consistent loss term (cons(θ))352for the corresponding training utterance pair301based on the first probability distribution311over possible speech recognition hypotheses and the second probability distribution394over possible non-synthetic speech recognition hypotheses. For instance, the training process300may employ a consistency loss term module350configured to receive, at each time step, the corresponding non-synthetic speech and speech recognition results311,394output by the auxiliary decoder390, and determine the consistency loss term352for the corresponding training utterance pair301at the time step.

In some examples, the consistency regularization part300cof the training process300determines the consistent loss term352based on a Kullback-Leibler divergence (DKL) between the first probability distribution311over possible speech recognition hypotheses and the second probability distribution394over possible non-synthetic speech recognition hypotheses. The consistent loss term352based on DKLmay be expressed by the following equation:

Here, the consistent loss term352determined for the training utterance pair301at each time step provides an “unsupervised” loss term that is independent of the accuracy of the auxiliary decoder390(e.g., independent of the supervised loss terms342,344ofFIG.3B), and thus, may be employed to update parameters of the audio encoder210for promoting consistency between non-synthetic speech representations and alignment outputs of the same utterances. In batch training, the consistent loss term352may correspond to an average loss term obtained for the batch. In other words, the consistent loss term352permits the audio encoder210to learn to behave the same, e.g., make consistent encoded representation predictions on both non-synthetic speech (e.g., real/human speech) and alignment outputs of a same training utterance, regardless of whether the training utterance belongs to non-synthetic speech or alignment outputs.

Lastly, the training process300may combine the unpaired data loss function (unpaired), the paired data loss function (J paired), and the consistent loss term (cons) to obtain an overall loss term,tts4pretrain2, that may be expressed as follows.

where λ1may be equal to 1.0 and λ2is equal to 0.1. The training process300may pre-train the audio encoder210using the overall loss term,tts4pretrain2, by updating parameters of the audio encoder210to effectively teach the audio encoder210to learn shared representations between speech and text in the target language even though no labeled training data in the target language is available. After training the audio encoder210, the training process300may fine-tune the pre-trained audio encoder on the labeled speech utterances that may include supervised training samples of both alignment outputs corresponding to unspoken textual utterance320and non-synthetic (e.g., human speech).

In some implementations, the training process300for training the audio encoder210applies encoder consistency regularization. Unlike decoder consistency regularization applied to auxiliary decoder(s) during the consistency regularization part300cthat requires hypothesized labels (e.g., transcripts302and unspoken textual utterances320), encoder consistency regularization does not require hypothesized labels and therefore has the advantage being allowed to be applied to all the training data307,306,320. Encoder consistency regularization may be applied via Hierarchical Contrastive consistency Regularization (HCCR) techniques where encoder activations e, e* from original/non-augmented and augmented speech are projected through an auxiliary network to generate z and z*. Thereafter, positive and negative pairs are constructive and a contrastive loss lt,z,z*is calculated as follows:

Specific to HCCR, a Convolutional Neural Network (CNN) projection network may calculate projections over increasing length segments of encoder activations e (30, 50, 120 ms) to yield 3 views (V) and draw negative examples from the same utterance for short segments, and from other utterances in the batches with 120 ms segments. Accordingly, an HCCR loss may be calculated over the labeled speech utterances307(paired speech), the un-transcribed non-synthetic speech utterances306(unpaired speech), and the alignment outputs402generated from the unspoken textual utterances320as follows:

The HCCR loss calculated by Equation 11 may be added to Equation 9 with a coefficient of 1e-3 as part of the overall loss term,tts4pretrain2, for use in pre-training the audio encoder210.

In some implementations, the training process300may be employed to train end-to-end ASR models with decoder structures (i.e., non-pre-training) or fine-tune an ASR model to perform downstream tasks such as speech translation or natural language understanding. Moreover, implementations described above describe the training process using each part300a-cof the training process300. Yet, it is understood any combination of the training parts300a-cmay be used to train the audio encoder210using any combination of unspoken textual utterances320, labeled speech utterances307, and/or untranscribed non-synthetic speech utterances306independently.

FIG.5illustrates an example training data generation process (e.g., generation process)500that generates the modified speech utterances305and the unspoken textual utterances320used by the training process300(FIGS.3A-3C) to train the audio encoder210. In some implementations, the training data also includes unmodified speech utterances502in the target domain. As will become apparent, the generation process500generates the modified speech utterances305and the unspoken textual utterances320from the unmodified speech utterances502. In contrast to the modified speech utterances305, the unmodified speech utterances502have not been modified to obfuscate one or more classes of sensitive information recited in the utterances. Simply put, the unmodified speech utterances502include utterances spoken in the target domain and non-redacted sensitive information. Notably, due to the sensitive information in the unmodified speech utterances502, the training process300(FIGS.3A-3C) may be unable to directly use the unmodified speech utterances502for various ethical and legal concerns.

In some examples, each unmodified speech utterance502is paired with an unmodified transcription representing the unmodified speech utterance502(e.g., including the sensitive information). In other examples, each unmodified speech utterance502is not paired with any corresponding transcription such that the unmodified speech utterances502include audio-only data. The target domain may be any speech-related domain that includes entity-identifying information, private information, and/or where ethical use of the sensitive information requires special care. Utterances within the target domain may include a mixture of non-sensitive information and one or more classes of sensitive information. For example, the one or more classes of sensitive information may include at least one of personally identifiable information (PII), protected health information (PHI), medical records, financial records, or dates. The non-sensitive information includes information that can be publicly known and is spoken in conjunction with the sensitive information. For instance, the unmodified speech utterance502of “John was admitted” includes the sensitive information of “John” and the non-sensitive information of “was admitted.”

In some implementations, the generation process500includes a redactor510and a tokenizer520. The redactor510is configured to process each unmodified speech utterance502in the training data and generate a corresponding modified speech utterance305and a corresponding modified transcription303. Since some portions of the unmodified speech utterances502include sensitive information, the redactor510identifies the sensitive information (if any) included in each unmodified speech utterance502and redacts the identified sensitive information. For instance, in some implementations, the redactor510generates each modified speech utterance305by replacing the identified sensitive information with silent audio data to redact the audio data associated with the sensitive information. In other implementations, the redactor510generates each modified speech utterance305by synthesizing speech corresponding to the identified sensitive information and replacing the audio data associated with the identified sensitive information with the synthesized speech. The synthesized speech may include information that is different from, but associated with, the identified sensitive information such that the synthesized speech does not reveal the actual sensitive information. Accordingly, the redactor510may optionally include a trained text-to-speech (TTS) or voice conversion model511.

Continuing with the above example, for the unmodified speech utterance502of “John was admitted”, the redactor510identifies “John” as sensitive information and samples a random fake name such as “Bill” and synthesizes the sampled random fake name instead of the actual name. Thereafter, the redactor510replaces audio data associated with “John” with synthesized audio data associated with “Bill.” Thus, in this example, the modified speech utterance305includes “Bill was admitted” where the term “Bill” is synthesized speech while the rest of the utterance is non-synthetic speech.

The redactor510also generates a corresponding modified transcription303for each unmodified speech utterance502. Thus, the modified transcriptions303correspond to the modified speech utterances305. As such, the redactor510may include a trained ASR model513configured to transcribe each unmodified speech utterance302. Notably, the redactor510replaces text associated with identified sensitive information with corresponding class identifiers (e.g., markup tags)512. That is, rather than generating text representing the sensitive information, the redactor510inserts class identifiers512in place of text representing the sensitive information thereby effectively redacting or obfuscating the sensitive information. Stated differently, the redactor510tags the redacted portions of the transcriptions303of the modified speech utterances305with the class identifier512. In some examples, each class identifier512indicates a particular class from the one or more classes of sensitive information that has been redacted. Classification identifiers512may indicate that the redacted text corresponds to a patient name, a date or time, a medical record, etc. For example, a respective classification identifier512for text representing a patient name may be “[PATIENT_NAME]” rather than text representing the actual name of the patient. Thus, the redactor510may classify the identified sensitive information and generate the classification identifier512, which indicates the particular class of the sensitive information, based on the classification.

The tokenizer520is configured to receive, as input, each modified transcription303and generate, as output, corresponding unspoken textual utterances320. In short, the tokenizer520inserts fake random sampling data535into each redacted portion of the transcriptions303of the modified speech utterances305. The inserted fake random sampling data535is associated with the class of sensitive information identified by the class identifier tag512at the redacted portion. More specifically, for each modified transcription303, the tokenizer520obtains corresponding fake random sampling data535from a database of fake random sampling data530. The database of fake random sampling data530,530a-nmay include a respective database of fake random sampling data530for each potential class identifier512. That is, each respective database of fake random sampling data530includes a random distribution of data corresponding to one of the class identifiers512. The particular class identifier512serves to indicate to the tokenizer520which respective database of fake random sampling data530to sample from. For example, a respective database of fake random sampling data530corresponding to a name class identifier512includes a random distribution of names (e.g., Mark, Matt, Adam, etc.).

Accordingly, the tokenizer520obtains corresponding fake random sampling data535for each respective modified transcription303based on the class identifiers512included in the respective modified transcription303. Thereafter, the tokenizer520replaces the class identifiers512with the obtained fake random sampling data535associated with the class identifier512. In some examples, the tokenizer520samples multiple instances of fake random sampling data535for a single particular class identifier512. In these examples, the tokenizer replaces the particular class identifier512with each instance of fake random sampling data535such that the tokenizer520generates multiple unspoken textual utterances320from a single modified transcription303.

In the example shown, the redactor510receives an unmodified speech utterance502of “Joe had no past surgeries” where “Joe” represents sensitive information and the rest of the utterance represents non-sensitive information. The redactor510generates a modified speech utterance305and a modified transcription303based on the unmodified speech utterance502. The redactor510generates the modified speech utterance305by replacing audio data associated with “Joe” with silence audio data or synthesizing speech using another name (e.g., Jared) and replacing the audio data associated with “Joe” with the synthesized speech. Moreover, the redactor510generates the modified transcription303by identifying the sensitive information of “Joe,” classifying the sensitive information as a name, and replacing the identified sensitive information with the class identifier512indicating that the sensitive information corresponds to a name. As such, the modified transcription303results in “[NAME] had no past surgeries.”

Subsequently, the tokenizer520receives the modified transcription303and obtains fake random sampling data535by sampling data associated with the particular class identifier512(e.g., names) and replaces the particular class identifier512with the fake random sampling data535. Here, since the particular class identifier512represents a name, the tokenizer520samples data from a random distribution of fake names and inserts the fake random sampling data535in place of the particular class identifier512resulting in the unspoken textual utterance320of “Jared had no past surgeries.” Notably, the unspoken textual utterance320does not reveal the sensitive information from the unmodified speech utterance502. Yet, the fake random sampling data535is closely related to the sensitive information such that, by using the unspoken textual utterances320to train the ASR model200, the ASR model200will be able to accurately recognize phrases within the one or more classes of sensitive information.

Advantageously, the generation process500receives unmodified speech utterances502that include sensitive information and outputs unspoken textual utterances320that replace the sensitive information with fake random sampling data535. Training on the unspoken textual utterances320teaches the ASR model200to recognize speech in the target domain. Moreover, the fake random sampling data535inserted into the unspoken textual utterances320teaches the ASR model200to recognize terms or phrases within the one or more classes of sensitive information without training the ASR model200directly on the sensitive information included in the unmodified speech utterances502. The training process300(FIGS.3A-3C), uses the text-only of the unspoken textual utterances320to train the ASR model200without synthesizing the unspoken textual utterances320.

FIG.6is a flowchart of an example arrangement of operations for a computer-implemented method600of training a speech recognition model using text-injection. The method600may execute on data processing hardware710(FIG.7) using instructions stored on memory hardware720(FIG.7). The data processing hardware710and the memory hardware720may reside on the remote computing device201and/or the user device102ofFIG.1each corresponding to a computing device700(FIG.7).

At operation602, the method600includes receiving training data including transcribed speech utterance304spoken in a general domain, modified speech utterances305in a target domain, and unspoken textual utterances320. Each transcribed speech utterance304is paired with a corresponding transcription302. The modified speech utterances305include utterances spoken in the target domain that have been modified to obfuscate one or more classes of sensitive information recited in the utterances. Moreover, each modified speech utterance305is paired with a corresponding transcription303that redacts the sensitive information obfuscated from the modified speech utterance305. The unspoken textual utterances320correspond to the transcriptions of the modified speech utterances205in the target domain. The unspoken textual utterances320include fake random data535inserted into redacted portions of the transcriptions303of the modified speech utterances305where the sensitive information recited in the modified speech utterance has been redacted. At operation604, the method600includes generating, using an alignment model400, a corresponding alignment output402for each unspoken textual utterance320of the received training data. At operation606, the method600includes training a speech recognition model200on the alignment outputs402generated for the unspoken textual utterances320, the modified speech utterances305, and the transcribed speech utterances304to teach the speech recognition model200to learn to recognize speech in the target domain and phrases within the one or more classes of sensitive information.

The computing device700includes a processor710, memory720, a storage device730, a high-speed interface/controller740connecting to the memory720and high-speed expansion ports750, and a low speed interface/controller760connecting to a low speed bus770and a storage device730. Each of the components710,720,730,740,750, and760, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor710can process instructions for execution within the computing device700, including instructions stored in the memory720or on the storage device730to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display780coupled to high speed interface740. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices700may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The storage device730is capable of providing mass storage for the computing device700. In some implementations, the storage device730is a computer-readable medium. In various different implementations, the storage device730may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory720, the storage device730, or memory on processor710.

The high speed controller740manages bandwidth-intensive operations for the computing device700, while the low speed controller760manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller740is coupled to the memory720, the display780(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports750, which may accept various expansion cards (not shown). In some implementations, the low-speed controller760is coupled to the storage device730and a low-speed expansion port790. The low-speed expansion port790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device700may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server700aor multiple times in a group of such servers700a, as a laptop computer700b, or as part of a rack server system700c.