Synthesized data augmentation using voice conversion and speech recognition models

A method for training a speech conversion model personalized for a target speaker with atypical speech includes obtaining a plurality of transcriptions in a set of spoken training utterances and obtaining a plurality of unspoken training text utterances. Each spoken training utterance is spoken by a target speaker associated with atypical speech and includes a corresponding transcription paired with a corresponding non-synthetic speech representation. The method also includes adapting, using the set of spoken training utterances, a text-to-speech (TTS) model to synthesize speech in a voice of the target speaker and that captures the atypical speech. For each unspoken training text utterance, the method also includes generating, as output from the adapted TTS model, a synthetic speech representation that includes the voice of the target speaker and that captures the atypical speech. The method also includes training the speech conversion model based on the synthetic speech representations.

TECHNICAL FIELD

This disclosure relates to synthesized data augmentation using voice conversion and speech recognition models.

BACKGROUND

Automatic speech recognition (ASR), the process of taking an audio input and transcribing it into text, has greatly been an important technology that is used in mobile devices and other devices. In general, automatic speech recognition attempts to provide accurate transcriptions of what a person has said by taking an audio input (e.g., speech utterance) and transcribing the audio input into text.

One challenge in developing deep learning-based speech conversion models and ASR models is that parameters of these models tend to over fit the training data, thereby resulting in difficulties generalizing unseen data when the training data is not extensive enough. While training speech conversion models and ASR models on larger training datasets improves accuracy, there is a lack of sufficient training data including utterances targeting specific domains (i.e., linguistic diversity) that are spoken by speakers with atypical speech patterns (i.e., acoustic diversity) to achieve acceptable accuracy by these models.

SUMMARY

One aspect of the disclosure provides a method for training a speech conversion model personalized for a target speaker associated with atypical speech. The method includes obtaining, by data processing hardware, a plurality of training text utterances. A first portion of the plurality of training text utterances includes a plurality of transcriptions in a set of spoken training utterances. Each spoken training utterance is spoken by a target speaker associated with atypical speech and includes a corresponding transcription paired with a corresponding non-synthetic speech representation of the corresponding spoken training utterance. A second portion of the plurality of training text utterances includes a plurality of unspoken training text utterances pertaining to a specific domain in which the speech conversion model is trained to learn. Each unspoken training text utterance is not paired with any corresponding spoken utterance. The method also includes adapting, by the data processing hardware, using the set of spoken training utterances, a text-to-speech (TTS) model to synthesize speech in a voice of the target speaker and that captures the atypical speech associated with the target speaker. For each unspoken training text utterance of the plurality unspoken training text utterances, the method includes generating, by the data processing hardware, as output from the adapted TTS model, a synthetic speech representation of the corresponding unspoken training text utterance. The synthetic speech representation includes the voice of the target speaker and captures the atypical speech associated with the target speaker. The method also includes training, by the data processing hardware, the speech conversion model based on the synthetic speech representation generated by the adapted TTS model for each unspoken training text utterance of the plurality of unspoken training text utterances.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, for each synthetic speech representation generated by the adapted TTS model, the method further includes generating, by the data processing hardware, as output from the speech conversion model, a corresponding audio waveform of synthesized canonical fluent speech in the voice of the target speaker, generating, by the data processing hardware, as output from a text decoder, a textual representation for the corresponding audio waveform of synthesized canonical fluent speech generated as output from the speech conversion model; and determining, by the data processing hardware, a word error rate loss associated with the corresponding synthetic speech representation. The word error rate loss is based on the textual representation generated as output form the text decoder for the corresponding audio waveform of synthesized canonical fluent speech and the corresponding unspoken training text utterance. In these implementations, the method also includes, identifying, by the data processing hardware, a filtered set of synthetic speech representations. Each filtered set of synthetic speech representations corresponds to a respective one of the synthetic speech representations generated as output from the speech conversion model that has a word error rate loss that satisfies a word error rate loss threshold. In these implementations, training the speech conversion model based on the synthetic speech representation generated by the adapted TTS model for each unspoken text utterance of the plurality of unspoken text utterances includes training the speech conversion model on the filtered set of synthetic speech representations. The speech conversion model does not train on any of the synthetic speech representations generated as output form the speech conversion model that have word error rate losses that do not satisfy the word error rate loss threshold.

In some examples, the method further includes, when the speech conversion model is not previously trained to convert audio waveforms of input utterances spoken by speakers having a same type of atypical speech as the atypical speech associated with the target speaker, adapting, by the data processing hardware, using the set of spoken training utterances, the speech conversion model to convert audio waveforms of input utterances spoken by the target speaker with atypical speech into audio waveforms of synthesized canonical fluent speech. Here, generating the corresponding audio waveform of synthesized canonical fluent speech includes generating, as output from the adapted speech conversation model, the corresponding audio waveform of synthesized canonical fluent speech in the voice of the target speaker. In some examples, the text decoder resides on the speech conversion model. In other examples, the text decoder resides on a reference automated speech recognition model separate from the speech conversion model.

In some implementations, the speech conversion model includes an end-to-end neural network configured to convert input audio waveforms directly into corresponding output audio waveforms. In these implementations, after training the speech conversion model, the method may also include receiving, at the data processing hardware, an input audio waveform corresponding to an utterance spoken by the target speaker associated with atypical speech, and converting, by the data processing hardware, using the trained speech conversion model, the input audio waveform corresponding to the utterance spoken by the target speaker associated with atypical speech into an output audio waveform corresponding to a synthesized canonical fluent speech representation of the utterance spoken by the target speaker.

In other implementations, the speech conversion model includes an automated speech recognition model configured to convert speech into corresponding text. In these implementations, after training the speech conversion model, the method may also include receiving, by the data processing hardware, audio data corresponding to an utterance spoken by the target speaker associated with atypical speech; and converting, by the data processing hardware, using the trained speech conversion model, the audio data corresponding to the utterance spoken by the target speaker associated with atypical speech into a canonical textual representation of the utterance spoken by the target speaker.

At least a portion of the plurality of unspoken training text utterances in the second portion of the plurality of training text utterances may include manually written text targeting particular phrases that pertain to the particular domain. Optionally, the method may include executing, by the data processing hardware, an unspoken text selection process to obtain the unspoken training text utterances in the unspoken training text utterances in the second portion of the plurality of training text utterances. The text selection process is configured to obtain a corpus of unspoken text utterances. For each unspoken text utterance in the corpus of unspoken text utterances, the text selection process is configured to determine a first probability associated with the unspoken text utterance appearing in a domain-specific language model and determine a second probability associated with the unspoken text utterance appearing in a background language model. The background language model is trained on every unspoken text utterance in the corpus of unspoken text utterances. For each unspoken text utterance in the corpus of unspoken text utterance, the text selection process is also configured to determine a score based on the first probability, the second probability, and a number of words appearing in the corresponding unspoken text utterance. Finally, the text selection process is configured to select, as the unspoken training text utterances in the second portion of the plurality of training text utterances, the unspoken text utterances in the corpus of unspoken text utterances that have the N-best scores.

In some implementations, the TTS model includes a pre-trained reference TTS model that includes an encoder portion and a decoder portion. In these implementations, adapting the TTS model includes adapting the pre-trained reference TTS model by tuning parameters of the decoder portion while parameters of the encoder portion remain fixed.

Another aspect of the disclosure provides a system for training a speech conversion model personalized for a target speaker associated with atypical speech. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include obtaining a plurality of training text utterances. A first portion of the plurality of training text utterances includes a plurality of transcriptions in a set of spoken training utterances. Each spoken training utterance is spoken by a target speaker associated with atypical speech and includes a corresponding transcription paired with a corresponding non-synthetic speech representation of the corresponding spoken training utterance. A second portion of the plurality of training text utterances includes a plurality of unspoken training text utterances pertaining to a specific domain in which the speech conversion model is trained to learn. Each unspoken training text utterance is not paired with any corresponding spoken utterance. The operations also include adapting, using the set of spoken training utterances, a text-to-speech (TTS) model to synthesize speech in a voice of the target speaker and that captures the atypical speech associated with the target speaker. For each unspoken training text utterance of the plurality unspoken training text utterances, the operations include generating, as output from the adapted TTS model, a synthetic speech representation of the corresponding unspoken training text utterance. The synthetic speech representation includes the voice of the target speaker and captures the atypical speech associated with the target speaker. The operations also include training the speech conversion model based on the synthetic speech representation generated by the adapted TTS model for each unspoken training text utterance of the plurality of unspoken training text utterances.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, for each synthetic speech representation generated by the adapted TTS model, the operations further include generating, as output from the speech conversion model, a corresponding audio waveform of synthesized canonical fluent speech in the voice of the target speaker, generating as output from a text decoder, a textual representation for the corresponding audio waveform of synthesized canonical fluent speech generated as output from the speech conversion model; and determining a word error rate loss associated with the corresponding synthetic speech representation. The word error rate loss is based on the textual representation generated as output form the text decoder for the corresponding audio waveform of synthesized canonical fluent speech and the corresponding unspoken training text utterance. In these implementations, the operations also include, identifying a filtered set of synthetic speech representations. Each filtered set of synthetic speech representations corresponds to a respective one of the synthetic speech representations generated as output from the speech conversion model that has a word error rate loss that satisfies a word error rate loss threshold. In these implementations, training the speech conversion model based on the synthetic speech representation generated by the adapted TTS model for each unspoken text utterance of the plurality of unspoken text utterances includes training the speech conversion model on the filtered set of synthetic speech representations. The speech conversion model does not train on any of the synthetic speech representations generated as output form the speech conversion model that have word error rate losses that do not satisfy the word error rate loss threshold.

In some examples, the operations further include, when the speech conversion model is not previously trained to convert audio waveforms of input utterances spoken by speakers having a same type of atypical speech as the atypical speech associated with the target speaker, adapting, using the set of spoken training utterances, the speech conversion model to convert audio waveforms of input utterances spoken by the target speaker with atypical speech into audio waveforms of synthesized canonical fluent speech. Here, generating the corresponding audio waveform of synthesized canonical fluent speech includes generating, as output from the adapted speech conversation model, the corresponding audio waveform of synthesized canonical fluent speech in the voice of the target speaker. In some examples, the text decoder resides on the speech conversion model. In other examples, the text decoder resides on a reference automated speech recognition model separate from the speech conversion model.

In some implementations, the speech conversion model includes an end-to-end neural network configured to convert input audio waveforms directly into corresponding output audio waveforms. In these implementations, after training the speech conversion model, the operations may also include receiving an input audio waveform corresponding to an utterance spoken by the target speaker associated with atypical speech; and converting, using the trained speech conversion model, the input audio waveform corresponding to the utterance spoken by the target speaker associated with atypical speech into an output audio waveform corresponding to a synthesized canonical fluent speech representation of the utterance spoken by the target speaker.

In other implementations, the speech conversion model includes an automated speech recognition model configured to convert speech into corresponding text. In these implementations, after training the speech conversion model, the method may also include receiving audio data corresponding to an utterance spoken by the target speaker associated with atypical speech; and converting, using the trained speech conversion model, the audio data corresponding to the utterance spoken by the target speaker associated with atypical speech into a canonical textual representation of the utterance spoken by the target speaker.

At least a portion of the plurality of unspoken training text utterances in the second portion of the plurality of training text utterances may include manually written text targeting particular phrases that pertain to the particular domain. Optionally, the operations may include executing an unspoken text selection process to obtain the unspoken training text utterances in the unspoken training text utterances in the second portion of the plurality of training text utterances. The text selection process is configured to obtain a corpus of unspoken text utterances. For each unspoken text utterance in the corpus of unspoken text utterances, the text selection process is configured to determine a first probability associated with the unspoken text utterance appearing in a domain-specific language model and determine a second probability associated with the unspoken text utterance appearing in a background language model. The background language model is trained on every unspoken text utterance in the corpus of unspoken text utterances. For each unspoken text utterance in the corpus of unspoken text utterance, the text selection process is also configured to determine a score based on the first probability, the second probability, and a number of words appearing in the corresponding unspoken text utterance. Finally, the text selection process is configured to select, as the unspoken training text utterances in the second portion of the plurality of training text utterances, the unspoken text utterances in the corpus of unspoken text utterances that have the N-best scores.

In some implementations, the TTS model includes a pre-trained reference TTS model that includes an encoder portion and a decoder portion. In these implementations, adapting the TTS model includes adapting the pre-trained reference TTS model by tuning parameters of the decoder portion while parameters of the encoder portion remain fixed.

DETAILED DESCRIPTION

Automated speech recognition (ASR) has made tremendous strides with the introduction of end-to-end (E2E) deep learning-based models to recognize speech from speakers with atypical speech patterns for conversion into accurate transcriptions. For instance, atypical speech patterns may include, without limitation, impaired speech due to physical or neurological conditions (e.g., speakers with Amyotrophic Lateral Sclerosis (ALS) disease), heavily-accented speech, and deaf speech. Similar deep learning-based models can be applied by speech-to-speech conversion systems to convert speech with atypical speech patterns into canonical fluent output speech. As used herein, and unless specified otherwise, the terms “speech conversion system” and “speech conversion model” can refer to either an ASR system/model, in which input atypical speech is recognized and converted into corresponding text (e.g., transcription), or a speech-to-speech conversion system/model, in which the input atypical speech is directly converted into canonical fluent synthesized speech without performing speech recognition. Stated differently, the speech-to-speech conversion system/model is configured to convert an input audio waveform or spectrograms corresponding to the atypical speech directly into an output audio waveform or spectrograms corresponding to the canonical fluent speech without convening the input audio waveform into an intermediate representation (e.g., text or phonemes). As will become apparent, speech conversion models, as well as techniques for training speech conversion models, will enable a user with atypical speech to speak to, and be understood by, both other humans and speech interfaces (e.g., digital assistants) by enabling recognition and/or reproduction of the user's intended speech.

One challenge in developing deep learning-based speech conversion models and ASR models is that parameters of these models tend to over fit the training data, thereby resulting in difficulties generalizing unseen data when the training data is not extensive enough. While training speech conversion models and ASR models on larger training datasets improves accuracy, there is a lack of training data that provides both linguistic diversity and acoustic diversity sufficient for personalization toward a target speaker having atypical speech and in a specific target domain. For instance, to attain training data with sufficient acoustic diversity, the target speaker having the atypical speech would have to record hours of spoken utterances each paired with a corresponding transcription. Moreover, attaining sufficient linguistic diversity for the specific target domain would require the utterances recorded from the speaker to include terms associated with the specific target domain. For example, the specific target domain may include, without limitation, an occupational field (e.g., a retina specialized physician), an education discipline (e.g., a lecturer on psychology), music, navigation, or weather. The specific target domain could similarly be a personalized domain associated with the target speaker, in which specific terms associated with the personalized domain could include names of family members, contact names, music artist/album/songs in a music library associated with the target speaker, etc.

Text-to-speech (TTS) or speech syntheses systems have successfully applied Seq2Seq models to obtain state of the art natural, realistic sounding synthesized speech that can be indistinguishable to the human ear from human speech. Advantageously, unspoken text utterances, or text-only data, can be easily and cheaply obtained to produce synthesized speech for improving training of speech conversion models. For instance, not only can unspoken text utterances be used to increase the volume of training data sets, but the unspoken text utterances can increase linguistic diversity in the training data without the difficulty of having to obtain transcribed speech (e.g., human spoken audio and corresponding transcriptions).

Implementations herein are directed toward improving training data used for training a speech conversion model (i.e., ASR or speech-to-speech conversion) personalized for a target speaker with atypical speech and targeting a specific domain of interest for the target speaker. Specifically, implementations include sampling initial personalized seed data corresponding to transcribed acoustic data of recorded utterances spoken by the target speaker with atypical speech and using the sampled seed data to adapt/tune a baseline text-to-speech (TTS) model. Here, the “baseline TTS model” simply refers to a reference/existing TTS model previously trained to convert input text into synthesized canonical speech in the voice of one or more predefined speakers. Here, the personalized seed data sampled from the target speaker tunes/adapt the baseline TTS model to convert input text into output synthesized speech in the voice of the target speaker and having the atypical speech pattern of the target speaker. The pre-trained baseline TTS model includes an encoder portion and a decoder portion, whereby adapting the TTS model may include tuning/re-training parameters of the decoder portion while parameters of the encoder portion remain fixed. By using the personalized seed data to adapt the TTS model in this manner, the adapted TTS model may be used to convert text utterances, including terms or phrases associated with the specific domain, into synthetic training utterances that include synthesized speech in the voice of the target speaker and having the associated atypical speech patterns of the target speaker. As will become apparent, the adapted TTS model may generate a multitude of synthetic training utterances that target the specific domain and with atypical speech in the voice (i.e., synthesized voice) of the target speaker to increase both linguistic diversity and acoustic diversity in training the speech conversion model.

The synthetic training utterances produced by the adapted TTS model and corresponding transcriptions are used to adapt/tune a baseline speech conversion model. Here, a “baseline speech conversion model” refers to either a reference/existing ASR model, pre-trained on a general corpus of transcribed acoustic data to recognize typical/canonical speech, or a reference/existing speech-to-speech conversion model, trained to map input audio waveforms (or spectrograms) for each of a plurality of utterances from a corpus spanning a variety of speakers and recording conditions to corresponding output audio waveforms (or spectrograms) in a voice of a predefined canonical speaker. Accordingly, the synthetic training utterances provide linguistic diversity and acoustic diversity sufficient for adapting/tuning the general speech conversion model to recognize and/or convert atypical speech spoken by the target speaker, and targeting a specific domain, into canonical text and/or canonical fluent synthesized speech. In these implementations, the sampled seed data corresponding to the transcribed acoustic data of recorded utterances spoken by the target speaker may be further used to adapt/tune the baseline speech conversion model. In other implementations, a combination of the synthetic training utterances produced by the adapted TTS model and the sampled seed data are used to train a speech conversional model from scratch.

FIG. 1Aillustrates a speech conversion model300,300aconfigured to convert input audio data102corresponding to an utterance108spoken by a target speaker104associated with atypical speech into output audio data106corresponding to a synthesized canonical fluent speech representation of the same utterance114spoken by the target speaker104. An associated speech conversion model300of the speech conversion system100aincludes a speech-to-speech (S2S) conversion model300aconfigured to convert the input audio data102(e.g., input spectrogram) directly into the output audio data106(e.g., output spectrogram) without performing speech recognition, or otherwise without requiring the generation of any intermediate discrete representations (e.g., text or phonemes) from the input audio data102. The S2S conversion model300aincludes a spectrogram encoder310configured to encode the input audio data102into a hidden feature representation (e.g., a series of vectors) and a spectrogram decoder320configured to decode the hidden representation into the output audio data106corresponding to the synthesized canonical fluent speech representation. For instance, as the spectrogram encoder310receives the input audio data102of the utterance108, the spectrogram encoder310may process five frames of audio and convert those five frames of audio to ten vectors. The vectors are not a transcription of the frames of audio data102, but rather a mathematical representation of the frames of the audio data102. In turn, the spectrogram decoder320may generate the output audio data106corresponding to the synthesized canonical fluent speech representation based on the vectors received from the spectrogram encoder310. For example, the spectrogram decoder320may receive the ten vectors from the spectrogram encoder310that represent the five frames of audio. Here, the spectrogram decoder320may generate five frames of output audio data106corresponding to the synthesized canonical fluent speech representation of the utterance114that includes the intended words or parts of words as the five frames of the input audio data102, but without the disfluencies of the atypical speech.

In some examples, the S2S conversion model300aalso includes a text decoder (FIG. 2D)250that decodes the hidden representation into a textual representation, e.g., phonemes or graphemes. In these examples, the spectrogram decoder320and the text decoder250may correspond to parallel decoding branches of the S2S conversion model300athat each receive the hidden representation encoded by the spectrogram encoder310and emit their respective one of the output audio data106or the textual representation in parallel. The S2S conversion system100amay further include a synthesizer375to synthesize the output audio data106into a time-domain waveform for audible output as a same utterance114of fluent canonical fluent speech. A time-domain audio waveform includes an audio waveform that defines an amplitude of an audio signal over time. The synthesizer375may include a unit selection module or a WaveNet module for synthesizing the output audio data106into time-domain waveforms of synthesized canonical fluent speech. In some implementations, the synthesizer375includes a vocoder network, i.e., neural vocoder, that is separately trained and conditioned on mel-frequency spectrograms for conversion into time-domain audio waveforms.

In the example shown, the target speaker104is associated with atypical speech such that the target speaker104speaks with an atypical speech pattern that may be difficult to understand. Atypical speech patterns may include, without limitation, impaired speech due to physical or neurological conditions (e.g., speakers with Amyotrophic Lateral Sclerosis (ALS) disease), heavily-accented speech, and deaf speech. By way of example, the target speaker104has ALS disease and is associated with atypical speech due to ALS disease. The speech-to-speech conversion system100ais accordingly trained to covert the input audio data102corresponding to utterances108spoken by the target speaker104associated with ALS speech directly into the output audio data106corresponding to the synthesized canonical fluent speech representation of the same utterance108. Thus, the synthesized canonical fluent speech representation provided by the output audio data106improves intelligibility of the ALS speech spoken by the targets speaker104. Without departing from the scope of the present disclosure, the S2S conversion model300amay be trained to convert input audio data102corresponding to utterances108associated with atypical speech in a first language directly into output audio data106corresponding to a synthesized canonical fluent speech representation of the same utterance108in the same voice, but in a different second language.

A computing device110associated with the target speaker104may capture the utterance108spoken by the target speaker104and transmit the corresponding input audio data102to the speech-to-speech conversion system100afor conversion into the output audio data106. Thereafter, the speech-to-speech conversion system100amay transmit the output audio data106corresponding to the synthesized canonical fluent speech representation of the same utterance114spoken by the target speaker104to another computing device116associated with user118, whereby the other computing device116audibly outputs the synthesized canonical fluent speech representation of the utterance108spoken by the target speaker104. In this example, the target speaker104and the user118are speaking with each other through their respective computing devices110,116, such as over a telephone call or other type of voice communication protocol, for example, voice over internet protocol. While the target speaker104and the other user118may speak the same language, it may be difficult for the other user118to understand the target speaker104because the target speaker104has atypical speech due to ALS disease. Thus, while the target speaker104speaks with atypical speech (e.g., ALS speech) that may be difficult to understand, the other user118hearing the synthesized canonical fluent speech representation will have an easier time understanding the utterance108intended by the target speaker104. Stated differently, the synthesized canonical fluent speech representation provides a more consistent cadence that may be easier for another user to understand than the original utterance108spoken by the target speaker with the atypical speech. Notably, the synthesized canonical fluent speech representation is in the voice of the target speaker104.

In some other examples, the S2S conversion system100amay instead pass the output audio data106corresponding to the synthesized canonical fluent speech representation of the utterance spoken by the target speaker104to an output audio device for audibly outputting the synthesized canonical fluent speech representation in the voice of the target speaker104to an audience. For instance, the target speaker104may be a psychology professor providing a lecture to a class of students, in which utterances spoken by the target speaker104include medical terminology belonging to a particular specific domain, e.g., psychology. As will become apparent, the speech-to-speech conversion model300ais trained to learn linguistic diversity associated with particular domains, as well as to learn acoustic diversity associated with particular types of atypical speech associated with target speakers104.

Alternatively, the other computing device116may be associated with down-stream automated speech recognition (ASR) system in which the speech-to-speech conversion system100afunctions as a front-end to provide the output audio data106corresponding to the synthesized canonical fluent speech representation as an input to the ASR system for conversion into recognized text. The recognized text could be presented to the other user118and/or could be provided to a natural language understanding (NLU) system for further processing. The functionality of the speech-to-speech conversion system100acan reside on a remote server112, on either or both of the computing devices110,116, or any combination of the remote server and computing devices110,116. In some implementations, the S2S conversion model300acontinuously generates output audio data106corresponding to synthesized canonical fluent speech representations of an utterance as the target speaker104speaks corresponding portions of the utterance as atypical speech. By continuously generating output audio data106corresponding to synthesized canonical fluent speech representations of portions of the utterance108spoken by the target speaker104, the conversation between the target speaker104and the user118(or audience) may be more naturally paced. In some additional implementations, the S2S conversion model300awaits to determine/detect when the target speaker104stops speaking, using techniques such as voice activity detection, end pointing, end of query detection, etc., and before converting the corresponding input audio data102of the utterance108with atypical speech into the corresponding output audio data106corresponding to synthesized canonical fluent speech representation of the same utterance114.

Referring now toFIG. 1B, in some implementations, the speech conversion system100,100bincludes a speech-to-text conversion system100bconfigured to convert the input audio data102corresponding to the utterance108spoken by the target speaker104associated with the atypical speech into a canonical textual representation (i.e., a transcription)120of the utterance108. As with the S2S conversion system100aofFIG. 1A, the speech-to-text conversion system100bis not only configured to recognize the particular type of atypical speech (e.g., ALS speech) associated with the target speaker104, but also recognize particular words and/or phrases associated with a particular domain. These particular words and/or phrases may include proper nouns or other terminology that is generally not present, or insufficiently represented, in a general training corpus used to train a general/baseline speech-to-text system.

Accordingly, the speech-to-text conversion system100bmay correspond to a personalized automated speech recognition (ASR) system for the target speaker104that can recognize the target speaker's particular type of atypical speech patterns, as well as linguistic information for a particular domain, for conversion into a corresponding canonical textual representation120that captures the intent of the original utterance108spoken by the target speaker104associated with the atypical speech. Another user118(FIG. 1A) may obtain the canonical textual representation120of the utterance108. In some configurations, the canonical textual representation120output from the system100bis processed, e.g., by a natural language understanding (NLU) module executing on the user device110or the remote server112, to execute a user command. Additionally or alternatively, a text-to-speech system (e.g., executing on any combination of the user device110or the remote server112) may convert the transcription into synthesized speech for audible output by another device. The functionality of the speech-to-text conversion system100bcan reside on the remote server112, on either or both of the computing device110, or any combination of the remote server112and computing device110.

A speech conversion model300associated speech-to-text conversion system100bmay include a speech-to-text conversion model300b(interchangeably referred to as an automated speech recognition (ASR) model300b) configured to perform speech recognition on the utterance108of atypical speech by converting the input audio data102into the canonical textual representation (i.e., transcription)120of the utterance108. The S2S conversion model300aincludes an encoder350configured to encode the input audio data102into a hidden feature representation (e.g., a series of vectors) and a text decoder250configured to decode the hidden representation into the canonical transcription120. For instance, as the text encoder350receives the input audio data102of the utterance108, the encoder350may process five frames of audio and convert those five frames of audio to ten vectors. The vectors are not a transcription of the frames of audio data102, but rather a mathematical representation of the frames of the audio data102. In turn, the text decoder250may generate the canonical transcription120based on the vectors received from the encoder350. For example, the text decoder250may generate a sequence of words corresponding to the canonical transcription120of the utterance180that includes the intended words or parts of words in the five frames of the input audio data102. Without departing from the scope of the present disclosure, the ASR model300bmay be trained to convert input audio data102corresponding to utterances108associated with atypical speech in a first language into a corresponding canonical transcription of the utterance108a different second language.

Referring toFIGS. 1A and 1B, the speech conversion system100executes a training process200configured to train the speech conversion model300, i.e., the S2S conversion model300aofFIG. 1Aand/or the speech-to-text conversion model300bofFIG. 1B. As will be described in greater detail below with reference toFIGS. 2A-2E, the training process200includes a personalized seed data collection stage200a(FIG. 2A), a data generation stage200b(FIG. 2B), an adaption stage200c(FIG. 2C), a validation and filtering stage200d(FIG. 2D), and final training stage200e(FIG. 2E).

Referring toFIG. 2A, the personalized seed data collection stage200aof the training process200includes obtaining a set of spoken training utterances305,305a-nfor the target speaker104associated with atypical speech. Here, each spoken training utterance305is spoken by the target speaker104and includes a corresponding transcription302apaired with a corresponding non-synthetic speech representation304of the corresponding spoken training utterance305. As such, the non-synthetic speech representation304is in a voice of the target speaker104and includes the atypical speech patterns for the type of atypical speech (e.g., ALS speech) associated with the target speaker104. The transcription302ain the set of spoken training utterances305may form a first portion of a plurality of training text utterances302. Each transcription302amay be a canonical transcription in the native speaking language of the target speaker104. In some examples, some or all of the spoken training utterances305include words and/or phrases pertaining to a specific domain in which the speech conversion model300is trained to learn.

In some implementations, the personalized seed data collection stage200aprompts the target speaker104to speak each spoken training utterance305and records the utterance to obtain the corresponding non-synthetic speech representation304. Each non-synthetic speech representation304obtained for the target speaker104may be paired with the corresponding transcription302aof the spoken training utterance305. As such, each spoken training utterance305includes manually-transcribed acoustic data302a,304spoken by the target speaker104. In the example shown, the personalized seed data collection stage200aprompts the user to speak each spoken training utterance305by displaying the corresponding transcription302aon a graphical user interface of the computing device110associated with the target speaker104. This may include a separate prompt for each spoken training utterance, or may include prompting the target speaker to speak any number of consecutive spoken training utterances at a time. Additionally or alternatively, the computing device110may audibly output a prompt for the target speaker to speak each training utterance (e.g., “Please speak the following phrase”. The set of spoken training utterances305may be stored in a data store202overlain on memory hardware420(FIG. 4). In some examples, the personalized seed data collection stage200acollects about five-hundred (500) spoken training utterances305.

Referring toFIG. 2B, the data generation stage200bof the training process200includes obtaining a plurality of unspoken training text utterances302bpertaining to the specific domain in which the speech conversion model300is being trained to learn. For example, the target speaker104may be a psychology professor such that the specific domain includes psychology terminology for college-level psychology courses. Each unspoken training text utterance302bis not paired with any corresponding spoken utterance. The plurality of unspoken training text utterances302bmay form a second portion of the plurality of training text utterances302.

In some implementations, the data generation stage200bis configured to select the unspoken training text utterances302bfrom a corpus of unspoken text402. The corpus of unspoken text402includes a multitude of unspoken training text utterances302bfrom across a large range of domains, and includes a far greater linguistic diversity than the specific domain in which the speech conversion model300is being trained to learn. As mentioned previously, the set of spoken training utterances305may be domain-specific in that they pertain to the specific domain. The corpus of unspoken text402may be stored in the same or different data store202as the spoken training utterances305. The corpus of unspoken text402may dynamically change to incorporate new unspoken training text utterances302b. Simply using all unspoken training text utterances302bin the unspoken text corpus402is not feasible for the following reasons: i) for each sentence, the speech modality needs much more memory to be encoded than text, thereby making converting all text in the corpus402impractical; ii) conversion between speech and text modalities in TTS inference and speech conversion model training also requires large computation; and iii) the vast amount of difference between the transcriptions302ain the spoken training utterances305and the unspoken training text utterances302bin the unspoken text corpus402requires intelligent strategies to balance their contributions.

The data generation stage200baims to select a subset of the available unspoken training text utterances302bfrom the unspoken text corpus402as the data for TTS synthesis described in greater detail below with reference toFIGS. 2D and 2E. Stated differently, the data generation stage200baims to improve the match between the selected subset of the available unspoken training text utterances302band the specific domain being targeted (e.g., Psychology terminology), which in turn reduces the computational resources required to exploit a large amount of non-domain-specific data. Accordingly, the data generation stage200breduces computational and memory costs by selecting unspoken training text utterances302bthat best match the specific domain the speech conversion model300is being trained to learn.

In some examples, the data generation stage200bselects the subset of the available unspoken training text utterances302bfrom the corpus402that best match the specific domain by simply providing a domain identifier (not shown) associated with the specific domain as an input to a background language model (LM)406previously trained on the entire unspoken text corpus402. As mentioned previously, the unspoken text corpus402spans a multitude of different domains. In these examples, the background LM406may include a maximum entropy (MaxEnt LM) capable of optionally accepting the domain identifier as input as described in U.S. Pat. No. 9,842,592, filed on Feb. 12, 2014, the contents of which is incorporated herein by reference in its entirety. Here, the domain identifier associated with the specific domain may allow the MaxEnt LM to output a subset of the available unspoken training text utterances302bfrom the corpus402that are likely to include words and/or phrases pertaining to the specific domain. In some configurations, rather than evaluating likelihood of words, a statistical language model operates in reverse mode to randomly generate a text phrase that matches a statistical distribution of words pertaining to the specific domain.

In additional examples, and as depicted inFIG. 2A, the data generation stage200bexecutes an unspoken text selection process that uses the transcriptions302ain the set of spoken training utterances305obtained from the target speaker104to select the subset of the available unspoken training text utterances302bfrom the corpus402that best match the specific domain. Here, the spoken training utterances305spoken by the target speaker104include words, phrases, and/or other terminology pertaining to the specific domain. Optionally, in addition to, or in lieu of the transcriptions302ain the set of spoken training utterances305, a set of different transcribed utterances that pertain to the specific domain can be used for selecting the unspoken training text utterances302b. This would provide the advantage of not requiring all the spoken training utterances305to belong to the specific domain.

During a first stage (STAGE A) of the unspoken text selection process, the data generation stage200bbuilds two language models404,406to enable contrastive selection of the unspoken training text utterances302bHere, a domain-specific language model (LM)410is trained on each transcription302ain the set of spoken training utterances305. The set of spoken training utterances305are assumed to belong to the specific-domain for which the speech conversion model300is being trained. On the other hand, the background LM406is trained on each unspoken training text utterance302bin the entire unspoken text corpus402. As mentioned previously, the unspoken text corpus402spans a multitude of different domains. In some examples, the first stage uses n-gram language model training to build the two language models404,406. In other examples, the first stage uses neural network language model training to build the two language models404,406.

During a second state (STAGE B) of the unspoken text selection process, the data generation stage200buses the two contrastive LMs404,406to evaluate each unspoken training text utterance302bin the unspoken text corpus402by determining a first probability. P(w|), associated with each word in the unspoken training text utterance302bappearing in the domain-specific LM404and determining a second probability, P(w|), associated with each word in the unspoken training text utterance302bappearing in in the background LM406. Thereafter, for each unspoken training text utterance302bin the unspoken text corpus402, the process200determines, at a scorer408, a score, S, based on the first probability, the second probability, and a number of words, #(w), appearing in the corresponding unspoken training text utterance302b. For example, the score S for each unspoken training text utterance302bmay be calculated as follows.

After determining the scores, the data generation process200bselects the unspoken training text utterances302bwith the N-best scores S as these unspoken training text utterances302bbest match the specific domain. The text corpus402may include billions of text utterances302b. In lieu of, or in addition to, selecting from the available text corpus402, the unspoken training text utterances302bmay include manually-written text not generated form a LM, to target certain phrases/improper nouns (e.g., family member names, contact names, games, etc.), and/or the unspoken training text utterances302bmay be derived from a particular topic of interest using a topic classifier associated with the specific domain. The unspoken training text utterances302bgenerated during the data generation stage200bcan include millions of utterances, and thus, far exceed the number of spoken training utterances305collected from the speaker. As will become apparent, the content of the unspoken training text utterances302bincreases linguistic diversity for the specific domain the speech conversion model300is being trained to learn, while corresponding synthetic speech representations generated from the unspoken training text utterances302bincreases acoustic diversity for the atypical speech the speech conversion model300is converting.

Referring toFIG. 2C, the adaption stage200cof the training process200includes using the set of spoken training utterances305collected during the personalized seed data collection stage200aofFIG. 2Ato adapt both a text-to-speech (ITS) model210and a reference S2S conversion model301to synthesize speech in the voice of the target speaker104and that captures the atypical speech (e g. ALS speech) associated with the target speaker104. The adaption stage200cmay occur before, after, or concurrently with the data generation stage200bofFIG. 2B.

The TTS model210may be pre-trained on input text to generate synthesized canonical fluent speech in the voices of one or more predefined speakers. As such, ground-truth speech samples used to train the TTS model210may be obtained from speakers with typical speech.

Similarly, the reference S2S conversion model301is pre-trained on input audio data corresponding to a multitude of utterances spoken by various different speakers into corresponding output audio data that captures the same content in the voice of a single predefined speaker. Notably, the utterances from the various different speakers may include typical speech patterns, a variety of different types of atypical speech patterns (e.g., heavy accents spanning different dialects, irregular speech spanning different neurological conditions), as well as background noise. For example, the reference S2S conversion model301can include the end-to-end-trained speech-to-speech conversion model described in,Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation, available at https://arxiv.org/pdf/1904.04169.pdf, and incorporated herein by reference. The reference S2S conversion model301can use a sequence-to-sequence to normalize arbitrary speech, potentially including background noise, and generate the same content in the voice of the single predefined target speaker. The source speech can be from any speaker or accent, and may contain complex prosodic patterns, imperfections, atypical speech, and background noise, all of which are removed through the normalization process as the first audio data is converted into clean second audio data with a fixed accent and consistent articulation and prosody. In other words, the system may be used to project away all non-linguistic information, including speaker characteristics, and to retain only what is been said, not who, how, or where it is said.

Since the TTS model210is pre-trained to generate synthesized canonical fluent speech in the voice other than the target speaker104and the reference S2S conversion model301is pre-trained on utterances from a variety of different speakers associated with both typical speech and various types of atypical speech, the adaption stage200caims to adapt the models210,301to both the voice of the targets speaker104and the particular type of atypical speech (e g. ALS speech) associated with the target speaker104. In some examples, however, when the reference S2S conversion model is pre-trained to convert input audio data associated with the particular type of atypical speech associated with the target speaker104, the adaption stage200cforegoes adapting the reference S2S conversion model301since the model301is already trained to convert the same type of atypical speech associated with the target speaker104into canonical fluent speech.

The adaption stage200cadapts the TTS model210to convert the transcriptions302ain the set of spoken training utterances305into corresponding synthetic speech306in the voice of the target speaker104and that captures the atypical speech associated with the target speaker104. In some implementations, the TTS model210includes an encoder312and a decoder314that cooperate to process the transcriptions302ato adapt the TTS model210to generate time-domain audio waveforms of synthesized speech306in the voice of the target speaker104and that captures the atypical speech associated with the target speaker104. A time-domain audio waveform is an audio waveform that defines an audio signal's amplitude over time.

The encoder212may be an encoder neural network212configured to receive the transcription304as a sequence of characters and generate a fixed-length context vector213for each mel-frequency spectrogram that the decoder214will later generate. Since TTS model210is being adapted to produce synthesized speech capturing the atypical speech in the voice of the target speaker104, the adaption stage200cmay include tuning/re-training parameters of the decoder214while parameters of the encoder212remain fixed. The decoder214may a neural network configured to receive, as input, the fixed-length context vectors213generated by the encoder neural network212and generate, as output for each fixed-length context vector213, a corresponding frame of a mel-frequency spectrogram. A mel-frequency spectrogram is a frequency-domain representation of sound. Mel-frequency spectrograms emphasize lower frequencies, which are critical to speech intelligibility, while de-emphasizing high frequency, which are dominated by fricatives and other noise bursts and generally do not need to be modeled with high fidelity. The synthesized speech306may include a synthesized speech representation associated with the mel-frequency spectrograms output from the decoder214, or the synthesized speech306may be a time-domain audio waveform generated by a vocoder (not shown) based on the mel-frequency spectrograms output from the decoder214. The decoder214may include a post-net that may be adapted to the target speaker104by refining acoustic characteristics of the mel-frequency spectrograms generated by the decoder to better match the voice and atypical speech associated with the target speaker104.

The adaption stage200cadapts the reference S2S conversion model301to convert the non-synthetic speech representations304from the set spoken training utterances305into synthesized canonical fluent speech in the voice of the target speaker104. As mentioned previously, the non-synthetic speech representations304are associated with the utterances spoken by the target speaker104, and as such, capture the atypical speech associated with the target speaker104. Here, the adaptation stage200cmay use the corresponding transcriptions302aas a ground truth for the spectrogram decoder320to accurately decode/emit the synthesized canonical fluent speech307that conveys the intended content of the input non-synthetic speech representation304.

Referring toFIG. 2D, for each unspoken training text utterance302bof the plurality of unspoken training text utterances obtained during the data generation stage200bofFIG. 2B, the validation and filtering stage200dgenerates, as output from adapted TTS model210, a synthetic speech representation306of the corresponding unspoken training text utterance302b. Since the spoken training utterances305were used to adapt the TTS model210, each synthetic speech representation306includes the voice of the target speaker and captures the atypical speech associated with the target speaker104. As such, each synthetic speech representation306output from the adapted TTS model210is paired with a corresponding one of the plurality of unspoken training text utterances302b.

In the example shown, the validation and filtering stage200dfurther uses the adapted S2S conversion model301to generate, for each synthetic speech representation306output from the adapted TTS model210, a corresponding audio waveform of synthesized canonical fluent speech316in the voice of the target speaker104, and thereafter uses a text decoder250to generate a textual representation318for the corresponding audio waveform of synthesized canonical fluent speech316generated as output from the adapted S2S conversion model301. As previously mentioned, if the reference S2S speech conversion model301is previously trained for converting the same type of atypical speech, the reference S2S speech conversion model301need not (but still could) be adapted prior to generating audio waveforms of synthesized canonical fluent speech from the synthetic speech representations306. In some examples, the text decoder250resides on the adapted S2S speech conversion model301in which the S2S speech conversion model301emits the textual representation318from the text decoder250in parallel with emitting the corresponding audio waveform of synthesized canonical fluent speech316from the spectrogram decoder320. In other examples, the text decoder250resides on a reference automated speech recognition model separate from the speech conversion model.

Thereafter, for each synthetic speech representation306output from the adapted TTS model210, the validation and filtering stage200dapplies a supervised loss term module340to determine a word error rate loss342associated with the corresponding synthetic speech representation306. Specifically, the word error rate loss342is based on the textual representation318output from text decoder250for the synthesized canonical fluent speech306and the corresponding unspoken training text utterance302b. Notably, the unspoken training text utterance302bserves as both an input to the adapted TTS model210for conversion into the resulting synthetic speech representation306, and as a ground-truth for verifying the corresponding textual representation318output from the decoder250. In the example shown, the validation and filtering stage200dvalidates each synthetic speech representation306output from the adapted TTS model210by determining whether or not the corresponding word error rate loss342satisfies a word error rate loss threshold. When the corresponding word error rate loss342satisfies the word error rate loss threshold, the corresponding synthetic speech representation306is stored in a filtered set of synthetic speech representations306A for use in training the speech conversion model300. When the corresponding word error rate loss342fails to satisfy the word error rate loss threshold, the corresponding synthetic speech representation306is discarded an not used to train the speech conversion model300.

In the example shown, the supervised loss term module340determines the word error rate loss342based on the number of misrecognized words between the corresponding textual representation318and the corresponding unspoken training text utterance302bserving as ground-truth. For example, a word error rate loss of 60% indicates that 40% of the words in the corresponding textual representation318were misrecognized by the text decoder250from the corresponding synthesized canonical fluent speech316. The word error rate loss threshold can be set to any value, and may be adjusted as needed. In one example, the word error rate loss threshold is 70% indicating that at least 70% of the words in the corresponding textual representation318must be accurately recognized in order for the corresponding synthetic speech representation306to satisfy the word error rate loss threshold, and thus, be accepted in the filtered set of synthetic speech representations306A. The word error rate loss342and value set for the word error rate loss threshold serve as a proxy for identifying only the synthetic speech representations306(i.e., the filtered set of synthetic speech representations306A) that are suitable for training the speech conversion model300, and discarding any synthetic speech representations306that are not suitable for training. The discarded synthetic speech representations306are indicative of input audio waveforms (or spectrograms) that the adapted S2S conversion model301is unable to accurately convert into intelligible synthesized canonical fluent speech316due to the text decoder250producing corresponding textual representations318with word error rate losses342that fail to satisfy the word error rate loss threshold. Simply put, the discarded synthetic speech representations306are associated with a small percentage of the total number of synthetic speech representations306generated by the adapted TTS model210that are indicative of being unintelligible or far from ideal in terms of quality of content and style.

While it is understood that the steps of using the adapted S2S conversion model301to produce synthesized canonical fluent speech316from each atypical synthetic speech representation306and applying speech recognition on the synthesized canonical fluent speech316by the text encoder250to produce the textual representation318help validate the synthetic speech representations306for training the speech conversion model300, these steps may be optional. For instance, the adapted TTS model210may be used to generate a corresponding synthetic speech representation306for each unpaired unspoken training text utterance302b, and all of the synthetic speech representations306may be used to train the speech conversion model300without applying the adapted S2S conversion model301, the text decoder250, and the supervised loss term module340to identify the filtered set of synthetic speech representations306A.

Referring now toFIG. 2E, the final training stage200eincludes training the speech conversion model300based on the synthetic speech representation306generated by the adapted TTS model210for each unspoken training text utterance302bof the plurality of unspoken training text utterances302b. More specifically, the final training stage200eof the training process200trains the speech conversion model300on the filtered set of synthetic speech representations306A and not training the speech conversion model300on any of the synthetic speech representations306that have been discarded for having word error rate losses342not satisfying the word error rate loss threshold as discussed above in the validation and filtering stage200dofFIG. 2D.

In some implementations, training the speech conversion model300includes training a S2S speech conversion model300apersonalized for the target speaker104to convert input audio waveforms associated with the atypical speech of the speaker directly into corresponding output waveforms of canonical fluent speech in the voice of the speaker. The number of synthetic speech representations306A in the filtered set of synthetic speech representations306A provides sufficient acoustic diversity for training the S2S speech conversion model300ato learn both the voice of the target speaker104and the type of atypical speech associated with the target speaker104. Moreover, since each synthetic speech representation306used to train the S2S conversion model300apertains to the specific domain, the number of synthetic speech representations306A in the filtered set of synthetic speech representations306A also provides sufficient linguistic diversity for training the S2S speech conversion model300ato learn specific words, names, phrases, or other terminology associated with the specific domain that are like to be present in atypical speech utterances108spoken by the target speaker104.

In some examples, training the S2S conversion model300aincludes adapting the reference S2S conversion model301(FIG. 2C) on the filtered set of synthetic speech representations306A each paired with a corresponding one of the unspoken training text utterances302b. As mentioned previously, the reference S2S conversion model301was previously trained using utterances spoken from a variety of different speakers with different speaking styles and different voices to produce canonical fluent speech in the voice of a predefined speaker. As such, adapting the reference S2S conversion model301on the filtered set of synthetic speech representations306A provides the trained S2S conversion model300athat is personalized for the target speaker104to convert any input audio waveform (e.g., input audio data102ofFIG. 1A) corresponding to an utterance spoken by the target speaker104associated with the particular type of atypical speech into an output audio waveform (e.g., output audio data106ofFIG. 1A) corresponding to a synthesized canonical fluent speech representation316of the same utterance spoken by the target speaker. In these examples, the trained S2S speech conversion model300amay be further adapted/tuned on the non-synthetic speech representations304from the set of spoken training utterances305collected from the target speaker104during the personalized seed data collection stage200aofFIG. 2A.

In other examples, training the S2S conversion model300aincludes training the S2S conversion model300afrom scratch. Training the S2S conversion model300afrom scratch may include training on a mixture of the filtered set of synthetic speech representations306A, each paired with a corresponding one of the unspoken training text utterances302b, and the non-synthetic speech representations304in the set of spoken training utterances305, each paired with a corresponding transcription302a. Here, the model300may be trained on corresponding batches of non-synthetic and synthetic speech representations304,306in parallel such that the model300aoutputs corresponding synthesized canonical fluent speech representations316in parallel.

When training the S2S conversion model300a, whether by adapting the reference S2S conversion model301or from scratch, the final training stage200eof the training process200may use a stochastic optimization algorithm, such as stochastic gradient decent, to train the model300athrough backpropagation. For example, an automated speech recognizer (e.g., text decoder250ofFIG. 2D) may produce a textual representation318for each corresponding synthesized canonical fluent speech representation316output from the S2S conversion model300athat may be compared with the corresponding training text utterance302a,302bserving as ground truth to obtain a word error rate loss342(FIG. 2D). While the automated speech recognizer may be a separately trained automated speech recognizer, the automated speech recognizer may also include a text decoder of the S2S conversion model300athat emits the textual representations318in parallel with the corresponding synthetic canonical fluent speech representations318emitted by the spectrogram decoder320. Regardless from where the resulting textual representations318are produced, the stochastic optimization algorithm may use the word error rate losses342to define respective loss functions (e.g., cross-entropy loss functions) based on a difference between actual outputs (e.g., the textual representations318) and target outputs (the transcriptions and unspoken training text utterances302a,302b). For example, the loss function may be computed for each batch of training examples, and then differentiated with respect to each weight in the model300a.

In some additional implementations, training the speech conversion model300additionally or alternatively includes training a speech-to-text conversion model300b(interchangeably referred to as ‘ASR model’) personalized for the target speaker104to convert input audio waveforms associated with the atypical speech of the speaker into corresponding text that includes canonical transcriptions120of utterances spoken by the target speaker104. The number of synthetic speech representations306A in the filtered set of synthetic speech representations306A provides sufficient acoustic diversity for training the ASR model300bto learn recognize utterances108spoken with the type of atypical speech associated with the target speaker104. Moreover, since each synthetic speech representation306used to train the ASR model300bpertains to the specific domain, the number of synthetic speech representations306A in the filtered set of synthetic speech representations306A also provides sufficient linguistic diversity for training the ASR model300bto learn to recognize particular words, names, phrases, or other terminology associated with the specific domain that are like to be present in atypical speech utterances108spoken by the target speaker104.

As with the S2S speech conversion model300a, training the ASR model300bmay include adapting a reference ASR model300bthat was previously trained on a general corpus of training utterances spoken by a variety of different speakers with different speaking styles. Here, the reference ASR model300bmay be adapted on the filtered set of synthetic speech representations306A each paired with a corresponding one of the unspoken training text utterances302b, and then further adapted/tuned on the non-synthetic speech representations304from the set of spoken training utterances305collected from the target speaker104during the personalized seed data collection stage200aofFIG. 2A. On the other hand, the ASR model300bmay be trained from scratch using a mixture of the filtered set of synthetic speech representations306A, each paired with a corresponding one of the unspoken training text utterances302b, and the non-synthetic speech representations304in the set of spoken training utterances305, each paired with a corresponding transcription302a.

In other examples, training the S2S conversion model300aincludes training the S2S conversion model300afrom scratch. Training the S2S conversion model300afrom scratch may include training on a mixture of the filtered set of synthetic speech representations306A, each paired with a corresponding one of the unspoken training text utterances302b, and the non-synthetic speech representations304in the set of spoken training utterances305, each paired with a corresponding transcription302aHere, the model300may be trained on corresponding batches of non-synthetic and synthetic speech representations304,306in parallel such that the model300aoutputs corresponding synthesized canonical fluent speech representations316in parallel.

The final training stage200eof the training process200may use a stochastic optimization algorithm, such as stochastic gradient decent, to train the ASR model300bthrough backpropagation. Details of using the stochastic optimization algorithm for training the ASR model300bare discussed above with respect to training the S2S conversion model300a.

FIG. 3provides a flowchart of an example arrangement of operations for a method380of training a speech conversion model personalized for a target speaker associated with atypical speech. The flowchart may be described with respect to the training process200ofFIGS. 1A, 1B, andFIGS. 2A-2E. Data processing hardware410(FIG. 4) may perform the operations for the method380by executing instructions stored on memory hardware420(FIG. 4) in communication with the data processing hardware410. The data processing hardware410and the memory hardware420may reside on a computing device400(FIG. 400), such as a remote server112and/or the user computing device110ofFIGS. 1A and 1B. At operation382, the method380obtains a plurality of training text utterances302a,302bHere, a first portion of the plurality of training text utterances includes a plurality of transcriptions302ain a set of spoken training utterances305,305a-n. Each spoken training utterance305is spoken by a target speaker104associated with atypical speech and includes a corresponding transcription302apaired with a corresponding non-synthetic speech representation304of the corresponding spoken training utterance305. The set of spoken training utterances305may be obtained during the personalized seed data collection stage200aofFIG. 2A. A second portion of the plurality of training text utterances includes a plurality of unspoken training text utterances302bpertaining to a specific domain in which the speech conversion model300is trained to learn. Each unspoken training text utterance302bis not paired with any corresponding spoken utterance. The plurality of unspoken training text utterances302bmay be obtained during the data generation stage200bofFIG. 2B.

At operation384, the method380includes adapting, using the set of spoken training utterances305, a text-to-speech (TTS) model210to synthesize speech in a voice of the target speaker and that captures the atypical speech associated with the target speaker. Details of adapting the TTS model210are described with reference to the adaption stage200cofFIG. 2C.

At operation386, for each unspoken training text utterance302bof the plurality unspoken training text utterances, the method380also includes generating, as output from the adapted TTS model210, a synthetic speech representation306of the corresponding unspoken training text utterance302b. Here, each synthetic speech representation306output from the adapted TTS model210is in the voice of the target speaker104and captures the atypical speech associated with the target speaker104. At operation388, the method380also includes training the speech conversion model300based on the synthetic speech representation306generated by the adapted TTS model210for each unspoken training utterance302bof the plurality of unspoken training text utterances. Training the speech conversion model300includes training at least one of a S2S conversion model300aor a speech-to-text (e g, ASR) model300b.

The computing device400includes a processor410, memory420, a storage device430, a high-speed interface/controller440connecting to the memory420and high-speed expansion ports450, and a low speed interface/controller460connecting to a low speed bus470and a storage device430. Each of the components410,420,430,440,450, and460, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor410can process instructions for execution within the computing device400, including instructions stored in the memory420or on the storage device430to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display480coupled to high speed interface440. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices400may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The storage device430is capable of providing mass storage for the computing device400. In some implementations, the storage device430is a computer-readable medium. In various different implementations, the storage device430may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory420, the storage device430, or memory on processor410.

The high speed controller440manages bandwidth-intensive operations for the computing device400, while the low speed controller460manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller440is coupled to the memory420, the display480(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports450, which may accept various expansion cards (not shown). In some implementations, the low-speed controller460is coupled to the storage device430and a low-speed expansion port490. The low-speed expansion port490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device400may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server400aor multiple times in a group of such servers400a, as a laptop computer400b, or as part of a rack server system400c.