Patent ID: 12260851

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Text-to-speech (TTS) systems, often used by speech synthesis systems, are generally only given text inputs without any reference acoustic representation at runtime, and must impute many linguistic factors that are not provided by the text inputs in order to produce realistically sounding synthesized speech. A subset of these linguistic factors are collectively referred to as prosody and may include intonation (pitch variation), stress (stressed syllables vs. non-stressed syllables), duration of sounds, loudness, tone, rhythm, and style of speech. Prosody may indicate the emotional state of the speech, the form of the speech (e.g., statement, question, command, etc.), the presence of irony or sarcasm of the speech, uncertainty in the knowledge of the speech, or other linguistic elements incapable of being encoded by grammar or vocabulary choice of the input text. The linguistic factors may also convey an accent/dialect associated with how speakers in a particular geographical region pronounce words/terms in a given language. For example, English speakers from Boston, Mass. have a “Boston accent,” and pronounce words/terms differently than how English speakers from Fargo, N. Dak. pronounce the same terms. Accordingly, a given text input can produce synthesized speech in a given language across various different accents/dialects and/or different speaking styles, as well as produce synthesized speech across different languages.

In some instances, TTS systems are trained using human speech spoken by one or more target speakers. For example, each target speaker may be a professional voice actor that speaks with a particular style and in a particular accent/dialect (e.g., American English accent) native to the target speaker. Using a corpus of training utterances spoken by a target speaker (e.g., professional advertisement reader), TTS systems can learn to generate synthesized speech that matches the voice, speaking style, and accent/dialect associated with the target speaker. In some situations, it may be useful for the TTS system to generate synthesized speech that clones the voice of the target speaker but in a different speaking style and/or accent/dialect than the speaking style and/or accent/dialect native to the target speaker. Returning to the example where the target speaker includes a professional voice actor who speaks with an American English accent, it may be desirable for the TTS system to generate synthesized speech that includes the voice of the voice actor (e.g., target speaker) but in a British English accent. Here, the TTS system will not be able generate the synthesized speech that clones the voice of the target speaker in the British English accent unless the TTS system was trained on reference utterances spoken by the target speaker in the British English accent. Moreover, the professional voice actor who natively speaks in the American English accent may not be able to produce speech that accurately pronounces/enunciates terms associated with the British English accent, leaving no ability to even train the TTS system on reference utterances spoken by the voice actor in the British English accent. This inability to attain sufficient training data is further compounded in situations when it is desirable for the TTS system to generate synthesized speech that clones the voice of a target speaker across multiple different accents/dialects, none of which, are native to the voice actor.

Implementations herein are directed toward leveraging a trained voice cloning system to generate training synthesized speech representations that clone a voice of a target speaker in a target accent/dialect that the target speaker does not natively speak and using the training synthesized speech representations to train a TTS system to learn to produce synthesized expressive speech that clones the voice of the targets speaker in the target accent/dialect. More specifically, the trained voice cloning system obtains training data including a plurality of training audio signals and corresponding transcripts, whereby each training audio signal corresponds to a reference utterance spoken by the target speaker in a first accent/dialect native to the target speaker. For each training audio signal, the trained voice cloning system generates a training synthesized speech representation of the of the corresponding reference utterance spoken by the target speaker. Here, the training synthesized speech representation includes the voice of the target speaker in a second accent/dialect different than the first accent/dialect. That is, the training synthesized speech representation is associated with a different accent/dialect than the first accent/dialect associated with the reference utterance spoken by the target speaker.

An untrained TTS system trains on the transcript of the training audio signal and the training synthesized speech representation to learn how to generate synthesized speech that clones the voice of the target speaker in the second accent/dialect. That is, in the untrained state, the TTS system is unable to transfer the voice of the target speaker across different accents/dialects in synthesized speech generated from input text. However, after leveraging the voice cloning system to produce training synthesized speech representations that clone the voice of the target speaker in a different accent/dialect and using the training synthesized speech representations to train the TTS system, the trained TTS system can be employed during inference to convert an input text utterance into corresponding synthesized expressive speech cloning the voice of the target speaker in the second accent/dialect. Here, during inference, the trained TTS system may receive conditioning inputs that include a speaker embedding representing voice characteristics of the target speaker and an accent/dialect identifier identifying the second accent/dialect so that the TTS system can convert the input text utterance into an output audio waveform that clones the voice of the target speaker in the second accent/dialect.

FIG.1shows an example system100for training an untrained text-to-speech system (TTS)300and executing the trained TTS system300to synthesize an input text utterance320into expressive speech152that includes a voice of a target target speaker in a target accent/dialect. While examples herein are directed toward generating synthesized speech152in a particular voice for different accents/dialects, implementations herein can be similarly applied for generating synthesized speech152in the particular voice for different speaking styles in addition to, or in lieu of, different accents/dialects. The system100includes a computing system (interchangeable referred to as ‘computing device’)120having data processing hardware122and memory hardware124in communication with the data processing hardware122and storing instructions executable by the data processing hardware122to cause the data processing hardware122to perform operations.

In some implementations, the computing system120(e.g., data processing hardware122) provides a trained voice cloning system200configured to generate training synthesized speech representations202for use in training the untrained TTS system300. The trained voice cloning system200obtains training data10that includes a plurality of training audio signals102and corresponding transcripts106. Each training audio signal102includes an utterance of human speech spoken by a target speaker in a first accent/dialect. For example, the training audio signals102may be spoken by a target speaker in an American English accent. Thus, the first accent/dialect associated with the utterances of human speech spoken by the target speaker may correspond to the native accent/dialect of the target speaker. Each transcript106includes a textual representation for each corresponding reference utterance. The training data10may also include a plurality of speaker embeddings (also referred to as “speaker identifiers”)108that each represent speaker characteristics (e.g., native accent, a speaker identifier, male/female, etc.) of the corresponding target speaker. That is, the speaker embedding/identifier108may represent speaker characteristics of the target speaker. The speaker embedding/identifier108may include a numerical vector representing the speaker characteristics of the target speaker or simply include an identifier associated with the target speaker that instructs the trained voice cloning system200to produce the training synthesized speech representation202in the voice of the target speaker. In the case of the latter, the speaker identifier may be translated to the corresponding speaker embedding for use by the system200. In some examples, the trained voice cloning system200includes a voice conversion system that converts each training audio signal102(e.g., reference utterance of human speech) directly into a corresponding training synthesized speech representation202. In other examples, the trained voice cloning system200includes a text-to-speech voice cloning system that converts the corresponding transcript106into a corresponding training synthesized speech representation106that clones the voice of the reference utterance in a second accent/dialect different than the first accent/dialect associated with the training audio signal102.

For simplicity, examples herein are directed toward the trained voice cloning system200generating training synthesized speech representations202that clone the voice of a target speaker in a target accent/dialect (e.g., second accent/dialect). However, implementations herein are equally applicable to the trained voice cloning system200generating training synthesized speech representations202that clone the voice of the target speaker and having any target speech characteristic. Thus, the target speech characteristic may include at least one of a target accent/dialect, a target prosody/style, or some other speech characteristic. As will become apparent, the training synthesized speech representations202generated by the trained voice cloning system that have the target speech characteristic are used to train the untrained TTS system300to learn how to produce synthesized speech202having the target speech characteristic.

For each training audio signal102of the training data10, the trained voice cloning system200generates a training synthesized speech representation202of the corresponding reference utterance spoken by the target speaker. Here, the training synthesized speech representation202includes the voice of the target speaker in a second accent/dialect different than the first accent/dialect of the training audio signals102. That is, the trained voice cloning system200takes the training audio signal102corresponding to the reference utterance spoken by the target speaker in the first accent/dialect as input, and generates the training synthesized speech representation202of the training audio signal102in the second accent/dialect as output Thus, the trained voice cloning system200generates a corresponding training synthesized speech representation202for each of the plurality of training audio signals102of the training data10to create a plurality of training synthesized speech representations202for use in training the untrained TTS system300. In some examples, the trained voice cloning system200determines the speaker characteristics of the training synthesized speech representation202from the speaker embedding/identifier108.

In some implementations, when the trained voice cloning system200includes the TTS voice cloning system200, the training data10includes a plurality of training text utterances106and the TTS voice cloning system200converts each training text utterance106into the training synthesized speech representation202in a target speech characteristic. The target speech characteristic may include the second accent/dialect. Alternatively, the target speech characteristic may include a target prosody/style. That is, the TTS voice cloning system200may produce training synthesized speech representations202from text alone. Accordingly, the training text utterances106may correspond to unspoken text utterances that are not paired from any corresponding audio signal of human speech. As such, the unspoken text utterances could be derived manually or from a language model. The TTS voice cloning system200may also receive a speaker embedding/identifier108that conditions the TTS voice cloning system200to clone the voice of the target speaker to produce training synthesized speech representations202in the voice of the target speaker and having the target speech characteristic. The TTS voice cloning system200may also receive a target speech characteristic identifier that identifies a target speech characteristic. For instance, the target speech characteristic identifier may include an accent/dialect identifier109identifying a target accent/dialect (e.g., second accent/dialect) of the resulting training synthesized speech representations202and/or include a prosody/style identifier (i.e., utterance embedding204) that indicates a target prosody/style of the resulting training synthesized speech representation202.

For each training audio signal102of the training data10, the untrained TTS system300trains based on the corresponding transcript106of the training audio signal102and the corresponding training synthesized speech representation202output from the trained voice cloning system200that includes the voice of the targets speaker in the second dialect/language. More specifically, training the untrained TTS system300may include, for each training audio signal102of the training data10, training both a TTS model400and a synthesizer150of the untrained TTS system300to learn how to generate synthesized speech from input text such that the synthesized speech clones the voice of the target speaker in the second dialect/accent. That is, the TTS system300, including the TTS model400and the synthesizer150, is trained to produce synthesized speech152that matches each training synthesized speech representation202. During training, the TTS system300may learn to predict utterance embeddings204for the training synthesized speech representation202. Here, each utterance embedding204may represent prosody information and/or accent/dialect information associated with the training synthesized speech representation202the TTS system300aims to replicate. Moreover, multiple TTS systems300,300A-N may train on training synthesized speech representation202output form the trained voice cloning system200. Here, each TTS system300trains on a corresponding set of training synthesized speech representations202that may include voices of different target speaker, different speaking styles/prosodies, and/or different accent/dialect. Thereafter, each of the multiple trained TTS systems300are configured to generate expressive speech152for a respective target voice in a corresponding accent/dialect. The computing device120may store each trained TTS system300on data storage180(e.g., memory hardware124) for later use during inference.

During inference, the computing device120may use the trained TTS system300to synthesize an input text utterance320into expressive speech152that clones the voice of the target speaker in a target accent/dialect (or conveying some other target speech characteristic in addition to or in lieu of the target accent/dialect). In particular, a TTS model400of the trained TTS system300may obtain conditioning inputs including a speaker embedding/identifier108that represents voice characteristics of the target speaker and an accent/dialect identifier109that identifies the intended accent/dialect (e.g., British English or American English). Conditioning inputs could further include a speaking prosody/style identifier representing a particular speaking style vertical that the resulting synthesized speech152should include. The TTS model400, conditioned on the speaker embedding/identifier108and accent/dialect identifier109, processes the input text utterance320to generate an output audio waveform402. Here, the speaker embedding/identifier108includes speaker characteristics of the target speaker and the accent/dialect identifier109includes the target accent/dialect (e.g., American English, British English, etc.). The output audio waveform402conveys the target accent/dialect and the voice characteristics of the target speaker to enable the speech synthesizer150to generate the synthetic speech152from the output audio waveform402. The TTS model400may also generate a number of predicted frames280corresponding to the output audio waveform402.

FIG.2Ashows an example of the trained voice cloning system200,200aof the system100. The trained voice cloning system200areceives a training audio signal102corresponding to a reference utterance spoken by the targets speaker in a first accent/dialect and a corresponding transcription106of the reference utterance, and generates a training synthesized speech representation202that clones the voice of the target speaker in a second accent/dialect different than the first accent/dialect. The trained voice cloning system200aincludes an inference network210, a synthesizer220, and an adversarial loss module230. The inference network210includes a residual encoder212that is configured to consume the input training audio signal102corresponding to the reference utterance spoken by the target speaker in the first accent/dialect and outputs a residual encoding214of the training audio signal102. The training audio signal102may include mel spectrogram representations. In some examples, a feature representation (i.e., mel spectrogram sequence) is extracted from the training audio signal102and provided as input to the residual encoder212to generate the corresponding residual encoding214therefrom.

The synthesizer220includes a text encoder222, speaker embeddings/identifier108, language embeddings224, a decoder neural network500, and a waveform synthesizer228. The text encoder222may include an encoder neural network having a convolutional subnetwork and a bidirectional long short-term memory (LSTM) layer. The decoder neural network500is configured to receive, as input, outputs225from the from the text encoder222, the speaker embedding/identifier108, and the language embedding224to generate an output mel spectrogram502. The speaker embedding/identifier108may represent voice characteristics of the targets speaker and the language embedding224may specify language information associated with at least one of a language of the training audio signal, a language of the training synthesized speech utterance204to be produced, accent/dialect identifiers109identifying the accent/dialects associated with the training audio signal102and the training synthesized speech representation. Finally, the waveform synthesizer228may convert the mel spectrograms502output from the decoder neural network500into a time-domain waveform (e.g., training synthesized speech representation202). The training synthesized speech representation202includes the voice of the target speaker in the second accent/dialect different than the first accent/dialect spoken in the reference utterance of the training data by the same target speaker. Accordingly, the voice cloning system200aoutputs training synthesized speech representations202that retain the voice of the target speaker that spoke the reference utterance in the first accent/dialect and convert the first accent/dialect spoken in the reference utterance into the second/accent dialect. Each training synthesized speech representation202generated by the voice cloning system200amay also be associated with the language embedding224, the accent/dialect identifier109, and/or the speaker embedding/identifier108for use as conditioning inputs when training the TTS system300on the training synthesized speech representation202. In some implementations, the waveform synthesizer228is a Griffin-Lim synthesizer. In some other implementations, the waveform synthesizer228is a vocoder. For instance, the waveform synthesizer228may include a WaveRNN vocoder. Here, the WaveRNN vocoder may generate 16-bit signals sampled at 24 kHz conditioned on spectrograms predicted by the trained voice cloning system200. In some other implementations, the waveform synthesizer228is a trainable spectrogram to waveform inverter. After the waveform synthesizer125generates the waveform, an audio output system can generate the training synthesized speech representations202using the waveform. In some examples, a WaveNet neural vocoder replaces the waveform synthesizer228. A WaveNet neural vocoder may provide different audio fidelity of the training synthesized speech representation202in comparison to the training synthesized speech representation202produced by the waveform synthesizer228.

The text encoder222is configured to encode the corresponding transcription106of the training audio signal102into a sequence of text encodings225,225a-n. In some implementations, the text encoder includes an attention network that is configured to receive a sequential feature representation of the transcription106to generate a corresponding text encoding as a fixed-length context vector for each output step of the decoder neural network500. That is, the attention network at the text encoder222may generate a fixed-length context vector225,225a-nfor each frame of a mel spectrogram502that the decoder neural network500will later generate. A frame is a unit of the mel spectrogram502that is based on a small portion of the input signal, e.g., a 10 millisecond sample of the input signal. The attention network may determine a weight for each element of the text encoder222output and generate the fixed-length vector225by determining a weighted sum of each element. The attention weights may change for each decoder neural network500time step.

Accordingly, the decoder neural network500is configured to receive, as input, the fixed-length vectors (e.g., text encodings)225and generate as output a corresponding frame of a mel-frequency spectrogram502. The mel-frequency spectrogram502is a frequency-domain representation of sound. Mel-frequency spectrograms emphasize lower frequencies, which are critical to speech intelligibility, while de-emphasizing high frequency, which are dominated by fricatives and other noise bursts and generally do not need to be modeled with high fidelity.

In some implementations, the decoder neural network500includes an attention-based sequence-to-sequence model configured to generate a sequence of output log-mel spectrogram frames, e.g., output mel spectrogram502, based on a transcription106. For instance, the decoder neural network500may be based on the Tacotron 2 model (See “Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions,” by J. Shen, et al., at, e.g., https://arxiv.org/abs/1712.05884, which is incorporated herein by reference). The trained voice cloning system200aprovides an enhanced, multilingual trained voice cloning system that augments the decoder neural network500with additional speaker inputs (e.g., speaker embeddings/identifiers108), and optionally, language embeddings224, an adversarial-trained speaker classifier (e.g., speaker classifier234), and a variational autoencoder-style residual encoder (e.g., the residual encoder212).

The enhanced, trained voice cloning system200a, that augments the attention-based sequence-to-sequence decoder neural network500with one or more of the speaker classifier234, the residual encoder212, the speaker embedding/identifiers108, and/or the language embedding224notably provides many positive results. Namely, the trained voice cloning system200aenables the use of a phonemic input representation for the transcriptions106to encourage sharing of model capacity across different natural languages and different accent/dialects, and incorporates an adversarial loss term233to encourage the trained voice cloning system200ato disentangle how the trained voice cloning system200arepresents speaker identity, which perfectly correlates with the language used in the training data10, from the speech content.

FIG.2Billustrates an example trained voice cloning system200,200bconfigured to convert input training audio signal102corresponding to a reference utterance spoken by a target speaker in a first accent/dialect into an output mel-spectrogram502representing the voice of the target speaker in a second accent/dialect. That is, the trained voice cloning system200bincludes a speech-to-speech (S2S) conversion model. The training voice cloning system200bis contrasted with the training voice cloning system200a(FIG.2A) that uses the corresponding transcription106as an input for generating the output mel-spectrogram502. The S2S conversion model200bis configured to convert the training audio signal102directly into the output mel-spectrogram502without performing speech recognition, or otherwise without requiring the generation of any intermediate discrete representations (e.g., text or phonemes) from the training audio signal102. The S2S conversion model200bincludes a spectrogram encoder240configured to encode the training audio signal102into a hidden feature representation (e.g., a series of vectors) and a spectrogram decoder500configured to decode the hidden representation into the output mel-spectrogram502. For instance, as the spectrogram decoder500receives the input training audio signal102corresponding to the reference utterance, the spectrogram decoder500may process give frames of audio and convert those five frames of audio to ten vectors. The vectors are not a transcription of the frames of training audio signal102, but rather a mathematical representation of the frames of the training audio signal102. In turn, the spectrogram decoder500may generate the output mel spectrogram502corresponding to the training synthesized speech representation based on the vectors received from the spectrogram encoder240. For example, the spectrogram decoder500may receive the ten vectors from the spectrogram encoder240that represent the five frames of audio. Here, the spectrogram decoder500may generate five frames of the output mel spectrogram502corresponding to the speech representation of the reference utterance that includes the intended words or parts of words as the five frames of the training audio signal102in the second/accent dialect.

In some examples, the S2S conversion model200balso includes a text decoder (not shown) that decodes the hidden representation into a textual representation, e.g., phonemes or graphemes. In these examples, the spectrogram decoder500and the text decoder may correspond to parallel decoding branches of the trained voice cloning system200that each receive the hidden representation encoded by the spectrogram encoder240and emit their respective one of the output mel spectrogram502or the textual representation in parallel. As with the TTS-based voice cloning system200aofFIG.2A, the S2S conversion system200bmay further include a waveform synthesizer228, or alternatively a vocoder, to synthesize the output mel spectrogram502into a time-domain waveform for audible output. A time-domain audio waveform includes an audio waveform that defines an amplitude of an audio signal over time. The waveform synthesizer228may include a unit selection module or a WaveNet module for synthesizing the output mel spectrogram502into time-domain waveforms of training synthesized speech representations202. In some implementations, the vocoder228i.e., neural vocoder, is separately trained and conditioned on mel-frequency spectrograms for conversion into time-domain audio waveforms (e.g., training synthesized speech representation202).

In the example shown, the target speaker associated with the training data10speaks in a first accent/dialect (e.g., American English accent). The trained voice cloning system (e.g., S2S voice conversion model)200bis trained to convert the training audio signal102of the training data10spoken in the first accent/dialect directly into training synthesized speech representations202including the voice of the target speaker in a second accent/dialect (e.g., British English accent). Without departing from the scope of the present disclosure, the trained voice cloning system200bmay be trained to convert training audio signal102corresponding to a reference utterance spoken by a target speaker in a first language or speaking style into training synthesized speech representations202that retain the voice of the target speaker but in a different second language or speaking style.

FIG.3illustrates an example training process301for training the TTS system300on training synthesized speech representations202generated by the trained voice cloning system200. The trained voice cloning system200obtains the training data10including training audio signals102and corresponding transcripts106. Each training signal102may be associated with the conditioning inputs that include the speaker embedding/identifiers108and the accent/dialect identifier109. Here, the training audio signals102of the training data10represent human speech in a first accent/dialect (e.g., American English). Based on the training audio signal102(and optionally the corresponding transcript), the trained voice cloning system200is configured to generate a training synthesized speech representation202including the voice of the target speaker in a second accent/dialect different than the first accent/dialect. The training synthesized speech representation202may include an audio waveform or a sequence of mel-frequency spectrograms. The trained voice cloning system200provides the training synthesized speech representation202for training the untrained TTS model300.

The untrained TTS system300includes a TTS model400and a synthesizer150. The TTS model400includes an encoder portion400aand a decoder portion400b. The TTS model400may additionally include a variation layer. The encoder portion400ais trained to learn how to encode the training synthesized speech representation202into a corresponding utterance embedding204that represents a prosody and/or the second accent/dialect captured by the training synthesized speech representation202. During training, the decoder portion400bis conditioned on the transcript106and the conditioning inputs (e.g., speaker embedding/identifiers108and accent/dialect identifier) and configured to decode the utterance embedding204encoded by the encoder portion400afrom the training synthesized speech representation202into a predicted output audio signal402. During training, the decoder portion400breceives the transcript106and the utterance embedding204of the training data to generate the predicted output audio signal. The goal of training is to minimize any loss between the predicted output audio signal402and the training synthesized speech representation202. The decoder portion400bmay also generate a number of predicted frames280corresponding to the predicted output audio signal402. That is, the decoder portion400bdecodes the utterance embedding204into the sequence of fixed-length predicted frames280(referred to interchangeably as ‘predicted frames’) that provide prosodic features and/or accent/dialect information. The prosodic features represent the prosody of the training synthesized speech representation202and include duration, pitch contour, energy contour, and/or mel-frequency spectrogram contour.

In some implementations, the synthesizer150is trained to learn how to generate a predicted synthesized speech representation152from the predicted number of frames280corresponding the predicted output audio signal402from the TTS model400. Here, predicted synthesized speech representation clones the voice of the target speaker in the second accent/dialect and may further include the prosody captured by the training synthesized speech representation202. More specifically, the synthesizer150, like the TTS model400, receives the training synthesized speech representation202output from the voice cloning system200as a ground-truth label to teach the synthesizer to150to generate the predicted synthesized speech representation152that matches the training synthesized speech representation202. The synthesizer150generates gradients/losses154between the predicted synthesized speech representation152and the training synthesized speech representation202during training. In some examples, the synthesizer150back-propagates the gradients/losses154through the TTS model400and the synthesizer150.

Once the TTS model400and the synthesizer of the TTS system300are trained, the trained TTS system300only applies the decoder portion400bto generate synthesized speech152in the second accent/dialect from an input text utterance320. That is, the decoder portion400bmay decode a selected utterance embedding204conditioned on the input text utterance320and the conditioning inputs108,109into an output audio waveform402and corresponding predicted number of frames280. Thereafter, the synthesizer150uses the predicted number of frames280to generate synthesized speech152that clones the voice of the target speaker in the second accent/dialect.

FIGS.4A and4Bshow the TTS model400ofFIG.3represented by a hierarchical linguistic structure for synthesizing an input text utterance320into expressive speech that clones the voice of the target speaker in the target accent/dialect. As will become apparent, the TTS model400may be trained to jointly predict, for each syllable of given input text utterance320, a duration of the syllable and pitch (F0) and energy (C0) contours for the syllable without relying on any unique mapping from the given input text utterance or other linguistic specification to produce synthesized speech152having the target accent/dialect and in the voice of the target speaker.

During training, hierarchical linguistic structure of the TTS model400includes the encoder portion400a(FIG.4A) that encodes a plurality of fixed-length reference frames211sampled from the training synthesized speech representation202into the fixed-length utterance embedding204, and the decoder portion400b(FIG.4B) that learns how to decode the fixed-length utterance embedding204. The decoder portion400bmay decode the fixed-length utterance embedding204into the output audio waveform402including a number of predicted frames280of expressive speech As will become apparent, the TTS model400is trained so that the number of predicted frames280output from the decoder portion400bis equal to the number of reference frames211input to the encoder portion400a. Moreover, the TTS model400is trained so that accent/dialect and prosody information associated with the reference frames211and predicted frames280substantially match one another.

Referring toFIGS.3and4A, the encoder portion400areceives the sequence of fixed-length reference frames211sampled from the synthesized speech representation202output from the trained voice cloning system200. The training synthesized speech representation202includes the voice of the target speaker in the target accent/dialect. The reference frames211may include a duration of 5 milliseconds (ms) and represent one of a contour of pitch (F0) or a contour of energy (C0) (and/or contour of spectral characteristics (M0)) for the synthesized speech representation202. In parallel, the encoder portion400amay also receive a second sequence of reference frames211each including a duration of 5 ms and representing the other one of the contour of pitch (F0) or the contour of energy (C0) (and/or contour of spectral characteristics (M0)) for the synthesized speech representation202. Accordingly, the sequence of reference frames211sampled from the synthesized speech representation202provide a duration, pitch contour, energy contour, and/or spectral characteristics contour to represent the target accent/dialect and/or prosody of the synthesized speech representation202. The length or duration of the synthesized speech representation202correlates to a sum of the total number of reference frames211.

The encoder portion400aincludes hierarchical levels of reference frames211, phonemes421,421a, syllables430,430a, words440,440a, and sentences450,450afor the synthesized speech representation202that clock relative to one another. For instance, the level associated with the sequence of reference frames211clocks faster than the next level associated with the sequence of phonemes421. Similarly, the level associated with the sequence of syllables430clocks slower than the level associated with the sequence of phonemes421and faster than the level associated with the sequence of words440. Accordingly, the slower clocking layers receive, as input, an output from the faster clocking layers so that the output after the final clock (i.e., state) of a faster later is taken as the input to the corresponding slower layer to essentially provide a sequence-to-sequence encoder. In the examples shown, the hierarchical levels include Long Short-Term Memory (LSTM) levels.

In the example shown, the synthesized speech representation202includes one sentence450,450A with three words440,440A-C. The first word440,440A includes two syllables430,430Aa-Ab. The second word440,440B incudes one syllable430,430Ba. The third word440,440aincludes two syllables430,430Ca-Cb. The first syllable430,430Aa of the first word440,440A includes two phonemes421,421Aa1-Aa2. The first syllable430,430Ba of the second word440,440B includes three phonemes421,421Ba1-Ba3. The first syllable430,430Ca of the third word440,440C incudes one phoneme421,421Ca1. The second syllable430,430Cb of the third word440,440C includes two phonemes421,421Cb1-Cb2.

In some implementations, the encoder portion400afirst encodes the sequence of reference frames211into frame-based syllable embeddings432,432Aa-Cb. Each frame-based syllable embedding432may indicate reference prosodic features represented as a numerical vector indicative of a duration, pitch (F0), and/or energy (C0) associated with the corresponding syllable430. In some implementations, the reference frames211define a sequence of phonemes421Aa1-421Cb2. Here, instead of encoding a subset of reference frames211into one or more phonemes, the encoder portion400ainstead accounts for the phonemes421by encoding phone level linguistic features422,422Aa1-Cb2into phone feature-based syllable embeddings434,434Aa-Cb. Each phoneme-level linguistic feature422may indicate a position of the phoneme, while each phoneme feature-based syllable embedding434includes a vector indicating the position of each phoneme within the corresponding syllable430as well as the number of phonemes421within the corresponding syllable430. For each syllable430, the respective syllable embeddings432,434may be concatenated and encoded with the respective syllable-level linguistic features436,436Aa-Cb for the corresponding syllable430. Moreover, each syllable embedding432,434is indicative of a corresponding state for the level of syllables430.

With continued reference toFIG.4A, the blocks in the hierarchical layers that include a diagonal hatching pattern correspond to linguistic features (except for the word level440) for a particular level of the hierarchy. The hatching pattern at the word-level440includes word embeddings442extracted as linguistic features from the input text utterance320(during inference) or WP embeddings442output from the bidirectional encoder representations from transformers (BERT) model470based on word units472obtained from the transcript106. Since the recurrent neural network (RNN) portion of the encoder400ahas no notion of wordpieces, the WP embeddings442corresponding to the first wordpeice of each word may be selected to represent the word which may contain one or more syllables430. With the frame-based syllable embeddings432and the phone feature-based syllable embeddings434, the encoder portion400aconcatenates and encodes these syllable embeddings432,434with other linguistic features436,453,442(or WP embeddings442). For example, the encoder portion400aencodes the concatenated syllable embeddings432,434with syllable-level linguistic features436,436Aa-Cb, word-level linguistic features (or WP embeddings432,432A-C output from the BERT model470), and/or sentence level linguistic features452,452A. By encoding the syllable embeddings432,434with the linguistic features436,452,442(or WP embeddings442), the encoder portion400agenerates an utterance embedding204for the synthesized speech representation202. The utterance embedding204may be stored in the data storage180(FIG.1) along with the transcription106(e.g., textual representation) of the synthesized speech representation202. From the training data10, the linguistic features432,442,452may be extracted and stored for use in conditioning the training of the hierarchical linguistic structure. The linguistic features (e.g., linguistic features422,436,442,452) may include, without limitation, individual sounds for each phoneme and/or the position of each phoneme in a syllable, whether each syllable is stressed or un-stressed, syntactic information for each word, and whether the utterance is a question or phrase and/or a gender of a speaker of the utterance. As used herein, any reference of word-level linguistic feature442with respect to the encoder and decoder portions400a,400bof the TTS model400can be replaced with WP embeddings from the BERT model470.

In the example ofFIG.4A, encoding blocks422,422Aa-Cb are shown to depict the encoding between the linguistic features436,442,452and the syllable embeddings432,434. Here, the blocks422are sequence encoded at a syllable rate to generate the utterance embedding204. As an illustration, the first block422Aa is fed as input into a second block422Ab. The second block422Ab is fed as an input into a third block422Ba. The third block422Ba is fed as an input into the fourth block422Ca. The fourth block422Ca is fed into the fifth block422Cb. In some configurations, the utterance embedding204includes a mean μ and the standard deviation σ are with respect to the training data of multiple training synthesized speech representations202.

In some implementations, each syllable430receives, as input, a corresponding encoding of a subset of references frames211and includes a duration equal to the number of reference frames211in the encoded subset. In the example shown, the first seven fixed-length reference frames211are encoded into syllable430Aa; the next four fixed-length reference frames211are encoded into syllable430Ab; the next eleven fixed-length references frames211are encoded into syllable430Ba; the next three fixed-length reference frames211are encoded into syllable430Ca; and the final six fixed-length reference frames211are encoded into syllable430Cb. Thus, each syllable430in the sequence of syllables430may include a corresponding duration based on the number of reference frames211encoded into the syllable430and corresponding pitch and/or energy contours. For instance, syllable430Aa includes a duration equal to 35 ms (i.e., seven reference frames211each having the fixed-length of five milliseconds) and syllable430Ab includes a duration equal to 20 ms (i.e., four reference frames211each having the fixed-length of five milliseconds). Thus, the level of reference frames211clocks a total of ten times for a single clocking between the syllable430Aa and the next syllable430Ab at the level of syllables430. The duration of the syllables430may indicate timing of the syllables430and pauses in between adjacent syllables430.

In some examples, the utterance embedding204generated by the encoder portion400ais a fixed-length utterance embedding204that includes a numerical vector representing an accent/dialect and/or prosody of the synthesized speech representation202. In some examples, the fixed-length utterance embedding204includes a numerical vector having a value equal to “128” or “256.”

Referring now toFIGS.3and4B, during training the decoder portion400bof the TTS model400is configured to produce a plurality of fixed-length syllable embeddings435by initially decoding the fixed-length utterance embedding204that specifies the target accent/dialect and prosody for the transcript106. More specifically, the utterance embedding204represents the target accent/dialect and prosody possessed by the synthesized speech representation202output from the trained voice cloning system200. Moreover, the decoder portion400bdecodes the fixed-length utterance embedding204associated with the transcript106using the received speaker embedding/identifier108that indicates the voice characteristics of the target speaker and/or the accent/dialect identifier109that indicates the target accent/dialect for the resulting synthesized speech152. Thus, the decoder portion400bis configured to back-propagate the utterance embedding204to generate the plurality of fixed-length predicted frames280that closely match the plurality of fixed-length reference frames211encoded by the encoder portion400aofFIG.4A. For instance, fixed-length predicted frames280for both pitch (F0) and energy (C0) may be generated in parallel to represent the target accent/dialect (e.g., predicted accent) that substantially matches the target accent/dialect prosody possessed by the training synthesized speech representation202. In some examples, the speech synthesizer150uses the fixed-length predicted frames280to produce the synthesized speech152that clones the voice of the target speaker in the intended accent/dialect based on the fixed-length utterance embedding204. For instance, a unit selection module or a WaveNet module of the speech synthesizer150may use the number of predicted frames280to produce the synthesized speech152having the intended accent and/or intended prosody. Notably, and as mentioned previously, the intended accent/dialect produced in the synthesized speech152includes an accent/dialect that is not native to the target speaker and not spoken by the target speaker in any of the reference utterances of the training data10.

In the example shown, the decoder portion400bdecodes the utterance embedding204received from the encoder portion400ainto hierarchical levels of words440,440b, syllables430,430b, phonemes421,421b, and fixed-length predicted frames280. Specifically, the fixed-length utterance embedding204corresponds to a variational layer of hierarchical input data for the decoder portion400band each of the stacked hierarchical levels include Long Short-Term Memory (LSTM) processing cells variably clocked to a length of the hierarchical input data. For instance, the syllable level430clocks faster than the word level440and slower than the phoneme level421. The rectangular blocks in each level correspond to LSTM processing cells for respective words, syllables, phonemes, or frames. Advantageously, the trained voice cloning system200gives the LSTM processing cells of the word level440memory over the last 1000 words, gives the LSTM cells of the syllable level430memory over the last 100 syllables, gives the LSTM cells of the phoneme level421memory over the last 100 phonemes, and gives the LSTM cells of the fixed-length pitch and/or energy frames280memory over the last 100 fixed-length frames280. When the fixed-length frames280include a duration (e.g., frame rate) of five milliseconds each, the corresponding LSTM processing cells provide memory over the last 500 milliseconds (e.g., a half second).

In the example shown the decoder portion400bof the hierarchical linguistic structure simply back-propagates the fixed-length utterance embedding204encoded by the encoder portion400ainto the sequence of three words440A-440C, the sequence of five syllables430Aa-430Cb, and the sequence of nine phonemes421Aa1-421Cb2to generate the sequence of predicted fixed-length frames280. The decoder portion400bis conditioned upon linguistic features of the training data10during training and the input text utterance320during inference. By contrast to the encoder portion400aofFIG.4Awhere outputs from faster clocking layers are received as inputs by slower clocking layers, the decoder portion400bincludes outputs from slower clocking layers feeding faster clocking layers such that the output of a slower clocking layer is distributed to the input of the faster clocking layer at each clock cycle with a timing signal appended thereto. Additional details of the TTS model400are described with reference to U.S patent application Ser. No. 16/867,427, filed on May 5, 2020, the contents of which are incorporated by reference in their entirety.

Referring toFIG.4B, in some implementations, the hierarchical linguistic structure for the TTS model400is adapted to provide a controllable model for predicting mel spectral information for an input text utterance320during inference, while at the same time effectively controlling the accent/dialect and prosody implicitly represented in the mel spectral information. Specifically, the TTS model400may predict a mel-frequency spectrogram502for the input text utterance and provide the mel-frequency spectrogram502as input to a vocoder network155of the speech synthesizer150for conversion into a time-domain audio waveform. A time-domain audio waveform includes an audio waveform that defines an amplitude of an audio signal over time. As will become apparent, the speech synthesizer150can generate synthesized speech152from the input text utterance320using the TTS system300trained on sample transcripts106and training synthesized speech representation202output from the trained voice cloning system200. That is, the TTS system300does not receive complex linguistic and acoustic features that require significant domain expertise to produce, but rather is able to convert input text utterances320to mel-frequency spectrograms502using an end-to-end deep neural network. The vocoder network155, i.e., neural vocoder, may be separately trained and conditioned on mel-frequency spectrograms for conversion into tie-domain audio waveforms.

A mel-frequency spectrogram includes a frequency-domain representation of sound. Mel-frequency spectrograms emphasize lower frequencies, which are critical to speech intelligibility, while de-emphasizing high frequency, which are dominated by fricatives and other noise bursts and generally do not need to be modeled with high fidelity. The vocoder network155can be any network that is configured to receive mel-frequency spectrograms and generate audio output samples based on the mel-frequency spectrograms. For example, the vocoder network155can be based on the parallel feed-forward neural network described in van den Oord,Parallel WaveNet: Fast High-Fidelity Speech Synthesis, available at https://arxiv.org/pdf/1711.10433.pdf, and incorporated herein by reference. Alternatively, the vocoder network155can be an autoregressive neural network.

Referring now to FIG.5, the spectrogram decoder500(interchangeably referred to as decoder portion500) of the trained voice cloning system200may include an architecture having a pre-net510, a Long Short-Term Memory (LSTM) subnetwork520, a linear projection530, and a convolutional post-net540. The pre-net510, through which a mel-frequency prediction for a previous time step passes, may include two fully-connected layers of hidden rectified linear units (ReLUs). The pre-net510acts as an information bottleneck for learning attention to increase convergence speed and to improve generalization capability of the speech synthesis system during training. In order to introduce output variation, dropout with probability 0.5 may be applied to later in the pre-net510.

The LSTM subnetwork520may include two or more LSTM layers. At each time step, the LSTM subnetwork520receives a concatenation of the output of the pre-net510, the fixed-length context vector225(e.g., the text encoding output from the encoder ofFIGS.2A and2B) is projected to a scalar and passed through a sigmoid activation to predict that the output sequence of mel spectrograms502has completed. The LSTM layers may be regularized using zoneout with probability of, for example, 0.1. The linear projection receives as input the output of the LSTM subnetwork520and produces a prediction of a mel-frequency spectrogram502,502P.

The convolutional post-net540with one or more convolutional layers processes the predicted mel-frequency spectrogram502P for the time step to predict a residual542to add to the predicted mel-frequency spectrogram502P at adder550. This improves the overall reconstruction. Each convolutional layer except for the final convolutional layer may be followed by batch normalization and hyperbolic tangent (TanH) activations. The convolutional layers are regularized using dropout with a probability of, for example, 0.5. The residual542is added to the predicted mel-frequency spectrogram502P generated by the linear projection520, and the sum (i.e., the mel-frequency spectrogram502) may be provided to the speech synthesizer150. In some implementations, in parallel to the decoder portion500predicting mel-frequency spectrograms502for each time step, a concatenation of the output of the LSTM subnetwork520, [the utterance embedding], and the portion of the training data10(e.g., a character embedding generated by a text encoder (not shown)) is projected to a scalar and passed through a sigmoid activation to predict the probability that the output sequence of mel frequency spectrograms502has completed. The output sequence mel-frequency spectrograms502corresponds to the training synthesized speech representation202for the training data10and includes the intended prosody and intended accent of the target speaker.

This “stop token” prediction is used during inference to allow the trained voice cloning system200to dynamically determine when to terminate generation instead of always generating for a fixed duration. When the stop token indicates that generation has terminated, i.e., when the stop token probability exceeds a threshold value, the decoder portion500stops predicting mel-frequency spectrograms502P and returns the mel-frequency spectrograms predicted up to that point as the training synthesized speech representation202. Alternatively, the decoder portion500may always generate mel-frequency spectrograms502of the same length (e.g., 10 seconds).

FIG.6is a flowchart of an exemplary arrangement of operations for a method600of synthesizing an input text utterance into expressive speech having an intended accent/dialect and cloning a voice of a target speaker432. The data processing hardware122(FIG.1) may execute the operations for the method600by executing instructions stored on the memory hardware124. At operation602, the method600includes obtaining training data10including a plurality of training audio signals102and corresponding transcripts106. Each training audio signal102corresponds to a reference utterance spoken by a target speaker in a first accent/dialect. Each transcript106includes a textual representation of the corresponding reference utterance. For each training audio signal102of the training audio signals102, the method600performs operations604and606. At operation604, the method600includes generating, by a trained voice cloning system200configured to receive the training audio signal102corresponding to the reference utterance spoken by the target speaker in the first accent/dialect as input, a training synthetic speech representation202of the corresponding reference utterance spoken by the target speaker. Here, the training synthesized speech representation202includes a voice of the target speaker in a second accent/dialect that is different than the first accent/dialect. At operation606, the method600includes training a text-to-speech (TTS) system300based on the corresponding transcript106of the training audio signal102and the training synthesized speech representation202of the corresponding reference utterance generated by the trained voice cloning system200.

At operation608, the method600includes receiving an input text utterance320to be synthesized into expressive speech152in the second accent/dialect. At operation610, the method600includes obtaining conditioning inputs including a speaker embedding/identifier108that represents voice characteristics of the target speaker and an accent/dialect identifier109that identifies the second accent/dialect. At operation612, the method600includes generating, using the trained TTS system300conditioned on the obtained conditioning inputs, by processing the input text utterance320an output audio waveform402corresponding to a synthesized speech representation202of the input text utterance320that clones the voice of the target speaker in the second accent/dialect.

A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an “application,” an “app,” or a “program.” Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

The non-transitory memory may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. The non-transitory memory may be volatile and/or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

FIG.7is schematic view of an example computing device700that may be used to implement the systems and methods described in this document. The computing device700is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

The computing device700includes a processor710, memory720, a storage device730, a high-speed interface/controller740connecting to the memory720and high-speed expansion ports750, and a low speed interface/controller760connecting to a low speed bus770and a storage device730. Each of the components710,720,730,740,750, and760, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor710can process instructions for execution within the computing device700, including instructions stored in the memory720or on the storage device730to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display780coupled to high speed interface740. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices700may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory720stores information non-transitorily within the computing device700. The memory720may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory720may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

The storage device730is capable of providing mass storage for the computing device700. In some implementations, the storage device730is a computer-readable medium. In various different implementations, the storage device730may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory720, the storage device730, or memory on processor710.

The high speed controller740manages bandwidth-intensive operations for the computing device700, while the low speed controller760manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller740is coupled to the memory720, the display780(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports750, which may accept various expansion cards (not shown). In some implementations, the low-speed controller760is coupled to the storage device730and a low-speed expansion port790. The low-speed expansion port790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device700may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server700aor multiple times in a group of such servers700a, as a laptop computer700b, or as part of a rack server system700c.

Various implementations of the systems and techniques described herein can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

The processes and logic flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.