Patent Description:
Speech synthesis systems use text-to-speech (TTS) models to generate speech from textual input. The generated/synthesized speech should accurately convey the message (intelligibility) while sounding like human speech (naturalness) with an intended prosody (expressiveness). While traditional concatenative and parametric synthesis models are capable of providing intelligible speech and recent advances in neural modeling of speech have significantly improved the naturalness of synthesized speech, most existing TTS models are ineffective at modeling a full variety of prosodic styles, thereby causing synthesized speech used by important applications to lack expressiveness. For instance, it is desirable for applications such as conversational assistants and long-form readers to produce realistic speech by imputing prosody features not conveyed in textual input, such as intonation, stress, and rhythm and style. For example, a simple statement can be spoken in many different ways depending on whether the statement is a question, an answer to a question, there is uncertainty in the statement, or to convey any other meaning about the environment or context which is unspecified by the input text.

It may be useful in some scenarios to transfer prosody modeled from various different speakers in a specific prosodic domain/vertical, such as news reading, sports commentators, educational lecturers, etc., to an existing target voice. Applying the target voice to a new prosodic domain/vertical in this manner can be particularly difficult since the amount of training data associated with the target voice in the new domain/vertical is insufficient.

<CIT>describes a method for representing an intended prosody in synthesized speech includes receiving a text utterance having at least one word, and selecting an utterance embedding for the text utterance. Each word in the text utterance has at least one syllable and each syllable has at least one phoneme. The utterance embedding represents an intended prosody. For each syllable, using the selected utterance embedding, the method also includes: predicting a duration of the syllable by encoding linguistic features of each phoneme of the syllable with a corresponding prosodic syllable embedding for the syllable; predicting a pitch contour of the syllable based on the predicted duration for the syllable; and generating a plurality of fixed-length predicted pitch frames based on the predicted duration for the syllable. Each fixed-length predicted pitch frame represents part of the predicted pitch contour of the syllable.

<CIT> describes a multilingual text-to-speech synthesis method and system. The method includes receiving first learning data including a learning text of a first language and learning speech data of the first language corresponding to the learning text of the first language, receiving second learning data including a learning text of a second language and learning speech data of the second language corresponding to the learning text of the second language, and generating a single artificial neural network text-to-speech synthesis model by learning similarity information between phonemes of the first language and phonemes of the second language based on the first learning data and the second learning data.

One aspect of the disclosure provides a method for synthesizing an input text utterance into expressive speech having an intended prosody and a target voice. The method includes receiving, at data processing hardware, the input text utterance to be synthesized into expressive speech having the intended prosody and the target voice. The method also includes generating, by the data processing hardware, using a first text-to-speech (TTS) model, an intermediate synthesized speech representation for the input text utterance. The intermediate synthesized speech representation possesses the intended prosody. The method also includes providing, by the data processing hardware, the intermediate synthesized speech representation to a second TTS model including an encoder portion and a decoder portion. The encoder portion is configured to encode the intermediate synthesized speech representation into an utterance embedding that specifies the intended prosody. The decoder portion is configured to process the input text utterance and the utterance embedding to generate an output audio signal of expressive speech. The output audio signal has the intended prosody specified by the utterance embedding and speaker characteristics of the target voice.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the method also includes sampling, by the data processing hardware, from the intermediate synthesized speech representation, a sequence of fixed-length reference frames providing prosodic features that represent the intended prosody possessed by the intermediate synthesized speech representation. Here, providing the intermediate synthesized speech representation to the second TTS model includes providing the sequence of fixed-length reference frames sampled from the intermediate synthesized speech representation to the encoder portion, whereby the encoder portion is configured to encode the sequence of fixed-length reference frames into the utterance embedding. The prosodic features that represent the intended prosody possessed by the intermediate synthesized speech representation may include duration, pitch contour, energy contour, and/or mel-frequency spectrogram contour. In these implementations, the encoder portion may be configured to encode the sequence of fixed-length reference frames into the utterance embedding by, for each syllable in the intermediate synthesized speech representation: encoding phone-level linguistic features associated with each phoneme in the syllable into a phoneme feature-based syllable embedding; encoding the fixed-length reference frames associated with the syllable into a frame-based syllable embedding, the frame-based syllable embedding indicative of a duration, pitch, and/or energy associated with the corresponding syllable; and encoding, into a corresponding prosodic syllable embedding for the syllable, the phoneme feature-based and frame-based syllable embeddings with syllable-level linguistic features associated with the syllable, sentence-level linguistic features associated with the intermediate synthesized speech representation, and word-level linguistic features associated with a word that includes the corresponding syllable.

The word-level linguistic features may include a wordpiece embedding obtained from a sequence of wordpiece embeddings generated by a Bidirectional Encoder Representations from Transformers (BERT) model from the input text utterance. In some examples, the decoder portion is configured to process the input text utterance and the utterance embedding to generate the output audio signal by decoding, using the input text utterance, the corresponding utterance embedding into a sequence of fixed-length predicted frames providing a prosodic representation of the input text utterance. Here, the prosodic representation represents the intended prosody specified by the utterance embedding. The second TTS model may be trained so that a number of the fixed-length predicted frames decoded by the decoder portion is equal to a number of the fixed-length reference frames sampled from the intermediate synthesized speech representation.

In some examples, the utterance embedding includes a fixed-length numerical vector. The intermediate synthesized speech representation may include an audio waveform or a sequence of mel-frequency spectrograms that captures the intended prosody such that providing the intermediate synthesized speech representation to the second TTS model may include providing the audio waveform or the sequence of mel-frequency spectrograms to the encoder portion. Here, the encoder portion is configured to encode the audio waveform or the sequence of mel-frequency spectrograms into the utterance embedding.

In some implementations, the method also includes obtaining, by the data processing hardware, a speaker embedding representing the speaker characteristics of the target voice, and providing, by the data processing hardware, the speaker embedding to the decoder portion of the second TTS model to process the input text utterance, the utterance embedding, and the speaker embedding to generate the output audio signal of expressive speech. The intermediate synthesized speech representation generated using the first TTS model may include an intermediate voice that lacks the speaker characteristics of the target voice and includes undesirable acoustic artifacts.

The method may also include receiving, at the data processing hardware, training data including a plurality of training audio signals and corresponding transcripts, each training audio signal includes an utterance of human speech having the intended prosody spoken by a corresponding speaker in a prosodic domain/vertical associated with the intended prosody. Each transcript includes a textual representation of the corresponding training audio signal. For each corresponding transcript of the training data: the method also includes training, by the data processing hardware, the first TTS model generate a corresponding reference audio signal including a training synthesized speech representation that captures the intended prosody of the corresponding utterance of human speech; training, by the data processing hardware, the encoder portion of the second TTS model by encoding the corresponding training synthesized speech representation into a corresponding utterance embedding representing the intended prosody captured by the training synthesized speech representation; training, by the data processing hardware, using the corresponding transcript of the training data, the decoder portion of the second TTS model by decoding the corresponding utterance embedding encoded by the encoder portion into a predicted output audio signal of expressive speech having the intended prosody; generating gradients/losses between the predicted output audio signal and the corresponding reference audio signal; and back-propagating the gradients/losses through the second TTS model.

The first TTS model and the second TTS model may be trained separately or jointly. In some examples, the first TTS model includes a first neural network architecture and the second TTS model includes a second neural network architecture that is different than the first neural network architecture. In other examples, the first TTS model and the second TTS model include a same neural network architecture.

Another aspect of the disclosure provides a system for synthesizing an input text utterance into expressive speech having an intended prosody and a target voice. The system includes data processing hardware and memory hardware in communication with the data processing hardware and storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations. The operations include receiving the input text utterance to be synthesized into expressive speech having the intended prosody and the target voice. The operations also include generating, using a first text-to-speech (TTS) model, an intermediate synthesized speech representation for the input text utterance. The intermediate synthesized speech representation possesses the intended prosody. The operations also include providing the intermediate synthesized speech representation to a second TTS model including an encoder portion and a decoder portion. The encoder portion is configured to encode the intermediate synthesized speech representation into an utterance embedding that specifies the intended prosody. The decoder portion is configured to process the input text utterance and the utterance embedding to generate an output audio signal of expressive speech. The output audio signal has the intended prosody specified by the utterance embedding and speaker characteristics of the target voice.

This aspect may include one or more of the following optional features. In some implementations, the operations also include sampling, from the intermediate synthesized speech representation, a sequence of fixed-length reference frames providing prosodic features that represent the intended prosody possessed by the intermediate synthesized speech representation. Here, providing the intermediate synthesized speech representation to the second TTS model includes providing the sequence of fixed-length reference frames sampled from the intermediate synthesized speech representation to the encoder portion, whereby the encoder portion is configured to encode the sequence of fixed-length reference frames into the utterance embedding. The prosodic features that represent the intended prosody possessed by the intermediate synthesized speech representation may include duration, pitch contour, energy contour, and/or mel-frequency spectrogram contour. In these implementations, the encoder portion may be configured to encode the sequence of fixed-length reference frames into the utterance embedding by, for each syllable in the intermediate synthesized speech representation: encoding phone-level linguistic features associated with each phoneme in the syllable into a phoneme feature-based syllable embedding; encoding the fixed-length reference frames associated with the syllable into a frame-based syllable embedding, the frame-based syllable embedding indicative of a duration, pitch, and/or energy associated with the corresponding syllable; and encoding, into a corresponding prosodic syllable embedding for the syllable, the phoneme feature-based and frame-based syllable embeddings with syllable-level linguistic features associated with the syllable, sentence-level linguistic features associated with the intermediate synthesized speech representation, and word-level linguistic features associated with a word that includes the corresponding syllable.

In some implementations, the operations also include obtaining a speaker embedding representing the speaker characteristics of the target voice and providing the speaker embedding to the decoder portion of the second TTS model to process the input text utterance, the utterance embedding, and the speaker embedding to generate the output audio signal of expressive speech. The intermediate synthesized speech representation generated using the first TTS model may include an intermediate voice that lacks the speaker characteristics of the target voice and includes undesirable acoustic artifacts.

The operations also include receiving training data including a plurality of training audio signals and corresponding transcripts, each training audio signal includes an utterance of human speech having the intended prosody spoken by a corresponding speaker in a prosodic domain/vertical associated with the intended prosody. Each transcript includes a textual representation of the corresponding training audio signal. For each corresponding transcript of the training data: the operations also include training the first TTS model generate a corresponding reference audio signal including a training synthesized speech representation that captures the intended prosody of the corresponding utterance of human speech; training the encoder portion of the second TTS model by encoding the corresponding training synthesized speech representation into a corresponding utterance embedding representing the intended prosody captured by the training synthesized speech representation; training, using the corresponding transcript of the training data, the decoder portion of the second TTS model by decoding the corresponding utterance embedding encoded by the encoder portion into a predicted output audio signal of expressive speech having the intended prosody; generating gradients/losses between the predicted output audio signal and the corresponding reference audio signal; and back-propagating the gradients/losses through the second TTS model.

Text to-speech (TTS) models, often used by speech synthesis systems, are generally only given text inputs without any reference acoustic representation at runtime, and must impute many linguistic factors that are not provided by the text inputs in order to produce realistically sounding synthesized speech. A subset of these linguistic factors are collectively referred to as prosody and may include intonation (pitch variation), stress (stressed syllables vs. non-stressed syllables), duration of sounds, loudness, tone, rhythm, and style of the speech. Prosody may indicate the emotional state of the speech, the form of the speech (e.g., statement, question, command, etc.), the presence of irony or sarcasm of the speech, uncertainty in the knowledge of the speech, or other linguistic elements incapable of being encoded by grammar or vocabulary choice of the input text. Accordingly, a given text input that is associated with a high degree of prosodic variation can produce synthesized speech with local changes in pitch and speaking duration to convey different semantic meanings, and also with global changes in the overall pitch trajectory to convey different moods and emotions.

Specific domains/verticals, such as, without limitation, news reading (e.g., newscasters), sports commentators, educational lecturers, each include utterances spoken by a variety of different speakers/voices that have different voice characteristics (e.g., male/female, language, accent, etc.) but possess a same prosodic style associated with the specific domain/vertical. For example, prosodic representations of utterances spoken by sports commentators may convey a lot of emotion, whereas prosodic representations of utterances spoken by newscasters may convey a slower speaking rate and clearer enunciation of words. While recent advances in neural modeling of speech have significantly improved the naturalness of synthesized speech and provide potential for robustly synthesizing speech by predicting linguistic factors corresponding to prosody that are not provided by text inputs, the task of modeling prosody only is very difficult since disentangling prosody from speaker characteristics such as accent is a non-trivial process. Moreover, neural network-based prosody models tasked with modeling a specific prosody using training utterances from a large corpus of various voices belonging to a particular domain/vertical associated with the specific prosody, inherently generate synthesized speech with an imperfect voice that contains undesirable/unwanted acoustic artifacts due to the large corpus lacking a sufficient number of training utterances spoken by a same speaker.

Speech synthesis systems may employ TTS models capable of producing synthesized speech in a particular target voice. For instance, it may be desirable for an e-reader application to produce synthesized speech from input text in a voice of Bob Dylan, or as another example, a text message application could produce a synthesized speech representation of a received text message in that of a voice of the person that sent the text message. However, training TTS models to produce synthesized speech in a particular target voice and also having an intended prosody, is a non-trivial task, especially when sufficient training utterances spoken by a speaker of the target voice and having the intended prosody are not available. For example, in a scenario to produce synthesized speech in the voice of deceased newscaster Peter Jennings from a play-by-play transcript of Super Bowl LIV between the San Francisco 49ers and the Kansas City Chiefs, it would be desirable for the resulting synthesized speech in the voice of Peter Jennings to also have a prosody in a sports commentator vertical. While a multitude of utterances spoken by Peter Jennings could be sampled from recordings of ABC World News Tonight in which Peter Jennings was the anchor from <NUM> to <NUM>, these sampled utterances would have very little value as training examples for modeling prosody in the sports commentator vertical since the utterances are sampled from an entirely different vertical, e.g., a newscasters vertical. Even more problematic, since Peter Jennings has been deceased since August <NUM>, <NUM>, Peter Jennings is not available to provide any training utterances of value that would include him speaking with the prosody style in the sports commentator vertical.

Implementations herein are directed toward learning a specific prosody possessed by a corpus of training utterances spoken in various voices by different speakers and generating synthetic expressive speech from input text such that the synthetic expressive speech reproduces the learned specific prosody in a target voice. Here, no utterances possessing the specific prosody in the target voice are required for training. More specifically, implementations are directed toward a two-level speech prosody transfer system in which a first text-to-speech (TTS) model is tasked with only learning an intended prosody possessed by the corpus of training utterances and a second TTS model is tasked with transferring the intended prosody learned by the first TTS model to synthesized speech representations in the target voice.

Described in greater detail below, the first TTS model is trained to produce an intermediate speech representation that captures the intended prosody without attempting to disentangle the intended prosody and speaker characteristics. As such, the resulting intermediate synthesized speech representation produced by the first TTS model captures the intended prosody (expressiveness) that was conveyed in the training utterances, but may include an imperfect voice having reduced quality (e.g., noise artifacts) and lacking speaker characteristics (e.g., accent). As such, the intermediate synthesized speech representation is not suitable for a human listener since it is not intended to accurately convey a message (intelligibility), nor is the intermediate synthesized speech representation intended to sound like human speech (naturalness). Despite the intermediate synthesized speech representation having the imperfect voice, and thus not conveying speaker characteristics representative of the target voice, the second TTS model is trained to reproduce the intended prosody captured by the intermediate speech representation and generate expressive speech having the intended produced in the target voice. That is, the second TTS model generates expressive speech with the intended prosody and having speaker characteristics associated with the target voice. Here, the target voice may be associated with an actor that never spoke any of the training utterances possessing the intended prosody.

The second TTS model may correspond to a prosody transfer model that includes an encoder portion and a decoder portion. Here, the prosody transfer model may correspond to a variational autoencoder (VAE) architecture or a sequence-to-sequence feature prediction network architecture. The encoder portion is configured to encode an intermediate synthesized speech representation produced by the first TTS model into an utterance embedding that specifies the intended prosody captured by the intermediate synthesized speech representation, while the decoder portion is configured to decode the utterance embedding to predict prosodic features, such as durations of phonemes and pitch and energy contours for each syllable. In some examples, the decoder portion is configured to decode the utterance embedding to predict mel-spectral spectrograms in addition to or in lieu of the prosodic features. The mel-spectral spectrograms may inherently convey the intended prosody.

The first TTS system may train on training utterances of human speech and corresponding transcripts to produce training synthesized speech representations from the transcripts that capture the prosody of the corresponding training utterance of human speech. The training utterances may all be sampled from a particular prosodic vertical such that each training utterance possesses an intended prosody associated with the particular prosodic vertical. The encoder portion of the second TTS model may train on utterance embeddings representing the captured prosody by encoding the numerous training synthesized speech representations produced by the first TTS model conditioned on prosodic features and linguistic features embeddings representing the training synthesized speech representations. The prosodic features may represent acoustic information about the reference audio signals in terms of pitch (F0), phoneme duration, and energy (C0). For instance, the prosodic features may include phoneme durations and fixed-length frames of pitch and energy sampled from the reference audio signal. The linguistic features may include, without limitation: phoneme-level linguistic features, containing information about the position of a phoneme in a syllable, the phoneme identity, and a number of phonemes in a syllable; syllable-level linguistic features, containing information such as whether a syllable identify and whether the syllable is stressed or un-stressed; word-level linguistic features encoding syntactic information about each word, and sentence-level linguistic features containing information about a speaker, a gender of the speaker, and/or whether the utterance is a question or phrase. The linguistic features may be extracted from the corresponding transcript for each training utterance. In some examples, the second TTS model incorporates a Bidirectional Encoder Representations from Transformers (BERT) model that is configured to output wordpiece embeddings. In these examples, the wordpiece embeddings may replace the word-level linguistic features that would otherwise encode syntactic information about each word explicitly.

Each utterance embedding encoded by the encoder portion may be represented by a fixed-length numerical vector. In some implementations, the fixed-length numerical vector includes a value equal to <NUM>. However, other implementations may use fixed-length numerical vectors having values greater than or less than <NUM>. For a given input text utterance, the decoder portion may process the input text utterance and a fixed-length utterance embedding to generate an output audio signal of expressive speech. Here, the output audio signal has an intended prosody specified by the utterance embedding. The output audio signal may include a sequence of predicted fixed-length frames (e.g., five milliseconds) of pitch, energy, and/or phoneme durations, or the output audio signal may include mel-frequency spectrogram frames that convey the intended prosody. Additionally, the decoder portion may receive a speaker embedding that provides speaker characteristics of a target voice. As such, the output audio signal having the intended prosody may also include the speaker characteristics of the target voice. A synthesizer may receive, as input, the output audio signal produced by the second TTS model and generate, as output, a synthesized speech representation of the input text utterance that has the intended prosody and is spoken in the target voice.

<FIG> shows an example system <NUM> for training a two-stage prosody transfer system <NUM> to synthesize a text utterance <NUM> into expressive speech <NUM> in a target voice and having a prosodic representation <NUM> that represents an intended prosody associated with a particular prosodic vertical <NUM>. The system <NUM> includes a computing system (interchangeably referred to as 'computing device') <NUM> having data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions executable by the data processing hardware <NUM> to cause the data processing hardware <NUM> to perform operations. In some implementations, the computing system <NUM> (e.g., data processing hardware <NUM>) provides a two-level prosody transfer system <NUM> trained to generate an output audio signal <NUM> of expressive speech from an input text utterance <NUM> such that the output audio signal <NUM> has an intended prosody from a particular prosodic vertical <NUM> and speaker characteristics of the target voice. The output audio signal <NUM> conveys the prosodic representation <NUM> representing the intended prosody to enable a speech synthesizer <NUM> to generate an audio waveform of synthesized speech <NUM> having the intended prosody in the target voice.

The prosody transfer system <NUM> includes a first text-to-speech (TTS) system <NUM> having a first TTS model <NUM> and a second TTS system <NUM> having a second TTS model <NUM>. The first and second TTS systems <NUM>, <NUM> may each include a speech synthesizer <NUM>. The first and second TTS models <NUM>, <NUM> may each include respective neural network architectures that may be the same or different. The first TTS system <NUM> is configured to use the first TTS model <NUM> for generating an intermediate synthesized speech representation <NUM> of the input text utterance <NUM>. For example, the first TTS model <NUM> may generate an intermediate output signal <NUM>, such as a sequence of mel-frequency spectrograms, that captures/possesses the intended prosody from the particular prosodic vertical <NUM>. The speech synthesizer <NUM> may then generate the intermediate synthesized speech representation <NUM> from the intermediate output signal <NUM>, and provide the intermediate synthesized speech representation <NUM> to the second TTS model <NUM>. The second TTS system <NUM> is configured to use the second TTS model <NUM> for transferring, or reproducing, the intended prosody captured by the intermediate synthesized speech representation <NUM> into the output audio signal <NUM> of expressive speech to convey the input text utterance <NUM> as a spoken representation having the intended prosody in the target voice. The second TTS model <NUM> may receive a speaker embedding Z that includes the speaker characteristics of the target voice. In some examples, the second TTS model <NUM> receives the intermediate output signal <NUM> (e.g., mel-frequencey spectrograms) produced by the first TTS model <NUM> in addition to, or in lieu of, the intermediate synthesized speech representation <NUM> for transferring the intended prosody into the output audio signal <NUM> of expressive speech. Since the input text utterance <NUM> has no way of conveying context, semantics, and pragmatics to guide the intended prosody of the synthesized speech <NUM>, the prosody transfer system <NUM> may predict the prosodic representation <NUM> for the input text utterance <NUM> by conditioning the second TTS model <NUM> on linguistic features extracted from the text utterance <NUM> and using a fixed-length utterance embedding <NUM> as a latent variable representing the intended prosody for the text utterance <NUM>. Described in greater detail below, the intermediate synthesized speech representation <NUM> produced by the first TTS system <NUM> is tasked with only capturing/possessing the intended prosody from the particular prosodic vertical <NUM> so that the second TTS model <NUM> can reproduce the intended prosody for the text utterance <NUM> by encoding the intermediate synthesized speech representation <NUM> into the utterance embedding <NUM>. The computing system <NUM> may include a distributed system (e.g., cloud computing environment). The synthesizer <NUM> may include a vocoder <NUM>.

In some implementations, the system <NUM> trains multiple prosody transfer systems <NUM>, 200A-N each configured to transfer a different respective intended prosody from a corresponding prosodic vertical <NUM>, 20A-N into expressive speech <NUM> in a target voice. For example, each of the different prosodic verticals <NUM> may include utterances spoken by a variety of different humans that have different voice characteristics (e.g., male/female, language, accent, etc.) that possess a same prosodic style associated with the corresponding prosodic vertical <NUM>. For instance, prosodic vertical 20A may correspond to utterances of human speech associated with news reading (e.g., newscasters), prosodic vertical 20B may correspond to utterances of human speech associated with sports commentators, and prosodic vertical 20N may correspond to utterances of human speech associated with educational lecturers. While the sports commentator vertical 20B could broadly contain utterances sampled from speakers commentating on multitude of different sports, each specific prosodic vertical <NUM> can convey an intended prosody sampled from a more narrow set of utterances. For instance, a multitude of different sports commentator prosodic verticals <NUM> could exist where each corresponds to utterances of human speech associated with a particular sport. This could be advantageous since prosodic style may vary between utterances spoken by sports commentators for the sport of curling compared to that of sports commentators for the sport of rugby.

With continued reference to <FIG>, for each prosodic vertical <NUM>, 20A-N, the computing device <NUM> (e.g., data processing hardware <NUM>) receives corresponding training data <NUM> including a plurality of training audio signals <NUM> and corresponding transcripts <NUM>. Each training audio signal <NUM> includes an utterance of human speech having the intended prosody spoken by a corresponding speaker in the prosodic vertical (interchangeably referred to as 'prosodic domain') associated with the intended prosody. Each transcript <NUM> includes a textual representation of the corresponding training audio signal <NUM>. For each corresponding transcript <NUM> of the training data <NUM>, the computing device <NUM> trains the first TTS model <NUM> to generate a corresponding reference audio signal 202T that includes a training synthesized speech representation that captures the intended prosody of the corresponding utterance <NUM> of human speech. Thereafter, the computing device <NUM> trains the second TTS model <NUM> of the second TTS system <NUM>. More specifically, training the second TTS model <NUM> may include, for each corresponding transcript <NUM> of the training data <NUM>, training both an encoder portion <NUM> and a decoder portion <NUM> of the second TTS model <NUM>. Training the encoder portion <NUM> includes encoding the corresponding training synthesized speech representation 202T (e.g., either an audio waveform or mel-frequency spectrograms) into a corresponding utterance embedding <NUM> representing the intended prosody captured by the training synthesized speech representation 202T. Notably, the first TTS system <NUM>, and more particularly parameters of the first TTS model <NUM>, are optimized to only produce synthesized speech representations 202T that accurately capture the prosody of the ground-truth utterance <NUM> of human speech, and thus, are permitted to include an intermediate voice that lacks sufficient voice characteristics and have reduced audio quality (e.g., contain acoustic artifacts).

Training the decoder portion <NUM> includes using the corresponding transcript <NUM> of the training data <NUM> to decode the utterance embedding <NUM> encoded by the encoder portion <NUM> into a predicted output audio signal <NUM> of expressive speech having the intended prosody. That is, the predicted output audio signal <NUM> is associated with a prosodic representation <NUM> that reproduces the intended prosody captured by the training intermediate synthesized speech representation 202T from the corresponding ground-truth utterance <NUM> of human speech. The decoder portion <NUM> may be further trained to learn speaker characteristics of a particular target voice so that the audio signal <NUM> of expressive speech has the intended prosody and the speaker characteristics of the target voice. In some examples, the first TTS model <NUM> and the second TTS model <NUM> are trained jointly. In other examples, the first TTS model <NUM> and the second TTS model <NUM> are trained separately.

Lastly, for each training audio signal <NUM> and corresponding transcript <NUM> in the training data <NUM> for the corresponding prosodic vertical <NUM>, the computing device <NUM> (e.g., data processing hardware <NUM>) generates gradients/losses between the predicted output audio signal <NUM> and the corresponding reference audio signal 202T and back-propagates the gradients/losses through the second TTS model <NUM>. Accordingly, the computing device <NUM> may train a corresponding prosodic transfer system <NUM>, 200A-N for each of a multitude of different prosodic verticals <NUM>, 20A-N, such that each prosodic transfer system <NUM> is configured to synthesize a text utterance <NUM> into expressive speech <NUM> in a target voice and having a prosodic representation <NUM> that represents an intended prosody associated with the corresponding particular prosodic vertical <NUM>. For instance, applying the example above, a trained prosodic transfer system 200A may synthesize expressive speech <NUM> in the target voice with an intended prosody associated with the news reader prosodic vertical 20A, a trained prosodic transfer system 200B may synthesize expressive speech in the target voice with an intended prosody associated with the sports commentators prosodic vertical 20B, and a trained prosodic transfer system 200N may synthesize expressive speech in the target voice with an intended prosody associated with the educational lecturers prosodic vertical 20N. The computing device <NUM> may store each trained prosodic transfer system <NUM> on data storage <NUM> (e.g., memory hardware <NUM>) for later use during inference.

During inference, the computing device <NUM> may use a trained prosodic transfer system <NUM> to synthesize a text utterance <NUM> into expressive speech <NUM> in a target voice and having a prosodic representation <NUM> that represents an intended prosody associated with a particular prosodic vertical <NUM>. The prosodic representation <NUM> may correspond to predicted prosodic features of pitch, energy, and duration of each phoneme. Namely, during a first level/stage, the trained prosody transfer system <NUM> uses the first TTS model <NUM> to generate an intermediate synthesized speech representation <NUM> for the input text utterance <NUM>, whereby the intermediate synthesized speech representation <NUM> possesses the intended prosody and is permitted to have a reduced audio quality and lack speaker characteristics. During a second level/stage, the trained prosody transfer system <NUM> provides the intermediate synthesized speech representation <NUM> to the second TTS model <NUM>. Here, the encoder portion <NUM> of the second TTS model <NUM> is configured to encode the intermediate synthesized speech representation <NUM> into an utterance embedding <NUM> that specifies the intended prosody, while the decoder portion <NUM> of the second TTS model <NUM> is configured to process the input text utterance <NUM> and the utterance embedding <NUM> to generate the output audio signal <NUM> of expressive speech. The output audio signal <NUM> has the intended prosody specified by the utterance embedding <NUM> and speaker characteristics of the target voice. The decoder portion <NUM> may receive a speaker embedding Z associated with the target voice that conveys the speaker characteristics (e.g., accent, male/female, and accent). In the example shown, the speech synthesizer <NUM> uses the output audio signal <NUM> to produce synthesized speech <NUM> from the text utterance <NUM> and having the intended prosody in the target voice.

<FIG> provides a schematic view of a prosody transfer system <NUM>, 200a in which the second TTS model 222a at the second TTS system <NUM> corresponds to a variational autoencoder (VAE)-based second TTS model 222a. More specifically, the second TTS model 222a may provide a hierarchical linguistic structure for a clockwork hierarchical variational autoencoder (CHiVE). However, the second TTS model 222a may include other types of VAEs. In the example shown, the first TTS system <NUM> receives, as input, a text utterance <NUM> and optional other inputs <NUM>, that may include speaker characteristics (e.g., speaker embedding Z) of the target voice. The other inputs <NUM> may additionally or alternatively include one or more of a language identifier, text normalization, or a prosodic vertical identifier of the corresponding prosodic domain. Using the input text utterance and the optional other inputs <NUM>, the first TTS model <NUM> generates an intermediate output audio signal <NUM> that may include a sequence of mel-frequency spectrograms inherently possessing the intended prosody for the input text utterance <NUM>. The first TTS system <NUM> may synthesize (e.g., using a speech synthesizer <NUM>) the intermediate output audio signal <NUM> into the intermediate synthesized speech representation <NUM>. As set forth above, the intermediate synthesized speech representation <NUM> is configured to accurately capture the intended prosody, and is permitted to include reduced audio quality and lack speaker characteristics for recognizing the target voice. Stated differently, the intermediate synthesized speech representation <NUM> may not necessarily be suitable for human listening, but rather, serves as a carrier of prosodic information conveying the intended prosody for use by the second TTS model <NUM> to reproduce and incorporate into expressive speech in the target voice.

An extractor <NUM> may then sample, from the intermediate synthesized speech representation <NUM>, a sequence of fixed-length reference frames <NUM> providing prosodic features that represent the intended prosody possessed by the intermediate synthesized speech representation <NUM>. The encoder portion <NUM>, 300a of the VAE-based second TTS model 222a is configured to encode the sequence of fixed-length reference frames <NUM> into the utterance embedding <NUM> that specifies the intended prosody. The prosodic features that represent the intended prosody possessed by the intermediate synthesized speech representation <NUM> may include duration, pitch contour, energy contour, and/or mel-frequency spectrogram contour.

With continued reference to <FIG>, the decoder portion <NUM>, 400a of the CHiVE-based second TTS model 222a is configured to process the input text utterance <NUM> and the utterance embedding <NUM> to generate the output audio signal <NUM> by decoding, using the input text utterance <NUM>, the corresponding utterance embedding <NUM> into a sequence of fixed-length predicted frames <NUM> providing the prosodic representation <NUM> of the input text utterance <NUM>.

<FIG> and <FIG> show the hierarchical linguistic structure for the CHiVE-based second TTS model 222a of <FIG> for providing a controllable model of prosody transfer. The model 222a may jointly predict, for each syllable of given input text <NUM>, a duration of the syllable and pitch (F0) and energy (C0) contours for the syllable without relying on any unique mappings from the given input text or other linguistic specification to produce synthesized speech <NUM> having an intended prosody in the target voice. The second TTS model 222a includes the encoder portion 300a (<FIG> and <FIG>) that encodes the plurality of fixed-length reference frames <NUM> sampled from the intermediate synthesized speech representation <NUM> (or from the intermediate output) into the fixed-length utterance embedding <NUM>, and the decoder portion 400a (<FIG> and <FIG>) that learns how to decode the fixed-length utterance embedding <NUM>. The decoder portion 400a may decode the fixed-length utterance embedding <NUM> into the output audio signal <NUM> of expressive speech that may include a plurality of fixed-length predicted frames <NUM> (e.g., to predict pitch (F0), energy (C0), or spectral characteristics (M0) for the utterance embedding <NUM>). As will become apparent, the second TTS model 222a is trained so that the number of predicted frames <NUM> output from the decoder portion 400a is equal to the number of reference frames <NUM> input to the encoder portion 300a. Moreover, the second TTS model 222a is trained so that prosody information associated with the reference and predicted frames <NUM>, <NUM> substantially match one another.

Referring to <FIG> and <FIG>, the encoder portion 300a receives the sequence of fixed-length reference frames <NUM> sampled from the intermediate synthesized speech representation <NUM> output from the first TTS system <NUM>. The intermediate synthesized speech representation <NUM> captures the intended prosody for the input text utterance <NUM>. The reference frames <NUM> may each include a duration of <NUM> milliseconds (ms) and represent one of a contour of pitch (F0) or a contour of energy (C0) (and/or contour of spectral characteristics (M0)) for the intermediate synthesized speech representation <NUM>. In parallel, the encoder portion 300a may also receive a second sequence of reference frames <NUM> each including a duration of <NUM> and representing the other one of the contour of pitch (F0) or the contour of energy (C0) (and/or contour of spectral characteristics (M0)) for the intermediate synthesized speech representation <NUM>. Accordingly, the sequence reference frames <NUM> sampled from the intermediate synthesized speech representation <NUM> provide a duration, pitch contour, energy contour, and/or spectral characteristics contour to represent the intended prosody captured by the intermediate synthesized speech representation <NUM>. The length or duration of the intermediate synthesized speech representation <NUM> correlates to a sum of the total number of reference frames <NUM>.

The encoder portion 300a includes hierarchical levels of reference frames <NUM>, phonemes <NUM>, 321a, syllables <NUM>, 330a, words <NUM>, 340a, and sentences <NUM>, 350a for the intermediate synthesized speech representation <NUM> that clock relative to one another. For instance, the level associated with the sequence of reference frames <NUM> clocks faster than the next level associated with the sequence of phonemes <NUM>. Similarly, the level associated with the sequence of syllables <NUM> clocks slower than the level associated with the sequence of phonemes <NUM> and faster than the level associated with the sequence of words <NUM>. Accordingly, the slower clocking layers receive, as input, an output from faster clocking layers so that the output after the final clock (i.e., state) of a faster layer is taken as the input to the corresponding slower layer to essentially provide a sequence-to-sequence encoder. In the examples shown, the hierarchical levels include Long Short-Term Memory (LSTM) levels.

In the example shown, the intermediate synthesized speech representation <NUM> includes one sentence <NUM>, 350A with three words <NUM>, 340A-C. The first word <NUM>, 340A includes two syllables <NUM>, 330Aa-Ab. The second word <NUM>, 340B includes one syllable <NUM>, 330Ba. The third word <NUM>, 340a includes two syllables <NUM>, 330Ca-Cb. The first syllable <NUM>, 330Aa of the first word <NUM>, 340A includes two phonemes <NUM>, 321Aa1-Aa2. The second syllable <NUM>, 330Ab of the first word <NUM>, 340A includes one phoneme <NUM>, 321Ab1. The first syllable <NUM>, 330Ba of the second word <NUM>, 340B includes three phonemes <NUM>, 321Ba1-Ba3. The first syllable <NUM>, 330Ca of the third word <NUM>, 340C includes one phoneme <NUM>, 321Ca1. The second syllable <NUM>, 330Cb of the third word <NUM>, 340C includes two phonemes <NUM>, 321Cb1-Cb2.

In some implementations, the encoder portion 300a first encodes the sequence of reference frames <NUM> into frame-based syllable embeddings <NUM>, 332Aa-Cb. Each frame-based syllable embedding <NUM> may indicate reference prosodic features represented as a numerical vector indicative of a duration, pitch (F0), and/or energy (C0) associated with the corresponding syllable <NUM>. In some implementations, the reference frames <NUM> define a sequence of phonemes 321Aa1-321Cb2. Here, instead of encoding a subset of reference frames <NUM> into one or more phonemes <NUM>, the encoder portion 300a instead accounts for the phonemes <NUM> by encoding phone level linguistic features <NUM>, 322Aa1-Cb2 into phone feature-based syllable embeddings <NUM>, 334Aa-Cb. Each phoneme-level linguistic feature <NUM> may indicate a position of the phoneme, while each phoneme feature-based syllable embedding <NUM> include a vector indicating the position of each phoneme within the corresponding syllable <NUM> as well as the number of phonemes <NUM> within the corresponding syllable <NUM>. For each syllable <NUM>, the respective syllable embeddings <NUM>, <NUM> may be concatenated and encoded with respective syllable-level linguistic features <NUM>, 336Aa-Cb for the corresponding syllable <NUM>. Moreover, each syllable embedding <NUM>, <NUM> is indicative of a corresponding state for the level of syllables <NUM>.

With continued reference to <FIG>, the blocks in the hierarchical layers that include a diagonal hatching pattern correspond to linguistic features (except for the word level <NUM>) for a particular level of the hierarchy. The hatching pattern at the word-level <NUM> include word embeddings <NUM> extracted as linguistic features from the input text utterance <NUM> or WP embeddings <NUM> output from the BERT model <NUM> based on word units <NUM> obtained from the transcript <NUM>. Since the recurrent neural network (RNN) portion of the encoder 300a has no notion of wordpiecies, the WP embedding <NUM> corresponding to the first wordpiece of each word may be selected to represent the word which may contain one or more syllables <NUM>. With the frame-based syllable embeddings <NUM> and the phone feature-based syllable embeddings <NUM>, the encoder portion 300a concatenates and encodes these syllable embeddings <NUM>, <NUM> with other linguistic features <NUM>, <NUM>, <NUM> (or WP embeddings <NUM>). For example, the encoder portion 300a encodes the concatenated syllable embeddings <NUM>, <NUM> with syllable-level linguistic features <NUM>, 336Aa-Cb, word-level linguistic features (or WP embeddings <NUM>, 342A-C output from the BERT model <NUM>), and/or sentence-level linguistic features <NUM>, 352A. By encoding the syllable embeddings <NUM>, <NUM> with the linguistic features <NUM>, <NUM>, <NUM> (or WP embeddings <NUM>), the encoder portion 300a generates an utterance embedding <NUM> for the intermediate synthesized speech representation <NUM>. The utterance embedding <NUM> may be stored in the data storage <NUM> (<FIG>) along with the input text utterance <NUM> (e.g., textual representation) of the intermediate synthesized speech representation <NUM>. From the input text utterance <NUM>, the linguistic features <NUM>, <NUM>, <NUM>, <NUM> may be extracted and stored for use in conditioning the training of the hierarchical linguistic structure. The linguistic features (e.g., linguistic features <NUM>, <NUM>, <NUM>, <NUM>) may include, without limitation, individual sounds for each phoneme and/or the position of each phoneme in a syllable, whether each syllable is stressed or un-stressed, syntactic information for each word, and whether the utterance is a question or phrase and/or a gender of a speaker of the utterance. As used herein, any reference of word-level linguistic features <NUM> with respect to the encoder and decoder portions 300a, 400a of the VAE-based second TTS model 222a can be replaced with WP embeddings from the BERT model <NUM>.

In the example of <FIG>, encoding blocks <NUM>, 322Aa-Cb are shown to depict the encoding between the linguistic features <NUM>, <NUM>, <NUM> and the syllable embeddings <NUM>, <NUM>. Here, the blocks <NUM> are sequence encoded at a syllable rate to generate the utterance embedding <NUM>. As an illustration, the first block 322Aa is fed as an input into a second block 322Ab. The second block 322Ab is fed as an input into a third block 322Ba. The third block 322Ca is fed as an input into the fourth block 322Ca. The fourth block 322Ca is fed into the fifth block 322Cb. In some configurations, the utterance embedding <NUM> includes a mean µ and a standard deviation σ for the intermediate synthesized speech representation <NUM> where the mean µ and the standard deviation σ are with respect to the training data of multiple intermediate synthesized speech representations <NUM>.

In some implementations, each syllable <NUM> receives, as input, a corresponding encoding of a subset of reference frames <NUM> and includes a duration equal to the number of reference frames <NUM> in the encoded subset. In the example shown, the first seven fixed-length reference frames <NUM> are encoded into syllable 330Aa; the next four fixed-length reference frames <NUM> are encoded into syllable 330Ab; the next eleven fixed-length reference frames <NUM> are encoded into syllable 330Ba; the next three fixed-length reference frames <NUM> are encoded into syllable 330Ca; and the final six fixed-length reference frames <NUM> are encoded into syllable 330Cb. Thus, each syllable <NUM> in the sequence of syllables <NUM> may include a corresponding duration based on the number of reference frames <NUM> encoded into the syllable <NUM> and corresponding pitch and/or energy contours. For instance, syllable 330Aa includes a duration equal to <NUM> (i.e., seven reference frames <NUM> each having the fixed-length of five milliseconds) and syllable 330Ab includes a duration equal to <NUM> (i.e., four reference frames <NUM> each having the fixed-length of five milliseconds). Thus, the level of reference frames <NUM> clocks a total of ten times for a single clocking between the syllable 330Aa and the next syllable 330Ab at the level of syllables <NUM>. The duration of the syllables <NUM> may indicate timing of the syllables <NUM> and pauses in between adjacent syllables <NUM>.

In some examples, the utterance embedding <NUM> generated by the encoder portion 300a is a fixed-length utterance embedding <NUM> that includes a numerical vector representing a prosody of the intermediate synthesized speech representation <NUM>. In some examples, the fixed-length utterance embedding <NUM> includes a numerical vector having a value equal to "<NUM>" or "<NUM>".

Referring now to <FIG> and <FIG>, the decoder portion 400a of the VAE-based second TTS model 222a is configured to produce a plurality of fixed-length syllable embeddings <NUM> by initially decoding the fixed-length utterance embedding <NUM> that specifies the intended prosody for the input text utterance <NUM>. More specifically, the utterance embedding <NUM> represents the intended prosody possessed by the intermediate synthesized speech representation <NUM> output from the first TTS system <NUM> for the input text utterance <NUM>. Thus, the decoder portion 400a is configured to back-propagate the utterance embedding <NUM> to generate the plurality of fixed-length predicted frames <NUM> that closely match the plurality of fixed-length reference frames <NUM>. For instance, fixed-length predicted frames <NUM> for both pitch (F0) and energy (C0) may be generated in parallel to represent the intended prosody (e.g., predicted prosody) that substantially matches the intended prosody possessed by the training data. In some examples, the speech synthesizer <NUM> uses the fixed-length predicted frames <NUM> to produce synthesized speech <NUM> with the intended prosody and in the target voice based on the fixed-length utterance embedding <NUM>. For instance, a unit selection module or a WaveNet module of the speech synthesizer <NUM> may use the frames <NUM> to produce the synthesized speech <NUM> having the intended prosody.

In the example shown, the decoder portion 400a decodes the utterance embedding <NUM> (e.g., numerical value of "<NUM>") received from the encoder portion 300a (<FIG> and <FIG>) into hierarchical levels of words <NUM>, 340b, syllables <NUM>, 330b, phonemes <NUM>, 321b, and the fixed-length predicted frames <NUM>. Specifically, the fixed-length utterance embedding <NUM> corresponds to a variational layer of hierarchical input data for the decoder portion 400a and each of the stacked hierarchical levels include Long Short-Term Memory (LSTM) processing cells variably clocked to a length of the hierarchical input data. For instance, the syllable level <NUM> clocks faster than the word level <NUM> and slower than the phoneme level <NUM>. The rectangular blocks in each level correspond to LSTM processing cells for respective words, syllables, phonemes, or frames. Advantageously, the VAE-based second TTS model 222a gives the LSTM processing cells of the word level <NUM> memory over the last <NUM> words, gives the LSTM cells of the syllable level <NUM> memory over the last <NUM> syllables, gives the LSTM cells of the phoneme level <NUM> memory over the last <NUM> phonemes, and gives the LSTM cells of the fixed-length pitch and/or energy frames <NUM> memory over the last <NUM> fixed-length frames <NUM>. When the fixed-length frames <NUM> include a duration (e.g., frame rate) of five milliseconds each, the corresponding LSTM processing cells provide memory over the last <NUM> milliseconds (e.g., a half second).

In the example shown, the decoder portion 400a of the hierarchical linguistic structure simply back-propagates the fixed-length utterance embedding <NUM> encoded by the encoder portion 300a into the sequence of three words 340A-340C, the sequence of five syllables 330Aa-330Cb, and the sequence of nine phonemes 321Aa1-321Cb2 to generate the sequence of predicted fixed-length frames <NUM>. The decoder portion 400a is conditioned upon linguistic features of the input text utterance <NUM>. By contrast to the encoder portion 300a of <FIG> where outputs from faster clocking layers are received as inputs by slower clocking layers, the decoder portion 400a includes outputs from slower clocking layers feeding faster clocking layers such that the output of a slower clocking layer is distributed to the input of the faster clocking layer at each clock cycle with a timing signal appended thereto.

Referring to <FIG>, <FIG>, and <FIG>, in some implementations, the hierarchical linguistic structure for the clockwork hierarchical variational autoencoder 222a is adapted to provide a controllable model for predicting mel spectral information for an input text utterance <NUM>, while at the same time effectively controlling the prosody implicitly represented in the mel spectral information. Specifically, the second TTS model 222a may predict a mel-frequency spectrogram <NUM> for the input text utterance (simply referred to as "input text" <NUM>) and provide the mel-frequency spectrogram <NUM> as input to a vocoder network <NUM> of the speech synthesizer for conversion into a time-domain audio waveform. A time-domain audio waveform includes an audio waveform that defines an amplitude of an audio signal over time. As will become apparent, the speech synthesizer <NUM> can generate synthesized speech <NUM> from input text <NUM> using the autoencoder 222a trained on sample input text and corresponding mel-frequency spectrograms <NUM> output from the first TTS model <NUM> alone. That is, the VAE-based second TTS model 222a does not receive complex linguistic and acoustic features that require significant domain expertise to produce, but rather is able to convert input text <NUM> to mel-frequency spectrograms <NUM> using an end-to-end deep neural network. The vocoder network <NUM>, i.e., neural vocoder, is separately trained and conditioned on mel-frequency spectrograms for conversion into time-domain audio waveforms.

A mel-frequency spectrogram includes a frequency-domain representation of sound. Mel-frequency spectrograms emphasize lower frequencies, which are critical to speech intelligibility, while de-emphasizing high frequency, which are dominated by fricatives and other noise bursts and generally do not need to be modeled with high fidelity. The vocoder network <NUM> can be any network that is configured to receive mel-frequency spectrograms and generate audio output samples based on the mel-frequency spectrograms. For example, the vocoder network <NUM> can be, or can be based on the parallel feed-forward neural network described in van den Oord, Parallel WaveNet: Fast High-Fidelity Speech Synthesis, available at https://arxiv. org/pdf/<NUM>. Alternatively, the vocoder network <NUM> can be an autoregressive neural network.

As described above with reference to <FIG>, <FIG>, and <FIG>, the VAE-based second TTS model 222a includes the encoder portion 300a and the decoder portion 400a. The encoder portion 300a is configured to encode a plurality of fixed-length reference mel-frequency spectrogram frames <NUM> sampled/extracted from the intermediate synthesized speech representation <NUM> into the utterance embedding <NUM>. The decoder portion 400a is configured to learn how to decode the utterance embedding into a plurality of fixed-length predicted mel-frequency spectrogram frames 280M0. The VAE-based TTS model 222a may be trained so that the number of predicted mel-frequency spectrogram frames <NUM> output from the decoder portion 400a is equal to the number of reference mel-frequency spectrogram frames <NUM> input to the encoder portion 300a. Moreover, the VAE-based TTS model 222a is trained so that prosody information associated with the reference and predicted mel-frequency spectrogram frames <NUM>, <NUM> substantially match one another. The predicted mel-frequency spectrogram frames <NUM> may implicitly provide a prosodic representation of the intermediate synthesized speech representation <NUM>. The reference mel-frequency spectrogram frames <NUM> may be sampled from the intermediate output audio signal <NUM> output from the first TTS system <NUM> in addition to, or in lieu of, the intermediate synthesized speech representation <NUM>. Additional details of the VAE-based second TTS model 222a are described with reference to <CIT>.

<FIG> provides a schematic view of a prosody transfer system <NUM>, 200b in which the second TTS model <NUM> at the second TTS system <NUM> corresponds to a sequence-to-sequence feature prediction network-based second TTS model 222b (hereinafter S2S-based second TTS model 222b). In the example shown, the first TTS system <NUM> receives, as input, a text utterance <NUM> and optional other inputs <NUM>, that may include, speaker characteristics (e.g., speaker embedding Z) of the target voice. The other inputs <NUM> may additionally or alternatively include one or more of a language identifier, text normalization, or a prosodic vertical identifier of the corresponding prosodic domain. Using the input text utterance and the optional other inputs <NUM>, the first TTS model <NUM> generates an intermediate output audio signal <NUM> that may include a sequence of mel-frequency spectrograms possessing the intended prosody for the input text utterance <NUM>. The first TTS system <NUM> may synthesize (e.g., using a speech synthesizer <NUM>) the intermediate output audio signal <NUM> into the intermediate synthesized speech representation <NUM>. As set forth above, the intermediate synthesized speech representation <NUM> is configured to accurately capture the intended prosody, and is permitted to include reduced audio quality and lack speaker characteristics for recognizing the target voice. Stated differently, the intermediate synthesized speech speech representation <NUM> is not suitable for human listening, but rather, serves as a carrier of prosodic information conveying the intended prosody for use by the second TTS model <NUM> to reproduce and incorporate into expressive speech in the target voice.

The encoder portion <NUM>, 300b of the S2S-based second TTS model 222b is configured to encode the intermediate synthesized speech representation <NUM> (or the intermediate output audio signal <NUM>) into the utterance embedding <NUM> that specifies the intended prosody. The intermediate synthesized speech representation <NUM> (or the intermediate output audio signal <NUM>) fed to the encoder portion 300b may implicitly represent the intended prosody for the input text utterance <NUM>. In some implementations, the encoder portion 300b corresponds to a variational autoencoder that encodes the intended prosody as latent factors into the utterance embedding <NUM>. In these implementations, the utterance embedding <NUM> may correspond to a latent embedding. These latent factors are generally not represented in conditioning inputs to the decoder portion 400b, whereby the conditioning inputs may include an input text utterance <NUM> and other inputs <NUM> such as a speaker embedding <NUM> associated with speaker characteristics of the target voice, a language embedding associated with a native language of the input text utterance <NUM>, and a prosodic vertical identifier identifying the particular prosodic vertical <NUM> (<FIG>) conveying the intended prosody. Accordingly, the encoder portion 300b passes the utterance embedding <NUM> to the decoder 400b.

Referring now to <FIG> and <FIG>, the decoder portion 400a of the S2S-based second TTS model 222b may include an architecture having a pre-net <NUM>, a Long Short-Term Memory (LSTM) subnetwork <NUM>, a linear projection <NUM>, and a convolutional post-net <NUM>. The pre-net <NUM>, through which a mel-frequency spectrogram prediction for a previous time step passes, may include two fully-connected layers of hidden ReLUs. The pre-net <NUM> acts as an information bottleneck for learning attention to increase convergence speed and to improve generalization capability of the speech synthesis system during training. In order to introduce output variation at inference time, dropout with probability <NUM> may be applied to layers in the pre-net.

The LSTM subnetwork <NUM> may include two or more LSTM layers. At each time step, the LSTM subnetwork <NUM> receives a concatenation of the output of the pre-net <NUM>, the utterance embedding <NUM>, and a portion of the text utterance <NUM> for the time step. The LSTM layers may be regularized using zoneout with probability of, for example, <NUM>. The linear projection <NUM> receives as input the output of the LSTM subnetwork <NUM> and produces a prediction of a mel-frequency spectrogram 118P.

The convolutional post-net <NUM> with one or more convolutional layers processes the predicted mel-frequency spectrogram 118P for the time step to predict a residual <NUM> to add to the predicted mel-frequency spectrogram 118P at adder <NUM>. This improves the overall reconstruction. Each convolutional layer except for the final convolutional layer may be followed by batch normalization and hyperbolic tangent (TanH) activations. The convolutional layers are regularized using dropout with a probability of, for example, <NUM>. The residual <NUM> is added to the predicted mel-frequency spectrogram 118P generated by the linear projection <NUM>, and the sum (i.e., the mel-frequency spectrogram <NUM>) may be provided to the speech synthesizer <NUM>. In some implementations, in parallel to the decoder portion 400b predicting mel-frequency spectrograms <NUM> for each time step, a concatenation of the output of the LSTM subnetwork <NUM>, the utterance embedding <NUM>, and the portion of the text utterance <NUM> (e.g., a character embedding generated by a text encoder (not shown)) is projected to a scalar and passed through a sigmoid activation to predict the probability that the output sequence of mel frequency spectrograms <NUM> has completed. The output sequence mel-frequency spectrograms <NUM> corresponds to the output audio signal <NUM> of expressive speech for the input text utterance <NUM> and includes the intended prosody and speaker characteristics associated with the target voice.

This "stop token" prediction is used during inference to allow the model 222b to dynamically determine when to terminate generation instead of always generating for a fixed duration. When the stop token indicates that generation has terminated, i.e., when the stop token probability exceeds a threshold value, the decoder portion 400b stops predicting mel-frequency spectrograms 118P and returns the mel-frequency spectrograms predicted up to that point as the output audio signal <NUM> of expressive speech. Alternatively, the decoder portion 400b may always generate mel-frequency spectrograms <NUM> of the same length (e.g., <NUM> seconds). In some implementations, the speech synthesizer is a Griffin-Lim synthesizer. In some other implementations, the speech synthesizer includes the vocoder <NUM>. For instance, the speech synthesizer <NUM> may include a WaveRNN vocoder <NUM>. Here, the WaveRNN vocoder <NUM> may generate <NUM>-bit signals sampled at <NUM> conditioned on spectrograms <NUM> predicted by the TTS model 222b. In some other implementations, the waveform synthesizer is a trainable spectrogram to waveform inverter. After the waveform synthesizer <NUM> generates the waveform, an audio output system can generate the speech <NUM> using a waveform and provide the generated speech <NUM> for playback, e.g., on a user device, or provide the generated waveform to another system to allow the other system to generate and play back the speech <NUM>. In some examples, a WaveNet neural vocoder <NUM> replaces the waveform synthesizer <NUM>. A WaveNet neural vocoder may provide different audio fidelity of synthesized speech in comparison to synthesized speech produced by the waveform synthesizer <NUM>. Thus, in some examples, the first TTS system <NUM> may employ a conventional waveform synthesizer <NUM> to generate the intermediate synthesized speech representation <NUM> in the intermediate voice of reduced quality, but accurately possessing an intended prosody, while the second TTS system <NUM> may transfer the intended prosody from the intermediate synthesized speech representation <NUM> into the synthesized speech in the target voice produced by a WaveNet neural vocoder <NUM>.

In some implementations, the decoder portion 400b includes an attention-based sequence-to-sequence model configured to generate a sequence of output log-mel spectrogram frames, e.g., output mel spectrogram <NUM>, based on the input text utterance <NUM> and additional inputs such as a speaker embedding Z providing speaker characteristics associated with the target voice. For instance, the decoder portion 400b may be based on the Tacotron <NUM> model (See "<NPL>). Thus, the S2S-based second TTS model 222b provides an enhanced, TTS model for transferring the intended prosody possessed in the intermediate synthesized speech representation <NUM> into the utterance embedding <NUM> and processing the utterance embedding <NUM> and the input text utterance <NUM> with additional inputs <NUM> (e.g. a speaker embedding z) to produce the output audio signal <NUM> of expressive speech having the intended prosody in the target voice. The additional inputs <NUM> such as the speaker embedding z, the language identifier, and the prosodic vertical identifier helps permit transfer of different voices across different languages for any intended prosody the system <NUM> is trained on.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> synthesizing an input text utterance into expressive speech having an intended prosody in a target voice. The data processing hardware <NUM> (<FIG>) may perform the operations for the method <NUM> by executing instructions stored on the memory hardware <NUM>. At operation <NUM>, the method <NUM> includes receiving an input text utterance <NUM> to be synthesized into expressive speech <NUM> having an intended prosody and a target voice. At operation <NUM>, the method <NUM> includes generating, using a first text-to-speech (TTS) model <NUM>, an intermediate synthesized speech representation <NUM> for the input text utterance <NUM>. Here, the intermediate synthesized speech representation <NUM> possesses the intended prosody. The intermediate synthesized speech representation <NUM> may include an audio waveform or a sequence of mel-frequency spectrograms that captures the intended prosody. Further, the intermediate synthesized speech representation <NUM> may include an intermediate voice that lacks the speaker characteristics of the target voice and includes undesirable acoustic artifacts. Thus, the intermediate synthesized speech representation <NUM> provides expressiveness, but may lack intelligibility and naturalness.

At operation <NUM>, the method <NUM> includes providing the intermediate synthesized speech representation to a second TTS model <NUM> that includes an encoder portion <NUM> and a decoder portion <NUM>. The encoder portion <NUM> is configured to encode the intermediate synthesized speech representation <NUM> into an utterance embedding <NUM> that specifies the intended prosody. The decoder portion <NUM> is configured to process the input text utterance <NUM> and the utterance embedding <NUM> to generate an output audio signal <NUM> of expressive speech <NUM>. Here, the output audio signal has the intended prosody specified by the utterance embedding <NUM> and speaker characteristics of the target voice.

In some examples, the method <NUM> also includes obtaining an additional input <NUM> of a speaker embedding, Z, representing the speaker characteristics of the target voice. In these examples, the decoder portion <NUM> is configured to process the input text utterance <NUM>, the utterance embedding <NUM>, and the speaker embedding, Z, to generate the output audio signal of expressive speech. The first TTS model <NUM> and the second TTS model <NUM> may each include the same or different types of neural network architectures.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory device; magnetic disks, e.g. internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>), an input text utterance (<NUM>) to be synthesized into expressive speech (<NUM>) having an intended prosody and a target voice;
generating, by the data processing hardware (<NUM>), using a first trained text-to-speech, TTS, model (<NUM>), an intermediate synthesized speech representation (<NUM>) for the input text utterance (<NUM>), the intermediate synthesized speech representation (<NUM>) possessing the intended prosody; and
providing, by the data processing hardware (<NUM>), the intermediate synthesized speech representation (<NUM>) to a second trained TTS model (<NUM>), the second trained TTS model (<NUM>) comprising:
an encoder portion (<NUM>) configured to encode the intermediate synthesized speech representation (<NUM>) into an utterance embedding (<NUM>) that specifies the intended prosody; and
a decoder portion (<NUM>) configured to process the input text utterance (<NUM>) and the utterance embedding (<NUM>) to generate an output audio signal (<NUM>) of expressive speech (<NUM>), the output audio signal (<NUM>) having the intended prosody specified by the utterance embedding (<NUM>) and speaker characteristics of the target voice.