Patent Description:
Speech synthesis systems use text-to-speech (TTS) models to generate speech from textual input. The generated/synthesized speech should accurately convey the message (intelligibility) while sounding like human speech (naturalness) with an intended prosody (expressiveness). While traditional concatenative and parametric synthesis models are capable of providing intelligible speech and recent advances in neural modeling of speech have significantly improved the naturalness of synthesized speech, most existing models are ineffective at modeling prosody, thereby causing synthesized speech used by important applications to lack expressiveness. For instance, it is desirable for applications such as conversational assistants and long-form readers to produce realistic speech by imputing prosody features not conveyed in textual input, such as intonation, stress, and rhythm and style. For example, a simple statement can be spoken in many different ways depending on whether the statement is a question, an answer to a question, there is uncertainty in the statement, or to convey any other meaning about the environment or context which is unspecified by the input text. "Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis", Wang et. A1 proposes "global style tokens" (GSTs), a bank of embeddings that are jointly trained within Tacotron, a state-of-the-art end-to-end speech synthesis system. The embeddings are trained with no explicit labels, yet learn to model a large range of acoustic expressiveness.

One aspect of the disclosure provides a method of representing an intended prosody in synthesized speech according to claim <NUM>. The method includes receiving, at data processing hardware, a text utterance having at least one word, and selecting, by the data processing hardware, an utterance embedding for the text utterance. Each word in the text utterance has at least one syllable and each syllable has at least one phoneme. The utterance embedding represents an intended prosody. For each syllable, using the selected utterance embedding, the method also includes: predicting, by the data processing hardware, a duration of the syllable by encoding linguistic features of each phoneme of the syllable with a corresponding prosodic syllable embedding for the syllable; predicting, by the data processing hardware, a pitch contour of the syllable based on the predicted duration for the syllable; and generating, by the data processing hardware, a plurality of fixed-length predicted pitch frames based on the predicted duration for the syllable. Each fixed-length predicted pitch frame represents part of the predicted pitch contour of the syllable.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, a network representing a hierarchical linguistic structure of the text utterance includes a first level including each syllable of the text utterance, a second level including each phoneme of the text utterance, and a third level including each fixed-length predicted pitch frame for each syllable of the text utterance. In these implementations, the first level of the network may include a long short-term memory (LSTM) processing cell representing each syllable of the text utterance, the second level of the network may include a LSTM processing cell representing each phoneme of the text utterance, and the third level of the network may include a LSTM processing cell representing each fixed-length predicted pitch frame. Here, the LSTM processing cells of the second level clock relative to and faster than the LSTM processing cells of the first level, while the LSTM processing cells of the third level clock relative to and faster than the LSTM processing cells of the second level.

In some examples, predicting the duration of the syllable includes: for each phoneme associated with the syllable, predicting a duration of the corresponding phoneme by encoding the linguistic features of the corresponding phoneme with the corresponding prosodic syllable embedding for the syllable; and determining the duration of the syllable by summing the predicted durations for each phoneme associated with the syllable. In these examples, predicting the pitch contour of the syllable based on the predicted duration for the syllable may include combining the corresponding prosodic syllable embedding for the syllable with each encoding of the corresponding prosodic syllable embedding and the phone-level linguistic features of each corresponding phoneme associated with the syllable.

In some implementations, the method also includes, for each syllable, using the selected utterance embedding: predicting, by the data processing hardware, an energy contour of each phoneme in the syllable based on a predicted duration for the phoneme; and for each phoneme associated with the syllable, generating, by the data processing hardware, a plurality of fixed-length predicted energy frames based on the predicted duration for the phoneme. Here, each fixed-length energy frame represents the predicted energy contour of the corresponding phoneme. In these implementations, a hierarchical linguistic structure represents the text utterance and the hierarchical linguistic structure includes a first level including each syllable of the text utterance, a second level including each phoneme of the text utterance, a third level including each fixed-length predicted pitch frame for each syllable of the text utterance, and a fourth level parallel to the third level and including each fixed-length predicted energy frame for each phoneme of the text utterance. The first level may include a long short-term memory (LSTM) processing cell representing each syllable of the text utterance, the second level may include a LSTM processing cell representing each phoneme of the text utterance, the third level may include a LSTM processing cell representing each fixed-length predicted pitch frame, and the fourth level may include a LSTM processing cell representing each fixed-length predicted energy frame. Here, the LSTM processing cells of the second level clock relative to and faster than the LSTM processing cells of the first level, the LSTM processing cells of the third level clock relative to and faster than the LSTM processing cells of the second level, and the LSTM processing cells of the fourth level clock at the same speed as the LSTM processing cells of the third level and clock relative to and faster than the LSTM processing cells of the second level.

In some examples, the third level of the hierarchical linguistic structure includes a feed-forward layer that predicts the predicted pitch frames for each syllable in a single pass and/or the fourth level of the hierarchical linguistic structure includes a feed-forward layer that predicts the predicted energy frames for each phoneme in a single pass. Moreover, the lengths of the fixed-length predicted energy frames and the fixed-length predicted pitch frames may be the same. Additionally or alternatively, a total number of fixed-length predicted energy frames generated for each phoneme of the received text utterance may be equal to a total number of the fixed-length predicted pitch frames generated for each syllable of the received text utterance.

In some implementations, the method also includes: receiving, by the data processing hardware, training data including a plurality of reference audio signals, each reference audio signal including a spoken utterance of human speech and having a corresponding prosody; and training, by the data processing hardware, a deep neural network for a prosody model by encoding each reference audio signal into a corresponding fixed-length utterance embedding representing the corresponding prosody of the reference audio signal. The utterance embedding may include a fixed-length numerical vector.

Another aspect of the disclosure provides a system for representing an intended prosody in synthesized speech according to claim <NUM>. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving a text utterance having at least one word, and selecting an utterance embedding for the text utterance. Each word in the text utterance has at least one syllable and each syllable has at least one phoneme. The utterance embedding represents an intended prosody. For each syllable, using the selected utterance embedding, the operations also include: predicting a duration of the syllable by encoding linguistic features of each phoneme of the syllable with a corresponding prosodic syllable embedding for the syllable; predicting a pitch contour of the syllable based on the predicted duration for the syllable; and generating a plurality of fixed-length predicted pitch frames based on the predicted duration for the syllable. Each fixed-length predicted pitch frame represents part of the predicted pitch contour of the syllable.

This aspect may include one or more of the following optional features. In some implementations, a network representing a hierarchical linguistic structure of the text utterance includes a first level including each syllable of the text utterance, a second level including each phoneme of the text utterance, and a third level including each fixed-length predicted pitch frame for each syllable of the text utterance. In these implementations, the first level of the network may include a long short-term memory (LSTM) processing cell representing each syllable of the text utterance, the second level of the network may include a LSTM processing cell representing each phoneme of the text utterance, and the third level of the network may include a LSTM processing cell representing each fixed-length predicted pitch frame. Here, the LSTM processing cells of the second level clock relative to and faster than the LSTM processing cells of the first level, while the LSTM processing cells of the third level clock relative to and faster than the LSTM processing cells of the second level.

In some implementations, the operations also include, for each syllable, using the selected utterance embedding: predicting an energy contour of each phoneme in the syllable based on a predicted duration for the phoneme; and for each phoneme associated with the syllable, generating a plurality of fixed-length predicted energy frames based on the predicted duration for the phoneme. Here, each fixed-length energy frame represents the predicted energy contour of the corresponding phoneme. In these implementations, a hierarchical linguistic structure represents the text utterance and the hierarchical linguistic structure includes a first level including each syllable of the text utterance, a second level including each phoneme of the text utterance, a third level including each fixed-length predicted pitch frame for each syllable of the text utterance, and a fourth level parallel to the third level and including each fixed-length predicted energy frame for each phoneme of the text utterance. The first level may include a long short-term memory (LSTM) processing cell representing each syllable of the text utterance, the second level may include a LSTM processing cell representing each phoneme of the text utterance, the third level may include a LSTM processing cell representing each fixed-length predicted pitch frame, and the fourth level may include a LSTM processing cell representing each fixed-length predicted energy frame. Here, the LSTM processing cells of the second level clock relative to and faster than the LSTM processing cells of the first level, the LSTM processing cells of the third level clock relative to and faster than the LSTM processing cells of the second level, and the LSTM processing cells of the fourth level clock at the same speed as the LSTM processing cells of the third level and clock relative to and faster than the LSTM processing cells of the second level.

In some implementations, the operations also includes: receiving training data including a plurality of reference audio signals, each reference audio signal including a spoken utterance of human speech and having a corresponding prosody; and training a deep neural network for a prosody model by encoding each reference audio signal into a corresponding fixed-length utterance embedding representing the corresponding prosody of the reference audio signal. The utterance embedding may include a fixed-length numerical vector.

Text-to-speech (TTS) models, often used by speech synthesis systems, are generally only given text inputs without any reference acoustic representation at runtime, and must impute many linguistic factors that are not provided by the text inputs in order to produce realistically sounding synthesized speech. A subset of these linguistic factors are collectively referred to as prosody and may include intonation (pitch variation), stress (stressed syllables vs. non-stressed syllables), duration of sounds, loudness, tone, rhythm, and style of the speech. Prosody may indicate the emotional state of the speech, the form of the speech (e.g., statement, question, command, etc.), the presence of irony or sarcasm of the speech, uncertainty in the knowledge of the speech, or other linguistic elements incapable of being encoded by grammar or vocabulary choice of the input text. Accordingly, a given text input that is associated with a high degree of prosodic variation can produce synthesized speech with local changes in pitch and speaking duration to convey different semantic meanings, and also with global changes in the overall pitch trajectory to convey different moods and emotions.

Neural network models provide potential for robustly synthesizing speech by predicting linguistic factors corresponding to prosody that are not provided by text inputs. As a result, a number of applications, such as audiobook narration, news readers, voice design software, and conversational assistants can produce realistically sounding synthesized speech that is not monotonous-sounding. Implementations herein are directed toward a neural network model that includes a variational autoencoder (VAE) having an encoder portion for encoding a reference audio signal corresponding to a spoken utterance into an utterance embedding that represents the prosody of the spoken utterance, and a decoder portion that decodes the utterance embedding to predict durations of phonemes and pitch and energy contours for each syllable.

The encoder portion may train utterance embeddings representing prosody by encoding numerous reference audio signals conditioned on linguistic features representing the utterances. The linguistic features may include, without limitation, individual sounds for each phoneme, whether each syllable is stressed or un-stressed, the type of each word (e.g., noun/adjective/verb) and/or the position of the word in the utterance, and whether the utterance is a question or phrase. Each utterance embedding is represented by a fixed-length numerical vector. In some implementations, the fixed-length numerical vector includes a value equal to <NUM>. However, other implementations may use fixed-length numerical vectors having values greater than or less than <NUM>. The decoder portion may decode a fixed-length utterance embedding into a sequence of phoneme durations via a first decoder and into a sequence of fixed-length frames (e.g., five millisecond) of pitch and energy using the phoneme durations. During training, the phoneme durations and fixed-length frames of pitch and energy predicted by the decoder portion closely match the phoneme durations and fixed-length frames of pitch and energy sampled from the reference audio signal associated with the fixed-length utterance embedding.

The VAE of the present disclosure includes a Clockwork Hierarchal Variational Autoencoder (CHiVE) that incorporates hierarchical stacked layers of long-short term-memory (LSTM) cells, with each layer of LSTM cells incorporating structure of the utterance such that one layer represents phonemes, a next layer represents syllables, and another layer represents words. Moreover, the hierarchy of stacked layers of LSTM cells are variably clocked to a length of hierarchical input data. For instance, if the input data contains a word of three syllables followed by a word of four syllables, then the syllable layer of the CHiVE would clock three times relative to a single clock of the word layer for the first input word, and then the syllable layer would clock four more times relative to a subsequent single clock of the word layer for the second word. Thus, rather than using frame-based techniques where memory associated with given LSTM cell is only effective for about a half second (i.e., <NUM> times steps with a five (<NUM>) millisecond frame rate), and thus, only providing the LSTM cell memory for two or three syllables of speech, the phoneme, word, and syllable layers of the CHiVE clock with phonemes, words, and syllables, respectively, giving the LSTM cells of the stacked layers memory over the last <NUM> words, syllables, or phonemes.

During inference, the CHiVE is configured to receive a text utterance and select an utterance embedding for the text utterance. The received text utterance has at least one word, each word has at least one syllable, and each syllable has at least one phoneme. Since the text utterance is missing context, semantic information, and pragmatic information to guide the appropriate prosody for producing synthesized speech from the utterance, the CHiVE uses that selected utterance embedding as the latent variable to represent an intended prosody. Thereafter, the CHiVE uses the selected utterance embedding to predict a duration of each syllable by encoding linguistic features of each phoneme contained in the syllable with a corresponding prosodic syllable embedding for the syllable, and predict a pitch of each syllable based on the predicted duration for the syllable. Lastly, the CHiVE is configured to generate a plurality of fixed-length pitch frames based on the predicted duration for each syllable such that each fixed-length pitch frame represents the predicted pitch of the syllable. The CHiVE may similarly predict energy (e.g., loudness) of each syllable based on the predicted duration for the syllable and generate a plurality of fixed-length energy frames each representing the predicted energy of the syllable. The fixed-length pitch and/or energy frames may be provided to a unit-selection model or wave-net model of a TTS system to produce the synthesized speech with the intended prosody provided by the input fixed-length utterance embedding.

<FIG> shows an example system <NUM> for training a deep neural network <NUM> to provide a controllable prosody model <NUM>, and for predicting a prosodic representation <NUM> for a text utterance <NUM> using the prosody model <NUM>. The system <NUM> includes a computing system <NUM> having data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions that cause the data processing hardware <NUM> to perform operations. In some implementations, the computing system <NUM> (e.g., the data processing hardware <NUM>) provides a prosody model <NUM> based on a trained deep neural network <NUM> to a text-to-speech (TTS) system <NUM> for controlling prosody of synthesized speech <NUM> from an input text utterance <NUM>. Since the input text utterance <NUM> has no way of conveying context, semantics, and pragmatics to guide the appropriate prosody of the synthesized speech <NUM>, the prosody model <NUM> may predict a prosodic representation <NUM> for the input text utterance <NUM> by conditioning the model <NUM> on linguistic features extracted from the text utterance <NUM> and using a fixed-length utterance embedding <NUM> as a latent variable representing an intended prosody for the text utterance <NUM>. In some examples, the computing system <NUM> implements the TTS system <NUM>. In other examples, the computing system <NUM> and the TTS system <NUM> are distinct and physically separate from one another. The computing system may include a distributed system (e.g., cloud computing environment).

In some implementations, the deep neural network <NUM> is trained on a large set of reference audio signals <NUM>. Each reference audio signal <NUM> may include a spoken utterance of human speech recorded by a microphone and having a prosodic representation. During training, the deep neural network <NUM> may receive multiple reference audio signals <NUM> for a same spoken utterance, but with varying prosodies (i.e., the same utterance can be spoken in multiple different ways). Here, the reference audio signals <NUM> are of variable-length such that the duration of the spoken utterances varies even though the content is the same. The deep neural network <NUM> is configured to encode/compress the prosodic representation associated with each reference audio signal <NUM> into a corresponding fixed-length utterance embedding <NUM>. The deep neural network <NUM> may store each fixed-length utterance embedding <NUM> in an utterance embedding storage <NUM> (e.g., on the memory hardware <NUM> of the computing system <NUM>) along with a corresponding transcript <NUM> of the reference audio signal <NUM> associated the utterance embedding <NUM>. The deep neural network <NUM> may be further trained by back-propagating the fixed-length utterance embeddings <NUM> conditioned upon linguistic features extracted from the transcripts <NUM> to generate fixed-length frames of pitch, energy, and duration of each syllable.

During inference, the computing system <NUM> may use the prosody model <NUM> to predict a prosodic representation <NUM> for a text utterance <NUM>. The prosody model <NUM> may select an utterance embedding <NUM> for the text utterance <NUM>. The utterance embedding <NUM> represents an intended prosody of the text utterance <NUM>. Described in greater detail below with reference to <FIG> and <FIG>, the prosody model <NUM> may predict the prosodic representation <NUM> for the text utterance <NUM> using the selected utterance embedding <NUM>. The prosodic representation <NUM> may include predicted pitch, predicted timing, and predicted loudness (e.g., energy) for the text utterance <NUM>. In the example shown, the TTS system <NUM> uses the prosodic representation <NUM> to produce synthesized speech <NUM> from the text utterance <NUM> and having the intended prosody.

<FIG> and <FIG> show a hierarchical linguistic structure (e.g., deep neural network of <FIG>) <NUM> for a clockwork hierarchal variational autoencoder (CHiVE) <NUM> ('autoencoder <NUM>') that provides a controllable model of prosody that jointly predicts, for each syllable of given input text, a duration of all phonemes in the syllable and pitch (F0) and energy (C0) contours for the syllable without relying on any unique mappings from the given input text or other linguistic specification to produce synthesized speech <NUM> having an intended/selected prosody. The autoencoder <NUM> an encoder portion <NUM> (<FIG>) that encodes a plurality of fixed-length reference frames <NUM> sampled from a reference audio signal <NUM> into a fixed-length utterance embedding <NUM>, and a decoder portion <NUM> (<FIG>) that learns how to decode the fixed-length utterance embedding <NUM> into a plurality of fixed-length predicted frames <NUM>. As will become apparent, the autoencoder <NUM> is trained so that the number of predicted frames <NUM> output from the decoder portion <NUM> is equal to the number of reference frames <NUM> input to the encoder portion <NUM>. Moreover, the autoencoder <NUM> is trained so that data associated with the reference and predicted frames <NUM>, <NUM> substantially match one another.

Referring to <FIG>, the encoder portion <NUM> receives the sequence of fixed-length reference frames <NUM> from the input reference audio signal <NUM>. The input reference audio signal <NUM> may include a spoken utterance of human speech recorded by a microphone that includes a target prosody. The encoder portion <NUM> may receive multiple reference audio signals <NUM> for a same spoken utterance, but with varying prosodies (i.e., the same utterance can be spoken in multiple different ways). For example, the same spoken utterance may vary in prosody when the spoken reference is an answer to a question compared to when the spoken utterance is a question. The reference frames <NUM> may each include a duration of <NUM> milliseconds (ms) and represent one of a contour of pitch (F0) or a contour of energy (C0) for the reference audio signal <NUM>. In parallel, the encoder portion <NUM> may also receive a second sequence of reference frames <NUM> each including a duration of <NUM> and representing the other one of the contour of pitch (F0) or the contour of energy (C0) for the reference audio signal <NUM>. Accordingly, the sequence reference frames <NUM> sampled from the reference audio signal <NUM> provide a duration, pitch contour, and/or energy contour to represent prosody for the reference audio signal <NUM>. The length or duration of the reference audio signal <NUM> correlates to a sum of the total number of reference frames <NUM>.

The encoder portion <NUM> includes hierarchical levels of reference frames <NUM>, phonemes <NUM>,230a, syllables <NUM>, 240a, and words <NUM>, 250a for the reference audio signal <NUM> that clock relative to one another. For instance, the level associated with the sequence of reference frames <NUM> clocks faster than the next level associated with the sequence of phonemes <NUM>. Similarly, the level associated with the sequence of syllables <NUM> clocks slower than the level associated with the sequence of phonemes <NUM> and faster than the level associated with the sequence of words <NUM>. Accordingly, the slower clocking layers receive, as input, an output from faster clocking layers so that the output after the final clock (i.e., state) of a faster layer is taken as the input to the corresponding slower layer to essentially provide a sequence-to-sequence encoder. In the examples shown, the hierarchical levels include Long Short-Term Memory (LSTM) levels.

In some examples, the encoder portion <NUM> first encodes the sequence of reference frames <NUM> into the sequence of phonemes <NUM>. Each phoneme <NUM> receives, as input, a corresponding encoding of a subset of reference frames <NUM> and includes a duration equal to the number of reference frames <NUM> in the encoded subset. In the example shown, the first four fixed-length reference frames <NUM> are encoded into phoneme 230Aa1; the next three fixed-length reference frames <NUM> are encoded into phoneme 230Aa2; the next four fixed-length reference frames <NUM> are encoded into phoneme 230Ab <NUM>; the next two fixed-length reference frames <NUM> are encoded into phoneme 230Ba1, the next five fixed-length reference frames <NUM> are encoded into phoneme 230Ba2; the next four fixed-length reference frames <NUM> are encoded into phoneme 230Ba3; the next three fixed-length reference frames <NUM> are encoded into phoneme 230Ca1; the next four fixed-length reference frames <NUM> are encoded into phoneme 230Cb1; and the final two fixed-length reference frames <NUM> are encoded into phoneme 230Cb2. Thus, each phoneme <NUM> in the sequence of phonemes <NUM> includes a corresponding duration based on the number of reference frames <NUM> encoded into the phoneme <NUM> and corresponding pitch and/or energy contours. For instance, phoneme 230Aa1 includes a duration equal to <NUM> (i.e., four reference frames <NUM> each having the fixed-length of five milliseconds) and phoneme 230Aa2 includes a duration equal to <NUM> (i.e., three reference frames <NUM> each having the fixed-length of five milliseconds). Thus, the level of reference frames <NUM> clocks a total of seven times for a single clocking between the phoneme 230Aa1 and the next phoneme 230Aa2 for the level of phonemes <NUM>.

After encoding the fixed-length frames <NUM> into corresponding phonemes <NUM>, the encoder portion <NUM> is further configured to encode the sequence of phonemes <NUM> into the sequence of syllables <NUM> for the reference audio signal <NUM>. Here, each syllable <NUM> receives, as input, a corresponding encoding of one or more phonemes <NUM> and includes a duration equal to a sum of the durations for the one or more phonemes <NUM> of the corresponding encoding. The duration of the syllables <NUM> may indicate timing of the syllables <NUM> and pauses in between adjacent syllables <NUM>. In the example shown, the first two phonemes 230Aa1, 230Aa2 are encoded into syllable 240Aa; the next phoneme 230Ab1 is encoded into syllable 240Ab; each of phonemes 230Ba1, 230Ba2, 230Ba3 are encoded into syllable 240Ba; phoneme 230Ca1 is encoded into syllable 240Ca; and phonemes 230Cb1, 230Cb2 are encoded into syllable 240Cb. Each syllable 240Aa-240Cb in the level of syllables <NUM> may correspond to a respective syllable embedding (e.g., a numerical vector) that indicates a duration, pitch (F0), and/or energy (C0) associated with the corresponding syllable <NUM>. Moreover, each syllable is indicative of a corresponding state for the level of syllables <NUM>. For instance, syllable 240Aa includes a duration equal to <NUM> (i.e., the sum of the <NUM> duration for phoneme 230Aa1 and the <NUM> duration for phone 230A2) and syllable 240Ab includes a duration equal to <NUM> (i.e., the <NUM> duration for phoneme 230Ab1). Thus, the level of reference frames <NUM> clocks a total of eleven times and the level of phonemes <NUM> clocks a total of three times for a single clocking between the syllable 240Aa and the next syllable 240Ab for the level of syllables <NUM>.

With continued reference to <FIG>, in some implementations, the encoder portion <NUM> further encodes the sequence of syllables <NUM> into the sequence of words <NUM> for the reference audio signal <NUM>. Here, syllables 240Aa, 240Ab are encoded into word 250A; syllable 240Ba is encoded into word 250B; and syllables 240Ca, 240Cb are encoded into word 250C. Finally, the encoder portion <NUM> encodes the sequence of words <NUM> into the fixed-length utterance embedding <NUM>. The fixed-length utterance embedding <NUM> includes a numerical vector representing a prosody of the reference audio signal <NUM>. In some examples, the fixed-length utterance embedding <NUM> includes a numerical vector having a value equal to "<NUM>". The encoder portion <NUM> may repeat this process for each reference audio signal <NUM>. In some examples, the encoder portion <NUM> encodes a plurality of reference audio signals <NUM> each corresponding to a same spoken utterance/phrase but with varying prosodies, i.e., each reference audio signal <NUM> conveys the same utterance but is spoken differently. The fixed-length utterance embedding <NUM> may be stored in the data storage <NUM> (<FIG>) along with a respective transcript <NUM> (e.g., textual representation) of the reference audio signal <NUM>. From the transcript <NUM>, linguistic features may be extracted and stored for use in conditioning the training of the hierarchical linguistic structure <NUM>. The linguistic features may include, without limitation, individual sounds for each phoneme, whether each syllable is stressed or unstressed, the type of each word (e.g., noun/adjective/verb) and/or the position of the word in the utterance, and whether the utterance is a question or phrase.

Referring to <FIG>, in some implementations, the hierarchical linguistic structure <NUM> omits the level associated with the sequence of phonemes <NUM> and allows the encoder portion <NUM> to simply encode a corresponding subset of reference frames <NUM> into each syllable <NUM> of the syllable level <NUM> during training. For instance, the first seven reference frames <NUM> may be encoded directly into syllable 240Aa without having to encode into corresponding phonemes 230Aa1, 230Aa2 (<FIG>) as an intermediary step. Similarly, during training, the hierarchical linguistic structure <NUM> may optionally omit the level associated with the sequence of words <NUM> and allow the encoder portion <NUM> to encode the sequence of syllables <NUM> directly into the fixed-length utterance embedding <NUM>. In other examples, training may instead optionally include the level of associated with the sequence of phonemes <NUM> and allow the encoder portion <NUM> to simply encode a corresponding subset of reference frames <NUM> into each phoneme <NUM> of the level of phonemes <NUM> and then encode a corresponding subset of phonemes <NUM> directly into the fixed-length utterance embedding <NUM> without having to encode corresponding syllables <NUM> and/or words <NUM>.

Referring to <FIG>, the decoder portion <NUM> of the variational autoencoder <NUM> is configured to produce a plurality of fixed-length syllable embeddings <NUM> by initially decoding a fixed-length utterance embedding <NUM> that represents a prosody for an utterance. During training, the utterance embedding <NUM> may include the utterance embedding <NUM> output from the encoder portion <NUM> of <FIG> and <FIG> by encoding the plurality of fixed-length reference frames <NUM> sampled from the reference audio signal <NUM>. Thus, the decoder portion <NUM> is configured to back-propagate the utterance embedding <NUM> during training to generate the plurality of fixed-length predicted frames <NUM> that closely match the plurality of fixed-length reference frames <NUM>. For instance, fixed-length predicted frames <NUM> for both pitch (F0) and energy (C0) may be generated in parallel to represent a target prosody (e.g., predicted prosody) that substantially matches the reference prosody of the reference audio signal <NUM> input to the encoder portion <NUM> as training data. In some examples, a TTS system <NUM> (<FIG>) uses the fixed-length predicted frames <NUM> to produce synthesized speech <NUM> with a selected prosody based on the fixed-length utterance embedding <NUM>. For instance, a unit selection module or a WaveNet module of the TTS system <NUM> may use the frames <NUM> to produce the synthesized speech <NUM> having the intended prosody.

In the example shown, the decoder portion <NUM> decodes the utterance embedding <NUM> (e.g., numerical value of "<NUM>") received from the encoder portion <NUM> (<FIG> or <FIG>) into hierarchical levels of words <NUM>, 250b, syllables <NUM>, 240b, phonemes <NUM>, 230b, and the fixed-length predicted frames <NUM>. Specifically, the fixed-length utterance embedding <NUM> corresponds to a variational layer of hierarchical input data for the decoder portion <NUM> and each of the stacked hierarchical levels include Long Short-Term Memory (LSTM) processing cells variably clocked to a length of the hierarchical input data. For instance, the syllable level <NUM> clocks faster than the word level <NUM> and slower than the phoneme level <NUM>. The rectangular blocks in each level correspond to LSTM processing cells for respective words, syllables, phonemes, or frames. Advantageously, the autoencoder <NUM> gives the LSTM processing cells of the word level <NUM> memory over the last <NUM> words, gives the LSTM cells of the syllable level <NUM> memory over the last <NUM> syllables, gives the LSTM cells of the phoneme level <NUM> memory over the last <NUM> phonemes, and gives the LSTM cells of the fixed-length pitch and/or energy frames <NUM> memory over the last <NUM> fixed-length frames <NUM>. When the fixed-length frames <NUM> include a duration (e.g., frame rate) of five milliseconds each, the corresponding LSTM processing cells provide memory over the last <NUM> milliseconds (e.g., a half second).

In the example shown, the decoder portion <NUM> of the hierarchical linguistic structure <NUM> simply back-propagates the fixed-length utterance embedding <NUM> encoded by the encoder portion <NUM> into the sequence of three words 250A-250C, the sequence of five syllables 240Aa-240Cb, and the sequence of nine phonemes 230Aa1-230Cb2 to generate the sequence of predicted fixed-length frames <NUM>. The decoder portion <NUM> is conditioned upon linguistic features of the input text. By contrast to the encoder portion <NUM> of <FIG> and <FIG> where outputs from faster clocking layers are received as inputs by slower clocking layers, the decoder portion <NUM> includes outputs from slower clocking layers feeding faster clocking layers such that the output of a slower clocking layer is distributed to the input of the faster clocking layer at each clock cycle with a timing signal appended thereto.

Referring to <FIG>, in some implementations, the autoencoder <NUM> uses the hierarchical linguistic structure <NUM> to predict a prosodic representation for a given text utterance <NUM> during inference by jointly predicting durations of phonemes <NUM> and pitch and/or energy contours for each syllable <NUM> of the given text utterance <NUM>. Since the text utterance <NUM> does not provide any context, semantic information, or pragmatic information to indicate an appropriate prosody for the text utterance, the autoencoder <NUM> selects an utterance embedding <NUM> as a latent variable to represent an intended prosody for the text utterance <NUM>.

The utterance embedding <NUM> may be selected from the utterance embedding data storage <NUM> (<FIG>). Each utterance embedding <NUM> in the storage <NUM> may be encoded by the encoder portion <NUM> (<FIG> and <FIG>) from a corresponding variable-length reference audio signal <NUM> (<FIG> and <FIG>) during training. Specifically, the encoder portion <NUM> compresses prosody of variable-length reference audio signals <NUM> into fixed-length utterance embeddings <NUM> during training and stores each utterance embedding <NUM> together with a transcript <NUM> of the corresponding reference audio signal <NUM> in the utterance embedding data storage <NUM> for use by the decoder portion <NUM> at inference. In the example shown, the autoencoder <NUM> may first locate utterance embeddings <NUM> having transcripts <NUM> that closely match the text utterance <NUM> and then select one of the utterance embeddings <NUM> to predict the prosodic representation <NUM> (<FIG>) for the given text utterance <NUM>. In some examples, the fixed-length utterance embedding <NUM> is selected by picking a specific point in a latent space of embeddings <NUM> that likely represents particular semantics and pragmatics for a target prosody. In other examples, the latent space is sampled to choose a random utterance embedding <NUM> for representing the intended prosody for the text utterance <NUM>. In yet another example, the autoencoder <NUM> models the latent space as multidimensional unit Gaussian by choosing a mean of the utterance embeddings <NUM> having closely matching transcripts <NUM> for representing a most likely prosody for the linguistic features of the text utterance <NUM>. If the prosody variation of the training data is reasonably neutral, the last example of choosing the mean of utterance embeddings <NUM> is a reasonable choice.

<FIG> and <FIG> show the text utterance <NUM> having three words 250A, 250B, 250C represented in the word level <NUM> of the hierarchical linguistic structure <NUM>. The first word 250A contains syllables 240Aa, 240Ab, the second word 250B contains one syllable 240Ba, and the third word 250C contains syllables 240Ca, 240Cb. Accordingly, the syllable level <NUM> of the hierarchical linguistic structure <NUM> includes a sequence of five syllables 240Aa-240Cb of the text utterance <NUM>. At the syllable level <NUM> of LTSM processing cells, the autoencoder <NUM> is configured to produce/output a corresponding syllable embedding 245Aa, 245Ab, 245Ba, 245Ca, 245Cb for each syllable <NUM> from the following inputs: the fixed-length utterance embedding <NUM>; utterance-level linguistic features <NUM> associated with the text utterance <NUM>; word-level linguistic features <NUM> associated with the word <NUM> that contains the syllable <NUM>; and syllable-level linguistic features <NUM> for the syllable <NUM>. The utterance-level linguistic features <NUM> may include, without limitation, whether or not the text utterance <NUM> is a question, an answer to a question, a phrase, a sentence, etc. The word-level linguistic features <NUM> may include, without limitation, a word type (e.g., noun, pronoun, verb, adjective, adverb, etc.) and a position of the word in the text utterance <NUM>. The syllable-level linguistic features <NUM> may include, without limitation, whether the syllable <NUM> is stressed or unstressed.

In the example shown, each syllable 240Aa, 240Ab, 240Ba, 240Ca, 240Cb in the syllable level <NUM> may be associated with a corresponding LTSM processing cell that outputs a corresponding syllable embedding 245Aa, 245Ab, 245Ba, 245Ca, 245Cb to the faster clocking phoneme level <NUM> for decoding the individual fixed-length predicted pitch (F0) frames <NUM>, 280F0 (<FIG>) and for decoding the individual fixed-length predicted energy (C0) frames <NUM>, 280C0 (<FIG>) in parallel. <FIG> shows each syllable in the syllable level <NUM> including a plurality of fixed-length predicted pitch (F0) frames 280F0 that indicate a duration (timing and pauses) and a pitch contour for the syllable <NUM>. Here, the duration and pitch contour correspond to a prosodic representation of the syllable <NUM>. <FIG> shows each phoneme in the phoneme level <NUM> including a plurality of fixed-length predicted energy (C0) frames 280C0 that indicate a duration and an energy contour for the phoneme.

The first syllable 240Aa (i.e., LTSM processing cell Aa) in the syllable level <NUM> receives the fixed-length utterance embedding <NUM>, utterance-level linguistic features <NUM> associated with the text utterance <NUM>, word-level linguistic features 252A associated with the first word 250A, and the syllable-level linguistic features 242Aa for the syllable 240Aa as inputs for producing the corresponding syllable embedding 245Aa. The second syllable 240Ab in the syllable level <NUM> receives the fixed-length utterance embedding <NUM>, the utterance-level linguistic features <NUM> associated with the text utterance <NUM>, the word-level linguistic features 252A associated with the first word 250A, and corresponding syllable-level linguistic features <NUM> (not shown) for the syllable 240Ab as inputs for producing the corresponding syllable embedding 245Aa. While the example only shows syllable-level linguistic features <NUM> associated with the first syllable 240Aa, the corresponding syllable-level linguistic features <NUM> associated with each other syllable 240Ab-240Cb in the syllable level <NUM> are only omitted from the views of <FIG> and <FIG> for the sake of clarity.

For simplicity, the corresponding syllable-level linguistic features <NUM> input to the processing block for syllable 240Ab are not shown. The LTSM processing cell (e.g., rectangle Ab) associated with the second syllable 240Ab also receives the state of the preceding first syllable 240Aa. The remaining sequence of syllables 240Ba, 240Ca, 240Cb in the syllable level <NUM> each produce corresponding syllable embeddings 245Ba, 245Ca, 245Cb in a similar manner. For simplicity, the corresponding syllable-level linguistic features <NUM> input to the processing block for each of the syllables 240Ba, 240Ca, 240Cb are not shown. Moreover, each LTSM processing cell of the syllable level <NUM> receives the state of the immediately preceding LTSM processing cell of the syllable level <NUM>.

Referring to <FIG>, the phoneme level <NUM> of the hierarchical linguistic structure <NUM> includes the sequence of nine phonemes 230Aa1-230Cb2 each associated with a corresponding predicted phoneme duration <NUM>. Moreover, the autoencoder <NUM> encodes the phoneme-level linguistic features <NUM> associated with each phoneme 230Aa1-230Cb2 with the corresponding syllable embedding <NUM> for predicting the corresponding predicted phoneme duration <NUM> and for predicting the corresponding pitch (f0) contour for the syllable containing the phoneme. The phoneme-level linguistic features <NUM> may include, without limitation, an identity of sound for the corresponding phoneme <NUM>. While the example only shows phoneme-level linguistic features <NUM> associated with the first phoneme 230Aa1, the phoneme-level linguistic features <NUM> associated with the other phonemes 230Aa2-230Cb2 in the phoneme level <NUM> are only omitted from the views of <FIG> and <FIG> for the sake of clarity.

The first syllable 240Aa contains phonemes 230Aa1, 230Aa2 and includes a predicted syllable duration equal to the sum of the predicted phone durations <NUM> for the phonemes 230Aa1, 230Aa2. Here, the predicted syllable duration for the first syllable 240Aa determines the number of fixed-length predicted pitch (F0) frames 280F0 to decode for the first syllable 240Aa. In the example shown, the autoencoder <NUM> decodes a total of seven fixed-length predicted pitch (F0) frames 280F0 for the first syllable 240Aa based on the sum of the predicted phoneme durations <NUM> for the phonemes 230Aa1, 230Aa2. Accordingly, the faster clocking syllable layer <NUM> distributes the first syllable embedding 245Aa as an input to each phoneme 230Aa1, 230Aa2 included in the first syllable 240Aa. A timing signal may also be appended to the first syllable embedding 245Aa. The syllable level <NUM> also passes the state of the first syllable 240Aa to the second syllable 240Ab.

The second syllable 240Ab contains a single phoneme 230Ab1 and therefore includes a predicted syllable duration equal to the predicted phoneme duration <NUM> for the phoneme 230Ab1. Based on the predicted syllable duration for the second syllable 240Ab, the autoencoder <NUM> decodes a total of four fixed-length predicted pitch (F0) frames 280F0 for the second syllable 240Ab. Accordingly, the faster clocking syllable layer <NUM> distributes the second syllable embedding 245Ab as an input to the phoneme 230Ab1. A timing signal may also be appended to the second syllable embedding 245Aa. The syllable level <NUM> also passes the state of the second syllable 240Ab to the third syllable 240Ba.

The third syllable 240Ba contains phonemes 230Ba1, 230Ba2, 230Ba3 and includes a predicted syllable duration equal to the sum of the predicted phoneme durations <NUM> for the phonemes 230Ba1, 230Ba2, 230Ba3. In the example shown, the autoencoder <NUM> decodes a total of eleven fixed-length predicted pitch (F0) frames 280F0 for the third syllable 240Ba based on the sum of the predicted phoneme durations <NUM> for the phonemes 230Ba1, 230Ba2, 230Ba3. Accordingly, the faster clocking syllable layer <NUM> distributes the third syllable embedding 245Ba as an input to each phoneme 230Ba1, 230Ba2, 230Ba3 included in the third syllable 240Ba. A timing signal may also be appended to the third syllable embedding 245Ba. The syllable level <NUM> also passes the state of the third syllable 240Ba to the fourth syllable 240Ca.

The fourth syllable 240Ca contains a single phoneme 230Ca1 and therefore includes a predicted syllable duration equal to the predicted phoneme duration <NUM> for the phoneme 230Ca1. Based on the predicted syllable duration for the fourth syllable 240Ca, the autoencoder <NUM> decodes a total of three fixed-length predicted pitch (F0) frames 280F0 for the fourth syllable 240Ca. Accordingly, the faster clocking syllable layer <NUM> distributes the fourth syllable embedding 245Ca as an input to the phoneme 230Ca1. A timing signal may also be appended to the fourth syllable embedding 245Ca. The syllable level <NUM> also passes the state of the fourth syllable 240Ba to the fifth syllable 240Cb.

Lastly, the fifth syllable 240Cb contains phonemes 230Cb1, 230Cb2 and includes a predicted syllable duration equal to the sum of the predicted phoneme durations <NUM> for the phonemes 230Cb1, 230Cb2. In the example shown, the autoencoder <NUM> decodes a total of six fixed-length predicted pitch (F0) frames 280F0 for the fifth syllable 240Cb based on the sum of the predicted phoneme durations <NUM> for the phonemes 230Cb1, 230Cb2. Accordingly, the faster clocking syllable layer <NUM> distributes the fifth syllable embedding 245Cb as an input to each phoneme 230Cb1, 230Cb2 included in the fifth syllable 240Cb. A timing signal may also be appended to the fifth syllable embedding 245Cb.

<FIG> provides a detailed view within dashed box <NUM> of <FIG> to show the decoding of the first syllable embedding 245Aa into individual fixed-length predicted pitch (F0) frames 280F0 for the first syllable 240Aa. As set forth above with reference to <FIG>, the autoencoder <NUM> determines the number of fixed-length predicted pitch (F0) frames <NUM> to decode based on the predicted syllable duration for the first syllable 240Aa. The first syllable 240Aa generates the corresponding first syllable embedding 245Aa for distribution as an input to each of the first and second phonemes 230Aa1, 230Aa2 of the faster clocking syllable level <NUM>.

At the phoneme level <NUM> of the hierarchical linguistic structure <NUM>, the autoencoder <NUM> predicts the phoneme duration <NUM> for the first phoneme 230Aa1 by encoding the phoneme-level linguistic features <NUM> associated with the first phoneme 230Aa1 with the first syllable embedding 245Aa. Likewise, the autoencoder <NUM> predicts the phoneme duration <NUM> for the second phoneme 230Aa2 by encoding the phoneme-level linguistic features (not shown) associated with the second phoneme 230Aa2 with the first syllable embedding 245Aa. The second phoneme 230Aa2 also receives the previous state from the first phoneme 230Aa1. The predicted syllable duration for the first syllable 230Aa is equal to the sum of the predicted phone durations <NUM> for the first and second phonemes 230Aa1, 230Aa2. The encodings of the first syllable embedding 245Aa with the corresponding phoneme-level linguistic features <NUM> associated with each of the phonemes 230Aa1, 230Aa2 is further combined with the first syllable embedding 245Aa at the output of the phoneme level <NUM> to predict the pitch (F0) for the first syllable 240Aa and generate the fixed-length predicted pitch (F0) frames 280F0 for the first syllable 240Aa. In the example shown, the autoencoder <NUM> determines the total number (e.g., seven) of fixed-length predicted pitch (F0) frames 280F0 to decode/generate based on the predicted syllable duration for the first syllable 240Aa. Thus, the fixed-length predicted pitch (F0) frames <NUM> decoded from the first syllable embedding 245Aa collectively indicate a corresponding duration and pitch contour for the first syllable 240Aa of the text utterance <NUM>.

Referring back to <FIG>, the autoencoder <NUM> similarly decodes each of the remaining syllable embeddings 245Ab, 245Ba, 245Ca, 245Cb output from the syllable level <NUM> into individual fixed-length predicted pitch (F0) frames <NUM> for each corresponding syllable 240Ab, 240Ba, 240Ca, 240Cb. For instance, the second syllable embedding 245Ab is further combined at the output of the phoneme level <NUM> with the encoding of the second syllable embedding 245Ab and the corresponding phoneme-level linguistic features <NUM> associated with the phoneme 230Ab1, while the third syllable embedding 245Ba is further combined at the output of the phoneme level <NUM> with the encodings of the third syllable embedding 245Ba and the corresponding phoneme-level linguistic features <NUM> associated with each of the phonemes 230Ba1, 230Ba2, 230Ba3. Moreover, the fourth syllable embedding 245Ca is further combined at the output of the phoneme level <NUM> with the encodings of the fourth syllable embedding 245Ca and the corresponding phoneme-level linguistic features <NUM> associated with the phoneme 230Ca1, while the fifth syllable embedding 245Cb is further combined at the output of the phoneme level <NUM> with the encodings of the fifth syllable embedding 245Cb and the corresponding phoneme-level linguistic features <NUM> associated with each of the phonemes 230Cb1, 230Cb2. While the fixed-length predicted pitch (F0) frames 280F0 generated by the autoencoder <NUM> include frame-level LSTM, other configurations may replace the frame-level LSTM of pitch (F0) frames 280F0 with a feed-forward layer so that the pitch (F0) of every frame in a corresponding syllable is predicted in one pass.

Referring now to <FIG>, the autoencoder <NUM> is further configured to encode the phoneme-level linguistic features <NUM> associated with each phoneme 230Aa1-230Cb2 with the corresponding syllable embedding <NUM> for predicting the corresponding energy (C0) contour for each phoneme <NUM>. The phoneme-level linguistic features <NUM> associated with phonemes 230Aa2-230Cb2 in the phoneme level <NUM> are only omitted from the view of <FIG> for the sake of clarity. The autoencoder <NUM> determines the number of fixed-length predicted energy (C0) frames <NUM>, 280C0 to decode for each phoneme <NUM> based on the corresponding predicted phoneme duration <NUM>. For instance, the autoencoder <NUM> decodes/generates four (<NUM>) predicted energy (C0) frames 280C0 for the first phoneme 230Aa1, three (<NUM>) predicted energy (C0) frames 280C0 for the second phoneme 230Aa2, four (<NUM>) predicted energy (C0) frames 280C0 for the third phoneme 230Ab1, two (<NUM>) predicted energy (C0) frames 280C0 for the fourth phoneme 230Ba1, five (<NUM>) predicted energy (C0) frames 280C0 for the fifth phoneme 230Ba2, four (<NUM>) predicted energy (C0) frames 280C0 for the sixth phoneme 230Ba3, three (<NUM>) predicted energy (C0) frames 280C0 for the seventh phoneme 230Ca1, four (<NUM>) predicted energy (C0) frames 280C0 for the eighth phoneme 230Cb1, and two (<NUM>) predicted energy (C0) frames 280C0 for the ninth phoneme 230Cb2. Accordingly, as with the predicted phoneme duration <NUM>, the predicted energy contour for each phoneme in the phoneme level <NUM> is based on an encoding between the syllable embedding <NUM> input from the corresponding syllable in the slower clocking syllable level <NUM> that contains the phoneme and the linguistic features <NUM> associated with the phoneme.

<FIG> is a flow chart of an example arrangement of operations for a method <NUM> of predicting a prosodic representation <NUM> for a text utterance <NUM>. The method <NUM> may be described with reference to <FIG>. The memory hardware <NUM> residing on the computer system <NUM> of <FIG> may store instructions that when executed by the data processing hardware <NUM> cause the data processing hardware <NUM> to execute the operations for the method <NUM>. At operation <NUM>, the method <NUM> includes receiving the text utterance <NUM>. The text utterance <NUM> has at least one word, each word having at least one syllable, each syllable having at least one phoneme. At operation <NUM>, the method <NUM> includes selecting an utterance embedding <NUM> for the text utterance <NUM>. The utterance embedding <NUM> represents an intended prosody. As used herein, the selected utterance embedding <NUM> is used to predict the prosodic representation <NUM> of the text utterance <NUM> for use by a TTS system <NUM> to produce synthesized speech <NUM> from the text utterance <NUM> and having the intended prosody. The utterance embedding <NUM> may be represented by a fixed-length numerical vector. The numerical vector may include a value equal to "<NUM>". To select the utterance embedding <NUM> for the text utterance <NUM>, the data processing hardware <NUM> may first query the data storage <NUM> to locate utterance embeddings <NUM> having transcripts <NUM> that closely match the text utterance <NUM> and then select the utterance embeddings <NUM> to predict the prosodic representation <NUM> for the given text utterance <NUM>. In some examples, the fixed-length utterance embedding <NUM> is selected by picking a specific point in a latent space of embeddings <NUM> that likely represents particular semantics and pragmatics for a target prosody. In other examples, the latent space is sampled to choose a random utterance embedding <NUM> for representing the intended prosody for the text utterance <NUM>. In yet another example, the data processing hardware <NUM> models the latent space as multidimensional unit Gaussian by choosing a mean of the utterance embeddings <NUM> having closely matching transcripts <NUM> for representing a most likely prosody for the linguistic features of the text utterance <NUM>. If the prosody variation of the training data is reasonably neutral, the last example of choosing the mean of utterance embeddings <NUM> is a reasonable choice.

At operation <NUM>, for each syllable <NUM>, using the selected utterance embedding <NUM>, the method <NUM> includes predicting a duration of the syllable by encoding linguistic features <NUM> of each phoneme <NUM> of the syllable with a corresponding prosodic syllable embedding <NUM> for the syllable. For instance, for each phoneme <NUM> associated with the syllable <NUM>, the method <NUM> may predict a duration <NUM> of the corresponding phoneme <NUM> by encoding the linguistic features <NUM> of the corresponding phoneme <NUM> with the corresponding prosodic syllable embedding <NUM> for the syllable <NUM>. Thereafter, the method <NUM> may predict the duration of the syllable <NUM> by summing the predicted durations <NUM> for each phoneme <NUM> associated with the syllable <NUM>.

At operation <NUM>, for each syllable <NUM>, using the selected utterance embedding <NUM>, the method <NUM> includes predicting a pitch contour of the syllable based on the predicted duration for the syllable. At operation <NUM>, for each syllable, using the selected utterance embedding <NUM>, the method <NUM> also includes generating a plurality of fixed-length predicted pitch frames <NUM>, 280F0 based on the predicted duration for the syllable <NUM>. Each fixed-length predicted pitch frame 280F0 represents part of the predicted contour of the syllable <NUM>.

Additional operations for the method <NUM> may further include, for each syllable <NUM>, using the selected utterance embedding <NUM>, predicting an energy contour of each phoneme <NUM> in the syllable 240based on a predicted duration <NUM> for the corresponding phoneme <NUM>. For each phoneme <NUM> associated with the syllable, the method <NUM> may generate a plurality of fixed-length predicted energy frames <NUM>, 280C0 based on the predicted duration <NUM> for the corresponding phoneme <NUM>. Here, each fixed-length energy frame 280C0 represents the predicted energy contour of the corresponding phoneme <NUM>.

<FIG> is schematic view of an example computing device <NUM> (e.g., computing system <NUM> of <FIG>) that may be used to implement the systems and methods described in this document.

The processor <NUM> (e.g., data processing hardware <NUM> of <FIG>)) can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> (e.g., memory hardware <NUM> of <FIG>) or on the storage device <NUM> (e.g., memory hardware <NUM> of <FIG>) to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display <NUM> coupled to high speed interface <NUM>.

Claim 1:
A method (<NUM>) of representing an intended prosody in synthesized speech using a prosody model using a hierarchical linguistic structure of a text utterance, the prosody model including a variational autoencoder comprising an encoder portion and a decoder portion, the method comprising:
receiving, at data processing hardware (<NUM>), a text utterance (<NUM>) having at least one word (<NUM>), each word (<NUM>) having at least one syllable (<NUM>), each syllable (<NUM>) having at least one phoneme (<NUM>);
selecting, by the data processing hardware (<NUM>), an utterance embedding (<NUM>) for the text utterance (<NUM>), the utterance embedding (<NUM>) representing an intended prosody for the utterance and encoded by the encoder portion during training; and
for each syllable (<NUM>), using the selected utterance embedding (<NUM>):
predicting, by the decoder portion, a duration of the syllable (<NUM>) by encoding linguistic features (<NUM>) of each phoneme (<NUM>) of the syllable (<NUM>) with a corresponding prosodic syllable embedding (<NUM>) for the syllable (<NUM>);
predicting, by the decoder portion, a pitch contour (F0) of the syllable (<NUM>) based on the predicted duration for the syllable (<NUM>); and
generating, by the decoder portion, a plurality of fixed-length predicted pitch frames (<NUM>) based on the predicted duration for the syllable (<NUM>), each fixed-length predicted pitch frame (<NUM>) representing part of the predicted pitch contour (F0) of the syllable (<NUM>).