Patent Description:
Text-to-speech (TTS) systems read aloud digital text to a user and are becoming increasingly popular on mobile devices. Certain TTS models aim to synthesize various aspects of speech, such as speaking styles, to produce human-like, natural sounding speech. Synthesis in TTS models is a one-to-many mapping problem, as there can be multiple possible speech outputs for the different prosodies of text inputs. Many TTS systems utilize an autoregressive model that predicts current values based on previous values. While autoregressive TTS models can synthesize text and generate highly natural speech outputs, the hundreds of calculations required reduce efficiency during inference.

<NPL>) describes a non-autoregressive neural text-to-speech model augmented with a variational autoencoder-based residual encoder.

<CIT> describes a method for providing a frame-based mel spectral representation of speech including receiving a text utterance having at least one word, and selecting a mel spectral embedding for the text utterance. Each word in the text utterance has at least one syllable and each syllable has at least one phoneme. For each phoneme, using the selected mel spectral embedding, the method also includes: predicting a duration of the corresponding phoneme by encoding linguistic features of the corresponding phoneme with a corresponding syllable embedding for the syllable that includes the corresponding phoneme; and generating a plurality of fixed-length predicted mel-frequency spectrogram frames based on the predicted duration for the corresponding phoneme. Each fixed-length predicted mel-frequency spectrogram frame representing mel-spectral information of the corresponding phoneme.

<CIT> describes an end-to-end voice synthesis method and system based on a DNN-HMM bimodal alignment network. According to the method, a frame length prediction module is used for replacing a traditional end-to-end attention autoregressive structure, and a convolutional change module and a bidirectional long-short-term memory network are used for constructing an encoder and a decoder, so that a large number of model parameters are reduced. On the basis that a phoneme frame length sequence is obtained through DNN-HMM bimodal alignment network training, an end-to-end voice synthesis model is trained, and thus the process that a traditional end-to-end voice synthesis model obtains text and audio alignment information in an autoregressive attention mode is avoided.

One aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations for training a non-autoregressive text-to-speech (TTS) model. The operations include obtaining a sequence representation of an encoded text sequence concatenated with a variational embedding. Using a duration model network, the operations also include predicting, based on the sequence representation, a phoneme duration for each phoneme represented by the encoded text sequence. Based on the predicted phoneme durations, the operations include learning, using a first function conditioned on the sequence representation, an interval representation and learning, using a second function conditioned on the sequence representation, an auxiliary attention context representation. The operations also include upsampling, using the interval representation matrix and the auxiliary attention context representation, the sequence representation into an upsampled output specifying a number of frames. The operations also include generating, as output from a spectrogram decoder that includes a stack of one or more self-attention blocks and based on the upsampled output, one or more predicted mel-frequency spectrogram sequences for the encoded text sequence. The operations also include determining a final spectrogram loss based on the one or more predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence and training the TTS model based on the final spectrogram loss.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the first function and the second function each include a respective multi-layer perception-based learnable function. The operations may further include determining a global phoneme duration loss based on the predicted phoneme durations and an average phoneme duration. Here, training the TTS model is further based on the global phoneme duration loss. In some examples, training the TTS model based on the final spectrogram loss and the global phoneme duration loss includes training the duration model network to predict the phoneme duration for each phoneme without using supervised phoneme duration labels extracted from an external aligner.

In some implementations, the operations further include using the duration model network by generating, based on the predicted phoneme durations for each phoneme represented by the encoded text sequence, respective start and end boundaries and mapping, based on a number of phonemes represented by the encoded text sequence and a number of reference frames in the reference mel-frequency spectrogram sequence, the respective start and end boundaries generated for each phoneme into respective grid matrices. Here, learning the interval representation is based on the respective grid matrices mapped from the start and end boundaries and learning the auxiliary attention context is based on the respective grid matrices mapped from the start and end boundaries. Upsampling the sequence representation into the upsampled output may include determining a product of the interval representation matrix and the sequence representation, determining an Einstein summation (einsum) of the interval representation matrix and the auxiliary attention context representation, and summing the product of the interval representation matrix and the sequence representation and a projection of the einsum to generate the upsampled output.

In some implementations, the operations further include: receiving training data that includes a reference audio signal and a corresponding input text sequence, the reference audio signal includes a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal; encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal; and encoding, using a text encoder, the input text sequence into the encoded text sequence. In some examples the residual encoder includes a global variational autoencoder (VAE). In these examples, encoding the reference audio signal into the variational embedding includes sampling the reference mel-frequency spectrogram sequence from the reference audio signal and encoding, using the global VAE, the reference mel-frequency spectrogram sequence into the variational embedding. Optionally, the residual encoder may include a phoneme-level fine-grained variational autoencoder (VAE). Here, encoding the reference audio signal into the variational embedding includes: sampling the reference mel-frequency spectrogram sequence from the reference audio signal; aligning the reference mel-frequency spectrogram sequence with each phoneme in a sequence of phonemes extracted from the input text sequence; and encoding, using the phoneme-level fine-grained VAE and based on aligning the reference mel-frequency spectrogram sequence with each phoneme in the sequence of phonemes, a sequence of phoneme-level variational embeddings.

The residual encoder includes a stack of lightweight convolution (LConv) blocks, each LConv block in the stack of LConv blocks may include a gated linear unit (GLU) layer, a LConv layer configured to receive an output of the GLU layer, a residual connection configured to concatenate an output of the LConv layer with an input to the GLU layer, and a final feedforward layer configured to receive, as input, the residual connection concatenating the output of the LConv layer with the input to the GLU layer. In some implementations, the operations further include: concatenating the encoded text sequence, the variational embedding, and a reference speaker embedding that represents an identity of a reference speaker that uttered the reference audio signal; and generating the sequence representation based on the duration modeling network receiving, as input, the concatenation of the encoded text sequence, the variational embedding, and the reference speaker embedding. In some examples, the input text sequence includes a sequence of phonemes. In these examples, encoding the input text sequence into the encoded text sequence includes: receiving, from a phoneme look-up table, a respective embedding of each phoneme in the sequence of phonemes; for each phoneme in the sequence of phonemes, processing, using an encoder pre-net neural network of the text encoder, the respective embedding to generate a respective transformed embedding of the phoneme; processing, using a bank of convolutional blocks, the respective transformed embeddings to generate convolution outputs; and processing, using a stack of self-attention blocks, the convolution outputs to generate the encoded text sequence. Optionally, each self-attention block in the stack of self-attention blocks includes an identical lightweight convolution (LConv) block. Each self-attention block in the stack of self-attention blocks includes an identical transformer block.

Another aspect of the disclosure provides a system for training a non-autoregressive text-to-speech (TTS) model that includes data processing hardware and memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include obtaining a sequence representation of an encoded text sequence concatenated with a variational embedding. Using a duration model network, the operations also include predicting, based on the sequence representation, a phoneme duration for each phoneme represented by the encoded text sequence. Based on the predicted phoneme durations, the operations include learning, using a first function conditioned on the sequence representation, an interval representation and learning, using a second function conditioned on the sequence representation, an auxiliary attention context representation. The operations also include upsampling, using the interval representation matrix and the auxiliary attention context representation, the sequence representation into an upsampled output specifying a number of frames. The operations also include generating, as output from a spectrogram decoder that includes a stack of one or more self-attention blocks and based on the upsampled output, one or more predicted mel-frequency spectrogram sequences for the encoded text sequence. The operations also include determining a final spectrogram loss based on the one or more predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence and training the TTS model based on the final spectrogram loss.

The synthesis of realistic human speech is an underdetermined problem in that a same text input has an infinite number of reasonable spoken realizations. While end-to-end neural network-based approaches are advancing to match human performance for short assistant-like utterances, neural network models are sometimes viewed as less interpretable or controllable than more conventional models that include multiple processing steps each operating on refined linguistic or phonetic representations. Sources of variability in speech include prosodic characteristics of intonation, stress, rhythm, and style, as well as speaker and channel characteristics. The prosodic characteristics of a spoken utterance convey linguistic, semantic, and emotional meaning beyond what is present in a lexical representation (e.g., a transcript of the spoken utterance).

For instance, neural network-based end-to-end text-to-speech (TTS) models may convert input text to output speech. Neural network TTS models provide potential for robustly synthesizing speech by predicting linguistic factors corresponding to prosody that are not provided by text inputs. As a result, a number of applications, such as audiobook narration, news readers, voice design software, and conversational assistants can produce realistically sounding synthesized speech that is not monotonous-sounding.

Many neural end-to-end TTS models utilize an autoregressive model that predicts current values based on previous values. For instance, many autoregressive models are based on recurrent neural networks that use some or all of an internal state of the network from a previous time step in computing an output at a current time step. An example of a recurrent neural network is a long short term (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more cells that each include an input gate, a forget gate, and an output gate that allows the cell to store previous states for the cell, e.g., for use in generating a current activation or to be provided to other components of the LSTM neural network.

While autoregressive TTS models can synthesize text and generate highly natural speech outputs, their architecture through a series of uni-directional LSTM-based decoder blocks with soft attention inherently makes both training and inference less efficient when implemented on modern parallel hardware compared to fully-feedforward architectures. Moreover, as autoregressive models train via teacher forcing by applying ground truth labels for each time step, autoregressive models are additionally prone to producing discrepancies between training and when the trained model is applied during inference. Together with the soft attention mechanism, these discrepancies can lead to synthesized speech output with reduced quality, such as the synthesized speech exhibiting robustness errors such as babbling, early cut-off, word repetition, and word skipping. The reduction in quality of synthesized speech in autoregressive TTS models may be further exacerbated as a size of the synthesized text increases.

To alleviate the aforementioned drawbacks of autoregressive-based TTS models, implementations herein are directed toward a non-autoregressive neural TTS model augmented with a variational autoencoder (VAE)-based residual encoder. As will become apparent, the VAE-based residual encoder may disentangle latent representations/states from reference audio signals that convey residual information, such as style/prosody information, which cannot be represented by the input text (e.g., phoneme sequence) to be synthesized or speaker identifiers (IDs) for the speakers that spoke the reference audio signals. That is to say, a latent representation enables the output synthesized speech produced by the TTS model to sound like the reference audio signal that was input to the residual encoder.

The non-autoregressive neural TTS model augmented with the VAE-based residual encoder provides a controllable model for predicting mel spectral information (e.g., a predicted mel-frequency spectrogram sequence) for an input text utterance, while at the same time effectively controlling the prosody/style represented in the mel spectral information. For instance, using a selected variational embedding learned by the VAE-based residual encoder to represent an intended prosody/style for synthesizing a text utterance into expressive speech, a spectrogram decoder of the TTS model may predict a mel-frequency spectrogram for the text utterance and provide the mel-frequency spectrogram as input to a synthesizer (e.g., a waveform synthesizer or a vocoder network) for conversion into a time-domain audio waveform indicative of synthesized speech having the intended prosody/style. As will become apparent, the non-autoregressive TTS model is trained on sample input text sequences and corresponding reference mel-frequency spectrogram sequences of human speech alone so that the trained TTS model can convert an input text utterance to a mel-frequency spectrogram sequence having an intended prosody/style conveyed by a learned prior variational embedding.

<FIG> shows an example system <NUM> for training a deep neural network <NUM> that is augmented with a VAE-based residual encoder <NUM> to provide a non-autoregressive neural TTS model (or simply 'TTS model') <NUM>, and for predicting a spectrogram (i.e., mel-frequency spectrogram sequence) <NUM> for a text utterance <NUM> using the TTS model <NUM>. The system <NUM> includes a computing system <NUM> having data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions that cause the data processing hardware <NUM> to perform operations. In some implementations, the computing system <NUM> (e.g., the data processing hardware <NUM>) or a user computing device <NUM> executing the trained TTS model <NUM> provides the predicted mel-frequency spectrogram <NUM> predicted by the TTS model <NUM> from the input text utterance <NUM> to a synthesizer <NUM> for conversion into a time-domain audio waveform indicative of synthesized speech <NUM> that may be audibly output as a spoken representation of the input text utterance <NUM>. A time-domain audio waveform includes an audio waveform that defines an amplitude of an audio signal over time. The synthesizer <NUM> may be separately trained and conditioned on mel-frequency spectrograms for conversion into time-domain audio waveforms.

A mel-frequency spectrogram includes a frequency-domain representation of sound. Mel-frequency spectrograms emphasize lower frequencies, which are critical to speech intelligibility, while de-emphasizing high frequency, which are dominated by fricatives and other noise bursts and generally do not need to be modeled with high fidelity. The synthesizer <NUM> may include a vocoder neural network that may include any network that is configured to receive mel-frequency spectrograms and generate audio output samples (e.g., time-domain audio waveforms) based on the mel-frequency spectrograms. For example, the vocoder network <NUM> can be based on the parallel feedforward neural network described in van den Oord, Parallel WaveNet: Fast High-Fidelity Speech Synthesis, available at https://arxiv. org/pdf/<NUM>. Alternatively, the vocoder network <NUM> can be an autoregressive neural network. The synthesizer <NUM> may include a waveform synthesizer such as a Griffin-Lim synthesizer or a trainable spectrogram to waveform inverter. The choice of the synthesizer <NUM> has no impact on resulting prosody/style of the synthesized speech <NUM>, and in practice, only impacts audio fidelity of the synthesized speech <NUM>.

Since the input text utterance <NUM> has no way of conveying context, semantics, and pragmatics to guide a desired prosody/style of the synthesized speech <NUM>, the TTS model <NUM> may apply a variational embedding <NUM> as a latent variable specifying an intended prosody/style in order to predict a mel-frequency spectrogram <NUM> for the text utterance <NUM> that conveys the intended prosody/style specified by the variational embedding <NUM>. In some examples, the computing system <NUM> implements the TTS model <NUM>. Here, a user may access the TTS model <NUM> through a user computing device <NUM> and provide the input text utterance <NUM> for the TTS model <NUM> to synthesize into expressive speech <NUM> having an intended prosody/style specified by a variational embedding <NUM>. The variational embedding <NUM> may be selected by the user and correspond to a prior variational embedding <NUM> sampled from the residual encoder <NUM>. The variational embedding <NUM> may be a per-speaker variational embedding <NUM> that the user may select by providing a speaker identifier (ID) (i.e., through an interface executing on the user computing device <NUM>) that identifies a speaker who speaks with the intended prosody/style. Here, each speaker ID may map to a respective per-speaker variational embedding <NUM> previously learned by the residual encoder <NUM>. Additionally or alternatively, the user could provide an input specifying a particular vertical associated with a respective prosody/style. Here, different verticals (e.g., newscasters, sportscasters, etc.) may each map to a respective variational embedding <NUM> previously learned by the residual encoder <NUM> that conveys the respective prosody/style associated with the vertical. In these examples, the synthesizer <NUM> may reside on the computing system <NUM> or the user computing device <NUM>. When the synthesizer <NUM> resides on the computing system <NUM>, the computing system <NUM> may transmit a time-domain audio waveform representing the synthesized speech <NUM> to the user computing device <NUM> for audible playback. In other examples, the user computing device <NUM> implements the TTS model <NUM>. The computing system may include a distributed system (e.g., cloud computing environment).

In some implementations, the deep neural network <NUM> is trained on a large set of reference audio signals <NUM>. Each reference audio signal <NUM> may include a spoken utterance of human speech recorded by a microphone and having a prosodic/style representation. During training, the deep neural network <NUM> may receive multiple reference audio signals <NUM> for a same spoken utterance, but with varying prosodies/styles (i.e., the same utterance can be spoken in multiple different ways). Here, the reference audio signals <NUM> are of variable-length such that the duration of the spoken utterances varies even though the content is the same. The deep neural network <NUM> may also receive multiple sets of reference audio signals <NUM> where each set includes reference audio signals <NUM> for utterances having similar prosodies/styles spoken by a same respective speaker, but conveying different linguistic content. The deep neural network <NUM> augmented with the VAE-based residual encoder <NUM> is configured to encode/compress the prosodic/style representation associated with each reference audio signal <NUM> into a corresponding variational embedding <NUM>. The variational embedding <NUM> may include a fixed-length variational embedding <NUM>. The deep neural network <NUM> may store each variational embedding <NUM> in storage <NUM> (e.g., on the memory hardware <NUM> of the computing system <NUM>) along with a corresponding speaker embedding, ys, representing a speaker identity <NUM> (<FIG>) of a reference speaker that uttered the reference audio signal <NUM> associated the variational embedding <NUM>. The variational embedding <NUM> may be a per-speaker variational embedding that includes an aggregation (e.g., mean) of multiple variational embeddings <NUM> encoded by the residual encoder <NUM> from reference audio signals <NUM> spoken by a same speaker.

During inference, the computing system <NUM> or the user computing device <NUM> may use the trained TTS model <NUM> to predict a mel-frequency spectrogram sequence <NUM> for a text utterance <NUM>. The TTS model <NUM> may select a variational embedding <NUM> from the storage <NUM> that represents an intended prosody/style for the text utterance <NUM>. Here, the variational embedding <NUM> may correspond to a prior variational embedding <NUM> sampled from the VAE-based residual encoder <NUM>. The TTS model <NUM> may predict the mel-frequency spectrogram sequence <NUM> for the text utterance <NUM> using the selected variational embedding <NUM>. In the example shown, the synthesizer <NUM> uses the predicted mel-frequency spectrogram sequence <NUM> to produce synthesized speech <NUM> having the intended prosody/style specified by the variational embedding <NUM>.

In a non-limiting example, an individual could train the deep neural network <NUM> to learn a per-speaker variational embedding that conveys a prosodic/style representation associated with a particular speaker. For instance, the host of the Techmeme Ride Home podcast, Brian McCullough, could train the deep neural network <NUM> on reference audio signals <NUM> that include previous episodes of the podcast along with input text sequences <NUM> corresponding to transcripts of the reference audio signals <NUM>. During training, the VAE-based residual encoder <NUM> may learn a per-speaker variational embedding <NUM> that represents the prosodic/style of Brian narrating the Ride Home podcast. Brian could then apply this per-speaker variational embedding <NUM> for use by the trained TTS model <NUM> (executing on the computing system <NUM> or the user computing device <NUM>) to predict mel-frequency spectrogram sequences <NUM> for text utterances <NUM> corresponding to a transcript for a new episode of the Ride Home podcast. The predicted mel-frequency spectrogram sequences <NUM> may be provided as input to the synthesizer <NUM> for producing synthesized speech <NUM> having Brian's unique prosody/style as specified by his per-speaker variational embedding <NUM>. That is, the resulting synthesized speech <NUM> may sound exactly like Brian's voice and possess Brian's prosody/style for narrating the episode of the Ride Home podcast. Accordingly, to air the new episode, Brian only has to provide a transcript for the episode and use the trained TTS model <NUM> to produce synthesized speech <NUM> that may be streamed to the loyal listeners (also referred to as Mutant Podcast Army) of the Ride Home podcast.

<FIG> shows a non-autoregressive neural network (e.g., the deep neural network of <FIG>) <NUM> for training the non-autoregressive TTS model <NUM>. The deep neural network <NUM> includes the VAE-based residual encoder <NUM>, a text encoder <NUM>, a duration model network <NUM>, and a spectrogram decoder <NUM>. The deep neural network <NUM> may be trained on training data that includes multiple reference audio signals <NUM> and corresponding input text sequences <NUM>. Each reference audio signal <NUM> includes a spoken utterance of human speech and the corresponding input text sequence <NUM> corresponds to a transcript of the reference audio signal <NUM>. In the example shown, the VAE-based residual encoder <NUM> is configured to encode the reference audio signal <NUM> into a variational embedding (z) <NUM>. Specifically, the residual encoder <NUM> receives a reference mel-frequency spectrogram sequence <NUM> sampled from the reference audio signal <NUM> and encodes the reference mel-frequency spectrogram sequence <NUM> into the variational embedding <NUM>, whereby the variational embedding <NUM> disentangles style/prosody information from the reference audio signal <NUM> corresponding to the spoken utterance of human speech. As such, the variational embedding <NUM> corresponds to a latent state of a reference speaker, such as affect and intent, which contributes to the prosody, emotion, and/or speaking style of the reference speaker. As used herein, the variational embedding <NUM> includes both style information and prosody information. In some examples, the variational embedding <NUM> includes a vector of numbers having a capacity represented by a number of bits in the variational embedding <NUM>. The reference mel-frequency spectrogram sequence <NUM> sampled from the reference audio signal <NUM> may have a length LR and a dimension DR. As used herein, the reference mel-frequency spectrogram sequence <NUM> includes a plurality of fixed-length reference mel-frequency spectrogram frames sampled/extracted from the reference audio signal <NUM>. Each reference mel-frequency spectrogram frame may include a duration of five milliseconds.

The VAE-based residual encoder <NUM> corresponds to a posterior network that enables unsupervised learning of latent representations (i.e. variational embeddings (z)) of speaking styles. Learning variational embeddings through the use of VAE networks provides favorable properties of disentangling, scaling, and combination for simplifying style control compared to heuristic-based systems. Here, the residual encoder <NUM> includes a phoneme-level fine-grained VAE network that includes a projection layer with a rectified linear unit (ReLU) <NUM> and a stack of lightweight convolution (LConv) blocks <NUM> having multi-headed attention. The phoneme-level fine-grained VAE network <NUM> is configured to encode spectrogram frames from the reference mel-frequency spectrogram sequence <NUM> associated with each phoneme in the input text sequence <NUM> into a respective phoneme-level variational embedding <NUM>. More specifically, the phoneme-level fine-grained VAE network may align the reference mel-frequency spectrogram sequence <NUM> with each phoneme in a sequence of phonemes extracted from the input text sequence <NUM> and encode a sequence of phoneme-level variational embeddings <NUM>. Accordingly, each phoneme-level variational embedding <NUM> in the sequence of phoneme-level variational embeddings <NUM> encoded by the phoneme-level fine-grained VAE network <NUM> encodes a respective subset of one or more spectrogram frames from the reference mel-frequency spectrogram sequence <NUM> that includes a respective phoneme in the sequence of phonemes extracted from the input text sequence <NUM>. The phoneme-level fine-grained VAE network <NUM> may initially concatenate the reference mel-frequency spectrogram sequence <NUM> with a speaker embedding ys, representing a speaker that spoke the utterance associated with the reference mel-frequency spectrogram sequence <NUM>, and sinusoidal positional embeddings <NUM> indicating phoneme position information for each phoneme in the sequence of phonemes extracted from the input text sequence <NUM>. In some examples, the residual encoder <NUM> includes positional embeddings (not shown) in place of sinusoidal positional embeddings <NUM>. Each sinusoidal positional embedding <NUM> may include a fixed-length vector that contains information about a specific position of a respective phoneme in the sequence of phonemes extracted from the input text sequence <NUM>. Subsequently, the concatenation is applied to a stack of five (<NUM>) <NUM>-headed <NUM> x <NUM> LConv blocks <NUM> to compute attention with a layer normalized encoded text sequence <NUM> output from the text encoder <NUM>.

<FIG> shows a schematic view <NUM> of an example LConv block (e.g., stack of LConv blocks <NUM>) having a gated linear unit (GLU) layer <NUM>, a LConv layer <NUM> configured to receive an output of the GLU layer <NUM>, and a feedforward (FF) layer <NUM>. The example LConv block <NUM> also includes a first residual connection <NUM> (e.g., first concatenator <NUM>) configured to concatenate an output of the LConv layer <NUM> with an input to the GLU layer <NUM>. The FF layer <NUM> is configured to receive, as input, the first residual connection <NUM> that concatenates the output of the LConv layer <NUM> with the input to the GLU layer <NUM>. The example LConv block <NUM> also includes a second residual connection <NUM> (e.g., second concatenator <NUM>) configured to concatenate the output of the FF layer <NUM> with the first residual connection <NUM>. The example LConv block <NUM> may perform FF mixing in the FF layer <NUM> using a structure ReLU(W<NUM>X + b<NUM>)W<NUM> + b<NUM> where W<NUM> increases the dimension by a factor of <NUM>.

In other implementations, the VAE-based residual encoder <NUM> includes a global VAE network that the non-autoregressive deep neural network <NUM> may employ in lieu of the phoneme-level fine-grained VAE network depicted in <FIG>. The global VAE network encodes the reference mel-frequency spectrogram sequence <NUM> into a global variational embedding <NUM> at the utterance level. Here, the global VAE network includes two stacks of lightweight convolution (LConv) blocks each having multi-headed attention. Each LConv block in the first and second stacks of the global VAE network <NUM> may include eight (<NUM>) heads. In some examples, the first stack of LConv blocks includes three (<NUM>) <NUM> x <NUM> LConv blocks and the second stack of LConv blocks following the first stack includes five <NUM> x <NUM> LConv blocks interleaved with <NUM> x <NUM> convolutions. This configuration of two stacks of LConv blocks permits the global VAE network <NUM> to successively down sample latent representations before applying global average pooling to obtain the final global variational embedding <NUM>. A projection layer may project a dimension of the global variational embedding <NUM> output from the second stack of LConv blocks. For instance, the global variational embedding <NUM> output from the second stack of LConv blocks may have a dimension of eight (<NUM>) and the projection layer may project the dimension to thirty-two (<NUM>).

With continued reference to <FIG>, the text encoder <NUM> encodes the input text sequence <NUM> into a text encoding sequence <NUM>. The text encoding sequence <NUM> includes an encoded representation of a sequence of speech units (e.g., phonemes) extracted from the input text sequence <NUM>. The input text sequence <NUM> may include words each having one or more phonemes, silences at all word boundaries, and punctuation marks. Thus, the input text sequence <NUM> includes a sequence of phonemes and the text encoder <NUM> may receive, from a token embedding look-up table <NUM>, a respective token embedding for each phoneme in the sequence of phonemes. Here, the respective token embedding includes a phoneme embedding. However, in other examples, the token embedding look-up table <NUM> may obtain token embeddings for other types of speech inputs associated with the input text sequence <NUM> instead of phonemes, such as, without limitation, sub-phonemes (e.g., senomes), graphemes, word pieces, or words in the utterance. After receiving the respective token embedding of each phoneme in the sequence of phonemes, the text encoder <NUM> uses an encoder pre-net neural network <NUM> to process each respective token embedding to generate a respective transformed embedding <NUM> of each phoneme. Thereafter, a bank of convolutional (Conv) blocks <NUM> may process the respective transformed embeddings <NUM> to generate convolution outputs <NUM>. In some examples, the bank of Conv blocks <NUM> includes three (<NUM>) identical <NUM> x <NUM> Conv blocks. <FIG> shows a schematic view <NUM> an example Conv block having a Conv layer <NUM>, a batch normalization layer <NUM>, and a dropout layer <NUM>. During training, the batch normalization layer <NUM> may apply batch normalization to reduce internal covariate shift. The dropout layer <NUM> may reduce overfitting. Finally, a stack of self-attention blocks <NUM> process the convolution outputs 213to generate the encoded text sequence <NUM>. In the example shown, the stack of self-attention blocks <NUM> includes six (<NUM>) transformer blocks. In other examples, the self-attention blocks <NUM> may include LConv blocks in lieu of transformer blocks.

Notably, since each convolution output <NUM> flows through the stack of self-attention blocks <NUM> simultaneously, the stack of self-attention blocks <NUM> have no knowledge of position/order of each phoneme in the input text utterance <NUM>. Thus, in some examples, sinusoidal positional embeddings <NUM> are combined with the convolution output <NUM> to inject necessary position information indicating the order of each phoneme in the input text sequence <NUM>. In other examples, encoded positional embeddings are used in place of the sinusoidal positional embeddings <NUM>. By contrast, autoregressive encoders that incorporate recurrent neural networks (RNNs) inherently take the order of each phoneme into account since each phoneme is parsed from the input text sequence in a sequential matter. However, the text encoder <NUM> integrating the stack of self-attention blocks <NUM> that employ multi-head self-attention avoids the recurrence of auto-regressive encoders to result in drastically reduced training time, and theoretically, capture longer dependences in the input text sequence <NUM>.

With continued reference to <FIG>, a concatenator <NUM> concatenates the variational embedding <NUM> from the residual encoder <NUM>, the encoded text sequence <NUM> output from the text encoder <NUM>, and a reference speaker embedding ys representing a speaker identity <NUM> of a reference speaker that uttered the reference audio signal into a concatenation <NUM>. The duration model network <NUM> receives the concatenation <NUM> and is configured to generate an upsampled output <NUM> that specifies a number of frames for the encoded text sequence <NUM> from the concatenation <NUM>. In some implementations, the duration model network <NUM> includes a stack of self-attention blocks <NUM> followed by two independent small convolution blocks <NUM>, <NUM> and a projection with a softplus activation <NUM>. In the example shown, the stack of self-attention blocks includes four (<NUM>) <NUM> x <NUM> LConv blocks <NUM>. As described above with reference to <FIG>, each LConv block in the stack of self-attention blocks <NUM> may include a GLU unit <NUM>, a LConv layer <NUM>, and a FF layer <NUM> with residual connections. The stack of self-attention blocks <NUM> generates a sequence representation V based on the concatenation <NUM> of the encoded text sequence <NUM>, variational embedding <NUM>, and reference speaker embedding ys. Here, the sequence representation V represents a sequence of M x <NUM> column vectors (e.g., V = {v<NUM>,.

In some implementations, the first convolution block <NUM> generates an output <NUM> and the second convolution block <NUM> generates an output <NUM> from the sequence representation V. The convolution blocks <NUM>, <NUM> may each include a <NUM> x <NUM> Conv block that has a kernel-width of <NUM> and output dimension of <NUM>. The projection with the softplus activation <NUM> may predict a phoneme duration <NUM> (e.g., {d<NUM>,. , dK}) for each phoneme represented by the encoded text sequence <NUM>. Here, the softplus activation <NUM> receives, as input, the sequence representation V to predict the phoneme durations <NUM>. The duration model network <NUM> computes a global phoneme duration loss <NUM> (e.g., L1 loss term <NUM>) between the predicted phoneme durations <NUM> and a target average duration <NUM> represented by: <MAT> where <IMG> represents the global phoneme duration loss <NUM> (e.g., L1 loss term <NUM>), K represents the total number of phonemes (e.g., tokens) of the input text sequence <NUM>, dk represents a phoneme duration <NUM> for a particular phoneme k from the total number of phonemes K, and T represents the total target frame duration from the reference mel-frequency spectrogram sequence <NUM>.

In some examples, training the TTS model <NUM> is based on the global phoneme duration loss <NUM> from Equation <NUM>. The individual target duration for each phoneme is unknown, thus, the duration model network <NUM> determines the target average duration <NUM> based on a proportion of the T total frame duration from the entire reference mel-frequency spectrogram sequence <NUM> and K total number of phonemes (e.g., tokens) in the input text sequence <NUM>. That is, the target average duration <NUM> is the average duration for all phonemes using the reference mel-frequency spectrogram sequence <NUM> and input text sequence <NUM>. The L1 loss term <NUM> is then determined between the predicted phoneme durations <NUM> and the target average duration <NUM> determined using the reference mel-frequency spectrogram sequence <NUM> and input text sequence <NUM>. As such, the duration model network <NUM> learns to predict phoneme durations <NUM> in an unsupervised manner without the use of supervised phoneme duration labels provided from an external aligner. While external aligners are capable of providing reasonable alignments between phonemes and mel-spectral frames, phoneme duration rounding is required by a length regulator to upsample phonemes in the input text sequence <NUM> according to their duration which leads to rounding errors that may persist. In some instances, using supervised duration labels from the external aligner during training and using predicted durations during inference creates phoneme duration discrepancies between training the TTS model <NUM> (<FIG>) and inference of the TTS model <NUM> (<FIG>). Moreover, such rounding operations are not differentiable, and thus, an error gradient is unable to propagate through the duration model network <NUM>.

The duration model network <NUM> includes a matrix generator <NUM> to define token boundaries <MAT> from the predicted phoneme durations <NUM>. The matrix generator <NUM> determines the token boundaries (e.g., phoneme boundaries) from the predicted phoneme durations <NUM> as follows: <MAT> <MAT>.

In Equation <NUM>, sk represents the start of a token boundary (also referred to herein as start boundary sk) for a particular phoneme k. In Equation <NUM>, ek represents the end of a token boundary (also referred to herein as end boundary ek) for a particular phoneme k. The matrix generator <NUM>, using the start and end boundaries sk, ek from Equations <NUM> and <NUM>, maps the token boundaries into two token boundary grid matrices S and E as follows. <MAT> <MAT>.

Equation <NUM> maps each start boundary sk to the Stk grid matrix giving the distances to the start boundaries sk of token k at time t. Equation <NUM> maps each end boundary ek to the Etk grid matrix giving the distances to the end boundaries ek of token k at time t. The matrix generator <NUM> generates the start and end boundaries sk, ek (also referred to collectively as token boundaries sk, ek) and maps the token boundaries sk, ek to a start token boundary grid matrix S and an end token boundary grid matrix E (referred to collectively as grid matrices <NUM>) respectively. Here, the grid matrices <NUM> are of size T x K where K represents the number of phonemes in the input text sequence <NUM> and T represents the number of frames in the reference mel-frequency spectrogram sequence <NUM> and the total frame duration. The matrix generator <NUM> may map, based on a number of phonemes represented by the encoded text sequence <NUM> and a number of reference frames in the reference mel-frequency spectrogram sequence <NUM>, the respective start and end boundaries sk, ek generated for each phoneme in the sequence of phonemes into respective grid matrices <NUM> (e.g., start token boundary grid matrix S and end token boundary grid matrix E).

The duration model network <NUM> includes a first function <NUM> to learn an interval representation matrix W. In some implementations, the first function <NUM> includes two (<NUM>) projection layers with bias and Swish-activation. In the example shown, both projections of the first function <NUM> project and output with a dimension of <NUM> (i.e., P = <NUM>). The first function <NUM> receives a concatenation <NUM>, as input, from a concatenator <NUM>. The concatenator <NUM> concatenates the output <NUM> from the convolution block <NUM> and the grid matrices <NUM> to generate the concatenation <NUM>. The first function <NUM> projects an output <NUM> using the concatenation <NUM>. Subsequently, the first function <NUM> generates the interval representation matrix W (e.g., T x K attention matrix) from a projection with a softplus activation <NUM> of the output <NUM>. The interval representation matrix W may be learned based on the respective grid matrices <NUM> mapped from the start and end boundaries sk, ek as follows: <MAT>.

Equation <NUM> computes the interval representation W (e.g., T x K attention matrix) using a Softmax function and multi-layer perceptron-based (MLP) learnable function of the grid matrices <NUM> (e.g., the start token boundary grid matrix S and the end token boundary grid matrix E) and the sequence representation V. The MLP learnable function includes a third projection layer with an output dimension of <NUM> which is fed to the Softmax activation function and the Conv1D(V) includes a kernel-width of <NUM>, an output dimension of <NUM>, batch normalization, and Swish-activation. Here, the (k, t)-th element of the grid matrices <NUM> gives an attention probability between the k-th token (e.g., phoneme) and the t-th frame and W() is a learnable function mapping the grid matrices <NUM> and the sequence representation V with a small 1D convolution layer.

The duration model network <NUM> may learn an auxiliary attention context tensor C (e.g., C = [C<NUM>,. , CP]) using a second function <NUM> conditioned on the sequence representation V. Here, Cp includes a T x K matrix from the auxiliary attention context tensor C. The auxiliary attention context tensor C may include auxiliary multi-headed attention-like information for the spectrogram decoder <NUM>. A concatenator <NUM> concatenates the grid matrices <NUM> and the output <NUM> to generate a concatenation <NUM>. The second function <NUM> may include two (<NUM>) projection layers with bias and Swish-activation. In the example shown, the projection of the second function <NUM> projects an output with a dimension of <NUM> (i.e., P = <NUM>). The second function <NUM> receives, as input, the concatenation <NUM> to generate the auxiliary attention context tensor C based on the respective grid matrices <NUM> mapped from the start and end boundaries sk, ek and the sequence representation V as follows: <MAT>.

Equation <NUM> computes the auxiliary attention context tensor C using a multi-layer perceptron-based (MLP) learnable function of the grid matrices <NUM> (e.g., the start token boundary grid matrix S and the end token boundary grid matrix E) and the sequence representation V. The auxiliary attention context tensor C may help smooth the optimization and converge the stochastic gradient descent (SGD). The duration model network <NUM> may upsample the sequence representation V into the upsampled output <NUM> (e.g., O = {o<NUM>,. , oT}) with a number of frames. Here, the number of frames of the upsampled output <NUM> corresponds to the predicted length of the predicted mel-frequency spectrogram <NUM> determined by the predicted phoneme duration <NUM> of the corresponding input text sequence <NUM>. Upsampling the sequence representation V into the upsampled output <NUM> includes determining a product <NUM> of the interval representation matrix W and the sequence representation V using a multiplier <NUM>.

The duration model network <NUM> determines an Einstein summation (einsum) <NUM> of the interval representation matrix W and the auxiliary attention context tensor C using an einsum operator <NUM>. A projection <NUM> projects the einsum <NUM> into a projected output <NUM> that is added to the product <NUM> at an adder <NUM> to generate the upsampled output <NUM>. Here, the upsampled output <NUM> may be represented as follows: <MAT> where O denotes the upsampled output <NUM>, ⊙ denotes element-wise multiplication, <NUM>k represents a K x <NUM> column vector that includes elements all equal to <NUM>, and A is a P x M projection matrix.

With continued reference to <FIG>, the spectrogram decoder <NUM> is configured to receive, as input, the upsampled output <NUM> of the duration model network <NUM>, and generate, as output, one or more predicted mel-frequency spectrogram sequences <NUM> for the input text sequence <NUM>. The spectrogram decoder <NUM> may include a stack of multiple self-attention blocks <NUM>, 262a-n with multi-headed attention. In some examples, the spectrogram decoder <NUM> includes six (<NUM>) eight-headed <NUM> x <NUM> LConv blocks with <NUM> dropout. The spectrogram decoder <NUM> may include more or less than six LConv blocks. As described above with reference to <FIG>, each LConv block in the stack of self-attention blocks <NUM> may include a GLU unit <NUM>, a LConv layer <NUM>, and a FF layer <NUM> with residual connections. In other examples, each self-attention block <NUM> in the stack includes an identical transformer block.

In some implementations, the spectrogram decoder <NUM> generates a respective predicted mel-frequency spectrogram sequence <NUM>, 302a-n as output from each self-attention block <NUM> in the stack of self-attention blocks <NUM>, 262a-n. The network <NUM> may be trained so that a number of frames in each respective predicted mel-frequency spectrogram sequence <NUM> is equal to a number of frames in the reference mel-frequency spectrogram sequence <NUM> input to the residual encoder <NUM>. In the example shown, each self-attention block 262a-n is paired with a corresponding projection layer 264a-n that projects an output <NUM> from the self-attention block <NUM> to generate the respective predicted mel-frequency spectrogram sequence 302a-n having a dimension that matches a dimension of the reference mel-frequency spectrogram sequence <NUM>. In some examples, the projection layer <NUM> projects a <NUM>-bin predicted mel-frequency spectrogram sequence <NUM>. By predicting multiple mel-frequency spectrogram sequences 302a-n, the non-autoregressive neural network <NUM> may be trained using a soft dynamic time warping (soft-DTW) loss. That is, because the predicted mel-frequency spectrogram sequence <NUM> may be of a different length (e.g., number of frames) from the reference mel-frequency spectrogram sequence <NUM> the spectrogram decoder <NUM> cannot determine a regular Laplace loss. Rather, the spectrogram decoder <NUM> determines a soft-DTW loss between the reference mel-frequency spectrogram sequence <NUM> and the predicted mel-frequency spectrogram sequence <NUM> that may include different lengths. In particular, for each respective predicted mel-frequency spectrogram sequence 302a-n, the spectrogram decoder <NUM> determines a respective spectrogram loss <NUM>, 270a-n based on the corresponding predicted mel-frequency spectrogram sequence <NUM> and the reference mel-frequency spectrogram sequence <NUM>. The respective spectrogram loss <NUM> may include a soft-DTW loss term determined by the recursion as follows: <MAT>.

In Equation <NUM>, ri,j represents the distance between the reference mel-frequency spectrogram sequence frames from <NUM> to i and the predicted mel-frequency spectrogram sequence frames from <NUM> to j with the best alignment. Here, minγ includes a generalized minimum operation with a smoothing parameter γ, warp includes a warp penalty, and xi and xj are the reference mel-frequency spectrogram frames and the predicted mel-frequency spectrogram sequence frames in time i and j, respectively.

The soft-DTW loss term recursion is computationally intensive and may include a complexity of O(T<NUM>), a diagnal band width fixed at <NUM>, a warp penalty of <NUM>, and a smoothing parameter γ of <NUM>. For instance, a first spectrogram loss 270a may be determined based on a first predicted mel-frequency spectrogram sequence 302a and the reference mel-frequency spectrogram sequence <NUM>. Here, the first predicted mel-frequency spectrogram sequence 302a and the reference mel-frequency spectrogram sequence <NUM> may be the same or different lengths. A second spectrogram loss 270b may be determined based on a first predicted mel-frequency spectrogram sequence 302a and the reference mel-frequency spectrogram sequence <NUM>, and so on until all of the respective spectrogram losses 270a-n are iteratively determined the predicted mel-frequency spectrogram sequence 302a-n. The spectrogram losses <NUM>, including the soft-DTW loss terms, may be aggregated to generate a final soft-DTW loss <NUM>. The final soft-DTW loss <NUM> may correspond to an iterative soft-DTW loss term <NUM>. The final soft-DTW loss <NUM> may be determined from any combination of predicted mel-frequency spectrogram sequences <NUM> and the reference mel-frequency spectrogram sequences <NUM> of the same and/or different lengths as follows: <MAT>.

In Equation <NUM>, <IMG> includes the final soft-DTW loss <NUM>, <MAT> includes the soft-DTW L1 spectrogram reconstruction loss for the l-th iteration in the spectrogram decoder, <IMG> includes the average duration L1 loss, and DKL includes the KL divergence between prior and posterior from the residual encoder. Training the deep neural network <NUM> aims to minimize the final soft-DTW loss <NUM> to reduce the difference between phoneme durations of the predicted mel-frequency spectrogram sequences <NUM> and the reference mel-frequency spectrogram sequences <NUM>. By minimizing the final soft-DTW loss <NUM>, the trained TTS model <NUM> may generate predicted mel-frequency spectrogram sequences <NUM> that includes the intended prosody/style based on the reference mel-frequency spectrogram sequences <NUM>. Aggregating the spectrogram losses 270a-n may include summing the spectrogram losses 270a-n to obtain the final soft-DTW loss <NUM>. Optionally, aggregating the spectrogram losses <NUM> may include averaging the spectrogram losses <NUM>.

The deep neural network <NUM> may be trained so that a number of frames in each respective predicted mel-frequency spectrogram sequence <NUM> is equal to a number of frames in the reference mel-frequency spectrogram sequence <NUM> input to the residual encoder <NUM>. Moreover, the deep neural network <NUM> is trained so that data associated with the reference and predicted mel-frequency spectrogram sequences <NUM>, <NUM> substantially match one another. The predicted mel-frequency spectrogram sequence <NUM> may implicitly provide a prosodic/style representation of the reference audio signal <NUM>.

<FIG> shows an example of the non-autoregressive TTS model <NUM> trained by the non-autoregressive deep neural network <NUM> of <FIG>. Specifically, <FIG> depicts the TTS model <NUM> using a selected variational embedding <NUM> to predict a mel-frequency spectrogram sequence <NUM> for an input text utterance <NUM>, whereby the selected variational embedding <NUM> represents an intended prosody/style for the text utterance <NUM>. During inference, the trained TTS model <NUM> executes on the computing system <NUM> or the user computing device <NUM> and may use the selected variational embedding <NUM> to predict the corresponding mel-spectrogram sequence <NUM> for the input text utterance <NUM>. Here, the TTS model <NUM> selects a variational embedding <NUM> from the storage <NUM> that represents an intended prosody/style for the text utterance <NUM>. In some examples, a user provides a user input indication indicating selection of the intended prosody/style the user wants the resulting synthesized speech <NUM> to convey for the text utterance <NUM> and the TTS model <NUM> selects the appropriate variational embedding <NUM> from the data storage <NUM> that represents the intended prosody/style. In these examples, the intended prosody/style may be selected by the user by indicating a speaker identity <NUM> associated with a particular speaker who speaks with the intended prosody/style and/or specifying a particular prosodic vertical (e.g., newscaster, sportscaster, etc.,) that corresponds to the intended prosody/style. The selected variational embedding <NUM> may correspond to a prior variational embedding <NUM> sampled from the VAE-based residual encoder <NUM>. The trained TTS model <NUM> generates synthesized speech <NUM>, using a synthesizer <NUM>, with the intended prosody/style (e.g., selected variational embedding <NUM>) for the respective input text utterance <NUM>. That is, the selected variational embedding <NUM> may include an intended prosody/style (e.g., newscaster, sportscaster, etc.) stored on the residual encoder <NUM>. The selected variational embedding <NUM> conveys the intended prosody/style via synthesized speech <NUM> for the input text utterance <NUM>.

In additional implementations, the trained TTS model <NUM> employs the residual encoder <NUM> during inference to extract/predict a variational embedding <NUM> on the fly for use in predicting a mel-frequency spectrogram sequence <NUM> for the input text utterance <NUM>. For instance, the residual encoder <NUM> may receive a reference audio signal <NUM> (<FIG>) uttered by a human user that conveys the intended prosody/style (e.g., "Say it like this") and extract/predict a corresponding variational embedding <NUM> that represents the intended prosody/style. Thereafter, the trained TTS model <NUM> may use the variational embedding <NUM> to effectively transfer the intended prosody/style conveyed by the reference audio signal <NUM> to the mel-frequency spectrogram sequence <NUM> predicted for the input text utterance <NUM>. Accordingly, the input text utterance <NUM> to be synthesized into expressive speech <NUM> and the reference audio signal <NUM> conveying the intended prosody/style to be transferred to the expressive speech <NUM> may include different linguistic content.

In particular, the text encoder <NUM> encodes a sequence of phonemes extracted from the text utterance <NUM> into an encoded text sequence <NUM>. The text encoder <NUM> may receive, from the token embedding look-up table <NUM>, a respective token embedding for each phoneme in the sequence of phonemes extracted from the text utterance <NUM>. After receiving the respective token embedding of each phoneme in the sequence of phonemes extracted from the text utterance <NUM>, the text encoder <NUM> uses the encoder pre-net neural network <NUM> to process each respective token embedding to generate a respective transformed embedding <NUM> of each phoneme. Thereafter, the bank of Conv blocks <NUM> (e.g., three (<NUM>) identical <NUM> x <NUM> Conv blocks) processes the respective transformed embeddings <NUM> to generate convolution outputs <NUM>. Finally, a stack of self-attention blocks <NUM> process the convolution outputs 213to generate the encoded text sequence <NUM>. In the example shown, the stack of self-attention blocks <NUM> includes six (<NUM>) transformer blocks. In other examples, the self-attention blocks <NUM> may include LConv blocks in lieu of transformer blocks. Notably, since each convolution output <NUM> flows through the stack of self-attention blocks <NUM> simultaneously, the stack of self-attention blocks <NUM> have no knowledge of position/order of each phoneme in the input text utterance. Thus, in some examples, sinusoidal positional embeddings <NUM> are combined with the convolution output <NUM> to inject necessary position information indicating the order of each phoneme in the input text sequence <NUM>. In other examples, encoded positional embeddings are used in place of the sinusoidal positional embeddings <NUM>.

With continued reference to <FIG>, the concatenator <NUM> concatenates the selected variational embedding <NUM>, the encoded text sequence <NUM>, and optionally, a reference speaker embedding ys to generate the concatenation <NUM>. Here, the reference speaker embedding ys may represent the speaker identity <NUM> of a reference speaker that uttered one or more reference audio signal <NUM> associated with the selected variational embedding <NUM> or the speaker identity <NUM> of some other reference speaker having voice characteristics to be conveyed in the resulting synthesized speech <NUM>. The duration model network <NUM> is configured to decode the concatenation <NUM> of the encoded text sequence <NUM>, the selected variational embedding <NUM>, and the reference speaker embedding ys to generate an upsampled output <NUM> of the duration model network <NUM> for predicting a phoneme duration <NUM> for each phoneme in the sequence of phonemes in the input text utterance <NUM>.

The duration model network <NUM> is configured to generate the upsampled output <NUM> that specifies a number of frames for the encoded text sequence <NUM> from the concatenation <NUM>. In some implementations, the duration model network <NUM> includes a stack of self-attention blocks <NUM> followed by two independent small convolution blocks <NUM>, <NUM> and a projection with a softplus activation <NUM>. In the example shown, the stack of self-attention blocks includes four (<NUM>) <NUM> x <NUM> LConv blocks <NUM>. As described above with reference to <FIG>, each LConv block in the stack of self-attention blocks <NUM> may include a GLU unit <NUM>, a LConv layer <NUM>, and a FF layer <NUM> with residual connections. The stack of self-attention blocks <NUM> generates a sequence representation V based on the concatenation <NUM> of the encoded text sequence <NUM>, variational embedding <NUM>, and reference speaker embedding ys.

In some implementations, the first convolution block <NUM> generates an output <NUM> and the second convolution block <NUM> generates an output <NUM> from the sequence representation V. The convolution blocks <NUM>, <NUM> may include a <NUM> x <NUM> Conv block that has a kernel-width of <NUM> and output dimension of <NUM>. The duration model network <NUM> includes a projection with a softplus activation <NUM> to predict a phoneme duration <NUM> (e.g., {d<NUM>,. , dK}) for each phoneme represented by the encoded text sequence <NUM>. Here, the softplus activation <NUM> receives, as input, the sequence representation V to predict the phoneme durations <NUM>.

The duration model network <NUM> includes a matrix generator <NUM> to define token boundaries <MAT> from the predicted phoneme durations <NUM> as described above with reference to <FIG>. The duration model network <NUM> includes a first function <NUM> to learn an interval representation matrix W. In some implementations, the first function <NUM> includes two (<NUM>) projection layers with bias and Swish-activation. In the example shown, both projections of the first function <NUM> project and output with a dimension of <NUM> (i.e., P = <NUM>). The first function <NUM> receives a concatenation <NUM>, as input, from the concatenator <NUM>. The concatenator <NUM> concatenates the output <NUM> from the convolution block <NUM> and the grid matrices <NUM> to generate the concatenation <NUM>. Subsequently, the first function <NUM> generates the interval representation matrix W (e.g., T x K attention matrix) from a projection with a softplus activation <NUM> of the output <NUM> using Equation <NUM>.

The duration model network <NUM> may learn an auxiliary attention context tensor C (e.g., C = [C<NUM>,. , CP]) using a second function <NUM> conditioned on the sequence representation V. Here, Cp is a T x K matrix from the auxiliary attention content tensor C. The auxiliary attention context tensor C may include auxiliary multi-headed attention like information for the spectrogram decoder <NUM>. A concatenator <NUM> concatenates the grid matrices <NUM> and the output <NUM> to generate a concatenation <NUM>. The second function <NUM> may include by two (<NUM>) projection layers with bias and Swish-activation. The second function <NUM> receives, as input, the concatenation <NUM> to generate the auxiliary attention context tensor C based on the respective grid matrices <NUM> mapped from the start and end boundaries sk, ek and the sequence representation V using Equation <NUM>.

The duration model network <NUM> may upsample the sequence representation V into the upsampled output <NUM> (e.g., O = {o<NUM>,. , oT}) with a number of frames. Here, the number of frames of the upsampled output corresponds to the predicted length of the predicted mel-frequency spectrogram <NUM> determined by the predicted phoneme duration <NUM> of the corresponding input text sequence <NUM>. Upsampling the sequence representation V into the upsampled output <NUM> is based on determining a product <NUM> of the interval representation W and the sequence representation V using the multiplier <NUM>, and determining an einsum <NUM> of the interval representation matrix W and the auxiliary attention context sensor C using the einsum operator <NUM>. A projection <NUM> projects the einsum <NUM> into a projected output <NUM> that the adder <NUM> sums with the product <NUM> to generate the upsampled output <NUM> represented by Equation <NUM>.

The spectrogram decoder <NUM> is configured to generate, based on the upsampled output <NUM> of the duration model network <NUM> and the predicted phoneme durations <NUM>, a predicted mel-frequency spectrogram sequence <NUM> for the text utterance <NUM>. Here, the predicted mel-frequency spectrogram sequence <NUM> has the intended prosody/style specified by the selected variational embedding <NUM>. The predicted mel-frequency spectrogram sequence <NUM> for the text utterance <NUM> is based on the auxiliary attention context tensor C, the sequence representation V interval representation w, and the upsampled output <NUM> of the duration model network <NUM> into the number of frames.

The spectrogram decoder <NUM> generates a respective predicted mel-frequency spectrogram sequence <NUM> as output from the last self-attention block <NUM> in the stack of self-attention blocks 262a-n. Here, each self-attention block <NUM> in the stack of self-attention blocks <NUM> of the spectrogram decoder <NUM> includes one of an identical LConv block or an identical transformer block. In some examples, the spectrogram decoder <NUM> includes six (<NUM>) eight-headed <NUM> x <NUM> LConv blocks with <NUM> dropout. The output of the last feedforward (FF) layer <NUM> (<FIG>) for each self-attention block <NUM> is provided as input to the subsequent self-attention block <NUM>. That is, the GLU unit <NUM> (<FIG>) and the LConv layer <NUM> (<FIG>) of the first self-attention block 262a in the stack of self-attention blocks 262a-n processes the output <NUM> from the duration model network <NUM> and the predicted phoneme durations <NUM> and output from the last FF layer <NUM> of the first self-attention block 262a is provided as input to the subsequent second self-attention block 262b in the stack of self-attention blocks <NUM>. The output of the last FF layer of each self-attention block <NUM> is provided as input to the subsequent self-attention block <NUM> until the last self-attention block 262n in the stack of self-attention blocks <NUM> is reached. The last self-attention block 262n (e.g., the sixth self-attention block <NUM>) in the stack of self-attention blocks <NUM> is paired with a corresponding projection layer <NUM> that projects an output <NUM> from the last self-attention block <NUM> to generate the respective predicted mel-frequency spectrogram sequence <NUM>.

The predicted mel-frequency spectrogram sequence <NUM> generated by the spectrogram decoder <NUM> corresponds to the input text utterance <NUM> and conveys the intended prosody/style indicated by the selected variational embedding <NUM>. The trained TTS model <NUM> provides the predicted mel-frequency spectrogram sequence <NUM> for the input text utterance <NUM> to the synthesizer <NUM> for conversion into a time-domain audio waveform indicative of synthesized speech <NUM>. The synthesized speech <NUM> may be audibly output as spoken representation of the input text utterance <NUM> including the intended prosody/style as indicated by the selected variational embedding <NUM>.

<FIG> is a flowchart of an exemplary arrangement of operations for a computer-implemented method <NUM> for training a non-autoregressive text-to-speech (TTS) model. At operation <NUM>, the method <NUM> includes obtaining a sequence representation V of an encoded text sequence <NUM> concatenated with a variational embedding <NUM>. At operation <NUM>, using a duration model network the method <NUM> includes predicting, based on the sequence representation V, a phoneme duration <NUM> for each phoneme represented by the encoded text sequence <NUM>. Based on the predicted phoneme durations <NUM>, the method <NUM> includes, at operation <NUM>, learning, using a first function <NUM> conditioned on the sequence representation V, an interval representation matrix W. At operation <NUM>, the method <NUM> includes learning, using a second function <NUM> conditioned on the sequence representation V, an auxiliary attention context representation C.

At operation <NUM>, the method <NUM> includes upsampling, using the interval representation matrix W and the auxiliary attention context representation C, the sequence representation V into an upsampled output <NUM> specifying a number of frames. At operation <NUM>, the method <NUM> includes generating, as output from a spectrogram decoder <NUM> that includes a stack of one or more self-attention blocks <NUM>, 262a-n, based on the upsampled output <NUM>, one or more predicted mel-frequency spectrogram sequences <NUM> for the encoded text sequence <NUM>. At operation <NUM>, the method <NUM> includes determining a final spectrogram loss <NUM> based on the one or more predicted mel-frequency spectrogram sequences <NUM> and a reference mel-frequency spectrogram sequence <NUM>. At operation <NUM>, the method <NUM> includes training the TTS model <NUM> based on the final spectrogram loss <NUM>.

For example, it may be implemented as a standard server 700a or multiple times in a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

Claim 1:
A computer-implemented method (<NUM>) when executed on data processing hardware (<NUM>) causes the data processing hardware (<NUM>) to perform operations for training a non-autoregressive text-to-speech model (<NUM>), the operations comprising:
obtaining a sequence representation (<NUM>) of an encoded text sequence (<NUM>) concatenated with a variational embedding (<NUM>);
using a duration model network (<NUM>):
predicting, based on the sequence representation (<NUM>), a phoneme duration (<NUM>) for each phoneme represented by the encoded text sequence (<NUM>);
based on the predicted phoneme durations (<NUM>):
learning, using a first function (<NUM>) conditioned on the sequence representation (<NUM>), an interval representation matrix; and
learning, using a second function (<NUM>) conditioned on the sequence representation (<NUM>), an auxiliary attention context representation;
upsampling, using the interval representation and the auxiliary attention context representation, the sequence representation (<NUM>) into an upsampled output (<NUM>) specifying a number of frames; and
generating, as output from a spectrogram decoder (<NUM>) comprising a stack of one or more self-attention blocks, based on the upsampled output (<NUM>), one or more predicted mel-frequency spectrogram sequences (<NUM>) for the encoded text sequence (<NUM>);
determining a final spectrogram loss (<NUM>) based on the one or more predicted mel-frequency spectrogram sequences (<NUM>) and a reference mel-frequency spectrogram sequence (<NUM>); and
training the text-to-speech model (<NUM>) based on the final spectrogram loss (<NUM>).