Patent Description:
Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input For instance, neural networks may convert input text to output speech.

Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from a previous time step in computing an output at a current time step. An example of a recurrent neural network is a long short term (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more cells that each include an input gate, a forget gate, and an output gate that allows the cell to store previous states for the cell, e.g., for use in generating a current activation or to be provided to other components of the LSTM neural network.

<NPL>, describes embedding capacity as a unified method of analysing the behavior of latent variable models of speech, comparing existing heuristic (non-variational) methods to variational methods that are able to explicitly constrain capacity using an upper bound on representational mutual information.

<NPL>, describes the introduction of a Variational Autoencoder (VAE) to an end-to-end speech synthesis model, to learn the latent representation of speaking styles in an unsupervised manner. The style representation learned through VAE shows properties such as disentangling, scaling, and combination, for style control. Style transfer can be achieved in this framework by first inferring style representation through the recognition network of VAE, then feeding it into TTS network to guide the style in synthesising speech. To avoid Kullback-Leibler (KL) divergence collapse in training, several techniques are adopted.

<NPL>, describes intuitions and theoretical assessments of the emergence of disentangled representation in variational autoencoders. Taking a rate-distortion theory perspective, circumstances under which representations aligned with the underlying generative factors of variation of data emerge when optimising the modified ELBO bound in β-VAE, as training progresses is shown. From these insights, a modification to the training regime of β-VAE is proposed.

One aspect of the invention provides a computer-implemented method in accordance with claim <NUM>. Further preferred embodiments are provided in the dependent claims. The specified capacity of the variational embedding is based on an adjustable variational bound of the variational posterior. In these examples, the adjustable variational bound includes an adjustable KL term that provides an upper bound on the variational embedding. Optionally, the adjustable variational bound may include a tunable KL weight that provides an upper bound on the variational embedding. Increasing the adjustable variational bound increases the specified capacity of the variational embedding while decreasing the adjustable variational bound decreases the specified capacity of the variational embedding.

Another aspect of the disclosure provides a system in accordance with claim <NUM> as well as a computer-readable medium in accordance with claim <NUM>.

Like reference symbols in the various drawings indicate like elements,.

The synthesis of realistic human speech is an underdetermined problem in that a same text input has an infinite number of reasonable spoken realizations. While End-to-end neural network-based approaches are advancing to match human performance for short assistant-like utterances, neural network models are sometimes viewed as less interpretable or controllable than more conventional models that include multiple processing steps each operating on refined linguistic or phonetic representations. Accordingly, implementations herein are directed toward producing end-to-end models that can probabilistically model and/or directly control remaining variability in synthesized speech.

Sources of variability include prosodic characteristics of intonation, stress, rhythm, and style, as well as speaker and channel characteristics. The prosodic characteristics of a spoken utterance convey linguistic, semantic, and emotional meaning beyond what is present in a lexical representation (e.g., a transcript of the spoken utterance). Providing an ability to transfer these characteristics from one utterance to another enables users to control how synthesized speech sounds by using their own voice (e.g., "say it like this"), rather than having to manipulate complicated acoustic or linguistic parameters by hand. Implementations herein are further directed toward methods that enable sampling from a distribution over likely prosodic realizations of an utterance in order to allow users to experience the variety present in natural speech. Implementations herein may include.

Referring to <FIG>, in some implementations, an example text-to-speech (TTS) conversion system <NUM> includes a subsystem <NUM> that is configured to receive input text <NUM> as an input and to process the input text <NUM> to generate speech <NUM> as an output. The input text <NUM> includes a sequence of characters in a particular natural language. The sequence of characters may include alphabet letters, numbers, punctuation marks, and/or other special characters. The input text <NUM> can be a sequence of characters of varying lengths. The text-to-speech conversion system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented For instance, the system <NUM> may execute on the computer system <NUM> of <FIG>.

The system <NUM> may include a user interface <NUM> that allows users to input text <NUM> for conversion into synthesized speech and/or provide reference speech <NUM> (<FIG>) using their own voice so that a variational embedding associated with the reference speech can control how speech synthesized from input text sounds. The user interface <NUM> may also allow a user to select a target speaker that is different than a voice of the user providing the reference speech <NUM> so that the synthesized speech sounds like the target speaker, but having the prosody/style conveyed in the reference speech uttered by the user. The user interface <NUM> may further permit the user to select/sample from a distribution over likely prosodic realizations of an utterance in order to allow users to experience the variety present in natural speech.

To process the input text <NUM>, the subsystem <NUM> is configured to interact with an end-to-end text-to-speech model <NUM> that includes a sequence-to-sequence recurrent neural network <NUM> (hereafter "seq2seq network <NUM>"), a post-processing neural network <NUM>, and a waveform synthesizer <NUM>.

After the subsystem <NUM> receives input text <NUM> that includes a sequence of characters in a particular natural language, the subsystem <NUM> provides the sequence of characters as input to the seq2seq network <NUM>. The seq2seq network <NUM> is configured to receive the sequence of characters from the subsystem <NUM> and to process the sequence of characters to generate a spectrogram of a verbal utterance of the sequence of characters in the particular natural language.

In particular, the seq2seq network <NUM> processes the sequence of characters using (i) an encoder neural network <NUM>, which includes an encoder pre-net neural network <NUM> and an encoder CBHG neural network <NUM>, and (ii) an attention-based decoder recurrent neural network <NUM>. Each character in the sequence of characters can be represented as a one-hot vector and embedded into a continuous vector. That is, the subsystem <NUM> can represent each character in the sequence as a one-hot vector and then generate an embedding, i.e., a vector or other ordered collection of numeric values, of the character before providing the sequence as input to the seq2seq network <NUM>.

The encoder pre-net neural network <NUM> is configured to receive a respective embedding of each character in the sequence and process the respective embedding of each character to generate a transformed embedding of the character. For example, the encoder pre-net neural network <NUM> can apply a set of non-linear transformations to each embedding to generate a transformed embedding. In some cases, the encoder pre-net neural network <NUM><NUM> includes a bottleneck neural network layer with dropout to increase convergence speed and improve generalization capability of the system during training.

The encoder CBHG neural network <NUM> is configured to receive the transformed embeddings from the encoder pre-net neural network <NUM> and process the transformed embeddings to generate encoded representations of the sequence of characters. The encoder CBHG neural network <NUM> includes a CBHG neural network <NUM> (<FIG>), which is described in more detail below with respect to <FIG>. The use of the encoder CBHG neural network <NUM> as described herein may reduce overfitting. In addition, the encoder CBHB neural network <NUM> may result in fewer mispronunciations when compared to, for instance, a multi-layer RNN encoder.

The attention-based decoder recurrent neural network <NUM> (herein referred to as "The decoder neural network <NUM>'') is configured to receive a sequence of decoder inputs. For each decoder input in the sequence, the decoder neural network <NUM> is configured to process the decoder input and the encoded representations generated by the encoder CBHG neural network <NUM> to generate multiple frames of the spectrogram of the sequence of characters That is, instead of generating (predicting) one frame at each decoder step, the decoder neural network <NUM> generates r frames of the spectrogram, with r being an integer greater than one to many cases, there is no overlap between sets of r frames.

In particular, at decoder step t, at least the last frame of the r frames generated at decoder step t-<NUM> is fed as input to the decoder neural network <NUM> at decoder step t+<NUM>. In some implementations, all of the r frames generated at the decoder step t-<NUM> are fed as input to the decoder neural network <NUM> at the decoder step t+<NUM>. The decoder input for the first decoder step can be an all-zero frame (i.e. a <GO> frame). Attention over the encoded representations is applied to all decoder steps, e.g., using a conventional attention mechanism. The decoder neural network <NUM> may use a fully connected neural network layer with a linear activation to simultaneously predict r frames at a given decoder step. For example, to predict <NUM> frames, each frame being an <NUM>-D (<NUM>-Dimension) vector, the decoder neural network <NUM> uses the fully connected neural network layer with the linear activation to predict a <NUM>-D vector and to reshape the <NUM>-D vector to obtain the <NUM> frames.

By generating r frames at each time step, the decoder neural network <NUM> divides the total number of decoder steps by r, thus reducing model size, training time, and inference time. Additionally, this technique substantially increases convergence speed, i e. , because it results in a much faster (and more stable) alignment between frames and encoded representations as learned by the attention mechanism. This is because neighboring speech frames are correlated and each character usually corresponds to multiple frames. Emitting multiple frames at a time step allows the decoder neural network <NUM> to leverage this quality to quickly learn how to, i.e., be trained to, efficiently attend to the encoded representations during training.

The decoder neural network <NUM> may include one or more gated recurrent unit neural network layers. To speed up convergence, the decoder neural network <NUM> may include one or more vertical residual connections. In some implementations, the spectrogram is a compressed spectrogram such as a mel-scale spectrogram. Using a compressed spectrogram instead of, for instance, a raw spectrogram may reduce redundancy, thereby reducing the computation required during training and inference.

The post-processing neural network <NUM> is configured to receive the compressed spectrogram and process the compressed spectrogram to generate a waveform synthesizer input. To process the compressed spectrogram, the post-processing neural network <NUM> includes the CBHG neural network <NUM> (<FIG>). In particular, the CBHG neural network <NUM> includes a <NUM>-D convolutional subnetwork, followed by a highway network, and followed by a bidirectional recurrent neural network. The CBHG neural network <NUM> may include one or more residual connections. The <NUM>-D convolutional subnetwork may include a bank of <NUM>-D convolutional filters followed by a max pooling along time layer with stride one In some cases, the bidirectional recurrent neural network is a gated recurrent unit neural network. The CBHG neural network <NUM> is described in more detail below with reference to <FIG>.

In some implementations, the post-processing neural network <NUM> and the sequence-to-sequence recurrent neural network <NUM> are trained jointly. That is, during training, the system <NUM> (or an external system) trains the post-processing neural network <NUM> and the seq2seq network <NUM> on the same training dataset using the same neural network training technique, e.g., a gradient descent-based training technique. More specifically, the system <NUM> (or an external system) can backpropagate an estimate of a gradient of a loss function to jointly adjust the current values of all network parameters of the post-processing neural network <NUM> and the seq2seq network <NUM>. Unlike conventional systems that have components that need to be separately trained or pre-trained and thus each component's errors can compound, systems that have the post-processing neural network <NUM> and seq2seq network <NUM> that are jointly trained are more robust (e.g., they have smaller errors and can be trained from scratch). These advantages enable the training of the end-to-end text-to-speech model <NUM> on a very large amount of rich, expressive yet often noisy data found in the real world.

The waveform synthesizer <NUM> is configured to receive the waveform synthesizer input, and process the waveform synthesizer input to generate a waveform of the verbal utterance of the input sequence of characters in the particular natural language. In some implementations, the waveform synthesizer is a Griffin-Lim synthesizer. In some other implementations, the waveform synthesizer is a vocoder. In some other implementations, the waveform synthesizer is a trainable spectrogram to waveform inverter. After the waveform synthesizer <NUM> generates the waveform, the subsystem <NUM> can generate speech <NUM> using the waveform and provide the generated speech <NUM> for playback, e.g., on a user device, or provide the generated waveform to another system to allow the other system to generate and play back the speech In some examples, a WaveNet neural vocoder replaces the waveform synthesizer <NUM>. A WaveNet neural vocoder may provide different audio fidelity of synthesized speech in comparison to synthesized speech produced by the waveform synthesizer <NUM>.

<FIG> shows an example CBHG neural network <NUM>. The CBHG neural network <NUM> can be the CBHG neural network included in the encoder CBHG neural network <NUM> or the CBHG neural network included in the post-processing neural network <NUM> of <FIG>. The CBHG neural network <NUM> includes a <NUM>-D convolutional subnetwork <NUM>, followed by a highway network <NUM>, and followed by a bidirectional recurrent neural network <NUM>. The CBHG neural network <NUM> may include one or more residual connections, e g. , the residual connection <NUM>.

The <NUM>-D convolutional subnetwork <NUM> may include a bank of <NUM>-D convolutional filters <NUM> followed by a max pooling along time layer with a stride of one <NUM>. The bank of <NUM>-D convolutional filters <NUM> may include K sets of <NUM>-D convolutional filters, in which the k-th set includes Ck filters each having a convolution width of k. The <NUM>-D convolutional subnetwork <NUM> is configured to receive an input sequence <NUM>, for example, transformed embeddings of a sequence of characters that are generated by an encoder pre-net neural network <NUM> (<FIG>) The subnetwork <NUM> processes the input sequence <NUM> using the bank of <NUM>-D convolutional filters <NUM> to generate convolution outputs of the input sequence <NUM>. The subnetwork <NUM> then stacks the convolution outputs together and processes the stacked convolution outputs using the max pooling along time layer with stride one <NUM> to generate max-pooled outputs. The subnetwork <NUM> then processes the max-pooled outputs using one or more fixed-width <NUM>-D convolutional filters to generate subnetwork outputs of the subnetwork <NUM>.

After the <NUM>-D convolutional subnetwork <NUM> generates the subnetwork outputs, the residual connection <NUM> is configured to combine the subnetwork outputs with the original input sequence <NUM> to generate convolution outputs. The highway network <NUM> and the bidirectional recurrent neural network <NUM> are then configured to process the convolution outputs to generate encoded representations of the sequence of characters. In particular, the highway network <NUM> is configured to process the convolution outputs to generate high-level feature representations of the sequence of characters. In some implementations, the highway network includes one or more fully-connected neural network layers.

The bidirectional recurrent neural network <NUM> is configured to process the high-level feature representations to generate sequential feature representations of the sequence of characters A sequential feature representation represents a local structure of the sequence of characters around a particular character. A sequential feature representation may include a sequence of feature vectors. In some implementations, the bidirectional recurrent neural network is a gated recurrent unit neural network.

During training, one or more of the convolutional filters of the <NUM>-D convolutional subnetwork <NUM> can be trained using batch normalization method, which is described in detail in<NPL>. In some implementations, one or more convolutional filters in the CBHG neural network <NUM> are non-causal convolutional filters, i.e., convolutional filters that, at a given time step T, can convolve with surrounding inputs in both directions (e.g.,. , T-<NUM>, T-<NUM> and T+<NUM>, T+<NUM>,. In contrast, a causal convolutional filter can only convolve with previous inputs (. T-<NUM>, T-<NUM>, etc.). In some other implementations, all convolutional filters in the CBHG neural network <NUM> are non-causal convolutional filters. The use of non-causal convolutional filters, batch normalization, residual connections, and max pooling along time layer with stride one improves the generalization capability of the CBHG neural network <NUM> on the input sequence and thus enables the text-to-speech conversion system to generate high-quality speech.

<FIG> is an example arrangement of operations for a method <NUM> of generating speech from a sequence of characters. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a text-to-speech conversion system (e.g., the text-to-speech conversion system <NUM> of <FIG>) or a subsystem of a text-to-speech conversion system (e.g., the subsystem <NUM> of <FIG>), appropriately programmed, can perform the process <NUM>.

At operation <NUM>, the method <NUM> includes the system receiving a sequence of characters in a particular natural language, and at operation <NUM>, the method <NUM> includes the system providing the sequence of characters as input to a sequence-to-sequence (seq2seq) recurrent neural network <NUM> to obtain as output a spectrogram of a verbal utterance of the sequence of characters in the particular natural language. In some implementations, the spectrogram is a compressed spectrogram, e.g., a mel-scale spectrogram. In particular, the seq2seq recurrent neural network <NUM> processes the sequence of characters to generate a respective encoded representation of each of the characters in the sequence using an encoder neural network <NUM> that includes an encoder pre-net neural network <NUM> and an encoder CBHG neural network <NUM>.

More specifically, each character in the sequence of characters can be represented as a one-hot vector and embedded into a continuous vector. The encoder pre-net neural network <NUM> receives a respective embedding of each character in the sequence and processes the respective embedding of each character in the sequence to generate a transformed embedding of the character. For example, the encoder pre-net neural network <NUM> can apply a set of non-linear transformations to each embedding to generate a transformed embedding. The encoder CBHG neural network <NUM> then receives the transformed embeddings from the encoder pre-net neural network <NUM> and processes the transformed embeddings to generate the encoded representations of the sequence of characters.

To generate a spectrogram of a verbal utterance of the sequence of characters, the seq2seq recurrent neural network <NUM> processes the encoded representations using an attention-based decoder recurrent neural network <NUM> In particular, the attention-based decoder recurrent neural network <NUM> receives a sequence of decoder inputs. The first decoder input in the sequence is a predetermined initial frame For each decoder input in the sequence, the attention-based decoder recurrent neural network <NUM> processes the decoder input and the encoded representations to generate r frames of the spectrogram, in which r is an integer greater than one. One or more of the generated r frames can be used as the next decoder input in the sequence In other words, each other decoder input in the sequence is one or more of the r frames generated by processing a decoder input that precedes the decoder input in the sequence.

The output of the attention-based decoder recurrent neural network thus includes multiple sets of frames that form the spectrogram, in which each set includes r frames. In many cases, there is no overlap between sets of r frames. By generating r frames at a time, the total number of decoder steps performed by the attention-based decoder recurrent neural network is reduced by a factor of r, thus reducing training and inference time. This technique also helps to increase convergence speed and learning rate of the attention-based decoder recurrent neural network and the system in general.

At operation <NUM>, the method <NUM> includes generating speech using the spectrogram of the verbal utterance of the sequence of characters in the particular natural language. In some implementations, when the spectrogram is a compressed spectrogram, the system can generate a waveform from the compressed spectrogram and generate speech using the waveform.

At operation <NUM>, the method <NUM> includes providing the generated speech for playback. For example, the method <NUM> may provide the generated speech for playback by transmitting the generated speech from the system to a user device (e.g.. audio speaker) over a network for playback.

Implementations herein are directed toward introducing a number extensions to latent variable models based on the TTS conversions system <NUM> for expressive speech synthesis (e.g., control and transfer of prosody and style) that allow the models to make more effective use of latent/variational embeddings. The use of latent variable models enables probabilistically modeling and/or directly controlling remaining variability in synthesized speech. Sources of variability include prosodic characteristics of intonation, stress, rhythm, and style, as well as speaker and channel characteristics. The prosodic characteristics of a spoken utterance convey linguistic, semantic, and emotional meaning beyond what is present in a lexical representation (e.g., a transcript of the spoken utterance). Providing an ability to transfer these characteristics from one utterance to another enables users to control how synthesized speech sounds by using their own voice (e.g., "say it like this"), rather than having to manipulate complicated acoustic or linguistic parameters by hand. In some implementations, methods include varying a capacity target of a reconstruction loss term in a variational reference encoder to allow transferring a prosody of reference speech at a fine-grained level (e.g., prioritizing precision) to a similar piece of text (i.e., text having a similar number of syllables as the reference speech), or at a coarse-grained level (e. g, prioritizing generalization) to an arbitrary piece of text (i.e., text of any length and syllabic content).

<FIG> shows an example prosody-style transfer model <NUM> for transferring style and/or prosody of a reference speaker to a target speaker and/or controlling style and/or prosody of synthesized speech <NUM> produced from input text. The transfer model <NUM> allows users to synthesize natural speech with a particular speaking style or prosody In a variety of different but natural ways. As will become apparent, the transfer model <NUM> enables transferring of prosodic/style characteristics from one utterance to another by using a reference utterance (e.g., "say it like this''). Additionally, the transfer model <NUM> permits randomly sampling prosodic characteristics from a distribution over likely prosodic realizations of an utterance in order to provide natural variety across longer sections of speech.

The transfer model <NUM> includes a variational autoencoder (VAE) network for unsupervised learning of latent representations (i.e., variational embeddings (z)) of speaking styles. Learning variational embeddings through the use of the VAE provides favorable properties of disentangling, scaling, and combination for simplifying style control compared to heuristic-based systems. The transfer system <NUM> includes a reference encoder <NUM> and an end-to-end TTS model <NUM> configured to receive a reference audio signal (X, Xref) <NUM> as input and determine a variational embedding (z) <NUM> for the reference audio signal <NUM> as output. The TTS model <NUM> receives the variational embedding <NUM> output from the reference encoder <NUM> for converting input text <NUM> into synthesized speech <NUM> (e.g., output audio signal <NUM> (X, Xtgt)) having a style/prosody specified by the variational embedding <NUM>. That is to say, the variational embedding <NUM> enables the synthesized speech <NUM> produced by the TTS model <NUM> to sound like the reference audio signal <NUM> input to the reference encoder <NUM>.

The TTS model <NUM> includes an encoder <NUM>, an attention module <NUM>, a decoder <NUM>, and a synthesizer <NUM>. In some implementations, the TTS model <NUM> includes the TTS model <NUM> of <FIG>. For instance, the encoder <NUM>, the attention module <NUM>. and the decoder <NUM> may collectively correspond to the seq2seq recurrent neural network <NUM> and the synthesizer <NUM> may include the waveform synthesizer <NUM> or a WaveNet neural vocoder. However, the choice of synthesizer <NUM> has no impact on resulting prosody and/or style of the synthesized speech, and in practice, only impacts audio fidelity of the synthesized speech <NUM>. The attention module <NUM> may include Gaussian Mixture Model (GMM) attention to improve generalization to long utterances. Accordingly, the encoder <NUM> of the TTS model <NUM> may use a CBHG neural network <NUM> (<FIG>) to encode the input text <NUM><NUM> and modify the GMM attention of the attention model <NUM> to compute the parameters using a softplus function instead of exp.

The input text <NUM> may include phoneme inputs produced by a text normalization front-end and lexicon since prosody is being addressed, rather than the model's ability to learn pronunciation from graphemes. The decoder <NUM> may include the decoder recurrent neural network <NUM> (<FIG>) and use a reduction factor equal to two (<NUM>), thereby producing two spectrogram frames (e.g.. output audio signal <NUM>) per timestep. In some examples, two layers of <NUM>-cell long short term memory (LSTM) using zoneout with probability equal to <NUM> may replace GRU cells of the decoder <NUM>. In other implementations, the TTS model <NUM> includes the speech synthesis system disclosed in <CIT>, the contents of which are incorporated by reference in their entirety.

The variational embedding <NUM> corresponds to a latent state of a target speaker, such as affect and intent, which contributes to the prosody, emotion, and/or speaking style of the target speaker. As used herein, the variational embedding <NUM> includes both style information and prosody information. In some examples, the variational embedding <NUM> includes a vector of numbers having a capacity represented by a number of bits in the variational embedding <NUM>. Generally, increasing a capacity of the variational embedding <NUM> increases precision of the synthesized speech <NUM> such that the target speaker represented by the synthesized speech <NUM> closely resembles the reference audio signal <NUM>. As such, high capacity variational embeddings <NUM> prioritize precision and are better suited for inter-speaker transfer scenarios. However, one-drawback with achieving these increases in precision is that the input text <NUM> (i e. , target text) converted by the TTS model <NUM> must closely resemble reference text corresponding to the reference audio signal <NUM>, whereby the reference text is input to the TTS model <NUM> during training of the transfer system <NUM>. As used herein, input text <NUM> closely resembles reference text when the input text <NUM> includes a similar number of vowels as the reference text. On the other hand, decreasing a capacity of the variational embedding <NUM> increases generality of the variational embedding <NUM> such that the variational embedding <NUM> works well for producing synthesized speech <NUM> from different input texts <NUM> (i.e., inter-text transfer). Accordingly, low capacity variational embeddings <NUM> prioritize generality and are better suited for text-agnostic style transfer.

In some implementations, the reference encoder <NUM> also receives conditional dependencies <NUM> to balance the tradeoff between precision and generality such that both style and prosody are controllable and transferable. By contrast to heuristic-based encoders <NUM> that are only capable of computing prosody/style embeddings from reference audio, the reference encoder <NUM> in the VAE permits sampling of variational embeddings <NUM> previously produced by the encoder <NUM> so that a greater variety of prosodic and style information is capable of representing the input text <NUM> input to the TTS model <NUM> for conversion to synthesized speech <NUM>. As such, a reference audio signal <NUM> is not needed for computing the variational embedding <NUM> because the variational embedding <NUM> can be sampled. Of course, the reference encoder <NUM> can compute variational embeddings <NUM> from a reference audio signal <NUM> (e g. , "say it like this"). As described in greater detail below, hierarchical fractions containing different style/prosody information can be decomposed from the variational embedding <NUM>, enabling later sampling of these hierarchical fractions of the variational embedding <NUM> in order to control a trade-off between reference similarity and sample variability in inter-speech transfer scenarios. The conditional dependencies <NUM> include reference/target text yT characterizing the reference audio signal <NUM> and/or a reference/target speaker ys indicating an identity of the speaker that uttered the reference audio signal <NUM>. The reference/target speaker ys allows an identity of a target speaker (or reference speaker) to be preserved during inter-speaker transfer. For instance, when a reference speaker has a different pitch range than the target speaker, the synthesized speech <NUM> may still sound like the target speaker since a suitable variational embedding <NUM> can be sampled by the reference encoder <NUM> when the reference speaker ys is provided.

The input text <NUM> and the reference text tr of the conditional dependences <NUM> may include character sequences while the reference and output audio signals <NUM>, <NUM> correspond to acoustic features that may include mel-frequency spectrograms. The reference encoder <NUM> includes a neural network.

Referring to <FIG>, in some implementations, the reference encoder <NUM> modifies a deterministic reference encoder <NUM> disclosed by "<NPL>, the contents of which are incorporated by reference in their entirety In some implementations, the reference encoder <NUM> is configured to receive a reference audio signal <NUM> and generate/predict a fixed-length prosody embedding PE <NUM> (also referred to as 'prosodic embedding') from the reference audio signal <NUM>. The prosody embedding PE <NUM> may capture characteristics of the reference audio signal <NUM> independent of phonetic information and idiosyncratic speaker traits such as, stress, intonation, and timing. The prosody embedding PE <NUM> may used as an input for preforming prosody transfer in which synthesized speech is generated for a completely different speaker than the reference speaker, but exhibiting the prosody of the reference speaker.

In the example shown, the reference audio signal <NUM> may be represented as spectrogram slices having a length LR and dimension DR. The spectrogram slices associated with the reference audio signal <NUM> may be indicative of a Mel-warped spectrum. In the example shown, the reference encoder <NUM> includes a six-layer convolutional layer network <NUM> with each layer including <NUM>×<NUM> filters with 2x2 stride, SAME padding, and ReLU activation. Batch normalization is applied to every layer and the number of filters in each layer doubles at half the rate of downsampling: <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. A recurrent neural network <NUM> with a single <NUM>-width Gated Recurrent Unit (GRU-RNN) layer receives the output <NUM> from the last convolutional layer and outputs a <NUM>-dimentional output <NUM> applied to a fully connected layer <NUM> followed by an activation function <NUM> that outputs the predicted prosody embedding PE <NUM>. The recurrent neural network <NUM> may include other types of bidirectional recurrent neural networks.

The choice of activation function <NUM> (e.g., a softmax or tanh) in reference encoder <NUM> may constrain the information contained in the style embedding SE <NUM> and help facilitate learning by controlling the magnitude of the predicted prosody embedding PE <NUM>. Moreover, the choice of the length LR and the dimension DR of the reference audio signal <NUM> input to the reference encoder <NUM> impacts different aspects of prosody learned by the encoder <NUM>. For instance, a pitch track representation may not permit modeling of prominence in some language since the encoder does not contain energy information, while a Mel Frequency Cepstral Coefficient (MFCC) representation may, at least to some degree depending on the number of coefficients trained, prevent the encoder <NUM> from modeling intonation.

While the prosody embedding PE <NUM> output from the reference encoder <NUM> can be used in a multitude of different TTS architectures for producing synthesized speech, a seed signal (e.g., reference audio signal <NUM>) is required for producing the prosody embedding PE <NUM> at inference time. For instance, the seed signal could be a "Say it like this" reference audio signal <NUM> Alternatively, to convey synthesized speech with an intended prosody/style, some TTS architectures can be adapted to use a manual style embedding selection at inference time instead of using the reference encoder <NUM> to output a prosody embedding PE <NUM> front a seed signal.

The reference encoder <NUM> of <FIG> corresponds to a heuristic-based model (non-variational) for predicting fixed-length prosodic embeddings, and includes a six-layer convolutional layer network with each layer including 3x3 filters with 2x2 stride. SAME padding, and ReLU activation. Batch normalization is applied to every layer and the number of filters in each layer doubles at half the rate of downsampling: <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. A recurrent neural network with single <NUM>-width Gated Recurrent U nit (GRU) layer receives the output from the last convolutional layer and outputs a <NUM>-dimentional output applied to a fully connected layer followed by a softmax. tanh activation function. Referring back to <FIG>, the reference encoder <NUM> corresponds to a 'variational posterior' that replaces the tanh bottleneck layer of the deterministic reference encoder <NUM> of <FIG> with multilayer perception (MLP) <NUM> having linear activation to predict the parameters (i e. , mean µ and standard deviation σ of latent variables) of the reference encoder <NUM>. When used, the conditional dependencies <NUM> (reference text yT and/or reference speaker ys) may feed into the MLP <NUM> as well The variability embedding (z) <NUM> may be derived using reparameterization based on the mean µ and standard deviation σ of the latent variables output from the MLP <NUM> of the reference encoder <NUM>. The encoder states input to the text encoder <NUM> of the TTS model <NUM> include the variability embedding (z) <NUM> and the sequence of characters in the input text <NUM>, such that an encoded sequence <NUM> output from the encoder <NUM> includes a summation of the input text <NUM> and the variability embedding <NUM> which is consumed by the attention module <NUM>. The text encoder <NUM> may also receive a target speaker yS identifying a specific speaker for how the synthesized speech <NUM> should sound. In some examples, the attention module <NUM> is configured to convert the encoded sequence <NUM> to a fixed-length context vector <NUM> for each output step of the decoder <NUM> to produce the output audio signal <NUM>.

During training, a transcript of the reference audio signal <NUM> matches the sequence of characters of the input text sequence <NUM> input to the encoder <NUM> of the TTS model <NUM> so that the output audio signal <NUM> output from the decoder <NUM> will match the reference audio signal <NUM>. During inference, the transfer system <NUM> may perform inter-text transfer by including different input text sequences <NUM> to the encoder <NUM> that do not match the transcript of the reference audio signal <NUM>. Similarly, the transfer system <NUM> may perform inter-speaker transfer by specifying speakers for the synthesized speech that are different than the speaker uttering the reference audio signal.

While <FIG> shows the deterministic reference encoder <NUM> for computing fixed-length prosody embeddings best suited for same or similar-text prosody transfer, i.e., input text for conversion includes a similar number of syllables as a transcript of the reference audio signal In this heuristic approach, prosody transfer precision is controlled by the dimensionality of the prosody embedding and choice of non-linearity (tanh vs. softmax). Referring to <FIG>. another heuristic-based model <NUM> modifies the architecture of the deterministic reference encoder <NUM> by implementing a style token layer <NUM> disclosed by ''<NPL>, the contents of which are incorporated by reference in their entirety. Here, the style token layer <NUM> receives the prosody embedding PE <NUM> output from the deterministic reference encoder <NUM> and uses the prosody embedding PE <NUM> as a query vector to an attention module <NUM><NUM> configured to learn a similarity measure between the prosody embedding PE <NUM> and each token <NUM> in a bank of randomly initialized embeddings <NUM>, 614a-n (also referred to as global style tokens (GSTs) or token embeddings) The set of token embeddings (also referred to as "style tokens") <NUM> is shared across all training sequences. Thus, the attention module <NUM> outputs a set of combination weights <NUM>, 616a-n that represent the contribution of each style token <NUM> to the encoded prosody embedding PE <NUM>. The weighted sum of token embeddings corresponds to a style embedding SE <NUM> that is input to the text encoder (e. g , encoder <NUM>, <NUM>) of the TTS model <NUM> for conditioning at every time step.

During inference, the text encoder of the TTS model <NUM> may be directly conditioned on a specific token/style embedding <NUM> (e.g., Token B) to allow for style control and manipulation without a reference audio signal <NUM>. On the other hand, when a reference audio signal <NUM> for a target speaker is used whose transcript does not match input text <NUM> to be synthesized into synthesized speech <NUM>, the style token layer <NUM> is conditioned upon the reference audio signal <NUM> represented by the prosodic embedding PE <NUM> output from the reference encoder <NUM>. The prosodic embedding PE <NUM>, style embeddings <NUM>, and tokens <NUM> affect information capacity of the respective embeddings and allow these heuristic-based models <NUM>. <NUM> to target a specific trade-off between transfer precision (how closely the output resembles the references) and generality (how well an embedding works with arbitrary text).

In some implementations, embedding capacity of heuristic-based approaches, such as the deterministic reference encoder <NUM> of <FIG> and the heuristic-based model <NUM> of FIGS. 6A and 6B implementing the deterministic reference encoder <NUM> and the style token layer <NUM>. is estimated by measuring a test-time reconstruction loss of the deterministic encoder <NUM>. Specifically, these heuristic-based approaches may start with a teacher-forced reconstruction loss represented by Equation <NUM> (expressed below) to train their sequence-to-sequence model and then augment their model with the deterministic reference encoder <NUM> (denoted ge (x)) represented by Equation <NUM> (expressed below). Equations <NUM> and <NUM> are expressed as follows:
<MAT>
<MAT>
where x is an audio spectrogram <NUM>. yT is the input text <NUM>, ys is the target speaker (if training a multi-speaker model), fθ(·) is a deterministic function that maps the inputs to spectrogram predictions, and K is a normalization constant. Teacher-forcing implies that fθ(·) is dependent upon x<t when predicting spectrogram xt. Because an ℓ<NUM> reconstruction loss is typically used, the likelihood is equivalent to a Laplace random vector with fixed diagonal covariance and means provided by fθ(·) (though in practice, the deterministic output of fθ(·) serves as the output). Transfer is accomplished by pairing the embedding PE <NUM> computed by the reference encoder <NUM> with different text or speakers during synthesis.

Referring to <FIG> and <FIG>, plots 700a, 700b each show reconstruction loss ℓ<NUM>(y-axis) varies with embedding dimensionality (x-axis) of the prosody embedding PE <NUM> and choice of non-linearity (tanh vs. softmax) for heuristic-based (e g. , non-variational) prosody transfer using the deterministic reference encoder <NUM>. Here, the bottleneck of softmax non-linearity prosody embedding PE <NUM> is more severe than the tanh non-linearity prosody embedding. Similarly, plots 700a, 700b each show reconstruction loss ℓ<NUM> (y-axis) varies with embedding dimensionality (x-axis) of the style embedding for heuristic-based style transfer The more restrictive bottleneck of the style embedding (eg. , Style Token) compared to prosody embeddings shows how embedding capacity affects the precision/generality trade-off.

Referring back to <FIG> and <FIG>, plot 700a further depicts reconstruction loss varying with embedding dimensionality for variational embeddings <NUM> with different KL weights, β, while plot 700b further depicts reconstruction loss varying with embedding dimensionality for variational embeddings <NUM> with different capacity limits, C, whereby the capacity limits are controlled via the KL term directly using Equation <NUM>. Plot 700a shows that the reference decoder <NUM> using KL weight β = <NUM> produces a variational embedding <NUM> that matches the loss of the tanh non-linearity prosody embedding from the heuristic-based prosody transfer model and using KL weight β = <NUM> produces a variational embedding <NUM> similar to the style embedding Further, using KI. weight β = <NUM> produces a variational embedding <NUM> with a loss very similar to a baseline of the TTS model <NUM> since capacity of the variational embedding <NUM> is effectively squashed to zero.

By specifying target capacities of the variational embeddings, one can estimate capacity of deterministic embeddings (prosody or style embeddings) computed by the deterministic reference encoder <NUM> via comparing/matching reconstruction loss measurements versus embedding dimensionality. Thus, with the ability to now estimate capacity of deterministic embeddings (also referred to as reference embeddings) output from the deterministic reference encoder based on the comparison of reconstruction loss vs embedding dimensionality relationship to variational embeddings with computable/controllable capacity, capacity of these deterministic embeddings can also be controlled by adjusting a dimension of the reference embeddings computed by the deterministic reference encoder. Thus, deterministic embeddings can now provide a tradeoff between precision/fidelity and generality/transferability using these techniques to estimate and control capacity.

Since the KL term corresponds to an upper bound on embedding capacity (Equation <NUM>), a specific limit on the embedding capacity may be targeted by constraining the KL term using Equation <NUM>. For instance, and with continued reference to <FIG> and <FIG>, plot 700b shows that reconstruction loss flattens out when the embedding z reaches a certain dimensionality. This allows the reference encoder <NUM> to control a target representational capacity in the variational embedding <NUM> as long as the reference encoder has a sufficient structural capacity (at least C). In some examples, the variational embeddings <NUM> include fixed-length <NUM>-dimensional embeddings to accommodate a range of targeted capacities for balancing the tradeoff between precision and generality. A number of bits in the variational embedding <NUM> may represent the capacity.

Thus during training of the transfer system <NUM>, the capacity of the variational embedding <NUM> output from the reference encoder <NUM> may be controlled by using the upper bound (e.g., variational bound) corresponding to a KL term to control a quantity of information within the variational embedding <NUM>. In this manner, desirable tradeoffs between precision and generality may be obtained by controlling the capacity of the variational embedding <NUM> alone and without requiring any altering of the architecture of the reference encoder to target specific precision/generality points.

Referring back to the transfer system <NUM> of <FIG>, implementations herein are further directed toward estimating and quantifying the capacity of the variational embedding <NUM> output from the variational reference encoder <NUM> ('variational posterior') using an upper bound (i.e., a variational bound of the reference encoder <NUM>) on representative mutual information The reference encoder <NUM> may augment the reconstruction loss for the deterministic reference encoder <NUM> in Equation <NUM> with a KL term to align the variational reference encoder <NUM>, q(z|x), with a prior, p(z), as represented by Equation <NUM> (expressed below) Equation <NUM> (expressed below) represents the overall loss of the reference encoder being equivalent to a negative lower bound (negative ELBO) of the representative mutual information corresponding to x, yT, ys. Equations <NUM> and <NUM> are expressed as follows:
<MAT>
<MAT>.

In some examples, adjusting the KL term in Equation <NUM> controls capacity of the variational embedding <NUM> of the reference encoder <NUM><NUM>, whereby the KL term provides an upper bound on the mutual information between the data, x, and the latent embedding, z~ q(z|x). This relationship between the KL term and the capacity of the variational embedding <NUM>, z, is expressed as follows. <MAT>
<MAT>
<MAT>
Where pD(x) is data distribution, R (e.g. "rate") is the KL term in Equation <NUM>, RAVG is the KI. term averaged over the data distribution, Iq(X;Z) the representational mutual information that corresponds to the capacity of z, and q(z) (e.g., aggregated posterior) is q(z|x) marginalized over the data distribution. The bound in Equation <NUM> follows from Equation <NUM> and the non-negativity of the KL divergence, wherein Equation <NUM> shows that the slack on the bound is the KL divergence between the aggregated posterior, q(z), and the prior, p(z) In some examples, lowering R (e.g., the KL term) provides for better sampling of variational embeddings <NUM>, z, from the model via the prior since samples of z that the decoder <NUM> sees during training will be substantially similar to samples from the prior.

In some implementations, a specific capacity of the variational embedding <NUM> is targeted by applying a Lagrange multiplier-based, dual-optimizer approach to the KL term rather than the reconstruction term Applying the Lagrange multiplier-based, dual optimizer to the KL term may be expressed as follows:
<MAT>.

where θ denotes the model parameters, λ is the Lagrange multiplier, and C denotes a capacity limit By constraining λ to be non-negative by passing an unconstrained parameter though a softplus non-linearity, the capacity constraint C corresponds to a limit/threshold rather than a target. As result, the optimization prevents attempts to increase the KL term by moving q(z) away from q(z). Advantageously, this dual optimizer approach is much less tedious than tuning the KL weight by hand, while at the same time, leads to more stable optimization compared to directly penalizing the ℓ<NUM> reconstruction loss deviation from the target KL.

Referring back to <FIG> and <FIG>, plot 700a further depicts reconstruction loss varying with embedding dimensionality for variational embeddings <NUM> with different KL weights, β, while plot <NUM>(X)b further depicts reconstruction loss varying with embedding dimensionality for variational embeddings <NUM> with different capacity limits, C, whereby the capacity limits are controlled via the KL term directly using Equation <NUM> Plot 700a shows that the reference decoder <NUM> using KL weight β = <NUM> produces a variational embedding <NUM> that matches the loss of the tanh non-linearity prosody embedding from the heuristic-based prosody transfer model and using KL weight β = <NUM> produces a variational embedding <NUM> similar to the style embedding. Further, using KL weight β = <NUM> produces a variational embedding <NUM> with a loss very similar to a baseline of the TTS model <NUM> since capacity of the variational embedding <NUM> is effectively squashed to zero.

Referring back to <FIG>, in some implementations, conditional dependencies <NUM> are input to the reference encoder <NUM> to balance the tradeoff between precision and generalization. The conditional dependencies <NUM> include reference text, yT, and/or reference speaker, ys. By applying the reference speaker, an identity of a target speaker may be preserved in synthesized speech <NUM> so that the target speaker does not mimic a reference speaker having a different pitch range than the target speaker. During training, the reference text yT and the target text associated with the input text sequence <NUM> input to the encoder <NUM> are the same Similarly, the reference speaker may also be input to the <NUM> to the encoder during training. However during inference, the reference text and target text may be different and/or the reference speaker and the target speaker may be different. For instance, the conditional dependencies <NUM> and reference audio signal <NUM> may be input to the reference encoder <NUM> to produce the variational embedding <NUM> having both prosody and style information. The input text sequence <NUM> input to the encoder <NUM> of the TTS model <NUM> may include target text yT that is different than the reference text yT to change what is said by the synthesized speech. Additionally or alternatively, a different target speaker may be input to the text encoder <NUM> of the TTS model <NUM> to change who spoke. Here, the variational embedding <NUM> is paired with the target text and/or the target speaker. As a result, this variational embedding <NUM> could be sampled at a later time when there is no reference audio signal but the conditional dependencies <NUM> match the target speaker and target text paired with the variational embedding <NUM>.

Referring to <FIG>, a conditional generative model corresponding to the decoder <NUM> of the TTS model that produces an output/target audio signal X from a variational embedding z, target text yT, and a target speaker ys. The conditional generative model is represented by the form p(x|z, yT, yS) p(z). <FIG> shows a variational posterior missing the conditional dependencies present in <FIG> shows the variational posterior (e.g., reference encoder <NUM>) including conditional posteriors to match the form of <FIG>. Here, the matching variational posterior of <FIG>. Speaker information is represented as learned speaker-wise embedding vectors, while the text information is summarized into a vector by passing the output of the text encoder <NUM> through a unidirectional RNN. A simple diagonal Gaussian may be used for the approximate posterior, q(z|x; yT ; yS) and a standard normal distribution for the prior, p(z) These distributions are chosen for simplicity and efficiency, but more powerful distributions such as Gaussian mixtures and normalizing flows could be used.

Generally, while a variational embedding <NUM> fully specifies variation of the prosodic and style information, the synthesized speech <NUM> based on the variational embedding <NUM> will always sound the same with the same input text sequence <NUM> even though there are an infinite number of ways the input text sequence <NUM> can be expressed for a given style. In some implementations, decomposing the variational embedding z <NUM> into hierarchical fractions zs, zp allows one to specify how a a joint capacity, Iq(X: [Zs, Zp]), is divided between the hierarchical fractions zs, zp. In some examples, the hierarchical fraction zs represents style information associated with the variational embedding z and the hierarchical fraction zp represents prosodic information associated with the variational embedding z. However, the hierarchical fractions decomposed may be used to denote other types of information without departing from the scope of the present disclosure.

Equation <NUM> shows the KL term providing the upper bound on capacity Iq(X;Z). The following equations may be used to derive capacity of the prosodic fraction zp as follows:
<MAT>
<MAT>.

The following equations may be used to derive capacity of the style fraction zs as follows. <MAT>
<MAT>
where Rs makes up a portion of the overall joint KL term. If Rp = R-Rs, the following bounds include:
<MAT>.

In order to specify how joint capacity is distributed between the fractions (e.g., latent variables), Equation <NUM> is extended to have two Lagrange multipliers and capacity targets as follows. <MAT>
where capacity target Cs limits information capacity of zs and Cp limits how much capacity zp has in excess of zs, wherein the total capacity of zp is capped at Cs + Cp. In some examples, a reference hierarchical fraction zs is inferred by the reference encoder <NUM> from a reference audio signal <NUM> and used to sample multiple realizations Untuitively, the higher Cs is, the more the output will resemble the reference, and the higher Cp is, the more variation from sample to sample for the same reference hierarchical fraction zs.

With reference to <FIG>, <FIG>, in some implementations, when only conditional dependencies <NUM> of reference text yT and reference speaker ys are input to the reference encoder <NUM> without reference audio signal <NUM>, the zs is sampled from a train model, and zp is sampled and sent to decoder of <FIG> along with the conditional dependencies to compute the target output audio X. The sampled zp is paired as a prior with the conditional dependencies <NUM>. Here, using the same conditional dependencies <NUM> of reference text yT and reference speaker yS, the variational reference decoder (variational posterior) of <FIG> will output this zp and use the zp to compute zs. As such, the decoder of <FIG> may now regenerate the target audio signal X using the computed zs, the reference text yT, and the reference speaker yS as inputs. Advantageously, the hierarchal traction zs represents variation in the reference encoder specified by zs so that different capacities of zp can be sampled to result in synthesized speech of a given style sounding different. Thus, the zp and zp correspond to thresholds for balancing the tradeoff between precision and generalization. Accordingly, by using conditional dependencies <NUM>, prior variational embeddings <NUM> learned by the reference encoder <NUM> may be sampled to synthesize speech without reference audio signals <NUM> and/or to sample prosodic characteristics from a distribution over likely prosodic realizations of an utterance with a specified style in order to provide natural variety across longer sections of speech.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of estimating a capacity of a reference embedding. At operation <NUM>, the method <NUM> includes receiving, at a deterministic reference encoder <NUM>, a reference audio signal <NUM>, and at operation <NUM>, the method <NUM> includes determining a reference embedding <NUM>, <NUM> corresponding to the reference audio signal <NUM>. Here, the reference embedding <NUM>, <NUM> has a corresponding embedding dimensionality.

At operation <NUM>, the method <NUM> includes measuring a reconstruction loss as a function of the corresponding embedding dimensionality of the reference embedding <NUM>,<NUM>. At operation <NUM>, the method <NUM> includes obtaining a variational embedding <NUM> from a variational posterior. The variational embedding <NUM> has a corresponding dimensionality and a specified capacity, whereby the specified capacity is based on an adjustable variational bound of the variational posterior.

At operation <NUM>, the method <NUM> includes measuring reconstruction loss as a function of the corresponding embedding dimensionality of the variational embedding. At operation <NUM>, the method <NUM> includes estimating a capacity of the reference embedding <NUM>, <NUM> by comparing the measured reconstruction loss for the reference embedding <NUM>. <NUM> relative to the measured reconstruction loss for the variational embedding <NUM> having the specified capacity.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of targeting a specific capacity of a variational embedding <NUM>. At operation <NUM>, the method <NUM> includes adjusting a KL term of a reference encoder <NUM> to provide an upper bound on a capacity of variational embedding <NUM> computed by the reference encoder <NUM>. Adjusting the KL term may include increasing the KL term to increase the capacity of the variational embedding <NUM> or decreasing the KL term to decrease the capacity of the variational embedding. Increasing the capacity of the variational embedding increases precision of synthesized speech <NUM>, while decreasing the capacity of the variational embedding <NUM> increases generality of the variational embedding for converting different input texts into synthesized speech <NUM>. In some implementations, adjusting the KL term includes applying a Lagrange multiplier to the KL term and specifying a capacity limit. Adjusting the KL term may include tuning a weight of the KL term.

At operation <NUM>, the method <NUM> includes receiving, at the reference encoder <NUM>, a reference audio signal <NUM>. At operation <NUM>, the method <NUM> includes determining, by the reference encoder <NUM>, a variational embedding <NUM> associated with the reference audio signal <NUM>. The variational embedding <NUM> having a capacity bounded by the upper bound provided by the adjusted KL term. At operation <NUM>, the method <NUM> includes providing the variational embedding <NUM> associated with the reference audio signal <NUM> to a text-to-speech synthesis model <NUM>. Here, the text-to-speech synthesis model <NUM> is configured to convert input text <NUM> into synthesized speech <NUM> based on the variational embedding <NUM> associated with the reference audio signal <NUM>. A number of bits represents the capacity of the variational embedding <NUM>.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of sampling hierarchical fractions associated with variational embeddings <NUM> to vary how synthesized speech sounds for a given style. The method <NUM> may permit the controlling of a specified fraction of variation represented in the variational embedding <NUM> to allow a rest of variation to be sampled from a text-to-speech model <NUM>. At operation <NUM>, the method <NUM> includes obtaining a variational embedding <NUM> output from a reference encoder <NUM>, and at operation <NUM>, the method <NUM> includes decomposing the variational embedding into hierarchical fractions and generating synthesized speech <NUM> based on the variational embedding <NUM>, target text, and a target speaker.

At operation <NUM>, the method <NUM> includes pairing the variational embedding <NUM> with the target text and the target speaker. At operation <NUM>, the method <NUM> includes receiving the target text and the target speaker at a reference encoder <NUM> without a reference audio signal and computing a first hierarchical fraction decomposed from the variational embedding paired with the target text and target speaker, the first hierarchical fraction providing a given style. At operation <NUM>, the method <NUM> includes sampling a second hierarchical fraction associated with the variational embedding <NUM> using the first hierarchical fraction Here, sampling the second hierarchical fraction varies how synthesized speech <NUM> sounds for the same given style.

The non-transitory memory may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device The non-transitory memory may be volatile and/or non-volatile addressable semiconductor memory.

<FIG> is schematic view of an example computing device <NUM> that may be used to implement the systems and methods described in this document The computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers.

The computing device <NUM> includes a processor <NUM>, memory <NUM>, a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM> Each of the components <NUM>, <NUM>. <NUM>, <NUM>, <NUM>. and <NUM>, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g.. as a server bank, a group of blade servers, or a multi-processor system).

In various different implementations, the storage device <NUM> may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations In additional implementations, a computer program product is tangibly embodied in an information carrier.

The low-speed expansion port <NUM>, which may include various communication ports (e.g., USB, Bluetooth. Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

For example, it may be implemented as a standard server 1300a or multiple times in a group of such servers 1300a, as a laptop computer 1300b, or as part of a rack server system 1300c.

However, a computer need not have such devices Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks. g, internal hard disks or removable disks, magneto optical disks; and CD ROM and DVD-ROM disks.

Other kinds of devices can be used to provide interaction with a user as well, for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user, for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

Claim 1:
A computer-implemented method for targeting a specified capacity of a variational embedding, the method comprising:
adjusting a KL term of a reference encoder to provide an upper bound on variational embedding capacity, the KL term configured to align the reference encoder with a prior embedding;
receiving, at the reference encoder, a reference audio signal;
determining, by the reference encoder, a variational embedding associated with the reference audio signal, the variational embedding having a capacity bounded by the upper bound provided by the adjusted KL term; and
providing the variational embedding associated with the reference audio signal to a text-to-speech (TTS) synthesis model, the TTS synthesis model configured to convert input text into synthesized speech based on the variational embedding associated with the reference audio signal.