Patent Description:
Speech synthesis systems use text-to-speech (TTS) models to generate speech from textual input. The generated/synthesized speech should accurately convey the message (intelligibility) while sounding like human speech (naturalness) with an intended prosody (expressiveness). While traditional concatenative and parametric synthesis models were capable of providing intelligible speech, recent advances in neural modeling of speech have significantly improved the naturalness and fidelity of synthesized speech. Yet even with these advances, often times the accuracy of these neural network models depends on the corpus of training examples that are available to teach the neural network model how to synthesize speech. As such, when a limited amount of training examples exist, neural network models lack the speech synthesis accuracy that users of speech synthesis systems expect or even demand. This may be especially true as speech synthesis systems (e.g., personal assistants) become more integrated in daily human-computer interaction.

<NPL>) describes Probability Density Distillation, a method for training a parallel feed-forward network from a trained WaveNet.

One aspect of the disclosure provides a method of self-training WaveNet. The method includes receiving, at data processing hardware, a plurality of recorded speech samples and training, by the data processing hardware, a first autoregressive neural network using the plurality of recorded speech samples. The trained first autoregressive neural network is configured to output synthetic speech as an audible representations of a text input. The method further includes generating, by the data processing hardware, a plurality of synthetic speech samples using the trained first autoregressive neural network. The method additionally includes training, by the data processing hardware, a second autoregressive neural network using the plurality of synthetic speech samples from the trained first autoregressive neural network and distilling, by the data processing hardware, the trained second autoregressive neural network into a feedforward neural network, wherein the feedforward neural network is configured to output synthetic speech without knowledge of one or more prior synthetic speech outputs.

Yet another aspect of the disclosure provides a self-training WaveNet system. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving a plurality of recorded speech samples and training a first autoregressive neural network using the plurality of recorded speech samples. The trained first autoregressive neural network is configured to output synthetic speech as an audible representations of a text input. The operations further include generating a plurality of synthetic speech samples using the trained first autoregressive neural network. The operations additionally includes training a second autoregressive neural network using the plurality of synthetic speech samples from the trained first autoregressive neural network and distilling the trained second autoregressive neural network into a feedforward neural network, wherein the feedforward neural network is configured to output synthetic speech without knowledge of one or more prior synthetic speech outputs.

Implementations of any of the disclosures may include one or more of the following optional features. In some implementations, the second autoregressive neural network includes a different autoregressive neural network than the first autoregressive neural network or the same autoregressive neural network as the first autoregressive neural network. When the first and second autoregressive neural networks are the same, training the second autoregressive neural network using the plurality of synthetic speech samples includes re-training the first autoregressive neural network using the plurality of synthetic speech samples. In some examples, the plurality of recorded speech samples include a respective number of recorded speech samples and the plurality of synthetic speech samples include a respective number of synthetic speech samples, the respective number of recorded speech samples less than the respective number of synthetic speech samples. In these examples, the respective number of synthetic speech samples may be at least one multiple greater than the respective number of recorded speech samples.

In some configurations, distilling the trained second autoregressive neural network into a feedforward neural network includes training the feedforward neural network based on a probability distribution from the trained second autoregressive neural network. In these configurations, training the feedforward neural network includes optimizing a loss function based on a Kullback-Leibler (KL) divergence between the feedforward neural network and the second autoregressive neural network. The loss function may include a weighted sum of the KL divergence between the feedforward neural network and the second autoregressive neural network, a mean squared error, a phoneme classification error, and a contrastive loss.

In some implementations, the feedforward neural network is configured to output synthetic speech without knowledge of one or more prior synthetic speech outputs. Each of the second autoregressive neural network and the feedforward network may include a plurality of dilated residual blocks where each dilated residual block includes layers of dilated convolutions. The feedforward network may include a plurality of inverse autoregressive flows (IAF). When the feedforward network includes a plurality of inverse autoregressive flows (IAF), each IAF flow of the plurality of IAF flows include one or more dialed residual blocks where each dilated residual block includes layers of dilated convolutions.

Deep neural networks have increasingly been used to advance an ability of a computing device to understand natural speech. Yet in addition to understanding natural speech, people often interact with computing devices with the expectation that the computing device (e.g., a speech-enabled device) generates synthesized speech as a response. For instance, a user of speech-enabled device submits a query to the speech-enabled device or a request that the speech-enabled device generates speech based on some form of text. In order to output a response to the user, the speech-enabled device employs a speech synthesis system or text-to-speech (TTS) system. Over time, speech synthesis has shifted from concatenative or statistical parametric synthesis to synthesis performed by deep neural network models. During this shift, speech synthesis or TTS systems have evolved to produce high-fidelity audio with near human parity.

A model that has proven to be a popular choice to generate seemingly realistic speech is WaveNet. WaveNet originally referred to a deep neural network that generates raw audio waveforms. When first developed, WaveNet was a model that was fully probabilistic and autoregressive where a predictive distribution for an audio sample was conditioned on all prior audio sample distributions (i.e., used ancestral sampling). As an autoregressive network, WaveNet used dilated convolutions to model the probabilistic distribution for a speech sample. For instance, WaveNet's convolutional layers had various dilation factors to allow a receptive field to grow exponentially with depth in order to cover thousands of time steps. With convolutional layers, WaveNet is able to process its input in parallel; enabling its architecture to be trained much more quickly when compared to a recurrent neural network based model. Yet although WaveNet proved capable of modeling thousands of raw audio samples, as an autoregressive network, WaveNet proved to be too slow during inference for real-time speech synthesis. Unlike training that is able to be performed in parallel, during inference or generation of the waveform, a fully autoregressive WaveNet generates a synthesized output in a sequential fashion. For instance, WaveNet was only capable of generating speech at about <NUM> time steps per second. While this slower-than-real-time inference speed may be acceptable for offline synthesized speech generation, a fully autoregressive WaveNet proves too slow for real-time applications.

To remedy the slow inference speed, a parallel variation of WaveNet (referred to as Parallel WaveNet) emerged to produce audio at a rate faster than real-time speech while maintaining the high-fidelity and realistic speech manner of the original autoregressive WaveNet. To be able to generate synthesized audio during inference at real-time speech rates, Parallel WaveNet distilled an autoregressive network of the original autoregressive WaveNet into a parallel feed-forward neural network described in van den Oord, Parallel WaveNet: Fast High-Fidelity Speech Synthesis, available at https://arxiv. org/pdf/<NUM><NUM>. Here, the autoregressive network is referred to as a "teacher" network (or autoregressive teacher) because the feedforward network is taught by the autoregressive network; thus, the feedforward network is also referred to as a "student" network or feedforward student network. In other words, Parallel WaveNet uses an already trained autoregressive neural network as a teacher network to train the student network. In this respect, Parallel WaveNet takes advantage of the parallel training speed of the autoregressive WaveNet, but generates a feedforward network that does not rely on ancestral sampling (e.g., like the trained autoregressive teacher network). As a feedforward network, the student network is thereafter able to implicitly infer outputs of prior time steps rather than requiring actual knowledge of these outputs. With this teacher-student architecture, Parallel WaveNet is capable of generating samples at about <NUM>,<NUM> time steps per second; addressing the speed deficiencies of a fully autoregressive WaveNet.

Unfortunately, Parallel WaveNet is not without its shortcomings. One such shortcoming is that Parallel WaveNet requires a large amount of recorded speech data to produce high quality student networks. For instance, a single-speaker version of WaveNet has been shown to require about twenty-four hours of recorded speech to result in a high fidelity student network. Moreover, other neural vocoders, such as single-speaker Tacotron, have greater fidelity when trained on more utterances (e.g., twenty-five thousand utterances) than less utterances (e.g., fifteen thousand or even eight thousand utterances). Yet there are inevitable circumstances where a large amount of recorded speech data is not available. When the teacher network trains a student network with a small amounts of recorded speech data, the synthesized output may contain artifacts, such as static noise, which become more prominent when fewer training samples of recorded speech area available.

Since speech synthesis models do not always have the luxury of a large amount of recorded speech data, a version of WaveNet that generates high-fidelity audio from a low-data regime would increase the robustness and/or universal effectiveness of WaveNet. Here, in order to further evolve WaveNet to be effective with a low-data regime, a self-training model leverages the high-fidelity audio produced by an autoregressive WaveNet and the faster-than-real-time synthesis capability of Parallel WaveNet. In other words, in low-data regimes, a Parallel WaveNet may be trained on speech synthesized by an autoregressive WaveNet teacher to form a self-training model (i.e., a self-training WaveNet). By generating high-fidelity synthetic speech data from an autoregressive teacher network to train a student network, the self-training WaveNet is able to train the student network without compromising fidelity when low amounts of recorded speech are available. Here, self-training refers to the technique of using an already trained system to generate outputs on unseen input examples and using these generated outputs as targets for subsequent training/re-training. Using a self-training approach, high fidelity synthetic speech samples produced by a trained autoregressive teacher network train (or distill) the feedforward student network. This approach takes advantage of the fact that the autoregressive WaveNet (e.g., the autoregressive teacher network) produces high quality synthetic examples; allowing a self-training technique to not further degrade the student network in response to the synthetic examples.

Referring to <FIG>, in some implementations, a speech environment <NUM> includes a user <NUM> communicating a spoken utterance <NUM> to a speech-enabled device <NUM> (also referred to as a device <NUM> or a user device <NUM>). The user <NUM> (i.e., speaker of the utterance <NUM>) may speak the utterance <NUM> as a query or a command to solicit a response from the device <NUM>. The device <NUM> is configured to capture sounds from one or more users <NUM> within the speech environment <NUM>. Here, the audio sounds may refer to a spoken utterance <NUM> by the user <NUM> that functions as an audible query, a command for the device <NUM>, or an audible communication captured by the device <NUM>. Speech-enabled systems of the device <NUM> or associated with the device <NUM> may field the query for the command by answering the query and/or causing the command to be performed.

Here, the device <NUM> captures an audio signal <NUM> (also referred to as audio data) of the spoken utterance <NUM> by the user <NUM>. The device <NUM> may correspond to any computing device associated with the user <NUM> and capable of receiving audio signals <NUM>. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, and internet of things (IoT) devices, smart speakers, etc. The device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations. In some examples, the device <NUM> includes one or more applications (i.e., software applications) where each application may utilize one or more speech processing systems <NUM>, <NUM>, <NUM> associated with device <NUM> to perform various functions within the application. For instance, the device <NUM> includes an assistant application configured to communicate synthesized playback audio <NUM> to the user <NUM> to assist the user <NUM> with various tasks.

The device <NUM> further includes an audio subsystem with an audio capturing device (e.g., a microphone) <NUM> for capturing and converting spoken utterances <NUM> within the speech environment <NUM> into electrical signals and a speech output device (e.g., a speaker) <NUM> for communicating an audible audio signal (e.g., a synthesized playback signal <NUM> from the device <NUM>). While the device <NUM> implements a single audio capturing device <NUM> in the example shown, the device <NUM> may implement an array of audio capturing devices <NUM> without departing from the scope of the present disclosure, whereby one or more audio capturing devices <NUM> in the array may not physically reside on the device <NUM>, but be in communication with the audio subsystem (e.g., peripherals of the device <NUM>). For example, the device <NUM> may correspond to a vehicle infotainment system that leverages an array of microphones positioned throughout the vehicle.

Furthermore, the device <NUM> is configured to communicate via a network <NUM> with a remote system <NUM>. The remote system <NUM> may include remote resources <NUM>, such as remote data processing hardware <NUM> (e.g., remote servers or CPUs) and/or remote memory hardware <NUM> (e.g., remote databases or other storage hardware). The device <NUM> may utilize the remote resources <NUM> to perform various functionality related to speech processing and/or synthesized playback communication. For instance, the device <NUM> is configured to perform speech recognition using a speech recognition system <NUM> and/or conversion of text to speech using a TTS system <NUM> (e.g., using the self-training model <NUM>). These systems <NUM>, <NUM>, <NUM> may reside on the device <NUM> (referred to as on-device systems) or reside remotely (e.g., reside on the remote system <NUM>), but in communication with the device <NUM>. In some examples, some of these systems <NUM>, <NUM>, <NUM> reside locally or on-device while others reside remotely. In other words, any of these systems <NUM>, <NUM><NUM> may be local or remote in any combination. For instance, when a system <NUM>, <NUM>, <NUM> is rather large in size or processing requirements, the system <NUM>, <NUM>, <NUM> may reside in the remote system <NUM>. Yet when the device <NUM> may support the size or the processing requirements of one or more systems <NUM>, <NUM>, <NUM>, the one or more systems <NUM>, <NUM>, <NUM> may reside on the device <NUM> using the data processing hardware <NUM> and/or the memory hardware <NUM>. Optionally, the one or more of the systems <NUM>, <NUM>, <NUM> may reside on both locally/on-device and remotely. For instance, one or more of the systems <NUM>, <NUM>, <NUM> may default to execute on the remote system <NUM> when a connection to the network <NUM> between the device <NUM> and remote system <NUM> is available, but when the connection is lost or the network <NUM> is unavailable, the systems <NUM>, <NUM>, <NUM> instead execute locally on the device <NUM>.

A speech recognition system <NUM> receives an audio signal <NUM> as an input and transcribes that audio signal into a transcription <NUM> as an output. Generally speaking, by converting the audio signal <NUM> into a transcription <NUM>, the speech recognition system <NUM> allows the device <NUM> to recognize when a spoken utterance <NUM> from the user <NUM> corresponds to a query, a command, or some other form of audio communication. The transcription <NUM> refers to a sequence of text that the device <NUM> may then use to generate a response to the query or the command. For instance, if the user <NUM> asks the device <NUM> the question of "what will the weather be like today," the device <NUM> passes the audio signal corresponding to the question "what will the weather be like today" to the speech recognition system <NUM>. The speech recognized system <NUM> converts the audio signal into a transcript that includes the text of "what will the weather be like today?" The device <NUM> may then determine a response to the query using the text or portions of the text. For instance, in order to determine the weather for the current day (i.e., today), the device <NUM> passes the text (e.g., "what will the weather be like today?") or identifying portions of the text (e.g., "weather" and "today") to a search engine. The search engine may then return one or more search results that the device <NUM> interprets to generate a response for the user <NUM>.

In some implementations, the device <NUM> or a system associated with the device <NUM> identifies text <NUM> that the device <NUM> will communicate to the user <NUM> as a response to a query of the spoken utterance <NUM>. The device <NUM> may then use the TTS system <NUM> to convert the text <NUM> into corresponding synthesized playback audio <NUM> for the device <NUM> to communicate to the user <NUM> (e.g., audibly communicate to the user <NUM>) as the response to the query of the spoken utterance <NUM>. In other words, the TTS system <NUM> receives, as input, text <NUM> and converts the text <NUM> to an output of synthesized playback audio <NUM> where the synthesized playback audio <NUM> is an audio signal defining an audible rendition of the text <NUM>. Here, the TTS system <NUM> (or other speech synthesis system) includes a self-training model <NUM> (e.g., the self-training model of <FIG>) that utilizes a deep neural network (e.g., the self-training WaveNet) to generate the synthesized playback audio <NUM>. Once generated, the TTS system <NUM> communicates the synthesized playback audio <NUM> to the device <NUM> to allow the device <NUM> to output the synthesized playback audio <NUM>. For instance, the device <NUM> outputs the synthesized playback audio <NUM> of "today is sunny" at a speaker <NUM> of the device <NUM>.

Referring to <FIG>, the TTS system <NUM> includes the self-training model <NUM> as a neural vocoder in order to generate the synthesized playback audio <NUM>. As a neural vocoder, the self-training model <NUM> may be conditioned on features that encode linguistic and/or prosodic information. Here, the linguistic conditioning may include phoneme, syllable, word, phrase, and/or utterance level features. In some examples, these features are derived through text normalization (e.g., static rule-based text normalization) in conjunction with rule-based feature computations. The prosody conditioning may be provided by an autoencoder such as a hierarchical variational autoencoder. The autoencoder may also use the linguistic conditioning to generate, as an output, a fundamental frequency per frame.

In some implementations, such as <FIG>, the self-training model <NUM> includes a first teacher network <NUM>, a second or synthesized teacher network <NUM>, and a synthesized student network <NUM>. Each network <NUM>, <NUM>, <NUM> may have dilated residual blocks as its constituent unit(s). In some examples, a dilated residual block includes one or more layers of convolutions. For example, the dilated residual blocks of the networks <NUM>, <NUM>, <NUM> correspond to ten layers of convolutions (e.g., increasing by a factor of two for every layer). In some configurations, the teacher networks <NUM>, <NUM> include three dilated residual blocks while the student network <NUM> includes four inverse autoregressive flows (IAF) where each IAF may have some number of residual blocks. For instance, the first, second, and third IAFs each include one dilated block while the fourth IAF includes three dilated residual blocks.

As depicted by <FIG>, this self-training model <NUM> is in contrast to a conventional Parallel WaveNet that does not train with synthesized speech. Generally, the conventional method of training Parallel WaveNet is a two-step procedure. In the first step, an autoregressive teacher network is trained to model the probability distribution of a sample based on previous samples of recorded speech. Here, since all the recorded speech samples are available, training may occur in parallel with the technique of teacher forcing. During the second step to train a conventional Parallel WaveNet, the training process distills the density distribution of the autoregressive teacher network into a feedforward student network. Distillation generally refers to a process of training a neural network using a pre-trained network. Using distillation, neurons of the pre-trained network that are less critical to a desired output (e.g., similar to deadweight) may be reduced to form a more streamlined neural network (i.e., the distilled neural network). Distillation may enable the distilled neural network to be more accurate and/or a more compact size when compared to the pre-trained network. In other words, when the pre-trained network was formed, the pre-trained network may have formed neurons that ultimately resulted in having less impact on the desired output by the time the training of the pre-trained network was complete; therefore, the pre-trained network includes neurons that may be removed or modified to reduce any detrimental impact from these neurons or to remove unnecessary neurons.

In some examples, the self-training model <NUM> includes a first teacher <NUM> that functions as an autoregressive neural network. Much like the conventional Parallel WaveNet, the first teacher <NUM> trains using recorded speech samples <NUM>, 242a-n from a volume <NUM> (e.g., shown as a database or other type of audio data depository) of recorded speech samples <NUM>. Here though, the self-training model <NUM> may be utilized when the number of recorded speech samples <NUM> is relatively low (i.e., a low-data regime). For instance, a low-data regime refers where the volume <NUM> of recorded speech samples <NUM> is less than twenty-five thousand samples (i.e., a healthy amount of recorded samples <NUM> for Parallel WaveNet), but greater than five thousand samples (i.e., an unhealthy amount of recorded samples <NUM> for Parallel WaveNet). In some examples, the low-data regime is between five thousand to fifteen thousand recorded speech samples <NUM>. Even though the low-data regime has less recorded speech samples <NUM>, the first teacher <NUM> trains with the recorded speech samples <NUM> to form a trained first teacher <NUM>.

With the trained first teacher <NUM>, the self-training model <NUM> has the trained first teacher <NUM> generate a volume <NUM> of synthetic speech samples <NUM>, 252a-n as an output <NUM>. In some implementations, the generation of the synthetic speech samples <NUM> is a one-time processing task that can be performed offline with little to no limits on the amount of synthetic speech samples <NUM> that the trained teacher <NUM> generates. In some configurations, the self-training model <NUM> uses the trained autoregressive teacher from Parallel WaveNet and not the feedforward student (e.g., shown as a dotted box in <FIG>) to generate the synthetic speech samples <NUM> because the autoregressive teacher network <NUM> generates synthetic speech samples <NUM> having higher fidelity than the feedforward student network. Therefore, although it is possible to also generate synthetic speech samples with the feedforward student network of Parallel WaveNet, it may compromise the fidelity of the self-training model <NUM>.

In some examples, with a TTS system <NUM>, the self-training model <NUM> is configured to generate the synthetic speech samples <NUM> from readily available text samples (e.g., text <NUM> from the TTS system <NUM>). In some configurations, the text samples may be unlabeled text samples. The teacher <NUM> may generate the synthetic speech samples <NUM> from a corpus of text samples with sufficient phonetic coverage for the language of the synthetic speech samples <NUM>. In some examples, the synthetic speech samples <NUM> undergo a pruning process to prevent the synthesized student <NUM> from learning using a noisy or a corrupt synthetic speech sample <NUM> (i.e., a synthetic speech sample <NUM> that would be detrimental to learning). For instance, to avoid an issue where the synthetic speech samples <NUM> include bias with respect to phoneme distribution, a script selection methodology may be applied to the dataset of generated synthetic speech samples <NUM>. Additionally or alternatively, during or after the process of generating the synthetic speech samples <NUM>, the training process may generate phoneme alignments for the synthetic speech samples <NUM>. By generating phoneme alignments for the synthetic speech samples <NUM>, a pruning process may reject synthetic speech samples <NUM> that produce a phoneme alignment score below a particular threshold (i.e., minimum allowable alignment score). In other words, phoneme alignment scores may indicate which synthetic speech samples <NUM> may need pruning from the training set of synthetic speech samples <NUM> that will be used to train the synthesized teacher <NUM> (or, in some instances, re-train the teacher <NUM>).

In some configurations, once the trained teacher <NUM> generates the synthetic speech samples <NUM>, the model training process uses the synthetic speech samples <NUM> to train the synthesized teacher <NUM>. For instance, the model training process uses a training dataset that includes synthetic speech samples <NUM> that survived the pruning process. Here, the training process for the synthesized teacher <NUM> is identical to that of the first teacher <NUM> except that the training process uses the synthetic speech samples <NUM> instead of the recorded speech samples <NUM>. With this training process, the synthesized teacher <NUM> may be trained with a large set of samples (i.e., the synthetic speech samples <NUM>) when compared to the small number of recorded speech samples <NUM>. For instance, the teacher <NUM> generates upwards of twenty-five to fifty thousand synthetic speech samples while the low-data regime includes a fraction of this amount (e.g., between five thousand to fifteen thousand recorded speech samples <NUM>); therefore, even though both the first teacher <NUM> and the synthesized teacher <NUM> are autoregressive neural networks, the synthesized teacher <NUM> is trained on an amount of speech data that is one or more multiples greater than the amount of speech data that trained the teacher <NUM>.

Optionally, in some examples, the training process uses the synthetic speech samples <NUM> to train or to distill the student <NUM> without training a synthesized teacher <NUM>. Although this approach is feasible, it may not be ideal. This is especially true since the trained teacher <NUM> was trained only on a relatively small amount of recorded speech samples <NUM> in order to generate the synthetic speech samples <NUM>. This means that immediately training the student <NUM> would likely not have the level of fidelity that the synthesized student <NUM> has when it is trained by a teacher (e.g., the synthesized teacher <NUM>) that has been trained with a larger data regime (e.g., the synthetic speech samples <NUM>). Additionally or alternatively, the first teacher <NUM> and the second teacher or synthesized teacher <NUM> may be the same neural network. In other words, the synthesized teacher <NUM> is simply the teacher <NUM> that has been re-trained using the synthetic speech samples <NUM> such that the neural network forming the teacher <NUM> has then been retrained on a larger corpus of speech samples.

When the synthesized teacher <NUM> has been trained, the trained synthesized teacher <NUM> may then be distilled into the synthesized student <NUM>. This means that the trained synthesized teacher <NUM> trains the synthesized student <NUM> according to probability distributions that correspond to synthetic speech. The training process by the trained synthesized teacher <NUM> distills the synthesized teacher <NUM> into a feedforward synthesized student <NUM>. As a feedforward neural network, the synthesized student <NUM>, much like a feedforward student of a Parallel WaveNet, is able to generate an output of synthesized speech (e.g., synthesized playback audio <NUM>) without requiring knowledge of one or more prior synthesized speech outputs during inference (e.g., like an autoregressive network). Although the training process is generally described with respect to a single speaker, the training process for the model <NUM> may be scaled for multiple speakers. For example, when the model <NUM> is for multiple speakers, the synthesized teacher <NUM> distills a synthesized student <NUM> for each speaker of the multiple speakers such that the model <NUM> for multiple speakers includes multiple synthesized students <NUM> (e.g., proportional to the number of multiple speakers).

In some implementations, the training process trains a component of the model <NUM> by optimizing a loss function. For instance, to train either teacher model <NUM>, <NUM>, the training process may use a loss function represented as a negative log-likelihood of a predicted mixture distribution. In some examples, the distillation process trains the student <NUM> using a loss function based on a Kullback-Leibler (KL) divergence. For example, the loss function is based on the KL divergence between the student <NUM> (e.g., a feedforward neural network) and the synthesized teacher <NUM> (e.g., an autoregressive neural network trained by synthetic speech samples <NUM>). In some configurations, the distillation process trains the student <NUM> with a loss function that is a weighted sum of several different loss functions (or errors), such as a KL divergence (e.g., between the student <NUM> and the synthesized teacher <NUM> distributions), a mean squared error (e.g., between predicted and target signal powers in a moving window), a phoneme classification error, and/or a contrastive loss that maximizes the difference between the KL divergence of probability distributions between the student <NUM> and the teacher <NUM> when obtained with correct conditionings and the KL divergence of probability distributions between the student <NUM> and the teacher <NUM> when obtained with incorrect conditionings.

<FIG> is a flowchart of an example arrangement of operations for a method self-training WaveNet. At operation <NUM>, the method <NUM> receives a plurality of recorded speech samples <NUM>. At operation <NUM>, the method <NUM> trains a first autoregressive neural network <NUM> using the plurality of recorded speech samples <NUM>. The trained first autoregressive neural network <NUM> is configured to output synthetic speech as an audible representations of a text input. At operation <NUM>, the method <NUM> generates a plurality of synthetic speech samples <NUM> using the trained first autoregressive neural network <NUM>. At operation <NUM>, the method <NUM> trains a second autoregressive neural network <NUM> using the plurality of synthetic speech samples <NUM> from the trained first autoregressive neural network <NUM>. At operation <NUM>, the method <NUM> distills the trained second autoregressive neural network <NUM> into a feedforward neural network <NUM>.

<FIG> is another flowchart of an example arrangement of operations for another method self-training WaveNet. At operation <NUM>, the method <NUM> receives a plurality of recorded speech samples <NUM>. At operation <NUM>, the method <NUM> trains an autoregressive neural network <NUM> using the plurality of recorded speech samples <NUM>. The trained autoregressive neural network <NUM> is configured to output synthetic speech as an audible representations of a text input. At operation <NUM>, the method <NUM> generates a plurality of synthetic speech samples <NUM> using the trained autoregressive neural network <NUM>. At operation <NUM>, the method <NUM> distills the trained autoregressive neural network <NUM> into a feedforward neural network <NUM>.

<FIG> is schematic view of an example computing device <NUM> that may be used to implement the systems (e.g., the speech recognition system <NUM>, the TTS system <NUM>, and/or the self-training model <NUM>) and methods (e.g.,, methods <NUM>, <NUM>) described in this document.

The computing device <NUM> includes a processor <NUM> (e.g., data processing hardware), memory <NUM> (e.g., memory hardware), a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>), a plurality of recorded speech samples (<NUM>);
training, by the data processing hardware (<NUM>), a first autoregressive neural network (<NUM>) using the plurality of recorded speech samples (<NUM>), the trained first autoregressive neural network (<NUM>) configured to output (<NUM>) synthetic speech as an audible representation of a text input (<NUM>);
generating, by the data processing hardware (<NUM>), a plurality of synthetic speech samples (<NUM>) using the trained first autoregressive neural network (<NUM>);
training, by the data processing hardware (<NUM>), a second autoregressive neural network (<NUM>) using the plurality of synthetic speech samples (<NUM>) from the trained first autoregressive neural network (<NUM>); and
distilling, by the data processing hardware (<NUM>), the trained second autoregressive neural network (<NUM>) into a feedforward neural network (<NUM>);
wherein the feedforward neural network (<NUM>) is configured to output (<NUM>) synthetic speech without knowledge of one or more prior synthetic speech outputs.